Packages

package packer

The public traits and classes in this package are used to provide configuration for the Mothra packing tools. (mothra-packer, mothra-invariantpacker, and mothra-repacker.)

All three tools may be configured using PartitionerConfigurator objects, which provide a sequence of Partitioners to determine which single partition directory within a Mothra repository should receive each record. This is the recommended method for configuring partitioning.

For more specialized needs, outside the standard Mothra repository structure, PackingLogic objects may produce any number of output PackableRecords for each input Record, allowing for arbitrary directory structures, record re-writing, record filtering, and other possibilities. We do not recommend using this configuration method unless you have very specific needs.

Linear Supertypes
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. packer
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. case class PackableRecord(record: Record, relativePackedPath: String, startTime: RecordStartTime, observationDomain: Int) extends Product with Serializable

    A Record and the information required to generate the file name where the Record will be stored in a Mothra repository.

    A Record and the information required to generate the file name where the Record will be stored in a Mothra repository.

    record

    The record to write to the file.

    relativePackedPath

    The relative path of the file to return; does not include time prefix or filename suffix, but does include a version number

    startTime

    The date+time to use for time-based parts of file path

    observationDomain

    The domain used for all IPFIX Messages in the file; this is needed in case a new file must created and opened.

  2. trait PackingLogic extends AnyRef

    A method for Mothra packers to map Records to storage locations in a Mothra repository.

    A method for Mothra packers to map Records to storage locations in a Mothra repository. It permits storing a Record zero, one, or multiple times, and it also allows the opportunity to modify the Record before it is stored.

    See also

    PartitionerConfigurator instead. While some packing tools (mothra-packer and mothra-repacker) still allow general PackingLogic definitions to be used, the more structured approach of splitting records into a partition structure is more generally useful and safe. To use a PackingLogic object for the packing logic configuration of one of the Mothra packer tools, use a file containing something like the following:

        import org.cert.netsa.io.ipfix.Record
    import org.cert.netsa.mothra.packer.{PackableRecord, PackingLogic}
    new PackingLogic {
      override def pack(rec: Record): Iterator[PackableRecord] = ...
    }

    Those tools will then compile this and use the partition configuration for packing records.

  3. trait Partitioner extends AnyRef

    A method for Mothra packers to determine into which single partition (if any) on one level of a partitioning hierarchy a Record should be stored.

    A method for Mothra packers to determine into which single partition (if any) on one level of a partitioning hierarchy a Record should be stored. Partitioners are used to partition (split) records into various directories in the long-term data storage location.

    A single Partitioner typically splits on one attribute of a Record, such as its source IP address, IP protocol, or destination port.

    A sequence of multiple Partitioners is used by PartitionerConfigurator to partition on multiple attributes in order.

    See also

    the companion object for numerous ways to define simple Partitioners on various data types. These allow you to quickly specify ranges or specific values for different partitions.

  4. trait PartitionerConfigurator extends AnyRef

    A method for Mothra packers to map Records to storage locations in a Mothra repository by using multiple Partitioner definitions.

    A method for Mothra packers to map Records to storage locations in a Mothra repository by using multiple Partitioner definitions. Each Partitioner is called in order to produce a single optional path element to be included in the final storage location. The combined list of path elements (excluding any which are None) is used to construct the final path.

    To use a PartitionerConfigurators for the packing logic configuration of one of the Mothra packer tools, use a file containing something like the following:

    import org.cert.netsa.mothra.packer.{Partitioner, PartitionerConfigurator}
    new PartitionerConfigurator {
      override def partitioners: Seq[Partitioner] = { ... }
    }

    Those tools will then compile this and use the partition configuration for packing records.

  5. case class PartitionerPackLogic(partitioners: Seq[Partitioner]) extends PackingLogic with Product with Serializable

    An implementation of the PackingLogic trait.

    An implementation of the PackingLogic trait. The pack() method stores Records into files based on a sequence of Partitioners that is passed to this class's constructor.

    partitioners

    How records are to be partitioned.

  6. case class ProtoSportPackLogic() extends PackingLogic with Product with Serializable

    An implementation of the PackingLogic trait.

    An implementation of the PackingLogic trait. The pack() method stores Records into files based on the record's IP Protocol and source transport port.

    In this version, a directory is created for each unique protocol and source port. For a version that groups uncommon protocols and ports into larger files, see ProtoSportPartitionerPackLogic.

    See also

    ProtoSportPartitionerPackLogic for a better way to pack Records

  7. class ProtoSportPartitionerPackLogic extends PartitionerConfigurator

    A simple class whose partitioners() method needs to be called to get a sequence of Partitioners.

    A simple class whose partitioners() method needs to be called to get a sequence of Partitioners. Those results are used to configure a PartitionerPackLogic.

    The sequence of Partitioners returned by this class split first by protocol and then by source port.

    See the comments in the VlanPartitionerPackLogic source file for instructions on how this file needs to be edited to support loading these partitioners at run-time.

    See also

    ProtoSportPackLogic for an older version of this concept.

  8. case class ProtocolPackLogic() extends PackingLogic with Product with Serializable

    An implementation of the PackingLogic trait.

    An implementation of the PackingLogic trait. The pack() method stores records into files based on the record's IP Protocol.

  9. final case class RecordStartTime(date: LocalDate, time: LocalTime) extends Product with Serializable

    The start time of a Record.

    The start time of a Record. All returned values use UTC.

    date

    The date (YEAR, MONTH, DAY) part of the start time.

    time

    The time (HOUR, MINUTE, SECOND) part of the start time.

  10. class VlanPartitionerPackLogic extends PartitionerConfigurator

    A simple class whose partitioners() method needs to be called to get a sequence of Partitioners.

    A simple class whose partitioners() method needs to be called to get a sequence of Partitioners. Those results are used to configure a PartitionerPackLogic.

    The sequence of Partitioners returned by this class split by vlanId.

    See also

    ProtoSportPartitionerPackLogic for a different set of partitioners.

Value Members

  1. object PackableRecord extends Serializable

    A PackableRecord factory.

  2. object Partitioner

    An object to hold classes that extend the [Partitioner] trait.

  3. object RecordStartTime extends Serializable

    A RecordStartTime factory.

Inherited from AnyRef

Inherited from Any

Ungrouped