package packer
The public traits and classes in this package are used to provide
configuration for the Mothra packing tools. (mothra-packer,
mothra-invariantpacker, and mothra-repacker.)
All three tools may be configured using PartitionerConfigurator objects, which provide a sequence of Partitioners to determine which single partition directory within a Mothra repository should receive each record. This is the recommended method for configuring partitioning.
For more specialized needs, outside the standard Mothra repository structure, PackingLogic objects may produce any number of output PackableRecords for each input Record, allowing for arbitrary directory structures, record re-writing, record filtering, and other possibilities. We do not recommend using this configuration method unless you have very specific needs.
- Alphabetic
- By Inheritance
- packer
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- case class PackableRecord(record: Record, relativePackedPath: String, startTime: RecordStartTime, observationDomain: Int) extends Product with Serializable
A Record and the information required to generate the file name where the Record will be stored in a Mothra repository.
A Record and the information required to generate the file name where the Record will be stored in a Mothra repository.
- record
The record to write to the file.
- relativePackedPath
The relative path of the file to return; does not include time prefix or filename suffix, but does include a version number
- startTime
The date+time to use for time-based parts of file path
- observationDomain
The domain used for all IPFIX Messages in the file; this is needed in case a new file must created and opened.
- trait PackingLogic extends AnyRef
A method for Mothra packers to map Records to storage locations in a Mothra repository.
A method for Mothra packers to map Records to storage locations in a Mothra repository. It permits storing a Record zero, one, or multiple times, and it also allows the opportunity to modify the Record before it is stored.
- See also
PartitionerConfigurator instead. While some packing tools (
mothra-packerandmothra-repacker) still allow general PackingLogic definitions to be used, the more structured approach of splitting records into a partition structure is more generally useful and safe. To use a PackingLogic object for the packing logic configuration of one of the Mothra packer tools, use a file containing something like the following:import org.cert.netsa.io.ipfix.Record import org.cert.netsa.mothra.packer.{PackableRecord, PackingLogic} new PackingLogic { override def pack(rec: Record): Iterator[PackableRecord] = ... }
Those tools will then compile this and use the partition configuration for packing records.
- trait Partitioner extends AnyRef
A method for Mothra packers to determine into which single partition (if any) on one level of a partitioning hierarchy a Record should be stored.
A method for Mothra packers to determine into which single partition (if any) on one level of a partitioning hierarchy a Record should be stored. Partitioners are used to partition (split) records into various directories in the long-term data storage location.
A single Partitioner typically splits on one attribute of a Record, such as its source IP address, IP protocol, or destination port.
A sequence of multiple Partitioners is used by PartitionerConfigurator to partition on multiple attributes in order.
- See also
the companion object for numerous ways to define simple Partitioners on various data types. These allow you to quickly specify ranges or specific values for different partitions.
- trait PartitionerConfigurator extends AnyRef
A method for Mothra packers to map Records to storage locations in a Mothra repository by using multiple Partitioner definitions.
A method for Mothra packers to map Records to storage locations in a Mothra repository by using multiple Partitioner definitions. Each Partitioner is called in order to produce a single optional path element to be included in the final storage location. The combined list of path elements (excluding any which are None) is used to construct the final path.
To use a PartitionerConfigurators for the packing logic configuration of one of the Mothra packer tools, use a file containing something like the following:
import org.cert.netsa.mothra.packer.{Partitioner, PartitionerConfigurator} new PartitionerConfigurator { override def partitioners: Seq[Partitioner] = { ... } }
Those tools will then compile this and use the partition configuration for packing records.
- case class PartitionerPackLogic(partitioners: Seq[Partitioner]) extends PackingLogic with Product with Serializable
An implementation of the PackingLogic trait.
An implementation of the PackingLogic trait. The
pack()method stores Records into files based on a sequence of Partitioners that is passed to this class's constructor.- partitioners
How records are to be partitioned.
- case class ProtoSportPackLogic() extends PackingLogic with Product with Serializable
An implementation of the PackingLogic trait.
An implementation of the PackingLogic trait. The
pack()method stores Records into files based on the record's IP Protocol and source transport port.In this version, a directory is created for each unique protocol and source port. For a version that groups uncommon protocols and ports into larger files, see ProtoSportPartitionerPackLogic.
- See also
ProtoSportPartitionerPackLogic for a better way to pack Records
- class ProtoSportPartitionerPackLogic extends PartitionerConfigurator
A simple class whose
partitioners()method needs to be called to get a sequence of Partitioners.A simple class whose
partitioners()method needs to be called to get a sequence of Partitioners. Those results are used to configure a PartitionerPackLogic.The sequence of Partitioners returned by this class split first by protocol and then by source port.
See the comments in the VlanPartitionerPackLogic source file for instructions on how this file needs to be edited to support loading these partitioners at run-time.
- See also
ProtoSportPackLogic for an older version of this concept.
- case class ProtocolPackLogic() extends PackingLogic with Product with Serializable
An implementation of the PackingLogic trait.
An implementation of the PackingLogic trait. The
pack()method stores records into files based on the record's IP Protocol. - final case class RecordStartTime(date: LocalDate, time: LocalTime) extends Product with Serializable
The start time of a Record.
The start time of a Record. All returned values use UTC.
- date
The date (YEAR, MONTH, DAY) part of the start time.
- time
The time (HOUR, MINUTE, SECOND) part of the start time.
- class VlanPartitionerPackLogic extends PartitionerConfigurator
A simple class whose
partitioners()method needs to be called to get a sequence of Partitioners.A simple class whose
partitioners()method needs to be called to get a sequence of Partitioners. Those results are used to configure a PartitionerPackLogic.The sequence of Partitioners returned by this class split by vlanId.
- See also
ProtoSportPartitionerPackLogic for a different set of partitioners.
Value Members
- object PackableRecord extends Serializable
A PackableRecord factory.
- object Partitioner
An object to hold classes that extend the [Partitioner] trait.
- object RecordStartTime extends Serializable
A RecordStartTime factory.