Object to implement the Reacker application.

Typical Usage in a Spark environment:

spark-submit --class org.cert.netsa.mothra.packer.tools.RepackerMain mothra-tools.jar <partition-conf> <dest-dir> <work-dir> <s1> [<s2> <s3> ...]

where:

partition-conf: Partitioning configuration file as Hadoop URI

dest-dir: Root destination directory as Hadoop URI

work-dir: Working directory on the local disk (not file://)

s1..sn: Source directories as Hadoop URIs

Makes a single recursive scan of the source directories <s1>,<s2>,... for IPFIX files. Splits the IPFIX records in the source files into output file(s) in a time-based directory structure based on the partitioning rules in the partitioning configuration file <partition-conf>. The output files are initially created in the working directory <work-dir>, and, once ALL input files have been read, are moved to the destination directory and the initial source files removed. The dest-dir may be a source directory.

Repacker runs as a batch process; not as a daemon.

Example/Intended uses for the Repacker include:

(1)Changing how the records are packed---for example packing by the silkAppLabel instead of the protocolIdentifier.

(2)Combining multiple files for an hour into a single file for that hour, merging hourly files into a file that covers a longer duration, or spliting a longer duration file into smaller files.

(3)Changing the compression algorithm used on the IPFIX files.

Currently the repacker does NOT support modifying the records, it only moves the records into different files.

Repacker uses multiple threads. By default, each source directory specified on the command line gets a dedicated thread to scanning that directory and its subdirectories recursively for IPFIX files, and another thread decidated to reading those files and repacking them. The repacker does not support having multiple threads scan a directory, but it does allow multiple threads to process a single directory's files.

The <work-dir> must NOT be a source directory or a subdirectory of a source directory. To repack the files in an existing working directory, use a different working directory. The repacker ignores any files in the <work-dir> that exist when the repacker is started, and it ignores files placed there by other programs.

The property values that are used by the repacker are:

mothra.repacker.compression -- the compression algorithm used for the new IPFIX files. Values typically supported by Hadoop include bzip2, gzip, lz4, lzo, lzop, snappy, and default. The empty string indicates no compression.

mothra.repacker.hoursPerFile -- The number of hours covered by each file in the repository. The valid range is 1 (a file for each hour) to 24 (one file per day). The default is 1.

mothra.repacker.maxScanJobs -- the maximum number of threads dedicated to scanning the source directories. The default (and maximum) value is the number of source directories.

mothra.repacker.readersPerScanner -- the number of reader/repacker threads to create for each source directory. The default is 1.

mothra.repacker.maxThreads -- the maximum number of worker (scanner and repacker) threads to create. The default value is computed using the formula: (maxScanJobs * (1 + readersPerScanner)).

mothra.repacker.maximumSize -- the (approximate) maximum file size to create. When specified, a work-file that exceeds this size is closed and moved into the repository. NOTES: (1)This value uses the uncompressed file size, and does not consider any compression that may occur when the file is moved from the workDir to the tgtDir. In addition, a file's size tends to grow in large steps because of buffering by the Java stream code. (2)Specifying a maximumSize may temporarially cause duplicate records to appear in the repository because of some records in the original files and some in the new file. Once Repacker finishes scanning all files, the original files are removed and only the newly packed files are left. This issue of temporary having duplicate records in the repository will be resolved in a future release.

mothra.repacker.archiveDirectory -- the root directory into which working files are moved after the repacker has finished running, as a Hadoop URI. If not specified, the working files are deleted.

mothra.repacker.fileCacheSize -- The maximum size of the open file cache. This is the maximum number of open files maintained by the file cache for writing to files in the work directory. The repacker does not limit the number of files in the work directory; this only limits the number of open files. Once the cache reaches this number of open files and the packer needs to (re-)open a file, the packer closes the least-recently-used file. This value does not include the file handles required when reading incoming files or when copying files from the work directory to the data directory. The default is 2000; the minimum permitted is 128.