Object to implement the FileJoiner application.

Typical Usage in a Spark environment:

spark-submit --class org.cert.netsa.mothra.packer.tools.FileJoinerMain mothra-tools.jar <s1> [<s2> <s3> ...]

where:

s1..sn: Directories to process, as Hadoop URIs

FileJoiner reduces the number of data files in a Mothra repository. It may also be used to modify the files' compression.

FileJoiner runs as a batch process, not as a daemon.

FileJoiner makes a single recursive scan of the source directories <s1>, <s2>, ... for files whose names match the pattern "YYYYMMDD.HH." or "YYYYMMDD.HH-PTddH." (It looks for files matching the regular expression ^\d{8}\.\d{2}(?:-PT\d\d?H)?\.) Files whose names match that pattern are processed by FileJoiner to create a single new file in the same directory that has the same prefix as the originals, and then the original file(s) are removed.

By default, files that share the same prefix are only processed when there are two or more files. To force re-writing when there is a single file, set the Java property mothra.filejoiner.minCountToJoin to a value less than 2. The property may also be used to create a new file only when an "excessive" number of files share the same prefix.

There is always a single thread that recursively scans the directories. The number of threads that joins the files may be set by specifying the mothra.filejoiner.maxThreads Java property. If not specified, the default is 6.

FileJoiner may be run so that either it spawns a thread for every directory that contains files to be joined or it spawns a thread for each set of files in a directory that have the same prefix. The behavior is controlled whether the mothra.filejoiner.spawnThread Java property is set to by-prefix or by-directory. The default is by-directory. (For backwards compatibility, by-hour is an alias for by-prefix.)

By default, FileJoiner does not compress the files it writes. (NOTE: It should support writing the output using the same compression as the input.) To specify the compression codec that it should use, specify the mothra.filejoiner.compression Java property. Values typically supported by Hadoop include bzip2, gzip, lz4, lzo, lzop, snappy, and default. The empty string indicates no compression.

FileJoiner joins files sharing the same prefix into a single file by default. The mothra.filejoiner.maximumSize Java property may be used to limit the maximum file size. The size is for the compressed file if compression is active. The value is approximate since it is only checked after the data appears on disk which occurs in large blocks because of buffering by the Java stream code and the compression algorithm. (By setting that property and mothra.filejoiner.minCountToJoin to 1, you can force large files to be split into smaller ones, making the FileJoiner a file-splitter.)