FileSanitizerMain
Related Doc:
package tools
The default compression codec to use for files written to HDFS.
The default compression codec to use for files written to HDFS. This may be modified by specifying the following property: mothra.filesanitizer.compression.
Values typically supported by Hadoop include bzip2, gzip, lz4,
lzo, lzop, snappy, and default. The empty string indicates no
compression.
The default number of threads to run for sanitizing files when the
mothra.filesanitizer.maxThreads Java property is not set.
The default number of threads to run for sanitizing files when the
mothra.filesanitizer.maxThreads Java property is not set. (The
scanning task always runs in its own thread.)
The default value for spawnThread when the
mothra.filesanitizer.spawnThread Java property is not specified.
The compression codec used for files written to HDFS.
The compression codec used for files written to HDFS. This may be set by setting the "mothra.filesanitizer.compression" property. If that property is not set, DEFAULT_COMPRESSION is used.
The Hadoop configuration
The information model
How often to print log messages regarding the number of tasks, in seconds.
The maximum number of filesanitizer threads to start.
The maximum number of filesanitizer threads to start. It defaults to
the value DEFAULT_MAX_THREADS.
This run-time behavior may be modified by setting the mothra.filesanitizer.maxThreads property.
The (approximate) maximum size file to create.
The (approximate) maximum size file to create. The default is no maximum. When a file's size exceeds this value, the file is closed and a new file is started. Typically a file's size will not exceed this value by more than the maximum size of an IPFIX message, 64k.
The behavior as to whether a file-sanitizing thread is spawned...
The behavior as to whether a file-sanitizing thread is spawned...
by-directory: for every directory that contains files to be sanitized,
or
by-prefix: for every unqiue basename prefix (that is, the file name
without the UUID) (in a single directory) that contains files to be
sanitized. by-hour is an alias for by-prefix.
The default is specified by the DEFAULT_SPAWN_THREAD variable. The
run-time behavior may be modified by setting the
mothra.filesanitizer.spawnThread Java property to one of those values.
Mapping from spawnThread value to threadPerDirectory.
Object to implement the FileSanitizer application.
Typical Usage in a Spark environment:
spark-submit --class org.cert.netsa.mothra.packer.tools.FileSanitizerMain mothra-tools.jar <f1>[,<f2>[,<f3>...]] <s1> [<s2> <s3> ...]where:
f1..fn: Names of InfoElements to be removed from the files s1..sn: Directories to process, as Hadoop URIs
FileSanitizer removes Information Element fields from the data files in a Mothra repository. In addition, when multiple files share the same name except for the UUID, FileSanitizer combines those files together.
The IE fields to be removed must be specified in a single argument, as a comma-separated list of names, such as
sourceTransportPort,destinationTransportPort.Each remaining argument is a single directory to process.
FileSanitizer runs as a batch process, not as a daemon.
FileSanitizer makes a single recursive scan of the source directories <s1>, <s2>, ... for files whose names match the pattern "YYYYMMDD.HH." or "YYYYMMDD.HH-PTddH." (It looks for files matching the regular expression
^\d{8}\.\d{2}(?:-PT\d\d?H)?\.) Files whose names match that pattern are processed by FileSanitizer to remove the named Information Elements. All files where the regular expression matched the same string are joined into a single file, similar to the FileJoiner. Finally, the original files are removed.There is always a single thread that recursively scans the directories. The number of threads that sanitizes and joins the files may be set by specifying the
mothra.filesanitizer.maxThreadsJava property. If not specified, the default is 6.FileSanitizer may be run so that either it spawns a thread for every directory that contains files to process or it spawns a thread for each set of files in a directory that have the same prefix. The behavior is controlled whether the
mothra.filesanitizer.spawnThreadJava property is set toby-prefixorby-directory. The default isby-directory. (For backwards compatibility,by-houris an alias forby-prefix.)By default, FileSanitizer does not compress the files it writes. (NOTE: It should support writing the output using the same compression as the input.) To specify the compression codec that it should use, specify the
mothra.filesanitizer.compressionJava property. Values typically supported by Hadoop includebzip2,gzip,lz4,lzo,lzop,snappy, anddefault. The empty string indicates no compression.FileSanitizer joins the files sharing the same prefix into a single file by default. The
mothra.filesanitizer.maximumSizeJava property may be used to limit the maximum file size. The size is for the compressed file if compression is active. The value is approximate since it is only checked after the data appears on disk which occurs in large blocks because of buffering by the Java stream code and the compression algorithm.