o

org.cert.netsa.mothra.tools

InvariantPackerMain

object InvariantPackerMain extends App with StrictLogging

Object to implement the InvariantPacker application.

Typical Usage in a Spark environment:

spark-submit --class org.cert.netsa.mothra.packer.tools.InvariantPackerMain mothra-tools.jar [--one-shot] <sourceDir> <destinationDir> <partitionerFile>

Processes files created by super_mediator running in invariant mode and writes them into HDFS.

Linear Supertypes
StrictLogging, App, DelayedInit, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. InvariantPackerMain
  2. StrictLogging
  3. App
  4. DelayedInit
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val DEFAULT_COMPRESSION: String

    The default compression codec to use for files written to HDFS.

    The default compression codec to use for files written to HDFS. This may be modified by specifying the following property: mothra.invariantpacker.compression.

    Values typically supported by Hadoop include bzip2, gzip, lz4, lzo, lzop, snappy, and default. The empty string indicates no compression.

  5. val DEFAULT_FILE_CACHE_SIZE: Int

    The default values for the maximum number output files allowed to be open at once.

    The default values for the maximum number output files allowed to be open at once.

    This value may be set at run time via the mothra.invariantpacker.fileCacheSize Java property.

  6. val DEFAULT_MAXIMUM_AGE_SECONDS: Int

    When an output file has not been opened yet, it is not opened until the oldest input file reaches a particular age, the number of input files reaches a count, or the sum of the sizes of the input files reaches a value.

    When an output file has not been opened yet, it is not opened until the oldest input file reaches a particular age, the number of input files reaches a count, or the sum of the sizes of the input files reaches a value. This holds the default setting for the age of the oldest input file, in seconds.

    The run-time behavior may be modified by setting the 'mothra.invariantpacker.maxInputAgeSeconds' Java property.

  7. val DEFAULT_MAX_THREADS: Int

    The default maximum number of threads that attempt to write to output files simultaneously.

    The default maximum number of threads that attempt to write to output files simultaneously.

    This run-time behavior may be modified by setting the mothra.invariantpacker.maxThreads property.

  8. val DEFAULT_MINIMUM_INPUT_COUNT: Int

    When an output file has not been opened yet, it is not opened until the oldest input file reaches a particular age, the number of input files reaches a count, or the sum of the sizes of the input files reaches a value.

    When an output file has not been opened yet, it is not opened until the oldest input file reaches a particular age, the number of input files reaches a count, or the sum of the sizes of the input files reaches a value. This holds the default setting for the minimum count of input files.

    The run-time behavior may be modified by setting the 'mothra.invariantpacker.minInputCount' Java property.

  9. val DEFAULT_MINIMUM_INPUT_SIZE: Long

    When an output file has not been opened yet, it is not opened until the oldest input file reaches a particular age, the number of input files reaches a count, or the sum of the sizes of the input files reaches a value.

    When an output file has not been opened yet, it is not opened until the oldest input file reaches a particular age, the number of input files reaches a count, or the sum of the sizes of the input files reaches a value. This holds the default setting for sum of the sizes of the input files, in octets.

    The run-time behavior may be modified by setting the 'mothra.invariantpacker.minInputSize' Java property.

  10. val DEFAULT_OBSERVATION_DOMAIN_ID: Int

    The default observationDomainId to use for output files.

    The default observationDomainId to use for output files.

    The run-time behavior may be modified by setting the 'mothra.invariantpacker.observationDomainId' Java property.

  11. val DEFAULT_OUTPUT_FILE_IDLE_SECONDS: Int

    The default maximum number of seconds to allow an idle output file to remain open so additional incoming records may appended to it.

    The default maximum number of seconds to allow an idle output file to remain open so additional incoming records may appended to it.

    The run-time behavior may be modified by setting the mothra.invariantpacker.outputIdleSeconds Java property.

  12. val DEFAULT_POLL_INTERVAL: Int

    Default value for often the source directory is scanned (polled) for IPFIX files to process.

    Default value for often the source directory is scanned (polled) for IPFIX files to process. This value may be specified at run-time by specifying the mothra.invariantpacker.pollingInterval Java property.

  13. val MINIMUM_FILE_CACHE_SIZE: Int

    The minimum value allowed for the maximum number of output files that may be opened at once.

  14. val MINIMUM_MAXIMUM_SIZE: Long

    The minimum value allowed for the maximum file size as specified by the mothra.invariantpacker.maximumSize Java property.

  15. val OUTPUT_FILE_IDLE_SECONDS_MINIMUM: Int

    The minimum allowed value for the mothra.invariantpacker.outputIdleSeconds Java property.

  16. def args: Array[String]
    Attributes
    protected
    Definition Classes
    App
    Annotations
    @deprecatedOverriding( "args should not be overridden" , "2.11.0" )
  17. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  18. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  19. val compressCodec: Option[CompressionCodec]

    The compression codec used for files written to HDFS.

    The compression codec used for files written to HDFS. This may be set by setting the "mothra.invariantpacker.compression" property. If that property is not set, DEFAULT_COMPRESSION is used.

  20. val destinationDir: Path
  21. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  22. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  23. val executionStart: Long
    Definition Classes
    App
    Annotations
    @deprecatedOverriding( ... , "2.11.0" )
  24. val fileCacheSize: Int

    The maximum number of output files allowed to be open at once.

    The maximum number of output files allowed to be open at once. Defaults to DEFAULT_FILE_CACHE_SIZE.

    This value may be set at run time via the mothra.invariantpacker.fileCacheSize Java property.

  25. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  26. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  27. implicit val hadoopConf: Configuration

    The Hadoop configuration

  28. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  29. val ignored: String
  30. val incomingDir: Path
  31. implicit val infoModel: InfoModel

    The information model

  32. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  33. val logger: Logger
    Attributes
    protected
    Definition Classes
    StrictLogging
  34. def main(args: Array[String]): Unit
    Definition Classes
    App
    Annotations
    @deprecatedOverriding( "main should not be overridden" , "2.11.0" )
  35. val maxInputAge: Int

    Have an unopened output file process its input files when there exists an input file whose age is older than this (number of milliseconds), regardless of the number or sizes of the input files.

    Have an unopened output file process its input files when there exists an input file whose age is older than this (number of milliseconds), regardless of the number or sizes of the input files. It defaults to the value DEFAULT_MAX_AGE_SECONDS (converted to milliseonds).

    The run-time behavior may be modified by setting the 'mothra.invariantpacker.maxInputAgeSeconds' Java property.

  36. val maxThreads: Int

    The maximum number of threads that attempt to write to output files simultaneously.

    The maximum number of threads that attempt to write to output files simultaneously. It defaults to the value DEFAULT_MAX_THREADS.

    This run-time behavior may be modified by setting the mothra.invariantpacker.maxThreads property.

  37. val maximumSize: Option[Long]

    The (approximate) maximum size file to create.

    The (approximate) maximum size file to create. The default is no maximum. When a file's size exceeds this value, the file is closed and a new file is started. Typically a file's size will not exceed this value by more than the maximum size of an IPFIX message, 64k.

  38. val minInputCount: Int

    Have an unopened output file process its input files when there are more than this number of input files.

    Have an unopened output file process its input files when there are more than this number of input files.

    The run-time behavior may be modified by setting the 'mothra.invariantpacker.minInputCount' Java property.

  39. val minInputSize: Long

    Have an unopened output file process its input files when the sum of the sizes of the input files is larger than this number of octets.

    Have an unopened output file process its input files when the sum of the sizes of the input files is larger than this number of octets.

    The run-time behavior may be modified by setting the 'mothra.invariantpacker.minInputSize' Java property.

  40. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  41. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  42. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  43. val observationDomain: Int

    The observationDomainId to use for output files.

    The observationDomainId to use for output files.

    The run-time behavior may be modified by setting the 'mothra.invariantpacker.observationDomainId' Java property.

  44. var oneshot: Boolean
  45. val outputIdleSeconds: Int

    The maximum number of seconds to allow an idle output file to remain open so additional incoming records may appended to it.

    The maximum number of seconds to allow an idle output file to remain open so additional incoming records may appended to it. It defaults to the value DEFAULT_OUTPUT_FILE_IDLE_SECONDS.

    This run-time behavior may be modified by setting the mothra.invariantpacker.outputIdleSeconds Java property.

  46. val packLogicPath: Path
  47. val pollingInterval: Int

    How often the incoming directory is scanned for files to process, in seconds.

    How often the incoming directory is scanned for files to process, in seconds. Defaults to DEFAULT_POLL_INTERVAL.

    The run-time behavior may be modified by setting the 'mothra.invariantpacker.pollingInterval' Java property.

  48. val positionalArgs: Array[String]
  49. var running: Boolean
  50. val switches: Array[String]
  51. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  52. def toString(): String
    Definition Classes
    AnyRef → Any
  53. def usage(full: Boolean = false): Unit
  54. def version(): Unit
  55. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  56. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  57. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Deprecated Value Members

  1. def delayedInit(body: ⇒ Unit): Unit
    Definition Classes
    App → DelayedInit
    Annotations
    @deprecated
    Deprecated

    (Since version 2.11.0) the delayedInit mechanism will disappear

Inherited from StrictLogging

Inherited from App

Inherited from DelayedInit

Inherited from AnyRef

Inherited from Any

Ungrouped