package silk
SiLK file formats, data types, and methods to read them, including support for reading them from Spark.
RWRec is the type of SiLK flow records.
You can use RWRecReader to read SiLK files from Scala, including compressed files if Hadoop native libraries are available. For example:
import org.cert.netsa.io.silk.RWRecReader import java.io.FileInputStream val inputFile = new FileInputStream("path/to/silk/rw/file") for ( rec <- RWRecReader.ofInputStream(inputFile) ) { println(rec.sIP) }
- See also
org.cert.netsa.mothra.datasources.silk.flow for working with SiLK data in Spark using the Mothra SiLK datasource.
- Alphabetic
- By Inheritance
- silk
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
sealed abstract
class
BagDataType extends ShortEnumEntry
A SiLK bag data type.
A SiLK bag data type. May be converted to and from Short values.
- See also
the companion object for more details
-
abstract
class
BagReader[T] extends Iterator[(T, Long)]
A reader of binary SiLK Bag files.
A reader of binary SiLK Bag files. This is usable as an Iterator over a pair representing the key and counter in the Bag. Depending on the contents of the Bag file, the key is either an IPAddress or an Int. The counter is always a Long.
Since a Bag file may contain different key types, the BagReader factory methods return a BagResult wrapper over BagReader.
The
keyTypeandcounterTypemethods return a BagDataType that specifies the type of the key and the counter that were specified in the Bag file's header.This example uses the single argument form of the companion object's
ofInputStream()method to read the Bag file "example.bag". This code may be used outside of Hadoop when it is known that the Bag's key is an IPAddress.val stream = new java.io.FileInputStream("example.bag") val bagresult = BagReader.ofInputStream(stream) val bag = bagresult match { case BagResult.IPAddressBag(iter) => iter case _ => null } bag.hasNext val (ipaddr, counter) = bag.next()
- Note
While the SiLK command lines tools display an unspecified key type as an IPv4 address, the BagReader class treats an unspecified key type as an integer.
- See also
the companion object for more details.
Example: -
sealed abstract
class
BagResult extends AnyRef
Return type for the
ofInputStream()method of the companion object to the BagReader class. -
class
BagWriter extends AnyRef
A writer of binary SiLK Bag files.
A writer of binary SiLK Bag files.
To include a header in the Bag file that specifies the type of the and counter, run
setKeyType()and/orsetCounterType()prior to writing the Bag.This example reads the contents of "example.bag" and writes it to "copy.bag", where the keys are IP addresses:
val in = new java.io.FileInputStream("example.bag") val out = new java.io.FileOutputStream("copy.bag") val bagresult = BagReader.ofInputStream(in) val bag = bagresult match { case BagResult.IPAddressBag(iter) => iter case _ => null } val writer = BagWriter.toOutputStream(out) if ( None != bag.keyType ) { writer.setKeyType(bag.keyType) } if ( None != bag.counterType ) { writer.setCounterType(bag.counterType) } writer.appendIPAddresses(bag) writer.close()
- See also
the companion object for more details
Example: -
sealed abstract
class
CompressionMethod extends ByteEnumEntry
A SiLK compression method.
A SiLK compression method. May be converted to and from Byte values.
See the companion object for more details.
-
sealed abstract
class
FileFormat extends ByteEnumEntry
A SiLK file format.
A SiLK file format. May be converted to and from Byte values. Equipped with a partial function mapping Short file versions to Unpackers.
See the companion object for more details.
-
final
case class
FlowType(toByte: Byte) extends AnyVal with Product with Serializable
A SiLK flow type (type and class), as represented by a Byte value.
A SiLK flow type (type and class), as represented by a Byte value.
- toByte
The byte value representing this flow type.
-
case class
Header(fileFlags: Byte, fileFormat: FileFormat, fileVersion: Byte, compressionMethod: CompressionMethod, silkVersion: SilkVersion, recordSize: Short, recordVersion: Short, headerEntries: IndexedSeq[HeaderEntry]) extends Product with Serializable
A SiLK file header, including contained header entries.
A SiLK file header, including contained header entries. Supports only "new-style" header format (SiLK versions 1.0+).
- fileFlags
The bits encoding file flags. Currently only whether the file is big-endian.
- fileFormat
The SiLK file format contained within this file.
- fileVersion
The SiLK file version--specifically the version of the header format.
- compressionMethod
The compression method used by data in this file.
- silkVersion
The version of SiLK used to create this file.
- recordSize
The size of individual (uncompressed) records in this file.
- recordVersion
The record version of the file format.
- headerEntries
Sequence of additional extensible header records of various types.
- See also
Header.isBigEndian
-
sealed abstract
class
HeaderEntry extends AnyRef
One of a variety of SiLK extended header entries.
One of a variety of SiLK extended header entries.
See the companion object for more details.
-
abstract
class
IPSetReader extends Iterator[IPBlock]
A reader of binary SiLK IPset files.
A reader of binary SiLK IPset files. This is usable as an Iterator over IPBlock objects.
This example uses the single argument form of the companion object's
ofInputStream()method to read the IPset file "example.set"; the code may be used outside of Hadoop.val stream = new java.io.FileInputStream("example.set") val ipset = IPSetReader.ofInputStream(stream) ipset.hasNext val ipblock = ipset.next() println(ipblock.min + "/" + ipblock.prefixLength)
- See also
the companion object for more details
Example: -
class
IPSetWriter extends AnyRef
A writer of binary SiLK IPset files.
A writer of binary SiLK IPset files.
This example reads the contents of "example.set" and writes it to "copy.set":
val in = new java.io.FileInputStream("example.set") val out = new java.io.FileOutputStream("copy.set") val ipset = IPSetReader.ofInputStream(in) val writer = IPSetWriter.toOutputStream(out) writer.append(ipset) writer.close()
- See also
the companion object for more details
Example: -
final
case class
PrefixMapProtocolPortPair(startProtocol: Protocol, startPort: Port, endProtocol: Protocol, endPort: Port) extends Product with Serializable
A type representing a range of (org.cert.netsa.data.net.Protocol, org.cert.netsa.data.net.Port) Pairs.
A type representing a range of (org.cert.netsa.data.net.Protocol, org.cert.netsa.data.net.Port) Pairs.
This is the type of the key when using an Iterator over the contents of a PrefixMap file that contains (Protocol, Port) Pairs.
- See also
-
abstract
class
PrefixMapReader[T] extends Iterator[(T, String)]
A reader of binary SiLK PrefixMap files.
A reader of binary SiLK PrefixMap files. This is usable as an Iterator over a pair representing a key and value. Depending on the contents of the PrefixMap file, the key is either an IPBlock or a PrefixMapProtocolPortPair. The value is always a String.
Because the PrefixMap may contain different key types, the PrefixMapReader factory methods return a PrefixMapResult wrapper over PrefixMapReader.
The mapName method returns the map name that was specified in the PrefixMap file's header, if any.
This example uses the single argument form of the companion object's
ofInputStream()method to read the PrefixMap file "example.pmap". The code may be used outside of Hadoop when it is known that the key is an IPBlock.val stream = new java.io.FileInputStream("example.pmap") val pmapresult = PrefixMapReader.ofInputStream(stream) val pmap = pmapresult match { case PrefixMapResult.IPBlockPrefixMap(iter) => iter case _ => null } pmap.hasNext val (ipblock, name) = pmap.next()
- See also
the companion object for more details
Example: -
sealed abstract
class
PrefixMapResult extends AnyRef
Return type for the
ofInputStream()method of the companion object to the PrefixMapReader class. -
case class
RWRec(startTime: Instant, elapsed: Duration, sPort: Port, dPort: Port, protocol: Protocol, flowType: FlowType, sensor: Sensor, flags: TCPFlags, initFlags: TCPFlags, restFlags: TCPFlags, tcpState: TCPState, application: Port, memo: Short, input: SNMPInterface, output: SNMPInterface, packets: Long, bytes: Long, sIP: IPAddress, dIP: IPAddress, nhIP: IPAddress) extends Product with Serializable
A SiLK flow record.
A SiLK flow record.
Note that in addition to the fields of the case class, some derived fields are also provided. (See below.)
- startTime
The instant that the first packet in this flow was observed.
- elapsed
The duration between the instants the first and last packets in this flow were observed.
- sPort
The source port of this flow, or zero if this flow is neither a TCP nor a UDP flow.
- dPort
The destination port of this flow, or zero if this flow is neither a TCP nor a UDP flow.
- protocol
The IP protocol of this flow.
- flowType
The SiLK flow type (class and type) of this flow, or
FlowType(0)if unknown.- sensor
The SiLK sensor that observed this flow, or
Sensor(0)if unknown.- flags
The union of all TCP flags observed in this flow, or
TCPFlags(0)if this flow is not a TCP flow.- initFlags
The flags observed in the initial packet of this TCP flow, or
TCPFlags(0)if this flow is not a TCP flow or if extended flags are not available.- restFlags
The union of all TCP flags observed after the initial packet of this flow, or
TCPFlags(0)if this flow is not a TCP flow or if extended flags are not available.- tcpState
Flags relating to the observed status of this flow, including whether extended TCP flags are available. See TCPState for more details.
- application
The detected application of this flow, expressed as the common port number for that application, or
Port(0)if no application was detected.- memo
A Short value stored as a memo on this flow, or zero if no such memo has been set.
- input
The input SNMP routing interface for this flow, or
SNMPInterface(0)if routing information is not available.- output
The output SNMP routing interface for this flow, or
SNMPInterface(0)if routing information is not available.- packets
The number of IP packets observed in this flow.
- bytes
The number of bytes in packets observed in this flow.
- sIP
The source IP address of packets in this flow.
- dIP
The destination IP address of packets in this flow.
- nhIP
The next-hop IP address of packets in this flow, or
IPAddress("0.0.0.0")orIPAddress("::")if routing information is not available.
-
class
RWRecReader extends Iterator[RWRec]
A reader of SiLK flow records.
A reader of SiLK flow records. This is usable as an Iterator over RWRec records.
See the companion object for more details.
-
class
RWRecWriter extends AnyRef
A writer of binary SiLK RWRec files that are readable by SiLK.
A writer of binary SiLK RWRec files that are readable by SiLK.
This example reads the contents of "example.rw" and writes it to "copy.rw":
val in = new java.io.FileInputStream("example.rw") val out = new java.io.FileOutputStream("copy.rw") val reader = RWRecReader.ofInputStream(in) val writer = RWRecWriter.toOutputStream(out) writer.append(reader) writer.close()
- See also
the companion object for more details
Example: -
final
case class
Sensor(toShort: Short) extends AnyVal with Product with Serializable
A SiLK sensor, as represented by a Short value.
A SiLK sensor, as represented by a Short value.
- toShort
The short value representing this sensor.
-
case class
SilkConfig(version: Option[Int], defaultClassName: Option[String], packingLogicPath: Option[String], pathFormat: String, groups: Map[String, GroupConfig], sensors: SensorMap, classes: Map[String, ClassConfig]) extends Product with Serializable
SiLK data spool configuration.
SiLK data spool configuration.
- version
The version of the config file format used.
- defaultClassName
The default class to be examined if none is specified.
- packingLogicPath
The path to the plugin to be loaded by the packer for determining where to pack flows.
- pathFormat
The format used for filenames in the data spool.
- groups
The sensor groups defined in this configuration.
- sensors
The sensors defined in this configuration, usable as a value of type
Map[Sensor, SensorConfig].- classes
The classes defined in this configuration.
-
class
SilkDataFormatException extends IOException
Signals that a SiLK input file has been found to be malformed.
-
final
case class
SilkVersion(toInt: Int) extends AnyVal with Product with Serializable
A SiLK version number, as represented by an Int value encoding the major version, minor version, and patch numbers.
A SiLK version number, as represented by an Int value encoding the major version, minor version, and patch numbers.
- toInt
The int value representing this SiLK version.
-
final
case class
TCPState(toByte: Byte) extends AnyVal with Product with Serializable
A SiLK TCP state flag vector, encoding various properties of TCP packets in a TCP flow, as encoded in a Byte value.
A SiLK TCP state flag vector, encoding various properties of TCP packets in a TCP flow, as encoded in a Byte value.
- toByte
The byte value representing this state vector.
Value Members
- object BagDataType extends ShortEnum[BagDataType] with Product with Serializable
-
object
BagReader
The BagReader object provides support for creating a BagReader.
-
object
BagResult
The object returned by the
ofInputStream()method of the companion object to the BagReader class. -
object
BagWriter
The BagWriter companion object provides support for creating an BagWriter.
- object CompressionMethod extends ByteEnum[CompressionMethod] with Product with Serializable
- object FileFormat extends ByteEnum[FileFormat]
- object FlowType extends Serializable
- object Header extends Serializable
- object HeaderEntry
-
object
IPSetReader
The IPSetReader object provides support for creating an IPSetReader.
-
object
IPSetWriter
The IPSetWriter companion object provides support for creating an IPSetWriter.
-
object
PrefixMapProtocolPortPair extends Serializable
The PrefixMapProtocolPortPair object provides support for creating a PrefixMapProtocolPortPair.
-
object
PrefixMapReader
The PrefixMapReader object provides support for creating an PrefixMapReader.
-
object
PrefixMapResult
The object returned by the
ofInputStream()method of the companion object to the PrefixMapReader class. - object RWRecReader
-
object
RWRecWriter
The RWRecWriter companion object provides support for creating an RWRecWriter.
- object Sensor extends Serializable
- object SilkConfig extends Serializable
- object SilkVersion extends Serializable
- object TCPState extends Serializable