pipeline

Type Members

1. trait Ai2CodeInfo extends HasCodeInfo

   Reads the version number and GitHub URL from configuration file bundled into the jar.

2. trait Ai2Signature extends PipelineRunnerSupport with Ai2CodeInfo

   For convenience, case classes can mix in this single trait to implement PipelineRunnerSupport

3. trait Artifact extends AnyRef

   Represents data in a persistent store.

4. trait ArtifactIo[T, -A <: Artifact] extends HasCodeInfo

   Interface for defining how to persist a data type.

5. class ArtifactStreamWriter extends AnyRef

   Class for writing that exposes a more restrictive interface than OutputStream In particular, we don't want clients to close the stream Also, we force character encoding to UTF-8.

6. trait CachingDisabled extends CachingEnabled

7. trait CachingEnabled extends AnyRef

8. case class CodeInfo(className: String, buildId: String, unchangedSince: String, srcUrl: Option[URI], binaryUrl: Option[URI]) extends Product with Serializable

   Contains information about the origin of the compiled class implementing a Producer

9. trait ColumnFormats extends AnyRef

   Support for persisting to a column-delimited file.

10. class DirectoryArtifact extends StructuredArtifact

    Directory of files.

11. class FileArtifact extends FlatArtifact

    Flat file.

12. trait FlatArtifact extends Artifact

    Generic data blob.

13. trait HasCodeInfo extends AnyRef

14. class LineCollectionIo[T] extends ArtifactIo[Iterable[T], FlatArtifact] with Ai2CodeInfo

    Persist a collection of string-serializable objects to a flat file, one line per object.

15. class LineIteratorIo[T] extends ArtifactIo[Iterator[T], FlatArtifact] with Ai2CodeInfo

    Persist an iterator of string-serializable objects to a flat file, one line per object.

16. case class Link(fromId: String, toId: String, name: String) extends Product with Serializable

    Represents dependency between Producer instances

17. case class MavenVersionId(major: Int, minor: Option[Int] = scala.None, incremental: Option[Int] = scala.None, build: Option[Int] = scala.None, qualifier: Option[String] = scala.None) extends Comparable[MavenVersionId] with Product with Serializable

    Maven-style version id

18. trait NoPipelineRunnerSupport extends PipelineRunnerSupport

    Producer implementations that do not need to be executed by PipelineRunner can mix in this convenience trait.

19. case class Node(info: CodeInfo, params: Map[String, String], outputPath: Option[URI]) extends Product with Serializable

    Represents a Producer instance with PipelineRunnerSupport

20. class PersistedProducer[T, A <: Artifact] extends Producer[T]

21. class PipelineRunner extends pipeline.IoHelpers.ArtifactFactory[(Signature, String)]

    Executes a pipeline represented by a set of Producer instances Inspects the meta-info about the pipeline steps (represented by PipelineRunnerSupport interface) and builds a DAG representation of the pipeline.

22. trait PipelineRunnerSupport extends HasCodeInfo

    This information is used by PipelineRunner to construct and visualize the DAG for a pipeline

23. trait Producer[T] extends Logging with CachingEnabled with PipelineRunnerSupport

    An individual step in a data processing pipeline.

24. trait ReadHelpers extends ColumnFormats

25. trait S3Artifact[A <: Artifact] extends Logging

26. case class S3Config(service: AmazonS3Client, bucket: String) extends Product with Serializable

27. class S3FlatArtifact extends FlatArtifact with S3Artifact[FileArtifact]

    Artifact implementations using S3 storage.

28. class S3ZipArtifact extends StructuredArtifact with S3Artifact[ZipFileArtifact]

    Zip file stored in S3.

29. case class Signature(name: String, unchangedSinceVersion: String, dependencies: Map[String, PipelineRunnerSupport], parameters: Map[String, String]) extends Product with Serializable

    Acts as an identifier for a Producer instance.

30. class SingletonIo[T] extends ArtifactIo[T, FlatArtifact] with Ai2CodeInfo

    Persist a single object to a flat file.

31. trait StringSerializable[T] extends AnyRef

    Serialize an object to/from a String

32. trait StructuredArtifact extends Artifact

    Artifact with nested structure, containing multiple data blobs identified by String names.

33. trait UnknownCodeInfo extends HasCodeInfo

    Represents code from an unspecified location

34. case class Workflow(nodes: Map[String, Node], links: Iterable[Link]) extends Product with Serializable

    DAG representation of the execution of a set of Producers

35. trait WriteHelpers extends AnyRef

36. class ZipFileArtifact extends StructuredArtifact

    Zip file.