DataBag

Abstract Value Members

abstract def collect(): Seq[A]

Converts the DataBag back into a scala Seq.
Converts the DataBag back into a scala Seq. Warning: Do not call this on DataBags that are too large to fit on one machine!
returns
The contents of the DataBag as a scala Seq.
abstract def distinct: DataBag[A]

Removes duplicate entries from the bag, e.g.
Removes duplicate entries from the bag, e.g.
```
DataBag(Seq(1,1,2,3)).distinct = DataBag(Seq(1,2,3))
```
returns
A version of this DataBag without duplicate entries.
abstract def flatMap[B](f: (A) ⇒ DataBag[B])(implicit arg0: Meta[B]): DataBag[B]

Monad flatMap.
Monad flatMap.
B
Type of the output DataBag.
f
Function to be applied on the collection. The resulting bags are flattened.
returns
A DataBag containing the union (flattening) of the DataBags f(x) produced for each element of the input.

abstract def fold[B](agg: Alg[A, B])(implicit arg0: Meta[B]): B

Structural recursion over the bag.

Structural recursion over the bag. Assumes an algebraic specification of the DataBag type using three constructors:

sealed trait DataBag[A]
case class Sng[A](x: A) extends DataBag[A]
case class Union[A](xs: DataBag[A], ys: Bag[A]) extends DataBag[A]
case object Empty extends DataBag[Nothing]

The function then denotes the following recursive computation:

this match {
  case Empty => agg.zero
  case Sng(x) => agg.init(x)
  case Union(xs, ys) => p(xs.fold(agg), ys.fold(agg))
}

agg: The algebra parameterizing the recursion scheme.

abstract def groupBy[K](k: (A) ⇒ K)(implicit arg0: Meta[K]): DataBag[Group[K, DataBag[A]]]

Groups the bag by key.
Groups the bag by key.
K
Key type.
k
Key selector function.
returns
A version of this bag with the entries grouped by key.
implicit abstract def m: Meta[A]
abstract def map[B](f: (A) ⇒ B)(implicit arg0: Meta[B]): DataBag[B]

Monad map.
Monad map.
B
Type of the output DataBag.
f
Function to be applied on the collection.
returns
A DataBag containing the elements f(x) produced for each element x of the input.
abstract def sample(k: Int, seed: Long = 5394826801L): Vector[A]

Creates a sample of up to k elements using reservoir sampling initialized with the given seed.
Creates a sample of up to k elements using reservoir sampling initialized with the given seed.
If the collection represented by the DataBag instance contains less then n elements, the resulting collection is trimmed to a smaller size.
The method should be deterministic for a fixed DataBag instance with a materialized result. In other words, calling xs.sample(n)(seed) two times in succession will return the same result.
The result, however, might vary between program runs and DataBag implementations.
abstract def union(that: DataBag[A]): DataBag[A]

Union operator.
Union operator. Respects duplicates, e.g.:
```
DataBag(Seq(1,1,2,3)) plus DataBag(Seq(1,2,5)) = DataBag(Seq(1,1,2,3,1,2,5))
```
that
The second addend parameter.
returns
The set-theoretic union (with duplicates) between this DataBag and the given one.
abstract def withFilter(p: (A) ⇒ Boolean): DataBag[A]

Monad filter.
Monad filter.
p
Predicate to be applied on the collection. Only qualifying elements are passed down the chain.
abstract def writeCSV(path: String, format: CSV)(implicit converter: CSVConverter[A]): Unit

Writes a DataBag into the specified path in a CSV format.
Writes a DataBag into the specified path in a CSV format.
path
The location where the data will be written.
format
The CSV format configuration
converter
A converter to use for element serialization.
abstract def writeParquet(path: String, format: Parquet)(implicit converter: ParquetConverter[A]): Unit

Writes a DataBag into the specified path in a CSV format.
Writes a DataBag into the specified path in a CSV format.
path
The location where the data will be written.
format
The CSV format configuration
converter
A converter to use for element serialization.
abstract def writeText(path: String): Unit

Writes a DataBag into the specified path as plain text.
Writes a DataBag into the specified path as plain text.
The serialization logic is backend-specific.
path
The location where the data will be written.
abstract def zipWithIndex(): DataBag[(A, Long)]

Zips the elements of this collection with a unique dense index.
Zips the elements of this collection with a unique dense index.
The method should be deterministic for a fixed DataBag instance with a materialized result. In other words, calling xs.zipWithIndex() two times in succession will return the same result.
The result, however, might vary between program runs and DataBag implementations.

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def as[DColl[_]](implicit conv: (DataBag[A]) ⇒ DColl[A]): DColl[A]

Converts this bag into a distributed collection of type DColl[A].
final def asInstanceOf[T0]: T0

Definition Classes
Any
def bottom(n: Int)(implicit o: Ordering[A]): List[A]

Find the bottom n elements in the collection with respect to the natural ordering of the elements' type.
Find the bottom n elements in the collection with respect to the natural ordering of the elements' type.
n
number of elements to return
o
the implicit Ordering of elements
returns
an ordered (ascending) List of the bottom n elements
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def count(p: (A) ⇒ Boolean): Long

Count the number of elements in the collection that satisfy a predicate.
Count the number of elements in the collection that satisfy a predicate.
p
the predicate to test against
returns
the number of elements that satisfy p
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(o: Any): Boolean

Definition Classes
DataBag → AnyRef → Any
def exists(p: (A) ⇒ Boolean): Boolean

Test if at least one element of the collection satisfies p.
Test if at least one element of the collection satisfies p.
p
predicate to test against the elements of the collection
returns
true if the collection contains an element that satisfies the predicate
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def find(p: (A) ⇒ Boolean): Option[A]

Finds some element in the collection that satisfies a given predicate.
Finds some element in the collection that satisfies a given predicate.
p
the predicate to test against
returns
Some element if one exists, None otherwise
def fold[B](zero: B)(init: (A) ⇒ B, plus: (B, B) ⇒ B)(implicit arg0: Meta[B]): B

Delegates to fold(Alg(zero, init, plus)).
def forall(p: (A) ⇒ Boolean): Boolean

Test if all elements of the collection satisfy p.
Test if all elements of the collection satisfy p.
p
predicate to test against the elements of the collection
returns
true if all the elements of the collection satisfy the predicate
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
DataBag → AnyRef → Any
def isEmpty: Boolean

Test the collection for emptiness.
Test the collection for emptiness.
returns
true if the collection contains no elements at all
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def max(implicit o: Ordering[A]): A

Find the largest element in the collection with respect to the natural ordering of the elements' type.
Find the largest element in the collection with respect to the natural ordering of the elements' type.
o
the implicit natural Ordering of the elements

Exceptions thrown
Exception if the collection is empty
def min(implicit o: Ordering[A]): A

Find the smallest element in the collection with respect to the natural ordering of the elements' type.
Find the smallest element in the collection with respect to the natural ordering of the elements' type.
o
the implicit natural Ordering of the elements

Exceptions thrown
Exception if the collection is empty
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def nonEmpty: Boolean

Tet the collection for emptiness.
Tet the collection for emptiness.
returns
true if the collection has at least one element
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def product(implicit n: Numeric[A]): A

Calculate the product over all elements in the collection.
Calculate the product over all elements in the collection.
n
implicit Numeric operations of the elements
returns
one if the collection is empty
def reduce[B >: A](zero: B)(plus: (B, B) ⇒ B)(implicit arg0: Meta[B]): B

Shortcut for fold(z)(identity, f).
Shortcut for fold(z)(identity, f).
B
return type (super class of the element type)
returns
the result of combining all elements into one
def reduceOption(plus: (A, A) ⇒ A): Option[A]

Shortcut for fold(None)(Some, Option.lift2(f)), which is the same as reducing the collection to a single element by applying a binary operator.
Shortcut for fold(None)(Some, Option.lift2(f)), which is the same as reducing the collection to a single element by applying a binary operator.
returns
the result of reducing all elements into one
def size: Long

returns
the number of elements in the collection
def sum(implicit n: Numeric[A]): A

Calculate the sum over all elements in the collection.
Calculate the sum over all elements in the collection.
n
implicit Numeric operations of the elements
returns
zero if the collection is empty
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
def top(n: Int)(implicit o: Ordering[A]): List[A]

Find the top n elements in the collection with respect to the natural ordering of the elements' type.
Find the top n elements in the collection with respect to the natural ordering of the elements' type.
n
number of elements to return
o
the implicit Ordering of elements
returns
an ordered (descending) List of the bottom n elements
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object DataBag | package api

trait DataBag[A] extends Serializable

Abstract Value Members

abstract def collect(): Seq[A]

abstract def distinct: DataBag[A]

abstract def flatMap[B](f: (A) ⇒ DataBag[B])(implicit arg0: Meta[B]): DataBag[B]

abstract def fold[B](agg: Alg[A, B])(implicit arg0: Meta[B]): B

abstract def groupBy[K](k: (A) ⇒ K)(implicit arg0: Meta[K]): DataBag[Group[K, DataBag[A]]]

implicit abstract def m: Meta[A]

abstract def map[B](f: (A) ⇒ B)(implicit arg0: Meta[B]): DataBag[B]

abstract def sample(k: Int, seed: Long = 5394826801L): Vector[A]

abstract def union(that: DataBag[A]): DataBag[A]

abstract def withFilter(p: (A) ⇒ Boolean): DataBag[A]

abstract def writeCSV(path: String, format: CSV)(implicit converter: CSVConverter[A]): Unit

abstract def writeParquet(path: String, format: Parquet)(implicit converter: ParquetConverter[A]): Unit

abstract def writeText(path: String): Unit

abstract def zipWithIndex(): DataBag[(A, Long)]

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def as[DColl[_]](implicit conv: (DataBag[A]) ⇒ DColl[A]): DColl[A]

final def asInstanceOf[T0]: T0

def bottom(n: Int)(implicit o: Ordering[A]): List[A]

def clone(): AnyRef

def count(p: (A) ⇒ Boolean): Long

final def eq(arg0: AnyRef): Boolean

def equals(o: Any): Boolean

def exists(p: (A) ⇒ Boolean): Boolean

def finalize(): Unit

def find(p: (A) ⇒ Boolean): Option[A]

def fold[B](zero: B)(init: (A) ⇒ B, plus: (B, B) ⇒ B)(implicit arg0: Meta[B]): B

def forall(p: (A) ⇒ Boolean): Boolean

final def getClass(): Class[_]

def hashCode(): Int

def isEmpty: Boolean

final def isInstanceOf[T0]: Boolean

def max(implicit o: Ordering[A]): A

def min(implicit o: Ordering[A]): A

final def ne(arg0: AnyRef): Boolean

def nonEmpty: Boolean

final def notify(): Unit

final def notifyAll(): Unit

def product(implicit n: Numeric[A]): A

def reduce[B >: A](zero: B)(plus: (B, B) ⇒ B)(implicit arg0: Meta[B]): B

def reduceOption(plus: (A, A) ⇒ A): Option[A]

def size: Long

def sum(implicit n: Numeric[A]): A

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def top(n: Int)(implicit o: Ordering[A]): List[A]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped