SecondarySortGroupingPairRDDFunctions

Instance Constructors

new SecondarySortGroupingPairRDDFunctions(rdd: RDD[(K, V)])(implicit arg0: Ordering[K], arg1: ClassTag[K], arg2: Ordering[V], arg3: ClassTag[V])

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def mapValuesWithKeyedPreparedResource[R, R1, A](resources: RDD[(K, R)], prepareResource: (R) ⇒ R1, op: (R1, V) ⇒ A)(implicit arg0: ClassTag[R]): RDD[(K, A)]

Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
R
the type of resources being used
A
the type returned by applying the operation with the resource to each value
resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
prepareResource
a function to transform the resource into what will be used in op
op
the operation to apply to each value. The operation takes a resource and value and returns the transformed value
returns
PairRDD of values transformed by applying the operation with the appropriate resource
def mapValuesWithKeyedPreparedResource[R, R1, A](resources: RDD[(K, R)], prepareResource: (R) ⇒ R1, op: (R1, V) ⇒ A, numPartitions: Int)(implicit arg0: ClassTag[R]): RDD[(K, A)]

Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
R
the type of resources being used
A
the type returned by applying the operation with the resource to each value
resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
prepareResource
a function to transform the resource into what will be used in op
op
the operation to apply to each value. The operation takes a resource and value and returns the transformed value
numPartitions
the number of partitions for shuffling
returns
PairRDD of values transformed by applying the operation with the appropriate resource
def mapValuesWithKeyedPreparedResource[R, R1, A](resources: RDD[(K, R)], prepareResource: (R) ⇒ R1, op: (R1, V) ⇒ A, partitioner: Partitioner)(implicit arg0: ClassTag[R]): RDD[(K, A)]

Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
R
the type of resources being used
A
the type returned by applying the operation with the resource to each value
resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
prepareResource
a function to transform the resource into what will be used in op
op
the operation to apply to each value. The operation takes a resource and value and returns the transformed value
partitioner
the partitioner for shuffling
returns
PairRDD of values transformed by applying the operation with the appropriate resource
def mapValuesWithKeyedPreparedResource[R, A](resources: RDD[(K, R)], op: (R) ⇒ (V) ⇒ A)(implicit arg0: ClassTag[R]): RDD[(K, A)]

Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
R
the type of resources being used
A
the type returned by applying the operation with the resource to each value
resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
op
The operation to apply to each value. The operation takes a resource and returns a function, which will then be applied to each value.
returns
PairRDD of values transformed by applying the operation with the appropriate resource
def mapValuesWithKeyedPreparedResource[R, A](resources: RDD[(K, R)], op: (R) ⇒ (V) ⇒ A, numPartitions: Int)(implicit arg0: ClassTag[R]): RDD[(K, A)]

Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
R
the type of resources being used
A
the type returned by applying the operation with the resource to each value
resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
op
The operation to apply to each value. The operation takes a resource and returns a function, which will then be applied to each value.
numPartitions
the number of partitions for shuffling
returns
PairRDD of values transformed by applying the operation with the appropriate resource
def mapValuesWithKeyedPreparedResource[R, A](resources: RDD[(K, R)], op: (R) ⇒ (V) ⇒ A, partitioner: Partitioner)(implicit arg0: ClassTag[R]): RDD[(K, A)]

Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
R
the type of resources being used
A
the type returned by applying the operation with the resource to each value
resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
op
The operation to apply to each value. The operation takes a resource and returns a function, which will then be applied to each value.
partitioner
the partitioner for shuffling
returns
PairRDD of values transformed by applying the operation with the appropriate resource
def mapValuesWithKeyedResource[R, A](resources: RDD[(K, R)], op: (R, V) ⇒ A)(implicit arg0: ClassTag[R]): RDD[(K, A)]

Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
R
the type of resources being used
A
the type returned by applying the operation with the resource to each value
resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
op
the operation to apply to each value. Takes a resource and value and returns the transformed value
returns
PairRDD of values transformed by applying the operation with the appropriate resource
def mapValuesWithKeyedResource[R, A](resources: RDD[(K, R)], op: (R, V) ⇒ A, numPartitions: Int)(implicit arg0: ClassTag[R]): RDD[(K, A)]

Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
R
the type of resources being used
A
the type returned by applying the operation with the resource to each value
resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
op
the operation to apply to each value. Takes a resource and value and returns the transformed value
numPartitions
the number of partitions for shuffling
returns
PairRDD of values transformed by applying the operation with the appropriate resource
def mapValuesWithKeyedResource[R, A](resources: RDD[(K, R)], op: (R, V) ⇒ A, partitioner: Partitioner)(implicit arg0: ClassTag[R]): RDD[(K, A)]

Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
R
the type of resources being used
A
the type returned by applying the operation with the resource to each value
resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
op
the operation to apply to each value. Takes a resource and value and returns the transformed value
partitioner
the partitioner for shuffling
returns
PairRDD of values transformed by applying the operation with the appropriate resource
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def sortedFoldLeftByKey[A](startValue: A, op: (A, V) ⇒ A): RDD[(K, A)]

Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
A
the result type of the folding operation
startValue
the start value for the fold
op
the binary operation for folding
returns
PairRDD with keys and values, where values are the result of applying foldLeft across the sorted values
def sortedFoldLeftByKey[A](startValue: A, op: (A, V) ⇒ A, numPartitions: Int): RDD[(K, A)]

Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
A
the result type of the folding operation
startValue
the start value for the fold
op
the binary operation for folding
numPartitions
the number of partitions for shuffling
returns
PairRDD with keys and values, where values are the result of applying foldLeft across the sorted values
def sortedFoldLeftByKey[A](startValue: A, op: (A, V) ⇒ A, partitioner: Partitioner): RDD[(K, A)]

Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
A
the result type of the folding operation
startValue
the start value for the fold
op
the binary operation for folding
partitioner
the partitioner for shuffling
returns
PairRDD with keys and values, where values are the result of applying foldLeft across the sorted values
def sortedGroupByKey: RDD[(K, Iterable[V])]

Groups by key and sorts the values by some implicit ordering
Groups by key and sorts the values by some implicit ordering
returns
a PairRDD of keys and sorted values
def sortedGroupByKey(numPartitions: Int): RDD[(K, Iterable[V])]

Groups by key and sorts the values by some implicit ordering
Groups by key and sorts the values by some implicit ordering
numPartitions
the number of partitions for shuffling
returns
PairRDD of keys and sorted values
def sortedGroupByKey(partitioner: Partitioner): RDD[(K, Iterable[V])]

Groups by key and sorts the values by some implicit ordering
Groups by key and sorts the values by some implicit ordering
partitioner
the partitioner for shuffling
returns
PairRDD of keys and sorted values
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package sorting

final class SecondarySortGroupingPairRDDFunctions[K, V] extends Serializable

Instance Constructors

new SecondarySortGroupingPairRDDFunctions(rdd: RDD[(K, V)])(implicit arg0: Ordering[K], arg1: ClassTag[K], arg2: Ordering[V], arg3: ClassTag[V])

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

def mapValuesWithKeyedPreparedResource[R, R1, A](resources: RDD[(K, R)], prepareResource: (R) ⇒ R1, op: (R1, V) ⇒ A)(implicit arg0: ClassTag[R]): RDD[(K, A)]

def mapValuesWithKeyedPreparedResource[R, R1, A](resources: RDD[(K, R)], prepareResource: (R) ⇒ R1, op: (R1, V) ⇒ A, numPartitions: Int)(implicit arg0: ClassTag[R]): RDD[(K, A)]

def mapValuesWithKeyedPreparedResource[R, R1, A](resources: RDD[(K, R)], prepareResource: (R) ⇒ R1, op: (R1, V) ⇒ A, partitioner: Partitioner)(implicit arg0: ClassTag[R]): RDD[(K, A)]

def mapValuesWithKeyedPreparedResource[R, A](resources: RDD[(K, R)], op: (R) ⇒ (V) ⇒ A)(implicit arg0: ClassTag[R]): RDD[(K, A)]

def mapValuesWithKeyedPreparedResource[R, A](resources: RDD[(K, R)], op: (R) ⇒ (V) ⇒ A, numPartitions: Int)(implicit arg0: ClassTag[R]): RDD[(K, A)]

def mapValuesWithKeyedPreparedResource[R, A](resources: RDD[(K, R)], op: (R) ⇒ (V) ⇒ A, partitioner: Partitioner)(implicit arg0: ClassTag[R]): RDD[(K, A)]

def mapValuesWithKeyedResource[R, A](resources: RDD[(K, R)], op: (R, V) ⇒ A)(implicit arg0: ClassTag[R]): RDD[(K, A)]

def mapValuesWithKeyedResource[R, A](resources: RDD[(K, R)], op: (R, V) ⇒ A, numPartitions: Int)(implicit arg0: ClassTag[R]): RDD[(K, A)]

def mapValuesWithKeyedResource[R, A](resources: RDD[(K, R)], op: (R, V) ⇒ A, partitioner: Partitioner)(implicit arg0: ClassTag[R]): RDD[(K, A)]

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def sortedFoldLeftByKey[A](startValue: A, op: (A, V) ⇒ A): RDD[(K, A)]

def sortedFoldLeftByKey[A](startValue: A, op: (A, V) ⇒ A, numPartitions: Int): RDD[(K, A)]

def sortedFoldLeftByKey[A](startValue: A, op: (A, V) ⇒ A, partitioner: Partitioner): RDD[(K, A)]

def sortedGroupByKey: RDD[(K, Iterable[V])]

def sortedGroupByKey(numPartitions: Int): RDD[(K, Iterable[V])]

def sortedGroupByKey(partitioner: Partitioner): RDD[(K, Iterable[V])]

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped