final class SecondarySortGroupingPairRDDFunctions[K, V] extends Serializable
- Alphabetic
- By Inheritance
- SecondarySortGroupingPairRDDFunctions
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new SecondarySortGroupingPairRDDFunctions(rdd: RDD[(K, V)])(implicit arg0: Ordering[K], arg1: ClassTag[K], arg2: Ordering[V], arg3: ClassTag[V])
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def mapValuesWithKeyedPreparedResource[R, R1, A](resources: RDD[(K, R)], prepareResource: (R) => R1, op: (R1, V) => A)(implicit arg0: ClassTag[R]): RDD[(K, A)]
Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
- R
the type of resources being used
- A
the type returned by applying the operation with the resource to each value
- resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
- prepareResource
a function to transform the resource into what will be used in op
- op
the operation to apply to each value. The operation takes a resource and value and returns the transformed value
- returns
PairRDD of values transformed by applying the operation with the appropriate resource
- def mapValuesWithKeyedPreparedResource[R, R1, A](resources: RDD[(K, R)], prepareResource: (R) => R1, op: (R1, V) => A, numPartitions: Int)(implicit arg0: ClassTag[R]): RDD[(K, A)]
Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
- R
the type of resources being used
- A
the type returned by applying the operation with the resource to each value
- resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
- prepareResource
a function to transform the resource into what will be used in op
- op
the operation to apply to each value. The operation takes a resource and value and returns the transformed value
- numPartitions
the number of partitions for shuffling
- returns
PairRDD of values transformed by applying the operation with the appropriate resource
- def mapValuesWithKeyedPreparedResource[R, R1, A](resources: RDD[(K, R)], prepareResource: (R) => R1, op: (R1, V) => A, partitioner: Partitioner)(implicit arg0: ClassTag[R]): RDD[(K, A)]
Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
- R
the type of resources being used
- A
the type returned by applying the operation with the resource to each value
- resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
- prepareResource
a function to transform the resource into what will be used in op
- op
the operation to apply to each value. The operation takes a resource and value and returns the transformed value
- partitioner
the partitioner for shuffling
- returns
PairRDD of values transformed by applying the operation with the appropriate resource
- def mapValuesWithKeyedPreparedResource[R, A](resources: RDD[(K, R)], op: (R) => (V) => A)(implicit arg0: ClassTag[R]): RDD[(K, A)]
Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
- R
the type of resources being used
- A
the type returned by applying the operation with the resource to each value
- resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
- op
The operation to apply to each value. The operation takes a resource and returns a function, which will then be applied to each value.
- returns
PairRDD of values transformed by applying the operation with the appropriate resource
- def mapValuesWithKeyedPreparedResource[R, A](resources: RDD[(K, R)], op: (R) => (V) => A, numPartitions: Int)(implicit arg0: ClassTag[R]): RDD[(K, A)]
Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
- R
the type of resources being used
- A
the type returned by applying the operation with the resource to each value
- resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
- op
The operation to apply to each value. The operation takes a resource and returns a function, which will then be applied to each value.
- numPartitions
the number of partitions for shuffling
- returns
PairRDD of values transformed by applying the operation with the appropriate resource
- def mapValuesWithKeyedPreparedResource[R, A](resources: RDD[(K, R)], op: (R) => (V) => A, partitioner: Partitioner)(implicit arg0: ClassTag[R]): RDD[(K, A)]
Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
- R
the type of resources being used
- A
the type returned by applying the operation with the resource to each value
- resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
- op
The operation to apply to each value. The operation takes a resource and returns a function, which will then be applied to each value.
- partitioner
the partitioner for shuffling
- returns
PairRDD of values transformed by applying the operation with the appropriate resource
- def mapValuesWithKeyedResource[R, A](resources: RDD[(K, R)], op: (R, V) => A)(implicit arg0: ClassTag[R]): RDD[(K, A)]
Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
- R
the type of resources being used
- A
the type returned by applying the operation with the resource to each value
- resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
- op
the operation to apply to each value. Takes a resource and value and returns the transformed value
- returns
PairRDD of values transformed by applying the operation with the appropriate resource
- def mapValuesWithKeyedResource[R, A](resources: RDD[(K, R)], op: (R, V) => A, numPartitions: Int)(implicit arg0: ClassTag[R]): RDD[(K, A)]
Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
- R
the type of resources being used
- A
the type returned by applying the operation with the resource to each value
- resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
- op
the operation to apply to each value. Takes a resource and value and returns the transformed value
- numPartitions
the number of partitions for shuffling
- returns
PairRDD of values transformed by applying the operation with the appropriate resource
- def mapValuesWithKeyedResource[R, A](resources: RDD[(K, R)], op: (R, V) => A, partitioner: Partitioner)(implicit arg0: ClassTag[R]): RDD[(K, A)]
Applies op to every value with some resource, where values and resources share the same key.
Applies op to every value with some resource, where values and resources share the same key. This allows you to send data to executors based on key, so that: (1) the entire set of resources are not held in memory on all executors; and, (2) a specific resource is not duplicated; it is reused for all corresponding data values One example usage might be when conducting a geospatial operation. If the keys indicate a geographic area, and the value contains geospatial resources in that geographic area, one can apply a method using geospatially local resources to all values while reducing data duplication and shuffling.
- R
the type of resources being used
- A
the type returned by applying the operation with the resource to each value
- resources
a PairRDD of keys and resources, where keys are used to determine what data the resource is associated with for the operation. There must be exactly one resource for each key in the RDD this method is applied to
- op
the operation to apply to each value. Takes a resource and value and returns the transformed value
- partitioner
the partitioner for shuffling
- returns
PairRDD of values transformed by applying the operation with the appropriate resource
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def sortedFoldLeftByKey[A](startValues: RDD[(K, A)], op: (A, V) => A)(implicit arg0: ClassTag[A]): RDD[(K, A)]
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
- A
the result type of the folding operation
- startValues
an RDD of start values by key
- op
the binary operation for folding
- returns
PairRDD with keys and values, where values are the result of applying foldLeft across the sorted values
- def sortedFoldLeftByKey[A](startValues: RDD[(K, A)], op: (A, V) => A, numPartitions: Int)(implicit arg0: ClassTag[A]): RDD[(K, A)]
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
- A
the result type of the folding operation
- startValues
an RDD of start values by key
- op
the binary operation for folding
- returns
PairRDD with keys and values, where values are the result of applying foldLeft across the sorted values
- def sortedFoldLeftByKey[A](startValues: RDD[(K, A)], op: (A, V) => A, partitioner: Partitioner)(implicit arg0: ClassTag[A]): RDD[(K, A)]
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
- A
the result type of the folding operation
- startValues
an RDD of start values by key
- op
the binary operation for folding
- returns
PairRDD with keys and values, where values are the result of applying foldLeft across the sorted values
- def sortedFoldLeftByKey[A](startValue: A, op: (A, V) => A): RDD[(K, A)]
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
- A
the result type of the folding operation
- startValue
the start value for the fold
- op
the binary operation for folding
- returns
PairRDD with keys and values, where values are the result of applying foldLeft across the sorted values
- def sortedFoldLeftByKey[A](startValue: A, op: (A, V) => A, numPartitions: Int): RDD[(K, A)]
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
- A
the result type of the folding operation
- startValue
the start value for the fold
- op
the binary operation for folding
- numPartitions
the number of partitions for shuffling
- returns
PairRDD with keys and values, where values are the result of applying foldLeft across the sorted values
- def sortedFoldLeftByKey[A](startValue: A, op: (A, V) => A, partitioner: Partitioner): RDD[(K, A)]
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
Groups by key and applies a binary operation using foldLeft over the values sorted by some implicit ordering
- A
the result type of the folding operation
- startValue
the start value for the fold
- op
the binary operation for folding
- partitioner
the partitioner for shuffling
- returns
PairRDD with keys and values, where values are the result of applying foldLeft across the sorted values
- def sortedGroupByKey: RDD[(K, Iterable[V])]
Groups by key and sorts the values by some implicit ordering
Groups by key and sorts the values by some implicit ordering
- returns
a PairRDD of keys and sorted values
- def sortedGroupByKey(numPartitions: Int): RDD[(K, Iterable[V])]
Groups by key and sorts the values by some implicit ordering
Groups by key and sorts the values by some implicit ordering
- numPartitions
the number of partitions for shuffling
- returns
PairRDD of keys and sorted values
- def sortedGroupByKey(partitioner: Partitioner): RDD[(K, Iterable[V])]
Groups by key and sorts the values by some implicit ordering
Groups by key and sorts the values by some implicit ordering
- partitioner
the partitioner for shuffling
- returns
PairRDD of keys and sorted values
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()