Package org.pipecraft.pipes.terminal

Class AsyncSharderByHashPipe<T>

java.lang.Object

org.pipecraft.pipes.terminal.TerminalPipe

org.pipecraft.pipes.terminal.AsyncSharderPipe<T>

org.pipecraft.pipes.terminal.AsyncSharderByHashPipe<T>

Type Parameters:

T - The items' data type

All Implemented Interfaces:

Closeable, AutoCloseable, BasePipe

public class AsyncSharderByHashPipe<T>
extends AsyncSharderPipe<T>

An async sharder pipe, writing items to files based on the hash value of individual items. The callers can't assume anything regarding the hash function used internally.

Author:

Eyal Schneider

Constructor Summary

Constructors

Constructor	Description
AsyncSharderByHashPipe(AsyncPipe<T> input, EncoderFactory<? super T> encoderFactory, int shardCount, File folder)	Constructor Uses default file write options.
AsyncSharderByHashPipe(AsyncPipe<T> input, EncoderFactory<? super T> encoderFactory, int shardCount, File folder, FileWriteOptions writeOptions)	Constructor

Method Summary

Methods inherited from class org.pipecraft.pipes.terminal.AsyncSharderPipe

close, getProgress, getShardSizes, start

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AsyncSharderByHashPipe

public AsyncSharderByHashPipe(AsyncPipe<T> input,
                              EncoderFactory<? super T> encoderFactory,
                              int shardCount,
                              File folder,
                              FileWriteOptions writeOptions)

Constructor

Parameters:

input - The input pipe

encoderFactory - The encoder factory to use for writing items into the different shards

shardCount - The required number of shards (files) to produce. Their names will be numbers in the rage 0 to shardCount - 1.

folder - The folder where to place all shards. Must exist.

writeOptions - Specify how the shard files should be written

AsyncSharderByHashPipe

public AsyncSharderByHashPipe(AsyncPipe<T> input,
                              EncoderFactory<? super T> encoderFactory,
                              int shardCount,
                              File folder)

Constructor Uses default file write options.

Parameters:

input - The input pipe

encoderFactory - The encoder to use for writing items into the different shards

shardCount - The required number of shards (files) to produce. Their names will be numbers in the rage 0 to shardCount - 1.

folder - The folder where to place all shards. Must exist.