Package org.pipecraft.pipes.sync.source

Class StorageMultiFileReaderPipe<T,B>

java.lang.Object

org.pipecraft.pipes.sync.inter.CompoundPipe<T>

org.pipecraft.pipes.sync.source.StorageMultiFileReaderPipe<T,B>

Type Parameters:

T - The output item data type

B - The remote file metadata object type

All Implemented Interfaces:

Closeable, AutoCloseable, BasePipe, Pipe<T>

public class StorageMultiFileReaderPipe<T,B>
extends CompoundPipe<T>

A source pipe reading multiple remote binary files from the cloud. Supports file filtering and ordering, as well as automatic sharding of the data.

Author:

Eyal Schneider

Constructor Summary

Constructors

Constructor	Description
StorageMultiFileReaderPipe(StorageMultiFileReaderConfig<T,B> config)	Constructor

Method Summary

Modifier and Type	Method	Description
protected Pipe<T>	createPipeline()

Methods inherited from class org.pipecraft.pipes.sync.inter.CompoundPipe

close, getProgress, next, peek, start

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StorageMultiFileReaderPipe

public StorageMultiFileReaderPipe(StorageMultiFileReaderConfig<T,B> config)

Constructor

Parameters:

config - The multi reader config

Method Detail

createPipeline

protected Pipe<T> createPipeline()
                          throws PipeException,
                                 InterruptedException

Specified by:

createPipeline in class CompoundPipe<T>

Returns:

A new pipeline to represent the logic of this pipe

Throws:

PipeException - In case of a pipeline creation error

InterruptedException - In case that the thread is interrupted