Package org.pipecraft.pipes.sync.source

Class StorageTxtFileFetcherReaderPipe

java.lang.Object

org.pipecraft.pipes.sync.inter.CompoundPipe<String>

org.pipecraft.pipes.sync.source.StorageTxtFileFetcherReaderPipe

All Implemented Interfaces:

Closeable, AutoCloseable, BasePipe, Pipe<String>

public class StorageTxtFileFetcherReaderPipe
extends CompoundPipe<String>

A source pipe providing the lines of a text file from cloud storage. Supports decompression. In contrast to the streaming nature of StorageTxtFileReaderPipe, this implementation downloads the file completely first. This approach may be much faster for large files, since the download is optimized (sliced) and reading from local disk is usually faster than network read.

Author:

Eyal Schneider

Constructor Summary

Constructors

Constructor	Description
StorageTxtFileFetcherReaderPipe(Storage<?,?> storage, String bucket, String path, File tempFolder)	Constructor Uses the default read chunk size and default read options, and assumes UTF8
StorageTxtFileFetcherReaderPipe(Storage<?,?> storage, String bucket, String path, Charset charset, int chunkSize, FileReadOptions options, File tempFolder)	Constructor
StorageTxtFileFetcherReaderPipe(Storage<?,?> storage, String bucket, String path, Charset charset, File tempFolder)	Constructor Uses the default read chunk size and default read options
StorageTxtFileFetcherReaderPipe(Storage<?,?> storage, String bucket, String path, FileReadOptions options, File tempFolder)	Constructor Uses the default read chunk size and assumes UTF8

Method Summary

Modifier and Type	Method	Description
void	close()
protected Pipe<String>	createPipeline()

Methods inherited from class org.pipecraft.pipes.sync.inter.CompoundPipe

getProgress, next, peek, start

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StorageTxtFileFetcherReaderPipe

public StorageTxtFileFetcherReaderPipe(Storage<?,?> storage,
                                       String bucket,
                                       String path,
                                       Charset charset,
                                       int chunkSize,
                                       FileReadOptions options,
                                       File tempFolder)

Constructor

Parameters:

storage - The cloud storage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket

charset - The charset used

chunkSize - The size (in bytes) of each chunk read from storage at once, or 0 for using the default one.

options - The file read options

tempFolder - A folder where to place the temporary file for the download. The file is deleted by the pipe at the end of work.

StorageTxtFileFetcherReaderPipe

public StorageTxtFileFetcherReaderPipe(Storage<?,?> storage,
                                       String bucket,
                                       String path,
                                       Charset charset,
                                       File tempFolder)

Constructor Uses the default read chunk size and default read options

Parameters:

storage - The cloud storage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket

charset - The charset used

tempFolder - A folder where to place the temporary file for the download. The file is deleted by the pipe at the end of work.

StorageTxtFileFetcherReaderPipe

public StorageTxtFileFetcherReaderPipe(Storage<?,?> storage,
                                       String bucket,
                                       String path,
                                       File tempFolder)

Constructor Uses the default read chunk size and default read options, and assumes UTF8

Parameters:

storage - The cloud storage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket. The file is assumed to be UTF8.

tempFolder - A folder where to place the temporary file for the download. The file is deleted by the pipe at the end of work.

StorageTxtFileFetcherReaderPipe

public StorageTxtFileFetcherReaderPipe(Storage<?,?> storage,
                                       String bucket,
                                       String path,
                                       FileReadOptions options,
                                       File tempFolder)

Constructor Uses the default read chunk size and assumes UTF8

Parameters:

storage - The cloud torage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket. The file is assumed to be UTF8.

options - The file read options

tempFolder - A folder where to place the temporary file for the download. The file is deleted by the pipe at the end of work.

Method Detail

createPipeline

protected Pipe<String> createPipeline()
                               throws PipeException,
                                      InterruptedException

Specified by:

createPipeline in class CompoundPipe<String>

Returns:

A new pipeline to represent the logic of this pipe

Throws:

PipeException - In case of a pipeline creation error

InterruptedException - In case that the thread is interrupted

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Overrides:

close in class CompoundPipe<String>

Throws:

IOException