Package org.pipecraft.pipes.sync.source

Class StorageTxtFileReaderPipe

java.lang.Object

org.pipecraft.pipes.sync.source.InputStreamPipe<String>

org.pipecraft.pipes.sync.source.StorageTxtFileReaderPipe

All Implemented Interfaces:

Closeable, AutoCloseable, BasePipe, Pipe<String>

public class StorageTxtFileReaderPipe
extends InputStreamPipe<String>

A source pipe providing the lines of a text file in some Storage, in a streaming manner. Supports decompression. Note that for larger files StorageTxtFileFetcherReaderPipe may be much faster.

Author:

Eyal Schneider

Constructor Summary

Constructors

Constructor	Description
StorageTxtFileReaderPipe(Storage<?,?> storageConnector, String bucket, String path)	Constructor Uses the default read chunk size and default read options, and assumes UTF8
StorageTxtFileReaderPipe(Storage<?,?> storageConnector, String bucket, String path, Charset charset)	Constructor Uses the default read chunk size and default read options
StorageTxtFileReaderPipe(Storage<?,?> storageConnector, String bucketName, String path, Charset charset, int chunkSize, FileReadOptions options)	Constructor
StorageTxtFileReaderPipe(Storage<?,?> storageConnector, String bucket, String path, FileReadOptions options)	Constructor Uses the default read chunk size and assumes UTF8

Method Summary

Modifier and Type	Method	Description
void	close()
protected SizedInputStream	createInputStream()
String	next()
String	peek()
void	start()	Performs pre-processing prior to item flow throw the pipe.

Methods inherited from class org.pipecraft.pipes.sync.source.InputStreamPipe

getInputStream, getInputStreamSize, getProgress

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StorageTxtFileReaderPipe

public StorageTxtFileReaderPipe(Storage<?,?> storageConnector,
                                String bucketName,
                                String path,
                                Charset charset,
                                int chunkSize,
                                FileReadOptions options)

Constructor

Parameters:

storageConnector - The Storage connector

bucketName - The bucket to read the file from

path - The full path of the text file inside the bucket

charset - The charset used

chunkSize - The size (in bytes) of each chunk read from storage at once, or 0 for using the default one.

options - The file read options

StorageTxtFileReaderPipe

public StorageTxtFileReaderPipe(Storage<?,?> storageConnector,
                                String bucket,
                                String path,
                                Charset charset)

Constructor Uses the default read chunk size and default read options

Parameters:

storageConnector - The Storage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket

charset - The charset used

StorageTxtFileReaderPipe

public StorageTxtFileReaderPipe(Storage<?,?> storageConnector,
                                String bucket,
                                String path)

Constructor Uses the default read chunk size and default read options, and assumes UTF8

Parameters:

storageConnector - The Storage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket. The file is assumed to be UTF8.

StorageTxtFileReaderPipe

public StorageTxtFileReaderPipe(Storage<?,?> storageConnector,
                                String bucket,
                                String path,
                                FileReadOptions options)

Constructor Uses the default read chunk size and assumes UTF8

Parameters:

storageConnector - The Storage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket. The file is assumed to be UTF8.

options - The file read options

Method Detail

next

public String next()
            throws PipeException,
                   InterruptedException

Returns:

The next item in this pipe output, or null if the output end has been reached. May be a blocking operation.

Throws:

PipeException - In case of pipe errors in this pipe or somewhere up-stream while trying to prepare next item to return.

InterruptedException - In case that the operation has been interrupted by another thread.

peek

public String peek()

Returns:

The next item in the pipe's output. Does not remove it, so next call to next() will return it.

createInputStream

protected SizedInputStream createInputStream()
                                      throws IOException,
                                             InterruptedException

Specified by:

createInputStream in class InputStreamPipe<String>

Returns:

A new input stream based on ctor arguments. Called once in the start() method. Expected to be the raw input stream (without decompression or buffering layers), but should be initialized with expected size (see SizedInputStream). The size should be null if unknown.

Throws:

IOException - (Will be wrapped by IOPipeException)

InterruptedException - In case that the operation is interrupted

start

public void start()
           throws PipeException,
                  InterruptedException

Description copied from interface: BasePipe

Performs pre-processing prior to item flow throw the pipe. Implementations must call the same method for all their input pipes before accessing their items. This is typically done here.

Specified by:

start in interface BasePipe

Overrides:

start in class InputStreamPipe<String>

Throws:

PipeException - In case of pipe errors in this pipe or somewhere up-stream.

InterruptedException - In case that the operation has been interrupted by another thread.

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Overrides:

close in class InputStreamPipe<String>

Throws:

IOException