Package org.pipecraft.pipes.terminal

Class StorageTxtFileWriterPipe

java.lang.Object

org.pipecraft.pipes.terminal.TerminalPipe

org.pipecraft.pipes.terminal.StorageTxtFileWriterPipe

All Implemented Interfaces:

Closeable, AutoCloseable, BasePipe

public class StorageTxtFileWriterPipe
extends TerminalPipe

A terminal pipe writing text lines from the input pipe into a remote cloud storage file. Supports compression. Note that this class makes use of upload using IO streams, which is an optional feature of Bucket class. Therefore, some types of Storage won't be supported here.

Author:

Eyal Schneider

Constructor Summary

Constructors

Constructor	Description
StorageTxtFileWriterPipe(Pipe<String> input, Storage<?,?> storage, String bucket, String path)	Constructor Uses default write options and chunk size, and assumes UTF8
StorageTxtFileWriterPipe(Pipe<String> input, Storage<?,?> gs, String bucket, String path, Charset charset)	Constructor Uses the default write options and chunk size
StorageTxtFileWriterPipe(Pipe<String> input, Storage<?,?> storage, String bucket, String path, Charset charset, int chunkSize, FileWriteOptions options)	Constructor
StorageTxtFileWriterPipe(Pipe<String> input, Storage<?,?> storage, String bucket, String path, FileWriteOptions options)	Constructor Uses default chunk size and UTF8

Method Summary

Modifier and Type	Method	Description
void	close()
void	start()	Performs pre-processing prior to item flow throw the pipe.

Methods inherited from class org.pipecraft.pipes.terminal.TerminalPipe

getProgress

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StorageTxtFileWriterPipe

public StorageTxtFileWriterPipe(Pipe<String> input,
                                Storage<?,?> storage,
                                String bucket,
                                String path,
                                Charset charset,
                                int chunkSize,
                                FileWriteOptions options)

Constructor

Parameters:

input - The input pipe

storage - The cloud storage connector

bucket - The bucket to write to

path - The full path (excluding the bucket) of the target file

chunkSize - The size (in bytes) of each chunk written to GS at once, or 0 for using the default one.

charset - The charset to use

options - The file writing options. Note that append and temp aren't supported by GS, so these flag are ignored.

StorageTxtFileWriterPipe

public StorageTxtFileWriterPipe(Pipe<String> input,
                                Storage<?,?> gs,
                                String bucket,
                                String path,
                                Charset charset)

Constructor Uses the default write options and chunk size

Parameters:

input - The input pipe

gs - The cloud storage connector

bucket - The bucket to write to

path - The full path (excluding the bucket) of the target file

charset - The charset to use

StorageTxtFileWriterPipe

public StorageTxtFileWriterPipe(Pipe<String> input,
                                Storage<?,?> storage,
                                String bucket,
                                String path,
                                FileWriteOptions options)

Constructor Uses default chunk size and UTF8

Parameters:

input - The input pipe

storage - The cloud storage connector

bucket - The bucket to write to

path - The full path (excluding the bucket) of the target file

options - The file writing options. Note that append isn't supported by GS, so this flag is ignored.

StorageTxtFileWriterPipe

public StorageTxtFileWriterPipe(Pipe<String> input,
                                Storage<?,?> storage,
                                String bucket,
                                String path)

Constructor Uses default write options and chunk size, and assumes UTF8

Parameters:

input - The input pipe

storage - The cloud storage connector

bucket - The bucket to write to

path - The full path (excluding the bucket) of the target file

Method Detail

start

public void start()
           throws PipeException,
                  InterruptedException

Description copied from interface: BasePipe

Performs pre-processing prior to item flow throw the pipe. Implementations must call the same method for all their input pipes before accessing their items. This is typically done here.

Throws:

PipeException - In case of pipe errors in this pipe or somewhere up-stream.

InterruptedException - In case that the operation has been interrupted by another thread.

close

public void close()
           throws IOException

Throws:

IOException