Package org.pipecraft.pipes.source.google_cs

Class GSTxtFileFetcherReaderPipe

java.lang.Object

org.pipecraft.pipes.sync.inter.CompoundPipe<String>

org.pipecraft.pipes.sync.source.StorageTxtFileFetcherReaderPipe

org.pipecraft.pipes.source.google_cs.GSTxtFileFetcherReaderPipe

All Implemented Interfaces:

Closeable, AutoCloseable, org.pipecraft.pipes.BasePipe, org.pipecraft.pipes.sync.Pipe<String>

public class GSTxtFileFetcherReaderPipe
extends org.pipecraft.pipes.sync.source.StorageTxtFileFetcherReaderPipe

A source pipe providing the lines of a text file from Google-Storage. Supports decompression. In contrast to the streaming nature of GSTxtFileReaderPipe, this implementation downloads the file completely first. This approach may be much faster for large files, since the download is optimized (sliced) and reading from local disk is usually faster than network read.

Author:

Eyal Schneider

Constructor Summary

Constructors

Constructor	Description
GSTxtFileFetcherReaderPipe(GoogleStorage gs, String bucket, String path, File tempFolder)	Constructor Uses the default read chunk size and default read options, and assumes UTF8
GSTxtFileFetcherReaderPipe(GoogleStorage gs, String bucket, String path, Charset charset, int chunkSize, org.pipecraft.infra.io.FileReadOptions options, File tempFolder)	Constructor
GSTxtFileFetcherReaderPipe(GoogleStorage gs, String bucket, String path, Charset charset, File tempFolder)	Constructor Uses the default read chunk size and default read options
GSTxtFileFetcherReaderPipe(GoogleStorage gs, String bucket, String path, org.pipecraft.infra.io.FileReadOptions options, File tempFolder)	Constructor Uses the default read chunk size and assumes UTF8

Method Summary

Methods inherited from class org.pipecraft.pipes.sync.source.StorageTxtFileFetcherReaderPipe

close, createPipeline

Methods inherited from class org.pipecraft.pipes.sync.inter.CompoundPipe

getProgress, next, peek, start

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GSTxtFileFetcherReaderPipe

public GSTxtFileFetcherReaderPipe(GoogleStorage gs,
                                  String bucket,
                                  String path,
                                  Charset charset,
                                  int chunkSize,
                                  org.pipecraft.infra.io.FileReadOptions options,
                                  File tempFolder)

Constructor

Parameters:

gs - The Google-Storage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket

charset - The charset used

chunkSize - The size (in bytes) of each chunk read from GS at once, or 0 for using the default one.

options - The file read options

tempFolder - A folder where to place the temporary file for the download. The file is deleted by the pipe at the end of work.

GSTxtFileFetcherReaderPipe

public GSTxtFileFetcherReaderPipe(GoogleStorage gs,
                                  String bucket,
                                  String path,
                                  Charset charset,
                                  File tempFolder)

Constructor Uses the default read chunk size and default read options

Parameters:

gs - The Google-Storage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket

charset - The charset used

tempFolder - A folder where to place the temporary file for the download. The file is deleted by the pipe at the end of work.

GSTxtFileFetcherReaderPipe

public GSTxtFileFetcherReaderPipe(GoogleStorage gs,
                                  String bucket,
                                  String path,
                                  File tempFolder)

Constructor Uses the default read chunk size and default read options, and assumes UTF8

Parameters:

gs - The Google-Storage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket. The file is assumed to be UTF8.

tempFolder - A folder where to place the temporary file for the download. The file is deleted by the pipe at the end of work.

GSTxtFileFetcherReaderPipe

public GSTxtFileFetcherReaderPipe(GoogleStorage gs,
                                  String bucket,
                                  String path,
                                  org.pipecraft.infra.io.FileReadOptions options,
                                  File tempFolder)

Constructor Uses the default read chunk size and assumes UTF8

Parameters:

gs - The Google-Storage connector

bucket - The bucket to read the file from

path - The full path of the text file inside the bucket. The file is assumed to be UTF8.

options - The file read options

tempFolder - A folder where to place the temporary file for the download. The file is deleted by the pipe at the end of work.