Package org.pipecraft.pipes.utils.multi

Class MultiFileReaderUtils

java.lang.Object

org.pipecraft.pipes.utils.multi.MultiFileReaderUtils

public class MultiFileReaderUtils
extends Object

Shared helper methods for multi-file reader pipes.

Author:

Eyal Schneider

Constructor Summary

Constructors

Constructor	Description
MultiFileReaderUtils()

Method Summary

Modifier and Type	Method	Description
static <T,B> List<PipeSupplier<T>>	downloadAndGetReadPipes(Collection<B> files, Bucket<B> bucket, StorageMultiFileReaderConfig<T,B> config)
static <T> Collection<File>	getAllLocalFilesToRead(LocalMultiFileReaderConfig<T> config)
static <T,B> Collection<B>	getAllRemoteFilesToRead(StorageMultiFileReaderConfig<T,B> config)
static boolean	isOwned(String filePath, ShardSpecifier shardSpecifier)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultiFileReaderUtils

public MultiFileReaderUtils()

Method Detail

isOwned

public static boolean isOwned(String filePath,
                              ShardSpecifier shardSpecifier)

Parameters:

filePath - File path to check against the given shard

shardSpecifier - The shard

Returns:

true if and only if the file belongs to the given shard. Shards are determined by hashing of the file paths.

getAllLocalFilesToRead

public static <T> Collection<File> getAllLocalFilesToRead(LocalMultiFileReaderConfig<T> config)
                                                   throws IOException

Parameters:

config - The configuration of the multi file reader, as LocalMultiFileReaderConfig

Returns:

the set of local files that the reader should read, according to the locations and filters specified in the config

Throws:

IOException - If an IO error occures while listing local files

getAllRemoteFilesToRead

public static <T,B> Collection<B> getAllRemoteFilesToRead(StorageMultiFileReaderConfig<T,B> config)
                                                         throws IOException

Parameters:

config - The configuration of the multi file reader, as LocalMultiFileReaderConfig

Returns:

the set of remote files that the reader should read, according to the locations and filters specified in the config

Throws:

IOException - If an IO error occures while listing remote files

downloadAndGetReadPipes

public static <T,B> List<PipeSupplier<T>> downloadAndGetReadPipes(Collection<B> files,
                                                                        Bucket<B> bucket,
                                                                        StorageMultiFileReaderConfig<T,B> config)
                                                                 throws IOException,
                                                                        InterruptedException

Throws:

IOException

InterruptedException