cormoran.pepper.parquet

Class ParquetStreamFactory

java.lang.Object

cormoran.pepper.parquet.ParquetStreamFactory

All Implemented Interfaces:

IAvroStreamFactory

public class ParquetStreamFactory
extends Object
implements IAvroStreamFactory

Enable converting a Parquet file to a Stream of Map

Author:

Benoit Lacelle

Constructor Summary

Constructors

Constructor and Description
ParquetStreamFactory()
ParquetStreamFactory(org.apache.hadoop.conf.Configuration configuration)

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.conf.Configuration	cloneDefaultConfiguration()
org.apache.hadoop.conf.Configuration	getConfiguration()
protected org.apache.parquet.filter2.compat.FilterCompat.Filter	makeFilter()
static Stream<Map<String,?>>	readParquetAsStream(Path pathOnDisk, Map<String,?> exampleTypes)
protected org.apache.hadoop.fs.Path	toHadoopPath(Path javaPathOnDisk)
protected Stream<org.apache.avro.generic.GenericRecord>	toStream(org.apache.parquet.hadoop.ParquetReader<org.apache.avro.generic.GenericRecord> reader)
Stream<org.apache.avro.generic.GenericRecord>	toStream(Path javaPath)
Stream<org.apache.avro.generic.GenericRecord>	toStream(org.apache.hadoop.fs.Path hadoopPath)
Stream<? extends Map<String,?>>	toStream(Path path, Map<String,?> exampleTypes)
long	writeToPath(Path javaPathOnDisk, Stream<? extends org.apache.avro.generic.GenericRecord> rowsToWrite)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetStreamFactory

public ParquetStreamFactory()

ParquetStreamFactory

public ParquetStreamFactory(org.apache.hadoop.conf.Configuration configuration)

Method Detail

cloneDefaultConfiguration

public static org.apache.hadoop.conf.Configuration cloneDefaultConfiguration()

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

toStream

public Stream<org.apache.avro.generic.GenericRecord> toStream(Path javaPath)
                                                       throws IOException

Specified by:

toStream in interface IAvroStreamFactory

Throws:

IOException

toStream

public Stream<org.apache.avro.generic.GenericRecord> toStream(org.apache.hadoop.fs.Path hadoopPath)
                                                       throws IOException

Throws:

IOException

toStream

public Stream<? extends Map<String,?>> toStream(Path path,
                                                Map<String,?> exampleTypes)
                                         throws IOException

Throws:

IOException

makeFilter

protected org.apache.parquet.filter2.compat.FilterCompat.Filter makeFilter()

toStream

protected Stream<org.apache.avro.generic.GenericRecord> toStream(org.apache.parquet.hadoop.ParquetReader<org.apache.avro.generic.GenericRecord> reader)

readParquetAsStream

public static Stream<Map<String,?>> readParquetAsStream(Path pathOnDisk,
                                                        Map<String,?> exampleTypes)
                                                 throws FileNotFoundException,
                                                        IOException

Throws:

FileNotFoundException

IOException

writeToPath

public long writeToPath(Path javaPathOnDisk,
                        Stream<? extends org.apache.avro.generic.GenericRecord> rowsToWrite)
                 throws IOException

Specified by:

writeToPath in interface IAvroStreamFactory

Throws:

IOException

toHadoopPath

protected org.apache.hadoop.fs.Path toHadoopPath(Path javaPathOnDisk)