cormoran.pepper.parquet

Class ParquetStreamFactory

java.lang.Object

cormoran.pepper.avro.AvroStreamFactory

cormoran.pepper.parquet.ParquetStreamFactory

All Implemented Interfaces:

IAvroStreamFactory

public class ParquetStreamFactory
extends AvroStreamFactory

Enable converting a Parquet file to a Stream of Map

Author:

Benoit Lacelle

Field Summary

Fields inherited from class cormoran.pepper.avro.AvroStreamFactory

LOGGER

Constructor Summary

Constructors

Constructor and Description
ParquetStreamFactory()
ParquetStreamFactory(org.apache.hadoop.conf.Configuration configuration)

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.conf.Configuration	cloneDefaultConfiguration()
org.apache.hadoop.conf.Configuration	getConfiguration()
protected org.apache.parquet.filter2.compat.FilterCompat.Filter	makeFilter()
protected IGenericRecordConsumer	prepareRecordConsumer(org.apache.avro.Schema schema, URI uri)
static Stream<Map<String,?>>	readParquetAsStream(URI uriToParquet, Map<String,?> exampleTypes)
Stream<org.apache.avro.generic.GenericRecord>	stream(InputStream rawInputStream)
Stream<org.apache.avro.generic.GenericRecord>	stream(URI uri)
protected org.apache.hadoop.fs.Path	toHadoopPath(URI uri)
protected Stream<org.apache.avro.generic.GenericRecord>	toStream(org.apache.parquet.hadoop.ParquetReader<org.apache.avro.generic.GenericRecord> reader)
Stream<org.apache.avro.generic.GenericRecord>	toStream(org.apache.hadoop.fs.Path hadoopPath)

Methods inherited from class cormoran.pepper.avro.AvroStreamFactory

outputStream, prepareRecordConsumer, serialize, serialize, transcode

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface cormoran.pepper.avro.IAvroStreamFactory

toStream, toStream, writeToPath, writeToPath

Constructor Detail

ParquetStreamFactory

public ParquetStreamFactory()

ParquetStreamFactory

public ParquetStreamFactory(org.apache.hadoop.conf.Configuration configuration)

Method Detail

cloneDefaultConfiguration

public static org.apache.hadoop.conf.Configuration cloneDefaultConfiguration()

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

stream

public Stream<org.apache.avro.generic.GenericRecord> stream(URI uri)
                                                     throws IOException

Specified by:

stream in interface IAvroStreamFactory

Overrides:

stream in class AvroStreamFactory

Throws:

IOException

stream

public Stream<org.apache.avro.generic.GenericRecord> stream(InputStream rawInputStream)
                                                     throws IOException

Parameters:

rawInputStream - a stream of bytes associated to a Parquet file. It is sub-optimal as Parquet require a SeekableInputStream (i.e. an InputStream with RandomAccess). We will then copy the file in local FS

Returns:

Throws:

IOException

toStream

public Stream<org.apache.avro.generic.GenericRecord> toStream(org.apache.hadoop.fs.Path hadoopPath)
                                                       throws IOException

Throws:

IOException

makeFilter

protected org.apache.parquet.filter2.compat.FilterCompat.Filter makeFilter()

toStream

protected Stream<org.apache.avro.generic.GenericRecord> toStream(org.apache.parquet.hadoop.ParquetReader<org.apache.avro.generic.GenericRecord> reader)

readParquetAsStream

public static Stream<Map<String,?>> readParquetAsStream(URI uriToParquet,
                                                        Map<String,?> exampleTypes)
                                                 throws FileNotFoundException,
                                                        IOException

Throws:

FileNotFoundException

IOException

prepareRecordConsumer

protected IGenericRecordConsumer prepareRecordConsumer(org.apache.avro.Schema schema,
                                                       URI uri)
                                                throws IOException

Overrides:

prepareRecordConsumer in class AvroStreamFactory

Throws:

IOException

toHadoopPath

protected org.apache.hadoop.fs.Path toHadoopPath(URI uri)