org.bytedeco.arrow_dataset

Class ParquetFileFormat

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow_dataset.FileFormat

org.bytedeco.arrow_dataset.ParquetFileFormat

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow::dataset")
 @Properties(inherit=arrow_dataset.class)
public class ParquetFileFormat
extends FileFormat

\brief A FileFormat implementation that reads from Parquet files

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
ParquetFileFormat()
ParquetFileFormat(long size) Native array allocator.
ParquetFileFormat(Pointer p) Pointer cast constructor.
ParquetFileFormat(ReaderProperties reader_properties) Convenience constructor which copies properties from a parquet::ReaderProperties.

Method Summary

Modifier and Type	Method and Description
ParquetFileFormat	getPointer(long i)
FileReaderResult	GetReader(FileSource source)
FileReaderResult	GetReader(FileSource source, ScanOptions arg1, ScanContext arg2) \brief Return a FileReader on the given source.
SchemaResult	Inspect(FileSource source) \brief Return the schema of the file if possible.
BoolResult	IsSupported(FileSource source) \}
FileFragmentResult	MakeFragment(FileSource source, Expression partition_expression, RowGroupInfo row_groups)
FileFragmentResult	MakeFragment(FileSource source, Expression partition_expression, RowGroupInfo row_groups, Schema physical_schema) \brief Create a Fragment, restricted to the specified row groups.
FileFragmentResult	MakeFragment(FileSource source, Expression partition_expression, Schema physical_schema) \brief Create a Fragment targeting all RowGroups.
ParquetFileFormat	position(long position)
long	reader_options_buffer_size()
ParquetFileFormat	reader_options_buffer_size(long setter)
StringUnorderedSet	reader_options_dict_columns() \}
ParquetFileFormat	reader_options_dict_columns(StringUnorderedSet setter)
FileDecryptionProperties	reader_options_file_decryption_properties()
ParquetFileFormat	reader_options_file_decryption_properties(FileDecryptionProperties setter)
boolean	reader_options_use_buffered_stream() \defgroup parquet-file-format-reader-properties properties which correspond to members of parquet::ReaderProperties.
ParquetFileFormat	reader_options_use_buffered_stream(boolean setter)
ScanTaskIteratorResult	ScanFile(ScanOptions options, ScanContext context, FileFragment file) \brief Open a file for scanning
boolean	splittable() \brief Return true if fragments of this format can benefit from parallel scanning.
String	type_name() \brief The name identifying the kind of file format

Methods inherited from class org.bytedeco.arrow_dataset.FileFormat

MakeFragment, MakeFragment, MakeFragment, WriteFragment

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getPointer, getPointer, getPointer, hashCode, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetof, offsetof, parseBytes, physicalBytes, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ParquetFileFormat

public ParquetFileFormat(Pointer p)

Pointer cast constructor. Invokes Pointer.Pointer(Pointer).

ParquetFileFormat

public ParquetFileFormat(long size)

Native array allocator. Access with Pointer.position(long).

ParquetFileFormat

public ParquetFileFormat()

ParquetFileFormat

public ParquetFileFormat(@Const @ByRef
                         ReaderProperties reader_properties)

Convenience constructor which copies properties from a parquet::ReaderProperties. memory_pool will be ignored.

Method Detail

position

public ParquetFileFormat position(long position)

Overrides:

position in class Pointer

getPointer

public ParquetFileFormat getPointer(long i)

Overrides:

getPointer in class Pointer

type_name

@StdString
public String type_name()

Description copied from class: FileFormat

\brief The name identifying the kind of file format

Overrides:

type_name in class FileFormat

splittable

@Cast(value="bool")
public boolean splittable()

Description copied from class: FileFormat

\brief Return true if fragments of this format can benefit from parallel scanning.

Overrides:

splittable in class FileFormat

reader_options_use_buffered_stream

@Name(value="reader_options.use_buffered_stream")
 @Cast(value="bool")
public boolean reader_options_use_buffered_stream()

\defgroup parquet-file-format-reader-properties properties which correspond to members of parquet::ReaderProperties. We don't embed parquet::ReaderProperties directly because we get memory_pool from ScanContext at scan time and provide differing defaults. \{

reader_options_use_buffered_stream

public ParquetFileFormat reader_options_use_buffered_stream(boolean setter)

reader_options_buffer_size

@Name(value="reader_options.buffer_size")
 @Cast(value="int64_t")
public long reader_options_buffer_size()

reader_options_buffer_size

public ParquetFileFormat reader_options_buffer_size(long setter)

reader_options_file_decryption_properties

@Name(value="reader_options.file_decryption_properties")
 @SharedPtr
public FileDecryptionProperties reader_options_file_decryption_properties()

reader_options_file_decryption_properties

public ParquetFileFormat reader_options_file_decryption_properties(FileDecryptionProperties setter)

reader_options_dict_columns

@Name(value="reader_options.dict_columns")
 @ByRef
public StringUnorderedSet reader_options_dict_columns()

\}

\defgroup parquet-file-format-arrow-reader-properties properties which correspond to members of parquet::ArrowReaderProperties. We don't embed parquet::ReaderProperties directly because we get batch_size from ScanOptions at scan time, and we will never pass use_threads == true (since we defer parallelization of the scan). Additionally column names (rather than indices) are used to indicate dictionary columns. \{

reader_options_dict_columns

public ParquetFileFormat reader_options_dict_columns(StringUnorderedSet setter)

IsSupported

@ByVal
public BoolResult IsSupported(@Const @ByRef
                                      FileSource source)

\}

Overrides:

IsSupported in class FileFormat

Inspect

@ByVal
public SchemaResult Inspect(@Const @ByRef
                                    FileSource source)

\brief Return the schema of the file if possible.

Overrides:

Inspect in class FileFormat

ScanFile

@ByVal
public ScanTaskIteratorResult ScanFile(@SharedPtr
                                               ScanOptions options,
                                               @SharedPtr
                                               ScanContext context,
                                               FileFragment file)

\brief Open a file for scanning

Overrides:

ScanFile in class FileFormat

MakeFragment

@ByVal
public FileFragmentResult MakeFragment(@ByVal
                                               FileSource source,
                                               @SharedPtr @ByVal
                                               Expression partition_expression,
                                               @StdVector
                                               RowGroupInfo row_groups,
                                               @SharedPtr @ByVal(nullValue="std::shared_ptr<arrow::Schema>(nullptr)")
                                               Schema physical_schema)

\brief Create a Fragment, restricted to the specified row groups.

MakeFragment

@ByVal
public FileFragmentResult MakeFragment(@ByVal
                                               FileSource source,
                                               @SharedPtr @ByVal
                                               Expression partition_expression,
                                               @StdVector
                                               RowGroupInfo row_groups)

MakeFragment

@ByVal
public FileFragmentResult MakeFragment(@ByVal
                                               FileSource source,
                                               @SharedPtr @ByVal
                                               Expression partition_expression,
                                               @SharedPtr @ByVal
                                               Schema physical_schema)

\brief Create a Fragment targeting all RowGroups.

Overrides:

MakeFragment in class FileFormat

GetReader

@ByVal
public FileReaderResult GetReader(@Const @ByRef
                                          FileSource source,
                                          ScanOptions arg1,
                                          ScanContext arg2)

\brief Return a FileReader on the given source.

GetReader

@ByVal
public FileReaderResult GetReader(@Const @ByRef
                                          FileSource source)