org.bytedeco.arrow_dataset

Class ParquetFileFormat

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow_dataset.FileFormat

org.bytedeco.arrow_dataset.ParquetFileFormat

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow::dataset")
 @Properties(inherit=arrow_dataset.class)
public class ParquetFileFormat
extends FileFormat

\brief A FileFormat implementation that reads from Parquet files

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
ParquetFileFormat()
ParquetFileFormat(long size) Native array allocator.
ParquetFileFormat(Pointer p) Pointer cast constructor.
ParquetFileFormat(ReaderProperties reader_properties) Convenience constructor which copies properties from a parquet::ReaderProperties.

Method Summary

Modifier and Type	Method and Description
FileWriteOptions	DefaultWriteOptions()
boolean	Equals(FileFormat other)
ParquetFileFormat	getPointer(long i)
FileReaderResult	GetReader(FileSource source)
FileReaderResult	GetReader(FileSource source, ScanOptions arg1, ScanContext arg2) \brief Return a FileReader on the given source.
SchemaResult	Inspect(FileSource source) \brief Return the schema of the file if possible.
BoolResult	IsSupported(FileSource source) \brief Indicate if the FileSource is supported/readable by this format.
FileFragmentResult	MakeFragment(FileSource source, Expression partition_expression, Schema physical_schema) \brief Create a Fragment targeting all RowGroups.
ParquetFileFragmentResult	MakeFragment(FileSource source, Expression partition_expression, Schema physical_schema, int[] row_groups)
ParquetFileFragmentResult	MakeFragment(FileSource source, Expression partition_expression, Schema physical_schema, IntBuffer row_groups)
ParquetFileFragmentResult	MakeFragment(FileSource source, Expression partition_expression, Schema physical_schema, IntPointer row_groups) \brief Create a Fragment, restricted to the specified row groups.
FileWriterResult	MakeWriter(OutputStream destination, Schema schema, FileWriteOptions options)
ParquetFileFormat	position(long position)
long	reader_options_buffer_size()
ParquetFileFormat	reader_options_buffer_size(long setter)
StringUnorderedSet	reader_options_dict_columns() \}
ParquetFileFormat	reader_options_dict_columns(StringUnorderedSet setter)
boolean	reader_options_enable_parallel_column_conversion() \}
ParquetFileFormat	reader_options_enable_parallel_column_conversion(boolean setter)
FileDecryptionProperties	reader_options_file_decryption_properties()
ParquetFileFormat	reader_options_file_decryption_properties(FileDecryptionProperties setter)
boolean	reader_options_use_buffered_stream() \defgroup parquet-file-format-reader-properties properties which correspond to members of parquet::ReaderProperties.
ParquetFileFormat	reader_options_use_buffered_stream(boolean setter)
ScanTaskIteratorResult	ScanFile(ScanOptions options, ScanContext context, FileFragment file) \brief Open a file for scanning
boolean	splittable() \brief Return true if fragments of this format can benefit from parallel scanning.
String	type_name() \brief The name identifying the kind of file format

Methods inherited from class org.bytedeco.arrow_dataset.FileFormat

MakeFragment, MakeFragment, MakeFragment

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getPointer, getPointer, getPointer, hashCode, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetof, offsetof, parseBytes, physicalBytes, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ParquetFileFormat

public ParquetFileFormat(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

ParquetFileFormat

public ParquetFileFormat(long size)

Native array allocator. Access with Pointer.position(long).

ParquetFileFormat

public ParquetFileFormat()

ParquetFileFormat

public ParquetFileFormat(@Const @ByRef
                         ReaderProperties reader_properties)

Convenience constructor which copies properties from a parquet::ReaderProperties. memory_pool will be ignored.

Method Detail

position

public ParquetFileFormat position(long position)

Overrides:

position in class Pointer

getPointer

public ParquetFileFormat getPointer(long i)

Overrides:

getPointer in class Pointer

type_name

@StdString
public String type_name()

Description copied from class: FileFormat

\brief The name identifying the kind of file format

Overrides:

type_name in class FileFormat

splittable

@Cast(value="bool")
public boolean splittable()

Description copied from class: FileFormat

\brief Return true if fragments of this format can benefit from parallel scanning.

Overrides:

splittable in class FileFormat

Equals

@Cast(value="bool")
public boolean Equals(@Const @ByRef
                                           FileFormat other)

Overrides:

Equals in class FileFormat

reader_options_use_buffered_stream

@Name(value="reader_options.use_buffered_stream")
 @Cast(value="bool")
public boolean reader_options_use_buffered_stream()

\defgroup parquet-file-format-reader-properties properties which correspond to members of parquet::ReaderProperties. We don't embed parquet::ReaderProperties directly because we get memory_pool from ScanContext at scan time and provide differing defaults. \{

reader_options_use_buffered_stream

public ParquetFileFormat reader_options_use_buffered_stream(boolean setter)

reader_options_buffer_size

@Name(value="reader_options.buffer_size")
 @Cast(value="int64_t")
public long reader_options_buffer_size()

reader_options_buffer_size

public ParquetFileFormat reader_options_buffer_size(long setter)

reader_options_file_decryption_properties

@Name(value="reader_options.file_decryption_properties")
 @SharedPtr
public FileDecryptionProperties reader_options_file_decryption_properties()

reader_options_file_decryption_properties

public ParquetFileFormat reader_options_file_decryption_properties(FileDecryptionProperties setter)

reader_options_dict_columns

@Name(value="reader_options.dict_columns")
 @ByRef
public StringUnorderedSet reader_options_dict_columns()

\}

\defgroup parquet-file-format-arrow-reader-properties properties which correspond to members of parquet::ArrowReaderProperties. We don't embed parquet::ReaderProperties directly because we get batch_size from ScanOptions at scan time, and we will never pass use_threads == true (since we defer parallelization of the scan). Additionally column names (rather than indices) are used to indicate dictionary columns. \{

reader_options_dict_columns

public ParquetFileFormat reader_options_dict_columns(StringUnorderedSet setter)

reader_options_enable_parallel_column_conversion

@Name(value="reader_options.enable_parallel_column_conversion")
 @Cast(value="bool")
public boolean reader_options_enable_parallel_column_conversion()

\}

EXPERIMENTAL: Parallelize conversion across columns. This option is ignored if a scan is already parallelized across input files to avoid thread contention. This option will be removed after support is added for simultaneous parallelization across files and columns.

reader_options_enable_parallel_column_conversion

public ParquetFileFormat reader_options_enable_parallel_column_conversion(boolean setter)

IsSupported

@ByVal
public BoolResult IsSupported(@Const @ByRef
                                      FileSource source)

Description copied from class: FileFormat

\brief Indicate if the FileSource is supported/readable by this format.

Overrides:

IsSupported in class FileFormat

Inspect

@ByVal
public SchemaResult Inspect(@Const @ByRef
                                    FileSource source)

\brief Return the schema of the file if possible.

Overrides:

Inspect in class FileFormat

ScanFile

@ByVal
public ScanTaskIteratorResult ScanFile(@SharedPtr
                                               ScanOptions options,
                                               @SharedPtr
                                               ScanContext context,
                                               FileFragment file)

\brief Open a file for scanning

Overrides:

ScanFile in class FileFormat

MakeFragment

@ByVal
public FileFragmentResult MakeFragment(@ByVal
                                               FileSource source,
                                               @ByVal
                                               Expression partition_expression,
                                               @SharedPtr @ByVal
                                               Schema physical_schema)

\brief Create a Fragment targeting all RowGroups.

Overrides:

MakeFragment in class FileFormat

MakeFragment

@ByVal
public ParquetFileFragmentResult MakeFragment(@ByVal
                                                      FileSource source,
                                                      @ByVal
                                                      Expression partition_expression,
                                                      @SharedPtr @ByVal
                                                      Schema physical_schema,
                                                      @StdVector
                                                      IntPointer row_groups)

\brief Create a Fragment, restricted to the specified row groups.

MakeFragment

@ByVal
public ParquetFileFragmentResult MakeFragment(@ByVal
                                                      FileSource source,
                                                      @ByVal
                                                      Expression partition_expression,
                                                      @SharedPtr @ByVal
                                                      Schema physical_schema,
                                                      @StdVector
                                                      IntBuffer row_groups)

MakeFragment

@ByVal
public ParquetFileFragmentResult MakeFragment(@ByVal
                                                      FileSource source,
                                                      @ByVal
                                                      Expression partition_expression,
                                                      @SharedPtr @ByVal
                                                      Schema physical_schema,
                                                      @StdVector
                                                      int[] row_groups)

GetReader

@ByVal
public FileReaderResult GetReader(@Const @ByRef
                                          FileSource source,
                                          ScanOptions arg1,
                                          ScanContext arg2)

\brief Return a FileReader on the given source.

GetReader

@ByVal
public FileReaderResult GetReader(@Const @ByRef
                                          FileSource source)

MakeWriter

@ByVal
public FileWriterResult MakeWriter(@SharedPtr
                                           OutputStream destination,
                                           @SharedPtr @ByVal
                                           Schema schema,
                                           @SharedPtr
                                           FileWriteOptions options)

Overrides:

MakeWriter in class FileFormat

DefaultWriteOptions

@SharedPtr
public FileWriteOptions DefaultWriteOptions()

Overrides:

DefaultWriteOptions in class FileFormat