org.bytedeco.arrow_dataset

Class ParquetFileFragment

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow_dataset.Fragment

org.bytedeco.arrow_dataset.FileFragment

org.bytedeco.arrow_dataset.ParquetFileFragment

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow::dataset")
 @NoOffset
 @Properties(inherit=arrow_dataset.class)
public class ParquetFileFragment
extends FileFragment

\brief A FileFragment with parquet logic. ParquetFileFragment provides a lazy (with respect to IO) interface to scan parquet files. Any heavy IO calls are deferred to the Scan() method. The caller can provide an optional list of selected RowGroups to limit the number of scanned RowGroups, or to partition the scans across multiple threads. It can also attach optional statistics with each RowGroups, providing pushdown predicate benefits before invoking any heavy IO. This can induce significant performance boost when scanning high latency file systems.

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
ParquetFileFragment(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
Status	EnsureCompleteMetadata()
Status	EnsureCompleteMetadata(FileReader reader) \brief Ensure attached statistics are complete and the physical schema is cached.
boolean	HasCompleteMetadata() \brief Indicate if the attached statistics are complete and the physical schema is cached.
RowGroupInfo	row_groups() \brief Return the RowGroups selected by this fragment.
FragmentResult	SplitByRowGroup(Expression predicate)

Methods inherited from class org.bytedeco.arrow_dataset.FileFragment

format, Scan, source, splittable, type_name

Methods inherited from class org.bytedeco.arrow_dataset.Fragment

partition_expression, ReadPhysicalSchema

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getPointer, getPointer, getPointer, getPointer, hashCode, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetof, offsetof, parseBytes, physicalBytes, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ParquetFileFragment

public ParquetFileFragment(Pointer p)

Pointer cast constructor. Invokes Pointer.Pointer(Pointer).

Method Detail

SplitByRowGroup

@ByVal
public FragmentResult SplitByRowGroup(@Const @SharedPtr @ByRef
                                              Expression predicate)

row_groups

@StdVector
public RowGroupInfo row_groups()

\brief Return the RowGroups selected by this fragment. An empty list represents all RowGroups in the parquet file.

HasCompleteMetadata

@Cast(value="bool")
public boolean HasCompleteMetadata()

\brief Indicate if the attached statistics are complete and the physical schema is cached. The statistics are complete if the provided RowGroups (see row_groups()) is not empty / and all RowGroup return true on RowGroup::HasStatistics().

EnsureCompleteMetadata

@ByVal
public Status EnsureCompleteMetadata(FileReader reader)

\brief Ensure attached statistics are complete and the physical schema is cached.

EnsureCompleteMetadata

@ByVal
public Status EnsureCompleteMetadata()