org.bytedeco.arrow

Class ChunkedArray

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow.ChunkedArray

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow")
 @NoOffset
 @Properties(inherit=arrow.class)
public class ChunkedArray
extends Pointer

\class ChunkedArray \brief A data structure managing a list of primitive Arrow arrays logically as one large array Data chunking is treated throughout this project largely as an implementation detail for performance and memory use optimization. ChunkedArray allows Array objects to be collected and interpreted as a single logical array without requiring an expensive concatenation step. In some cases, data produced by a function may exceed the capacity of an Array (like BinaryArray or StringArray) and so returning multiple Arrays is the only possibility. In these cases, we recommend returning a ChunkedArray instead of vector of Arrays or some alternative. When data is processed in parallel, it may not be practical or possible to create large contiguous memory allocations and write output into them. With some data types, like binary and string types, it is not possible at all to produce non-chunked array outputs without requiring a concatenation step at the end of processing. Application developers may tune chunk sizes based on analysis of performance profiles but many developer-users will not need to be especially concerned with the chunking details. Preserving the chunk layout/sizes in processing steps is generally not considered to be a contract in APIs. A function may decide to alter the chunking of its result. Similarly, APIs accepting multiple ChunkedArray inputs should not expect the chunk layout to be the same in each input.

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
ChunkedArray(Array chunk) \brief Construct a chunked array from a single Array
ChunkedArray(ArrayVector chunks) \brief Construct a chunked array from a vector of arrays The vector must be non-empty and all its elements must have the same data type.
ChunkedArray(ArrayVector chunks, DataType type) \brief Construct a chunked array from a vector of arrays and a data type As the data type is passed explicitly, the vector may be empty.
ChunkedArray(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
Array	chunk(int i)
ArrayVector	chunks()
boolean	Equals(ChunkedArray other) \brief Determine if two chunked arrays are equal.
ChunkedArrayVectorResult	Flatten()
ChunkedArrayVectorResult	Flatten(MemoryPool pool) \brief Flatten this chunked array as a vector of chunked arrays, one for each struct field
long	length()
static ChunkedArrayResult	Make(ArrayVector chunks)
static ChunkedArrayResult	Make(ArrayVector chunks, DataType type)
long	null_count()
int	num_chunks()
ChunkedArray	Slice(long offset) \brief Slice from offset until end of the chunked array
ChunkedArray	Slice(long offset, long length) \brief Construct a zero-copy slice of the chunked array with the indicated offset and length
String	ToString()
DataType	type()
Status	Validate() \brief Perform cheap validation checks to determine obvious inconsistencies within the chunk array's internal data.
Status	ValidateFull() \brief Perform extensive validation checks to determine inconsistencies within the chunk array's internal data.
ChunkedArrayResult	View(DataType type) Construct a zero-copy view of this chunked array with the given type.

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getPointer, getPointer, getPointer, getPointer, hashCode, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetof, offsetof, parseBytes, physicalBytes, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ChunkedArray

public ChunkedArray(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

ChunkedArray

public ChunkedArray(@ByVal
                    ArrayVector chunks)

\brief Construct a chunked array from a vector of arrays The vector must be non-empty and all its elements must have the same data type.

ChunkedArray

public ChunkedArray(@SharedPtr @Cast(value={"","std::shared_ptr<arrow::Array>"})
                    Array chunk)

\brief Construct a chunked array from a single Array

ChunkedArray

public ChunkedArray(@ByVal
                    ArrayVector chunks,
                    @SharedPtr @Cast(value={"","std::shared_ptr<arrow::DataType>"})
                    DataType type)

\brief Construct a chunked array from a vector of arrays and a data type As the data type is passed explicitly, the vector may be empty.

Method Detail

Make

@ByVal
public static ChunkedArrayResult Make(@ByVal
                                              ArrayVector chunks,
                                              @SharedPtr @Cast(value={"","std::shared_ptr<arrow::DataType>"})
                                              DataType type)

Make

@ByVal
public static ChunkedArrayResult Make(@ByVal
                                              ArrayVector chunks)

length

@Cast(value="int64_t")
public long length()

Returns:

the total length of the chunked array; computed on construction

null_count

@Cast(value="int64_t")
public long null_count()

Returns:

the total number of nulls among all chunks

num_chunks

public int num_chunks()

chunk

@SharedPtr
 @Cast(value={"","std::shared_ptr<arrow::Array>"})
public Array chunk(int i)

Returns:

chunk a particular chunk from the chunked array

chunks

@Const
 @ByRef
public ArrayVector chunks()

Slice

@SharedPtr
 @Cast(value={"","std::shared_ptr<arrow::ChunkedArray>"})
public ChunkedArray Slice(@Cast(value="int64_t")
                                                                                                 long offset,
                                                                                                 @Cast(value="int64_t")
                                                                                                 long length)

\brief Construct a zero-copy slice of the chunked array with the indicated offset and length

Parameters:

offset - [in] the position of the first element in the constructed slice

length - [in] the length of the slice. If there are not enough elements in the chunked array, the length will be adjusted accordingly

Returns:

a new object wrapped in std::shared_ptr

Slice

@SharedPtr
 @Cast(value={"","std::shared_ptr<arrow::ChunkedArray>"})
public ChunkedArray Slice(@Cast(value="int64_t")
                                                                                                 long offset)

\brief Slice from offset until end of the chunked array

Flatten

@ByVal
public ChunkedArrayVectorResult Flatten(MemoryPool pool)

\brief Flatten this chunked array as a vector of chunked arrays, one for each struct field

Parameters:

pool - [in] The pool for buffer allocations, if any

Flatten

@ByVal
public ChunkedArrayVectorResult Flatten()

View

@ByVal
public ChunkedArrayResult View(@SharedPtr @Cast(value={"","std::shared_ptr<arrow::DataType>"})
                                       DataType type)

Construct a zero-copy view of this chunked array with the given type. Calls Array::View on each constituent chunk. Always succeeds if there are zero chunks

type

@SharedPtr
 @Cast(value={"","std::shared_ptr<arrow::DataType>"})
public DataType type()

Equals

@Cast(value="bool")
public boolean Equals(@Const @ByRef
                                           ChunkedArray other)

\brief Determine if two chunked arrays are equal. Two chunked arrays can be equal only if they have equal datatypes. However, they may be equal even if they have different chunkings.

ToString

@StdString
public String ToString()

Returns:

PrettyPrint representation suitable for debugging

Validate

@ByVal
public Status Validate()

\brief Perform cheap validation checks to determine obvious inconsistencies within the chunk array's internal data. This is O(k*m) where k is the number of array descendents, and m is the number of chunks.

Returns:

Status

ValidateFull

@ByVal
public Status ValidateFull()

\brief Perform extensive validation checks to determine inconsistencies within the chunk array's internal data. This is O(k*n) where k is the number of array descendents, and n is the length in elements.

Returns:

Status