org.bytedeco.arrow

Class StreamDecoder

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow.StreamDecoder

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow::ipc")
 @NoOffset
 @Properties(inherit=arrow.class)
public class StreamDecoder
extends Pointer

\brief Push style stream decoder that receives data from user. This class decodes the Apache Arrow IPC streaming format data. This API is EXPERIMENTAL.

Since:

0.17.0

See Also:

https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
StreamDecoder(Listener listener)
StreamDecoder(Listener listener, IpcReadOptions options) \brief Construct a stream decoder.
StreamDecoder(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
Status	Consume(ArrowBuffer buffer) \brief Feed data to the decoder as a Buffer.
Status	Consume(byte[] data, long size)
Status	Consume(ByteBuffer data, long size)
Status	Consume(BytePointer data, long size) \brief Feed data to the decoder as a raw data.
long	next_required_size() \brief Return the number of bytes needed to advance the state of the decoder.
Schema	schema()
ReadStats	stats() \brief Return current read statistics

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getPointer, getPointer, getPointer, getPointer, hashCode, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetof, offsetof, parseBytes, physicalBytes, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

StreamDecoder

public StreamDecoder(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

StreamDecoder

public StreamDecoder(@SharedPtr
                     Listener listener,
                     @ByVal(nullValue="arrow::ipc::IpcReadOptions::Defaults()")
                     IpcReadOptions options)

\brief Construct a stream decoder.

Parameters:

listener - [in] a Listener that must implement Listener::OnRecordBatchDecoded() to receive decoded record batches

options - [in] any IPC reading options (optional)

StreamDecoder

public StreamDecoder(@SharedPtr
                     Listener listener)

Method Detail

Consume

@ByVal
public Status Consume(@Cast(value="const uint8_t*")
                              BytePointer data,
                              @Cast(value="int64_t")
                              long size)

\brief Feed data to the decoder as a raw data. If the decoder can read one or more record batches by the data, the decoder calls listener->OnRecordBatchDecoded() with a decoded record batch multiple times.

Parameters:

data - [in] a raw data to be processed. This data isn't copied. The passed memory must be kept alive through record batch processing.

size - [in] raw data size.

Returns:

Status

Consume

@ByVal
public Status Consume(@Cast(value="const uint8_t*")
                              ByteBuffer data,
                              @Cast(value="int64_t")
                              long size)

Consume

@ByVal
public Status Consume(@Cast(value="const uint8_t*")
                              byte[] data,
                              @Cast(value="int64_t")
                              long size)

Consume

@ByVal
public Status Consume(@SharedPtr
                              ArrowBuffer buffer)

\brief Feed data to the decoder as a Buffer. If the decoder can read one or more record batches by the Buffer, the decoder calls listener->RecordBatchReceived() with a decoded record batch multiple times.

Parameters:

buffer - [in] a Buffer to be processed.

Returns:

Status

schema

@SharedPtr
 @ByVal
public Schema schema()

Returns:

the shared schema of the record batches in the stream

next_required_size

@Cast(value="int64_t")
public long next_required_size()

\brief Return the number of bytes needed to advance the state of the decoder. This method is provided for users who want to optimize performance. Normal users don't need to use this method. Here is an example usage for normal users: ~~~{.cpp} decoder.Consume(buffer1); decoder.Consume(buffer2); decoder.Consume(buffer3); ~~~ Decoder has internal buffer. If consumed data isn't enough to advance the state of the decoder, consumed data is buffered to the internal buffer. It causes performance overhead. If you pass next_required_size() size data to each Consume() call, the decoder doesn't use its internal buffer. It improves performance. Here is an example usage to avoid using internal buffer: ~~~{.cpp} buffer1 = get_data(decoder.next_required_size()); decoder.Consume(buffer1); buffer2 = get_data(decoder.next_required_size()); decoder.Consume(buffer2); ~~~ Users can use this method to avoid creating small chunks. Record batch data must be contiguous data. If users pass small chunks to the decoder, the decoder needs concatenate small chunks internally. It causes performance overhead. Here is an example usage to reduce small chunks: ~~~{.cpp} buffer = AllocateResizableBuffer(); while ((small_chunk = get_data(&small_chunk_size))) { auto current_buffer_size = buffer->size(); buffer->Resize(current_buffer_size + small_chunk_size); memcpy(buffer->mutable_data() + current_buffer_size, small_chunk, small_chunk_size); if (buffer->size() < decoder.next_required_size()) { continue; } std::shared_ptr chunk(buffer.release()); decoder.Consume(chunk); buffer = AllocateResizableBuffer(); } if (buffer->size() > 0) { std::shared_ptr chunk(buffer.release()); decoder.Consume(chunk); } ~~~

Returns:

the number of bytes needed to advance the state of the decoder

stats

@ByVal
public ReadStats stats()

\brief Return current read statistics