org.bytedeco.arrow.global

Class arrow_dataset

java.lang.Object

org.bytedeco.arrow.presets.arrow_dataset

org.bytedeco.arrow.global.arrow_dataset

All Implemented Interfaces:

InfoMapper

public class arrow_dataset
extends arrow_dataset

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.arrow.presets.arrow_dataset

arrow_dataset.CustomOpenWithCompression, arrow_dataset.EnumeratedRecordBatchGenerator, arrow_dataset.ExpressionVisitor, arrow_dataset.RecordBatchGenerator, arrow_dataset.TaggedRecordBatchGenerator, arrow_dataset.TaggedRecordBatchVisitor

Constructor Summary

Constructors

Constructor and Description
arrow_dataset()

Method Summary

Modifier and Type	Method and Description
static Expression	and_(Expression arg0)
static Expression	and_(Expression lhs, Expression rhs)
static Expression	call(BytePointer function, Expression arguments)
static Expression	call(BytePointer function, Expression arguments, FunctionOptions options)
static Expression	call(String function, Expression arguments)
static Expression	call(String function, Expression arguments, FunctionOptions options)
static ExpressionResult	Canonicalize(Expression arg0)
static ExpressionResult	Canonicalize(Expression arg0, ExecContext arg1) \defgroup expression-passes Functions for modification of Expressions \{ These transform bound expressions.
static Status	CheckProjectable(Schema from, Schema to) Logic for automatically determining the structure of multi-file dataset with possible partitioning according to available partitioning
static ExpressionResult	Deserialize(ArrowBuffer arg0)
static void	DieWithMessage(BytePointer msg)
static void	DieWithMessage(String msg) \brief Make a generator that always fails with a given error This overload allows inferring the return type from the argument.
static Expression	equal(Expression lhs, Expression rhs)
static boolean	equals(Expression l, Expression r) \brief Execute an expression that returns a Result, extracting its value into the variable defined by lhs (or returning a Status on error).
static DatumResult	ExecuteScalarExpression(Expression arg0, Datum input)
static DatumResult	ExecuteScalarExpression(Expression arg0, Datum input, ExecContext arg2) Execute a scalar expression against the provided state and input Datum.
static FieldRefDatumMapResult	ExtractKnownFieldValues(Expression guaranteed_true_predicate) Assemble a mapping from field references to known values.
static Expression	field_ref(FieldRef ref)
static FieldRef	FieldsInExpression(Expression arg0) Assemble a list of all fields referenced by an Expression at any depth.
static ExpressionResult	FoldConstants(Expression arg0) Simplify Expressions based on literal arguments (for example, add(null, x) will always be null so replace the call with a null literal).
static Expression	greater_equal(Expression lhs, Expression rhs)
static Expression	greater(Expression lhs, Expression rhs)
static void	InvalidValueOrDie(Status st)
static Expression	is_null(Expression lhs)
static Expression	is_valid(Expression lhs)
static String	kCsvTypeName()
static byte	kCsvTypeName(int i) \}
static int	kDefaultBackgroundMaxQ()
static int	kDefaultBackgroundQRestart()
static int	kDefaultBatchReadahead()
static long	kDefaultBatchSize() \defgroup dataset-scanning Scanning API \{
static int	kDefaultFragmentReadahead()
static String	kDefaultHiveNullFallback()
static byte	kDefaultHiveNullFallback(int i) \brief The default fallback used for null values in a Hive-style partitioning.
static String	kIpcTypeName()
static byte	kIpcTypeName(int i) \addtogroup dataset-file-formats \{
static String	kParquetTypeName()
static byte	kParquetTypeName(int i) \addtogroup dataset-file-formats \{
static Expression	less_equal(Expression lhs, Expression rhs)
static Expression	less(Expression lhs, Expression rhs)
static Expression	literal(Datum lit)
static Expression	not_(Expression operand)
static Expression	not_equal(Expression lhs, Expression rhs)
static boolean	notEquals(Expression l, Expression r)
static Expression	or_(Expression arg0)
static Expression	or_(Expression lhs, Expression rhs)
static Expression	project(Expression values, StringVector names)
static ExpressionResult	ReplaceFieldsWithKnownValues(FieldRefDatumMap known_values, Expression arg1) Simplify Expressions by replacing with known values of the fields which it references.
static ScanTaskIteratorResult	ScanTaskIteratorFromRecordBatch(RecordBatchVector batches, ScanOptions options) \}
static BufferResult	Serialize(Expression arg0)
static ExpressionResult	SimplifyWithGuarantee(Expression arg0, Expression guaranteed_true_predicate) Simplify an expression by replacing subexpressions based on a guarantee: a boolean expression which is guaranteed to evaluate to true.
static BytePointer	StripPrefixAndFilename(BytePointer path, BytePointer prefix)
static StringVector	StripPrefixAndFilename(FileInfoVector files, BytePointer prefix)
static StringVector	StripPrefixAndFilename(FileInfoVector files, String prefix) \brief Vector version of StripPrefixAndFilename.
static String	StripPrefixAndFilename(String path, String prefix) \brief Remove a prefix and the filename of a path.
static StringVector	StripPrefixAndFilename(StringVector paths, BytePointer prefix)
static StringVector	StripPrefixAndFilename(StringVector paths, String prefix) \brief Vector version of StripPrefixAndFilename.

Methods inherited from class org.bytedeco.arrow.presets.arrow_dataset

map

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

arrow_dataset

public arrow_dataset()

Method Detail

kDefaultBackgroundMaxQ

@Namespace(value="arrow")
 @MemberGetter
public static int kDefaultBackgroundMaxQ()

See Also:

MakeBackgroundGenerator

kDefaultBackgroundQRestart

@Namespace(value="arrow")
 @MemberGetter
public static int kDefaultBackgroundQRestart()

DieWithMessage

@Namespace(value="arrow::internal")
public static void DieWithMessage(@StdString
                                                                      String msg)

\brief Make a generator that always fails with a given error This overload allows inferring the return type from the argument.

DieWithMessage

@Namespace(value="arrow::internal")
public static void DieWithMessage(@StdString
                                                                      BytePointer msg)

InvalidValueOrDie

@Namespace(value="arrow::internal")
public static void InvalidValueOrDie(@Const @ByRef
                                                                         Status st)

equals

@Namespace(value="arrow::dataset")
 @Cast(value="bool")
 @Name(value="operator ==")
public static boolean equals(@Const @ByRef
                                                                                                                 Expression l,
                                                                                                                 @Const @ByRef
                                                                                                                 Expression r)

\brief Execute an expression that returns a Result, extracting its value into the variable defined by lhs (or returning a Status on error). Example: Assigning to a new value: ARROW_ASSIGN_OR_RAISE(auto value, MaybeGetValue(arg)); Example: Assigning to an existing value: ValueType value; ARROW_ASSIGN_OR_RAISE(value, MaybeGetValue(arg)); WARNING: ARROW_ASSIGN_OR_RAISE expands into multiple statements; it cannot be used in a single statement (e.g. as the body of an if statement without {})! WARNING: ARROW_ASSIGN_OR_RAISE std::moves its right operand. If you have an lvalue Result which you *don't* want to move out of cast appropriately.

notEquals

@Namespace(value="arrow::dataset")
 @Cast(value="bool")
 @Name(value="operator !=")
public static boolean notEquals(@Const @ByRef
                                                                                                                    Expression l,
                                                                                                                    @Const @ByRef
                                                                                                                    Expression r)

literal

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression literal(@ByVal
                                                                            Datum lit)

field_ref

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression field_ref(@ByVal
                                                                              FieldRef ref)

call

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression call(@StdString
                                                                         String function,
                                                                         @StdVector
                                                                         Expression arguments,
                                                                         @SharedPtr
                                                                         FunctionOptions options)

call

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression call(@StdString
                                                                         String function,
                                                                         @StdVector
                                                                         Expression arguments)

call

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression call(@StdString
                                                                         BytePointer function,
                                                                         @StdVector
                                                                         Expression arguments,
                                                                         @SharedPtr
                                                                         FunctionOptions options)

call

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression call(@StdString
                                                                         BytePointer function,
                                                                         @StdVector
                                                                         Expression arguments)

FieldsInExpression

@Namespace(value="arrow::dataset")
 @StdVector
public static FieldRef FieldsInExpression(@Const @ByRef
                                                                                         Expression arg0)

Assemble a list of all fields referenced by an Expression at any depth.

ExtractKnownFieldValues

@Namespace(value="arrow::dataset")
 @ByVal
public static FieldRefDatumMapResult ExtractKnownFieldValues(@Const @ByRef
                                                                                                        Expression guaranteed_true_predicate)

Assemble a mapping from field references to known values.

Canonicalize

@Namespace(value="arrow::dataset")
 @ByVal
public static ExpressionResult Canonicalize(@ByVal
                                                                                       Expression arg0,
                                                                                       ExecContext arg1)

\defgroup expression-passes Functions for modification of Expressions \{ These transform bound expressions. Some transforms utilize a guarantee, which is provided as an Expression which is guaranteed to evaluate to true. The guaranteed_true_predicate need not be bound, but canonicalization is currently deferred to producers of guarantees. For example in order to be recognized as a guarantee on a field value, an Expression must be a call to "equal" with field_ref LHS and literal RHS. Flipping the arguments, "is_in" with a one-long value_set, ... or other semantically identical Expressions will not be recognized.

Weak canonicalization which establishes guarantees for subsequent passes. Even equivalent Expressions may result in different canonicalized expressions. TODO this could be a strong canonicalization

Canonicalize

@Namespace(value="arrow::dataset")
 @ByVal
public static ExpressionResult Canonicalize(@ByVal
                                                                                       Expression arg0)

FoldConstants

@Namespace(value="arrow::dataset")
 @ByVal
public static ExpressionResult FoldConstants(@ByVal
                                                                                        Expression arg0)

Simplify Expressions based on literal arguments (for example, add(null, x) will always be null so replace the call with a null literal). Includes early evaluation of all calls whose arguments are entirely literal.

ReplaceFieldsWithKnownValues

@Namespace(value="arrow::dataset")
 @ByVal
public static ExpressionResult ReplaceFieldsWithKnownValues(@Const @ByRef
                                                                                                       FieldRefDatumMap known_values,
                                                                                                       @ByVal
                                                                                                       Expression arg1)

Simplify Expressions by replacing with known values of the fields which it references.

SimplifyWithGuarantee

@Namespace(value="arrow::dataset")
 @ByVal
public static ExpressionResult SimplifyWithGuarantee(@ByVal
                                                                                                Expression arg0,
                                                                                                @Const @ByRef
                                                                                                Expression guaranteed_true_predicate)

Simplify an expression by replacing subexpressions based on a guarantee: a boolean expression which is guaranteed to evaluate to true. For example, this is used to remove redundant function calls from a filter expression or to replace a reference to a constant-value field with a literal.

ExecuteScalarExpression

@Namespace(value="arrow::dataset")
 @ByVal
public static DatumResult ExecuteScalarExpression(@Const @ByRef
                                                                                             Expression arg0,
                                                                                             @Const @ByRef
                                                                                             Datum input,
                                                                                             ExecContext arg2)

Execute a scalar expression against the provided state and input Datum. This /** expression must be bound.

ExecuteScalarExpression

@Namespace(value="arrow::dataset")
 @ByVal
public static DatumResult ExecuteScalarExpression(@Const @ByRef
                                                                                             Expression arg0,
                                                                                             @Const @ByRef
                                                                                             Datum input)

Serialize

@Namespace(value="arrow::dataset")
 @ByVal
public static BufferResult Serialize(@Const @ByRef
                                                                                Expression arg0)

Deserialize

@Namespace(value="arrow::dataset")
 @ByVal
public static ExpressionResult Deserialize(@SharedPtr
                                                                                      ArrowBuffer arg0)

project

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression project(@StdVector
                                                                            Expression values,
                                                                            @ByVal
                                                                            StringVector names)

equal

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression equal(@ByVal
                                                                          Expression lhs,
                                                                          @ByVal
                                                                          Expression rhs)

not_equal

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression not_equal(@ByVal
                                                                              Expression lhs,
                                                                              @ByVal
                                                                              Expression rhs)

less

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression less(@ByVal
                                                                         Expression lhs,
                                                                         @ByVal
                                                                         Expression rhs)

less_equal

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression less_equal(@ByVal
                                                                               Expression lhs,
                                                                               @ByVal
                                                                               Expression rhs)

greater

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression greater(@ByVal
                                                                            Expression lhs,
                                                                            @ByVal
                                                                            Expression rhs)

greater_equal

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression greater_equal(@ByVal
                                                                                  Expression lhs,
                                                                                  @ByVal
                                                                                  Expression rhs)

is_null

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression is_null(@ByVal
                                                                            Expression lhs)

is_valid

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression is_valid(@ByVal
                                                                             Expression lhs)

and_

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression and_(@ByVal
                                                                         Expression lhs,
                                                                         @ByVal
                                                                         Expression rhs)

and_

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression and_(@StdVector
                                                                         Expression arg0)

or_

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression or_(@ByVal
                                                                        Expression lhs,
                                                                        @ByVal
                                                                        Expression rhs)

or_

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression or_(@StdVector
                                                                        Expression arg0)

not_

@Namespace(value="arrow::dataset")
 @ByVal
public static Expression not_(@ByVal
                                                                         Expression operand)

kDefaultHiveNullFallback

@Namespace(value="arrow::dataset")
 @MemberGetter
public static byte kDefaultHiveNullFallback(int i)

\brief The default fallback used for null values in a Hive-style partitioning.

kDefaultHiveNullFallback

@Namespace(value="arrow::dataset")
 @MemberGetter
public static String kDefaultHiveNullFallback()

StripPrefixAndFilename

@Namespace(value="arrow::dataset")
 @StdString
public static String StripPrefixAndFilename(@StdString
                                                                                           String path,
                                                                                           @StdString
                                                                                           String prefix)

\brief Remove a prefix and the filename of a path. e.g., StripPrefixAndFilename("/data/year=2019/c.txt", "/data") -> "year=2019"

StripPrefixAndFilename

@Namespace(value="arrow::dataset")
 @StdString
public static BytePointer StripPrefixAndFilename(@StdString
                                                                                                BytePointer path,
                                                                                                @StdString
                                                                                                BytePointer prefix)

StripPrefixAndFilename

@Namespace(value="arrow::dataset")
 @ByVal
public static StringVector StripPrefixAndFilename(@Const @ByRef
                                                                                             StringVector paths,
                                                                                             @StdString
                                                                                             String prefix)

\brief Vector version of StripPrefixAndFilename.

StripPrefixAndFilename

@Namespace(value="arrow::dataset")
 @ByVal
public static StringVector StripPrefixAndFilename(@Const @ByRef
                                                                                             StringVector paths,
                                                                                             @StdString
                                                                                             BytePointer prefix)

StripPrefixAndFilename

@Namespace(value="arrow::dataset")
 @ByVal
public static StringVector StripPrefixAndFilename(@Const @ByRef
                                                                                             FileInfoVector files,
                                                                                             @StdString
                                                                                             String prefix)

\brief Vector version of StripPrefixAndFilename.

StripPrefixAndFilename

@Namespace(value="arrow::dataset")
 @ByVal
public static StringVector StripPrefixAndFilename(@Const @ByRef
                                                                                             FileInfoVector files,
                                                                                             @StdString
                                                                                             BytePointer prefix)

CheckProjectable

@Namespace(value="arrow::dataset")
 @ByVal
public static Status CheckProjectable(@Const @ByRef
                                                                                 Schema from,
                                                                                 @Const @ByRef
                                                                                 Schema to)

Logic for automatically determining the structure of multi-file dataset with possible partitioning according to available partitioning

kDefaultBatchSize

@Namespace(value="arrow::dataset")
 @MemberGetter
 @Cast(value="const int64_t")
public static long kDefaultBatchSize()

\defgroup dataset-scanning Scanning API \{

kDefaultBatchReadahead

@Namespace(value="arrow::dataset")
 @MemberGetter
public static int kDefaultBatchReadahead()

kDefaultFragmentReadahead

@Namespace(value="arrow::dataset")
 @MemberGetter
public static int kDefaultFragmentReadahead()

ScanTaskIteratorFromRecordBatch

@Namespace(value="arrow::dataset")
 @ByVal
public static ScanTaskIteratorResult ScanTaskIteratorFromRecordBatch(@ByVal
                                                                                                                RecordBatchVector batches,
                                                                                                                @SharedPtr
                                                                                                                ScanOptions options)

\}

kCsvTypeName

@Namespace(value="arrow::dataset")
 @MemberGetter
public static byte kCsvTypeName(int i)

\}

kCsvTypeName

@Namespace(value="arrow::dataset")
 @MemberGetter
public static String kCsvTypeName()

kIpcTypeName

@Namespace(value="arrow::dataset")
 @MemberGetter
public static byte kIpcTypeName(int i)

\addtogroup dataset-file-formats \{

kIpcTypeName

@Namespace(value="arrow::dataset")
 @MemberGetter
public static String kIpcTypeName()

kParquetTypeName

@Namespace(value="arrow::dataset")
 @MemberGetter
public static byte kParquetTypeName(int i)

\addtogroup dataset-file-formats \{

kParquetTypeName

@Namespace(value="arrow::dataset")
 @MemberGetter
public static String kParquetTypeName()