package io
- Alphabetic
- By Inheritance
- io
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- class DeltaWriter extends Closeable with StrictLogging
Builds up dictionaries and write record batches.
Builds up dictionaries and write record batches. Dictionaries are encoded as deltas to minimize redundant messages.
- class DictionaryBuildingWriter extends Closeable
Writes an arrow file of simple features.
Writes an arrow file of simple features. Dictionaries will be built up as features are observed. Dictionaries are encoded based on
maxSize. Values will not be correctly encoded if more thanmaxSizedistinct values are encountered - trait SimpleFeatureArrowFileReader extends Closeable
For reading simple features from an arrow file written by SimpleFeatureArrowFileWriter.
For reading simple features from an arrow file written by SimpleFeatureArrowFileWriter.
Expects arrow streaming format (no footer). Can handle multiple 'files' in a single input stream
- class SimpleFeatureArrowFileWriter extends Closeable with Flushable with LazyLogging
For writing simple features to an arrow file.
For writing simple features to an arrow file.
Uses arrow streaming format (no footer). Closing the SimpleFeatureArrowFileWriter closes the given os.
Value Members
- def createFileFromBatches(sft: SimpleFeatureType, dictionaries: Map[String, ArrowDictionary], encoding: SimpleFeatureEncoding, ipcOpts: IpcOption, sort: Option[(String, Boolean)], batches: CloseableIterator[Array[Byte]], firstBatchHasHeader: Boolean): CloseableIterator[Array[Byte]]
Create an arrow file from record batches
Create an arrow file from record batches
- sft
simple feature type
- dictionaries
dictionaries
- encoding
feature encoding
- sort
sorting of the batches, if any
- batches
batches
- firstBatchHasHeader
does the first batch have the arrow file header or not
- def createRoot(vector: FieldVector, metadata: Map[String, String] = null): VectorSchemaRoot
Creates a vector schema root for the given vector
Creates a vector schema root for the given vector
- vector
vector
- metadata
field metadata
- def createTransferPair(sft: SimpleFeatureType, from: FieldVector, to: FieldVector): (Int, Int) => Unit
Create a transfer pair between two vectors.
Create a transfer pair between two vectors. This handles geometry vectors correctly, which the underlying arrow transfer pairs do not.
- from
from vector
- to
to vector
- returns
transfer(fromIndex, toIndex)
- def getSortAsMetadata(field: String, reverse: Boolean): Map[String, String]
Creates metadata for sort fields
Creates metadata for sort fields
- field
sort field
- reverse
reverse sorted or not
- returns
metadata map
- def getSortFromMetadata(metadata: Map[String, String]): Option[(String, Boolean)]
Checks schema metadata for sort fields
Checks schema metadata for sort fields
- metadata
schema metadata
- returns
(sort field, reverse sorted or not)
- def writeHeaderAndFirstBatch(result: SimpleFeatureVector, dictionaries: Map[String, ArrowDictionary], ipcOpts: IpcOption, sort: Option[(String, Boolean)], count: Int): Array[Byte]
Write out the header, dictionaries, and first batch of an arrow streaming file
Write out the header, dictionaries, and first batch of an arrow streaming file
- result
vector loaded with first batch
- dictionaries
dictionaries
- sort
sort
- count
number of records in first batch
- object BatchWriter
- object ConcatenatedFileWriter
- object DeltaWriter extends StrictLogging
- object DictionaryBuildingWriter
- object FormatVersion
- object Metadata
- object SimpleFeatureArrowFileReader
- object SimpleFeatureArrowFileWriter