Uses of Interface
org.pipecraft.pipes.sync.Pipe
-
-
Uses of Pipe in org.pipecraft.pipes.async.inter
Methods in org.pipecraft.pipes.async.inter with parameters of type Pipe Modifier and Type Method Description static <T> SyncToAsyncPipe<T>SyncToAsyncPipe. fromPipe(Pipe<T> input)Method parameters in org.pipecraft.pipes.async.inter with type arguments of type Pipe Modifier and Type Method Description static <T> SyncToAsyncPipe<T>SyncToAsyncPipe. fromPipes(Collection<Pipe<T>> inputs, int threadCount) -
Uses of Pipe in org.pipecraft.pipes.sync.inter
Classes in org.pipecraft.pipes.sync.inter that implement Pipe Modifier and Type Class Description classAsyncToSyncPipe<T>A pipe which acts as a converter from async pipe/s to a sync pipe.classCallbackPipe<T>A pipe invoking a callback method per item passed through it.classCompoundPipe<T>A pipe encapsulating a pipeline.classConcatPipe<T>An intermediate pipe consisting of concatenating the contents of a series of pipesclassCountPipeA pipe counting the items in the input pipe, and returning the count as a single output item.classCSVMapperPipe<T>An intermediate pipe that parses strings in CSV format, and binds each line to an objectclassDedupPipe<T>Uses item equality (equals() method) in input pipe items for performing a dedup operation.classDelegatePipe<T>A pipe delegating all work to another pipe given in the constructor.classFilterBasePipe<T>A base-class pipe implementation for pipes that filter an input pipe using some criteria.classFilterPipe<T>A pipe filtering an input pipe by some predicate on the pipe items.classFlexibleMapPipe<S,T>An intermediate pipe that maps an input item into zero or more items of another type.classGrouperPipe<T>Splits the input items into different families, and emits items from the same family sequentially.classHeadPipe<T>A pipe returning the first K items of its input pipe.classIntermediateBinFileWriterPipe<T>The intermediate version ofBinFileWriterPipe.classIntermediateSharderBySeqPipe<T>An intermediate version ofSharderBySeqPipeclassIntermediateTxtFileWriterPipe<T>A pipe write to a text file any item passed through it.classMapPipe<S,T>A pipe transforming each item in the input pipe into another, possibly with a different type.classMultiSortedBasePipe<T>A base class for pipes performing some set operation on a collection of sorted input pipes.classOrderValidationPipe<T>An intermediate pipe that validates the order of the consumed elements by a comparatorclassProgressPipe<T>A no-op pipe with only one side effect - progress tracking.classSkipPipe<T>A pipe skipping the first K items of its input pipe.classSortedIntersectionPipe<T>An intermediate pipe performing set intersection operation on a collection of sorted input pipes.classSortedMergePipe<T>An intermediate pipe sort-merging the contents of a collection of sorted input pipes.classSortedSubtractionPipe<T>An intermediate pipe performing a set subtraction operation on two given pipes (i.e.classSortedUnionPipe<T>An intermediate pipe performing set union operation on a collection of sorted input pipes.classSortPipe<T>An intermediate pipe performing a sort on an input pipe, without deduping.classTopKPipe<T>A pipe returning the top K items of the input pipe, according to some predefined order relation.Methods in org.pipecraft.pipes.sync.inter that return Pipe Modifier and Type Method Description protected abstract Pipe<T>CompoundPipe. createPipeline()protected Pipe<T>GrouperPipe. createPipeline()Pipe<T>DelegatePipe. getOriginPipe()Methods in org.pipecraft.pipes.sync.inter that return types with arguments of type Pipe Modifier and Type Method Description protected List<? extends Pipe<T>>MultiSortedBasePipe. getInputPipes()Methods in org.pipecraft.pipes.sync.inter with parameters of type Pipe Modifier and Type Method Description static <T> ConcatPipe<T>ConcatPipe. fromPipes(Pipe<T>... inputs)Method parameters in org.pipecraft.pipes.sync.inter with type arguments of type Pipe Modifier and Type Method Description static <T> ConcatPipe<T>ConcatPipe. fromPipes(List<Pipe<T>> inputs)Constructors in org.pipecraft.pipes.sync.inter with parameters of type Pipe Constructor Description CallbackPipe(Pipe<T> input, FailableConsumer<? super T,? extends PipeException> itemCallback)ConstructorCallbackPipe(Pipe<T> input, FailableConsumer<? super T,? extends PipeException> itemCallback, FailableRunnable<? extends PipeException> terminationCallback)ConstructorCallbackPipe(Pipe<T> input, FailableRunnable<? extends PipeException> terminationCallback)ConstructorCountPipe(Pipe<?> input)constructorCSVMapperPipe(Pipe<String> input, boolean hasHeader, String[] columnNames, FailableFunction<Map<String,String>,T,PipeException> rowConverter)Constructor Constructs a mapper with the default delimiter (',').CSVMapperPipe(Pipe<String> input, char delimiter, boolean hasHeader, String[] columnNames, FailableFunction<Map<String,String>,T,PipeException> rowConverter)ConstructorCSVMapperPipe(Pipe<String> input, FailableFunction<Map<String,String>,T,PipeException> rowConverter)Constructor Constructs a mapper with the default delimiter (','), expecting a header, and using the header column names for the row converter.DedupPipe(Pipe<T> input, CodecFactory<T> inputCodec, int partitionCount, File tmpFolder)ConstructorDelegatePipe(Pipe<T> originPipe)ConstructorFilterBasePipe(Pipe<T> input)ConstructorFilterPipe(Pipe<T> input, Predicate<? super T> predicate)ConstructorFlexibleMapPipe(Pipe<S> input, PipeFactory<S,? extends T> factory)ConstructorGrouperPipe(Pipe<T> input, FailableFunction<T,?,PipeException> discriminator, CodecFactory<T> inputCodec, int partitionCount, File tmpFolder)ConstructorHeadPipe(Pipe<T> input, int n)ConstructorIntermediateBinFileWriterPipe(Pipe<T> input, File outputFile, FileWriteOptions options, EncoderFactory<? super T> encoderFactory)ConstructorIntermediateBinFileWriterPipe(Pipe<T> input, File outputFile, EncoderFactory<? super T> encoderFactory)Constructor Uses default file write settings (see @{FileWriteOptions})IntermediateSharderBySeqPipe(Pipe<T> input, EncoderFactory<? super T> encoderFactory, FailableFunction<? super T,String,PipeException> shardSelectorFunction, File folder)Constructor Uses default file write optionsIntermediateSharderBySeqPipe(Pipe<T> input, EncoderFactory<? super T> encoderFactory, FailableFunction<? super T,String,PipeException> shardSelectorFunction, File folder, FileWriteOptions fileWriteOptions)ConstructorIntermediateTxtFileWriterPipe(Pipe<T> input, File output, Charset charset, FailableFunction<T,String,PipeException> stringMapper)Constructor Uses default file write options.IntermediateTxtFileWriterPipe(Pipe<T> input, File output, Charset charset, FileWriteOptions options)Constructor Uses the toString as the mapper from the object to StringIntermediateTxtFileWriterPipe(Pipe<T> input, File output, Charset charset, FileWriteOptions options, FailableFunction<T,String,PipeException> stringMapper)ConstructorIntermediateTxtFileWriterPipe(Pipe<T> input, File output, FailableFunction<T,String,PipeException> stringMapper)Constructor Uses default file write options and UTF8.IntermediateTxtFileWriterPipe(Pipe<T> input, File output, FileWriteOptions options)Constructor Uses default toString() as the string mapper, and UTF8IntermediateTxtFileWriterPipe(Pipe<T> input, File output, FileWriteOptions options, FailableFunction<T,String,PipeException> stringMapper)Constructor Uses UTF8MapPipe(Pipe<S> input, FailableFunction<? super S,T,PipeException> mapFunction)constructorMultiSortedBasePipe(Comparator<? super T> comparator, Pipe<T>... inputPipes)ConstructorOrderValidationPipe(Pipe<T> input, Comparator<T> comparator)ConstructorProgressPipe(Pipe<T> input, int sampleRate, Consumer<Integer> callback)ConstructorSkipPipe(Pipe<T> input, int n)ConstructorSortedIntersectionPipe(Comparator<? super T> comparator, Pipe<T>... inputPipes)ConstructorSortedMergePipe(Comparator<? super T> comparator, Pipe<T>... inputPipes)ConstructorSortedSubtractionPipe(Pipe<T> pipeA, Pipe<T> pipeB, Comparator<? super T> comparator)ConstructorSortedUnionPipe(Comparator<? super T> comparator, Pipe<T>... inputPipes)ConstructorSortPipe(Pipe<T> input, int maxItemsInMemory, File tmpFolder, CodecFactory<T> codecFactory, Comparator<T> comparator)Constructor Uses compression on temp files.SortPipe(Pipe<T> input, int maxItemsInMemory, File tmpFolder, EncoderFactory<? super T> encoderFactory, DecoderFactory<T> decoderFactory, Comparator<T> comparator)Constructor Uses compression on temp files.SortPipe(Pipe<T> input, int maxItemsInMemory, File tmpFolder, EncoderFactory<? super T> encoderFactory, DecoderFactory<T> decoderFactory, Comparator<T> comparator, Compression tempFilesCompression)ConstructorSortPipe(Pipe<T> input, int maxItemsInMemory, CodecFactory<T> codecFactory, Comparator<T> comparator)Constructor Uses compression on temp files, and uses system default temp folderSortPipe(Pipe<T> input, Comparator<T> comparator)Constructor Use this constructor for disabling disk usage and sorting in-memory only.TopKPipe(Pipe<T> input, int k, Comparator<? super T> comparator)ConstructorConstructor parameters in org.pipecraft.pipes.sync.inter with type arguments of type Pipe Constructor Description MultiSortedBasePipe(List<? extends Pipe<T>> inputPipes, Comparator<? super T> comparator)ConstructorSortedIntersectionPipe(List<? extends Pipe<T>> inputPipes, Comparator<? super T> comparator)ConstructorSortedMergePipe(List<? extends Pipe<T>> inputPipes, Comparator<? super T> comparator)ConstructorSortedUnionPipe(List<? extends Pipe<T>> inputPipes, Comparator<? super T> comparator)Constructor -
Uses of Pipe in org.pipecraft.pipes.sync.inter.join
Classes in org.pipecraft.pipes.sync.inter.join that implement Pipe Modifier and Type Class Description classHashJoinPipe<K,L,R>A pipe performing a join operation between a 'left' pipe of type L, and a list of 'right' pipes of type R.classLookupJoinPipe<K,L,R>A pipe performing a join operation between a 'left' pipe of type L, and a list of 'right' pipes of type R.classSortedJoinPipe<K,L,R>A pipe performing a join operation between a 'left' pipe of type L, and a list of 'right' pipes of type R.Methods in org.pipecraft.pipes.sync.inter.join that return Pipe Modifier and Type Method Description protected Pipe<JoinRecord<K,L,R>>HashJoinPipe. createPipeline()Methods in org.pipecraft.pipes.sync.inter.join with parameters of type Pipe Modifier and Type Method Description floatJoinMode. resolveProgress(Pipe<?> leftPipe, Collection<? extends Pipe<?>> rightPipes)Method parameters in org.pipecraft.pipes.sync.inter.join with type arguments of type Pipe Modifier and Type Method Description floatJoinMode. resolveProgress(Pipe<?> leftPipe, Collection<? extends Pipe<?>> rightPipes)Constructors in org.pipecraft.pipes.sync.inter.join with parameters of type Pipe Constructor Description HashJoinPipe(Pipe<L> leftPipe, FailableFunction<L,K,PipeException> leftKeyExtractor, List<? extends Pipe<R>> rightPipes, FailableFunction<R,K,PipeException> rightKeyExtractor, JoinMode joinMode, int partitionCount, CodecFactory<L> leftCodec, CodecFactory<R> rightCodec, File tmpFolder)ConstructorHashJoinPipe(Pipe<L> leftPipe, FailableFunction<L,K,PipeException> leftKeyExtractor, Pipe<R> rightPipe, FailableFunction<R,K,PipeException> rightKeyExtractor, JoinMode joinMode, int partitionCount, CodecFactory<L> leftCodec, CodecFactory<R> rightCodec, File tmpFolder)Constructor To be used when there's a single right pipe.LookupJoinPipe(Pipe<L> leftPipe, FailableFunction<L,K,PipeException> leftKeyExtractor, List<? extends Pipe<R>> rightPipes, FailableFunction<R,K,PipeException> rightKeyExtractor, JoinMode joinMode)ConstructorLookupJoinPipe(Pipe<L> leftPipe, FailableFunction<L,K,PipeException> leftKeyExtractor, Pipe<R> rightPipe, FailableFunction<R,K,PipeException> rightKeyExtractor, JoinMode joinMode)Constructor To be used when there's a single right pipe.SortedJoinPipe(Pipe<L> leftPipe, FailableFunction<L,K,PipeException> leftKeyExtractor, List<? extends Pipe<R>> rightPipes, FailableFunction<R,K,PipeException> rightKeyExtractor, Comparator<? super K> keyComparator, JoinMode joinMode)ConstructorSortedJoinPipe(Pipe<L> leftPipe, FailableFunction<L,K,PipeException> leftKeyExtractor, Pipe<R> rightPipe, FailableFunction<R,K,PipeException> rightKeyExtractor, Comparator<? super K> keyComparator, JoinMode joinMode)Constructor To be used when there's a single right pipe.Constructor parameters in org.pipecraft.pipes.sync.inter.join with type arguments of type Pipe Constructor Description HashJoinPipe(List<? extends Pipe<R>> rightPipes, FailableFunction<R,K,PipeException> rightKeyExtractor, int partitionCount, CodecFactory<R> rightCodec, File tmpFolder)Constructor A constructor for the case of no left pipe.HashJoinPipe(Pipe<L> leftPipe, FailableFunction<L,K,PipeException> leftKeyExtractor, List<? extends Pipe<R>> rightPipes, FailableFunction<R,K,PipeException> rightKeyExtractor, JoinMode joinMode, int partitionCount, CodecFactory<L> leftCodec, CodecFactory<R> rightCodec, File tmpFolder)ConstructorLookupJoinPipe(List<? extends Pipe<R>> rightPipes, FailableFunction<R,K,PipeException> rightKeyExtractor)Constructor A constructor for the case of no left pipe.LookupJoinPipe(Pipe<L> leftPipe, FailableFunction<L,K,PipeException> leftKeyExtractor, List<? extends Pipe<R>> rightPipes, FailableFunction<R,K,PipeException> rightKeyExtractor, JoinMode joinMode)ConstructorSortedJoinPipe(List<? extends Pipe<R>> rightPipes, FailableFunction<R,K,PipeException> rightKeyExtractor, Comparator<? super K> keyComparator)Constructor A constructor for the case of no left pipe.SortedJoinPipe(Pipe<L> leftPipe, FailableFunction<L,K,PipeException> leftKeyExtractor, List<? extends Pipe<R>> rightPipes, FailableFunction<R,K,PipeException> rightKeyExtractor, Comparator<? super K> keyComparator, JoinMode joinMode)Constructor -
Uses of Pipe in org.pipecraft.pipes.sync.inter.reduct
Classes in org.pipecraft.pipes.sync.inter.reduct that implement Pipe Modifier and Type Class Description classHashReductorPipe<I,O>Scans the input pipe and performs a reduction operation on families of items based on some discriminating property.classListReductorPipe<S,T>Scans the input pipe and groups sequential items based on some discriminating property.classSequenceReductorPipe<I,O>Scans the input pipe and performs a reduction operation on a sequence of items with some discriminating property.Methods in org.pipecraft.pipes.sync.inter.reduct that return Pipe Modifier and Type Method Description protected Pipe<O>HashReductorPipe. createPipeline()Constructors in org.pipecraft.pipes.sync.inter.reduct with parameters of type Pipe Constructor Description HashReductorPipe(Pipe<I> input, CodecFactory<I> inputCodec, int partitionCount, File tmpFolder, ReductorConfig<I,?,?,O> reductorConfig)ConstructorListReductorPipe(Pipe<S> input, FailableFunction<S,?,PipeException> discriminator, Function<List<S>,T> postProcessor)Constructor Creates a sequence reductor pipe with no protective threshold on group sizes.ListReductorPipe(Pipe<S> input, FailableFunction<S,?,PipeException> discriminator, Function<List<S>,T> postProcessor, int groupSizeLimit, ListReductorPipe.GroupSizeLimitPolicy sizeLimitPolicy)ConstructorSequenceReductorPipe(Pipe<I> input, ReductorConfig<I,?,?,O> reductorConfig)Constructor -
Uses of Pipe in org.pipecraft.pipes.sync.inter.sample
Classes in org.pipecraft.pipes.sync.inter.sample that implement Pipe Modifier and Type Class Description classExactSamplerPipe<T>A pipe sampling exactly m items of an input pipe containing n items, where n is known in advance.classPortionSamplerPipe<T>A pipe sampling about x% of the incoming items.Constructors in org.pipecraft.pipes.sync.inter.sample with parameters of type Pipe Constructor Description ExactSamplerPipe(Pipe<T> input, int n, int m)ConstructorExactSamplerPipe(Pipe<T> input, int n, int m, Random rnd)ConstructorPortionSamplerPipe(Pipe<T> input, double p)ConstructorPortionSamplerPipe(Pipe<T> input, double p, Random rnd)Constructor -
Uses of Pipe in org.pipecraft.pipes.sync.source
Classes in org.pipecraft.pipes.sync.source that implement Pipe Modifier and Type Class Description classBinInputReaderPipe<T>A source pipe reading items from a given binary input (file or input stream).classCollectionReaderPipe<T>A pipe based on a given collection of items.classEmptyPipe<T>A source pipe with no data in it.classErrorPipe<T>A source pipe with no data in it, which only produces a predefinedPipeException.classInputStreamPipe<T>A base class for source pipes based on some input stream.classIteratorReaderPipe<T>A pipe based on an iterator.classJdbcQueryResultsPipe<T>A source pipe reading rows from a DB query result set, and converting them to some entity typeclassMultiFileReaderPipe<T>A source pipe reading multiple local binary files.classMultiTxtFileReaderPipeReads data from multiple files in local disk under some folder, as if they were concatenated using some predefined order.classQueueReaderPipe<T>A source pipe reading the contents of aBlockingQueue.classReaderPipeA source pipe that reads lines from a given reader.classSeqGenPipe<T>A pipe based on a given sequence generator function.classStorageMultiFileReaderPipe<T,B>A source pipe reading multiple remote binary files from the cloud.classStorageMultiTxtFileReaderPipe<B>Reads data from multiple files in cloud storage under some folder, as if they were concatenated using some predefined order.classStorageTxtFileFetcherReaderPipeA source pipe providing the lines of a text file from cloud storage.classStorageTxtFileReaderPipeA source pipe providing the lines of a text file in someStorage, in a streaming manner.classTxtFileReaderPipeA source pipe reading lines from a given local text file or input stream containing text content.classURLTxtReaderPipeA source pipe providing the lines of a text resource defined by a URL.Methods in org.pipecraft.pipes.sync.source with type parameters of type Pipe Modifier and Type Method Description static <C extends Pipe<?>>
CEmptyPipe. instance()Methods in org.pipecraft.pipes.sync.source that return Pipe Modifier and Type Method Description protected Pipe<T>MultiFileReaderPipe. createPipeline()protected Pipe<String>MultiTxtFileReaderPipe. createPipeline()protected Pipe<T>StorageMultiFileReaderPipe. createPipeline()protected Pipe<String>StorageMultiTxtFileReaderPipe. createPipeline()protected Pipe<String>StorageTxtFileFetcherReaderPipe. createPipeline() -
Uses of Pipe in org.pipecraft.pipes.terminal
Constructors in org.pipecraft.pipes.terminal with parameters of type Pipe Constructor Description BinFileWriterPipe(Pipe<T> input, File outputFile, FileWriteOptions options, EncoderFactory<? super T> encoderFactory)ConstructorBinFileWriterPipe(Pipe<T> input, File outputFile, EncoderFactory<? super T> encoderFactory)Constructor Uses default file write settings (see @{FileWriteOptions})CollectionWriterPipe(Pipe<T> input)Constructor Collects items into anArrayListCollectionWriterPipe(Pipe<T> input, Collection<T> outputCollection)ConstructorConsumerPipe(Pipe<T> input)Constructor Builds a no-op consumerConsumerPipe(Pipe<T> input, Runnable terminationAction)ConstructorConsumerPipe(Pipe<T> input, FailableConsumer<? super T,PipeException> itemAction)ConstructorConsumerPipe(Pipe<T> input, FailableConsumer<? super T,PipeException> itemAction, Runnable terminationAction)ConstructorPercentilePipe(Pipe<T> input, double p, Function<T,C> sortingComponentExtractor)ConstructorQueueWriterPipe(Pipe<T> inputPipe, BlockingQueue<T> queue, T successMarker, T errorMarker)ConstructorSharderByHashPipe(Pipe<T> input, EncoderFactory<T> encoderFactory, FailableFunction<? super T,?,PipeException> featureSelectorFunction, int shardCount, File folder, FileWriteOptions writeOptions)ConstructorSharderByHashPipe(Pipe<T> input, EncoderFactory<T> encoderFactory, FailableFunction<? super T,?,PipeException> featureSelectorFunction, Function<Integer,String> fileNameFunction, int shardCount, File folder, FileWriteOptions writeOptions)ConstructorSharderByItemPipe(Pipe<T> input, EncoderFactory<? super T> encoderFactory, FailableFunction<? super T,String,PipeException> shardSelectorFunction, File folder)ConstructorSharderByItemPipe(Pipe<T> input, EncoderFactory<? super T> encoderFactory, FailableFunction<? super T,String,PipeException> shardSelectorFunction, File folder, FileWriteOptions writeOptions)ConstructorSharderBySeqPipe(Pipe<T> input, EncoderFactory<? super T> encoderFactory, FailableFunction<? super T,String,PipeException> shardSelectorFunction, File folder)Constructor Uses default file write optionsSharderBySeqPipe(Pipe<T> input, EncoderFactory<? super T> encoderFactory, FailableFunction<? super T,String,PipeException> shardSelectorFunction, File folder, FileWriteOptions fileWriteOptions)ConstructorStorageTxtFileWriterPipe(Pipe<String> input, Storage<?,?> storage, String bucket, String path)Constructor Uses default write options and chunk size, and assumes UTF8StorageTxtFileWriterPipe(Pipe<String> input, Storage<?,?> gs, String bucket, String path, Charset charset)Constructor Uses the default write options and chunk sizeStorageTxtFileWriterPipe(Pipe<String> input, Storage<?,?> storage, String bucket, String path, Charset charset, int chunkSize, FileWriteOptions options)ConstructorStorageTxtFileWriterPipe(Pipe<String> input, Storage<?,?> storage, String bucket, String path, FileWriteOptions options)Constructor Uses default chunk size and UTF8TxtFileWriterPipe(Pipe<String> input, File f)Constructor Uses default file write options and UTF8 charset encoding.TxtFileWriterPipe(Pipe<String> input, File f, Charset charset)Constructor Uses default file write options.TxtFileWriterPipe(Pipe<String> input, File f, Charset charset, FileWriteOptions options)ConstructorTxtFileWriterPipe(Pipe<String> input, File f, FileWriteOptions options)Constructor Uses UTF8 charset encoding.WriterPipe(Pipe<String> input, Writer w)ConstructorWriterPipe(Pipe<String> input, Writer w, int bufferSize)Constructor -
Uses of Pipe in org.pipecraft.pipes.utils
Methods in org.pipecraft.pipes.utils that return Pipe Modifier and Type Method Description Pipe<T>PipeFactory. get(P param)Pipe<T>PipeReaderSupplier. get(SizedInputStream is, M metadata)Pipe<T>PipeSupplier. get()
-