org.icij.extract.extractor

Class DocumentConsumer

java.lang.Object

org.icij.concurrent.ExecutorProxy

org.icij.extract.extractor.DocumentConsumer

All Implemented Interfaces:

java.util.function.Consumer<TikaDocument>, Shutdownable

public class DocumentConsumer
extends ExecutorProxy
implements java.util.function.Consumer<TikaDocument>

Base consumer for documents. Superclasses should call accept(TikaDocument). All tasks are sent to a work-stealing thread pool. The parallelism of the thread pool is defined in the call to the constructor. A task is defined as both the extraction from a file and the output of extracted data. Completion is only considered successful if both parts of the task complete with no exceptions. The final status of each task is saved to the reporter, if any is set.

Since:

1.0.0-beta

Field Summary

Fields

Modifier and Type	Field and Description
protected Extractor	extractor
protected Spewer	spewer

Fields inherited from class org.icij.concurrent.ExecutorProxy

executor

Constructor Summary

Constructors

Constructor and Description
DocumentConsumer(Spewer spewer, Extractor extractor) Create a new consumer with the default pool size, which is the number of available processors.
DocumentConsumer(Spewer spewer, Extractor extractor, ExecutorService executor) Create a new consumer that submits tasks to the given Executor.
DocumentConsumer(Spewer spewer, Extractor extractor, int poolSize) Create a new consumer with the given pool size.

Method Summary

Modifier and Type	Method and Description
void	accept(TikaDocument tikaDocument) Consume a file.
static int	defaultPoolSize() Returns the default thread pool size, which is equivalent to the number of available processors minus 1, or 1 - whichever is greater.
Reporter	getReporter() Get the reporter.
void	setReporter(Reporter reporter) Set the reporter.

Methods inherited from class org.icij.concurrent.ExecutorProxy

awaitTermination, shutdown, shutdownNow

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.function.Consumer

andThen

Field Detail

spewer

protected final Spewer spewer

extractor

protected final Extractor extractor

Constructor Detail

DocumentConsumer

public DocumentConsumer(Spewer spewer,
                        Extractor extractor,
                        ExecutorService executor)

Create a new consumer that submits tasks to the given Executor.

Parameters:

spewer - the Spewer used to write extracted text and metadata

extractor - the Extractor used to extract from files

executor - the executor used to run consuming tasks

DocumentConsumer

public DocumentConsumer(Spewer spewer,
                        Extractor extractor,
                        int poolSize)

Create a new consumer with the given pool size. Uses a BlockingThreadPoolExecutor, which means that calls to accept(org.icij.extract.document.TikaDocument) will block when the thread pool is full of running tasks.

Parameters:

spewer - the Spewer used to write extracted text and metadata

extractor - the Extractor used to extract from files

poolSize - the fixed size of the thread pool used to consume documents

DocumentConsumer

public DocumentConsumer(Spewer spewer,
                        Extractor extractor)

Create a new consumer with the default pool size, which is the number of available processors.

Parameters:

spewer - the Spewer used to write extracted text and metadata

extractor - the Extractor used to extract from files

Method Detail

defaultPoolSize

public static int defaultPoolSize()

Returns the default thread pool size, which is equivalent to the number of available processors minus 1, or 1 - whichever is greater.

Returns:

the default pool size

setReporter

public void setReporter(Reporter reporter)

Set the reporter.

Parameters:

reporter - reporter

getReporter

public Reporter getReporter()

Get the reporter.

Returns:

The reporter.

accept

public void accept(TikaDocument tikaDocument)

Consume a file. If a blocking executor such as BlockingThreadPoolExecutor is being used (the default when no ExecutorService is passed to the constructor) then this method will block until a thread becomes available. Otherwise the behaviour is similar to Executor.execute(Runnable), causing the task to be put in a queue.

Specified by:

accept in interface java.util.function.Consumer<TikaDocument>

Parameters:

tikaDocument - the tikaDocument to consume

Throws:

RejectedExecutionException - if unable to queue the consumer task for execution, including when the current thread is interrupted.