org.icij.extract

Class Scanner

java.lang.Object

org.icij.concurrent.ExecutorProxy

org.icij.extract.Scanner

All Implemented Interfaces:

Shutdownable

public class Scanner
extends ExecutorProxy

Scanner for scanning the directory tree starting at a given path. Each time scan(java.nio.file.Path) is called, the job is put in an unbounded queue and executed in serial. This makes sense as it's usually the file system which is a bottleneck and not the CPU, so parallelization won't help. The scan(java.nio.file.Path) method is non-blocking, which is useful for creating parallelized producer-consumer setups, where files are processed as they're scanned. Encountered documents are put in a given queue. This is a classic producer, putting elements in a queue which are then extracted by a consumer. The queue should be bounded, to avoid the scanner filling up memory, but the bound should be high enough to create a significant buffer between the scanner and the consumer. Documents are pushed into the queue synchronously and if the queue is bounded, only when a space becomes available. This implementation is thread-safe.

Since:

1.0.0-beta

Field Summary

Fields

Modifier and Type	Field and Description
protected BlockingQueue<TikaDocument>	queue

Fields inherited from class org.icij.concurrent.ExecutorProxy

executor

Constructor Summary

Constructors

Constructor and Description
Scanner(DocumentFactory factory, BlockingQueue<TikaDocument> queue)
Scanner(DocumentFactory factory, BlockingQueue<TikaDocument> queue, SealableLatch latch)
Scanner(DocumentFactory factory, BlockingQueue<TikaDocument> queue, SealableLatch latch, Notifiable notifiable) Creates a Scanner that sends all results straight to the underlying BlockingQueue on a single thread.

Method Summary

Modifier and Type	Method and Description
Scanner	configure(Options<String> options) Configure the scanner with the given options.
ScannerVisitor	createScannerVisitor(Path path)
void	exclude(String pattern) Add a glob pattern for excluding files and directories.
boolean	followSymLinks() Check whether symlinks will be followed.
void	followSymLinks(boolean followLinks) Set whether symlinks should be followed.
SealableLatch	getLatch() Get the latch.
int	getMaxDepth() Get the currently set maximum depth to recurse when scanning.
long	getNumberOfFiles(Path path)
boolean	ignoreHiddenFiles() Check whether hidden files will be ignored.
void	ignoreHiddenFiles(boolean ignoreHiddenFiles) Set whether hidden files should be ignored.
boolean	ignoreSystemFiles() Check whether system files will be ignored.
void	ignoreSystemFiles(boolean ignoreSystemFiles) Set whether system files should be ignored.
void	include(String pattern) Add a glob pattern for including files.
long	queued()
Future<Path>	scan(Path path) Queue a scanning job.
List<Future<Path>>	scan(Path[] paths) Submit all of the given paths to the scanner for execution, returning a list of Future objects representing those tasks.
List<Future<Path>>	scan(String[] paths)
void	setMaxDepth(int maxDepth) Set the maximum depth to recurse when scanning.

Methods inherited from class org.icij.concurrent.ExecutorProxy

awaitTermination, shutdown, shutdownNow

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

queue

protected final BlockingQueue<TikaDocument> queue

Constructor Detail

Scanner

public Scanner(DocumentFactory factory,
               BlockingQueue<TikaDocument> queue)

Scanner

public Scanner(DocumentFactory factory,
               BlockingQueue<TikaDocument> queue,
               SealableLatch latch)

Scanner

public Scanner(DocumentFactory factory,
               BlockingQueue<TikaDocument> queue,
               SealableLatch latch,
               Notifiable notifiable)

Creates a Scanner that sends all results straight to the underlying BlockingQueue on a single thread.

Parameters:

queue - results from the scanner will be put on this queue

latch - signalled when a document is queued

notifiable - receives notifications when new file documents are queued

Method Detail

configure

public Scanner configure(Options<String> options)

Configure the scanner with the given options.

Parameters:

options - options for configuring the scanner

Returns:

the scanner

include

public void include(String pattern)

Add a glob pattern for including files. Files not matching the pattern will be ignored.

Parameters:

pattern - the glob pattern

exclude

public void exclude(String pattern)

Add a glob pattern for excluding files and directories.

Parameters:

pattern - the glob pattern

followSymLinks

public void followSymLinks(boolean followLinks)

Set whether symlinks should be followed.

Parameters:

followLinks - whether to follow symlinks

followSymLinks

public boolean followSymLinks()

Check whether symlinks will be followed.

Returns:

whether symlinks will be followed

ignoreHiddenFiles

public void ignoreHiddenFiles(boolean ignoreHiddenFiles)

Set whether hidden files should be ignored. File names starting with a dot will always be ignored if set to true, but DOS hidden files will only be ignored if the filesystem supports the DOS hidden fileattribute.

Parameters:

ignoreHiddenFiles - whether to ignore hidden files

ignoreHiddenFiles

public boolean ignoreHiddenFiles()

Check whether hidden files will be ignored.

Returns:

whether hidden files will be ignored

ignoreSystemFiles

public void ignoreSystemFiles(boolean ignoreSystemFiles)

Set whether system files should be ignored.

Parameters:

ignoreSystemFiles - whether to ignore system files

ignoreSystemFiles

public boolean ignoreSystemFiles()

Check whether system files will be ignored.

Returns:

whether system files are ignore

setMaxDepth

public void setMaxDepth(int maxDepth)

Set the maximum depth to recurse when scanning.

Parameters:

maxDepth - maximum depth

getMaxDepth

public int getMaxDepth()

Get the currently set maximum depth to recurse when scanning.

Returns:

maximum depth

getLatch

public SealableLatch getLatch()

Get the latch.

Returns:

The latch or null if none is set.

queued

public long queued()

Returns:

The total number of queued documents over the lifetime of this scanner.

scan

public Future<Path> scan(Path path)

Queue a scanning job. Jobs are put in an unbounded queue and executed in serial, in a separate thread. This method doesn't block. Call ExecutorProxy.awaitTermination(long, TimeUnit) to block.

Parameters:

path - the path to scan

Returns:

A Future that can be used to wait on the result or cancel.

createScannerVisitor

public ScannerVisitor createScannerVisitor(Path path)

scan

public List<Future<Path>> scan(Path[] paths)

Submit all of the given paths to the scanner for execution, returning a list of Future objects representing those tasks.

Returns:

a Future for each path scanned

See Also:

scan(Path)

scan

public List<Future<Path>> scan(String[] paths)

See Also:

scan(Path[])

getNumberOfFiles

public long getNumberOfFiles(Path path)
                      throws IOException

Throws:

IOException