ai.preferred.venom.fetcher

Class HttpFetcher.Builder

java.lang.Object

ai.preferred.venom.fetcher.HttpFetcher.Builder

Enclosing class:

HttpFetcher

public static class HttpFetcher.Builder
extends Object

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	Builder()

Method Summary

Modifier and Type	Method and Description
HttpFetcher	build() Builds the fetcher with the options specified.
HttpFetcher.Builder	compressed(boolean compressed) Set whether to request for compress pages and to decompress pages after it is fetched.
HttpFetcher.Builder	connectionRequestTimeout(int connectionRequestTimeout) The timeout in milliseconds used when requesting a connection from the connection manager.
HttpFetcher.Builder	connectTimeout(int connectTimeout) Determines the timeout in milliseconds until a connection is established.
HttpFetcher.Builder	fileManager(FileManager fileManager) Sets the FileManager to be used.
HttpFetcher.Builder	headers(Map<String,String> headers) Sets the headers to be used when fetching items.
HttpFetcher.Builder	numIoThreads(int numIoThreads) Number of httpclient dispatcher threads.
HttpFetcher.Builder	proxyProvider(ProxyProvider proxyProvider) Sets the ProxyProvider to be used.
HttpFetcher.Builder	register(Callback callback) Register any callbacks that will be called when a page has been fetched.
HttpFetcher.Builder	router(ValidatorRouter router) Sets ValidatorRouter to be used.
HttpFetcher.Builder	socketTimeout(int socketTimeout) Defines the socket timeout (SO_TIMEOUT) in milliseconds, which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets).
HttpFetcher.Builder	soTimeout(int soTimeout) Determines the default socket timeout value for non-blocking I/O operations.
HttpFetcher.Builder	stopCodes(int... codes)
HttpFetcher.Builder	threadFactory(ThreadFactory threadFactory) Set the thread factory that creates the httpclient dispatcher threads.
HttpFetcher.Builder	userAgentGenerator(UserAgentGenerator userAgentGenerator) Sets the UserAgentGenerator to be used, if not set, default will be chosen.
HttpFetcher.Builder	validator(ResponseValidator validator) Sets the ResponseValidator to be used.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Builder

protected Builder()

Method Detail

register

public HttpFetcher.Builder register(Callback callback)

Register any callbacks that will be called when a page has been fetched.

Parameters:

callback - A set of FetcherCallback.

Returns:

this.

fileManager

public HttpFetcher.Builder fileManager(FileManager fileManager)

Sets the FileManager to be used. Defaults to none.

If fileManager is set, all items fetched will be saved to storage.

Parameters:

fileManager - file manager to be used.

Returns:

this.

headers

public HttpFetcher.Builder headers(Map<String,String> headers)

Sets the headers to be used when fetching items. Defaults to none.

Parameters:

headers - a map to headers to be used.

Returns:

this.

numIoThreads

public HttpFetcher.Builder numIoThreads(int numIoThreads)

Number of httpclient dispatcher threads.

Parameters:

numIoThreads - number of threads.

Returns:

this.

proxyProvider

public HttpFetcher.Builder proxyProvider(ProxyProvider proxyProvider)

Sets the ProxyProvider to be used. Defaults to none.

Parameters:

proxyProvider - proxy provider to be used

Returns:

this.

stopCodes

public HttpFetcher.Builder stopCodes(int... codes)

threadFactory

public HttpFetcher.Builder threadFactory(ThreadFactory threadFactory)

Set the thread factory that creates the httpclient dispatcher threads.

Parameters:

threadFactory - an instance of ThreadFactory.

Returns:

this.

userAgentGenerator

public HttpFetcher.Builder userAgentGenerator(UserAgentGenerator userAgentGenerator)

Sets the UserAgentGenerator to be used, if not set, default will be chosen.

Parameters:

userAgentGenerator - user agent generator to be used.

Returns:

this.

validator

public HttpFetcher.Builder validator(ResponseValidator validator)

Sets the ResponseValidator to be used. Defaults to none.

This will validate the fetched page and retry if page is not consistent with the specification set by the validator.

Parameters:

validator - validator to be used.

Returns:

this.

router

public HttpFetcher.Builder router(ValidatorRouter router)

Sets ValidatorRouter to be used. Defaults to none. Validator rules set in validator will always be used.

Parameters:

router - router validator router to be used.

Returns:

this.

connectionRequestTimeout

public HttpFetcher.Builder connectionRequestTimeout(int connectionRequestTimeout)

The timeout in milliseconds used when requesting a connection from the connection manager. A timeout value of zero is interpreted as an infinite timeout.

Parameters:

connectionRequestTimeout - timeout.

Returns:

this.

connectTimeout

public HttpFetcher.Builder connectTimeout(int connectTimeout)

Determines the timeout in milliseconds until a connection is established. A timeout value of zero is interpreted as an infinite timeout.

Parameters:

connectTimeout - timeout.

Returns:

this.

socketTimeout

public HttpFetcher.Builder socketTimeout(int socketTimeout)

Defines the socket timeout (SO_TIMEOUT) in milliseconds, which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets).

Parameters:

socketTimeout - timeout.

Returns:

this.

soTimeout

public HttpFetcher.Builder soTimeout(int soTimeout)

Determines the default socket timeout value for non-blocking I/O operations.

Parameters:

soTimeout - timeout.

Returns:

this.

compressed

public HttpFetcher.Builder compressed(boolean compressed)

Set whether to request for compress pages and to decompress pages after it is fetched. Defaults to true.

Parameters:

compressed - should request for compress pages

Returns:

this.

build

public HttpFetcher build()

Builds the fetcher with the options specified.

Returns:

an instance of Fetcher.