ai.preferred.venom.fetcher

Class HttpCloseableFetcher.Builder

java.lang.Object

ai.preferred.venom.fetcher.HttpCloseableFetcher.Builder

Enclosing class:

HttpCloseableFetcher

public static class HttpCloseableFetcher.Builder
extends Object

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	Builder()

Method Summary

Modifier and Type	Method and Description
HttpCloseableFetcher	build() Builds the fetcher with the options specified.
HttpCloseableFetcher.Builder	compressed(boolean compressed) Set whether to request for compress pages and to decompress pages after it is fetched.
HttpCloseableFetcher.Builder	connectionRequestTimeout(int connectionRequestTimeout) The timeout in milliseconds used when requesting a connection from the connection manager.
HttpCloseableFetcher.Builder	connectTimeout(int connectTimeout) Determines the timeout in milliseconds until a connection is established.
HttpCloseableFetcher.Builder	fileManager(@NotNull FileManager fileManager) Sets the FileManager to be used.
HttpCloseableFetcher.Builder	headers(@NotNull Map<String,String> headers) Sets the headers to be used when fetching items.
HttpCloseableFetcher.Builder	numIoThreads(int numIoThreads) Number of httpclient dispatcher threads.
HttpCloseableFetcher.Builder	proxyProvider(@NotNull ProxyProvider proxyProvider) Sets the ProxyProvider to be used.
HttpCloseableFetcher.Builder	register(@NotNull Callback callback) Register any callbacks that will be called when a page has been fetched.
HttpCloseableFetcher.Builder	router(@NotNull ValidatorRouter router) Sets ValidatorRouter to be used.
HttpCloseableFetcher.Builder	socketTimeout(int socketTimeout) Defines the socket timeout (SO_TIMEOUT) in milliseconds, which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets).
HttpCloseableFetcher.Builder	soTimeout(int soTimeout) Determines the default socket timeout value for non-blocking I/O operations.
HttpCloseableFetcher.Builder	stopCodes(int... codes)
HttpCloseableFetcher.Builder	threadFactory(@NotNull ThreadFactory threadFactory) Set the thread factory that creates the httpclient dispatcher threads.
HttpCloseableFetcher.Builder	userAgentGenerator(@NotNull UserAgentGenerator userAgentGenerator) Sets the UserAgentGenerator to be used, if not set, default will be chosen.
HttpCloseableFetcher.Builder	validator(@NotNull ResponseValidator validator) Sets the ResponseValidator to be used.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Builder

protected Builder()

Method Detail

register

public HttpCloseableFetcher.Builder register(@NotNull
                                             @NotNull Callback callback)

Register any callbacks that will be called when a page has been fetched.

Parameters:

callback - A set of FetcherCallback.

Returns:

this.

fileManager

public HttpCloseableFetcher.Builder fileManager(@NotNull
                                                @NotNull FileManager fileManager)

Sets the FileManager to be used. Defaults to none.

If fileManager is set, all items fetched will be saved to storage.

Parameters:

fileManager - file manager to be used.

Returns:

this.

headers

public HttpCloseableFetcher.Builder headers(@NotNull
                                            @NotNull Map<String,String> headers)

Sets the headers to be used when fetching items. Defaults to none.

Parameters:

headers - a map to headers to be used.

Returns:

this.

numIoThreads

public HttpCloseableFetcher.Builder numIoThreads(int numIoThreads)

Number of httpclient dispatcher threads.

Parameters:

numIoThreads - number of threads.

Returns:

this.

proxyProvider

public HttpCloseableFetcher.Builder proxyProvider(@NotNull
                                                  @NotNull ProxyProvider proxyProvider)

Sets the ProxyProvider to be used. Defaults to none.

Parameters:

proxyProvider - proxy provider to be used

Returns:

this.

stopCodes

public HttpCloseableFetcher.Builder stopCodes(int... codes)

threadFactory

public HttpCloseableFetcher.Builder threadFactory(@NotNull
                                                  @NotNull ThreadFactory threadFactory)

Set the thread factory that creates the httpclient dispatcher threads.

Parameters:

threadFactory - an instance of ThreadFactory.

Returns:

this.

userAgentGenerator

public HttpCloseableFetcher.Builder userAgentGenerator(@NotNull
                                                       @NotNull UserAgentGenerator userAgentGenerator)

Sets the UserAgentGenerator to be used, if not set, default will be chosen.

Parameters:

userAgentGenerator - user agent generator to be used.

Returns:

this.

validator

public HttpCloseableFetcher.Builder validator(@NotNull
                                              @NotNull ResponseValidator validator)

Sets the ResponseValidator to be used. Defaults to none.

This will validate the fetched page and retry if page is not consistent with the specification set by the validator.

Parameters:

validator - validator to be used.

Returns:

this.

router

public HttpCloseableFetcher.Builder router(@NotNull
                                           @NotNull ValidatorRouter router)

Sets ValidatorRouter to be used. Defaults to none. Validator rules set in validator will always be used.

Parameters:

router - router validator router to be used.

Returns:

this.

connectionRequestTimeout

public HttpCloseableFetcher.Builder connectionRequestTimeout(int connectionRequestTimeout)

The timeout in milliseconds used when requesting a connection from the connection manager. A timeout value of zero is interpreted as an infinite timeout.

Parameters:

connectionRequestTimeout - timeout.

Returns:

this.

connectTimeout

public HttpCloseableFetcher.Builder connectTimeout(int connectTimeout)

Determines the timeout in milliseconds until a connection is established. A timeout value of zero is interpreted as an infinite timeout.

Parameters:

connectTimeout - timeout.

Returns:

this.

socketTimeout

public HttpCloseableFetcher.Builder socketTimeout(int socketTimeout)

Defines the socket timeout (SO_TIMEOUT) in milliseconds, which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets).

Parameters:

socketTimeout - timeout.

Returns:

this.

soTimeout

public HttpCloseableFetcher.Builder soTimeout(int soTimeout)

Determines the default socket timeout value for non-blocking I/O operations.

Parameters:

soTimeout - timeout.

Returns:

this.

compressed

public HttpCloseableFetcher.Builder compressed(boolean compressed)

Set whether to request for compress pages and to decompress pages after it is fetched. Defaults to true.

Parameters:

compressed - should request for compress pages

Returns:

this.

build

public HttpCloseableFetcher build()

Builds the fetcher with the options specified.

Returns:

an instance of Fetcher.