ai.preferred.venom.fetcher

Class HttpClosableFetcher

java.lang.Object

ai.preferred.venom.fetcher.AbstractClosableFetcher

ai.preferred.venom.fetcher.HttpClosableFetcher

All Implemented Interfaces:

Fetcher, AutoCloseable

public class HttpClosableFetcher
extends AbstractClosableFetcher

This class holds the implementation to provide how items are fetched, to fetch the item, to validate the item and to store it if specified.

Author:

Maksim Tkachenko, Truong Quoc Tuan, Ween Jiann Lee

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HttpClosableFetcher.Builder

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HttpClosableFetcher(HttpClosableFetcher.Builder builder)

Method Summary

Modifier and Type	Method and Description
static HttpClosableFetcher	buildDefault()
static HttpClosableFetcher.Builder	builder()
void	close()
Future<Response>	fetch(Request request) Fetch the desired HTTP page given in Request.
Future<Response>	fetch(Request request, Crawler.CallbackProcessor callback) Fetch the desired HTTP page given in Request.
void	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HttpClosableFetcher

protected HttpClosableFetcher(HttpClosableFetcher.Builder builder)

Method Detail

buildDefault

public static HttpClosableFetcher buildDefault()

builder

public static HttpClosableFetcher.Builder builder()

fetch

public Future<Response> fetch(Request request)

Description copied from interface: Fetcher

Fetch the desired HTTP page given in Request.

Parameters:

request - information for the page to fetch.

Returns:

Response future

fetch

public Future<Response> fetch(Request request,
                              Crawler.CallbackProcessor callback)

Description copied from interface: Fetcher

Fetch the desired HTTP page given in Request. Executes callback upon completion.

Parameters:

request - information for the page to fetch.

callback - callback

Returns:

Response future

start

public void start()

Specified by:

start in class AbstractClosableFetcher

close

public void close()
           throws Exception

Throws:

Exception