Package org.lockss.laaws.crawler.wget

Class WgetCmdLineCrawler

java.lang.Object

org.lockss.laaws.crawler.impl.pluggable.CmdLineCrawler

org.lockss.laaws.crawler.wget.WgetCmdLineCrawler

All Implemented Interfaces:

PluggableCrawler

public class WgetCmdLineCrawler
extends CmdLineCrawler

The type Wget cmd line crawler.

Nested Class Summary

Nested classes/interfaces inherited from class org.lockss.laaws.crawler.impl.pluggable.CmdLineCrawler

CmdLineCrawler.CommandLineBuilder, CmdLineCrawler.RunnableCrawlJob

Field Summary

Fields

Modifier and Type	Field	Description
static final String	ATTR_OUTPUT_LEVEL	The constant ATTR_OUTPUT_LEVEL.
static final String	ATTR_SUCCESS_CODE	The constant ATTR_SUCCESS_CODE.

Fields inherited from class org.lockss.laaws.crawler.impl.pluggable.CmdLineCrawler

ATTR_COMPRESS_WARC, ATTR_COMPRESSED_WARC_FILE_EXTENSION, ATTR_CRAWL_EXECUTOR_SPEC, ATTR_ERROR_LOG_LEVEL, ATTR_EXCLUDE_STATUS_PATTERN, ATTR_JOIN_OUTPUT_STREAMS, ATTR_OUTPUT_LOG_LEVEL, ATTR_PROC_EXIT_WAIT, ATTR_UNCOMPRESSED_WARC_FILE_EXTENSION, ATTR_UNSUPPORTED_PARAMS, cmdLineBuilder, compressWarc, config, crawlMap, DEFAULT_CMDLINE_CRAWL_EXECUTOR_SPEC, DEFAULT_COMPRESS_WARC, DEFAULT_COMPRESSED_WARC_FILE_EXTENSION, DEFAULT_ERROR_LOG_LEVEL, DEFAULT_EXCLUDE_STATUS_PATTERN, DEFAULT_JOIN_OUTPUT_STREAMS, DEFAULT_OUTPUT_LOG_LEVEL, DEFAULT_PROC_EXIT_WAIT, DEFAULT_UNCOMPRESSED_WARC_FILE_EXTENSION, errorLogLevel, excludeStatusPattern, outputLogLevel, pcManager, PREFIX, procExitWait, START_URL_KEY, unsupportedParams, URL_STEMS_KEY, warcFileFilter

Constructor Summary

Constructors

Constructor	Description
WgetCmdLineCrawler()

Method Summary

Modifier and Type	Method	Description
protected boolean	didCrawlSucceed(int exitCode)
List<String>	getConfigOptions()
double	getConnectTimeout()
double	getFetchDelay()
long	getMaxRetries()
double	getReadTimeout()
double	getRetryDelay()
void	updateCrawlerConfig(CrawlerConfig crawlerConfig)	set the configuration parameters for this crawler

Methods inherited from class org.lockss.laaws.crawler.impl.pluggable.CmdLineCrawler

deleteAllCrawls, disable, getCmdLineBuilder, getCompressedWarcExtension, getConfig, getCrawl, getCrawlerConfig, getCrawlerId, getErrorLogLevel, getOutputLogLevel, getPluggableCrawlManager, getProcExitWait, getUncompressedWarcExtension, getUnsupportedParams, getWarcFileFilter, initCrawlScheduler, isCrawlerEnabled, isElgibleForCrawl, isJoinOutputStreams, requestCrawl, setCmdLineBuilder, setConfig, setCrawlManager, setNamespace, setPluggableCrawlManager, setV2Repo, shutdown, shutdownWithWait, stopCrawl, storeInRepository, updateAuConfig, useCompressWarc

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

ATTR_SUCCESS_CODE

public static final String ATTR_SUCCESS_CODE

The constant ATTR_SUCCESS_CODE.

See Also:

• Constant Field Values

ATTR_OUTPUT_LEVEL

public static final String ATTR_OUTPUT_LEVEL

The constant ATTR_OUTPUT_LEVEL.

See Also:

• Constant Field Values

Constructor Details

WgetCmdLineCrawler

public WgetCmdLineCrawler()

Method Details

updateCrawlerConfig

public void updateCrawlerConfig(CrawlerConfig crawlerConfig)

Description copied from interface: PluggableCrawler

set the configuration parameters for this crawler

Specified by:

updateCrawlerConfig in interface PluggableCrawler

Overrides:

updateCrawlerConfig in class CmdLineCrawler

Parameters:

crawlerConfig - the configuration parameters to use

getMaxRetries

public long getMaxRetries()

getRetryDelay

public double getRetryDelay()

getConnectTimeout

public double getConnectTimeout()

getReadTimeout

public double getReadTimeout()

getFetchDelay

public double getFetchDelay()

didCrawlSucceed

protected boolean didCrawlSucceed(int exitCode)

Overrides:

didCrawlSucceed in class CmdLineCrawler

getConfigOptions

public List<String> getConfigOptions()