Package org.lockss.laaws.crawler.impl.pluggable

Class CmdLineCrawler

java.lang.Object

org.lockss.laaws.crawler.impl.pluggable.CmdLineCrawler

All Implemented Interfaces:

PluggableCrawler

Direct Known Subclasses:

WgetCmdLineCrawler

public class CmdLineCrawler
extends Object
implements PluggableCrawler

A Base implementation of a CmdLineCrawler.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	CmdLineCrawler.CommandLineBuilder
static class	CmdLineCrawler.RunnableCrawlJob

Field Summary

Fields

Modifier and Type	Field	Description
static final String	ATTR_COMPRESS_WARC
static final String	ATTR_COMPRESSED_WARC_FILE_EXTENSION
static final String	ATTR_CRAWL_EXECUTOR_SPEC	Controls the number of AUs running cmd line crawls
static final String	ATTR_ERROR_LOG_LEVEL
static final String	ATTR_EXCLUDE_STATUS_PATTERN
static final String	ATTR_JOIN_OUTPUT_STREAMS
static final String	ATTR_OUTPUT_LOG_LEVEL
static final String	ATTR_PROC_EXIT_WAIT
static final String	ATTR_UNCOMPRESSED_WARC_FILE_EXTENSION
static final String	ATTR_UNSUPPORTED_PARAMS
protected CmdLineCrawler.CommandLineBuilder	cmdLineBuilder
protected boolean	compressWarc
protected CrawlerConfig	config	The Configuration for this crawler.
protected HashMap<String,CmdLineCrawl>	crawlMap	The map of crawls for this crawler.
static final String	DEFAULT_CMDLINE_CRAWL_EXECUTOR_SPEC
static final String	DEFAULT_COMPRESS_WARC
static final String	DEFAULT_COMPRESSED_WARC_FILE_EXTENSION
static final String	DEFAULT_ERROR_LOG_LEVEL
static final String	DEFAULT_EXCLUDE_STATUS_PATTERN
static final String	DEFAULT_JOIN_OUTPUT_STREAMS
static final String	DEFAULT_OUTPUT_LOG_LEVEL
static final long	DEFAULT_PROC_EXIT_WAIT
static final String	DEFAULT_UNCOMPRESSED_WARC_FILE_EXTENSION
protected String	errorLogLevel	The level to use when logging error from a process
protected String	excludeStatusPattern	The http response codes to exclude from warc import.
protected String	outputLogLevel	The level to use when logging output from a process
protected PluggableCrawlManager	pcManager
static final String	PREFIX
protected long	procExitWait
static final String	START_URL_KEY
protected List<String>	unsupportedParams
static final String	URL_STEMS_KEY
protected List<String>	warcFileFilter

Constructor Summary

Constructors

Constructor	Description
CmdLineCrawler()	Instantiates a new Cmd line crawler.

Method Summary

Modifier and Type	Method	Description
void	deleteAllCrawls()	Stop all crawls and clear the crawl queue managed by this crawler
protected boolean	didCrawlSucceed(int exitCode)
void	disable(boolean abortCrawling)	disable this crawler clearing any queued crawls.
protected CmdLineCrawler.CommandLineBuilder	getCmdLineBuilder()
String	getCompressedWarcExtension()
CrawlerConfig	getConfig()
PluggableCrawl	getCrawl(String crawlId)	Get a Crawl for a given crawl id.
CrawlerConfig	getCrawlerConfig()	Return the configuration for this crawler
String	getCrawlerId()	Return the unique Id for this crawler.
String	getErrorLogLevel()
String	getOutputLogLevel()
PluggableCrawlManager	getPluggableCrawlManager()
long	getProcExitWait()
String	getUncompressedWarcExtension()
List<String>	getUnsupportedParams()
List<String>	getWarcFileFilter()
protected void	initCrawlScheduler(String reqSpec)
boolean	isCrawlerEnabled()	is this crawler enabled
boolean	isElgibleForCrawl(String auId)
boolean	isJoinOutputStreams()
PluggableCrawl	requestCrawl(org.lockss.plugin.ArchivalUnit au, org.lockss.util.rest.crawler.CrawlJob crawlJob)
CmdLineCrawler	setCmdLineBuilder(CmdLineCrawler.CommandLineBuilder cmdLineBuilder)
CmdLineCrawler	setConfig(CrawlerConfig config)
CmdLineCrawler	setCrawlManager(PluggableCrawlManager pcManager)
CmdLineCrawler	setNamespace(String namespace)
void	setPluggableCrawlManager(PluggableCrawlManager pluggableCrawlManager)	Set the Crawl Manager which created and maintains this crawler.
CmdLineCrawler	setV2Repo(org.lockss.util.rest.repo.LockssRepository v2Repo)
void	shutdown()	Shutdown the crawler.
protected void	shutdownWithWait(ExecutorService scheduler)
PluggableCrawl	stopCrawl(String crawlId)	Stop a crawl a specific crawl
void	storeInRepository(String auId, File warcFile, boolean isCompressed)
void	updateAuConfig(org.lockss.plugin.ArchivalUnit au, boolean isRepairCrawl, List<String> reqUrls, List<String> crawlStems)
void	updateCrawlerConfig(CrawlerConfig crawlerConfig)	set the configuration parameters for this crawler
boolean	useCompressWarc()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

PREFIX

public static final String PREFIX

See Also:

• Constant Field Values

ATTR_CRAWL_EXECUTOR_SPEC

public static final String ATTR_CRAWL_EXECUTOR_SPEC

Controls the number of AUs running cmd line crawls

See Also:

• Constant Field Values

DEFAULT_CMDLINE_CRAWL_EXECUTOR_SPEC

public static final String DEFAULT_CMDLINE_CRAWL_EXECUTOR_SPEC

See Also:

• Constant Field Values

ATTR_EXCLUDE_STATUS_PATTERN

public static final String ATTR_EXCLUDE_STATUS_PATTERN

See Also:

• Constant Field Values

DEFAULT_EXCLUDE_STATUS_PATTERN

public static final String DEFAULT_EXCLUDE_STATUS_PATTERN

See Also:

• Constant Field Values

ATTR_OUTPUT_LOG_LEVEL

public static final String ATTR_OUTPUT_LOG_LEVEL

See Also:

• Constant Field Values

DEFAULT_OUTPUT_LOG_LEVEL

public static final String DEFAULT_OUTPUT_LOG_LEVEL

See Also:

• Constant Field Values

ATTR_ERROR_LOG_LEVEL

public static final String ATTR_ERROR_LOG_LEVEL

See Also:

• Constant Field Values

DEFAULT_ERROR_LOG_LEVEL

public static final String DEFAULT_ERROR_LOG_LEVEL

See Also:

• Constant Field Values

ATTR_JOIN_OUTPUT_STREAMS

public static final String ATTR_JOIN_OUTPUT_STREAMS

See Also:

• Constant Field Values

DEFAULT_JOIN_OUTPUT_STREAMS

public static final String DEFAULT_JOIN_OUTPUT_STREAMS

See Also:

• Constant Field Values

ATTR_PROC_EXIT_WAIT

public static final String ATTR_PROC_EXIT_WAIT

See Also:

• Constant Field Values

DEFAULT_PROC_EXIT_WAIT

public static final long DEFAULT_PROC_EXIT_WAIT

See Also:

• Constant Field Values

ATTR_COMPRESS_WARC

public static final String ATTR_COMPRESS_WARC

See Also:

• Constant Field Values

DEFAULT_COMPRESS_WARC

public static final String DEFAULT_COMPRESS_WARC

See Also:

• Constant Field Values

ATTR_COMPRESSED_WARC_FILE_EXTENSION

public static final String ATTR_COMPRESSED_WARC_FILE_EXTENSION

See Also:

• Constant Field Values

DEFAULT_COMPRESSED_WARC_FILE_EXTENSION

public static final String DEFAULT_COMPRESSED_WARC_FILE_EXTENSION

See Also:

• Constant Field Values

ATTR_UNCOMPRESSED_WARC_FILE_EXTENSION

public static final String ATTR_UNCOMPRESSED_WARC_FILE_EXTENSION

See Also:

• Constant Field Values

DEFAULT_UNCOMPRESSED_WARC_FILE_EXTENSION

public static final String DEFAULT_UNCOMPRESSED_WARC_FILE_EXTENSION

See Also:

• Constant Field Values

ATTR_UNSUPPORTED_PARAMS

public static final String ATTR_UNSUPPORTED_PARAMS

See Also:

• Constant Field Values

START_URL_KEY

public static final String START_URL_KEY

See Also:

• Constant Field Values

URL_STEMS_KEY

public static final String URL_STEMS_KEY

See Also:

• Constant Field Values

config

protected CrawlerConfig config

The Configuration for this crawler.

outputLogLevel

protected String outputLogLevel

The level to use when logging output from a process

errorLogLevel

protected String errorLogLevel

The level to use when logging error from a process

excludeStatusPattern

protected String excludeStatusPattern

The http response codes to exclude from warc import.

compressWarc

protected boolean compressWarc

warcFileFilter

protected List<String> warcFileFilter

procExitWait

protected long procExitWait

unsupportedParams

protected List<String> unsupportedParams

crawlMap

protected HashMap<String,CmdLineCrawl> crawlMap

The map of crawls for this crawler.

cmdLineBuilder

protected CmdLineCrawler.CommandLineBuilder cmdLineBuilder

pcManager

protected PluggableCrawlManager pcManager

Constructor Details

CmdLineCrawler

public CmdLineCrawler()

Instantiates a new Cmd line crawler.

Method Details

setCrawlManager

public CmdLineCrawler setCrawlManager(PluggableCrawlManager pcManager)

setV2Repo

public CmdLineCrawler setV2Repo(org.lockss.util.rest.repo.LockssRepository v2Repo)

setNamespace

public CmdLineCrawler setNamespace(String namespace)

setConfig

public CmdLineCrawler setConfig(CrawlerConfig config)

setCmdLineBuilder

public CmdLineCrawler setCmdLineBuilder(CmdLineCrawler.CommandLineBuilder cmdLineBuilder)

getConfig

public CrawlerConfig getConfig()

getCmdLineBuilder

protected CmdLineCrawler.CommandLineBuilder getCmdLineBuilder()

getCrawlerId

public String getCrawlerId()

Description copied from interface: PluggableCrawler

Return the unique Id for this crawler.

Specified by:

getCrawlerId in interface PluggableCrawler

Returns:

the id of the crawler

updateCrawlerConfig

public void updateCrawlerConfig(CrawlerConfig crawlerConfig)

Description copied from interface: PluggableCrawler

set the configuration parameters for this crawler

Specified by:

updateCrawlerConfig in interface PluggableCrawler

Parameters:

crawlerConfig - the configuration parameters to use

getCrawlerConfig

public CrawlerConfig getCrawlerConfig()

Description copied from interface: PluggableCrawler

Return the configuration for this crawler

Specified by:

getCrawlerConfig in interface PluggableCrawler

Returns:

the configuration parameters in use by this crawler.

getProcExitWait

public long getProcExitWait()

getWarcFileFilter

public List<String> getWarcFileFilter()

getCompressedWarcExtension

public String getCompressedWarcExtension()

getUncompressedWarcExtension

public String getUncompressedWarcExtension()

getUnsupportedParams

public List<String> getUnsupportedParams()

useCompressWarc

public boolean useCompressWarc()

requestCrawl

public PluggableCrawl requestCrawl(org.lockss.plugin.ArchivalUnit au,
 org.lockss.util.rest.crawler.CrawlJob crawlJob)

Specified by:

requestCrawl in interface PluggableCrawler

isElgibleForCrawl

public boolean isElgibleForCrawl(String auId)

stopCrawl

public PluggableCrawl stopCrawl(String crawlId)

Description copied from interface: PluggableCrawler

Stop a crawl a specific crawl

Specified by:

stopCrawl in interface PluggableCrawler

Parameters:

crawlId - The crawl id of the crawl to stop

Returns:

The PluggableCrawl containing the results of this crawl attempt.

getCrawl

public PluggableCrawl getCrawl(String crawlId)

Description copied from interface: PluggableCrawler

Get a Crawl for a given crawl id.

Specified by:

getCrawl in interface PluggableCrawler

Parameters:

crawlId - The crawl id of the crawl to stop

Returns:

The PluggableCrawl that matches a crawl id.

deleteAllCrawls

public void deleteAllCrawls()

Description copied from interface: PluggableCrawler

Stop all crawls and clear the crawl queue managed by this crawler

Specified by:

deleteAllCrawls in interface PluggableCrawler

isCrawlerEnabled

public boolean isCrawlerEnabled()

Description copied from interface: PluggableCrawler

is this crawler enabled

Specified by:

isCrawlerEnabled in interface PluggableCrawler

Returns:

true if this crawler is set to enabled.

shutdown

public void shutdown()

Description copied from interface: PluggableCrawler

Shutdown the crawler.

Specified by:

shutdown in interface PluggableCrawler

shutdownWithWait

protected void shutdownWithWait(ExecutorService scheduler)

disable

public void disable(boolean abortCrawling)

Description copied from interface: PluggableCrawler

disable this crawler clearing any queued crawls. if the crawler was running is now marked as disabled or is missing from the supported crawler ids in the configuration

Specified by:

disable in interface PluggableCrawler

Parameters:

abortCrawling - abort the currently running crawls.

setPluggableCrawlManager

public void setPluggableCrawlManager(PluggableCrawlManager pluggableCrawlManager)

Description copied from interface: PluggableCrawler

Set the Crawl Manager which created and maintains this crawler.

Specified by:

setPluggableCrawlManager in interface PluggableCrawler

getPluggableCrawlManager

public PluggableCrawlManager getPluggableCrawlManager()

storeInRepository

public void storeInRepository(String auId,
 File warcFile,
 boolean isCompressed)
                       throws IOException

Throws:

IOException

updateAuConfig

public void updateAuConfig(org.lockss.plugin.ArchivalUnit au,
 boolean isRepairCrawl,
 List<String> reqUrls,
 List<String> crawlStems)
                    throws IOException

Throws:

IOException

initCrawlScheduler

protected void initCrawlScheduler(String reqSpec)

didCrawlSucceed

protected boolean didCrawlSucceed(int exitCode)

getOutputLogLevel

public String getOutputLogLevel()

getErrorLogLevel

public String getErrorLogLevel()

isJoinOutputStreams

public boolean isJoinOutputStreams()