CmdLineCrawler (laaws-crawler-service 1.0.0 API)

java.lang.Object
- org.lockss.laaws.crawler.impl.pluggable.CmdLineCrawler

All Implemented Interfaces:

PluggableCrawler

Direct Known Subclasses:

WgetCmdLineCrawler
```
public class CmdLineCrawler
extends Object
implements PluggableCrawler
```
A Base implementation of a CmdLineCrawler.

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static interface CmdLineCrawler.CommandLineBuilder

static class CmdLineCrawler.RunnableCrawlJob

Nested Classes
Modifier and Type	Class and Description
`static interface`	`CmdLineCrawler.CommandLineBuilder`
`static class`	`CmdLineCrawler.RunnableCrawlJob`

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`ATTR_COMPRESS_WARC`
`static String`	`ATTR_COMPRESSED_WARC_FILE_EXTENSION`
`static String`	`ATTR_CRAWL_EXECUTOR_SPEC` Controls the number of AUs running cmd line crawls
`static String`	`ATTR_ERROR_LOG_LEVEL`
`static String`	`ATTR_EXCLUDE_STATUS_PATTERN`
`static String`	`ATTR_JOIN_OUTPUT_STREAMS`
`static String`	`ATTR_OUTPUT_LOG_LEVEL`
`static String`	`ATTR_PROC_EXIT_WAIT`
`static String`	`ATTR_UNCOMPRESSED_WARC_FILE_EXTENSION`
`static String`	`ATTR_UNSUPPORTED_PARAMS`
`protected CmdLineCrawler.CommandLineBuilder`	`cmdLineBuilder`
`protected boolean`	`compressWarc`
`protected CrawlerConfig`	`config` The Configuration for this crawler.
`protected HashMap<String,CmdLineCrawl>`	`crawlMap` The map of crawls for this crawler.
`static String`	`DEFAULT_CMDLINE_CRAWL_EXECUTOR_SPEC`
`static String`	`DEFAULT_COMPRESS_WARC`
`static String`	`DEFAULT_COMPRESSED_WARC_FILE_EXTENSION`
`static String`	`DEFAULT_ERROR_LOG_LEVEL`
`static String`	`DEFAULT_EXCLUDE_STATUS_PATTERN`
`static String`	`DEFAULT_JOIN_OUTPUT_STREAMS`
`static String`	`DEFAULT_OUTPUT_LOG_LEVEL`
`static long`	`DEFAULT_PROC_EXIT_WAIT`
`static String`	`DEFAULT_UNCOMPRESSED_WARC_FILE_EXTENSION`
`protected String`	`errorLogLevel` The level to use when logging error from a process
`protected String`	`excludeStatusPattern` The http response codes to exclude from warc import.
`protected String`	`outputLogLevel` The level to use when logging output from a process
`protected PluggableCrawlManager`	`pcManager`
`static String`	`PREFIX`
`protected long`	`procExitWait`
`static String`	`START_URL_KEY`
`protected List<String>`	`unsupportedParams`
`static String`	`URL_STEMS_KEY`
`protected List<String>`	`warcFileFilter`

Constructor Summary

Constructors
Constructor and Description

CmdLineCrawler()
Instantiates a new Cmd line crawler.

Constructors
Constructor and Description
`CmdLineCrawler()` Instantiates a new Cmd line crawler.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`deleteAllCrawls()` Stop all crawls and clear the crawl queue managed by this crawler
`protected boolean`	`didCrawlSucceed(int exitCode)`
`void`	`disable(boolean abortCrawling)` disable this crawler clearing any queued crawls.
`protected CmdLineCrawler.CommandLineBuilder`	`getCmdLineBuilder()`
`String`	`getCompressedWarcExtension()`
`CrawlerConfig`	`getConfig()`
`PluggableCrawl`	`getCrawl(String crawlId)` Get a Crawl for a given crawl id.
`CrawlerConfig`	`getCrawlerConfig()` Return the configuration for this crawler
`String`	`getCrawlerId()` Return the unique Id for this crawler.
`String`	`getErrorLogLevel()`
`String`	`getOutputLogLevel()`
`PluggableCrawlManager`	`getPluggableCrawlManager()`
`long`	`getProcExitWait()`
`String`	`getUncompressedWarcExtension()`
`List<String>`	`getUnsupportedParams()`
`List<String>`	`getWarcFileFilter()`
`protected void`	`initCrawlScheduler(String reqSpec)`
`boolean`	`isCrawlerEnabled()` is this crawler enabled
`boolean`	`isElgibleForCrawl(String auId)`
`boolean`	`isJoinOutputStreams()`
`PluggableCrawl`	`requestCrawl(org.lockss.plugin.ArchivalUnit au, org.lockss.util.rest.crawler.CrawlJob crawlJob)`
`CmdLineCrawler`	`setCmdLineBuilder(CmdLineCrawler.CommandLineBuilder cmdLineBuilder)`
`CmdLineCrawler`	`setConfig(CrawlerConfig config)`
`CmdLineCrawler`	`setCrawlManager(PluggableCrawlManager pcManager)`
`CmdLineCrawler`	`setNamespace(String namespace)`
`void`	`setPluggableCrawlManager(PluggableCrawlManager pluggableCrawlManager)` Set the Crawl Manager which created and maintains this crawler.
`CmdLineCrawler`	`setV2Repo(org.lockss.util.rest.repo.LockssRepository v2Repo)`
`void`	`shutdown()` Shutdown the crawler.
`protected void`	`shutdownWithWait(ExecutorService scheduler)`
`PluggableCrawl`	`stopCrawl(String crawlId)` Stop a crawl a specific crawl
`void`	`storeInRepository(String auId, File warcFile, boolean isCompressed)`
`void`	`updateAuConfig(org.lockss.plugin.ArchivalUnit au, boolean isRepairCrawl, List<String> reqUrls, List<String> crawlStems)`
`void`	`updateCrawlerConfig(CrawlerConfig crawlerConfig)` set the configuration parameters for this crawler
`boolean`	`useCompressWarc()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - PREFIX
```
public static final String PREFIX
```
    See Also:
    
    Constant Field Values
  - ATTR_CRAWL_EXECUTOR_SPEC
```
public static final String ATTR_CRAWL_EXECUTOR_SPEC
```
    Controls the number of AUs running cmd line crawls
    
    See Also:
    
    Constant Field Values
  - DEFAULT_CMDLINE_CRAWL_EXECUTOR_SPEC
```
public static final String DEFAULT_CMDLINE_CRAWL_EXECUTOR_SPEC
```
    See Also:
    
    Constant Field Values
  - ATTR_EXCLUDE_STATUS_PATTERN
```
public static final String ATTR_EXCLUDE_STATUS_PATTERN
```
    See Also:
    
    Constant Field Values
  - DEFAULT_EXCLUDE_STATUS_PATTERN
```
public static final String DEFAULT_EXCLUDE_STATUS_PATTERN
```
    See Also:
    
    Constant Field Values
  - ATTR_OUTPUT_LOG_LEVEL
```
public static final String ATTR_OUTPUT_LOG_LEVEL
```
    See Also:
    
    Constant Field Values
  - DEFAULT_OUTPUT_LOG_LEVEL
```
public static final String DEFAULT_OUTPUT_LOG_LEVEL
```
    See Also:
    
    Constant Field Values
  - ATTR_ERROR_LOG_LEVEL
```
public static final String ATTR_ERROR_LOG_LEVEL
```
    See Also:
    
    Constant Field Values
  - DEFAULT_ERROR_LOG_LEVEL
```
public static final String DEFAULT_ERROR_LOG_LEVEL
```
    See Also:
    
    Constant Field Values
  - ATTR_JOIN_OUTPUT_STREAMS
```
public static final String ATTR_JOIN_OUTPUT_STREAMS
```
    See Also:
    
    Constant Field Values
  - DEFAULT_JOIN_OUTPUT_STREAMS
```
public static final String DEFAULT_JOIN_OUTPUT_STREAMS
```
    See Also:
    
    Constant Field Values
  - ATTR_PROC_EXIT_WAIT
```
public static final String ATTR_PROC_EXIT_WAIT
```
    See Also:
    
    Constant Field Values
  - DEFAULT_PROC_EXIT_WAIT
```
public static final long DEFAULT_PROC_EXIT_WAIT
```
    See Also:
    
    Constant Field Values
  - ATTR_COMPRESS_WARC
```
public static final String ATTR_COMPRESS_WARC
```
    See Also:
    
    Constant Field Values
  - DEFAULT_COMPRESS_WARC
```
public static final String DEFAULT_COMPRESS_WARC
```
    See Also:
    
    Constant Field Values
  - ATTR_COMPRESSED_WARC_FILE_EXTENSION
```
public static final String ATTR_COMPRESSED_WARC_FILE_EXTENSION
```
    See Also:
    
    Constant Field Values
  - DEFAULT_COMPRESSED_WARC_FILE_EXTENSION
```
public static final String DEFAULT_COMPRESSED_WARC_FILE_EXTENSION
```
    See Also:
    
    Constant Field Values
  - ATTR_UNCOMPRESSED_WARC_FILE_EXTENSION
```
public static final String ATTR_UNCOMPRESSED_WARC_FILE_EXTENSION
```
    See Also:
    
    Constant Field Values
  - DEFAULT_UNCOMPRESSED_WARC_FILE_EXTENSION
```
public static final String DEFAULT_UNCOMPRESSED_WARC_FILE_EXTENSION
```
    See Also:
    
    Constant Field Values
  - ATTR_UNSUPPORTED_PARAMS
```
public static final String ATTR_UNSUPPORTED_PARAMS
```
    See Also:
    
    Constant Field Values
  - START_URL_KEY
```
public static final String START_URL_KEY
```
    See Also:
    
    Constant Field Values
  - URL_STEMS_KEY
```
public static final String URL_STEMS_KEY
```
    See Also:
    
    Constant Field Values
  - config
```
protected CrawlerConfig config
```
    The Configuration for this crawler.
  - outputLogLevel
```
protected String outputLogLevel
```
    The level to use when logging output from a process
  - errorLogLevel
```
protected String errorLogLevel
```
    The level to use when logging error from a process
  - excludeStatusPattern
```
protected String excludeStatusPattern
```
    The http response codes to exclude from warc import.
  - compressWarc
```
protected boolean compressWarc
```
  - warcFileFilter
```
protected List<String> warcFileFilter
```
  - procExitWait
```
protected long procExitWait
```
  - unsupportedParams
```
protected List<String> unsupportedParams
```
  - crawlMap
```
protected HashMap<String,CmdLineCrawl> crawlMap
```
    The map of crawls for this crawler.
  - cmdLineBuilder
```
protected CmdLineCrawler.CommandLineBuilder cmdLineBuilder
```
  - pcManager
```
protected PluggableCrawlManager pcManager
```
- Constructor Detail
  - CmdLineCrawler
```
public CmdLineCrawler()
```
    Instantiates a new Cmd line crawler.
- Method Detail
  - setCrawlManager
```
public CmdLineCrawler setCrawlManager(PluggableCrawlManager pcManager)
```
  - setV2Repo
```
public CmdLineCrawler setV2Repo(org.lockss.util.rest.repo.LockssRepository v2Repo)
```
  - setNamespace
```
public CmdLineCrawler setNamespace(String namespace)
```
  - setConfig
```
public CmdLineCrawler setConfig(CrawlerConfig config)
```
  - setCmdLineBuilder
```
public CmdLineCrawler setCmdLineBuilder(CmdLineCrawler.CommandLineBuilder cmdLineBuilder)
```
  - getConfig
```
public CrawlerConfig getConfig()
```
  - getCmdLineBuilder
```
protected CmdLineCrawler.CommandLineBuilder getCmdLineBuilder()
```
  - getCrawlerId
```
public String getCrawlerId()
```
    Description copied from interface: PluggableCrawler
    
    Return the unique Id for this crawler.
    
    Specified by:
    
    getCrawlerId in interface PluggableCrawler
    
    Returns:
    
    the id of the crawler
  - updateCrawlerConfig
```
public void updateCrawlerConfig(CrawlerConfig crawlerConfig)
```
    Description copied from interface: PluggableCrawler
    
    set the configuration parameters for this crawler
    
    Specified by:
    
    updateCrawlerConfig in interface PluggableCrawler
    
    Parameters:
    
    crawlerConfig - the configuration parameters to use
  - getCrawlerConfig
```
public CrawlerConfig getCrawlerConfig()
```
    Description copied from interface: PluggableCrawler
    
    Return the configuration for this crawler
    
    Specified by:
    
    getCrawlerConfig in interface PluggableCrawler
    
    Returns:
    
    the configuration parameters in use by this crawler.
  - getProcExitWait
```
public long getProcExitWait()
```
  - getWarcFileFilter
```
public List<String> getWarcFileFilter()
```
  - getCompressedWarcExtension
```
public String getCompressedWarcExtension()
```
  - getUncompressedWarcExtension
```
public String getUncompressedWarcExtension()
```
  - getUnsupportedParams
```
public List<String> getUnsupportedParams()
```
  - useCompressWarc
```
public boolean useCompressWarc()
```
  - requestCrawl
```
public PluggableCrawl requestCrawl(org.lockss.plugin.ArchivalUnit au,
                                   org.lockss.util.rest.crawler.CrawlJob crawlJob)
```
    Specified by:
    
    requestCrawl in interface PluggableCrawler
  - isElgibleForCrawl
```
public boolean isElgibleForCrawl(String auId)
```
  - stopCrawl
```
public PluggableCrawl stopCrawl(String crawlId)
```
    Description copied from interface: PluggableCrawler
    
    Stop a crawl a specific crawl
    
    Specified by:
    
    stopCrawl in interface PluggableCrawler
    
    Parameters:
    
    crawlId - The crawl id of the crawl to stop
    
    Returns:
    
    The PluggableCrawl containing the results of this crawl attempt.
  - getCrawl
```
public PluggableCrawl getCrawl(String crawlId)
```
    Description copied from interface: PluggableCrawler
    
    Get a Crawl for a given crawl id.
    
    Specified by:
    
    getCrawl in interface PluggableCrawler
    
    Parameters:
    
    crawlId - The crawl id of the crawl to stop
    
    Returns:
    
    The PluggableCrawl that matches a crawl id.
  - deleteAllCrawls
```
public void deleteAllCrawls()
```
    Description copied from interface: PluggableCrawler
    
    Stop all crawls and clear the crawl queue managed by this crawler
    
    Specified by:
    
    deleteAllCrawls in interface PluggableCrawler
  - isCrawlerEnabled
```
public boolean isCrawlerEnabled()
```
    Description copied from interface: PluggableCrawler
    
    is this crawler enabled
    
    Specified by:
    
    isCrawlerEnabled in interface PluggableCrawler
    
    Returns:
    
    true if this crawler is set to enabled.
  - shutdown
```
public void shutdown()
```
    Description copied from interface: PluggableCrawler
    
    Shutdown the crawler.
    
    Specified by:
    
    shutdown in interface PluggableCrawler
  - shutdownWithWait
```
protected void shutdownWithWait(ExecutorService scheduler)
```
  - disable
```
public void disable(boolean abortCrawling)
```
    Description copied from interface: PluggableCrawler
    
    disable this crawler clearing any queued crawls. if the crawler was running is now marked as disabled or is missing from the supported crawler ids in the configuration
    
    Specified by:
    
    disable in interface PluggableCrawler
    
    Parameters:
    
    abortCrawling - abort the currently running crawls.
  - setPluggableCrawlManager
```
public void setPluggableCrawlManager(PluggableCrawlManager pluggableCrawlManager)
```
    Description copied from interface: PluggableCrawler
    
    Set the Crawl Manager which created and maintains this crawler.
    
    Specified by:
    
    setPluggableCrawlManager in interface PluggableCrawler
  - getPluggableCrawlManager
```
public PluggableCrawlManager getPluggableCrawlManager()
```
  - storeInRepository
```
public void storeInRepository(String auId,
                              File warcFile,
                              boolean isCompressed)
                       throws IOException
```
    Throws:
    
    IOException
  - updateAuConfig
```
public void updateAuConfig(org.lockss.plugin.ArchivalUnit au,
                           boolean isRepairCrawl,
                           List<String> reqUrls,
                           List<String> crawlStems)
                    throws IOException
```
    Throws:
    
    IOException
  - initCrawlScheduler
```
protected void initCrawlScheduler(String reqSpec)
```
  - didCrawlSucceed
```
protected boolean didCrawlSucceed(int exitCode)
```
  - getOutputLogLevel
```
public String getOutputLogLevel()
```
  - getErrorLogLevel
```
public String getErrorLogLevel()
```
  - isJoinOutputStreams
```
public boolean isJoinOutputStreams()
```

Class CmdLineCrawler

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

PREFIX

ATTR_CRAWL_EXECUTOR_SPEC

DEFAULT_CMDLINE_CRAWL_EXECUTOR_SPEC

ATTR_EXCLUDE_STATUS_PATTERN

DEFAULT_EXCLUDE_STATUS_PATTERN

ATTR_OUTPUT_LOG_LEVEL

DEFAULT_OUTPUT_LOG_LEVEL

ATTR_ERROR_LOG_LEVEL

DEFAULT_ERROR_LOG_LEVEL

ATTR_JOIN_OUTPUT_STREAMS

DEFAULT_JOIN_OUTPUT_STREAMS

ATTR_PROC_EXIT_WAIT

DEFAULT_PROC_EXIT_WAIT

ATTR_COMPRESS_WARC

DEFAULT_COMPRESS_WARC

ATTR_COMPRESSED_WARC_FILE_EXTENSION

DEFAULT_COMPRESSED_WARC_FILE_EXTENSION

ATTR_UNCOMPRESSED_WARC_FILE_EXTENSION

DEFAULT_UNCOMPRESSED_WARC_FILE_EXTENSION

ATTR_UNSUPPORTED_PARAMS

START_URL_KEY

URL_STEMS_KEY

config

outputLogLevel

errorLogLevel

excludeStatusPattern

compressWarc

warcFileFilter

procExitWait

unsupportedParams

crawlMap

cmdLineBuilder

pcManager

Constructor Detail

CmdLineCrawler

Method Detail

setCrawlManager

setV2Repo

setNamespace

setConfig

setCmdLineBuilder

getConfig

getCmdLineBuilder

getCrawlerId

updateCrawlerConfig

getCrawlerConfig

getProcExitWait

getWarcFileFilter

getCompressedWarcExtension

getUncompressedWarcExtension

getUnsupportedParams

useCompressWarc

requestCrawl

isElgibleForCrawl

stopCrawl

getCrawl

deleteAllCrawls

isCrawlerEnabled

shutdown

shutdownWithWait

disable

setPluggableCrawlManager

getPluggableCrawlManager

storeInRepository

updateAuConfig

initCrawlScheduler

didCrawlSucceed

getOutputLogLevel

getErrorLogLevel

isJoinOutputStreams