Package org.lockss.laaws.crawler.impl

Class PluggableCrawlManager

java.lang.Object

org.lockss.app.BaseLockssManager

org.lockss.app.BaseLockssDaemonManager

org.lockss.laaws.crawler.impl.PluggableCrawlManager

All Implemented Interfaces:

org.lockss.app.ConfigurableManager, org.lockss.app.LockssManager

public class PluggableCrawlManager
extends org.lockss.app.BaseLockssDaemonManager
implements org.lockss.app.ConfigurableManager

The type Pluggable crawl manager.

Nested Class Summary

Nested classes/interfaces inherited from class org.lockss.app.BaseLockssManager

org.lockss.app.BaseLockssManager.MapMessageListener

Field Summary

Fields

Modifier and Type	Field	Description
static final String	ATTR_CRAWLER_ID	The constant CRAWLER_ID.
static final String	ATTR_CRAWLER_NAME	The constant CRAWLER_NAME.
static final String	ATTR_CRAWLING_ENABLED	The constant CRAWLING_ENABLED.
static final String	ATTR_STARTER_ENABLED	The constant CRAWL_STARTER_ENABLED.
static final String	CRAWLER_IDS	The constant CRAWLER_IDS.
static final String	CRAWLER_PREFIX
static final String	DB_FILENAME	The constant DB_FILENAME.
static final String	DEFAULT_CRAWL_DB_PATH	The constant DEFAULT_CRAWL_DB_PATH.
static boolean	DEFAULT_REQUEUE_ON_RESTART
static final List<String>	defaultCrawlerIds	The default list of known crawlers.
static final String	ENABLED	The constant ENABLED.
static final String	PARAM_CRAWL_DB_PATH	The constant PARAM_CRAWL_DB_PATH.
static final String	PARAM_REQUEUE_ON_RESTART
static final String	PREFIX	The constant PREFIX.

Fields inherited from class org.lockss.app.BaseLockssManager

isInited, isStarted, jmsConsumer, jmsProducer, shuttingDown, startedSem, theApp

Constructor Summary

Constructors

Constructor	Description
PluggableCrawlManager()

Method Summary

Modifier and Type	Method	Description
void	addCrawlJob(org.lockss.util.rest.crawler.CrawlJob crawlJob)	Insert crawl Job into persistent store.
void	deleteAllCrawls()	Delete all crawls.
long	getConnectTimeout()
PluggableCrawler	getCrawler(String crawlerId)	Gets crawler.
CrawlerConfig	getCrawlerConfig(String crawlerId)	Gets crawler config.
List<String>	getCrawlerIds()	Gets crawler ids.
org.lockss.util.rest.crawler.CrawlJob	getCrawlJob(String jobId)	Gets crawl status.
org.dizitart.no2.objects.Cursor<org.lockss.util.rest.crawler.CrawlJob>	getCrawlJobsWithAuId(String auId)	Gets crawl jobs with au id.
long	getFetchDelay()
int	getMaxRetries()
long	getReadTimeout()
long	getRetryDelay()
void	handleCrawlComplete(org.lockss.crawler.CrawlerStatus status)	handle the complete crawl.
void	handleCrawlComplete(org.lockss.crawler.CrawlEvent event)	handle the complete crawl.
boolean	isCrawlerEnabled()	is crawling enabled
boolean	isCrawlerEnabled(String crawlerId)	Is a specific crawler enabled
boolean	isCrawlStarterEnabled()	is the crawl starter enabled
boolean	isEligibleForCrawl(String auId)	Is eligible Au elgible for crawl boolean.
void	markInterruptedCrawls()
void	restartCrawls()	restart unfinished crawls.
void	setConfig(org.lockss.config.Configuration newConfig, org.lockss.config.Configuration prevConfig, org.lockss.config.Configuration.Differences changedKeys)
void	startService()
void	stopService()
void	updateCrawlJob(org.lockss.util.rest.crawler.CrawlJob crawlJob)	Update a crawl Jobs status.

Methods inherited from class org.lockss.app.BaseLockssDaemonManager

getDaemon, initService, initService, isDaemonInited

Methods inherited from class org.lockss.app.BaseLockssManager

getAlertManager, getApp, getClassName, getConfigManager, getDefaultTempRootDirectory, getJMSManager, getManagerByType, isAppInited, isInited, isRepoReady, isShuttingDown, isStarted, receiveMessage, resetConfig, serviceStarted, setUpJmsReceive, setUpJmsReceive, setUpJmsSend, stopJms, waitForRepo, waitForRepo, waitStarted, waitStarted

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.lockss.app.ConfigurableManager

auConfigChanged, auConfigRemoved

Field Details

PREFIX

public static final String PREFIX

The constant PREFIX.

See Also:

• Constant Field Values

CRAWLER_PREFIX

public static final String CRAWLER_PREFIX

See Also:

• Constant Field Values

PARAM_CRAWL_DB_PATH

public static final String PARAM_CRAWL_DB_PATH

The constant PARAM_CRAWL_DB_PATH.

See Also:

• Constant Field Values

DEFAULT_CRAWL_DB_PATH

public static final String DEFAULT_CRAWL_DB_PATH

The constant DEFAULT_CRAWL_DB_PATH.

See Also:

• Constant Field Values

PARAM_REQUEUE_ON_RESTART

public static final String PARAM_REQUEUE_ON_RESTART

See Also:

• Constant Field Values

DEFAULT_REQUEUE_ON_RESTART

public static boolean DEFAULT_REQUEUE_ON_RESTART

DB_FILENAME

public static final String DB_FILENAME

The constant DB_FILENAME.

See Also:

• Constant Field Values

CRAWLER_IDS

public static final String CRAWLER_IDS

The constant CRAWLER_IDS.

See Also:

• Constant Field Values

defaultCrawlerIds

public static final List<String> defaultCrawlerIds

The default list of known crawlers.

ATTR_CRAWLER_ID

public static final String ATTR_CRAWLER_ID

The constant CRAWLER_ID.

See Also:

• Constant Field Values

ATTR_CRAWLER_NAME

public static final String ATTR_CRAWLER_NAME

The constant CRAWLER_NAME.

See Also:

• Constant Field Values

ATTR_CRAWLING_ENABLED

public static final String ATTR_CRAWLING_ENABLED

The constant CRAWLING_ENABLED.

See Also:

• Constant Field Values

ATTR_STARTER_ENABLED

public static final String ATTR_STARTER_ENABLED

The constant CRAWL_STARTER_ENABLED.

See Also:

• Constant Field Values

ENABLED

public static final String ENABLED

The constant ENABLED.

See Also:

• Constant Field Values

Constructor Details

PluggableCrawlManager

public PluggableCrawlManager()

Method Details

startService

public void startService()

Specified by:

startService in interface org.lockss.app.LockssManager

Overrides:

startService in class org.lockss.app.BaseLockssManager

stopService

public void stopService()

Specified by:

stopService in interface org.lockss.app.LockssManager

Overrides:

stopService in class org.lockss.app.BaseLockssManager

setConfig

public void setConfig(org.lockss.config.Configuration newConfig,
 org.lockss.config.Configuration prevConfig,
 org.lockss.config.Configuration.Differences changedKeys)

Specified by:

setConfig in interface org.lockss.app.ConfigurableManager

getMaxRetries

public int getMaxRetries()

getRetryDelay

public long getRetryDelay()

getConnectTimeout

public long getConnectTimeout()

getReadTimeout

public long getReadTimeout()

getFetchDelay

public long getFetchDelay()

getCrawlerIds

public List<String> getCrawlerIds()

Gets crawler ids.

Returns:

the crawler ids

isEligibleForCrawl

public boolean isEligibleForCrawl(String auId)

Is eligible Au elgible for crawl boolean.

Parameters:

auId - the au id

Returns:

the boolean

isCrawlerEnabled

public boolean isCrawlerEnabled()

is crawling enabled

Returns:

true iff crawler (global is enabled)

isCrawlStarterEnabled

public boolean isCrawlStarterEnabled()

is the crawl starter enabled

Returns:

boolean are crawls allowed to start

getCrawlJob

public org.lockss.util.rest.crawler.CrawlJob getCrawlJob(String jobId)

Gets crawl status.

Parameters:

jobId - the id

Returns:

the crawl status

getCrawlJobsWithAuId

public org.dizitart.no2.objects.Cursor<org.lockss.util.rest.crawler.CrawlJob> getCrawlJobsWithAuId(String auId)

Gets crawl jobs with au id.

Parameters:

auId - the au id

Returns:

the crawl jobs with au id

addCrawlJob

public void addCrawlJob(org.lockss.util.rest.crawler.CrawlJob crawlJob)

Insert crawl Job into persistent store. throws if crawl job already exists.

Parameters:

crawlJob - the crawl job.

updateCrawlJob

public void updateCrawlJob(org.lockss.util.rest.crawler.CrawlJob crawlJob)

Update a crawl Jobs status. throws if crawl job does not exist.

Parameters:

crawlJob - the crawl job.

restartCrawls

public void restartCrawls()

restart unfinished crawls.

markInterruptedCrawls

public void markInterruptedCrawls()

deleteAllCrawls

public void deleteAllCrawls()

Delete all crawls.

getCrawlerConfig

public CrawlerConfig getCrawlerConfig(String crawlerId)

Gets crawler config.

Parameters:

crawlerId - the crawler id

Returns:

the crawler config

getCrawler

public PluggableCrawler getCrawler(String crawlerId)

Gets crawler.

Parameters:

crawlerId - the crawler id

Returns:

the crawler

isCrawlerEnabled

public boolean isCrawlerEnabled(String crawlerId)

Is a specific crawler enabled

Parameters:

crawlerId - the id of the crawler

Returns:

true if the crawler is found in the configuration map and is marked enabled.

handleCrawlComplete

public void handleCrawlComplete(org.lockss.crawler.CrawlEvent event)

handle the complete crawl.

Parameters:

event - the crawl complete event.

handleCrawlComplete

public void handleCrawlComplete(org.lockss.crawler.CrawlerStatus status)

handle the complete crawl.

Parameters:

status - the status of the completed crawl.