cn.wanghaomiao.seimi.def

Class BaseSeimiCrawler

java.lang.Object

cn.wanghaomiao.seimi.def.BaseSeimiCrawler

All Implemented Interfaces:

SeimiCrawler

public abstract class BaseSeimiCrawler
extends Object
implements SeimiCrawler

Since:

2015/8/17.

Author:

汪浩淼 et.tw@163.com

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.http.client.CookieStore	cookieStore
protected String	crawlerName
protected org.slf4j.Logger	logger
protected SeimiQueue	queue

Constructor Summary

Constructors

Constructor and Description
BaseSeimiCrawler()

Method Summary

Methods

Modifier and Type	Method and Description
String[]	allowRules() 用于设置允许的请求URL匹配规则
String[]	denyRules() 用于设置要放弃访问的请求URL匹配规则
org.apache.http.client.CookieStore	getCookieStore() 如果开启cookies通过此方法获取cookiesStore
String	getCrawlerName()
String	getUserAgent()
String	proxy() 可以自定义返回随机的代理
protected void	push(Request request)
void	setCrawlerName(String crawlerName)
void	setQueue(SeimiQueue queue)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface cn.wanghaomiao.seimi.core.SeimiCrawler

start, startUrls

Field Detail

queue

protected SeimiQueue queue

cookieStore

protected org.apache.http.client.CookieStore cookieStore

logger

protected org.slf4j.Logger logger

crawlerName

protected String crawlerName

Constructor Detail

BaseSeimiCrawler

public BaseSeimiCrawler()

Method Detail

push

protected void push(Request request)

getUserAgent

public String getUserAgent()

Specified by:

getUserAgent in interface SeimiCrawler

getCookieStore

public org.apache.http.client.CookieStore getCookieStore()

Description copied from interface: SeimiCrawler

如果开启cookies通过此方法获取cookiesStore

Specified by:

getCookieStore in interface SeimiCrawler

Returns:

CookieStore

allowRules

public String[] allowRules()

Description copied from interface: SeimiCrawler

用于设置允许的请求URL匹配规则

Specified by:

allowRules in interface SeimiCrawler

Returns:

白名单规则正则表达式列表

denyRules

public String[] denyRules()

Description copied from interface: SeimiCrawler

用于设置要放弃访问的请求URL匹配规则

Specified by:

denyRules in interface SeimiCrawler

Returns:

黑名单规则正则表达式列表

proxy

public String proxy()

Description copied from interface: SeimiCrawler

可以自定义返回随机的代理

Specified by:

proxy in interface SeimiCrawler

Returns:

setQueue

public void setQueue(SeimiQueue queue)

setCrawlerName

public void setCrawlerName(String crawlerName)

getCrawlerName

public String getCrawlerName()