| 限定符和类型 | 方法和说明 |
|---|---|
Request |
SeimiQueue.bPop(java.lang.String crawlerName)
阻塞式出队一个请求
|
| 限定符和类型 | 方法和说明 |
|---|---|
java.util.List<Request> |
SeimiCrawler.startRequests() |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
SeimiQueue.addProcessed(Request req)
记录一个处理过的请求
|
void |
SeimiCrawler.handleErrorRequest(Request request)
当一个请求处理异常次数超过开发者所设置或是默认设置的最大重新处理次数时会调用该方法记录异常请求
|
boolean |
SeimiQueue.isProcessed(Request req)
判断一个URL是否处理过了
|
Response |
SeimiDownloader.process(Request request)
处理抓取请求生成response
|
boolean |
SeimiQueue.push(Request req)
入队一个请求
|
| 限定符和类型 | 方法和说明 |
|---|---|
Request |
DefaultRedisQueue.bPop(java.lang.String crawlerName) |
Request |
DefaultLocalQueue.bPop(java.lang.String crawlerName) |
| 限定符和类型 | 方法和说明 |
|---|---|
java.util.concurrent.LinkedBlockingQueue<Request> |
DefaultLocalQueue.getQueue(java.lang.String crawlerName) |
java.util.List<Request> |
BaseSeimiCrawler.startRequests() |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
DefaultRedisQueue.addProcessed(Request req) |
void |
DefaultLocalQueue.addProcessed(Request req) |
void |
BaseSeimiCrawler.handleErrorRequest(Request request) |
boolean |
DefaultRedisQueue.isProcessed(Request req) |
boolean |
DefaultLocalQueue.isProcessed(Request req) |
boolean |
DefaultRedisQueue.push(Request req) |
boolean |
DefaultLocalQueue.push(Request req) |
protected void |
BaseSeimiCrawler.push(Request request) |
| 限定符和类型 | 方法和说明 |
|---|---|
static org.apache.http.client.methods.RequestBuilder |
HcRequestGenerator.getHttpRequestBuilder(Request request,
CrawlerModel crawlerModel) |
Response |
HcDownloader.process(Request request) |
| 限定符和类型 | 方法和说明 |
|---|---|
static okhttp3.Request.Builder |
OkHttpRequestGenerator.getOkHttpRequesBuilder(Request seimiReq,
CrawlerModel crawlerModel) |
Response |
OkHttpDownloader.process(Request request) |
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
CrawlerCache.consumeRequest(Request request) |
| 限定符和类型 | 方法和说明 |
|---|---|
static <T,A1> Request |
Request.build(java.lang.String url,
Request.SeimiCallbackFunc<T,A1> callBackFunc) |
static Request |
Request.build(java.lang.String url,
java.lang.String callBack) |
static Request |
Request.build(java.lang.String url,
java.lang.String callBack,
HttpMethod httpMethod,
java.util.Map<java.lang.String,java.lang.String> params,
java.util.Map<java.lang.String,java.lang.Object> meta) |
static Request |
Request.build(java.lang.String url,
java.lang.String callBack,
int maxReqCount) |
static <T,A1> Request |
Request.build(java.lang.String url,
java.lang.String callBack,
Request.SeimiCallbackFunc<T,A1> callBackFunc,
HttpMethod httpMethod,
java.util.Map<java.lang.String,java.lang.String> params,
java.util.Map<java.lang.String,java.lang.Object> meta,
int maxReqcount) |
Request |
Response.getRequest() |
Request |
Request.setCallBack(Request.SeimiCallbackFunc<SeimiCrawler,Response> cbFunc) |
Request |
Request.setCallBack(java.lang.String callBack) |
Request |
Request.setCrawlerName(java.lang.String crawlerName) |
Request |
Request.setCurrentReqCount(int currentReqCount) |
Request |
Request.setHeader(java.util.Map<java.lang.String,java.lang.String> header) |
Request |
Request.setHttpMethod(HttpMethod httpMethod) |
Request |
Request.setMaxReqCount(int maxReqCount) |
Request |
Request.setMeta(java.util.Map<java.lang.String,java.lang.Object> meta) |
Request |
Request.setParams(java.util.Map<java.lang.String,java.lang.String> params) |
Request |
Request.setSeimiAgentContentType(SeimiAgentContentType seimiAgentContentType) |
Request |
Request.setSeimiAgentRenderTime(long seimiAgentRenderTime) |
Request |
Request.setSeimiAgentScript(java.lang.String seimiAgentScript) |
Request |
Request.setSeimiAgentUseCookie(java.lang.Boolean seimiAgentUseCookie) |
Request |
Request.setSeimiCookies(java.util.List<SeimiCookie> seimiCookies) |
Request |
Request.setSkipDuplicateFilter(boolean skipDuplicateFilter) |
Request |
Request.setStop(boolean stop) |
Request |
Request.setUrl(java.lang.String url) |
Request |
Request.setUseSeimiAgent(boolean useSeimiAgent) |
Request |
Request.useSeimiAgent() |
| 限定符和类型 | 方法和说明 |
|---|---|
boolean |
CrawlerModel.checkRequest(Request request) |
void |
CrawlerModel.sendRequest(Request request) |
void |
Response.setRequest(Request request) |
| 限定符和类型 | 方法和说明 |
|---|---|
static java.lang.String |
GenericUtils.signRequest(Request request) |
Copyright © 2019. All Rights Reserved.