public class AdaptiveFetcherPolicy extends FetcherPolicy
FetcherPolicy.FetcherMode, FetcherPolicy.RedirectMode
_crawlDelay, DEFAULT_ACCEPT_LANGUAGE, DEFAULT_CRAWL_DELAY, DEFAULT_CRAWL_END_TIME, DEFAULT_MAX_CONNECTIONS_PER_HOST, DEFAULT_MAX_CONTENT_SIZE, DEFAULT_MAX_REDIRECTS, DEFAULT_MAX_REQUESTS_PER_CONNECTION, DEFAULT_MIN_PAGE_FETCH_INTERVAL, DEFAULT_MIN_RESPONSE_RATE, NO_CRAWL_END_TIME, NO_MIN_RESPONSE_RATE, NO_REDIRECTS
Constructor and Description |
---|
AdaptiveFetcherPolicy(int minResponseRate,
int maxContentSize,
long crawlEndTime,
long crawlDelay) |
AdaptiveFetcherPolicy(long crawlEndTime,
long crawlDelay) |
Modifier and Type | Method and Description |
---|---|
FetchRequest |
getFetchRequest(long now,
long crawlDelay,
int maxUrls) |
int |
getMaxRequestsPerConnection() |
addValidMimeType, addValidMimeTypes, calcMaxUrls, equals, getAcceptLanguage, getCrawlDelay, getCrawlEndTime, getDefaultCrawlDelay, getFetcherMode, getMaxConnectionsPerHost, getMaxContentSize, getMaxRedirects, getMaxUrls, getMinPageFetchInterval, getMinResponseRate, getRedirectMode, getRequestTimeout, getValidMimeTypes, hashCode, isTerminateFetch, seMinPageFetchInterval, setAcceptLanguage, setCrawlDelay, setCrawlEndTime, setFetcherMode, setMaxConnectionsPerHost, setMaxContentSize, setMaxRedirects, setMaxRequestsPerConnection, setMinResponseRate, setRedirectMode, setRequestTimeout, setValidMimeTypes, toString
public AdaptiveFetcherPolicy(long crawlEndTime, long crawlDelay)
public AdaptiveFetcherPolicy(int minResponseRate, int maxContentSize, long crawlEndTime, long crawlDelay)
public int getMaxRequestsPerConnection()
getMaxRequestsPerConnection
in class FetcherPolicy
public FetchRequest getFetchRequest(long now, long crawlDelay, int maxUrls)
Copyright © 2012 Bixo Labs