Package | Description |
---|---|
bixo.operations | |
bixo.pipes | |
bixo.robots |
Constructor and Description |
---|
FilterAndScoreByUrlAndRobots(BaseFetcher fetcher,
BaseRobotsParser parser,
BaseScoreGenerator scorer) |
FilterAndScoreByUrlAndRobots(UserAgent userAgent,
int maxThreads,
BaseRobotsParser parser,
BaseScoreGenerator scorer) |
ProcessRobotsTask(java.lang.String protocolAndDomain,
BaseScoreGenerator scorer,
java.util.Queue<GroupedUrlDatum> urls,
BaseFetcher fetcher,
BaseRobotsParser parser,
cascading.tuple.TupleEntryCollector collector,
com.scaleunlimited.cascading.LoggingFlowProcess flowProcess) |
Constructor and Description |
---|
FetchPipe(cascading.pipe.Pipe urlProvider,
BaseScoreGenerator scorer,
BaseFetcher fetcher,
BaseFetcher robotsFetcher,
BaseRobotsParser parser,
BaseFetchJobPolicy fetchJobPolicy,
int numReducers) |
Modifier and Type | Class and Description |
---|---|
class |
SimpleRobotRulesParser |
Modifier and Type | Method and Description |
---|---|
static BaseRobotRules |
RobotUtils.getRobotRules(BaseFetcher fetcher,
BaseRobotsParser parser,
java.net.URL robotsUrl)
Externally visible, static method for use in tools and for testing.
|
Copyright © 2012 Bixo Labs