Package | Description |
---|---|
bixo.robots |
Modifier and Type | Class and Description |
---|---|
class |
SimpleRobotRules
Result from parsing a single robots.txt file - which means we
get a set of rules, and a crawl-delay.
|
Modifier and Type | Method and Description |
---|---|
BaseRobotRules |
SimpleRobotRulesParser.failedFetch(int httpStatusCode) |
abstract BaseRobotRules |
BaseRobotsParser.failedFetch(int httpStatusCode)
The fetch of robots.txt failed, so return rules appropriate give the
HTTP status code.
|
static BaseRobotRules |
RobotUtils.getRobotRules(BaseFetcher fetcher,
BaseRobotsParser parser,
java.net.URL robotsUrl)
Externally visible, static method for use in tools and for testing.
|
BaseRobotRules |
SimpleRobotRulesParser.parseContent(java.lang.String url,
byte[] content,
java.lang.String contentType,
java.lang.String robotName) |
abstract BaseRobotRules |
BaseRobotsParser.parseContent(java.lang.String url,
byte[] content,
java.lang.String contentType,
java.lang.String robotName)
Parse the robots.txt file in
|
Copyright © 2012 Bixo Labs