public class RobotUtils
extends java.lang.Object
Constructor and Description |
---|
RobotUtils() |
Modifier and Type | Method and Description |
---|---|
static BaseFetcher |
createFetcher(BaseFetcher fetcher) |
static BaseFetcher |
createFetcher(UserAgent userAgent,
int maxThreads) |
static long |
getMaxFetchTime() |
static BaseRobotRules |
getRobotRules(BaseFetcher fetcher,
BaseRobotsParser parser,
java.net.URL robotsUrl)
Externally visible, static method for use in tools and for testing.
|
public static BaseFetcher createFetcher(BaseFetcher fetcher)
public static BaseFetcher createFetcher(UserAgent userAgent, int maxThreads)
public static long getMaxFetchTime()
public static BaseRobotRules getRobotRules(BaseFetcher fetcher, BaseRobotsParser parser, java.net.URL robotsUrl)
fetcher
- Fetcher for downloading robots.txt filerobotsUrl
- URL to robots.txt fileCopyright © 2012 Bixo Labs