Class | Description |
---|---|
BaseGroupGenerator | |
BaseScoreGenerator | |
FetchBuffer | |
FilterAndScoreByUrlAndRobots |
Filter out URLs by either domain (not popular enough) or if they're blocked by robots.txt
|
FixedScoreGenerator | |
GroupFunction | |
LoadUrlsFunction | |
MakeFetchSetsBuffer |
We get ScoredUrlDatums, grouped by server IP address.
|
NormalizeUrlFunction | |
ProcessRobotsTask | |
ResolveRedirectsTask | |
UrlFilter | |
UrlLengthener |
Copyright © 2012 Bixo Labs