public class FetchPipe
extends cascading.pipe.SubAssembly
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
CONTENT_PIPE_NAME |
static java.lang.String |
STATUS_PIPE_NAME |
Constructor and Description |
---|
FetchPipe(cascading.pipe.Pipe urlProvider,
BaseScoreGenerator scorer,
BaseFetcher fetcher,
BaseFetcher robotsFetcher,
BaseRobotsParser parser,
BaseFetchJobPolicy fetchJobPolicy,
int numReducers) |
FetchPipe(cascading.pipe.Pipe urlProvider,
BaseScoreGenerator scorer,
BaseFetcher fetcher,
int numReducers)
Generate an assembly that will fetch all of the UrlDatum tuples coming out of urlProvider.
|
Modifier and Type | Method and Description |
---|---|
cascading.pipe.Pipe |
getContentTailPipe() |
cascading.pipe.Pipe |
getStatusTailPipe() |
static java.util.Map<java.lang.String,cascading.tap.Tap> |
makeSinkMap(cascading.tap.Tap statusSink,
cascading.tap.Tap fetchedSink)
Utility routine that helps create the Cascading map needed when there are
multiple tails (like with this subassembly) and you need to build the Flow
|
getName, getPrevious, getTailNames, getTails, setPrevious, setTails, unwind
equals, getConfigDef, getHeads, getParent, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasStepConfigDef, id, isEquivalentTo, named, names, outgoingScopeFor, pipes, print, printInternal, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, setParent, toString
public static final java.lang.String CONTENT_PIPE_NAME
public static final java.lang.String STATUS_PIPE_NAME
public FetchPipe(cascading.pipe.Pipe urlProvider, BaseScoreGenerator scorer, BaseFetcher fetcher, int numReducers)
urlProvider
- scorer
- fetcher
- numReducers
- public FetchPipe(cascading.pipe.Pipe urlProvider, BaseScoreGenerator scorer, BaseFetcher fetcher, BaseFetcher robotsFetcher, BaseRobotsParser parser, BaseFetchJobPolicy fetchJobPolicy, int numReducers)
public cascading.pipe.Pipe getContentTailPipe()
public cascading.pipe.Pipe getStatusTailPipe()
public static java.util.Map<java.lang.String,cascading.tap.Tap> makeSinkMap(cascading.tap.Tap statusSink, cascading.tap.Tap fetchedSink)
statusSink
- Tap where status will be sent (can be null)fetchedSink
- Tap where fetched content will be sent (can be null)Copyright © 2012 Bixo Labs