org.eclipse.smila.importing.crawler.web

Class WebCrawlingContext

java.lang.Object

org.eclipse.smila.importing.util.CrawlingContext

org.eclipse.smila.importing.crawler.web.WebCrawlingContext

public class WebCrawlingContext
extends CrawlingContext

Context holding information needed throughout most of the web crawling process like mapper, filter configuration etc. for one task

Constructor Summary

Constructors

Constructor and Description
WebCrawlingContext(TaskContext taskContext) creates a crawling context from the taskContext.

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	getCurrentInputBulkId()
WebCrawlerConstants.ErrorHandling	getErrorHandling()
java.util.Set<java.lang.String>	getExtractedUrls()
FilterConfiguration	getFilterConfiguration()
int	getLinksPerBulk()
RobotsTxt	getRobotsTxt(java.lang.String hostAndPort, JobRunDataProvider jobRunDataProvider)
TaskLog	getTaskLog()
AnyMap	getTaskParameters()
java.util.Set<java.lang.String>	getVisitedUrls()
void	putRobotsTxt(java.lang.String hostAndPort, RobotsTxt robotsTxt, JobRunDataProvider jobRunDataProvider) store robots.txt in context and job run data, if available.
void	setCurrentInputBulkId(java.lang.String inputBulkId)

Methods inherited from class org.eclipse.smila.importing.util.CrawlingContext

getDataSource, getJobName, getJobRunId, getMapper, getNonNegativeParameter, getTaskContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WebCrawlingContext

public WebCrawlingContext(TaskContext taskContext)

creates a crawling context from the taskContext.

Method Detail

getFilterConfiguration

public FilterConfiguration getFilterConfiguration()

Returns:

the _filterConfiguration

getTaskLog

public TaskLog getTaskLog()

Returns:

the _taskLog

getTaskParameters

public AnyMap getTaskParameters()

Returns:

the _parameters

getCurrentInputBulkId

public java.lang.String getCurrentInputBulkId()

Returns:

the _jobRunId

setCurrentInputBulkId

public void setCurrentInputBulkId(java.lang.String inputBulkId)

Parameters:

inputBulkId -

getVisitedUrls

public java.util.Set<java.lang.String> getVisitedUrls()

Returns:

the visited urls of this task

getLinksPerBulk

public int getLinksPerBulk()

Returns:

the number of links per output bulk.

getErrorHandling

public WebCrawlerConstants.ErrorHandling getErrorHandling()

Returns:

how to handle recoverable fetch errors.

getExtractedUrls

public java.util.Set<java.lang.String> getExtractedUrls()

Returns:

set in which the Urls are collected that have been written to linksToCrawl in this task, for duplicate prevention.

getRobotsTxt

public RobotsTxt getRobotsTxt(java.lang.String hostAndPort,
                     JobRunDataProvider jobRunDataProvider)

Returns:

robots.txt checker for given host, if already fetched.

putRobotsTxt

public void putRobotsTxt(java.lang.String hostAndPort,
                RobotsTxt robotsTxt,
                JobRunDataProvider jobRunDataProvider)

store robots.txt in context and job run data, if available.