org.eclipse.smila.importing.crawler.web

Class WebCrawlingContext

java.lang.Object

org.eclipse.smila.importing.util.CrawlingContext

org.eclipse.smila.importing.crawler.web.WebCrawlingContext

public class WebCrawlingContext
extends CrawlingContext

Context holding information needed throughout most of the web crawling process like mapper, filter confiruration etc. for one task

Constructor Summary

Constructors

Constructor and Description
WebCrawlingContext(TaskContext taskContext) creates a crawling context from the taskContext.

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	getCurrentInputBulkId()
java.util.Set<java.lang.String>	getExtractedUrls()
FilterConfiguration	getFilterConfiguration()
int	getLinksPerBulk()
TaskLog	getTaskLog()
AnyMap	getTaskParameters()
java.util.Set<java.lang.String>	getVisitedUrls()
void	setCurrentInputBulkId(java.lang.String inputBulkId)

Methods inherited from class org.eclipse.smila.importing.util.CrawlingContext

getDataSource, getJobRunId, getMapper, getNonNegativeParameter, getTaskContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WebCrawlingContext

public WebCrawlingContext(TaskContext taskContext)

creates a crawling context from the taskContext.

Method Detail

getFilterConfiguration

public FilterConfiguration getFilterConfiguration()

Returns:

the _filterConfiguration

getTaskLog

public TaskLog getTaskLog()

Returns:

the _taskLog

getTaskParameters

public AnyMap getTaskParameters()

Returns:

the _parameters

getCurrentInputBulkId

public java.lang.String getCurrentInputBulkId()

Returns:

the _jobRunId

setCurrentInputBulkId

public void setCurrentInputBulkId(java.lang.String inputBulkId)

Parameters:

inputBulkId -

getVisitedUrls

public java.util.Set<java.lang.String> getVisitedUrls()

Returns:

the visited urls of this task

getLinksPerBulk

public int getLinksPerBulk()

Returns:

the number of links per output bulk.

getExtractedUrls

public java.util.Set<java.lang.String> getExtractedUrls()

Returns:

set in which the Urls are collected that have been written to linksToCrawl in this task, for duplicate prevention.