org.eclipse.smila.importing.crawler.feed

Class FeedCrawlerWorker

java.lang.Object

org.eclipse.smila.importing.crawler.feed.FeedCrawlerWorker

All Implemented Interfaces:

Worker

public class FeedCrawlerWorker
extends java.lang.Object
implements Worker

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	INPUT_ATTRIBUTE_FEED_URL (when using input slot) Name of the input record attribute containing the feed URL.
static java.lang.String	INPUT_SLOT_LINKS_TO_CRAWL Name of the (optional) input slot containing the links to crawl.
static java.lang.Long	MAX_RECORDS_PER_BULK_DEFAULT default: write up to 1000 records to one output bulk.
static java.lang.String	NAME Name of the worker, used in worker description and workflows.
static java.lang.String	OUTPUT_SLOT_CRAWLED_RECORDS name of output slot containing the crawled records.
static java.lang.String	TASK_PARAM_DELTA_PROPERTIES feed entry properties used for delta hash.
static java.lang.String	TASK_PARAM_FEED_URL Name of the task parameter that contains the feed URL(s) (may be more than one).
static java.lang.String	TASK_PARAM_MAX_RECORDS_PER_BULK Maximum number of records in one bulk object.

Constructor Summary

Constructors

Constructor and Description
FeedCrawlerWorker()

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	getName()
void	perform(TaskContext taskContext) Performs a computation on the data available in the TaskContext, such as a task for this worker, input and (if configured) output slots.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NAME

public static final java.lang.String NAME

Name of the worker, used in worker description and workflows.

See Also:

Constant Field Values

INPUT_SLOT_LINKS_TO_CRAWL

public static final java.lang.String INPUT_SLOT_LINKS_TO_CRAWL

Name of the (optional) input slot containing the links to crawl.

See Also:

Constant Field Values

OUTPUT_SLOT_CRAWLED_RECORDS

public static final java.lang.String OUTPUT_SLOT_CRAWLED_RECORDS

name of output slot containing the crawled records.

See Also:

Constant Field Values

TASK_PARAM_FEED_URL

public static final java.lang.String TASK_PARAM_FEED_URL

Name of the task parameter that contains the feed URL(s) (may be more than one).

See Also:

Constant Field Values

TASK_PARAM_MAX_RECORDS_PER_BULK

public static final java.lang.String TASK_PARAM_MAX_RECORDS_PER_BULK

Maximum number of records in one bulk object.

See Also:

Constant Field Values

TASK_PARAM_DELTA_PROPERTIES

public static final java.lang.String TASK_PARAM_DELTA_PROPERTIES

feed entry properties used for delta hash.

See Also:

Constant Field Values

INPUT_ATTRIBUTE_FEED_URL

public static final java.lang.String INPUT_ATTRIBUTE_FEED_URL

(when using input slot) Name of the input record attribute containing the feed URL.

See Also:

Constant Field Values

MAX_RECORDS_PER_BULK_DEFAULT

public static final java.lang.Long MAX_RECORDS_PER_BULK_DEFAULT

default: write up to 1000 records to one output bulk.

Constructor Detail

FeedCrawlerWorker

public FeedCrawlerWorker()

Method Detail

getName

public java.lang.String getName()

Specified by:

getName in interface Worker

Returns:

the name of the worker. The worker function will be executed for tasks tied to this worker name.

perform

public void perform(TaskContext taskContext)
             throws java.lang.Exception

Description copied from interface: Worker

Performs a computation on the data available in the TaskContext, such as a task for this worker, input and (if configured) output slots. An implementor must make sure, calls to this method must be thread-safe!

Specified by:

perform in interface Worker

Parameters:

taskContext - the TaskContext information with which this operation can be performed.

Throws:

java.lang.Exception