org.eclipse.smila.importing.crawler.jdbc

Class JdbcCrawlerWorker

java.lang.Object

org.eclipse.smila.importing.crawler.jdbc.JdbcCrawlerWorker

All Implemented Interfaces:

Worker

public class JdbcCrawlerWorker
extends java.lang.Object
implements Worker

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	INPUT_SLOT_SPLITS_TO_CRAWL name of input slot taking the splits to be crawled (optional).
static java.lang.Long	MAX_ATTACHMENT_SIZE_DEFAULT default max size of binary content is 1 GB.
static java.lang.Long	MAX_RECORDS_PER_BULK_DEFAULT default: write up to 1000 records to one file bulk.
static java.lang.String	NAME Name of the worker, used in worker description and workflows.
static java.lang.String	OUTPUT_SLOT_CRAWLED_RECORDS name of output slot taking the file records to process in ETL.
static java.lang.String	OUTPUT_SLOT_SPLITS_TO_CRAWL name of output slot taking the splits to be crawled (optional).
static java.lang.String	TASK_PARAM_CRAWL_SQL crawl sql statement
static java.lang.String	TASK_PARAM_DB_PROPS database properties, e.g. user, password
static java.lang.String	TASK_PARAM_DB_URL database url
static java.lang.String	TASK_PARAM_DELTA_COLUMNS delta columns
static java.lang.String	TASK_PARAM_ID_COLUMNS id columns
static java.lang.String	TASK_PARAM_MAX_ATTACHMENT_SIZE max. size of binary content (BLOB etc.) in bytes.
static java.lang.String	TASK_PARAM_MAX_RECORDS_PER_BULK Maximum number of records in one bulk object.
static java.lang.String	TASK_PARAM_SPLIT_INCREMENT increment for splitted database crawling.
static java.lang.String	TASK_PARAM_SPLIT_LIMITS_SQL SQL statement to determine split limits for splitted database crawling.

Constructor Summary

Constructors

Constructor and Description
JdbcCrawlerWorker()

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	getName()
void	perform(TaskContext taskContext) Performs a computation on the data available in the TaskContext, such as a task for this worker, input and (if configured) output slots.
void	setJdbcAccessService(JdbcAccessService jdbcAccess) DS service reference bind method.
void	unsetJdbcAccessService(JdbcAccessService jdbcAccess) DS service reference unbind method.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NAME

public static final java.lang.String NAME

Name of the worker, used in worker description and workflows.

See Also:

Constant Field Values

OUTPUT_SLOT_CRAWLED_RECORDS

public static final java.lang.String OUTPUT_SLOT_CRAWLED_RECORDS

name of output slot taking the file records to process in ETL.

See Also:

Constant Field Values

INPUT_SLOT_SPLITS_TO_CRAWL

public static final java.lang.String INPUT_SLOT_SPLITS_TO_CRAWL

name of input slot taking the splits to be crawled (optional).

See Also:

Constant Field Values

OUTPUT_SLOT_SPLITS_TO_CRAWL

public static final java.lang.String OUTPUT_SLOT_SPLITS_TO_CRAWL

name of output slot taking the splits to be crawled (optional).

See Also:

Constant Field Values

TASK_PARAM_MAX_RECORDS_PER_BULK

public static final java.lang.String TASK_PARAM_MAX_RECORDS_PER_BULK

Maximum number of records in one bulk object.

See Also:

Constant Field Values

TASK_PARAM_DB_URL

public static final java.lang.String TASK_PARAM_DB_URL

database url

See Also:

Constant Field Values

TASK_PARAM_DB_PROPS

public static final java.lang.String TASK_PARAM_DB_PROPS

database properties, e.g. user, password

See Also:

Constant Field Values

TASK_PARAM_CRAWL_SQL

public static final java.lang.String TASK_PARAM_CRAWL_SQL

crawl sql statement

See Also:

Constant Field Values

TASK_PARAM_ID_COLUMNS

public static final java.lang.String TASK_PARAM_ID_COLUMNS

id columns

See Also:

Constant Field Values

TASK_PARAM_DELTA_COLUMNS

public static final java.lang.String TASK_PARAM_DELTA_COLUMNS

delta columns

See Also:

Constant Field Values

TASK_PARAM_MAX_ATTACHMENT_SIZE

public static final java.lang.String TASK_PARAM_MAX_ATTACHMENT_SIZE

max. size of binary content (BLOB etc.) in bytes.

See Also:

Constant Field Values

TASK_PARAM_SPLIT_INCREMENT

public static final java.lang.String TASK_PARAM_SPLIT_INCREMENT

increment for splitted database crawling.

See Also:

Constant Field Values

TASK_PARAM_SPLIT_LIMITS_SQL

public static final java.lang.String TASK_PARAM_SPLIT_LIMITS_SQL

SQL statement to determine split limits for splitted database crawling.

See Also:

Constant Field Values

MAX_RECORDS_PER_BULK_DEFAULT

public static final java.lang.Long MAX_RECORDS_PER_BULK_DEFAULT

default: write up to 1000 records to one file bulk.

MAX_ATTACHMENT_SIZE_DEFAULT

public static final java.lang.Long MAX_ATTACHMENT_SIZE_DEFAULT

default max size of binary content is 1 GB.

Constructor Detail

JdbcCrawlerWorker

public JdbcCrawlerWorker()

Method Detail

getName

public java.lang.String getName()

Specified by:

getName in interface Worker

Returns:

the name of the worker. The worker function will be executed for tasks tied to this worker name.

perform

public void perform(TaskContext taskContext)
             throws java.lang.Exception

Description copied from interface: Worker

Performs a computation on the data available in the TaskContext, such as a task for this worker, input and (if configured) output slots. An implementor must make sure, calls to this method must be thread-safe!

Specified by:

perform in interface Worker

Parameters:

taskContext - the TaskContext information with which this operation can be performed.

Throws:

java.lang.Exception

setJdbcAccessService

public void setJdbcAccessService(JdbcAccessService jdbcAccess)

DS service reference bind method.

unsetJdbcAccessService

public void unsetJdbcAccessService(JdbcAccessService jdbcAccess)

DS service reference unbind method.