org.eclipse.smila.tika

Class TikaPipelet

java.lang.Object

org.eclipse.smila.processing.pipelets.ATransformationPipelet

org.eclipse.smila.tika.TikaPipelet

All Implemented Interfaces:

Pipelet

public class TikaPipelet
extends ATransformationPipelet

The TikaPipelet can be used for converting various document formats to text and extracting metadata.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TikaPipelet.StoreMode possible values for parameter 'storeMode'.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_MAX_LENGTH default length. -1 is "unlimited".
static java.lang.String	DIV_PAGE_END_TAG Page end tag after the content was exported as (X)HTML.
static java.lang.String	DIV_PAGE_START_TAG Page start tag after the content was exported as (X)HTML.
static java.lang.String	PROP_ATTACHMENT_CONTENT_TYPE_ATTRIBUTE (Optional) parameter referencing the attribute that contains the content type (e.g.
static java.lang.String	PROP_EXPORT_AS_HTML (Optional) parameter that defines if the content should be transformed to (X)HTML (true) or plain text (false).
static java.lang.String	PROP_EXTRACT_PROPERTIES (Optional) Parameter that defines what to extract from input and copy into record attributes with the name of the extracted properties.
static java.lang.String	PROP_FILE_NAME_ATTRIBUTE (Optional) parameter referencing the attribute that contains the file name that can give the Tika parser possibly a hint about how to parse the file.
static java.lang.String	PROP_KEEP_HYPHENS (Optional) parameter that defines whether the hyphens should be kept in the output (as in the input) (true) or whether the software should try to remove the hyphens by a heuristic approach.
static java.lang.String	PROP_MAPPING_METADATA_NAME name of the metadata field (will be matched case-insensitively).
static java.lang.String	PROP_MAPPING_SINGLE_RESULT will only one result (true) multiple, if available (false) be considered.
static java.lang.String	PROP_MAPPING_STORE_MODE (Optional) parameter that defines how the extracted properties are stored in the target attribute.
static java.lang.String	PROP_MAPPING_TARGET_ATTRIBUTE name of the target attribute for the metadata entry (optional, default: the value of PROP_MAPPING_METADATA_NAME with its original case.
static java.lang.String	PROP_MAX_LENGTH (Optional) parameter that defines how many characters of the content should be extracted to prevent out of memory leaks.
static java.lang.String	PROP_PAGE_BREAK (Optional) parameter that defines whether page breaks should be marked with a in the result.
static java.lang.String	PROP_PAGE_NUMBER_ATTRIBUTE (Optional) parameter referencing the attribute that that contains the page number.
static java.lang.String	PROP_PARTS_ATTRIBUTE (Optional) If the parameter is set, the content is split into multiple parts using page-breaks and the parts, represented by a sequence of maps, are stored to an output record.
static java.lang.String	SPLIT_ID_SEPARATOR separator used to create the record id of records split when pageBreak=true.

Fields inherited from class org.eclipse.smila.processing.pipelets.ATransformationPipelet

_config, ENCODING_ATTACHMENT, ENCODING_CHARSET, PROP_INPUT_NAME, PROP_INPUT_TYPE, PROP_OUTPUT_NAME, PROP_OUTPUT_TYPE, PROP_OUTPUT_VALUE_TYPE

Constructor Summary

Constructors

Constructor and Description
TikaPipelet()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(AnyMap configuration) set configuration of pipelet. called once after instantiation before the pipelet is actually used in a workflow.
java.lang.String[]	process(Blackboard blackboard, java.lang.String[] recordIds) process given records.

Methods inherited from class org.eclipse.smila.processing.pipelets.ATransformationPipelet

getInputName, getInputStream, getInputType, getOutputName, getOutputType, getOutputValueType, isReadFromAttribute, isStoreInAttribute, readInput, readStringInput, storeResult, storeResult, storeResult, storeResults

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PROP_ATTACHMENT_CONTENT_TYPE_ATTRIBUTE

public static final java.lang.String PROP_ATTACHMENT_CONTENT_TYPE_ATTRIBUTE

(Optional) parameter referencing the attribute that contains the content type (e.g. "text/plain") of the attachment content. The parameter (resp. attribute) may not be set (null/empty) and then a content type detection is performed.

See Also:

Constant Field Values

PROP_FILE_NAME_ATTRIBUTE

public static final java.lang.String PROP_FILE_NAME_ATTRIBUTE

(Optional) parameter referencing the attribute that contains the file name that can give the Tika parser possibly a hint about how to parse the file.

See Also:

Constant Field Values

PROP_EXTRACT_PROPERTIES

public static final java.lang.String PROP_EXTRACT_PROPERTIES

(Optional) Parameter that defines what to extract from input and copy into record attributes with the name of the extracted properties. This is a list of maps or one single map (if only one mapping is needed) describing the mapping of one metadata attribute, each. The metadata keys will be matched case-insensitively.

See Also:

Constant Field Values

PROP_EXPORT_AS_HTML

public static final java.lang.String PROP_EXPORT_AS_HTML

(Optional) parameter that defines if the content should be transformed to (X)HTML (true) or plain text (false). Default: false.

See Also:

Constant Field Values

PROP_PAGE_BREAK

public static final java.lang.String PROP_PAGE_BREAK

(Optional) parameter that defines whether page breaks should be marked with a in the result. Default: false.

See Also:

Constant Field Values

PROP_PAGE_NUMBER_ATTRIBUTE

public static final java.lang.String PROP_PAGE_NUMBER_ATTRIBUTE

(Optional) parameter referencing the attribute that that contains the page number. This is only evaluated if parameter page breaks is set to true. If not specified the attribute is not set. Default: not set

See Also:

Constant Field Values

PROP_PARTS_ATTRIBUTE

public static final java.lang.String PROP_PARTS_ATTRIBUTE

(Optional) If the parameter is set, the content is split into multiple parts using page-breaks and the parts, represented by a sequence of maps, are stored to an output record. The parameter defines the key for this sequence. Default: not set. The parameter is evaluated only if the page break parameter is set to true.

See Also:

Constant Field Values

PROP_KEEP_HYPHENS

public static final java.lang.String PROP_KEEP_HYPHENS

(Optional) parameter that defines whether the hyphens should be kept in the output (as in the input) (true) or whether the software should try to remove the hyphens by a heuristic approach. Default: true.

See Also:

Constant Field Values

PROP_MAX_LENGTH

public static final java.lang.String PROP_MAX_LENGTH

(Optional) parameter that defines how many characters of the content should be extracted to prevent out of memory leaks. Defaults to -1 ("unlimited").

See Also:

Constant Field Values

DEFAULT_MAX_LENGTH

public static final int DEFAULT_MAX_LENGTH

default length. -1 is "unlimited".

See Also:

Constant Field Values

PROP_MAPPING_METADATA_NAME

public static final java.lang.String PROP_MAPPING_METADATA_NAME

name of the metadata field (will be matched case-insensitively).

See Also:

Constant Field Values

PROP_MAPPING_TARGET_ATTRIBUTE

public static final java.lang.String PROP_MAPPING_TARGET_ATTRIBUTE

name of the target attribute for the metadata entry (optional, default: the value of PROP_MAPPING_METADATA_NAME with its original case.

See Also:

Constant Field Values

PROP_MAPPING_SINGLE_RESULT

public static final java.lang.String PROP_MAPPING_SINGLE_RESULT

will only one result (true) multiple, if available (false) be considered. Default: false.

See Also:

Constant Field Values

PROP_MAPPING_STORE_MODE

public static final java.lang.String PROP_MAPPING_STORE_MODE

(Optional) parameter that defines how the extracted properties are stored in the target attribute.

See Also:

Constant Field Values

SPLIT_ID_SEPARATOR

public static final java.lang.String SPLIT_ID_SEPARATOR

separator used to create the record id of records split when pageBreak=true.

See Also:

Constant Field Values

DIV_PAGE_START_TAG

public static final java.lang.String DIV_PAGE_START_TAG

Page start tag after the content was exported as (X)HTML.

See Also:

Constant Field Values

DIV_PAGE_END_TAG

public static final java.lang.String DIV_PAGE_END_TAG

Page end tag after the content was exported as (X)HTML.

See Also:

Constant Field Values

Constructor Detail

TikaPipelet

public TikaPipelet()

Method Detail

configure

public void configure(AnyMap configuration)
               throws ProcessingException

set configuration of pipelet. called once after instantiation before the pipelet is actually used in a workflow.

Specified by:

configure in interface Pipelet

Overrides:

configure in class ATransformationPipelet

Parameters:

configuration - configuration of pipelet.

Throws:

ProcessingException - configuration is not applicable for pipelet (missing properties, wrong datatypes)

process

public java.lang.String[] process(Blackboard blackboard,
                         java.lang.String[] recordIds)
                           throws ProcessingException

process given records.

Parameters:

blackboard - Blackboard holding and managing the records.

recordIds - Ids of records to process.

Returns:

Ids of records to be passed into the next pipelet. By default this should be the same as the passed in recordIds unless there is a specific (businesslogic) reason not to do so.

Throws:

ProcessingException - error during processing.