org.eclipse.smila.bulkbuilder.helper

Class BulkbuilderBase

java.lang.Object

org.eclipse.smila.bulkbuilder.helper.BulkbuilderBase

All Implemented Interfaces:

BulkTrackerCallback

Direct Known Subclasses:

Bulkbuilder

public abstract class BulkbuilderBase
extends java.lang.Object
implements BulkTrackerCallback

basic BulkBuilder implementation. Creates bulks from the pushed records on a time and size limit.

Constructor Summary

Constructors

Constructor and Description
BulkbuilderBase(ObjectStoreService objectStore, BulkbuilderTaskProvider taskProvider, MicroBulkbuilder microBulkbuilder) Creates a new BulkBuilder instance.

Method Summary

Methods

Modifier and Type	Method and Description
WorkflowRunInfo	addRecord(java.lang.String jobName, Record record) Write record to add bulk.
protected void	addResultCounters(java.util.Map<java.lang.String,java.lang.Number> counters, BulkOutput bulk, java.lang.String slotName) add bulk counters to result description counters from performance counter.
WorkflowRunInfo	addToMicroBulk(java.lang.String jobName, Record record, java.lang.String microBulkId) Add record to micro bulk.
void	checkBulks() check bulks.
WorkflowRunInfo	commitBulk(java.lang.String jobName) Commits active bulks for given job name.
protected BulkOutput	createBulk(java.lang.String jobName, BulkInfo bulkInfo, BulkType bulkType, AnyMap taskParameters) create an BulkOutput for the bulk.
WorkflowRunInfo	deleteRecord(java.lang.String jobName, Record record) Write record to delete bulk.
WorkflowRunInfo	finishMicroBulk(java.lang.String jobName, java.lang.String microBulkId) Finishes the micro bulk.
protected BulkOutput	getActiveBulk(java.lang.String jobName, BulkType bulkType) Get active bulk data for given job name and bulk type.
protected BulkInfo	getBulkInfo(java.lang.String jobName, java.util.Map<java.lang.String,java.util.List<BulkInfo>> outputBulks, BulkType bulkType) gets the bulk info from the output bulks for the output slots connected to this worker.
protected abstract long	getBulkSizeLimit(BulkType bulkType, AnyMap taskParameters)
protected abstract long	getCommitTimeoutMillis(AnyMap taskParameters)
protected MicroBulkbuilder	getMicroBulkbuilder() get reference to microbulk helper.
protected ObjectStoreService	getObjectStore() get reference to object store service.
protected java.util.Map<java.lang.String,java.lang.Number>	getResultCounters(java.lang.String jobName, BulkOutput... bulkDatas) create taskmanager result description counters from performance counters.
protected BulkbuilderTaskProvider	getTaskProvider() get reference to task provider helper.
protected void	releaseBulk(BulkOutput currBulk) Release active bulk.
void	removeMicroBulk(java.lang.String microBulkId) Removes the micro bulk with the given id.
protected void	setActiveBulk(java.lang.String jobName, BulkOutput bulkData) set active bulk data for given job name and bulk type.
void	shutdown() Shutdown builder and close all open bulks.
protected WorkflowRunInfo	writeMicroBulkWithRetry(java.lang.String jobName, byte[] microBulk, java.lang.Integer numberOfRecords, Task task) Adds microbulk with retry if something goes wrong.
protected WorkflowRunInfo	writeRecordWithRetry(java.lang.String jobName, Record record, Task task, BulkType bulkType) Adds record with retry if something goes wrong.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BulkbuilderBase

public BulkbuilderBase(ObjectStoreService objectStore,
               BulkbuilderTaskProvider taskProvider,
               MicroBulkbuilder microBulkbuilder)
                throws BlackboardAccessException

Creates a new BulkBuilder instance.

Parameters:

objectStore - a reference to the objectStore the object store service.

taskProvider - the task provider that generates tasks for the bulkbuilder

microBulkbuilder - helper class for constructing in-memory micro bulk that can be appended to the objectstore bulk in one atomic append step.

Throws:

BlackboardAccessException - cannot access blackboard.

Method Detail

addRecord

public WorkflowRunInfo addRecord(java.lang.String jobName,
                        Record record)
                          throws BulkbuilderException

Write record to add bulk.

Parameters:

jobName - The job name

record - The record to store

Throws:

BulkbuilderException - error writing to object store

deleteRecord

public WorkflowRunInfo deleteRecord(java.lang.String jobName,
                           Record record)
                             throws BulkbuilderException

Write record to delete bulk.

Parameters:

jobName - The job name

record - The record to store

Throws:

BulkbuilderException - error writing to object store

commitBulk

public WorkflowRunInfo commitBulk(java.lang.String jobName)
                           throws BulkbuilderException

Commits active bulks for given job name.

Parameters:

jobName - The job name

Throws:

BulkbuilderException - An exception if something goes wrong

addToMicroBulk

public WorkflowRunInfo addToMicroBulk(java.lang.String jobName,
                             Record record,
                             java.lang.String microBulkId)
                               throws BulkbuilderException

Add record to micro bulk.

Parameters:

jobName - The job name

record - The record to store

microBulkId - The id of the micro bulk

Throws:

BulkbuilderException - error writing to object store

finishMicroBulk

public WorkflowRunInfo finishMicroBulk(java.lang.String jobName,
                              java.lang.String microBulkId)
                                throws BulkbuilderException

Finishes the micro bulk.

Parameters:

jobName - The job name

microBulkId - The id of the micro bulk

Throws:

BulkbuilderException - error writing to object store

removeMicroBulk

public void removeMicroBulk(java.lang.String microBulkId)

Removes the micro bulk with the given id.

Parameters:

microBulkId - The id of the micro bulk

checkBulks

public void checkBulks()
                throws BulkbuilderException

check bulks.

Specified by:

checkBulks in interface BulkTrackerCallback

Throws:

BulkbuilderException - error checking or committing bulks.

shutdown

public void shutdown()

Shutdown builder and close all open bulks.

getCommitTimeoutMillis

protected abstract long getCommitTimeoutMillis(AnyMap taskParameters)

Returns:

the timeout for committing bulks either from task parameter or from some configuration or default value.

getBulkSizeLimit

protected abstract long getBulkSizeLimit(BulkType bulkType,
                    AnyMap taskParameters)

Returns:

the maximum bulk size either from task parameter or from some configuration or default value.

getObjectStore

protected ObjectStoreService getObjectStore()

get reference to object store service.

getTaskProvider

protected BulkbuilderTaskProvider getTaskProvider()

get reference to task provider helper.

getMicroBulkbuilder

protected MicroBulkbuilder getMicroBulkbuilder()

get reference to microbulk helper.

getActiveBulk

protected BulkOutput getActiveBulk(java.lang.String jobName,
                       BulkType bulkType)

Get active bulk data for given job name and bulk type.

Parameters:

jobName - The job name.

bulkType - bulk type

Returns:

BulkData object or null if none exists.

setActiveBulk

protected void setActiveBulk(java.lang.String jobName,
                 BulkOutput bulkData)

set active bulk data for given job name and bulk type.

Parameters:

jobName - The job name.

bulkData - bulk data to set

writeRecordWithRetry

protected WorkflowRunInfo writeRecordWithRetry(java.lang.String jobName,
                                   Record record,
                                   Task task,
                                   BulkType bulkType)
                                        throws BulkbuilderException

Adds record with retry if something goes wrong.

Parameters:

jobName - The job name

record - The record to store

task - The task

Throws:

BulkbuilderException - error writing to object store

writeMicroBulkWithRetry

protected WorkflowRunInfo writeMicroBulkWithRetry(java.lang.String jobName,
                                      byte[] microBulk,
                                      java.lang.Integer numberOfRecords,
                                      Task task)
                                           throws BulkbuilderException

Adds microbulk with retry if something goes wrong.

Parameters:

jobName - The job name

microBulk - The microbulk to store

numberOfRecords - number of records in this bulk.

task - The task

Throws:

BulkbuilderException - error writing to object store

getBulkInfo

protected BulkInfo getBulkInfo(java.lang.String jobName,
                   java.util.Map<java.lang.String,java.util.List<BulkInfo>> outputBulks,
                   BulkType bulkType)
                        throws BulkbuilderException

gets the bulk info from the output bulks for the output slots connected to this worker.

Parameters:

jobName - job name

bulkType - bulk type (add or delete)

Returns:

bulk info for the operation

Throws:

BulkbuilderException - task does not have the slot connected

createBulk

protected BulkOutput createBulk(java.lang.String jobName,
                    BulkInfo bulkInfo,
                    BulkType bulkType,
                    AnyMap taskParameters)

create an BulkOutput for the bulk.

Parameters:

jobName - the name of the current job

bulkInfo - the bulk info for the curent bulk

bulkType - the bulk's type

taskParameters - task parameters

Returns:

the BulkOutput for the bulk.

releaseBulk

protected void releaseBulk(BulkOutput currBulk)

Release active bulk.

Parameters:

currBulk - current bulk for index.

getResultCounters

protected java.util.Map<java.lang.String,java.lang.Number> getResultCounters(java.lang.String jobName,
                                                                 BulkOutput... bulkDatas)

create taskmanager result description counters from performance counters.

addResultCounters

protected void addResultCounters(java.util.Map<java.lang.String,java.lang.Number> counters,
                     BulkOutput bulk,
                     java.lang.String slotName)

add bulk counters to result description counters from performance counter.