Implementing a submittable service job

Implementing a new service job in g-Eclipse is quite easy. Service Jobs framework provides abstract implementations for the most common use cases:

1. Sending JSDL representing a service job.
2. Monitoring job status.
3. When job finishes, downloading job output and initiating computing job's result.

What is left to the developer is to provide few methods, mainly the ones which will:

provide JSDL(s) for submission
provide information needed to store and check job's status
compute the result of the service job when grid job finishes

Interfaces and abstract implementations

eu.geclipse.core.model.IServiceJob: This is the basic interface for all service implementations in service job framework. However, developers of the service job should use abstract implementation described below.
eu.geclipse.servicejob.model.tests.job.AbstractSubmittableServiceJob: This abstract implementation of the IServiceJob interface provides useful implementation for the most common use cases: submitting a JSDL, starting service job status updater and monitor status of the job, initialize computing of the result when service job finishes. Also see eu.geclipse.servicejob.model.AbstractServiceJob, which is used by AbstractSubmittableServiceJob, for useful methods and fields.
When extending this abstract class, developer is required to implement several abstract methods, which will provide all informations needed to submit, monitor and download outputs of the jobs. Below you can find a list of all required and optional methods to overwrite.

Abstract methods

Also see eu.geclipse.servicejob.model.submittable.job. AbstractSubmittableServiceJob abstract class for documentation.

Initializing service job data

This is a very important step. When service job is submitted using Service Job Framework several methods of the service job implementation are used to get needed information about the service job. Below are the methods you need to implement.

public void initData(): This method is invoked when service job is created. If the service job requires to run some piece of code, this is a good place to perform that operation.
public List getServiceJobResourcesNames(): Returns list of names of the resources where service job should be run. If you don't want to implement this method, the abstract implementation returns value of the serviceJobResources field of the AbstractServiceJob.
public String getResultType( final String singleServiceJobName ): Each service job can provide different kinds of results: text, images, charts etc. This method specifies name ( extension ) of the type of result which is returned by this service job.
public String getRawDataInput( final IGridJobID jobID, final String resourceName ): Every service job, especially the ones which use submission mechanism, need some informations to access the job and its results. When implementing this method developer needs to provide string XML containing all of the needed information, for example: ID of the job, job service URI which you need to ask about job status, VO used for job submission etc. Developer then need to provide service job specific parsers and writers to read and update the informations he needs.
Also see eu.geclipse.benchmarking.model.service.BenchmarkServiceJob for exemplary XML in case of benchmarking services in gLite, and eu.geclipse.benchmarking.model.service.BenchmarkServiceJobParser for XML parser in this case.

Providing JSDL for submission

Submittable service jobs are started by submitting appropriate job description to run on specified resource. Developer which will extend AbstractSubmittableServiceJob needs to provide such job description by implementing following method:

public IGridJobDescription getJSDLForSubmission( final String resourceName ): This method should provide a submittable job description which will be run on resourceName.

Provided job description will be then submitted to a submission service specified by the submissionService field of the AbstractSubmittableServiceJob. If you need to provide submission service in different way, you can overwrite public IGridJobService getSubmissionService() method.

Note: When using gEclipse UI the standard action "Run Service Job" will fetch list of available job submission services of this service job's VO. User selects one of them and the job is submitted using this submission service.

Creating middleware specific job ID instance

Service jobs are middleware-dependant. Since job implementations are quite different between each middleware, it's service job developer duty to know how to construct such job ID. To construct such middleware specific job you often need additional informations( for example string representing job ID, name of the VO, etc. ). It's often good idea to get XML containing service job specific information by invoking getResultRawData() and parsing the results.

public IGridJobID createJobID( final SubmittableServiceJobResult serviceJobResult ): Method which creates middleware-specific instance of IGridJobID implementation, given the serviceJobResult.

Computing service job result

After job associated with a service job finishes the corresponding service job's implementation of the method computeJobResult( final IGridJobID jobID, final IGridJobStatus jobStatus ) is invoked. The usual implementation proceeds as follows:

1. Check the jobStatus of the job. For jobs which were not successful go to point number 4.
2. Download output(s) of the job.
3. Parse the job results for service job result.
4. Set the result of the service job.

When job finishes Service Job Framework sets status of the job as the service job result. This often is not correct: failed jobs don't necessarily mean that the service job also fails. Thats why setting the service job result at the end of computeJobResult method is advised. To set result of the job use:

public void setJobResult( final IGridJobID jobID, final Date lastRefreshDate, final String status, final String besStatus ): Method used to set status for job with the given jobID, which was read from job service at lastRefreshDate. Last parameter, besStatus, should be one of the global job statuses. See eu.geclipse.servicejob.model.ServiceJobStates enumeration class for list of available statuses.

Note:Downloading outputs of the job is specific for each middleware. Because Service Job Framework is middleware-independent developers need to know how to download output of the job having middleware-specific implementation of the IGridJobID. For example you can check eu.geclipse.benchmarking.model.service.BenchmarkServiceJob class and protected IFile getOutput( final IGridJobID jobID, final IFolder folder, final String fileName ) method how this is done in gLite middleware case.

Computing test result

After the last job of the service job has finished, then whole service job result can be computed. This can mean checking if all sub-jobs were finished successfully or checking all outputs generated by the sub-jobs. Developer need to handle this case to compute whole service job result and proceed with the result.

: Method which is invoked after last job of this service job finishes and it's results are computed using computeJobResult(IGridJobID, IGridJobStatus). This is a method which should be used to store the results of the service job in a database or in another modules.

Exemplary implementations

All of the current implementations of the submittable service jobs are for gLite middleware. Below you can find list of all implementations with references to the most important classes.

Benchmarking of the grid resources

This service job is used to benchmark various resources on the grid. User needs to provide file used for benchmarking (you can find 2 benchmarks in the eu.geclipse.benchmarking/xml - bonnie.bsx and flops.bsx) and additional parameters, if needed. Results are stored in local database, and can be viewed using Benchmark View.

eu.geclipse.benchmarking.model.service.BenchmarkServiceJob: Class extending submittable service job abstraction. You can see how does raw input data in either getRawInputData method look like. Also you can check how you can download job output using eu.geclipse.glite.resources.GliteWMSJobID in the method IFile getOutput( final IGridJobID jobID, final IFolder folder, final String fileName ).
eu.geclipse.benchmarking.model.service.BenchmarkServiceJobParser: Parser dedicated to benchmarking. It creates GliteWMSJobID from the raw input data XML stored in the service job status file.
eu.geclipse.gbdl.ui.wizards.BenchmarkWizard: Implementation of eu.geclipse.servicejob.ui.wizard.AbstractServiceJobWizardNode, adding benchmarking pages to service job wizard.

Application deployment and uninstall

Application deployment deals with cases of software installation on computing elements(CEs), while uninstall is the reverse problem. User is asked to select target CEs and the appropriate application (in form of a file) he wants to deploy or uninstall.

eu.geclipse.glite.servicejob.deployment.DeploymentServiceJob: Class extending submittable service job abstraction. Main task is to create proper JDL file. You can check also the overwritten run() action to perform some additional tasks just before job submission.
eu.geclipse.glite.servicejob.deployment.UninstallServiceJob: Similar to previous class, but takes care of uninstalling of the application.
eu.geclipse.glite.servicejob.deployment.DeploymentParser: Parser for both deployment and uninstall tasks. Beside usual creation of job ID from the XML stored on file also parses several other data and informations.
eu.geclipse.glite.ui.servicejob.deployment.GliteDeploymentWizard/GliteUninstallWizard: Wizard nodes for deployment and uninstall, extending service job wizard node abstraction.

gLite version

This service job was the first implementation of the g-Eclipse service job framework. It tests selected resources for installed gLite version.

eu.geclipse.glite.servicejob.gliteversion.GliteVersionJobTest: Class extending submittable service job abstraction.
eu.geclipse.glite.servicejob.gliteversion.GTDLGliteVersionParser: Parser for gLite version test.