Camel jdbc component

Camel provides a camel-jdbc component, where SQL queries are sent in the message body to perform database operations.

The URI syntax is as follows:

jdbc:dataSourceName[?options]

To demonstrate the camel-jdbc concept, we will test the following examples:

Example using simple sql query,
Example using stored procedure,
Example for serializing query response objects to CSV, XML and JSON format.

Simple sql query

First, let's create the Employees table

Creating a Table

-- create Employees table
  create table Employees (
  employee_id   number primary key
  First_Name         VARCHAR2(10 BYTE),
  Last_Name          VARCHAR2(10 BYTE),
  Start_Date         DATE DEFAULT (sysdate),
  Salary             Number(8,2),
  City               VARCHAR2(10 BYTE)
);

Once the table is created, you can use the desc command to display its structure:

SQL> desc employees;
Name        Type         Nullable Default   Comments 
----------- ------------ -------- --------- -------- 
EMPLOYEE_ID NUMBER                                   
FIRST_NAME  VARCHAR2(10) Y                           
LAST_NAME   VARCHAR2(10) Y                           
START_DATE  DATE         Y        (sysdate)          
SALARY      NUMBER(8,2)  Y                           
CITY        VARCHAR2(10) Y                           

SQL>

Route example

In the following example, we poll a database using the JDBC component and a Timer to trigger the sql request. The JDBC component executes the sql commande. Then, the result will be marshaled using Apache Commons CSV library.

<camel:route id="testjdbc" shutdownRoute="Defer">
			<camel:from uri="timer://starttest?period=10000" />
			<camel:setBody>
				<camel:constant>select * from Employees</camel:constant>
			</camel:setBody>
			<camel:to uri="jdbc:dataSource"/>
			
			<camel:marshal>
				<camel:csv delimiter=","/>
			</camel:marshal>
			
			<camel:to uri="log:marshal" />
</camel:route>

Data source configuration:

A data source should be also created in Spring like this:

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
   <property name="driverClassName" value="oracle.jdbc.driver.OracleDriver"/>
        <property name="url"  value="jdbc:oracle:thin:@x.x.x.x:1521:myDataSource"/>
        <property name="username" value="scott" />
        <property name="password" value="tiger" />
</bean>

<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
        <constructor-arg ref="dataSource"/>
</bean>

Result

The sql query result is returned in the OUT body as an ArrayList<HashMap<String, Object>>. The data is then passed to Camel CSV component to transform it to CSV format.

[2012-03-27 17:38:25,925] INFO  : org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:1310) - Total 1 routes, of which 1 is started.
[2012-03-27 17:38:25,926] INFO  : org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:1311) - Apache Camel 2.6.0 (CamelContext: camel-1) started in 0.714 seconds
[2012-03-27 17:38:26,367] INFO  : org.apache.camel.processor.Logger.process(Logger.java:89) - Exchange[ExchangePattern:InOnly, BodyType:byte[], Body:
1,Jason,Martin,29/03/2012,1234,56,Toronto
9,Alison,Mathews,30/03/2012,6661,78,Vancouver
5,Linda,Green,30/03/2012,4322,78,New York
12,Celia,Rice,29/03/2012,1234,00 

]

Stored Procedure sample

In this example, a stored procedure to add a new employee record to a table will be used.

Creating a procedure

The following statement creates the procedure AddNewEmployee which add a new employee to the Employees Table.

the pl/sql procedure is shown bellow:

create or replace procedure AddNewEmployee(employee_id in number, p_FirstName in VARCHAR2,p_LastName   in VARCHAR2, p_Salary   in   Number) is
begin
  insert into Employees(employee_id,  First_Name, Last_Name, Salary) 
  values (employee_id,  p_FirstName, p_LastName, p_Salary);
  
  COMMIT;
end AddNewEmployee;

Remark: How to execute a Stored Procedure

After successfully compiling a program, you'll need to test it. Hence, there are tree ways to execute a procedure.

From the SQL prompt:
EXECUTE [or EXEC] procedure_name;
Within another procedure - simply use the procedure name.
procedure_name;
Finally, you can use PL/SQL Developer's Test Scripts: AddNewEmployee can be tested using the following PL/SQL block:

Setting up Spring to use a Camel route that uses the stroredProcedureBean

First you define StroredProcedureBean in the Spring XML file with the id stroredProcedureBean. you must define, also, your stored procedure name as property in this spring bean.

<bean id="stroredProcedureBean" class="com.infinity.***************StroredProcedureBean">
		<property name="applyAddEmployeeProc" ref="addEmployeeProc" />
</bean>

<bean id="addEmployeeProc" class="com.infinity.AddEmployeeProc">
        <constructor-arg ref="procDataSource" />
        <constructor-arg value="AddNewEmployee" />
</bean>

Remarks:

procDataSource: refer to the datasource defined in another XML file.
AddNewEmployee: is the stored procedure name.

DataSource configuration

Declaration of a DataSource in Spring like this:

    <context:property-placeholder location="classpath:mpc.properties"/>
	
    <bean id="dataSourceTarget"
        class="org.springframework.jdbc.datasource.DriverManagerDataSource">
        <property name="driverClassName" value="oracle.jdbc.OracleDriver" />
        <property name="url"
            value="jdbc:oracle:thin:@${mpc.datasource.host}:${mpc.datasource.port}:${mpc.datasource.sid}" />
        <property name="username" value="${mpc.datasource.user}" />
        <property name="password" value="${mpc.datasource.password}" />
    </bean>
    
    <bean id="procDataSource"
        class="org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy">
        <property name="targetDataSource" ref="dataSourceTarget" />
    </bean>

Camel route

The camel route code for calling this stored procedure is as follows:

<route id="TestStoredProc" startupOrder="1" shutdownRoute="Defer"> 
			<from uri="timer://toInboundQueue?period=6000"/> 
			<log message=" recieved msg: ${body}"/> 

			<setProperty propertyName="employee_id"> 
				<constant>12</constant> 
			</setProperty>

			<setProperty propertyName="p_FirstName">
				<constant>Celia</constant>
			</setProperty>
			<setProperty propertyName="p_LastName">
				<constant>Rice</constant>
			</setProperty>
			<setProperty propertyName="p_Salary">
				<constant>1234</constant>
			</setProperty>
			<bean ref="stroredProcedureBean">
			</bean>
			<log message=" End of addNewEmployee"/>
			<to uri="mock:testjdbc"/>
</route>

Bean declaration

First, create a class which extends from StoredProcedure Spring api:

public class AddEmployeeProc extends StoredProcedure {

	public AddEmployeeProc(DataSource dataSource, String procName) {

		setDataSource(dataSource);
		setSql(procName);
		JdbcTemplate jdbcTemplate = new JdbcTemplate();
		jdbcTemplate.setDataSource(dataSource);
		setJdbcTemplate(jdbcTemplate);
		// input
		declareParameter(new SqlParameter("employee_id", OracleTypes.NUMBER));
		declareParameter(new SqlParameter("p_FirstName", OracleTypes.VARCHAR));
		declareParameter(new SqlParameter("p_LastName", OracleTypes.VARCHAR));
		declareParameter(new SqlParameter("p_Salary", OracleTypes.NUMBER));
		compile();
	}

	@SuppressWarnings("rawtypes")
	@Override
	public Map execute(int employee_id, String p_FirstName, String p_LastName,
			int p_Salary) {
		Map inParameters = new HashMap();
		Map outParameters = new HashMap();

		inParameters.put("employee_id", employee_id);
		inParameters.put("p_FirstName", p_FirstName);
		inParameters.put("p_LastName", p_LastName);
		inParameters.put("p_Salary", p_Salary);

		outParameters = execute(inParameters);
		logger.debug("Data Inserted Successfully ...");

		return outParameters;
	}

StroredProcedureBean class:

public class StroredProcedureBean {

	AddEmployeeProc applyAddEmployeeProc;

	@Handler
	public void inParameters(@Property("employee_id") Integer employee_id,
			@Property("p_FirstName") String p_FirstName,
			@Property("p_LastName") String p_LastName,
			@Property("p_Salary") Integer p_Salary) {

		applyAddEmployeeProc.execute(employee_id, p_FirstName, p_LastName,
				p_Salary);
	}

	public AddEmployeeProc getApplyAddEmployeeProc() {
		return applyAddEmployeeProc;
	}

	public void setApplyAddEmployeeProc(AddEmployeeProc applyAddEmployeeProc) {
		this.applyAddEmployeeProc = applyAddEmployeeProc;
	}

}

Notice that we use @Handler in the bean to tell Camel which method it should pick and use.

Getting the example running

You can start the application by using the following Maven goal:

mvn camel:run

Notice: In order to run Enterprise Integration Patterns using Spring for Dependency Injection inside Maven, you can use Camel Maven Plugin. Hence, you must add following in your <build><plugins> section:

<plugin>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-maven-plugin</artifactId>
  <configuration>
    <applicationContextUri>META-INF/spring/*.xml;YOUR_FILE_NAME_IN_THE_CLASS_PATH.xml</applicationContextUri>
  </configuration>
</plugin>

After Camel has started, a new employee is added in the Employees table:

SQL> -- display data in the table
SQL> select * from employees;

EMPLOYEE_ID FIRST_NAME LAST_NAME  START_DATE      SALARY CITY
----------- ---------- ---------- ----------- ---------- ----------
         12 Celia      Rice       29/03/2012     1234,00 

SQL>

Generating XML Data from the Database

To explain how query result is converted to XML format, we will use the Employees table.

SQL> select * from Employees;

EMPLOYEE_ID FIRST_NAME LAST_NAME  START_DATE      SALARY CITY
----------- ---------- ---------- ----------- ---------- ----------
          1 Jason      Martin     29/03/2012     1234,56 Toronto
          9 Alison     Mathews    30/03/2012     6661,78 Vancouver
          5 Linda      Green      30/03/2012     4322,78 New York
         12 Celia      Rice       29/03/2012     1234,00

XStream library

XStream is a library that converts Java objects to XML format and back again from XML to Java (marshal and unmarshal). The query result will be converted to XML and saved in a file called 'target/received/report.xml'

Camel route

<!-- Specifies XStream data format -->
<dataFormats>
	<xstream id="xstream-utf8" encoding="UTF-8"/>
</dataFormats>
<!-- camel route -->
<route>
	<from uri="timer://startEndPoint?period=60000" />
	<camel:setBody>
		<camel:constant> select * from Employees </camel:constant>
	</camel:setBody>
	<camel:to uri="jdbc:dataSource" />
	<marshal ref="xstream-utf8" />
	<camel:to uri="file://target/received?fileName=report.xml" />
</route>

Generated XML file

<?xml version='1.0' encoding='UTF-8'?>
<list>
	<map>
		<entry>
			<string>START_DATE</string>
			<sql-timestamp>2012-03-29 14:36:53.0</sql-timestamp>
		</entry>
		<entry>
			<string>SALARY</string>
			<big-decimal>1234.56</big-decimal>
		</entry>
		<entry>
			<string>FIRST_NAME</string>
			<string>Jason</string>
		</entry>
		<entry>
			<string>CITY</string>
			<string>Toronto</string>
		</entry>
		<entry>
			<string>EMPLOYEE_ID</string>
			<big-decimal>1</big-decimal>
		</entry>
		<entry>
			<string>LAST_NAME</string>
			<string>Martin</string>
		</entry>
	</map>
	<map>
	...
	</map>
	...
</list>

JSON data format:

To use JSON in your camel routes, you need to add the following maven dependencies:

<!-- when using JSON data format -->
	<dependency>
		<groupId>org.apache.camel</groupId>
		<artifactId>camel-jackson</artifactId>
		<version>${camel-version}</version>
	</dependency>

	<dependency>
		<groupId>org.codehaus.jackson</groupId>
		<artifactId>jackson-mapper-asl</artifactId>
		<version>1.6.0</version>
	</dependency>

Adapting camel route as follow:

<dataFormats>
	<json id="json" library="Jackson" />
</dataFormats>
<route>
	<from uri="timer://testJSON?period=60000" />
	<camel:setBody>
		<camel:constant select * from Employees
		</camel:constant>
	</camel:setBody>
	<camel:to uri="jdbc:dataSource" />
	<camel:marshal ref="json"/>
	<camel:to uri="file://target/received?fileName=reportJson.txt" />
</route>

Here is the JSON out for the sql query:

[{"START_DATE":1333028213000,"SALARY":1234.56,"FIRST_NAME":"Jason","CITY":"Toronto","EMPLOYEE_ID":1,"LAST_NAME":"Martin"},{"START_DATE":1333122783000,"SALARY":1234,"FIRST_NAME":"Celia","CITY":null,"EMPLOYEE_ID":122,"LAST_NAME":"Rice"},{"START_DATE":1333105177000,"SALARY":6661.78,"FIRST_NAME":"Alison","CITY":"Vancouver","EMPLOYEE_ID":9,"LAST_NAME":"Mathews"},{"START_DATE":1333105188000,"SALARY":4322.78,"FIRST_NAME":"Linda","CITY":"New York","EMPLOYEE_ID":5,"LAST_NAME":"Green"},{"START_DATE":1333039175000,"SALARY":1234,"FIRST_NAME":"Celia","CITY":null,"EMPLOYEE_ID":12,"LAST_NAME":"Rice"}]

DBMS XMLGEN

DBMS XMLGEN is a PL/SQL package that allows to convert sql query result into XML data from Oracle tables.

The following example convert Employees table rows into XML and saves the XML as CLOB. Then we access to the message by using a processor, the payload conversion from clob to string are done by this processor.

Camel route

<bean id="clobProcessor" class="com.infinity.utils.ClobProcessor" />

<route id="XMLGenRoute" startupOrder="1" shutdownRoute="Defer">
			<from uri="timer://testOracleXMLGEN?period=60000" />
			<camel:setBody>
				<camel:constant>
				SELECT DBMS_XMLGEN.getXML('select * from Employees',0) FROM dual 
				</camel:constant>
			</camel:setBody>
			<camel:to uri="jdbc:dataSource" />
			<process ref="clobProcessor" />
			<log message="after execution: ${body} " />
			<camel:to uri="file://target/received?fileName=report.xml" />
</route>

Adding a Processor

Here the class which implements processor:

public class ClobProcessor implements Processor {
	public void process(Exchange exchange) throws Exception {
		char[] buffer;
		int length = 0;
		int count = 0;
		String data = "";
		String[] type;
		StringBuffer sb;
		List<Map<String, Object>> exchangeData = (List<Map<String, Object>>) exchange
				.getIn().getBody(List.class);
		Map<String, Object> row = exchangeData.get(0);
		Collection<Object> collection = row.values();
		Object[] table = collection.toArray();
		// Assign result set to CLOB variable.
		CLOB clob = (oracle.sql.CLOB) table[0];
		// Check that it is not null and read the character stream.
		if (clob != null) {
			Reader is = clob.getCharacterStream();
			sb = new StringBuffer();
			length = (int) clob.length();
			// Check CLOB is not empty.
			if (length > 0) {
				buffer = new char[length];
				count = 0;
				// Read stream and append to StringBuffer.
				try {
					while ((count = is.read(buffer)) != -1)
						sb.append(buffer);
					data = new String(sb);
				} catch (Exception e) {
				}
			} 
		}
		exchange.getIn().setBody(data);
	}
}

Result

Employees table records are now (resp. in the previous example) enclosed inside the ROW tag (resp. in map tag).

<?xml version="1.0"?>
<ROWSET>
 <ROW>
  <EMPLOYEE_ID>1</EMPLOYEE_ID>
  <FIRST_NAME>Jason</FIRST_NAME>
  <LAST_NAME>Martin</LAST_NAME>
  <START_DATE>29/03/12</START_DATE>
  <SALARY>1234,56</SALARY>
  <CITY>Toronto</CITY>
 </ROW>
 <ROW>
  <EMPLOYEE_ID>122</EMPLOYEE_ID>
  <FIRST_NAME>Celia</FIRST_NAME>
  <LAST_NAME>Rice</LAST_NAME>
  <START_DATE>30/03/12</START_DATE>
  <SALARY>1234</SALARY>
 </ROW>
 <ROW>
  <EMPLOYEE_ID>9</EMPLOYEE_ID>
  <FIRST_NAME>Alison</FIRST_NAME>
  <LAST_NAME>Mathews</LAST_NAME>
  <START_DATE>30/03/12</START_DATE>
  <SALARY>6661,78</SALARY>
  <CITY>Vancouver</CITY>
 </ROW>
...
</ROWSET>

We will use the same example used previously to transform the generated XML message into HTML format using XSLT style sheets.
Updating camel route as follow:

<route id="XMLGenRoute" startupOrder="1" shutdownRoute="Defer">
			<from uri="timer://testOracleXMLGEN?period=60000" />
			<camel:setBody>
				<camel:constant>
				SELECT DBMS_XMLGEN.getXML('select * from Employees',0) FROM dual 
				</camel:constant>
			</camel:setBody>
			<camel:to uri="jdbc:dataSource" />
			<process ref="clobProcessor" />
			<camel:to uri="xslt://mytransform.xsl" />
			<camel:to uri="file://target/received?fileName=employeeList.html" />
</route>

The result of the previous example is routed to the XSLT component, which transforms the payload using the style sheet. Then, the message is sent to file named 'employeeList.html'.
Here is the mytransform.xsl declaration:

<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
  <html>
  <body>
    <h2>Employee List</h2>
    <table border="1">
      <tr bgcolor="#9acd32">
        <th>ID</th>
        <th>First Name</th>
        <th>Last Name</th>
        <th>Start date</th>
        <th>Salary</th>
        <th>City</th>
      </tr>
      <xsl:for-each select="ROWSET/ROW">
        <tr>
          <td><xsl:value-of select="EMPLOYEE_ID"/></td>
          <td><xsl:value-of select="FIRST_NAME"/></td>
          <td><xsl:value-of select="LAST_NAME"/></td>
          <td><xsl:value-of select="START_DATE"/></td>
          <td><xsl:value-of select="SALARY"/></td>
          <td><xsl:value-of select="CITY"/></td>
        </tr>
      </xsl:for-each>
    </table>
  </body>
  </html>
</xsl:template>
</xsl:stylesheet>

Notice: Camel will look in the classpath by default for schema loction.

Result

Employee List

Intercept exception case

Exception policies are used to intercept and handle specific exceptions in particular ways. To understand how catch exceptions in camel, let's take the same example tested in the previous section and try to look how you can use onException to your advantage. You can learn more about this on the Camel website, at exception clause page.

Here, we intercept (catche) two exceptions:

java.net.ConnectException: thrown in the case of connection exceptions. The error caused this exception is recoverable. recoverable errors may be quickly resolved (connection established).
java.sql.SQLException: thrown in the case of sql query syntax exception. The error caused this exception is irrecoverable. Irrecoverable errors continue to be errors on subsequent attempts.

Catch SQLException

We trying to access a database table 'Employe' (instead of Employees) that doesn't exist, which would thrown an SQLException.
Updating route with the following onException:

<bean id="failureResponseProcessor" class="com.sungard.utils.FailureResponseProcessor" />
...

<!-- policy for when any exception occurred -->
<camel:onException>
		<camel:exception>java.sql.SQLException</camel:exception>
		<camel:redeliveryPolicy maximumRedeliveries="0" logHandled="false" />
		<camel:handled>
			<camel:constant>true</camel:constant>
		</camel:handled>
		<camel:convertBodyTo type="java.lang.String"/>
		<process ref="failureResponseProcessor" />
		<camel:log message="******** [sql exception:] ${body}" />
</camel:onException>

<route id="XMLGenRoute" startupOrder="1" shutdownRoute="Defer">
		<from uri="timer://testOracleXMLGEN?period=60000" />
		<camel:setBody>
			<camel:constant>
			SELECT DBMS_XMLGEN.getXML('select * from Employe',0) FROM dual 
			</camel:constant>
		</camel:setBody>
		<camel:to uri="jdbc:dataSource" />
		<process ref="clobProcessor" />
		<camel:to uri="xslt://mytransform.xsl" />
		<camel:to uri="file://target/received?fileName=employeeList.html" />
</route>

Using a processor to return a custom failure message, that indicates not only what the problem was but that also includes failure sql query:

public class FailureResponseProcessor implements Processor {
      public void process(Exchange exchange) throws Exception {
            String body = exchange.getIn().getBody(String.class);
            Exception e = exchange.getProperty(Exchange.EXCEPTION_CAUGHT,
                        Exception.class);
            StringBuilder sb = new StringBuilder();
            sb.append("ERROR: ");
            sb.append(e.getMessage());
            sb.append("\nSql query: ");
            sb.append(body);
            exchange.getIn().setBody(sb.toString());

      }
 }

The route creates a new HTML document having a unique name ID, this ID provided by generate method in GuidGenerator class.

public final class GuidGenerator {
      public static int generate() {
            Random ran = new Random();
            return ran.nextInt(10000000);
      }
}

This ID saved in the header of the message. updating ClobProcessor as follow:

public class ClobProcessor implements Processor {
	
	public void process(Exchange exchange) throws Exception {
	
	...

		exchange.getIn().setHeader("documentId", GuidGenerator.generate());
	}
}

At runtime you'll get the following stracktrace:

[2012-04-26 15:34:00,580] INFO  : org.apache.camel.processor.Logger.log(Logger.java:213) - ******** [sql exception:] ERROR: ORA-19206: Invalid value for query or REF CURSOR parameter
ORA-06512: at "SYS.DBMS_XMLGEN", line 121
ORA-06512: at line 1

Sql query: SELECT DBMS_XMLGEN.getXML('select * from Employe',0) FROM dual

Catch ConnectException

Let's make the example a bit more interesting and add a second onException definition.

<camel:onException>
                <camel:exception>java.net.ConnectException</camel:exception>
                <camel:redeliveryPolicy maximumRedeliveries="0" logHandled="true" />
                <camel:handled>
                      <camel:constant>true</camel:constant>
                </camel:handled>
                <camel:convertBodyTo type="java.lang.String"/>
                <camel:log message="******** The Network Adapter could not establish the connection" />
</camel:onException>

Build Processor that simulate ConnectException exception:

public class SimulateError implements Processor{

      public void process(Exchange arg0) throws Exception {
            
            throw new ConnectException("Simulated connection error");            
      }
}

Adapting route as follow:

<bean id="simulateError" class="com.sungard.utils.SimulateError" />
...

<route id="XMLGenRoute" startupOrder="1" shutdownRoute="Defer">
		<from uri="timer://testOracleXMLGEN?period=60000" />
		<process ref="simulateError"/>
		<camel:setBody>
			<camel:constant>
			SELECT DBMS_XMLGEN.getXML('select * from Employees',0) FROM dual 
			</camel:constant>
		</camel:setBody>
		<camel:to uri="jdbc:dataSource" />
		<process ref="clobProcessor" />
		<camel:to uri="xslt://mytransform.xsl" />
		<camel:to uri="file://target/received?fileName=employeeList.html" />
</route>

Camel logs:

[2012-04-26 17:43:09,047] INFO  : org.apache.camel.processor.Logger.log(Logger.java:213) - ******** The Network Adapter could not establish the connection
[2012-04-26 17:43:09,047] ERROR : org.apache.camel.processor.Logger.log(Logger.java:249) - Failed delivery for exchangeId: ID-emea-tun-ws0376-51683-1335458541624-0-2. Exhausted after delivery attempt: 4 caught: java.net.ConnectException: Simulated connection error. Processed by failure processor: Pipeline[[Channel[convertBodyTo[java.lang.String]], Channel[Log[******** The Network Adapter could not establish the connection]]]]
java.net.ConnectException: Simulated connection error
	at com.sungard.utils.SimulateError.process(SimulateError.java:12)
	at org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50)
	...