As computerisation permeated business processes many disparate systems were developed to service discrete areas of the organisation. The enclosed nature of the systems became an issue when modelling the flow of information through the entire ecosystems and a reliable way of sharing records between applications became paramount. File transfer and inter-database connectivity options became common place, but the increasing importance of underpinning modern data driven enterprise required a robust mechanism.
A study of the many facets of data integration was published in the book Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Woolf divided these into 65 patterns (EIP) describing the method of transfer between 2 end points. The patterns can be grouped into main categories such as file transfer, messaging , database connectivity, web services and these will have subdivisions according to direction and synchronicity. The Apache Camel project was created to supply Java libraries supporting each of the EIP’s and is a codification of the book. Programs can import the framework and it provides a rule-based routing & mediation engine for use either standalone or within an enterprise service bus (ESB).
Routes map flow and integration logic between different systems by connecting EIPs together and mediation handles the prerequisites to the allow connectivity. The following sample demonstrates the use of Camel in a java program.
- Line 17 The program defines a MySQL data source and registers it.
- Line 27 A Camel context is created.
- Line 29 A route added by overriding the default route builder.
- Line 34 From and to endpoints are added to the route and a bean to output the results.
- Line 41 The context is started.
- Line 43 A producer created via a template send a message containing the SQL query to the from endpoint.
The ResultHandler bean simply loops through the result data set returned by the to endpoint and prints the result to the console as shown below.
The camel dependencies used by the project are shown in the image below.
This is a trivial example to demonstrate Camel in use but by clicking on the Code tab within the Talend Studio design space for any routes, the generated code will be displayed and some of the concepts shown above will be evident within the listing.
The strength of camel lies within mediation and routing, but it is less suitable for data transformation. There are some endpoints such as XSLT transformation which can manipulate data and Processor which allows Java snippets to be added within any route but to undertake complex transformations through these methods would impinge on the clarity of the process. A better practice is to hand off the transformation logic to a system better optimised for that function. Understanding this helps to clarify why Camel is not an ESB, rather a framework that can be used to help create one. It may contain a lot of what are generally considered ESB functions but not all of them. Apache does have an ESB project built on top of Camel called ServiceMix which is full featured but is very hand-configuration heavy.
Talend Open Studio (TOS) provides a GUI for Talend development based on the Eclipse IDE. Drag and drop functionality allows components to be placed in the design workspace and create links between them. A runtime is built in which replicates either an OSGI container or the stand-alone environment depending on the task. There may be slight differences from the actual environments and programs should be tested outside of studio as well.
For their ESB offering Talend harness the power of the Camel Framework and combines it with a powerful data integration engine which can either work in collaboration with Camel or as a stand-alone product. Within TOS there 2 perspectives: mediation which creates camel based routes and integration producing data integration jobs. Earlier versions used to require manual switching of perspectives but since version 6 the change has been automatic based on the task selected. There is no obvious naming of perspectives and the visual appearance can be quite similar, even though the underlying composition is considerably different. A couple of points can instantly confirm if you are looking at a route or a job.
- Mediation component names all begin with lower case ‘c’ such as c_Timer, this is a nod to the Camel dependency. Integration component names begin with lower case ‘t’ I.e. t_LogRow.
- The connectors between the components will be labelled ‘route’ and mediation routes and ‘row’ for integration jobs.
Neither of these are fool proof as the names could be overridden by user defined labels and that is one of the reasons, I always recommend retaining the component name as part of any label. A further check can be made by looking at the active tab above the design surface which will indicate the type of process.
Mediation implements a selection of the camel components by installing the framework and all dependencies into the environment. The entire framework is supported but not all components are directly implemented. Where a suitable option can’t be found it can be created by assigning the correct endpoint URL to the cMessagingEndpoint component. Additional camel options can be found in the documentation and may be added via the Advanced Options tab. Camel uses Java Messaging Service (JMS) to transport data and the visualisation of the routes represent the encapsulation of data in a message as it passed through header values various operations. Routes can call integration jobs via the cTalendJob components allowing information in the JMS message to configure the job prior to running it by propagating message header values within the route into the job context. Return values from the job can be incorporated back into the route message.
Integration is used to transform data, and this may be passed from the route or accessed externally. The visual appearance of a job may seem like a route, but it is very different. In a job each row represents the connection between steps in the data transformation whereas the route connections show the physical flow of the information. When a job is called by a camel route it is effectively taking the role of a processor endpoint inserting java code into the route but a lot easier to create, test and maintain than hand coding a java snippet.
