Overview :
The retail proof of concept (POC) is a system designed for a hypothetical company demonstrating the usage of Talend DI and ESB products to integrate data. The client for this scenario is a supermarket with a number of branches returning sales data to a central point. The data should be recorded and disseminated to a number of sub systems and applications including; stock control, finance, marketing and analytics. Flow of information should not normally require any human intervention. Although the example specifies a retail business the concepts of data flow would be equally applicable to other areas: market research, sensor readings, vehicle usage; anything where discrete data is collected by a central point, collated and disseminated to other locations.
The Database
The main RDBMS for this project is MySQL version 8 in this case the standalone community version is used with development in the MySQL workbench. There are subscription versions of the software including full support and it is also available on cloud platforms such as AWS via RDS for managed support or BYOL on an EC2 instance or other providers including Microsoft Azure or GCP.
More information about the database structure can be found here.
Scripts to create and populate the POC databases are available on GitHub.
MongoDB database does not require a script, simply create a database called logging and within it, a collection saleslog.
The SQL Server RetailPOC database can be created using this script.
Talend Development
A podcast Talend Rocks describing and demonstrating the proof of concept can be found via this link.
The PowerPoint presentation together with the scene script used in the video are available for download.
Source code for the Talend project is available here.
Datasource files for adding to Karaf runtime container can also be downloaded.
Documentation for the Talend processes can be accessed via Github
Routes and jobs to control and transform the data. Further information about the difference between these types of procedure and how they build, can be found here.
Enterprise integration patterns (EIP) used include; database access both table and stored procedure, file transfer using csv and JSON formats, JMS messaging and web services, both SOAP and REST.
The completed routes and jobs are built in studio prior to being run standalone via a batch file or deployed into the OSGI container in Karaf. Information about the runtime can be found in this article.
