Mule 4 + MuleSoft + Enterprise Integration Patterns + Examples

Migration Integration Pattern

• This pattern talks about migrating data from one system to another system.
• Migration pattern is also know as unidirectional migration pattern as it only talk about transferring data from source system to target system, not vice-versa. For example:
• In below diagram if any Sales happen in Customer Web Portal then CRM application should also know about the Sales. Hence, we need to migrate data from Customer Web Portal to CRM application.

High Level Migration Integration Steps

We can define migration in following steps

• Source system/application where data reside and need to be migrated to target system.
• Frequency at which you want to migrate out data e.g. Real time – every second or once in day or any custom frequency.
• Criteria or condition which define the eligible data in source system e.g. customer modification date is greater than yesterday.
• Enrichment and transformation as per target system.
  • Enrichment – pulling more information which is not available in source system
  • Transformation – transforming it to desired target format which is acceptable by the target system e.g. Full name instead of First name and Last name
• End result
  • A report to show the status of current migration.

Migration Pattern Example

• Batch Jobs are the widely used activities in MuleSoft for migrating data from one system to another as it provide highest throughput because of parallel execution.

How Batch Jobs can be constructed in Mule Application

Source to hold

• Scheduling logic along with calling Source System API to retrieve data
• Polling logic – if source data/event is coming directly from the source system

Enrichment and Transformation – Batch Step – Accept Policy (ALL)

• Enrichment – for a record we call other APIs to get additional details
• Transformation – To convert the source system data required by the target system

Error Handling – Batch Step – Accept Policy (ONLY_FAILURES)

• To do something at record level failure
• To do certain activity for all the failed record only once – then add aggregator activity

Reporting – On Complete

• Send the migration status details as report.

Batch Jobs With Scheduler Example

Batch Jobs With Polling Example

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:batch="http://www.mulesoft.org/schema/mule/batch"
	xmlns="http://www.mulesoft.org/schema/mule/core"
	xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/batch http://www.mulesoft.org/schema/mule/batch/current/mule-batch.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd">
	<file:config name="File_Config1" doc:name="File Config" doc:id="c954c938-773d-4317-8a32-c24c04ddd037" >
		<file:connection workingDir="D:\" />
	</file:config>
	<flow name="batch-job-processing-sample-schedular-flow" doc:id="f7693791-2b1a-44e6-946c-db03b2f6579e" >
		<scheduler doc:name="Scheduler" doc:id="a385b53e-907e-4124-8672-effa2c476611" >
			<scheduling-strategy >
				<fixed-frequency />
			</scheduling-strategy>
		</scheduler>
		<logger level="INFO" doc:name="If event Source System API call" doc:id="41906c24-561d-415d-8547-d149a4e6f1b1" />
		<flow-ref doc:name="batch-job--processing-sample-flow" doc:id="ea2720d8-f067-4c39-958a-b7407d875e43" name="batch-job-processing-sample-flow"/>
	</flow>
	<flow name="batch-job-processing-sample-polling-flow" doc:id="e3d0d63f-c5eb-4555-93e0-728e062b59d1" >
		<file:listener doc:name="On New or Updated File" doc:id="05d17c0b-f21c-4b8e-818b-72bd96de6870" config-ref="File_Config1">
			<scheduling-strategy >
				<fixed-frequency />
			</scheduling-strategy>
		</file:listener>
		<flow-ref doc:name="batch-job-processing-sample-flow" doc:id="7ab3cf18-2cae-40ad-ab20-ac3651cb1ae2" name="batch-job-processing-sample-flow" />
	</flow>
	<flow name="batch-job-processing-sample-flow" doc:id="cb383444-7c1a-48d3-a4be-077893e07c14" >
		<batch:job jobName="batch-job-processing-sampleBatch_Job" doc:id="e33f8a3e-2c7d-44ad-ac26-43bd18714878" >
			<batch:process-records >
				<batch:step name="Enrichment_Batch_Step" doc:id="a23e07c9-c17c-4404-a32d-4792af442bf6" >
					<logger level="INFO" doc:name="Enrichment Logic" doc:id="90ef724c-893d-4c56-8e75-ac3cb8522ec8" />
				</batch:step>
				<batch:step name="Transformation_Target_System_Batch_Step" doc:id="62e174d0-d518-4619-aea9-4630e49cc7dd" >
					<logger level="INFO" doc:name="Transformation Logic" doc:id="f9363864-c376-4c11-b5ce-d70a0aa28721" />
					<logger level="INFO" doc:name="Target System API call" doc:id="07711bea-b7ac-4d8b-98bd-c69fd2aefeb4" />
				</batch:step>
				<batch:step name="Error_Handling" doc:id="1ae5bcb6-970d-41a3-a30e-4d8c2a6fd647" >
					<logger level="INFO" doc:name="For Every Failed Record" doc:id="e50f3c17-d97b-4355-8cc3-a2401a50889f" />
					<batch:aggregator doc:name="Batch Aggregator" doc:id="ea6bbb15-65fb-4cc4-ad37-2dfe1fd290ed" streaming="true">
						<logger level="INFO" doc:name="For All Failed Record" doc:id="017e5405-5eb9-4a21-999d-d27b877b01be" />
					</batch:aggregator>
				</batch:step>
			</batch:process-records>
			<batch:on-complete >
				<logger level="INFO" doc:name="Send Migration Report" doc:id="7547871f-5ad8-48c7-9cb8-1839c63cbed4" />
			</batch:on-complete>
		</batch:job>
	</flow>
</mule>

MULE_HOME is set to D:\Software\AnypointStudio\plugins\org.mule.tooling.server.4.2.2.ee_7.3.5.202001031809\mule
MULE_BASE is set to D:\Software\AnypointStudio\plugins\org.mule.tooling.server.4.2.2.ee_7.3.5.202001031809\mule
--> Wrapper Started as Console
 Copyright (C) 1999-2018 Tanuki Software, Ltd. All Rights Reserved.
    http://wrapper.tanukisoftware.com
  Licensed to MuleSoft Inc. for Mule Runtime Enterprise Edition

Launching a JVM...
Starting the Mule Container...
WrapperManager: Initializing...

Running Mule in testing mode

INFO  2021-09-10 20:48:42,025 [WrapperListener_start_runner] org.mule.runtime.core.internal.logging.LogUtil: 
**********************************************************************
*                                                                    *
*                         (((((((((((((((((((                        *
*                     (((((                 ((((#                    *
*                  &(((                        *(((                  *
*                &(((                             (((                *
*               (((     (((%               (((#     (((              *
*              ((     (((((((             (((((((    (((             *
*             ((     (((((((((           (((((((((     ((            *
*            ((    #(((((((((((        #(((((((((((    (((           *
*           #((    ((((((((((((((     ((((((((((((((    ((           *
*           ((    ((((((  /(((((((   (((((((  /((((((    ((          *
*           ((    ((((((    (((((((&(((((((    ((((((    ((          *
*           ((    (((((      ((((((((((((*     ((((((    ((          *
*           ((    ((((((      *(((((((((       ((((((    ((          *
*           ((    ((((((        (((((((        ((((((   (((          *
*            ((    ((((((                     ((((((    ((           *
*            ((#    ((((((#                 (((((((    ((            *
*             ((#    ((((((((             ((((((((    (((            *
*              (((    (((((((              (((((     ((.             *
*                ((      (((               (((     #((               *
*                 /((                            (((                 *
*                    (((                      &(((                   *
*                      *((((&             &((((                      *
*                           ((((((((((((((/                          *
*                                                                    *
*    ___  ___      _        ______            _   _                  *
*    |  \/  |     | |       | ___ \          | | (_)                 *
*    | .  . |_   _| | ___   | |_/ /   _ _ __ | |_ _ _ __ ___   ___   *
*    | |\/| | | | | |/ _ \  |    / | | | '_ \| __| | '_ ` _ \ / _ \  *
*    | |  | | |_| | |  __/  | |\ \ |_| | | | | |_| | | | | | |  __/  *
*    \_|  |_/\__,_|_|\___|  \_| \_\__,_|_| |_|\__|_|_| |_| |_|\___|  *
*                                                                    *
*                                                                    *
* Mule Runtime and Integration Platform                              *
* Version: 4.2.2 Build: e790fb3a                                     *
* MuleSoft, Inc.                                                     *
* For more information go to                                         *
* https://www.mulesoft.com/platform/soa/mule-esb-enterprise          *
*                                                                    *
* Server started: 9/10/21 8:48 PM                                    *
* JDK: 1.8.0_41 (mixed mode)                                         *
* JDK properties:                                                    *
*  - java.vendor = Oracle Corporation                                *
*  - java.vm.name = OpenJDK Client VM                                *
*  - java.home = D:\Software\java-se-8u41\jre                        *
* OS: Windows 8.1 -  (6.3, x86)                                      *
INFO  2021-09-10 20:48:54,549 [WrapperListener_start_runner] org.mule.runtime.core.internal.logging.LogUtil: 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Mule is up and kicking (every 5000ms)                                        +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
INFO  2021-09-10 20:48:54,597 [WrapperListener_start_runner] org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@1e2e45b{HTTP/1.1,[http/1.1]}{0.0.0.0:64281}
INFO  2021-09-10 20:48:54,600 [WrapperListener_start_runner] org.mule.runtime.core.internal.logging.LogUtil: 
**********************************************************************
*              - - + DOMAIN + - -               * - - + STATUS + - - *
**********************************************************************
* default                                       * DEPLOYED           *
**********************************************************************

Broadcast Integration Pattern

• Broadcast pattern is moving data from one source system to many target systems. E.g.
• In below diagram if any Sales happen in Customer Web Portal then CRM application should also know about the Sales, Inventory and Analytics. Hence, we need to broadcast data from Customer Web Portal to multiple target systems.

Target systems don’t have to send any response back to the Source system (Customer Web Portal).

Broadcast Integration Pattern Usecases

We can define Broadcast pattern in following steps

• Source system/application where data change result into a notification/event – Case 1 and 2 and Case 3 when integration go and check the Source system for change data.
  • Source system can send the notification including actual data – Case 1
  • Source system can send only the notification without actual data – Case 2
  • Integration layer will check the Source system for any change data – Case 3
• Integration layer
  • Case 1 – Integration will receive the notification along with actual data
  • Case 2 – Integration will receive the notification and pull the actual data from the Source system
  • Case 3 – After certain scheduled interval Integration will check the source system for change data
• Target systems will receive the source system data in their desired format.

Note: Every case has its pros and cons, let’s see

Case 1 – Source system send notification along with data

• Pros
  • Reduce API call between MuleSoft and Source system
  • MuleSoft don’t have to check for the change data
  • If MuleSoft is down then data can be lost
    • To overcome this problem Async approach is taken like introducing JMS queue between MuleSoft and Source system
• Cons
  • Notification and data creation logic lies in Source system
  • MuleSoft will not know in case Source system is down or any issue in Source system

Case 2 – Source system send only notification and MuleSoft will pull the actual data from the source system

• Pros
  • Light weight notification
  • Segregation of logic for pulling data from source system
• Cons
  • Notification creation logic lies in Source system
  • MuleSoft will not know in case Source system is down or any issue in Source system

Case 3 – Scheduler will invoke the MuleSoft integration after certain time interval and MuleSoft will pull the actual change data from source system

• Pros
  • Reliable approach as MuleSoft will know if Source system is down
  • Pull logic is within MuleSoft
  • Better approach in case changes are high as in one go it will pull all the change data
• Cons
  • More calls even in case there is no change of data in Source system

Bi-Directional Sync Integration Pattern

• Bi-Directional Sync Integration is a pattern where data flow from Source system to Target system and later from Target system to Source system. Idea is to make two system works together to achieve a business process or function. E.g.

1. Above example shows a Customer Portal which collects the Test Request. Once Customer Portal has all the required information then it can sends the details to Lab system.
2. Lab system take it’s time to conduct different test and later send the Test Results back to the Customer Portal to share it with the customer. Here, idea is to make these two application work together to provide the Lab Test functionality to customers.

Correlation Integration Pattern

• Correlation Integration is a pattern which is similar to bi-directional sync but this sync happen only whenever there is a requirement of having similar data set in another system. E.g.

• A large bank having different branches across cities. If any Customer A is register to Branch X then there is a no need to sync this Customer A information to other branches (if required then this can be done using bi-directional sync).

• Now, if same Customer A moves to different city and go to Branch Y to open his account then through Correlation pattern we can get the actual Customer A information from Branch X and sync it with Branch Y

• Correlation Integration pattern is more about intersection of data sets which means data which is common between two system (Customer A data available in both Branch X and Y) and happen naturally (Customer A move to different city)
• Example, finding a Customer on SSN will make sense over first name and last name or include both SSN and name (Correlation). if we only go with first name and last name there are chances we might end up creating wrong data set in another system. Hence, correlation sync requried in this case.

Aggregation Integration Pattern

It aggregates the data from different systems and merge them before processing the response.

Here, response can be inserted e.g. Reporting or as Response e.g. HTTP response in Database from different Source Systems to a Target system.

Reliability Integration Pattern

• Suppose, we are creating an API to receive payment and once we receive the payment information, if anything goes wrong then you are losing the payment and end up in losing customer experience also right?

To make this reliable we can introduce a connector (which support transaction) to divide the transactional and non-transactional activities as below.

Reliability Integration Pattern Example

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:vm="http://www.mulesoft.org/schema/mule/vm" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns="http://www.mulesoft.org/schema/mule/core"
	xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/vm http://www.mulesoft.org/schema/mule/vm/current/mule-vm.xsd">
	<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="7d4397c7-ad40-41d2-bb54-6ec0d3466baa">
		<http:listener-connection host="0.0.0.0" port="8081" />
	</http:listener-config>
	<vm:config name="VM_Config" doc:name="VM Config" doc:id="5db475fe-978a-4b58-9325-9e2bb2c424fb">
		<vm:connection />
		<vm:queues>
			<vm:queue queueName="reliableVM" />
		</vm:queues>
	</vm:config>
	<flow name="reliability-pattern-services-non-transactional-flow" doc:id="1220980b-c57e-4fa7-8f6c-10ace31885d0">
		<http:listener doc:name="Listener" doc:id="9e9d2e91-1f07-45a0-ac68-2abecff0ff98" config-ref="HTTP_Listener_config" path="reliable" />
		<vm:publish doc:name="Publish" doc:id="10cecc00-4bc2-4351-942f-c16d4bd87a98" config-ref="VM_Config" queueName="reliableVM" />
	</flow>
	<flow name="reliability-pattern-services-transactional-flow" doc:id="b8a07c5a-caa6-4c13-83ff-bd1dd670e4b0">
		<vm:listener doc:name="Listener" doc:id="f3fdfde1-f1c5-43b9-aeef-da9fed8cb854" config-ref="VM_Config" queueName="reliableVM" />
		<logger level="INFO" doc:name="DB Insert - 1" doc:id="9e325dc1-581d-48c0-9906-615010953373" />
		<logger level="INFO" doc:name="DB Insert - 2" doc:id="675fe923-8ba6-4f41-86ef-aa154bd87ba7" />
	</flow>
</mule>

INFO  2021-09-10 22:11:01,521 [WrapperListener_start_runner] org.mule.runtime.core.internal.logging.LogUtil: 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Mule is up and kicking (every 5000ms)                                        +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
INFO  2021-09-10 22:11:01,559 [WrapperListener_start_runner] org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@11e88b1{HTTP/1.1,[http/1.1]}{0.0.0.0:61097}
INFO  2021-09-10 22:11:01,561 [WrapperListener_start_runner] org.mule.runtime.core.internal.logging.LogUtil: 
**********************************************************************
*              - - + DOMAIN + - -               * - - + STATUS + - - *
**********************************************************************
* default                                       * DEPLOYED           *
**********************************************************************

JMS Messaging Integration Pattern

• This Integration Pattern is used for asynchronous communication between the Source and Target systems and the communication happens through messaging only.
• Systems are loosely coupled and they really don’t aware of each other. Source system can send the message to JMS without having knowledge of Target system.
• It is widely used Integration Pattern for asynchronous communication.

Advantages of JMS Integration Pattern

• Loosely coupled
• Asynchronous – Receiver will get the message as soon as it arrives on the JMS. Receiver don’t have to check the JMS on regular intervals.
• Reliable – JMS ensure/guarantee the delivery of messages and that to only once.

JMS Pattern as Point To Point Communication

Here, we can have one or multiple senders and only one receiver.

Key points to remember

• Point-to-Point JMS uses Queue for communication between Senders and Receiver.
• JMS sender/senders can send the message to queue without having knowledge of receiver
• JMS receiver will receive the message as and when it arrive on JMS queue
• There is no time dependency for the message to be consumed and can reside on the queue for long as per the Queue configuration
• JMS queue will store the message till the time it’s not consumed by the receiver. In case the JMS application restart then also message will not lost

JMS Pattern as Pub-Sub Communication

It uses JMS Topic for communication between Senders and Receivers. Here, we can have multiple publishers and subscribers. Copy of message is send to all the subscribers.

Key points to remember

• Pub/Sub uses JMS Topic instead of Queue for communication.
• Publishers can send the message to Topic and copy of message will be received by all the subscriber
• There is a time associated with the message where all subscriber should get the message before the time expired otherwise the message is not sent to the subscriber after time delay
• Topic doesn’t store the message for long

Keep Learning. Have, a great day :). In case of any query please do post.