Filter complex array using dw2-Mule4

Hello friends,

In this article I will show you how we can filter results from complex Arrays. In this use-case we will use select and descendent operators from Mulesoft dataweave.

In this use-case we will try to find out the Book details which is at-least rented once from Library. We will see some variations to dataweave expressions in below section.

Input Payload:

[
  {
    "bookId": "20200114",
    "bookType": "Fiction",
    "title": {
      "en": "Candide"
    },
    "message": {
      "en": ""
    },
    "bookDetails": [
      {
        "label": {
          "en": "Candide"
        },
        "Library": {
          "city": "Pune",
          "rented": {
            "count": "1"
          }
        }
      }
    ]
  },
  {
    "bookId": "20200115",
    "bookType": "Fiction",
    "title": {
      "en": "The Alchemist"
    },
    "message": {
      "en": ""
    },
    "bookDetails": [
      {
        "label": {
          "en": "The Alchemist"
        },
        "Library": {
          "city": "Kolkata",
          "rented": {
            "count": "0"
          }
        }
      }
    ]
  }
]

DW Script:

%dw 2.0
output application/json
---
payload[?($.bookDetails.Library.rented.count[0] >= "1")] default []

Output:

[
  {
    "bookId": "20200114",
    "bookType": "Fiction",
    "title": {
      "en": "Candide"
    },
    "message": {
      "en": ""
    },
    "bookDetails": [
      {
        "label": {
          "en": "Candide"
        },
        "Library": {
          "city": "Pune",
          "rented": {
            "count": "1"
          }
        }
      }
    ]
  }
]

Now let us understand the dataweave script in some details:

DW-Script

We can add multiple conditions inside red brackets based on our requirement.

For example if we want to add filter books based on Library location. We can use

expression like below:

%dw 2.0
output application/json
---
payload[?(($.bookDetails.Library.rented.count[0] >= "1") and ($.bookDetails.Library.city[0] == "Pune"))] default []

%dw 2.0
output application/json
---
payload[?(($.bookDetails.Library.rented.count[0] >= "1") and ($.bookDetails.Library.city contains "Pune"))] default []

Here, important thing to understand about [0] inside red-brackets. Actually we either need

to use [0], because as we have already said in diagram that it iterates and evaluates each

record and give output result or contains operator to evaluate the condition. Also, when

we are having any count check and compare like "> or <" contains operator will not work.

When we are using selectors for filtering data with one more variation in expression.

Let's see some more variation where we can use descendent operator to evaluate

the results.

%dw 2.0
output application/json
---
payload[?(($..count[0] >= "1") and ($..city[0] == "Pune"))] default []

%dw 2.0
output application/json
---
payload[?(($..count[0] >= "1") and ($..city contains "Pune"))] default []

Results will remain same with expression used with descendent and contains operator.

Happy Learning:)

Read More »

How to lookup CSV table using Dataweave2

 Hello Friends, 

In this article we will see how to lookup CSV file data without converting it to xml/json or any other message format.

Let's take one example where we need to lookup data from CSV file and retrieve/filter results from the dataset. Here, we will take one very common dataset with employee details information and then we will filter the data with some set of desired condition.

CSV table Input

Name	Location	Pincode
Niral	Pune	910002
Nikku	Pune	910005
Shruthi	Bhopal	843001
Manpreet	Chandigarh	890021
Little John	Mumbai	200011
Harry	Delhi	100011
Tom	Goa	500110

DW Script:

%dw 2.0
output application/json
---
(payload filter ((item, index) -> item.Location == "Pune"))

Output:

[
  {
    "Name": "Niral",
    "Location": "Pune",
    "Pincode": "910002"
  },
  {
    "Name": "Nikku",
    "Location": "Pune",
    "Pincode": "910005"
  }
]

In above example we are filtering data based on location. Here, we have not converted csv data in json before using filter. You can also store the results in dataweave vars and use those results based on your use-cases. 

Please find below some variations having same output as results:

%dw 2.0
output application/json
---
payload filter ($.Location == "Pune")

You can have multiple conditions also. Below expression, we are retrieving data from csv file inside project. You can use readUrl to load csv data from classpath and then use filter  based on Location and Pincode.

 

%dw 2.0
output application/json
var fiterResult= readUrl("classpath://csvFiles/ProfileDetails.csv","application/csv") filter ($.Location == "Pune" and $.Pincode == "910005")
---
fiterResult

Output:

[
  {
    "Name": "Nikku",
    "Location": "Pune",
    "Pincode": "910005"
  }
]

Happy Learning :)

Read More »

Validation module indepth - Mule4

 Introduction:

Mulesoft has provided a bunch of validation components which is very useful for message field validation during developments. These validations can be general or based on specific requirements. Validators are used to verify that parts of the Mule message/events to meet some kind of criteria which we specify.

If a message doesn't meet the validation criteria, validation component generates Validation error. Here, we can customize the exception message that is displayed in logs/responses.

Let's see some of important and frequently used validation components

mule4-validation-module

Is not null:

This components checks if the field value should be not null or else throws' error with "Name is Mandatory field"

<validation:is-not-null value="#[payload.name]" message="Name is Mandatory field"/>

Is email:

This components checks if the incoming email field is in valid format or not else throws validation error with message "Invalid Email format".

<validation:is-email email="#[payload.email]" message="Invalid Email format"/>

ALL vs ANY

To perform multiple validation at a time in mule flow. Mulesoft provides two scopes ALL and ANY which is used for these purpose.

ALL:- All validation should be passed from a set validation components defined in ALL scope.

<validation:all>
      <validation:is-not-null value="#[payload.name]" message="Name is Mandatory field"/>
      <validation:validate-size value="#[payload.name]" min="5" max="20" message="Name field size should be between 5 and 20"/>
</validation:all>

Here we are checking name field should not empty and name length should be minimum length 5 and max 20. It is similar to AND conditions like any programming language.

ANY:- Any of the validation components should be passed from a set of Validation components defined in ANY scope

<validation:any>
  <validation:is-not-null value="#[payload.name]" message="Name is Mandatory field"/>
  <validation:validate-size value="#[payload.name]" min="5" max="20" message="Name field size should be between 5 and 20"/>
<validation:any>

With Any scope we are ensuring any of the above condition should pass, it will pass the the validation. It is similar to OR conditions like any programming language.

Based on project requirement we can used these scopes with multiple validation components at a time.

Use of validation module Dataweave:

Now let us see how we can use these mule validation components can be used as Mule expression to make decisions without throwing any validation errors.

<choice>
   <when expression="#[ Validation::is-not-null(payload.name) and Validation::is-not-null(payload.name)]">
	<set-payload value='#[payload.name ++ " is a valid name."]'/>
   </when>
   <otherwise>
        <set-payload value='#[payload.name ++ " is NOT a valid name."]'/>
   </otherwise>
</choice>

Happy learning :)

Read More »

How to pass payload using Get method-Mule4

Hello Friends, 

I recently worked for one of API implementation where I needed to encrypt and send/receive one or more field in queryParms/headers while calling Get operation to external system.

As per API standard Get method should not pass payload as body while calling any Apis or 3rd party systems. Get method supports Uri Params, Query Params or Header Attributes.

In this article, I will show you the technique how you can pass encrypted payload using Http Get method. This is one of tricky interview question which normally asked in Mulesoft or Api Design related interviews.

As we know the when we encrypt any value it will not be a single string. Encrypted value may be of be multiple line as given below for PGP encryption for value 123456

-----BEGIN PGP MESSAGE-----

Version: BCPG v1.58

hIwDmCS94uDDx9kBA/9NIO0Kq9+yhXBLQveoykVjbECqXnGM8DargSsBxE2xBLQP

000W7mre5YpeSRJi4dvjVkAKPaxIFJhHlUNv38i5DkGD/z53SKtbKfYdfT+3oHT4

SVnIkcqWM+i+xawrKSk8Lz+j5GaJZ+wQrrxi8TP9RVW+wZlcbBNfxZVShDYp8dJX

AX9/x+1dFyO0EtHBipB0MLOqrb+sua028HGcg7E/sptn89JAgUcNEXOurVSfN6Ws

P/KxUMQY9+KKQQKavJ6LC92a6GJhVk4S+cnY1sxDLGtqKcGd+qCd

\u003dwGi2

-----END PGP MESSAGE-----

For both scenario(send/receive) we will be using (dw::core::Binaries) module from dataweave

toBase64(Binary): String

It takes Binary as input and returns result as base64 encoded String.

base64 Encoding

fromBase64(String): Binary

It takes base64 encoded String and returns original Binary data as a result.

base64 decoding

In this article we have seen how to convert encrypted payload to base64 encoded string and vice-versa. We can do the same conversion and pass with other payload type like JSON/XML etc..
These two dataweave functions are quite useful while dealing with payload with GET method.

Happy Learning :)

Read More »

splitBy and joinBy function dataweave2

 Hello friends, 

In this article I want show you the uses of two very common dataweave functions splitBy and joinBy. 

These two functions are very useful while doing transformation with Array and Strings. 

Here, we will see how we can use these dataweave functions converting data from String to Array and vice versa.

splitBy

Splits a string into a string array based on a value of input or matches part of that input string. It filters out the matching part from the returned array.

A string or a Java regular expression used to split the string. If it does not match some part of the string, the function will return the original, unsplit string in the array.

splitBy(String, Regex): Array<String>

Example 1:
------------------------------
Input
Hello World
------------------------------
Dataweave 2.0 Expression
------------------------------
%dw 2.0
output application/java
---
"Hello World" splitBy(/\s/)
------------------------------
Output
[Hello, World]

splitBy(String, String): Array<String>

Example 2:
------------------------------
Input
Manish Kumar
------------------------------
Dataweave 2.0 Expression
------------------------------
%dw 2.0
output application/java
---
payload splitBy(" ")
------------------------------
Output
[Manish, Kumar]

The regex/separator can match any character in the input. splitBy performs the opposite operation of joinBy.

joinBy

joinBy function Merges an array into a single string value and uses the provided string as a separator between each item in the list. 

You can use joinBy Dataweave 2.0 function to convert array to string.

Example 3:
------------------------------
Dataweave 2.0 Expression
------------------------------
%dw 2.0
output application/json
---
{ "joinName" : ["Manish","Kumar"] joinBy "-" }
------------------------------
Output
{
  "joinName": "Manish-Kumar"
}

joinBy performs the opposite task of splitBy.

Happy Learning :)

Read More »

Array comparison use case Dataweave2

Hello friends,

In this article we will see one of the common and challenging real-time use-case in IT industry during application development where we require to compare between arrays. These arrays could be from two different source or api responses.

Here we will see how we can compare two arrays results/payloads and get required results. Also we will see the complete example of our use-case.

To achieve this we will see how we can use the combination of some dataweave functions and how we can utilise these functions for our use-case.

• flatMap
• filter
• contains
• partation

There are many ways to achieve these functionality. Here, I have tried to show you two different methods to achieve this.

Method I:

flatMap

flatMap(Array<T>, (item: T, index: Number) -> Array<R>): Array<R>

It comes under dw::Core package from dataweave library.

Iterates over each item in an array and flattens the results. It produces output as Array. 

Instead of returning an array of arrays (as map does when you iterate over the values within an input like [ [1,2], [3,4] ]), flatMap returns a flattened array that looks like this: [1, 2, 3, 4]. flatMap is similar to flatten, but flatten only acts on the values of the arrays, while flatMap can act on values and indices of items in the array.

%dw 2.0

output application/json indent=false

---

[ ["a":1,"b":0], ["c":0.1,"d":7] ] flatMap (value, index) -> value

-------------------------------------------------------------------

Output:

[{"a": 1},{"b": 0},{"c": 0.1},{"d": 7}]

filter

filter(Array<T>, (item: T, index: Number) -> Boolean): Array<T>

It comes under dw::Core package from dataweave library. It produces output as array.

Iterates over an array and applies an expression that returns matching values. The expression must return true or false. If the expression returns true for a value or index in the array, the value gets captured in the output array. If it returns false for a value or index in the array, that item gets filtered out of the output. If there are no matches, the output array will be empty.

%dw 2.0

output application/json

---

[9,2,3,4,5] filter (value, index) -> (value > 4)

-------------------------------------------------

Output:

[

9,

5

]

Complete example below by using flatMap and filter function:

%dw 2.0

output application/json

var filterCriteria = [

{

"IDENTITY": "D40000",

"NM": "Delta",

"CODE": "D12"

},

{

"IDENTITY": "C30000",

"NM": "Charlie",

"CODE": "C11"

}

]

var responsePayload= [

{

"CODE": "A11",

"NAME": "Alpha",

"ID": "C10000"

},

{

"CODE": "B12",

"NAME": "Bravo",

"ID": "B20000"

},

{

"CODE": "C11",

"NAME": "Charlie",

"ID": "C30000"

},

{

"CODE": "D12",

"NAME": "Delta",

"ID": "D40000"

},

{

"CODE": "E12",

"NAME": "Echo",

"ID": "E50000"

}

]

---

filterCriteria flatMap(v) -> (

responsePayload filter (v.IDENTITY == $.ID and v.NM == $.NAME and v.CODE == $.CODE)

)

---------------------------------------------------------------------------------------------------------------

Output:

[

{

"CODE": "D12",

"NAME": "Delta",

"ID": "D40000"

},

{

"CODE": "C11",

"NAME": "Charlie",

"ID": "C30000"

}

]

----------------------------------------------------------------------------------------------------------------------

Method II:

Now, let use see the 2nd way to compare array by using contains and partition function. Here the input arrays can be from two different source or api responses. Let's see how we can compare and filter the incoming arrays by using these two useful function.

contains

contains(Array<T>, Any): Boolean

It comes under dw::Core package from dataweave library. It returns true if an input contains a given value, false if not.

This version of contains accepts an array as input. Other versions accept a string and can use another string or regular expression to determine whether there is a match.

partation

partition(Array<T>, (item: T) -> Boolean): { success: Array<T>, failure: Array<T> }


It comes under dw::core::Arrays package from dataweave library.

It separates the array into the elements that satisfy the condition from those that do not.
Elements that satisfies conditions will come under "success" array partation and which it don't satisfy comes "failure" array partation. So, you can retrieve the matching results under "success" partation node.


Note: Introduced in DataWeave 2.2.0. Supported by Mule 4.2 and later.

%dw 2.0

import partition from dw::core::Arrays

output application/json

var responsePayload = [

{

"CODE": "A11",

"NAME": "Alpha",

"ID": "C10000"

},

{

"CODE": "B12",

"NAME": "Bravo",

"ID": "B20000"

},

{

"CODE": "C11",

"NAME": "Charlie",

"ID": "C30000"

},

{

"CODE": "D12",

"NAME": "Delta",

"ID": "D40000"

},

{

"CODE": "E12",

"NAME": "Echo",

"ID": "E50000"

}

]

var filterCriteria = [

{

"IDENTITY": "D40000",

"NM": "Delta",

"CODE": "D12"

},

{

"IDENTITY": "C30000",

"NM": "Charlie",

"CODE": "C11"

}

]

---

responsePayload partition (e) -> filterCriteria contains {IDENTITY:e.ID,NM:e.NAME,CODE:e.CODE}

--------------------------------------------------------------------------------------------------------------------------

Output

{

"success": [

{

"CODE": "C11",

"NAME": "Charlie",

"ID": "C30000"

},

{

"CODE": "D12",

"NAME": "Delta",

"ID": "D40000"

}

],

"failure": [

{

"CODE": "A11",

"NAME": "Alpha",

"ID": "C10000"

},

{

"CODE": "B12",

"NAME": "Bravo",

"ID": "B20000"

},

{

"CODE": "E12",

"NAME": "Echo",

"ID": "E50000"

}

]

}

You can choose any of these methods while doing comparison of arrays for your use-cases.

Happy Learning :)

Read More »

Dataweave useful String functions-Part1

Hello friends, In this article we will see some common dataweave String functions from String core package(dw::core::Strings)  along with their use-cases.

To use these functions in dataweave scripts we need to add/import the module header as below.

you can either import complete package by using * or if you want to use specific function you can use just function name in place of placeholder as given below <functionName>

%dw 2.0

import * from dw::core::Strings or import <functionName> from dw::core::Strings 

output application/json

---

{}

Here, we will see 4 common String functions and their usage.

• capitalize(String): String
• camelize(String): String
• dasherize(String): String
• underscore(String): String 

capitalize(String): String

This function accepts input as String and returns results as String. It capitalizes the first letter of each word in a string. It also removes underscores between words and puts a space before each capitalized word.

%dw 2.0

import capitalize from dw::core::Strings

output application/json

---

{

"a" : capitalize("manish"),

"b" : capitalize("mr_manish_kumar"),

"c" : capitalize("manish-KUMAR"),

"d" : capitalize("manishKumar")

}

Output:

{

"a": "Manish",

"b": "Mr Manish Kumar",

"c": "Manish Kumar",

"d": "Manish Kumar"

}

camelize(String): String

This function accepts input as String and returns results as String.  It returns a string in camel case based on underscores in the string. All underscores are deleted, including any underscores at the beginning of the string.

%dw 2.0

import camelize from dw::core::Strings

output application/json

---

{

"a" : camelize("1_manish"),

"b" : camelize("mr_manish_kumar"),

"c" : camelize("_manish_kumar")

}

Output:

{

"a": "1Manish",

"b": "mrManishKumar",

"c": "manishKumar"

}

dasherize(String): String

This function accepts input as String and returns results as String. It replaces spaces, underscores, and camel-casing in a string with dashes (hyphens). If no spaces, underscores, and camel-casing are present, the output will match the input. 

%dw 2.0

import dasherize from dw::core::Strings

output application/json

---

{

"a" : dasherize("Manish"),

"b" : dasherize("Manish_first_name"),

"c" : dasherize("Manish Kumar"),

"d" : dasherize("ManishKumar")

}

Output:

{ "a": "manish", "b": "manish-first-name", "c": "manish-kumar", "d": "manish-kumar" }

Above function works based String input containing underscore. Below underscore function will help you to revert String  to underscore format. 

underscore(String): String 

This function accepts input as String and returns results as String. It replaces hyphens, spaces, and camel-casing in a string with underscores. If no hyphens, spaces, and camel-casing are present, the output will match the input. Also it converts string into lower case.

%dw 2.0

import underscore from dw::core::Strings

output json

---

{

"a" : underscore("Manish"),

"b" : underscore("Manish-first-name"),

"c" : underscore("Manish KUMAR"),

"d" : underscore("ManishKumar")

}

Output:

{ "a": "manish", "b": "manish_first_name", "c": "manish_kumar", "d": "manish_kumar" }

Also, If you want to use multiple functions in your transformation

you can import with either * or comma separated as example below:

%dw 2.0

import dasherize,underscore,capitalize,camelize from dw::core::Strings

output json

---

{

"a" : capitalize("manish_kumar"),

"b" : underscore("Mr-Manish-Kumar"),

"c" : dasherize("Manish KUMAR"),

"d" : camelize("manish_kumar")

}

Output:

{ "a": "Manish Kumar", "b": "mr_manish_kumar", "c": "manish-kumar", "d": "manishKumar" }

Note:: dasherize, underscore will convert result strings into lower case.

Happy Learning :)

Read More »

Optimize Json data using DataWeave

In this Article, we will see how we can optimize and allow Json data to accept and transform duplicate elements. In real-time projects optimizing data really matters when we talk about tuning performance. Either you say in terms of  logging data or in terms of processing data every byte matters.

In DataWeave, we have two properties which will hep us to optimize our application data.

• duplicateKeyAsArray: JSON data does not allow duplicate keys within same parent, so we may get exception in real-time project development scenario. With this attribute, if your dataweave finds any duplicate keys, it will create an array with all those values.

Sample XML Input:

<users>
 <user>
  <personal_information>
   <first_name version="V1">Manish</first_name>
   <first_name version="V1">Mak</first_name>
   <middle_name>Kumar</middle_name>
   <last_name>Sah</last_name>
  </personal_information>
 </user>
</users>

Dataweave Script:

%dw 2.0
output application/json duplicateKeyAsArray=true
---
payload

Sample Output:

{
  "users": {
    "user": {
      "personal_information": {
        "first_name": [
        "Manish",
        "MaK"
        ],
        "middle_name": "Kumar",
        "last_name": "Sah"
      }
    }
  }
}

• indent: It indicates whether you want to compress your JSON data into a single line or you need JSON data for better readability.

Sample Input:
<users>
 <user>
  <personal_information>
   <first_name version="V1">Manish</first_name>
   <first_name version="V1">Mak</first_name>
   <middle_name>Kumar</middle_name>
   <last_name>Sah</last_name>
  </personal_information>
 </user>
</users>

Dataweave Script:

%dw 2.0
output application/json duplicateKeyAsArray=true, indent=false
---
payload

Sample Output:

{"users": {"user": {"personal_information": {"first_name": ["Manish","MaK"],"middle_name": "Kumar","last_name": "Sah"}}}}


Happy Learning :)

Read More »