Combine two queries using WSO2 ESB - xpath

Question

I've been trying to figure out how to get WSO2's ESB to make calls to two different APIs and combine their results into a single response, and running into nothing but trouble. At its most basic, I've got two backends I'm making requests to that respond something like this:
http://example.com/items:
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<response xmlns="http://example.com/response">
<query name="items" xmlns="http://example.com/query">
<row>
<id>1</id>
<name>Item 1</name>
</row>
<row>
<id>2</id>
<name>Item 2</name>
</row>
</query>
</response>
</soapenv:Body>
</soapenv:Envelope>
http://example.com/parts:
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<response xmlns="http://example.com/response">
<query name="parts" xmlns="http://example.com/query">
<row>
<id>1</id>
<part>Part 1.1</part>
</row>
<row>
<id>1</id>
<part>Part 1.2</part>
</row>
<row>
<id>1</id>
<part>Part 1.3</part>
</row>
<row>
<id>2</id>
<part>Part 2.1</part>
</row>
<row>
<id>2</id>
<part>Part 2.2</part>
</row>
</query>
</response>
</soapenv:Body>
</soapenv:Envelope>
I'd like to request both of those, then combine their results to look something like this:
<items>
<item>
<id>1</id>
<name>Item 1</name>
<parts>
<part>
<id>1</id>
<name>Part 1.1</name>
</part>
<part>
<id>1</id>
<name>Part 1.2</name>
</part>
<part>
<id>1</id>
<name>Part 1.3</name>
</part>
</parts>
</item>
<item>
<id>2</id>
<name>Item 2</name>
<parts>
<part>
<id>2</id>
<name>Part 2.1</name>
</part>
<part>
<id>2</id>
<name>Part 2.2</name>
</part>
</parts>
</item>
</items>
Basically, every response from both APIs has a list of rows, each of which contains an id element. The ids in the call to /items are unique within that response, and each row in the response from parts has an id that ties it to a row from /items.
I've got the following API definition in the ESB:
<?xml version="1.0" encoding="UTF-8"?>
<api context="/item_list" name="ItemList" xmlns="http://ws.apache.org/ns/synapse">
<resource methods="POST" uri-template="/">
<inSequence>
<header name="Content-Type" scope="transport" value="text/xml; charset=utf-8"/>
<clone>
<target>
<sequence>
<send>
<endpoint>
<address format="soap11" uri="http://example.com/items"/>
</endpoint>
</send>
</sequence>
</target>
<target>
<sequence>
<send>
<endpoint>
<address format="soap11" uri="http://example.com/parts"/>
</endpoint>
</send>
</sequence>
</target>
</clone>
</inSequence>
<outSequence>
<aggregate>
<correlateOn expression="//*[name()='response']/*[name()='query']/*[name()='row']/*[name()='id']" />
<completeCondition>
<messageCount max="2" min="2"/>
</completeCondition>
<onComplete expression="//*[name()='response']/*[name()='query']/*[name()='row']">
<send/>
</onComplete>
</aggregate>
</outSequence>
<faultSequence/>
</resource>
</api>
The inSequence here is heavily simplified, but it does send two valid queries and gets back the expected responses. The outSequence as it's written here never sends a response to the client or logs an error on the server. If I remove the correlateOn element from aggregate, I get back a single row, seemingly at random, from one of the two API calls. I think correlateOn is something I want to be using here, but I can't find any useful documentation on it from either WSO2 or Apache, so I'm sure I'm using it incorrectly. My XPath background is pretty weak, so I'm sure that expression could also use some work.
Am I at least on the right track here with the clone/aggregate pattern? How would I go about combining the results from these two queries into something similar to my example? If I can get something even sort of close, I should be able to do the rest with XSLT.

Answer 1

take a look at this demo:
Backend 1 with the items response:
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="items"
transports="https http"
startOnLoad="true">
<target>
<inSequence>
<payloadFactory media-type="xml">
<format>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<response xmlns="http://example.com/response">
<query xmlns="http://example.com/query" name="items">
<row>
<id>1</id>
<name>Item 1</name>
</row>
<row>
<id>2</id>
<name>Item 2</name>
</row>
</query>
</response>
</soapenv:Body>
</soapenv:Envelope>
</format>
<args/>
</payloadFactory>
<log level="full"/>
<loopback/>
</inSequence>
<outSequence>
<send/>
</outSequence>
<faultSequence/>
</target>
</proxy>
Backend 2 with the parts response:
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="parts"
transports="https http"
startOnLoad="true">
<target>
<inSequence>
<payloadFactory media-type="xml">
<format>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<response xmlns="http://example.com/response">
<query xmlns="http://example.com/query" name="parts">
<row>
<id>1</id>
<part>Part 1.1</part>
</row>
<row>
<id>1</id>
<part>Part 1.2</part>
</row>
<row>
<id>1</id>
<part>Part 1.3</part>
</row>
<row>
<id>2</id>
<part>Part 2.1</part>
</row>
<row>
<id>2</id>
<part>Part 2.2</part>
</row>
</query>
</response>
</soapenv:Body>
</soapenv:Envelope>
</format>
<args/>
</payloadFactory>
<log level="full"/>
<loopback/>
</inSequence>
<outSequence>
<send/>
</outSequence>
<faultSequence/>
</target>
</proxy>
My API calling backend 1 and backend 2 and transforming with xslt:
<?xml version="1.0" encoding="UTF-8"?>
<api xmlns="http://ws.apache.org/ns/synapse"
name="ItemList"
context="/item_list">
<resource methods="POST" uri-template="/">
<inSequence>
<header name="Action" scope="default" value="urn:mediate"/>
<call>
<endpoint>
<address uri="http://localhost:8283/services/items.itemsHttpSoap11Endpoint"
format="soap11"/>
</endpoint>
</call>
<enrich>
<source type="inline" clone="true">
<Payloads/>
</source>
<target type="property" property="Items"/>
</enrich>
<enrich>
<source clone="true" xpath="$body/*"/>
<target action="child" xpath="$ctx:Items"/>
</enrich>
<payloadFactory media-type="xml">
<format>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body/>
</soapenv:Envelope>
</format>
<args/>
</payloadFactory>
<call>
<endpoint>
<address uri="http://localhost:8283/services/parts.partsHttpSoap11Endpoint"
format="soap11"/>
</endpoint>
</call>
<enrich>
<source clone="true" xpath="$body/*[name()='response']/*[name()='query']"/>
<target type="property" property="Parts"/>
</enrich>
<enrich>
<source type="property" clone="true" property="Parts"/>
<target action="child" xpath="$ctx:Items"/>
</enrich>
<enrich>
<source type="property" property="Items"/>
<target type="body"/>
</enrich>
<xslt key="transformTwoSourcesToOneResult"/>
<loopback/>
</inSequence>
<outSequence>
<send/>
</outSequence>
<faultSequence/>
</resource>
</api>
And my xslt transformation:
<?xml version="1.0" encoding="UTF-8"?>
<localEntry key="transformTwoSourcesToOneResult" xmlns="http://ws.apache.org/ns/synapse">
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:ns0="http://example.com/query"
xmlns:ns1="http://example.com/response"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:exslt="http://exslt.org/common"
xmlns:saxon="http://saxon.sf.net/"
xmlns:syn="http://ws.apache.org/ns/synapse"
exclude-result-prefixes="ns0 ns1 xs">
<xsl:output method="xml" encoding="UTF-8" indent="yes"/>
<xsl:template match="/">
<xsl:variable name="var1_instance_Payloads" select="."/>
<items>
<xsl:for-each select="$var1_instance_Payloads/syn:Payloads">
<xsl:variable name="var2_Payloads" select="."/>
<xsl:for-each select="$var2_Payloads/ns1:response/ns0:query/ns0:row">
<xsl:variable name="var2_row" select="."/>
<item>
<id>
<xsl:value-of select="number(string($var2_row/ns0:id))"/>
</id>
<name>
<xsl:value-of select="string($var2_row/ns0:name)"/>
</name>
<parts>
<xsl:for-each select="$var2_Payloads/ns0:query/ns0:row">
<xsl:variable name="var4_row" select="."/>
<xsl:if test="string((number(string($var2_row/ns0:id)) = number(string($var4_row/ns0:id)))) != 'false'">
<part>
<id>
<xsl:value-of select="number(string($var4_row/ns0:id))"/>
</id>
<name>
<xsl:value-of select="string($var4_row/ns0:part)"/>
</name>
</part>
</xsl:if>
</xsl:for-each>
</parts>
</item>
</xsl:for-each>
</xsl:for-each>
</items>
</xsl:template>
</xsl:stylesheet>
</localEntry>
The API response:
<items xmlns="http://ws.apache.org/ns/synapse" xmlns:syn="http://ws.apache.org/ns/synapse" xmlns:saxon="http://saxon.sf.net/" xmlns:exslt="http://exslt.org/common">
<item>
<id>1</id>
<name>Item 1</name>
<parts>
<part>
<id>1</id>
<name>Part 1.1</name>
</part>
<part>
<id>1</id>
<name>Part 1.2</name>
</part>
<part>
<id>1</id>
<name>Part 1.3</name>
</part>
</parts>
</item>
<item>
<id>2</id>
<name>Item 2</name>
<parts>
<part>
<id>2</id>
<name>Part 2.1</name>
</part>
<part>
<id>2</id>
<name>Part 2.2</name>
</part>
</parts>
</item>
</items>