Spring integration MessageTransformationException failed to transform message - amqp

I'm trying to run this (https://github.com/cloudfoundry-samples/wgrus/tree/master/spring-integration) wgrus sample application using spring integration v2.2.6.RELEASE and keep getting a MessageTransformationException. The application is split into 3 components - a store UI, an inventory service and a shipping service (this service is irrelevant here) and uses the AMQP protocol to send messages to RabbitMQ. In brief, you enter order details in the UI and send them to an order channel in the form of a message. Behind the scenes the application performs a claim check, which stores the message in Mongo (I can see that it gets stored) and returns a new message, whose payload is the id of the stored message - so far so good, the message is stored and I can see the amqpOut channel returning the Id of the stored message, i.e. 826bcbfb-21fa-424d-aecd-0bab3d1a690b - this can be seen in the debug shown at the bottom of my question. Next, the message is sent to RabbitMQ through an outbound AMQP channel and is supposed to be picked up by the inventory service and at this point the exception is thrown. I can see that at some point something happens with the id of the stored message as the debug prints Payload=???sr?java.util.UUID????m?/
Has anyone come across this issue and knows how to resolve it?
Config used by the store UI:
<int:object-to-json-transformer input-channel="orderChannel" output-channel="jsonOrders" content-type="text/x-json"/>
<int:claim-check-in input-channel="jsonOrders" output-channel="amqpOut" message-store="messageStore"/>
<amqp:outbound-channel-adapter id="amqpOut" amqp-template="rabbitTemplate" routing-key="orders"/>
...
Config used by the inventory service:
<int:claim-check-out message-store="messageStore" input-channel="orderChannel" output-channel="inventoryChannel" remove-message="true" />
<amqp:inbound-channel-adapter channel="orderChannel"
connection-factory="rabbitConnectionFactory"
queue-names="orders"
error-channel="errorLogger" />
<int:logging-channel-adapter id="errorLogger" log-full-message="true" level="INFO"/>
<int:chain input-channel="inventoryChannel" output-channel="amqpOut">
<int:json-to-object-transformer type="org.wgrus.Order"/>
<int:enricher request-channel="creditCheck">
<int:property name="approved" expression="payload.startsWith('OK')"/>
</int:enricher>
<int:enricher request-channel="inventoryRouter">
<int:property name="reserved" expression="payload"/>
</int:enricher>
<int:object-to-json-transformer content-type="text/x-json" />
<int:claim-check-in/>
</int:chain>
...
DEBUG:
DEBUG: http-bio-8080-exec-3
org.springframework.integration.channel.DirectChannel - postSend
(sent=true) on channel 'amqpOut', message:
[Payload=826bcbfb-21fa-424d-aecd-0bab3d1a690b][Headers={timestamp=1384617973920,
id=e98cd0c1-bd8c-4786-be31-1e77b0200934, content-type=text/x-json}]
DEBUG: http-bio-8080-exec-3
org.springframework.integration.channel.DirectChannel - postSend
(sent=true) on channel 'jsonOrders', message:
[Payload={"id":1,"approved":false,"reserved":false,"customerId":"","quantity":1,"productId":"widget"}][Headers={timestamp=1384617973852,
id=826bcbfb-21fa-424d-aecd-0bab3d1a690b, content-type=text/x-json}]
DEBUG: http-bio-8080-exec-3
org.springframework.integration.channel.DirectChannel - postSend
(sent=true) on channel 'orderChannel', message: [Payload=Order #1: 1
widgets for ][Headers={timestamp=1384617973824,
id=1b700a4b-35e1-4d16-8ca0-7cd20ccfb85e}]
DEBUG:
SimpleAsyncTaskExecutor-1
org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper -
headerName=[content-type] WILL be mapped, matched pattern=content-type
DEBUG: SimpleAsyncTaskExecutor-1
org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper -
headerName=[amqp_receivedRoutingKey] WILL be mapped, matched
pattern=amqp_receivedRoutingKey
DEBUG: SimpleAsyncTaskExecutor-1
org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper -
headerName=[amqp_deliveryMode] WILL be mapped, matched
pattern=amqp_deliveryMode
DEBUG: SimpleAsyncTaskExecutor-1
org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper -
headerName=[amqp_redelivered] WILL be mapped, matched
pattern=amqp_redelivered
DEBUG: SimpleAsyncTaskExecutor-1
org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper -
headerName=[amqp_deliveryTag] WILL be mapped, matched
pattern=amqp_deliveryTag
DEBUG: SimpleAsyncTaskExecutor-1
org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper -
headerName=[content-type] WILL be mapped, matched pattern=content-type
DEBUG: SimpleAsyncTaskExecutor-1
org.springframework.integration.channel.DirectChannel - preSend on
channel 'orderChannel', message:
[Payload=???sr?java.util.UUID????m?/?J?leastSigBitsJ?mostSigBitsxp???=i?k??!?][Headers={timestamp=1384617974260,
id=81a2fb77-0f1e-4be7-9148-84da86a30ed8, content-type=text/x-json,
amqp_receivedRoutingKey=orders, amqp_deliveryMode=PERSISTENT,
amqp_redelivered=false, amqp_deliveryTag=1}]
DEBUG: SimpleAsyncTaskExecutor-1
org.springframework.integration.transformer.MessageTransformingHandler
- org.springframework.integration.transformer.MessageTransformingHandler#1
received message: [Payload=???sr?java.util.UUID????m?/?J?

Well, your issue is here:
<int:object-to-json-transformer input-channel="orderChannel" output-channel="jsonOrders" content-type="text/x-json"/>
Can you explain what is the reason to set content-type, if you after that use <claim-check-in>, who just return UUID - id of the stored Message?
And what is happened?
Your UUID is convert to serialized bytes by SimpleMessageConverter and this converter sets
contentType as application/x-java-serialized-object to the MessageProperties.
But after that is called AmqpHeaderMapper, who changes contentType with your value text/x-json from MessageHeaders.
That's fine for Producer.
But Consumer can't convert the Body correctly, because here also works SimpleMessageConverter by default.
And it checks contentType.startsWith("text").
And just create simple String from bytes of serialized UUID.
Hope it is clear
UPDATE
Unfortunately <object-to-json-transformer> sets content-type header anyway, to application/json by default.
To prevent it you should configure like this content-type="".

Related

Unable to get DeliverToCompID (Tag128) value on acceptor application in QuickFix/J

I am trying to make the DeliverToCompID (tag128) mandatory on NewOrderSingle Message. I am able to send the message from Banzai with tag128 present, but still getting this --> 58=Required tag missing371=128 error message.
I have declared the DeliverToCompID (tag128) value in the config file
[session]
SocketConnectPort=9878
DeliverToCompID=FIXIMULATOR3
SenderCompID=BANZAI
TargetCompID=FIXIMULATOR
and in Banzai Application I am fetching the value from the config and setting it in the message.
String tag128 = settings.getString(sessionID,"DeliverToCompID");
System.out.println("tag 128 "+tag128);
message.getHeader().setField(new DeliverToCompID(tag128));
and I am sending a NewOrderSingle Message, and TAG 128 is present in the logs of both the banzai(sender) and fiximulator(acceptor) application.
BANZAI LOG <20221111-05:28:39, FIX.4.2:BANZAI->FIXIMULATOR, outgoing> (8=FIX.4.29=15435=D34=749=BANZAI52=20221111-05:28:39.66056=FIXIMULATOR128=FIXIMULATOR311=166814451964721=138=256340=154=155=AAA59=060=20221111-05:28:39.65810=238)
FIXIMULATOR LOG <20221111-05:28:39, FIX.4.2:FIXIMULATOR->BANZAI, incoming> (8=FIX.4.29=15435=D34=749=BANZAI52=20221111-05:28:39.66056=FIXIMULATOR128=FIXIMULATOR311=166814451964721=138=256340=154=155=AAA59=060=20221111-05:28:39.65810=238)
But I am getting this :: <20221111-05:28:39, FIX.4.2:FIXIMULATOR->BANZAI, event> (Message 7 Rejected: Required tag missing:128)
<20221111-05:28:39, FIX.4.2:FIXIMULATOR->BANZAI, outgoing> (8=FIX.4.29=12635=334=749=FIXIMULATOR52=20221111-05:28:39.66456=BANZAI115=FIXIMULATOR345=758=Required tag missing371=128372=D373=110=084)
I have used custom FIX42.xml file
<message name="NewOrderSingle" msgtype="D" msgcat="app">
<field name="DeliverToCompID" required="Y"/>
and in the FIXIMLATOR config I have made
ValidateIncomingMessage=Y
DataDictionary=config/FIX42.xml
UseDataDictionary=Y

How can I read and decode AVRO messages from Kafka along with their associated kafka key using Benthos?

I am using Benthos to read AVRO-encoded messages from Kafka which have the kafka_key metadata field set to also contain an AVRO-encoded payload. The schemas of these AVRO-encoded payloads are stored in Schema Registry and Benthos has a schema_registry_decode processor for decoding them. I'm looking to produce an output JSON message for each Kafka message containing two fields, one called content containing the decoded AVRO message and the other one called metadata containing the various metadata fields collected by Benthos including the decoded kafka_key payload.
It turns out that one can achieve this using a branch processor like so:
input:
kafka:
addresses:
- localhost:9092
consumer_group: benthos_consumer_group
topics:
- benthos_input
pipeline:
processors:
# Decode the message
- schema_registry_decode:
url: http://localhost:8081
# Populate output content field
- bloblang: |
root.content = this
# Decode kafka_key metadata payload and populate output metadata field
- branch:
request_map: |
root = meta("kafka_key")
processors:
- schema_registry_decode:
url: http://localhost:8081
result_map: |
root.metadata = meta()
root.metadata.kafka_key = this
output:
stdout: {}

Mulesoft EC2 *describeInstances* with *filter* option

I'm having problems using the EC2 connector with filters for DescribeInstances. Specifically, I'm trying to find all instances that have the tag "classId" set.
I've also tried to find all instances that have the classId tag with specific string, e.g. "123".
Below are the XMLs of the describeInstance for both scenarios.
tag-key ------
<ec2:describe-instances doc:name="Describe instances" doc:id="ca64b7d4-99bb-4045-bbb4-16c0c27b1df5" config-ref="Amazon_EC2_Configuration">
<ec2:filters>
<ec2:filter name="tag-key" values="#[['classId']]">
</ec2:filter>
</ec2:filters>
</ec2:describe-instances>
tag:classId:----
<ec2:describe-instances doc:name="Describe instances" doc:id="ca64b7d4-99bb-4045-bbb4-16c0c27b1df5" config-ref="Amazon_EC2_Configuration">
<ec2:filters>
<ec2:filter name="tag:classId">
<ec2:values >
<ec2:value value="#['123']" />
</ec2:values>
</ec2:filter>
</ec2:filters>
</ec2:describe-instances>
Each time I receive an error like the following (for tag:classId):
ERROR 2021-03-29 08:32:49,693 [[MuleRuntime].uber.04: [ec2-play].ec2-playFlow.BLOCKING #1092a5bc] [processor: ; event: df5e2df0-908a-11eb-94b5-38f9d38da5c3] org.mule.runtime.core.internal.exception.OnErrorPropagateHandler: 
********************************************************************************
Message        : The filter 'null' is invalid (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 33e3bbfb-99ea-4382-932f-647662810c92; Proxy: null)
Element        : ec2-playFlow/processors/0 # ec2-play:ec2-play.xml:33 (Describe instances)
Element DSL      : <ec2:describe-instances doc:name="Describe instances" doc:id="ca64b7d4-99bb-4045-bbb4-16c0c27b1df5" config-ref="Amazon_EC2_Configuration">
<ec2:filters>
<ec2:filter name="tag:classId">
<ec2:values>
<ec2:value value="#['123']"></ec2:value>
</ec2:values>
</ec2:filter>
</ec2:filters>
</ec2:describe-instances>
Error type      : EC2:INVALID_PARAMETER_VALUE
FlowStack       : at ec2-playFlow(ec2-playFlow/processors/0 # ec2-play:ec2-play.xml:33 (Describe instances))
 (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
NOTE: The code works without a filter, returning all instances. But, that isn't what I want or need. The more filtering I can do the faster the response.
Does anyone have samples of the filter option working? Can you tell me what I'm doing wrong?
Thanks!
This surely is a bug. I tried the same and it was not working for me as well. I enabled debug logging and found that the connector is not sending the filter.1.Name=tag:classId as a query parameter in the request. Here is the debug log that I found. (Notice there is no filter.1.Name=tag:classId in the query string)
DEBUG 2021-04-02 21:55:17,198 [[MuleRuntime].uber.03: [test-aws-connector].test-aws-connectorFlow.BLOCKING #2dff3afe] [processor: ; event: 91a34891-93d0-11eb-af49-606dc73d31d1] org.apache.http.wire: http-outgoing-0 >> "Action=DescribeInstances&Version=2016-11-15&Filter.1.Value.1=123"
However, I tried to use the Expression or Bean Reference option and set the expression directly as [{name: 'tag:classId', values:['123']}] like this:
and it worked correctly. Here is the same debug log after this change
DEBUG 2021-04-02 21:59:17,198 [[MuleRuntime].uber.03: [test-aws-connector].test-aws-connectorFlow.BLOCKING #2dff3afe] [processor: ; event: 91a34891-93d0-11eb-af49-606dc73d31d1] org.apache.http.wire: http-outgoing-0 >> "Action=DescribeInstances&Version=2016-11-15&Filter.1.Name=tag%3AclassId&Filter.1.Value.1=123"
Also, I want to point out very weird behaviour, this does not work if you try to format [{name: 'tag:classId',values: ['123']}] across multiple lines in the expression and will give an error during deployment.

Gateway and aggregator - CorrelationStrategy error

I have to access an external service using spring integration. The process is (1) I had to pass an id to get basic information (2) using basic info from step 1, I need to access to more services and merge the information into a singlr object.
integration-graph:
input: Channel1 outputs to : Channel1Out
I have recipient list router that puts message to the 2 channels Channel2 and Channel3.
Channel2 and Channel3's output channels use a xml xpath-transformer
and output to Channel 4
<int:aggregator id="aggregatorChannel"
correlation-strategy-expression="headers['jms_messageId']"
release-strategy-expression="size() == 2" method="mergeVO"
input-channel="channel4" output-channel="dest-channel">
<bean class="n.b.lbr.eai.vo.PojoAggregator"></bean>
</int:aggregator>
This is giving error.
java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
at org.springframework.util.Assert.state(Assert.java:70) ~[spring-core-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.handleMessageInternal(AbstractCorrelatingMessageHandler.java:385) ~[spring-integration-core-4.3.9.RELEASE.jar:4.3.9.RELEASE]
I did see some posts on this topic, but I do not understand how to solve the below error
{
"timestamp": 1533137160301,
"status": 500,
"error": "Internal Server Error",
"exception": "java.lang.IllegalStateException",
"message": "Null correlation not allowed. Maybe the CorrelationStrategy is failing?",
"path": "/w/b/search/11223"
}
please suggest, if this is a design issue or how to solve this problem
EDIT1:
Is the below valid scatter gather?
<bean id="messageStore" class="org.springframework.integration.store.SimpleMessageStore"/>
<int:scatter-gather id="scatterGather2" input-channel="drBInputChannel" gather-channel="gatherChannel" gather-timeout="5000">
<int:scatterer id="myScatterer" apply-sequence="true">
<int:recipient channel="bserviceInputChannel"/>
<int:recipient channel="aserviceInputChannel"/>
</int:scatterer>
<int:gatherer id="myGatherer"
**??**
message-store="messageStore"
correlation-strategy=**??**
release-strategy-expression="size() == 2"
>
<bean class="nd.wbr.eai.vo.PojoAggregator"></bean>
</int:gatherer>
</int:scatter-gather>
I need help to convert to xml and use in the above
#Bean
public MessageHandler gatherer() {
return new AggregatingMessageHandler(
***new ExpressionEvaluatingMessageGroupProcessor("^[payload gt 5] ?:-1D"),***
new SimpleMessageStore(),
***new HeaderAttributeCorrelationStrategy(
IntegrationMessageHeaderAccessor.CORRELATION_ID),***
new ExpressionEvaluatingReleaseStrategy("size() == 2"));
}
The "java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?" exception means that your correlation-strategy-expression="headers['jms_messageId']" doesn't produce anything meaningful. To be precise there is just no jms_messageId header in the message.
Not sure why you did such a choice for the correlation key, but there is definitely not going to be such a headers when you perform HTTP request. You may emulate it though, but it might be better to choose some other correlation strategy.
On the other hand, looking to your original task description, I would say that you need to take a look into the Scatter-Gather pattern and stop to worry about correlation key altogether!

IBM Content Collector error receiving response after external WS call

In my current project using IBM Content Collector 4.0.1 SP5 with IBM Filenet P8 Content Engine 5.2.1 I need to collect files from file system and add them to a certain P8's object store.
This issue is related and comes after this one.
The WS response must conform to a custom metadata source called Esito which contains two metadata properties called resultCode and message.
The call seems correct but when getting the response I can read this message in the log and the file is not uploaded into P8 Content Manager:
2017-11-10T08:54:05.708Z FINEST [52] [ctms-native]
2017-11-10T08:54:05Z Trace2 0x15a4 Feeding JavaTask with 1 TaskInputs
ufibridge.dll:0x114ac [com.ibm.afu.ufibridge.logging.LoggingAdapter
log] [CTMS-task-15a4 45] 2017-11-10T08:54:05.708Z FINEST [53] Prepared
content to send to
webservice:{"e:\report\amm_000001_00001\2017\11\201711_amm_000001_00001_qxn_report_00.pdf":{"esito":"KO","fileName":"201711_AMM_000001_00001_QXN_REPORT_00(1)(2)(3).pdf"},"e:\report\amm_000001_00001\2017\11\201711_amm_000001_00001_qxn_report_00.xml":{"esito":"KO","fileName":"201711_AMM_000001_00001_QXN_REPORT_00(1)(2)(3).xml"}}
[com.ibm.afu.connector.webservice.task.InvokeServiceTask
getInputHttpEntity] [CTMS-task-15a4 45] 2017-11-10T08:54:05.708Z
FINEST [54] Configuration:
property name="message" type="STRING" displayName="message"
property name="resultCode" type="STRING" displayName="resultCode"
http://192.168.8.29:8080/sirfAcq/filenet/notificaArchiviazione
fileName esito
[com.ibm.afu.connector.webservice.task.InvokeServiceTask execute]
[CTMS-task-15a4 45] 2017-11-10T08:54:05.708Z FINEST [55] Invoking
webservice
URI:http://192.168.8.29:8080/sirfAcq/filenet/notificaArchiviazione
[com.ibm.afu.connector.webservice.task.InvokeServiceTask execute]
[CTMS-task-15a4 45] 2017-11-10T08:54:05.739Z FINEST [56] Invocation
took time (ms): 31
[com.ibm.afu.connector.webservice.task.InvokeServiceTask execute]
[CTMS-task-15a4 45] 2017-11-10T08:54:05.739Z SEVERE [57]
java.lang.String incompatible with com.ibm.json.java.JSONObject
[com.ibm.afu.connector.webservice.task.InvokeServiceTask execute]
[CTMS-task-15a4 45]Exception [java.lang.ClassCastException]:
java.lang.String incompatible with com.ibm.json.java.JSONObject
com.ibm.afu.connector.webservice.task.InvokeServiceTask.processResponse(InvokeServiceTask.java:140)
com.ibm.afu.connector.webservice.task.InvokeServiceTask.execute(InvokeServiceTask.java:112)
Could someone please explain me the correct format of the WS response JSON?
Could it be possible to use the One-Way mode instead of the Request-Response mode?
This issue seems to happen because of the incompatibility between the produce and the response body definition, based on the code snippet I had for your first post, your problem should be here:
produces = {MediaType.APPLICATION_JSON_VALUE, MediaType.APPLICATION_XML_VALUE })
#ResponseBody
String notificaArchiviazione(#RequestParam("fileName") String fileName, #RequestParam("esito") String esito)
The first parameter in produce is sat as JSONMediaType.APPLICATION_JSON_VALUE while in the response body you expect a string #RequestParam("fileName") String fileName

Resources