Getting data elements from SOAP request - proxy

My Request comes as below
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hel="http://example.org/HelloService">
<soapenv:Header>
<sampler>Test</sampler>
</soapenv:Header>
<soapenv:Body>
<hel:LastName>
<lastName>Test</lastName>
</hel:LastName>
</soapenv:Body>
</soapenv:Envelope>
My flow which exposes my CXF:proxy-service is as below
<flow name="WS_In">
<http:inbound-endpoint address="http://localhost:8080/HelloService"
exchange-pattern="request-response">
<cxf:proxy-service wsdlLocation="classpath:helloservice.wsdl"
namespace="http://example.org/HelloService" service="ProxyService" >
<cxf:inInterceptors>
<spring:bean id="inLogger"
class="org.apache.cxf.interceptor.LoggingInInterceptor" />
<spring:bean id="msgInt" class="com.example.components.MessageInterceptor"/>
</cxf:inInterceptors>
</cxf:proxy-service>
</http:inbound-endpoint>
<component>
<spring-object bean="proxyService"></spring-object>
</component>
<echo-component></echo-component>
</flow>
But I am not able to get the elements in the soap header. I tried using interceprots with method like below
#Override
public void handleMessage(SoapMessage message) throws Fault {
System.out.println("The headers is " + message.getHeaders() );
}
But this is printing an emply Collection.
Please suggest how I can get this.

Try adding a custom-processor on your flow before the proxy-service
<flow name="WS_In">
<http:inbound-endpoint address="http://localhost:8080/HelloService"
exchange-pattern="request-response">
<custom-processor class="your.package.SoapHeaderObserver" />
<cxf:proxy-service wsdlLocation="classpath:helloservice.wsdl"
namespace="http://example.org/HelloService" service="ProxyService" >
<cxf:inInterceptors>
<spring:bean id="inLogger"
class="org.apache.cxf.interceptor.LoggingInInterceptor" />
<spring:bean id="msgInt" class="com.example.components.MessageInterceptor"/>
</cxf:inInterceptors>
</cxf:proxy-service>
</http:inbound-endpoint>
<component>
<spring-object bean="proxyService"></spring-object>
</component>
<echo-component></echo-component>
public class SoapHeaderObserver {
doSomething(SoapMessage message) {
//try to get header here
}
}
Also, your custom-processor can implement MessageProcessor or Callable. Take a look here on How to implement a Mule Message Observer?
And also, the framework have a lot of processors that you might use, instead of building your own.

I got the solution be adding a message processor before my proxy-service.
Given below is my processor.
public class SOAPHeaderExtractor implements MessageProcessor {
#Override
public MuleEvent process(MuleEvent event) {
try
{
MuleMessage inputMessage = event.getMessage();
SOAPMessage soapMsg = MessageFactory.newInstance().createMessage(null,
new java.io.ByteArrayInputStream(inputMessage.getPayloadAsString().getBytes()));
SOAPHeader header = soapMsg.getSOAPHeader();
System.out.println(header.getElementsByTagName("sampler").item(0).getTextContent() );
}
catch(Exception e){
e.printStackTrace();
}
return event ;
}
}
And the change to my flow.
<http:inbound-endpoint address="http://localhost:8080/HelloService"
exchange-pattern="request-response">
<custom-processor class="com.example.processors.SOAPHeaderExtractor" />
<cxf:proxy-service wsdlLocation="classpath:helloservice.wsdl"
namespace="http://example.org/HelloService" service="ProxyService" >
<cxf:inInterceptors >
<spring:bean id="inLogger" class="org.apache.cxf.interceptor.LoggingInInterceptor" />
</cxf:inInterceptors>
</cxf:proxy-service>
</http:inbound-endpoint>
Please do suggst if there is any better way.

Related

Setting up WS-Security in MULE without properties file (WSS4JInInterceptor with signaturePropRefId)

I'm tryring to set up WS-Security in a CXF Proxy MULE Project. I currently have it working with a properties file, but I would like to take some info out of the properties file and insert it in the DataBase so it can be secured there but I can't find a way to make it work.
The relevant data in my project now looks like this:
CXF Proxy in flow:
<cxf:proxy-service doc:name="CXF Server" wsdlLocation="${wss.http.protocol}://${wss.http.host}:${wss.http.port}${wss.http.base_path}?${wss.http.wsdl_file}" payload="envelope" bindingId="${wss.http.binding_id}" namespace="${wss.http.namespace}" service="${wss.http.service}" >
<cxf:inInterceptors>
<spring:bean class="org.apache.cxf.interceptor.LoggingInInterceptor" />
<spring:bean class="org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor">
<spring:constructor-arg>
<spring:map>
<spring:entry key="action" value="Signature" />
<spring:entry key="signaturePropFile" value="ws.properties" />
</spring:map>
</spring:constructor-arg>
</spring:bean>
</cxf:inInterceptors>
</cxf:proxy-service>
ws.properties file:
org.apache.ws.security.crypto.provider=org.apache.ws.security.components.crypto.Merlin
org.apache.ws.security.crypto.merlin.keystore.type=JKS
org.apache.ws.security.crypto.merlin.file=myTrustStore.jks
org.apache.ws.security.crypto.merlin.keystore.password=myTrustStorePass
I'd need to take those file and keystore.password parameters out of the files of the project and set them in the DataBase in order to be injected.
I have vainly (as labels like util:properties are throwing a 'The prefix "util" for element "util:properties" is not bound' error) tried an approach similar to the one shown here:
http://cxf.547215.n5.nabble.com/WS-Security-Properties-Reference-td5505704.html
I've also tried to set the properties file this way in order to get the relevant data from the BD, but this data is not getting injected:
org.apache.ws.security.crypto.provider=org.apache.ws.security.components.crypto.Merlin
org.apache.ws.security.crypto.merlin.keystore.type=JKS
org.apache.ws.security.crypto.merlin.file=${wss.trustKeystore.file}
org.apache.ws.security.crypto.merlin.keystore.password=${wss.trustKeystore.password}
Am I doing anything wrong?
Can this be solved in any of the ways exposed (or any other)? If so, how?
Thanks.
EDIT due to a new scenario for this problem:
When I load the configuration for the interceptor from a properties file everything works fine, but I need to inject those configuration properties from DB, so I decided to configure it by using a java.util.properties object in the XML file in order to inject the values later. As a previous step to stablishing the injections, this is the code I have:
<mule xmlns:mulexml="http://www.mulesoft.org/schema/mule/xml"
xmlns:tls="http://www.mulesoft.org/schema/mule/tls"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns:cxf="http://www.mulesoft.org/schema/mule/cxf"
xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.mulesoft.org/schema/mule/xml http://www.mulesoft.org/schema/mule/xml/current/mule-xml.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core
http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http
http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/cxf
http://www.mulesoft.org/schema/mule/cxf/current/mule-cxf.xsd
http://www.mulesoft.org/schema/mule/tls
http://www.mulesoft.org/schema/mule/tls/current/mule-tls.xsd">
<http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="5081" doc:name="HTTP Listener Configuration"/>
<http:request-config name="HTTP_Request_Configuration" host="${conf.prop.host}" port="${conf.prop.port}" doc:name="HTTP Request Configuration" connectionIdleTimeout="60000" responseTimeout="60000"/>
<spring:beans>
<spring:bean id="WrongResultException" name="WrongResultException" class="transform.WrongResultException"/>
<spring:bean name="wsCryptoProperties" class="java.util.Properties">
<spring:constructor-arg>
<spring:map>
<spring:entry key="org.apache.ws.security.crypto.provider" value="org.apache.ws.security.components.crypto.Merlin"/>
<spring:entry key="org.apache.ws.security.crypto.merlin.keystore.type" value="JKS"/>
<spring:entry key="org.apache.ws.security.crypto.merlin.keystore.password" value="my_truststore_password"/>
<spring:entry key="org.apache.ws.security.crypto.merlin.file" value="my_truststore.jks"/>
</spring:map>
</spring:constructor-arg>
</spring:bean>
</spring:beans>
<flow name="HttpsCall">
<http:listener config-ref="https-listener-configured-in-domain-app" path="/my_path/my_service" doc:name="HTTPS"/>
<logger message="HTTPS call" level="INFO" doc:name="Logger HTTPS"/>
<flow-ref name="HttpCall" doc:name="HttpCall"/>
</flow>
<flow name="HttpCall">
<http:listener config-ref="http-listener-configured-in-domain-app" path="/my_path/my_service" doc:name="HTTP"/>
<cxf:proxy-service doc:name="CXF Server" wsdlLocation="${service.protocol}://${service.host}:${service.port}${service.base_path}?${service.wsdl_file}" payload="envelope" bindingId="${service.binding_id}" namespace="${service.namespace}" service="${service.service}" >
<cxf:inInterceptors>
<spring:bean class="org.apache.cxf.interceptor.LoggingInInterceptor" />
<spring:bean class="org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor">
<spring:constructor-arg>
<spring:map>
<spring:entry key="action" value="Signature" />
<!-- This is how the Crypto object is configured when using a properties file
<spring:entry key="signaturePropFile" value="security_conf_file.properties" />-->
<spring:entry key="signaturePropRefId" value="wsCryptoProperties"/>
</spring:map>
</spring:constructor-arg>
</spring:bean>
</cxf:inInterceptors>
</cxf:proxy-service>
<message-properties-transformer doc:name="Message Properties">
<add-message-property key="SOAPAction" value="#[message.inboundProperties.SOAPAction]"/>
</message-properties-transformer>
<cxf:proxy-client payload="envelope" doc:name="CXF Client" />
<http:request config-ref="HTTP_Request_Configuration" path="${service.base_path}" method="POST" doc:name="HTTP" />
<exception-strategy ref="mule-serviceCatch_Exception_Strategy" doc:name="Reference Exception Strategy"/>
</flow>
<catch-exception-strategy name="mule-serviceCatch_Exception_Strategy">
<logger message="Exception: #[message]" level="INFO" doc:name="Logger"/>
<transformer ref="WrongResultException" doc:name="Transformer Reference"/>
<mulexml:object-to-xml-transformer doc:name="Object to XML"/>
</catch-exception-strategy>
The fact is that everything compiles properly even when configuring the properties via java.util.Properties but, when calling to the service, an error is thrown saying that the properties were not loaded correctly:
WARN 2017-05-03 12:08:27,448 [[mule_domain_app].http-listener-configured-in-domain-app.worker.01] org.apache.ws.security.handler.WSHandler: The Crypto reference wsCryptoProperties specified by signaturePropRefId could not be loaded
WARN 2017-05-03 12:08:27,467 [[mule_domain_app].http-listener-configured-in-domain-app.worker.01] org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor:
org.apache.ws.security.WSSecurityException: General security error (WSSecurityEngine: No crypto property file supplied to verify signature)
at
org.apache.ws.security.message.token.X509Security.getX509Certificate(X509Security.java:100) ~[wss4j-1.6.18.jar:1.6.18]
at
org.apache.ws.security.message.token.SecurityTokenReference.getKeyIdentifier(SecurityTokenReference.java:491) ~[wss4j-1.6.18.jar:1.6.18] ...
And so on...
Can anybody throw some light over this issue? I'm completely stuck.
maybe you can provide this two parameters when you are starting your mule runtime.
./mule -M-Dorg.apache.ws.security.crypto.merlin.file=myTrustStore.jks -M-Dorg.apache.ws.security.crypto.merlin.keystore.password=myTrustStorePass
inside your mule configuration file, you should be able to access the values with: ${org.apache.ws.security.crypto.merlin.file} and ${org.apache.ws.security.crypto.merlin.keystore.password}
Not pretty sure if this would be a valid solution in your use case but maybe it could give you new ideas...
Let's see if this will solve the problem... I could replicate the case of getting values from a database and setup a connector. I made it slightly different but I guess that the concept is the only important matter.
In my case, I get properties from a derby in memory database and then I use this props for setting up an http:listener-config
First, I added a springBean to hold my properties:
<spring:beans>
<spring:bean id="propertiesBean" name="propertiesBean" class="com.mmartinez.test.DerbyLoadProperties" />
</spring:beans>
Inside the DerbyLoadProperties class, is necessary to implements InitializingBean and FactoryBean. The factoryBean will allow you to return the Properties object containing your properties, in this case coming from the database.
public class DerbyLoadProperties implements InitializingBean, FactoryBean {
private Properties prop = new Properties();
#Override
public void afterPropertiesSet() throws Exception {
initializeDatabaseAndExtractProperties();
//Inside this method I initialize the in-memory-db and also add the host and port properties from DB
//prop.put("http.hostmario", host);
//prop.put("http.portmario", port);
}
#Override
public Class getObjectType() {
return Properties.class;
}
#Override
public Object getObject() throws Exception {
return prop;
}
#Override
public boolean isSingleton() {
return false;
}
Now I can use my propertiesBean as a property-placeholder
<context:property-placeholder properties-ref="propertiesBean" />
In the last step, I will setup my http:listener-config, in your case the cxf:proxy-service....
<http:listener-config port="${http.portmario}" host="${http.hostmario}" name="testListener" doc:name="HTTP Listener Configuration" />
Is working fine for me.
Solved!
This line was missing beetween the entries of the map passed in the WSS4JInInterceptor constructor:
<spring:entry key="wsCryptoProperties" value-ref="wsCryptoProperties"/>
Doing this, an injection in the values of the props of the java.util.Properties object works as a charm.
Hope this helps anybody in the future!

Delete File after successful persist to MongoDB in Spring Integration

I have a Spring Integration flow that reads a csv file from a directory, splits the lines, then processes each line and extracts 2 objects from each line. These two objects are then send to two seperate int-mongodb:outbound-channel-adapter. I want to delete the incoming file after all of the lines have been processed and persisted. I have seen example of using the Transaction Manager to do this with the inbound adapter, but nothing with the outbound adapter. Is there a way to do this?
My config looks something like this:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd
http://www.springframework.org/schema/integration/mongodb http://www.springframework.org/schema/integration/mongodb/spring-integration-mongodb.xsd
http://www.springframework.org/schema/data/mongo http://www.springframework.org/schema/data/mongo/spring-mongo.xsd"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xmlns:task="http://www.springframework.org/schema/task"
xmlns:int-mongodb="http://www.springframework.org/schema/integration/mongodb"
xmlns:mongo="http://www.springframework.org/schema/data/mongo">
<int:poller default="true" fixed-delay="50"/>
<int-file:inbound-channel-adapter id="filesInChannel"
directory="file:${file.ingest.directory}"
auto-create-directory="true">
<int:poller id="poller" fixed-rate="100">
</int:poller>
</int-file:inbound-channel-adapter>
<task:executor id="executor" pool-size="10" queue-capacity="50" />
<int:channel id="executorChannel">
<int:queue capacity="50"/>
</int:channel>
<int:splitter input-channel="filesInChannel" output-channel="executorChannel"
expression="T(org.apache.commons.io.FileUtils).lineIterator(payload)"/>
<int:service-activator id="lineParserActivator" ref="lineParser" method="parseLine"
input-channel="executorChannel" output-channel="lineChannel">
<int:poller task-executor="executor" fixed-delay="500">
</int:poller>
</int:service-activator>
<bean name="lineParser" class="com.xxx.LineParser"/>
<int:channel id="lineChannel">
<int:queue/>
</int:channel>
<int:channel id="lineMongoOutput">
<int:queue/>
</int:channel>
<int:channel id="actionMongoOutput">
<int:queue/>
</int:channel>
<int:transformer input-channel="lineChannel" output-channel="lineMongoOutput">
<bean id="lineTransformer" class="com.xxx.transformer.LineTransformer"></bean>
</int:transformer>
<int:transformer input-channel="lineChannel" output-channel="actionMongoOutput">
<bean id="actionTransformer" class="com.xxx.transformer.ActionTransformer"></bean>
</int:transformer>
<mongo:db-factory id="mongoDbFactory" dbname="${mongo.db.name}" password="${mongo.db.pass}" username="${mongo.db.user}" port="${mongo.db.port}" host="${mongo.db.host}"/>
<int-mongodb:outbound-channel-adapter id="lineMongoOutput"
collection-name="full"
mongodb-factory="mongoDbFactory" />
<int-mongodb:outbound-channel-adapter id="actionMongoOutput"
collection-name="action"
mongodb-factory="mongoDbFactory" />
</beans>
You can't really do it on the outbound adapter because you don't know when you're "done". Given you are asynchronously handing off to the downstream flow (via executors and queue channels), you can't do it on the inbound adapter either, because the poller thread will return to the adapter as soon as all the splits are sent.
Aside from that, I see some issues in your flow:
You seem to have an excessive amount of thread handoffs - you really don't need queue channels in the downstream flow because your executions are controlled by the exec. channel.
It is quite unusual to make every channel a QueueChannel.
Finally, you have 2 transformers subscribed to the same channel.
Do you realize that messages sent to lineChannel will alternate round-robin style.
Perhaps that is your intent, given your description, but it seems a little brittle to me; I would prefer to see the different data types going to different channels.
If you avoid using queue channels, and use gateways within your service activator to send out the data to the mongo adapters, your service activator would know when it is complete and be able to remove the file at that time.
EDIT:
Here is one solution (it writes to logs rather than mongo, but you should get the idea)...
<int-file:inbound-channel-adapter directory="/tmp/foo" channel="toSplitter">
<int:poller fixed-delay="1000">
<int:transactional synchronization-factory="sf" transaction-manager="ptxMgr" />
</int:poller>
</int-file:inbound-channel-adapter>
<int:transaction-synchronization-factory id="sf">
<int:after-commit expression="payload.delete()" />
<int:after-rollback expression="payload.renameTo(new java.io.File('/tmp/bad/' + payload.name))" />
</int:transaction-synchronization-factory>
<bean id="ptxMgr" class="org.springframework.integration.transaction.PseudoTransactionManager" />
<int:splitter input-channel="toSplitter" output-channel="processChannel">
<bean class="org.springframework.integration.file.splitter.FileSplitter" />
</int:splitter>
<int:service-activator input-channel="processChannel">
<bean class="foo.Foo">
<constructor-arg ref="gate" />
</bean>
</int:service-activator>
<int:gateway id="gate" service-interface="foo.Foo$Gate">
<int:method name="toLine" request-channel="toLine" />
<int:method name="toAction" request-channel="toAction" />
</int:gateway>
<int:channel id="toLine" />
<int:logging-channel-adapter channel="toLine" expression="'LINE:' + payload" level="WARN"/>
<int:channel id="toAction" />
<int:logging-channel-adapter channel="toAction" expression="'ACTION:' + payload" level="WARN"/>
.
public class Foo {
private final Gate gateway;
public Foo(Gate gateway) {
this.gateway = gateway;
}
public void parse(String payload) {
String[] split = payload.split(",");
if (split.length != 2) {
throw new RuntimeException("Bad row size: " + split.length);
}
this.gateway.toLine(split[0]);
this.gateway.toAction(split[1]);
}
public interface Gate {
void toLine(String line);
void toAction(String action);
}
}
.
#ContextConfiguration
#RunWith(SpringJUnit4ClassRunner.class)
public class FooTests {
#Test
public void testGood() throws Exception {
File file = new File("/tmp/foo/x.txt");
FileOutputStream fos = new FileOutputStream(file);
fos.write("foo,bar".getBytes());
fos.close();
int n = 0;
while(n++ < 100 && file.exists()) {
Thread.sleep(100);
}
assertFalse(file.exists());
}
#Test
public void testBad() throws Exception {
File file = new File("/tmp/foo/y.txt");
FileOutputStream fos = new FileOutputStream(file);
fos.write("foo".getBytes());
fos.close();
int n = 0;
while(n++ < 100 && file.exists()) {
Thread.sleep(100);
}
assertFalse(file.exists());
file = new File("/tmp/bad/y.txt");
assertTrue(file.exists());
file.delete();
}
}
Add a task executor to the <poller/> to process multiple files concurrently. Add a router as needed.

Spring integration and AMQP on RabbitMQ throws exception Method handleToken(byte[]) cannot be found

This is my context config:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int-amqp="http://www.springframework.org/schema/integration/amqp"
xmlns:rabbit="http://www.springframework.org/schema/rabbit"
xmlns:int="http://www.springframework.org/schema/integration"
xsi:schemaLocation="http://www.springframework.org/schema/integration/amqp http://www.springframework.org/schema/integration/amqp/spring-integration-amqp.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/rabbit http://www.springframework.org/schema/rabbit/spring-rabbit.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd">
<rabbit:connection-factory id="connectionFactory" host="localhost"/>
<rabbit:template id="amqpTemplate" connection-factory="connectionFactory"/>
<rabbit:queue name="upload-queue"/>
<rabbit:admin connection-factory="connectionFactory"/>
<rabbit:topic-exchange name="infrastructure-exchange">
<rabbit:bindings>
<rabbit:binding queue="upload-queue" pattern="ru.interosite.*"/>
</rabbit:bindings>
</rabbit:topic-exchange>
<int-amqp:outbound-channel-adapter
channel="toRabbit"
routing-key="ru.interosite.1"
amqp-template="amqpTemplate"
exchange-name="infrastructure-exchange"/>
<int-amqp:inbound-channel-adapter
channel="fromRabbit"
queue-names="upload-queue"
connection-factory="connectionFactory"
/>
<int:gateway id="infrastructureGateway" service-interface="com.dropbyke.web.api.service.InfrastructureGateway"/>
<int:channel id="toRabbit"/>
<int:channel id="fromRabbit"/>
<bean id="testInfrastructureServiceImpl" class="com.dropbyke.web.api.service.TestInfrastructureServiceImpl"/>
<int:service-activator input-channel="fromRabbit" ref="testInfrastructureServiceImpl" method="handleToken" />
<!-- A Spring Integration adapter that routes messages sent to the helloWorldChannel to the bean named "helloServiceImpl"'s hello() method -->
<!--<int:outbound-channel-adapter channel="fromRabbit" ref="testInfrastructureServiceImpl" method="handleToken"/>-->
</beans>
This is gateway interface:
public interface InfrastructureGateway {
#Gateway(requestChannel = "toRabbit")
void doUpload(UserTokenDTO tokenDTO);
}
Message endpoint that supposed to handle calls:
#MessageEndpoint
public class TestInfrastructureServiceImpl implements TestInfrastructureService {
public void handleToken( UserTokenDTO tokenDTO) {
System.out.println("Handle user token " + tokenDTO.getToken());
}
}
So when I invoke gateway's doUpload method I'm getting this error message:
Caused by: org.springframework.expression.spel.SpelEvaluationException: EL1004E:(pos 8): Method call: Method handleToken(byte[]) cannot be found on com.dropbyke.web.api.service.TestInfrastructureServiceImpl type
If I exclude Rabbit part then all works fine. What should I do to make it work with Rabbit? Should I implement some converter from byte[] to my domain object UserTokenDTO?
As nicolas-labrot suggested I had to make my UserTokenDTO to be a Serializable. After that all works fine.

How does MessageChannelPartitionHandler in spring batch behave if resource path does not exist in remote partitioning using spring integration?

I am using spring batch remote partitioning. Below is my configuration
<task:executor id="taskExecutor" pool-size="50" />
<rabbit:template id="computeAmqpTemplate"
connection-factory="rabbitConnectionFactory" routing-key="computeQueue"
reply-timeout="${compute.partition.timeout}">
</rabbit:template>
<int:channel id="computeOutboundChannel">
<int:dispatcher task-executor="taskExecutor" />
</int:channel>
<int:channel id="computeInboundStagingChannel" />
<amqp:outbound-gateway request-channel="computeOutboundChannel"
reply-channel="computeInboundStagingChannel" amqp-template="computeAmqpTemplate"
mapped-request-headers="correlationId, sequenceNumber, sequenceSize, STANDARD_REQUEST_HEADERS"
mapped-reply-headers="correlationId, sequenceNumber, sequenceSize, STANDARD_REQUEST_HEADERS" />
<beans:bean id="computeMessagingTemplate"
class="org.springframework.integration.core.MessagingTemplate"
p:defaultChannel-ref="computeOutboundChannel"
p:receiveTimeout="${compute.partition.timeout}" />
<beans:bean id="computePartitionHandler"
class="org.springframework.batch.integration.partition.MessageChannelPartitionHandler"
p:stepName="computeStep" p:gridSize="${compute.grid.size}"
p:messagingOperations-ref="computeMessagingTemplate" />
<int:aggregator ref="computePartitionHandler"
send-partial-result-on-expiry="true" send-timeout="${compute.step.timeout}"
input-channel="computeInboundStagingChannel" />
<amqp:inbound-gateway concurrent-consumers="${compute.consumer.concurrency}"
request-channel="computeInboundChannel"
reply-channel="computeOutboundStagingChannel" queue-names="computeQueue"
connection-factory="rabbitConnectionFactory"
mapped-request-headers="correlationId, sequenceNumber, sequenceSize, STANDARD_REQUEST_HEADERS"
mapped-reply-headers="correlationId, sequenceNumber, sequenceSize, STANDARD_REQUEST_HEADERS" />
<int:channel id="computeInboundChannel" />
<int:service-activator ref="stepExecutionRequestHandler"
input-channel="computeInboundChannel" output-channel="computeOutboundStagingChannel" />
<int:channel id="computeOutboundStagingChannel" />
<beans:bean id="computePartitioner"
class="org.springframework.batch.core.partition.support.MultiResourcePartitioner"
p:resources="file:${spring.tmp.batch.dir}/#{jobParameters[batch_id]}/shares_rics/shares_rics_*.txt"
scope="step" />
<beans:bean id="computeFileItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader"
p:resource="#{stepExecutionContext[fileName]}" p:lineMapper-ref="stLineMapper"
scope="step" />
<beans:bean id="computeItemWriter"
class="com.st.batch.foundation.writers.ComputeItemWriter"
p:symfony-ref="symfonyStepScoped" p:timeout="${compute.item.timeout}"
p:batchId="#{jobParameters[batch_id]}" scope="step" />
<step id="computeStep">
<tasklet transaction-manager="transactionManager">
<chunk reader="computeFileItemReader" writer="computeItemWriter"
commit-interval="${compute.commit.interval}" />
</tasklet>
</step>
<flow id="computeFlow">
<step id="computeStep.master">
<partition partitioner="computePartitioner"
handler="computePartitionHandler" />
</step>
</flow>
<job id="computeJob" restartable="true">
<flow id="computeJob.computeFlow" parent="computeFlow" />
</job>
The problem is, in multiresouce partitioner I am passign the pattern to look for files and create partitions equal to number of files. But The directory is created at run time and conditionally.
I want this step to make successfully if the directory does not exist (input files not available) and proceed to next step.
Right now, the jobs just hangs, doest do anything. Neither it considers the step successfull not it throws exception so doesnt even fail. It just becomes idle at this step.
<beans:bean id="computePartitioner"
class="org.springframework.batch.core.partition.support.MultiResourcePartitioner"
p:resources="file:${spring.tmp.batch.dir}/#{jobParameters[batch_id]}/shares_rics/shares_rics_*.txt"
scope="step" />
Is there any way to handle this ? I just want to consider this step as successful and proced to next step.
Upate:
Just tested it with creating partitioned step which runs on local server instead of remote, if files does not exist, it by default marks step as completed. So the problem is not with MultiResourcePartitioner, the problem occurs when we use partitioned step to run on remote server with above configuration.
I guess, its aggregation logic which keeps waiting for response even though there are no step execution messages sent ? Is it due to default SequenceSizeReleaseStrategy which relies on IntegrationMessageHeaderAccessor.SEQUENCE_SIZE header attribute and as there is no messages at all aggregator can't access to SEQUENCE_SIZE ?
#MessageEndpoint
public class MessageChannelPartitionHandler implements PartitionHandler {
public Collection<StepExecution> handle(StepExecutionSplitter stepExecutionSplitter,
StepExecution masterStepExecution) throws Exception {
Set<StepExecution> split = stepExecutionSplitter.split(masterStepExecution, gridSize);
int count = 0;
if (replyChannel == null) {
replyChannel = new QueueChannel();
}//end if
for (StepExecution stepExecution : split) {
Message<StepExecutionRequest> request = createMessage(count++, split.size(), new StepExecutionRequest(
stepName, stepExecution.getJobExecutionId(), stepExecution.getId()), replyChannel);
if (logger.isDebugEnabled()) {
logger.debug("Sending request: " + request);
}
messagingGateway.send(request);
}
Message<Collection<StepExecution>> message = messagingGateway.receive(replyChannel);
if (logger.isDebugEnabled()) {
logger.debug("Received replies: " + message);
}
Collection<StepExecution> result = message.getPayload();
return result;
}
private Message<StepExecutionRequest> createMessage(int sequenceNumber, int sequenceSize,
StepExecutionRequest stepExecutionRequest, PollableChannel replyChannel) {
return MessageBuilder.withPayload(stepExecutionRequest).setSequenceNumber(sequenceNumber)
.setSequenceSize(sequenceSize)
.setCorrelationId(stepExecutionRequest.getJobExecutionId() + ":" + stepExecutionRequest.getStepName())
.setReplyChannel(replyChannel)
.build();
}
}
If there are no step execution request i.e. split count is 0, then it won't go inside forloop so wont send any messages but still after for loop it will wait to receive for response.
What could be the solution ?
Use a decider to check for directory and return CONTINUE or SKIP (or any other meaningful value you want)
I had raised issue on spring batch bug tracker Jira, https://jira.spring.io/browse/BATCH-2283.
Its issue with MessageChannelPartitionHandler which doesn't handle this scenario as of now. The temporary fix is to override MessageChannelPartitionHandler.handle, check the size of message set and return NULL as proposed in above raised ticket.
This is how the method should be, it worked for me.
public Collection<StepExecution> handle(StepExecutionSplitter stepExecutionSplitter,
StepExecution masterStepExecution) throws Exception {
Set<StepExecution> split = stepExecutionSplitter.split(masterStepExecution, gridSize);
if(split.size() == 0) {
return null;
}
int count = 0;
if (replyChannel == null) {
replyChannel = new QueueChannel();
}//end if
for (StepExecution stepExecution : split) {
Message<StepExecutionRequest> request = createMessage(count++, split.size(), new StepExecutionRequest(
stepName, stepExecution.getJobExecutionId(), stepExecution.getId()), replyChannel);
if (logger.isDebugEnabled()) {
logger.debug("Sending request: " + request);
}
messagingGateway.send(request);
}
if (logger.isDebugEnabled()) {
logger.debug("No message sent but waiting for reply: ");
}
Message<Collection<StepExecution>> message = messagingGateway.receive(replyChannel);
if (logger.isDebugEnabled()) {
logger.debug("Received replies: " + message);
}
Collection<StepExecution> result = message.getPayload();
return result;
}

Spring Integration, how can I pass inbound http request through outbound gateway?

I'm trying to implement some sort of proxy as part of my data flow, I want to receive a http-request on my inbound gateway and pass it through outbound gateway. I want preserve all query string parameters. My gateways configuration is:
<int:channel id="searchRequestChannel" />
<int:channel id="searchReplyChannel" />
<int-http:inbound-gateway id="searchRequestInboundGateway"
supported-methods="GET"
request-channel="searchRequestChannel"
reply-channel="searchReplyChannel"
path="/services/normalization"
reply-timeout="50000"
/>
<int-http:outbound-gateway id="searchServiceGateway"
http-method="GET"
request-channel="searchRequestChannel"
url="http://localhost:8080/query"
extract-request-payload="false"
expected-response-type="java.lang.String"
reply-timeout="50000"
charset="UTF-8"
/>
I expected that it would work as follows:
Client send request to the inbound gateway /services/normalization:
GET /services/normalization q=cat&exclude=black
Inbound gateway receives request and send it through searchRequestChannel to the outbound gateway.
Outbound gateway sends whole request to the external service:
GET /query q=cat&exclude=black
But on practice, outbound gateway sends empty request that does not contains any query arguments:
GET /query
So my question, what's easiest way to send the http-request that was accepted on inbound gateway through outbound gateway. In other words how can I implement simple proxy by spring integration tools?
This is a bit of a kludge, but works; the DispatcherServlet binds the request to the thread...
<int-http:inbound-gateway id="searchRequestInboundGateway"
supported-methods="GET"
request-channel="searchRequestEnricherChannel"
reply-channel="searchReplyChannel"
path="/services/normalization{queryString}"
reply-timeout="50000"
/>
<int:header-enricher input-channel="searchRequestEnricherChannel" output-channel="searchRequestChannel">
<int:header name="queryString"
expression="T(org.springframework.web.context.request.RequestContextHolder).requestAttributes.request.queryString" />
</int:header-enricher>
and then on the outbound side, use
<int-http:outbound-gateway id="searchServiceGateway"
http-method="GET"
request-channel="searchRequestChannel"
url="http://localhost:8080/query?{queryString}"
encode-uri="false"
extract-request-payload="false"
expected-response-type="java.lang.String"
reply-timeout="50000"
charset="UTF-8">
<uri-variable name="queryString" expression="headers.queryString" />
</int-http:outbound-gateway>
However, this won't work with 2.2.x and earlier because the query string is encoded on the outbound side (foo=bar&baz=qux becomes foo%3Dbar%26baz%3Dqux). In 3.0 we have added the ability to not encode the URI using an attribute by using encode-uri="false". This is not yet available in a release, but it's available in 3.0.0.BUILD-SNAPSHOT.
EDIT:
The above is a general solution that will work for all query strings; if you know the actual parameters, another solution would be to extract each parameter separately and rebuild the query string on the outbound side...
<int-http:inbound-gateway ... >
<int-http:header name="foo" expression="#requestParams.foo.get(0)"/>
<int-http:header name="baz" expression="#requestParams.baz.get(0)"/>
</int-http:inbound-gateway>
<int-http:outbound-gateway request-channel="requestChannel"
url="http://localhost:18080/http/receiveGateway?foo={foo}&baz={baz}"
http-method="POST"
expected-response-type="java.lang.String">
<int-http:uri-variable name="foo" expression="headers.foo"/>
<int-http:uri-variable name="baz" expression="headers.baz"/>
</int-http:outbound-gateway>
On the inbound side, it would be better if we offered the queryString as a first class expression variable #queryString.
Please feel free to open an 'Improvement' JIRA Issue
My own workaround solution is use a transformer that transforms parameters in the message payload (map of query string parameters) to prepared query string and use an url-expression in an outbound-gateway to avoid a query string encoding:
<bean id="payloadToQueryString"
class="com.dph.integration.PayloadToQueryStringTransformer" />
<int-http:inbound-gateway id="searchRequestInboundGateway"
supported-methods="GET"
request-channel="searchRequestChannel"
path="/services/normalization"
reply-timeout="50000" />
<int:transformer input-channel="searchRequestChannel"
output-channel="searchGatewayChannel"
ref="payloadToQueryString" method="transform" />
<int-http:outbound-gateway id="searchServiceGateway"
http-method="GET"
request-channel="searchGatewayChannel"
url-expression="'http://localhost:8080/query?' + payload"
expected-response-type="java.lang.String"
reply-timeout="50000"
charset="UTF-8">
</int-http:outbound-gateway>
PayloadToQueryStringTransformer class is:
public class PayloadToQueryStringTransformer extends AbstractTransformer {
#Override
protected Object doTransform(final Message<?> message) throws Exception {
return MessageBuilder
.withPayload(urlEncodeUTF8(((MultiValueMap) message.getPayload()).toSingleValueMap()))
.copyHeaders(message.getHeaders())
.build();
}
private static String urlEncodeUTF8(final String s) {
try {
return URLEncoder.encode(s, "UTF-8");
} catch (final UnsupportedEncodingException e) {
throw new UnsupportedOperationException(e);
}
}
private static String urlEncodeUTF8(final Map<?,?> map) {
final StringBuilder sb = new StringBuilder();
for (final Map.Entry<?,?> entry : map.entrySet()) {
if (sb.length() > 0) {
sb.append("&");
}
sb.append(String.format("%s=%s",
urlEncodeUTF8(entry.getKey().toString()),
urlEncodeUTF8(entry.getValue().toString())
));
}
return sb.toString();
}
}

Resources