I use spring integration to execute 2 spring batch executable jar files wrapped in groovy script. If spring integration configured to execute first groovy script and then second it executes only first script and does not executes second. If I reverse it in spring integration configuration it executes second groovy script and does not execute first. My integration configuration is as follows
<int:logging-channel-adapter id="logger" log-full-message="true" level="INFO" logger-name="org.springframework"/>
<int:wire-tap channel="logger" />
<context:mbean-server />
<int-jmx:mbean-export/>
<int-file:inbound-channel-adapter id="getFile" directory="file:${xmlFileInputDirectory}"
prevent-duplicates="true" channel="channel1" filename-pattern="Data*">
<int:poller id="poller" max-messages-per-poll="1" fixed-rate="5000">
</int:poller>
</int-file:inbound-channel-adapter>
<int:channel id="channel1">
<int:interceptors>
<int:wire-tap channel="logger"/>
</int:interceptors>
</int:channel>
<int:channel id="channe2">
<int:interceptors>
<int:wire-tap channel="logger"/>
</int:interceptors>
</int:channel>
<int:service-activator input-channel="channel1" output-channel="channe2">
<int-groovy:script>
<![CDATA[
def proc = "java -Dfile.encoding=utf8 -Dlogging.config=log4j2.xml -jar one.jar".execute();
proc.waitForProcessOutput(System.out, System.err);
]]>
</int-groovy:script>
</int:service-activator>
<int:service-activator input-channel="channe2">
<int-groovy:script>
<![CDATA[
def proc = "java -Dfile.encoding=utf8 -Dlogging.config=log4j2.xml -jar two.jar --data.dictionary.path=data.csv".execute();
proc.waitForProcessOutput(System.out, System.err);
]]>
</int-groovy:script>
</int:service-activator>
<int:message-history/>
<bean id="mainClass"
class="com.company.Main" />
My main class that starts first batch job looks like this:
public final class Main {
private Main() { }
public static void main(final String... args) {
final AbstractApplicationContext context =
new ClassPathXmlApplicationContext("classpath:META-INF/int_file");
context.start();
}
}
My first batch job main class looks like this:
public class Application {
public static void main(String[] args)
throws ... {
SpringApplication.run(Application.class, args);
System.exit(0);
}
}
Related
[EDIT] Uploading complete configs:
rabbit.xml which dequeues from rabbit
<rabbit:connection-factory id="amqpConnectionFactoryInbound"
host="${rabbit.host}" port="${rabbit.port}"
username="${rabbit.username}" password="${rabbit.password}" channel-
cache-size="5"
connection-factory="rabbitConnectionFactoryInbound"/>
<beans:bean id="rabbitConnectionFactoryInbound"
class="com.rabbitmq.client.ConnectionFactory">
<beans:property name="requestedHeartbeat"
value="60" />
</beans:bean>
<!-- Inbound Adapter to AMQP RabbitMq and write to file -->
<int-amqp:inbound-channel-adapter id="rabbitMQInboundChannelAdapter"
channel="rabbitInboundMessageChannel"
concurrent-consumers="8" task-
executor="rabbit-executor" connection-
factory="amqpConnectionFactoryInbound"
message-converter="byteArrayToStringConverter" queue-
names="${rabbit.queue}" acknowledge-mode="MANUAL" error-
channel="errorChannelId"
prefetch-count="25" />
<header-enricher input-channel="rabbitInboundMessageChannel" output-
channel="rabbitOutboundboundMessageChannel">
<int:header name="Operation" value="${operation.rabbit}" />
<int:header name="GUID" expression="#{
'T(java.util.UUID).randomUUID().toString()' }" />
<int:header name="operationStartTime" expression="#{
'T(java.lang.System).currentTimeMillis()' }" />
</header-enricher>
<int:channel id="rabbitOutboundboundMessageChannel">
<int:interceptors>
<int:wire-tap channel="loggerChannel" />
</int:interceptors>
</int:channel>
<task:executor id="rabbit-executor" rejection-policy="CALLER_RUNS"
pool-size="10-30"
queue-capacity="25" />
</beans:beans>
The message is then sent to router channel: router.xml
<int:header-enricher input-channel="rabbitOutboundboundMessageChannel"
output-channel="routerChannel">
<int:header name="Operation" value="${operation.router}"
overwrite="true" />
<int:header name="file_name" expression="headers['GUID'] + '.xml'" />
<int:header name="operationStartTime" expression="#{
'T(java.lang.System).currentTimeMillis()' }"
overwrite="true" />
<int:error-channel ref="errorChannelId" />
</int:header-enricher>
<int:recipient-list-router id="rabbitMsgrouter" input-
channel="routerChannel">
<int:recipient channel="fileBackupChannel" selector-expression="new
String(payload).length()>0" />
<int:recipient channel="transformerChannel" />
</int:recipient-list-router>
<int:channel id="transformerChannel">
<int:interceptors>
<int:wire-tap channel="loggerChannel" />
</int:interceptors>
</int:channel>
<int:channel id="fileBackupChannel"/>
<int:channel id="loggerChannel"/>
</beans>
The message is now sent to persister.xml and transformer.xml. The following is persister.xml and I want to ack if persistence is successful. There are other downstream processes after transformer.xml
<int:header-enricher input-channel="fileBackupChannel" output-
channel="fileSaveChannel">
<int:header name="Operation" value="${operation.filePersister}"
overwrite="true" />
<int:header name="replyChannel" value="nullChannel" />
<int:header name="operationStartTime" expression="#{
'T(java.lang.System).currentTimeMillis()' }" />
<int:error-channel ref="errorChannelId" />
</int:header-enricher>
<int-file:outbound-gateway id="fileBackUpChannelAdapter"
directory="${file.location}"
request-channel="fileSaveChannel" reply-channel="rabbitAckChannel"/>
<int:service-activator input-channel="rabbitAckChannel" output-
channel="nullChannel" ref="ackRabbit" method="handleRabbitAcks" />
<bean id="ackRabbit"
class="com.expedia.dataloader.rabbit.RabbitAcknowledgement"/>
<int:channel id="rabbitAckChannel">
<int:interceptors>
<int:wire-tap channel="loggerChannel" />
</int:interceptors>
</int:channel>
<int:channel id="loggerChannel"/>
<int:channel id="fileSaveChannel"/>
</beans>
I'm having trouble manually acking payloads from rabbitmq.
This is my work flow:
1. Get message from rabbit using inbound-channel-adapter:
<int-amqp:inbound-channel-adapter id="rabbitMQInboundChannelAdapter"
channel="rabbitInboundMessageChannel"
concurrent-consumers="${rabbit.concurrentConsumers}" task-
executor="rabbit-executor" connection-
factory="amqpConnectionFactoryInbound"
message-converter="byteArrayToStringConverter" queue-
names="${rabbit.queue}" acknowledge-mode="MANUAL" error-
channel="errorChannelId"
prefetch-count="${rabbit.prefetchCount}" />
2. Persist message to disk using outbound-gateway:
<int-file:outbound-gateway id="fileBackUpChannelAdapter"
directory="${file.location}"
request-channel="fileSaveChannel" reply-channel="loggerChannel" />
3. ack from rabbit when persister (step 2) succeeds.
for step (3), i wrote the following code:
public class RabbitAcknowledgement {
public void handleRabbitAcks(Message<?> message) throws IOException {
com.rabbitmq.client.Channel channel = (Channel)
message.getHeaders().get("amqp_channel");
long deliveryTag = (long) message.getHeaders().get("amqp_deliveryTag");
channel.basicAck(deliveryTag, false);
}
which I'm calling from spring via:
<int:service-activator input-
channel="rabbitOutboundboundMessageChannel" output-
channel="routerChannel" ref="ackRabbit" method="handleRabbitAcks" />
This doesn't work and the the rabbit payloads in my queue are not acked.
My questions are:
Do I need MANUAL ack in this scenario?
What am I doing wrong?
It should work fine; I just ran a quick test and it works for me...
#SpringBootApplication
public class So44666444Application implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(So44666444Application.class, args).close();
}
#Autowired
private RabbitTemplate template;
private final CountDownLatch latch = new CountDownLatch(1);
#Override
public void run(String... args) throws Exception {
this.template.convertAndSend("foo", "bar");
latch.await();
}
#Bean
public AmqpInboundChannelAdapter adapter(ConnectionFactory cf) {
AmqpInboundChannelAdapter adapter = new AmqpInboundChannelAdapter(listenerContainer(cf));
adapter.setOutputChannelName("ack");
return adapter;
}
#Bean
public AbstractMessageListenerContainer listenerContainer(ConnectionFactory cf) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(cf);
container.setAcknowledgeMode(AcknowledgeMode.MANUAL);
container.setQueueNames("foo");
return container;
}
#ServiceActivator(inputChannel = "ack")
public void ack(#Header(AmqpHeaders.CHANNEL) Channel channel, #Header(AmqpHeaders.DELIVERY_TAG) Long tag)
throws IOException {
System.out.println("Acking: " + tag);
channel.basicAck(tag, false);
latch.countDown();
}
}
If I set a breakpoint on the basicAck, I see the message as unacked on the console; stepping over to the next line and the message is removed.
I would like to convert my XML config to Java dsl, but I can not find an example for int-ip:udp-inbound-channel-adapter in java dsl. There is TcpNetServerConnectionFactory for TCP but nothing for UDP. Only that class is subclassing AbstractServerConnectionFactory. XML config related to spring integration is bellow.
<int-ip:udp-inbound-channel-adapter
id="receiverChannel" channel="udpReceivedChannel" port="1206" multicast="false"
check-length="false" lookup-host="false" pool-size="20"/>
<int:transformer id="convertTransformer" input-channel="convertChannel"
output-channel="toProcessChannel" ref="transformer" method="transform">
</int:transformer>
<int:service-activator id="accumulateActivator" input-channel="udpReceivedChannel"
output-channel="convertChannel"
ref="accumulator" method="accumulate">
</int:service-activator>
<int:service-activator id="cssenderAcivator" input-channel="sendToCMSChannel"
ref="cssender" method="sendToCS">
</int:service-activator>
<int:service-activator id="jackpotRaiseActivator" input-channel="toProcessChannel"
ref="jackpotraise" method="raise" >
</int:service-activator>
<int:service-activator id="jackpotScreenActivator" input-channel="jackpotScreenChannel"
ref="jackpotscreen" method="updateJackpotsOnDisplay" >
</int:service-activator>
<int:channel id="udpReceivedChannel">
<int:dispatcher task-executor="accumulateExecutor"/>
<!--<int:queue message-store="redisMessageStore"/>--> <!-- ovo nam ne treba bez da nesto externo trpa u redis-->
</int:channel>
<int:channel id="toProcessChannel">
<int:dispatcher task-executor="jackpotRaiseExecutor"/>
<int:interceptors>
<int:wire-tap channel="sendToCMSChannel"/>
</int:interceptors>
</int:channel>
<int:channel id="convertChannel">
<int:dispatcher task-executor="transformerExecutor"/>
</int:channel>
<int:channel id="sendToCMS">
<int:dispatcher task-executor="cmsSenderExecutor"/>
</int:channel>
<int:channel id="jackpotScreenChannel">
<int:dispatcher task-executor="jackpotScreenExecutor"/>
</int:channel>
<task:executor id="accumulateExecutor" pool-size="20" keep-alive="120" />
<task:executor id="jackpotRaiseExecutor" pool-size="20" keep-alive="120" />
<task:executor id="transformerExecutor" pool-size="20" keep-alive="120" />
<task:executor id="cmsSenderExecutor" pool-size="20" keep-alive="120" />
<task:executor id="jackpotScreenExecutor" pool-size="400" keep-alive="500" />
Here's a quick boot app that creates an adapter and sends a packet to it...
#SpringBootApplication
public class So40286815Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So40286815Application.class, args);
DatagramSocket socket = new DatagramSocket();
DatagramPacket packet = new DatagramPacket("foo".getBytes(), 3);
packet.setAddress(InetAddress.getLocalHost());
packet.setPort(1206);
socket.send(packet);
Thread.sleep(10000);
socket.close();
context.close();
}
#Bean
public UnicastReceivingChannelAdapter inbound() {
UnicastReceivingChannelAdapter adapter = new UnicastReceivingChannelAdapter(1206);
adapter.setOutputChannelName("foo");
return adapter;
}
#ServiceActivator(inputChannel = "foo")
public void handle(byte[] bytes) {
System.out.println(new String(bytes));
}
}
and with the DSL...
#Bean
public UnicastReceivingChannelAdapter inbound() {
return new UnicastReceivingChannelAdapter(1206);
}
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(inbound())
.handle(System.out::println)
.get();
}
or, simply...
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(new UnicastReceivingChannelAdapter(1206))
.transform(new ObjectToStringTransformer())
.handle(m -> System.out.println(m.getPayload()))
.get();
}
I need to get all files from all directories and I want to use sftp:outbound-gateway with command = "mget" (of course with filters).
The sample code
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-sftp="http://www.springframework.org/schema/integration/sftp"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/sftp http://www.springframework.org/schema/integration/sftp/spring-integration-sftp.xsd">
<bean id="sftpSessionFactory" class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="host" />
<property name="user" value="username" />
<property name="password" value="pass" />
</bean>
<int:channel id="output">
<int:queue />
</int:channel>
<int:channel id="inboundMGetRecursive" />
<int:poller default="true" fixed-delay="5000">
</int:poller>
<int-sftp:outbound-gateway session-factory="sftpSessionFactory"
request-channel="inboundMGetRecursive"
command="mget"
expression="payload"
command-options="-R"
local-directory-expression="#remoteDirectory"
reply-channel="output"
remote-directory="${remote-directory}"/>
</beans>
works only when a code in java creates a message and send it.
public final class Main {
private static final Logger LOGGER = Logger.getLogger(Main.class);
private Main() { }
/**
* Load the Spring Integration Application Context
*
* #param args - command line arguments
*/
public static void main(final String... args) {
final AbstractApplicationContext context =
new ClassPathXmlApplicationContext("/main/resources/META-INF/spring/integration/spring-integration-context.xml");
context.registerShutdownHook();
final DirectChannel requestChannel = (DirectChannel) context.getBean("inboundMGetRecursive");
final PollableChannel replyChannel = (PollableChannel) context.getBean("output");
String dir = "/remote_directory/test/sample";
requestChannel.send(new GenericMessage<Object>(dir + "'*'"));
Message<?> result = replyChannel.receive(1000);
List<File> localFiles = (List<File>) result.getPayload();
for (File file : localFiles) {
System.out.println(file.getName());
}
System.exit(0);
}
}
I want the code to be called on start up with out a java code, like the
sftp:inbound-channel-adapter works.
Can it be done? if yes, is there a sample code or an example?
<int:inbound-channel-adapter channel="inboundMGetRecursive"
expression="'/remote_directory/test/sample'">
<int:poller fixed-delay="60000" />
</int:inbound-channel-adapter>
<int:sftp-outbound-gateway ...
Will invoke the gateway once a minute.
If you want to invoke it once, and once only, use a FireOnceTrigger as discussed in this answer
<int:poller trigger="myTrigger" />
am trying to implement Spring RabbitMq with XML based instead of configuration file. The message is sent ( I can see it in the RabbitMQ Admin Management ) but the consumer is not printing any output. Consumer is initialized ( can confirm that the constructor is called ). Can you please let me know what is the problem with the below:
rabbitConfiguration.xml:
<rabbit:connection-factory id="connectionFactory" host="localhost" username="guest" password="guest" />
<rabbit:template id="amqpTemplate" connection-factory="connectionFactory" exchange="my.exchange" routing-key="my.test.1" />
<rabbit:admin connection-factory="connectionFactory" />
<rabbit:queue id="myQueue" name="my.queue" />
<rabbit:topic-exchange id="myExchange" name="my.exchange">
<rabbit:bindings>
<rabbit:binding queue="myQueue" pattern="my.#.*">
</rabbit:binding>
</rabbit:bindings>
</rabbit:topic-exchange>
<rabbit:listener-container id="myListenerContainer" connection-factory="connectionFactory" >
<rabbit:listener ref="aListener" method="printer" queues="myQueue"/>
</rabbit:listener-container>
<bean id="aListener" class="rabbitmq.MyReceiver" />
rabbitmq.MyReceiver.java
Public class MyReceiver {
public MyReceiver() {
System.out.println("init..");
}
public void printer(String msg){
System.out.println("message: " + msg);
}
}
Here is Producer code in different Class:
ConfigurableApplicationContext context = new ClassPathXmlApplicationContext("rabbitConfiguration.xml");//loading beans
RabbitTemplate rb = (RabbitTemplate) context.getBean("amqpTemplate");
rb.convertAndSend("blah balh");
context.close();
The workflow of my application is adding a message header, then routing the message depending on another header and sending the message further downstream. My tests for this workflow succeed individually but not when executed together, although it seems that I have reset all relevant objects.
The spring integration workflow:
<int:header-enricher input-channel="workflowStart" output-channel="headerEnriched">
<int:header name="correlationId" expression="headers.id.toString()" />
</int:header-enricher>
<int:header-value-router input-channel="headerEnriched" header-name="messageType">
<int:mapping value="stp" channel="stpChannel" />
<int:mapping value="nonStp" channel="nonStpChannel" />
</int:header-value-router>
<int:publish-subscribe-channel id="nonStpChannel" />
<int:publish-subscribe-channel id="stpChannel" apply-sequence="true"/>
<int:chain input-channel="nonStpChannel" output-channel="systemChannel1" id="nonStpChainBlockA">
<!-- do something -->
</int:chain>
<int:chain input-channel="stpChannel" output-channel="systemChannel3" id="stpChainBlockA">
<!-- do something -->
</int:chain>
The Java test class:
#Autowired
private MessageChannel workflowStart;
#Autowired
private MessageHandler systemChannel1OutputHandler;
#Autowired
private MessageHandler systemChannel3OutputHandler;
#Value("/xml/tradeStp.xml")
private Resource stpMessageResource;
private String stpMessage;
#Value("/xml/tradeNonStp.xml")
private Resource nonStpMessageResource;
private String nonStpMessage;
#Before
public void setup() throws Exception {
reset(systemChannel1OutputHandler);
reset(systemChannel3OutputHandler);
stpMessage = IOUtils.toString(stpMessageResource.getInputStream());
nonStpMessage = IOUtils.toString(nonStpMessageResource.getInputStream());
}
#Test
public void nonStpMessageWorkflow1Test() {
Message<String> m = MessageBuilder
.withPayload(nonStpMessage)
.setHeaderIfAbsent("messageType", "nonStp")
.build();
workflowStart.send(m);
verify(systemChannel1OutputHandler, times(1)).handleMessage(any(Message.class));
}
#Test
public void stpMessageWorkflow2Test() {
Message<String> m = MessageBuilder
.withPayload(stpMessage)
.setHeaderIfAbsent("messageType", "stp")
.build();
workflowStart.send(m);
verify(systemChannel3OutputHandler, times(1)).handleMessage(any(Message.class));
}
And the test context:
<import resource="classpath:/spring/workflow.xml"/>
<int:outbound-channel-adapter channel="systemChannel1" ref="systemChannel1OutputHandler" method="handleMessage"/>
<bean id="systemChannel1OutputHandler" class="org.mockito.Mockito" factory-method="mock">
<constructor-arg value="org.springframework.integration.core.MessageHandler"/>
</bean>
<int:outbound-channel-adapter channel="systemChannel3" ref="systemChannel3OutputHandler" method="handleMessage"/>
<bean id="systemChannel3OutputHandler" class="org.mockito.Mockito" factory-method="mock">
<constructor-arg value="org.springframework.integration.core.MessageHandler"/>
</bean>
<int:outbound-channel-adapter channel="headerEnriched" ref="headerEnricherOutputHandler" method="handleMessage"/>
<bean id="headerEnricherOutputHandler" class="org.mockito.Mockito" factory-method="mock">
<constructor-arg value="org.springframework.integration.core.MessageHandler"/>
</bean>
If I run both tests together, the first test succeeds, while the second one fails with the message:
Wanted but not invoked:
messageHandler.handleMessage(<any>);
-> at com.company.integration.com.company.export.config.InterfaceTest.stpMessageWorkflow2Test(InterfaceTest.java:81)
Actually, there were zero interactions with this mock.
I have tried to debug in various ways. The problem seems to be just after the header-value-router and before the chain. The last debug output for the second test is a "postSend" on the headerEnriched channel.
Any advice is very welcome.
UPDATE:
I failed to include an additional outputHandler that I have in my context for another test, which I have included now. By removing the headerEnricherOutputHandler all tests run fine. However, I still don't understand the reason why this causes problems when tests are being run together.
I've had problems with Mockito using any(class). Where you have
verify(systemChannel3OutputHandler, times(1)).handleMessage(any(Message.class));
does it work with if you remove the class, like:
verify(systemChannel3OutputHandler, times(1)).handleMessage(any());
I cannot reproduce your issue with the following, which does not appear materially different to your tests...
<int:header-value-router input-channel="headerEnriched" header-name="messageType">
<int:mapping value="stp" channel="stpChannel" />
<int:mapping value="nonStp" channel="nonStpChannel" />
</int:header-value-router>
<int:publish-subscribe-channel id="nonStpChannel" />
<int:publish-subscribe-channel id="stpChannel" apply-sequence="true"/>
<int:chain input-channel="stpChannel" output-channel="systemChannel1">
<int:transformer expression="'bar'"/>
</int:chain>
<int:chain input-channel="nonStpChannel" output-channel="systemChannel3">
<int:transformer expression="'bar'"/>
</int:chain>
<int:channel id="systemChannel1" />
<int:channel id="systemChannel3" />
<int:outbound-channel-adapter channel="systemChannel1" ref="systemChannel1OutputHandler" method="handleMessage"/>
<bean id="systemChannel1OutputHandler" class="org.mockito.Mockito" factory-method="mock">
<constructor-arg value="org.springframework.integration.core.MessageHandler"/>
</bean>
<int:outbound-channel-adapter channel="systemChannel3" ref="systemChannel3OutputHandler" method="handleMessage"/>
<bean id="systemChannel3OutputHandler" class="org.mockito.Mockito" factory-method="mock">
<constructor-arg value="org.springframework.integration.core.MessageHandler"/>
</bean>
.
#ContextConfiguration
#RunWith(SpringJUnit4ClassRunner.class)
public class Foo {
#Autowired
private MessageChannel headerEnriched;
#Autowired
private MessageHandler systemChannel1OutputHandler;
#Autowired
private MessageHandler systemChannel3OutputHandler;
#Test
public void test10() {
Message<String> foo = MessageBuilder.withPayload("foo")
.setHeader("messageType", "stp").build();
headerEnriched.send(foo);
verify(systemChannel1OutputHandler).handleMessage(any(Message.class));
}
#Test
public void test30() {
Message<String> foo = MessageBuilder.withPayload("foo")
.setHeader("messageType", "nonStp").build();
headerEnriched.send(foo);
verify(systemChannel3OutputHandler).handleMessage(any(Message.class));
}
}
Both tests run fine for me.