I am using Tomcat and I have http:inbound-gateway as gateway and channel named request configured as it's request channel. Sometimes there are lots of requests on my service which cause occurring Too many open files Error which comes from OS.
I tried to make request channel a QueChannel and set capacity for it but it does not work. Then I tried to set request channel as a polling-consumer channel and set a poller with fixed-delay to poll and it did not work again. Is there a conventional way to limit number of input requests?
And how can I customize response in discarded requests?
<int:channel id="request">
<int:queue capacity="100"/>
</int:channel>
...
<int-http:inbound-gateway id="RESTServiceGateway"
supported-methods="GET"
request-channel="request"
error-channel="errorResolver" />
<int:chain input-channel="request" output-channel="response">
<int:poller fixed-delay="1" max-messages-per-poll=""/>
...
Or
<task:executor id="requestExecutor" pool-size="1-10" queue-capacity="10"/>
<int:channel id="request">
<int:dispatcher task-executor="requestExecutor"/>
</int:channel>
<int-http:inbound-gateway id="RESTServiceGateway"
supported-methods="GET"
request-channel="request"
error-channel="errorResolver" />
<int:chain input-channel="request" output-channel="response">
...
Too many open files is not related to your integration flow configuration. This is about opened sockets from the HTTP clients to your Tomcat. This one is already concurrent and can handle many requests in parallel. Therefore I would say that your paralleling logic in the flow doesn't bring too much value and definitely doesn't effect a number of opened sockets.
You can configure concurrency in Tomcat though: maxConnections in server.xml .
Another option is to increase an amount for those opened file on your Linux using ulimit tool: https://www.tecmint.com/increase-set-open-file-limits-in-linux/
Related
There is one request queue and the reply queues are created by the client server instances and pinned to each instance rather than using temporary queues.
The use case needs to get an inbound jms message and then send that message to an asynchronous process. Once the async reply messsage is received from the service I need to take those results and reply back to the original message's jmsReplyTo. The Jms gateway would not work in this instance AFAIK>
I am using a jms message driven channel adapter for the message in with a series of channels and service activators to handle the out of process calls and async replies. I am trying to use the DynamicDestinationResolver to no avail. Additionally I have tried to set the outbound destination address programatically but could not figure out a good way to do this.
This seems like a common pattern but I could not find a good example for a completely disconnected async request response. Disconnected meaning that the usual async jms request reply did not seem to fit the need.
Context Config:
<!-- Input from Amq -->
<amq:queue id="requestQueue" physicalName="${request.queue}" />
<int-jms:message-driven-channel-adapter id="jmsIn"
connection-factory="jmsConnectionFactory"
destination="requestQueue"
channel="queueRequestChannel" concurrent-consumers="5" />
<int:channel id="queueRequestChannel" />
<int:service-activator input-channel="queueRequestChannel" ref="switchMessageHandler" method="processSwitchMessage"
output-channel="cardNetworkOutChannel"/>
<!-- Output to Card Network-->
<int:channel id="cardNetworkOutChannel" />
<!--<int:service-activator input-channel="cardNetworkOutChannel" ref="cardNetworkHandler" method="send8583Message" />-->
<!-- Simply used to mock the card network by transforming a SwithMessage to a SwitchMessageResponse * Not needed for target solution -->
<int:transformer id="requestResponseTransformer" ref="nettyCardNetworkClientMock" input-channel="cardNetworkOutChannel"
method="process" output-channel="cardNetworkInChannel"/>
<!-- Input from Card Network -->
<int:channel id="cardNetworkInChannel" />
<int:service-activator input-channel="cardNetworkInChannel" ref="switchMessageHandler" method="sendSwitchMessage"
output-channel="queueReplyChannel"/>
<int:channel id="queueReplyChannel"/>
<int-jms:outbound-channel-adapter
destination-resolver="simpleDestinationResolver" connection-factory="jmsConnectionFactory"
channel="queueReplyChannel" destination-expression="headers.jms_replyTo" />
I just updated the jms sample app to make the server side use independent adapters instead of the inbound gateway and it works just fine...
<!-- <jms:inbound-gateway id="jmsin" -->
<!-- request-destination="requestQueue" -->
<!-- request-channel="demoChannel"/> -->
<channel id="demoChannel"/>
<jms:message-driven-channel-adapter destination="requestQueue" channel="demoChannel" />
<service-activator input-channel="demoChannel" ref="demoBean" output-channel="reply" />
<channel id="reply" />
<jms:outbound-channel-adapter channel="reply" destination-expression="headers['jms_replyTo']" />
Turn on DEBUG logging - we put out lots of useful stuff.
EDIT
I just made it async by...
<channel id="reply">
<queue/>
</channel>
<jms:outbound-channel-adapter channel="reply" destination-expression="headers['jms_replyTo']">
<poller fixed-delay="3000"/>
</jms:outbound-channel-adapter>
EDIT2
Depending on what you are using on the client side, many clients require the inbound message id to be used as the correlation id. (This is true for the outbound gateway by default, with named reply queue, unless you provide a correlation-key).
So, to set up the correlationId, you can use a header enricher; I just tested this...
<chain input-channel="reply">
<header-enricher>
<header name="jms_correlationId" expression="headers['jms_messageId']" />
</header-enricher>
<jms:outbound-channel-adapter destination-expression="headers['jms_replyTo']"/>
<poller fixed-delay="1000" />
</chain>
This is not the issue if the client side is setting the correlation id header itself.
I am trying to poll local directory using ftp inbound adapter to fetch files for further processing. I want to move the file to another local directory and delete it from origination. Not getting a way to achieve it. Here's what I have so far:
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel" session-factory="ftpClientFactory"
filename-pattern="*.xml" auto-create-local-directory="false"
delete-remote-files="false" remote-directory="/" local-directory="//C://FBS//testmq">
<int:poller fixed-rate="20000" />
</int-ftp:inbound-channel-adapter>
<int:channel id="ftpChannel">
<int:queue />
</int:channel>
Use transaction synchronization with a pseudo transaction manager; see the file example in the documentation. Here's the configuration from that section of the docs:
<int-file:inbound-channel-adapter id="inputDirPoller"
channel="someChannel"
directory="/foo/bar"
filter="filter"
comparator="testComparator">
<int:poller fixed-rate="5000">
<int:transactional transaction-manager="transactionManager" synchronization-factory="syncFactory" />
</int:poller>
</int-file:inbound-channel-adapter>
<int:transaction-synchronization-factory id="syncFactory">
<int:after-commit expression="payload.renameTo(new java.io.File('/success/' + payload.name))"
channel="committedChannel" />
<int:after-rollback expression="payload.renameTo(new java.io.File('/failed/' + payload.name))"
channel="rolledBackChannel" />
</int:transaction-synchronization-factory>
Continue reading into the next section...
Referring to the above section, you may be thinking it would be useful to take these 'success' or 'failure' actions when a flow completes, even if there is no 'real' transactional resources (such as JDBC) downstream of the poller. For example, consider a followed by an ftp:outbout-channel-adapter/. Neither of these components is transactional but we might want to move the input file to different directories, based on the success or failure of the ftp transfer.
To provide this functionality, the framework provides a PseudoTransactionManager, enabling the above configuration even when there is no real transactional resource involved. If the flow completes normally, the beforeCommit and afterCommit synchronizations will be called, on failure the afterRollback will be called. Of course, because it is not a real transaction there will be no actual commit or rollback. The pseudo transaction is simply a vehicle used to enable the synchronization features.
I'm relatively new to spring integration but I have been tasked with implementing a tcp gateway that needs to:
Listen for messages on a socket
Receive a message
Process the message and write some data to a queue
Return a response message to the original socket.
In my experience with spring integration, the message flow has not been bi-directional. I've only configured routers to listen, handle messages, and output to queue/topic. In this case, though, I need to accept messages and return a response while also forwarding on some message to a queue. Suggestions?
This is my integration xml so far.
<int:chain input-channel="tcpChannel">
<int:service-activator ref="tcpHandler" method="handleInput" />
</int:chain>
<int-ip:tcp-inbound-gateway
id="tcpGateway"
connection-factory="tcpServer"
request-channel="tcpChannel" />
How can I have the output of handleInput be forwarded to a queue but also have some response sent back from the gateway?
Edit: After the conversation below with Gary, this seems to be the pattern we want to follow:
<int-ip:tcp-inbound-gateway id="tcpGateway"
connection-factory="tcpServer"
request-channel="tcpChannel"
reply-channel="tcpReplyChannel"/>
<int:publish-subscribe-channel id="tcpChannel" />
<int:chain input-channel="tcpChannel">
<!-- int:json-to-object-transformer type="com.heb.revo.events.RxPosCredit" /-->
<int:service-activator ref="tcpHandler" method="handleInputToQueue" />
<jms:outbound-channel-adapter destination-name="${queue.response}" />
</int:chain>
<int:service-activator id="tcpResponseHandler"
ref="tcpHandler" method="replyToSocket"
input-channel="tcpChannel"
output-channel="tcpReplyChannel" />
<int:publish-subscribe-channel id="tcpReplyChannel" />
Since your chain has no output-channel the framework will route the return value of the handleInput method back to the gateway automatically.
If you want to capture the result and send it somewhere else (as well as a reply), create a <int:publish-subscribe-channel id="foo"/>, set the output-channel of the chain to foo, set the reply-channel on the gateway to foo, and subscribe another endpoint to foo (as an input-channel.
I have a production application that seems to be losing messages intermittently. One example that I have identified is the sending system logs 3 messages that are sent milliseconds apart. The receiving system logs that it got message 1 and message 3...but not message 2. The queue's depth is 0. I see no errors in the logs of the application that reads the messages to indicate something bad happened.
I have verified that there are no other "rogue" clients creating a race condition for reading messages.
Here is my configuration; we have a primary and secondary/failover queue...
<bean id="primaryProficiencyInboundQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueManagerName" value="${lsm.primary.outbound.manager}"/>
<property name="baseQueueName" value="${lsm.proficiency.inbound.queue}"/>
</bean>
<bean id="secondaryProficiencyInboundQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueManagerName" value="${lsm.secondary.outbound.manager}"/>
<property name="baseQueueName" value="${lsm.proficiency.inbound.queue}"/>
</bean>
<int:channel id="inboundProficiencyChannel">
<int:interceptors>
<int:wire-tap channel="logger" />
</int:interceptors>
</int:channel>
<!-- adapter that connects the inbound proficiency queues to the inboundProficiencyChannel -->
<jms:message-driven-channel-adapter id="primaryProficiencyInboundAdapter"
connection-factory="primaryConnectionFactory"
destination="primaryProficiencyInboundQueue"
channel="inboundProficiencyChannel" />
<jms:message-driven-channel-adapter id="secondaryProficiencyInboundAdapter"
connection-factory="secondaryConnectionFactory"
destination="secondaryProficiencyInboundQueue"
channel="inboundProficiencyChannel" />
<int:service-activator id="proficiencyInboundActivator"
input-channel="inboundProficiencyChannel">
<bean class="com.myactivator.LSMInboundProficiencyActivator" />
</int:service-activator>
Is it possible that something is failing silently?
One other noteworthy thing is that there are two production servers that are both configured as above....meaning that there are two servers that have JMS adapters configured to read the messages. I mentioned above that I see message 1 and message 3 being read. In that case, I see server one process message one and server two process mesage two...so it seems that having two servers configured like above has no negative impact. Any other thoughts on what could be happening or how I can debug the case of the missing/dropped messages?
Version information:
Spring - 3.0.5.RELEASE
Spring Integration - 2.0.3.RELEASE
Nothing springs to mind that would cause lost messages; I don't recall that ever being reported; but 2.0.3 is quite ancient (3rd birthday coming up soon). We don't support 2.0.x any more but you should at least update to 2.0.6 which is the last release of 2.0.x. But preferably move up to the latest 3.0.0 release (or at least 2.2.6).
we are using Spring Integration 2.1 for persisting messages into database sent by clients.
There is a queue which will be filled by a custom adapter. The configured service activator polls this queue and releases the message to a spring managed #Repository bean. All errors will be captured to an error channel and will be handled by a service. The configuration works so far fine.
My concern is that if the database is not available the service-activators polls all incoming message from the queue and puts them into the error channel. Is there a way to prevent the service-activator to poll the message if the database is obviously not available, for example by sending a test query ?
My configuraton:
<int:channel id="inChannel">
<int:queue />
</int:channel>
<bean id="service" class="some.service.Service" />
<int:service-activator ref="service"
method="write" input-channel="inChannel">
<int:poller fixed-rate="100" task-executor="srvTaskExecutor"
receive-timeout="90" error-channel="errChannel" />
</int:service-activator>
<task:executor id="srvTaskExecutor" pool-size="2-10"
queue-capacity="0" rejection-policy="DISCARD" />
<int:channel id="errChannel" />
<int:service-activator input-channel="errChannel"
ref="errorService" method="write"/>
Regards.
If you give the polling service-activator an "id", you can refer to that instance and call start() or stop() on it based on the DB being available or not. Most likely you'd want to set auto-startup="false" on that service-activator as well.
Additionally, you can even define a "control-bus" element and then send messages like "myActivator.start()" and "myActivator.stop()" to that control bus' input-channel.
Hope that helps,
Mark