I'm reading the Spring Integration documentation to figure out how to implment a bridge with polleable and subscribable channels (which will be one of my SpringXD modules).
So the documentation related to the bridge says that basically, I should do something like:
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd">
<channel id="pollable" />
<bridge input-channel="pollable" output-channel="subscribable">
<poller max-messages-per-poll="10" fixed-rate="5000" />
</bridge>
<channel id="subscribable" />
Then I'm creating a SpringXD Stream like:
stream create source-customBridge-sink-testing --definition "kafka-source:kafka --zkconnect=localhost:2181 --topic=first-queue | custom-bridge | kafka-sink:kafka --topic=regular-queue" --deploy
But I'm getting:
2017-03-16T12:52:06-0300 1.3.0.RELEASE INFO DeploymentSupervisor-0 zk.ZKStreamDeploymentHandler - Deployment status for stream 'source-customBridge-sink-testing': DeploymentStatus{state=failed,error(s)=org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.sprin
gframework.integration.config.ConsumerEndpointFactoryBean#0': Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: A poller should not be specified for endpoint 'org.springframework.integration.config.ConsumerEndpointFactoryBean#0', since 'pollable' is a Subscrib.
What am I missing?
The problem that you are missing that the pollable must be declared as :
<channel id="pollable">
<queue/>
</channel>
Related
Recently we introduced task executor to spring integrations pollers to fasten our file reading process . However introduction of task executor led to unexpected problems where in our service stopped processing messages in spring integration channels
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:integration="http://www.springframework.org/schema/integration"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xmlns:int-jdbc="http://www.springframework.org/schema/integration/jdbc"
xmlns:int-amqp="http://www.springframework.org/schema/integration/amqp"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file
http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
http://www.springframework.org/schema/integration/jdbc
http://www.springframework.org/schema/integration/jdbc/spring-integration-jdbc.xsd
http://www.springframework.org/schema/integration/amqp
http://www.springframework.org/schema/integration/amqp/spring-integration-amqp.xsd">
<integration:channel id="filesIn" />
<integration:channel id="toArchive" />
<integration:channel id="toRabbitForRO" />
<integration:channel id="outputFilesROIn" />
<integration:channel id="toRabbitForSA" />
<integration:channel id="outputFilesSAIn" />
<int-file:inbound-channel-adapter directory="${rnr.file.directory}" auto-startup="true"
filter="filterFiles" channel="filesIn">
<integration:poller
cron="0 0,5,10,15,20,25,30,35,40,45,50,55 0-9,18-23 * * ?"
task-executor="largeFileTaskExecutor"
max-messages-per-poll="${max-messages}"/>
</int-file:inbound-channel-adapter>
<integration:service-activator input-channel="filesIn" output-channel="toArchive"
ref="processSingleLargeFile" method="process"></integration:service-activator>
<int-file:outbound-channel-adapter channel="toArchive" delete-source-files="true"
directory="file:${rnr.file.directory}/archive">
</int-file:outbound-channel-adapter>
<int-file:inbound-channel-adapter directory="${roOutputDir}"
auto-startup="true" filename-pattern="*.xml" channel="outputFilesROIn">
<integration:poller fixed-delay="200"
task-executor="smallFileTaskExecutor"
max-messages-per-poll="${max-messages}" ></integration:poller>
</int-file:inbound-channel-adapter>
<integration:service-activator input-channel="outputFilesROIn"
output-channel="toRabbitForRO" ref="processMultipleFiles" method="processROFile"></integration:service-activator>
<int-amqp:outbound-channel-adapter
channel="toRabbitForRO" amqp-template="rabbitTemplate" exchange-name="sample-excahnge"
routing-key="sample-key" />
</beans>
Te first task executor introduced in poller works perfectly. However the second poller doesn't seem to work. It does not read file from the directory mentioned
<int-file:inbound-channel-adapter directory="${roOutputDir}"
auto-startup="true" filename-pattern="*.xml" channel="outputFilesROIn">
<integration:poller fixed-delay="200"
task-executor="smallFileTaskExecutor"
max-messages-per-poll="${max-messages}" ></integration:poller>
</int-file:inbound-channel-adapter>
// this file channel adapter is not working . No message appear in output channel **outputFilesROIn**
smallFileTaskExecutor and largeFileTaskexecutor have 2 threads as core pool size .
max-messages-per-poll for each poller is defined as 2
Thread dump when our service is not processing messages : https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjAvMDgvMTkvLS1hcGktZWQzZTJmYzMtMWFkYy00Mzk5LWJkZjgtNzk0NGQwMzdjNjIwMjg2Njk5ZDMtYTFmNC00YzIzLThmZTQtYzQ4Nzg4NmNhNGM1LnR4dC0t&
PS : followed this How to read and process multiple files concurrently in spring? while implementing concurrency in reading files
According to your thread dump, it looks like a default task scheduler with pool of 10 threads is fully busy. Probably your cron is very aggressive and with that largeFileTaskExecutor it has a opportunity to fire scheduled tasks like a crazy. This way your second poller just don't have resources in the task scheduler to do its job.
Consider to adjust your cron or reconfigure that task scheduler to bigger thread pool. Or just don't use executors since it confirms to us that it doesn't increase your performance.
See docs for scheduler thread pool: https://docs.spring.io/spring-integration/docs/current/reference/html/configuration.html#namespace-taskscheduler
I am attempting to define a simple message flow in Spring Integration that reads from one channel and then dumps the messages into a Kafka queue. To do this, I am using spring-integration-kafka. The problem is I get an EvaluationContext error I can't decipher.
Here is my configuration in XML:
<int:channel id="myStreamChannel"/>
<int:gateway id="myGateway" service-interface="com.myApp.MyGateway" >
<int:method name="process" request-channel="myStreamChannel"/>
</int:gateway>
<int:channel id="activityOutputChannel"/>
<int:transformer input-channel="myStreamChannel" output-channel="activityOutputChannel" ref="activityTransformer"/>
<int-kafka:outbound-channel-adapter id="kafkaOutboundChannelAdapter"
kafka-producer-context-ref="kafkaProducerContext"
auto-startup="false"
channel="activityOutputChannel"
topic="my-test"
message-key-expression="header.messageKey">
<int:poller fixed-delay="1000" time-unit="MILLISECONDS" receive-timeout="0" task-executor="taskExecutor"/>
</int-kafka:outbound-channel-adapter>
<task:executor id="taskExecutor"
pool-size="5-25"
queue-capacity="20"
keep-alive="120"/>
<int-kafka:producer-context id="kafkaProducerContext" producer-properties="producerProperties">
<int-kafka:producer-configurations>
<int-kafka:producer-configuration broker-list="kafkaserver.com:9092"
key-class-type="java.lang.String"
value-class-type="java.lang.String"
topic="my-test"
key-encoder="stringEncoder"
value-encoder="stringEncoder"
compression-codec="snappy"/>
</int-kafka:producer-configurations>
</int-kafka:producer-context>
When I run my application via Spring Boot, I get this exception:
Error creating bean with name 'org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler#0': Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: [Assertion failed] - this argument is required; it must not be null
This is the offending line in the stack trace:
at org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler.onInit(KafkaProducerMessageHandler.java:68)
Here is what happens at Line 68:
Assert.notNull(this.evaluationContext);
So the EvaluationContext is null. I have no idea why.
By the way, when I replace the Kafka endpoint with a trivial stdout endpoint to print the message body, everything works fine.
Can you tell me what is wrong with my Kafka endpoint configuration that there is no EvaluationContext available?
Your issue belongs to the version mismatch hell.
The Spring Boot pulls that Spring Integration Core version, which doesn't support IntegrationEvaluationContextAware to populate the EvaluationContext in the KafkaProducerMessageHandler.
So, you should upgrade Spring Integration Kafka to most recent release: https://github.com/spring-projects/spring-integration-kafka/releases
I'm trying to configure Spring XD's mail sink to send messages to an outlook account. This is my stream definition:
stream create outlookMailSink --definition "http | mail --to='\"email#address.com\"' --host=outlook.office365.com --subject=payload+' world'" --deploy
I'm testing using this shell command:
http post --data Hello
I am getting the following error message:
Failed message 1: com.sun.mail.smtp.SMTPSendFailedException: 530 5.7.57 SMTP; Client was not authenticated to send anonymous mail during MAIL FROM
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:84)
I have investigated this in the Spring XD documentation and internet searches but I haven't found a solution that works. Can anyone help me with this please?
I found a solution which involves creating a new mail sink module which is a slight modification of spring XD's supplied mail sink module. The stream must also include to, from, host, port, username and password options.
Copy the ..\spring-xd-<version>\xd\modules\sink\mail folder and rename to secure-mail.
In ..\spring-xd-<version>\xd\modules\sink\secure-mail rename both mail.properties and mail.xml to secure-mail.properties and secure-mail.xml respectively.
Replace the contents of secure-mail.xml with the following:
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/integration"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:beans="http://www.springframework.org/schema/beans"
xmlns:file="http://www.springframework.org/schema/integration/file"
xmlns:int-mail="http://www.springframework.org/schema/integration/mail"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/integration/mail http://www.springframework.org/schema/integration/mail/spring-integration-mail.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
<channel id="input" />
<int-mail:header-enricher input-channel="input" output-channel="send">
<int-mail:to expression="${to:null}" />
<int-mail:from expression="${from:null}" />
<int-mail:subject expression="${subject:null}" />
<int-mail:cc expression="${cc:null}" />
<int-mail:bcc expression="${bcc:null}" />
<int-mail:reply-to expression="${replyTo:null}" />
<int-mail:content-type expression="${contentType:null}" />
</int-mail:header-enricher>
<channel id="send" />
<int-mail:outbound-channel-adapter
channel="send" host="${host}" port="${port}" username="${username:}"
password="${password:}" java-mail-properties="javaMailProperties"/>
<util:properties id="javaMailProperties">
<beans:prop key="mail.smtp.starttls.enable">true</beans:prop>
</util:properties>
</beans:beans>
Create the stream as follows:
stream create outlookMailSink --definition "http | secure-mail --to='\"email#address.com\"' --from='\"email#address.com\"' --host=outlook.office365.com --port=587 --username=email#address.com --password=password --subject=payload+' world'" --deploy
Test using shell command: http post --data Hello
The contents of secure-mail.xml is almost identical to mail.xml, the key is to set the property mail.smtp.starttls.enable to be true in order to enable TLS encryption for communication over port 587. OF course you could just modify Spring XD's mail sink module directly and use this - it's up to you.
I'd be interested to hear if anyone has a better solution to this? For example, is it possible to set the mail.smtp.starttls.enable property on start-up of Spring XD thereby allowing you to use the original mail sink module? I tried this by modifying the xd-singlenode startup script - the property was set but it didn't affect the mail sink module.
References:
https://stackoverflow.com/a/24765052/2124106
https://stackoverflow.com/a/23823863/2124106
http://docs.spring.io/spring-integration/reference/html/mail.html
I seem to be having some trouble configuring my Spring MVC backend to receive and send TCP messages. Looking at the configuration a user suggests in this question - how to plug a TCP-IP client server in a spring MVC application - I tried to place this configuration into my root-context.xml. However, for all of the tags it displays a message such as:
Unable to locate Spring NamespaceHandler for element 'int-ip:tcp-outbound-gateway' of schema namespace 'http://www.springframework.org/schema/integration/ip'
int-ip:tcp-outbound-gateway and int:gateway both display cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'int:gateway' (replace int:gateway with int-ip:tcp-outbound-gateway).
Here is my root-context.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-ip="http://www.springframework.org/schema/integration/ip"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration/ip http://www.springframework.org/schema/integration/ip/spring-integration-ip.xsd
http://www.springframework.org/schema/integration/ http://www.springframework.org/schema/integration/spring-integration.xsd">
<!-- Root Context: defines shared resources visible to all other web components -->
<int:gateway id="gw"
service-interface="org.springframework.integration.samples.tcpclientserver.SimpleGateway"
default-request-channel="input"/>
<int-ip:tcp-connection-factory id="client"
type="client"
host="localhost"
port="1234"
single-use="true"
so-timeout="10000"/>
<int:channel id="input" />
<int-ip:tcp-outbound-gateway id="outGateway"
request-channel="input"
reply-channel="clientBytes2StringChannel"
connection-factory="client"
request-timeout="10000"
remote-timeout="10000"/>
<int:transformer id="clientBytes2String"
input-channel="clientBytes2StringChannel"
expression="new String(payload)"/>
</beans>
What am I doing incorrectly? Also, some general tips as to how I could use Spring to send and receive TCP communications would be appreciated :)
It appears you don't have the spring-integration-ip and spring-integration-core jars on your classpath. You need to bundle them into your war or otherwise make them available on the classpath according to your app server's requirements.
Is there any example of using spring xd with pure java config (no xml).
I created a simple tasklet, jarred the same (whojob.jar) and put it into the lib folder.
The xml config is below. I put this in a file called whojob which is in the modules/job folder and also jarred with my whojob.jar.
When I try to create a job:- (:>job create --name mywhojob --definition "whojob") I get the error :
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.integration.config.TransformerFactoryBean#0': Cannot create inner bean 'org.springframework.xd.dirt.plugins.job.JobLaunchRequestTransformer#0' of type [org.springframework.xd.dirt.plugins.job.JobLaunchRequestTransformer] while setting bean property 'targetObject'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.xd.dirt.plugins.job.JobLaunchRequestTransformer#0' defined in class path resource [META-INF/spring-xd/plugins/job/job-module-beans.xml]: Cannot resolve reference to bean 'jobFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jobFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: No Batch Job found in registry for the provided key 'mywhojob.job'.
My xml config is below. If possible I would like to have a pure java solution.
Thanks
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:batch="http://www.springframework.org/schema/batch"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/batch
http://www.springframework.org/schema/batch/spring-batch.xsd">
<batch:job id="whojob">
<batch:step id="whojobStep">
<batch:tasklet ref="whojobXDTasklet" />
</batch:step>
</batch:job>
<bean id="whojobXDTasklet"
class=" com.auction.data.batch.job.WhoTasklet" >
<property name= "who" ref="whoBean"/>
</bean>
<bean id="whoBean"
class="com.auction.data.batch.service.Who" >
</bean>
</beans>
Here is a sample that uses XD job with the batch tasklet defined in an external jar file.
https://github.com/ilayaperumalg/spring-xd-batch-sample
To create jar, from the repo root directory, just run: ./gradlew build
To make it easier, I have also copied the job module "myjob" (which uses the above tasklet) into the repo. You can just copy this 'myjob' into your $XD_HOME/modules/job/.
Give it a try using XD version M6 or the latest master, and let us know if you have any question.