Header Enricher for spring xd to call http-client t pass headers - spring-xd

I am using http header enricher component to pass the headers but I am not able to make a http request using http-client
I have a stream like below I am trying to make a http rest Post request and log it to the logger.I have a API-Key header for which I used this below header enricher module
https://github.com/spring-projects/spring-xd-modules/tree/master/header-enricher
stream create --name test24 --definition "http |header-enricher --headers={\"Api-Key\":\"'xyz'\"}| http-client --url='''http://xyz:/abc''' --httpMethod=POST | log " --deploy
http post --data '{"profileReferenceID":"pPPpe85Ht91e%2FpCvlkJMkniwiM%2BGvPjATkjc6HGqaJFS065txVj%2BhqDPsHz54KEm1RrCXftjNCSl2fpBKKKdrie9n6t9jetWhk29ELCmbw0%3D","templateURL":{"type":"type","href":"/content/notification/templates/EOSWEL?firstName=Abcd&userName=dddd&accountLast=2144&phone=1234&SvcgLOBCd=MS","method":"POST"}}'
> POST (text/plain;Charset=UTF-8) http://localhost:9000 {"profileReferenceID":"pPPpe85Ht91e%2FpCvlkJMkniwiM%2BGvPjATkjc6HGqaJFS065txVj%2BhqDPsHz54KEm1RrCXftjNCSl2fpBKKKdrie9n6t9jetWhk29ELCmbw0%3D","templateURL":{"type":"type","href":"/content/notification/templates/EOSWEL?firstName=Abcd&userName=dddd&accountLast=2144&phone=1234&SvcgLOBCd=MS","method":"POST"}}
2015-11-16T13:05:24-0600 1.2.1.RELEASE WARN xdbus.test24.1-1 retry.RejectAndDontRequeueRecoverer - Retries exhausted for message (Body:'{"profileReferenceID":"pPPpe85Ht91e%2FpCvlkJMkniwiM%2BGvPjATkjc6HGqaJFS065txVj%2BhqDPsHz54KEm1RrCXftjNCSl2fpBKKKdrie9n6t9jetWhk29ELCmbw0%3D","templateURL":{"type":"type","href":"/content/notification/templates/EOSWEL?firstName=Abcd&userName=dddd&accountLast=2144&phone=1234&SvcgLOBCd=MS","method":"POST"}}'MessageProperties [headers={requestMethod=POST, User-Agent=Java/1.8.0_60, Host=localhost:9000, Content-Length=305, contentType=text/plain, Api-Key=xyz, requestPath=/, originalContentType=text/plain;Charset=UTF-8}, timestamp=null, messageId=null, userId=null, appId=null, clusterId=null, type=null, correlationId=null, replyTo=null, contentType=text/plain, contentEncoding=null, contentLength=0, deliveryMode=null, expiration=null, priority=0, redelivered=false, receivedExchange=, receivedRoutingKey=xdbus.test24.1, deliveryTag=2, messageCount=0])
I am getting 403 forbidden i get this error from my webservice if my client api key is not passed which i was able to reproduce in post man .So I assume my API-Key header is not passed.
caused by: org.springframework.messaging.MessageHandlingException: HTTP request execution failed for URI [http://xyx/abc]; nested exception is org.springframework.web.client.HttpClientErrorException: 403 Forbidden
at org.springframework.integration.http.outbound.HttpRequestExecutingMessageHandler.handleRequestMessage(HttpRequestExecutingMessageHandler.java:410) ~[spring-integration-http-4.1.6.RELEASE.jar:na]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:99) ~[spring-integration-core-4.1.6.RELEASE.jar:na]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78) ~[spring-integration-core-4.1.6.RELEASE.jar:na]
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) ~[spring-integration-core-4.1.6.RELEASE.jar:na]
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101) ~[spring-integration-core-4.1.6.RELEASE.jar:na]
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97) ~[spring-integration-core-4.1.6.RELEASE.jar:na]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) ~[spring-integration-core-4.1.6.RELEASE.jar:na]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:287) ~[spring-integration-core-4.1.6.RELEASE.jar:na]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:245) ~[spring-integration-core-4.1.6.RELEASE.jar:na]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.1.7.RELEASE.jar:4.1.7.RELEASE]

You should add your Api-Key header to be mapped to the HTTP request header:
http-client --url='''http://xyz:/abc''' --httpMethod=POST --mappedRequestHeaders=HTTP_REQUEST_HEADERS,Api-Key
mappedRequestHeaders
request message header names to be propagated to/from the adpater/gateway (String, default: HTTP_REQUEST_HEADERS)
Where HTTP_REQUEST_HEADERS is a set of only standard HTTP headers which are appropriate for request.
http://docs.spring.io/spring-xd/docs/current/reference/html/#http-client

Related

elasticsearch - Incorrect HTTP method for uri [/] and method [PUT], allowed: [DELETE, GET, HEAD]

This is my JSON data.
"{\"athena\":{\"settings\":{\"index\":{\"max_ngram_diff\":\"49\",\"mapping\":{\"total_fields\":{\"limit\":\"3000\"},\"ignore_malformed\":\"true\"},\"refresh_interval\":\"30s\",\"number_of_shards\":\"1\",\"max_shingle_diff\":\"4\",\"max_result_window\":\"100000\",\"analysis\":{\"filter\":{\"searchkick_index_shingle\":{\"token_separator\":\"\",\"type\":\"shingle\"},\"analyzed_ngram\":{\"token_chars\":[\"letter\",\"digit\"],\"min_gram\":\"2\",\"type\":\"edge_ngram\",\"max_gram\":\"10\"},\"location_synonym_filter\":{\"type\":\"synonym\",\"synonyms\":[\"e=>east\",\"w=>west\",\"n=>north\",\"s=>south\",\"rd=>road\",\"st=>street\",\"av=>avenue\",\"pkwy=>parkway\",\"hwy=>highway\"]},\"searchkick_stemmer\":{\"type\":\"snowball\",\"language\":\"English\"}},\"char_filter\":{\"ampersand\":{\"type\":\"mapping\",\"mappings\":[\"&=> and \"]}},\"normalizer\":{\"lowercase_normalizer\":{\"filter\":[\"lowercase\"],\"type\":\"custom\"}},\"analyzer\":{\"searchkick_index\":{\"filter\":[\"lowercase\",\"asciifolding\",\"searchkick_index_shingle\",\"searchkick_stemmer\"],\"char_filter\":[\"ampersand\"],\"type\":\"custom\",\"tokenizer\":\"standard\"},\"autocomplete_analyzer\":{\"filter\":[\"lowercase\",\"location_synonym_filter\"],\"token_chars\":[\"letter\",\"digit\"],\"type\":\"custom\",\"stopwords\":\"_none_\",\"tokenizer\":\"standard\"}}},\"number_of_replicas\":\"0\"}}}}"
and I am running this command.
elasticdump --input=./athena-analyzer.json --output=http://localhost:9200 --type=analyzer
and i am getting this error.
{"error":"Incorrect HTTP method for uri [/] and method [PUT], allowed: [DELETE, GET, HEAD]","status":405}

How to resolve HTTPS connection errors in mule?

Below is the end system(Eloqua) connection code.
<http:request config-ref="Eloqua_Bulk_API" path="/customObjects/{customObjectId}/fields" method="GET" doc:name="Eloqua Get CDO fields">
<http:request-builder>
<http:uri-param paramName="customObjectId" value="#[flowVars.cdo.id]"/>
</http:request-builder>
</http:request>
But not not getting proper repose, it is throwing below error message.
<CustomObjectFieldSearchResponse xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="">
<ResponseStatus>
<ErrorCode>ArgumentNullException</ErrorCode>
<Message>As result 'CustomObjectFieldSearchResponse' is not a supported responseType, a defaultAction must be supplied
Parameter name: defaultAction</Message>
</ResponseStatus>
</CustomObjectFieldSearchResponse>
Message : Response code 500 mapped as failure.
Payload : org.glassfish.grizzly.utils.BufferInputStream#1f0131c5
Element : /SAPBWServiceReqInsertUpdate/input/0/0/EloquaLookupServiceRequestCDOBulk/subprocessors/1/EloquaLookupFields/subprocessors/0/0/1/2 # nol-integration-nonmaven:bulk-integration.xml:92 (Eloqua Get CDO fields)
Element XML : <http:request config-ref="Eloqua_Bulk_API" path="/customObjects/{customObjectId}/fields" method="GET" doc:name="Eloqua Get CDO fields">
<http:request-builder>
<http:uri-param paramName="customObjectId" value="#[flowVars.cdo.id]"></http:uri-param>
</http:request-builder>
</http:request>
--------------------------------------------------------------------------------
Root Exception stack trace:
org.mule.module.http.internal.request.ResponseValidatorException: Response code 500 mapped as failure.
at org.mule.module.http.internal.request.SuccessStatusCodeValidator.validate(SuccessStatusCodeValidator.java:37)
at org.mule.module.http.internal.request.DefaultHttpRequester.validateResponse(DefaultHttpRequester.java:413)
at org.mule.module.http.internal.request.DefaultHttpRequester.innerProcess(DefaultHttpRequester.java:401)
at org.mule.module.http.internal.request.DefaultHttpRequester.processBlocking(DefaultHttpRequester.java:221)
at org.mule.processor.AbstractNonBlockingMessageProcessor.process(AbstractNonBlockingMessageProcessor.java:43)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:108)
when i try from SOAPUI it is fetching the data properly.
I added added the below header in the request(http request/component). It is working fine.
name: Content-Type
value: application/json

Does Spring Cloud Stream Kafka supports embedded headers?

According to this topic:
Kafka Spring Integration: Headers not coming for kafka consumer -
this is no headers support for Kafka
But documentation says:
spring.cloud.stream.kafka.binder.headers
The list of custom headers that will be transported by the binder.
Default: empty.
I can't get it working with spring-cloud-stream-binder-kafka: 1.2.0.RELEASE
SENDING LOG:
MESSAGE (e23885fd-ffd9-42dc-ebe3-5a78467fee1f) SENT :
GenericMessage [payload=...,
headers={
content-type=application/json,
correlationId=51dd90b1-76e6-4b8d-b667-da25f214f383,
id=e23885fd-ffd9-42dc-ebe3-5a78467fee1f,
contentType=application/json,
timestamp=1497535771673
}]
RECEIVING LOG:
MESSAGE (448175f5-2b21-9a44-26b9-85f093b33f6b) RECEIVED BY HANDLER 1:
GenericMessage [payload=...,
headers={
kafka_offset=36,
id=448175f5-2b21-9a44-26b9-85f093b33f6b,
kafka_receivedPartitionId=0,
contentType=application/json;charset=UTF-8,
kafka_receivedTopic=new_patient, timestamp=1497535771715
}]
MESSAGE (448175f5-2b21-9a44-26b9-85f093b33f6b) RECEIVED BY HANDLER 2 :
GenericMessage [payload=...,
headers={
kafka_offset=36,
id=448175f5-2b21-9a44-26b9-85f093b33f6b,
kafka_receivedPartitionId=0,
contentType=application/json;charset=UTF-8,
kafka_receivedTopic=new_patient, timestamp=1497535771715
}]
I expect to see the same message id and get correlationId on receiving side.
application.properties:
spring.cloud.stream.kafka.binder.headers=correlationId
spring.cloud.stream.bindings.newTest.destination=new_test
spring.cloud.stream.bindings.newTestCreated.destination=new_test
spring.cloud.stream.default.consumer.headerMode=embeddedHeaders
spring.cloud.stream.default.producer.headerMode=embeddedHeaders
SENDING MESSAGE:
#Publisher(channel = "testChannel")
public Object newTest(Object param) {
...
return myObject;
}
Yes, it does: http://docs.spring.io/spring-cloud-stream/docs/Chelsea.SR2/reference/htmlsingle/index.html#_consumer_properties
headerMode
When set to raw, disables header parsing on input. Effective only for messaging middleware that does not support message headers natively and requires header embedding. Useful when inbound data is coming from outside Spring Cloud Stream applications.
Default: embeddedHeaders
But that is already Spring Cloud Stream story, not Spring Kafka per se.

ClientHandlerException while handling 401 Unauthorised from server

I am hitting a multipart API which is working fine in all 2XX and 4XX cases except 401s, it is not able to map to ClientResponse class and throws ClientHandlerException, this is how I am making the call:
ClientResponse response = client.resource(uri)
.accept("application/json")
.header("X-FORWARDED-FOR", "X.X.X.X")
.header("Authorization", authorization)
.type(MediaType.MULTIPART_FORM_DATA_TYPE)
.post(ClientResponse.class, part);
athorization: Basic authorization header
part: FormDataMultiPart object
here are the error logs:
ERROR [2017-06-02 21:46:01,362] [ ] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: 91277b4e45d2df98
! com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: Error writing to server
! at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
! at com.sun.jersey.api.client.Client.handle(Client.java:652)
! at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
! at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
! at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570)
... some more application logs and then..
! Caused by: java.io.IOException: Error writing to server
! at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:666)
! at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:666)
! at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1534)
Jersey-client version: 1.18.1

NullPointerException from AbstractAggregatingMessageGroupProcessor when using Kafka Message Bus

Pretty sure this a bug in Spring XD.
Running Spring XD 1.3.0.RELEASE in SingleNode mode. All configuration is default except I am using Kafka instead of local transport. Relevant XD configuration:
spring:
profiles: singlenode
xd:
transport: kafka
messagebus:
kafka:
brokers: localhost:9092
zkAddress: localhost:2181
mode: embeddedHeaders
offsetManagement: kafkaTopic
socketBufferSize: 2097152
offsetStoreTopic: SpringXdOffsets
offsetStoreSegmentSize: 25000000
offsetStoreRetentionTime: 60000
offsetStoreRequiredAcks: 1
offsetStoreMaxFetchSize: 1048576
offsetStoreBatchBytes: 16384
offsetStoreBatchTime: 1000
offsetUpdateTimeWindow: 10000
offsetUpdateCount: 0
offsetUpdateShutdownTimeout: 2000
default:
batchSize: 16384
batchTimeout: 0
replicationFactor: 1
concurrency: 1
requiredAcks: 1
compressionCodec: none
queueSize: 8192 # must be a power of 2
maxWait: 100
fetchSize: 1048576
minPartitionCount: 1
durableSubscription: false
Create a stream using an aggregator (this one straight from the reference docs):
stream create --name aggregates --definition "http | aggregator --count=3 --aggregation=T(org.springframework.util.StringUtils).collectionToDelimitedString(#this.![payload],' ') | log" --deploy
Then send 3 POST's:
xd:> http post --data Hello
xd:> http post --data World
xd:> http post --data !
The result is this stacktrace:
2015-12-10T17:07:11-0800 1.3.0.RELEASE ERROR pool-13-thread-1 listener.LoggingErrorHandler - Error while processing: KafkaMessage [Message(magic = 0, attributes = 0, crc = 344940496, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=81 cap=81]), KafkaMessageMetadata [offset=2, nextOffset=3, Partition[topic='aggregates.0', id=0]]
org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.config.AggregatorFactoryBean#0]; nested exception is java.lang.NullPointerException
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:139) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:147) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:120) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:442) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:392) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:231) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:154) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:102) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:105) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.channel.FixedSubscriberChannel.send(FixedSubscriberChannel.java:69) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.channel.FixedSubscriberChannel.send(FixedSubscriberChannel.java:63) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:105) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$300(KafkaMessageDrivenChannelAdapter.java:43) ~[spring-integration-kafka-1.3.0.RELEASE.jar:na]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$AutoAcknowledgingChannelForwardingMessageListener.doOnMessage(KafkaMessageDrivenChannelAdapter.java:171) ~[spring-integration-kafka-1.3.0.RELEASE.jar:na]
at org.springframework.integration.kafka.listener.AbstractDecodingMessageListener.onMessage(AbstractDecodingMessageListener.java:50) ~[spring-integration-kafka-1.3.0.RELEASE.jar:na]
at org.springframework.integration.kafka.listener.QueueingMessageListenerInvoker$KafkaMessageDispatchingSubscriber.onNext(QueueingMessageListenerInvoker.java:221) [spring-integration-kafka-1.3.0.RELEASE.jar:na]
at org.springframework.integration.kafka.listener.QueueingMessageListenerInvoker$KafkaMessageDispatchingSubscriber.onNext(QueueingMessageListenerInvoker.java:209) [spring-integration-kafka-1.3.0.RELEASE.jar:na]
at reactor.core.processor.util.RingBufferSubscriberUtils.route(RingBufferSubscriberUtils.java:67) [reactor-core-2.0.5.RELEASE.jar:na]
at reactor.core.processor.RingBufferProcessor$BatchSignalProcessor.run(RingBufferProcessor.java:789) [reactor-core-2.0.5.RELEASE.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_66]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
Caused by: java.lang.NullPointerException: null
at org.springframework.integration.aggregator.AbstractAggregatingMessageGroupProcessor.aggregateHeaders(AbstractAggregatingMessageGroupProcessor.java:115) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.aggregator.AbstractAggregatingMessageGroupProcessor.processMessageGroup(AbstractAggregatingMessageGroupProcessor.java:79) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.completeGroup(AbstractCorrelatingMessageHandler.java:648) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.handleMessageInternal(AbstractCorrelatingMessageHandler.java:405) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
... 32 common frames omitted
Is this a Spring XD bug? Is there a workaround?
More details.
The header causing the NPE is kafka_messageKey. The value of this header is null so AbstractAggregatingMessageGroupProcessor:115 throws the NPE.
Looking further at KafkaMessageBus, it appears that the Kafka messageKey is intentionally set to null from the Producer.
One possible workaround would be to add a header filter to the aggregator module, by inserting a header filter at the beginning of $XD_HOME/modules/processor/aggregator/config/aggregator.xml, by altering its start as follows:
<channel id="input" />
<channel id="aggregatorInput"/>
<int:header-filter input-channel="inputChannel"
output-channel="aggregatorInput" header-names="kafka_messageKey"/>
<aggregator input-channel="aggregatorInput" output-channel="output"
correlation-strategy-expression="${correlation}"
release-strategy-expression="${release}" expression="${aggregation}"
send-partial-result-on-expiry="true" expire-groups-upon-completion="true"
message-store="messageStore">
</aggregator>
... rest of the module definition remains unchanged ...
As for how to fix this issue on the log run, I will comment on the JIRA issue.
Cheers,
Marius
This is a bug in Spring XD. See INT-3908 for details.
Following the suggestion by Marius, the following is a suitable workaround:
Edit $XD_HOME/modules/processor/aggregator/config/aggregator.xml to include:
<channel id="aggregatorInput"/>
<header-enricher input-channel="input" output-channel="aggregatorInput" default-overwrite="true">
<header name="kafka_messageKey" value="."/>
</header-enricher>
<aggregator input-channel="aggregatorInput" output-channel="output"
correlation-strategy-expression="${correlation}"
release-strategy-expression="${release}" expression="${aggregation}"
send-partial-result-on-expiry="true" expire-groups-upon-completion="true"
message-store="messageStore">
</aggregator>
Note: using a header-filter will not work as the mechanism SI uses for removing a header will only remove the header if it is not already null.
Alternatively, if you don't want to edit the module XML directly, you could use Module Composition to include a Header Enricher before the standard aggregator module.
module compose --name kafa-aggregator --definition "header-enricher --headers={\"kafka_messageKey\":\"'.'\"} --overwrite=true | aggregator --count=3 --aggregation=T(org.springframework.util.StringUtils).collectionToDelimitedString(#this.![payload],' ')"
stream create --name aggregates --definition "http | kafa-aggregator | log" --deploy

Resources