SQS Listener #Headers getting body content instead of Message Attributes - spring-boot

I am using Spring Cloud SQS messaging for listening to a specified queue. Hence using #SqsListener annotation as below:
#SqsListener(value = "${QUEUE}", deletionPolicy = SqsMessageDeletionPolicy.ALWAYS )
public void receive(#Headers Map<String, String> header, #Payload String message) {
try {
logger.logInfo("Message payload is: "+message);
logger.logInfo("Header from SQS is: "+header);
if(<Some condition>){
//Dequeue the message once message is processed successfully
awsSQSAsync.deleteMessage(header.get(LOOKUP_DESTINATION), header.get(RECEIPT_HANDLE));
}else{
logger.logInfo("Message with header: " + header + " FAILED to process");
logger.logError(FLEX_TH_SQS001);
}
} catch (Exception e) {
logger.logError(FLEX_TH_SQS001, e);
}
}
I am able to connect the specified queue successfully and read the message as well. I am setting a message attribute as "Key1" = "Value1" along with message in aws console before sending the message. Following is the message body:
{
"service": "ecsservice"
}
I am expecting "header" to receive a Map of all the message attributes along with the one i.e. Key1 and Value1. But what I am receiving is:
{service=ecsservice} as the populated map.
That means payload/body of message is coming as part of header, although body is coming correctly.
I wonder what mistake I am doing due to which #Header header is not getting correct message attributes.
Seeking expert advice.
-PC

I faced the same issue in one of my spring projects.
The issue for me was, SQS configuration of QueueMessageHandlerFactory with Setting setArgumentResolvers.
By default, the first argument resolver in spring is PayloadArgumentResolver.
with following behavior
#Override
public boolean supportsParameter(MethodParameter parameter) {
return (parameter.hasParameterAnnotation(Payload.class) || this.useDefaultResolution);
}
Here, this.useDefaultResolution is by default set to true – which means any parameter can be converted to Payload.
And Spring tries to match your method actual parameters with one of the resolvers, (first is PayloadArgumentResolver) - Indeed it will try to convert all the parameters to Payload.
Source code from Spring:
#Nullable
private HandlerMethodArgumentResolver getArgumentResolver(MethodParameter parameter) {
HandlerMethodArgumentResolver result = this.argumentResolverCache.get(parameter);
if (result == null) {
for (HandlerMethodArgumentResolver resolver : this.argumentResolvers) {
if (resolver.supportsParameter(parameter)) {
result = resolver;
this.argumentResolverCache.put(parameter, result);
break;
}
}
}
return result;
}
How I solved this,
The overriding default behavior of Spring resolver
factory.setArgumentResolvers(
listOf(
new PayloadArgumentResolver(converter, null, false),
new HeaderMethodArgumentResolver(null, null)
)
)
Where I set, default flag to false and spring will try to convert to payload only if there is annotation on parameter.
Hope this will help.

Apart from #SqsListener, you need to add #MessageMapping to the method. This annotation will helps to resolve method arguments.

I had this issue working out of a rather large codebase. It turned out that a HandlerMethodArgumentResolver was being added to the list of resolvers that are used to basically parse the message into the parameters. In my case it was the PayloadArgumentResolver, which usually always resolves an argument to be the payload regardless of the annotation. It seems by default it's supposed to come last in the list but because of the code I didn't know about, it ended up being added to the front.
Anyway, if you're not sure take a look around your code and see if you're doing anything regarding spring's QueueMessageHandler or HandlerMethodArgumentResolver.
It helped me to use a debugger and look at HandlerMethodArgumentResolver.resolveArgument method to start tracing what happens.
P.S. I think your #SqsListener code looks fine except that I think #Headers is supposed to technically resolve to a Map of < String, Object >", but I'm not sure that would cause the issue you're seeing.

Related

Mirror #RequestPart behavior in WebFlux functional router definitions with different content types

Problem
We're developing a Spring Boot service to upload data to different back end databases. The idea is that, in one multipart/form-data request a user will send a "model" (basically a file) and "modelMetadata" (which is JSON that defines an object of the same name in our code).
We got the below to work in the WebFlux annotated controller syntax, when the user sends the "modelMetadata" in the multipart form with the content-type of "application/json":
#PostMapping(consumes = [MediaType.MULTIPART_FORM_DATA_VALUE], produces = [MediaType.APPLICATION_JSON_VALUE])
fun saveModel(#RequestPart("modelMetadata") monoModelMetadata: Mono<ModelMetadata>,
#RequestPart("model") monoModel: Mono<FilePart>,
#RequestHeader headers: HttpHeaders) : Mono<ResponseEntity<ModelMetadata>> {
return modelService.saveModel(monoModelMetadata, monoModel, headers)
}
But we can't seem to figure out how to do the same thing in Webflux's functional router definition. Below are the relevant code snippets we have:
#Bean
fun modelRouter() = router {
accept(MediaType.MULTIPART_FORM_DATA).nest {
POST(ROOT, handler::saveModel)
}
}
fun saveModel(r: ServerRequest): Mono<ServerResponse> {
val headers = r.headers().asHttpHeaders()
val monoModelPart = r.multipartData().map { multiValueMap ->
it["model"] // What do we do with this List<Part!> to get a Mono<FilePart>
it["modelMetadata"] // What do we do with this List<Part!> to get a Mono<ModelMetadata>
}
From everything we've read, we should be able to replicate the same functionality found in the annotation controller syntax with the router functional syntax, but this particular aspect doesn't seem to be well documented. Our goal was to move over to use the new functional router syntax since this is a new application we're developing and there are some nice forward thinking features/benefits as described here.
What we've tried
Googling to the ends of the Earth for a relevant example
this is a similar question, but hasn't gained any traction and doesn't relate to our need to create an object from one piece of the multipart request data
this may be close to what we need for uploading the file component of our multipart request data, but doesn't handle the object creation from JSON
Tried looking at the #RequestPart annotation code to see how things are done on that side, there's a nice comment that seems to hint at how they are converting the parts to objects, but we weren't able to figure out where that code lives or any relevant example of how to use an HttpMessageConverter on the ``
the content of the part is passed through an {#link HttpMessageConverter} taking into consideration the 'Content-Type' header of the request part.
Any and all help would be appreciated! Even just some links for us to better understand Part/FilePart types and there role in multipart requests would be helpful!
I was able to come up with a solution to this issue using an autowired ObjectMapper. From the below solution I could turn the modelMetadata and modelPart into Monos to mirror the #RequestPart return types, but that seems ridiculous.
I was also able to solve this by creating a MappingJackson2HttpMessageConverter and turning the metadataDataBuffer into a MappingJacksonInputMessage, but this solution seemed better for our needs.
fun saveModel(r: ServerRequest): Mono<ServerResponse> {
val headers = r.headers().asHttpHeaders()
return r.multipartData().flatMap {
// We're only expecting one Part of each to come through...assuming we understand what these Parts are
if (it.getOrDefault("modelMetadata", listOf()).size == 1 && it.getOrDefault("model", listOf()).size == 1) {
val modelMetadataPart = it["modelMetadata"]!![0]
val modelPart = it["model"]!![0] as FilePart
modelMetadataPart
.content()
.map { metadataDataBuffer ->
// TODO: Only do this if the content is JSON?
objectMapper.readValue(metadataDataBuffer.asInputStream(), ModelMetadata::class.java)
}
.next() // We're only expecting one object to be serialized from the buffer
.flatMap { modelMetadata ->
// Function was updated to work without needing the Mono's of each type
// since we're mapping here
modelService.saveModel(modelMetadata, modelPart, headers)
}
}
else {
// Send bad request response message
}
}
Although this solution works, I feel like it's not as elegant as the one alluded to in the #RequestPart annotation comments. Thus I will accept this as the solution for now, but if someone has a better solution please let us know and I will accept it!

Spring Cloud Function - Separate routing-expression for different Consumer

I have a service, which receives different structured messages from different message queues. Having #StreamListener conditions we can choose at every message type how that message should be handled. As an example:
We receive two different types of messages, which have different header fields and values e.g.
Incoming from "order" queue:
Order1: { Header: {catalog:groceries} }
Order2: { Header: {catalog:tools} }
Incoming from "shipment" queue:
Shipment1: { Header: {region:Europe} }
Shipment2: { Header: {region:America} }
There is a binding for each queue, and with according #StreamListener I can process the messages by catalog and region differently
e.g.
#StreamListener(target = OrderSink.ORDER_CHANNEL, condition = "headers['catalog'] == 'groceries'")
public void onGroceriesOrder(GroceryOder order){
...
}
So the question is, how to achieve this with the new Spring Cloud Function approach?
At the documentation https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/3.0.2.RELEASE/reference/html/spring-cloud-stream.html#_event_routing it is mentioned:
Also, for SpEL, the root object of the evaluation context is Message so you can do evaluation on individual headers (or message) as well …​.routing-expression=headers['type']
Is it possible to add the routing-expression to the binding like (in application.yml)
onGroceriesOrder-in-0:
destination: order
routing-expression: "headers['catalog']==groceries"
?
EDIT after first answer
If the above expression at this location is not possible, what the first answer implies, than my question goes as follows:
As far as I understand, an expression like routing-expression: headers['catalog'] must be set globally, because the result maps to certain (consumer) functions.
How can I control that the 2 different messages on each queue will be forwarted to their own consumer function, e.g.
Order1 --> MyOrderService.onGroceriesOrder()
Order2 --> MyOrderService.onToolsOrder()
Shipment1 --> MyShipmentService.onEuropeShipment()
Shipment2 --> MyShipmentService.onAmericaShipment()
That was easy with #StreamListener, because each method gets their own #StreamListener annotation with different conditions. How can this be achieved with the new routing-expression setting?
?
Aside from the fact that the above is not a valid expression, but I think you meant headers['catalog']==groceries. If so, what would you expect to happen from evaluating it as the only two option could be true/false. Anyway, these are rhetorical but helps to understand the problem and how to fix it.
The expression must result in a value of a function to route TO. So. . .
routing-expression: headers['catalog'] - assumes that the actual value of catalog header is the name of the function to invoke
routing-expression: headers['catalog']==groceries ? 'processGroceries' : 'processOther' - maps value 'groceries' to 'processGroceries' function.
For a specific routing, you can use MessageRoutingCallback strategy:
MessageRoutingCallback
The MessageRoutingCallback is a strategy to assist with determining
the name of the route-to function definition.
public interface MessageRoutingCallback {
FunctionRoutingResult routingResult(Message<?> message);
. . .
}
All you need to do is implement and register it as a bean to be picked
up by the RoutingFunction. For example:
#Bean
public MessageRoutingCallback customRouter() {
return new MessageRoutingCallback() {
#Override
FunctionRoutingResult routingResult(Message<?> message) {
return new FunctionRoutingResult((String) message.getHeaders().get("func_name"));
}
};
}
Spring Cloud Function

Spring Cloud Contract with Spring AMQP

So I've been trying to use Spring Cloud Contract to test RabbitListener.
So far I have found out that by defining "triggeredBy" in contract, the generated test will call the method provided there and so we will need to provide the actual implementation of what that method do in the TestBase.
Another thing is "outputMessage", where we can verify whether the method call before have correctly resulting on some message body sent to certain exchange.
Source: documentation and sample
My question is, is there any way to produce the input message from the contract, instead of triggering own custom method?
Perhaps something similar like Spring Integration or Spring Cloud Stream example in the documentation:
Contract.make {
name("Book Success")
label("book_success")
input {
messageFrom 'input.exchange.and.maybe.route'
messageHeaders {
header('contentType': 'application/json')
header('otherMessageHeader': '1')
}
messageBody ([
bookData: someData
])
}
outputMessage {
sentTo 'output.exchange.and.maybe.route'
headers {
header('contentType': 'application/json')
header('otherMessageHeader': '2')
}
body([
bookResult: true
])
}
}
I couldn't find any examples in their sample project that show how to do this.
Having used spring cloud contract to document and test rest api services, if possible I would like to stay consistent by defining both the input and expected output in contract files for event based services.
Never mind, actually its already supported.
For unknown reason the documentation in "Stub Runner Spring AMQP" does not list the scenario like others previous sample.
Here is how I make it works:
Contract.make {
name("Amqp Contract")
label("amqp_contract")
input {
messageFrom 'my.exchange'
messageHeaders {
header('contentType': 'text/plain')
header('amqp_receivedRoutingKey' : 'my.routing.key')
}
messageBody(file('request.json'))
}
outputMessage {
sentTo 'your.exchange'
headers {
header('contentType': 'text/plain')
header('amqp_receivedRoutingKey' : 'your.routing.key')
}
body(file('response.json'))
}
}
This will create a test that will call your listener based on "my.exchange" and "my.routing.key" triggering the handler method.
It will then capture the message and routing key on your RabbitTemplate call to "your.exchange".
verify(this.rabbitTemplate, atLeastOnce()).send(eq(destination), routingKeyCaptor.capture(),
messageCaptor.capture(), any(CorrelationData.class));
Both message and routing key then will be asserted.

AWS Lex receives Invalid Response from lambda function - Can not construct instance of IntentResponse

Using Java8 in eclipse AWS SDK, I've created and uploaded a lambda function that is hooked in upon fulfillment of my lex intent.
Lambda has not problem receiving JSON request and parsing.
Then, I format a simple "Close" dialogAction response and send back to lex and receive the following error from the Test Bot page in the lex console:
An error has occurred: Received invalid response from Lambda:
Can not construct instance of IntentResponse:
no String-argument constructor/factory method to deserialize
from String value
('{"dialogAction
{"type":"Close","fulfillmentState":"Fulfilled","message":
{"contentType":"PlainText","content":"Thanks I got your info"}}}')
at [Source: "{\"dialogAction\":
{\"type\":\"Close\",\"fulfillmentState\":\"Fulfilled\",\"message\":
{\"contentType\":\"PlainText\",\"content\":\"Thanks I got your
info\"}}}";line: 1, column: 1]
It seems to have a problem right away with the format (line 1, column 1), but my JSON string looks ok to me. Before returning the output string in the handleRequest java function, I am writing the it to the Cloudwatch log and it writes as follows:
{
"dialogAction": {
"type": "Close",
"fulfillmentState": "Fulfilled",
"message": {
"contentType": "PlainText",
"content": "Thanks I got your info"
}
}
}
Things I've tried:
Removing the message element as it's not required
Adding in non-required properties like sessionAttributes,
responseCard, etc
removing the double quotes
replacing double quotes with single quotes
hardcoding json from sample response format message in documentation
Is there something hidden at the http headers level or is java8 doing something to the JSON that is not visible?
Not sure if this is because I'm using Java8 or not, but a return value of "String" from the RequestHandler class handleRequest method will not work.
Yes, String is an object, but the constructors on the Lex side are expecting an "Object". I was converting my lex response POJO to a String before returning it in the handleRequest method. That was my mistake.
I fixed it by changing the return type of the handleRequest method to be "Object" instead of "String".
public Object handleRequest(Object input, Context context)
instead of
public String handleRequest(Object input, Context context)
You also have to implement the
public class LambdaFunctionHandler implements RequestHandler<Object, Object>
not
public class LambdaFunctionHandler implements RequestHandler<Object, String>
This solved my issue.
In my case I was facing exactly the same issue and was able to fix it by creating specific response POJO type and using this POJO as the return type for 'handleRequest' method. E.g. BotResponse.java as follow:
public class BotResponse implements Serializable{
private static final long serialVersionUID = 1L;
public DialogAction dialogAction = new DialogAction();
public DialogAction getDialogAction() {
return dialogAction;
}
public void setDialogAction(DialogAction dialogAction) {
this.dialogAction = dialogAction;
}
}
Note, I have also added the 'implements Serializable' just to be on safer side. Probably it is an overkill.
Not sure why but for me returning a well formatted JSON String object did not worked even after changing the return type of 'handleRequest' method to 'Object'.
I know this is an old question however thought this might help some else
#Mattbob Solution dint fix my issue, However he is in the right path. Best approach is to use a Response object, a custom response object and make the lambda return the custom response object. So i went to the Documentation and created a custom object that looks Response format here
http://docs.aws.amazon.com/lex/latest/dg/lambda-input-response-format.html
At the time of answering question i couldnt find an object in SDK that matched the response Object so i had to recreate but if some one knows please comment below
Class xxxxx implements RequestHandler<Object, AccountResponse> {
#Override
public AccountResponse handleRequest(Object input, Context context) {
}
}
Lambda will look somewhat like this and just populate and return the object to match response structure and error goes away. Hope this helps.
Whenever we are returning the object to the bot from the backend make sure we need to pass content type along with content. But here we are passing wrong. So wE need to pass as like below. It is in Node.js
let message = {
contentType: "PlainText",
content: 'Testing bot'
};

Spring Integration and returning schema validation errors

We are using Spring Integration to process a JSON payload passed into a RESTful endpoint. As part of this flow we are using a filter to validate the JSON:
.filter(schemaValidationFilter, s -> s
.discardFlow(f -> f
.handle(message -> {
throw new SchemaValidationException(message);
}))
)
This works great. However, if the validation fails we want to capture the parsing error and return that to the user so they can act on the error. Here is the overridden accept method in the SchemaValidationFilter class:
#Override
public boolean accept(Message<?> message) {
Assert.notNull(message);
Assert.isTrue(message.getHeaders().containsKey(TYPE_NAME));
String historyType = (String)message.getHeaders().get(TYPE_NAME);
JSONObject payload = (JSONObject) message.getPayload();
String jsonString = payload.toJSONString();
try {
ProcessingReport report = schemaValidator.validate(historyType, payload);
return report.isSuccess();
} catch (IOException | ProcessingException e) {
throw new MessagingException(message, e);
}
}
What we have done is in the catch block we throw a MessageException which seems to solve the problem. However this seems to break what a filter should do (simply return a true or false).
Is there a best practice for passing the error details from the filter to the client? Is the filter the right solution for this use case?
Thanks for your help!
John
I'd say you go correct way. Please, refer to the XmlValidatingMessageSelector, so your JsonValidatingMessageSelector should be similar and must follow the same design.
Since we have a throwExceptionOnRejection option we always can be sure that throwing Exception instead of just true/false is correct behavior.
What Gary says is good, too, but according to the existing logic in that MessageSelector impl we can go ahead with the same and continue to use .filter(), but, of course, already without .discardFlow(), because we won't send invalid message to the discardChannel.
When your JsonValidatingMessageSelector is ready, feel free to contribute it back to the Framework!
It's probably more correct to do the validation in a <service-activator/>...
public Message<?> validate(Message<?> message) {
...
try {
ProcessingReport report = schemaValidator.validate(historyType, payload);
return message;
}
catch (IOException | ProcessingException e) {
throw new MessagingException(message, e);
}
}
...since you're never really filtering.

Resources