I have a case to publish message from remote host to rabbitmq using rabbitmq/api and i'd like to pass handling of published message to MassTransit consumer.
As i could notice, "minimal needed properties" of success handled message with MassTransit consumer looks like this:
Payload:
{
'messageId':'86fd0000-17e1-ac9e-4bde-08d3f1e8553e',
'correlationId':'86fd0000-17e1-ac9e-4165-08d3f1e8553e',
'conversationId':'86fd0000-17e1-ac9e-4d99-08d3f1e8553e',
'messageType':[
'urn:message:Tm.Core.Integration.RabbitMq.Bus.Extension:EndScenario',
'urn:message:Tm.Core.Integration.RabbitMq.Command:IEndScenario',
'urn:message:Tm.Core.Integration.RabbitMq.Command:ICommand'
],
'message':{
'correlationId':'86fd0000-17e1-ac9e-4165-08d3f1e8553e',
'scenarioId':'6c3fda36-8ded-41f0-9536-dd76cd146963'
}
}
As you can see, messageType property contains information, that helps to pass it into responsible consumer (isn't it?).
I'd like to ask you, is there a way i could remove that property and continue to handle passed like this way messages using MassTransit consumer? I'm really afraid that sooner or later some of refactoring will break my code (messageType depends on runtime type of message, isn't it?) and i'd like to prevent it.
I understand that i have to write some code of routing (mb get this message and publish another with more information) ?
My goal is to pass message like this:
{
'messageId':'86fd0000-17e1-ac9e-4bde-08d3f1e8553e',
'correlationId':'86fd0000-17e1-ac9e-4165-08d3f1e8553e',
'conversationId':'86fd0000-17e1-ac9e-4d99-08d3f1e8553e',
'message':{
'correlationId':'86fd0000-17e1-ac9e-4165-08d3f1e8553e',
'scenarioId':'6c3fda36-8ded-41f0-9536-dd76cd146963'
}
}
The messageType section of the message envelope is required. If not present, the message cannot be deserialized into a class.
The only way you can subscribe to a message with no types is to use a consumer that looks at the JToken itself, such as:
public JsonConsumer :
IConsumer<JToken>
{}
Related
I'm creating client side streaming method which proto file definition should be looking similar to this:
service DataService {
rpc Send(stream SendRequest) returns (SendResponse) {}
}
message SendRequest {
string id = 1;
bytes data = 2;
}
message SendResponse {
}
The problem here is that ID is sent with each streaming message even it is needed only once. What's your recommendation and most optimal way for such a use cases?
One hacky approach would be to set ID only once within first message and after that left if blank. But this API is supposed to be used by third party and method definition like above doesn't explaining that well.
I don't think something like this is supported either:
service DataService {
rpc Send(InitialSendRequest, stream DataOnlyRequest) returns (SendResponse) {}
}
I'm currently considering SendRequest message to be something like this, but will have to check how optimal is this compared to the first case considering proto marshaling:
message SendRequest {
oneof request{
string id = 1;
bytes data = 2;
}
}
Your approach of using a oneof with the fields clearly documented saying id is expected only in the first message on the stream and that server implementations will terminate the stream if id is set on subsequent messages on the stream sounds good to me.
The following is a usage of the above described pattern in grpc-lb-v1. Even though the grpc team is moving away from grpc-lb-v1, the above mentioned pattern is a commonly used one.
I'm not very sure about its implications with respect to proto marshaling. That might be a question for the protobuf team.
Hope this helps.
I would like to send a stream of different protobuf messages through the wires and be able to differentiate them at arrival as they are coming.
Let's say I have a *.proto like that:
message Book {
//...
}
message BlueRay{
//...
}
And then and the sender side, I serialize let's say this sequence (pseudo code in C#):
Book1.WriteDelimitedTo(myStream);
BlueRay1.WriteDelimitedTo(myStream);
Book2.WriteDelimitedTo(myStream);
How can I do to know the order/types of messages I'm getting on the receiver side? (The contract is available on both sender and receiver side of course)
Depending of my sender's state I can not presume/tell what is going to be sent and in which order...
I understood that there is no built-in way to do that like stated in the documentation, but for instance for the size of the message there was a helper (C# API helper WriteDelimited method to embedd size).
How can I do to get/map the type of a received message?
My server will be written in a given language (C# actually), but my clients should be "implementable" in any protobuf supported target, so I don not want to set up something that would serialize C#/CLR specific stuff in between...
I'm maybe using protobuf in a weird way? I'm trying to set up a kind of protocol.
I think I finally found out how to do that (only in C# at the moment).
Basically, I'm writing the following to the stream:
the coming message descriptor's target name (in proto file, as both end share this definition), using the primitive for string serilization
the coming message size, using the primitive for size serialization
the serialized message into the stream
This results in a method like the following:
public static void WriteToStream(Stream outputStream, IMessage message)
{
MessageDescriptor stateMsgDescriptor = message.Descriptor;
using (CodedOutputStream codedOutStr = new CodedOutputStream(outputStream, true))
{
codedOutStr.WriteString(stateMsgDescriptor.FullName);
int size = message.CalculateSize();
codedOutStr.WriteLength(size);
message.WriteTo(codedOutStr);
codedOutStr.Flush();
}
}
As stated here,
The Protocol Buffer wire format is not self-delimiting, so protocol
buffer parsers cannot determine where a message ends on their own
I'm trying to create a new queue, but when using
cf create-service aws-sqs standard my-q
the name of the queue in AWS is automatically assigned and is just an id composed of random letters and numbers.
This is fine when using the normal java client. However, we want to use spring-cloud-aws-messaging (#SqsListener annotation), because it offers us deletion policies out of the box, and a way to extend visibility, so that we can implement retries easily.
#SqsListener(value = "my-q", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void listen(TestItem item, Visibility visibility) {
log.info("received message: " + item);
//do business logic
//if call fails
visibility.extend(1000);
//throw exception
//if no failure, message will be dropped
}
The queue name on the annotation is declared, so we can't change it dynamically after reading the VCAP_SERVICE environment variable injected by PCF on the application.
The only alternative we can think of is use reflection to set accessibility on value of the annotation, and set the value to the name on the VCAP_SERVICE, but that's just nasty, and we'd like to avoid it if possible.
Is there any way to change the name of the queue to something specific on creation? This suggests that it's possible, as seen below:
cf create-service aws-sqs standard my-q -c '{ "CreateQueue": { "QueueName": “my-q”, "Attributes": { "MaximumMessageSize": "1024"} } }'
However, this doesn't work. It returns:
Incorrect Usage: Invalid configuration provided for -c flag. Please
provide a valid JSON object or path to a file containing a valid JSON
object.
How do I set the name on creation of the queue? Or the only way to achieve my end goal is to use reflection?
EDIT: As pointed out by Daniel Mikusa, the double quotes were not real double quotes, and that was causing the error. The command is successful now, however it doesn't create the queue with the intended name. I'm now wondering if this name needs to be set on bind-service instead. The command has a -c option too but I cannot find any documentation to support which parameters are available for a aws-sqs service.
Is it possible to extend the visibility time out of a message that is in flight.
See:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html.
Section: Changing a Message's Visibility Timeout.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQSClient.html#changeMessageVisibility-com.amazonaws.services.sqs.model.ChangeMessageVisibilityRequest-
In summary I want to be able to extend the first set visibility timeout for a given message that is in flight.
Example if 15secs have passed I then want to extend the timeout by another 20secs. Better example in java docs above.
From my understanding in the links above you can do this on the amazon side.
Below are my current settings;
SqsMessageDrivenChannelAdapter adapter =
new SqsMessageDrivenChannelAdapter(queue);
adapter.setMessageDeletionPolicy(SqsMessageDeletionPolicy.ON_SUCCESS);
adapter.setMaxNumberOfMessages(1);
adapter.setSendTimeout(2000);
adapter.setVisibilityTimeout(200);
adapter.setWaitTimeOut(20);
Is it possible to extend this timeout?
Spring Cloud AWS supports this starting with Version 2.0. Injecting a Visiblity parameter in your SQS listener method does the trick:
#SqsListener(value = "my-sqs-queue")
void onMessageReceived(#Payload String payload, Visibility visibility) {
...
var extension = visibility.extend(20);
...
}
Note, that extend will work asynchronously and will return a Future. So if you want to be sure further down the processing, that the visibility of the message is really extended at the AWS side of things, either block on the Future using extension.get() or query the Future with extension.isDone()
OK. Looks like I see your point.
We can change visibility for particular message using API:
AmazonSQS.changeMessageVisibility(String queueUrl, String receiptHandle, Integer visibilityTimeout)
For this purpose in downstream flow you have to get access to (inject) AmazonSQS bean and extract special headers from the Message:
#Autowired
AmazonSQS amazonSqs;
#Autowired
ResourceIdResolver resourceIdResolver;
...
MessageHeaders headers = message.getHeaders();
DestinationResolver destinationResolver = new DynamicQueueUrlDestinationResolver(this.amazonSqs, this.resourceIdResolver);
String queueUrl = destinationResolver.resolveDestination(headers.get(AwsHeaders.QUEUE));
String receiptHandle = headers.get(AwsHeaders.RECEIPT_HANDLE);
amazonSqs.changeMessageVisibility(queueUrl, receiptHandle, YOUR_DESIRED_VISIBILITY_TIMEOUT);
But eh, I agree that we should provide something on the matter as out-of-the-box feature. That may be even something similar to QueueMessageAcknowledgment as a new header. Or even just one more changeMessageVisibility method to this one.
Please, raise a GH issue for Spring Cloud AWS project on the matter with link to this SO topic.
Using WMQ7.0 with WMB 6.1
I have one flow where I am transforming a message and using MQRFH2.usr for holding some data.
But, I am facing the issue where the MQRFH2.usr is coming in the main message body.
I have deployed the same code in different environments, but I am getting this issue only in one environment.
So, it doesn't seems to be a code issue. It has something to do with configurations.
Kindly, suggest what could be the possible cause.
Check the queue's PROPCTL setting. If this is set to NONE then the behavior is as follows:
If the application does not create a message handle, all the message
properties are removed from the MQRFH2. Name/value pairs in the MQRFH2
headers are left in the message.
Be sure to read the doc page through a couple of times and maybe test with different settings to understand fully how PROPCTL modifies the message content your app receives.
The MQRFH2 headers, if present, always come in the payload part of the message (that's the way webpshere organizes it). You can receive one or more MQRFH2 headers (structures).
Perhaps you are expecting only one and are receiving two? This would explain your message data being left with gibberish.
I use the following code to handler these heards upon receiving a message
MQRFH2 header = null;
// Find and store message length
int msglen = replyMessage.getMessageLength();
MQHeaderList list = new MQHeaderList(replyMessage);
int indexOf = list.indexOf("MQRFH2");
if (indexOf >= 0) {
header = (MQRFH2) list.get(indexOf);
msglen = msglen - header.size();
}
String msgText = replyMessage.readStringOfCharLength(msglen);
Hope it helps.
Martins