I'm trying to set up a DLQ for a Kinesis.
I used SQS and set it as the Kinesis on failure destination.
The Kinesis is attached to a lambda that always throws an error so the event will go right away to the SQS DLQ.
I can see the events in the SQS, but that payload of the event is missing ( the json I send as part of the event ), in the lambda if I print the event before throwing the exception, I can see the base64 encoded data, but not in my DLQ.
Is there a way to send the event data to the DLQ as well? I want to be able to examine the cause of the error correctly and put the event back to the Kinesis after I finished fixing the issue in the lambda.
https://docs.aws.amazon.com/lambda/latest/dg//with-kinesis.html#services-kinesis-errors
The actual records aren't included, so you must process this record and retrieve them from the stream before they expire and are lost.
According to the above the event payload won't be sent to the DLQ event so "missing event data" is expected here.
Therefore, in order to retrieve the actual record back, you might want to try something like
1) assuming we have the following kinesis batch info
{
"KinesisBatchInfo": {
"shardId": "shardId-000000000001",
"startSequenceNumber": "49601189658422359378836298521827638475320189012309704722",
"endSequenceNumber": "49601189658422359378836298522902373528957594348623495186",
"approximateArrivalOfFirstRecord": "2019-11-14T00:38:04.835Z",
"approximateArrivalOfLastRecord": "2019-11-14T00:38:05.580Z",
"batchSize": 500,
"streamArn": "arn:aws:kinesis:us-east-2:123456789012:stream/mystream"
}
}
2) we can get the record back by doing something like
import AWS from 'aws-sdk';
const kinesis = new AWS.Kinesis();
const ShardId = 'shardId-000000000001';
const ShardIteratorType = 'AT_SEQUENCE_NUMBER';
const StreamName = 'my-awesome-stream';
const StartingSequenceNumber =
'49601189658422359378836298521827638475320189012309704722';
const { ShardIterator } = await kinesis
.getShardIterator({
ShardId,
ShardIteratorType,
StreamName,
StartingSequenceNumber,
})
.promise();
const records = await kinesis
.getRecords({
ShardIterator,
})
.promise();
console.log('Records', records);
NOTE: don't forget to make sure your process has permission to 1) kinesis:GetShardIterator 2) kinesis:GetRecords
Hope that helps!
Related
In MassTransit Send and RequestClient will be mapped to exchange or queue, That will be handled by LoadBalanced Consumer.
But for Publish Message, It will be consumed by all the instances that are running and waiting for the Message.
So, In StateMachine, Consumer has to publish the Events, That will make if more than once StateMachine Instance running it will be Picked by both StateMachine and Process will be duplicated? This is what happening at my work. So, We end up running Single StateMachine Instance.
await context.Publish(new
{
context.Message.OrderId,
context.Message.Timestamp,
context.Message.CustomerNumber,
context.Message.PaymentCardNumber,
context.Message.Notes
});
This publishes the events to Saga, if Saga is running in LoadBalancer. Both Instance will be receiving the SameEvent. And Start Processing the Event and changing the Next State.
If this is the Case, How to solve this. Only one StateMachine Should Pick the published message at once.
We end up running Single StateMachine Instance. So, the Published message wont be picked by both instance and will endup haivng duplicate process.
The Current Implmentation:
Have a REST Api - That receives the request to Start the Initial State.
var sendToUri =
new Uri(
$"rabbitMq://{_rabbitMqConfig.Host}/{_rabbitMqConfig.VirtualHost}-{_rabbitMqConfig.WfSagaQueue}");
var endPoint = await bus.GetSendEndpoint(sendToUri);
var req = wfRequest;
await endPoint.Send<IWfExecRequest>(req);
In the StateMachine :
services.AddMassTransit(x =>
{
x.AddConsumer<WfExecRequestConsumer>();
x.AddConsumer<WfTaskCompletedConsumer>();
x.UsingRabbitMq((context, cfg) =>
{
var wfTaskExecHandler = context.GetRequiredService<IWfTaskExecHandler>();
var wfManagementClient = context.GetRequiredService<IWfManagementClient>();
var wfSagaStateMachine = new MsrAutomationStateMachine(wfTaskExecHandler, wfManagementClient);
cfg.Host(HostCredets);
cfg.ReceiveEndpoint(queueName: "msr-automation-wf-exec-request", configureEndpoint: e =>
{
e.PrefetchCount = 1;
e.ConfigureConsumer<WfExecRequestConsumer>(context);
e.StateMachineSaga(wfSagaStateMachine, repo);
});
cfg.ReceiveEndpoint(queueName: "WfTaskCompleted", configureEndpoint: e =>
{
e.PrefetchCount = 1;
e.ConfigureConsumer<WfTaskCompletedConsumer>(context);
});
});
});
This StateMachine Receives , WfExecRequest (Inital Event), TaskCompleted and TaskFaulted (From Muliple Consumer Saga/Consumer) - This was done at Consumer Side as Context.Publish.
So, What I see if we Run more than one Instance of the same StateMachine the TaskCompled Message getting Consumed by both Instances.
Thanks Again.
First, clearly something is wrong with your configuration. If the saga state machine is running on a single queue (receive endpoint, regardless of how many instances of your service are running) it will automatically load balance on that single queue across all running instances.
If you are running multiple instances of the saga state machine on different queues, well yeah, you're basically doing it wrong.
Second, I'm not sure what "LoadBalancer" is but typically something that unnecessary when using a message broker. If "LoadBalancer" is something for your HTTP/API endpoints, that's fine, but the broker and the queue are the scale out points in a message-based system.
If you had posted actual code, or shared some explicit details that would help as it is now it's entirely based on supposition.
How to connect an endpoint(exchange-exchange-queue) in masstransit, accumulate data in the queue, and then, after some time, connect a consumer to this endpoint?
I wanted to do something like:
Task.Run(async () =>
{
for (var i = 0;; i++)
{
await _bus.Publish(new Event(i), stoppingToken);
await Task.Delay(1_000, stoppingToken);
}
});
// a command comes to connect the consumer
var endpoint = _bus.ConnectReceiveEndpoint();
await endpoint.Ready;
// I prepare the consumer, as soon as it is ready, I connect it,
then I read the data that has accumulated during the preparation
(my consumer needs to load the state before reading the data,
and also cannot skip the data during its preparation)
endpoint.ReceiveEndpoint.ConnectConsumer(() =>
_serviceProvider.GetRequiredService<EventConsumer>());
but this code will not create an exchange-exchange relationship, so the queue will be empty
If the exchange bindings (wired to the receive endpoint, which ultimately is a queue) do not exist when messages are published, they are discarded by RabbitMQ.
You would need to connect the receive endpoint in advance, so that the messages end up in the queue.
I'm running a MassTransit configuration with AmazonSQS. In my program I start by creating a receiveenpoint with the queue "input-queue1", I subscribe this SQS queue to an SNS topic named "topic1" and associate a consumer to this receiveendpoint that does some standard printing of the messages it receives. After starting the bus i want to subscribe the already created queue "input-queue1" to another SNS topic, named "topic2", but I couldn't find a way of doing this after starting the Bus (It's important to me that i can do this after the Bus is started). Is there a way of doing this and i'm just missing something, or is it not possible at all? (I tried with the commented portion of the code but it didn't work)
class Program
{
static async Task Main(string[] args)
{
var bus = Bus.Factory.CreateUsingAmazonSqs(x =>
{
x.Host(Constants.Region, h =>
{
h.AccessKey(Constants.AccesskeyId);
h.SecretKey(Constants.SecretAccessKey);
});
x.ReceiveEndpoint("input-queue1", e =>
{
e.Subscribe("topic1", callback => { });
e.Consumer(() => new Handler());
});
});
bus.StartAsync().Wait();
/*var handle = bus.ConnectReceiveEndpoint("input-queue1", e => {
e.Subscribe("topic2", callback => { });
});
var ready = await handle.Ready;*/
Console.WriteLine("Listening to messages...");
Console.WriteLine("Press enter to quit");
Console.ReadLine();
}
}
You can't change the topology of a receive endpoint once it has been created. This means that no new topic subscriptions can be created, and existing subscriptions cannot be removed.
If you need to change the configuration of the receive endpoint, you would need to do it yourself by using the SNS API to add the subscription yourself. I would question why you would want to do this though. If the consumer isn't able to consume the message forwarded to the queue, it would be moved to the skipped queue.
I have created a Lambda function which is triggered by a DynamoDB stream. I am trying to process Dynamodb events and put them into a Kinesis stream after some transformation. The Lambda has full access to both DynamoDB and Kinesis stream.
I am using Cloudwatch to check the logs and can see that the DynamoDb events are successfully processed. But when I try to create the Kinesis client (present in a different class), the code fails. I tried logging the error and even printing it but it did not help. Sometimes the logs end with this message
END RequestId: {some request id}
Other times, I get the following error
log4j:WARN No appenders could be found for logger (com.amazonaws.AmazonWebServiceClient).
The code fails at the time of creation of Kinesis client. I can see the log messages / print statements before the creation of Kinesis client. But right at that line code fails. I am not sure what the problem is. Can someone please help me out?
Here is the class in which the code fails
private AmazonKinesis kinesisClient;
private String streamName;
public TestKinesisPut(String streamName) {
this.streamName = streamName;
BasicAWSCredentials awsCreds = new BasicAWSCredentials("ACCESS_KEY", "SECRET_KEY");
System.out.println("aws creds are: " + awsCreds);
clientBuilder = AmazonKinesisClientBuilder.standard().withRegion(Regions.AP_SOUTH_1).
withCredentials(new AWSStaticCredentialsProvider(awsCreds));
System.out.println("Credentials are set: \n " + clientBuilder);
try {
System.out.println("This one is new \n About to build new kinesis client");
// the code fails after this line
kinesisClient = clientBuilder.build();
System.out.println("failed to build client");
}
catch(Exception e) {
System.out.println("failed to initialize producer: " + e.getMessage());
kinesisClient = null;
}
}
Thanks
After a few days of head scratching I decided to tinker with the configuration of my Lambda function. Looks like the problem was caused by OutOfMemoryError. I increased the memory of my Lambda function and it started working.
It seems that at the time of creation of the KinesisClient, the JVM was getting out of metaspace. I did some research and found this stackoverflow thread. Please refer the link to view a detailed discussion on a similar scenario.
When a subscriber of a topic throws an exception non-handled message lands in {subscribername}_error queue.
Given the example:
const string subsriberName = "AnotherSubscriber";
cfg.SubscriptionEndpoint<AnotherThingHappened>(host, subsriberName, configurator =>
{
configurator.Handler<AnotherThingHappened>(context =>
{
Console.WriteLine(context.Message.AnotherThingType);
if (Random.NextDouble() < 0.1)
{
throw new Exception("Oups, I failed :(");
}
return Task.CompletedTask;
});
});
It created "AnotherSubscriber" subscription on topic ObjectCreatedA. But when it fails the message goes to the queue anothersubscriber_error. It makes it harder to diagnose, monitor and replay messages. Because from ASB perspective this is just an ordinary queue.
How do I route failures to the DLQ of topic ObjectCratedA/AnotherSubscriber instead of **_error one?
Thanks in advance.
This is now possible as of MassTransit 6.2, see the related GitHub issue.
Your configuration will now need to look something like:
cfg.SubscriptionEndpoint(
"my-subscription",
"my-topic",
e =>
{
e.ConfigureConsumer<MyConsumer>(provider);
// Send failures to built-in Azure Service Bus Dead Letter queue
e.ConfigureDeadLetterQueueDeadLetterTransport();
e.ConfigureDeadLetterQueueErrorTransport();
});