At times we are observing issue with Kafka Streams where the same record in getting processed multiple times. below snapshot from log will help :
{
"timestamp": "1652121164192",
"level": "INFO",
"thread": "EntityChangeTopic-7c6877ac-32ee-4e31-8a0b-b9ae02602d72-StreamThread-17",
"mdc": {
"sourceId": "UU",
"serviceType": "k2k",
"recordId": "875812581",
"External_Input_Topic_Name": "topic",
"Action": "AddOrUpdateRequest",
"clientId": "0.2.16",
"elasticapmtraceparent": "\u0000\u0000\ufffd\ufffd[\ufffd\ufffd\u0000&ҼX\u0010>I\ufffd\u0007\u0001\ufffdWh9z\nF\ufffd\u0002\u0000",
"output_format": "ecm",
"format": "icj",
"correlationId": "3591e033-cfb7-11ec-951a-75c1535081ae",
"external_correlationId": "0ad01109-b278-4a69-910d-7acf7db24bab",
"tenantRecord": "true"
},
"logger": "com.optum.hcm.kafka.streams.ECRProcessor",
"message": "processing ecr kafka message with key, correlationId : 0.2.16, 3591e033-cfb7-11ec-951a-75c1535081ae",
"context": "default"
Around 300sec(default max.poll.interval.ms) later I see another log with same message getting processed.
{
"timestamp": "1652121470436",
"level": "INFO",
"thread": "EntityChangeTopic-7c6877ac-32ee-4e31-8a0b-b9ae02602d72-StreamThread-18",
"mdc": {
"sourceId": "UU",
"serviceType": "k2k",
"recordId": "875812581",
"External_Input_Topic_Name": "topic",
"Action": "AddOrUpdateRequest",
"clientId": "0.2.16",
"elasticapmtraceparent": "\u0000\u0000\ufffd\ufffd[\ufffd\ufffd\u0000&ҼX\u0010>I\ufffd\u0007\u0001\ufffdWh9z\nF\ufffd\u0002\u0000",
"output_format": "ecm",
"format": "icj",
"correlationId": "3591e033-cfb7-11ec-951a-75c1535081ae",
"external_correlationId": "0ad01109-b278-4a69-910d-7acf7db24bab",
"tenantRecord": "true"
},
"logger": "com.optum.hcm.kafka.streams.ECRProcessor",
"message": "processing ecr kafka message with key, correlationId : 0.2.16, 3591e033-cfb7-11ec-951a-75c1535081ae",
"context": "default"
}
I am trying to understand the behavior of in case the processing time took why the same message is getting processed again and again by different thread?
Also if the consumer(Stream Thread) is removed after max.poll.interval.ms and all assigned task are given to another thread, when it's determined to bring the original consumer(Stream thread) back? Will original consumer reprocess the record?
Related
I have an AWS Lamda that is configured to receive messages from an Amazon MQ (Apache ActiveMQ) via an MQ trigger. I depend on the destination object shown in the example MQ record event in the documentation to know from which queue the message is coming. However, in reality, the message object has an empty destination object, and I am unsure why. The object looks good otherwise.
This is what the example looks like in the AWS documentation.
{
"eventSource": "aws:mq",
"eventSourceArn": "arn:aws:mq:us-west-2:111122223333:broker:test:b-9bcfa592-423a-4942-879d-eb284b418fc8",
"messages": [
{
"messageID": "ID:b-9bcfa592-423a-4942-879d-eb284b418fc8-1.mq.us-west-2.amazonaws.com-37557-1234520418293-4:1:1:1:1",
"messageType": "jms/text-message",
"deliveryMode": 1,
"replyTo": null,
"type": null,
"expiration": "60000",
"priority": 1,
"correlationId": "myJMSCoID",
"redelivered": false,
"destination": {
"physicalname": "testQueue"
},
"data":"QUJDOkFBQUE=",
"timestamp": 1598827811958,
"brokerInTime": 1598827811958,
"brokerOutTime": 1598827811959
},
{
"messageID": "ID:b-9bcfa592-423a-4942-879d-eb284b418fc8-1.mq.us-west-2.amazonaws.com-37557-1234520418293-4:1:1:1:1",
"messageType": "jms/bytes-message",
"deliveryMode": 1,
"replyTo": null,
"type": null,
"expiration": "60000",
"priority": 2,
"correlationId": "myJMSCoID1",
"redelivered": false,
"destination": {
"physicalname": "testQueue"
},
"data":"LQaGQ82S48k=",
"timestamp": 1598827811958,
"brokerInTime": 1598827811958,
"brokerOutTime": 1598827811959
}
]
}
This is what the event object looks like in reality as JSON when logged out.
This is the MQ Trigger.
This is what the object looks like in the Active MQ console.
I am currently building an application for which it is important to check the existence of resources with a certain profile.
As we need to check this for 40+ profiles I'd like to put this all in 1 batch request and let our HAPI-FHIR server implementation handle this, as opposed to querying them one by one. This would get too chatty otherwise.
Because I only need to know about whether the resource exists I'd like to use _summary=count. I am assuming this increases the performance of the request.
Example request
{
"resourceType": "Bundle",
"type": "batch",
"entry": [
{
"request": {
"method": "GET",
"url": "/Observation?_profile=http://nictiz.nl/fhir/StructureDefinition/zib-DrugUse&_summary=true"
}
},
{
"request": {
"method": "GET",
"url": "/RelatedPerson?_profile=http://fhir.nl/fhir/StructureDefinition/nl-core-relatedperson&_summary=count"
}
}
]
}
Response
{
"resourceType": "Bundle",
"id": "fd66cfd9-4693-496d-86fc-98289067480b",
"type": "batch-response",
"link": [
{
"relation": "self",
"url": "<redacted>"
}
],
"entry": [
{
"resource": {
"resourceType": "Bundle",
"id": "2647a49f-0503-496b-b274-07d4e9163f1b",
"meta": {
"lastUpdated": "2021-02-15T11:44:18.035+00:00",
"tag": [
{
"system": "http://hl7.org/fhir/v3/ObservationValue",
"code": "SUBSETTED",
"display": "Resource encoded in summary mode"
}
]
},
"type": "searchset",
"total": 48
},
"response": {
"status": "200 OK"
}
},
{
"resource": {
"resourceType": "Bundle",
"id": "2f9cc861-5d20-4da1-aa9f-12153b75539d",
"meta": {
"lastUpdated": "2021-02-15T11:44:18.151+00:00",
"tag": [
{
"system": "http://hl7.org/fhir/v3/ObservationValue",
"code": "SUBSETTED",
"display": "Resource encoded in summary mode"
}
]
},
"type": "searchset",
"total": 10
},
"response": {
"status": "200 OK"
}
}
]
}
Can I assume that the ordering of the batch-response is the same as that of the batch-request?
Or is there a method to annotate the batch entries which are persisted onto the batch-response?
Or finally, is there a flag I can turn on to make the response include the request.url part?
I'm using HAPI-FHIR 5.1.0 both for client and server.
Apparently I didn't look well enough in the specs, as I just found the following:
From the FHIR spec
For a batch, or a successful transaction, the response the server SHALL return a Bundle with type set to batch-response or transaction-response that contains one entry for each entry in the request, in the same order, with the outcome of processing the entry.
i am trying to understand the receiving model from the kafka which is emitted from neo4j streams to one kafka topic.
Actually that data is looing like below.
{
"meta": {
"timestamp": 1608725179199,
"username": "neo4j",
"txId": 16443,
"txEventId": 0,
"txEventsCount": 1,
"operation": "created",
"source": {
"hostname": "9945"
}
},
"payload": {
"id": "1000",
"before": null,
"after": {
"properties": {
"name": "sdfdf"
},
"labels": [
"aaq"
]
},
"type": "node"
},
"schema": {
"properties": {
"name": "String"
},
"constraints": []
}
}
so to consume this type of complex structured data in springboot from kafka we need to create nested model?
i mean nearly 4 classes nested each other?
From my understanding i am trying to create the below classes.
meta (**This is 1st class**)
"operation": "created",
payload(**This is 2ndclass which is nested inside the Meta**)
id
before
after(**it is 3rd class which is nested to payload**)
properties(**it is 4th class which is nested with in after**)
these data only we needed to store
labels
type
Actually i didn't faced thee kind of nested issue before so don't have idea to procees?
so is the above method is right or any other possiblities are there?
Have to consume the data from kafka topic which is emitted by neo4j streams is the ultimate goal?
Language : Java
Framework: Springboot
I have one PostConfirmation lambda trigger that adds data to DynamoDB via GraphQL (appsync https call) and then I query for that info in the PreTokenGeneration lambda.
When I test manually via my app UI things work.
But when executing via Jest tests, 50%+ of the time I get an error due to the supposed record info not being found in DynamoDB.
The problem doesn't occur when I test manually via UI app. Only when executing via Jest test.
I checked the Cloudwatch timestamps for the PostConfirmation DynamoDB record addition, PreTokenGeneration and checked the createDate in DynamoDB. The timestamps look ok.
For instance:
The PostConfirmation log entry that says the record was added has the timestamp at 2020-08-24T17:51:06.463Z.
The DynamoDB createdAt for the added record (createdAt) says the record was created at 2020-08-24T17:51:06.377Z.
The PostConfirmation lambda "END RequestId: xxxxx" has the timestamp at 2020-08-24T17:51:06.465-05:00
The PreTokenGeneration lambda starts at 2020-08-24T17:51:12.866Z and at 2020-08-24T17:51:13.680Z the query result says it didn't find any record.
Can someone help me or give me a hint about why this happen and/or how can I troubleshoot this problem? Thank you in advance.
Taking into account the answers from #noel-llevares I modified the VTL template to include the ConsistentRead=true but the problem remains.
Heres is the RequestMapping logged for the save operation
{
"logType": "RequestMapping",
"path": [
"createAccountMember"
],
"fieldName": "createAccountMember",
"resolverArn": "arn:aws:appsync:xx-xxxx-x:111111111:apis/<redacted>/types/Mutation/resolvers/createAccountMember",
"requestId": "<redacted>",
"context": {
"arguments": {
"input": {
"id": "<redacted>",
"userID": "<redacted>",
"accountID": "<redacted>",
"membershipStatus": "active",
"groupsEnrolledIn": [
<redacted>
],
"recordOwner": "<redacted>",
"createdAt": "2020-08-25T05:11:10.917Z",
"updatedAt": "2020-08-25T05:11:10.917Z",
"__typename": "AccountMember"
}
},
"stash": {},
"outErrors": []
},
"fieldInError": false,
"errors": [],
"parentType": "Mutation",
"graphQLAPIId": "<redacted>",
"transformedTemplate": "\n\n\n\n\n\n\n\n{\n \"version\": \"2018-05-29\",\n \"operation\": \"PutItem\",\n \"key\": {\n \"id\": {\"S\":\"<redacted>\"}\n} ,\n \"attributeValues\": {\"accountID\":{\"S\":\"<redacted>\"},\"createdAt\":{\"S\":\"2020-08-25T05:11:10.917Z\"},\"recordOwner\":{\"S\":\"<redacted>\"},\"__typename\":{\"S\":\"AccountMember\"},\"id\":{\"S\":\"<redacted>\"},\"membershipStatus\":{\"S\":\"active\"},\"userID\":{\"S\":\"<redacted>\"},\"groupsEnrolledIn\":{\"L\":[{\"S\":\"<redacted>\"},{\"S\":\"<redacted>\"},{\"S\":\"<redacted>\"}]},\"updatedAt\":{\"S\":\"2020-08-25T05:11:10.917Z\"}},\n \"condition\": {\"expression\":\"attribute_not_exists(#id)\",\"expressionNames\":{\"#id\":\"id\"}}\n}\n"
}
The ResponseMapping logged for the save operation
{
"logType": "ResponseMapping",
"path": [
"createAccountMember"
],
"fieldName": "createAccountMember",
"resolverArn": "<redacted>",
"requestId": "<redacted>",
"context": {
"arguments": {
"input": {
"id": "<redacted>",
"userID": "<redacted>",
"accountID": "<redacted>",
"membershipStatus": "active",
"groupsEnrolledIn": [
"<redacted>",
"<redacted>",
"<redacted>"
],
"recordOwner": "<redacted>",
"createdAt": "2020-08-25T05:11:10.917Z",
"updatedAt": "2020-08-25T05:11:10.917Z",
"__typename": "AccountMember"
}
},
"result": {
"accountID": "<redacted>",
"createdAt": "2020-08-25T05:11:10.917Z",
"recordOwner": "<redacted>",
"__typename": "AccountMember",
"id": "<redacted>",
"membershipStatus": "active",
"userID": "<redacted>",
"groupsEnrolledIn": [
"<redacted>",
"<redacted>",
"<redacted>"
],
"updatedAt": "2020-08-25T05:11:10.917Z"
},
"stash": {},
"outErrors": []
},
"fieldInError": false,
"errors": [],
"parentType": "Mutation",
"graphQLAPIId": "<redacted>",
"transformedTemplate": "{\"accountID\":\"<redacted>\",\"createdAt\":\"2020-08-25T05:11:10.917Z\",\"recordOwner\":\"<redacted>\",\"__typename\":\"AccountMember\",\"id\":\"<redacted>\",\"membershipStatus\":\"active\",\"userID\":\"<redacted>\",\"groupsEnrolledIn\":[\"<redacted>\",\"<redacted>\",\"<redacted>\"],\"updatedAt\":\"2020-08-25T05:11:10.917Z\"}\n"
}
Here's is the Request mapping logged for the list operation. You can see the consistentRead=true
{
"logType": "RequestMapping",
"path": [
"listAccountMembers"
],
"fieldName": "listAccountMembers",
"resolverArn": "<redacted>",
"requestId": "<redacted>",
"context": {
"arguments": {
"filter": {
"userID": {
"eq": "<redacted>"
}
}
},
"stash": {},
"outErrors": []
},
"fieldInError": false,
"errors": [],
"parentType": "Query",
"graphQLAPIId": "<redacted>,
"transformedTemplate": " \n{\"version\":\"2018-05-29\",\"limit\":100,\"consistentRead\":true,\"filter\":{\"expression\":\"(#userID = :userID_eq)\",\"expressionNames\":{\"#userID\":\"userID\"},\"expressionValues\":{\":userID_eq\":{\"S\":\"<redacted>\"}}},\"operation\":\"Scan\"}"
}
Here is the responseMapping logged. You can see the result is an empty array (items:[]) even though the record has been added previously and we have specified consistentRead=true for the query.
{
"logType": "ResponseMapping",
"path": [
"listAccountMembers"
],
"fieldName": "listAccountMembers",
"resolverArn": "<redacted>",
"requestId": "<redacted>",
"context": {
"arguments": {
"filter": {
"userID": {
"eq": "<redacted>"
}
}
},
"result": {
"items": [],
"nextToken": "<redacted>",
"scannedCount": 100
},
"stash": {},
"outErrors": []
},
"fieldInError": false,
"errors": [],
"parentType": "Query",
"graphQLAPIId": "<redacted>",
"transformedTemplate": "\n{\"items\":[],\"nextToken\":\"<redacted>",\"scannedCount\":100,\"startedAt\":null}\n"
}
What else could I be missing?
UPDATE02
I found the possible cause. It's beacause I'm new to how DynamoDB works. The query or scan operations get the results by key. In this case there is no key involved, so it gets all the records taking the limit into account. In my case it's 100, and then it applies the filter. So if the record added is not in the first 100 results it can't find it unless I go through the paging (not good for muy specific need).
TL;DR: I changed the query to use a #key directive with userID as the key field and the problem has gone because que field is a GSI and the number of records I expect to retrieve with such partition is much less than the 100 limit. I'll add this as part of the answer as soon as I finish undoing the tweaks i previously made.
DynamoDB is eventually consistent by default.
According to the documentation,
When you read data from a DynamoDB table, the response might not reflect the results of a recently completed write operation. The response might include some stale data. If you repeat your read request after a short time, the response should return the latest data.
If you need to read what you just wrote immediately, you can opt to use strongly consistent reads. This is usually done by specifying ConsistentRead to true in your DynamoDB calls.
I found the root cause. The query or scan operations get the results by key. In this case there is no key involved, so it gets all the records taking the limit into account, in my case it's 100, and then it applies the filter. So if the record added is not in the first 100 results it can't find it unless I go through paging (not good for my specific need). I didn't notice this because I'm new to how DynamoDB works. But thank to #noel-llevares I went through more in-depth research to find a solution.
The solution was to change the query to use a #key directive with name "byUsername" that exists in the AccountMember type with userID as the key field and the problem has gone because que field is a GSI and the number of records I expect to retrieve with such partition is much less than the 100 limit.
Based on Kinesis documentation, sequence number is supposed to be unique, however we see the same value being reused across multiple records. Our event producer is Spring Boot application that uses KPL internally, consumers are AWS lambdas. We have performed a re-sharding a couple times during the test. Below you can see sample sequence number reused more than once. How that's even possible?
"Records": [{
"kinesis": {
"kinesisSchemaVersion": "1.0",
"partitionKey": "00000000000000002",
"sequenceNumber": "49596124085897508159438713510240079964989152308217511954",
"data": "************************",
"approximateArrivalTimestamp": 1558991793.009
},
"eventSource": "aws:kinesis",
"eventVersion": "1.0",
"eventID": "shardId-000000000001:49596124085897508159438713510240079964989152308217511954",
"eventName": "aws:kinesis:record",
"invokeIdentityArn": "-----------------",
"awsRegion": "us-east-1",
"eventSourceARN": "-----------------"
}, {
"kinesis": {
"kinesisSchemaVersion": "1.0",
"partitionKey": "00000000000000003",
"sequenceNumber": "49596124085897508159438713510240079964989152308217511954",
"data": ""************************",",
"approximateArrivalTimestamp": 1558991793.009
},
"eventSource": "aws:kinesis",
"eventVersion": "1.0",
"eventID": "shardId-000000000001:49596124085897508159438713510240079964989152308217511954",
"eventName": "aws:kinesis:record",
"invokeIdentityArn": "-----------------",
"awsRegion": "us-east-1",
"eventSourceARN": "-----------------"
}, {
"kinesis": {
"kinesisSchemaVersion": "1.0",
"partitionKey": "00000000000000004",
"sequenceNumber": "49596124085897508159438713510240079964989152308217511954",
"data": ""************************",",
"approximateArrivalTimestamp": 1558991793.009
},
"eventSource": "aws:kinesis",
"eventVersion": "1.0",
"eventID": "shardId-000000000001:49596124085897508159438713510240079964989152308217511954",
"eventName": "aws:kinesis:record",
"invokeIdentityArn": "-----------------",
"awsRegion": "us-east-1",
"eventSourceARN": "-----------------"
}]
When Kinesis stream writers use KPL with user record aggregation (see Consumer De-aggregation) user records are batched together and delivered as a single Kinesis record to regular Kinesis consumers. Kinesis record sequence numbers are unique in this case, but we need to implement de-aggregation.
However, in case the enhanced fan-out is enabled for Lambdas, user records are delivered as individual Kinesis records (no de-aggregation is required) and they share the same sequence number.
So the Kinesis record sequence number is not always unique.