I have a DynamoDB that is getting data written to it from a java app
mapper.save(row, DynamoDBMapperConfig.builder().withSaveBehavior(SaveBehavior.CLOBBER).build());
I want to have a lambda trigger off of new items in the DDB so that their keys can be put onto SNS. However, if an item in DynamoDB is being overwritten (IE we received duplicate data) we do NOT want to do anything with it.
How to handle that? I control both the lambda and the code writing to DDB, but not the source of the data.
I don't think we can prevent the Lambda from being triggered when an item is overwritten in DynamoDB. But once the Lambda is triggered, we can identify if it's a new record or an existing record.
The input to the Lambda function will be a DynamoDBStreamEvent which contains an OldImage attribute. If that is present, it indicates that it's an existing record that got modified. In this case, we can just return from the Lambda without doing any processing.
Also, the event contains the entire snapshot before and after the insert in the OldImage and NewImage attributes. So we can also check whether some other attribute value has changed or not to decide whether it's an overwrite.
You need to have an IF, CASE, or something that looks at the stream record's eventName and if it is INSERT, which means new if I recall correctly, then it will run your code. If it is something like MODIFY, then it will not. There is an example in the DynamoDB documentation.
Related
my problem:
I am busy developing a Vue web app using Amplify DataStore.
In the web app, when I save a new Item to the DataStore I am unable to immediately access that item using a Lambda function and AWS.DynamoDB.DocumentClient() to query the dynamoDb table using await docClient.query(params).promise(). The result is empty.
If I wait a minute or two after saving the item, I am able to access the item in my lambda function.
I assume that there is a delay in syncing the locally saved item to the backend.
The lambda function is called by a third party via an API immediately after the new item is saved in the web app.
How can I ensure that the item is available to access in my lambda function?
You should wait for a 200 response when saving the item, and then issue a strongly consistent read from DynamoDB which will ensure it returns the latest result:
ConsistentRead: true
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html#query-property
DynamoDB works on a leader/follower architecture, all writes are persisted by the leader and later replicated to the followers (in milliseconds). It's possible that you are reading from a follower and thus getting an empty response and DynamoDB has not yet replicated that write. Setting ConsistentRead to true will force the read to happen on the leader, which will return the item, so long as your save API had already returned a 200.
found a solution, I use DataStore.observe to observe changes in the item (model) that I have saved - msg.opType === 'INSERT' is triggered on first saving (in local storage) and msg.opType === 'UPDATE' when the item is written in the backend dynamoDb table.
Also, these attributes and values are returned in update
"createdAt": "2022-12-23T13:23:57.978Z",
"updatedAt": "2022-12-23T13:23:57.978Z",
"owner": "test",
"_version": 1,
"_lastChangedAt": 1671801837996,
"_deleted": null
and not in first save. So by using a conditional statement I can ensure that the observed item matches mine and then I can trigger my code knowing that the data is available in the dynamoDb table.
Hope this helps anyone coming across a similar issue as mine.
I would like to do some cleanup after a record has been deleted in my DynamoDB table. It would be pretty great if I could use triggers to do this. Unfortunately it seems that OldImage is not passed into "REMOVE" events. The problem is that I need some record attributes other than the keys in order to perform my cleanup and I can't actually read the record anymore to get these attributes once the event has triggered. Is there any other way I can still read attributes of a record that has been deleted in a trigger?
Change the DynamoDB stream to include new and old images.
Is it possible to write a validation function to ensure a field of a new document is unique?
Imagine I'm trying to write a validation function that does not allow for two users to have the same email. Every time I create a new user, the validation function will be called and will probably look something like this:
function (newDoc, oldDoc) {
//How do I get this array to contain the emails of all the users?
var allEmail;
if (allEmail.indexOf(newDoc.email) !== -1) {
throw "This email adress is already taken";
}
};
How can I fill the array allEmail to contain all emails of the users?
Is it possible to call views in the validation function?
Not possible. Validation function only operates with updated doc and his previous revision and it cannot access to other documents. The only field that guaranteed to be unique is document _id. If it's possible and doesn't produce security/privacy issues, use email as doc _id to be ensure that it's unique.
Otherwise you have to create a view with target field as a key and check for it existence first before create a new doc on the client side. However, this logic easily becomes ruined when you'll replicate docs from other instance.
If the application is in offline, the above suggested solution how will react. Local pouch view can check and return the pouch results alone. There may be a high chance of same value entered from some other end and updated to couch db.
Do you have workaround for this case ?
I have some questions regarding Cloud Code (Couldn't find details in the docs).
Can triggers (afterSave...) trigger other triggers ? Example: afterSave("Post"...) creates a new row in Comments where there's also an afterSave function attached.
Is there an afterInsert event ? Or do we need to use beforeSave and check if objectId is null ? Or is there another way to check such thing ? (I need to trigger certain function only after insert, not modification)
Thanks
The triggers will fire regardless of the source of the change. So afterSave would fire for a row created in another afterSave handler.
There isn't an afterInsert; only a afterSave. You can't check if objectId === null in afterSave because it would always be false (it was just saved, after all). You can use Parse.Object.isNew to check if it is new. I believe it still returns true in afterSave for a new object.
There is custom field "Lock Flag" in Account BC, namely in S_ORG_EXT_X table. This field is made available in Opportunity BC using join to above table. The join specification is as follows: Opportunity.Account Id = Account.Id. Account Id is always populated when creating new opportunity. The requirement is that for newly created records in Opportunity BC if "Lock Flag" is equal to 'Y', then we should not allow to create the record and we should show custom error message.
My initial proposal was to use a Runtime Event that is calling Data Validation Manager business service where validation rule is evaluated and error message shown. Assuming that we have to decide whether to write record or not, the logic should be placed in PreWriteRecord event handler as long as WriteRecord have row already commited to database.
The main problem was how to determine if it is new record or updated one. We have WriteRecordNew and WriteRecordUpdated runtime events but they are fired after record is actually written so it doesn't prevent user from saving record. My next approach was to use eScript: write custom code in BusComp_PreWriteRecord server script and call BC's method IsNewRecordPending to determine if it is new record, then check the flag and show error message if needed.
But unfortunately I am faced with another problem. That joined field "Lock Flag" is not populated for newly created opportunity records. Remember we are talking about BC Opportunity and field is placed in S_ORG_EXT_X table. When we create new opportunity we pick account that it belongs to. So it reproduceable: OpportunityBC.GetFieldValue("Lock Flag") returns null for newly created record and returns correct value for the records that was saved previously. For newly created opportunities we have to re-query BC to see "Lock Flag" populated. I have found several documents including Oracle's recomendation to use PreDefaultValue property if we want to display joined field value immediately after record creation. The most suitable expression that I've found was Parent: BCName.FieldName but it is not the case, because active BO is Opportunity and Opportunity BC is the primary one.
Thanks for your patience if you read up to here and finally come my questions:
Is there any way to handle PreWrite event and determine if it is new record or not, without using eScript and BC.IsNewRecordPending method?
How to get value of joined field for newly created record especially in PreWriteRecord event handler?
It is Siebel 8.1
UPDATE: I have found an answer for the first part of my question. Now it seems so simple to me that I am wondering how I haven't done it initially. Here is the solution.
Create Runtime Event triggered on PreWriteRecord. Specify call to Data Validation Manager business service.
In DVM create a ruleset and a rule where condition is
NOT(BCHasRows("Opportunity", "Opportunity", "[Id]='"+[Id]+"'", "AllView"))
That's it. We are searching for record wth the same Row Id. If it is new record there should't be anything in database yet (remember that we are in PreWriteRecord handler) and function returns FALSE. If we are updating some row then we get TRUE. Reversing result with NOT we make DVM raise an error for new records.
As for second part of my question credits goes to #RanjithR who proposed to use PickMap to populate joined field (see below). I have checked that method and it works fine at least when you have appropriate PickMap.
We Siebel developers have used scripting to correctly determine if record is new. One non scripting way you could try is to use RuntimeEvents to set a profileattribute during the BusComp NewRecord event, then check that in the PreWrite event to see if the record is new. However, there is always a chance that user might undo a record, those scenarios are tricky.
Another option, try invokine the BC Method:IsNewRecordPending from RunTime event. I havent tried this.
For the second part of the query, I think you could easily solve your problem using a PickMap.
On Opportunity BC, when your pick Account, just add one more pickmap to pick the Locked flag from Account and set it to the corresponding field on Opportunity BC. When the user picks the Account, he will also pick the lock flag, and your script will work in PreWriteRecord.
May I suggest another solution, again, I haven't tried it.
When new records are created, the field ModificationNumber will be set to 0. Every time you modify it, the ModificationNumber will increment by 1.
Set a DataValidationManager ruleset, trigger it from PreSetFieldValue event of Account field on Opportunity BC. Check for the LockFlag = Y AND (ModificationNumber IS NULL OR ModificationNumber = 0)) and throw error. DVM should throw error when new records are created.
Again, best practices say don't use the ModNumbers. You could set a ProfileAttribute to signal NewRecord, then use that attribute in the DVM. But please remember to clear the value of ProfileAttribute in WriteRecord and UndoRecord.
Let us know how it went !