I have a global number that gets incremented frequently but gets reset yearly. Right now I'm storing this global number as a single row:
class GlobalCounter {
Integer counter = 0
static constraints = {
}
}
When I increment it, I use a lock in a transaction:
def globalCounter = GlobalCounter.lock(1)
globalCounter.counter = globalCounter.counter + 1
globalCounter.save()
globalCounter.discard()
The problem is that when I update it frequently, I get an exception instead of the code waiting:
Transaction (Process ID 61) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
I know that when you add a new row, the database gives the option of using a primary key with auto increment, so I think this should be possible without deadlocks. How can I do it?
Related
i have an issue :
i have a product entity which have 2 columns id and quantity
_so i have 2 api
one for update this will update product entity quantity (quantity = quantity - 1)
one for update this will update product entity quantity (quantity = quantity + 1)
the issue is then I call 2 api in the same time, this result not my expect here is my diagram
enter image description here
can anyone help my thank you
Well for your particular scenario there is a concept called locking. And there is two type of locking
Optimistic
Pessimistic
The idea is when one transaction is updating a row of a db table, you should not allow another transaction to update that row util the previous one is committed.
In application there are several ways to achieve this type of locking. We can describe this concurrent updating process as a Collision. In a system where a collision is not going to happen very frequently you can use the optimisting locking approach.
In optimistic approach you keep a version number in your row. When you perform an update you increase the version by 1. Let's analyse your scenario now and call your two service I (increase) and D (decrease). You have a product row P in your database table where quantity = 3, version = 0. When I and D is called for both of them when they fetch P from database the state of P is as below
quantity = 3, version = 0
Now D executes first and decrease and save P
Your update query should be like below
UPDATE Product p set p.quantity = :newQuantity
, p.version = p.version + 1 where p.version = :oldVersion and p.id = :id
For case of D value of newQuantity = 2 (oldQty - 1) and value of oldVersion = 0 (we fetched it at the beginning)
Now the current state of P is like below
quantity = 2, version = 1
Now when I tries to execute you should generate the same update query but for this case value of newQuantity = 4 (oldQty +1) and value of oldVersion = 0 (we fetched it at the beginning).
If you put these value to the update query your row won't be updated as it the version checking part will be false. From this you can then throw any locking exception to notify your client that the request could not be completed and can try again. This is basically the core concept of optimistic locking and there is much more efficient ways to handle it with frameworks like Hibernate
Here you can notice that we have not denied any of the read requests while updating the row but in the approach of Pessimistic locking you deny any read request when another transaction on going. So basically when D is on process of decreasing I would not be able to read the value and from there you can return to your client saying that the request was not completed. But this approach takes a toll on read heavy tables in exchange of tight data integrity.
Set Up
Kafka 2.5
Apache KStreams 2.4
Deployment to Openshift(Containerized)
Objective
Group a set of messages from a topic using a set of value attributes & assign a unique group identifier
-- This can be achieved by using selectKey and groupByKey
originalStreamFromTopic
.selectKey((k,v)-> String.join("|",v.attribute1,v.attribute2))
.groupByKey()
groupedStream.mapValues((k,v)->
{
v.setGroupKey(k);
return v;
});
For each message within a specific group , create a new message with an itemCount number as one of the attributes
e.g. A group with key "keypart1|keyPart2" can have 10 messages and each of the message should have an incremental id from 1 through 10.
aggregate?
count and some additional StateStore based implementation.
One of the options (that i listed above), can make use of a couple of state stores
state store 1-> Mapping of each groupId and individual Item (KTable)
state store 2 -> Count per groupId (KTable)
A join of these 2 tables to stamp a sequence on the message as they get published to the final topic.
Other statistics:
Average number of messages per group would be in some 1000s except for an outlier case where it can go upto 500k.
In general the candidates for a group should be made available on the source within a span of 15 mins max.
Following points are of concern from the optimum solution perspective .
I am still not clear how i would be able to stamp a sequence number on the messages unless some kind of state store is used for keeping track of messages published within a group.
Use of KTable and state stores (either explicit usage or implicitly by the use of KTable) , would add to the state store size considerably.
Given the problem involves some kind of tasteful processing , the state store cant be avoided but any possible optimizations might be useful.
Any thoughts or references to similar patterns would be helpful.
You can use one state store with which you maintain the ID for each composite key. When you get a message you select a new composite key and then you lookup the next ID for the composite key in the state store. You stamp the message with the new ID that you just looked up. Finally, you increase the ID and write it back to the state store.
Code-wise, it would be something like:
// create state store
StoreBuilder<KeyValueStore<String,String>> keyValueStoreBuilder = Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore("idMaintainer"),
Serdes.String(),
Serdes.Long()
);
// add store
builder.addStateStore(keyValueStoreBuilder);
originalStreamFromTopic
.selectKey((k,v)-> String.join("|",v.attribute1,v.attribute2))
.repartition()
.transformValues(() -> new ValueTransformer() {
private StateStore state;
void init(ProcessorContext context) {
state = context.getStateStore("idMaintainer");
}
NewValueType transform(V value) {
// your logic to:
// - get the ID for the new composite key,
// - stamp the record
// - increase the ID
// - write the ID back to the state store
// - return the stamped record
}
void close() {
}
}, "idMaintainer")
.to("output-topic");
You do not need to worry about concurrent access to the state store because in Kafka Streams same keys are processed by one single task and tasks do not share state stores. That means, your new composite keys with the same value will be processed by one single task that exclusively maintains the IDs for the composite keys in its state store.
Need some opinion/help around one use case of KStream/KTable usage.
Scenario:
I have 2 topics with common key--requestId.
input_time(requestId,StartTime)
completion_time(requestId,EndTime)
The data in input_time is populated at time t1 and the data in completion_time is populated at t+n.(n being the time taken for a process to complete).
Objective
To compare the time taken for a request by joining data from the topics and raised alert in case of breach of a threshold time.
It may happen that the process may fail and the data may not arrive on the completion_time topic at all for the request.
In that case we intend to use a check that if the currentTime is well past a specific(lets say 5s) threshold since the start time.
input_time(req1,100) completion_time(req1,104) --> no alert to be raised as 104-100 < 5(configured value)
input_time(req2,100) completion_time(req2,108) --> alert to be raised with req2,108 as 108-100 >5
input_time(req3,100) completion_time no record--> if current Time is beyond 105 raise an alert with req3,currentSysTime as currentSysTime - 100 > 5
Options Tried.
1) Tried both KTable-KTable and KStream-Kstream outer joins but the third case always fails.
final KTable<String,Long> startTimeTable = builder.table("input_time",Consumed.with(Serdes.String(),Serdes.Long()));
final KTable<String,Long> completionTimeTable = builder.table("completion_time",Consumed.with(Serdes.String(),Serdes.Long()));
KTable<String,Long> thresholdBreached =startTimeTable .outerJoin(completionTimeTable,
new MyValueJoiner());
thresholdBreached.toStream().filter((k,v)->v!=null)
.to("finalTopic",Produced.with(Serdes.String(),Serdes.Long()));
Joiner
public Long apply(Long startTime,Long endTime){
// if input record itself is not available then we cant use any alerting.
if (null==startTime){
log.info("AlertValueJoiner check: the start time itself is null so returning null");
return null;
}
// current processing time is the time used.
long currentTime= System.currentTimeMillis();
log.info("Checking startTime {} end time {} sysTime {}",startTime,endTime,currentTime);
if(null==endTime && currentTime-startTime>5000){
log.info("Alert:No corresponding record from file completion yet currentTime {} startTime {}"
,currentTime,startTime);
return currentTime-startTime;
}else if(null !=endTime && endTime-startTime>5000){
log.info("Alert: threshold breach for file completion startTime {} endTime {}"
,startTime,endTime);
return endTime-startTime;
}
return null;
}
2) Tried the custom logic approach recommended as per the thread
How to manage Kafka KStream to Kstream windowed join?
-- This approach stopped working for scenarios 2 and 3.
Is there any case of handling all three scenarios using DSL or Processors?
Not sure of we can use some kind of punctuator to listen to when the window changes and check for the stream records in current window and if there is no matching records found,produce a result with systime.?
Due to the nature of the logic involve it surely had to be done with combination of DSL and processor API.
Used a custom transformer and state store to compare with configured
values.(case 1 &2)
Added a punctuator based on wall clock for
handling the 3rd case
I have a scenario like i want to read a spreadsheet which consists of around 2000 records and enter it into database.
Currently we are using Executor framework. We have limitation that no of tasks should be only 5. Each task reads 20 rows from the excel. We provide the start index and end index of the rows to be read from the excel to each task.
Say, currently,
Task 1 handles 1-20
Task-2 handles 21-40
Task-3 handles 41-60
Task-4 handles 61-80
Task-5 handles 81-100
If Task-1 finishes its execution, it takes the next 20 rows thats 101-120. Suppose if Task-2 finishes before Task-1, it will start reading from 121-140 and not 101-120.
Can i handle this scenario more effectively in a Fork-Join framework only with the restriction of 5 tasks and each task 20 rows?
Need some insight into the performance issues.
No need to switch the thread pool. To make the load more balanced you can just maintain atomic variable which points to the first not taken row:
AtomicInteger currentRow = new AtomicInteger(); // shared between tasks
final int maxRow = 2000;
final int batchSize = 20;
// Inside every task:
while(true) {
int row = currentRow.getAndAdd(batchSize);
if(row >= maxRow) return;
int from = row+1;
int to = Math.min(row+batchSize, maxRow);
// process rows from..to; it's guaranteed that other threads
// do not process the same rows.
}
The body of every task is absolutely the same. Also this implementation does not depend on number of tasks created. If you later decide to have 3 tasks or 7 tasks, just adjust the thread pool size and submit more (or less) tasks.
I have a quartz job that updates some records in a database. Since the number of records to update is huge, it will do it in batches of 100 for a period of 2 hours every night between 1 and 3 AM.
So if it is processing a job when the 3 AM deadline reaches, I would like it to finish processing the current batch of 100 records and then exit cleanly. How can this be achieved? Can it be done using a cron expression alone or is there some mechanism to achieve this graceful shutdown?
I am using spring's SchedulerFactoryBean and MethodInvokingJobDetailFactoryBean classes to configure the job in my application context xml.
Some pseudo code of my job implementation method.
public void updateRecords()
{
while(true) // i need to replace true with some other logic to exit from loop at scheduled time.
{
// 1. select 100 records
// 2. update 100 records
// 3. commit
}
}
When you run the updateRecords, store the System.currentTimeMillis and see if the time has exceeded that time + 2*60*60*1000 ms in the while loop.
public void updateRecords()
{
long jobStartedAt = System.currentTimeMillis();
long twoHoursLater = jobStartedAt + 2 * 60 * 60 * 1000; //2 hours, 60 minutes/hour, 60 seconds/minute, 1000 ms/second
while(System.currentTimeMillis() < twoHoursLater)
{
// 1. select 100 records
// 2. update 100 records
// 3. commit
}
}
Also:
Depending on your database properties, 100 jobs in a batch could be a bit to small to efficient. Don't be afraid to try larger batches.
And if the jobs takes very long it could be because of heavy indexing. If you have the possibility, drop the indexes before doing large scale (dumb) batching and rebuild them afterwards, if they are not explicitly needed for the batch processing. Every update of indexes can render many more diskwrites than the single change of a record.