We are trying a scenario of Rate Limiting the total no. of JSON records requested in a month to 10000 for an API.
We are storing the total count of records in a table against client_id and a Timestamp(which is primary key).
Per request we fetch record from table for that client with Timestamp with in that month.
From this record we get the current count, then increment it with no. of current records in request and update the DB.
Using the Spring Transaction, the pseudocode is as below
#Transactional(propagation=Propagation.REQUIRES_NEW, isolation=Isolation.REPEATABLE_READ)
public void updateLimitData(String clientId, currentRecordCount) {
//step 1
startOfMonthTimestamp = getStartOfMonth();
endOfMonthTimestamp = getEndOfMonth();
//step 2
//read from DB
latestLimitDetails = fetchFromDB(startOfMonthTimestamp, endOfMonthTimestamp, clientId);
latestLimitDetails.count + currentRecordCount;
//step 3
saveToDB(latestLimitDetails)
}
We want to make sure that in case of multiple threads accessing the "updateLimitData()" method, each thread get the updated data for a clientId for a month and it do not overwrite the count wrongly.
In the above scenario if multiple threads access the method "updateLimitData()" and reach the "step 3". First thread will update "count" in DB, then the second thread update "count" in DB which may not have latest count.
I understand from Isolation.REPEATABLE_READ that "Write Lock" is placed in the rows when update is called at "Step 3" only(by that time other thread will have stale data). How I can ensure that always threads get he latest count from table in multithread scenario.
One solution came to my mind is synchronizing this block but this will not work well in multi server scenario.
Please provide a solution.
A transaction would not help you unless you lock the table/row whilst doing this operation (don't do that as it will affect performance).
You can migrate this to the database, doing this increment within the database using a stored procedure or function call. This will ensure ACID and transactional safety as this is built into the database.
I recommend doing this using standard Spring Actuator to produce a count of API calls however, this will mean re-writing your service to use the actuator endpoint and not the database. You can link this to your Gateway/Firewall/Load-balancer to deny access to the API once their quote is reached. This means that your API endpoint is pure and this logic is removed from your API call. All new API's you developer will automatically get this functionality.
Related
The spring boot application that I am working on
pools 1000 messages from table X [ This table X is populated by another service s1]
From each message get the account number and query table Y to get additional information about account.
I am using spring integrating to pool messages from table X and reading additional information for account, I am planning to use Spring JDBC.
We are expecting about 10k messages very day.
Is above approach, to query table Y for each message, a good approach ?
No, that indeed not. If all of that data is in the same database, consider to write a proper SELECT to join those tables in a single query performed by that source polling channel adapter.
Another approach is to implement a stored procedure which will do that job for you and will return the whole needed data: https://docs.spring.io/spring-integration/reference/html/jdbc.html#stored-procedures.
Although if the memory for that number of records to handle at once is a limit in your environment or you don't care how fast all of them are processed, then indeed an integration flow with parallel processing of splitted polling result is OK. For that goal you can use a JdbcOutboundGateway as a service in your flow instead of playing with plain JdbcTemplate: https://docs.spring.io/spring-integration/reference/html/jdbc.html#jdbc-outbound-gateway
I'm using Java DynamoDBMapper into my SpringBoot microservice where I'm using DynamoDB.
I have a doubt about the transactions management:
For a REST API (POST), before allowing the creation of the entity I have to do some checks about the status of the user objects that are currently saved on the DB. This is not a check on the status of a specific object fields.
What I mean.. I have to do something like this:
I have to retrieve the count of the objects that are currently assigned to the user
In case this count is <= N I have to allow the creation of the new object.
Basically I would like to encapsulate these steps into a single 'atomic' operation in order to avoid creating objects for the user if he already reached the limit or block the operation if, at the same time the user has deleted one saved object.
I'm not able to understand if I can do this using the transactions.
Basically, I would like to understand if it's possible to do a sort of lock:
I mean, If I'm doing this operation I would like to block other operations for the same user: e.g. delete an object (using a dedicated API) when in the middle of step 1 and 2.
Should I use the transactions (and, in case, how?) or should I use a different approach like this: Building Distributed Locks with the DynamoDB Lock Client
I have the following scenario. In a database A I have a table with huge amount of records (several millions); these records increase day by day very rapidly (also 100.000 records at day).
I need to fetch these records, check if these records are valid and import them in my own database. At the first interaction I should take all the stored records. Then I can take only the new records saved. I have a timestamp column I can use for this filter but I can't figure how to create a JpaPagingItemReader or a JdbcPagingItemReader and pass the dynamic filter based on the date (e.g. select all records where timestamp is greater than job last execution date)
I'm using spring boot, spring data jpa and spring batch.I'm configuring the Job instance in chunks with dimension 1000. I can also use a paging query (is it useful if I use chunks?)
I have a micro service (let's call this MSA) with all the business logic needed to check if records are valid and insert the valid records.
I have another service on a separate server. This service contains all the batch operation (let's call this MSB).
I'm wondering what is the best approach to the batch. I was thinking to these solutions:
in MSB I duplicate all the entities, repositories and services I use in the MSA. Then in MSB I can make all needed queries
in MSA I create all the rest API needed. The ItemProcessor of MSB will call these rest API to perform checks on items to be processed and finally in the ItemWriter I'll call the rest API for saving data
The first solution would avoid the http calls but it forces me to duplicate all repositories and services between the 2 micro services. Sadly I can't use a common project where to place all the common objects.
The second solution, on the other hand, would avoid the code duplication but it would imply a lot of http calls (above all in the ItemProcessor to check if an item is valid or less).
Do you have any other suggestion? Is there a better approach?
Thank you
Angelo
I have two Frontends consuming JSON from two different Backends using the JSON Web Token. These backends act on the same database.
In the db for example I have the Driver, Customer and Trip tables. The customer or the driver can cancel a trip only if it has not been canceled beforehand by one of them. Some transactions are recorded during a cancellation.
How to prevent having a double execution in this case when simultaneously, the customer and the driver launch a request for trip cancellation?
Am usin' Spring Boot (RESTful) and Spring JPA.
Any help will be greatly appreciated.
Edit:
Assuming these backends are A & B, Customer is requesting cancellation from the backend A, and Driver from B.
Use optimistic locking. Your code would look as follows:
#Entity
public class Trip {
#Version
#NotNull
private Long version;
...
}
It works as follows. Each change modifies the version. Suppose two users (or two services) loaded the same version of the trip. Now they both try to cancel it, i.e. they both try to modify it. Besides changes they both send the version. The JPA checks if the version in the update statement is the same as in the database. So the first request wins and will be executed. During the execution the version will be incremented.
Now the 2nd request arrives and wants also to cancel the trip. The JPA will see that the version attribute in the update statement is older (less) than the version value in database. Thus the 2nd request will not be executed and an OptimisticLockException will be thrown.
You can catch this exception and inform the user that the data were change in the meanwhile and suggest user to reload the data. The user reloads the data and sees that the trip has already been cancelled.
I have a use case where i want to record data in rows and display to the user.
Multiple users can add these records and they have to be displayed in order of insertion AND - MOST IMPORTANTLY - with a sequence number starting from 1.
I have a Spring boot microservice architecture at the backend, which obviously means i cannot hold state in my boot application as i'm gonna have multiple running instances.
Another method was to fetch all existing records in the db,count them,increment the count by 1 and use that as my sequence. I need to do this every time i am doing an insert.
But the problem with the second approach is with parallel requests, which could result in same sequence number being given to 2 records.
Third approach is to configure the counter in a db , but since i am using cosmos DB, apparently that is also not an option.
Any suggestions as to how i can implement a static, shared counter ?