we have been using spring integration aggregator (version 5.1.4.RELEASE) in our project with the following configuration
<int:aggregator id="adobeJdbcAggregator" input-channel="aggregatorInputChannel"
output-channel="aggregatorOutputChannel"
correlation-strategy-expression="headers['eventType']"
message-store="ccsJdbcMessageStore"
send-partial-result-on-expiry="true"
group-timeout-expression="#{aggregateGroupTimeoutValue}"
expire-groups-upon-completion="true"
expire-groups-upon-timeout="true"
release-strategy-expression="#{aggregateReleaseStrategyExpressionValue}"
ref="ccsAggregatorBean"
method="processMessage"
auto-startup="true"/>
We occasionally notice some messages pending in INT_MESSAGE table as there are not supposed to be any pending after aggregation, could u suggest what could be the issue with it? Please note that I don't see any pending messages in either INT_MSG_GROUP or INT_GROUP_TO_MSG tables, we need to fix this issue where some messages are not aggregated.
Note:
We use group-timeout expression which expires the group based on a timestamp
When message group is expired, the JdbcMessageStore performs these update queries:
public void removeMessageGroup(Object groupId) {
String groupKey = getKey(groupId);
this.jdbcTemplate.update(getQuery(Query.DELETE_MESSAGES_FROM_GROUP), groupKey, this.region, this.region);
if (logger.isDebugEnabled()) {
logger.debug("Removing relationships for the group with group key=" + groupKey);
}
this.jdbcTemplate.update(getQuery(Query.REMOVE_GROUP_TO_MESSAGE_JOIN), groupKey, this.region);
if (logger.isDebugEnabled()) {
logger.debug("Deleting messages with group key=" + groupKey);
}
this.jdbcTemplate.update(getQuery(Query.DELETE_MESSAGE_GROUP), groupKey, this.region);
}
So, the first one belongs to the mentioned INT_MESSAGE:
DELETE from INT_MESSAGE where MESSAGE_ID in
(SELECT MESSAGE_ID from INT_GROUP_TO_MESSAGE where GROUP_KEY = ? and REGION = ?) and REGION = ?
Then as you notices an INT_GROUP_TO_MESSAGE is cleaned from the group and so on for the INT_MESSAGE_GROUP.
Therefore it is not clear how those messages can be stale, if they belong to any group you are cleaning from the store.
On the other hand the INT_MESSAGE table is used for other scenarios where plain message manipulation is necessary (no group correlation). See more info in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#message-store. And a MessageStore interface. Therefore those messages could be stored from some other endpoints and they are not related to the mentioned aggregator anyway. For example Claim Check pattern uses a plain MessageStore abstraction.
Related
Like I have a page where data are showing like first name, middle name, last name, address, city, state, country, age, salary.
There are a filter implemented that have 4 fields like city, age, salary, state. now I have to made a controller in Spring boot that takes all 4 fields as input params and find data from database using Spring Data JPA.
But my problem is that I want to filter Data sometime by salary only, sometime by city, state, sometime with all 4 params.
So what will be controller code and JPA Repository query to do this filter process.
Please Help me
Thanks in Advance
test if your parameters are null :
"where (:name is null or name = :name) and (:city is null or city = :city)"
I would suggest to use JPA Specification instead,
to know more on specification code here is my thread on stackoverflow
If you want to stick with JPA then here's the query which can help you
where (:field1 is null or table.field1 = :field1)
and (:field2 is null or table.field2 = :field2)
...
...
...
Explanation: The above code will check if the parameter is null, then return true otherwise compare with the column.
You can modify Query with below clause Where you can write query in Repository and in
where clause you can add conditions. So this will work as expected.
#Query(value = "SELECT user from User user WHERE ((:name IS NULL) OR (user.name = :name)) AND ((:city IS NULL) OR (user.city = :city)))
I have a simple kafka consumer that collects events and based on the data in them inserts or updates a record in the database - table has a unique ID constraint on ID column and also in the entity field.
Everything works fine when the table is pre-populated and inserts happen every now and then. However when i truncate the table and send a couple thousand events with limited number of ID (i was doing 50 unique ID within 3k events) then events are processed simultaneously and the save() method randomly fails with Unique constraint violation exception. I debugged it and the outcome is pretty simple.
event1={id = 1 ... //somedata} gets picked up, service method saveOrUpdateRecord() looks for the record by ID=1, finds none, inserts a new record.
event2={id = 1 ... //somedata} gets picked up almost at the same time, service method saveOrUpdateRecord() looks for the record by ID=1, finds none (previous one is mid-inserting), tries to insert and fails with constraint violation exception - should find this record and merge it with the input from the event based on my conditions.
How can i get the saveOrUpdateRecord() to run only when the previous one was fully executed to prevent such behaviour? I really dont want to slow kafka consumer down with poll size etc, i just want my service to execute one transaction at a time.
The service method:
public void saveOrUpdateRecord(Object input) {
Object output = repository.findById(input.getId));
if (output == null) {
repository.save(input);
} else {
mergeRecord(input, output);
repository.save(output);
}
}
Will #Transactional annotaion on method do the job?
Make your service thread safe.
Use this:
public synchronized void saveOrUpdateRecord(Object input) {
Object output = repository.findById(input.getId));
if (output == null) {
repository.save(input);
} else {
mergeRecord(input, output);
repository.save(output);
}
}
I have a spring boot kafka consumer which consume data from a topic and store it in a Database and acknowledge it once stored.
It is working fine but the problem is happening if the application failed to get the DB connection after consuming the record ,in this case we are not sending the acknowledgement but still the message never consumed until or unless we change the group id and restart the consumer
My consumer looks like below
#KafkaListener(id = "${group.id}", topics = {"${kafka.edi.topic}"})
public void onMessage(ConsumerRecord record, Acknowledgment acknowledgment) {
boolean shouldAcknowledge = false;
try {
String tNo = getTrackingNumber((String) record.key());
log.info("Check Duplicate By Comparing With DB records");
if (!ediRecordService.isDuplicate(tNo)) {---this checks the record in my DB
shouldAcknowledge = insertEDIRecord(record, tNo); --this return true
} else {
log.warn("Duplicate record found.");
shouldAcknowledge = true;
}
if (shouldAcknowledge) {
acknowledgment.acknowledge();
}```
So if you see the above snippet we did not sent acknowledgment.
That is not how kafka offset works here
The records in the partitions are each assigned a sequential id number called the offset that uniquely identifies each record within the partition.
From the above statement For example, from the first poll consumer get the message at offset 300 and if it failed to persist into database because of some issue and it will not submit the offset.
So in the next poll it will get the next record where offset is 301 and if it persist data into database successfully then it will commit the offset 301 (which means all records in that partitions are processed till that offset, in above example it is 301)
Solution for this : use retry mechanism until it successfully stores data into database with some limited retries or just save failed data into error topic and reprocess it later, or save the offset of failed records somewhere so later you can reprocess them.
In my hyperledger-composer app with angular front-end I want to send a query to the REST server.
query selectEmployeesByProject {
description: "Select all employees with access to the specified project"
statement:
SELECT org.comp.app.Employee
WHERE (projects CONTAINS [_$project])
ORDER BY [lastName, firstName]
}
Employee is defined as follows:
participant Employee {
o String lastName
o String firstName
--> Project[] projects optional
}
The http-request that is sent is the following:
this.httpClient.get<any[]>(requestURL, {withCredentials: true});
whereby the request url is the following:
http://localhost:4200/api/queries/selectEmployeesByProject?projects=resource:org.comp.app.Project#project1ID
In the console I get the following error message:
Failed to load resource
http://localhost:4200/api/queries/selectEmployeesByProject?projects=resource:org.comp.app.Project#project1ID
the server responded with a status of 400 (Bad Request)
Why is the query not working?
In general, httpRequests to the REST-server work in my app. Even other queries work. However, queries with the "CONTAINS" operator (such as the one above) do not work.
more likely to be
query selectClientsByProjects {
description: "Select all clients with access to a certain project"
statement:
SELECT org.myComp.myApp.Client
WHERE (projects CONTAINS [_$project])
ORDER BY [lastName ASC, firstName ASC]
}
Note: At the time of writing - CouchDB does not allow the ORDER BY clause to contain multiple items of different sort direction (obviously not the case above).
Also - not tested but your http call may need to be:
this.http.get('http://localhost:4200/api/queries/selectClientsByProjects?projects=resource:org.myComp.myApp.Project%231')
(%23 is the ASCII for '#')
You can try it out your definition first anyway (eg. without ORDER BY maybe) etc etc. Note: CONTAINS must match one entry ('array contains an element, with complete string match (above), in any element, in that array)
my RDBMS is PostgreSQL. I am using SpringBoot and Hibernate as JPA.
Let's consider a very simple one-column table:
Man
Age: integer
And I would like to implement a such method that add a man to the table. That method should satisfy the following condition:
At most one man in table can be 80 years old
.
#Transactional
void addMan(int age){
....
}
It looks like I need to take a exclusive lock for whole table, yes? How to do it?
There is a second solution.
Use SERIALIZABLE transactions that look like this:
START TRANSACTION ISOLATION LEVEL SERIALIZABLE;
SELECT count(*) FROM man WHERE age = 80;
Now, if the result is 0, continue:
INSERT INTO man VALUES (80);
COMMIT;
If two transactions try to do this concurrently, one will fail with a serialization error.
In that case, you just have to retry the transaction until it succeeds.
Serializable transactions perform worse than transactions with a lower isolation level, but they will still perform better than actually serializing the transactions.
I would add a unique constraint / index to the table and let the database do the work.
create unique index man_age_unique_idx on man (age);
If a record with that age does not already exist, no problem.
If it does exist, you should get back a PersistenceException where the cause is a Hibernate ConstraintViolationException. From that you can get the name of the constraint that was violated, e.g. man_age_unique_idx in the example above.
try {
entityManager.persist(man);
} catch (PersistenceException e) {
if (e.getCause() instanceof ConstraintViolationException) {
ConstraintViolationException cve = (ConstraintViolationException) e.getCause();
if (Objects.equals(cve.getConstraintName(), "man_age_unique_idx")) {
// handle as appropriate... e.g. throw some custom business exception
throw new DuplicateAgeException("Duplicate age: " + man.getAge(), e);
}
} else {
// handle other causes ...
}
}