I am trying to use Oracle SOA BPEL to synch data of about 1000 employees between an HR service and our local db. I get IDs of all employees with a findEmp call and loop through it empCount times to getEmp(empID) from the same HR service and update/insert into our db in every loop. This times out after about 60 odd employees, though this process is an asynch process. How should I redesign the process flow?
The timeout is occuring because you don't have any dehydration points in your BPEL code. Oracle BPEL needs to dehydrate before the Java transaction times out.
If you are using the Oracle BPEL DB Adapter, you can actually submit many objects at once for processing to the database, simply put more than one in the element from the DB Adapter. This may help a lot, since you can fetch all your data at once, then write it all at once.
Additionally, you can extend the transaction timeout for Oracle BPEL- it's a configuration parameter in transaction-manager.xml (there's also some tweaks to the EJB timeouts you need to do on 10.1.3.3.x & 10.1.3.4.x). The Oracle BPEL docs tell you how to change this variable.
Related
The spring boot application that I am working on
pools 1000 messages from table X [ This table X is populated by another service s1]
From each message get the account number and query table Y to get additional information about account.
I am using spring integrating to pool messages from table X and reading additional information for account, I am planning to use Spring JDBC.
We are expecting about 10k messages very day.
Is above approach, to query table Y for each message, a good approach ?
No, that indeed not. If all of that data is in the same database, consider to write a proper SELECT to join those tables in a single query performed by that source polling channel adapter.
Another approach is to implement a stored procedure which will do that job for you and will return the whole needed data: https://docs.spring.io/spring-integration/reference/html/jdbc.html#stored-procedures.
Although if the memory for that number of records to handle at once is a limit in your environment or you don't care how fast all of them are processed, then indeed an integration flow with parallel processing of splitted polling result is OK. For that goal you can use a JdbcOutboundGateway as a service in your flow instead of playing with plain JdbcTemplate: https://docs.spring.io/spring-integration/reference/html/jdbc.html#jdbc-outbound-gateway
I understand jdbcTemplate.batchUpdate is used for sending several records to data base in one communication.
Lets say i have 1000 records to be updated, instead of 1000 communications from Application to database, the Application will send 1000 records in request.
Coming to JdbcBatchItemWriterBuilder its combination of Tasks in a job.
My question is, if there is 1000 records to be processed(INSERT statements) via JdbcBatchItemWriterBuilder, all INSERTS executed in one go? or one after one?
If one after one, connecting to database 1000 times using JdbcBatchItemWriterBuilder causes perf issues? hows that handled?
i would like to understand if Spring batch performs better than running 1000 INSERT staments using jdbcTemplate.update ?
The JdbcBatchItemWriter uses java.sql.PreparedStatement#addBatch and java.sql.Statement#executeBatch internally (See https://github.com/spring-projects/spring-batch/blob/c4010fbffa6b71cbcfe79d523023251ce73666a4/spring-batch-infrastructure/src/main/java/org/springframework/batch/item/database/JdbcBatchItemWriter.java#L189-L195), so there will be a single batch insert for all items of the chunk.
Moreover, this will be executed in a single transaction as described in the Chunk-oriented Processing section of the reference documentation.
I am working with a EJB(3.0)-Hibernate(3) project along with Oracle 11g DB.
First of all due to the security reason I am unable to share my code, I am really sorry for that.
Issue is :
In my Application from different locations, DB has been called for retrieving, persisting, merging records which deals with a number of tables in DB.
But, for a particular retrieve query(select query which is fetching only a single record by putting a primary key data in where clause) from my Application, it is taking too much time(almost 4 minutes) for getting the response from DB(response is proper with a single record).
I can track the time by debugging from calling point to DB inside Application and the retrieving response from DB to my Application.
So, I want to know why for a single record fetching, it is taking so much time where for other queries it's fetching within seconds or micro-seconds.
And also want to know how to track the time-stamp of [query request from Application just hitting the Database after connecting DB through Hibernate Layer] and also what is going on inside the DB for this flow.
Please give me some advice or suggestions from your entire work experience if you facing such kind of issue and also help me how to track the whole flow
Application <-> Hibernate Layer <-> Database
Thanks in advance!!!
I have an HSQLDB embedded database, which I use to store statistics of some measurements. The statistics are expected to be arriving about every second from the same thread, but be fetched from a few different threads (from the pool of threads) every several seconds.
I do not have much experience with jdbc, so my questions may sound trivial:
What is the price of creating/disposing a new connection every second? Recall, that the database is embedded, so there is no TCP/IP involved.
What is the price of creating/disposing prepared statements every second?
Please, note that some inserts are bulk inserts, where I think to use addBatch and executeBatch methods of a prepared statement.
You should try to reuse the connections and the prepared statements.
To do this, each thread will have a single connection, and each connection will reuse a set of prepared statements. The connection is committed after the unit of work is complete, but not closed. The connection is closed when your app is closed or completes its work.
You should use executeBatch for bulk inserts.
With HSQLDB, the price of creating / disposing a new connection / prepred statements every second is not high, but you should still avoid this if you can.
I am using Spring in my Web Application , with the underlying database as Sybase.
I have 3 complex stored procedures to be executed.
The procs , have create table and drop table commands to hold temporary result sets.
The tables are created in the user db space , rather that in the tempdb space. Hence, I am faced with the need to ensure that the entire service operation from the service bean , that would have DAO objects calling the stored procs, to be serialized. Does simply making the service bean method a Spring Transaction, ensure a solution to potential concurrency related problems in my case?
Another thing that I noticed is that, annotating my service method as #Transactional , made the sybase database throw an error : "Create table command cannot be executed within a transaction". Does this mean that Spring makes the entire database operation a transaction?
I am really not clear about this , and any explanation would be welcome.
Meaning if I have a stored proc named myproc . The sybase statement would be exec myproc. This,say, is executed by the DAOobject from the service method, annotated as #Transactional. Now does Spring make the database operation as "begin tran
exec myproc
end tran". My observation seems to suggest that. Please explain.
And also explain, if just annotation of #Transactional , will solve my concurrency issues. I , actually don't want 2 instances of my stored proc to be running on the database , at a time.
You've asked a number of questions at once, but I'll do the best I can to answer them.
Marking a service as #Transactional associates it with the current JTA (Java Transaction API) transaction (or creates one if required)
Depending on how your datasources are configured, JDBC connections will typically also be associated (enlisted) into the transaction
If a connection is associated with a JTA transaction then anything that is executed on it will take place within a database transaction.
In Sybase ASE, you cannot create (or drop) a table inside a transaction.
So, marking your service as #Transactional will prevent you from executing a proc that contains create table statements.
However, that won't solve the problem you're facing anyway.
Marking something #Transactional, simply means that it executes inside a JTA transaction. And that means that it either commits, or rolls-back, but it doesn't guard against concurrent access.
You have a few options
If you know that your application will only ever run on a single VM, then you can mark the code as serialized. This will make sure the VM only ever has 1 thread inside that code at a time.
You can implement concurrency controls inside the proc, (e.g. use lock table), but that will require a transaction, which will prevent you from creating a table inside the procedure.
Or you can redesign your application to not have to jump through all these hoops.
There are probably easier ways of achieving the outcome you're looking for - creating and dropping tables inside a proc, and then trying to prevent concurrent access to that proc is not a typical way of solving a problem.