We are using JPA/Hibernate for project where we need to insert, say 10000 records in database with multiple joins/relations etc. The functionality is fine but the performance is really slow.
Just wondering if migrating to JDBC will help in some performance gain?
Thanks,
-csn
You can do batch inserts with JPA/Hibernate - see "Batch processing".
You would almost certainly get better performance by doing batch inserts in JDBC (addBatch(), etc), but the Hibernate method may be more desirable if your schema is complex.
If you use JDBC batches, make sure that you do all of your addBatch() calls in a transaction.
If you happen to be using MySQL, be sure to add rewriteBatchedStatements=true to your connection parameters.
Related
I have written a spring batch solution which currently uses the embedded H2 in-memory database.
The read and write operations uses SOLR Cloud API calls.
Ideally, we dont want to introduce an proper relational database as a job repo database, for the read-write batch operation.
I read that H2 in-memory databases are best used for Dev and Test in spring batch.
Does anyone have experience of using this H2 database in spring batch on a proper live environment dealing with millions of records in the batch processing, but batch job will ran only once a day at most?
If H2 is not stable for prod, I might have to ditch spring batch OMG, or anyother alternatives?
Open to any ideas or references.
Thanks in advance.
H2 is a light-weight Java database, as you mentioned yourself, that it is ideal for dev testing !
When considering production, you might be missing on lot of features which a RDBMS , NoSQL databases provide!
For e.g. Replication, memory and performance optimizations etc.
If frequent reads and writes are concerned and you don't want RDBMS, you may choose MongoDB or Couchbase to manipulate records , they are fast too !
So considering Millions of records, I don't think H2 would be a good choice for production databases
A similar article might throw some light & help you decide !
Are there any reasons why h2 database shouldn't be used in production?
I have a Spring boot application with Hibernate, connecting to a PostgreSQL database. After some time, some users experience a problem with slow requests or requests without any response.
I suspect some long-running SQL queries, but how can I check which queries run long? I would like to log down execution times of queries. I know show-sql parameter for Hibernate, but it doesn't display arguments for SQL statements, nor it doesn't log execution times. Is there any other way to achieve that?
You can log slow queries in hibernate by setting the property:
spring.jpa.properties.hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS=100
Also, if you want to log SQL statements with parameters, you can use:
logging.level.org.hibernate.SQL=DEBUG
logging.level.org.hibernate.type.descriptor.sql.BasicBinder=TRACE
Am working on a Spring boot application using spring data JPA. The DB is a legacy one. There is a staging table which contains insert/update queries as CLOB data.
I have written a pojo for the staging table. The pojo implements CRUDRepository interface. I fetched the records from list() method and iterate the CLOB data. The queries present in the CLOB needs to be executed in other DB. The queries are insert/update queries related to 20+ tables.
Without creating pojos for that 20+ tables, how can I execute those SQL's.
The SQL's need no modifications, just need to execute the same that I fetch from the staging table. Is
EntityManager.createNativeQuery("insert/update")
a possible solution, or is there a better approach to handle it.
You could do that with JPA native queries for sure. Or even with plain JDBC.
But I recommend having a look at jOOQ. jOOQ generates POJOs for accessing and modifying the data and has a DSL that leads to compile time checked data access:
https://www.jooq.org/
jOOQ is free for OpenSource databases like MySQL, PostgreSQL etc. and affordable for commercial ones like Oracle.
What are the tools and approaches you suggest to use for monitoring performance? My app is running on Tomcat, Spring 3, JPA, Hibernate and solr. I've noticed some lag/slow activity on a certain page.
The app does not have any code of cache setup. Or even connection pooling.
Pardon my beginner-style questions, I'm only entering the "performance monitoring" world just now...
I know couple of things to look at is- dabatase calls, connection pooling, indexed tables, caching etc.
You absolutely need connection pooling. It's a no-brainer, there are libraries like Bone CP, Commons DBCP or c3p0 that do this transparently.
You must implemented automated load/stress test. JMeter is pretty easy to use, other tools like that are Gatling and Grinder.
Enable SQL logging and statistics. Most likely too many or too complex queries are slowing down your page.
Use a profiler. Either commercial (JProfiler, YourKit) or the one included in JVisualVM.
You can use this application to monitor your application:
http://newrelic.com/
He has a free plan!
I'm writing a Clojure program using clojure.java.jdbc. I'm using DBCP to pool connections to HSQL 2.2.8. I have a (transaction) block in which I test if a schema exists, and if not, creates it and a bunch of tables. One of the statements after the schema create (I believe a MERGE statement) throws a BatchUpdateException.
The issue is that the schema create is not rolled back on the BatchUpdateException, even though they're part of the same (transaction) block.
Are there known issues with Clojure JDBC interacting with DBCP or HSQL?
Never mind.
Transactions don't apply to schema changes, apparently. WTF?