We want to upgrade our current batch processing system to Spring Batch processing to use its parallel processing feature. But the underlaying database infrastructre is old like we are using sybase that support only 256 connections. The application server we are using is Jboss. Does adding parallel processing would need upgradation to underlying infrastructre like sybase (Sybase ASE15) or jboss(jboss-eap-5.1)? Please Suggest.
the database is almost always the bottleneck, not the Application Server, so i would update (or completely change) the database
you can change the spring batch database to the other that you like!
And then config a business datasource with your sybase
Related
I have written a spring batch solution which currently uses the embedded H2 in-memory database.
The read and write operations uses SOLR Cloud API calls.
Ideally, we dont want to introduce an proper relational database as a job repo database, for the read-write batch operation.
I read that H2 in-memory databases are best used for Dev and Test in spring batch.
Does anyone have experience of using this H2 database in spring batch on a proper live environment dealing with millions of records in the batch processing, but batch job will ran only once a day at most?
If H2 is not stable for prod, I might have to ditch spring batch OMG, or anyother alternatives?
Open to any ideas or references.
Thanks in advance.
H2 is a light-weight Java database, as you mentioned yourself, that it is ideal for dev testing !
When considering production, you might be missing on lot of features which a RDBMS , NoSQL databases provide!
For e.g. Replication, memory and performance optimizations etc.
If frequent reads and writes are concerned and you don't want RDBMS, you may choose MongoDB or Couchbase to manipulate records , they are fast too !
So considering Millions of records, I don't think H2 would be a good choice for production databases
A similar article might throw some light & help you decide !
Are there any reasons why h2 database shouldn't be used in production?
i'm the developper of an open-source application designed for self-hosting. As such, it is used by many people on various systems, OSes, and hardware. It's built using Kotlin and Spring Boot, and uses H2 as the database. The problem i have is that many users are facing database corruption from times to times (it happened to me also a few times).
The corruptions are always of the form:
org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.IllegalStateException: Chunk 20221 not found [1.4.200/9]" [50000-200]
I can't pinpoint any particular reason for those corruptions. The application doesn't use any dangerous H2 options. I couldn't find anything on the web or Stackoverflow that relates to my (imho) mundane use of H2.
I'm using:
Spring Boot 2.2.6.RELEASE
jOOQ 3.13.1 (but i had the same issues before with Hibernate)
H2 1.4.200
The connection string is pretty straightforward:
jdbc:h2:~/.komga/database.h2
The application is doing heavy writes once in a while (during file system scanning), but afterwards it's doing mostly reads.
Would you have suggestions on how to better configure H2 to avoid those issues ?
If your application uses Thread.interrupt() for threads doing calls into embedded database, this is the reason of corruption, don't do this or use the async: filesystem (jdbc:h2:async:…).
If classloader with H2 can be forcibly unloaded (on some application servers, for example) or application is going to be terminated in some abnormal way, you need to close all connections before it or execute the SHUTDOWN command and wait for its completion.
If you try to open the database file with some older version of H2, it can corrupt the file created by a more recent version.
You may run into some bug in H2, so if you can build a standalone test case (Java / JDBC / SQL only, no third-party libraries), you need to fill a new issue on GitHub. You can also try to build H2 from its current sources, there were some changes in the storage backend, but current H2 is very different from 1.4.200 in other aspects and third-partly libraries that you use may be not yet ready to work with it.
You can use the legacy PageStore backend instead of default one by appending ;MV_STORE=FALSE to the JDBC URL when you create your database. This backend uses table-level locks (you need to lock tables in the same order in all your transactions) and doesn't execute commands from different sessions in parallel, but it is more reliable.
You can use a separate H2 Server process; such configuration is usually more reliable than embedded databases, but it works slower.
In any case, with any DBMS, you should create backup copies on regular basis. H2 has BACKUP command for this purpose.
Is Spring Data JDBC v1.1.5 recommended for Oracle Database and Enterprise Applications? Lot of samples around the net based on Open Source RDBMS (H2 or PostgreSQL). We are using Spring Data JDBC in a Spring Boot Microservice Application, facing following problems.
Force to write custom converters for oracle.sql.TIMESTAMP, oracle.sql.TIMESTAMPTZ and oracle.sql.DATE and oracle.sql.ROWID etc..
Can't type cast oracle.sql.ROWID to java.lang.Number
Identity must not be null after save.
Spring Data JDBC is absolutely recommended for Enterprise Applications.
Not so much for use with Oracle.
Since the necessary resources (database & JDBC driver) weren't available in a form that could be easily used in integration tests on public platforms, Oracle isn't included in regular builds.
Therefore it is likely that one encounters issues when working with Oracle.
Some are already known, for others issues in Jira or even PRs are highly appreciated.
I am looking for a migration path for a Java-based project which uses Apache Cassandra 2.2 to Oracle Coherence 12 – and Oracle 12 backend.
The existing application uses CQL to interact with a 3 node Cassandra cluster.
Elswhere we specifically do not use any ORM (e.g. Hibernate/JPA) but use JDBC to interact with the database directly.
Yes, Cassandra is free while the Oracle solution is quite expensive but this is outside the scope of this question.
Any technical suggestions are welcomed.
You have a couple of options depending on your use case.
If you are using the SQL to interact with Cassandra for standard request/response interactions and need to migrate it to use Oracle DB which would require the least code changes and still use a standard approach would be to use an Object Relational Mapping (ORM) tool like Hibernate/JPA and use Coherence as the L2 cache (personally I like MyBatis since you have complete control over the SQL code. You may be able to use this Coherence integration with MyBatis ).
If you have other applications/ops users updating the database directly and need those changes to be available to your application then you will need to implement a CacheStore (use your favorite ORM here if you like) to save updates to the database and use Oracle Golden Gate Hotcache feature to push updates made to the database outside your application to Coherence. Your application will need to be changed to interact with Coherence directly using either their Map interface or using the Coherence Query Language (CQL) which is "SQL like". This approach will have an additional advantage of being able to support any asynchronous use cases you may have as Coherence API supports listening to cache changes (using MapListeners) similar to Cassandra's executeAsync.
I hope this helps.
I need to schedule regular database backup for my web application using Spring.
Does Spring Data offer any particular support for backupping? I plan to use TaskScheduler and TaskExecutor.
No, there is no specific support for that. Spring Data is intended for transactional use, not batch operations. Of course there is findAll() method that you can iterate over and store results somewhere.
Spring Batch is probably a bit better choice as it focuses on long-running, heavy batch processes. But IMHO your application is not a good place for running backup. Why not use database or OS support? It'll be faster and more reliable.
If you really need to backup your database from application level, consider your database manual, maybe there is some simple command to dump the contents of the database to a file. For example in h2 I am using SCRIPT SQL command from JdbcTemplate to dump the database to arbitrary file. But I use this technique to reset database after each integration test. I use JdbcTemplate to minimize overhead. That's why Spring Data is not the best tool for the job.
In MySQL there is a mysqldump process, so it's a bit more cumbersome to run from Java.
Please follow this link and get an idea regarding your matter
Spring support it
add the following dependency to pom.xml if ur build a maven project
<dependency>
<groupId>com.smattme</groupId>
<artifactId>mysql-backup4j</artifactId>
<version>1.0.0</version>
</dependency>
if gradle spring project
add the dependency to the build.gradle
compile group: 'com.smattme', name: 'mysql-backup4j', version: '1.0.0'
this article may help to understand
https://dzone.com/articles/how-to-backup-mysql-database-programmatically-usin