I have a Java Spring project which does a lot of database reading. The database I have available is a shared postgresql database on a remote machine and it's quite slow to get data from it, especially when I'm doing a lot of back-and-forth.
When I'm doing local testing and development, I use the embedded H2 in-memory database. Things easily go 100 times faster.
This made me wonder. Is there a way to use the embedded H2 in-memory database so that:
Data manipulation (INSERT, UPDATE, DELETE) is ("eventually") replicated to the PostgreSQL database
Upon start/restart of the Spring project, the H2 database is automatically filled with the data of the PostgreSQL server
This would allow me to use the fast H2 to provide a seamless user experience while at the same time also storing the data in a longer-term data storage. In a way, I'd be using the H2 as a fully cached version of the PostgreSQL database.
Is there a known / standardized way to approach this? Google for H2 and PostgreSQL and replication gives me results on transitioning from one to the other, but I'm not finding much as to using one as a sort of cache for the other.
I remain on the lookout for a Spring / JPA / Hibernate focused answer, but if none comes: I may have found an alternative domain to investigate. Dedicated database replication software might be able to manage this. Specifically, I've discovered SymmetricDS, which seems (I've only given the documentation a cursory glance) like it might be able to be embedded into my Spring application, do an initial load of my embedded in-memory H2 database on startup and then trickle feed data changes to the remote database.
(I'm not in any way affiliated with SymmetricDS, it just looks like it might be a solution.)
Related
I am using IntelliJ IDEA and JPA Buddy to generate entities from my database. However, every time I open an Entity from DB wizard, it takes a very very long time. Is it okay? Or something wrong with my database/IntelliJ IDEA or JPA Buddy?
My setup is:
Database: Oracle (~2000 tables)
IntelliJ IDEA: 2022.3.1
JPA Buddy: 2022.5.3
I have tried to recreate db connection and invalidate caches in the IntelliJ IDEA, same result.
It may happen due to a slow internet connection or many tables in the database (probably it is your case, 2000 is great number). Also, some database drivers are not showing their best side in this matter. The one way you can speed up your development process – is a "schema cache" option from JPA Buddy (1). Using it, you can generate the data model snapshot once and then use its local copy.
Just don't forget to refresh it when the database gets changed (2).
i am working on designing constructure in Greenplum database.
we have many clinets which need to store data for them.
there are two ways to design database constructure. we build one database and different schemas in this database for each clients
or build different databases for each clients, which way is better?
waht is nmore ,we need to migrate databases or schemas from dev environment to environment
Thanks
William
William,
Either way will work. If you are keeping multi-tenants in Greenplum and there is no data sharing, you might be better off keeping them in separate databases - easier for security and for backups. If there is a requirement that they share some common data, then using multiple schemas in one database is the better option.
I am not sure what version of Greenplum you are on, but you should be able to backup a schema from dev and restore it with gprestore using the --redirect option to put it in the database you want it to be in.
Jim McCann
Pivotal
I imagine that it nowadays is quite common to use one RDBMS during development and another RDBMS in production. I'd like to use H2 in development and MariaDB in production for a Spring Boot and JOOQ based application.
Is there some clever way to make the same generated JOOQ code work in both development and production environments, or do I need to generate two sets of code depending on the target environment? If the latter is true, how to do that in a sane way e.g. using the nu.studer.jooq gradle plugin?
Exceptions like this are thrown whenever I try to use the sources generated from a H2 database against a MariaDB server:
org.mariadb.jdbc.internal.util.dao.QueryException: SELECT command denied to user 'foo'#'localhost' for table 'FOO'
Query is: select `PUBLIC`.`FOO`.`ID`, `PUBLIC`.`FOO`.`NAME`, `PUBLIC`.`FOO`.`INFO` from `PUBLIC`.`FOO`
I use the same flyway initialization/migration scripts for both H2 and MariaDB.
You don't need to generate two sets of classes for each production environment. jOOQ's generated classes are pretty vendor agnostic, unless you use vendor specific features, e.g. like MariaDB's enum type or stored procedures, etc.
The error you're getting is probably related to one of these things:
You might not have a PUBLIC schema in your MariaDB database. You can either make sure the schema names match between H2 and MariaDB, or you can turn off schema name generation in jOOQ by using either Settings.renderSchema on your configuration, or by using a schema mapping.
Different databases have different default case sensitivity settings. In H2, by default, all tables are upper case, but this might not be the case in your MariaDB installation. You can either make sure the casing is the same in both databases, or you turn off the generation of the backticks / quotes. This can be done with Settings.renderNameStyle, setting it to AS_IS
It might be unrelated to jOOQ and you simply don't have the appropriate privilege to query the table.
Unrelated, a short note on using different vendors for development and production
You said:
I imagine that it nowadays is quite common to use one RDBMS during development and another RDBMS in production. I'd like to use H2 in development and MariaDB in production for a Spring Boot and JOOQ based application.
I really really advise against this practice. You can very easily set up your production database in docker and work directly against it. While H2 can emulate a couple of MariaDB features, it is nowhere near the same. By artificially restricting yourself to the least common denominator between H2 and MariaDB, you're missing out on a lot of cool MariaDB features, including CTE, window functions, stored procedures, etc. etc. And you will constantly fight the subtle differences between the vendors on various levels of your stack.
You should only do this when:
You actually need to support several databases in production
You really really really benefit from the slightly increased performance, e.g. for integration testing (but I doubt it, with docker).
We have multiple oracle schema which we want to import in to somekind of inmemory db so that when we run our integration test we can use that db and run our tests faster.
Is there anyway we this can be achieved using something like HSQL db. We are using spring framework and it does support inmemory db.
Any link to some resource would be highly appreciated.
Try force full database caching mode, if you're using 12.1.0.2. It's not exactly the same as a full in-memory database, but it should be closer.
alter database force full database caching;
In-memory database performance is over-rated anyway. Oracle's "old-fashioned" asynchronous IO and caching often work just fine. For example, in this question, accessing a temporary table (which is stored on disk) runs faster than an equivalent solution using in-memory data structures. And I've seen a small Oracle database handle petabytes of IO with the "boring" old buffer cache.
Or when you say "run our tests faster", are you referring to a more agile database; one that can be controlled by an individual, instead of the typical monolithic Oracle database installed on a server? I see that issue a lot, and there's no technical reason why Oracle can't be installed on your desktop. But that can be a tough cultural battle.
Yes, you can use HSQLDB for the purpose of unit testing - see this post for more information on how to integrate with Spring.
Also, see this list as a good starting point for different usages of HSQLDB.
My delayed job has something to do with exporting slightly edited version of most of the tables in the app's database, and while doing so, it is critical that none of the current data is being edited.
Is it possible to lock the entire database while running this delayed job?
More Information:
The database to be exported is in PostgreSQL, Heroku's postgresql database, to be more specific.
The flow is something like (all below should be done automatically by the code):
site would be put in maintenance mode,
freeze then export the database, then
when exporting is complete, re-activate the site back
Given there is not a lot of information with your question, I am going to answer you as best I can.
1) What is the database type and model? Is it a standalone DB like MS Access or Informix SE?
2) If not a standalone engine, does this database support replication. I used to work a lot with MS SQL Server, and replication had implications while the database was live and being edited. That is the implications were whether edited data was replicated. In this case, consult the docs. Is it an option to use replication to preserve the current database?
3) What kind of task is this? It sounds like maintenance. Our Informix SE databases lock when being imported or exported. On the production server, it is my job to make sure no local server applications are trying to access the locked DB, and that our external payments web site cannot interfere while the db is locked.
4) If this is a production site that is not in maintenance mode, then I suggest you probably do not want to lock an entire database.
I am sorry for not answering your question directly, but more information is needed like are you asking if this can be done from the Ruby DB interface on some model of db.