I want to migrate a subset of customer data from one shared database environment to another shared database environment. I use hibernate and have quite a few ID and FK_ID columns which are auto generated from an oracle sequence.
I have a liquibase change log that I exported from jailer which has the customer specific data.
I want to be able to rewrite all of the sequence ID columns so that they don't clash with what's already in the target database.
I would like to avoid building something that my company has to manage, and would prefer to upstream this to liquibase.
Is anyone aware of anything within liquibase that might be a good place to start.
I would like to either do this on the liquidbase xml before passing it to 'update' command, or as part of the update command itself. Ideally as part of the update command itself.
I am aware that I would need to make liquibase aware of which columns are PK sequence columns and the related FK columns. The database structure does have this all well defined, so I should be able to read this into the update process.
Alternatively I had thought I could use the extraction model csv from jailer
Jailer - http://jailer.sourceforge.net/
I would suggest that for one-time data migrations like this, Liquibase is not the best tool. It is really better for schema management rather than data management. I think that an ETL tool such as Pentaho would be a better solution.
I actually managed to figure it out for myself with the command line 'update' command of liquibase by using a custom change exec listener.
1) I pushed a MR to liquibase to allow registration of a change exec listener
2) I implemented my own change exec listener that intercepts each insert statement and rewrites each FK and PK field to one that is not as yet allocated in the target database. I achieve this by using a oracle sequence. In order to avoid having to go back to the database each time for a new sequence, I implemented my own version of the hibernate sequence caching
https://github.com/liquibase/liquibase/pull/505
https://github.com/pellcorp/liquibase-extensions
This turned out to be quite a generic solution and in concert with some fixes upstreamed to jailer to improve the liquibase export support its a very viable and reusable solution.
Basic workflow is:
1) Export a subset of data from source db using jailer to liquibase xml
2) Run the liquibase update command, with the custom exec change listener against the target.
3) TODO Run the jailer export on the target db and compare with the original source data.
Related
my project has large oracle sql scripts. liquibase locks the schema (DATABASECHANGELOGLOCK table) when installing a single patch. How do I install multiple patches in parallel without a queue?
P.S. Oracle will independently make locks at its discretion.
Any DDL is make the new schema state that is based on previous state. If the previous state is not valid, you cant apply next DDL (it is impossible to add new constrain to the column that not exist). To check the previous state, you use precondition in your changesets.
So, in general it is impossible to parallelise the schema update, because the schema changes should be applied in order and the order can't be changed.
The lock on DATABASECHANGELOGLOCK is aimed to be sure that it is impossible to run two schema update process in one time, and it is reasonable restriction, so don't try to get around it.
If update process takes to much time, just be sure that you:
not use liquibase to change database state (add data to tables)
not use liquibase to update code objects (functions, procedures and etc.) in the database
not use liquibase for migrate large amount of data
So, I have a specific problem and I can't find the Spring best practice for it.
I have a Spring instance in front of a Postgres DB. I have a domain entity that I'm adding some fields to (using flyway).
Essentially, imagine I have the following object:
Book
last_text_update // 2018-11-25 07:00:00
last_writer // PUBLISHER
is_finished // true
contract_closed_timestamp // NULL
From this, we do weird calculations like
"If last_writer == PUBLISHER && is_finished == true, return IS_SELLING_IN_STORES"
As you can imagine, this is ugly, and I'm refactoring it to the following:
Book
processedStatus: PUBLISHED/EDITING/PROPOSAL
workStatus: AWAITING_EDITOR_FEEDBACK/AWAITING_CHANGES/FINISHED
etc.etc.
So, I've prepared the SQL migrations that have added the necessary status columns. The way I see it, my options are to either:
1) Figure out all of the column transitions in raw SQL and run that migration on the production server like any other migration. i.e. where timestamp = blah and other timestamp = blah, set status as follows
2) Run a one-off batch job that cycles through each object and updates it.
It seems like it would be easier to do in Kotlin code via batch job vs. doing it in SQL, but what is the consensus on this? Is there a recommended to do a one-off batch job (i.e. ssh into an instance and manually run it, start it with HTTP over the API, etc.), or do I just suck it up and use SQL?
Migration using SQL (Option 1) seems more reasonable, given that
you are already using Flyway and
It seems ugly to me to pollute the entity model to allow it to do the data migration work. e.g.
you will need the entity model to behave given an "incomplete" persistence form, and
you need to keep the obsolete fields, and etc.
Don't forget the performance difference if you are cycling through lots of entities too. Unless there are some special requirement that need you to do the modification using entity model (e.g. sending out events etc), it seems more reasonable to do the data migration using SQL.
To generate insert statements from my databases (oracle,db2) i've used liquibase generateChangeLog command with argument
--diffTypes="data"
This command generate correct xml with insert statements, however this is not aware of foreign constraints, so I cannot use this file to again to fill my databases. Similar problems has been described here : Is there a way to generate Liquibase data in the right order?. The proposed workaround unfortunatelly is not possibile for my databases, because there is no any command to switch of constraint checks.
My question is, if exists any other solution to this problem ? Why can I generate data insert statements changelog, but cannot use it because of foreign key constraints ?
Jens is right in the comment. Liquibase has no way of determining dependencies because the main use case is tracking ran changeSets. GenerateChangeLog is a useful capability but it is not intended to handle all cases and managing dependencies is a complex task that is definitely out of scope.
My normal recommendation is that the output of generateChangeLog should be considered a useful first step for working with the changeLog and if you have dependency issues just reorder the changeSets before executing them. If there many of them, you may want to write a script to reorder them based on your knowledge of your dependencies.
If you can export schema + data, a good solution is separate DDL, DML and this part of DDL that create the constraints. You have to reorder those in the following order:
DDL (without constraints)
DML (data)
DDL (the constraints removed from step 1)
It requires a little bit of manual editing but you'll have to do it once for your project.
And, as a side note, for you next project, start using liquibase from the start, this way you will never encounter this problem anymore.
The jailer export liquidbase feature is excellent for this. It generates a change log in topological order.
http://jailer.sourceforge.net/
I am using H2Database With ORMLite. we have 60 tables all created with ORMLite "create if not exists", Now we are going to provide a major release and requirement is to update old version database. But I need to know how to do this with ormLite as in new version some of Tables will be new and some is existing old tables with some modifications e.g we have an table of job in previous version db, in this release we added 2 more columns and change the datatype of one column. any suggestions. I have seen some other posts regarding OrmLite for Android SqlLite. How can this approach be used for other db. e.g Like this post
ORMLite update of the database
But I need to know how to do this with ormLite as in new version some of Tables will be new and some is existing old tables with some modifications e.g we have an table of job in previous version db, in this release we added 2 more columns and change the datatype of one column.
I'm not sure there is any easy answer here. ORMLite doesn't directly provide any magic capabilities to make the migration of data any easier. Here are some thoughts however:
You will need to use some sort of SQL logic to determine whether your application has the "old" or "new" schema installed. You could use raw SQL to look for the existance of particular tables or columns. Might be a good idea going forward to store a meta table with database version which Android gets for free.
You can create new and old versions of each of your entities (OldAccount versus Account) and map them both to the same table with the #DatabaseTable(tableName = "accounts"). Then you can read the old entities using the oldAccountDao.iterator(), convert them to new entities and (as long as you aren't mucking with the primary key) update them using the new accountDao.update(...).
You can certain come up with a series of SQL statements that will need to be performed in the proper order to change the schema. Then call the dao.exectuteRaw(...) with them in order.
Obviously the new entities will just be created.
You might want to consider dumping a backup file of all tables somewhere before the conversion process and telling the user about it in case there is some failure so your users could revert and run the old version of your application.
Hopefully something here is helpful.
We have many tables in our database with autoincrement primary key ids setup the way they are in MySQL since we are in the process of migrating to Oracle from MySQL.
Now in oracle I recently learned that implementing this requires creating a sequence and a trigger on the id field for each such table. We have like 30 -40 tables in our schema and we want to avoid using database triggers in our product, since management of database is out of scope for our software appliance.
What are my options in implementing the auto increment id feature in oracle... apart from manually specifying the id in the code and managing it in the code which would change a lot of existing insert statements.
... I wonder if there is a way to do this from grails code itself? (by the way the method of specifying id as increment in domain class mapping doesnt work - only works for mysql)
Some info about our application environement: grails-groovy, hibernate, oracle,mysql support
This answer will have Grails/Hibernate handle the sequence generation by itself. It'll create a sequence per table for the primary key generation and won't cache any numbers, so you won't lose any identifiers if and when the cache times out. Grails/Hibernate calls the sequence directly, so it doesn't make use of any triggers either.
If you are using Grails hibernate will handle this for you automatically.
You can specify which sequence to use by putting the following in your domain object:
static mapping = {
id generator:'sequence', params:[sequence:'MY_SEQ']
}