I have a multi-maven module project and I have two modules that contain some Liquibase configuration with tables.
I want to add a foreign key between two tables but table A is in module A and table B is in module B. Problem is that during compilation I got an error because Liquibase doesn't see table A...
Is some solution for that?
I'm assuming this problem only appears when you use jOOQ's LiquibaseDatabase, which simulates your liquibase migrations for code generation. I can think of a few solutions to this problem:
Use a test containers based code generation instead. The blog post before uses Flyway, but it will work the same with Liquibase. This means that during your build of both modules, you'll have a running database instance, which will be up to date by the time you generate jOOQ code. That also removes any simulation related issues, such as allowing you to use vendor specific features of your RDBMS.
Extract the database change management and schema into another module, making the schema "global", instead of module dependent. Your foreign key kinda hints at the schema being global anyway. The relational model isn't really directed in a tree form. It's a graph. Sooner rather than later, you'll have a key from A to B and your dependencies will become cyclic.
Stay purely "modular" and make the schemas independent, removing the key. Of course, data integrity is a good thing, so the price to pay for this is high, but it would solve your immediate problem.
Remove the key from jOOQ's code generation and declare a synthetic foreign key in jOOQ's code generation (though, B would still somehow have to know about A in the generation process)
Run both liquibase migrations for A and B when you generate code for module B. That way, B has all the information available again.
Which to pick is a subjective choice and doesn't have a clear answer. But at least, this gives you options.
Related
my project has large oracle sql scripts. liquibase locks the schema (DATABASECHANGELOGLOCK table) when installing a single patch. How do I install multiple patches in parallel without a queue?
P.S. Oracle will independently make locks at its discretion.
Any DDL is make the new schema state that is based on previous state. If the previous state is not valid, you cant apply next DDL (it is impossible to add new constrain to the column that not exist). To check the previous state, you use precondition in your changesets.
So, in general it is impossible to parallelise the schema update, because the schema changes should be applied in order and the order can't be changed.
The lock on DATABASECHANGELOGLOCK is aimed to be sure that it is impossible to run two schema update process in one time, and it is reasonable restriction, so don't try to get around it.
If update process takes to much time, just be sure that you:
not use liquibase to change database state (add data to tables)
not use liquibase to update code objects (functions, procedures and etc.) in the database
not use liquibase for migrate large amount of data
I want to migrate a subset of customer data from one shared database environment to another shared database environment. I use hibernate and have quite a few ID and FK_ID columns which are auto generated from an oracle sequence.
I have a liquibase change log that I exported from jailer which has the customer specific data.
I want to be able to rewrite all of the sequence ID columns so that they don't clash with what's already in the target database.
I would like to avoid building something that my company has to manage, and would prefer to upstream this to liquibase.
Is anyone aware of anything within liquibase that might be a good place to start.
I would like to either do this on the liquidbase xml before passing it to 'update' command, or as part of the update command itself. Ideally as part of the update command itself.
I am aware that I would need to make liquibase aware of which columns are PK sequence columns and the related FK columns. The database structure does have this all well defined, so I should be able to read this into the update process.
Alternatively I had thought I could use the extraction model csv from jailer
Jailer - http://jailer.sourceforge.net/
I would suggest that for one-time data migrations like this, Liquibase is not the best tool. It is really better for schema management rather than data management. I think that an ETL tool such as Pentaho would be a better solution.
I actually managed to figure it out for myself with the command line 'update' command of liquibase by using a custom change exec listener.
1) I pushed a MR to liquibase to allow registration of a change exec listener
2) I implemented my own change exec listener that intercepts each insert statement and rewrites each FK and PK field to one that is not as yet allocated in the target database. I achieve this by using a oracle sequence. In order to avoid having to go back to the database each time for a new sequence, I implemented my own version of the hibernate sequence caching
https://github.com/liquibase/liquibase/pull/505
https://github.com/pellcorp/liquibase-extensions
This turned out to be quite a generic solution and in concert with some fixes upstreamed to jailer to improve the liquibase export support its a very viable and reusable solution.
Basic workflow is:
1) Export a subset of data from source db using jailer to liquibase xml
2) Run the liquibase update command, with the custom exec change listener against the target.
3) TODO Run the jailer export on the target db and compare with the original source data.
To generate insert statements from my databases (oracle,db2) i've used liquibase generateChangeLog command with argument
--diffTypes="data"
This command generate correct xml with insert statements, however this is not aware of foreign constraints, so I cannot use this file to again to fill my databases. Similar problems has been described here : Is there a way to generate Liquibase data in the right order?. The proposed workaround unfortunatelly is not possibile for my databases, because there is no any command to switch of constraint checks.
My question is, if exists any other solution to this problem ? Why can I generate data insert statements changelog, but cannot use it because of foreign key constraints ?
Jens is right in the comment. Liquibase has no way of determining dependencies because the main use case is tracking ran changeSets. GenerateChangeLog is a useful capability but it is not intended to handle all cases and managing dependencies is a complex task that is definitely out of scope.
My normal recommendation is that the output of generateChangeLog should be considered a useful first step for working with the changeLog and if you have dependency issues just reorder the changeSets before executing them. If there many of them, you may want to write a script to reorder them based on your knowledge of your dependencies.
If you can export schema + data, a good solution is separate DDL, DML and this part of DDL that create the constraints. You have to reorder those in the following order:
DDL (without constraints)
DML (data)
DDL (the constraints removed from step 1)
It requires a little bit of manual editing but you'll have to do it once for your project.
And, as a side note, for you next project, start using liquibase from the start, this way you will never encounter this problem anymore.
The jailer export liquidbase feature is excellent for this. It generates a change log in topological order.
http://jailer.sourceforge.net/
What are the implications (on performance and other aspects) of having Rails Associations between ActiveRecord Models that are not mapped to a Database relationship
I have seen this in a real project and have searched for these implications without any luck
These are some of the documents I have found
http://guides.rubyonrails.org/association_basics.html
http://www.ibm.com/developerworks/library/os-railsn1/
May not be an answer but it would make for a long comment and may add value to the OP.
Correct me if I am wrong but I believe what the OP is getting at is that the code creates the relation but the database does not have a key relationship using foreign and primary keys inside the actual database. Since rails uses Object-Relational (ORM) structure instead of a Database-Relational structure.
Not sure if there are really any draw back implications (although I am sure many would disagree) and the second article seems to focus mostly on n+1 issues which you can resolve by writing appropriate code. Many n+1 issues spring out of code where people do not want to be explicit with their attribute selections and thus rails obliges by returning all attributes of a record. then when you request that objects relationship rails graciously runs another query for you and returns what you've asked for but when you are doing this in large iterations this can cause a large load and create performance issues.
You can avoid these issues by explicitly selecting exactly what you want by using select and group as well as using include or eager_load statements. Yes these are more complicated to write originally as many of them will require more SQL and less railsy syntax but the performance increase can be drastic when querying large tables or multiple relationships.
These are statements you would generally write by hand in stored-procedures or queries in SQL. Rails still gives you these freedoms but many times they are ignored because they look more like SQL than ruby and I have seen more than once people complaining about both the fact that if they change the DB they will have to change this too and that it "looks bad". Well if you were writing this from scratch in a standard database both these issues still apply.
As many have stated "ORM is just a tool" and any tool that is used improperly can have drastic implications. As long as you understand the tools you are using ORM can be very powerful and far more concise but if you ignore features your tool provides expect to have consequences with an extensive reach.
If you use a chainsaw as a hammer expect to lose an arm
UPDATE
I am unaware of any improvements in performance due to relationships inside the database other than it offers a hint as to where to index values. (You can create indexes during rails migrations although the primary ones are created for you the relational ones generally are not). Adding indexing to relational fields will generally speed up the database performance whether or not the specific relationship is actually defined in the database. Someone Asked a Similar Question
Database Relationships are more about Data Integrity than anything else and these issues should be handled inside your models in an ORM design not in the database its self.
Here is another resource to take a look at
And a few more SO questions on the subject
In SQL Server 2008, do relationships make queries faster?
Does Foreign Key improve query performance?
What's wrong with foreign keys?
Is there a severe performance hit for using Foreign Keys in SQL Server?
SQL Server Foreign Key constraint benefits
You get the idea just about asking the right question.
SQL Developer supports unit testing of DML but I've not found a way to create unit tests for DDL. What would be a good approach to this problem? The schema I'm starting with is small, less than a dozen tables with larger projects on the horizon. Google isn't returning much to the application of unit tests to DDL. Any ideas on an approach to testing DDL or other tools that exist for unit testing DDL?
What do you want to test about DDL? Either the table is created as defined or it is not.
What you could do is write a series of tests that queries the Data Dictionary to ensure the tables are present, have the columns with the sizes and datatype you want etc. This would be more of a schema verification script than unit tests however, and I am not sure how valuable it would be.
If you maintain a schema build script (or a series of migrations to add new objects to add objects to your schema), then if it applies without errors you know the schema has been created as it was defined.
Then if you have stored procedures, some of them will fail to compile if the schema is not 100% correct. Getting the procedures in cleanly would be another verification step for the schema.
Finally, the unit tests that you write to test the DML and stored procedures will verify that the correct data goes into the correct tables.
You might want some tests to ensure that a table can only accept certain values or columns can be unique etc (ie test the constraints are correct) but that would be down to standard unit tests too.
I am a big believer in writing unit tests for DB code, but I don't like SQL Developers GUI approach of doing it. Right now I am writing tests for an application, but I am coding the tests in Ruby and it seems to be working well. It will also be easily built into our build and automated test process.
Another alternative is UT_PLSQL which I have used before, however simply due to the nature of PLSQL is makes the tests very verbose, which is why I decided to use Ruby for my current project.
I know this is an older question, but I've recently been working to solve the same problem. I think it's useful to define tests for DDL prior to creating objects and then creating those objects to pass those tests.
I've done some of this using an assert "pattern" -- i.e., tdd.ddlunit.assert_tableexists(p_schema_name, p_table_name) which raises an exception if the table doesn't exist, and silently runs when it does.
Other assertions I've created are for things like making sure all varchar2 columns use character semantics instead of byte length semantics, and making sure all tables and columns are commented.
These get checked in to the code repository and can be run via continuous integration frameworks to make sure we have a valid database per what we expect.