I've always thought that if a migration applied with flywaydb is removed (both: entry from DB schema_version table and migration sql file) then application (written in spring framework with flyway in classpath) will crash at startup because of invalid checksums.
Actually (I tested it today) when both migration file and DB entry is removed application starts just fine, which I found a bit confusing.
How does flyway calculate the checksums? Is it based only on content of migration file being applied or previous migrations are also taken into account?
Also, is it safe to remove already applied migrations if both file and table entry is removed?
Many questions here. For your main question, this behavior is controlled by the ignoreMissingMigrations flag. See https://flywaydb.org/documentation/commandline/migrate
Checksums are calculated based on the content of the migrations files (ignoring line-endings due to Git CRLF transformations).
If both a migration file and its entry in the metadata table are removed Flyway doesn't know anything about it anymore. This however doesn't mean it is safe, as it may impact your ability to recreate your database schema later.
Related
We are working in a very big organization, many Databases (of many types), many schemas, many users.
Does LB has to work with some Source Control (for locking the files
when many users exist in the organization and using the same DB,
same Schema, etc)?
What is the best practice of working with LB in a very big
organization, many concurrent users?
Can SQLCL general sql format type or just xml format type?
Is there some integration with SQL Developer? I mean, suppose a user
changes an objects via sql developer, what happens then?
We get this type of question all the time, after folks get a handle of how to automate DB changes, next step is typically to add it into an existing CI/CD workflow.
Yes, Liquibase works with any source control. Most users are using
Git. But you can use Git, TFS, SVN, CVS... Once you are up and
running with Liquibase, you just need to make sure that your scripts
are in source control and you are good to go.
Besides 3rd party source control tools, Liquibase has tracking tables called "DATABASECHANGELOG" tables that keep track of the changes applied to your database when using Liquibase deployments.
Here is some more information about getting started and How Liquibase Works. https://www.liquibase.org/get_started/how-lb-works.html
Liquibase has one more table that it uses internally called "DATABASECHANGELOGLOCK" table.
This table was designed to prevent multiple Liquibase users running deployments concurrently - potentially leaving the Database in a bad state. Once the Liquibase deployment (the liquibase update command) is done, the "DATABASECHANGELOGLOCK" will allow the next Liquibase user to deploy.
You can use both SQL and XML formats (or even JSON and YAML formats).
When using SQL, you have a few options:
Best option is to use Formatted SQL changeLogs https://www.liquibase.org/documentation/sql_format.html
https://www.liquibase.org/get_started/quickstart_sql.html
You can use plain raw SQL files referenced from an XML changeLog
https://www.liquibase.org/documentation/changes/sql_file.html
When using XML, can find all the available change types (also called changeSets) available in the following page (on the left of the page)
https://www.liquibase.org/documentation/changes/
XML changeLog are more agnostic and sometimes can be used for different Database platforms when doing migrations. Also, many of the change types in XML have the ability to be rolled back automatically. The reason that this is possible with XML is because Liquibase uses it own built in functions to figure out inverse statements like "create table" to be "drop table".
For each of those changeSets you can find out if they are auto rollback eligible (at the bottom of the page). For example, create table changeSet will be Auto Rollback = yes.
https://www.liquibase.org/documentation/changes/create_table.html
I try use flyway for migration. I find this option
spring.flyway.target= # Target version up to which migrations should
be considered.
But I need set version with which to start migration. For exmple I need start migration from V3_foo.sql
Can I do it?
Briefly why I need it. I have a database with data. No migration tools have been used before. Everything was done manually. Now I have created an init.sql and placed in it the creation of the entire base structure. Now, when adding changes, I will start the migration from version 2. And if you need to run on a new empty database - from version 1
Baseline is the flyway feature you need.
If your case, baseline your database with flyway.baselineVersion=2 will tell flyway that your database is already at the version 2. Any subsequent flyway migrate will only process migrations greater than 2.
Note: If previous migrations failed, it may be necessary to drop table flyway_schema_history first.
I'd like to have my version only the latest table definition in my repository, (no change sets), and have liquibase figure out which changes are needed when patching my databases. Please take note that I have a very big database schema (1000+ tables) installed in hundreds of customer sites, with different versions each one, and I really don't know which objects each version has
How can I make a liquibase-based installer for my application, given my set of table definitions, and hundreds of databases with about 12 different versions of objects on each one?
To be more specific, I'd like liquibase to compare my table definitions with the production database, and emit the alter table statements required to make the database current with my latest version.
I could contribute code if necessary in order to get this done
Liquibase and tools like it (for example flyway) are primarily designed to support database migrations. A migration is where every change to the DB is tracked so that it can be replayed on target environments thereby keeping them in sync with development (although time-shifted). It's all about keeping your schema under revision control.
Your use case is a little different. If I understand correctly you're trying to retrofit Liquibase onto a series of environments that you are not 100% certain match your application's current schema?
I would only recommend migration tools like liquibase if you intend to use them going forwards. If all you want is a DB diff tool, I would suggest you look elsewhere.
To perform an initial sync then I would suggest you investigate the diffLog command, coupled with changeLogSync command to initialize liquibase on the target DB.
comparing databases and genrating sql script using liquibase
In my project, current approach is to create database if not already exists using CreateDatabaseIfNotExists and doing seeding initial data from that Intializer as well. I also added Code First Migration support after upgrade to Entity 4.4, so that in the future when we change the modle/database structure we can update client side database without drop their exist database.
However it didn't seems to working well, for example, I am now stuck on design time where forms wouldn't load and the error message is something like The model backing the 'myEntities' context has changed since the database was created. Consider using Code First Migrations to update the database (http://go.microsoft.com/fwlink/?LinkId=238269).. But the model and database is indeed the updated version, just seems Migration didn't recognize the database generated by CreateDatabaseIfNotExists, but at the same time all seems working well at run time.
Also after that I noticed that if I let CreateDatabaseIfNotExists initialize a database, Add-migration afterwards will fail and complain that pending migration and ask me to do a update-database. When I try to do a the update-database, it will fail as well because the migration path seems assume the database is in initial setup state and will trying to running all the migration scripts while none should be run as the database generated by CreateDatabaseIfNotExists is indeed sync with current model and should not be migrated at all.
I discovered that there is a MigrationHistory table in System Tables, that table will always save the database initialize history regardless of whether the initializer is CreateDatabaseIfNotExists or MigrateDatabaseToLatestVersion. The difference is that if database is initialized by CreateDatabaseIfNotExists1, everytime the database initialized, the migriationId for that initialize record will be different, butMigrateDatabaseToLatestVersion` will always save same set of migrationId for each of the Migration steps. I guess that is how Entity Framework 5.0 works.
So in the end, I give up and rewrite my DB Access code to seeding the initial database data
in other part of my code rather than the CreateDatabaseIfNotExists or MigrationsConfiguration` class because either suit my need.
I have a local instance of a database that I recently created using DbContext.Database.Create(), so the __MigrationHistory table exists with an InitalCreate entry that matches the code at the moment.
Some code-based migrations exist in the Migrations folder, however. These will be run in our development and staging environments to bring those databases in line with the code. I don't need to apply them locally, however, since I created the database using the current code.
I now need to make a change to the model and create the corresponding migration. But when I run Add-Migration TestMigration, I get the following error
Unable to generate an explicit migration because the following explicit
migrations are pending:
[201203271113060_AddTableX,
201203290856574_AlterColumnY]
Apply the pending explicit migrations before attempting to generate
a new explicit migration.
What should I do in this case? I can't point the Add-Migration tool at another environment because it's not guaranteed that version matches what I have locally. I want a migration that matches only the changes I've made.
It seems I have a few options but none are ideal:
Delete the other migrations from the Migrations folder, run the Add-Migration command, upgrade the database, then restore the old migrations. This is simple but seems a bit hackish.
Revert to the version of the model in source control that the first migration was applied to, then build this and use it to create the database. Then get the latest version, apply all the migrations, then I'm ready to add my migration. This seems like a lot of effort!
Create the migration manually.
Does anyone have any suggestions about how to manage this?
We are planning to use a variant of your Option #1...
Our Standard Operating Procedure is to generate a SQL script for each migration (using the -script option of update-database), in order to have SQL scripts to be applied to end-user "production" databases by InstallShield (we plan to use EF update-database only for developer databases).
Thus, we have both the Migration .cs files and the corresponding .sql files for all migrations in our Migrations folder.
So rather than deleting the migrations from the Migrations folder (as you proposed in #1), we use SQL Mgmt Studio to manually apply just the parts of the .sql files that do the inserts into _MigrationHistory.
That brings the _MigrationHistory of the local database up-to-date with the changes that are already incorporated into that database.
But it's a kludge, and we're still looking for a better solution.
DadCat
What I've found works best is very simple: don't use DbContext.Database.Create() once you've enabled migrations. If you want to programmatically create a new database, use the migrations API instead.
var migrator = new DbMigrator(new Configuration());
migrator.Update();
Then you've got the full migration history and adding further migrations works just as expected.
You either need to run "update-database" from the package manager console to push your changes to the database OR you can delete the pending migration file ([201203271113060_AddTableX]) from your Migrations folder and then re-run "add-migration" to create a brand new migration based off of your edits.
I have encountered the same problem.
If you run
Update-database
and then run
Add-Migration YourMigrationName
This solves the problem
simply exclude the old migration file from the solution files.