I searching for an way to do a different migration in production and development.
I want to create a Spring Webapplication with Maven.
In development i want to update database schema AND load test data.
In production when a new version of the application is deployed i want only change the schema and don't load test data.
My first idea was to save the schema update and insert statements into different folders.
I think every body has solved this problem and can help me, thank you very much.
Basically, you have two options:
You could use different locations for your migrations in your flyway.locations property, i.e.:
for Test
flyway.locations=sql/structure,sql/test
for Production
flyway.locations=sql/structure
That way, you include your test data in the sql/test folder. You would have to take care with numbering, of course.
The second option (the one I prefer), is don't include test data in your migrations at all.
Rather, create your testdata any way you want and create an sql-dump of this data, which you keep separate from your migrations.
This works best if you have a separate database (instance, schema, whatever) containing your pristine testdata, where you apply each migration as part of your build process. This build job could then create a dump always matching the current migration.
When preparing your test machine, you first apply your migrations, then you load the contents of the matching dump.
I think this is a lot cleaner than the first version, especially because your test data can be prepared using other tools (your application) and has not to be handcoded.
Related
Im looking at Sqitch and so far it seems like a great tool, however I have an existing project that I want to use it with, Is there a way to create a baseline?
For example, I take a backup of my schema then add it to the deploy script, I then want to run a command that will not run the this script on the database as it already exists, but would apply everything after this point?
I need the full base schema in there so that we can re-deploy the whole schema if required
You can use the --log-only option of sqitch deploy command
From the docs: https://sqitch.org/docs/manual/sqitch-deploy/
--log-only
Log the changes as if they were deployed, but without actually running the deploy scripts. Useful for an existing database that is being converted to Sqitch, and you need to log changes as deployed because they have been deployed by other means in the past.
We are planning to move our application in production which is using flyway with spring boot .Most of the them time we are facing Validate database exception doing application start
org.flywaydb.core.api.FlywayException: Validate failed:
Migration checksum mismatch for migration version .
To recover from this exception we need to correct data on database or last option is reset database .But when we move to production to fight with this exception it will be nightmare .So we want to follow best practice for configuring flyway in production .We need answer from expert who has been using flyaway for several years in production .Thank You .
What is it?
The checksum validation from Flyway is basically a check between the checksum of the current migration file in your app against the checksum from the same migrtion it already run in the past. You can check this list on your database, under flyway_schema_history table created and used by Flyway.
What it means?
It means that the script you app has when it starts is not the same Flyway already applied in the past and since it can't figure out if that is correct or not, it fails. Ideally, you should never change a script you already applied, you should always evolve and create new ones, that's the whole idea about migrations.
How to avoid it?
As said before, you should never change scripts that were already executed before. You should always create new ones. Of course if that happens on a dev environment and you figure out changes are needed.
Extra
I see in your error message it says version ., which means that probably you haven't defined a properly name for your migration script. By default, and as a good practice, names are in this format: VyyyyMMd_HHmmss__action_you_performed_on_your_database and all that will be translated to the Flyway table as version and description.
Based on my experience using flyway,
Flyway tries to compare the checksum of your SQL script with the checksum previously run. This exception usually happens if you edit an SQL script that has already been applied by Flyway, causing a discrepancy in the checksum.
Development environment, you can delete your database and start migrations again.
Production environment, you must never edit SQL scripts that have already been applied on Production environment. Just create a few new SQL scripts in the future.
For example I have script V1__InitScript.sql with query:
create table if not exists users
(
guid varchar(36) not null primary key,
name varchar(255) not null,
description varchar(500)
);
I start My app on clear DB and flyway run this script. Now I have emty table with name users and 3 colums. I start work and I need add one column, for example age. what should i do?
1) I can add this column to table. And after that add this query to V2__Add_column.sql. But When I start app flyway try make this V2 script because it not exist in flyway_schema_history table.
2) When I want to add a column, I immediately add it with flyway. But when I do active development I can often change the data. So I have to run flyway for every change?
It seems reasonable to change the structure of the database, and the scripts to collect in a separate file. and when preparing a new release, add all the necessary scripts to flyway. But how can I be on my developer machine? Or do I not need to run the flyway on my machine, but only on test and production?
and another question - how to build a process correctly, if the work is done with the same base for all developers. Ie the developer has no local database on the computer
For me the most convenient way was to have a development database, which is not populated by Flyway and could be modified manually at anytime, and some database for testing, which is populated by Flyway only and never manually, and is used for any testing purposes (include tests automation). And for sure, Flyway should be used in the production.
This way you are free to modify your database during development and can still have all the advantages of Flyway for your deployment.
You should aim to keep all your environments as identical as possible. Flyway usage is no exception here. Use it everywhere, even in dev.
As you use Spring Boot, make sure Flyway is run automatically when the ApplicationContext is started. This will ensure that the database is automatically migrated both on app and on unit test startup.
To iterate rapidly in dev you can active Flyway's cleanOnValidationError mode. This way, every time you modify your latest migration script which you haven't committed yet, its checksum will change, which in turn causes Flyway's validation to fail, which will then with this property trigger a clean of the dev database, which will be immediately followed by migrate to recreate it fully according to the latest version of your scripts.
I've just started learning .NET MVC so this may be a silly question, but I've yet to find a good answer.
I'm following the Code First approach using the Entity Framework to build my database for me. I've included the following in my Application_Start() method in order to allow me to edit my database by making changes to my Model objects.
Database.SetInitializer<ContactManagerDB>(new DropCreateDatabaseIfModelChanges<ContactManagerDB>());
I was just wondering what would happen if I pushed this application to a production environment and then made a few changes to my models and then updated the application? Would this really drop and recreate the database in the production environment?
What's the best practice for pushing changes to production env. using the Code First approach?
DropCreateDatabaseIfModelChanges should only be use early on in development, never on a production machine. If you pushed to a production machine and made schema changes, you'd loose all your data.
You could delete the EdmMetadata table in your production environment. In that case, EF would not know the current schema to compare to the new, so it would just assume you know what you are doing and it would not touch the database schema.
Code first does not have the ability to upgrade your database while keeping your data intact.
I'm starting an ASP.NET MVC project using SubSonic 3 ActiveRecord. I added a table Users with a primary key ID and recompiled T4 files to generate User class.
I want to make sure that, as I go along with the development, I can regenerate/migrate the database at any point. It looks like I have to create tables and relationships in the database, regenerating ActiveRecord classes and doing migration as described in http://subsonicproject.com/docs/3.0_Migrations. The old 2.x way of defining migrations doesn't seem to be available any more.
Is there a way to drive development from the code rather than database, by changing model classes, and have the database migrated accordingly, without using SimpleRepository? I don't want to put generated code into source code repository, but if I don't, I lose database schema (unless I export and save it manually).
You can still use SubSonic 3 as a DAL and let SubSonic 2.2 generate the migrations for you.
You just need sonic.exe and it's dependencies to execute the migration files.
To be more precise, I have folder Migrations in my DataLayer Project, that keeps all the migration Files but set the BuildAction to none. So I have Syntax Highlighting (but no error checking unless I set BuildAction back to compile) but the code does not mess my project.
You can keep them in an own project, of course. But I like it this way to have it under version control and be sure that my current DAL is matching my migration version.
In addition, I have named my config file migration.config (instead of app.config/web.config) and use this commandline to execute my migrations.
Command: /path/to/sonic.exe
Arguments: migrate /config migration.config
Working Directory: $(SolutionDir)MyProject.Datalayer
Notice the /config switch to override the config file sonic.exe is looking for.
You could have a look at SimpleRepository:
http://subsonicproject.com/docs/Simple_Repo_5_Minute_Demo
The document you linked to does state:
"Bottom line: if you're a developer that is concerned about database design, migrations might not be for you"
I suspect that for the detail of design you want (and I would too), the migrations may not be suitable?