So, I just read and learnt about migrations in Flyway.
I developed a spring-boot project and included Flyway in the projects POM file.
Now I wrote a SQL script in a file and put it in the resources/db folder of the project and Flyway seems to do its job.
Sadly the SQL script has some errors and the project won't compile. Instead it gives me a SQL error. After that i change my SQL script and resolved my errors. When I run the same project it now throws an error saying that validation checksum failed. I checked back the flyway_schema_history and it shows me the previous run script.
Shouldn't Flyway store only those migrations which are correct and run properly? Because if by mistake I have some errors in a SQL script, i will have to make one script and copy my corrected SQL code in that and run it. Is there anything like this present in Flyway?
Flyways purpose is to control the versions of your database. This is only possible with the help of strict versioning rules, like every published version gets stored with a matching checksum. Once a version is published, it cannot just be deleted or manipulated because this would effect the database status.
The validation checksum failed error Flyway prints you is exactly the effect of manipulating an existing and versioned SQL file.
However - there are some options to revert the faulty SQL file:
Undo the last migration:
So go ahead and invoke
flyway undo This will give you the following result:
Database: jdbc:h2:file:./foobardb (H2 1.4) Current version of schema
"PUBLIC": 2 Undoing migration of schema "PUBLIC" to version 2 - Add
people Successfully undid 1 migration to schema "PUBLIC" (execution
time 00:00.030s)
(https://flywaydb.org/getstarted/undo#undoing-the-last-migration)
If you are still in a early testing phase, you can also delte the complete database and start with a fresh one...
Related
I tried to deploy a keystone app to Heroku and I did it but while I tried to open the app I got the following error:
An error occurred handling a request for the Admin UI: Error: Prisma error: The table main.User does not exist in the current database.
Here's a screenshot containing more details about the error:
I tried to locate the database and create the User table.
I expect to know the steps of how to solve this issue.
It looks like your DB hasn't been initialised properly. The error you've included is failing to count the items in the User list which (if you don't have sessions configured) is likely the first query to run – a count of items in each list is shown on the Admin UI the landing page so that's the first thing it does.
So something about how your migrations are being generated or applied in production isn't setup right. Most of the relevant docs on how this works are in the CLI guide, specifically, see the section about database migrations and the db.useMigrations flag.
Having db.useMigrations turned off can be handy if you're just playing around in dev. Keystone will automatically sync your DB structure to what's defined in your list configs whenever it starts, and does so without creating any physical migration files. If you're prototyping some change or just mucking around, this may be what you want but – if you're deploying somewhere – better to turn db.useMigrations on. Then, if Keystone detects changes to the DB when it runs, it'll prompt you to create a migration file, which can be tweaked to protect existing data if needed, tracked under version control (eg. git) and deployed.
Getting these migrations to run in an environment like Heroku is a little slightly weird as (assuming it's enabled for your app) Heroku can auto-scale. Migrations on the other hand need to be run exactly once. You also can't just lock the DB and run migrations when the first instance of the app starts – this delays the start up of the HTTP server so, if the migrations run for too long, Heroku may think the deployment has failed.
The way we suggest getting around this is to run migrations in the build staging. Fans of the 12-factor app methodology will notice this violated the separation of build and release stages but, for a simple Heroku deploy, it works fine. For larger/more serious apps, creating and applying migrations usually an area that needs significant thought and attention. The specific infrastructure and rollout processes required will be project dependant.
I'd also encourage you to check out the Keystone 6 Heroku example codebase if you haven't already. It's a little out of date but it shows the migrations and package.json scripts in action.
Im looking at Sqitch and so far it seems like a great tool, however I have an existing project that I want to use it with, Is there a way to create a baseline?
For example, I take a backup of my schema then add it to the deploy script, I then want to run a command that will not run the this script on the database as it already exists, but would apply everything after this point?
I need the full base schema in there so that we can re-deploy the whole schema if required
You can use the --log-only option of sqitch deploy command
From the docs: https://sqitch.org/docs/manual/sqitch-deploy/
--log-only
Log the changes as if they were deployed, but without actually running the deploy scripts. Useful for an existing database that is being converted to Sqitch, and you need to log changes as deployed because they have been deployed by other means in the past.
We are planning to move our application in production which is using flyway with spring boot .Most of the them time we are facing Validate database exception doing application start
org.flywaydb.core.api.FlywayException: Validate failed:
Migration checksum mismatch for migration version .
To recover from this exception we need to correct data on database or last option is reset database .But when we move to production to fight with this exception it will be nightmare .So we want to follow best practice for configuring flyway in production .We need answer from expert who has been using flyaway for several years in production .Thank You .
What is it?
The checksum validation from Flyway is basically a check between the checksum of the current migration file in your app against the checksum from the same migrtion it already run in the past. You can check this list on your database, under flyway_schema_history table created and used by Flyway.
What it means?
It means that the script you app has when it starts is not the same Flyway already applied in the past and since it can't figure out if that is correct or not, it fails. Ideally, you should never change a script you already applied, you should always evolve and create new ones, that's the whole idea about migrations.
How to avoid it?
As said before, you should never change scripts that were already executed before. You should always create new ones. Of course if that happens on a dev environment and you figure out changes are needed.
Extra
I see in your error message it says version ., which means that probably you haven't defined a properly name for your migration script. By default, and as a good practice, names are in this format: VyyyyMMd_HHmmss__action_you_performed_on_your_database and all that will be translated to the Flyway table as version and description.
Based on my experience using flyway,
Flyway tries to compare the checksum of your SQL script with the checksum previously run. This exception usually happens if you edit an SQL script that has already been applied by Flyway, causing a discrepancy in the checksum.
Development environment, you can delete your database and start migrations again.
Production environment, you must never edit SQL scripts that have already been applied on Production environment. Just create a few new SQL scripts in the future.
For example I have script V1__InitScript.sql with query:
create table if not exists users
(
guid varchar(36) not null primary key,
name varchar(255) not null,
description varchar(500)
);
I start My app on clear DB and flyway run this script. Now I have emty table with name users and 3 colums. I start work and I need add one column, for example age. what should i do?
1) I can add this column to table. And after that add this query to V2__Add_column.sql. But When I start app flyway try make this V2 script because it not exist in flyway_schema_history table.
2) When I want to add a column, I immediately add it with flyway. But when I do active development I can often change the data. So I have to run flyway for every change?
It seems reasonable to change the structure of the database, and the scripts to collect in a separate file. and when preparing a new release, add all the necessary scripts to flyway. But how can I be on my developer machine? Or do I not need to run the flyway on my machine, but only on test and production?
and another question - how to build a process correctly, if the work is done with the same base for all developers. Ie the developer has no local database on the computer
For me the most convenient way was to have a development database, which is not populated by Flyway and could be modified manually at anytime, and some database for testing, which is populated by Flyway only and never manually, and is used for any testing purposes (include tests automation). And for sure, Flyway should be used in the production.
This way you are free to modify your database during development and can still have all the advantages of Flyway for your deployment.
You should aim to keep all your environments as identical as possible. Flyway usage is no exception here. Use it everywhere, even in dev.
As you use Spring Boot, make sure Flyway is run automatically when the ApplicationContext is started. This will ensure that the database is automatically migrated both on app and on unit test startup.
To iterate rapidly in dev you can active Flyway's cleanOnValidationError mode. This way, every time you modify your latest migration script which you haven't committed yet, its checksum will change, which in turn causes Flyway's validation to fail, which will then with this property trigger a clean of the dev database, which will be immediately followed by migrate to recreate it fully according to the latest version of your scripts.
I'm starting an ASP.NET MVC project using SubSonic 3 ActiveRecord. I added a table Users with a primary key ID and recompiled T4 files to generate User class.
I want to make sure that, as I go along with the development, I can regenerate/migrate the database at any point. It looks like I have to create tables and relationships in the database, regenerating ActiveRecord classes and doing migration as described in http://subsonicproject.com/docs/3.0_Migrations. The old 2.x way of defining migrations doesn't seem to be available any more.
Is there a way to drive development from the code rather than database, by changing model classes, and have the database migrated accordingly, without using SimpleRepository? I don't want to put generated code into source code repository, but if I don't, I lose database schema (unless I export and save it manually).
You can still use SubSonic 3 as a DAL and let SubSonic 2.2 generate the migrations for you.
You just need sonic.exe and it's dependencies to execute the migration files.
To be more precise, I have folder Migrations in my DataLayer Project, that keeps all the migration Files but set the BuildAction to none. So I have Syntax Highlighting (but no error checking unless I set BuildAction back to compile) but the code does not mess my project.
You can keep them in an own project, of course. But I like it this way to have it under version control and be sure that my current DAL is matching my migration version.
In addition, I have named my config file migration.config (instead of app.config/web.config) and use this commandline to execute my migrations.
Command: /path/to/sonic.exe
Arguments: migrate /config migration.config
Working Directory: $(SolutionDir)MyProject.Datalayer
Notice the /config switch to override the config file sonic.exe is looking for.
You could have a look at SimpleRepository:
http://subsonicproject.com/docs/Simple_Repo_5_Minute_Demo
The document you linked to does state:
"Bottom line: if you're a developer that is concerned about database design, migrations might not be for you"
I suspect that for the detail of design you want (and I would too), the migrations may not be suitable?