Im looking at Sqitch and so far it seems like a great tool, however I have an existing project that I want to use it with, Is there a way to create a baseline?
For example, I take a backup of my schema then add it to the deploy script, I then want to run a command that will not run the this script on the database as it already exists, but would apply everything after this point?
I need the full base schema in there so that we can re-deploy the whole schema if required
You can use the --log-only option of sqitch deploy command
From the docs: https://sqitch.org/docs/manual/sqitch-deploy/
--log-only
Log the changes as if they were deployed, but without actually running the deploy scripts. Useful for an existing database that is being converted to Sqitch, and you need to log changes as deployed because they have been deployed by other means in the past.
Related
We are planning to move our application in production which is using flyway with spring boot .Most of the them time we are facing Validate database exception doing application start
org.flywaydb.core.api.FlywayException: Validate failed:
Migration checksum mismatch for migration version .
To recover from this exception we need to correct data on database or last option is reset database .But when we move to production to fight with this exception it will be nightmare .So we want to follow best practice for configuring flyway in production .We need answer from expert who has been using flyaway for several years in production .Thank You .
What is it?
The checksum validation from Flyway is basically a check between the checksum of the current migration file in your app against the checksum from the same migrtion it already run in the past. You can check this list on your database, under flyway_schema_history table created and used by Flyway.
What it means?
It means that the script you app has when it starts is not the same Flyway already applied in the past and since it can't figure out if that is correct or not, it fails. Ideally, you should never change a script you already applied, you should always evolve and create new ones, that's the whole idea about migrations.
How to avoid it?
As said before, you should never change scripts that were already executed before. You should always create new ones. Of course if that happens on a dev environment and you figure out changes are needed.
Extra
I see in your error message it says version ., which means that probably you haven't defined a properly name for your migration script. By default, and as a good practice, names are in this format: VyyyyMMd_HHmmss__action_you_performed_on_your_database and all that will be translated to the Flyway table as version and description.
Based on my experience using flyway,
Flyway tries to compare the checksum of your SQL script with the checksum previously run. This exception usually happens if you edit an SQL script that has already been applied by Flyway, causing a discrepancy in the checksum.
Development environment, you can delete your database and start migrations again.
Production environment, you must never edit SQL scripts that have already been applied on Production environment. Just create a few new SQL scripts in the future.
In looking at the sqitch docs, there’s a situation I don’t immediately understand how to deal with.
Like probably many organizations, we progress changes through several environments before they reach production. In our situation, we have a different DBA user on a different Oracle server for each environment, each with its own credentials.
As I understand it, sqitch uses database tables to track what changes have been applied to a server. Maybe I’m dumb, but it just doesn’t jump out at me how sqitch can tell me if a change has been applied to a UAT server, but not yet to a production server.
So basically, I’d like to organize a repository to move changes from one DB environment to the next. Might this be what “sqitch target” and plan files are for? Are there examples I can look at?
If I were you, I would create a centralized DB with DB links to points to each database. After that, I would create an Union of all Repositories and a View (with PIVOT function) to see the deployement path of each patch.
To deploy to multiple environments it would depend on how you're running your sqitch deploy command.
You use the sqitch.conf to declare various targets.
eg.
[core]
engine = oracle
top_dir = SQL
[engine "snowflake"]
reworked_dir = SQL/rework
[target "DEV"]
uri = "db:oracle:DEV_DB"
[target "QA"]
uri = "db:oracle:QA_DB"
[target "PROD"]
uri = "db:oracle:PROD_DB"
With that sqitch.conf setup you can now run deploy to target the required environments.
eg.
sqitch deploy --target DEV
sqitch deploy --target QA
sqitch deploy --target PROD
You won't be able to compare deployments from one environment to another unfortunately.
You can use the sqitch check --target <xxx> command to check for divergences between the planned and deployed changes as stated here: https://sqitch.org/docs/manual/sqitch-check/.
However I've found this to not work properly at times. I haven't been able to determine the exact cause as yet but you're welcome to run the command to check.
For example I have script V1__InitScript.sql with query:
create table if not exists users
(
guid varchar(36) not null primary key,
name varchar(255) not null,
description varchar(500)
);
I start My app on clear DB and flyway run this script. Now I have emty table with name users and 3 colums. I start work and I need add one column, for example age. what should i do?
1) I can add this column to table. And after that add this query to V2__Add_column.sql. But When I start app flyway try make this V2 script because it not exist in flyway_schema_history table.
2) When I want to add a column, I immediately add it with flyway. But when I do active development I can often change the data. So I have to run flyway for every change?
It seems reasonable to change the structure of the database, and the scripts to collect in a separate file. and when preparing a new release, add all the necessary scripts to flyway. But how can I be on my developer machine? Or do I not need to run the flyway on my machine, but only on test and production?
and another question - how to build a process correctly, if the work is done with the same base for all developers. Ie the developer has no local database on the computer
For me the most convenient way was to have a development database, which is not populated by Flyway and could be modified manually at anytime, and some database for testing, which is populated by Flyway only and never manually, and is used for any testing purposes (include tests automation). And for sure, Flyway should be used in the production.
This way you are free to modify your database during development and can still have all the advantages of Flyway for your deployment.
You should aim to keep all your environments as identical as possible. Flyway usage is no exception here. Use it everywhere, even in dev.
As you use Spring Boot, make sure Flyway is run automatically when the ApplicationContext is started. This will ensure that the database is automatically migrated both on app and on unit test startup.
To iterate rapidly in dev you can active Flyway's cleanOnValidationError mode. This way, every time you modify your latest migration script which you haven't committed yet, its checksum will change, which in turn causes Flyway's validation to fail, which will then with this property trigger a clean of the dev database, which will be immediately followed by migrate to recreate it fully according to the latest version of your scripts.
I'm fairly new to server administration. I have my Laravel app up and running and I want to make sure it has proper backups. I have researched some backup packages and I have settled on https://github.com/spatie/laravel-backup.
However, once the server fails, I need to know how to use the most recent backup (which will be on AWS S3) to restore the database on the rebuilt server. Are there any suggestions for guides on how to do this? I can't seem to find any unless it doesn't really require much learning and instead just a couple mySQL commands.
Thanks!
I would use replication and within Laravel i would try to switch connection to the replica database server so things can run smoothly until the problem is resolved.
Take a look at this Cross-Region Replication
A typical production environment is automatically running backups on most important things that your deployment needs in order to recover from a failure. Those parts would commonly be your database and storage folder, and configuration files.
Also when you deploy a laravel application there aren't many things that are "worth" backing up , you can choose the entire disk to be mirrored somewhere or you can schedule a backup script which run every N times and backups the things that are more important to your application.
Personally i wouldn't rely on an package from laravel to handle my backups , you can always use other backup utilities, replication and so on.
Update
Take a look at the link below:
User Guide » Amazon RDS DB Instance Lifecycle » Backing Up and Restoring
Backing Up and Restoring
You can call the API function RestoreDBInstanceFromDBSnapshot as showed on example.
But i don't think something automated exists that would auto restore or magically make everything work, you need to do a lot of security checks if something like that would even be attempted. Final word i believe a good solution manually entering or sending the request would be the most solid solution.
I searching for an way to do a different migration in production and development.
I want to create a Spring Webapplication with Maven.
In development i want to update database schema AND load test data.
In production when a new version of the application is deployed i want only change the schema and don't load test data.
My first idea was to save the schema update and insert statements into different folders.
I think every body has solved this problem and can help me, thank you very much.
Basically, you have two options:
You could use different locations for your migrations in your flyway.locations property, i.e.:
for Test
flyway.locations=sql/structure,sql/test
for Production
flyway.locations=sql/structure
That way, you include your test data in the sql/test folder. You would have to take care with numbering, of course.
The second option (the one I prefer), is don't include test data in your migrations at all.
Rather, create your testdata any way you want and create an sql-dump of this data, which you keep separate from your migrations.
This works best if you have a separate database (instance, schema, whatever) containing your pristine testdata, where you apply each migration as part of your build process. This build job could then create a dump always matching the current migration.
When preparing your test machine, you first apply your migrations, then you load the contents of the matching dump.
I think this is a lot cleaner than the first version, especially because your test data can be prepared using other tools (your application) and has not to be handcoded.