I tried following through the source code and the docs, but after a lost morning I give up: at the same time it's as if not enough assumptions are made in the SchemaDumper and at the same time there's no SchemaLoader and following through the sequel command source code it seems that it clobbers migration information to-date (since it has no "migrations to date" kind of information in the resulting file).
The motivation to do this is a failed migration in tests (Sequel thinks the tables are not there, yet they are so it breaks both on migrating to new versions and the check pending migrations check fails) - and a previous experience that running all migrations from start of history to today is generally a bad way to put up a database.
I've got this thus far:
namespace :schema do
task :dump => :migrations_environment do
schema = without_sequel_logging{ DB.dump_schema_migration }
File.open("db/schema.rb", 'w') {|f| f.write(schema) }
end
task :load => :migrations_environment do
Sequel::Migrator.run(DB, "db/schema.rb")
end
end
normally the load fails since the Migrator makes a load of assumptions ranging starting from that it will be given a folder full of files in a specific order, yet this is apparently exactly what sequel -m and sequel -d should do according to current source code - and sequel -m and sequel -d combo are apparently what you should use when you want to do schema dump & schema load.
Any ideas?
I think you are misunderstanding the point of Sequel's schema dump and load. Schema dumping should only be used if you have an existing database and would like to produce a migration from it, either for review of what tables/columns exist, or for loading into an empty database. Loading a migration dumped by the schema dumper should only be done on an empty database.
If you already have an existing test database that is not empty (i.e. previous migrations have been applied to it), you shouldn't be using schema dump and load, you should just run the migrator on the test database. In general, it's best to migrate your test database before you migrate your development database, so you can then run your tests and see if the migration breaks anything.
The only time you should have to run all migrations since the beginning is if you have an empty database. If you migrate your test databases similar to the way you migrate your development and production databases, you are generally only applying a single migration at a time.
Note that the schema dumper only handles a small fraction of what is possible, and will only work correctly for the simplest cases. It doesn't handle dumping views, functions, triggers, partial/functional indexes, and a whole range of other things. For all but the simplest cases, use the database's tools to dump and load schema.
Related
My reasoning, you can skip to below if not interested
I understand how the general procedure of migrations work and the purpose they serve, and very happy to use them in the way that is expected, that is by adding and removing fields as nessesary throughout the applications life.
My query is that at the very beginning of a project I rarely know many of the fields I will need in a given table, and at the very early stages of my projects I want to get the main features and relationships set up, and maybe just use some dud fields before the client makes up their minds on things.
The bottom line is it hurts my OCD knowing there are extra migration files sitting there that potentially look nothing like v1.0 of the project... once im at v0.5 I may decide im far enough along to start properly managing migrations.
Thats my thoughts, but here is the question:
What is the cleanest steps to reuse the same migration script again and again in the early stages of a project while there is no worry about loss of data or rolling back.
Just to add to this i would not want to refresh the entire migration because I would really prefer to keep any data I am playing around and especially the user table for staying logged in to the backend etc.
Would it be wrong to do this:
Could I just remove the migration row in the table, then run the migration?
This feels like it would have side effects, and possibly screw up rolling back, is that the case? What part does the migration table play, as this seems to work in practice?
Final words
Please bear in mind this is just a concept I'm trying to get my head around. If its absolutely bad practice no matter the circumstance I can accept it!
Edit:
Create a new seeder with php artisan make:seeder UserSeeder
Edit the seeder to "seed" the necessary data. Ex:
DB::table('users')->insert([
'name' => 'John Doe',
'email' => 'johndoe#example.com',
'password' => Hash::make('password'),
]);
Then call the built-in artisan function php artisan migrate:fresh --seed which will drop all tables and re-run all of your migrations, then seed your data with your seeders.
You can read more about this process here.
Original:
If you plan to support a live application in the long run, chances are you will have very many of these separate migration files that will still hurt your OCD every time you make a new one. It happens to me every time I create a new one migration to alter a table, haha.
However, in development I can understand your point if you're working on a private codebase and no other developers will be trying to keep up with your changes. If you do, any changes you make to the old migration files will be very tough for them to mimic as the migrations table keeps track of what migrations need to be run (if any), so if someone else tried to migrate after you've changed a previous migration, nothing would happen.
What I would do, is either set up a database seeder in Laravel so you can quickly reseed the data in the table if you rollback a migration, or get a sql dump of your table and reinsert it after you've migrated again.
Another thing you could consider is, not to worry about the migrations directory at this point in development and once you're ready to deploy or push, go through your table alterations and kind of "refactor" them into your desired migrations. But definitely run some thorough testing after this to ensure you're not missing any columns or alterations.
Where's the best place to store the version of a table in Oracle? Is it possible to store the version in the table itself, e. g. similar to the comment assigned to a table?
I don't think you can store that information in Oracle, except maybe in a comment on the table, but that would be error prone.
But personally I think you shouldn't want to keep track of versions of tables. After all, to get from a version 1 to a version 2, you may need to modify data as well, or other objects like triggers and procedures that use to new version of the table.
So in a way, it's better to version the entire database, so you can 'combine' multiple changes in one atomic version number.
There are different approaches to this, and different tools that can help you with that. I think Oracle even has some built-in feature, but with Oracle, that means that you will be charged gold bars if you use it, so I won't get into that, and just describe the two that I have tried:
Been there, done that: saving schema structure in Git
At some point we wanted to save our database changes in GitHub, where our other source is too.
We've been using Red Gate Source Control for Oracle (and Schema Compare, a similar tool), and have been looking into other similar tools as well. These tools use version control like Git to keep the latest structure of the database, and it can help you get your changes from your development database to scripts folder or VCS, and it can generate migration scripts for you.
Personally I'm not a big fan, because those tools and scripts focus only on the structure of the database (like you would with versioning individual tables). You'd still need to know how to get from version 1 to version 2, and sometimes only adding a column isn't enough; you need to migrate your data too. This isn't covered properly by tools like this.
In addition, I thought they were overall quite expensive for the work that they do, they don't work as easy as promised on the box, and you'd need different tools for different databases.
Working with migrations
A better solution would be to have migration script. You just make a script to get your database from version 1 to version 2, and another script to get it from version 2 to version 3. These migrations can be about table structure, object modifications, or even just data, it doesn't matter. All you need to do is remember which script was executed last, and execute all versions after that.
Executing migrations can be done by hand, or you can simply script it. But there are tools for this as well. One of them is Flyway, a free tool (paid pro support should you need it) that does exactly this. You can feed it SQL scripts from a folder, which are sorted and executed in order. Each script is a 'version'. Meta data about the process is stored in a separate table in your database. The whole process is described in more detail on Flyway's website.
The advantage of this tool is that it's really simple and flexible, because you just write the migration scripts yourself. All the tool does is execute them and keep track of it. And it can do it for all kinds of databases, so you can introduce the same flow for each database you have.
One way is to define a comment on the table:
comment on table your_table is 'some comment';
Then you can read that meta information using all_tab_comments table.
See
How to get table comments via SQL in Oracle?
For further reading, see:
https://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_4009.htm
I want to migrate a subset of customer data from one shared database environment to another shared database environment. I use hibernate and have quite a few ID and FK_ID columns which are auto generated from an oracle sequence.
I have a liquibase change log that I exported from jailer which has the customer specific data.
I want to be able to rewrite all of the sequence ID columns so that they don't clash with what's already in the target database.
I would like to avoid building something that my company has to manage, and would prefer to upstream this to liquibase.
Is anyone aware of anything within liquibase that might be a good place to start.
I would like to either do this on the liquidbase xml before passing it to 'update' command, or as part of the update command itself. Ideally as part of the update command itself.
I am aware that I would need to make liquibase aware of which columns are PK sequence columns and the related FK columns. The database structure does have this all well defined, so I should be able to read this into the update process.
Alternatively I had thought I could use the extraction model csv from jailer
Jailer - http://jailer.sourceforge.net/
I would suggest that for one-time data migrations like this, Liquibase is not the best tool. It is really better for schema management rather than data management. I think that an ETL tool such as Pentaho would be a better solution.
I actually managed to figure it out for myself with the command line 'update' command of liquibase by using a custom change exec listener.
1) I pushed a MR to liquibase to allow registration of a change exec listener
2) I implemented my own change exec listener that intercepts each insert statement and rewrites each FK and PK field to one that is not as yet allocated in the target database. I achieve this by using a oracle sequence. In order to avoid having to go back to the database each time for a new sequence, I implemented my own version of the hibernate sequence caching
https://github.com/liquibase/liquibase/pull/505
https://github.com/pellcorp/liquibase-extensions
This turned out to be quite a generic solution and in concert with some fixes upstreamed to jailer to improve the liquibase export support its a very viable and reusable solution.
Basic workflow is:
1) Export a subset of data from source db using jailer to liquibase xml
2) Run the liquibase update command, with the custom exec change listener against the target.
3) TODO Run the jailer export on the target db and compare with the original source data.
What is the difference between truncation, transaction and deletion database strategies when using Rspec? I can't find any resources explaining this. I read the Database Cleaner readme but it doesn't explain what each of these do.
Why do we have to use truncation strategy for Capybara? Do I have to clean up my database when testing or can I disable it. I dont understand why I should clean up my database after each test case, wouldn't it just slow down testing?
The database cleaning strategies refer to database terminology. I.e. those terms come from the (SQL) database world, so people generally familiar with database terminology will know what they mean.
The examples below refer to SQL definitions. DatabaseCleaner however supports other non-SQL types of databases too, but generally the definitions will be the same or similar.
Deletion
This means the database tables are cleaned using the SQL DELETE FROM statement. This is usually slower than truncation, but may have other advantages instead.
Truncation
This means the database tables are cleaned using the TRUNCATE TABLE statement. This will simply empty the table immediately, without deleting the table structure itself or deleting records individually.
Transaction
This means using BEGIN TRANSACTION statements coupled with ROLLBACK to roll back a sequence of previous database operations. Think of it as an "undo button" for databases. I would think this is the most frequently used cleaning method, and probably the fastest since changes need not be directly committed to the DB.
Example discussion: Rspec, Cucumber: best speed database clean strategy
Reason for truncation strategy with Capybara
The best explanation was found in the Capybara docs themselves:
# Transactional fixtures do not work with Selenium tests, because Capybara
# uses a separate server thread, which the transactions would be hidden
# from. We hence use DatabaseCleaner to truncate our test database.
Cleaning requirements
You do not necessarily have to clean your database after each test case. However you need to be aware of side effects this could have. I.e. if you create, modify, or delete some records in one step, will the other steps be affected by this?
Normally RSpec runs with transactional fixtures turned on, so you will never notice this when running RSpec - it will simply keep the database automatically clean for you:
https://www.relishapp.com/rspec/rspec-rails/v/2-10/docs/transactions
SQL Developer supports unit testing of DML but I've not found a way to create unit tests for DDL. What would be a good approach to this problem? The schema I'm starting with is small, less than a dozen tables with larger projects on the horizon. Google isn't returning much to the application of unit tests to DDL. Any ideas on an approach to testing DDL or other tools that exist for unit testing DDL?
What do you want to test about DDL? Either the table is created as defined or it is not.
What you could do is write a series of tests that queries the Data Dictionary to ensure the tables are present, have the columns with the sizes and datatype you want etc. This would be more of a schema verification script than unit tests however, and I am not sure how valuable it would be.
If you maintain a schema build script (or a series of migrations to add new objects to add objects to your schema), then if it applies without errors you know the schema has been created as it was defined.
Then if you have stored procedures, some of them will fail to compile if the schema is not 100% correct. Getting the procedures in cleanly would be another verification step for the schema.
Finally, the unit tests that you write to test the DML and stored procedures will verify that the correct data goes into the correct tables.
You might want some tests to ensure that a table can only accept certain values or columns can be unique etc (ie test the constraints are correct) but that would be down to standard unit tests too.
I am a big believer in writing unit tests for DB code, but I don't like SQL Developers GUI approach of doing it. Right now I am writing tests for an application, but I am coding the tests in Ruby and it seems to be working well. It will also be easily built into our build and automated test process.
Another alternative is UT_PLSQL which I have used before, however simply due to the nature of PLSQL is makes the tests very verbose, which is why I decided to use Ruby for my current project.
I know this is an older question, but I've recently been working to solve the same problem. I think it's useful to define tests for DDL prior to creating objects and then creating those objects to pass those tests.
I've done some of this using an assert "pattern" -- i.e., tdd.ddlunit.assert_tableexists(p_schema_name, p_table_name) which raises an exception if the table doesn't exist, and silently runs when it does.
Other assertions I've created are for things like making sure all varchar2 columns use character semantics instead of byte length semantics, and making sure all tables and columns are commented.
These get checked in to the code repository and can be run via continuous integration frameworks to make sure we have a valid database per what we expect.