I have an application with several code based migrations (EF5 code first) and an initializer that inherits from CreateDatabaseIfNotExists then I install that application on a production machine for the first time leting CodeFirst create the database from scratch. After that I add another code based migration and generate the migration script on my development machine and apply that script on the production machine. Then when I run my app on the production machine I'm getting errors. Those errors were generated because the code on my migrations are not being executed and there are code on those migrations that must be executed (for instance I have File Stream on my db).
Then, to solve that I changed my initializer to MigrateDatabaseToLatestVersion with AutomaticMigrationsEnabled = false. That solution solved the problem of executing all migrations code on first create but now I have this problem: VS2013 migrates the database automaticaly (if I add another migration) what I expect is to have an exception because AutomaticMigrationsEnabled = false in VS2012 that is happening BUT VS2013 IS MIGRATING AUTOMATICALLY.
Why is that happening? What I'm doing wrong?
Thanks
Automatic Migrations Enabled refers to the process of trying to automatically migrate your database to match your DbContext. For example, say you create your DbContext and run it once. Then you add a new table. With Automatic Migrations, you don't need to run Add-Migration, you can just run your app again and EF will automagically modify your database (note: it doesn't do this for all schema changes, but it makes a good effort. Sometimes you will still need to use Add-Migration).
Since your initializer is set to MigrateDatabaseToLatestVersion, any time the DbContext initialized it will run all of the migrations that are available. If there is any mismatch between your DbContext and the schema your migrations generate, you'll get an exception.
It sounds like the scenario you're expecting is when there is a mismatch between your DbContext and your database, you should get an exception. If this is what you want, you should not set your initializer to MigrateDatabaseToLatestVersion.
Related
Background
I am using PhpStorm to debug a test that creates a database entry. My goal is to set a breakpoint, then inspect the database manually.
I have confirmed this so far:
Step debugging is properly configured
Can connect via a forwarding port set up in docker-compose.yml (Fig. 1)
Laravel reports the entry exists in the database (Fig. 2)
Relevant Code
The star below indicates my breakpoint.
...
use Illuminate\Foundation\Testing\LazilyRefreshDatabase;
...
class ObfuscatedTestClass extends TestCase
{
use LazilyRefreshDatabase;
...
/** #test */
public function obfuscated_test_name() {
Queue::fake();
ObfuscatedModelName::factory()->create();
* Queue::assertPushed(SyncLeaseWithAccountingApp::class);
}
Hypotheses
Maybe I'm misunderstanding how databases are handled during these tests. I know Laravel has the ability to use database transactions to speed up tests, but I expect it to be modifying the database here when I'm using LazilyRefreshDatabase. Why else would I need to set up a database for testing?
Figures
Figure 1
Figure 2
I've been in the same boat all day, I also suspected transactions as a likely cause.
Then I came to realize all that was needed is to refresh the database connection when stopped at a breakpoint.
I'm not sure if this will be helpful to anyone but I thought I should answer because my situation is so similar to OP's - also running Docker and PhpStorm.
We're typically not in the habit of refreshing the db connection on a regular basis so it's easy to overlook.
EDIT:
To tag a little bit more onto my answer from earlier.. The data will persist if your test does not wipe it which can be useful. You can then tinker with that data by creating a file named ".env.testing" & setting APP_ENV=testing. Then run tinker like so: php artisan tinker --env=testing
I am not using outOfOrder.
I would like to be able to add a migration script, that would not be the latest, (e.g. to bugfix an existing script, without changing that script).
I would like the new script to be run, as part of the normal ordering, on databases that haven't been migrated yet.
Any databases that are up to date (e.g. manually repaired) should ignore the new script.
From the documentation:
OutOfOrder - Allows migrations to be run "out of order". If you
already have versions 1 and 3 applied, and now a version 2 is found,
it will be applied too instead of being ignored.
This suggests that the new script will be ignored, but I get the error:
ERROR: Validate failed: Detected resolved migration not applied to database
Will the new script only be ignored if the db baseline is ahead of it?
Is this the expected behaviour?
If so, I guess my solution here is either to:
Use outOfOrder, and complicate all my scripts to be idempotent.
Baseline my db after every migration.
There is a pull request for this that will be merged in time for Flyway 5.1.0: https://github.com/flyway/flyway/pull/1866
Until then you also have the option to disable validation by setting validateOnMigrate to false.
after upgrading Flyway Maven plugin from 2.3 to 3.0 I get:
[ERROR] Failed to execute goal
org.flywaydb:flyway-maven-plugin:3.0:migrate (default-cli) on project
xxx: org.flywaydb.core.api.FlywayException: Validate failed. Found
differences between applied migrations and available migrations:
Migration Checksum mismatch for migration
V003__data_feed_sources_locations.sql: DB=942424992,
Classpath=1117634405 -> [Help 1]
Got a similar error on some other project.
If I downgrade back to 2.3 the migration runs ok. Does this has something to do with different platform encoding for calculating checksums?
Any workaround, or better yet, proper solution?
Flyway 3.0 changed the default of validateOnMigrate to true.
This is however a good thing, as in the spirit of fail fast, errors are discovered sooner.
In your case some scripts did change since they were applied, which is what Flyway is reporting.
You have two options:
suppress the error by setting validateOnMigrate to false (2.3 default behavior)
invoke Flyway.repair() to reallign the checksums
Reference
Flyway Repair
To add to Axel Fontaine's answer:
I was able to use mvn flyway:repair but I had to point the flyway.locations config property at the folder that contains my db migration scripts. Otherwise I would get the message "Repair of metadata table xyz.schema_version not necessary. No failed migration detected." like other folks mentioned.
I used mvn -Dflyway.locations=filesystem:<project dir>/src/main/resources/db/migrations flyway:repair and I saw the checksum updated in the metadata table, fixing my problem.
I found the easiest way to resolve this issue was to literally update the checksum in the schema table to the value flyway expected. I knew for a fact that my migration files had not changed and that the current state of the database was what it needed to be. I also did not want to spend time reading documentation and messing around with Flyway.repair() or other methods that could potentially mess things up even more. A simple sql update resolved it right away
First, it looks for checksum changes. These changes occur if we update migration files which are already applied to a db instance.
FlywayException: Validate failed: Migration checksum mismatch for migration version 18.2.6
-> Applied to database : 90181454
-> Resolved locally : 717386176
repair() method would fix up checksum issue by updating the flyway_schema_history table with local checksum value.
However, it would neglect updated statements in same migration file. So, new changes in same file would be neglected as there is already an entry for version in flyway_schema_history table. setValidateOnMigrate() method has no effect in this scenario. We should follow incremental approach, schema changes should be supplied through new files.
The issue happen right after I changed the V1_2__books.sql ddl file, There should be a better way to force flyway to recognize the new changes!!!
I tried to run mvn flyway:repair but it did not work, I ended up changing the schema url in the application.properties file [datasource.flyway.url] to books2
I removed the below files as well (books is my old schema name )
~ #~:rm books.mv.db
~ #~:rm -r books.trace.db
datasource.flyway.url=jdbc:h2:file:~/books2
datasource.flyway.username=sa
datasource.flyway.password=
datasource.flyway.driver-class-name=org.h2.Driver
I ran the application and BINGO :)
Just wanted to add, that in order for the checksum to be updated by repair. Flyway has to have access to the directory where all the migrations are. Otherwise flyway just goes about it's business and outputs
"Repair of failed migration in metadata table xyz.schema_version not necessary. No failed migration detected."
The checksum needs to be updated using the flyway repair command (run the Flyway command as stated in “Upgrade procedure” but replace “migrate” with “repair”).
I recommend you do not intrude directly to database, sql scripts, etc. It can be dangerous
Example:
./flyway repare -user=root -password=changeme -url=jdbc:mysql://localhost/mypath -table=my_flyway_schema_version_table -locations=filesystem:/mypath_sql_scripts
if you are running on local db u can delete the flyway_schema_history table
Invoke Flyway.repair() directly from configurations.
Add below bean to your configuration class or create a new class with #Congiguration annotation and add the below code.
#Bean
public FlywayMigrationStrategy repairFlyway() {
return flyway -> {
// repair each script's checksum
flyway.repair();
// before new migrations are executed
flyway.migrate();
};
}
There is yet another solution. You can drop your migration from schema_version table.
I recently added a scheduler job and program to my development schema. When I tried to refresh the schema, I did a flyway clean, and then a flyway migrate.
I got the following error:
ERROR: Found non-empty schema "TESTDATA" without metadata table! Use init() or set initOnMigrate to true to initialize the metadata table.
When I dropped the job and program by hand, I was then able to run migrate again.
I've been using flyway for a while, and it's always been very straightforward - but I'm not sure how to convince it to properly clean my schema, now that I have an overnight batch job.
Note: I see the option -initOnMigrate, but this causes me two problems:
I have a lot of batch files which would be sensitive to trying to add another runline option.
I use flyway both to update existing schemas and to refresh schemas from scratch. If I need to modify the job or program, I could only include initOnMigrate (and have it bomb on the update), or not include it, and have it bomb on refresh (my current problem).
Thank you
You can work around this by implementing FlywayCallback.afterClean() and do the cleanup yourself.
Also, please file an issue in the issue tracker so we can fix this in time for 3.1.
Has anyone ever used Solrnet inside CLR Stored Procedures? I would really appreciate pointers to some good tutorials.
One problem I'm facing is that I can't find a way to include the SolrNet library as a reference to the VS (2008) Database (SQL Server) Project.
Update:
So it seems that when you have an SQL Server Project, and you want to add a reference to a library, it has to first exist in SQL Server itself, which makes sense. This is done by creating an assembly in SQL Server from the DLL itself with the following SQL:
CREATE ASSEMBLY SolrNet FROM 'C:\CLR_SP\SolrNet.dll'
WITH PERMISSION_SET = UNSAFE
(Note that UNSAFE might have some repercussions on the security of the database however it is ok for me for now)
However the SolrNet.dll requires other library dependencies such as Castle.Windsor.dll (which in itself requires System.Core.dll) and Ninject.dll. I found the required version of Castle.Windsor.dll (which is 2.5.1.0) and also System.Core.Dll (which was in the .Net folder of Windows), however I cannot find the required version of Ninject.dll (which should be 2.1.0.76). I have attempted to create assembly version 2.2 but, as expected, it did not do the job.
I searched for it on different repositories but could not find it. Does anyone know where I could find this version of DLL?
Update 2:
So after lots of searching over the net, I still didn't manage to find Ninject.dll v2.1.0.76. My next attempt was to use the next version of SolrNet (which is v0.4.0.2002). This version required Ninject.dll v2.2 which I had already found. So my current status is registering all other libraries in SQL Server which are dependencies of SolrNet.dll. I will leave this open to document my process just in case there will be someone having the same problem.
Update 3:
I have managed to register all required libraries (some of which I got from SolrNet source on GitHub). So now, SolrNet is registered as an assembly in SQL Server, and therefore I can reference it from the .NET SQL Server Project (for creating the CLR Stored Procedure). So I have written a very simple CLR SP which connects to SOLR and retrieves a piece of data. Code below:
[Microsoft.SqlServer.Server.SqlProcedure]
public static void PrintToday()
{
SqlPipe p;
p = SqlContext.Pipe;
p.Send("Helloooo");
// Open Solr instance
SolrNet.Startup.Init<ActiveProduct>("http://192.168.2.190:8983/solr");
// Get instance of ActiveProduct
ISolrOperations<ActiveProduct> operations = ActiveProduct.GetActiveProductSolrOperations();
// Prepare QueryOptions. This will be passed as a parameter into the query() method.
SolrNet.Commands.Parameters.QueryOptions qo = new QueryOptions();
qo.Start = 0;
qo.Rows = 20;
// Query Solr
SolrQueryResults<ActiveProduct> results = operations.Query(new SolrQueryByField("SearchDescription", "pants"), qo);
// Read results
String s = "Docs found: " + results.NumFound;
p.Send(s);
}
My next problem is that when I deploy and run the CLR SP, an error is popping stating that Solr is already registered in container. The exact output that I see in SQL Server Management Studio is the following:
Helloooo
Msg 6522, Level 16, State 1, Procedure PrintToday, Line 0
A .NET Framework error occurred during execution of user defined routine or aggregate 'PrintToday':
System.ApplicationException: Key 'SolrNet.Impl.SolrConnection.CLRStoredProcedures2.ActiveProduct.SolrNet.Impl.SolrConnection' already registered in container
System.ApplicationException:
at SolrNet.Utils.Container.Register(String key, Type serviceType, Converter`2 factory)
at SolrNet.Utils.Container.Register[T](String key, Converter`2 factory)
at SolrNet.Startup.Init[T](ISolrConnection connection)
at SolrNet.Startup.Init[T](String serverURL)
at StoredProcedures.PrintToday()
PrintToday is the name of the CLR StoredProcedure
CLRStoredProcedures2 is the name of the .NET SQL Server project and default namespace in VS 2008
ActiveProduct is the name of the document in Solr, and the cs class with Solr annotations
As can be seen from the output, the first Pipe.send("Hellooo") is doing its job therefore the SP works fine until there.
When I searched for the above error, I found out that it will show when one tries to register Solr instance twice in the same application. Now I don't see where I am registering the instance twice. Am I missing something here?
Note that the above cs function worked fine when executed in a cs console application developed on my machine. Another detail which might be important is that the SOLR Server is being hosted on my machine which is on the same network of my SQL Server 2005.
Update 4:
For starters, the error I mentioned above (in update 3) does not fire when the SP is executed the 1st time just after deployment (let's assume that the 1st time works for now since I have another error which I'm currently working on fixing), the error fires when the SP is executed again afterwards. So it seems that whatever SolrNet.Startup.Init<ActiveProduct>("http://192.168.2.190:8983/solr"); is doing (creating some sort of session that has to do with a container I think) when called from the SP it is not releasing the "session" therefore the 2nd time (and each time afterwards) the SP is executed, the error is fired. Is there a way to sort of stop the session or releasing from the container. What I can do as a workaround is to try - catch the SolrNet.Startup.Init part, however this is not clean.
Thanks.
I'll try to summarize the steps / requirements with explanations whenever possible:
For some reason SQL Server seems to trigger the "hidden" dependencies of the merged SolrNet. Usually you can just use the merged SolrNet (which includes all integration modules) and if you ignore the integration classes the dependencies won't trigger. But in this case the unmerged SolrNet is necessary.
SolrNet does HTTP requests to the Solr server, so the DLL must be registered with UNSAFE permissions in SQL-CLR.
SolrNet Initialization (Startup.Init) must happen only once per application, but you don't have a 'root' context in SQL-CLR to place this initialization. A workaround is to use a singleton or a Lazy type. SQL-CLR doesn't run .NET 4 yet, but you can backport the Lazy type or use FSharp.Core.dll's implementation (I blogged about this some time ago)