I am trying to run liquibase update on oracle database in command line and it couldn't able to identify the already executed change sets and it tries to start update from the beginning of the change log file
java -jar liquibase.jar --driver=oracle.jdbc.driver.OracleDriver --classpath=ojdbc14-11.2.0.3.0.jar --changeLogFile=ParentDBChangeLog.xml --url="jdbc:oracle:thin:#172.25.XX.XXX:1521:ora11g" --username=xxxx --password=xxxxx update
It is getting failed from the first change set which tries to execute and the error which I got is
Error: java.sql.SQLSyntaxErrorException: ORA-00955: name is already used by an existing object
How I can resolve this problem
You should check that the database still contains the DATABASECHANGELOG and DATABASECHANGELOGLOCK tables and that they are appropriately populated (DATABASECHANGELOG should have one row of data for each of the changesets that has been applied to the database, and the ID, AUTHOR, and FILENAME columns should match the values in your changelog.xml).
I have seen many instances of well-meaning DBAs seeing those tables, not recognizing what they are, and removing them. I have also seen instances where someone was starting to introduce Liquibase but not everyone knows about that, and continues to use whatever process is alrady being used to manage schema change, and that process removes or alters those tables.
Related
I have an old Liquibase .xml file for adding an index to two columns. It is already in the DATABASECHANGELOG table and in Production, since years. But now i updated the H2 Database for my integration tests and they fail because of "article_id ". There is a blank space in the column name.
<createIndex tableName="order_journal" indexName="IDX_ArticleId_Customer">
<column name="article_id "/>
<column name="customer_id"/>
</createIndex>
My datasource configuration:
I removed the blankspace and the tests worked. Of course the application doesnt start because i edited an already commited file in the changelock.
What is the common way to edit an old Liquibase file or is there an approach for the H2 database?
There are a couple of options, which one suites you best depends on conventions in your project.
preconditions do not affect checksums: you may add precondition to the existing changeset, which prevents execution on test database, and write correct changeset for test database with opposite precondition (that is not clear how to properly distinguish test and prod databases though) - check Prevent error when dropping not existing sequences, creating existing users for example
you may specify <validCheckSum> for modified changeset - seems to be exactly your case
you may create another changeset for test database and mark both changesets as failOnError="false" - if changeset contains only createIndex change that seems to be safe, this scenario is also ~described in liquibase blog and Prevent error when dropping not existing sequences, creating existing users
you may specify <validCheckSum>ANY</validCheckSum> - in that case you do not need to figure out what previous checksum was, however it seems not to be safe.
I am using H2 Database to test my SpringBoot application. I do not use the file to store the data. instead I just use the In Memory datatabase. in the properties file, my JdbcUrl look like below:
spring.datasource.url=jdbc:h2:mem:;MODE=MSSQLServer;INIT=runscript from 'classpath:/schema.sql'\\;runscript from 'classpath:/data.sql'
Now When I run the tests, I have the following test scenario
Add Some Entities in a table (This adds some rows in a table)
search those entities by some criteria
Do the assertion
Now, sometime this runs successfully, but sometimes what happens is, the search query returns empty list, which causes the test to be failed.
I tried to add print statements just to check whether my entities are getting inserted properly, so in the insert function. after each insertion, I run the below query
SELECT * FROM tableName;
Which returns correct list. means each insertion is inserting in the table correctly. Now in the search function, before running the actual search query, I run the same query again
SELECT * from tableName;
And Surprisingly this is returning empty also, which means there is no data in the table. Please suggest what I check for?
Pretty sure #Evgenij Ryazanov's comment is correct here.
Closing the last connection to a database closes the database.
When using in-memory databases this means the content is lost.
After step 1) Add Some Entities in a table - is the connection closing?
If so to keep the database open, add ;DB_CLOSE_DELAY=-1 to the database URL.
e.g.
spring.datasource.url=jdbc:h2:mem:;DB_CLOSE_DELAY=-1;MODE=MSSQLServer;INIT=runscript from 'classpath:/schema.sql'\\;runscript from 'classpath:/data.sql'
Note, this can create a memory leak!
see: http://www.h2database.com/html/features.html#in_memory_databases
I have a basic spring batch app which is trying to load the data from a csv file to mysql. the program does load the file into db during the first run. However when I accidently re-run the job/app again, it had thrown the primary key violation (for the right reasons).
What is the best way to avoid reloading the data that is present on the target system? when the batch job is scheduled, if for any good reason, the source file has not changed since the previous run, I want to see 0 record processed message rather than a primary key violation error. hope it makes sense.
more information:
Thanks. I have probably not understood the answer. Let me explain my requirement in a better way. I have a file contains the data from external data source (say new hire data) with a fixed name of hire.csv. The file should be updated with the delta changes for every run. As there is a possibility of manual error of not removing all loaded rows, some new hires from previous run would also be present on current run. Is there a mechanism available within itemreader or itemprocessor to skip those records that are already present on the target db? I can do "insert into tb where not in (select from tb)" but this run for every row which I dont want to use. Hope it is clear now. thanks again.
However when I accidently re-run the job/app again, it had thrown the primary key violation (for the right reasons). What is the best way to avoid reloading the data that is present on the target system?
The file you are ingesting should be a (identifying) job parameter. This way, when the first run succeeds, the job instance is complete and cannot be run again. This is by design in Spring Batch for this very use case: preventing accidental job execution twice by error.
Edit: Add further options based on comments
If deleting the file is an option, then you can use a job listener or a final step to delete the file after ingesting it. With this option, you need to add a second identifying paramter (since the file name is always hire.csv) to make sure you have a different job instance for each run. This option does not require having a different file name for each run.
If the file can be renamed to hire-${timestamp}.csv and will be unique, then deleting the file after ingesting it and using a single job parameter with the filename is enough
Side note: I have seen people using a business key to identify records in the input file and using an item processor to query the database and filter items that have been already ingested. This works for small datasets but performs poorly with large datasets due to the additional query for each item.
I want to migrate a subset of customer data from one shared database environment to another shared database environment. I use hibernate and have quite a few ID and FK_ID columns which are auto generated from an oracle sequence.
I have a liquibase change log that I exported from jailer which has the customer specific data.
I want to be able to rewrite all of the sequence ID columns so that they don't clash with what's already in the target database.
I would like to avoid building something that my company has to manage, and would prefer to upstream this to liquibase.
Is anyone aware of anything within liquibase that might be a good place to start.
I would like to either do this on the liquidbase xml before passing it to 'update' command, or as part of the update command itself. Ideally as part of the update command itself.
I am aware that I would need to make liquibase aware of which columns are PK sequence columns and the related FK columns. The database structure does have this all well defined, so I should be able to read this into the update process.
Alternatively I had thought I could use the extraction model csv from jailer
Jailer - http://jailer.sourceforge.net/
I would suggest that for one-time data migrations like this, Liquibase is not the best tool. It is really better for schema management rather than data management. I think that an ETL tool such as Pentaho would be a better solution.
I actually managed to figure it out for myself with the command line 'update' command of liquibase by using a custom change exec listener.
1) I pushed a MR to liquibase to allow registration of a change exec listener
2) I implemented my own change exec listener that intercepts each insert statement and rewrites each FK and PK field to one that is not as yet allocated in the target database. I achieve this by using a oracle sequence. In order to avoid having to go back to the database each time for a new sequence, I implemented my own version of the hibernate sequence caching
https://github.com/liquibase/liquibase/pull/505
https://github.com/pellcorp/liquibase-extensions
This turned out to be quite a generic solution and in concert with some fixes upstreamed to jailer to improve the liquibase export support its a very viable and reusable solution.
Basic workflow is:
1) Export a subset of data from source db using jailer to liquibase xml
2) Run the liquibase update command, with the custom exec change listener against the target.
3) TODO Run the jailer export on the target db and compare with the original source data.
I use Entity framework 5 code first with enabled migrations. I made many changes to my model classes and now I have too much migrations classes because after every change I updated the database.
Now I want to merge all my updates to get one "initial class" or 2 so that I could run the update-database command only once if I have to create my database again.
Is that possible with no code (too heavy), I mean with a command for instance ?
Thanks.
The solution is based on whether you want to keep existing data in databases (if you have production databases this is definitely a must) or you can simply drop your database.
I. Cuurrent database can be droped
First you need to delete all migration steps and delete your current database then run the command
add-migration Initial
This way you will have only one migration step instead of a lot.
II. Data must be kept
First create a backup of the current database (that is set as the default database in your solution), then drop the database so when you run the add-migration command the Entity Framework will think that no migrations were applied yet.
After this do the steps as described in the first part and then you will have only one migration step named Initial. After this run the
update-database
command which will create a database matching your current model however only with one row in the __MigrationHistory table. Save this row from the __MigrationHistory table.
Now you can restore the database you've just backed up, delete all rows in the __MigrationHistory table and insert that one row you have saved before. After this the Entity Framework will correctly think that the database and your model is up-to-date and that this was only achieved by running the migration step initial and this way you can keep all your data.
Of course for doing this to multiple databases you only need to do these steps once and for the other databases you just need to delete the current rows in the __MigrationHistory table and insert the new row for the Initial migration step.