Can we update sqoop.last.value in sqoop through scripts in automated way. It currently loads the run time. I dont want to miss the updates b/w runtime and source system update time. so looking to see if we can update the last value. I know manually we can do it, but looking for an automated way.
Related
We are working on Joomla3.9 project, have different environments and are using git as vcs. So every developer works on is own branch. It would be nice to have a database compare function like in TYPO3 or Contao (see the database differences after updating the project and apply the database changes just by one click). Or like the laravel migration system.
Any developer should easily update his own lokal database after database changes where made due an extension update via backend or by another developer. And of course the staging or live system must be updated easily too. We don't want to execute sql-scripts with the changes in phpMyAdmin.
We have tried https://dbv.vizuina.com/ . This is not the 100% solution. Like there is no cli support to start the migration process by an update script on the server.
Does anyone have a solution or knows an extension that can solve this problem? Or can this be handled with core Joomla functions (maybe with a little adjustment)?
So far, I've seen three possibilities to execute modifications to one ore many extension tables
1: Use the extension - revision control in the schema table. So add a new sql-file with an increased version number compared to the version number in the schema-table for this extension. Increase also the version in the manifest.xml and zip the extension again.
Reinstall the extension via extension->manage->install. So the new sql-file with the increased version number will be executed.
2: like the point above, but install the extension via joomla update mechanism (update server).
3.: create a new sql-file in sql/folder of the extension. No version name is needed for the new file, just update.sql oder another filename. Execute this script in script.php in update()-method, after the extension is installed (in this case it's an update) again.
The third possibility might be interesting. It should be possible to trigger the update()-method with a cli command / function, so that the method can be triggered via a script on the server.
But how can I get the info, which update-scripts have already been executed? Let's say I have 3 update files in sql-folder. update-1.sql, update-2.sql and update-3.sql.
update-1.sql has already been executed. So I don't want to execute this sql-file again - only the other two.
The schema-table is only used with the first two options. Do I have the info somewhere or must I manage the infos which update-scripts have been executed myself?
The answer related to versioning database for extensions depends on whether these extensions are tightly coupled to the application or need to be reusable to other applications also.
The latter case normally means that each extension accesses its own custom tables, in which case you should keep separate versioning for the database than for the extensions.
App version history can be kept in a db_version table. Then an insert statement is added at the end of each update script (adding an incremental version number). e.g.
insert into db_version(version,author,description) values(003,'Verna.Collins', 'removing obsolete column');
Provided that you need to apply data migration on extensions also, you need to maintain a db_version_extensions table which keeps version history for each of the extensions separately. e.g.
'001' 'extension1','Mandy.Aguilar','initial version'
'002' 'extension1','Mandy.Aguilar','adding extra column'
'001' 'extension2','Edna.Potter','initial version'
'002' 'extension2','Elvira.Townsend','dropping unused table'
..etc
Each extension zip should keep initial creation script and all sql-update files(which should normally not interfere with the rest of the app tables).
After pull it will be relatively easy to execute all the scripts with filename version greater than the last version number written in the database. This should be done for the app and for each extension separately.
Now if the extensions are tightly coupled to the app, it means that they might be using/updating tables of the app. For extensions of this type, you can add the updates as part of the application updates. These extensions could even be developed at the same repo, and be kept as directories instead of zip files.
Not sure if joomla supports any tools for automating the process of performing incremental db updates, but a nice tool is flyway, with ports for command-line, maven and graddle. See: how does flyway work
With EF code-based migrations, after performing a "Get Latest Version" of the code, it isn't always obvious if you have migrations to run. You could look in the migrations folder to see if there are any new ones, or you can just wait until something breaks and perhaps it will become obvious that your DB is out of date. Neither of those options are appealing to me.
I don't want to enable automatic migrations, and I also prefer not to have migrations always run on application startup because I may not be ready for them yet. I could run update-database every time I get latest (and don't have pending DB changes of my own), but that would be unnecessary most of the time.
Is there a simple way to know that my DB is out of date, particularly after pulling down the latest code?
Ideally I'm envisioning some sort of build event that only happens after a get latest.
You have to compare your model with the __MigrationsHistory table.
A simple way to do that is using update-databse with the -script switch. In that way you are doing the compare, but possible changes will not take effect instead they are generated as SQL in a new file.
update-database -script
Depending on your IDE and source control system I'm sure you can trigger a script from an "pull" or "get latest" event that does your compare.
I have created database project in visual studio 2013 from existing database. Then I have done lot of changes in database project like modify stored procedures, post deployment script, table structure, etc . Now database project is ready to deploy. But I am worry if any script fails then How I can retain the original state though it build properly.
Please suggest that if any query fails then I want ROLLBACK the all changes that I have made in database project.
Firstly you need to trust your tools and either believe they will work or find other tools.
While you are building the trust I would add a create backup to the pre-deployment script or run a backup before you deploy then if anything goes wrong you can restore and figure out what went wrong.
As David said to roll-back, you would get the previously deployed dacpac and generate a new deployment script from that but fixing forward is almost always the correct thing to do rather than rolling back to a previous version.
ed
Have you been checking your changes into version control? If so, all you need to do is to revert back to the last known good version.
Or... simply work out why it's failing now and fix the root cause?
I used Db projects some time ago and as far as I remember the deploy script was wrapped in a transaction. It is possible to generate sql script without executing it. That setting was somewhere in DB project settings. You can take a look inside that script and make sure that it'll rollback on error.
Doing a backup would still be a recommended practice especially when you deploy to production.
when working on important scripts I developed a habit of always starting a transaction and commenting out the commit.
If you accidentally run it, it won't take effect. The commented out commit would only come out when the thing was done.
While this answer indicates that you CAN revert in source control (Assuming SSDT at this point) it would be nice to get a pointer to the exact process to do this. On a file by file basis the history works the same but how to revert the entire database at once isn't immediately obvious.
I'm working on creating a deploy script to migrate new development from our dev server to our uat server. Unfortunately, the devs that made the changes didn't script them out at they coded. The easiest way for me to approach this is to use SqlDeveloper's database diff functionality. It does a good job of highlighting the differences and creating a script that I can run on UAT. However, I've noticed that it doesn't take into account any dependencies. For example, a it will put a command to create a table below a table that references it in a foreign key constraint. Because the referenced table doesn't exist yet, the first table create command fails. I've seen this with views referencing packages, packages referencing packages, etc. Is there any easy to way either 1) force sql to export in a "smarter" order or 2) manually calculate the dependencies (ex: querying USER_DEPENDENCIES, etc) so that i can manually sort the file of create commands without resorting to trial and error? I guess we could consider purchasing a commercial product as long as it matched exactly what we are looking for.
Note: we will probably have to deploy to UAT multiple times in order to support testing by end-users. I am trying to automate this as much as possible so I don't have to manually recreate this script every single time!
Thanks!
I'm using the Database project in visual studio 2010 to generate a script to deploy my database (and it's changes). This works great.
Is there a way to have Visual Studio database project generate a rollback script as well as the deployment script.
I'm not looking for rolling back the transaction while deploying; but say I deploy it and my stored procedure has an overlooked performance issue that comes up a week later that requires a rollback to the previous version of the database.
Is there a way to generate the rollback script at build/deploy time that will undo whatever changes the deployment script made.
EDIT: If we ignore that I'm using a database project: What is a good way to have an upgrade and downgrade path for a database generated?
This generation needs to be part of an automated build process.
To create a rollback script While doing a schema compare using VS2010, It is as simple as swapping the db names specified in the source and target.
This way VS2010 would create a rollback script which would have drop statements against your stored proc.
I've not seen anything like that.
I think you need to reconsider this approach, as you'd still need to fix the stored proc in your database project, otherwise you'd just be re-deploying the "bad" version the next time you deploy. (I'm sure you're already aware of that, but it doesn't help to point out the obvious sometimes!)
If you need to restore an old version of the sproc to the server in the mean time, I would have thought that the easiest thing to do would be to get the previous version from source control and manually deploy that.
You could create a backup of the database before the release and then just restore from the backup if things go wrong. Obviously you'd also loose any data changes (either made as part of the release or subsequently) since the backup was taken.
Another idea I had was to create a snapshot before the release. The operation to create a snapshot is very light weight. I'm not sure you'd want to keep the snapshot for a week, but if the release went wrong then I think it would be quicker to restore from a snapshot than from a full backup. I would be interested to hear any comments people have on this idea.