As per this question: Sonar throwing error BadDatabaseVersion it is not possible to run two sonar instances using the same database. Everything I've read so far implies the only solution is to shut down both instances and only restart the one you want to keep. Is my only other option to run two sonar instances, to have another sonar database? This seems pretty costly, and it seems that the only thing holding back sonar from running another instance is sharing server.core.id
So I guess I have two questions:
1) Why is sonar built with this dependency?
2) Are there any other options to run two instances on the same db?
Indeed, SonarQube currently can't have 2 servers started on the same DB. This limitation (that is referenced in this JIRA ticket) has beeen here by design since the very beginning - to make sure that you can't start 2 servers having a different set of plugins but pointing to the same DB.
To answer your second question, there's no way to have 2 instances pointing to the same DB. But we've starting a big refactoring to eventually make it possible to have a cluster of SonarQube instances, so feel free to watch the SONAR-5391 ticket and vote for it.
Related
I am looking a this Amazon page - https://aws.amazon.com/rds/aurora/serverless/ and it has this quote:
You pay on a per-second basis for the database capacity you use when
the database is active, and migrate between standard and serverless
configurations with a few clicks in the AWS Management Console.
I have a few normal Aurora clusters and want to switch them to serverless. I have looked and looked and cannot find the "migrate with a few clicks" bit in the Amazon user interface. I made a new serverless cluster just fine and so I could do a stop, backup, and restore with a short outage - but If I can do this without an outage - that would be far superior.
So where are these "few clicks" - or perhaps you will tell me the "few clicks" means stop, backup, and restore. Either way I think a lot of folks could benefit from knowing what "few clicks" make this happen.
As a comment on #drchuck's approach - We've learned this the hard way that AWS Database Migration Service does a bad job at creating the schema in the target database. However - there's a simple workaround:
1) Run mysqldump --no-data to get the exact schema from the source database.
2) Execute the dump'd schema on the target database.
3) Within your DMS task, under target table preparation mode, choose "Truncate" instead of "Drop tables on target". (https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html)
With this in place, DMS doesn't create the schema on the target side, and things work pretty well (all existing data is loaded, and then ongoing changes are sync'd in near-real-time).
We've used this approach for minimal downtime cutovers a few times.
It took more than a while to figure out those few clicks.
I'm here initially as I too could not find them and yes I saw the exact quote on the AWS page you indicated saying that yes you could.
First you take a snapshot and then you restore it. In the process of restoring it you can select a serverless instance. (At least under SOME conditions. I do not think that a 5.7.12 (just confirmed actually) can be restored to a serverless configuration).
I suspect that 5.7.12 will happen in due time.
Right now the magic bullet is to start with a 5.6.10a version, take a snapshot and then restore that to a serveless instance.
For what it's worth after the long time:
Apparently Amazon Aurora Serverless is only compatible with MySQL 5.6 - this explains why 5.7 snapshots cannot be recovered.
So the two options are
downgrading the MySQL version to 5.6 first or
dumping and importing the data (after I read the other answers, I'd go for the second option).
Further reading:
https://aws.amazon.com/rds/aurora/serverless/?nc1=h_ls#How_to_Get_Started
When I did not get an answer in a few days, I did the conversion two ways with different results so I figured I would share my results here. I would still love to hear a better approach. (1) When I did the conversion using mysqldump and restore, with a short outage things were fine. (2) When I used AWS Database Migration Service it went pretty badly.
First, you have to get the binary log format as "ROW" and retention to 24 hours. That necessitated server restarts on my old clusters. Then when the data migration worked, I lost all my auto increments, then NULLness in my columns, the UNIQUE clauses and foreign keys in the new tables. Literally the only thing that migrated correctly was that the actual data and PRIMARY KEY indications. Also, I would recommend migrating one database at a time (i.e. schema) and don't try to migrate the mysql internal schemas. I said "migrate everything" and the migration tool tried to migrate the MySQL stuff - sheesh.
The one thing the AWS Database Migration Service did that was really cool was the migrate and monitor (made possible by the binary logging on the rows). You could watch it moving rows.
Just for the record, AWS amended the quoted documentation in mid-2022 by changing 'few clicks' to 'few steps'.🤣
You pay on a per-second basis for the database capacity that you use
when the database is active, and migrate between standard and
serverless configurations with a few steps in the Amazon Relational
Database Service (Amazon RDS) console.
Currently the documentation states that there are two (multi-step) methods that can be used to migrate from provisioned to serverless, and serverless to provisioned:
Snapshot restore.
Logical backup and restore.
Details here.
We are having separate databases for each tenant which is creating a lot of downtime when we are deploying changes on cloud. The steps(in brief) what we follow whenever we have to deploy the changes on cloud are:
Put down the client site.
Take a snapshot of the current RDS instance(in case anything goes south).
Run the migration scripts(Changes) on each tenant database on RDS instance.
If everything goes well, then we make the client site live again.
Now the problem is, we are having around 250 tenants as of now and the 3rd step which is running the update script is taking too much time which in turn increases the downtime. Any suggestions on how to improve this process or if we are suppose to do it in some other way. There is a clear lack of enterprise level expertise here on our end, so any help will be appreciated. Thanks!
Without knowing anything about your application, here are some things to think about:
If your application would still have some value when running in a 'read-only' mode, you could limit the actual downtime by doing the following.
Make sure all of your RDS databases have a read-replica.
Set your application into 'read-only' mode (i.e. thru some application code).
Let your read replica catchup with your master
promote your read-replica to a stand-alone DB.
Run your updates against this copy of the database.
redirect your application to the new master.
create a new read replica from this new master
delete/archive your old database.
You still have to do all the work, and it still takes a while to run, but the actual downtime for the user should be minimal.
I am in the process of setting up SQ for a large organization. I plan to have two separate systems one for update testing and rule development. The second would be the production system where real work occurs. I will be using SQL 2014 typically when I do that I use a SQL always On group to sync to a DR server in another datacenter. My question is with a SonarQube instance does it make sense to DR the application to that level. If my organization can wait for a period of time to stand up a new server in a DR event would that be possible with a proper backup of the DB? Further if there were no backups of the DB what would be lost with a fresh new SonarQube server besides setup/config time? Is there historical value of code scans that would be lost or would the next scan of the code base have us right back to where we were in terms of critical issues found etc.?
Thanks for your replies.
All the data is stored in the database so using DR on the database is a good idea. You should make backup of the database and restoring the database is also a good solution (note that you should do backup of installed plugins).
If you loose the database, you will also loose all the configuration (quality profiles, credentials, etc.) and the history of the analyzed projects.
So to restore a SonarQube instance, you have to :
Restore the database
Restore SonarQube or install the same version
Restore the plugins (${SONAR_HOME}/extensions/plugins)
During the first start, the ES files (${SONAR_HOME}/data/es) will be regenerated and you're instance will then be up and running.
If you have commercial plugins or if you are working with large SonarQube instance you may contact the sales team to have support on this setup.
Disclaimer : I'm working at SonarSource
I am just wondering how Flyway deals with the fact that several dynos can try to run a database migration several times in a clustered environment such as Heroku thereby causing a conflict?
It seems Flyway uses locking in order to deal with this issue.
To quote the documentation:
Can multiple nodes migrate in parallel? Yes! Flyway uses the locking
technology of your database to coordinate multiple nodes. This ensures
that even if even multiple instances of your application attempt to
migrate the database at the same time, it still works. Cluster
configurations are fully supported.
This question explains a bit more how the locking behaviour works. It appears to acquire a lock on the Flyway schema table: select * from dbschema.schema_version for update which might cause problems for longer running migrations as the innodb_lock_wait_timeout setting might cause a timeout, at least in the case of MySQL.
We currently have two SonarQube servers (v4.5.1) running on two separate Windows 2012 servers each with its own MS SQL database server. One is our Development server and the other is our production server. The idea being that we test out all rule changes on the development server first, once we are happy that they are correct we port them to the Production server.
When we first setup the two servers we simply took a backup of the Development server database and restored it on the Production server. At this point both systems were in sync.
We have recently made some modifications to the Development rules set, however when we tried the same approach to move these to the production server it did not work.
The production box seemed to remember the previous rule set. There seems to be a cache of the previous rules that we can't work out how to clear.
Before restarting SonarQube with the new DB in place we deleted the temp folder as that appears to keep a cached H2 database, but that did not solve the issue. We also tried starting it up and using the /setup url but this did not appear to work either.
Is there a way to completely reset the SonarQube server prior to restoring the database so that it has no knowledge of the previous rule set?
Alternatively is there a better way to export and re-import the entire rule set between two servers?
We looked at exporting the rule profile, but this did not appear to contain the full detail of the rules.
Thanks
Pete
For the moment, this is not possible to fully synchronize rules and quality profiles between 2 servers because of SONAR-5366. You can watch and vote for this ticket.
Concerning the cache that you seem to have, this is probably the E/S indexes which are located in <install_dir>/data/es folder. What you can do is:
stop you server
fully delete the <install_dir>/data folder
restart the server: your rules should be in sync with the DB