Hi I am trying to deploy my application with zero downtime. My app is quite frequent with database ddl changes. What are all the possible ways to achieve it with zero transaction failure in the app. Though we can use kubernetes to achieve zero downtime of the application, I don't want any failures in service request happening at the time of deployment due to database change like dropping the columns, dropping the table and changing the datatype
TechStack
Kubernetes - Deployment
Spring boot Java -app
Oracle -database
This has nothing to do with Kubernetes. You will have the same problems or challenges when you install your application on bare metal servers, on VMs or on plain Docker. Have a look at https://spring.io/blog/2016/05/31/zero-downtime-deployment-with-a-database this describes the problem pretty good.
Related
I have a project that I'm working on, in which I have found myself facing quite an interesting debacle that I'm not sure which is the most efficient way to solve. I have this Spring Application that is randomly pulling some data from a PostgreSQL database and sending it to clients. On the other hand, I have a gitlab repo in which multiple individuals are submitting data which is manually entered every once in a while into the Postgres DB for the aforementioned purpose. What would be the best way to automate this?
Cheers
I have provisioned postgres on my heroku app and also installed postgres locally to maintain parity (as the documentation recommends) with the online database but I'm also not understanding how this will work. Am I supposed to be accessing a local copy of a database when running on my own computer (while building and before deploying) and then using heroku's separate postgres database once it is deployed? If it is parity, shouldn't they both be using the heroku postgres database?
In other words, will my local app (during production) and heroku app (deployed and live) be using the same online postgres database?
Thanks.
Am I supposed to be accessing a local copy of a database when running on my own computer (while building and before deploying) and then using heroku's separate postgres database once it is deployed?
Yes, that's exactly it. Without seeing what bit of documentation you're referencing it's hard to say what they mean but perhaps there's another way to explain it.
In your local development environment, you may find that you need to test database schema changes (this is just one example, there are many). If you only had the one heroku postgres database you'd be forced to test these changes in production, which might result in poor usability for your users and that doesn't even account for the possibility of making a mistake and accidentally destroying your production data. There are a number of other shortcomings and challenges with this single database configuration.
For these reasons and more, it's best to keep your production data completely separated from your development/staging/test environment by creating a local/staging database. You might reasonably ask, "What about the data? I need data to test!". There are many ways to put together your test database and which you choose will likely depend on your needs. A shortlist of possibilities:
Use a seed file to generate mock data in your db
Use a model factory (usually runs in conjunction with your testing framework)
Take a dump of your production database, anonymize and redact sensitive information and use that for local testing.
I am looking a this Amazon page - https://aws.amazon.com/rds/aurora/serverless/ and it has this quote:
You pay on a per-second basis for the database capacity you use when
the database is active, and migrate between standard and serverless
configurations with a few clicks in the AWS Management Console.
I have a few normal Aurora clusters and want to switch them to serverless. I have looked and looked and cannot find the "migrate with a few clicks" bit in the Amazon user interface. I made a new serverless cluster just fine and so I could do a stop, backup, and restore with a short outage - but If I can do this without an outage - that would be far superior.
So where are these "few clicks" - or perhaps you will tell me the "few clicks" means stop, backup, and restore. Either way I think a lot of folks could benefit from knowing what "few clicks" make this happen.
As a comment on #drchuck's approach - We've learned this the hard way that AWS Database Migration Service does a bad job at creating the schema in the target database. However - there's a simple workaround:
1) Run mysqldump --no-data to get the exact schema from the source database.
2) Execute the dump'd schema on the target database.
3) Within your DMS task, under target table preparation mode, choose "Truncate" instead of "Drop tables on target". (https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html)
With this in place, DMS doesn't create the schema on the target side, and things work pretty well (all existing data is loaded, and then ongoing changes are sync'd in near-real-time).
We've used this approach for minimal downtime cutovers a few times.
It took more than a while to figure out those few clicks.
I'm here initially as I too could not find them and yes I saw the exact quote on the AWS page you indicated saying that yes you could.
First you take a snapshot and then you restore it. In the process of restoring it you can select a serverless instance. (At least under SOME conditions. I do not think that a 5.7.12 (just confirmed actually) can be restored to a serverless configuration).
I suspect that 5.7.12 will happen in due time.
Right now the magic bullet is to start with a 5.6.10a version, take a snapshot and then restore that to a serveless instance.
For what it's worth after the long time:
Apparently Amazon Aurora Serverless is only compatible with MySQL 5.6 - this explains why 5.7 snapshots cannot be recovered.
So the two options are
downgrading the MySQL version to 5.6 first or
dumping and importing the data (after I read the other answers, I'd go for the second option).
Further reading:
https://aws.amazon.com/rds/aurora/serverless/?nc1=h_ls#How_to_Get_Started
When I did not get an answer in a few days, I did the conversion two ways with different results so I figured I would share my results here. I would still love to hear a better approach. (1) When I did the conversion using mysqldump and restore, with a short outage things were fine. (2) When I used AWS Database Migration Service it went pretty badly.
First, you have to get the binary log format as "ROW" and retention to 24 hours. That necessitated server restarts on my old clusters. Then when the data migration worked, I lost all my auto increments, then NULLness in my columns, the UNIQUE clauses and foreign keys in the new tables. Literally the only thing that migrated correctly was that the actual data and PRIMARY KEY indications. Also, I would recommend migrating one database at a time (i.e. schema) and don't try to migrate the mysql internal schemas. I said "migrate everything" and the migration tool tried to migrate the MySQL stuff - sheesh.
The one thing the AWS Database Migration Service did that was really cool was the migrate and monitor (made possible by the binary logging on the rows). You could watch it moving rows.
Just for the record, AWS amended the quoted documentation in mid-2022 by changing 'few clicks' to 'few steps'.🤣
You pay on a per-second basis for the database capacity that you use
when the database is active, and migrate between standard and
serverless configurations with a few steps in the Amazon Relational
Database Service (Amazon RDS) console.
Currently the documentation states that there are two (multi-step) methods that can be used to migrate from provisioned to serverless, and serverless to provisioned:
Snapshot restore.
Logical backup and restore.
Details here.
We are having separate databases for each tenant which is creating a lot of downtime when we are deploying changes on cloud. The steps(in brief) what we follow whenever we have to deploy the changes on cloud are:
Put down the client site.
Take a snapshot of the current RDS instance(in case anything goes south).
Run the migration scripts(Changes) on each tenant database on RDS instance.
If everything goes well, then we make the client site live again.
Now the problem is, we are having around 250 tenants as of now and the 3rd step which is running the update script is taking too much time which in turn increases the downtime. Any suggestions on how to improve this process or if we are suppose to do it in some other way. There is a clear lack of enterprise level expertise here on our end, so any help will be appreciated. Thanks!
Without knowing anything about your application, here are some things to think about:
If your application would still have some value when running in a 'read-only' mode, you could limit the actual downtime by doing the following.
Make sure all of your RDS databases have a read-replica.
Set your application into 'read-only' mode (i.e. thru some application code).
Let your read replica catchup with your master
promote your read-replica to a stand-alone DB.
Run your updates against this copy of the database.
redirect your application to the new master.
create a new read replica from this new master
delete/archive your old database.
You still have to do all the work, and it still takes a while to run, but the actual downtime for the user should be minimal.
I have tried all the basics of Kubernetes and if you want to update your application all you can use kubectl rolling-update to update the pods one by one without downtime. Now, I have read the kubernetes documentation again and I have found a new feature called Deployment on version v1beta1. I am confused since I there is a line on the Deployment docs:
Next time we want to update pods, we can just update the deployment again.
Isn't this the role for rolling-update? Any inputs would be very useful.
Deployment is an Object that lets you define a declarative deploy.
It encapsulates
DeploymentStatus object, that is in charge of managing the number of replicas and its state.
DeploymentSpec object, which holds number of replicas, templateSpec , Selectors, and some other data that deal with deployment behaviour.
You can get a glimpse of actual code here:
https://github.com/kubernetes/kubernetes/blob/5516b8684f69bbe9f4688b892194864c6b6d7c08/pkg/apis/extensions/v1beta1/types.go#L223-L253
You will mostly use Deployments to deploy services/applications, in a declarative manner.
If you want to modify your deployment, update the yaml/json you used without changing the metadata.
In contrast, kubectl rolling-update isn't declarative, no yaml/json involved, and needs an existing replication controller.
I have been testing rolling update of a service using both replication controller and declarative deployment objects. I found using rc there appears to be no downtime from a client perspective. But when the Deployment is doing a rolling update, the client gets some errors for a while until the update stabilizes.
This is with kubernetes 1.2.1
The main difference is that "kubectl rolling-update" is client-driven rolling update, whereas the Deployment object gives you server-side rolling update.