Marathon - How to resolve a hanging deployment - mesos

I have an application that was running smoothly. I pushed a newly updated Docker image and restarted the app through Marathon's UI. The redeployment is now frozen, stuck on Progress 1/2.
First, what does Progress indicate and why is it 1/2? When I restart other applications on Marathon, they all have a progress ratio of 1/1.
Second, how can I debug this? When I run the container locally, I don't have any issues.

Related

Docker backend service is suddenly stopping while docker service stays running, why is this happening and how to resolve it?

I am running docker on a windows server 2019 VM. Docker has been working fine on that machine for a year now. But suddenly, last week docker engine was suddenly stopping and so all the containers were not running. Note that the server was not restarted since December (2 months ago).
When I log into the machine as administrator and click on the whale icon, everything restarts but I want to know what the issue is and how I can resolve it without having to log into the server on a daily basis and restart the service.
I set up windows monitoring for com.docker.backend and com.docker.service with elapsed time counter which shows how long the services have been up for.
The graphs clearly show that at one point docker backend dies and only restarts when I log in as Administrator to the server.
I also restart docker engine in debug mode hoping that that will give us more information but unfortunately all it shows is the logs when the docker engine is running and not a single piece of info on why it stopped.

How to manually remove Kuberntes cluster from docker

My Kubernetes cluster on docker for desktop on Mac is non responsive.
So I tried to reset Kubernetes as was suggested in
delete kubernetes cluster on docker-for-desktop OSX
The results are:
All Kubernetes resources are deleted
Kubernetes restart hangs
GUI to disable Kubernetes is grayed out and non responsive
I would like to avoid reseting docker so I can keep my image repository
How do I manually remove Kubernetes from docker VM?
You can try disabling Docker Kubernetes in the settings file. You can find settings file in path ~/Library/Group\ Containers/group.com.docker/settings.json. Edit kubernetesEnabled property to false.
"kubernetesEnabled" : false,
I have ended up in situation where k8s is partly deleted and was not able to start docker. Restarting and/or changing this setting helped and did not delete images. I was not able to reproduce the situation later.
Also make sure you are running latest version of Docker.
How about this?
‚docker rm -f $(docker ps -aq)‘
This deletes and removes all containers.
I can't give you a technical answer that immediately fixes your problem, and this text is too long for a comment... but as someone who also had the same issue (couldn't disable k8s without a factory reset in Docker for Mac), my recommendation is:
Is it really worth for you to keep the image repository? Consider, what's a container? A program. It's not a VM. Would you backup your ls, ssh, vim... binaries when you want to initialize your OS? No right? But this is the same, you should view the container like another binary.
Odds here are that if you mess with manual actions, you will end up with a docker daemon in an undesired state. So, IMO, just go ahead and purge the Docker for Mac and start over, it's not really a big deal.
If you have tons of own images, you can build them right away. If you have tons of downloaded images, consider this as a good thing to do some cleaning. Also, notice that images work in layers, so if your images are correctly build leveraging the use of layers, the building process will be quite fast.
To remove the kuberntes cluster from docker desktop you need to run: rm -rf ~/.kube

Stop and uninstall Xcode services

This XCSDeviceService process is always on 25% CPU or more.
I think it's a background service for detecting devices to test on for Xcode Server. I installed Xcode server once on my local machine to test/play around with it. Now I have a second machine for CI, so i don't need it on my local machine anymore. I uninstalled the Server app, but the Xcode services are still running all the time. Also the Xcode Server symbol in the status bar is still there:
Any ideas how to stop/uninstall the Xcode services?
I installed the server app again and setup Xcode server and turned it off again (I didn't turn Xcode off before uninstalling the server app the last time). That stops the XCSDeviceService only for this session. As soon as i log out and in again it restarts.
I simply had to remove the whole folder at:
/Library/Developer/XcodeServer
Or run this command
sudo rm -r /Library/Developer/XcodeServer
Then log out and in again.

Why does updating a stopped Azure deployment take so long?

Complete Azure deployment takes a while for a reason, but here's what I do.
I stop the deployment, then while it is in "stopped" state I click "upgrade" and choose the service package. The deployment gets into "upgrading" state and it takes up to three minutes to complete and after that it gets into "stopped" state. I have to click "start" and wait for several more minutes to start the deployment.
I don't get it. The deployment doesn't need to be stopped prior to upgrading - it is already stopped. The deployment isn't started after upgrading. What is taking three minutes then?
Why are you stopping the deployment? The point of the upgrade is to get zero-downtime. You should simply choose upgrade and either do a manual or automatic rolling upgrade across your upgrade domains. Only new deployments require a stop/delete.
I think the reason is because the automated upgrade is careful to make sure that you could start the Azure service again at any time during the upgrade - so it still takes its time upgrading the individual upgrade zones separately.

Problems with Windows EC2 snapshots

I am getting serious access problems every single time I take a Windows snapshot from the EC2 console. After taking the snapshot neither the original machine, nor the images snapshot, are avaialable. And by this I mean that there is no RDP, HTTP or HTTPS connectivity, all of which were accessible ports before the snapshot. There is nothing to explain why this error occurs, as the sys logs are either blank or seem to show a successful snapshot and machine launch. Note that I have also had scenarios where I reboot the machine and again NOTHING is available.
What am I doing wrong? These are the steps I take.
1) Launch a default Win2k8 with IIS7 image. This is my machine: ami-c5e40dac
2) Install .NET 4.0
3) Activate the database (turn on the service).
4) Install my application and the database. This include an HTTPS certificate (I think I read somewhere that Windows has a restart problem if a cert is in the machine store - WTF?)
5) Take a snapshot or reboot --- Bang, everything is dead!
Anyone come across such problems?
I had a similar problem earlier this week; it turned out that my instance was just taking an age to boot (1 hour +).
Is it possible that you had some pending windows updates that wanted to run on startup?

Resources