I'm planning to run a .sh script after some containers are up to make some REST requests.
Created a job to wait for these initial containers, filled initContainers tag in .yaml.
First, i thought creating a container with using a Linux distro as a base image. Then it didn't seem quite right.
Isn't that waste of resources ? What is the best practice for this situation ?
Thanks in advance.
You can use any kind of image as base image. The only resource you're consuming (not wasting) is disk space.
Kubernetes will pull the image and that will consume disk space on the node. As soon as the init container has done it's work, it will be stopped and won't use any more RAM or CPU which are the real precious resources in the cloud or on bare metal.
If you're worried about the image size, you can also try to use the same image as the user container (the main container) but starting a different command from it.
This will make the node just pull on image and no additional space appart from your script will be consumed.
Another option could be to use a very small distro like alpine.
If you can write your initialization routine in go you can also use your go binary as image like discribed here.
Related
I want to instantiate different containers from the same image to serve different requests and process different data.
Once a request is received by Docker, it has to instantiate a container (C1) from image (I), to work on a related dataset file D1.
For the second request, Docker has to instantiate a container (C2) from image (I) as well, to work on dataset file D2.
And so on ...
Does Docker have a built-in facility to orchestrate this kind of business or I have to write my own service to receive requests and start the corresponding containers to serve them?
Kindly provide your guidance on what is the best way to do so.
I think what you're looking for is a serverless / function as a service framework. AFAIK Docker doesn't have anything like this built in. Try to take a look at OpenFaaS or Kubeless. Both are frameworks that should help you get started with implementing your case.
I am running 1000 docker containers sequentially. Each container instance runs a specific job. After the execution of the job, I kill the container to release resources and run another job within another instance and so on.
I would like to get the memory usage of each container. One value per container expressing the average memory usage.
How is it possible?
May be with prometheus, but I don't know how to use it
May be with prometheus, but I don't know how to use it
Prometheus and Cadvisor is one possibility, I gave a talk earlier in the week with an example doing this using Swarm. See http://www.slideshare.net/brianbrazil/prometheus-and-docker-docker-galway-november-2015
The metrics you're interested in will then be available with the cadvisor job label.
As per your statement, you are running multiple containers and want to scrape each containers memory usage. This can be done with prom query.
Container_memory_in_bytes will give you the complete information about how much memory is used by each container in real time.
Please find the link for setting up the complete monitoring infrastructure. enter link description here
With Promquery you can achieve this.
Hope this answered your question.
I'd like to use a GCE server and VNC to run some desktop programs I've written. However, I'd like to not have the GCE server running all the time (to save money) and I'd like to have the setup of my VNC desktop persisted (to save time). It seems like the easiest way to accomplish this is to somehow hibernate the GCE instance. Is there a way to do this either from a script or from the command line?
The only currently supported technique that I'm aware of would be to use a bootable persistent disk, which outlives the virtual machine so you could delete and restart the VM with moderately quick (under 30s) startup time but I don't think that would meet your needs because unless your app has built-in, robust checkpoint/resume logic, you might not be able to pick up where you left off on the desktop.
Another way to approach this would be to use a lightweight container, like Docker, which has suspend/resume and checkpointing built into its design. You can't currently run Docker containers on Google Compute Engine but we're interested in exploring the potential of Docker on GCE so stay tuned.
Sounds like what you'd really like is a native suspend capability with near instantaneous resume and no or very low charge for VMs in the suspended state. I think that's an interesting idea - I've submittted an internal feature request to the Compute Engine engineering team for future consideration.
Compute Engine has alpha support for suspending virtual machines. A quote from that documentation:
gcloud alpha compute instances suspend is used to suspend a Google Compute Engine virtual machine. Suspending a VM is the equivalent of sleep or standby mode: the guest receives an ACPI S3 suspend signal, after which all VM state is saved to temporary storage. An instance can only be suspended while it is in the RUNNING state. A suspended instance will be put in SUSPENDED state.
The command is not yet available in the Compute Engine web UI but can be invoked from the command line like:
gcloud alpha compute instances suspend my-vm
I've browsed through other questions on SO concerning backing up EC2, and it has provided me with a good basis, but I'm still a bit confused as to how I should approach my solution and develop a contingency plan. Most questions are fairly specific, but I have a pretty vanilla setup, and I think this information will be beneficial to future users. Let me provide my basic setup:
Basic small instance
Pushing files to S3
Running MongoDB
Running nginx
Now, due to the ephemeral nature of EC2, it's apparent that I need to bind my EC2 instance to EBS to ensure persistent storage. The reason I'm attempting to develop a contingency plan is that I'm worried my instance may disappear at any time (due to outages, etc). If my instance were to disappear, I'm concerned that I would have to spin up a new instance and reinstall all my applications before getting everything up an running again. A few questions:
How do I backup my instance to ensure that if it were to disappear, I could quickly bring it back up (preferably without having to reinstall all previous software)? I don't need a series of backups, just the previous days (or weeks) backup to ensure that a previous working version exists that can be started quickly.
If I use EBS instead of instance storage, it essentially acts in place of my hard drive right? So if I have MongoDB installed, I'm assuming the database it is writing to cab placed on EBS?
If I go with the small instance with 160 GB of storage and I use EBS, would I need to allocate 160 GB of EBS out of the gate, or is the 160 GB for instance storage only?
So, in summary, I would like a solution (either manual or automatic) that can create a snapshot of my EC2 instance to ensure that if it were to disappear, it could be reconstructed without having to spend the time to manually reconstruct everything.
In an ideal world, if my instance were to disappear, I could spin up an version of my instance with everything intact (to a point where things were backed up). Any resources or suggestions? Thanks in advance.
OK, here we go:
For backups:
Create your instance from one of the stock AWS images. Make sure it is an EBS-backed VM - depending on the size of the VM you pick, you'll get a volume assigned with 'n' GB of space, attached as the boot volume (/dev/sda1).
Configure the VM with whatever software you need, apply patches, tune for disk fragmentation, CPU consumption (task priorities, etc), and any other configuration you need that makes the VM tailored to your requirements.
Stop the VM and take a snapshot of the EBS volume, then restart it (re-assigning the Elastic IP is there is one). This is your backup snapshot - repeat as desired at whatever frequency you like. Remember to stop the VM when you snap it to prevent the OS writing to the volume whilst you're taking a copy of it.
For recovery:
Your VM will fail, eventually. You'll break something and render it damaged or inoperative, or the hardware it's running on will suffer a fault. It will happen.
When it does, terminate it (if it didn't self-terminate) and spin-up a new VM of the same type from the AWS stock list. Wait until it shows as 'Running', and then stop it.
Detach its EBS volume and delete it.
Create a new EBS volume from whichever backup snapshot you last created, and attach that new volume to the VM as /dev/sda1.
Start the VM and assign your EIP if appropriate.
About EBS storage:
It's a chunk of storage space. If you format it to look like a standard disk, you can use it exactly as you would a physical disk. Install stuff on it, point software at it for use as storage space, whatever.
you have two options: (BUT NOT EXACTLY AS YOU WANT ;( )
1- Have an 'external' EBS attached into your EC2 instance, and manually (you can do automatically through cronjobs), make snapshots from it ! But it's not what you want, why ? If your EC2 instance disappears, you will need to re-create all your environment again and re-attach your EBS... So it's a nice way of having backups of HUGE datas on your EC2, but your enviroment is destructed...
2- The best way, but not so perfect, is after you finished configuring your EC2, make a private AMI from it, so anytime you want you can launch more instances like that , from that AMI, so everything is cloned.... BUT the worst part of it, is that every time you change a configuration from the instance, you still need to make a new AMI, and every time you make a new AMI, you need to reboot your instance to grant data integrity on your new private AMI !
I recommend you to take a closer look into RESERVED EC2 instances, that have a better stability from normal ones. BUT YOU STILL CAN HAVE HARDWARE DISASTERS as normal instances too...
I plan to insert some initialization code into OnStart() method of my class derived from RoleEntryPoint. This code will make some permanent changes to the host machine, so in case it is run for the second time on the same machine it will have to detect those changes are already there and react appropriately and this will require some extra code on my part.
Is it possible OnStart() is run for the second time before the host machine is cleared? Do I need this code to be able to run for the second time on the same machine?
Is it possible OnStart() is run for
the second time before the host
machine is cleared?
Not sure how to interpret that.
As far as permanent changes go: Any installed software, registry changes, and other modifications should be repeated with every boot. If you're writing files to local (non-durable storage), you have a good chance of seeing those files next time you boot, but there's no guarantee. If you are storing something in Windows Azure Storage (blobs, tables, queues) or SQL Azure, then your storage changes will persist through a reboot.
Even if you were guaranteed that local changes would persist through a reboot, these changes wouldn't be seen on additional instances if you scaled out to more VMs.
I think the official answer is that the role instance will not run it's Job more than once in each boot cycle.
However, I've seen a few MSDN articles that recommend you make startup tasks idempotent - e.g. http://msdn.microsoft.com/en-us/library/hh127476.aspx - so probably best to add some simple checks to your code that would anticipate multiple executions.