I am trying to create a docker image from Dockerfile on Windows 10. Being new to it, it crashed multiple times due to one or more syntax errors in the Dockerfile. I tried to clear all the images by using docker system prune --all. I got some disk space cleared up (If I am right, the system here means HDD rather than RAM?). Anyway, I see that Docker.Service seems to be using 6+ GB of memory.
My question is, is there a way to clear the memory in Docker.Service? Why is it using so much of memory when no image is being used? I know that it can be cleared by exiting Docker or force closing it.
Update
By the way, I am using Linux container, there is an option when right click on the docker icon from the tray.
Update 2
I tried all the commands from their documentation page - https://docs.docker.com/config/pruning/ - No effect.
Update 3
Doesn't seem to clear even when the image is created and saved. Looks like a bug?
Docker creates an image for each command in dockerfile, it creates image layers and stores it in the cache, So whenever your dockerfile execution is interrupted the image remains in the cache. Hence you see the memory consumption.
Try the following command which will remove all the images
docker rmi $(docker images -a -q)
Docker runs Linux containers in a VM on Windows. The 6G of ram is likely what you have assigned to that VM. Use docker stats to see the resource usage of containers running inside that VM.
Related
I've got a VM running Windows Nano and Docker containers. The Docker containers are all running ASP.NET Core 5 apps. I'm coming across this really weird bug where the VM is running out of memory, and the Task Manager, Process Explorer do not show what is taking up all this memory:
What I'm observing is that when I kill one of the containers, a lot of memory gets returned so it's definitely caused by a Docker container. The question is, how do I go about diagnosing this problem?
I've tried to take a memory dump of the app in the container I killed, but the dump is no larger than 1GB.
I'm using Docker for Mac and for saving disk space I want to get rid of some unused local images. As far as I know the local files are stored in ~/Library/Containers/com.docker.docker.
But even if I remove all images and containers from docker the folder keeps it's size of some GB and not a single byte is released. The only option is to remove the com.docker.docker folder completely but this makes only sense if you want to remove all the data and not only the unused images. What am I doing wrong?
sudo docker system prune -af
Just simple put the command given above.
It will delete all the image automatically.
To remove all unused images, use docker images prune.
There's also another way to remove all unused image (dangling): docker rmi $(docker images -aq --filter dangling=true)
To remove images you do no need, use docker rmi <image>. You may use multiple images, eg. docker rmi abcdef ghijkl, where abcdef and ghijkl are image hashes.
If this does not help immediately, try restarting your docker service (sudo service docker restart). In some cases images/containers are only marked as deleted and are not removed. But they are after restart.
What is going on here is that your docker for mac application is really running a virtual machine. That virtual machine has its own disk, which is what that qcow2 file is. That disk gets lazily allocated and won't take up all it space all at once. Once the space is used inside the vm, then the disk will grow on the host.
The disk image will grow until it reaches the size of the disk. Just because a file gets deleted inside the vm, that will not cause the disk image on the host to shrink again.
There is a possible solution to shrink your disk image: https://forums.docker.com/t/where-does-docker-keep-images-containers-so-i-can-better-track-my-disk-usage/8370/7
I have seen a few folks use this trick with some success.
List your images with:
docker images
Delete a single image with:
docker rmi image_id
I have a large (100gb+) database that I'm trying to run with the official postgres image.
I can't store the data in a docker volume because the ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 file in Docker for Mac has a size limitation of about 60gb.
I'm hesitant to mount a host directory as a volume because file access in mounted host directory volumes is much slower than regular volumes.
These are some useful links that go into more detail on these issues:
This discusses the size limitation of the Docker.qcow2 file
This also discusses the size limitation of the Docker.qcow2 file
This discusses the mounted host directory volume speed issue
This gives a nice description of how to replace the Docker.qcow2 file with a file that can grow larger
This discusses how the Docker.qcow2 file doesn't shrink as its contents are removed (this isn't directly related, but can further complicate the problem)
Do you all just eat the speed loss and mount a host directory? Do you manually create a qcow2 file that can grow larger with qemu (if you do this, do you need to maintain this file between upgrades)? Do you do something else to handle this issue?
The following script delete and re-create a new shrinked Docker.qcow2 file preserving the images passed as arguments.
https://blog.mrtrustor.net/post/clean-docker-for-mac/
Hope this helps.
For a situation like this, I would definitely recommend creating a qcow image that can grow larger.
It is a relatively straightforward process, and you get the performance benefits that are generally quite necessary when running a large database.
Docker for Mac 18.06 switches from the qcow2 file format to the raw file format, which improves speed and disk usage. The core issue still persists, in that Docker has a limited amount of space that it can use for all of its data. However, you can now set what that limit is within Preferences -> Disk -> Disk image size.
Docker for Mac version 17.12 was the first version to introduce the raw file format, but there were some bugs in the initial release that caused them to remove it as a feature temporarily. You can search that page for 'raw' to look back through the feature's history.
There's a great note here about how Docker for Mac reports its disk usage...
This will display the logical size:
ls -alh ~/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw
While this will display the physical size:
du -h ~/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw
In Docker is there a way to put Go shared libraries that are needed by containerized go apps in a read only shared memory area? The objective is to allow many containers to the execute the same code reducing memory requirements in the containers. I expect a side effect would be smaller container images.
Where would these Go shared libraries be in memory?
The image size is less about memory and more about disk space.
You could set those shared dynamic library in an image of their own, with a shared VOLUME path.
You then docker create a container based on that image: that is a data volume container.
Finally, you reuse that created container as many time as you want, with a --volumes-from=<yourCreatedContainer> for each go app docker run containers.
I know there are lots of docker experts around but I spent considerable time to find out some facts and figure about Docker's run time performance, but unfortunately i could not get any concrete answer. Let me start with telling you my System's configuration:
(a) Running CentOS 6.5 on a machine having 48GB RAM, 1TB Disc and 12 Core CPUs.
(b) I build up a Docker image which is having size almost 6.5GB
Below are questions if someone can answer for the benefit of readers:
(a) Now with the given configuration, question comes that how many containers I can run in parallel without break any functionality?
(b) Assume I have two Images each having size 3.5GB, then is it suggested to run multiple small size images or we get a good performance with big sized image?
(c) What is the best file systems option to use with Docker?
EDIT: more information
(d) Actually I'm trying to put many compilers inside a container and trying to give facility to users to compile their languages online. This tool is under development and will replace my existing website compileonlone.com. Things are going fine, I build up two images with few compilers in each. I'm able to run around 250 containers successfully and after that I start getting too many files opened. After 250 containers, my RAM is reaching somewhere 40GB and CPU utilization is around 50%,.
Main problem I'm facing is removal of the old containers. Because user will come and compile his code and then will go away, so I need to remove those container after certain period of time but when I'm trying to remove such stopped containers using docker rm -v, its slowing down main docker process and its almost hanging. I mean docker -d daemon which is listening at /var/run/docker.sock. Not sure if there is any other way around to clean these containers or I have a bug. Here is the detail of Docker:
# docker info
Containers: 1016
Images: 41
Storage Driver: devicemapper
Pool Name: docker-0:20-258-pool
Pool Blocksize: 64 Kb
Data file: /var/lib/docker/devicemapper/devicemapper/data
Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 17820.7 Mb
Data Space Total: 102400.0 Mb
Metadata Space Used: 102.4 Mb
Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.17.2-1.el6.elrepo.x86_64
Operating System: <unknown>
WARNING: No swap limit support
If someone can help me on how to delete old containers in fastest way then it will be great. Simple shell script and all are not working. I already have tried like
#docker rm -v $(docker ps -a |grep Exited | awk '{print $1}')
but its completely slowing down main docker process and its unable to create new containers while this removal process is running.
Thanks for your time taken to answer these questions, which will help me as well as many others in going ahead with Docker.
a): A container is like a process. This question is like asking "how many processes can I run in parallel". It is not answerable without knowing what the processes are doing. Please add this information to your question.
b) Both 3.5GB and 6.5GB are very large for a Docker image. Best practice is to put one application in one container: if you have an application that size, then great. If not, maybe you have put your application's data into the image. This is not a good idea because the layered filesystem is slower than a regular filesystem, and you won't be wanting any of the features of layering or snapshotting on your transactional data.
The documentation on managing data explains how to mount regular disk so it is accessible from your containers.
Edit, after more information was supplied
d) Using up RAM implies the containers are still running. If there is some way within the logic of your site to know when a container is no longer needed you can docker kill it, then docker rm to remove the disk storage. Or docker rm -f does those two operations in one.
After a lot R&D and discussing with many experts, I found a solution to delete containers with lightening speed. Its simple you have run your docker daemon with dm.blkdiscard=false option as follow.
docker -d --storage-opt dm.blkdiscard=false
By the way here is what I have developed. Here I need to create and delete containers with a high speed
http://codingground.tutorialspoint.com
Hope this will help many others.