I wanted to install CVAT for training an Object Detection AI using Docker. The install failed for some reason in the middle and it wasn't installed. But all the files were still occupying space on my machine. I tried reinstalling the CVAT and the files keep adding to the occupied space. How do I remove all of these files? I am using a MacBook Pro with MacOS Big Sur Beta 4.
Edit: https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/installation.md#mac-os-mojave
These are the commands I am running to install CVAT.
docker-compose build output: https://pastebin.com/7EkeQ289
docker-compose up -d output: https://pastebin.com/hF3GFDkX
docker exec -it cvat bash -ic 'python3 ~/manage.py createsuperuser output: https://pastebin.com/Mfh8CivL
If you are trying to remove the containers, attempt the following:
1. docker ps -a - lists all containers
2. docker stop [label or SHA of the containers you want to remove]
docker-compose down [YAML configuration file you targeted with docker-compose up] - this should stop all containers, teardown networks, etc. that docker-compose started with 'up'
docker container prune - removes all stopped containers
NOTE: If you have other stopped containers that you want to keep, do not run this, but remove them individually, as I suggested in the stricken-through step two above, or Konrad Botor's comment
https://docs.docker.com/compose/reference/down/
https://docs.docker.com/engine/reference/commandline/container_prune/
If you want to remove the images:
docker images
docker rmi [label or SHA] (RMI is the remove image command)
https://docs.docker.com/engine/reference/commandline/images/
https://docs.docker.com/engine/reference/commandline/rmi/
To speed up this process, analyze the YAML configuration file being targeted for your docker-compose build command, and/or reference the documentation for that specific project (CVAT) if available, to determine what containers (software) it is initializing (and how it is doing so, if necessary). It might help to paste its contents in the question.
Note: what is taking up space may be volumes which are not cleaned up properly by the docker build scripts for this project. See the following documentation on how to remove those:
https://docs.docker.com/engine/reference/commandline/volume_rm/
I might be missing some context, as I cannot access your pastebin links (behind a firewall at the moment).
Related
I've reviewed the documentation here:
https://docs.docker.com/docker-for-mac/install/#install-and-run-docker-for-mac
It doesn't say anything about boot2docker, although some other questions along these lines talk about this:
Mount volume to Docker image on OSX
So the question is – the Docker for Mac application provides File Sharing via Preferences -> File Sharing; how does one make use of these shared folders from the docker image (for example if one ssh's into the docker image)? When I say how, I don't mean "what are the use-cases", I mean "please show me an example of how to access a shared folder from the command line of the running container".
Ideally I'm trying to create a similar scenario to Vagrant's synched folders whereby I can edit files on my Host env, independently of the Docker Image but these are updated automatically to the Docker image on save.
UPDATE:
To be clear, the reason for asking this question is because I couldn't get the -v docker command to work. E.g.
docker run -v /Users/geoidesic/Documents/projects/arc/mysite/djangocms_demo:/home/djangocms/djangocms/djangocms_demo -d -p 8001:8000 --name test_shared_volumes bluszcz/djangocms
With the above command the container immediately stops, so if I run docker ps the list of running containers is empty.
However, if I run the container without the -v command, then it stays running as expected:
docker run -d -p 8001:8000 --name test_shared_volumes bluszcz/djangocms
Updated:
Well, if you want to share file/directory between host and container, you're gonna use Docker's bind-mount.
For example, if I want to share my host's /etc/resolv.conf to my container, I do the following:
docker run -v /etc/resolv.conf:/etc/resolv.conf <IMAGE>
In which the -v ... part tells the container to reuse host's /etc/resolve.conf. And whenever I edit this file, the changes will be immediately visible to the container.
In Linux, you can use this way to share almost any of your host files to containers. Unfortunately, this is not the case for Mac. As I mentioned in my old answer, by default you can only share /Users/, /Volumes/, /private/, and /tmp directly.
On my Mac, saying, I want to share the /data directory to a container. I run below command:
docker run -it --rm -v /data:/data busybox sh
Then it pops up an unhappy error:
docker: Error response from daemon: Mounts denied:
The path /data
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
So you see, this is where File Sharing comes up.
Then comes my answers to your questions:
File Sharing does not provide you a ready-to-use way to do the sharing as you have experienced in Vagrant;
To share file/folder between host and container, use Dockers bind-mount.
Hope that helps.
Old answer:
File Sharing is used by Docker's bind-mount feature. By default, you can bind-mount files in /Users/, /Volumes/, /private/, and /tmp directly. For other paths, you need to add them to Preferences -> File Sharing first.
Use cases for bind-mount:
Persisting data generated by the running container, so that you can backup or migrate data.
Sharing data amount multiple running containers.
Share host configuration files to containers.
Share source code between host and containers, to make debugging easier.
Note: For cases #1 and #2, consider using volumes instead of bind-mount.
I downloaded an image of ubuntu os in my system and after about committing 3 images from it(every image has an incremental change from before image), I now have my final image. Now I want the Dockerfile of this final image so that I can include commands like starting service at the start of container.
I have used this command,
sudo docker commit --change='CMD service apache2 start' 172d6dc34471 server_startup
to make the apache service start when the container is run. This starts the service in the container but doesn't go inside the container. It just starts the apache and exits to my local environment.
I would like to know how to get the Dockerfile for my final image so that I can include startup commands.
I have also tried this dockerfile from image, but its not working. It just throws error that .sock file is missing. I have tried this with both the new image and the parent image that I first downloaded.
Any help is much appreciated. Thanks.
you can use
docker history --no-trunc your_image
it will show you (in reverse order) what has been done
It is less user friendly than dockerfile from image, but it is basically the same thing.
I have somewhere a Python script that does that cleanly, I will check and post it.
Edit: Just this should show a lot
docker history your_image | docker inspect $(awk ' NR>1 {print $1}')
As CMD you need to provide a no-daemon program that keeps in foreground. That is not accomplished by service.
You should try:
sudo docker commit --change='CMD apache2 -DFOREGROUND' 172d6dc34471 server_startup
Sorry for the late reply, I used the --change argument while committing the image to point to the custom script(which I added within the container with the list of commands to be run at the start) that I want to run when the image is run for the first time.
sudo docker commit --change='ENTRYPOINT ["/test.sh"]' containerId autostart
I didn't know much about the dockerfile that everyone is suggesting but it is also a very good solution.
I have some containers that hold sensitive information in its file system during their existence. I create those containers using a line such as:
docker run -it --rm someImage bash
Whenever I exit bash my container disappears along with all the sensitive files, which is precisely what I wanted. If I use a Docker container then I do not need to remember to delete those files. This solution is also better than putting files in /tmp since all files are removed right after I stopped using them.
The problem is when my physical machine dies for some reason. For instance, if I trip on the power cable. When this happens, my container goes to the stopped state and someone else can restart the container using
docker start myContainerId
Is there any way to improve the way I create my container to achieve what I want? One way to go would be to put something on the host machine's SystemV to erase those containers at startup, but I wonder if Docker does not have a better way to deal with that.
Just use docker rm
docker rm <containerid>
(it only works on stopped containers)
I'm trying to create my own docker image in a ubuntu-14 system.
My docker file is like the following:
FROM scratch
RUN /bin/bash -c 'echo "hello"'
I got the error message when I run docker build .:
exec: "/bin/sh": stat /bin/sh: no such file or directory
I guess it is because /bin/sh doesn't exist in the base image "scratch". How should I solve this problem?
Docker is basically a containerising tool that helps to build systems and bring them up and running in a flash without a lot of resource utilisation as compared to Virtual Machines.
A Docker container is basically a layered container. In case you happen to read a Dockerfile, each and every command in that file will lead to a creation of a new layer in container and the final layer is what your container actually is after all the commands in the Dockerfile has been executed.
The images available on the Dockerhub are specially optimised for this sort of environment and are very easy to setup and build. In case you are building a container right from scratch i.e. without any base image, then what you basically have is an empty container. An empty container does not understand what /bin/bash actually is and hence it won't work for you.
The Docker container does not use any specifics from your underlying OS. Multiple docker containers will make use of the same underlying kernel in an effective manner. That's it. Nothing else.
( There is however a concept of volumes wherein the container shares a specific volume on the local underlying system )
So in case you want to use /bin/bash, you need a base image which will setup the nitty gritties of this command for your container and then you can successfully execute it.
However, it is recommended that you use official Docker images for say Ubuntu and then install your custom stuff on top of it. The official images are right from the makers and are highly optimised for this environment.
Base image scratch does not use /bin/bash. So you should change to:
FROM ubuntu:14.04
RUN /bin/sh -c 'echo "hello"'
Trying to fix errors and debug problems with my application that is split over several containers, I frequently edit files in containers:
either I am totally lazy and install nano and edit directly in container or
I docker cp the file out of the container, edit it, copy it back and restart the container
Those are intermediate steps before coming to new content for container build, which takes a lot longer than doing the above (which of course is only intermediate/fiddling around).
Now I frequently break the starting program of the container, which in the breaking cases is either a node script or a python webserver script, both typically fail from syntax errors.
Is there any way to save those containers? Since they do not start, I cannot docker exec into them, and thus they are lost to me. I then go the rm/rmi/build/run route after fixing the offending file in the build input.
How can I either edit files in a stopped container, or cp them in or start a shell in a stopped container - anything that allows me to fix this container?
(It seems a bit like working on a remote computer and breaking the networking configuration - connection is lost "forever" this way and one has to use a fallback, if that exists.)
How to edit Docker container files from the host? looks relevant but is outdated.
I had a problem with a container which wouldn't start due to a bad config change I made.
I was able to copy the file out of the stopped container and edit it. something like:
docker cp docker_web_1:/etc/apache2/sites-enabled/apache2.conf .
(correct the file)
docker cp apache.conf docker_web_1:/etc/apache2/sites-enabled/apache2.conf
Answering my own question.. still hoping for a better answer from a more knowledgable person!!
There are 2 possibilities.
1) Editing file system on host directly. This is somewhat dangerous and has a chance of completely breaking the container, possibly other data depending on what goes wrong.
2) Changing the startup script to something that never fails like starting a bash, doing the fixes/edits and then changing the startup program again to the desired one (like node or whatever it was before).
More details:
1) Using
docker ps
to find the running containers or
docker ps -a
to find all containers (including stopped ones) and
docker inspect (containername)
look for the "Id", one of the first values.
This is the part that contains implementation detail and might change, be aware that you may lose your container this way.
Go to
/var/lib/docker/aufs/diff/9bc343a9..(long container id)/
and there you will find all files that are changed towards the image the container is based upon. You can overwrite files, add or edit files.
Again, I would not recommend this.
2) As is described at https://stackoverflow.com/a/32353134/586754 you can find the configuration json config.json at a path like
/var/lib/docker/containers/9bc343a99..(long container id)/config.json
There you can change the args from e. g. "nodejs app.js" to "/bin/bash". Now restart the docker service and start the container (you should see that it now correctly starts up). You should use
docker start -i (containername)
to make sure it does not quit straight away. You can now work with the container and/or later attach with
docker exec -ti (containername) /bin/bash
Also, docker cp is rather useful for copying files that were edited outside of the container.
Also, one should only fall back to those measures if the container is more or less "lost" anyway, so any change would be an improvement.
You can edit container file-system directly, but I don't know if it is a good idea.
First you need to find the path of directory which is used as runtime root for container.
Run docker container inspect id/name.
Look for the key UpperDir in JSON output.
That is your directory.
If you are trying to restart an stopped container and need to alter the container because of misconfiguration but the container isn't starting you can do the following which works using the "docker cp" command (similar to previous suggestion). This procedure lets you remove files and do any other changes needed. With luck you can skip a lot of the steps below.
Use docker inspect to find entrypoint, (named Path in some versions)
Create a clone of the using docker run
Enter clone using docker exec -ti bash (if *nix container)
Locate entrypoint file location by looking though the clone to find
Copy the old entrypoint script using docker cp : ./
Modify or create a new entrypoint script for instance
#!/bin/bash
tail -f /etc/hosts
ensure the script has execution rights
Replace the old entrypoint using docker cp ./ :
start the old container using start
redo steps 6-9 until the starts
Fix issues in container
Restore entrypoint if needed and redo steps 6-9 as required
Remove clone if needed