I am trying to set up rocker/rstudio docker on a linux ubuntu 14.04.5 with a data volume so all my data is outside of the docker. I have looked at Manage data in containers for some some guidance.
sudo docker run -d -p 8787:8787 rocker/rstudio -v ~/data/
I get back the following error:
docker: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"exec: \\"-v\\": executable file not found in $PATH\"\n".
Your -v flag should appear before the name of the image your want to run. If you list it after the image name docker will interpret it as the command used to launch the container.
You shouldn't use ~ when referring to the container volume. A better approach would be to use an absolute path like /data
If you are using a data volume in order to get persistence, consider mounting a host directory as your data volume (as seen in the tutorial you've linked to under Mount a host directory as a data volume.
Your final command should look something like this -
sudo docker run -d -p 8787:8787 -v /src/data:/data/ rocker/rstudio
Related
Sometimes the Docker container where Memgraph is running just stops working or says that the process was aborted with exit code 137. How can I fix this?
You should check the Memgraph logs, where you'll probably find the reason why the process was aborted.
Since you said that you're using Memgraph with Docker, there are two options:
If you run Memgraph with Docker using the volume for logs, that is with
-v mg_log:/var/log/memgraph, then mg_log folder usually can be found at \\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\ (Windows) or /var/lib/docker/volumes/ (Linux and macOS).
If you run Memgraph without using the volume for logs, then you need to enter the Docker container. In order to do that, first you have to find out the container ID by running docker ps. Then you have to copy the container ID and run docker exec -it <containerID> bash. For example, if container ID is 83d76fe4df5a, then you run docker exec -it 83d76fe4df5a bash. Next, you need find the folder where logs are located. You can do that by running cd /var/log/memgraph. To read the logs, run cat <memgraph_date>.log, that is, if you have log file memgraph_2022-03-02.log located inside the log folder, then run cat memgraph_2022-03-02.log.
Hopefully, when you read the logs, you'll be able to fix your problem.
I am running Memgraph on Windows 11 WSL using this command:
docker run -it -p 7687:7687 -p 3000:3000 -e MEMGRAPH="--bolt-port=7687" -v mg_lib:/mnt/c/temp/memgraph/lib -v mg_log:/mnt/c/temp/memgraph/log -v mg_etc:/mnt/c/temp/memgraph/etc memgraph
Then I created a node,
but I checked and those folders still empty.
How to persist Memgraph data to local hard drive?
Memgraph uses two mechanisms to ensure data durability:
write-ahead logs (WAL) and
periodic snapshots.
Snapshots are taken periodically during the entire runtime of Memgraph. When a snapshot is triggered, the whole data storage is written to the disk. Write-ahead logs save all database modifications that happened to a file. When running Memgraph with Docker, both of these mechanisms rely on the user to create volumes that will store this data when starting Memgraph.
There are two fields to specify for each volume.
The first is the name of the volume, and it's unique on a given host machine. In your case, that would be mg_lib, mg_log, and mg_etc.
The second field is the path where the file or directory is mounted in the container. In the case of Memgraph, that would be:
/var/lib/memgraph (this is where the durability related files are saved)
/var/log/memgraph (logs)
/etc/memgraph (configuration settings)
Given these paths, the command to run Memgraph with Docker is:
sudo docker run -it -p 7687:7687 -p 3000:3000 -v mg_lib:/var/lib/memgraph -v mg_log:/var/log/memgraph -v mg_etc:/etc/memgraph memgraph
By default, the volumes on the host machine can be found in:
\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes
I hope this answer can provide some clarity.
I'm using Windows 10 with native docker installation.
I'm looking for the location where docker save the containers logs.
In Linux, the Docker containers log files are in this location:
/var/lib/docker/containers/container-id/container-id-json.log
But where can I find it in windows 10 ?
For Windows 10 + WSL 2 (Ubuntu 20.04), Docker version 20.10.2, build 2291f61
Lets DOCKER_ARTIFACTS == \\wsl$\docker-desktop-data\version-pack-data\community\docker
Container logs can be found in the following location
DOCKER_ARTIFACTS\containers\[Your_container_ID]\[Your_container_ID]-json.log
Here is an example :
Check first if those logs are in (as suggested here):
C:\ProgramData\docker\containers\[container_ID]\[container_ID]-json.log
The Docker C:\ProgramData\docker is the Root Dir reported by docker info.
Regarding Docker Linux through Hyper-v, check if "How to Delete Docker Container Log Files (Windows or Linux) " can help (from Jon Gallant):
Run docker inspect to find your Docker log file location
Find the “Docker Root Dir” Value, mine is /var/lib/docker
Your docker log file path should be /var/lib/docker, but if it isn’t, then change it in the command below.
find /var/lib/docker/containers/ -type f -name "*.log"
The command you see in this image is based on "How to SSH into the Docker VM (MobyLinuxVM) on Windows"
We aren’t technically going to SSH into the VM, we’ll create a container that has full root access and then access the file system from there.
Get container with access to Docker Daemon
Run container with full root access
Switch to host file system
Open a Command prompt and execute the following:
docker run --privileged -it -v /var/run/docker.sock:/var/run/docker.sock jongallant/ubuntu-docker-client
docker run --net=host --ipc=host --uts=host --pid=host -it --security-opt=seccomp=unconfined --privileged --rm -v /:/host alpine /bin/sh
chroot /host
Execute the find command there, and you should find the logs.
For Windows 10 + Docker Desktop version 3.6.0, the virtual path for logs and data (artifacts) is \\wsl$\docker-desktop-data\version-pack-data\community\docker (you can copy/paste it in Explorer navigation bar).
The logs are at \\wsl$\docker-desktop-data\version-pack-data\community\docker\containers\[containerID]\[containerID]-json.logs
and the data is under \\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\[volumeID]\_data
How to get containerID:
You can find container ID (truncated) by running docker ps in a command prompt. You can also find it by clicking the CLI button on DockerDesktop next to the container name, the id will be in the title of the cmd window that will pop up.
Once you have the id, you can navigate to containers\[containerID] under the artifacts directory (\\wsl$\docker-... above). The log file will have the .log extension and its name will have the containerID in it. Keep in mind that it will be an enriched json format though, so not easily readable.
How to get volumeID:
To find a container's data (for example kafka broker topics), you need to find the id of the volumes where the data is stored. For that you will need to click on the container in DockerDesktop, then click INSPECT (top right). You can then scroll down to find Mounts configuration entries. Each mount will have a volumeID (different from containerID), and that volumeID will be part of a path similar to this one /var/lib/docker/volumes/71f7a5992c58fdcf229c3848acb014712f34fab380bc7c712cf5a0a632fe9110/_data. volumeID here being 71f7a5992c58fdcf229c3848acb014712f34fab380bc7c712cf5a0a632fe9110.
You can then take volumeID and navigate to volumes\[volumeID] under the artifacts directory (\\wsl$\docker-... above) where the data will be located.
For me, using Docker Desktop for Windows on version 4.9.1 (81317), Windows 10 21H2, WSL 2 mode, the containers' folders were at
\\wsl$\docker-desktop-data\data\docker\containers
Slightly different from the others.
For Windows Users who wants to delete all Docker log files on WSL 2.
The path to the docker container is correct thx to #craftsmannadeem
\\wsl$\docker-desktop-data\version-pack-data\community\docker\containers
Here a command to execute on Windows to delete all log files:
del /s \\wsl$\docker-desktop-data\version-pack-data\community\docker\containers\*-json.log
Bye bye Docker logs:
File was deleted - \\wsl$\docker-desktop-data\version-pack-data\community\docker\containers\2012efd0ccfb8aed6291dd9a3b7b5aef507b6af4fce5b85e9306f45980db9531\2012efd0ccfb8aed6291dd9a3b7b5aef507b6af4fce5b85e9306f45980db9531-json.log
File was deleted - \\wsl$\docker-desktop-data\version-pack-data\community\docker\containers\9e627f1fe8f3c3ab85c64f85f93942d1f077e9a6e2896b51df782b0c0c3777d1\9e627f1fe8f3c3ab85c64f85f93942d1f077e9a6e2896b51df782b0c0c3777d1-json.log
File was deleted - \\wsl$\docker-desktop-data\version-pack-data\community\docker\containers\6ea8f3cb354c199bc719701f8f1e75c333f81cd2f03dca0c7a626cbcbf9ed5a0\6ea8f3cb354c199bc719701f8f1e75c333f81cd2f03dca0c7a626cbcbf9ed5a0-json.log
...
For window system logs are located at C:\ProgramData\Docker\containers or %APPDATA%\Docker
For Linux system logs are located at /var/lib/docker/containers
I couldn't find where the logs were stored locally. (Good chance they aren't plain text any more. However, if you just need the output of the logs, you can run a command like this:
docker logs --details [container-name] > container-name.log
This will grab the logs for the container and write them to a log file in the current directory.
Note: The --details adds additional info to the logs like environment variables and the like, but is not required for the command to work.
If you use docker-compose with windows+wsl, In my case the log monitoring agent (also running as a container in docker) was not able to find the log files eventhough the path for logs was mounted as volume.
volumes:
- /var/run/docker.sock:/var/run/docker.sock
-/var/lib/docker:/var/lib/docker
The log monitoring agent could not find logs in
/var/lib/docker/containers/**/*.log
The problem in my case was, I was running docker-compose up command for the log monitoring agent from within wsl shell. When I ran it from windows powershell or cmd, the agent was able to find the logs in mounted path.
I have a similar issue to the following link, but in powershell as I am running a clickhouse docker container in windows 10.
Data directory permissions on host for Clickhouse installation via docker
My setup is run as such:
docker run -d -p 8124:8124 --name my_database --ulimit nofile=262144:262144 --volume=E:/:/var/lib/clickhouse yandex/clickhouse-server
E drive is one of the drives on my windows computer.
I cannot seem to access /var/lib/clickhouse/data when running a mergetree table creation. It seems that clickhouse client is not being given adequate permissions to reach this file system. The error looks as such:
Access to file denied: /var/lib/clickhouse/data/default/test_mergetree/tmp_insert_20150731_20150731_8_8_0
Since I am in powershell I am unsure how I might approach solving this. I am attempting to access the file system to give powershell permissions:
Something like this
ICACLS "var/lib/clickhouse/data" /setowner "administrator"
But then since clickhouse is dockerized it seems I cannot find the path:
The system cannot find the path specified.
Would I have to run docker compose? Or am I approaching this the wrong way?
ATTEMPT 1
I've tried running the following:
docker run --rm -i --entrypoint /bin/sh yandex/clickhouse-server -c id clickhouse
#got back:
uid=0(root) gid=0(root) groups=0(root)
#went into the system and ran
docker exec -it container-id bash
chown -R 0:0 /var/lib/clickhouse
#got back
chown: cannot read directory '/var/lib/clickhouse/System Volume Information': Permission denied
You should run docker and clickhouse in Linux instead of Windows.
Turns out this is an issue which has not been repaired in windows docker desktop:
https://github.com/docker/for-win/issues/39
Volume mounts are necesarry. But I got around it by changing the disk image to the target host drive. Under settings -> advanced -> change the virtual hard disk image to the drive you want and you can write within that drive. Note you still won't have access to raw data.
Still an issue but the practical solution is to move to Docker Volumes.
With bind mount there is a problem with wsl2
I migrated from Linux to Windows and tried to setup a postgres container with a mounted directory (copied from my Linux install) containing the database.
This does not work.
Windows mounts are always owned by root
Postgres does not run under root
How to get this unholy combination to work?
You don't provide much details so it is difficult to tell what actually went wrong. However there is a known issue with Postgres setup on Windows Docker using a windows mount for database data files. In that case, running docker logs will show something along the following lines
waiting for server to start....FATAL: data directory "/var/lib/postgresql/data" has wrong ownership
HINT: The server must be started by the user that owns the data directory.
stopped waiting
pg_ctl: could not start server
Unfortunately there is no way to overcome this issue so you cannot use Windows mount, see Postgres Data has wrong ownership. You may use docker volumes in order to make database data indipendent from docker postgres container, using the following commands
docker create -v /var/lib/postgresql/data --name PostgresData alpine
docker run -p 5432:5432 --name yourPostgres -e POSTGRES_PASSWORD=yourPassword -d --volumes-from PostgresData postgres
You may find a more thoroughful explanation at Setup Postgresql on Windows with Docker