Run postgres container with data volumes through docker-machine - macos

I have an issue with running postgres container with set up volumes for data folder on my Mac OS machine.
I tried to run it such like this:
docker run \
--name my-postgres \
-e POSTGRES_USER=admin \
-e POSTGRES_PASSWORD=password \
-e POSTGRES_DB=some_db_dev \
-v $PG_LOCAL_DATA:/var/lib/postgresql/data \
-d postgres:9.5.1
Every time I got the following result in logs:
* Starting PostgreSQL
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are enabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
initdb: could not create directory "/var/lib/postgresql/data/pg_xlog": Permission denied
initdb: removing contents of data directory "/var/lib/postgresql/data"
Versions of docker, docker-machine, virtualbox and boot2docker are:
docker-machine version 0.6.0, build e27fb87
Docker version 1.10.2, build c3959b1
VirtualBox Version 5.0.16 r105871
boot2docker 1.10.3
I saw many publications about this topic but the most of them are outdated. I had tried do the similar solution as for mysql but it did not help.
Maybe somebody can updated me: does some solution exist to run postgres container with data volumes through docker-machine?
Thanks!

If you are running docker-machine on a Mac, at this time, you cannot mount to a directory that is not part of your local user space (/Users/<user>/) without extra configuration.
This is because on the Mac, Docker makes a bind mount automatically with the home ~ directory. Remember that since Docker is being hosted in a VM that isn't your local Mac OS, any volume mounts are relative to the host VM - not your machine. That means by default, Docker cannot see your Mac's directories since it is being hosted on a separate VM from your Mac OS.
Mac OS => Linux Virtual Machine => Docker
^------------------^
Docker Can See VM
^-----------------X----------------^
Docker Can't See Here
If you open VirtualBox, you can create other mounts (i.e. shared folders) to your local machine to the host VM and then access them that way.
See this issue for specifics on the topic: https://github.com/docker/machine/issues/1826
I believe the Docker team is adding these capabilities in upcoming releases (especially since a native Mac version is in short works).

You should use docker named volumes instead of folders on your local file system.
Try creating a volume:
docker volume create my_vol
Then mount the data directory in your above command:
docker run \
--name my-postgres \
-e POSTGRES_USER=admin \
-e POSTGRES_PASSWORD=password \
-e POSTGRES_DB=some_db_dev \
-v my_vol:/var/lib/postgresql/data \
-d postgres:9.5.1
Checkout my blog post for a whole postgres, node ts docker setup for both dev and prod: https://yzia2000.github.io/blog/2021/01/15/docker-postgres-node.html
For more on docker volumes: https://docs.docker.com/storage/volumes/

Related

How to persist Memgraph data to local hard drive?

I am running Memgraph on Windows 11 WSL using this command:
docker run -it -p 7687:7687 -p 3000:3000 -e MEMGRAPH="--bolt-port=7687" -v mg_lib:/mnt/c/temp/memgraph/lib -v mg_log:/mnt/c/temp/memgraph/log -v mg_etc:/mnt/c/temp/memgraph/etc memgraph
Then I created a node,
but I checked and those folders still empty.
How to persist Memgraph data to local hard drive?
Memgraph uses two mechanisms to ensure data durability:
write-ahead logs (WAL) and
periodic snapshots.
Snapshots are taken periodically during the entire runtime of Memgraph. When a snapshot is triggered, the whole data storage is written to the disk. Write-ahead logs save all database modifications that happened to a file. When running Memgraph with Docker, both of these mechanisms rely on the user to create volumes that will store this data when starting Memgraph.
There are two fields to specify for each volume.
The first is the name of the volume, and it's unique on a given host machine. In your case, that would be mg_lib, mg_log, and mg_etc.
The second field is the path where the file or directory is mounted in the container. In the case of Memgraph, that would be:
/var/lib/memgraph (this is where the durability related files are saved)
/var/log/memgraph (logs)
/etc/memgraph (configuration settings)
Given these paths, the command to run Memgraph with Docker is:
sudo docker run -it -p 7687:7687 -p 3000:3000 -v mg_lib:/var/lib/memgraph -v mg_log:/var/log/memgraph -v mg_etc:/etc/memgraph memgraph
By default, the volumes on the host machine can be found in:
\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes
I hope this answer can provide some clarity.

Data directory permissions on host for Clickhouse installation via docker on Windows

I have a similar issue to the following link, but in powershell as I am running a clickhouse docker container in windows 10.
Data directory permissions on host for Clickhouse installation via docker
My setup is run as such:
docker run -d -p 8124:8124 --name my_database --ulimit nofile=262144:262144 --volume=E:/:/var/lib/clickhouse yandex/clickhouse-server
E drive is one of the drives on my windows computer.
I cannot seem to access /var/lib/clickhouse/data when running a mergetree table creation. It seems that clickhouse client is not being given adequate permissions to reach this file system. The error looks as such:
Access to file denied: /var/lib/clickhouse/data/default/test_mergetree/tmp_insert_20150731_20150731_8_8_0
Since I am in powershell I am unsure how I might approach solving this. I am attempting to access the file system to give powershell permissions:
Something like this
ICACLS "var/lib/clickhouse/data" /setowner "administrator"
But then since clickhouse is dockerized it seems I cannot find the path:
The system cannot find the path specified.
Would I have to run docker compose? Or am I approaching this the wrong way?
ATTEMPT 1
I've tried running the following:
docker run --rm -i --entrypoint /bin/sh yandex/clickhouse-server -c id clickhouse
#got back:
uid=0(root) gid=0(root) groups=0(root)
#went into the system and ran
docker exec -it container-id bash
chown -R 0:0 /var/lib/clickhouse
#got back
chown: cannot read directory '/var/lib/clickhouse/System Volume Information': Permission denied
You should run docker and clickhouse in Linux instead of Windows.
Turns out this is an issue which has not been repaired in windows docker desktop:
https://github.com/docker/for-win/issues/39
Volume mounts are necesarry. But I got around it by changing the disk image to the target host drive. Under settings -> advanced -> change the virtual hard disk image to the drive you want and you can write within that drive. Note you still won't have access to raw data.
Still an issue but the practical solution is to move to Docker Volumes.
With bind mount there is a problem with wsl2

Docker Postgres with windows share

I migrated from Linux to Windows and tried to setup a postgres container with a mounted directory (copied from my Linux install) containing the database.
This does not work.
Windows mounts are always owned by root
Postgres does not run under root
How to get this unholy combination to work?
You don't provide much details so it is difficult to tell what actually went wrong. However there is a known issue with Postgres setup on Windows Docker using a windows mount for database data files. In that case, running docker logs will show something along the following lines
waiting for server to start....FATAL: data directory "/var/lib/postgresql/data" has wrong ownership
HINT: The server must be started by the user that owns the data directory.
stopped waiting
pg_ctl: could not start server
Unfortunately there is no way to overcome this issue so you cannot use Windows mount, see Postgres Data has wrong ownership. You may use docker volumes in order to make database data indipendent from docker postgres container, using the following commands
docker create -v /var/lib/postgresql/data --name PostgresData alpine
docker run -p 5432:5432 --name yourPostgres -e POSTGRES_PASSWORD=yourPassword -d --volumes-from PostgresData postgres
You may find a more thoroughful explanation at Setup Postgresql on Windows with Docker

Where exactly, are files in docker container stored on the host machine

I am using docker on windows. With the use of kitematic, I have created an ubuntu container. This ubuntu image has postgresql installed on it.
Wondering if there is any possibility to access the postgres configuration files available in the container from the host (windows machine)?
Where exactly does the container store its file system on the host machine?
I hope it would be part of image file with format VMDK.
Please correct me if I'm wrong.
Wondering if there is any possibility to access the postgres configuration files available in the container from the host (windows machine)
That is not how Docker would allow you to modify a file in a container.
For that, you should mount a host (Windows) folder when starting (docker run -v) your container.
See "Mount a host directory as a data volume"
docker run -d -P --name web -v /c/Users/<myACcount>/src/webapp:/opt/webapp training/webapp python app.py
Issue 247 mentions ~/Library/Application Support/Kitematic for App data, and ~/Kitematic "for easy access to volume data".

Where is /var/lib/docker on Mac/OS X

I´m looking for the folder /var/lib/docker on my Mac after installing docker for Mac.
With docker info I get
Containers: 5
...
Server Version: 1.12.0-rc4
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 339
Dirperm1 Supported: true
...
Name: moby
ID: LUOU:5UHI:JFNI:OQFT:BLKR:YJIC:HHE5:W4LP:YHVP:TT3V:4CB2:6TUS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
....
But I don´t have a directory /var/lib/docker on my host.
I have checked /Users/myuser/Library/Containers/com.docker.docker/ but couldn´t find anything there. Any idea where it is located?
As mentioned in the above answers, you will find it in:
screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
Once you get the tty running you can navigate to /var/lib/docker
As of 2021 is the dance going, Mac Users get easily to the VM with the documented methods, and hence to the volumes.
There's a way Rocky Chen found to get inside the VM in Mac. With this you can actually inspect the famous /var/lib/docker/volumes.
docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
Let examine the method:
-it goes for Keep STDIN open even if not attached + Allocate pseudo-TTY
--privileged "gives all capabilities to the container. Allows special cases like running docker" .
--pid defines to use the host VM namespace.
debian the actual image to use.
nsenter a debian's tool to run programs in different namespaces
-t is the target PID
-m mount the provided PID namespace.
-u enter the Unix Time Sharing (UTS) namespace.
-n enter the provided PID network namespace.
-i enter the provided PID IPC namespace.
Once run, go to /var/lib/docker/volumes/and you'll find your volumes.
The next question to address for me is:
How to take those volumes and back them up in the host?
I appreciate ideas in the comments!
UPDATE FOR VSCODE USERS
If you downloaded the Official Docker extension, sun will shine for you.
Just inspect the volumes in Visual Studio Code. Right-click the files you want to have in your local, and download them. That easy!
2nd UPDATE
As of July 2021, Docker Desktop for Mac is announcing we will be able to access volumes directly from the GUI, but only for Pro and Team accounts.
The other answers here are outdated if you're using Docker for Mac.
Here's how I was able to get into the VM. Run the command:
screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
This is the default path, but you may need to first do:
cd ~/Library/Containers/com.docker.docker/Data/vms
and then ls to see which directory your VM is in and replace the "0" accordingly.
When you're in, you might just see a blank screen. Hit your "Enter" key.
This page explains that to exit from the VM you need to "Ctrl-a" then "d"
See this answer
When using Docker for Mac Application, it appears that the containers are stored within the VM located at:
~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
Just as #Dmitriy said:
screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
and can use ctrl a + d to detach the screen
and use screen -dr to re-attach the screen again(since if you simply attach screen again, the terminal text will be garbled.)
Reference
or if you want to exit, use ctrl + a + k,then choose y to kill the screen.
some what of a zombie thread but as I just found it here is another solution that doesn't need screen nor messes up shell etc.
The path listed from a docker volume inspect <vol_name>
returns the path for the container, something like:
"Mountpoint": "/var/lib/docker/volumes/coap_service_db_data/_data"
the _data component being the last component of the path you setup in the volumes: section of the service using a given volume eg:
volumes:
- db_data:/var/lib/postgresql/data , obvs your mileage will vary.
To get there on the mac the easiest method I have found is to actually start a small container running and mount the root of the host to the /docker directory in the image, this gives you access to the volumes used on the host.
docker run --rm -it -v /:/docker alpine:edge
from this point you can cd to the volume
cd /var/lib/docker/volumes/coap_service_db_data/_data
I think the new version of docker (my version is 20.10.5) uses socket instead of TTY to communicate with the virtual machine so you can use the nc command instead of the screen command.
nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock
Looks like the new version of docker for Mac has moved this to a UI element which you can see here. Clicking on that button which says CLI will launch a terminal which you can use to browse the docker file system.
Run:
docker run -it --privileged --pid=host debian nsenter -t 1 -a bash
ls /var/lib/docker
For MacOS I use the following steps:
login into docker virtual-machine (on MacOS docker can be run only inside virtual machine, in my case I have VirtualBox tool with docker VM): docker-machine ssh
as soon as I logged-in I need to switch to super user from docker user: sudo -i
now I'm able to check /var/lib/docker directory
I would say that the file:
/var/run/docker.sock
Is actually at:
/Volumes/{DISKNAME}/var/run/docker.sock
If you run this, it should prove it, as long as your running VirtualBox 5.2.8 or later and the share for /Volumes is setup to be auto-mounted and permanent AND you generated the default docker-machine while on that version of Virtualbox:
#!/bin/bash
docker run -d --restart unless-stopped -p 9000:9000 \
-v /var/run/docker.sock:/var/run/docker.sock portainer/portainer \
--no-auth
Then, access Portainer at: 192.168.99.100:9000 or localhost:9000
This path comes from Docker Host (not from MacOS)
before "Docker for Mac Application" times, where there was a VirtualBox VM "default" and inside this VM, the mentioned path exists (for sure), now in "Docker for Mac Application" times there is a Docker.qcow2 image, which is qemu base vm.
To jump inside this VM #mik-jagger way is ok (but there are few more)
Docker logs are not in /var/lib/docker on MacOS.
MacOs users can find the docker logs on this path;
/Users/Barrack.Kenya/Library/Containers/com.docker.docker/Data/log/host
job_name: docker
static_configs:
targets:
docker
labels:
job: dockerlogs
path: (Please put the path)
pipeline_stages:
docker: {}

Resources