I'm trying to run the official 5.4.3 Filebeat docker container via VirtualBox on a Windows host. Rather than creating a custom image, I'm using a volume mapping to pass the filebeat.yml file to the container using the automatically created VirtualBox mount /c/Users which points to C:\Users on my host.
Unfortunately I'm stuck on this error:
Exiting: error loading config file: config file ("filebeat.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/filebeat/filebeat.yml')
My docker-compose config is:
filebeat:
image: "docker.elastic.co/beats/filebeat:5.4.3"
volumes:
- "/c/Users/Nathan/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
- "/c/Users/Nathan/log:/mnt/log:ro"
I've tried SSH-ing into the machine and running the chmod go-w command but no change. Is this some kind of permission limitation when working with VirtualBox shared folders on a Windows host?
It looks like this is a side effect of the Windows DACL permissions system. Fortunately I only need this for a development environment so I've simply disabled the permission check by overriding the container entry point and passing the strict.perms argument.
filebeat:
image: "docker.elastic.co/beats/filebeat:5.4.3"
entrypoint: "filebeat -e -strict.perms=false"
volumes:
- "/c/Users/Nathan/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
- "/c/Users/Nathan/log:/mnt/log:ro"
Related
I have Docker installed on my Windows OS. There is my volumes filed of docker-compose.yml:
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
I just can't figure out how /var/run/docker.sock::/var/run/docker.sock path works for windows as I have no /var/run/ on my windows files where I can find docker.sock. So how this volume binding works at all?
The /var/run/docker.sock file on Docker for Mac and Windows for Linux images is inside the VM that Docker uses to run Linux containers. Those volume mounts happen from inside of that VM to the containers running in the VM. This is also why you can get an empty directory if you try to run a volume mount to a directory that you have not shared with the embedded VM.
You cannot see this file directly from the Windows environment (at least not that I'm aware of), though you can mount it into a container and see it that way.
For more details on how this VM is created, you can see the LinuxKit project: https://github.com/linuxkit/linuxkit
I'm trying to get the Hue docker image from gethue/hue, but it seems to ignore the configuration I give him and always look for HDFS on localhost instead of the docker container I ask him to look for.
Here is some context:
I'm using the following docker compose to launch a HDFS cluster:
hdfs-namenode:
image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8
hostname: namenode
environment:
- CLUSTER_NAME=davidov
ports:
- "8020:8020"
- "50070:50070"
volumes:
- ./data/hdfs/namenode:/hadoop/dfs/name
env_file:
- ./hadoop.env
hdfs-datanode1:
image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
depends_on:
- hdfs-namenode
links:
- hdfs-namenode:namenode
volumes:
- ./data/hdfs/datanode1:/hadoop/dfs/data
env_file:
- ./hadoop.env
This launches images from BigDataEurope, which are already properly configured, including:
- the activation of webhdfs (in /etc/hadoop/hdfs-site.xml):
- dfs.webhdfs.enabled set to true
- the hue proxy user (in /etc/hadoop/core-site.xml):
- hadoop.proxyuser.hue.hosts set to *
- hadoop.proxyuser.hue.groups set to *
The, I launch hue following their instructions:
First, I launch a bash prompt inside the docker container:
docker run -it -p 8888:8888 gethue/hue:latest bash
Then, I modify desktop/conf/pseudo-distributed.ini to point to the correct hadoop "node" (in my case a docker container with the address 172.30.0.2:
[hadoop]
# Configuration for HDFS NameNode
# ------------------------------------------------------------------------
[[hdfs_clusters]]
# HA support by using HttpFs
[[[default]]]
# Enter the filesystem uri
fs_defaultfs=hdfs://172.30.0.2:8020
# NameNode logical name.
## logical_name=
# Use WebHdfs/HttpFs as the communication mechanism.
# Domain should be the NameNode or HttpFs host.
# Default port is 14000 for HttpFs.
## webhdfs_url=http://172.30.0.2:50070/webhdfs/v1
# Change this if your HDFS cluster is Kerberos-secured
## security_enabled=false
# In secure mode (HTTPS), if SSL certificates from YARN Rest APIs
# have to be verified against certificate authority
## ssl_cert_ca_verify=True
And then I launch hue using the following command (still inside the hue container):
./build/env/bin/hue runserver_plus 0.0.0.0:8888
I then point my browser to localhost:8888, create a new user ('hdfs' in my case), and launch the HDFS file browser module. I then get the following error message:
Cannot access: /user/hdfs/.
HTTPConnectionPool(host='localhost', port=50070): Max retries exceeded with url: /webhdfs/v1/user/hdfs?op=GETFILESTATUS&user.name=hue&doas=hdfs (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 99] Cannot assign requested address',))
The interesting bit is that it still tries to connect to localhost (which of course cannot work), even though I modified its config file to point to 172.30.0.2.
Googling the issue, I found another config file: desktop/conf.dist/hue.ini. I tried modifying this one and launching hue again, but same result.
Does any one know how I could correctly configure hue in my case?
Thanks in advance for your help.
Regards,
Laurent.
Your one-off docker run command is not on the same network as the docker-compose containers.
You would need something like this, replacing [projectname] with the folder you started docker-compose up in
docker run -ti -p 8888:8888 --network="[projectname]_default" gethue/hue bash
I would suggest using Docker Compose also for the Hue container and volume mount for a INI files under desktop/conf/ that you can specify simply
fs_defaultfs=hdfs://namenode:8020
(since you put hostname: namenode in the compose file)
You'll also need to uncomment the WebHDFS line for your changes to take affect
All INI files are merged in the conf folder for Hue
I'm trying to do data persistence in postgres. But when I want to mount the data folder into my local file, I get this error.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
LOG: could not link file "pg_xlog/xlogtemp.25" to "pg_xlog/000000010000000000000001": Operation not permitted
FATAL: could not open file "pg_xlog/000000010000000000000001": No such file or directory
child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/data"
running bootstrap script ...
Here's my YAML file
version: '3.1'
services:
postgres:
restart: always
image: postgres:9.6.4-alpine
ports:
- 8100:5432
volumes:
- ./pgdata:/var/lib/postgresql/data
environment:
POSTGRES_PASSWORD: root
I'm using docker toolbox on windows. The docker machine in Virtual Box.
It looks like you use a shared data directory (host dir shared into the virtual) for database storage.
Only two options make sense:
1) you have a trivial issue with directory permissions
2) you hit a known problem (google!) with some VirtualBox and also VmWare versions that on some Windows versions, you cannot create symlinks in directories shared from the host to virtual machine.
for (2), a workaround is to NOT use shared folder to keep data.
Either way, it's a problem which should be solved by the provider of the docker image itself, or by the provider of virtualizer (vbox, vmware etc).
This is NOT a fault of Windows OS, or PostgreSQL.
Looks like it has to be /mnt/sda1/var/lib/docker/volumes/psql/_data for windows docker toolbox. This worked for me
docker run -it --name psql -p 5432:5432 -v psql:/var/lib/postgresql/data postgres:9.5-alpine
"Mounts": [
{
"Type": "volume",
"Name": "psql",
"Source": "/mnt/sda1/var/lib/docker/volumes/psql/_data",
"Destination": "/var/lib/postgresql/data",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
docker volume ls
DRIVER VOLUME NAME
local 65f253d220ad390337daaacf39e4d17000c36616acfe1707e41e92ab26a6a23a
local 761f7eceaed5525b70d75208a1708437e0ddfa3de3e39a6a3c069b0011688a07
local 8a42268e965e6360b230d16477ae78035478f75dc7cb3e789f99b15a066d6812
local a37e0cf69201665b14813218c6d0441797b50001b70ee51b77cdd7e5ef373d6a
local psql
Please refer this for more info: bad mount
I'm trying to invoke docker on my OSX host running Docker for Mac 17.06.0-ce-mac17 from inside a running jenkins docker container (jenkins:latest), per the procedure described at http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/.
I mount /var/run/docker.sock into the container, I stick a ubuntu docker binary inside it, and it's able to execute - but from inside the container as user "jenkins" when I run e.g. "docker ps" I get
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.30/containers/json?all=1: dial unix /var/run/docker.sock: connect: permission denied.
If I connect to the container as root (docker exec -u 0) it works though.
I need the jenkins user to be able to run this. I tried adding a docker group and adding jenkins to it inside the ubuntu container but that didn't help, since it's got nothing to do with the outside and Docker for Mac doesn't work like running this on linux where you can do semi easy uid/gid matching. I want to distribute this container so answers that go and hack part of my Docker for Mac install won't really help me. I'd rather not run the whole jenkins setup as root if I can help it. (I also tried running the container as privileged, that didn't help.)
Per the advice in Permission Denied while trying to connect to Docker Daemon while running Jenkins pipeline in Macbook I chowned the /var/run/docker.sock file inside the container manually to jenkins and now jenkins can run docker. But I'm having trouble coming up with a solution for a distributable container - I can't do that chown in the Dockerfile because the file doesn't exist yet, and shimming in into the entrypoint doesn't help because that runs as jenkins.
What do I need to do in order to build and run an image that will run external docker containers on my Mac as a non-root user from inside the container?
Follow this: https://forums.docker.com/t/mounting-using-var-run-docker-sock-in-a-container-not-running-as-root/34390
Basically, all you need to do is to change /var/run/docker.sock permissions inside your container and run the docker with sudo.
I've created a Dockerfile that can be used to help:
FROM jenkinsci/blueocean:latest
USER root
# change docker sock permissions after moutn
RUN if [ -e /var/run/docker.sock ]; then chown jenkins:jenkins /var/run/docker.sock; fi
I got this working, at least automated but currently only working on docker for Mac. Docker for Mac has a unique file permission model. Chowning /var/run/docker.sock to the jenkins user manually works, and it persists across container restarts and even image regeneration, but not past docker daemon restarts. Plus, you can't do the chown in the Dockerfile because docker.sock doesn't exist yet, and you can't do it in the entrypoint because that runs as jenkins.
So what I did was add jenkins to the "staff" group, because on my Mac, /var/run/docker.sock is symlinked down into /Users//Library/Containers/com.docker.docker/Data/s60 and is uid and gid staff. This lets the jenkins user run docker commands on the host.
Dockerfile:
FROM jenkins:latest
USER root
RUN \
apt-get update && \
apt-get install -y build-essential && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
COPY docker /usr/bin/docker
# To allow us to access /var/run/docker.sock on the Mac
RUN gpasswd -a jenkins staff
USER jenkins
ENTRYPOINT ["/bin/tini", "--", "/usr/local/bin/jenkins.sh"]
docker-compose.yml file:
version: "3"
services:
jenkins:
build: ./cd_jenkins
image: cd_jenkins:latest
ports:
- "8080:8080"
- "5000:5000"
volumes:
- ./jenkins_home:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock
This is, however, not portable to other systems (and depends on that docker for mac group staying "staff," which I imagine isn't guaranteed). I'd love suggested improvements to make this solution work across host systems. Other options suggested in questions like Execute docker host command inside jenkins docker container include:
Install sudo and let jenkins sudo and run all docker commands with sudo: adds security issues
"Add jenkins to the docker group" - UNIX only and probably relies on matching up gids from host to container right?
Setuid'ing the included docker executable might work, but has the same security elevation issues as sudo.
Another approach that worked for me - set the uid argument to the uid that owns /var/run/docker.sock (501 in my case). Not sure of the syntax for Dockerfile, but for docker-compose.yml, it's like this:
version: 3
services:
jenkins:
build:
context: ./JENKINS
dockerfile: Dockerfile
args:
uid: 501
volumes:
- /var/run/docker.sock:/var/run/docker.sock
...
Note this is based on using a Dockerfile to build the jenkins image, so many details left out. The key bit here is the uid: 501 under args.
We're working to create a standard "data science" image in Docker in order to help our team maintain a consistent environment. In order for this to be useful for us, we need the containers to have read/write access to our company's network. How can I mount a network drive to a docker container?
Here's what I've tried using the rocker/rstudio image from Docker Hub:
This works:
docker run -d -p 8787:8787 -v //c/users/{insert user}:/home/rstudio/foobar rocker/rstudio
This does not work (where P is the mapped location of the network drive):
docker run -d -p 8787:8787 -v //p:/home/rstudio/foobar rocker/rstudio
This also does not work:
docker run -d -p 8787:8787 -v //10.1.11.###/projects:/home/rstudio/foobar rocker/rstudio
Any suggestions?
I'm relatively new to Docker, so please let me know if I'm not being totally clear.
I know this is relatively old - but for the sake of others - here is what usually works for me. for use - we use a windows file-server so we use cifs-utils in order to map the drive. I assume that below instructions can be applied to nfs or anything else as well.
first - need to run the container in privileged mode so that you can mount remote folders inside of the container (--dns flag might not be required)
docker run --dns <company dns ip> -p 8000:80 --privileged -it <container name and tag>
now, (assuming centos with cifs and being root in the container) - hop into the container and run:
install cifs-utils if not installed yet
yum -y install cifs-utils
create the local dir to be mapped
mkdir /mnt/my-mounted-folder
prepare a file with username and credentials
echo "username=<username-with-access-to-shared-drive>" > ~/.smbcredentials
echo "password=<password>" > ~/.smbcredentials
map the remote folder
mount <remote-shared-folder> <my-local-mounted-folder> -t cifs -o iocharset=utf8,credentials=/root/.smbcredentials,file_mode=0777,dir_mode=0777,uid=1000,gid=1000,cache=strict
now you should have access
hope this helps..
I will write my decision. I have a Synology NAS. The shared folder uses the smb protocol.
I managed to connect it in the following way. The most important thing was to write version 1.0 (vers=1.0). It didn't work without it! I tried to solve the issue for 2 days.
version: "3"
services:
redis:
image: redis
restart: always
container_name: 'redis'
command: redis-server
ports:
- '6379:6379'
environment:
TZ: "Europe/Moscow"
celery:
build:
context: .
dockerfile: celery.dockerfile
container_name: 'celery'
command: celery --broker redis://redis:6379 --result-backend redis://redis:6379 --app worker.celery_worker worker --loglevel info
privileged: true
environment:
TZ: "Europe/Moscow"
volumes:
- .:/code
- nas:/mnt/nas
links:
- redis
depends_on:
- redis
volumes:
nas:
driver: local
driver_opts:
type: cifs
o: username=user,password=pass,**vers=1.0**
device: "//192.168.10.10/main"
I have been searching the solution the last days and I just get one working.
I am running docker container on an ubuntu virtual machine and I am mapping a folder on other host on the same network which is running windows 10, but I am almost sure that the operative system where the container is running is not a problem because the mapping is from the container itself so I think this solution should work in any SO.
Let's code.
First you should create the volume
docker volume create
--driver local
--opt type=cifs
--opt device=//<network-device-ip-folder>
--opt o=user=<your-user>,password=<your-pw>
<volume-name>
And then you have to run a container from an image
docker run
--name <desired-container-name>
-v <volume-name>:/<path-inside-container>
<image-name>
After this a container is running with the volume assignated to it,
and is mapped to .
You create some file in any of this folders and it will be replicated
automatically to the other.
In case someone wants to get this running from docker-compose I leave
this here
services:
<image-name>:
build:
context: .
container_name: <desired-container-name>
volumes:
- <volume-name>:/<path-inside-container>
...
volumes:
<volume-name>:
driver: local
driver_opts:
type: cifs
device: //<network-device-ip-folder>
o: "user=<your-user>,password=<your-pw>"
Hope I can help
Adding to the solution by #Александр Рублев, the trick that solved this for me was reconfiguring the Synology NAS to accept the SMB version used by docker. In my case I had to enable SMBv3
I know this is old, but I found this when looking for something similar but see that it's receiving comments for others, like myself, who find it.
I have figured out how to get this to work for a similar situation that took me awhile to figure out.
The answers here are missing some key information that I'll include, possibly because they weren't available at the time
The CIFS storage is, I believe, only for when you are connecting to a Windows System as I do not believe it is used by Linux at all unless that system is emulating a Windows environment.
This same thing can be done with NFS, which is less secure, but is supported by almost everything.
you can create an NFS volume in a similar way to the CIFS one, just with a few changes. I'll list both so they can be seen side by side
When using NFS on WSL2 you 1st need to install the NFS service into the Linux Host OS. I believe CIFS requires a similar one, most likely the cifs-utils mentioned by #LevHaikin, but as I don't use it I'm not certain. In my case the Host OS is Ubuntu, but you should be able to find the appropriate one by finding your system's equivalent for nfs-common (or cifs-utils if that's correct) installation
sudo apt update
sudo apt install nfs-common
That's it. That will install the service so NFS works on Docker (It took me forever to realize that was the problem since it doesn't seem to be mentioned as needed anywhere)
If using NFS, On the network device you need to have set NFS permissions for the NFS folder, in my case this would be done at the folder folder with the mount then being to a folder inside it. That's fine. (In my case the NAS that is my server mounts to #IP#/volume1/folder, within the NAS I never see the volume1 in the directory structure, but that full path to the shared folder is shown in the settings page when I set the NFS permissions. I'm not including the volume1 part as your system will likely be different) & you want the FULL PATH after the IP (use the IP as the numbers NOT the HostName), according to your NFS share, whatever it may be.
If using a CIFS device the same is true just for CIFS permissions.
The nolock option is often needed but may not be on your system. It just disables the ability to "lock" files.
The soft option means that if the system cannot connect to the mount directory it will not hang. If you need it to only work if the mount is there you can change this to hard instead.
The rw (read/write) option is for Read/Write, ro (read-only) would be for Read Only
As I don't personally use the CIFS volume the options set are just ones in the examples I found, whether they are necessary for you will need to be looked into.
The username & password are required & must be included for CIFS
uid & gid are Linux user & group settings & should be set, I believe, to what your container needs as Windows doesn't use them to my knowledge
file_mode=0777 & dir_mode=0777 are Linux Read/Write Permissions essentially like chmod 0777 giving anything that can access the file Read/Write/Execute permissions (More info Link #4) & this should also be for the Docker Container not the CIFS host
noexec has to do with execution permissions but I don't think actually function here, but it was included in most examples I found, nosuid limits it's ability to access files that are specific to a specific user ID & shouldn't need to be removed unless you know you need it to be, as it's a protection I'd recommend leaving it if possible, nosetuids means that it won't set UID & GUID for newly created files, nodev means no access to/creation of devices on the mount point, vers=1.0 I think is a fallback for compatibility, I personally would not include it unless there is a problem or it doesn't work without it
In these examples I'm mounting //NET.WORK.DRIVE.IP/folder/on/addr/device to a volume named "my-docker-volume" in Read/Write mode. The CIFS volume is using the user supercool with password noboDyCanGue55
NFS from the CLI
docker volume create --driver local --opt type=nfs --opt o=addr=NET.WORK.DRIVE.IP,nolock,rw,soft --opt device=:/folder/on/addr/device my-docker-volume
CIFS from CLI (May not work if Docker is installed on a system other than Windows, will only connect to an IP on a Windows system)
docker volume create --driver local --opt type=cifs --opt o=user=supercool,password=noboDyCanGue55,rw --opt device=//NET.WORK.DRIVE.IP/folder/on/addr/device my-docker-volume
This can also be done within Docker Compose or Portainer.
When you do it there, you will need to add a Volumes: at the bottom of the compose file, with no indent, on the same level as services:
In this example I am mounting the volumes
my-nfs-volume from //10.11.12.13/folder/on/NFS/device to "my-nfs-volume" in Read/Write mode & mounting that in the container to /nfs
my-cifs-volume from //10.11.12.14/folder/on/CIFS/device with permissions from user supercool with password noboDyCanGue55 to "my-cifs-volume" in Read/Write mode & mounting that in the container to /cifs
version: '3'
services:
great-container:
image: imso/awesome/youknow:latest
container_name: totally_awesome
environment:
- PUID=1000
- PGID=1000
ports:
- 1234:5432
volumes:
- my-nfs-volume:/nfs
- my-cifs-volume:/cifs
volumes:
my-nfs-volume:
name: my-nfs-volume
driver_opts:
type: "nfs"
o: "addr=10.11.12.13,nolock,rw,soft"
device: ":/folder/on/NFS/device"
my-cifs-volume:
driver_opts:
type: "cifs"
o: "username=supercool,password=noboDyCanGue55,uid=1000,gid=1000,file_mode=0777,dir_mode=0777,noexec,nosuid,nosetuids,nodev,vers=1.0"
device: "//10.11.12.14/folder/on/CIFS/device/"
More details can be found here:
https://docs.docker.com/engine/reference/commandline/volume_create/
https://www.thegeekdiary.com/common-nfs-mount-options-in-linux/
https://web.mit.edu/rhel-doc/5/RHEL-5-manual/Deployment_Guide-en-US/s1-nfs-client-config-options.html
https://www.maketecheasier.com/file-permissions-what-does-chmod-777-means/