I am experiencing an issue where MSYS2 running in docker can create and modify files in a mounted volume, but not delete them. This can be observed with commands where the deletion is explicit, such as rm a.txt, or with commands where the deletion is implicit, such as sed -i 's/foo/bar/' b.txt (b.txt should be edited in-place, and appears to be read, removed and recreated from the output (1)). The same files can be deleted with the equivalent DOS commands, such as del a.txt.
Using the following Dockerfile:
# Altered from https://github.com/StefanScherer/dockerfiles-windows/blob/main/msys2/Dockerfile
FROM mcr.microsoft.com/windows/servercore:20H2
RUN powershell -Command \
$ProgressPreference = 'SilentlyContinue' ; \
Write-Output 'Downloading msys2' ; \
curl.exe -o msys2.tar.xy http://repo.msys2.org/distrib/msys2-x86_64-latest.tar.xz ; \
Install-Package -Scope CurrentUser -Force 7Zip4Powershell ; \
Write-Output 'Extracting tar.xz' ; \
Expand-7zip msys2.tar.xy . ; \
Write-Output 'Extracting tar' ; \
Expand-7zip msys2.tar C:/ ; \
Write-Output 'Done'
RUN powershell -Command \
setx /M PATH $('C:\msys64\mingw64\bin;C:\msys64\usr\bin;C:\msys64\mingw32\bin;' + $Env:PATH)
Build and launch the container with:
docker pull mcr.microsoft.com/windows/servercore:20H2
docker build -t docker-msys .
mkdir local
docker run -ti -v "$(pwd)/local:C:/local" docker-msys cmd.exe
Using MSYS2 commands from within the container, lets manipulate files with:
touch a.txt
cp a.txt b.txt
rm a.txt
rm -f b.txt
ls *.txt
If launched from a non-mounted directory, the following is observed:
C:\>touch a.txt
C:\>cp a.txt b.txt
C:\>rm a.txt
C:\>rm -f b.txt
C:\>ls *.txt
ls: cannot access '*.txt': No such file or directory
Both files have been deleted as expected.
If launched from a mounted directory, here C:\local, the following is observed:
cd C:\local
C:\local>touch a.txt
C:\local>cp a.txt b.txt
C:\local>rm a.txt
rm: cannot remove 'a.txt': Invalid argument
C:\local>rm -f b.txt
C:\local>ls *.txt
a.txt b.txt
Both files still remain.
Other MSYS commands will have unexpected behavior in the mounted directory:
C:\local>sed -i 's/foo/bar/' b.txt
sed: cannot remove ./sed7E6QmG: Invalid argument
The files can be removed from the mounted directory with the equivalent dos command:
C:\local>del /Q *
C:\local>dir
Volume in drive C has no label.
Volume Serial Number is 541F-A7C7
Directory of C:\local
11/26/2020 04:05 PM <DIR> .
11/26/2020 04:05 PM <DIR> ..
0 File(s) 0 bytes
2 Dir(s) 48,827,674,624 bytes free
Changing from a mounted directory to a named volume affect in nothing the situation. The previous output is the same when the container is launched with:
docker volume create local-volume
docker run -ti -v local-volum:C:/local docker-msys cmd
The problem doesn't appear related to my Dockerfile. I was able to reproduce it using a container from https://hub.docker.com/r/mizarjp/winci-msys2 .
docker pull mizarjp/winci-msys2:1909
docker run -ti --rm -v "$(pwd)/local:C:/local" mizarjp/winci-msys2:1909 cmd.exe
Once in the container, perform set PATH=C:\msys64\mingw64\bin;C:\msys64\usr\bin;C:\msys64\mingw32\bin;%PATH%; before repeating the previous tests to obtain the same results.
Finally, rather than installing MSYS2 inside docker, mounting it's directory inside a container produce the same situation.
Is there anything I am missing?
Details:
Windows 10 Pro, 20H2, 19042.746
Docker Desktop 3.1.0 (51484)
MSYS2 (http://repo.msys2.org/distrib/msys2-x86_64-latest.tar.xz) 2020-07-03 19:34
(1) My humble interpretation!
Adding --isolation=process fixed it. rm no longer fails for me!
docker pull mcr.microsoft.com/windows/servercore:20H2
docker build -t docker-msys .
mkdir local
docker run --rm -ti --isolation=process -v "$(pwd)/local:C:/local" docker-msys cmd.exe
Re-running the previous commands produces:
C:\>cd local
C:\local>touch a.txt
C:\local>cp a.txt b.txt
C:\local>rm a.txt
C:\local>rm -f b.txt
C:\local>ls *.txt
ls: cannot access '*.txt': No such file or directory
Related
I am trying to run a bash script which should load data into jena. This script comes from a github repository and was allegedly working on the owner's machine but on mine it won't run, even though I followed the instructions. So let me first describe what the script does based on my understanding: It should load .nt data (RDF data) into Jena using docker by using the docker image of jena, named stain/jena. Here is the script:
#/bin/bash
files=$(echo $(pwd)/rawdata-bearb/hour/alldata.IC.nt/*.nt | sed "s%$(pwd)/rawdata-bearb/hour/alldata.IC.nt%/var/data/in%g")
mkdir output # added
for file in $files; do
v=$(echo $file | sed "s/^.*\/\([0-9][0-9]*\)\.nt$/\1-1/" | bc)
echo "$v"
mkdir -p /var/data/out/ic/$v
time docker run \
-it \
--rm \
-v $(pwd)/tdb-bearb-hour/:/var/data/out/ \
-v $(pwd)/rawdata-bearb/hour/alldata.IC.nt/:/var/data/in/ \
stain/jena /jena/bin/tdbloader2 \
--sort-args "-S=16G" \
--loc /var/data/out/ic/$v $file \
> output/load-bearb-hour-ic-$v-.txt
done
However, when I execute the script, I get following message from the saved log file:
13:12:46 INFO -- TDB Bulk Loader Start
mkdir: cannot create directory ‘/var/data/out/ic/0’: No such file or directory
13:12:46 ERROR Failed during data phase
According to the tdbloader2 manual the --loc parameter should create the directory if it does not exist
-- loc: Sets the location in which the database should be created.
This location must be a directory and must be empty,
if a non-existent path is specified it will be created as a new directory.
I created the directories /var/data/out/ic/0 - /var/data/out/ic/10 manually and re-executed the script. Still, I got the same error message. My first guess was that tdbloader2 or docker use the mkdir command without the -p parameter but since I manually created the directories, thus, they existed before the execution and I still got the same error, it must be something else. I am kindly asking for your help
My script is running perfectly on co-workers devices (MacOSX with Docker Desktop same as me), but gives me every time the same error and it does not move or only half, the libraries in the deps directory:
OSError: [Errno 18] Invalid cross-device link: '/tmp/pip-target-dzwe_2kc/lib/python/numpy' ->
'/foo/python/numpy'
My script :
#!/bin/bash
export PKG_DIR='python'
export SIDE_DEPS_DIR='deps'
rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}
rm -rf ${SIDE_DEPS_DIR} && mkdir -p ${SIDE_DEPS_DIR}
docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.8 \
pip3 install -r requirements.txt -t ${PKG_DIR}
# move stuff to deps
find /${PKG_DIR} -maxdepth 1 -type d \
\( -name "pandas*" -o -name "numpy*" -o -name "numpy.libs*" -o -name "scipy*" -o -name "scipy.libs*" \) -exec mv '{}' ${SIDE_DEPS_DIR} \;
# zip side dependencies
zip -r ge_deps.zip deps
# zip layer
zip -r layers-python38-great-expectations.zip python
It's a script which uses a public lambda docker image to create a lambda layer (basically a zip that contains libraries) and which removes unwanted libraries to put them in another folder deps.
The above code will use the public Docker image lambci / lambda and will install in the empty python directory, libraries which come from a python package which is called 'great-expectations' and which helps to test pipelines of data (which is specified in requirements.txt and is great-expectations==0.12.7)
I have been stuck with this problem for a while and have not found a solution.
Had this exact problem just now.
/tmp and /foo are different devices - /tmp is within the docker OS and /foo is mapped to your local OS.
pip seems to be using shutil.rename() to move the built package from tmp to the final output location (/foo). This fails because they are different devices. Ideally pip would use shutil.move() instead, which will deal with a cross-device move.
As a workaround, you can change the temp folder used by PIP by setting TMPDIR before invoking the pip command. i.e. export TMPDIR=/foo/tmp before calling pip in the docker image. So, the whole command might be something like
docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.8 \
/bin/bash -c "export TMPDIR=/foo/tmp && pip3 install -r requirements.txt -t ${PKG_DIR}"
(multiple commands soln taken from https://www.edureka.co/community/10736/how-to-run-multiple-commands-in-docker-at-once - open to better suggestions!)
This will likely be slower because it's using the local OS for temp files, but it avoids the attempted 'rename' across devices from the temp folder to the final output folder.
I am new to Docker, Debezium, Bash, and Kafka. I am attempting to run the Debezium tutorial/example for MSSQL Server on Windows 10 here:
https://github.com/debezium/debezium-examples/blob/master/tutorial/README.md#using-sql-server
I am able to start the topology, per step one. However, when I go to step two and execute the following command:
cat debezium-sqlserver-init/inventory.sql | docker exec -i tutorial_sqlserver_1 bash -c '/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD'
I get the following error:
bash: C:/Program: No such file or directory
I do not have the foggiest idea why it would even drag C:/Program in to this. I do not see it in the command nor do I see it in the *.sql file. Does anyone know why this is happening and what the fix is?
Note 1: I am already in the current directory where this command should be runnable and there are no spaces in the folder/file path
Note 2: I am running the commands in Git Bash
When using set -x to log how the command is run, there's still no C:/Program anywhere in it, as can be seen by the following log:
$ cat debezium-sqlserver-init/inventory.sql | docker exec -i tutorial_sqlserver_1 bash -c '/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD'
+ cat debezium-sqlserver-init/inventory.sql
+ docker exec -i tutorial_sqlserver_1 bash -c '/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD'
bash: C:/Program: No such file or directory
I had a similar problem yesterday, the solution was adding a backslash before the absolute path, like :
cat debezium-sqlserver-init/inventory.sql | docker exec -i tutorial_sqlserver_1 bash -c '\/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD'
\/opt/mssql-tools/bin/sqlcmd prevents conversion to Windows path.
I am trying to create a shell script that will check for a new file then cp to a Docker Container. The code I have so far is...
#!/bin/sh
source="/var/www/html/"
dest="dev_ubuntu:/var/www/html/"
inotifywait -m "/var/www/html" -e create -e moved_to |
while read file; do
sudo docker cp /var/www/html/$file dev_ubuntu:/var/www/html
done
But this code gives the following error:
Setting up watches.
Watches established.
"docker cp" requires exactly 2 argument(s).
See 'docker cp --help'.
Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
Copy files/folders between a container and the local filesystem
What am I doing wrong?
Do you have spaces in your file names? Use double quotes to avoid separating filenames by words:
echo $file
sudo docker cp "$file" dev_ubuntu:"$file"
I've also echoed the file name to see what is happening.
This works because it's copying some_file:
docker run --rm -v target-data:/target -v ~/source:/source alpine cp source/some_file target/
This does not (Using wildcard):
docker run --rm -v target-data:/target -v ~/source:/source alpine cp source/* target/
cp: can't stat 'source/*': No such file or directory
How do copy all the files in the souce volume to the target volume?
The thing is who expand that *:
docker run --rm -v target-data:/target -v ~/source:/source alpine sh -c 'cp -r source/* target/'
You need someone (sh) to expand the * before the program is launched (that is what the shell do before exec cp)