Makefile:
SHELL=/bin/bash
.PONY: stamp diff
.DEFAULT_GOAL := all
diff:
diff <(./stamp.txt) <(docker-compose -f docker-compose.test.yml up)
stamp:
docker-compose -f docker-compose.test.yml up > stamp.txt
all: stamp diff
Output:
$ make
docker-compose -f docker-compose.test.yml up > stamp.txt
Starting ci-test ... done
diff <(./stamp.txt) <(docker-compose -f docker-compose.test.yml up)
/bin/bash: ./stamp.txt: Permission denied
Starting ci-test ... done
0a1,4
> Attaching to ci-test
> ci-test | Python 3.7.3
> ci-test exited with code 0
>
\ No newline at end of file
Makefile:26: recipe for target 'diff' failed
make: *** [diff] Error 1
$ ll
total 32
drwxrwxr-x 4 mirror mirror 4096 May 26 07:38 ./
drwxrwxr-x 47 mirror mirror 4096 May 24 15:58 ../
-rw-rw-r-- 1 mirror mirror 210 May 26 07:34 docker-compose.test.yml
-rw-rw-r-- 1 mirror mirror 103 May 26 07:33 .env
drwxrwxr-x 9 mirror mirror 4096 May 26 07:36 .git/
drwxrwxr-x 3 mirror mirror 4096 May 24 03:29 .idea/
-rw-rw-r-- 1 mirror mirror 664 May 26 07:38 Makefile
-rw-rw-r-- 1 mirror mirror 92 May 26 07:36 stamp.txt
Questions. Why I got /bin/bash: ./stamp.txt: Permission denied and how to fix it?
You have a bug in your makefile. Here is a simplified version
SHELL=/bin/bash
.PHONY: stamp diff
.DEFAULT_GOAL := all
diff:
#diff <(./stamp.txt) <(echo "aaa")
stamp:
#echo > stamp.txt
all: stamp diff
This line: #diff <(./stamp.txt) <(echo "aaa") will pass result of execution of ./stamp.txt. I guess, what you are looking for is
SHELL=/bin/bash
.PHONY: stamp diff
.DEFAULT_GOAL := all
diff:
#diff <(cat ./stamp.txt) <(echo "aaa")
stamp:
#echo > stamp.txt
all: stamp diff
Related
Dockerfile:
FROM ubuntu:latest
RUN apt install -y bash
CMD []
build and run:
docker build -t test .
docker run -it test bash
minimal reproduction:
root#8807902e27b4:/# mkdir parent
root#8807902e27b4:/# cd parent
root#8807902e27b4:/parent# mkdir example
root#8807902e27b4:/parent# chmod 000 example
root#8807902e27b4:/parent# ls -la
total 12
drwxr-xr-x 3 root root 4096 Apr 28 19:33 .
drwxr-xr-x 1 root root 4096 Apr 28 19:32 ..
d--------- 2 root root 4096 Apr 28 19:33 example
root#8807902e27b4:/parent# cd example
root#8807902e27b4:/parent/example# echo "test" > test.txt
root#8807902e27b4:/parent/example# chmod 100 test.txt
root#8807902e27b4:/parent/example# cat test.txt
test
root#8807902e27b4:/parent/example# ls -la
total 12
d--------- 2 root root 4096 Apr 28 19:33 .
drwxr-xr-x 3 root root 4096 Apr 28 19:33 ..
---x------ 1 root root 5 Apr 28 19:33 test.txt
In the above example, the cd example command should fail, and even if it doesn't, running cat test.txt should fail. Anyone know what's up?
Here are the same (working) commands run in osx:
beaushinkle#Beaus-MBP ~/p/example-docker> mkdir parent
beaushinkle#Beaus-MBP ~/p/example-docker> cd parent
beaushinkle#Beaus-MBP ~/p/e/parent> mkdir example
beaushinkle#Beaus-MBP ~/p/e/parent> chmod 000 example
beaushinkle#Beaus-MBP ~/p/e/parent> cd example
cd: Permission denied: 'example'
beaushinkle#Beaus-MBP ~/p/e/parent [1]> chmod 777 example
beaushinkle#Beaus-MBP ~/p/e/parent> cd example
beaushinkle#Beaus-MBP ~/p/e/p/example> echo "test" > test.txt
beaushinkle#Beaus-MBP ~/p/e/p/example> chmod 100 test.txt
beaushinkle#Beaus-MBP ~/p/e/p/example> cat test.txt
cat: test.txt: Permission denied
If the prompt is anything to go by, we are logged in as root in the minimal reproduction. Thus, we have root privileges and can read and write all files (external link).
I'm trying to iterate over url entries in a file and use each file as an input for a crawler tool. It's result should be written to a file.
here is the gitlab-ci.yml file:
stages:
- test
test:
stage: test
tags:
- shell-docker
script:
- wget https://github.com/FaKeller/sireg/releases/download/v0.3.1/sireg-linux
- chmod 775 sireg-linux
- mkdir output
- ls -alF
- while read line; do
echo $line;
./sireg-linux exec --loader-sitemap-sitemap \"$line\" >> ./output/${line##*/}_out.txt;
done < sitemap-index
- ls -alF output
artifacts:
paths:
- output/*
expire_in: 1 hrs
and here is the sitemap-index file (only one entry):
http://example.com/sitemap.xml
both files are in the same directory. I expect a file sitemap.xml_out.txt to be written into the output folder(also the same directory). I am pretty sure the ./sireg-linux script does not execute because it usually takes few minutes to complete (tested locally).
the output of the stage looks like this:
2020-04-02 18:22:21 (4,26 MB/s) - »sireg-linux« saved [62566347/62566347]
$ chmod 775 sireg-linux
$ mkdir output
$ ls -alF
total 61128
drwxrwxr-x 4 gitlab-runner gitlab-runner 4096 Apr 2 18:22 ./
drwxrwxr-x 10 gitlab-runner gitlab-runner 4096 Apr 2 15:46 ../
drwxrwxr-x 5 gitlab-runner gitlab-runner 4096 Apr 2 18:22 .git/
-rw-rw-r-- 1 gitlab-runner gitlab-runner 512 Apr 2 18:22 .gitlab-ci.yml
drwxrwxr-x 2 gitlab-runner gitlab-runner 4096 Apr 2 18:22 output/
-rw-rw-r-- 1 gitlab-runner gitlab-runner 30 Apr 2 15:46 README.md
-rwxrwxr-x 1 gitlab-runner gitlab-runner 62566347 Nov 11 2017 sireg-linux*
-rw-rw-r-- 1 gitlab-runner gitlab-runner 55 Apr 2 18:08 sitemap-index
$ while read line; do echo $line; ./sireg-linux **exec** --loader-sitemap-sitemap \"$line\" >>
./output/${line##*/}_out.txt; done < sitemap-index
$ ls -alF output
total 8
drwxrwxr-x 2 gitlab-runner gitlab-runner 4096 Apr 2 18:22 ./
drwxrwxr-x 4 gitlab-runner gitlab-runner 4096 Apr 2 18:22 ../
Uploading artifacts...
Runtime platform arch=amd64 os=linux pid=23813 revision=1f513601 version=11.10.1
WARNING: output/*: no matching files
ERROR: No files to upload
Job succeeded
update
tried to move all steps into a separate script but that did not work either.
update 2
forgot to add exec in the command:
./sireg-linux exec --loader-sitemap-sitemap \"$line\" >>
./output/${line##*/}_out.txt;
unfortunately it didn't help.
what can I do to make it working?
Try changing ./sireg-linux --loader-sitemap-sitemap \"$line\" to ./sireg-linux exec --loader-sitemap-sitemap "$line". Hope this helps!
EDIT: Also, it looks like the script doesn't enter the while loop at all. Maybe the file sitemap-index is empty or it has only one line without a newline at the end?
EDIT 2: The back-slashes in the command line are wrong. corrected my answer
You can of course painfully debug multi-line commands in YAML.
You can even use YAML multi-line strings:
How do I break a string over multiple lines?
https://gitlab.com/snippets/1717579
But I would just wrap code into a shell script, store it in the same GitLab repo, and just call it in .gitlab-ci.yml.
This way you can run this script exactly the same way both locally and in CI, which is a best practice in Continuous Delivery.
- ./script.sh
I have a directory which contains a lot of files with .sum extension.
I used the below script to list the contents of all the .sum files to a temp.log file. While the first .sum file gets written to temp.log the awk utility seems to give error for the remaining .sum files. Please help, what i am missing in here.
cd $HOME/aphp/result/${test}
for filename in *.sum
do
tempdir=$filename
awk '/Failed/' "${filename}" > temp.log
awk '/Error/' "${filename}" >> temp.log
if [ -s temp.log ]
then
mkdir -p ${scanresult}/${tempdir}
mv temp.log ${scanresult}/${tempdir}/temp.log
cd ${scanresult}/${tempdir}
mv temp.log ${tempdir}_failed.txt
else
echo Skipping ${tempdir} scanning as it is executed 100 percent with no fail or error.
rm temp.log
fi
done
Errors:
awk: fatal: cannot open file `dss154.sum' for reading (No such file or directory)
awk: fatal: cannot open file `dss235.sum' for reading (No such file or directory)
awk: fatal: cannot open file `dss287.sum' for reading (No such file or directory)
ls -l *.sum
-rwxrwxrwx 1 smruti smruti 1844 Mar 25 16:23 dss103.sum
-rwxrwxrwx 1 smruti smruti 2353 Mar 25 16:40 dss154.sum
-rwxrwxrwx 1 smruti smruti 1023 Mar 25 16:43 dss235.sum
-rwxrwxrwx 1 smruti smruti 908 Mar 25 16:45 dss287.sum
-rwxrwxrwx 1 smruti smruti 867 Mar 25 16:45 dss288.sum
-rwxrwxrwx 1 smruti smruti 1064 Mar 25 16:47 dss350.sum
You are getting into this problem due to this line:
cd ${scanresult}/${tempdir}
Which is changing your current working directory to something else. After which rest of the files cannot be read after 1st file. It is not really clear why are you changing directory inside the loop.
You can use this line to go back to original path:
cd -
However most of your code after awk command looks suspicious and redundant.
I have a directory with the following files
➜ hrrep git:(master) ✗ ls -l
total 1000
drwxrwxrwx 22 zmjones staff 748 May 28 23:28 data
-rw-rw-rw- 1 zmjones staff 7180 May 20 16:17 data.R
drwxrwxrwx 9 zmjones staff 306 May 28 23:38 figures
-rw-r--r-- 1 zmjones staff 85841 May 28 23:23 hill_jones_hr.bib
-rw-r--r-- 1 zmjones staff 29193 May 28 22:23 hill_jones_hr.tex
-rw-r--r-- 1 zmjones staff 572 May 28 23:46 makefile
-rw-rw-rw- 1 zmjones staff 9588 May 28 22:47 models.R
drwxrwxrwx 3 zmjones staff 102 May 28 23:28 papers
drwxrwxrwx 9 zmjones staff 306 May 28 23:28 tex
-rw-r--r-- 1 zmjones staff 1483 May 20 12:58 un_data.py
These files are described by a makefile (using GNU Make 3.81)
all: ./data/cat_un_data.csv ./data/rep.csv ./figures/*.png ./hj-hr.pdf
./data/cat_un_data.csv: un_data.py #refreshing one will refresh all
python un_data.py
./data/rep.csv: data.R ./data/cat_un_data.csv
R CMD BATCH --no-save --no-restore data.R
./figures/*.png: models.R ./data/rep.csv
R CMD BATCH --no-save --no-restore models.R
TEXMCD := pdflatex -interaction=batchmode
hill_jones_hr.pdf: hill_jones_hr.tex ./figures/*.png
$(TEXCMD) $<
]bibtex *.aux
$(TEXCMD) $<
$(TEXCMD) $<
find . | egrep ".*((\.(aux|log|blg|bbl|out|Rout|Rhistory|DS_Store))|~)$$" | xargs rm
rm -rf auto
The makefile seems to work fine until it gets to the hll_jones_hr.pdf target, where it fails with an error:
hill_jones_hr.tex
make: hill_jones_hr.tex: No such file or directory
make: *** [hill_jones_hr.pdf] Error 1
I don't understand what the problem is. The document compiles fine when I execute pdflatex and bibtex manually. I tried adding the ignore error flag to no avail. I presume this is some stupid make error that I am making. This folder is in Dropbox if that makes any difference. I also tried adding the ./ prefix to both the .tex file and the .pdf target; the error was the same.
The problem is that the $(TEXCMD) variable is not defined, so your command:
$(TEXCMD) $<
is expanding to just $< or just the filename, which is not a legal command by itself.
use ./hill_jones_hr.tex instead of hill_jones_hr.tex, and try again.
What does the command cp $1/. $2 do? I know cp is used for copying from source(stored in variable $1) to destination(stored in variable $2). I am just confused with the /. used along with the variable. Can someone please help me understand this?
The command:
$ cp -R $1/. $2
copies contents of directory pointed by $1 to the directory $2.
Without -R switch this command would fail both when $1 is a file or directory.
In general, . points to the current directory. You can see that by comparing inode's shown by ls:
$ mkdir test
$ ls -ali
9525121 drwxr-xr-x 3 IU wheel 102 23 mar 12:31 .
771046 drwxrwxrwt 21 root wheel 714 23 mar 12:30 ..
9525312 drwxr-xr-x 2 IU wheel 68 23 mar 12:31 test
$ cd test
$ ls -ali
9525312 drwxr-xr-x 2 IU wheel 68 23 mar 12:31 .
9525121 drwxr-xr-x 3 IU wheel 102 23 mar 12:31 ..
Note that inode 9525312 points to test when viewed from the parent directory, and points to . when viewed from inside the test directory.