In my Makefile I want to execute multiple steps as a part of a single target. Those steps should be done sequentially, because they depend on one another. This is the simplified case:
target:
git archive --remote=some-remote master --format zip -o ./zipfile.zip
echo "$(VARIABLE_IN_MAKE):$(shell unzip -z ./zipfilezip | tail -1)" > ./textfile
$(shell cat ./textfile)
The problem here is that the shell command - $(shell unzip -z ./zipfilezip | tail -1) is executed as soon as the rule is "loaded", i.e. before the zipfile even exists. That returns errors. The cat command is also expanded too early.
What is the correct way to execute a subshell not before, but only after all the steps above have finished? Do I have to wrap all the commands in a bash -c call? Or chain them via &&?
Get rid of these $(shell...). Each line of a make recipe is already a shell script:
target:
git archive --remote=some-remote master --format zip -o ./zipfile.zip
echo "$(VARIABLE_IN_MAKE):$$(unzip -z ./zipfilezip | tail -1)" > ./textfile
cat ./textfile
Note that, in the second line, $$(unzip -z ./zipfilezip | tail -1) is expanded twice: a first time by make before passing the recipe to the shell, leading to $(unzip -z ./zipfilezip | tail -1), and a second time by the shell that treats it as a command substitution. This is why the $$ is needed: to escape the first expansion by make. If you were using $(unzip -z ./zipfilezip | tail -1) directly, make would expand it as the empty string (unless you have a make variable which name is unzip -z ./zipfilezip | tail -1, but this is very unlikely).
Related
I would like to convert and execute
if egrep -r 'my_pattern' ./template_builder
then exit 1
elif egrep -r 'my_second_pattern' ./template_builder
then exit 1
fi
in a Makefile, without success for now.
To build this:
cd /tmp;
mkdir template_builder;
echo "not_pattern" >> ./template_builder/test.txt
# Do the command at the top, nothing happens
echo "my_pattern" >> ./template_builder/test.txt
# Do the command at the top, terminal stops
touch Makefile
In a Makefile, I thought this would work :
check:
if egrep -r 'my_pattern' ./template_builder
then exit 1
elif egrep -r 'my_second_pattern' ./template_builder
then exit 1
fi
make check
if egrep -r 'my_pattern' ./template_builder
/bin/sh: -c: line 1: syntax error: unexpected end of file
make: *** [template] Error 2
How can I fix this?
Your attempt was not far from working!
Add backslashes at the end of every line, and ;s as explicit command separators (and of course use real tabs instead of the 8-space indents below):
check:
if egrep -r 'my_pattern' ./template_builder; \
then exit 1; \
elif egrep -r 'my_second_pattern' ./template_builder; \
then exit 1; \
fi
If I understand you correctly, if the directory template_builder located in /tmp does not contain a file matching the string 'my_pattern' or 'my_second_pattern', you want to exit from make with an error code.
You can achieve this with this rule in Makefile:
check:
egrep -r -v 'my_pattern' /tmp/template_builder || egrep -r -v 'my_second_pattern' /tmp/template_builder
Explanation: the first egrep is going to return an error in the case he finds a match. Due to the presence of the || operator, the second egrep will be invoked. The result of this second command will be the result that make will see. If it returns an error, the execution of make is aborted, which seems to be the behaviour you are expecting.
Caution: I edited my answer. The right boolean operator is || and not &&.
As others have already noted, make runs each separate line in a recipe in a new shell subprocess. (For the record, it uses sh out of the box, not Bash.) The trivial fix is to add a backslash to escape the newline at the end of each line which should be executed in the same shell as the next one. (You need to add a semicolon as well in some places, like before then and else and fi.) But you really want to refactor to use the facilities and idioms of make.
The default logic of make is to terminate the recipe if any line fails. So, your code can be reduced to simply
check: template_builder
! egrep -r 'my_pattern' $<
! egrep -r 'my_second_pattern' $<
The explicit exit 1 is not necessary here (negating a zero exit code produces exactly that); but if you wanted to force a particular exit code, you could do that with
egrep -r 'my_pattern' $< && exit 123 || true
Modern POSIX prefers grep -E over legacy egrep; of course, with these simple patterns, you can use just grep, or even grep -F (née fgrep).
Moreover, if you want to search for both patterns in the same set of files, it's much more efficient to search for them both at once.
check: template_builder
! egrep -e 'my_pattern' -e 'my_second_pattern' -r $<
... or combine them into a single regex my_(second_)?pattern (which requires egrep / grep -E).
Notice also how I factored out the dependency into $< and made it explicit; but you probably want to make this recipe .PHONY anyway, so that it gets executed even if nothing has changed.
(You can't directly copy/paste this code, because Stack Overflow stupidly renders the literal tabs in the markdown source as spaces.)
I found this exercise that my professor proposed in an old exam.
"Write a script in bash that matches a string to a directory passed as a function argument so that running the script on the same directory again makes the string the same if the directory contents have not changed.
So, i thought about compacting the directory into an archive, and extracting its md5 digest. The problem is that if you re-run the script, the digest changes. Why?
I enclose my attempt.
tar -cvzf archive.tgz $1 >> /dev/null;`
openssl dgst -md5 archive.tgz`
Pathname has relevance
First, make the filename canonical. Otherwise the same directory throws different strings when called with different pathnames. The pathname itself becomes part of "same". Then prof dir
dir="( readlink -m "$1" )
[[ -d $dir ]] || exit 1
Now build a checksum and output the result
tar cf - "$dir" | sha1sum | cut -d\ -f1
Any compression options like -j slows down the process, but doesn't help. The cut command removed the leading filename of sha1sum.
Pathname has no relevance
If the pathname has nor relevance, change the script to:
#!/usr/env bash
[[ -d $1 ]] || exit 1
tar cf - -C "$dir" . | sha1sum | cut -d\ -f1
How do you use a command line argument as a file path and check for file existence in Bash?
I have the simple Bash script test.sh:
#!/bin/bash
set -e
echo "arg1=$1"
if [ ! -f "$1" ]
then
echo "File $1 does not exist."
exit 1
fi
echo "File exists!"
and in the same directory, I have a data folder containing stuff.txt.
If I run ./test.sh data/stuff.txt I see the expected output:
arg1=data/stuff.txt
"File exists!"
However, if I call this script from a second script test2.sh, in the same directory, like:
#!/bin/bash
fn="data/stuff.txt"
./test.sh $fn
I get the mangled output:
arg1=data/stuff.txt
does not exist
Why does the call work when I run it manually from a terminal, but not when I run it through another Bash script, even though both are receiving the same file path? What am I doing wrong?
Edit: The filename does not have spaces. Both scripts are executable. I'm running this on Ubuntu 18.04.
The filename was getting an extra whitespace character added to it as a result of how I was retrieving it in my second script. I didn't note this in my question, but I was retrieving the filename from folder list over SSH, like:
fn=$(ssh -t "cd /project/; ls -t data | head -n1" | head -n1)
Essentially, I wanted to get the filename of the most recent file in a directory on a remote server. Apparently, head includes the trailing newline character. I fixed it by changing it to:
fn=$(ssh -t "cd /project/; ls -t data | head -n1" | head -n1 | tr -d '\n' | tr -d '\r')
Thanks to #bigdataolddriver for hinting at the problem likely being an extra character.
This question already has an answer here:
How do I expand commands within a bash alias?
(1 answer)
Closed 4 years ago.
I've been using cd "$(\ls -1dt ./*/ | head -n 1)" in some scripts to get into a new directory after creating it. I decided to put an alias in my bash_profile:
alias newest="cd $(\ls -1dt ./*/ | head -n 1)"
But when I run newest from the command line, it goes to a different directory, which happens to be the first one alphabetically though I don't know if that's why it's choosing that directory.
Pasting cd "$(\ls -1dt ./*/ | head -n 1)" directly into the command line works correctly. What's going on here?
Don't use ls -t in scripts at all -- see ParsingLs on why it's unreliable, and BashFAQ #3 on what to do instead. But ignoring that, the smallest fix for the immediate, narrow issue is to use a function:
newest() { cd "$(command ls -1dt ./*/ | head -n 1)"; }
Your alias was having the command substitution run at time of definition, not on invocation. If you really want it to still be an alias, you could use single quotes on the outside to prevent that command substitution from happening early:
alias newest='cd "$(\ls -1dt ./*/ | head -n 1)"'
What would a reliable, best-practice approach look like? Perhaps:
cdNewest() {
local latest='' candidate
set -- */
[[ -d $1 ]] || return # handle case where no directories exist so glob did not expand
latest=$1; shift
for candidate; do
[[ $candidate -nt $latest ]] && latest=$candidate
done
cd -- "$latest"
}
...which, instead of running two external commands (ls and head), runs none at all (and also avoids the need for command substitutions and pipelines, both of which are quite high-overhead, altogether).
Here's a Makefile
roman#debian ~/D/O/devops> cat Makefile
install:
-cat projects.txt | xargs -n 2 bash -c 'git clone $0 $1'
Here's projects.txt
roman#debian ~/D/O/devops> cat projects.txt
git#github.com:xxx/xxx1.git app-xxx1
git#github.com:xxx/xxx2.git app-xxx2
Here's what happens when I just copy this command to bash - it works:
roman#debian ~/D/O/devops> cat projects.txt | xargs -n 2 bash -c 'git clone $0 $1'
fatal: destination path 'app-xxx1' already exists and is not an empty directory.
It's using git clone properly it's just repo exists.
Now when you do make install this fails, all variables are blank:
roman#debian ~/D/O/devops> make install
cat projects.txt | xargs -n 2 bash -c 'git clone '
You must specify a repository to clone.
I'd like to use only xargs method in here, else it becomes too wordy, also there's even more problems when using loops. I've also tried to use $(1) but no luck
make is interpreting $0 and $1 as make variables and trying to expand them. Try replacing $ with $$...
install:
-cat projects.txt | xargs -n 2 bash -c 'git clone $$0 $$1'
Use -I {} to read lines and use that for the input inside Makefile
install:
-cat projects.txt | xargs -I '{}' git clone '{}'
make expands dollar sign references to be its own variables. While it is usual to write $(name), it is possible to write $c where c is a single character (as long as it is not a $), so when you use $0 and $1 in your command line, they will be substituted with whatever values those variables have inside make. (make does not know anything about bash quoting, so $-references will be expanded anywhere in the line.)
Since make does not automatically define $0 and $1 and it is unlikely that you have defined them in your makefile, the most likely is that they will be substituted with the empty string, resulting in the error you observe. To avoid this problem, escape the dollar signs by doubling them:
-cat projects.txt | xargs -n 2 bash -c 'git clone $$0 $$1'
Having said that, it is not at all clear to me why you want to use bash -c here. What's wrong with
xargs -n2 git clone < projects.txt
Worked for me when I used two $.
$ cat Makefile
install:
cat projects.txt | xargs -n 2 bash -c 'echo git clone $$0 $$1'
$ make install
cat projects.txt | xargs -n 2 bash -c 'echo git clone $0 $1'
git clone git#github.com:xxx/xxx1.git app-xxx1
git clone git#github.com:xxx/xxx2.git app-xxx2