I am trying to do variable interpolation inside a command substitution in a Makefile.
I have this code:
setup:
mkdir -p data_all ; \
for i in $(shell jq -r 'keys | #tsv' assets.json) ; do \
git_url=$(shell jq -r ".$$i" assets.json) ; \
git clone $$git_url data_all/$$i ; \
done
The code is failing, however, because $$i does not expand in the "shell" line that sets git_url.
How do I interpolate the variable $i in the "shell" line that sets git_url?
You mixed up make functions ($(shell ...)) and true shell constructs. When writing a recipe the simplest is to write it first in plain shell:
mkdir -p data_all ; \
for i in $( jq -r 'keys | #tsv' assets.json ) ; do \
git_url=$( jq -r ".$i" assets.json ) ; \
git clone $git_url data_all/$i ; \
done
And then escaping the unwanted $ expansion by make:
mkdir -p data_all ; \
for i in $$( jq -r 'keys | #tsv' assets.json ) ; do \
git_url=$$( jq -r ".$$i" assets.json ) ; \
git clone $$git_url data_all/$$i ; \
done
Related
fzf/ripgrep can search an alternate directory from the command line with something like.
rg something ./sompath | fzf
The fzf documentation has a nice little bash script for generating a preview of rg's results that I call with a shell alias and then open the file with vim:
#!/usr/bin/env bash
RG_PREFIX="rg --column --line-number --no-heading --color=always --smart-case "
INITIAL_QUERY="${*:-}"
IFS=: read -ra selected < <(
FZF_DEFAULT_COMMAND="$RG_PREFIX $(printf %q "$INITIAL_QUERY")" \
fzf --ansi \
--color "hl:-1:underline,hl+:-1:underline:reverse" \
--disabled --query "$INITIAL_QUERY" \
--bind "change:reload:sleep 0.1; $RG_PREFIX {q} || true" \
--bind "alt-enter:unbind(change,alt-enter)+change-prompt(2. fzf> )+enable-search+clear-query" \
--prompt '1. ripgrep> ' \
--delimiter : \
--preview "$BAT_CMD --color=always {1} --highlight-line {2}" \
--preview-window 'up,60%,border-bottom,+{2}+3/3,~3'
)
[ -n "${selected[0]}" ] && nvim "${selected[0]}" "+${selected[1]}"
It's very cool and works great. Unfortunately, there's no way to pass in a directory argument into this script. So you have to cd into the desired directory, do the search, and cd back.
Instead, I'd like to do something like this:
search_script initial_query ./some_dir
But my bash skills are weak and I'm not really sure what the best approach is for processing an optional directory argument.
The script has to somehow be smart enough to recognize when a directory argument is passed and when it isn't. I'm not sure if some kind of option string like --dir is the best way to go or what. And I'm wondering if I might be missing something really obvious solution, too.
Thanks.
I was able to cargo cult some little bash snippets and piece something that seems to do the trick and detect if last argument is a directory:
RG_PREFIX="rg --column --line-number --no-heading --color=always --smart-case "
INITIAL_QUERY="${*:-}"
last="${#: -1}"
if [[ -d $last ]]; then
INITIAL_QUERY="${#:1:$#-1}";
dir=$last
echo $INITIAL_QUERY
fi
IFS=: read -ra selected < <(
FZF_DEFAULT_COMMAND="$RG_PREFIX $(printf %q "$INITIAL_QUERY") $dir" \
fzf --ansi \
--color "hl:-1:underline,hl+:-1:underline:reverse" \
--disabled --query "$INITIAL_QUERY" \
--bind "change:reload:sleep 0.1; $RG_PREFIX {q} $dir || true" \
--bind "alt-enter:unbind(change,alt-enter)+change-prompt(2. fzf> )+enable-search+clear-query" \
--prompt '1. ripgrep> ' \
--delimiter : \
--preview "bat --color=always {1} --highlight-line {2}" \
--preview-window 'up,60%,border-bottom,+{2}+3/3,~3'
)
[ -n "${selected[0]}" ] && nvim "${selected[0]}" "+${selected[1]}"
If you think there is a simpler way or a better, I'd love to hear.
This is my target, where i want to run a loop for each element in the list variable.
The problem is the loop runs but the test variable value is passed as empty
list = mlflow emr
common=$(someDir)/common
.PHONY:build
build:
for var in $(list); do \
cd ${common}; \
test=$(git diff --name-only --diff-filter=AM master | grep ^$(var)/); \
if [ "$(test)" != "" ]; then \
echo "condition met"; \
else \
echo "It is Not Setup"; \
fi \
done
Error:
bash-5.0# sudo make build n=1
for var in mlflow emr; do \
cd /mak/epa-toolchain/common; \
test=; \
if [ "" != "" ]; then \
echo "condition met"; \
else \
echo "It is Not Setup"; \
fi \
done
It is Not Setup
It is Not Setup
The $ is a special character to make: it introduces a make variable reference. So this:
$(git diff --name-only --diff-filter=AM master | grep ^$(var)/)
is not a shell $(...) command, it's a make variable with a very strange name. Wherever you want the shell to see $ you have to escape it as $$:
$$(git diff --name-only --diff-filter=AM master | grep ^$$var/)
(note you have to change $(var) to $$var because the former is a reference to a make variable var, but you are looping in the shell which sets the shell variable var).
Ditto this:
if [ "$(test)" != "" ]; then \
has to be:
if [ "$$test" != "" ]; then \
because test is a shell variable you just assigned, not a make variable.
I got this script which reads a delimited part of my .gitignore file and remove all files after the given mark # #:
# https://stackoverflow.com/questions/55527923/how-to-stop-makefile-from-expanding-my-shell-output
RAW_GITIGNORE_CONTENTS := $(shell while read -r line; do printf "$$line "; done < ".gitignore")
GITIGNORE_CONTENTS := $(shell echo "$(RAW_GITIGNORE_CONTENTS)" | sed -E $$'s/[^\#]+\# //g')
# https://stackoverflow.com/questions/4210042/exclude-directory-from-find-command
DIRECTORIES_TO_CLEAN := $(shell /bin/find -not -path "./**.git**" -not -path "./pictures**" -type d)
clean:
# https://stackoverflow.com/questions/10586153/split-string-into-an-array-in-bash
# https://stackoverflow.com/questions/11289551/argument-list-too-long-error-for-rm-cp-mv-commands
readarray -td' ' GARBAGE_DIRECTORIES <<<"$(DIRECTORIES_TO_CLEAN) "; \
unset 'GARBAGE_DIRECTORIES[-1]'; \
declare -p GARBAGE_DIRECTORIES; \
readarray -td' ' GARBAGE_EXTENSIONS <<<"$(GITIGNORE_CONTENTS) "; \
unset 'GARBAGE_EXTENSIONS[-1]'; \
declare -p GARBAGE_EXTENSIONS; \
for filename in "$${GARBAGE_DIRECTORIES[#]}"; \
do \
arraylength="$${#GARBAGE_EXTENSIONS[#]}"; \
printf 'Cleaning %s extensions on %s\n' "$${arraylength}" "$$filename"; \
for extension in "$${GARBAGE_EXTENSIONS[#]}"; \
do \
[[ ! -z "$$filename" ]] || continue; \
[[ ! -z "$$extension" ]] || continue; \
full_expression="$${filename}/$${extension}" ;\
printf '%s\n' "$$full_expression"; \
rm -v "$$full_expression"; \
done; \
done;
Running it with the following .gitignore file:
*.txt
*.var
# Comment #
*.aux
The rm command is not expanding the wildcards and keeps telling me rm: cannot remove './*.aux': No such file or directory and do not remove the *.aux files from the ./ directory.
Update
After asked on a comment by #Beta, I simplified the Makefile to this:
GITIGNORE_CONTENTS := "*.aux" "*.lof"
DIRECTORIES_TO_CLEAN := "./setup/cache" "./setup/cache/chapters"
clean:
readarray -td' ' GARBAGE_DIRECTORIES <<<"$(DIRECTORIES_TO_CLEAN) "; \
unset 'GARBAGE_DIRECTORIES[-1]'; \
declare -p GARBAGE_DIRECTORIES; \
readarray -td' ' GARBAGE_EXTENSIONS <<<"$(GITIGNORE_CONTENTS) "; \
unset 'GARBAGE_EXTENSIONS[-1]'; \
declare -p GARBAGE_EXTENSIONS; \
for filename in "$${GARBAGE_DIRECTORIES[#]}"; \
do \
arraylength="$${#GARBAGE_EXTENSIONS[#]}"; \
printf 'Cleaning %s extensions on %s\n' "$${arraylength}" "$$filename"; \
for extension in "$${GARBAGE_EXTENSIONS[#]}"; \
do \
[[ ! -z "$$filename" ]] || continue; \
[[ ! -z "$$extension" ]] || continue; \
full_expression="$${filename}/$${extension}" ;\
printf '%s\n' "$$full_expression"; \
rm -vf "$$full_expression"; \
done; \
done;
Which results on this output after running it:
$ make
readarray -td' ' GARBAGE_DIRECTORIES <<<""./setup/cache" "./setup/cache/chapters" "; \
unset 'GARBAGE_DIRECTORIES[-1]'; \
declare -p GARBAGE_DIRECTORIES; \
readarray -td' ' GARBAGE_EXTENSIONS <<<""*.aux" "*.lof" "; \
unset 'GARBAGE_EXTENSIONS[-1]'; \
declare -p GARBAGE_EXTENSIONS; \
for filename in "${GARBAGE_DIRECTORIES[#]}"; \
do \
arraylength="${#GARBAGE_EXTENSIONS[#]}"; \
printf 'Cleaning %s extensions on %s\n' "${arraylength}" "$filename"; \
for extension in "${GARBAGE_EXTENSIONS[#]}"; \
do \
[[ ! -z "$filename" ]] || continue; \
[[ ! -z "$extension" ]] || continue; \
full_expression="${filename}/${extension}" ;\
printf '%s\n' "$full_expression"; \
rm -vf "$full_expression"; \
done; \
done;
declare -a GARBAGE_DIRECTORIES=([0]="./setup/cache" [1]="./setup/cache/chapters")
declare -a GARBAGE_EXTENSIONS=([0]="*.aux" [1]="*.lof")
Cleaning 2 extensions on ./setup/cache
./setup/cache/*.aux
./setup/cache/*.lof
Cleaning 2 extensions on ./setup/cache/chapters
./setup/cache/chapters/*.aux
./setup/cache/chapters/*.lof
More simplification
I reduced to the more simple version it could be:
clean:
rm -v "./setup/cache/*.aux";
Running this, also do not remove the files:
$ make
rm -v "./setup/cache/*.aux";
rm: cannot remove './setup/cache/*.aux': No such file or directory
make: *** [Makefile:3: clean] Error 1
$ ls ./setup/cache/*.aux
./setup/cache/main.aux
On above, after running ls, you can see the file still exists and it is there.
I managed to fix it by changing:
rm -vf "$$full_expression"; \
To:
rm -vf $${full_expression}; \
I am using bash to call tool written in java (gatk) and I need to pass multiple arguments from array as input arguments. I tried it this way, but it seems not working. Could you please help me, how to solve it?
Code:
java $GATK \
-T GenotypeGVCFs \
-R $ref \
-o output.vcf \
for foo in array
do
--variant $foo \
done
What i want to be called:
java $GATK \
-T GenotypeGVCFs \
-R $ref \
-o output.vcf \
for foo in array
do
--variant file1 \
--variant file2 \
--variant file3 ...etc
done
edit: sorry for misunderstandings
array=("file1","file2","file3"...)
Thanks
I assume that what you actually want is that if array contains a b c, to have the command
java $GATK \
-T GenotypeGVCFs \
-R $ref \
-o output.vcf \
--variant a --variant b --variant c
If that is so, you can prepare a second array:
array=("file 1" "file 2" "file 3")
declare -a fullarray
for i in "${array[#]}"
do
fullarray+=( --variant "$i" )
done
And then
java $GATK \
-T GenotypeGVCFs \
-R $ref \
-o output.vcf \
"${fullarray[#]}"
This will also make sure that if any of the names in array contains a space, it will still be passed as a proper parameter and not split into two (assuming that you didn't mess it up when you added it to the array).
With echo and $():
array=(file1 file2 file3)
java $GATK \
-T GenotypeGVCFs \
-R $ref \
-o output.vcf \
$(for foo in ${array[*]}
do
echo -n " --variant $foo"
done)
You can do this with the following:
java $GATK \
-T GenotypeGVCFs \
-R $ref \
-o output.vcf \
${array[*]/#/ --variant }
#RealSkeptic's answer is the best. I'd write, for readability:
array=( "file 1" "file 2" "file 3" )
args=(
"$GATK"
-T GenotypeGVCFs
-R "$ref"
-o output.vcf
)
for foo in "${array[#]}"; do args+=( --variant "$foo" ); done
java "${args[#]}"
I'm having difficulties with makefiles.
So in a recipe, I'm making a file (with a name and a .ujc extension) in a for loop and would like to have a text file at the end which contains all the created files. Purpose is to feed it to an application.
For example, in a semi high-level example,
List= [Class1,Class2,Class3]
foreach(Class C in List) {
#do operations on C > outputs a ClassX.ujc file
# add name of file to a text file named "list_of_files"
}
At the end I should have a text file, list_of_files.txt, which contains the following string:
Class1.ujc Class2.ujc Class3.ujc
As a reference, the code I have at the moment (and which does a bit of the stuff above but does not work is) is:
pc: $(APP)
$(foreach C, $(shell echo $(CLASS) | tr ',' ' '), \
make -C BUILDENV CLASS=$(C) BUILD=just_filelist OUTPUT=filelist.txt SKIPSELF=yes && \
../classCvt/classCvt <./Applications/$(C).class> ./Applications/$(C).ujc && \
cat app_file_list.txt | xargs echo ./Applications/$(C).ujc >app_file_list.txt && \
) true
time -p ./$(APP) `cat app_file_list.txt` `cat filelist.txt`
The internal make does make a filelist which is fed to the app, but I'd also like to feed the app_file_list but its construction goes totally wrong.
Probably simple, but I'm not getting there.
Edit:
The code below does what I want:
pc: $(APP)
rm -f cat app_file_list.txt
$(foreach C, $(shell echo $(CLASS) | tr ',' ' '), \
make -C BUILDENV CLASS=$(C) BUILD=just_filelist OUTPUT=filelist.txt SKIPSELF=yes && \
../classCvt/classCvt <./Applications/$(C).class> ./Applications/$(C).ujc && \
cat app_file_list.txt | echo ./Applications/$(C).ujc >>app_file_list.txt && \
) true
time -p ./$(APP) `cat app_file_list.txt` `cat filelist.txt`
Notable mistake I made was the xargs.
(Also in the post)
The solution turned out to be not-so-difficult. I needed to remove the xargs command and do the correct operation (i.e., >> instead of >) in the 'cat app_file_list.txt | etc...' line.
The code below does what I want:
pc: $(APP)
rm -f cat app_file_list.txt
$(foreach C, $(shell echo $(CLASS) | tr ',' ' '), \
make -C BUILDENV CLASS=$(C) BUILD=just_filelist OUTPUT=filelist.txt SKIPSELF=yes && \
../classCvt/classCvt <./Applications/$(C).class> ./Applications/$(C).ujc && \
cat app_file_list.txt | echo ./Applications/$(C).ujc >>app_file_list.txt && \
) true
time -p ./$(APP) `cat app_file_list.txt` `cat filelist.txt`
Notable mistake I made was the xargs which caused strings to repeat into the .txt file.