I am trying to run 'make' command from my bash script to build the code.
I can see that all parameters got assigned and able to display the command that i am trying to run. I could not see any issue with the command. But the issue is when it tries to run the command via bash script it fails.
My command is :- ./build_script LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm"
Script Snippet of parsing all the arguments and constructing make command:-
makeargs=""
for build_flag do
if [[ "$build_flag " = "PROJECTS="* ]]; then
apps =`echo $build_flag |sed "s/PROJECTS=//" `
makeargs="$makeargs PROJECTS=\"$apps \""
else
makeargs="$makeargs $build_flag"
fi
done
echo "make DCOV=1 $makeargs $maketest"
make DCOV=1 $makeargs $maketest
When i run the script, I can see the build command has constructed properly.
Output :-
make DCOV=1 LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm" run_all
GNUmakefile:16: warning: overriding commands for target `/'
GNUmakefile:19: warning: ignoring old commands for target `/'
make: *** No rule to make target `bfm"'. Stop.
I try to print PROJECTS variable in my 'GNUmakefile' present in build_main folder. I can see the output : PROJECTS is "bfm . It is not taking whole "cuda bfm" as a whole string.
Note:- When i try to run the same build command :- make DCOV=1 LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm" run_all explicitly it works fine.
Seems like issue with Interpreting variables with makefile.
Any solution for this ? Please help.
Thanks!
Change makeargs string to array before passing it as an arguments group.
eval makeargs_array=( $makeargs )
make UVC=1 "${makeargs_array[#]}" $maketest
Without converting to array, if you enable debug, it shows last line interpretation as
make DCOV=1 LIC=1 DOCUMNETS=1 'PROJECTS="cuda' bfm '"'
Which is clearly ignoring double-quote and considering space as separator.
Even double-quote is getting passed as a separate argument in this case.
Explanation:
Word-splitting
It says,
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
If we use "$makeargs" i.e. surrounded by double-quote, it is not considered by word-splitting and results in LIC=1 DOCUMNETS=1 "PROJECTS=cuda bfm"
But again its a complete string, while requirement is to split the string to use as arguments.
So now using $makeargs.
Word-splitting gets in action as per the default IFS (space, tab, newline), we get result as LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm "
Double-quoted part of string didn't affect the word-splitting since, subject to splitting is complete string here.
Why array worked here?
Array itself expands each element as separate word when using # and here no further word-splitting requires after expansion.
Arrays
Related
I have a Bash script in which I call rsync in order to perform a backup to a remote server. To specify that my Downloads folder be backed up, I'm passing "'${HOME}/Downloads'" as an argument to rsync which produces the output:
rsync -avu '/Volumes/Norman Data/Downloads' me#example.com:backup/
Running the command with the variable expanded as above (through the terminal or in the script) works fine, but because of the space in the expanded variable and the fact that the quotes (single ticks) are ignored when included in the variable being passed as part of an argument (see here), the only way I can get it not to choke on the space is to do:
stmt="rsync -avu '${HOME}/Downloads' me#examle.com:backup/"
eval ${stmt}
It seems like there would be some vulnerabilities presented by running eval on anything not 100% private to that script. Am I correct in thinking I should be doing it a different way? If so, any hints for a bash-script-beginner would be greatly appreciated.
** EDIT ** - I actually have a bit more involved use case than. the example above. For the paths passed, I have an array of them, each containing spaces, that I'm then combining into 1 string kind of like
include_paths=(
"'${HOME}/dir_a'"
"'${HOME}/dir_b' --exclude=video"
)
for item in "${include_paths[#]}"
do
inc_args="${inc_args}" ${item}
done
inc_args evaluates to '/Volumes/Norman Data/me/dir_a' '/Volumes/Norman Data/me/dir_b' --exclude=video
which I then try to pass as an argument to rsync but the single ticks are read as literals and it breaks after the 1st /Volumes/Norman because of the space.
rsync -avu "${inc_args}" me#example.com:backup/
Using eval seems to read the single ticks as quotes and executes:
rsync -avu '/Volumes/Norman Data/me/dir_a' '/Volumes/Norman Data/me/dir_b' --exclude=video me#example.com:backup/
like I need it to. I can't seem to get any other way to work.
** EDIT 2 - SOLUTION **
So the 1st thing I needed to do was modify the include_paths array to:
remove single ticks from within double quoted items
move any path-specific flags (ex. --exclude) to their own items directly after the path it should apply to
I then built up an array containing the rsync command and its options, added the expanded include_paths and exclude_paths arrays and the connection string to the remote host.
And finally expanded that array, which ran my entire, properly quoted rsync command. In the end the modified array include_paths is:
include_paths=(
"${HOME}/dir_a"
"${HOME}/dir_b"
"--exclude=video"
"${HOME}/dir_c"
)
and I put everything together with:
cmd=(rsync -auvzP)
for item in "${exclude_paths[#]}"
do
cmd+=("--exclude=${item}")
done
for item in "${include_paths[#]}"
do
cmd+=("${item}")
done
cmd+=("me#example.com:backup/")
set -x
"${cmd[#]}"
Use an array for the commands/option instead of a plain variable.
stmt=(rsync -avu "${HOME}/Dowloads" me#example.com:backup/)
Execute it using the builtin command
command "${stmt[#]}"
...Or I personally just put the options/arguments in an array.
options=(-avu "${HOME}/Download" me#example.com:backup/)
The execute it using rsync
rsync "${options[#]}"
If you have newer version of bash which that supports the additional P.E. parameter expansion, then you could probably quote the array.
options=(-avu "${HOME}/Download" me#example.com:backup/)
Check the output by applying the P.E.
echo "${options[#]#Q}"
Should print
'-avu' '/Volumes/Norman Data/Downloads' 'me#examle.com:backup/'
Then you can just
rsync "${options[#]#Q}"
I need to output the first X characters of the content of OLD_ENTRY, say 33 chars. I grab the number of chars with another script. What is, in the following command in a Windows cmd script, the correct syntax to use a variable, say POS, instead of the hardcoded value 33?
echo %OLD_ENTRY:~0,33%
Thanks for any help,
Rip
Alright, I tend to call something like this as "nested variables". Anyway, to expand such nested variables, you need to establish a second parsing or expansion phase, and you need to ensure that the inner variable (POS) becomes expanded first, and the outer one (OLD_ENTRY) becomes expanded during the second phase. There are some options:
Using call:
This option avoids delayed variable expansion, which could be problematic with literal ! symbols, but it is quite slow, and it doubles quoted ^ characters:
In command prompt window:
call echo %^OLD_ENTRY:~0,%POS%%
This looks like "escaping" (^) the outer variable, but actually, this has got nothing to do with true escaping. In command prompt, an undefined variable does not become replaced by an empty string, it is just kept literally. So in the first pass, the undefined variable ^OLD_ENTRY is simply kept (you can verify that by defining such a variable by set "^OLD_ENTRY=something"), scanning for the closing % is skipped after the : for undefined variables strangely, the variable %POS% becomes expanded, and the last (orphaned) % is kept too; immediately after this phase, the escape sequence ^O is recognised, which results in a literal O; so we get echo %OLD_ENTRY:~0,33%, which becomes expanded in the second pass.
In a batch file:
call echo %%OLD_ENTRY:~0,%POS%%%
In the first pass, the first two consecutive percent symbols become replaced by one literal % sign, the variable %POS% becomes expanded, and the remaining two consecutive percent symbols become replaced by one literal % sign too, so we have echo %OLD_ENTRY:~0,33%, which becomes expanded in the second pass.
Using delayed variable expansion:
This is the better option, I think, because it is faster and does not mess around with ^:
echo !OLD_ENTRY:~0,%POS%!
This option works in both command prompt window and batch files. Here the first pass is the normal/immediate expanssion (%) which handles the %POS% variable, so the second pass the delayed expansion (!) receives echo !OLD_ENTRY:~0,33! to expand.
Refer also to this post: How does the Windows Command Interpreter (CMD.EXE) parse scripts?
I am trying to write a Makefile for GNU make. I can't figure out what the problem is here:
foo := this|works
bar := "I lost my 'single quotes'"
baz := 'make|will|not|accept|this|with|the|single|quotes'
whatIWant := "A string with its 'single|quoted|regex|alternatives'"
this-almost-works: #But the single quotes are lost.
#printf '%s ' "$(whatIWant)"
this-fails-horribly:
#printf '$(whatIWant)'
I get the following error message
/bin/sh: 1: quoted: not found
/bin/sh: 1: /bin/sh: 1: regex: not foundalternatives": not found
blah blah Error 127
Why is it trying to run parts of this string in the shell?
How can I define a variable to contain exactly the contents of whatIWant?
Might be worth looking in detail at the expansion.
When defining variables,
just about the only character that has an effect is $.
Everything else is taken literally.
It's worth nothing that white space around the assignment operator (= or :=) is ignored.
foo := this|works
foo is assigned the literal text this|works.
Similarly,
baz := 'make|will|not|accept|this|with|the|single|quotes'
assigns the literal text 'make|will|not|accept|this|with|the|single|quotes' to baz.
Fine and dandy.
Now, when make decides to build this-fails-horribly
(possibly because you said to the shell make this-fails-horribly)
it expands the block of commands before doing anything.
Not unreasonably,
$(whatIWant) is replaced by "A string with its 'single|quoted|regex|alternatives'".
Again, fine and dandy.
What is left is passed verbatim, one line at a time, to the shell.
The shell sees
printf '"A string with its 'single|quoted|regex|alternatives'"'
(which make would have helpfully echoed to you if you had left off the # prefix).
Now we are in the land of shell quoting.
The printf command is passed one parameter: "A string with its single:
'"A string with its ' is a single quoted string. The shell strips the 's and is left with the text "A string with its.
single has no metacharacters in it, so the shell leaves this alone.
The output is piped to the quoted command
The output is piped to the regex command
The output is piped to the alternatives" command
The shell sees the single quoted string '=', strips the quotes leaving you with a literal = which it appends to the word alternatives
No syntax error.
When the shell attempts to set up the pipeline it looks for the alternatives" command.
It doesn't find one in the directories it its $PATH, so it stops with the message /bin/sh: 1: /bin/sh: 1: regex: not foundalternatives": not found.
One possible encoding:
.PHONY: this-workes-nicely
this-workes-nicely:
echo $(whatIWant)
though you'll probably find it's cleaner to leave the quotes outside the variable definition in the first place.
I read this question: Makefile: $subst in dependency list, but I still can't make my shell script work correctly.
I have a makefile with a line with the contents:
##public_detailed#|test_create|Syntax: commoncmdsyntax test_create test_name=<test-name>
A target runs a multiline bash script, where the commoncmdsyntax must be replaced by a string containing words and spaces.
In the script, I use cut to assign to a variable desc the following string:
Syntax: commoncmdsyntax test_create test_name=<test-name>
The problem is that commoncmdsyntax is not replaced by new text here:
$(subst commoncmdsyntax,new text,$$desc)
I also tried to replace it by a single word, like XX, but it also does not work.
The subst function (as in $(subst commoncmdsyntax,new text,$$desc)) is a Make function, so Make will perform the substitution before running any rule and therefore before your script assigns a value to desc. So even if secondary expansion worked the way you seem to think it will, this approach would still fail.
If you want to perform a substitution within something made by a shell script (in a recipe), the sensible way is to do so within the recipe:
echo $dest | sed 's/commoncmdsyntax/new text/'
We can give you a more detailed solution if you give us a minimal complete example of the problem.
I am working on a shell script to retrieve variable content from a JSON file via JQ. The JSON file is in string format (no matter whether this is a real string or a number) and to retrieve the variable in my bash script I did something like this
my_domain=$(cat /vagrant/data_bags/config.json | jq ."app"[0]."domain")
The above code once echoed results in "mydomain" with a beginning and a trailing quote sign. I though this was a normal behaviour of the echo command. However, while concatenating my variable with another shell command the system raise an error. For instance, the following command
cp /vagrant/public_html/index.php "/var/www/"+$my_domain+"/index.php"
fails with the following error
cp: cannot create regular file `/var/www/+"mydomain"+/index.php': No such file or directory
At this stage, I wasn't able to identify whether it's me doing the wrong concatenation with the plus sign or the variable is effectively including the quotes that in any case will end up generating an error.
I have tried to replace the quotes in my variable, but I ended up getting the system raising a "Command not found" error.
Can somebody suggest what am I doing wrong?
+ is not used for string concatenation in bash (or perl, or php). Just:
cp /vagrant/public_html/index.php "/var/www/$my_domain/index.php"
Embedding a variable inside a double-quoted text string is known as interpolation, and is one of the reasons why we need the $ prefix, to indicate that this is a variable. Interpolation is specifically not done inside single quoted strings.
Braces ${my_domain} are not required because the / directory separators are not valid characters in a variable name, so there is no ambiguity.
For example:
var='thing'
echo "Give me your ${var}s" # Correct, appends an 's' after 'thing'
echo "Give me your $vars" # incorrect, looks for a variable called vars.
If a variable (like 'vars') does not exist then (by default) it will not complain, it will just give an empty string. Braces (graph brackets) are required more in c-shell (csh or tcsh) because of additional syntax for modifying variables, which involves special trailing characters.
You don't need to use + to concatenate string in bash, change your command to
cp /vagrant/public_html/index.php "/var/www/"${my_domain}"/index.php"
My problem was not related only to the wrong concatenation, but also to the JQ library that after parsing the value from the JSon file was returning text between quotes.
In order to avoid JQ doing this, just add the -rawoutput parameter when calling JQ.