This question already has answers here:
How do you pass on filenames to other programs correctly in bash scripts?
(3 answers)
Closed 7 years ago.
I am attempting to parse the parameters sent to shell script. For example the values sent to the script are as follows:
-to someone#somewhere.com -a file1.txt file2.txt "new file.txt"
I can parse the string so that I get -a as my operator, but I want to reformat the parameter part file1.txt file2.txt "new file.txt" so that it looks like 'file1.txt' 'file2.txt' 'new file.txt' so that I can pass it down to the zip utility.
Right now I am using the following to parse the parameter, but it is not getting me the results I want. It is close but not quite right.
for file in `echo $PARM`
do
FILE_LIST="$FILE_LIST '"$file"'"
done
This gives me 'file1.txt' file2.txt' 'new' 'file.txt' How can I rework the above code to give me what I want.
Thank you
First, you need to understand the sequence of operations when the shell parses a command line. Here's a partial list: first, it interprets quotes and escapes, then removes them (after they've had their effects), then expands any variable references (and similar things like backquote expressions), word-splits and wildcard-expands the expanded variable values, then finally treats the result of all of that as a command and its arguments.
This has two important implications for what you're trying to do: by the time your script receives its arguments, they no longer have quotes; the quotes have had their effect (new file.txt is a single argument rather than two), but the quotes themselves are gone. Also, when putting quotes in a variable is useless because by they time the variable gets expanded and the quotes are part of the command line, it's too late for them to do anything useful -- they aren't parsed as quotes, they're just passed on to the command as part of the argument (which you don't want).
Fortunately, the answer is easy (and Stephen P summarized it in his comment): put double-quotes around all variable references. This prevents the word-splitting and wildcard-expansion phases from messing with their values, which means that whatever was passed to your script as a single argument (e.g. new file.txt) gets passed on as a single argument. If you need to pass on all of your arguments, use "$#". If you need to pass on only some, you can either use shift to get rid of the options and then "$#" will pass on the remaining ones, or use e.g. "${#:4}" to pass all argument starting at #4, or "${#:4:3}" to pass on three arguments starting at #4.
Related
My company has a tool that dynamically generates commands to run based on an input json. It works very well when all arguments to the compiled command are single words, but is failing when we attempt multi word args. Here is the minimal example of how it fails.
# Print and execute the command.
print_and_run() { local command=("$#")
if [[ ${command[0]} == "time" ]]; then
echo "Your command: time ${command[#]:1}"
time ${command[#]:1}
fi
}
# How print_and_run is called in the script
print_and_run time docker run our-conainer:latest $generated_flags
# Output
Your command: time docker run our-container:latest subcommand --arg1=val1 --arg2="val2 val3"
Usage: our-program [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
Try 'our-program --help' for help.
Error: No such command 'val3"'.
But if I copy the printed command and run it myself it works fine (I've omitted docker flags). Shelling into the container and running the program directly with these arguments works as well, so the parsing logic there is solid (It's a python program that uses click to parse the args).
Now, I have a working solution that uses eval, but my entire team jumped down my throat at that suggestion. I've also proposed a solution using delineating characters for multi-word arguments, but that was shot down as well.
No other solutions proposed by other engineers have worked either. So can I ask someone to perhaps explain why val3 is being treated as a separate command, or to help me find a solution to get bash to properly evaluate the dynamically determined command without using eval?
Your command after expanding $generated_flags is:
print_and_run time docker run our-conainer:latest subcommand --arg1=val1 --arg2="val2 val3"
Your specific problem is that in --arg2="val2 val3" the quotes are literal, not syntactical, because quotes are processed before variables are expanded. This means --arg2="val2 and val3" are being split into two separate arguments. Then, I assume, docker is trying to interpret val3" as some kind of docker command because it's not part of any argument, and it's throwing out an error because it doesn't know what that means.
Normally you'd fix this via an array to properly maintain the string boundary.
generated_flags=( "subcommand" "--arg1=val1" "--arg2=val2 val3" )
print_and_run time docker run our-container:latest "${generated_flags[#]}"
This will maintain --arg2=val2 val3 as a single argument as it gets passed into print_and_run, then you just have to expand your command array correctly inside the function (make sure to quote the expansion).
The question is:
why val3 is being treated as a separate command
Unquoted variable expansion undergo word splitting and filename expansion. Word splitting splits the result of the variable expansion on spcaes, tabs and newlines. Splits it into separate "words".
a="something else"
$a # results in two "words"; 'something' and 'else'
It is irrelevent what you put inside the variable value or how many quotes or escape sequences you put inside. Every consecutive spaces splits it into words. Quotes " ' and escapes \ are parsed when part of the input line, not when part of the result of unquoted expansion.
help me find a solution to
Write a parser that will actually parse the commands and split it according to the rules that you want to use and then execute the command split into separate words. For example, a very crude such parser is included in xargs:
$ echo " 'quotes quotes' not quotes" | xargs printf "'%s'\n"
'quotes quotes'
'not'
'quotes'
For example, python has shlex.split which you can just use, and at the same time introduce python which is waaaaay easier to manage than badly written Bash scripts.
tool that dynamically generates commands to run based on an input json
Overall, the proper way forward would is to upgrade the tool to generate a JSON array that represents the words of the command to be executed. Than you can just execute that array of words, which is, again, trivial to do properly in python with json and subprocess.run, and will require some gymnastics with jq and read and Bash arrays in shell.
Check your scripts with shellcheck.
I am trying to give an argument to my python program through the terminal.
For this I am using the lines:
import sys
something = sys.argv[1]
I now try to put in a string like this through the bash terminal:
python my_script.py 2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
This returns a bash error because some of the characters in the string are bash special characters.
How can I use the string exactly as it is?
You can put the raw string into a file, for example like this, with cat and a here document.
cat <<'EOF' > file.txt
2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
EOF
and then run
python my_script.py "$(< file.txt)"
You can also use the text editor of your choice for the first step if you prefer that.
If this is a reoccurring task, which you have to perform from time to time, you can make your life easier with a little alias in your shell:
alias escape='read -r string ; printf "Copy this:\n%q\n" "${string}"'
It is using printf "%q" to escape your input string.
Run it like this:
escape
2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
Copy this:
2m+\{N7HiwH3\[\>\!\"4y\?t9\*y#\;/\$Ar3wF9+k\$\[3hK/WA=aMzF°L0PaZTM\]t\*P\|I_AKAqIb0O4#\ cm=sl\)WWYwEg10DDv%k/\"c\{LrS\)oVd§4\>8bs:\;9u\$\ \*W_SGk3CXe7hZMm\$nXyhAuHDi-q+ug5+%ioou.\,IhC\]-_O§V\]\^\,2q:VBVyTTD6\'aNw9:oan\(s2SzV
You can use the escaped string directly in your shell, without additional quotes, like this:
python my_script.py 2m+\{N7HiwH3\[\>\!\"4y\?t9\*y#\;/\$Ar3wF9+k\$\[3hK/WA=aMzF°L0PaZTM\]t\*P\|I_AKAqIb0O4#\ cm=sl\)WWYwEg10DDv%k/\"c\{LrS\)oVd§4\>8bs:\;9u\$\ \*W_SGk3CXe7hZMm\$nXyhAuHDi-q+ug5+%ioou.\,IhC\]-_O§V\]\^\,2q:VBVyTTD6\'aNw9:oan\(s2SzV
In order to make life easier, shells like bash do a little bit of extra work to help users pass the correct arguments to the programs they instruct it to execute. This extra work usually results in predictable argument arrays getting passed to programs.
Oftentimes, though, this extra help results in unexpected arguments getting passed to programs; and sometimes results in the execution of undesired additional commands. In this case, though, it ended up causing Bash to emit an error.
In order to turn off this extra work, Bash allows users to indicate where arguments should begin and end by surrounding them by quotation marks. Bash supports both single quotes (') and double quotes (") to delimit arguments. As a last resort, if a string may contain single and double quotes (or double quotes are required but aren't aggressive enough), Bash allows you to indicate that a special- or whitespace-character should be part of the adjacent argument by preceding it with a backslash (\\).
If this method of escaping arguments is too cumbersome, it may be worth simplifying your program's interface by having it consume this data from a file instead of a command line argument. Another option is to create a program that loads the arguments from a more controlled location (like a file) and directly execs the target program with the desired argument array.
I am trying to study both bash and ruby and I am trying to pass a variable that contains a JSON object from ruby to be printed using a bash script.
I tried storing the JSON object in a variable then used that variable as an argument for my bash script but I am not getting the result that I wanted.
so in my ruby code I have this:
param = request.body
cmd = "ruby_to_bash #{param}"
exec cmd
and in my bash script, I am simply outputting the value of the argument:
echo $1
For example I have this JSON:
{"first":"one","second":"two","third":"three"}
my code only gives me this output:
{first:
I want to display the whole JSON object. How can I do this? Any kind of help will be much appreciated.
Both bash and ruby treat double quotes in a kinda special way: double-quoted strings are interpolated before passed to the receiver. I strongly encourage you to start with learning somewhat one in the first place instead of doing zero progress in both due to induced errors.
Answering the question stated, one must escape double quotes when passing back and forth since both ruby and bash treat them as content delimiters and discard when the content is already parsed in the resulting string.
This would work:
exec %|ruby_to_bash #{param.gsub('"', '\"')}|
I use %| and | as string delimiters to avoid clashes against inner quotes.
Sidenote: the above won’t work for the input containing spaces, but I purposely avoid to show how to deal with spaces since it leads to the dead end; once handled spaces with surrounding the interpolated param with single quotes, we are prone to be screwed up with inner single quotes in JSON object and so forth.
The code above is not someone should ever produce in production.
I think your best bet is, as usual in such cases, not to involve a shell at all.
When you say this:
cmd = "ruby_to_bash #{param}"
exec cmd
Your exec call is invoking a shell which parses cmd and then runs ruby_to_bash so you're doing this:
Build a command line.
Replace the current process with /bin/sh.
/bin/sh parses the command line that you should mashed together. This is where you need to worry about quoting and escaping to get past the shell.
/bin/sh exec's ruby_to_bash.
You could bypass the whole problem by using the multi-argument form of Kernel#exec:
exec(cmdname, arg1, ...)
command name and one or more arguments (no shell)
which doesn't bother with a shell at all. If you say:
exec 'ruby_to_bash', param
You won't involve a shell and the quotes or spacing in param won't matter to anyone other than ruby_to_bash.
I ham having difficulty understanding how to pass a variable to a ./configure command that includes variable expansion and quotes.
myvars.cfg
myFolderA="/home/myPrefix"
myFolderB="/home/stuffB"
myFolderC="/home/stuffC"
optsA="--prefix=${myFolderA}"
optsB="CPPFLAGS=\"-I${myFolderB} -I${myFolderC}\""
cmd="/home/prog/"
myScript.sh
#!/bin/bash
. /home/myvars.cfg
doCmd=("$cmd/configure" "${optsA}" "${optsB}")
${doCmd[#]}
The doCmd should look like this
/home/prog/configure --prefix=/home/myPrefix CPPFLAGS="-I/home/stuffB -I/home/stuffC"
however it seems when running bash it is adding single quotes
/home/prog/configure --prefix=/home/myPrefix 'CPPFLAGS="-I/home/stuffB' '-I/home/stuffC"'
causing an error of
configure: error: unrecognized option: `-I/home/stuffC"'
Is there a way to pass a variable that needs top be expanded and contains double quotes?
As your script is written, there is no point to using the doCmd array. You could simply write the command:
"$cmd/configure" "${optsA}" "${optsB}"
Or, more simply:
"$cmd/configure" "$optsA" "$optsB"
However, it is possible that you've simplified the script in a way which hides the need for the array. In any case, if you use the array, you need to ensure that its elements are not word-split and filepath expanded, so you must quote its expansion:
"${doCmd[#]}"
Also, you need to get rid of the quotes in optsB. You don't want to pass
CPPFLAGS="-I/home/stuffB -I/home/stuffC"
to the configure script. You want to pass what the shell would pass if you typed the above string. And what the shell would pass would be a single command-line argument with a space in it, looking like this:
CPPFLAGS=-I/home/stuffB -I/home/stuffC
In order to get that into optsB, you just write:
optsB="CPPFLAGS=-I${myFolderB} -I${myFolderC}"
Finally, the shell is not "adding single quotes" into the command line. It is showing you a form of the command whch you could type at the command-line. Since the argument (incorrectly) contains a quote symbol, the shell shows you the command with its arguments skingle-quoted, so that you can see that the optB has been (incorrectly) split into two arguments, each of which contains (incorrectly) one double quote.
You could have found much of the above and more by pasting your script into https://shellcheck.net. As the bash tag summary suggests, you should always try that before asking a shell question here because a lot of the time, it will solve your problem instantly.
I am new to scripting in Linux and I think I'm getting confused with using variables inside command substitution the more I learn and read about it. Can someone explain to me the following scenario?
In my ksh script, I am trying to use a ksh variable inside an sqlplus script as follows:
temp_var="'a', 'b'"
randomVar=$(sqlplus -s $con_details <<EOF
update table ABC
Set field1='val'
Where field2 NOT IN ("${temp_var}");
EOF)
But the above syntax leads to an error in the query and it fails with code 1.
However when I unquote the variable and simply write
Where field2 NOT IN (${temp_var});
The query runs fine. I have seen a lot of examples on SO and Unix and Linux advising to always quote your variables used inside command substitution, but it seems the opposite works for me.
I don't seem to get why using quotes inside $() give an error as opposed to not using them.
Also, the query runs fine when I don't use the ksh variable in it (i.e. Without the WHERE clause).
This is a different situation than where the usual advice applies -- you're using the variable in a here-document, rather than as part of the command line. The difference is in how it gets parsed.
When you use a variable on a command line (something like ls $file), the variable gets replaced by its value partway through the process of parsing the command, with weird and generally undesirable results. The standard solution is to double-quote the variable (ls "$file") to prevent it from being parsed at all, just used directly. The standard mistake people make is putting quotes in the variable's value, which doesn't work because the variable gets replaced after quotes have already been parsed.
But you're using the variable in a here-document, and those work a lot differently. What happens is that the shell just does variable expansion (and some escape parsing) in the here-document, but doesn't do any more extensive parsing. In particular, it doesn't parse quotes in the here-document, just treats them like any other characters. The document then gets passed as input to the command (sqlplus in your case), and it parses the document according to whatever its syntax rules are. Since the parsing happens after variable replacement, it doesn't matter if the quotes are in the variable or around it; they work the same either way. But you can't do both, which is what was happening with double-quotes around the variable. Essentially, you were sending this document to sqlplus:
update table ABC
Set field1='val'
Where field2 NOT IN ("'a', 'b'");
... and sqlplus doesn't like that double-quotes around single-quotes thing, and complains.