I was reading the doc of Ruby's system method here. If I put in the option -P /public/google (specifying the directory for downloading), does that count as one argument or two arguments, or either?
It actually depends which form of system you use. If you pass a command string, ala:
system 'somecmd -P /public/google'
The command string will be interpreted through the shell which will parse it, tokenizing on whitespace resulting in two arguments to somecmd. Likewise if you use the argument list form and you break the string up into whitespace delimited tokens, like:
system 'somecmd', '-P', '/public/google'
system *%w{ somecmd -P /public/google }
But if you use the argument list form, and send -P /public/google as a single argument it will appear to somecmd as a one argument with embedded whitespace:
system 'somecmd', '-P /public/google'
It counts as two. Since there's a space in between the two, the shell sees it and splits it as -P and /public/google (if you've worked with arguments, even in ruby, the shell would pass those in as separate arguments to the script).
Related
My company has a tool that dynamically generates commands to run based on an input json. It works very well when all arguments to the compiled command are single words, but is failing when we attempt multi word args. Here is the minimal example of how it fails.
# Print and execute the command.
print_and_run() { local command=("$#")
if [[ ${command[0]} == "time" ]]; then
echo "Your command: time ${command[#]:1}"
time ${command[#]:1}
fi
}
# How print_and_run is called in the script
print_and_run time docker run our-conainer:latest $generated_flags
# Output
Your command: time docker run our-container:latest subcommand --arg1=val1 --arg2="val2 val3"
Usage: our-program [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
Try 'our-program --help' for help.
Error: No such command 'val3"'.
But if I copy the printed command and run it myself it works fine (I've omitted docker flags). Shelling into the container and running the program directly with these arguments works as well, so the parsing logic there is solid (It's a python program that uses click to parse the args).
Now, I have a working solution that uses eval, but my entire team jumped down my throat at that suggestion. I've also proposed a solution using delineating characters for multi-word arguments, but that was shot down as well.
No other solutions proposed by other engineers have worked either. So can I ask someone to perhaps explain why val3 is being treated as a separate command, or to help me find a solution to get bash to properly evaluate the dynamically determined command without using eval?
Your command after expanding $generated_flags is:
print_and_run time docker run our-conainer:latest subcommand --arg1=val1 --arg2="val2 val3"
Your specific problem is that in --arg2="val2 val3" the quotes are literal, not syntactical, because quotes are processed before variables are expanded. This means --arg2="val2 and val3" are being split into two separate arguments. Then, I assume, docker is trying to interpret val3" as some kind of docker command because it's not part of any argument, and it's throwing out an error because it doesn't know what that means.
Normally you'd fix this via an array to properly maintain the string boundary.
generated_flags=( "subcommand" "--arg1=val1" "--arg2=val2 val3" )
print_and_run time docker run our-container:latest "${generated_flags[#]}"
This will maintain --arg2=val2 val3 as a single argument as it gets passed into print_and_run, then you just have to expand your command array correctly inside the function (make sure to quote the expansion).
The question is:
why val3 is being treated as a separate command
Unquoted variable expansion undergo word splitting and filename expansion. Word splitting splits the result of the variable expansion on spcaes, tabs and newlines. Splits it into separate "words".
a="something else"
$a # results in two "words"; 'something' and 'else'
It is irrelevent what you put inside the variable value or how many quotes or escape sequences you put inside. Every consecutive spaces splits it into words. Quotes " ' and escapes \ are parsed when part of the input line, not when part of the result of unquoted expansion.
help me find a solution to
Write a parser that will actually parse the commands and split it according to the rules that you want to use and then execute the command split into separate words. For example, a very crude such parser is included in xargs:
$ echo " 'quotes quotes' not quotes" | xargs printf "'%s'\n"
'quotes quotes'
'not'
'quotes'
For example, python has shlex.split which you can just use, and at the same time introduce python which is waaaaay easier to manage than badly written Bash scripts.
tool that dynamically generates commands to run based on an input json
Overall, the proper way forward would is to upgrade the tool to generate a JSON array that represents the words of the command to be executed. Than you can just execute that array of words, which is, again, trivial to do properly in python with json and subprocess.run, and will require some gymnastics with jq and read and Bash arrays in shell.
Check your scripts with shellcheck.
What are the differences between these terms: "option", "argument", and "parameter"? In man pages these terms often seem to be used interchangeably.
A command is split into an array of strings named arguments. Argument 0 is (normally) the command name, argument 1, the first element following the command, and so on. These arguments are sometimes called positional parameters.
$ ls -la /tmp /var/tmp
arg0 = ls
arg1 = -la
arg2 = /tmp
arg3 = /var/tmp
An option is a documented1 type of argument modifying the behavior of a command, e.g. -l commonly means "long", -v verbose. -lv are two options combined in a single argument. There are also long options like --verbose (see also Using getopts to process long and short command line options). As their name suggests, options are usually optional. There are however some commands with paradoxical "mandatory options".
$ ls -la /tmp /var/tmp
option1= -l
option2= -a
A parameter is an argument that provides information to either the command or one of its options, e.g. in -o file, file is the parameter of the -o option. Unlike options, whose possible values are hard coded in programs, parameters are usually not, so the user is free to use whatever string suits his/her needs. Should you need to pass a parameter that looks like an option but shouldn't be interpreted as such, you can separate it from the beginning of the command line with a double dash: --2.
$ ls -la /tmp /var/tmp
parameter1= /tmp
parameter2= /var/tmp
$ ls -l -- -a
option1 = -l
parameter1 = -a
A shell parameter is anything that store a value in the context of the shell. This includes positional parameters (e.g. $1, $2...), variables (e.g. $foo, $bar...) and special character ones (e.g. $#)
Finally, there are subcommands, also known as functions / (low-level) commands, which are used with "metacommands" that embed multiple separate commands, like busybox, git, apt-get, openssl, and the likes. With them, you might have global options preceeding the subcommand, and subcommand specific options that follow the subcommand. Unlike parameters, the list of possible subcommands is hardcoded in the command itself. e.g.:
$ busybox ls -l
command = busybox
subcommand = ls
subcommand option1 = -l
$ git --git-dir=a.git --work-tree=b -C c status -s
command = git
command option1 = --git-dir=a.git
command option2 = --work-tree=b
command option3 = -C c
subcommand = status
subcommand option1 = -s
Note that some commands like test, tar, dd and find have more complex argument parsing syntax than the ones described previously and can have some or all of their arguments parsed as expressions, operands, keys and similar command specific components.
Note also that optional variable assignments and redirections, despite being processed by the shell for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal like other command line parameters are not taken into account in my reply because they have disappeared when the command is actually called and passed its arguments.
1 I should have written usually documented because of course, undocumented options are still options.
2 The double dash feature need to be implemented by the program though.
The man page for a typical Unix command often uses the terms argument, option and parameter. At the lowest level, we have argument and everything is an argument, including the (filesystem path to the) command itself.
In a shell script you access arguments using the special variables $0 .. $n. Other languages have similar ways to access them (commonly through an array with a name like argv).
Arguments may be interpreted as options if you wish. How this is done is implementation-specific. You can either roll your own, for exampe a shell (such as bash) script can use provided getopts or getopt commands.
These typically define an option as an argument beginning with a hyphen (-) and some options may use proceeding arguments as its parameters. More capable parsers (e.g getopt) support mixing short-form (-h) and long-form (--help) options.
Typically, most options take zero or one parameter. Such parameters are also sometimes called values.
The supported options are coded in the program code (e.g in the invocation of getopts within a shell script). Any remaining arguments after the options have been consumed are commonly called positional parameters when the order in which they are given is significant (this is in contrast to options which usually can be given in any order).
Again, the script defines what the positional parameters are by how it consumes and uses them.
So a typical command
$ ls -I README -l foo 'bar car' baz
has seven arguments: /usr/bin/ls, -I, README, -l, foo, bar car, and baz accessible as $0 thru $6. The -l and -I are interpreted as options, the latter having a parameter (or value) of README. What remains are positional parameters (foo, bar car and baz).
Option parsing may alter the argument list by removing those it consumes (e.g using shift or set) so that only the positional parameters remain and are thereafter accessible as $1 .. $n.
Since the question is tagged "bash", I looked for relevant sections in the Bash manual. I list these as quoted passages below together with my own one sentence summaries.
Arguments
Everything following the command is an argument.
A simple shell command such as echo a b c consists of the command itself followed by arguments, separated by spaces.
A simple command is the kind of command encountered most often. It’s just a sequence of words separated by blanks, terminated by one of the shell’s control operators (see Definitions). The first word generally specifies a command to be executed, with the rest of the words being that command’s arguments.
Parameters
Arguments are referred to as parameters during function execution.
When a function is executed, the arguments to the function become the positional parameters during its execution
A parameter is an entity that stores values. It can be a name, a number, or one of the special characters listed below. A variable is a parameter denoted by a name.
A positional parameter is a parameter denoted by one or more digits, other than the single digit 0. Positional parameters are assigned from the shell’s arguments when it is invoked, and may be reassigned using the set builtin command. Positional parameter N may be referenced as ${N}, or as $N when N consists of a single digit.
Options
There is no dedicated section to defining what an option is, but they are referred to as hyphen-prefixed characters throughout the manual.
The -p option changes the output format to that specified by POSIX
What are the differences between these terms: "option", "argument", and "parameter"? In man pages these terms often seem to be used interchangeably.
A command is split into an array of strings named arguments. Argument 0 is (normally) the command name, argument 1, the first element following the command, and so on. These arguments are sometimes called positional parameters.
$ ls -la /tmp /var/tmp
arg0 = ls
arg1 = -la
arg2 = /tmp
arg3 = /var/tmp
An option is a documented1 type of argument modifying the behavior of a command, e.g. -l commonly means "long", -v verbose. -lv are two options combined in a single argument. There are also long options like --verbose (see also Using getopts to process long and short command line options). As their name suggests, options are usually optional. There are however some commands with paradoxical "mandatory options".
$ ls -la /tmp /var/tmp
option1= -l
option2= -a
A parameter is an argument that provides information to either the command or one of its options, e.g. in -o file, file is the parameter of the -o option. Unlike options, whose possible values are hard coded in programs, parameters are usually not, so the user is free to use whatever string suits his/her needs. Should you need to pass a parameter that looks like an option but shouldn't be interpreted as such, you can separate it from the beginning of the command line with a double dash: --2.
$ ls -la /tmp /var/tmp
parameter1= /tmp
parameter2= /var/tmp
$ ls -l -- -a
option1 = -l
parameter1 = -a
A shell parameter is anything that store a value in the context of the shell. This includes positional parameters (e.g. $1, $2...), variables (e.g. $foo, $bar...) and special character ones (e.g. $#)
Finally, there are subcommands, also known as functions / (low-level) commands, which are used with "metacommands" that embed multiple separate commands, like busybox, git, apt-get, openssl, and the likes. With them, you might have global options preceeding the subcommand, and subcommand specific options that follow the subcommand. Unlike parameters, the list of possible subcommands is hardcoded in the command itself. e.g.:
$ busybox ls -l
command = busybox
subcommand = ls
subcommand option1 = -l
$ git --git-dir=a.git --work-tree=b -C c status -s
command = git
command option1 = --git-dir=a.git
command option2 = --work-tree=b
command option3 = -C c
subcommand = status
subcommand option1 = -s
Note that some commands like test, tar, dd and find have more complex argument parsing syntax than the ones described previously and can have some or all of their arguments parsed as expressions, operands, keys and similar command specific components.
Note also that optional variable assignments and redirections, despite being processed by the shell for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal like other command line parameters are not taken into account in my reply because they have disappeared when the command is actually called and passed its arguments.
1 I should have written usually documented because of course, undocumented options are still options.
2 The double dash feature need to be implemented by the program though.
The man page for a typical Unix command often uses the terms argument, option and parameter. At the lowest level, we have argument and everything is an argument, including the (filesystem path to the) command itself.
In a shell script you access arguments using the special variables $0 .. $n. Other languages have similar ways to access them (commonly through an array with a name like argv).
Arguments may be interpreted as options if you wish. How this is done is implementation-specific. You can either roll your own, for exampe a shell (such as bash) script can use provided getopts or getopt commands.
These typically define an option as an argument beginning with a hyphen (-) and some options may use proceeding arguments as its parameters. More capable parsers (e.g getopt) support mixing short-form (-h) and long-form (--help) options.
Typically, most options take zero or one parameter. Such parameters are also sometimes called values.
The supported options are coded in the program code (e.g in the invocation of getopts within a shell script). Any remaining arguments after the options have been consumed are commonly called positional parameters when the order in which they are given is significant (this is in contrast to options which usually can be given in any order).
Again, the script defines what the positional parameters are by how it consumes and uses them.
So a typical command
$ ls -I README -l foo 'bar car' baz
has seven arguments: /usr/bin/ls, -I, README, -l, foo, bar car, and baz accessible as $0 thru $6. The -l and -I are interpreted as options, the latter having a parameter (or value) of README. What remains are positional parameters (foo, bar car and baz).
Option parsing may alter the argument list by removing those it consumes (e.g using shift or set) so that only the positional parameters remain and are thereafter accessible as $1 .. $n.
Since the question is tagged "bash", I looked for relevant sections in the Bash manual. I list these as quoted passages below together with my own one sentence summaries.
Arguments
Everything following the command is an argument.
A simple shell command such as echo a b c consists of the command itself followed by arguments, separated by spaces.
A simple command is the kind of command encountered most often. It’s just a sequence of words separated by blanks, terminated by one of the shell’s control operators (see Definitions). The first word generally specifies a command to be executed, with the rest of the words being that command’s arguments.
Parameters
Arguments are referred to as parameters during function execution.
When a function is executed, the arguments to the function become the positional parameters during its execution
A parameter is an entity that stores values. It can be a name, a number, or one of the special characters listed below. A variable is a parameter denoted by a name.
A positional parameter is a parameter denoted by one or more digits, other than the single digit 0. Positional parameters are assigned from the shell’s arguments when it is invoked, and may be reassigned using the set builtin command. Positional parameter N may be referenced as ${N}, or as $N when N consists of a single digit.
Options
There is no dedicated section to defining what an option is, but they are referred to as hyphen-prefixed characters throughout the manual.
The -p option changes the output format to that specified by POSIX
This question already has answers here:
How do you pass on filenames to other programs correctly in bash scripts?
(3 answers)
Closed 7 years ago.
I am attempting to parse the parameters sent to shell script. For example the values sent to the script are as follows:
-to someone#somewhere.com -a file1.txt file2.txt "new file.txt"
I can parse the string so that I get -a as my operator, but I want to reformat the parameter part file1.txt file2.txt "new file.txt" so that it looks like 'file1.txt' 'file2.txt' 'new file.txt' so that I can pass it down to the zip utility.
Right now I am using the following to parse the parameter, but it is not getting me the results I want. It is close but not quite right.
for file in `echo $PARM`
do
FILE_LIST="$FILE_LIST '"$file"'"
done
This gives me 'file1.txt' file2.txt' 'new' 'file.txt' How can I rework the above code to give me what I want.
Thank you
First, you need to understand the sequence of operations when the shell parses a command line. Here's a partial list: first, it interprets quotes and escapes, then removes them (after they've had their effects), then expands any variable references (and similar things like backquote expressions), word-splits and wildcard-expands the expanded variable values, then finally treats the result of all of that as a command and its arguments.
This has two important implications for what you're trying to do: by the time your script receives its arguments, they no longer have quotes; the quotes have had their effect (new file.txt is a single argument rather than two), but the quotes themselves are gone. Also, when putting quotes in a variable is useless because by they time the variable gets expanded and the quotes are part of the command line, it's too late for them to do anything useful -- they aren't parsed as quotes, they're just passed on to the command as part of the argument (which you don't want).
Fortunately, the answer is easy (and Stephen P summarized it in his comment): put double-quotes around all variable references. This prevents the word-splitting and wildcard-expansion phases from messing with their values, which means that whatever was passed to your script as a single argument (e.g. new file.txt) gets passed on as a single argument. If you need to pass on all of your arguments, use "$#". If you need to pass on only some, you can either use shift to get rid of the options and then "$#" will pass on the remaining ones, or use e.g. "${#:4}" to pass all argument starting at #4, or "${#:4:3}" to pass on three arguments starting at #4.
There is a executbale called app, and it can take some options and command line args, -l -v, to name a few. Now I'm writing a bash script inside which app will be invoked with some options, and I did it this way,
opt_string="-l -v" # this string might change according to different conditions used in if-else
# HERE is my problem
./app ${opt_string}
Look how I invoked app, typically I just invoke it in prompt shell like this:
./app -l -v
But now in this script, would it be actually this:
./app "-l -v"
cuz ${opt_string} is a STRING quoted by "", if so I doubt whether app will run normally.
I know there might be a way around this by using eval "./app ${opt_string}", but is there any way to strip the ""?
BASH FAQ entry #50: "I'm trying to put a command in a variable, but the complex cases always fail!"
opt_string=(-l -v)
./app "${opt_string[#]}"
The way you're invoking (./app ${opt_string}) it is fine. You define opt_string as a single string because you quote the value in the assignment. However, when you dereference it, you have not used quotes, so the shell will substitute its value and then split it into individual words.
when you say
./app "$opt_string"
you are passing one single argument to the app. When you say
./app $opt_string
you are passing multiple arguments to the app.
See word splitting in the bash manual.
Note that braces are not quotes. Curly braces (in this context) merely serve to disambiguate the variable name from the surrounding text, i.e. echo "$opt_string_blah" versus echo "${opt_string}_blah"