Is there any difference, logically or semantically, between the following?
$ eval 'echo hello >/dev/null'
$ eval "echo hello" >/dev/null
More generally: does the shell ever redirect the output eval itself? Or is eval always given the redirect as an argument (to then pass back to the shell)?
And does this behavior depend on the type of shell (sh/bash/zsh/etc)?
In all POSIX-compliant shells, eval is treated exactly like every other command at the syntax-parsing layer -- it receives a list of individual arguments (after quote-removal and other processing stages common to all commands), and only after being invoked concatenates them together into a single string to be evaluated. [1]
Similarly, like every other command, redirections specified as shell syntax (a description which applies to >/dev/null but not ">/dev/null") are applied before the command itself is executed.
This means that in the eval "echo hello" >/dev/null case, the >/dev/null redirection happens before eval is executed.
By contrast, eval 'echo hello >/dev/null' runs the redirection from within eval, before the echo.
[1] - (This is the cause of many bugs in end-user code: People who expect (broken) eval printf '%s\n' $args to be the same as (working) eval "printf '%s\n' $args" are often quite surprised; if testing those two, consider args='"first word" "second word"' as a simple case).
Related
I have the following case where exec and eval will handle variables passed as arguments differently.
Here, eval seems to output something which is intended.
But is there any alternative to using that?
$ cat arg.sh
#!/bin/bash
eval ./argtest $*
$ ./arg.sh "arg1 'subarg1 subarg2'"
Args: 2
Arg1: arg1
Arg2: subarg1 subarg2
But at the same time if I use exec instead of eval call, the single quotes are not getting honored.
$ ./arg.sh "arg1 'subarg1 subarg2'"
Args: 3
Arg1: arg1
Arg2: 'subarg1
Arg3: subarg2'
You should do:
#!/bin/bash
./argtest "$#"
To properly pass unchanged arguments.
Then do:
$ ./arg.sh arg1 'subarg1 subarg2'
As you would do with any other command.
Research when to use quoting in shell, how is $# positional arguments expansions handled specially in quotes, research how does $* and $# differ and research word splitting. Also research what is variable expansion and in which contexts it happens and how does single quotes differ from double quotes. And because exec is mentioned see bashfaq Eval command and security issues. Remember to check your scripts with https://shellcheck.net .
Is there any alternative to using eval in a shell script to achieve variable expansion
Yes - use envsubst for variable expansion, it's a tool just for that.
#!/bin/bash
arg=$(VARIABLE=something envsubst '$VARIABLE' <<<"$1")
./argtest "$arg"
$ bash -x ./arg.sh 'string with **not-expanded** $VARIABLE'
+ ./argtest 'string with **not-expanded** something'
Is there any alternative to using eval in a shell script to achieve *single quotes parsing
Yes - you would potentially write your own parser, probably in awk, that would split the string and then reload. A very very crude example:
#!/bin/bash
readfile -t args < <(sed "s/ *'\([^']*\)' */\n\1\n/; s/\n$//" <<<"$*")
./argtest "${args[#]}"
$ bash -x ./arg.sh "arg1 'subarg1 subarg2'"
+ ./argtest 'arg1' 'subarg1 subarg2'
Using $*, the shell applies word splitting to the parameters and passes the effect after word splitting to eval, repsepcitvely exec. What happens after, differs between them:
exec simply replaces the current process by a new one, based on the first parameter it gets. Than in passes the remaining parameters unmodified to this process.
eval on the other hand catenates the parameters together to a single string (using one space as a separator between those strings), then treats this resulting string as a new command where the usual expansion and word splitting mechanism of bash are applied, and finally runs this command.
The mechanism is completely different, which is not surprising, since these commands serve a different purpose.
I have a setup script that needs to be run remotely on an arbitrary machine (can be windows). So I had something along the lines of bash -c "do things that need environmental variables".
I found some strange things happening with nested quotes + enviornmental variables that I don't understand (demonstrated below)
# This worked because my environment was polluted.
bash -c "NAME=me echo $NAME"
> me
# I think this was a weird cross platform issue with how I was running.
# I couldn't reproduce it locally.
bash -c "NAME=me echo "Hi $NAME""
> Hi $NAME
# This was my workaround, and I have no clue why this works.
# I get that "Start "" end" does string concatenation in bash,
# but I have no clue why that would make this print 'Hi me' instead
# of 'Hi'.
#
# This works because echo Hi name prints "Hi name". I thought echo only
# took the first argument passed in.
bash -c "NAME=me echo Hi "" $NAME"
> Hi me
# This is the same as the first case. NAME was just empty this time.
bash -c "NAME=me echo Hi $NAME"
> Hi
Edit: A bunch of people have pointed out that the variables get expanded in double quotes before bash -c gets run. This makes sense, but I feel like it doesn't explain why case 1 works.
shouldn't bash -c "NAME=me echo $NAME" be expanded to bash -c "NAME=me echo ", since NAME isn't set before we run this?
Edit 2: A bunch of this stuff worked because my environment was polluted. I've tried to describe what mistakes I made in my assumptions
There are at least three sources of confusion here: quotes don't (generally) nest, $variable references are expanded by the shell even if they're in double-quotes, and variable references are resolved before var=value assignments are done.
Let me look at the second problem first. Here's an interactive example showing the effect:
$ NAME=Gordon
$ bash -c "NAME=me echo $NAME"
Gordon
Here, the outer (interactive) shell expanded $NAME before passing it to bash -c, so the command essentially became bash -c "NAME=me echo Gordon". There are several ways to avoid this: you can escape the $ to remove its normal effect (but the escape gets removed, so the inner shell will see it and apply it normally), or use single-quotes instead of double (which remove the special effect of all characters, except for another single-quote which ends the single-quoted string). So let's try those:
$ bash -c "NAME=me echo \$NAME"
$ bash -c 'NAME=me echo $NAME'
(You can't really see it, but there's a blank line after the second command as well, because it didn't print anything either.) What happened here is that the inner shell (the one created by bash -c) indeed got the command NAME=me echo $NAME, but when executing it expands $NAME first (giving nothing, because it's not defined in that shell), and then executes NAME=me echo which runs the echo command with NAME set to "me" in its environment. Let's try that interactively:
$ NAME=me echo $NAME
Gordon
(Remember that I set NAME=Gordon in my interactive shell earlier.) To get the intended effect, you'd need to set NAME and then as a separate command use it in an echo command:
$ bash -c "NAME=me; echo \$NAME"
me
$ bash -c 'NAME=me; echo $NAME'
me
Ok, with that out of the way let's move on to the original question about quoting. As I said, quotes don't (generally) nest. To understand what's going on, let's analyze some of the example commands. You can get a better idea how the shell interprets things by using set -x, which makes the shell print each command's equivalent just before it's executed:
$ set -x
$ bash -c "NAME=me echo "Hi $NAME""
+ bash -c 'NAME=me echo Hi' Gordon
Hi
What happened here is that the shell parsed "NAME=me echo "Hi as a double-quoted string immediately followed by two unquoted characters; since there's no gap between them, they get merged into a single argument to bash -c. It may seem a little weird having only part of an argument quoted, but it's actually entirely normal in shell syntax. It's even normal to have part of a single argument be unquoted, part single-quoted, part double-quoted, and even part in ANSI-C mode ($'ANSI-c-escaped stuff goes here').
With set -x, bash will print something equivalent to the command being executed. All of these commands are equivalent in shell syntax:
bash -c "NAME=me echo "Hi Gordon
bash -c "NAME=me echo Hi" Gordon
bash -c 'NAME=me echo Hi' Gordon
bash -c NAME=me\ echo\ Hi Gordon
bash -c NAME=me' 'echo' 'Hi Gordon
bash -c 'NAME=me'\ "echo Hi" Gordon
...and lots more. With set -x, bash will print one of these equivalents, and it just happens to choose the one with single-quotes around the entire argument.
Just for completeness, what happened to $NAME""? It's treated as an unquoted variable reference (which expands to Gordon) immediately followed by a zero-length double-quoted string, which doesn't do anything at all.
But... why does that just print "Hi"? Well, bash -c treats the next argument as a command to run, and any further arguments as the argument vector ($0, $1, etc) for that command's environment. Here's an illustration:
$ bash -c 'echo "Args: $0 $1 $2"' zeroth first second third
+ bash -c 'echo "Args: $0 $1 $2"' zeroth first second third
Args: zeroth first second
("third" doesn't get printed because the command doesn't print $3.)
Thus, when you run bash -c 'NAME=me echo Hi' Gordon, it executes NAME=me echo Hi with $0 set to "Gordon".
Ok, here's the last example I'll look at:
$ bash -c "NAME=me echo Hi "" $NAME"
+ bash -c 'NAME=me echo Hi Gordon'
Hi Gordon
What's happening here is that there's a double-quoted section "NAME=me echo Hi " immediately followed by another one, " $NAME", so they get merged into a single long argument (which happens to contain two spaces in a row -- one part of the first quoted section, one part of the second). Essentially, the "" in the middle ends one double-quotes section and immediately starts another, thus having no overall effect. And again, the shell decided to print a single-quoted equivalent rather than any of the various other possible equivalents.
So how do we actually get this to work right? Here's what I'd actually recommend:
$ bash -c 'NAME=me; echo "Hi $NAME"'
+ bash -c 'NAME=me; echo "Hi $NAME"'
Hi me
Since the entire command string is in single-quotes, none of these problems occur. The double-quotes are just normal characters being passed as part of the argument (so double-quotes sort of nest inside single-quotes -- and vice versa -- but it's really just 'cause they're ignored), and the $ doesn't get its special meaning to the outer shell either. Oh, and the ; makes this two separate commands, so the NAME=me part can take effect before the echo "$NAME" part uses it.
Another equivalent would be:
$ bash -c "NAME=me; echo \"Hi \$NAME\""
+ bash -c 'NAME=me; echo "Hi $NAME"'
Hi me
Here the escapes remove the special meanings of the $ and enclosed double-quotes. Note that the shell prints exactly the same thing as last time for its set -x output, indicating that this really is equivalent to the single-quoted version.
Suppose you have the following command stored in a variable:
COMMAND='echo hello'
What's the difference between
$ eval "$COMMAND"
hello
$ bash -c "$COMMAND"
hello
$ $COMMAND
hello
? Why is the last version almost never used if it is shorter and (as far as I can see) does exactly the same thing?
The third form is not at all like the other two -- but to understand why, we need to go into the order of operations when bash in interpreting a command, and look at which of those are followed when each method is in use.
Bash Parsing Stages
Quote Processing
Splitting Into Commands
Special Operator Parsing
Expansions
Word Splitting
Globbing
Execution
Using eval "$string"
eval "$string" follows all the above steps starting from #1. Thus:
Literal quotes within the string become syntactic quotes
Special operators such as >() are processed
Expansions such as $foo are honored
Results of those expansions are split on characters into whitespace into separate words
Those words are expanded as globs if they parse as same and have available matches, and finally the command is executed.
Using sh -c "$string"
...performs the same as eval does, but in a new shell launched as a separate process; thus, changes to variable state, current directory, etc. will expire when this new process exits. (Note, too, that that new shell may be a different interpreter supporting a different language; ie. sh -c "foo" will not support the same syntax that bash, ksh, zsh, etc. do).
Using $string
...starts at step 5, "Word Splitting".
What does this mean?
Quotes are not honored.
printf '%s\n' "two words" will thus parse as printf %s\n "two words", as opposed to the usual/expected behavior of printf %s\n two words (with the quotes being consumed by the shell).
Splitting into multiple commands (on ;s, &s, or similar) does not take place.
Thus:
s='echo foo && echo bar'
$s
...will emit the following output:
foo && echo bar
...instead of the following, which would otherwise be expected:
foo
bar
Special operators and expansions are not honored.
No $(foo), no $foo, no <(foo), etc.
Redirections are not honored.
>foo or 2>&1 is just another word created by string-splitting, rather than a shell directive.
$ bash -c "$COMMAND"
This version starts up a new bash interpreter, runs the command, and then exits, returning control to the original shell. You don't need to be running bash at all in the first place to do this, you can start a bash interpreter from tcsh, for example. You might also do this from a bash script to start with a fresh environment or to keep from polluting your current environment.
EDIT:
As #CharlesDuffy points out starting a new bash shell in this way will clear shell variables but environment variables will be inherited by the spawned shell process.
Using eval causes the shell to parse your command twice. In the example you gave, executing $COMMAND directly or doing an eval are equivalent, but have a look at the answer here to get a more thorough idea of what eval is good (or bad) for.
There are at least times when they are different. Consider the following:
$ cmd="echo \$var"
$ var=hello
$ $cmd
$var
$ eval $cmd
hello
$ bash -c "$cmd"
$ var=world bash -c "$cmd"
world
which shows the different points at which variable expansion is performed. It's even more clear if we do set -x first
$ set -x
$ $cmd
+ echo '$var'
$var
$ eval $cmd
+ eval echo '$var'
++ echo hello
hello
$ bash -c "$cmd"
+ bash -c 'echo $var'
$ var=world bash -c "$cmd"
+ var=world
+ bash -c 'echo $var'
world
We can see here much of what Charles Duffy talks about in his excellent answer. For example, attempting to execute the variable directly prints $var because parameter expansion and those earlier steps had already been done, and so we don't get the value of var, as we do with eval.
The bash -c option only inherits exported variables from the parent shell, and since I didn't export var it's not available to the new shell.
In bash, you can treat a string as command (and run it) in two different ways:
#!/bin/bash
cmd="echo -n sometext"
eval $cmd # Not sure if quotes make a difference here
and
#!/bin/bash
cmd="echo -n sometext"
$cmd # Not sure if quotes make a difference here either
Is there any difference between the two? Is there a situation where quotes around cmd make a difference? What about performance?
Yes, there is a difference :)
You need to first understand how eval works. Basically, eval is a shell builtin command. Whatever argument passed to eval is first treated as a string.
Let's take below example:
cmd="echo -n sometext"
eval $cmd
The complete run process of this command is as follows:
eval $cmd
+ eval echo -n sometext
++ echo -n sometext
sometext
Here, first $cmd first got evaluated and then the whole string was passed to eval command as argument. Then eval evaluates the command considering the first argument as a "command or an executable file" and then run as a normal command. So, here there is 2 rounds of evaluation getting performed for the execution of the complete command.
(NOTE: The + symbol above shows the step wise execution when used in bash -x mode)
The main consequence lies in variable expansion. With eval we have two rounds of expansion. One of course, when cmd is defined, and one when eval is executed.
var="inital"
cmd="echo -n $var \$var"
var="chanded in the mean time"
eval $cmd
inital chanded in the mean time
However, when you use $cmd only without eval, bash takes care of everything from variable expansion to the final execution. Just see the debugging window details while running only $cmd
$cmd
+ echo -n sometext
sometext
Performance wise, direct use of $cmd is good enough. However, when you are trying to use some external command or a script which requires environment changes, you can use eval
In cmd="echo -n sometext", quotes are necessary, otherwise after "echo", bash will raise an error like below:
cmd=echo -n sometext
-n: command not found
I hope the explanation will be helpful.
Suppose you have the following command stored in a variable:
COMMAND='echo hello'
What's the difference between
$ eval "$COMMAND"
hello
$ bash -c "$COMMAND"
hello
$ $COMMAND
hello
? Why is the last version almost never used if it is shorter and (as far as I can see) does exactly the same thing?
The third form is not at all like the other two -- but to understand why, we need to go into the order of operations when bash in interpreting a command, and look at which of those are followed when each method is in use.
Bash Parsing Stages
Quote Processing
Splitting Into Commands
Special Operator Parsing
Expansions
Word Splitting
Globbing
Execution
Using eval "$string"
eval "$string" follows all the above steps starting from #1. Thus:
Literal quotes within the string become syntactic quotes
Special operators such as >() are processed
Expansions such as $foo are honored
Results of those expansions are split on characters into whitespace into separate words
Those words are expanded as globs if they parse as same and have available matches, and finally the command is executed.
Using sh -c "$string"
...performs the same as eval does, but in a new shell launched as a separate process; thus, changes to variable state, current directory, etc. will expire when this new process exits. (Note, too, that that new shell may be a different interpreter supporting a different language; ie. sh -c "foo" will not support the same syntax that bash, ksh, zsh, etc. do).
Using $string
...starts at step 5, "Word Splitting".
What does this mean?
Quotes are not honored.
printf '%s\n' "two words" will thus parse as printf %s\n "two words", as opposed to the usual/expected behavior of printf %s\n two words (with the quotes being consumed by the shell).
Splitting into multiple commands (on ;s, &s, or similar) does not take place.
Thus:
s='echo foo && echo bar'
$s
...will emit the following output:
foo && echo bar
...instead of the following, which would otherwise be expected:
foo
bar
Special operators and expansions are not honored.
No $(foo), no $foo, no <(foo), etc.
Redirections are not honored.
>foo or 2>&1 is just another word created by string-splitting, rather than a shell directive.
$ bash -c "$COMMAND"
This version starts up a new bash interpreter, runs the command, and then exits, returning control to the original shell. You don't need to be running bash at all in the first place to do this, you can start a bash interpreter from tcsh, for example. You might also do this from a bash script to start with a fresh environment or to keep from polluting your current environment.
EDIT:
As #CharlesDuffy points out starting a new bash shell in this way will clear shell variables but environment variables will be inherited by the spawned shell process.
Using eval causes the shell to parse your command twice. In the example you gave, executing $COMMAND directly or doing an eval are equivalent, but have a look at the answer here to get a more thorough idea of what eval is good (or bad) for.
There are at least times when they are different. Consider the following:
$ cmd="echo \$var"
$ var=hello
$ $cmd
$var
$ eval $cmd
hello
$ bash -c "$cmd"
$ var=world bash -c "$cmd"
world
which shows the different points at which variable expansion is performed. It's even more clear if we do set -x first
$ set -x
$ $cmd
+ echo '$var'
$var
$ eval $cmd
+ eval echo '$var'
++ echo hello
hello
$ bash -c "$cmd"
+ bash -c 'echo $var'
$ var=world bash -c "$cmd"
+ var=world
+ bash -c 'echo $var'
world
We can see here much of what Charles Duffy talks about in his excellent answer. For example, attempting to execute the variable directly prints $var because parameter expansion and those earlier steps had already been done, and so we don't get the value of var, as we do with eval.
The bash -c option only inherits exported variables from the parent shell, and since I didn't export var it's not available to the new shell.