This bug appears when using shell commands in a ruby script. The commands are not executed as what would happen in the bash terminal.
The first two commands are reasonable, the third is not.
`echo foo` # => "foo\n"
`echo -n foo` # => "foo"
`echo -n foo | cat` # => "-n foo\n"
In bash, we would have the following:
$ echo foo # => "foo\n"
$ echo -n foo # => "foo"
$ echo -n foo | cat # => "foo"
Is there a way I am messing up the parameter passing to the calls to echo in the Ruby commands? If you're unfamiliar, the command echo returns the string it is given. Without the -n flag, it appends a newline character, when you add the -n, it does not. cat repeats whatever it is given.
First, Ruby uses /bin/sh, not bash, for the back tick operator. It is likely that on your system, /bin/sh is a link to /bin/bash, but bash behaves differently when invoked as sh, for better POSIX compliance. However, it's not perfect, as your example shows. The POSIX specification states that echo "shall not support any options", but leaves handling of an initial argument -n to be implementation-specific.
In a quick test with /bin/sh as a link to /bin/bash,
bash -c "echo -n foo" and bash -c "echo -n foo | cat" produced identical, newline-free results, but sh -c "echo -n foo" and sh -c "echo -n foo | cat" showed the results you report. (I am not sure how other shells, such as ksh, dash, or zsh, behave when invoked as sh, but by default they all treat -n as a newline suppressor.)
For predictable results, use printf instead, which never prints a newline unless a \n is included in the format string.
printf 'foo\n'
printf 'foo'
printf 'foo' | cat
UPDATE: This appears to be a bug in bash 3.2 that was fixed at some point in the 4.x series. With 4.1 or later, sh -c "echo -n foo | cat and sh -c "echo -n foo" produce the same output.
Related
I think the two commands below should be identical, but given the heredoc, the shell produces an error. Is it possible to pass a heredoc to the -c argument of sh?
heredoc example
/bin/sh -c <<EOF
echo 'hello'
EOF
# ERROR: /bin/sh: -c: option requires an argument
simple string example
/bin/sh -c "echo 'hello'"
# prints hello
The commands are not equivalent.
/bin/sh -c <<EOF
echo 'hello'
EOF
is equivalent to
echo "echo 'hello'" | /bin/sh -c
or, with here-string:
/bin/sh -c <<< "echo 'hello'"
but sh -c requires an argument. It would work with
echo "echo 'hello'" | /bin/sh
I tried to post this as a comment but formatting code doesn't work well in comments. Using the accepted answer by #Benjamin W. as a guide, I was able to get it to work with this snippet
( cat <<EOF
echo 'hello'
EOF
) | /bin/sh
The magic is in how cat handles inputs. From the man page:
If file is a single dash (`-') or absent, cat reads from the standard input.
So cat can redirect stdin to stdout and that can be piped to /bin/sh
Suppose I have the following shell program.
#!/bin/sh
FOO="foo"
echo $FOO | cat
I want to generate another shell program that does the same thing as this one, except that all shell variables have been substituted. For example,
#!/bin/sh
echo "foo" | cat
I know that I can get close if I run the above program using #!/bin/sh -x, but that output does not preserve redirections. Instead, I get
+ FOO=foo
+ echo foo
+ cat
foo
Any ideas?
The following shell:
$ cat eval.sh
echo "#!/bin/sh"
FOO="foo"
echo "echo $FOO | cat"
will write a shell:
$ sh eval.sh
#!/bin/sh
echo foo | cat
which does what you need.
The script I am writing with parallel currently looks like this:
#!/bin/bash
seq ${2:-3} | parallel --tty -j0 sidplayfp -wch{}.wav '{=$_=join" ",map{"-u".$_}grep!/#{[seq()]}/,(1..total_jobs())=}' ${#:3} -q $1 '2>/dev/null'
For example, running ./sidrender.sh Stomp.sid is expected to execute the following commands:
sidplayfp -wch1.wav -u2 -u3 -q Stomp.sid 2>/dev/null
sidplayfp -wch2.wav -u1 -u3 -q Stomp.sid 2>/dev/null
sidplayfp -wch3.wav -u1 -u2 -q Stomp.sid 2>/dev/null
However, this does not work properly, and when looking with --dry-run, it turns out that parallel is quoting together the -u flags (e.g. sidplayfp -wch1.wav '-u2 -u3' -q Stomp.sid 2>/dev/null) because they come from the same Perl expression.
Here's a minimal example of what is happening:
$ parallel --dry-run 'echo {= $_="foo bar" =}' ::: 1
echo 'foo bar'
If you were to replace "foo bar" with "foo", the output would be echo foo without any quotes.
Because sidplayfp does not properly parse the arguments when they are quoted, I need a way to stop parallel from quoting the output, but I cannot find a way to do so in the man page.
Use eval:
seq ${2:-3} |
parallel --tty -j0 eval sidplayfp -wch{}.wav '{=$_=join" ",map{"-u".$_}grep!/#{[seq()]}/,(1..total_jobs())=}' ${#:3} -q $1 '2>/dev/null'
I cannot understand the behaviour of this bash script (which I cut it out of a longer real use case):
# This is test.sh
cmd="echo -e \"\n\n\n\t===== Hello World =====\n\n\""
sh -c "$cmd"
What it prints is:
$ ./test.sh
-e
===== Hello World =====
$
If I remove the -e flag, everything is printed correctly, with quoted chars correctly interpreted and without the '-e' spoil: but it shouldn't be like that.
My bash is: GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17), under macOS.
In Posix mode (when run as sh), bash 3.2's echo command takes no options; -e is just another argument to write to standard output. Compare:
$ bash -c 'echo -e "a\tb"'
a b
$ sh -c 'echo -e "a\tb"'
-e a b
A literal tab is printed in both cases because Posix echo behaves the same as bash echo -e.
For this reason, printf is almost always better to use than echo to provide consistent behavior.
cmd='printf "\n\n\n\t===== Hello World =====\n\n"'
sh -c "$cmd"
sh-4.2# cat test.sh
cmd="echo -e \"\n\n\n\t===== Hello World =====\n\n\""
sh -c "$cmd"
sh-4.2# ./test.sh
===== Hello World =====
sh-4.2#
It is getting printed correctly on my machine
OK, I think I found it myself, from here:
sh, the Bourne shell, is old. Its behaviour is specified by the POSIX standard. If you want new behaviour, you use bash, the Bourne Again shell, which gets new features added to it all the time. On many systems, sh is just bash, and bash turns on a compatibility mode when run under that name.
Groan...
When moving from Ubuntu 14.04 to 16.04, I've noticed several of my Bash scripts failing due to missing exported functions. I wonder whether this is related to the fixes for the Shellshock bug, even though I simply export -f the functions, and not relying on the Bash-internal function representation. The failure does not occur in a direct Bash subshell, only if there's another process in between. For example, Bash invoking awk / Perl / Vim invoking another Bash. Here's an example with Perl:
Good
$ foo() { echo "foobar"; }
$ export -f foo
$ export -f; foo
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ bash -c "export -f; foo"
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ perl -e 'system("bash -c \"export -f; foo\"")'
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ echo $BASH_VERSION
4.3.11(1)-release
Bad
$ foo() { echo "foobar"; }
$ export -f foo
$ export -f; foo
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ bash -c "export -f; foo"
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ perl -e 'system("bash -c \"export -f; foo\"")'
bash: foo: command not found
$ echo $BASH_VERSION
4.3.42(1)-release
Am I doing something wrong, or is this a bug?
Edit: #chepner pointed out that Bash uses specially-named shell identifiers to store the functions. When going through dash (0.5.8-2.1ubuntu2, working with 0.5.7-4ubuntu1), these identifiers are removed. With ksh, they are kept alive. I checked with
$ dash
$ sudo strings /proc/$$/environ | grep foo # Still passed from Bash to Dash
BASH_FUNC_foo%%=() { echo "foobar"
$ bash
$ sudo strings /proc/$$/environ | grep foo # But went missing from Dash to Bash
$ exit
$ exit
$ ksh
$ sudo strings /proc/$$/environ | grep foo
BASH_FUNC_foo%%=() { echo "foobar"
$ bash
$ sudo strings /proc/$$/environ | grep foo # Kept from Ksh to Bash
BASH_FUNC_foo%%=() { echo "foobar"
Likewise, the behavior of Vim can be changed via :set shell=/bin/bash / :set shell=/bin/ksh
So, is dash to blame?!
TL;DR: Known dash problem; gray area, might be fixed; better not rely on exports surviving a non-bash parent.
This is caused by a change in dash 0.5.8; cp. dash removes exported bash functions from the environment.
There is no consensus yet whether this will be fixed. POSIX seems to allow the stripping of invalid environment entries, other (more obscure) shells apparently do this as well, but it causes problems in various applications, especially because /bin/sh is symlinked to dash (and therefore the default shell) in Ubuntu.
My personal use case is some short utility functions that I've put in my ~/.profile, and which I reference in some shell scripts. One of those runs in an autostarted Conky daemon, and that one misses the functions because the autostart happens through dash. I can work around this. The FPATH autoload mechanism from Korn shell would be nice to have in Bash, too...
This is not ideal, but you can define your functions inside of Dash:
$ foo() { echo "foobar"; }
$ dash -c "$(declare -f); foo"
foobar