When moving from Ubuntu 14.04 to 16.04, I've noticed several of my Bash scripts failing due to missing exported functions. I wonder whether this is related to the fixes for the Shellshock bug, even though I simply export -f the functions, and not relying on the Bash-internal function representation. The failure does not occur in a direct Bash subshell, only if there's another process in between. For example, Bash invoking awk / Perl / Vim invoking another Bash. Here's an example with Perl:
Good
$ foo() { echo "foobar"; }
$ export -f foo
$ export -f; foo
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ bash -c "export -f; foo"
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ perl -e 'system("bash -c \"export -f; foo\"")'
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ echo $BASH_VERSION
4.3.11(1)-release
Bad
$ foo() { echo "foobar"; }
$ export -f foo
$ export -f; foo
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ bash -c "export -f; foo"
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ perl -e 'system("bash -c \"export -f; foo\"")'
bash: foo: command not found
$ echo $BASH_VERSION
4.3.42(1)-release
Am I doing something wrong, or is this a bug?
Edit: #chepner pointed out that Bash uses specially-named shell identifiers to store the functions. When going through dash (0.5.8-2.1ubuntu2, working with 0.5.7-4ubuntu1), these identifiers are removed. With ksh, they are kept alive. I checked with
$ dash
$ sudo strings /proc/$$/environ | grep foo # Still passed from Bash to Dash
BASH_FUNC_foo%%=() { echo "foobar"
$ bash
$ sudo strings /proc/$$/environ | grep foo # But went missing from Dash to Bash
$ exit
$ exit
$ ksh
$ sudo strings /proc/$$/environ | grep foo
BASH_FUNC_foo%%=() { echo "foobar"
$ bash
$ sudo strings /proc/$$/environ | grep foo # Kept from Ksh to Bash
BASH_FUNC_foo%%=() { echo "foobar"
Likewise, the behavior of Vim can be changed via :set shell=/bin/bash / :set shell=/bin/ksh
So, is dash to blame?!
TL;DR: Known dash problem; gray area, might be fixed; better not rely on exports surviving a non-bash parent.
This is caused by a change in dash 0.5.8; cp. dash removes exported bash functions from the environment.
There is no consensus yet whether this will be fixed. POSIX seems to allow the stripping of invalid environment entries, other (more obscure) shells apparently do this as well, but it causes problems in various applications, especially because /bin/sh is symlinked to dash (and therefore the default shell) in Ubuntu.
My personal use case is some short utility functions that I've put in my ~/.profile, and which I reference in some shell scripts. One of those runs in an autostarted Conky daemon, and that one misses the functions because the autostart happens through dash. I can work around this. The FPATH autoload mechanism from Korn shell would be nice to have in Bash, too...
This is not ideal, but you can define your functions inside of Dash:
$ foo() { echo "foobar"; }
$ dash -c "$(declare -f); foo"
foobar
Related
I do have a simple function (in "reality" it is "a bit" more complex - just for illustration)
cat test.sh
_foocomplete(){
local list="abc\ndef\nghi"
local cur=${COMP_WORDS[COMP_CWORD]}
COMPREPLY=( $(echo -e ${list} | grep "${cur}") )
}
foo(){
echo "$1"
}
complete -F _foocomplete foo
Using this in bash, gives me
~ ❯ ps -o cmd= $$
bash
~ ❯ type foo
bash: type: foo: not found
~ ❯ source test.sh
~ ❯ foo <TAB>
abc def ghi
~ ❯ foo a<TAB>
abc
~ ❯ foo e<TAB>
def
Using it in zsh, is very similar
~ ❯ ps -o cmd= $$
-zsh
~ ❯ type -f foo
foo not found
~ ❯ source test.sh
~ ❯ foo <TAB>
abc def ghi
~ ❯ foo a<TAB>
abc
BUT that
~ ❯ foo e<TAB>
does not work (as in: does not, like in bash, give me def as option).
Is there - besides rewriting and using compdef - an "easy" explanation (and "fix")?
This is with no .zshrc (but also with e.g. Oh My Zsh)
As reply to autocomplete bash vs. zsh
cat test2.sh
foo(){
local list="abc\ndef\nghi"
echo -e ${list} | grep e
}
foo
~ ❯ /bin/zsh test2.sh
def
~ ❯ /bin/bash test2.sh
def
Thank you in advance
Zsh completion does "matching" itself while bash expects you to filter completion candidates with, e.g. grep. "def" does not begin with "e" so it is not completed by default. You can think of the default matching as putting * at just the cursor position.
I assume you must be using bashcompinit for that example to have even come close to working in zsh. Unfortunately that means that the usual hooks for configuring zsh completions are not available. bashcompinit leaves zsh's default matching active which as you observe makes it not behave in exactly the same way as bash - by using compadd's -U option it could have done that. Zsh's matching can usually be controlled with the matcher and matcher=list styles. Passing -M to compadd also works. The following is a basic native zsh completion similar to your bash example:
_foo() { compadd -M 'l:|=*' abc def ghi }
compdef _foo foo
The man page for echo says:
-n Do not print the trailing newline character. This may also be
achieved by appending `\c' to the end of the string, as is done by
iBCS2 compatible systems. Note that this option as well as the
effect of `\c' are implementation-defined in IEEE Std 1003.1-2001
(``POSIX.1'') as amended by Cor. 1-2002. Applications aiming for
maximum portability are strongly encouraged to use printf(1) to
suppress the newline character.
However this doesn't seem to work in sh on Mac:
sh-3.2$ echo $0
/bin/sh
sh-3.2$ which echo
/bin/echo
sh-3.2$ echo -n foo
-n foo
It works properly in bash:
bash-3.2$ echo $0
bash
bash-3.2$ which echo
/bin/echo
bash-3.2$ echo -n foo
foobash-3.2
FWIW this only seems to happen on Mac, on Linux it works properly:
$ echo $0
sh
$ echo -n foo
foo$
-n is a bash extension to echo. In version 3.2 (which ships with macOS), bash does not support the extension when invoked as sh. Starting with version 4.0 (some version of which is likely on your Linux box), bash does honor -n when invoked as sh.
Update: the xpg_echo option determines if bash's built-in echo is POSIX-compliant or not. In bash 3.2 (or at least the macOS build of 3.2), this option defaults to on; in bash 4.x, it defaults to off.
% sh -c 'shopt xpg_echo'
xpg_echo on
% ./sh -c 'shopt xpg_echo'
xpg_echo off
(./sh is a symlink to /usr/local/bin/bash, a local installation of bash 4.4 on my machine.)
I cannot understand the behaviour of this bash script (which I cut it out of a longer real use case):
# This is test.sh
cmd="echo -e \"\n\n\n\t===== Hello World =====\n\n\""
sh -c "$cmd"
What it prints is:
$ ./test.sh
-e
===== Hello World =====
$
If I remove the -e flag, everything is printed correctly, with quoted chars correctly interpreted and without the '-e' spoil: but it shouldn't be like that.
My bash is: GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17), under macOS.
In Posix mode (when run as sh), bash 3.2's echo command takes no options; -e is just another argument to write to standard output. Compare:
$ bash -c 'echo -e "a\tb"'
a b
$ sh -c 'echo -e "a\tb"'
-e a b
A literal tab is printed in both cases because Posix echo behaves the same as bash echo -e.
For this reason, printf is almost always better to use than echo to provide consistent behavior.
cmd='printf "\n\n\n\t===== Hello World =====\n\n"'
sh -c "$cmd"
sh-4.2# cat test.sh
cmd="echo -e \"\n\n\n\t===== Hello World =====\n\n\""
sh -c "$cmd"
sh-4.2# ./test.sh
===== Hello World =====
sh-4.2#
It is getting printed correctly on my machine
OK, I think I found it myself, from here:
sh, the Bourne shell, is old. Its behaviour is specified by the POSIX standard. If you want new behaviour, you use bash, the Bourne Again shell, which gets new features added to it all the time. On many systems, sh is just bash, and bash turns on a compatibility mode when run under that name.
Groan...
I'm struggling to understand the difference between | and > operators
I've looked in places like:
https://www.gnu.org/software/bash/manual/html_node/Redirections.html
and
Pipe vs redirect into process
But can't make enough sense of the explanations.
Here is my practical example:
test-a.sh:
alias testa='echo "alias testa here"'
echo "testa echo"
echo "testa echo2"
test-b.sh:
alias testb='echo "alias testb here"'
echo "testa echo"
echo "testa echo2"
test-pipes.sh:
function indent() {
input=$(cat)
echo "$input" | perl -p -e 's/(.*)/ \1/'
}
source test-a.sh | indent
testa
source test-b.sh > >(indent)
testb
output:
$ source test-pipes.sh
testa echo
testa echo2
test-pipes.sh:10: command not found: testa
testa echo
testa echo2
alias testb here
Piping doesn't allow the alias to be set in the current process, but the redirection does.
Can someone give a simple explanation?
From the bash man page:
Each command in a pipeline is executed as a separate process (i.e., in a sub‐shell).
Many things child processes do are isolated from the parent. Among the list are: changing the current directory, setting shell variables, setting environment variables, and aliases.
$ alias foo='echo bar' | :
$ foo
foo: command not found
$ foo=bar | :; echo $foo
$ export foo=bar | :; echo $foo
$ cd / | :; $ pwd
/home/jkugelman
Notice how none of the changes took effect. You can see the same thing with explicit subshells:
$ (alias foo='echo bar')
$ foo
foo: command not found
$ (foo=bar); echo $foo
$ (export foo=bar); echo $foo
$ (cd /); pwd
/home/jkugelman
Redirections, on the other hand, do not create subshells. They merely change where the input and output of a command go. The same goes with function calls. Functions are executed in the current shell, no subshell, so they're able to create aliases.
This bug appears when using shell commands in a ruby script. The commands are not executed as what would happen in the bash terminal.
The first two commands are reasonable, the third is not.
`echo foo` # => "foo\n"
`echo -n foo` # => "foo"
`echo -n foo | cat` # => "-n foo\n"
In bash, we would have the following:
$ echo foo # => "foo\n"
$ echo -n foo # => "foo"
$ echo -n foo | cat # => "foo"
Is there a way I am messing up the parameter passing to the calls to echo in the Ruby commands? If you're unfamiliar, the command echo returns the string it is given. Without the -n flag, it appends a newline character, when you add the -n, it does not. cat repeats whatever it is given.
First, Ruby uses /bin/sh, not bash, for the back tick operator. It is likely that on your system, /bin/sh is a link to /bin/bash, but bash behaves differently when invoked as sh, for better POSIX compliance. However, it's not perfect, as your example shows. The POSIX specification states that echo "shall not support any options", but leaves handling of an initial argument -n to be implementation-specific.
In a quick test with /bin/sh as a link to /bin/bash,
bash -c "echo -n foo" and bash -c "echo -n foo | cat" produced identical, newline-free results, but sh -c "echo -n foo" and sh -c "echo -n foo | cat" showed the results you report. (I am not sure how other shells, such as ksh, dash, or zsh, behave when invoked as sh, but by default they all treat -n as a newline suppressor.)
For predictable results, use printf instead, which never prints a newline unless a \n is included in the format string.
printf 'foo\n'
printf 'foo'
printf 'foo' | cat
UPDATE: This appears to be a bug in bash 3.2 that was fixed at some point in the 4.x series. With 4.1 or later, sh -c "echo -n foo | cat and sh -c "echo -n foo" produce the same output.