bash caller builtin stops working from exported function, why? - bash

I have very strange issues using bash and exported function to give me a reliable answer to the call of the builtin caller.
Here's my setup to illustrate this issue: Bash script bar defines and exports function bar1 and bar2. bar2 calls bar1. Bash script bar then execute bash script foo which will call bar1.
The caller builtin will then break only after the call of bar1. Is this normal ? can you explain why ? can you simplify the following code that exposes the issue ?
To be perfectly clear, here's how to reproduce on your system: Build both file:
cd /tmp
cat <<"EOF" > foo
#!/bin/bash
bar1
EOF
chmod +x foo
cat <<"EOF" > bar
#!/bin/bash
bar2() {
echo "$FUNCNAME IN: $(caller 0)" >&2
}
export -f bar2
bar1() {
echo "$FUNCNAME BEFORE: $(caller 0)" >&2
bar2
echo "$FUNCNAME AFTER: $(caller 0)" >&2
}
export -f bar1
./foo
EOF
chmod +x bar
You can then fiddle and see:
$ ./bar
bar1 BEFORE: 3 main ./foo
bar2 IN:
bar1 AFTER:
I expected (with acceptable variations on the line numbers):
$ ./bar
bar1 BEFORE: 9 main ./foo
bar2 IN: 5 bar ./foo
bar1 AFTER: 9 main ./foo
Ultimately, my question would be: how could I circumvent this issue and get the caller in all cases ?
ADDITIONAL INFO:
bash version: 4.3.42(1)-release (x86_64-pc-linux-gnu) from ubuntu package 4.3-14ubuntu1.

This is a bug in bash. It was fixed in version 4.4.
In the presence of exported functions, the BASH_SOURCE variable is not properly maintained. You can check it by displaying the contents of the FUNCNAME, BASH_SOURCE, BASH_LINENO special variables:
cd /tmp
cat <<"EOF" > foo
#!/bin/bash
bar1
EOF
chmod +x foo
cat <<"EOF" > bar
#!/bin/bash
bar2() {
echo "$FUNCNAME IN: $(caller 0) [${FUNCNAME[#]}] [${BASH_SOURCE[#]}] [${BASH_LINENO[#]}]" >&2
}
export -f bar2
bar1() {
echo "$FUNCNAME BEFORE: $(caller 0) [${FUNCNAME[#]}] [${BASH_SOURCE[#]}] [${BASH_LINENO[#]}]" >&2
bar2
echo "$FUNCNAME AFTER: $(caller 0) [${FUNCNAME[#]}] [${BASH_SOURCE[#]}] [${BASH_LINENO[#]}]" >&2
}
export -f bar1
./foo
EOF
chmod +x bar
Output of ./bar:
bar1 BEFORE: 3 main ./foo [bar1 main] [./foo] [3 0]
bar2 IN: [bar2 bar1 main] [./foo] [1 3 0]
bar1 AFTER: [bar1 main] [] [3 0]
As you can see, stack frames corresponding to invocations of exported functions aren't added to BASH_SOURCE, but whenever a function returns the topmost stack frame is popped.
Note that the FUNCNAME variable is not affected by this bug. Thus, if you need only the name of the caller you can obtain it as ${FUNCNAME[1]}.

Related

access modifiers in bash

Say I have a bash script and I want some variables to appear when sourced and others to only be accessible from within the script (both functions and variables). What's the convention to achieve this?
Let's say test.sh is your bash script.
What you can do is extract all the common items and put them in common.sh which can be sourced by other scripts.
The BASH_SOURCE array helps you here:
Consider this script, source.sh
#!/bin/bash
if [[ ${BASH_SOURCE[0]} == "$0" ]]; then
# this code is run when the script is _executed_
foo=bar
privFunc() { echo "running as a script"; }
main() {
privFunc
publicFunc
}
fi
# this code is run when script is executed or sourced
answer=42
publicFunc() { echo "Hello, world!"; }
echo "$0 - ${BASH_SOURCE[0]}"
[[ ${BASH_SOURCE[0]} == "$0" ]] && main
Running it:
$ bash source.sh
source.sh - source.sh
running as a script
Hello, world!
Sourcing it:
$ source source.sh
bash - source.sh
$ declare -p answer
declare -- answer="42"
$ declare -p foo
bash: declare: foo: not found
$ publicFunc
Hello, world!
$ privFunc
bash: privFunc: command not found
$ main
bash: main: command not found

Shell function in makefile may not work

From the docs:
The 'shell' function performs the same function that backquotes ('`')
perform in most shells: it does "command expansion". This means that it
takes as an argument a shell command and evaluates to the output of the
command. The only processing 'make' does on the result is to convert
each newline (or carriage-return / newline pair) to a single space. If
there is a trailing (carriage-return and) newline it will simply be
removed.
Version 1 of makefile, is:
foo := $(shell echo 'bar'; foo)
all:
#echo 'foo is: $(foo)'
.PHONY: all
Running, we get:
/bin/sh: 1: foo: not found
bar
foo is:
Version 2 of makefile, is:
foo := $(shell echo 'bar'; false)
all:
#echo 'foo is: $(foo)'
.PHONY: all
Running, we get:
foo is: bar
Now, given the above quote from the documentation:
The 'shell' function performs the same function that backquotes ('`')
perform in most shells: it does "command expansion".
We try and compare the following:
# Equivalent to version 1 of makefile above.
$ foo=`echo 'bar'; foo`
sh: 1: foo: not found
$ echo ${foo}
bar
# Equivalent to version 2 of makefile above.
$ foo=`echo 'bar'; false`
$ echo ${foo}
bar
So, not only is the shell function not consistent, but it differs from "command expansion" of shells, referenced to, in the docs, cited above!
Is there a rationale behind all this?
Converting a comment into a substantiated answer.
The exit status from echo 'bar'; foo is 127; the exit status from echo 'bar'; false is 1. Exit status 127 means that the command failed to execute in some way.
In the comment, I said that my suspicion is that make treats the two exit statuses differently.
When I create makefile.v3 that contains:
foo := $(shell echo 'bar'; exit ${exit})
all:
#echo 'foo is: $(foo)'
.PHONY: all
and run it as shown:
$ make -f makefile.v3
foo is: bar
$ make -f makefile.v3 exit=0
foo is: bar
$ make -f makefile.v3 exit=1
foo is: bar
$ make -f makefile.v3 exit=127
bar
foo is:
$ make -f makefile.v3 exit=128
foo is: bar
$ make -f makefile.v3 exit=129
foo is: bar
$ make -f makefile.v3 exit=126
foo is: bar
$ make -f makefile.v3 exit=255
foo is: bar
$
As hypothesized in the comment, the exit status of 127 is treated differently by make.
(For the record: tested with GNU Make 3.81 on Mac OS X 10.10.4.)

Pass $# to a function in a shellscript

Problem description
In a shell script, I want to iterate over all command line arguments ("$#") from inside a function. However, inside a function, $# refers to the function arguments, not the command line arguments. I tried passing the arguments to the function using a variable, but that doesn't help, since it breaks arguments with whitespaces.
How can I pass $# to a function in such a way that it does not break whitespace? I am sorry if this has been asked before, I tried searching for this question and there are a lot similar ones, but I didn't find an answer nevertheless.
Illustration
I made a shell script to illustrate the problem.
print_args.sh source listing
#!/bin/sh
echo 'Main scope'
for arg in "$#"
do
echo " $arg"
done
function print_args1() {
echo 'print_args1()'
for arg in "$#"
do
echo " $arg"
done
}
function print_args2() {
echo 'print_args2()'
for arg in $ARGS
do
echo " $arg"
done
}
function print_args3() {
echo 'print_args3()'
for arg in "$ARGS"
do
echo " $arg"
done
}
ARGS="$#"
print_args1
print_args2
print_args3
print_args.sh execution
$ ./print_args.sh foo bar 'foo bar'
Main scope
foo
bar
foo bar
print_args1()
print_args2()
foo
bar
foo
bar
print_args3()
foo bar foo bar
As you can see, I can't get the last foo bar to appear as a single argument. I want a function that gives the same output as the main scope.
You can use this BASH function:
#!/bin/bash
echo 'Main scope'
for arg in "$#"
do
echo " $arg"
done
function print_args1() {
echo 'print_args1()'
for arg in "$#"; do
echo " $arg"
done
}
function print_args3() {
echo 'print_args3()'
for arg in "${ARGS[#]}"; do
echo " $arg"
done
}
ARGS=( "$#" )
print_args1 "$#"
print_args3
You can see use of bash shebang at top:
#!/bin/bash
required be able to use BASH arrays.
Output:
bash ./print_args.sh foo bar 'foo bar'
Main scope
foo
bar
foo bar
print_args1()
foo
bar
foo bar
print_args3()
foo
bar
foo bar

How to give correct suggestions to tab complete when my words contains colons

I am writing a bash tab completion file for a utility that some times requires full URLs on the form: protocol://host:port. This contains two colons, which have proven to be problematic for tab completion. This is because the colons are treated as word breaks. I have read that I should not change COMP_WORDBREAKS directly, so I want to use the _get_comp_words_by_ref and __ltrim_colon_completions as suggested here: How to reset COMP_WORDBREAKS without effecting other completion script?
This works for a single colon, but the second colon causes a small problem as demonstrated in this minimal example:
This example shows the problem. It occurs for any number of colons in the suggestions.
[root#2e3e8853cc0c /]# cat /etc/bash_completion.d/foo
_foo()
{
local cur
COMPREPLY=()
_get_comp_words_by_ref -n : -c cur
COMPREPLY=( $(compgen -W "http://host:1234/aaa http://host:1234/bbb http://host:1234/ccc" -- ${cur}) )
__ltrim_colon_completions "$cur"
return 0
}
complete -F _foo foo
Hitting tab after foo successfully completes the common part. Hitting tab twice after that, yields the following suggestions:
[root#2e3e8853cc0c /]# foo http://host:1234/
1234/aaa 1234/bbb 1234/ccc
The desired result is ofcourse:
[root#2e3e8853cc0c /]# foo http://host:1234/
http://host:1234/aaa http://host:1234/bbb http://host:1234/ccc
After that, hitting a, b, or c plus tab works as expected, it completes the full URL.
Any suggestions to how I can produce the right output? Do I need to manually change the COMPREPLY variable, or am I just using the functions wrong?
I came up with a solution based on one trick I'm always using. Hope it would help.
_bar()
{
local CUR=$2
local cur
local -a compreply=()
local -a urls=(ftp://gnu.org \
http://host1:1234/aaa \
http://host2:1234/bbb \
http://host2:1234/ccc)
_get_comp_words_by_ref -n : -c cur
compreply=( $(compgen -W "${urls[*]}" -- "$cur") )
COMPREPLY=( "${compreply[#]}" )
__ltrim_colon_completions "$cur"
if [[ ${#COMPREPLY[#]} -gt 1 ]]; then
local common_prefix
common_prefix=$( printf '%s\n' "${COMPREPLY[#]}" \
| sed '$q;N;s/^\(.*\).*\n\1.*$/\1/;h;G;D' )
if [[ $common_prefix == "$CUR" ]]; then
COMPREPLY=( "${compreply[#]}" " " )
fi
fi
return 0
}
complete -F _bar bar
Following is what it would look like (tested with Bash 4.3.33):
[STEP 101] $ bar <TAB><TAB>
http://host1:1234/aaa http://host2:1234/ccc
ftp://gnu.org http://host2:1234/bbb
[STEP 101] $ bar f<TAB>
[STEP 101] $ bar ftp://gnu.org␣
[STEP 101] $ bar ftp://gnu.org <ENTER>
bash: bar: command not found
[STEP 102] $ bar h<TAB>
[STEP 102] $ bar http://host
[STEP 102] $ bar http://host<TAB><TAB>
http://host2:1234/bbb
http://host1:1234/aaa http://host2:1234/ccc
[STEP 102] $ bar http://host2<TAB>
[STEP 102] $ bar http://host2:1234/
[STEP 102] $ bar http://host2:1234/<TAB><TAB>
http://host2:1234/bbb http://host2:1234/ccc
[STEP 102] $ bar http://host2:1234/b<TAB>
[STEP 102] $ bar http://host2:1234/bbb␣
[STEP 102] $ bar http://host2:1234/bbb <ENTER>
bash: bar: command not found
[STEP 103] $
And actually the problem is not specific about two or more colons. One colon has the similar problem too.

Strange Bash function export for the Shellshock bug

Why does the code
date
bash -c "date"
declare -x date='() { echo today; }' #aka export date='() { echo today; }'
date
bash -c "date"
print
Wed Sep 24 22:01:50 CEST 2014
Wed Sep 24 22:01:50 CEST 2014
Wed Sep 24 22:01:50 CEST 2014
today
?
Where (and why) does the evaluation
date$date
happen and getting
date() {echo today; }
Ad: #Etan Reisner
I exporting a variable - not a function. Bash makes a function from it. The
export date='someting'
is still a variable regardless of its content. So, why is
export date='() { echo something; }' #Note, it is a variable, not function.
converted to an function?
The mentioned security advisory talks about the execution of the command following the variable, for example,
x='() { echo I do nothing; }; echo vulnerable' bash -c ':'
^^^^^^^^^^^^^^^
This is executed - this vunerability is CLOSED in version 4.3.25(1).
The command after the env-definition isn't executed in the latest Bash.
But the question remains - Why does Bash convert the exported variable to a function?
It is a bug ;) Full demo, based on #chepner's answer:
#Define three variables
foo='() { echo variable foo; }' # ()crafted
qux='() { echo variable qux; }' # ()crafted
bar='variable bar' # Normal
export foo qux bar # Export
#Define the same name functions (but not qux!)
foo() { echo "function foo"; }
bar() { echo "function bar"; }
declare -fx foo bar #Export
#printouts
echo "current shell foo variable:=$foo="
echo "current shell foo function:=$(foo)="
echo "current shell bar variable:=$bar="
echo "current shell bar function:=$(bar)="
echo "current shell qux variable:=$qux="
echo "current shell qux function:=$(qux)="
#subshell
bash -c 'echo subshell foo variable:=$foo='
bash -c 'echo subshell foo command :=$(foo)='
bash -c 'echo subshell bar variable:=$bar='
bash -c 'echo subshell bar command :=$(bar)='
bash -c 'echo subshell qux variable:=$qux='
bash -c 'echo subshell qux command :=$(qux)='
prints
current shell foo variable:=() { echo variable foo; }=
current shell foo function:=function foo=
current shell bar variable:=variable bar=
current shell bar function:=function bar=
current shell qux variable:=() { echo variable qux; }=
tt: line 20: qux: command not found
current shell qux function:==
subshell foo variable:== #<-- LOST the exported foo variable
subshell foo command :=function foo=
subshell bar variable:=variable bar=
subshell bar command :=function bar=
subshell qux variable:== #<-- And the variable qux got converted to
subshell qux command :=variable qux= #<-- function qux in the subshell (!!!).
Avoiding the long comments, here is code from the Bash sources:
if (privmode == 0 && read_but_dont_execute == 0 && STREQN ("() {", string, 4))
^^^^^^^^ THE PROBLEM
{
string_length = strlen (string);
temp_string = (char *)xmalloc (3 + string_length + char_index);
strcpy (temp_string, name);
temp_string[char_index] = ' ';
strcpy (temp_string + char_index + 1, string);
if (posixly_correct == 0 || legal_identifier (name))
parse_and_execute (temp_string, name, SEVAL_NONINT|SEVAL_NOHIST);
/* Ancient backwards compatibility. Old versions of bash exported
functions like name()=() {...} */
The "ancient" (seems) was better... :)
if (name[char_index - 1] == ')' && name[char_index - 2] == '(')
name[char_index - 2] = '\0';
The key point to remember is that
foo='() { echo 5; }'
only defines a string parameter with a string that looks a lot like a function. It's still a regular string:
$ echo $foo
() { echo 5; }
And not a function:
$ foo
bash: foo: command not found
Once foo is marked for export,
$ export foo
any child Bash will see the following string in its environment:
foo=() { echo 5; }
Normally, such strings become shell variables, using the part preceding the = as the name and the part following the value. However, Bash treats such strings specially by defining a function instead:
$ echo $foo
$ foo
5
You can see that the environment itself is not changed by examining it with something other than Bash:
$ perl -e 'print $ENV{foo}\n"'
() { echo 5
}
(The parent Bash replaces the semicolon with a newline when creating the child's environment, apparently). It's only the child Bash that creates a function instead of a shell variable from such a string.
The fact that foo could be both a parameter and a function within the same shell;
$ foo=5
$ foo () { echo 9; }
$ echo $foo
5
$ foo
9
explains why -f is needed with export. export foo would cause the string foo=5 to be added to the environment of a child; export -f foo is used to add the string foo=() { echo 9; }.
You are essentially manually exporting a function with the name date. (Since that is the format that bash uses internally to export functions. Which is suggested by Barmar in his answer. This mechanism is mentioned here at the very least.)
Then when you run bash it sees that exported function and uses it when you tell it to run date.
Is the question then where is that mechanism specified? My guess is it isn't since it is an internal detail.
This should show the merging of the behaviours if that helps anything.
$ bar() { echo automatic; }; export -f bar
$ declare -x foo='() { echo manual; }'
$ declare -p foo bar
declare -x foo="() { echo manual; }"
-bash: declare: bar: not found
$ type foo bar
-bash: type: foo: not found
bar is a function
bar ()
{
echo automatic
}
$ bash -c 'type foo bar'
foo is a function
foo ()
{
echo manual
}
bar is a function
bar ()
{
echo automatic
}
The answer to your question comes directly from man bash:
The export and declare -x commands allow parameters and functions
to be added to and deleted from the environment. If the value of a
parameter in the environment is modified, the new value becomes part
of the environment, replacing the old.
Thus
declare -x date='() { echo today; }'
replaces date in the environment. The next immediate call to date gives date as it exists in the script (which is unchanged). The call to bash -c "date" creates a new shell and executes date as defined by declare -x.

Resources