Shell sourced file output piped vs redirected has different effects. - shell

I'm struggling to understand the difference between | and > operators
I've looked in places like:
https://www.gnu.org/software/bash/manual/html_node/Redirections.html
and
Pipe vs redirect into process
But can't make enough sense of the explanations.
Here is my practical example:
test-a.sh:
alias testa='echo "alias testa here"'
echo "testa echo"
echo "testa echo2"
test-b.sh:
alias testb='echo "alias testb here"'
echo "testa echo"
echo "testa echo2"
test-pipes.sh:
function indent() {
input=$(cat)
echo "$input" | perl -p -e 's/(.*)/ \1/'
}
source test-a.sh | indent
testa
source test-b.sh > >(indent)
testb
output:
$ source test-pipes.sh
testa echo
testa echo2
test-pipes.sh:10: command not found: testa
testa echo
testa echo2
alias testb here
Piping doesn't allow the alias to be set in the current process, but the redirection does.
Can someone give a simple explanation?

From the bash man page:
Each command in a pipeline is executed as a separate process (i.e., in a sub‐shell).
Many things child processes do are isolated from the parent. Among the list are: changing the current directory, setting shell variables, setting environment variables, and aliases.
$ alias foo='echo bar' | :
$ foo
foo: command not found
$ foo=bar | :; echo $foo
$ export foo=bar | :; echo $foo
$ cd / | :; $ pwd
/home/jkugelman
Notice how none of the changes took effect. You can see the same thing with explicit subshells:
$ (alias foo='echo bar')
$ foo
foo: command not found
$ (foo=bar); echo $foo
$ (export foo=bar); echo $foo
$ (cd /); pwd
/home/jkugelman
Redirections, on the other hand, do not create subshells. They merely change where the input and output of a command go. The same goes with function calls. Functions are executed in the current shell, no subshell, so they're able to create aliases.

Related

setting variable from Multiple sub variables bash [duplicate]

foo='abc'
bar='xyz'
var=bar
How to I get access to 'xyz' when I have var?
I've tried:
$(echo $var)
and
$(eval echo $var)
I just get bar: command not found
Use bash indirect variable reference:
${!var}
And of course can be done with eval, not recommended:
eval 'echo $'"$var"
Why:
$ bar=xyz
$ var='bar;whoami'
$ eval 'echo $'"$var"
xyz
spamegg
Th command whoami is being evaluated too as part of evaluation by eval, imagine a destructive command instead of whoami.
Example:
$ bar='xyz'
$ var=bar
$ echo "${!var}"
xyz
$ eval 'echo $'"$var"
xyz

Is there a way to print interpolated shell commands while preserving redirections?

Suppose I have the following shell program.
#!/bin/sh
FOO="foo"
echo $FOO | cat
I want to generate another shell program that does the same thing as this one, except that all shell variables have been substituted. For example,
#!/bin/sh
echo "foo" | cat
I know that I can get close if I run the above program using #!/bin/sh -x, but that output does not preserve redirections. Instead, I get
+ FOO=foo
+ echo foo
+ cat
foo
Any ideas?
The following shell:
$ cat eval.sh
echo "#!/bin/sh"
FOO="foo"
echo "echo $FOO | cat"
will write a shell:
$ sh eval.sh
#!/bin/sh
echo foo | cat
which does what you need.

Can I put a breakpoint in shell script?

Is there a way to suspend the execution of the shell script to inspect the state of the environment or execute random commands?
alias combined with eval gives you basic functionality of breakpoints in calling context:
#!/bin/bash
shopt -s expand_aliases
alias breakpoint='
while read -p"Debugging(Ctrl-d to exit)> " debugging_line
do
eval "$debugging_line"
done'
f(){
local var=1
breakpoint
echo $'\n'"After breakpoint, var=$var"
}
f
At the breakpoint, you can input
echo $var
followed by
var=2
then Ctrl-d to exit from breakpoint.
Due to eval in the while loop, use with caution.
Bash or shell scripts do not have such debugging capabilities as other programming languages like Java, Python, etc.
We can put the echo "VAR_NAME=$VAR_NAME" command in the code where we want to log the variable value.
Also, a little bit more flexible solution is to put this code somewhere at the beginning in the shell script we want to debug:
function BREAKPOINT() {
BREAKPOINT_NAME=$1
echo "Enter breakpoint $BREAKPOINT_NAME"
set +e
/bin/bash
BREAKPOINT_EXIT_CODE=$?
set -e
if [[ $BREAKPOINT_EXIT_CODE -eq 0 ]]; then
echo "Continue after breakpoint $BREAKPOINT_NAME"
else
echo "Terminate after breakpoint $BREAKPOINT_NAME"
exit $BREAKPOINT_EXIT_CODE
fi
}
export -f BREAKPOINT
and then later, at the line of code where we need to break we invoke this function like this:
# some shell script here
BREAKPOINT MyBreakPoint
# and some other shell script here
So then the BREAKPOINT function will log some output then launch /bin/bash where we can run any echo or some other shell command we want. When we want to continue running the rest of the shell script (release breakpoint) we just need to execute exit command. If we need to terminate script execution we would run exit 1 command.
There exist solutions like bash-debug.
A poor-man's solution which works for me is the interactive shell.
By adding three lines of code, you can introspect and alter variables as follows:
Let's assume, that you have the script test.bash
A=FOO
export B=BAR
echo $A
echo $B
$ test.bash
FOO
BAR
If you add an interactive shell at line 3, you can look around and inspect variables which have been exported before:
A=FOO
export B=BAR
bash -c "$SHELL"
echo $A
echo $B
$ test.bash
$ echo $A
$ echo $B
BAR
$ exit
FOO
BAR
If you want to see all variables in your interactive shell, you have to add set -a to the preamble of your script, such that all variables and functions are exported:
set -a
A=FOO
export B=BAR
bash -c "$SHELL"
echo $A
echo $B
$ test.bash
$ echo $A
FOO
$ echo $B
BAR
$ exit
FOO
BAR
Note, that you cannot change the variables in your interactive shell. The only solution for me is to source an additional script of variables, which will be sourced rightafter the interactive shell
set -a
A=FOO
export B=BAR
bash -c "$SHELL"
source /tmp/var
echo $A
echo $B
$ test.bash
$ echo "export A=alice" > /tmp/var
$ echo "export B=bob" >> /tmp/var
$ exit
alice
bob

Regression: Exported Bash function lost after going through another process

When moving from Ubuntu 14.04 to 16.04, I've noticed several of my Bash scripts failing due to missing exported functions. I wonder whether this is related to the fixes for the Shellshock bug, even though I simply export -f the functions, and not relying on the Bash-internal function representation. The failure does not occur in a direct Bash subshell, only if there's another process in between. For example, Bash invoking awk / Perl / Vim invoking another Bash. Here's an example with Perl:
Good
$ foo() { echo "foobar"; }
$ export -f foo
$ export -f; foo
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ bash -c "export -f; foo"
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ perl -e 'system("bash -c \"export -f; foo\"")'
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ echo $BASH_VERSION
4.3.11(1)-release
Bad
$ foo() { echo "foobar"; }
$ export -f foo
$ export -f; foo
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ bash -c "export -f; foo"
foo ()
{
echo "foobar"
}
declare -fx foo
foobar
$ perl -e 'system("bash -c \"export -f; foo\"")'
bash: foo: command not found
$ echo $BASH_VERSION
4.3.42(1)-release
Am I doing something wrong, or is this a bug?
Edit: #chepner pointed out that Bash uses specially-named shell identifiers to store the functions. When going through dash (0.5.8-2.1ubuntu2, working with 0.5.7-4ubuntu1), these identifiers are removed. With ksh, they are kept alive. I checked with
$ dash
$ sudo strings /proc/$$/environ | grep foo # Still passed from Bash to Dash
BASH_FUNC_foo%%=() { echo "foobar"
$ bash
$ sudo strings /proc/$$/environ | grep foo # But went missing from Dash to Bash
$ exit
$ exit
$ ksh
$ sudo strings /proc/$$/environ | grep foo
BASH_FUNC_foo%%=() { echo "foobar"
$ bash
$ sudo strings /proc/$$/environ | grep foo # Kept from Ksh to Bash
BASH_FUNC_foo%%=() { echo "foobar"
Likewise, the behavior of Vim can be changed via :set shell=/bin/bash / :set shell=/bin/ksh
So, is dash to blame?!
TL;DR: Known dash problem; gray area, might be fixed; better not rely on exports surviving a non-bash parent.
This is caused by a change in dash 0.5.8; cp. dash removes exported bash functions from the environment.
There is no consensus yet whether this will be fixed. POSIX seems to allow the stripping of invalid environment entries, other (more obscure) shells apparently do this as well, but it causes problems in various applications, especially because /bin/sh is symlinked to dash (and therefore the default shell) in Ubuntu.
My personal use case is some short utility functions that I've put in my ~/.profile, and which I reference in some shell scripts. One of those runs in an autostarted Conky daemon, and that one misses the functions because the autostart happens through dash. I can work around this. The FPATH autoload mechanism from Korn shell would be nice to have in Bash, too...
This is not ideal, but you can define your functions inside of Dash:
$ foo() { echo "foobar"; }
$ dash -c "$(declare -f); foo"
foobar

Bash script: how to get the whole command line which ran the script

I would like to run a bash script and be able to see the command line used to launch it:
sh myscript.sh arg1 arg2 1> output 2> error
in order to know if the user used the "std redirection" '1>' and '2>', and therefore adapt the output of my script.
Is it possible with built-in variables ??
Thanks.
On Linux and some unix-like systems, /proc/self/fd/1 and /proc/self/fd/2 are symlinks to where your std redirections are pointing to. Using readlink, we can query if they were redirected or not by comparing them to the parent process' file descriptor.
We will however not use self but $$ because $(readlink /proc/"$$"/fd/1) spawns a new shell so self would no longer refer to the current bash script but to a subshell.
$ cat test.sh
#!/usr/bin/env bash
#errRedirected=false
#outRedirected=false
parentStderr=$(readlink /proc/"$PPID"/fd/2)
currentStderr=$(readlink /proc/"$$"/fd/2)
parentStdout=$(readlink /proc/"$PPID"/fd/1)
currentStdout=$(readlink /proc/"$$"/fd/1)
[[ "$parentStderr" == "$currentStderr" ]] || errRedirected=true
[[ "$parentStdout" == "$currentStdout" ]] || outRedirected=true
echo "$0 ${outRedirected:+>$currentStdout }${errRedirected:+2>$currentStderr }$#"
$ ./test.sh
./test.sh
$ ./test.sh 2>/dev/null
./test.sh 2>/dev/null
$ ./test.sh arg1 2>/dev/null # You will lose the argument order!
./test.sh 2>/dev/null arg1
$ ./test.sh arg1 2>/dev/null >file ; cat file
./test.sh >/home/camusensei/file 2>/dev/null arg1
$
Do not forget that the user can also redirect to a 3rd file descriptor which is open on something else...!
Not really possible. You can check whether stdout and stderr are pointing to a terminal: [ -t 1 -a -t 2 ]. But if they do, it doesn't necessarily mean they weren't redirected (think >/dev/tty5). And if they don't, you can't distinguish between stdout and stderr being closed and them being redirected. And even if you know for sure they are redirected, you can't tell from the script itself where they point after redirection.

Resources