Related
I'm working with this:
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
I have a script like below:
#!/bin/bash
e=2
function test1() {
e=4
echo "hello"
}
test1
echo "$e"
Which returns:
hello
4
But if I assign the result of the function to a variable, the global variable e is not modified:
#!/bin/bash
e=2
function test1() {
e=4
echo "hello"
}
ret=$(test1)
echo "$ret"
echo "$e"
Returns:
hello
2
I've heard of the use of eval in this case, so I did this in test1:
eval 'e=4'
But the same result.
Could you explain me why it is not modified? How could I save the echo of the test1 function in ret and modify the global variable too?
When you use a command substitution (i.e., the $(...) construct), you are creating a subshell. Subshells inherit variables from their parent shells, but this only works one way: A subshell cannot modify the environment of its parent shell.
Your variable e is set within a subshell, but not the parent shell. There are two ways to pass values from a subshell to its parent. First, you can output something to stdout, then capture it with a command substitution:
myfunc() {
echo "Hello"
}
var="$(myfunc)"
echo "$var"
The above outputs:
Hello
For a numerical value in the range of 0 through 255, you can use return to pass the number as the exit status:
mysecondfunc() {
echo "Hello"
return 4
}
var="$(mysecondfunc)"
num_var=$?
echo "$var - num is $num_var"
This outputs:
Hello - num is 4
This needs bash 4.1 if you use {fd} or local -n.
The rest should work in bash 3.x I hope. I am not completely sure due to printf %q - this might be a bash 4 feature.
Summary
Your example can be modified as follows to archive the desired effect:
# Add following 4 lines:
_passback() { while [ 1 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; return $1; }
passback() { _passback "$#" "$?"; }
_capture() { { out="$("${#:2}" 3<&-; "$2_" >&3)"; ret=$?; printf "%q=%q;" "$1" "$out"; } 3>&1; echo "(exit $ret)"; }
capture() { eval "$(_capture "$#")"; }
e=2
# Add following line, called "Annotation"
function test1_() { passback e; }
function test1() {
e=4
echo "hello"
}
# Change following line to:
capture ret test1
echo "$ret"
echo "$e"
prints as desired:
hello
4
Note that this solution:
Works for e=1000, too.
Preserves $? if you need $?
The only bad sideffects are:
It needs a modern bash.
It forks quite more often.
It needs the annotation (named after your function, with an added _)
It sacrifices file descriptor 3.
You can change it to another FD if you need that.
In _capture just replace all occurances of 3 with another (higher) number.
The following (which is quite long, sorry for that) hopefully explains, how to adpot this recipe to other scripts, too.
The problem
d() { let x++; date +%Y%m%d-%H%M%S; }
x=0
d1=$(d)
d2=$(d)
d3=$(d)
d4=$(d)
echo $x $d1 $d2 $d3 $d4
outputs
0 20171129-123521 20171129-123521 20171129-123521 20171129-123521
while the wanted output is
4 20171129-123521 20171129-123521 20171129-123521 20171129-123521
The cause of the problem
Shell variables (or generally speaking, the environment) is passed from parental processes to child processes, but not vice versa.
If you do output capturing, this usually is run in a subshell, so passing back variables is difficult.
Some even tell you, that it is impossible to fix. This is wrong, but it is a long known difficult to solve problem.
There are several ways on how to solve it best, this depends on your needs.
Here is a step by step guide on how to do it.
Passing back variables into the parental shell
There is a way to pass back variables to a parental shell. However this is a dangerous path, because this uses eval. If done improperly, you risk many evil things. But if done properly, this is perfectly safe, provided that there is no bug in bash.
_passback() { while [ 0 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; }
d() { let x++; d=$(date +%Y%m%d-%H%M%S); _passback x d; }
x=0
eval `d`
d1=$d
eval `d`
d2=$d
eval `d`
d3=$d
eval `d`
d4=$d
echo $x $d1 $d2 $d3 $d4
prints
4 20171129-124945 20171129-124945 20171129-124945 20171129-124945
Note that this works for dangerous things, too:
danger() { danger="$*"; passback danger; }
eval `danger '; /bin/echo *'`
echo "$danger"
prints
; /bin/echo *
This is due to printf '%q', which quotes everything such, that you can re-use it in a shell context safely.
But this is a pain in the a..
This does not only look ugly, it also is much to type, so it is error prone. Just one single mistake and you are doomed, right?
Well, we are at shell level, so you can improve it. Just think about an interface you want to see, and then you can implement it.
Augment, how the shell processes things
Let's go a step back and think about some API which allows us to easily express, what we want to do.
Well, what do we want do do with the d() function?
We want to capture the output into a variable.
OK, then let's implement an API for exactly this:
# This needs a modern bash 4.3 (see "help declare" if "-n" is present,
# we get rid of it below anyway).
: capture VARIABLE command args..
capture()
{
local -n output="$1"
shift
output="$("$#")"
}
Now, instead of writing
d1=$(d)
we can write
capture d1 d
Well, this looks like we haven't changed much, as, again, the variables are not passed back from d into the parent shell, and we need to type a bit more.
However now we can throw the full power of the shell at it, as it is nicely wrapped in a function.
Think about an easy to reuse interface
A second thing is, that we want to be DRY (Don't Repeat Yourself).
So we definitively do not want to type something like
x=0
capture1 x d1 d
capture1 x d2 d
capture1 x d3 d
capture1 x d4 d
echo $x $d1 $d2 $d3 $d4
The x here is not only redundant, it's error prone to always repeate in the correct context. What if you use it 1000 times in a script and then add a variable? You definitively do not want to alter all the 1000 locations where a call to d is involved.
So leave the x away, so we can write:
_passback() { while [ 0 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; }
d() { let x++; output=$(date +%Y%m%d-%H%M%S); _passback output x; }
xcapture() { local -n output="$1"; eval "$("${#:2}")"; }
x=0
xcapture d1 d
xcapture d2 d
xcapture d3 d
xcapture d4 d
echo $x $d1 $d2 $d3 $d4
outputs
4 20171129-132414 20171129-132414 20171129-132414 20171129-132414
This already looks very good. (But there still is the local -n which does not work in oder common bash 3.x)
Avoid changing d()
The last solution has some big flaws:
d() needs to be altered
It needs to use some internal details of xcapture to pass the output.
Note that this shadows (burns) one variable named output,
so we can never pass this one back.
It needs to cooperate with _passback
Can we get rid of this, too?
Of course, we can! We are in a shell, so there is everything we need to get this done.
If you look a bit closer to the call to eval you can see, that we have 100% control at this location. "Inside" the eval we are in a subshell,
so we can do everything we want without fear of doing something bad to the parental shell.
Yeah, nice, so let's add another wrapper, now directly inside the eval:
_passback() { while [ 0 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; }
# !DO NOT USE!
_xcapture() { "${#:2}" > >(printf "%q=%q;" "$1" "$(cat)"); _passback x; } # !DO NOT USE!
# !DO NOT USE!
xcapture() { eval "$(_xcapture "$#")"; }
d() { let x++; date +%Y%m%d-%H%M%S; }
x=0
xcapture d1 d
xcapture d2 d
xcapture d3 d
xcapture d4 d
echo $x $d1 $d2 $d3 $d4
prints
4 20171129-132414 20171129-132414 20171129-132414 20171129-132414
However, this, again, has some major drawback:
The !DO NOT USE! markers are there,
because there is a very bad race condition in this,
which you cannot see easily:
The >(printf ..) is a background job. So it might still
execute while the _passback x is running.
You can see this yourself if you add a sleep 1; before printf or _passback.
_xcapture a d; echo then outputs x or a first, respectively.
The _passback x should not be part of _xcapture,
because this makes it difficult to reuse that recipe.
Also we have some unneded fork here (the $(cat)),
but as this solution is !DO NOT USE! I took the shortest route.
However, this shows, that we can do it, without modification to d() (and without local -n)!
Please note that we not neccessarily need _xcapture at all,
as we could have written everyting right in the eval.
However doing this usually isn't very readable.
And if you come back to your script in a few years,
you probably want to be able to read it again without much trouble.
Fix the race
Now let's fix the race condition.
The trick could be to wait until printf has closed it's STDOUT, and then output x.
There are many ways to archive this:
You cannot use shell pipes, because pipes run in different processes.
One can use temporary files,
or something like a lock file or a fifo. This allows to wait for the lock or fifo,
or different channels, to output the information, and then assemble the output in some correct sequence.
Following the last path could look like (note that it does the printf last because this works better here):
_passback() { while [ 0 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; }
_xcapture() { { printf "%q=%q;" "$1" "$("${#:2}" 3<&-; _passback x >&3)"; } 3>&1; }
xcapture() { eval "$(_xcapture "$#")"; }
d() { let x++; date +%Y%m%d-%H%M%S; }
x=0
xcapture d1 d
xcapture d2 d
xcapture d3 d
xcapture d4 d
echo $x $d1 $d2 $d3 $d4
outputs
4 20171129-144845 20171129-144845 20171129-144845 20171129-144845
Why is this correct?
_passback x directly talks to STDOUT.
However, as STDOUT needs to be captured in the inner command,
we first "save" it into FD3 (you can use others, of course) with '3>&1'
and then reuse it with >&3.
The $("${#:2}" 3<&-; _passback x >&3) finishes after the _passback,
when the subshell closes STDOUT.
So the printf cannot happen before the _passback,
regardless how long _passback takes.
Note that the printf command is not executed before the complete
commandline is assembled, so we cannot see artefacts from printf,
independently how printf is implemented.
Hence first _passback executes, then the printf.
This resolves the race, sacrificing one fixed file descriptor 3.
You can, of course, choose another file descriptor in the case,
that FD3 is not free in your shellscript.
Please also note the 3<&- which protects FD3 to be passed to the function.
Make it more generic
_capture contains parts, which belong to d(), which is bad,
from a reusability perspective. How to solve this?
Well, do it the desparate way by introducing one more thing,
an additional function, which must return the right things,
which is named after the original function with _ attached.
This function is called after the real function, and can augment things.
This way, this can be read as some annotation, so it is very readable:
_passback() { while [ 0 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; }
_capture() { { printf "%q=%q;" "$1" "$("${#:2}" 3<&-; "$2_" >&3)"; } 3>&1; }
capture() { eval "$(_capture "$#")"; }
d_() { _passback x; }
d() { let x++; date +%Y%m%d-%H%M%S; }
x=0
capture d1 d
capture d2 d
capture d3 d
capture d4 d
echo $x $d1 $d2 $d3 $d4
still prints
4 20171129-151954 20171129-151954 20171129-151954 20171129-151954
Allow access to the return-code
There is only on bit missing:
v=$(fn) sets $? to what fn returned. So you probably want this, too.
It needs some bigger tweaking, though:
# This is all the interface you need.
# Remember, that this burns FD=3!
_passback() { while [ 1 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; return $1; }
passback() { _passback "$#" "$?"; }
_capture() { { out="$("${#:2}" 3<&-; "$2_" >&3)"; ret=$?; printf "%q=%q;" "$1" "$out"; } 3>&1; echo "(exit $ret)"; }
capture() { eval "$(_capture "$#")"; }
# Here is your function, annotated with which sideffects it has.
fails_() { passback x y; }
fails() { x=$1; y=69; echo FAIL; return 23; }
# And now the code which uses it all
x=0
y=0
capture wtf fails 42
echo $? $x $y $wtf
prints
23 42 69 FAIL
There is still a lot room for improvement
_passback() can be elmininated with passback() { set -- "$#" "$?"; while [ 1 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; return $1; }
_capture() can be eliminated with capture() { eval "$({ out="$("${#:2}" 3<&-; "$2_" >&3)"; ret=$?; printf "%q=%q;" "$1" "$out"; } 3>&1; echo "(exit $ret)")"; }
The solution pollutes a file descriptor (here 3) by using it internally.
You need to keep that in mind if you happen to pass FDs.
Note thatbash 4.1 and above has {fd} to use some unused FD.
(Perhaps I will add a solution here when I come around.)
Note that this is why I use to put it in separate functions like _capture, because stuffing this all into one line is possible, but makes it increasingly harder to read and understand
Perhaps you want to capture STDERR of the called function, too.
Or you want to even pass in and out more than one filedescriptor
from and to variables.
I have no solution yet, however here is a way to catch more than one FD, so we can probably pass back the variables this way, too.
Also do not forget:
This must call a shell function, not an external command.
There is no easy way to pass environment variables out of external commands.
(With LD_PRELOAD= it should be possible, though!)
But this then is something completely different.
Last words
This is not the only possible solution. It is one example to a solution.
As always you have many ways to express things in the shell.
So feel free to improve and find something better.
The solution presented here is quite far from being perfect:
It was nearly not tested at all, so please forgive typos.
There is a lot of room for improvement, see above.
It uses many features from modern bash, so probably is hard to port to other shells.
And there might be some quirks I haven't thought about.
However I think it is quite easy to use:
Add just 4 lines of "library".
Add just 1 line of "annotation" for your shell function.
Sacrifices just one file descriptor temporarily.
And each step should be easy to understand even years later.
Maybe you can use a file, write to file inside function, read from file after it. I have changed e to an array. In this example blanks are used as separator when reading back the array.
#!/bin/bash
declare -a e
e[0]="first"
e[1]="secondddd"
function test1 () {
e[2]="third"
e[1]="second"
echo "${e[#]}" > /tmp/tempout
echo hi
}
ret=$(test1)
echo "$ret"
read -r -a e < /tmp/tempout
echo "${e[#]}"
echo "${e[0]}"
echo "${e[1]}"
echo "${e[2]}"
Output:
hi
first second third
first
second
third
What you are doing, you are executing test1
$(test1)
in a sub-shell( child shell ) and Child shells cannot modify anything in parent.
You can find it in bash manual
Please Check: Things results in a subshell here
I had a similar problem when I wanted to remove temporary files I had created automatically. The solution I came up with was not to use command substitution, but rather to pass the name of the variable, that should take the final result, into the function. E.g.
#!/usr/bin/env bash
# array that keeps track of tmp-files
remove_later=()
# function that manages tmp-files
new_tmp_file() {
file=$(mktemp)
remove_later+=( "$file" )
# assign value (safe form of `eval "$1=$file"`)
printf -v "$1" -- "$file"
}
# function to remove all tmp-files
remove_tmp_files() { rm -- "${remove_later[#]}"; }
# define trap to remove all tmp-files upon EXIT
trap remove_tmp_files EXIT
# generate tmp-files
new_tmp_file tmpfile1
new_tmp_file tmpfile2
So, adapting this to the OP, it would be:
#!/usr/bin/env bash
e=2
function test1() {
e=4
printf -v "$1" -- "hello"
}
test1 ret
echo "$ret"
echo "$e"
Works and has no restrictions on the "return value".
Assuming that local -n is available, the following script lets the function test1 modify a global variable:
#!/bin/bash
e=2
function test1() {
local -n var=$1
var=4
echo "hello"
}
test1 e
echo "$e"
Which gives the following output:
hello
4
I'm not sure if this works on your terminal, but I found out that if you don't provide any outputs whatsoever it gets naturally treated as a void function, and can make global variable changes.
Here's the code I used:
let ran1=$(( (1<<63)-1)/3 ))
let ran2=$(( (1<<63)-1)/5 ))
let c=0
function randomize {
c=$(( ran1+ran2 ))
ran2=$ran1
ran1=$c
c=$(( c > 0 ))
}
It's a simple randomizer for games that effectively modifies the needed variables.
It's because command substitution is performed in a subshell, so while the subshell inherits the variables, changes to them are lost when the subshell ends.
Reference:
Command substitution, commands grouped with parentheses, and asynchronous commands are invoked in a subshell environment that is a duplicate of the shell environment
A solution to this problem, without having to introduce complex functions and heavily modify the original one, is to store the value in a temporary file and read / write it when needed.
This approach helped me greatly when I had to mock a bash function called multiple times in a bats test case.
For example, you could have:
# Usage read_value path_to_tmp_file
function read_value {
cat "${1}"
}
# Usage: set_value path_to_tmp_file the_value
function set_value {
echo "${2}" > "${1}"
}
#----
# Original code:
function test1() {
e=4
set_value "${tmp_file}" "${e}"
echo "hello"
}
# Create the temp file
# Note that tmp_file is available in test1 as well
tmp_file=$(mktemp)
# Your logic
e=2
# Store the value
set_value "${tmp_file}" "${e}"
# Run test1
test1
# Read the value modified by test1
e=$(read_value "${tmp_file}")
echo "$e"
The drawback is that you might need multiple temp files for different variables. And also you might need to issue a sync command to persist the contents on the disk between one write and read operations.
You can always use an alias:
alias next='printf "blah_%02d" $count;count=$((count+1))'
I want to merge all files into one. Here, the last argument is the destination file name.
I want to take last argument and then in loop stop before last arguments.
Here code is given that I want to implement:
echo "No. of Argument : $#"
for i in $* - 1
do
echo $i
cat $i >> last argument(file)
done
How to achieve that?
Using bash:
fname=${!#}
for a in "${#:1:$# - 1}"
do
echo "$a"
cat "$a" >>"$fname"
done
In bash, the last argument to a script is ${!#}. So, that is where we get the file name.
bash also allows selecting elements from an array. To start with a simple example, observe:
$ set -- a b c d e f
$ echo "${#}"
a b c d e f
$ echo "${#:2:4}"
b c d e
In our case, we want to select elements from the first to the second to last. The first is number 1. The last is number $#. We want to select all but the last. WE thus want $# - 1 elements of the array. Therefore, to select the arguments from the first to the second to last, we use:
${#:1:$# - 1}
A POSIX-compliant method:
eval last_arg=\$$#
while [ $# -ne 1 ]; do
echo "$1"
cat "$1" >> "$last_arg"
shift
done
Here, eval is safe, because you are only expanding a read-only parameter in the string that eval will execute. If you don't want to unset the positional parameters via shift, you can iterate over them, using a counter to break out of the loop early.
eval last_arg=\$$#
i=1
for arg in "$#"; do
echo "$arg"
cat "$arg" >> "$last_arg"
i=$((i+1))
if [ "$i" = "$#" ]; then
break
fi
done
I'm working with this:
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
I have a script like below:
#!/bin/bash
e=2
function test1() {
e=4
echo "hello"
}
test1
echo "$e"
Which returns:
hello
4
But if I assign the result of the function to a variable, the global variable e is not modified:
#!/bin/bash
e=2
function test1() {
e=4
echo "hello"
}
ret=$(test1)
echo "$ret"
echo "$e"
Returns:
hello
2
I've heard of the use of eval in this case, so I did this in test1:
eval 'e=4'
But the same result.
Could you explain me why it is not modified? How could I save the echo of the test1 function in ret and modify the global variable too?
When you use a command substitution (i.e., the $(...) construct), you are creating a subshell. Subshells inherit variables from their parent shells, but this only works one way: A subshell cannot modify the environment of its parent shell.
Your variable e is set within a subshell, but not the parent shell. There are two ways to pass values from a subshell to its parent. First, you can output something to stdout, then capture it with a command substitution:
myfunc() {
echo "Hello"
}
var="$(myfunc)"
echo "$var"
The above outputs:
Hello
For a numerical value in the range of 0 through 255, you can use return to pass the number as the exit status:
mysecondfunc() {
echo "Hello"
return 4
}
var="$(mysecondfunc)"
num_var=$?
echo "$var - num is $num_var"
This outputs:
Hello - num is 4
This needs bash 4.1 if you use {fd} or local -n.
The rest should work in bash 3.x I hope. I am not completely sure due to printf %q - this might be a bash 4 feature.
Summary
Your example can be modified as follows to archive the desired effect:
# Add following 4 lines:
_passback() { while [ 1 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; return $1; }
passback() { _passback "$#" "$?"; }
_capture() { { out="$("${#:2}" 3<&-; "$2_" >&3)"; ret=$?; printf "%q=%q;" "$1" "$out"; } 3>&1; echo "(exit $ret)"; }
capture() { eval "$(_capture "$#")"; }
e=2
# Add following line, called "Annotation"
function test1_() { passback e; }
function test1() {
e=4
echo "hello"
}
# Change following line to:
capture ret test1
echo "$ret"
echo "$e"
prints as desired:
hello
4
Note that this solution:
Works for e=1000, too.
Preserves $? if you need $?
The only bad sideffects are:
It needs a modern bash.
It forks quite more often.
It needs the annotation (named after your function, with an added _)
It sacrifices file descriptor 3.
You can change it to another FD if you need that.
In _capture just replace all occurances of 3 with another (higher) number.
The following (which is quite long, sorry for that) hopefully explains, how to adpot this recipe to other scripts, too.
The problem
d() { let x++; date +%Y%m%d-%H%M%S; }
x=0
d1=$(d)
d2=$(d)
d3=$(d)
d4=$(d)
echo $x $d1 $d2 $d3 $d4
outputs
0 20171129-123521 20171129-123521 20171129-123521 20171129-123521
while the wanted output is
4 20171129-123521 20171129-123521 20171129-123521 20171129-123521
The cause of the problem
Shell variables (or generally speaking, the environment) is passed from parental processes to child processes, but not vice versa.
If you do output capturing, this usually is run in a subshell, so passing back variables is difficult.
Some even tell you, that it is impossible to fix. This is wrong, but it is a long known difficult to solve problem.
There are several ways on how to solve it best, this depends on your needs.
Here is a step by step guide on how to do it.
Passing back variables into the parental shell
There is a way to pass back variables to a parental shell. However this is a dangerous path, because this uses eval. If done improperly, you risk many evil things. But if done properly, this is perfectly safe, provided that there is no bug in bash.
_passback() { while [ 0 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; }
d() { let x++; d=$(date +%Y%m%d-%H%M%S); _passback x d; }
x=0
eval `d`
d1=$d
eval `d`
d2=$d
eval `d`
d3=$d
eval `d`
d4=$d
echo $x $d1 $d2 $d3 $d4
prints
4 20171129-124945 20171129-124945 20171129-124945 20171129-124945
Note that this works for dangerous things, too:
danger() { danger="$*"; passback danger; }
eval `danger '; /bin/echo *'`
echo "$danger"
prints
; /bin/echo *
This is due to printf '%q', which quotes everything such, that you can re-use it in a shell context safely.
But this is a pain in the a..
This does not only look ugly, it also is much to type, so it is error prone. Just one single mistake and you are doomed, right?
Well, we are at shell level, so you can improve it. Just think about an interface you want to see, and then you can implement it.
Augment, how the shell processes things
Let's go a step back and think about some API which allows us to easily express, what we want to do.
Well, what do we want do do with the d() function?
We want to capture the output into a variable.
OK, then let's implement an API for exactly this:
# This needs a modern bash 4.3 (see "help declare" if "-n" is present,
# we get rid of it below anyway).
: capture VARIABLE command args..
capture()
{
local -n output="$1"
shift
output="$("$#")"
}
Now, instead of writing
d1=$(d)
we can write
capture d1 d
Well, this looks like we haven't changed much, as, again, the variables are not passed back from d into the parent shell, and we need to type a bit more.
However now we can throw the full power of the shell at it, as it is nicely wrapped in a function.
Think about an easy to reuse interface
A second thing is, that we want to be DRY (Don't Repeat Yourself).
So we definitively do not want to type something like
x=0
capture1 x d1 d
capture1 x d2 d
capture1 x d3 d
capture1 x d4 d
echo $x $d1 $d2 $d3 $d4
The x here is not only redundant, it's error prone to always repeate in the correct context. What if you use it 1000 times in a script and then add a variable? You definitively do not want to alter all the 1000 locations where a call to d is involved.
So leave the x away, so we can write:
_passback() { while [ 0 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; }
d() { let x++; output=$(date +%Y%m%d-%H%M%S); _passback output x; }
xcapture() { local -n output="$1"; eval "$("${#:2}")"; }
x=0
xcapture d1 d
xcapture d2 d
xcapture d3 d
xcapture d4 d
echo $x $d1 $d2 $d3 $d4
outputs
4 20171129-132414 20171129-132414 20171129-132414 20171129-132414
This already looks very good. (But there still is the local -n which does not work in oder common bash 3.x)
Avoid changing d()
The last solution has some big flaws:
d() needs to be altered
It needs to use some internal details of xcapture to pass the output.
Note that this shadows (burns) one variable named output,
so we can never pass this one back.
It needs to cooperate with _passback
Can we get rid of this, too?
Of course, we can! We are in a shell, so there is everything we need to get this done.
If you look a bit closer to the call to eval you can see, that we have 100% control at this location. "Inside" the eval we are in a subshell,
so we can do everything we want without fear of doing something bad to the parental shell.
Yeah, nice, so let's add another wrapper, now directly inside the eval:
_passback() { while [ 0 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; }
# !DO NOT USE!
_xcapture() { "${#:2}" > >(printf "%q=%q;" "$1" "$(cat)"); _passback x; } # !DO NOT USE!
# !DO NOT USE!
xcapture() { eval "$(_xcapture "$#")"; }
d() { let x++; date +%Y%m%d-%H%M%S; }
x=0
xcapture d1 d
xcapture d2 d
xcapture d3 d
xcapture d4 d
echo $x $d1 $d2 $d3 $d4
prints
4 20171129-132414 20171129-132414 20171129-132414 20171129-132414
However, this, again, has some major drawback:
The !DO NOT USE! markers are there,
because there is a very bad race condition in this,
which you cannot see easily:
The >(printf ..) is a background job. So it might still
execute while the _passback x is running.
You can see this yourself if you add a sleep 1; before printf or _passback.
_xcapture a d; echo then outputs x or a first, respectively.
The _passback x should not be part of _xcapture,
because this makes it difficult to reuse that recipe.
Also we have some unneded fork here (the $(cat)),
but as this solution is !DO NOT USE! I took the shortest route.
However, this shows, that we can do it, without modification to d() (and without local -n)!
Please note that we not neccessarily need _xcapture at all,
as we could have written everyting right in the eval.
However doing this usually isn't very readable.
And if you come back to your script in a few years,
you probably want to be able to read it again without much trouble.
Fix the race
Now let's fix the race condition.
The trick could be to wait until printf has closed it's STDOUT, and then output x.
There are many ways to archive this:
You cannot use shell pipes, because pipes run in different processes.
One can use temporary files,
or something like a lock file or a fifo. This allows to wait for the lock or fifo,
or different channels, to output the information, and then assemble the output in some correct sequence.
Following the last path could look like (note that it does the printf last because this works better here):
_passback() { while [ 0 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; }
_xcapture() { { printf "%q=%q;" "$1" "$("${#:2}" 3<&-; _passback x >&3)"; } 3>&1; }
xcapture() { eval "$(_xcapture "$#")"; }
d() { let x++; date +%Y%m%d-%H%M%S; }
x=0
xcapture d1 d
xcapture d2 d
xcapture d3 d
xcapture d4 d
echo $x $d1 $d2 $d3 $d4
outputs
4 20171129-144845 20171129-144845 20171129-144845 20171129-144845
Why is this correct?
_passback x directly talks to STDOUT.
However, as STDOUT needs to be captured in the inner command,
we first "save" it into FD3 (you can use others, of course) with '3>&1'
and then reuse it with >&3.
The $("${#:2}" 3<&-; _passback x >&3) finishes after the _passback,
when the subshell closes STDOUT.
So the printf cannot happen before the _passback,
regardless how long _passback takes.
Note that the printf command is not executed before the complete
commandline is assembled, so we cannot see artefacts from printf,
independently how printf is implemented.
Hence first _passback executes, then the printf.
This resolves the race, sacrificing one fixed file descriptor 3.
You can, of course, choose another file descriptor in the case,
that FD3 is not free in your shellscript.
Please also note the 3<&- which protects FD3 to be passed to the function.
Make it more generic
_capture contains parts, which belong to d(), which is bad,
from a reusability perspective. How to solve this?
Well, do it the desparate way by introducing one more thing,
an additional function, which must return the right things,
which is named after the original function with _ attached.
This function is called after the real function, and can augment things.
This way, this can be read as some annotation, so it is very readable:
_passback() { while [ 0 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; }
_capture() { { printf "%q=%q;" "$1" "$("${#:2}" 3<&-; "$2_" >&3)"; } 3>&1; }
capture() { eval "$(_capture "$#")"; }
d_() { _passback x; }
d() { let x++; date +%Y%m%d-%H%M%S; }
x=0
capture d1 d
capture d2 d
capture d3 d
capture d4 d
echo $x $d1 $d2 $d3 $d4
still prints
4 20171129-151954 20171129-151954 20171129-151954 20171129-151954
Allow access to the return-code
There is only on bit missing:
v=$(fn) sets $? to what fn returned. So you probably want this, too.
It needs some bigger tweaking, though:
# This is all the interface you need.
# Remember, that this burns FD=3!
_passback() { while [ 1 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; return $1; }
passback() { _passback "$#" "$?"; }
_capture() { { out="$("${#:2}" 3<&-; "$2_" >&3)"; ret=$?; printf "%q=%q;" "$1" "$out"; } 3>&1; echo "(exit $ret)"; }
capture() { eval "$(_capture "$#")"; }
# Here is your function, annotated with which sideffects it has.
fails_() { passback x y; }
fails() { x=$1; y=69; echo FAIL; return 23; }
# And now the code which uses it all
x=0
y=0
capture wtf fails 42
echo $? $x $y $wtf
prints
23 42 69 FAIL
There is still a lot room for improvement
_passback() can be elmininated with passback() { set -- "$#" "$?"; while [ 1 -lt $# ]; do printf '%q=%q;' "$1" "${!1}"; shift; done; return $1; }
_capture() can be eliminated with capture() { eval "$({ out="$("${#:2}" 3<&-; "$2_" >&3)"; ret=$?; printf "%q=%q;" "$1" "$out"; } 3>&1; echo "(exit $ret)")"; }
The solution pollutes a file descriptor (here 3) by using it internally.
You need to keep that in mind if you happen to pass FDs.
Note thatbash 4.1 and above has {fd} to use some unused FD.
(Perhaps I will add a solution here when I come around.)
Note that this is why I use to put it in separate functions like _capture, because stuffing this all into one line is possible, but makes it increasingly harder to read and understand
Perhaps you want to capture STDERR of the called function, too.
Or you want to even pass in and out more than one filedescriptor
from and to variables.
I have no solution yet, however here is a way to catch more than one FD, so we can probably pass back the variables this way, too.
Also do not forget:
This must call a shell function, not an external command.
There is no easy way to pass environment variables out of external commands.
(With LD_PRELOAD= it should be possible, though!)
But this then is something completely different.
Last words
This is not the only possible solution. It is one example to a solution.
As always you have many ways to express things in the shell.
So feel free to improve and find something better.
The solution presented here is quite far from being perfect:
It was nearly not tested at all, so please forgive typos.
There is a lot of room for improvement, see above.
It uses many features from modern bash, so probably is hard to port to other shells.
And there might be some quirks I haven't thought about.
However I think it is quite easy to use:
Add just 4 lines of "library".
Add just 1 line of "annotation" for your shell function.
Sacrifices just one file descriptor temporarily.
And each step should be easy to understand even years later.
Maybe you can use a file, write to file inside function, read from file after it. I have changed e to an array. In this example blanks are used as separator when reading back the array.
#!/bin/bash
declare -a e
e[0]="first"
e[1]="secondddd"
function test1 () {
e[2]="third"
e[1]="second"
echo "${e[#]}" > /tmp/tempout
echo hi
}
ret=$(test1)
echo "$ret"
read -r -a e < /tmp/tempout
echo "${e[#]}"
echo "${e[0]}"
echo "${e[1]}"
echo "${e[2]}"
Output:
hi
first second third
first
second
third
What you are doing, you are executing test1
$(test1)
in a sub-shell( child shell ) and Child shells cannot modify anything in parent.
You can find it in bash manual
Please Check: Things results in a subshell here
I had a similar problem when I wanted to remove temporary files I had created automatically. The solution I came up with was not to use command substitution, but rather to pass the name of the variable, that should take the final result, into the function. E.g.
#!/usr/bin/env bash
# array that keeps track of tmp-files
remove_later=()
# function that manages tmp-files
new_tmp_file() {
file=$(mktemp)
remove_later+=( "$file" )
# assign value (safe form of `eval "$1=$file"`)
printf -v "$1" -- "$file"
}
# function to remove all tmp-files
remove_tmp_files() { rm -- "${remove_later[#]}"; }
# define trap to remove all tmp-files upon EXIT
trap remove_tmp_files EXIT
# generate tmp-files
new_tmp_file tmpfile1
new_tmp_file tmpfile2
So, adapting this to the OP, it would be:
#!/usr/bin/env bash
e=2
function test1() {
e=4
printf -v "$1" -- "hello"
}
test1 ret
echo "$ret"
echo "$e"
Works and has no restrictions on the "return value".
Assuming that local -n is available, the following script lets the function test1 modify a global variable:
#!/bin/bash
e=2
function test1() {
local -n var=$1
var=4
echo "hello"
}
test1 e
echo "$e"
Which gives the following output:
hello
4
I'm not sure if this works on your terminal, but I found out that if you don't provide any outputs whatsoever it gets naturally treated as a void function, and can make global variable changes.
Here's the code I used:
let ran1=$(( (1<<63)-1)/3 ))
let ran2=$(( (1<<63)-1)/5 ))
let c=0
function randomize {
c=$(( ran1+ran2 ))
ran2=$ran1
ran1=$c
c=$(( c > 0 ))
}
It's a simple randomizer for games that effectively modifies the needed variables.
It's because command substitution is performed in a subshell, so while the subshell inherits the variables, changes to them are lost when the subshell ends.
Reference:
Command substitution, commands grouped with parentheses, and asynchronous commands are invoked in a subshell environment that is a duplicate of the shell environment
A solution to this problem, without having to introduce complex functions and heavily modify the original one, is to store the value in a temporary file and read / write it when needed.
This approach helped me greatly when I had to mock a bash function called multiple times in a bats test case.
For example, you could have:
# Usage read_value path_to_tmp_file
function read_value {
cat "${1}"
}
# Usage: set_value path_to_tmp_file the_value
function set_value {
echo "${2}" > "${1}"
}
#----
# Original code:
function test1() {
e=4
set_value "${tmp_file}" "${e}"
echo "hello"
}
# Create the temp file
# Note that tmp_file is available in test1 as well
tmp_file=$(mktemp)
# Your logic
e=2
# Store the value
set_value "${tmp_file}" "${e}"
# Run test1
test1
# Read the value modified by test1
e=$(read_value "${tmp_file}")
echo "$e"
The drawback is that you might need multiple temp files for different variables. And also you might need to issue a sync command to persist the contents on the disk between one write and read operations.
You can always use an alias:
alias next='printf "blah_%02d" $count;count=$((count+1))'
I have something like that:
...
args=$*
echo $args
...
result is
unusable1 unusable2 useful useful ... useful unusable3
I need remove all "unusable" args. They always at first, second and last position.
After some investigation i find ${*:3} bash syntax. It help remove first two.
...
args=${*:3}
echo $args
...
result is
useful useful ... useful unusable3
But I can't find how to remove last word using same nice syntax.
You can use a function/script like this to print all but last arguments:
func() {
echo "${#:1:$#-1}";
}
func aa bb cc dd ee
aa bb cc dd
func foo bar baz hello how are you
foo bar baz hello how are
args=${*:3} flattens your argument list. You don't want to do that. Consider following the pattern given below instead:
# this next line sets "$#" for testing purposes; you don't need it in real life
set -- \
"first argument" \
"second argument" \
"third argument" \
"fourth argument" \
"fifth argument"
# trim the first two
args=( "${#:2}" )
# trim the last one
args=( "${args[#]:1:$(( ${#args[#]} - 2 ))}" )
# demonstrate the output content
printf '<%s>\n' "${args[#]}"
Running the above yields the following output:
<third argument>
<fourth argument>
...and, by doing so, demonstrates that it's correctly keeping arguments together, even when they contain spaces or wildcard characters.
For a shell completion script, you might also consider:
printf '%q ' "${args[#]}"
...which quotes content in such a way as to be eval'able by the shell.
Using your syntax, you can use this :
args=${*:3:$#-3}
Explanation :
${*:offset:length}
offset is 3 to begin to the third argument and length is the number of arguments minus 3 (two first and last one).
You could use awk:
args="unusable1 unusable2 useful useful ... useful unusable3"
args=$(awk '{$1=$2=$NF="";print}' <<< "$args")
echo "$args"
Output:
useful useful ... useful
The command sets the the first, second and the last ($NF) position to an empty string. NF holds the number of fields in awk. Therefore $NF is the last column.
bash doesn't really provide such filtering methods for arrays. The best option is to just use a loop and filter one at a time.
for arg; do # Implicitly iterate over $#
[[ $arg =~ unusable ]] && continue
args+=( "$arg" )
done
Try with this:
args=$*
useful=${args#* }
useful=${useful#* }
useful=${useful% *}
Then you wlill get the interesting parameters in the $result variable.
I'm trying to list script's argument by printf function. Arguments are counted by iteration of $i. What should be in printf function?
I need something like
eval echo \$$i
but in printf function.
Edit: Have while cycle with iteration of $i and among other code, I have
printf "%s" $i
But, instead of $i, I need some code, that shows me value of argument.
In my case, it is name of file, and I need list them. One file in one iteration.
As noted in a comment, you normally do that with a loop such as:
i=1
for arg in "$#"
do
echo "$i: [$arg]"
((i++))
done
(If echo isn't allowed, use printf "%s\n" … where the … is whatever would have followed echo.)
You might also use indirect expansion to avoid the use of eval:
for i in 1 2 3 4; do echo "$i: [${!i}]"; done
You can generalize that with:
for i in $(seq 1 $#); do echo "$i: [${!i}]"; done
or
for ((i = 1; i <= $#; i++)); do echo "$i: [${!i}]"; done
For example, given:
set -- a b 'c d' ' e f '
all the loops produce the output:
1: [a]
2: [b]
3: [c d]
4: [ e f ]
The square brackets are merely there to delimit the argument values; it allows you to see the trailing blanks on the fourth line of output.
You might also be able to use:
printf "[%s]\n" "$#"
to get:
[a]
[b]
[c d]
[ e f ]
It is not very clear what you are asking for, but this will list the arguments passed to the script:
while [ $# -gt 0 ]; do
printf "%s\n" "$1"
shift
done
I guess what you are asking for is how to make the indirect reference to positional parameter i (i containing the position):
print "%s\n" ${!i}
To get your arguments in such a way that they can be fed back into the shell with the exact same value, the following bash extension can be used:
printf '%q ' "$#"; printf '\n'
This works even for rather unusual cases. Let's say that one of your arguments contains a newline:
./your-script 'hello
world' 'goodbye world'
This will be represented in the printf output:
$'hello\nworld' goodbye\ world
...with something you can use again in the shell:
$ echo $'hello\nworld' goodbye\ world
hello
world goodbye world