In my bash script I use while read loop and a helper function fv():
fv() {
case "$1" in
out) echo $VAR
;;
* ) VAR="$VAR $1"
;;
esac
}
cat "$1" | while read line
do
...some processings...
fv some-str-value
done
echo "`fv out`"
in a hope that I can distil value from while read loop in a variable accessible in rest of the script.
But above snippet is no good, as I get no output.
Is there easy way to solve this - output string from this loop in a variable that would be accessible in rest of the script - without reformatting my script?
As no one has explained to you why your code didn't work, I will.
When you use cat "$1" |, you are making that loop execute in a subshell. The VAR variable used in that subshell starts as a copy of VAR from the main script, and any changes to it are limited to that copy (the subshell's scope), they don't affect the script's original VAR. By removing the useless use of cat, you remove the pipeline and so the loop is executed in the main shell, so it can (and does) alter the correct copy of VAR.
Replace your while loop by while read line ; do ... ; done < $1:
#!/bin/bash
function fv
{
case "$1" in
out) echo $VAR
;;
* ) VAR="$VAR $1"
;;
esac
}
while read line
do
fv "$line\n"
done < "$1"
echo "$(fv out)"
Stop piping to read.
done < "$1"
Related
I'm trying to script some long/repetitive configuration & build operations in bash.
Started with a function that displays given command plus args and then executes it with given args.
Function definition follows:
runit () {
cmd=${#}
echo "${cmd}"
${cmd}
}
runit touch /tmp/cltconf1
Above (not involving redirection operator) displays the command and touches the target file as expected.
runit echo "gEnableSecureClient=True" > clt1.conf
Above (involving redirection operator) doesn't display command before execution and the content of clt1.conf file after the execution is:
echo gEnableSecureClient=True
gEnableSecureClient=True
I could understand that the redirection is not being controlled and thus causing the echo ${cmd} to actually write content echo gEnableSecureClient=True to clt1.conf and then actual command execution then writes content gEnableSecureClient=True.
I want to find out if this redirection operator can be controlled for my requirement.
Any shopts or escape sequence handling would help.
Your question is:
How to control redirection operator passed as one of the arguments to a bash function?
Invent your own invention and use it to pass the context (filename to be redirected to) to your function. You could use a global variable or you could use positional arguments with some rarely used argument style, like for example ++, for example:
runit() {
# parse arguments - detect `++` followed by `>` and filename
local cmd
while (($#)); do
case "$1" in
"++") break; ;;
*) cmd+=("$1"); ;;
esac
shift
done
local outf=/dev/stdout
if (($#)) && [[ "$1" == "++" ]]; then
shift
while (($#)); do
case "$1" in
">") outf=$2; shift; ;;
*) echo "ERROR: Invalid arguments: expected >" >&2; return 1; ;;
esac
shift
done
fi
if (($#)); then
echo "ERROR: internal error when parsing arguments" >&2; return 1
fi
echo "Running ${cmd[*]} > $outf"
"${cmd[#]}" > "$outf"
}
runit echo "gEnableSecureClient=True" ++ '>' clt1.conf
or example with global var, way simpler, but more spagetti:
runit() {
if [[ -n "${runit_outf:-}" ]]; then
echo "$* > $runit_outf"
"$#" > "$runit_outf"
runit_outf= # let's clear it after use
else
echo "$*"
"$#"
fi
}
# outputs to stdout
runit echo 123
runit_outf=clt1.conf # spaghetti code
runit echo 123 # outputs to file
This is just a template code that I did not test and have written in StackOverflow. It will not handle file descriptors - for that, you could write your own logic that would either parse the expression - ie. detect & in >&1, or write very unsafe code and call eval.
The presented above way is not recommended in any way, rather I wouldn't ever write code like that and would strongly discourage writing such complicated logic to handle simple cases. Instead, you should differentiate the output of a command from the logging stream of command, typically you would:
runit() {
local cmd
cmd=("$#")
echo "${cmd[*]}" >&2
"${cmd[#]}"
}
or use a dedicated beforehand-opened file descriptor in your code or redirect output to /dev/tty depending on the situation.
Before continuing further, you should definitely research what are and when to use bash arrays, research word splitting, and filename expansion, and check your script with https://shellcheck.net . Your function as it is now will break in surprisingly many ways when passing arguments with spaces and with special characters like * or ?.
I am writing bash script given below (Please ignore the capital letters variable names, this is just my test file):
#!/bin/bash
create_nodes_directories(){
HOSTS=(192.168.110.165 192.168.110.166 192.168.110.167)
accounts=('accountnum11' 'accountnum12' 'accountnum13')
for i in "${!HOSTS[#]}"; do
read -r curhost _ < <(hostname -I)
printf 'Enter the key pair for the %s node\n' "${accounts[i]}"
printf "Enter public key\n"
read -r EOS_PUB_KEY
printf "Enter private key\n"
read -r EOS_PRIV_KEY
PRODUCER=${accounts[i]}
args=()
args+=("$curhost")
for j in "${!HOSTS[#]}"; do
if [[ "$i" != "$j" ]]; then
args+=("${HOSTS[$j]}")
else
continue;
fi
done
#echo 'Array before test:'"${args[*]}"
create_genesis_start_file "$EOS_PUB_KEY" "$EOS_PRIV_KEY" "${HOSTS[$i]}" "$PRODUCER" args
create_start_file "$EOS_PUB_KEY" "$EOS_PRIV_KEY" "${HOSTS[$i]}" "$PRODUCER" args
done
}
create_genesis_start_file(){
EOS_PUB_KEY=$1
EOS_PRIV_KEY=$2
CURRENTHOST=$3
PRODUCER=$4
peerags="$5[#]"
peers=("${!peerags}")
echo 'Genesis Currenthost is:'"$CURRENTHOST"
#echo "${peers[*]}"
VAR=""
length=${#peers[#]}
last=$((length - 1))
for i in "${!peers[#]}" ; do
if [[ "$i" == "$last" ]]; then
VAR+="--p2p-peer-address ${peers[$i]}:8888 \\"
else
VAR+=$"--p2p-peer-address ${peers[$i]}:8888 \\"$'\n\t'
fi
done
}
create_start_file(){
EOS_PUB_KEY=$1
EOS_PRIV_KEY=$2
CURRENTHOST=$3
PRODUCER=$4
peerags="$5[#]"
peers=("${!peerags}")
echo 'Start file Currenthost is:'"$CURRENTHOST"
#echo "${peers[*]}"
}
create_nodes_directories
For every iteration of the first for loop, I am displaying the third argument $CURRENTHOST which is passed to functions create_genesis_start_file and create_start_file.
For first iteration, output is:
Genesis Currenthost is:192.168.110.165
Start file Currenthost is:192.168.110.167
Second iteration:
Genesis Currenthost is:192.168.110.166
Start file Currenthost is:192.168.110.167
Third iteration,
Genesis Currenthost is:192.168.110.167
Start file Currenthost is:192.168.110.167
Genesis Currenthost is as expected and Start file Currenthost should be same with it. I am not getting why the Start file Currenthost is always set as 192.168.110.167.
If I remove the below code from create_genesis_start_file it is working fine:
VAR=""
length=${#peers[#]}
last=$((length - 1))
for i in "${!peers[#]}" ; do
if [[ "$i" == "$last" ]]; then
VAR+="--p2p-peer-address ${peers[$i]}:8888 \\"
else
VAR+=$"--p2p-peer-address ${peers[$i]}:8888 \\"$'\n\t'
fi
done
I am not getting the exact problem why the variable value is getting changed? Please help.
The "$5[#]" looks odd to me. You can't use a scalar $5 as if it were an array.
It seems that you want to pass a whole array as parameter. Since bash does not have a native way to do this, I suggest that on the calling side, you pass "${args[#]}" as parameter, and inside your function, you do a
shift 4
peers=( "$#" )
Another possibility, which however violates the idea of encapsulation, is to treet peers as a global variable, which is accessible to all functions. With this approach, you would on the caller side collect the information already in the variable peers instead of args.
From a programming style, global variables (accross function boundaries) are usually disliked for good reasons, but in my personal opinion, if you just do simple shell scripting, I would find it an acceptable solution.
Bash allows to use: cat <(echo "$FILECONTENT")
Bash also allow to use: while read i; do echo $i; done </etc/passwd
to combine previous two this can be used: echo $FILECONTENT | while read i; do echo $i; done
The problem with last one is that it creates sub-shell and after the while loop ends variable i cannot be accessed any more.
My question is:
How to achieve something like this: while read i; do echo $i; done <(echo "$FILECONTENT") or in other words: How can I be sure that i survives while loop?
Please note that I am aware of enclosing while statement into {} but this does not solves the problem (imagine that you want use the while loop in function and return i variable)
The correct notation for Process Substitution is:
while read i; do echo $i; done < <(echo "$FILECONTENT")
The last value of i assigned in the loop is then available when the loop terminates.
An alternative is:
echo $FILECONTENT |
{
while read i; do echo $i; done
...do other things using $i here...
}
The braces are an I/O grouping operation and do not themselves create a subshell. In this context, they are part of a pipeline and are therefore run as a subshell, but it is because of the |, not the { ... }. You mention this in the question. AFAIK, you can do a return from within these inside a function.
Bash also provides the shopt builtin and one of its many options is:
lastpipe
If set, and job control is not active, the shell runs the last command of a pipeline not executed in the background in the current shell environment.
Thus, using something like this in a script makes the modfied sum available after the loop:
FILECONTENT="12 Name
13 Number
14 Information"
shopt -s lastpipe # Comment this out to see the alternative behaviour
sum=0
echo "$FILECONTENT" |
while read number name; do ((sum+=$number)); done
echo $sum
Doing this at the command line usually runs foul of 'job control is not active' (that is, at the command line, job control is active). Testing this without using a script failed.
Also, as noted by Gareth Rees in his answer, you can sometimes use a here string:
while read i; do echo $i; done <<< "$FILECONTENT"
This doesn't require shopt; you may be able to save a process using it.
Jonathan Leffler explains how to do what you want using process substitution, but another possibility is to use a here string:
while read i; do echo "$i"; done <<<"$FILECONTENT"
This saves a process.
This function makes duplicates $NUM times of jpg files (bash)
function makeDups() {
NUM=$1
echo "Making $1 duplicates for $(ls -1 *.jpg|wc -l) files"
ls -1 *.jpg|sort|while read f
do
COUNT=0
while [ "$COUNT" -le "$NUM" ]
do
cp $f ${f//sm/${COUNT}sm}
((COUNT++))
done
done
}
Bash allows to use: cat <(echo "$FILECONTENT")
Bash also allow to use: while read i; do echo $i; done </etc/passwd
to combine previous two this can be used: echo $FILECONTENT | while read i; do echo $i; done
The problem with last one is that it creates sub-shell and after the while loop ends variable i cannot be accessed any more.
My question is:
How to achieve something like this: while read i; do echo $i; done <(echo "$FILECONTENT") or in other words: How can I be sure that i survives while loop?
Please note that I am aware of enclosing while statement into {} but this does not solves the problem (imagine that you want use the while loop in function and return i variable)
The correct notation for Process Substitution is:
while read i; do echo $i; done < <(echo "$FILECONTENT")
The last value of i assigned in the loop is then available when the loop terminates.
An alternative is:
echo $FILECONTENT |
{
while read i; do echo $i; done
...do other things using $i here...
}
The braces are an I/O grouping operation and do not themselves create a subshell. In this context, they are part of a pipeline and are therefore run as a subshell, but it is because of the |, not the { ... }. You mention this in the question. AFAIK, you can do a return from within these inside a function.
Bash also provides the shopt builtin and one of its many options is:
lastpipe
If set, and job control is not active, the shell runs the last command of a pipeline not executed in the background in the current shell environment.
Thus, using something like this in a script makes the modfied sum available after the loop:
FILECONTENT="12 Name
13 Number
14 Information"
shopt -s lastpipe # Comment this out to see the alternative behaviour
sum=0
echo "$FILECONTENT" |
while read number name; do ((sum+=$number)); done
echo $sum
Doing this at the command line usually runs foul of 'job control is not active' (that is, at the command line, job control is active). Testing this without using a script failed.
Also, as noted by Gareth Rees in his answer, you can sometimes use a here string:
while read i; do echo $i; done <<< "$FILECONTENT"
This doesn't require shopt; you may be able to save a process using it.
Jonathan Leffler explains how to do what you want using process substitution, but another possibility is to use a here string:
while read i; do echo "$i"; done <<<"$FILECONTENT"
This saves a process.
This function makes duplicates $NUM times of jpg files (bash)
function makeDups() {
NUM=$1
echo "Making $1 duplicates for $(ls -1 *.jpg|wc -l) files"
ls -1 *.jpg|sort|while read f
do
COUNT=0
while [ "$COUNT" -le "$NUM" ]
do
cp $f ${f//sm/${COUNT}sm}
((COUNT++))
done
done
}
How do I set a variable in the parent shell, from a subshell?
a=3
(a=4)
echo $a
The whole point of a subshell is that it doesn't affect the calling session. In bash a subshell is a child process, other shells differ but even then a variable setting in a subshell does not affect the caller. By definition.
Do you need a subshell? If you just need a group then use braces:
a=3
{ a=4;}
echo $a
gives 4 (be careful of the spaces in that one). Alternatively, write the variable value to stdout and capture it in the caller:
a=3
a=$(a=4;echo $a)
echo $a
avoid using back-ticks ``, they are deprecated and can be difficult to read.
There is the gdb-bash-variable hack:
gdb --batch-silent -ex "attach $$" -ex 'set bind_variable("a", "4", 0)';
although that always sets a variable in the global scope, not just the parent scope
You don't. The subshell doesn't have access to its parent's environment. (At least within the abstraction that Bash provides. You could potentially try to use gdb, or smash the stack, or whatnot, to gain such access clandestinely. I wouldn't recommend that, though.)
One alternative is for the subshell to write assignment statements to a temporary file for its parent to read:
a=3
(echo 'a=4' > tmp)
. tmp
rm tmp
echo "$a"
If the problem is related to a while loop, one way to fix this is by using Process Substitution:
var=0
while read i;
do
# perform computations on $i
((var++))
done < <(find . -type f -name "*.bin" -maxdepth 1)
as shown here: https://stackoverflow.com/a/13727116/2547445
To change variables in a script called from a parent script, you can call the script preceded with a "."
(EDIT - for explanation)
In most shells "." is an alias for "source". the source command just inserts the text of another file at that position in the executing script. In the context of this question this answer avoids a sub-shell
a=3
echo $a
. ./calledScript.sh
echo $a
in calledScript.sh
a=4
Expected output
3
4
By reading the answer from #ruakh (thank you) with a temporary file approach and the comments asking for a file descriptors solution, I got the following idea:
a=3
. <(echo a=4; echo b=5)
echo $a
echo $b
It allows returning different variables at once (which could be an issue in the subshell variant of the accepted answer).
No iteration is needed,
No temporary file to take care of.
Close to the syntax proposed by the OP.
Result:
4
5
With xtrace enabled is visible that we are sourcing from the file descriptor created for the output of the subshell:
+ a=3
+ . /dev/fd/63 # <-- the file descriptor ;)
++ echo a=4
++ echo b=5
++ a=4
++ b=5
+ echo 4
4
+ echo 5
5
You can output the value in the subshell and assign the subshell output to a variable in the caller script:
# subshell.sh
echo Value
# caller
myvar=$(subshell.sh)
If the subshell has more to output you can separate the variable value and other messages by redirecting them into different output streams:
# subshell.sh
echo "Writing value" 1>&2
echo Value
# caller
myvar=$(subshell.sh 2>/dev/null) # or to somewhere else
echo $myvar
Alternatively, you can output variable assignments in the subshell, evaluate them in the caller script and avoid using files to exchange information:
# subshell.sh
echo "a=4"
# caller
# export $(subshell.sh) would be more secure, since export accepts name=value only.
eval $(subshell.sh)
echo $a
The last way I can think of is to use exit codes but this covers the integer values exchange only (and in a limited range) and breaks the convention for interpreting exit codes (0 for success non-0 for everything else).
Instead of accessing the variable from the parent shell, change the order of the commands and use the process substitution:
a=3
echo 5 | (read a)
echo $a
prints 3
a=3
read a < <(echo 5)
echo $a
prints 5
Another example:
let i=0
seq $RANDOM | while read r
do
let i=r
done
echo $i
vs
let i=0
while read r
do
let i=r
done < <(seq $RANDOM)
echo $i
Alternatively, when job control is inactive (e.g. in scripts) you can use the lastpipe shell option to achieve the same result without changing the order of the commands:
#!/bin/bash
shopt -s lastpipe
let i=0
seq $RANDOM | while read r
do
let i=r
done
echo $i
Unless you can apply all io to pipes and use file handles, basic variable updating is impossible within $(command) and any other sub-process.
Regular files, however, are bash's global variables for normal sequential processing. Note: Due to race conditions, this simple approach is not good for parallel processing.
Create an set/get/default function like this:
globalVariable() { # NEW-VALUE
# set/get/default globalVariable
if [ 0 = "$#" ]; then
# new value not given -- echo the value
[ -e "$aRam/globalVariable" ] \
&& cat "$aRam/globalVariable" \
|| printf "default-value-here"
else
# new value given -- set the value
printf "%s" "$1" > "$aRam/globalVariable"
fi
}
"$aRam" is the directory where values are stored. I like it to be a ram disk for speed and volatility:
aRam="$(mktemp -td $(basename "$0").XXX)" # temporary directory
mount -t tmpfs ramdisk "$aRam" # mount the ram disk there
trap "umount "$aRam" && rm -rf "$aRam"" EXIT # auto-eject
To read the value:
v="$(globalVariable)" # or part of any command
To set the value:
globalVariable newValue # newValue will be written to file
To unset the value:
rm -f "$aRam/globalVariable"
The only real reason for the access function is to apply a default value because cat will error given a non-existent file. It is also useful to apply other get/set logic. Otherwise, it would not be needed at all.
An ugly read method avoiding cat's non-existent file error:
v="$(cat "$aRam/globalVariable 2>/dev/null")"
A cool feature of this mess is that you can open another terminal and examine the contents of the files while the program is running.
While it's harder to get multiple variables out of a subshell, you can set multiple variables inside a function without using globals.
You can pass the name of a variable into a function that uses local -n to turn it into a special variable called a nameref:
myfunc() {
local -n OUT=$1
local -n SIDEEFFECT=$2
OUT='foo'
SIDEEFFECT='bar'
}
myfunc A B
echo $A
> foo
echo $B
> bar
This is the technique I ended up using instead of getting subshell FOO=$(myfunc) working setting multiple variables.
A very simple and practical method that allows multiple variables is as follows, eventually may add parameters to the call:
function ComplexReturn(){
# do your processing...
a=123
b=456
echo -n "AAA=${a}; BBB=${b};"
}
# ... this can be internal function or any subshell command
eval $(ComplexReturn)
echo $AAA $BBB