Read variable outside while loop [duplicate] - bash

Bash allows to use: cat <(echo "$FILECONTENT")
Bash also allow to use: while read i; do echo $i; done </etc/passwd
to combine previous two this can be used: echo $FILECONTENT | while read i; do echo $i; done
The problem with last one is that it creates sub-shell and after the while loop ends variable i cannot be accessed any more.
My question is:
How to achieve something like this: while read i; do echo $i; done <(echo "$FILECONTENT") or in other words: How can I be sure that i survives while loop?
Please note that I am aware of enclosing while statement into {} but this does not solves the problem (imagine that you want use the while loop in function and return i variable)

The correct notation for Process Substitution is:
while read i; do echo $i; done < <(echo "$FILECONTENT")
The last value of i assigned in the loop is then available when the loop terminates.
An alternative is:
echo $FILECONTENT |
{
while read i; do echo $i; done
...do other things using $i here...
}
The braces are an I/O grouping operation and do not themselves create a subshell. In this context, they are part of a pipeline and are therefore run as a subshell, but it is because of the |, not the { ... }. You mention this in the question. AFAIK, you can do a return from within these inside a function.
Bash also provides the shopt builtin and one of its many options is:
lastpipe
If set, and job control is not active, the shell runs the last command of a pipeline not executed in the background in the current shell environment.
Thus, using something like this in a script makes the modfied sum available after the loop:
FILECONTENT="12 Name
13 Number
14 Information"
shopt -s lastpipe # Comment this out to see the alternative behaviour
sum=0
echo "$FILECONTENT" |
while read number name; do ((sum+=$number)); done
echo $sum
Doing this at the command line usually runs foul of 'job control is not active' (that is, at the command line, job control is active). Testing this without using a script failed.
Also, as noted by Gareth Rees in his answer, you can sometimes use a here string:
while read i; do echo $i; done <<< "$FILECONTENT"
This doesn't require shopt; you may be able to save a process using it.

Jonathan Leffler explains how to do what you want using process substitution, but another possibility is to use a here string:
while read i; do echo "$i"; done <<<"$FILECONTENT"
This saves a process.

This function makes duplicates $NUM times of jpg files (bash)
function makeDups() {
NUM=$1
echo "Making $1 duplicates for $(ls -1 *.jpg|wc -l) files"
ls -1 *.jpg|sort|while read f
do
COUNT=0
while [ "$COUNT" -le "$NUM" ]
do
cp $f ${f//sm/${COUNT}sm}
((COUNT++))
done
done
}

Related

Adding a piped input to an array during a while read [duplicate]

Bash allows to use: cat <(echo "$FILECONTENT")
Bash also allow to use: while read i; do echo $i; done </etc/passwd
to combine previous two this can be used: echo $FILECONTENT | while read i; do echo $i; done
The problem with last one is that it creates sub-shell and after the while loop ends variable i cannot be accessed any more.
My question is:
How to achieve something like this: while read i; do echo $i; done <(echo "$FILECONTENT") or in other words: How can I be sure that i survives while loop?
Please note that I am aware of enclosing while statement into {} but this does not solves the problem (imagine that you want use the while loop in function and return i variable)
The correct notation for Process Substitution is:
while read i; do echo $i; done < <(echo "$FILECONTENT")
The last value of i assigned in the loop is then available when the loop terminates.
An alternative is:
echo $FILECONTENT |
{
while read i; do echo $i; done
...do other things using $i here...
}
The braces are an I/O grouping operation and do not themselves create a subshell. In this context, they are part of a pipeline and are therefore run as a subshell, but it is because of the |, not the { ... }. You mention this in the question. AFAIK, you can do a return from within these inside a function.
Bash also provides the shopt builtin and one of its many options is:
lastpipe
If set, and job control is not active, the shell runs the last command of a pipeline not executed in the background in the current shell environment.
Thus, using something like this in a script makes the modfied sum available after the loop:
FILECONTENT="12 Name
13 Number
14 Information"
shopt -s lastpipe # Comment this out to see the alternative behaviour
sum=0
echo "$FILECONTENT" |
while read number name; do ((sum+=$number)); done
echo $sum
Doing this at the command line usually runs foul of 'job control is not active' (that is, at the command line, job control is active). Testing this without using a script failed.
Also, as noted by Gareth Rees in his answer, you can sometimes use a here string:
while read i; do echo $i; done <<< "$FILECONTENT"
This doesn't require shopt; you may be able to save a process using it.
Jonathan Leffler explains how to do what you want using process substitution, but another possibility is to use a here string:
while read i; do echo "$i"; done <<<"$FILECONTENT"
This saves a process.
This function makes duplicates $NUM times of jpg files (bash)
function makeDups() {
NUM=$1
echo "Making $1 duplicates for $(ls -1 *.jpg|wc -l) files"
ls -1 *.jpg|sort|while read f
do
COUNT=0
while [ "$COUNT" -le "$NUM" ]
do
cp $f ${f//sm/${COUNT}sm}
((COUNT++))
done
done
}

Bash Script Not Concatenating String as Intended [duplicate]

I have a code like that
var="before"
echo "$someString" | sed '$someRegex' | while read line
do
if [ $condition ]; then
var="after"
echo "$var" #first echo
fi
done
echo "$var" #second echo
Here first echo print "after", but second is "before". How can I make second echo print "after". I think it is because of pipe buy I don't know how figure out.
Thanks for any solutions...
answer edit:
I corrected it and it works fine. Thanks eugene for your useful answer
var="before"
while read line
do
if [ $condition ]; then
var="after"
echo "$var" #first echo
fi
done < <(echo "$someString" | sed '$someRegex')
echo "$var" #second echo
The reason for this behaviour is that a while loop runs in a subshell when it's part of a pipeline. For the while loop above, a new subshell with its own copy of the variable var is created.
See this article for possible workarounds: I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?.

How to pipe input to a Bash while loop and preserve variables after loop ends

Bash allows to use: cat <(echo "$FILECONTENT")
Bash also allow to use: while read i; do echo $i; done </etc/passwd
to combine previous two this can be used: echo $FILECONTENT | while read i; do echo $i; done
The problem with last one is that it creates sub-shell and after the while loop ends variable i cannot be accessed any more.
My question is:
How to achieve something like this: while read i; do echo $i; done <(echo "$FILECONTENT") or in other words: How can I be sure that i survives while loop?
Please note that I am aware of enclosing while statement into {} but this does not solves the problem (imagine that you want use the while loop in function and return i variable)
The correct notation for Process Substitution is:
while read i; do echo $i; done < <(echo "$FILECONTENT")
The last value of i assigned in the loop is then available when the loop terminates.
An alternative is:
echo $FILECONTENT |
{
while read i; do echo $i; done
...do other things using $i here...
}
The braces are an I/O grouping operation and do not themselves create a subshell. In this context, they are part of a pipeline and are therefore run as a subshell, but it is because of the |, not the { ... }. You mention this in the question. AFAIK, you can do a return from within these inside a function.
Bash also provides the shopt builtin and one of its many options is:
lastpipe
If set, and job control is not active, the shell runs the last command of a pipeline not executed in the background in the current shell environment.
Thus, using something like this in a script makes the modfied sum available after the loop:
FILECONTENT="12 Name
13 Number
14 Information"
shopt -s lastpipe # Comment this out to see the alternative behaviour
sum=0
echo "$FILECONTENT" |
while read number name; do ((sum+=$number)); done
echo $sum
Doing this at the command line usually runs foul of 'job control is not active' (that is, at the command line, job control is active). Testing this without using a script failed.
Also, as noted by Gareth Rees in his answer, you can sometimes use a here string:
while read i; do echo $i; done <<< "$FILECONTENT"
This doesn't require shopt; you may be able to save a process using it.
Jonathan Leffler explains how to do what you want using process substitution, but another possibility is to use a here string:
while read i; do echo "$i"; done <<<"$FILECONTENT"
This saves a process.
This function makes duplicates $NUM times of jpg files (bash)
function makeDups() {
NUM=$1
echo "Making $1 duplicates for $(ls -1 *.jpg|wc -l) files"
ls -1 *.jpg|sort|while read f
do
COUNT=0
while [ "$COUNT" -le "$NUM" ]
do
cp $f ${f//sm/${COUNT}sm}
((COUNT++))
done
done
}

Set a parent shell's variable from a subshell

How do I set a variable in the parent shell, from a subshell?
a=3
(a=4)
echo $a
The whole point of a subshell is that it doesn't affect the calling session. In bash a subshell is a child process, other shells differ but even then a variable setting in a subshell does not affect the caller. By definition.
Do you need a subshell? If you just need a group then use braces:
a=3
{ a=4;}
echo $a
gives 4 (be careful of the spaces in that one). Alternatively, write the variable value to stdout and capture it in the caller:
a=3
a=$(a=4;echo $a)
echo $a
avoid using back-ticks ``, they are deprecated and can be difficult to read.
There is the gdb-bash-variable hack:
gdb --batch-silent -ex "attach $$" -ex 'set bind_variable("a", "4", 0)';
although that always sets a variable in the global scope, not just the parent scope
You don't. The subshell doesn't have access to its parent's environment. (At least within the abstraction that Bash provides. You could potentially try to use gdb, or smash the stack, or whatnot, to gain such access clandestinely. I wouldn't recommend that, though.)
One alternative is for the subshell to write assignment statements to a temporary file for its parent to read:
a=3
(echo 'a=4' > tmp)
. tmp
rm tmp
echo "$a"
If the problem is related to a while loop, one way to fix this is by using Process Substitution:
var=0
while read i;
do
# perform computations on $i
((var++))
done < <(find . -type f -name "*.bin" -maxdepth 1)
as shown here: https://stackoverflow.com/a/13727116/2547445
To change variables in a script called from a parent script, you can call the script preceded with a "."
(EDIT - for explanation)
In most shells "." is an alias for "source". the source command just inserts the text of another file at that position in the executing script. In the context of this question this answer avoids a sub-shell
a=3
echo $a
. ./calledScript.sh
echo $a
in calledScript.sh
a=4
Expected output
3
4
By reading the answer from #ruakh (thank you) with a temporary file approach and the comments asking for a file descriptors solution, I got the following idea:
a=3
. <(echo a=4; echo b=5)
echo $a
echo $b
It allows returning different variables at once (which could be an issue in the subshell variant of the accepted answer).
No iteration is needed,
No temporary file to take care of.
Close to the syntax proposed by the OP.
Result:
4
5
With xtrace enabled is visible that we are sourcing from the file descriptor created for the output of the subshell:
+ a=3
+ . /dev/fd/63 # <-- the file descriptor ;)
++ echo a=4
++ echo b=5
++ a=4
++ b=5
+ echo 4
4
+ echo 5
5
You can output the value in the subshell and assign the subshell output to a variable in the caller script:
# subshell.sh
echo Value
# caller
myvar=$(subshell.sh)
If the subshell has more to output you can separate the variable value and other messages by redirecting them into different output streams:
# subshell.sh
echo "Writing value" 1>&2
echo Value
# caller
myvar=$(subshell.sh 2>/dev/null) # or to somewhere else
echo $myvar
Alternatively, you can output variable assignments in the subshell, evaluate them in the caller script and avoid using files to exchange information:
# subshell.sh
echo "a=4"
# caller
# export $(subshell.sh) would be more secure, since export accepts name=value only.
eval $(subshell.sh)
echo $a
The last way I can think of is to use exit codes but this covers the integer values exchange only (and in a limited range) and breaks the convention for interpreting exit codes (0 for success non-0 for everything else).
Instead of accessing the variable from the parent shell, change the order of the commands and use the process substitution:
a=3
echo 5 | (read a)
echo $a
prints 3
a=3
read a < <(echo 5)
echo $a
prints 5
Another example:
let i=0
seq $RANDOM | while read r
do
let i=r
done
echo $i
vs
let i=0
while read r
do
let i=r
done < <(seq $RANDOM)
echo $i
Alternatively, when job control is inactive (e.g. in scripts) you can use the lastpipe shell option to achieve the same result without changing the order of the commands:
#!/bin/bash
shopt -s lastpipe
let i=0
seq $RANDOM | while read r
do
let i=r
done
echo $i
Unless you can apply all io to pipes and use file handles, basic variable updating is impossible within $(command) and any other sub-process.
Regular files, however, are bash's global variables for normal sequential processing. Note: Due to race conditions, this simple approach is not good for parallel processing.
Create an set/get/default function like this:
globalVariable() { # NEW-VALUE
# set/get/default globalVariable
if [ 0 = "$#" ]; then
# new value not given -- echo the value
[ -e "$aRam/globalVariable" ] \
&& cat "$aRam/globalVariable" \
|| printf "default-value-here"
else
# new value given -- set the value
printf "%s" "$1" > "$aRam/globalVariable"
fi
}
"$aRam" is the directory where values are stored. I like it to be a ram disk for speed and volatility:
aRam="$(mktemp -td $(basename "$0").XXX)" # temporary directory
mount -t tmpfs ramdisk "$aRam" # mount the ram disk there
trap "umount "$aRam" && rm -rf "$aRam"" EXIT # auto-eject
To read the value:
v="$(globalVariable)" # or part of any command
To set the value:
globalVariable newValue # newValue will be written to file
To unset the value:
rm -f "$aRam/globalVariable"
The only real reason for the access function is to apply a default value because cat will error given a non-existent file. It is also useful to apply other get/set logic. Otherwise, it would not be needed at all.
An ugly read method avoiding cat's non-existent file error:
v="$(cat "$aRam/globalVariable 2>/dev/null")"
A cool feature of this mess is that you can open another terminal and examine the contents of the files while the program is running.
While it's harder to get multiple variables out of a subshell, you can set multiple variables inside a function without using globals.
You can pass the name of a variable into a function that uses local -n to turn it into a special variable called a nameref:
myfunc() {
local -n OUT=$1
local -n SIDEEFFECT=$2
OUT='foo'
SIDEEFFECT='bar'
}
myfunc A B
echo $A
> foo
echo $B
> bar
This is the technique I ended up using instead of getting subshell FOO=$(myfunc) working setting multiple variables.
A very simple and practical method that allows multiple variables is as follows, eventually may add parameters to the call:
function ComplexReturn(){
# do your processing...
a=123
b=456
echo -n "AAA=${a}; BBB=${b};"
}
# ... this can be internal function or any subshell command
eval $(ComplexReturn)
echo $AAA $BBB

Make use of variable from while read loop

In my bash script I use while read loop and a helper function fv():
fv() {
case "$1" in
out) echo $VAR
;;
* ) VAR="$VAR $1"
;;
esac
}
cat "$1" | while read line
do
...some processings...
fv some-str-value
done
echo "`fv out`"
in a hope that I can distil value from while read loop in a variable accessible in rest of the script.
But above snippet is no good, as I get no output.
Is there easy way to solve this - output string from this loop in a variable that would be accessible in rest of the script - without reformatting my script?
As no one has explained to you why your code didn't work, I will.
When you use cat "$1" |, you are making that loop execute in a subshell. The VAR variable used in that subshell starts as a copy of VAR from the main script, and any changes to it are limited to that copy (the subshell's scope), they don't affect the script's original VAR. By removing the useless use of cat, you remove the pipeline and so the loop is executed in the main shell, so it can (and does) alter the correct copy of VAR.
Replace your while loop by while read line ; do ... ; done < $1:
#!/bin/bash
function fv
{
case "$1" in
out) echo $VAR
;;
* ) VAR="$VAR $1"
;;
esac
}
while read line
do
fv "$line\n"
done < "$1"
echo "$(fv out)"
Stop piping to read.
done < "$1"

Resources