I need to read from a variable line by line, do some operations with every line and then work with the data afterwards. I have to work in sh.
I already tried this, but $VAR is empty since I assume, that I saved into it in a subshell
#!/bin/sh POSIXLY_CORRECT=yes
STRING="a\nb\nc\n"
echo $STRING | while read line; do
*some operations with line*
VAR=$(echo "$VAR$line")
done
echo $VAR
I also tried redirecting a variable...
done <<< $STRING
done < $(echo $STRING)
done < < $(echo $STRING)
...and so on, but only got No such file or Redirection unexpected
Please help.
As you've guessed, variable assignments in a subshell aren't seen by the parent, and the commands in a pipelines are run in subshells.
Having to work in plain sh is a real buzzkill. All right, all right, here are a few ideas. One is to extend the life of the subshell and do your work after the loop ends:
string="a
b
c"
echo "$string" | {
var=
while IFS= read -r line; do
*some operations with line*
var=$var$line
done
echo "$var"
}
Another is to use a heredoc (<<), since sh doesn't have herestrings (<<<).
var=
while IFS= read -r line; do
*some operations with line*
var=$var$line
done <<STR
$var
STR
Other improvements:
Escapes aren't interpreted in string literals, so if you want literal newlines put literal newlines, not \n.
Make sure to empty out $var with var=. You don't want an environment variable leaking into your script.
Quote variable expansions. Not needed in assignments, though.
$(echo foo) is an anti-pattern: you can just write foo.
It's best to keep variables lowercase. All uppercase is reserved for shell variables.
Use IFS= and -r to keep read from stripping leading whitespace or interpreting backslashes.
Related
Seems that the recommended way of doing indirect variable setting in bash is to use eval:
var=x; val=foo
eval $var=$val
echo $x # --> foo
The problem is the usual one with eval:
var=x; val=1$'\n'pwd
eval $var=$val # bad output here
(and since it is recommended in many places, I wonder just how many scripts are vulnerable because of this...)
In any case, the obvious solution of using (escaped) quotes doesn't really work:
var=x; val=1\"$'\n'pwd\"
eval $var=\"$val\" # fail with the above
The thing is that bash has indirect variable reference baked in (with ${!foo}), but I don't see any such way to do indirect assignment -- is there any sane way to do this?
For the record, I did find a solution, but this is not something that I'd consider "sane"...:
eval "$var='"${val//\'/\'\"\'\"\'}"'"
A slightly better way, avoiding the possible security implications of using eval, is
declare "$var=$val"
Note that declare is a synonym for typeset in bash. The typeset command is more widely supported (ksh and zsh also use it):
typeset "$var=$val"
In modern versions of bash, one should use a nameref.
declare -n var=x
x=$val
It's safer than eval, but still not perfect.
Bash has an extension to printf that saves its result into a variable:
printf -v "${VARNAME}" '%s' "${VALUE}"
This prevents all possible escaping issues.
If you use an invalid identifier for $VARNAME, the command will fail and return status code 2:
$ printf -v ';;;' '%s' foobar; echo $?
bash: printf: `;;;': not a valid identifier
2
eval "$var=\$val"
The argument to eval should always be a single string enclosed in either single or double quotes. All code that deviates from this pattern has some unintended behavior in edge cases, such as file names with special characters.
When the argument to eval is expanded by the shell, the $var is replaced with the variable name, and the \$ is replaced with a simple dollar. The string that is evaluated therefore becomes:
varname=$value
This is exactly what you want.
Generally, all expressions of the form $varname should be enclosed in double quotes, to prevent accidental expansion of filename patterns like *.c.
There are only two places where the quotes may be omitted since they are defined to not expand pathnames and split fields: variable assignments and case. POSIX 2018 says:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
This list of expansions is missing the parameter expansion and the field splitting. Sure, that's hard to see from reading this sentence alone, but that's the official definition.
Since this is a variable assignment, the quotes are not needed here. They don't hurt, though, so you could also write the original code as:
eval "$var=\"the value is \$val\""
Note that the second dollar is escaped using a backslash, to prevent it from being expanded in the first run. What happens is:
eval "$var=\"the value is \$val\""
The argument to the command eval is sent through parameter expansion and unescaping, resulting in:
varname="the value is $val"
This string is then evaluated as a variable assignment, which assigns the following value to the variable varname:
the value is value
The main point is that the recommended way to do this is:
eval "$var=\$val"
with the RHS done indirectly too. Since eval is used in the same
environment, it will have $val bound, so deferring it works, and since
now it's just a variable. Since the $val variable has a known name,
there are no issues with quoting, and it could have even been written as:
eval $var=\$val
But since it's better to always add quotes, the former is better, or
even this:
eval "$var=\"\$val\""
A better alternative in bash that was mentioned for the whole thing that
avoids eval completely (and is not as subtle as declare etc):
printf -v "$var" "%s" "$val"
Though this is not a direct answer what I originally asked...
Newer versions of bash support something called "parameter transformation", documented in a section of the same name in bash(1).
"${value#Q}" expands to a shell-quoted version of "${value}" that you can re-use as input.
Which means the following is a safe solution:
eval="${varname}=${value#Q}"
Just for completeness I also want to suggest the possible use of the bash built in read. I've also made corrections regarding -d'' based on socowi's comments.
But much care needs to be exercised when using read to ensure the input is sanitized (-d'' reads until null termination and printf "...\0" terminates the value with a null), and that read itself is executed in the main shell where the variable is needed and not a sub-shell (hence the < <( ... ) syntax).
var=x; val=foo0shouldnotterminateearly
read -d'' -r "$var" < <(printf "$val\0")
echo $x # --> foo0shouldnotterminateearly
echo ${!var} # --> foo0shouldnotterminateearly
I tested this with \n \t \r spaces and 0, etc it worked as expected on my version of bash.
The -r will avoid escaping \, so if you had the characters "\" and "n" in your value and not an actual newline, x will contain the two characters "\" and "n" also.
This method may not be aesthetically as pleasing as the eval or printf solution, and would be more useful if the value is coming in from a file or other input file descriptor
read -d'' -r "$var" < <( cat $file )
And here are some alternative suggestions for the < <() syntax
read -d'' -r "$var" <<< "$val"$'\0'
read -d'' -r "$var" < <(printf "$val") #Apparently I didn't even need the \0, the printf process ending was enough to trigger the read to finish.
read -d'' -r "$var" <<< $(printf "$val")
read -d'' -r "$var" <<< "$val"
read -d'' -r "$var" < <(printf "$val")
Yet another way to accomplish this, without eval, is to use "read":
INDIRECT=foo
read -d '' -r "${INDIRECT}" <<<"$(( 2 * 2 ))"
echo "${foo}" # outputs "4"
Seems that the recommended way of doing indirect variable setting in bash is to use eval:
var=x; val=foo
eval $var=$val
echo $x # --> foo
The problem is the usual one with eval:
var=x; val=1$'\n'pwd
eval $var=$val # bad output here
(and since it is recommended in many places, I wonder just how many scripts are vulnerable because of this...)
In any case, the obvious solution of using (escaped) quotes doesn't really work:
var=x; val=1\"$'\n'pwd\"
eval $var=\"$val\" # fail with the above
The thing is that bash has indirect variable reference baked in (with ${!foo}), but I don't see any such way to do indirect assignment -- is there any sane way to do this?
For the record, I did find a solution, but this is not something that I'd consider "sane"...:
eval "$var='"${val//\'/\'\"\'\"\'}"'"
A slightly better way, avoiding the possible security implications of using eval, is
declare "$var=$val"
Note that declare is a synonym for typeset in bash. The typeset command is more widely supported (ksh and zsh also use it):
typeset "$var=$val"
In modern versions of bash, one should use a nameref.
declare -n var=x
x=$val
It's safer than eval, but still not perfect.
Bash has an extension to printf that saves its result into a variable:
printf -v "${VARNAME}" '%s' "${VALUE}"
This prevents all possible escaping issues.
If you use an invalid identifier for $VARNAME, the command will fail and return status code 2:
$ printf -v ';;;' '%s' foobar; echo $?
bash: printf: `;;;': not a valid identifier
2
eval "$var=\$val"
The argument to eval should always be a single string enclosed in either single or double quotes. All code that deviates from this pattern has some unintended behavior in edge cases, such as file names with special characters.
When the argument to eval is expanded by the shell, the $var is replaced with the variable name, and the \$ is replaced with a simple dollar. The string that is evaluated therefore becomes:
varname=$value
This is exactly what you want.
Generally, all expressions of the form $varname should be enclosed in double quotes, to prevent accidental expansion of filename patterns like *.c.
There are only two places where the quotes may be omitted since they are defined to not expand pathnames and split fields: variable assignments and case. POSIX 2018 says:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
This list of expansions is missing the parameter expansion and the field splitting. Sure, that's hard to see from reading this sentence alone, but that's the official definition.
Since this is a variable assignment, the quotes are not needed here. They don't hurt, though, so you could also write the original code as:
eval "$var=\"the value is \$val\""
Note that the second dollar is escaped using a backslash, to prevent it from being expanded in the first run. What happens is:
eval "$var=\"the value is \$val\""
The argument to the command eval is sent through parameter expansion and unescaping, resulting in:
varname="the value is $val"
This string is then evaluated as a variable assignment, which assigns the following value to the variable varname:
the value is value
The main point is that the recommended way to do this is:
eval "$var=\$val"
with the RHS done indirectly too. Since eval is used in the same
environment, it will have $val bound, so deferring it works, and since
now it's just a variable. Since the $val variable has a known name,
there are no issues with quoting, and it could have even been written as:
eval $var=\$val
But since it's better to always add quotes, the former is better, or
even this:
eval "$var=\"\$val\""
A better alternative in bash that was mentioned for the whole thing that
avoids eval completely (and is not as subtle as declare etc):
printf -v "$var" "%s" "$val"
Though this is not a direct answer what I originally asked...
Newer versions of bash support something called "parameter transformation", documented in a section of the same name in bash(1).
"${value#Q}" expands to a shell-quoted version of "${value}" that you can re-use as input.
Which means the following is a safe solution:
eval="${varname}=${value#Q}"
Just for completeness I also want to suggest the possible use of the bash built in read. I've also made corrections regarding -d'' based on socowi's comments.
But much care needs to be exercised when using read to ensure the input is sanitized (-d'' reads until null termination and printf "...\0" terminates the value with a null), and that read itself is executed in the main shell where the variable is needed and not a sub-shell (hence the < <( ... ) syntax).
var=x; val=foo0shouldnotterminateearly
read -d'' -r "$var" < <(printf "$val\0")
echo $x # --> foo0shouldnotterminateearly
echo ${!var} # --> foo0shouldnotterminateearly
I tested this with \n \t \r spaces and 0, etc it worked as expected on my version of bash.
The -r will avoid escaping \, so if you had the characters "\" and "n" in your value and not an actual newline, x will contain the two characters "\" and "n" also.
This method may not be aesthetically as pleasing as the eval or printf solution, and would be more useful if the value is coming in from a file or other input file descriptor
read -d'' -r "$var" < <( cat $file )
And here are some alternative suggestions for the < <() syntax
read -d'' -r "$var" <<< "$val"$'\0'
read -d'' -r "$var" < <(printf "$val") #Apparently I didn't even need the \0, the printf process ending was enough to trigger the read to finish.
read -d'' -r "$var" <<< $(printf "$val")
read -d'' -r "$var" <<< "$val"
read -d'' -r "$var" < <(printf "$val")
Yet another way to accomplish this, without eval, is to use "read":
INDIRECT=foo
read -d '' -r "${INDIRECT}" <<<"$(( 2 * 2 ))"
echo "${foo}" # outputs "4"
On my GNU bash, version 4.3.42(1)-release I am doing some tests to answer a question. The idea is to split a :-separated string and and each of its elements into an array.
For this, I try to set the IFS to : in the scope of the command, so that the split is automatic and IFS remains untouched:
$ myvar="a:b:c"
$ IFS=: d=($myvar)
$ printf "%s\n" ${d[#]}
a
b
c
And apparently IFS remains the same:
$ echo $IFS
# empty
The BASH reference says that:
If IFS is unset, the parameters are separated by spaces. If IFS is
null, the parameters are joined without intervening separators.
However, then I notice that the IFS is kind of broken, so that echo $myvar returns a b c instead of a:b:c.
Unsetting the value solves it:
$ unset IFS
$ echo $myvar
a:b:c
But I wonder: what is causing this? Isn't IFS=: command changing IFS just in the scope of the command being executed?
I see in Setting IFS for a single statement that this indeed works:
$ IFS=: eval 'd=($myvar)'
$ echo $myvar
a:b:c
But I don't understand why it does and IFS=: d=($myvar) does not.
I was going to comment on this when I saw you use it, but it didn't occur to me until just now what the problem was. The line
IFS=: d=($myvar)
doesn't temporarily set IFS; it simply sets two variables in the current shell. (Simple commands can be prefixed with local environment settings, but an assignment statement itself is not a simple command.)
When you write
echo $IFS
IFS expands to :, but because : is the first character of IFS, it is removed during word splitting. Using
echo "$IFS"
would show that IFS is still set to :.
IFS behaves fine with read on the same line:
myvar="a:b:c"
IFS=: read -ra d <<< "$myvar"
printf "%s\n" "${d[#]}"
a
b
c
Check value of IFS:
declare -p IFS
-bash: declare: IFS: not found
so clearly IFS has not been tampered in current shell.
Now check original input:
echo $myvar
a:b:c
or:
echo "$myvar"
a:b:c
In bash, you can set set a variable that is valid for a single statement only if that statement is not itself a variable assignment. For example:
$ foo=one bar=two
$ echo $foo
one
To make the second assignment part of a statement, you need ... some other statement. As you've noticed, eval works. In addition, read should work:
$ foo="one:two:three"
$ IFS=: read -r -a bar <<< "$foo"
$ declare -p bar
declare -a bar='([0]="one" [1]="two" [2]="three")'
I'm trying to evaluate multiple lines of shell commands using eval, but when I try to resolve variables with eval separated by a newline \n the variables are not resolved.
x='echo a'
y='echo b'
z="$x\n$y"
eval $x
eval $y
eval $z
Which outputs:
a
b
anecho b
The last command gives anecho b, and apparently \n was treated as n there. So is there a way to evaluate multiple lines of commands (say, separated by \n)?
\n is not a newline; it's an escape sequence that in some situations will be translated into a newline, but you haven't used it in one of those situations. The variable $z doesn't wind up containing a newline, just backslash followed by "n". As a result, this is what's actually being executed:
$ echo a\necho b
anecho b
You can either use a semicolon instead (which requires no translation), or use \n in a context where it will be translated into a newline:
$ newline=$'\n'
$ x='echo a'
$ y='echo b'
$ z="$x$newline$y"
$ eval "$z"
a
b
Note the double-quotes around "$z" -- they're actually critical here. Without them, bash will word-split the value of $z, turning all whitespace (spaces, tabs, newlines) into word breaks. If that happens, eval will receive the words "echo" "a" "echo" b", effectively turning the newline into a space:
$ eval $z
a echo b
This is yet another in the long list of cases where it's important to double-quote variable references.
You are passing the newline into eval. So it's like you are on the console typing this:
el#voyager$ echo a\necho b
anecho b
So the first echo is understood correctly, and it thinks you want quotes around the rest. The backslash seems to be ignored. Perhaps you meant something like this:
el#voyager$ echo -e 'a\n'; echo b
a
b
Option 1:
delimit statements passed into eval with a semicolon like this:
x='echo a'
y='echo b'
z="$x;$y"
eval $x
eval $y
eval $z
prints:
a
b
a
b
Option 2:
Put the newline in the place where it will be interpreted by the echo, like this:
x='echo -e "a\n"'
y='echo b'
z="$x;$y"
eval $x
eval $y
eval $z
prints:
a
b
a
b
Now the newline is preserved and interpreted by the echo, not the eval.
Not necessarily the optimal way as it will fail if the x and y variables contain sequences processed by printf like %s and similar but anyway, here is a method to do it while keeping \n as a separator:
x='echo a'
y='echo b'
z="$x\n$y"
eval $x
eval $y
export IFS=" "
eval $(printf "$z")
prints:
a
b
a
b
A slightly different approach:
read -r -d '' script <<'EOF'
echo a
echo b
EOF
eval "$script"
outputs
a
b
Explanation
read -r -d '' script
-r - do not allow backslashes to escape any characters
-d '' - continue until the first character of DELIM is read, rather than newline (makes it read until EOF)
script - the name of the variable to save the result in
<<'EOF' - use a heredoc WITHOUT variable expansion (the single quotes around the EOF stops the variable expansion)
Alternative
This could also be done using $(cat <<'EOF'...EOF), but this way does not needlessly use cat and does not use a subshell.
Example with useless cat:
script=$(cat <<'EOF'
echo a
echo b
EOF
)
eval "$script"
On my FreeBSD box, I was trying do something in a Bourne script which at first seemed fairly trivial - but clouded my mind for a few moments. Since this page is what I referred to trying to fix my problem, I will explain what I needed to do and how I got it done :
a=A
b=B
eval ${a}_${b}="something"
No problems so far. I get a new variable A_B that stores "something"
But if I spread the assignment over 2 lines as under :
eval ${a}_${b}="some
thing"
The shell barks back at me that it could find no command called 'thing'. It is important to understand that eval tries to evaluate RHS as a command. To get eval to evaluate RHS as a string, you have to double-double quote RHS :
eval ${a}_${b}="\"some
thing\""
Hope this helps someone.
Manish Jain
Are you thinking of using the more powerful zsh instead of bash?
If so, you may want to take advantage of process substitution and heredoc. The following code works similarly to the title.
source <(cat << EOF
command1
command2
EOF
)
A simple yet annoying thing:
Using a script like this:
while read x; do
echo "$x"
done<file
on a file containing whitespace:
text
will give me an output without the whitespace:
text
The problem is i need this space before text (it's one tab mostly but not always).
So the question is: how to obtain identical lines as are in input file in such a script?
Update: Ok, so I changed my while read x to while IFS= read x.
echo "$x" gives me correct answer without stripping first tab, but, eval "echo $x" strips this tab.
What should I do then?
read is stripping the whitespace. Wipe $IFS first.
while IFS= read x
do
echo "$x"
done < file
The entire contents of the read are put into a variable called REPLY. If you use REPLY instead of 'x', you won't have to worry about read's word splitting and IFS and all that.
I ran into the same trouble you are having when attempting to strip spaces off the end of filenames. REPLY came to the rescue:
find . -name '* ' -depth -print | while read; do mv -v "${REPLY}" "`echo "${REPLY}" | sed -e 's/ *$//'`"; done
I found the solution to the problem 'eval "echo $x" strips this tab.' This should fix it:
eval "echo \"$x\""
I think this causes the inner (escaped) quotes will be evaluated with the echo, whereas I think that both
eval "echo $x"
and
eval echo "$x"
cause the quotes to be evaluated before the echo, which means that the string passed to echo has no quotes, causing the white space to be lost. So the complete answer is:
while IFS= read x
do
eval "echo \"$x\""
done < file