I have a variable that I receive from an external service that is in JSON format, and I want to use it inside a HEREDOC. I tried to use jq but got a parse error. This happens because the variable inside the HEREDOC don't have quotes.
Below is an example of what happens:
list="[{\"a\"=\"b\"}]"
echo "out"
echo "$list"
echo "len=${#list}"
echo ""
/bin/bash <<-SHELL
echo "in - 1"
echo "$list"
echo "len=${#list}"
echo ""
list2="$list"
echo "in - 2"
echo "\$list2"
echo "len=\${#list2}"
echo ""
list3="[{\"a\"=\"b\"}]"
echo "in - 3"
echo "\$list3"
echo "len=\${#list3}"
SHELL
And the output:
out
[{"a"="b"}]
len=11
in - 1
[{a=b}]
len=11
in - 2
[{a=b}]
len=7
in - 3
[{"a"="b"}]
len=11
I assume this happens because the external variables are expanded before the HEREDOC is executed, but is there a way to make HEREDOC preserve the quotes in the variable that was generated outside it?
Quote the delimiter, so that no parameter expansions happen in the here document. Then pass your JSON value as an argument, rather than embedding it in the script.
list='[{"a"="b"}]'
echo "out"
echo "$list"
echo "len=${#list}"
echo ""
/bin/bash <<-'SHELL' -s "$list"
echo "in - 1"
echo "$1"
echo "len=${#1}"
SHELL
The -s option allows you to provide arguments to a shell that reads its command from standard input, so that "$list" isn't mistaken for the name of the script to execute.
Related
In the following program, if I set the variable $foo to the value 1 inside the first if statement, it works in the sense that its value is remembered after the if statement. However, when I set the same variable to the value 2 inside an if which is inside a while statement, it's forgotten after the while loop. It's behaving like I'm using some sort of copy of the variable $foo inside the while loop and I am modifying only that particular copy. Here's a complete test program:
#!/bin/bash
set -e
set -u
foo=0
bar="hello"
if [[ "$bar" == "hello" ]]
then
foo=1
echo "Setting \$foo to 1: $foo"
fi
echo "Variable \$foo after if statement: $foo"
lines="first line\nsecond line\nthird line"
echo -e $lines | while read line
do
if [[ "$line" == "second line" ]]
then
foo=2
echo "Variable \$foo updated to $foo inside if inside while loop"
fi
echo "Value of \$foo in while loop body: $foo"
done
echo "Variable \$foo after while loop: $foo"
# Output:
# $ ./testbash.sh
# Setting $foo to 1: 1
# Variable $foo after if statement: 1
# Value of $foo in while loop body: 1
# Variable $foo updated to 2 inside if inside while loop
# Value of $foo in while loop body: 2
# Value of $foo in while loop body: 2
# Variable $foo after while loop: 1
# bash --version
# GNU bash, version 4.1.10(4)-release (i686-pc-cygwin)
echo -e $lines | while read line
...
done
The while loop is executed in a subshell. So any changes you do to the variable will not be available once the subshell exits.
Instead you can use a here string to re-write the while loop to be in the main shell process; only echo -e $lines will run in a subshell:
while read line
do
if [[ "$line" == "second line" ]]
then
foo=2
echo "Variable \$foo updated to $foo inside if inside while loop"
fi
echo "Value of \$foo in while loop body: $foo"
done <<< "$(echo -e "$lines")"
You can get rid of the rather ugly echo in the here-string above by expanding the backslash sequences immediately when assigning lines. The $'...' form of quoting can be used there:
lines=$'first line\nsecond line\nthird line'
while read line; do
...
done <<< "$lines"
UPDATED#2
Explanation is in Blue Moons's answer.
Alternative solutions:
Eliminate echo
while read line; do
...
done <<EOT
first line
second line
third line
EOT
Add the echo inside the here-is-the-document
while read line; do
...
done <<EOT
$(echo -e $lines)
EOT
Run echo in background:
coproc echo -e $lines
while read -u ${COPROC[0]} line; do
...
done
Redirect to a file handle explicitly (Mind the space in < <!):
exec 3< <(echo -e $lines)
while read -u 3 line; do
...
done
Or just redirect to the stdin:
while read line; do
...
done < <(echo -e $lines)
And one for chepner (eliminating echo):
arr=("first line" "second line" "third line");
for((i=0;i<${#arr[*]};++i)) { line=${arr[i]};
...
}
Variable $lines can be converted to an array without starting a new sub-shell. The characters \ and n has to be converted to some character (e.g. a real new line character) and use the IFS (Internal Field Separator) variable to split the string into array elements. This can be done like:
lines="first line\nsecond line\nthird line"
echo "$lines"
OIFS="$IFS"
IFS=$'\n' arr=(${lines//\\n/$'\n'}) # Conversion
IFS="$OIFS"
echo "${arr[#]}", Length: ${#arr[*]}
set|grep ^arr
Result is
first line\nsecond line\nthird line
first line second line third line, Length: 3
arr=([0]="first line" [1]="second line" [2]="third line")
You are asking this bash FAQ. The answer also describes the general case of variables set in subshells created by pipes:
E4) If I pipe the output of a command into read variable, why
doesn't the output show up in $variable when the read command finishes?
This has to do with the parent-child relationship between Unix
processes. It affects all commands run in pipelines, not just
simple calls to read. For example, piping a command's output
into a while loop that repeatedly calls read will result in
the same behavior.
Each element of a pipeline, even a builtin or shell function,
runs in a separate process, a child of the shell running the
pipeline. A subprocess cannot affect its parent's environment.
When the read command sets the variable to the input, that
variable is set only in the subshell, not the parent shell. When
the subshell exits, the value of the variable is lost.
Many pipelines that end with read variable can be converted
into command substitutions, which will capture the output of
a specified command. The output can then be assigned to a
variable:
grep ^gnu /usr/lib/news/active | wc -l | read ngroup
can be converted into
ngroup=$(grep ^gnu /usr/lib/news/active | wc -l)
This does not, unfortunately, work to split the text among
multiple variables, as read does when given multiple variable
arguments. If you need to do this, you can either use the
command substitution above to read the output into a variable
and chop up the variable using the bash pattern removal
expansion operators or use some variant of the following
approach.
Say /usr/local/bin/ipaddr is the following shell script:
#! /bin/sh
host `hostname` | awk '/address/ {print $NF}'
Instead of using
/usr/local/bin/ipaddr | read A B C D
to break the local machine's IP address into separate octets, use
OIFS="$IFS"
IFS=.
set -- $(/usr/local/bin/ipaddr)
IFS="$OIFS"
A="$1" B="$2" C="$3" D="$4"
Beware, however, that this will change the shell's positional
parameters. If you need them, you should save them before doing
this.
This is the general approach -- in most cases you will not need to
set $IFS to a different value.
Some other user-supplied alternatives include:
read A B C D << HERE
$(IFS=.; echo $(/usr/local/bin/ipaddr))
HERE
and, where process substitution is available,
read A B C D < <(IFS=.; echo $(/usr/local/bin/ipaddr))
Hmmm... I would almost swear that this worked for the original Bourne shell, but don't have access to a running copy just now to check.
There is, however, a very trivial workaround to the problem.
Change the first line of the script from:
#!/bin/bash
to
#!/bin/ksh
Et voila! A read at the end of a pipeline works just fine, assuming you have the Korn shell installed.
This is an interesting question and touches on a very basic concept in Bourne shell and subshell. Here I provide a solution that is different from the previous solutions by doing some kind of filtering. I will give an example that may be useful in real life. This is a fragment for checking that downloaded files conform to a known checksum. The checksum file look like the following (Showing just 3 lines):
49174 36326 dna_align_feature.txt.gz
54757 1 dna.txt.gz
55409 9971 exon_transcript.txt.gz
The shell script:
#!/bin/sh
.....
failcnt=0 # this variable is only valid in the parent shell
#variable xx captures all the outputs from the while loop
xx=$(cat ${checkfile} | while read -r line; do
num1=$(echo $line | awk '{print $1}')
num2=$(echo $line | awk '{print $2}')
fname=$(echo $line | awk '{print $3}')
if [ -f "$fname" ]; then
res=$(sum $fname)
filegood=$(sum $fname | awk -v na=$num1 -v nb=$num2 -v fn=$fname '{ if (na == $1 && nb == $2) { print "TRUE"; } else { print "FALSE"; }}')
if [ "$filegood" = "FALSE" ]; then
failcnt=$(expr $failcnt + 1) # only in subshell
echo "$fname BAD $failcnt"
fi
fi
done | tail -1) # I am only interested in the final result
# you can capture a whole bunch of texts and do further filtering
failcnt=${xx#* BAD } # I am only interested in the number
# this variable is in the parent shell
echo failcnt $failcnt
if [ $failcnt -gt 0 ]; then
echo $failcnt files failed
else
echo download successful
fi
The parent and subshell communicate through the echo command. You can pick some easy to parse text for the parent shell. This method does not break your normal way of thinking, just that you have to do some post processing. You can use grep, sed, awk, and more for doing so.
I use stderr to store within a loop, and read from it outside.
Here var i is initially set and read inside the loop as 1.
# reading lines of content from 2 files concatenated
# inside loop: write value of var i to stderr (before iteration)
# outside: read var i from stderr, has last iterative value
f=/tmp/file1
g=/tmp/file2
i=1
cat $f $g | \
while read -r s;
do
echo $s > /dev/null; # some work
echo $i > 2
let i++
done;
read -r i < 2
echo $i
Or use the heredoc method to reduce the amount of code in a subshell.
Note the iterative i value can be read outside the while loop.
i=1
while read -r s;
do
echo $s > /dev/null
let i++
done <<EOT
$(cat $f $g)
EOT
let i--
echo $i
How about a very simple method
+call your while loop in a function
- set your value inside (nonsense, but shows the example)
- return your value inside
+capture your value outside
+set outside
+display outside
#!/bin/bash
# set -e
# set -u
# No idea why you need this, not using here
foo=0
bar="hello"
if [[ "$bar" == "hello" ]]
then
foo=1
echo "Setting \$foo to $foo"
fi
echo "Variable \$foo after if statement: $foo"
lines="first line\nsecond line\nthird line"
function my_while_loop
{
echo -e $lines | while read line
do
if [[ "$line" == "second line" ]]
then
foo=2; return 2;
echo "Variable \$foo updated to $foo inside if inside while loop"
fi
echo -e $lines | while read line
do
if [[ "$line" == "second line" ]]
then
foo=2;
echo "Variable \$foo updated to $foo inside if inside while loop"
return 2;
fi
# Code below won't be executed since we returned from function in 'if' statement
# We aready reported the $foo var beint set to 2 anyway
echo "Value of \$foo in while loop body: $foo"
done
}
my_while_loop; foo="$?"
echo "Variable \$foo after while loop: $foo"
Output:
Setting $foo 1
Variable $foo after if statement: 1
Value of $foo in while loop body: 1
Variable $foo after while loop: 2
bash --version
GNU bash, version 3.2.51(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.
Though this is an old question and asked several times, here's what I'm doing after hours fidgeting with here strings, and the only option that worked for me is to store the value in a file during while loop sub-shells and then retrieve it. Simple.
Use echo statement to store and cat statement to retrieve. And the bash user must chown the directory or have read-write chmod access.
#write to file
echo "1" > foo.txt
while condition; do
if (condition); then
#write again to file
echo "2" > foo.txt
fi
done
#read from file
echo "Value of \$foo in while loop body: $(cat foo.txt)"
The below code does not give any output:
$echo `cat time`
19991213100942
$a=$(echo `cat time`) | echo $a | echo ${a:0:4}
Please tell where I am making mistake.
a=$(echo `cat time`)
assigns the output of the command inside the brackets $(...) to the variable $a.
Later in the script, you can print the variable:
echo $a
That prints: 19991213100942
echo ${a:0:4}
That prints: 1999
You can reference the varibale by its name $a.
First, you don't need to echo the output of cat time: just cat time.
Second, as #Etan says (kind of), replace the pipes with semicolons or newlines
a=$(< time) # a bash builtin, equivalent to but faster than: a=$(cat time)
echo $a
echo ${a:0:4}
I want to implement a bash function which test is the 1st argument is actually a variable, defined somewhere.
For instance, in my .bashrc :
customPrompt='yes';
syntaxOn='no';
[...]
function my_func {
[...]
# I want to test if the string $1 is the name of a variable defined up above
# so something like:
if [[ $$1 == 'yes' ]];then
echo "$1 is set to yes";
else
echo "$1 is not set or != to yes";
fi
# but of course $$1 doesn't work
}
output needed :
$ my_func customPrompt
> customPrompt is set to yes
$ my_func syntaxOn
> syntaxOn is set but != to yes
$ my_func foobar
> foobar is not set
I tried a lot of test, like -v "$1", -z "$1", -n "$1", but all of them test $1 as a string not as a variable.
(please correct me if I make not myself clear enought)
In the bash you can use the indirect variable subtituion.
t1=some
t2=yes
fufu() {
case "${!1}" in
yes) echo "$1: set to yes. Value: ${!1}";;
'') echo "$1: not set. Value: ${!1:-UNDEF}";;
*) echo "$1: set to something other than yes. Value: ${!1}";;
esac
}
fufu t1
fufu t2
fufu t3
prints
t1: set to something other than yes. Value: some
t2: set to yes. Value: yes
t3: not set. Value: UNDEF
The ${!variablename} in bash mean indirect variable expansion. Described in the e.g. https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
Whrere:
The basic form of parameter expansion is ${parameter}. The value of
parameter is substituted. The braces are required when parameter is a
positional parameter with more than one digit, or when parameter is
followed by a character that is not to be interpreted as part of its
name.
If the first character of parameter is an exclamation point (!), a
level of variable indirection is introduced. Bash uses the value of
the variable formed from the rest of parameter as the name of the
variable; this variable is then expanded and that value is used in the
rest of the substitution, rather than the value of parameter itself.
This is known as indirect expansion. The exceptions to this are the
expansions of ${!prefix } and ${!name[#]} described below. The
exclamation point must immediately follow the left brace in order to
introduce indirection.
Also, check this: https://stackoverflow.com/a/16131829/632407 how to modify in a function a value of the variable passed indirectly.
You can check variable set or not by simply like
if [[ $var ]]
then
echo "Sorry First set variable"
else
echo $var
fi
You can do something like this for your script
customPrompt='yes';
syntaxOn='no';
function my_func
{
if [[ ${!1} ]];then
echo "$1 is set to ${!1}";
else
echo "$1 is not set";
fi
}
my_func customPrompt
my_func syntaxOn
my_func foobar
Output:
customPrompt is set to yes
syntaxOn is set to no
foobar is not set
You can customize the function as per you requirement by simply making some comparison conditions.
For more details you can check this answer
If you really want to check if your variable is set or unset (not just empty), use this format:
function my_func {
if [[ -z ${!1+.} ]]; then
echo "$1 is not set."
elif [[ ${!1} == yes ]]; then
echo "$1 is set to yes"
else
echo "$1 is set to \"${!1}\"."
fi
}
You're going to have problems...
The Bash shell is a very wily creature. Before you execute anything, Bash comes in and interpolates your command. Your command or shell script never sees whether or not you have a variable as a parameter.
$ set -x
set -x
$ foo=bar
+ foo=bar
$ echo "$foo"
+ echo bar
bar
$ set +x
The set -x turns on debugging mode in the shell. It shows you what a command actually executes. For example, I set foo=bar and then do echo $foo. My echo command doesn't see $foo. Instead, before echo executes, it interpolates $foo with bar. All echo sees at this point is that it's suppose to take bar as its argument (not $foo).
This is awesomely powerful. It means that your program doesn't have to sit there and interpret the command line. If you typed echo *.txt, echo doesn't have to expand *.txt because the shell has already done the dirty work.
For example, here's a test shell script:
#! /bin/sh
if [[ $1 = "*" ]]
then
echo "The first argument was '*'"
else
"I was passed in $# parameters"
fi
Now, I'll run my shell script:
$ test.sh *
I was passed in 24 parameters
What? Wasn't the first parameter of my script a *? No. The shell grabbed * and expanded it to be all of the files and directories in my directory. My shell script never saw the *. However, I can do this:
$ test.sh '*'
The first argument was '*'
The single quotes tell the shell not to interpolate anything. (Double quotes prevent globbing, but still allow for environment variable expansion).
This if I wanted to see if my first parameter is a variable, I have to pass it in single quotes:
$ test.sh '$foo'
And, I can do this as a test:
if [[ $1 != ${1#$} ]]
then
echo "The first parameter is the variable '$1'"
fi
The ${1#$} looks a bit strange, but it's just ${var#pattern}. This removes pattern from the left most side of $var. I am taking $1 and removing the $ if it exists. This gets expanded in the shell as:
if [[ $foo != foo ]]
which is true.
So, several things:
First, you've got to stop the shell from interpolating your variable. That means you have to use single quotes around the name.
You have to use pattern matching to verify that the first parameter starts with a $.
Once you do that, you should be able to use your variable with ${$1} in your script.
I have a environment config file where i have defined the environment variables. I use source to get these variables inside my shell script(bash).
I use a checkout command in my shell script which checks out the files from location defined in an environment variable. Now i need to use multiple locations to checkout the files which can be any number for different run of shell script.
For eg. User gives two paths, PATH1 and PATH2 in config file and a NUM_OF_PATHS as 2.
In my shell script I want to do something like below for using path.
i=0
echo ${NUM_OF_PATHS}
while [ $i -lt ${NUM_OF_PATHS} ]
do
checkout $PATH{$i}
i=`expr $i + 1`
done
How can I use the variable i to form an environment variable PATH1 or PATH2 etc.?
i=1
while [ $i -le ${NUM_OF_PATHS} ]
do
CPATH=$(eval echo \$\{PATH$i\})
echo "PATH$i: $CPATH"
let i++
done
eval combines and evaluates its parameters and executes the combined expression. Here, eval executes: echo ${PATH1}. In order to do this, we first escape the ${...} so that echo can receive them after eval. The only un-escaped special character is $ before i. eval expands this and strips off the escaped characters and executes echo with the result.
So, CPATH=$(eval echo \$\{PATH$i\}) becomes CPATH=$(echo ${PATH1}) and CPATH gets the echo output.
Here is a complete example that I think will do what you want. Below is my original message that was voted down for a reason I don't completely understand:
$ cat test.sh
function checkout() {
echo "Registering $1"
}
PATH1="Path1;/usr/bin"
PATH2="Path2;/bin"
NUM_OF_PATHS=2
i=1
echo $NUM_OF_PATHS
while [ $i -le $NUM_OF_PATHS ]
do
eval "checkout \$PATH$i"
i=`expr $i + 1`
done
$ bash test.sh
2
Registering Path1;/usr/bin
Registering Path2;/bin
$
Original Message
You can use the "eval" command. Here is an example (works on GNU bash, version 4.2.37(2)-release):
$ A1="Variable 1"
$ A2="Variable 2"
$ for i in {1..2}; do eval "echo \$A$i"; done
Variable 1
Variable 2
$
The string will evaluate to "echo $A1" and "echo $A2", and then eval will do what you want.
Why don't you use array variable? Use one array MYPATH[] instead multi PATH1 PATH2 variables
MYPATH=(
path0
path1
path2
)
MYPATH[3]=path3
NUM_OF_PATHS="${#MYPATH[#]}"
echo ${NUM_OF_PATHS}
for ((i=0; i < NUM_OF_PATHS; i++))
do
checkout ${MYPATH[$i]}
done
Use curly braces around the name: ${PATH$i}
I'm having a problem with a shell script (POSIX shell under HP-UX, FWIW). I have a function called print_arg into which I'm passing the name of a parameter as $1. Given the name of the parameter, I then want to print the name and the value of that parameter. However, I keep getting an error. Here's an example of what I'm trying to do:
#!/usr/bin/sh
function print_arg
{
# $1 holds the name of the argument to be shown
arg=$1
# The following line errors off with
# ./test_print.sh[9]: argval=${"$arg"}: The specified substitution is not valid for this command.
argval=${"$arg"}
if [[ $argval != '' ]] ; then
printf "ftp_func: $arg='$argval'\n"
fi
}
COMMAND="XYZ"
print_arg "COMMAND"
I've tried re-writing the offending line every way I can think of. I've consulted the local oracles. I've checked the online "BASH Scripting Guide". And I sharpened up the ol' wavy-bladed knife and scrubbed the altar until it gleamed, but then I discovered that our local supply of virgins has been cut down to, like, nothin'. Drat!
Any advice regarding how to get the value of a parameter whose name is passed into a function as a parameter will be received appreciatively.
You could use eval, though using direct indirection as suggested by SiegeX is probably nicer if you can use bash.
#!/bin/sh
foo=bar
print_arg () {
arg=$1
eval argval=\"\$$arg\"
echo "$argval"
}
print_arg foo
In bash (but not in other sh implementations), indirection is done by: ${!arg}
Input
foo=bar
bar=baz
echo $foo
echo ${!foo}
Output
bar
baz
This worked surprisingly well:
#!/bin/sh
foo=bar
print_arg () {
local line name value
set | \
while read line; do
name=${line%=*} value=${line#*=\'}
if [ "$name" = "$1" ]; then
echo ${value%\'}
fi
done
}
print_arg foo
It has all the POSIX clunkiness, in Bash would be much sorter, but then again, you won't need it because you have ${!}. This -in case it proves solid- would have the advantage of using only builtins and no eval. If I were to construct this function using an external command, it would have to be sed. Would obviate the need for the read loop and the substitutions. Mind that asking for indirections in POSIX without eval, has to be paid with clunkiness! So don't beat me!
Even though the answer's already accepted, here's another method for those who need to preserve newlines and special characters like Escape ( \033 ): Storing the variable in base64.
You need: bc, wc, echo, tail, tr, uuencode, uudecode
Example
#!/bin/sh
#====== Definition =======#
varA="a
b
c"
# uuencode the variable
varB="`echo "$varA" | uuencode -m -`"
# Skip the first line of the uuencode output.
varB="`NUM=\`(echo "$varB"|wc -l|tr -d "\n"; echo -1)|bc \`; echo "$varB" | tail -n $NUM)`"
#====== Access =======#
namevar1=varB
namevar2=varA
echo simple eval:
eval "echo \$$namevar2"
echo simple echo:
echo $varB
echo precise echo:
echo "$varB"
echo echo of base64
eval "echo \$$namevar1"
echo echo of base64 - with updated newlines
eval "echo \$$namevar1 | tr ' ' '\n'"
echo echo of un-based, using sh instead of eval (but could be made with eval, too)
export $namevar1
sh -c "(echo 'begin-base64 644 -'; echo \$$namevar1 | tr ' ' '\n' )|uudecode"
Result
simple eval:
a b c
simple echo:
YQpiCmMK ====
precise echo:
YQpiCmMK
====
echo of base64
YQpiCmMK ====
echo of base64 - with updated newlines
YQpiCmMK
====
echo of un-based, using sh instead of eval (but could be made with eval, too)
a
b
c
Alternative
You also could use the set command and parse it's output; with that, you don't need to treat the variable in a special way before it's accessed.
A safer solution with eval:
v=1
valid_var_name='[[:alpha:]_][[:alnum:]_]*$'
print_arg() {
local arg=$1
if ! expr "$arg" : "$valid_var_name" >/dev/null; then
echo "$0: invalid variable name ($arg)" >&2
exit 1
fi
local argval
eval argval=\$$arg
echo "$argval"
}
print_arg v
print_arg 'v; echo test'
Inspired by the following answer.