How preserve words containing spaces while expanding variables in bash script? - bash

My scheme is the following:
I have a shell script that executes a command which calls a C program :
name=$1
StringWithSpaces=$name
command="someprogram.out $StringWithSpaces $otherarguments"
$command
where name is a string with spaces s.a. "String With Spaces" passed to the shell from another python script.
My problem is that when I read that argument in C, it is passed as several arguments instead of just one. I have tried $#, $* and all that stuff. I have also tried to make a function in C that separate the several argv[i] within the StringWithSpaces one, but I am a bit stuck. I wish I could read the variable in C just as a single argument, to make the program as simple as I can.
This is the exact shell code (bash):
#!/bin/bash
#$1 Nombre de la base de datos
#$2 $3 gps coordinates
#$4 geoDistance (radio en km)
#$5 imDownload (a 0: solo se descargan GPS, a 1 también imágenes y tags)
#$Disabled keywords (opcional, lista de keywords separados por comas)
#Generamos el base path
BASE_DIR=`pwd`
#BASE_DIR=${BASE_DIR%/*}
EXE_DIR=$BASE_DIR/FlickrAPI/bin
DB_PATH=$BASE_DIR/data
#Exportamos las librerías, necesario para que funcione el código
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${BASE_DIR}/FlickrAPI/lib
cont=1;
#General DB
name=$1
gps="$2 $3"
geoDistance=$4
numImagesDB=3751
ncores=400;
imDownload=$5
dbDir=$DB_PATH/$name
mkdir "$dbDir";
rm -rf "$dbDir"/imagesDB
rm -rf "$dbDir"/tagsDB
rm -rf "$dbDir"/gps.txt
mkdir "$dbDir"/imagesDB
mkdir "$dbDir"/tagsDB
#tidx=`seq 7 $#`;
#keywords="";
#for ((i=7;i<=$#;i++))
#do
# keywords=$keywords" "${!i};
#done
keywords=$6;
export LD_LIBRARY_PATH=/home/user/anaconda2/lib/
command="$EXE_DIR/get_GPS_bimestral $dbDir $gps z $numImagesDB $ncores $geoDistance $imDownload $keywords"
echo $command
$command

Put the command in an array with:
Command=(someprogram.out "$StringWithSpaces" "$otherarguments")
When array is expanded in quotes using # to request all its members, as in "${Command[#]}", then bash expands its array member to a single word. So the desired strings with spaces are kept as single strings.
Here is a sample script:
#!/bin/bash -e
function ShowArguments()
{
for argument in "$#"; do
echo "Argument: $argument"
done
}
Argument1="abc def"
Argument2="pdq xyz"
Command=(ShowArguments "$Argument1" "$Argument2")
echo "${Command[#]}"
"${Command[#]}"
The output from the above is:
ShowArguments abc def pdq xyz
Argument: abc def
Argument: pdq xyz
You may want "$otherarguments" to be $otherarguments. Or, if it contains a string with spaces that should be kept as a string, you should handle it the same way, as an array that is expanded with ${otherarguments[#]} in quotes. Here is an example with quoting of one of the variables used to hold arguments:
#!/bin/bash -e
function ShowArguments()
{
for argument in "$#"; do
echo "Argument: $argument"
done
}
Argument1="Single argument with multiple words"
Argument2=("Multiple argument" with various "numbers of words")
Command=(ShowArguments "$Argument1" "${Argument2[#]}")
echo "${Command[#]}"
"${Command[#]}"
It produces:
ShowArguments Single argument with multiple words Multiple argument with various numbers of words
Argument: Single argument with multiple words
Argument: Multiple argument
Argument: with
Argument: various
Argument: numbers of words
After that, you might want to replace echo with a fancier command that quotes its arguments, to show the proper quoting to the user. But that is an aesthetic or user-interface choice; it will not affect the command executed.

My problem is that when I read that argument in C, it is passed as several arguments instead of just one.
in your script replace
command="$EXE_DIR/get_GPS_bimestral $dbDir $gps z $numImagesDB $ncores $geoDistance $imDownload $keywords"
echo $command
$command
by
command="$EXE_DIR/get_GPS_bimestral \"$dbDir $gps z $numImagesDB $ncores $geoDistance $imDownload $keywords\""
echo $command
echo $command | bash
Explanations :
if the shell script b is :
./a.out "$* 3 4"
and c.c is
#include <stdio.h>
int main(int argc, char ** argv)
{
printf("argc = %d, argv[1] = '%s'\n", argc, argv[1]);
}
then :
pi#raspberrypi:/tmp $ ./b 1 2
argc = 2, argv[1] = '1 2 3 4'
the C program receives only one arg, that is ok
but if the script is modified to be for instance :
command="./a.out \"$* 3 4\""
echo $command
$command
that doesn't work :
pi#raspberrypi:/tmp $ ./b 1 2
./a.out "1 2 3 4"
argc = 5, argv[1] = '"1'
so if you want to store in a var to echo then execute it you can do :
command="./a.out \"$* 3 4\""
echo $command
echo $command | bash
execution:
pi#raspberrypi:/tmp $ ./b 1 2
./a.out "1 2 3 4"
argc = 2, argv[1] = '1 2 3 4'
of course if you want for example 4 received as a separate argument just put it outside the string :
command="./a.out \"$* 3\" 4"
echo $command
echo $command | bash
execution :
pi#raspberrypi:/tmp $ ./b 1 2
./a.out "1 2 3" 4
argc = 3, argv[1] = '1 2 3'

Related

How do you get GNU Parallel to parse quoted command line arguments?

This is one sample program in the GNU parallel documentation for executing via the shell script shebang.
#!/usr/bin/parallel --shebang-wrap --colsep " " /bin/bash
echo Arguments: $#
The output for
./bash_echo.parl gracias 'buenos dias'
is
gracias
buenos
dias
The above script does not handle command line arguments that are quoted and contain spaces. The arguments are expanded instead and treated as individual inputs.
How do I obtain the correct output as for the bash script below?
#!/usr/bin/env bash
for i in "$#"; do
echo "$i"
done
This, obviously, handles quoted command line args.
Output:
gracias
buenos dias
I've tried using the option 'colseps' setting the separator to ' ' but that isn't the solution.
You have found a bug: --shebang-wrap was never tested with spaces.
Possible fix:
diff --git a/src/parallel b/src/parallel
index 69adfdac..e7c0d930 100755
--- a/src/parallel
+++ b/src/parallel
## -3302,9 +3302,10 ## sub read_options() {
#options = shift #ARGV;
}
my $script = Q(shift #ARGV);
+ my #args = map{ Q($_) } #ARGV;
# exec myself to split $ARGV[0] into separate fields
- exec "$0 --_pipe-means-argfiles #options #parser $script ".
- "::: #ARGV";
+ exec "$0 --_pipe-means-argfiles #options #parser $script ".
+ "::: #args";
}
}
if($ARGV[0] =~ / --shebang(-?wrap)? /) {
It seems to fix your issue, but it may introduce others.
For updates: Follow https://savannah.gnu.org/bugs/index.php?63703

Syntax error: end of file unexpected

I define a function in Makefile
define write_file
for i in $( seq 1 10 )
do
echo "1234567" >> "tmp/test.txt"
done
endef
And
pre:
mkdir -p exe tmp
${call write_file}
But when I make pre,I got error:
mkdir -p exe tmp
for i in
/bin/sh: 1: Syntax error: end of file unexpected
Each line of a make-recipe is a single command that make runs in a single
new shell. So you need to make the body of the macro write_file into a single
shell command.
In addition, make expands anything of the unescaped form $(....),
treating .... as a defined or undefined make expression. So in your
case, make expands $( seq 1 10 ) to nothing. To stop make doing
that and let the shell expand $( seq 1 10 ), you need to escape $ for make,
which you do by writing $$ instead. The same goes for any $ in a make-recipe
that you intend to be expanded by the shell.
Putting these points together, you want:
define write_file
for i in $$( seq 1 10 ); \
do \
echo "1234567" >> "tmp/test.txt"; \
done
endef

Multiple shell parameter assignments prefixing a bash command

In the bash manual section 3.7.4, it states
The environment for any simple command or function may be augmented
temporarily by prefixing it with parameter assignments, as described
in Shell Parameters [section 3.4].
And a trivial example of this is
MYVAR=MYVALUE mycommand
I have read section 3.4 and I still can't figure out how to specify multiple parameter assignments. Note that the statement in 3.7.4 is definitely plural, implying that it is possible.
The following does not seem to work:
MYVAR1=abc MYVAR2=xyz mycommand
I am using bash version 4.1, 23 December 2009.
It should work. That is acceptable syntax. Here's an example:
$ cat a
#!/bin/sh
echo $MYVAR1
echo $MYVAR2
$ ./a
$ MYVAR1=abc MYVAR2=xyz ./a
abc
xyz
$
UPDATE: Your updated example given in your answer will work if you precede the simple command with the variables as required:
mycommand () { echo MYVAR1=[$MYVAR1]; echo MYVAR2=[$MYVAR2]; }
for f in ~/*.txt ; do MYVAR1=abc MYVAR2=xyz mycommand; done
Oops, my example was over-simplified. This works fine:
mycommand () { echo MYVAR1=[$MYVAR1]; echo MYVAR2=[$MYVAR2]; }
MYVAR1=abc MYVAR2=xyz mycommand
but this does not:
mycommand () { echo MYVAR1=[$MYVAR1]; echo MYVAR2=[$MYVAR2]; }
MYVAR1=abc MYVAR2=xyz for f in ~/*.txt; do mycommand; done
and the key difference is this phrase:
for any simple command or function
A for command is a complex command, not "a simple command or function", according to section 3.2.
I've never seen that method used.
You could always just pass in regular parameters to the script.
Updating for new example:
This works in both situations:
> mycommand () { echo MYVAR1=[$1]; echo MYVAR2=[$2]; }
> mycommand a b
MYVAR1=[a]
MYVAR2=[b]
> for f in ~/file*.txt; do mycommand a b; done
MYVAR1=[a]
MYVAR2=[b]
MYVAR1=[a]
MYVAR2=[b]

idioms for returning multiple values in shell scripting

Are there any idioms for returning multiple values from a bash function within a script?
http://tldp.org/LDP/abs/html/assortedtips.html describes how to echo multiple values and process the results (e.g., example 35-17), but that gets tricky if some of the returned values are strings with spaces in.
A more structured way to return would be to assign to global variables, like
foo () {
FOO_RV1="bob"
FOO_RV2="bill"
}
foo
echo "foo returned ${FOO_RV1} and ${FOO_RV2}"
I realize that if I need re-entrancy in a shell script I'm probably doing it wrong, but I still feel very uncomfortable throwing global variables around just to hold return values.
Is there a better way? I would prefer portability, but it's probably not a real limitation if I have to specify #!/bin/bash.
In the special case where your values never contain spaces, this read trick can be a simple solution:
get_vars () {
#...
echo "value1" "value2"
}
read var1 var2 < <(get_vars)
echo "var1='$var1', var2='$var2'"
But of course, it breaks as soon as there is a space in one of the values. You could modify IFS and use a special separator in your function's echo, but then the result is not really simpler than the other suggested solutions.
This question was posted 5 years ago, but I have some interesting answer to post. I have just started learning bash, and I also encounter to the same problem as you did. I think this trick might be helpful:
#!/bin/sh
foo=""
bar=""
my_func(){
echo 'foo="a"; bar="b"'
}
eval $(my_func)
echo $foo $bar
# result: a b
This trick is also useful for solving a problem when a child process can not send back a value to its parent process.
Much as I love shell, it's probably the case that as soon as you're throwing arbitrary structured data around, Unix bourne/posix shell is not the right choice.
If there are characters which do not occur inside fields, then separate with one of those. The classic example is /etc/passwd, /etc/group and various other files which use a colon as a field separator.
If using a shell which can handle a NUL character inside strings, then joining on the NUL and separating on it (via $IFS or whatever) can work well. But several common shells, including bash, break on NUL. A test would be an old .sig of mine:
foo=$'a\0b'; [ ${#foo} -eq 3 ] && echo "$0 rocks"
Even if that would work for you, you've just reached one of the warning signs that it's time to switch to a more structured language (Python, Perl, Ruby, Lua, Javascript ... pick your preferred poison). Your code is likely to become hard to maintain; even if you can, there's a smaller pool of people who'll understand it well enough to maintain it.
Yet another way:
function get_tuple()
{
echo -e "Value1\nValue2"
}
IFS=$'\n' read -d '' -ra VALUES < <(get_tuple)
echo "${VALUES[0]}" # Value1
echo "${VALUES[1]}" # Value2
In order version of Bash which doesn't support nameref (introduced in Bash 4.3-alpha) I may define helper function in which the return value is assigned to the given variable. It's sort of like using eval to do the same kind of variable assignment.
Example 1
## Add two complex numbers and returns it.
## re: real part, im: imaginary part.
##
## Helper function named by the 5th positional parameter
## have to have been defined before the function is called.
complexAdd()
{
local re1="$1" im1="$2" re2="$3" im2="$4" fnName="$5" sumRe sumIm
sumRe=$(($re1 + $re2))
sumIm=$(($im1 + $im2))
## Call the function and return 2 values.
"$fnName" "$sumRe" "$sumIm"
}
main()
{
local fooRe='101' fooIm='37' barRe='55' barIm='123' bazRe bazIm quxRe quxIm
## Define the function to receive mutiple return values
## before calling complexAdd().
retValAssign() { bazRe="$1"; bazIm="$2"; }
## Call comlexAdd() for the first time.
complexAdd "$fooRe" "$fooIm" "$barRe" "$barIm" 'retValAssign'
## Redefine the function to receive mutiple return values.
retValAssign() { quxRe="$1"; quxIm="$2"; }
## Call comlexAdd() for the second time.
complexAdd "$barRe" "$barIm" "$bazRe" "$bazIm" 'retValAssign'
echo "foo = $fooRe + $fooIm i"
echo "bar = $barRe + $barIm i"
echo "baz = foo + bar = $bazRe + $bazIm i"
echo "qux = bar + baz = $quxRe + $quxIm i"
}
main
Example 2
## Add two complex numbers and returns it.
## re: real part, im: imaginary part.
##
## Helper functions
## getRetRe(), getRetIm(), setRetRe() and setRetIm()
## have to have been defined before the function is called.
complexAdd()
{
local re1="$1" im1="$2" re2="$3" im2="$4"
setRetRe "$re1"
setRetRe $(($(getRetRe) + $re2))
setRetIm $(($im1 + $im2))
}
main()
{
local fooRe='101' fooIm='37' barRe='55' barIm='123' bazRe bazIm quxRe quxIm
## Define getter and setter functions before calling complexAdd().
getRetRe() { echo "$bazRe"; }
getRetIm() { echo "$bazIm"; }
setRetRe() { bazRe="$1"; }
setRetIm() { bazIm="$1"; }
## Call comlexAdd() for the first time.
complexAdd "$fooRe" "$fooIm" "$barRe" "$barIm"
## Redefine getter and setter functions.
getRetRe() { echo "$quxRe"; }
getRetIm() { echo "$quxIm"; }
setRetRe() { quxRe="$1"; }
setRetIm() { quxIm="$1"; }
## Call comlexAdd() for the second time.
complexAdd "$barRe" "$barIm" "$bazRe" "$bazIm"
echo "foo = $fooRe + $fooIm i"
echo "bar = $barRe + $barIm i"
echo "baz = foo + bar = $bazRe + $bazIm i"
echo "qux = bar + baz = $quxRe + $quxIm i"
}
main
you can make use of associative arrays with you have bash 4 eg
declare -A ARR
function foo(){
...
ARR["foo_return_value_1"]="VAR1"
ARR["foo_return_value_2"]="VAR2"
}
you can combine them as strings.
function foo(){
...
echo "$var1|$var2|$var3"
}
then whenever you need to use those return values,
ret="$(foo)"
IFS="|"
set -- $ret
echo "var1 one is: $1"
echo "var2 one is: $2"
echo "var3 one is: $3"
I would go for the solution I suggested here, but using an array variable instead. Older bash:es don't support associative arrays.
E.g.,
function some_func() # ARRVAR args...
{
local _retvar=$1 # I use underscore to avoid clashes with return variable names
local -a _out
# ... some processing ... (_out[2]=xxx etc.)
eval $_retvar='("${_out[#]}")'
}
Calling site:
function caller()
{
local -a tuple_ret # Do not use leading '_' here.
# ...
some_func tuple_ret "arg1"
printf " %s\n" "${tuple_ret[#]}" # Print tuple members on separate lines
}
Later version of Bash supports nameref. Use declare -n var_name to give var_name the nameref attribute. nameref gives your function the ability to "pass by reference" which is commonly used in C++ functions to return multiple values. According to Bash man page:
A variable can be assigned the nameref attribute using the -n option to the declare or local builtin commands to create a nameref, or a reference to another variable. This allows variables to be manipulated indirectly. Whenever the nameref variable is referenced or assigned to, the operation is actually performed on the variable specified by the nameref variable's value. A nameref is commonly used within shell functions to refer to a variable whose name is passed as an argument to the function.
The following are some interactive command line examples.
Example 1:
$ unset xx yy
$ xx=16
$ yy=xx
$ echo "[$xx] [$yy]"
[16] [xx]
$ declare -n yy
$ echo "[$xx] [$yy]"
[16] [16]
$ xx=80
$ echo "[$xx] [$yy]"
[80] [80]
$ yy=2016
$ echo "[$xx] [$yy]"
[2016] [2016]
$ declare +n yy # Use -n to add and +n to remove nameref attribute.
$ echo "[$xx] [$yy]"
[2016] [xx]
Example 2:
$ func()
> {
> local arg1="$1" arg2="$2"
> local -n arg3ref="$3" arg4ref="$4"
>
> echo ''
> echo 'Local variables:'
> echo " arg1='$arg1'"
> echo " arg2='$arg2'"
> echo " arg3ref='$arg3ref'"
> echo " arg4ref='$arg4ref'"
> echo ''
>
> arg1='1st value of local assignment'
> arg2='2st value of local assignment'
> arg3ref='1st return value'
> arg4ref='2nd return value'
> }
$
$ unset foo bar baz qux
$
$ foo='value of foo'
$ bar='value of bar'
$ baz='value of baz'
$ qux='value of qux'
$
$ func foo bar baz qux
Local variables:
arg1='foo'
arg2='bar'
arg3ref='value of baz'
arg4ref='value of qux'
$
$ {
> echo ''
> echo '2 values are returned after the function call:'
> echo " foo='$foo'"
> echo " bar='$bar'"
> echo " baz='$baz'"
> echo " qux='$qux'"
> }
2 values are returned after the function call:
foo='value of foo'
bar='value of bar'
baz='1st return value'
qux='2nd return value'
I am new to bash, But found this code helping.
function return_multiple_values() {
eval "$1='What is your name'"
eval "$2='my name is: BASH'"
}
return_var=''
res2=''
return_multiple_values return_var res2
echo $return_var
echo $res2
Shell script functions can only return the exit status of last command executed or the exit status of that function specified explicitly by a return statement.
To return some string one way may be this:
function fun()
{
echo "a+b"
}
var=`fun` # Invoke the function in a new child shell and capture the results
echo $var # use the stored result
This may reduce your discomfort although it adds the overhead of creation of a new shell and hence would be marginally slower.

Putting the output of a command with interaction inside a variable while using grep in bash

This program I use has it's own variables to set when you run it, so I want to set those variables and then greping the output then storing it inside a variable. However, I don't know how to go about this the correct way. The idea I have doesn't work. The focus is on lines 7 through 14.
1 #!/usr/local/bin/bash
2 source /home/gempak/NAWIPS/Gemenviron.profile
3 FILENAME="$(date -u '+%Y%m%d')_sao.gem"
4 SFFILE="$GEMDATA/surface/$FILENAME"
5 echo -n "Enter the station ID: "
6 read -e STATION
7 OUTPUT=$(sflist << EOF
8 SFFILE = $SFFILE
9 AREA = #$STATION
10 DATTIM = all
11 SFPARM = TMPF;DWPF
12 run
13 exit
14 EOF)
15 echo $OUTPUT
But I get this:
./listweather: line 7: unexpected EOF while looking for matching `)'
./listweather: line 16: syntax error: unexpected end of file
Putting together everyone's answers, I came across a working solution myself. This code works for me:
#!/usr/local/bin/bash
source /home/gempak/NAWIPS/Gemenviron.profile
FILENAME="$(date -u '+%Y%m%d')_sao.gem"
SFFILE="$GEMDATA/surface/$FILENAME"
echo -n "Enter the station ID: "
read -e STATION
OUTPUT=$(sflist << EOF
SFFILE = $SFFILE
AREA = #$STATION
DATTIM = ALL
SFPARM = TMPF;DWPF
run
exit
EOF
)
echo $OUTPUT | grep $STATION
Thanks everyone!
I'd put your program to run in a separate .sh script file, and then run the script from your first file, passing the arguments you want to pass as command line arguments. That way you can test them separately.
You could also do it in a function, but I like the modularity of the second script. I don't udnerstand exactly what you are trying to do above, but something like:
runsflist.sh:
#!/bin/bash
FILENAME="$(date -u '+%Y%m%d')_sao.gem"
SFFILE="$GEMDATA/surface/$FILENAME"
AREA = #$STATION
DATTIM = all
SFPARM = TMPF;DWPF
grep $STATION | sflist
main.sh:
#!/bin/bash
echo -n "Enter the station ID: "
read -e STATION
OUTPUT=`runsflist.sh`
echo $OUTPUT
If sflist needs interaction, I'd try something like this:
SFFILE=$(
( echo SFFILE = "$SFFILE"
echo AREA = "#$STATION"
echo DATTIM = all
echo SFPARM = TMPF;DWPF
echo run
cat
) | sflist)
Unfortunately, you have to type exit as part of the interaction.

Resources