I'd like to store a command line in a variable, and then execute that command line. The problem is, the command line has arguments with spaces in them. If I do
$ x='command "complex argument"'
$ $x
it calls command with "complex and argument". I tried using "$x" thinking it would preserve the argument splittings, but it only tries to execute a program with the file name command "complex argument". I also tried variations of the quotes (' vs ") and using exec, but it didn't help. Any ideas?
Edit: eval "$x" almost works, but if the whitespaces separating arguments are newlines and not spaces, then it treats the lines as separate commands.
Edit2: The extra " quotes were too much, and made eval interpret the newlines not as spaces, but as command delimiters. The solutions in the answers all work.
For testing purposes, I define:
$ function args() { while [[ "$1" != "" ]]; do echo arg: $1; shift; done }
This works as expected:
$ args "1 2" 3
arg: 1 2
arg: 3
$ x="arg 4 5 6"
$ $x
arg: 4
arg: 5
arg: 6
This doesnt:
$ x="args \"3 4\" 5"
$ $x
arg: "3
arg: 4"
arg: 5
A safe approach is to store your command line in a BASH array:
arr=( command "complex argument" )
Then execute it:
"${arr[#]}"
OR else another approach is to use BASH function:
cmdfunc() {
command "complex argument"
}
cmdfunc
Another solution is
x='command "complex argument"'
eval $x
Related
I have a script which installs a tool, if it's not already on my system:
#!/bin/bash
ASCIIDOCTOR_BIN=`command -v asciidoctor`
if ! [ -x $ASCIIDOCTOR_BIN ]; then ......
The problem is, that it doesn't seem to be working: the conditional expression always evaluates to false.
Even with
#!/bin/bash
ASCIIDOCTOR_BIN=`command -v asciidoctor`
if ! [ false ]; then ......
the condition turns out to be just false.
As I was playing around with it I found out that if I replace the brackets with double brackets, it works:
#!/bin/bash
ASCIIDOCTOR_BIN=`command -v asciidoctor`
if ! [[ -x $ASCIIDOCTOR_BIN ]]; then ......
Can any1 explain why is that? I was told to avoid double brackets and curious, if there are other ways to make this script work.
I am working on Ubuntu 20.04.
You problem is that you are missing quotes, and don't understand how parameter expansion works:
Consider the following, assuming you enter a bash prompt:
$ a=""
$ b="helloWorld"
$ bash -c 'for arg; do echo "got arg: $arg"; done' _ $a $b
got arg: helloWorld
You would assume that bash is called with the positional parameters:
$0 = _
$1 = "" # empty string
$2 = "helloWorld"
But in reality the parameter $a expands to nothing resulting in bash being called with:
$0 = _
$1 = "helloWorld"
Applying our knowledge to your example you will notice that if $ASCIIDOCTOR_BIN is empty, which it is when the command doesn't exists, the if statement will become:
if [ -x ]; then ...
Which will test if the string -x is empty, which it clearly isn't.
The solution to this is to always quote parameter expansions:
$ a=""
$ b="helloWorld"
$ bash -c 'for arg; do echo "got arg: $arg"; done' _ "$a" "$b"
got arg:
got arg: helloWorld
One thing you should know about [ is that it isn't part of the language syntax but rather a command, usually implemented in the shell.
Same goes for true and false, these are also command, which means [ false ] is simply testing if the string false is empty, which it clearly isn't.
So parameter expansion etc applies to all it's arguments.
A note on [[. [[ is a special bash syntax where in parameter expansion works differently, and therefore you code works.
You can find a nice writeup on how parameter expansion works in bash: https://mywiki.wooledge.org/BashGuide/Parameters
I have two bash string built in commands that work fine independently but when nested generate an error message no matter what I try. Here's the two individual commands that work:
$ A="etc/.java"
$ echo $A
/etc/.java
$ B="${A//$'\057\056'/$'\057'}"
$ echo $B
/etc/java
$ B="${A^^}"
$ echo $B
/ETC/.JAVA
Now trying to combine the two commands together I get errors:
$ B="${${A^^}//$'\057\056'/$'\057'}"
bash: ${${A^^}///.//}: bad substitution
$ B="${ ${A^^}//$'\057\056'/$'\057'}"
bash: ${ ${A^^}///.//}: bad substitution
$ B="${ ${A^^} //$'\057\056'/$'\057'}"
bash: ${ ${A^^} ///.//}: bad substitution
$ B="${"${A^^}"//$'\057\056'/$'\057'}"
bash: ${"${A^^}"//'/.'/'/'}: bad substitution
$ B="${ "${A^^}" //$'\057\056'/$'\057'}"
bash: ${ "${A^^}" //'/.'/'/'}: bad substitution
$ B="${${A^^} //$'\057\056'/$'\057'}"
bash: ${${A^^} ///.//}: bad substitution
Simplified examples are presented above so one can copy and paste to their own terminal. Piping or redirection would be complicated because my real world code is this:
while [[ $i -lt $DirsArrCnt ]] ; do
DirsArr[$i]=false
CurrNdx=$i
CurrKey="${DirsArr[$(( $i + 1 ))]}"
# ^^ = convert to upper-case
# ${Variable//$'\041\056'/$'\041'} = Change /. to / for hidden directory sorting
if [[ "${"${CurrKey^^}"//$'\041\056'/$'\041'}" > \
"${"${LastKey^^}"//$'\041\056'/$'\041'}" ]] || \
[[ "${"${CurrKey^^}"//$'\041\056'/$'\041'}" = \
"${"${LastKey^^}"//$'\041\056'/$'\041'}" ]] ; then
LastNdx=$CurrNdx
LastKey="$CurrKey"
i=$(( $i + $OneDirArrCnt))
continue
fi
In the special case of one of the expansions being upper casing, it can be done in a single expansion, using declare -u (introduced in Bash 4.0). declare -u converts to uppercase on assignment.
Combining upper casing and substitution then becomes this:
$ declare -u A='/etc/.java'
$ echo "${A//\/./\/}"
/ETC/JAVA
There is the analogous -l for lower casing and the (undocumented) -c for title casing, but these are the only cases where you can do "nested" parameter expansion.
I've got a program that I want to call by passing parameters from a shell variable. Throughout this question, I am going to assume that it is given by
#!/bin/sh
echo $#
i.e. that it prints out the number of arguments that are passed to it. Let's call it count-args.
I call my program like this:
X="arg1 arg2"
count-args $X
This works quite well. But now one of my arguments has a whitespace in it and I can't find a way to escape it, e.g. the following things do not work:
X="Hello\ World"
X="Hello\\ World"
X="'Hello World'"
In all of the cases, my program count-args prints out 2. I want to find a way so I can pass the string Hello World and that it returns 1 instead. How?
Just for clarification: I do not want to pass all parameters as a single string, e.g.
X="Hello World"
count-args $X
should print out 2. I want a way to pass parameters that contain whitespaces.
Use an array to store multiple, space-containing arguments.
$ args=("first one" "second one")
$ count-args "${args[#]}"
2
This can be solved with xargs. By replacing
count-args $X
with
echo $X | xargs count-args
I can use backslashes to escape whitespaces in $X, e.g.
X="Hello\\ World"
echo $X | xargs count-args
prints out 1 and
X="Hello World"
echo $X | xargs count-args
prints out 2.
count-args "$X"
The quotes ensure in bash, that the whole content of variable X is passed as a single parameter.
Your Counting script:
$ cat ./params.sh
#!/bin/sh
echo $#
For completeness here is what happens with various arguments:
$ ./params.sh
0
$ ./params.sh 1 2
2
$ ./params.sh
0
$ ./params.sh 1
1
$ ./params.sh 1 2
2
$ ./params.sh "1 2"
1
And here is what you get with variables:
$ XYZ="1 2" sh -c './params.sh $XYZ'
2
$ XYZ="1 2" sh -c './params.sh "$XYZ"'
1
Taking this a bit further:
$ cat params-printer.sh
#!/bin/sh
echo "Count: $#"
echo "1 : '$1'"
echo "2 : '$2'"
We get:
$ XYZ="1 2" sh -c './params-printer.sh "$XYZ"'
Count: 1
1 : '1 2'
2 : ''
This looks like what you wanted to do.
Now: If you have a script you cannot control and neither can you control the way the script is invoked. Then there is very little you can do to prevent a variable with spaces turning into multiple arguments.
There are quite a few questions around this on StackOverflow which indicate that you need the ability to control how the command is invoked else there is little you can do.
Passing arguments with spaces between (bash) script
Passing a string with spaces as a function argument in bash
Passing arguments to a command in Bash script with spaces
And wow! this has been asked so many times before:
How to pass argument with spaces to a shell script function
I have a Perl script that gives me a defined list of random numbers that correspond to the lines of a file. Next I want to extract those lines from the file using sed.
#!/bin/bash
count=$(cat last_queries.txt | wc -l)
var=$(perl test.pl test2 $count)
The variable var returns an output like: cat last_queries.txt | sed -n '12p;500p;700p'. The problem is that I can't run this last command. I tried with $var, but the output is not correct (if I run manually the command it works fine, so no problem there). What is the correct way to do this?
P.S: Sure I could do all the work in Perl, but I'm trying to learn this way, because it could help me in other situations.
You just need to do:
#!/bin/bash
count=$(cat last_queries.txt | wc -l)
$(perl test.pl test2 $count)
However, if you want to call your Perl command later, and that's why you want to assign it to a variable, then:
#!/bin/bash
count=$(cat last_queries.txt | wc -l)
var="perl test.pl test2 $count" # You need double quotes to get your $count value substituted.
...stuff...
eval $var
As per Bash's help:
~$ help eval
eval: eval [arg ...]
Execute arguments as a shell command.
Combine ARGs into a single string, use the result as input to the shell,
and execute the resulting commands.
Exit Status:
Returns exit status of command or success if command is null.
You're are probably looking for eval $var.
line=$((${RANDOM} % $(wc -l < /etc/passwd)))
sed -n "${line}p" /etc/passwd
just with your file instead.
In this example I used the file /etc/password, using the special variable ${RANDOM} (about which I learned here), and the sed expression you had, only difference is that I am using double quotes instead of single to allow the variable expansion.
There're 2 basic ways of executing a string command in a shell script whether it's given as parameter or not here's.
COMMAND="ls -lah"
$(echo $COMMAND)
or
COMMAND="ls -lah"
bash -c $COMMAND
In the case where you have multiple variables containing the arguments for a command you're running, and not just a single string, you should not use eval directly, as it will fail in the following case:
function echo_arguments() {
echo "Argument 1: $1"
echo "Argument 2: $2"
echo "Argument 3: $3"
echo "Argument 4: $4"
}
# Note we are passing 3 arguments to `echo_arguments`, not 4
eval echo_arguments arg1 arg2 "Some arg"
Result:
Argument 1: arg1
Argument 2: arg2
Argument 3: Some
Argument 4: arg
Note that even though "Some arg" was passed as a single argument, eval read it as two.
Instead, you can just use the string as the command itself:
# The regular bash eval works by jamming all its arguments into a string then
# evaluating the string. This function treats its arguments as individual
# arguments to be passed to the command being run.
function eval_command() {
"$#";
}
Note the difference between the output of eval and the new eval_command function:
eval_command echo_arguments arg1 arg2 "Some arg"
Result:
Argument 1: arg1
Argument 2: arg2
Argument 3: Some arg
Argument 4:
Better ways to do it
Using a function:
# define it
myls() {
ls -l "/tmp/test/my dir"
}
# run it
myls
Using an array:
# define the array
mycmd=(ls -l "/tmp/test/my dir")
# run the command
"${mycmd[#]}"
cmd="ls -atr ${HOME} | tail -1" <br/>
echo "$cmd" <br/>
VAR_FIRST_FILE=$( eval "${cmd}" ) <br/>
or
cmd=("ls -atr ${HOME} | tail -1") <br/>
echo "$cmd" <br/>
VAR_FIRST_FILE=$( eval "${cmd[#]}" )
Let's say I have a really simple shell script 'foo':
#!/bin/sh
echo $#
If I invoke it like so:
foo 1 2 3
It happily prints:
1 2 3
However, let's say one of my arguments is double-quote enclosed and contains whitespace:
foo 1 "this arg has whitespace" 3
foo happily prints:
1 this arg has whitespace 3
The double-quotes have been stripped! I know shell thinks its doing me a favor, but... I would like to get at the original version of the arguments, unmolested by shell's interpretation. Is there any way to do so?
First, you probably want quoted version of $#, i.e. "$#". To feel the difference, try putting more than one space inside the string.
Second, quotes are element of shell's syntax -- it doesn't do you a favor. To preserve them, you need to escape them. Examples:
foo 1 "\"this arg has whitespace\"" 3
foo 1 '"this arg has whitespace"' 3
Double quote $#:
#!/bin/sh
for ARG in "$#"
do
echo $ARG
done
Then:
foo 1 "this arg has whitespace" 3
will give you:
1
this arg has whitespace
3
What i'd do is to quote all the arguments received with spaces that might help your case.
for x in "${#}" ; do
# try to figure out if quoting was required for the $x
if [[ "$x" != "${x%[[:space:]]*}" ]]; then
x="\""$x"\""
fi
echo $x
_args=$_args" "$x
done
echo "All Cmd Args are: $_args"
You need to quote the quotes:
foo 1 "\"this arg has whitespace\"" 3
or (more simply)
foo 1 '"this arg has whitespace"' 3
You need to quote the double quotes to make sure that the shell doesn't remove them when parsing word arguments.
Let's suppose you are in a more rigid set-up and you CANNOT change your command line, and make it more "friendly" by escaping the double quotes. For example:
example_script.sh argument_without_quotes "argument with quotes i cannot escape"
First consider that inside your script you can't tell if an argument is passed with or without quotes, because the shell strips them.
So what you can possibly do is rebuilding double quotes for arguments containing whitespaces
This example rebuilds the whole command line, double-quoting arguments that have white spaces
#!/bin/sh
#initialize the variable that will contain the whole argument string
argList=""
#iterate on each argument
for arg in "$#"
do
#if an argument contains a white space, enclose it in double quotes and append to the list
#otherwise simply append the argument to the list
if echo $arg | grep -q " "; then
argList="$argList \"$arg\""
else
argList="$argList $arg"
fi
done
#remove a possible trailing space at the beginning of the list
argList=$(echo $argList | sed 's/^ *//')
#pass your argument list WITH QUOTES
echo "my_executable" $argList
#my_executable $argList
Note this limitation. If you run this example
example_script.sh "argument with spaces" argument_without_spaces "argument_doublequoted_but_without_spaces"
you will get this output
my_executable "argument with spaces" argument_without_spaces argument_doublequoted_but_without_spaces
Note the last argument: since it had no spaces, it has not been enclosed again in double quotes, but this shouldn't be an issue.
The most reliable method that I've found to do this is to leverage the logic that's built into the shell for "xtrace". This is a subsystem that's turned on via set -x which will print every command that's run by the shell. If we turn on xtrace then run a command that does nothing (true) with the same arguments, the shell (most shells...) will quote the arguments for us.
show_quoted() {
(
# use > as the xtrace prefix (this is the default)
PS4='+'
exec 2>&1 # send the xtrace (on stderr, 2) to stdout (1) for processing
# turn on xtrace -- all commands are printed, with $PS4 as a prefix
set -x
# `true` is the command that does nothing, successfully
true "$#"
) |
sed '
# remove the xtrace prefix -- any number of + characters
s/^+*//
# hide the "true" command
s/^true //
'
}
Same thing, but with a normal amount of comments:
show_quoted() {
(
PS4='+' # reset to default
exec 2>&1 # send xtrace to stdout
set -x
true "$#"
) | sed 's/^+*true //' # remove the xtrace prefix
}
Here's my test results: (these vary a bit but are all valid)
$ sh show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
1 '2 3' 'he said: "hi"' 'she said: '\''bye'\'''
$ bash show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
1 '2 3' 'he said: "hi"' 'she said: '\''bye'\'''
$ zsh show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
1 '2 3' 'he said: "hi"' 'she said: '\''bye'\'
$ busybox sh show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
1 '2 3' 'he said: "hi"' 'she said: '"'"'bye'"'"
These shells (below) weren't up to the task however. To be fair, the toybox shell is "80% done".
$ dash show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
1 2 3 he said: "hi" she said: 'bye'
$ ./toybox sh show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
"$#"