I'm trying to pass 2 arguments to a command and each argument contains spaces, I've tried escaping the spaces in the args, I've tried wrapping in single quotes, I've tried escaping \" but nothing will work.
Here's a simple example.
#!/bin/bash -xv
ARG="/tmp/a b/1.txt"
ARG2="/tmp/a b/2.txt"
ARG_BOTH="\"$ARG\" \"$ARG2\""
cat $ARG_BOTH
I'm getting the following when it runs:
ARG_BOTH="$ARG $ARG2"
+ ARG_BOTH='/tmp/a\ b/1.txt /tmp/a\ b/2.txt'
cat $ARG_BOTH
+ cat '/tmp/a\' b/1.txt '/tmp/a\' b/2.txt
cat: /tmp/a\: No such file or directory
cat: b/1.txt: No such file or directory
cat: /tmp/a\: No such file or directory
cat: b/2.txt: No such file or directory
See http://mywiki.wooledge.org/BashFAQ/050
TLDR
Put your args in an array and call your program as myutil "${arr[#]}"
#!/bin/bash -xv
file1="file with spaces 1"
file2="file with spaces 2"
echo "foo" > "$file1"
echo "bar" > "$file2"
arr=("$file1" "$file2")
cat "${arr[#]}"
Output
file1="file with spaces 1"
+ file1='file with spaces 1'
file2="file with spaces 2"
+ file2='file with spaces 2'
echo "foo" > "$file1"
+ echo foo
echo "bar" > "$file2"
+ echo bar
arr=("$file1" "$file2")
+ arr=("$file1" "$file2")
cat "${arr[#]}"
+ cat 'file with spaces 1' 'file with spaces 2'
foo
bar
This might be a good use-case for the generic "set" command, which sets the top-level shell parameters to a word list. That is, $1, $2, ... and so also $* and $# get reset.
This gives you some of the advantages of arrays while also staying all-Posix-shell-compatible.
So:
set "arg with spaces" "another thing with spaces"
cat "$#"
The most straightforward revision of your example shell script that will work correctly is:
#! /bin/sh
ARG="/tmp/a b/1.txt"
ARG2="/tmp/a b/2.txt"
cat "$ARG" "$ARG2"
However, if you need to wrap up a whole bunch of arguments in one shell variable, you're up a creek; there is no portable, reliable way to do it. (Arrays are Bash-specific; the only portable options are set and eval, both of which are asking for grief.) I would consider a need for this as an indication that it was time to rewrite in a more powerful scripting language, e.g. Perl or Python.
Related
I am trying to understand what a parameter expansion does inside a bash script.
third_party_bash_script
#!/bin/sh
files="${*:--}"
# For my understanding I tried to print the contents of files
echo $files
pkill bb_stream
if [ "x$VERBOSE" != "" ]; then
ARGS=-v1
fi
while [ 1 ]; do cat $files; done | bb_stream $ARGS
When I run ./third_party_bash_script, all it prints is a hyphen - and nothing else. Since it did not make sense to me, I also tried to experiment with it in the terminal
$ set one="1" two="2" three="3"
$ files="${*:--}"
$ echo $files
one="1" two="2" three="3"
$ set four="4"
$ files="${*:--}"
four="4"
I can't seem to understand what it's doing. Could someone kindly help me with the interpretation of ${*:--} by the sh?
"$#" is an array of the arguments passed to your script, "$*" is a string of all of those arguments concatenated with blanks in between.
"${*:--}" is the string of arguments if any were provided (:-), or - otherwise which means "take input from stdin" otherwise.
"${#:--}" is the array of arguments if any were provided (:-), or - otherwise which means "take input from stdin" otherwise.
$ cat file
foo
bar
$ cat tst.sh
#!/usr/bin/env bash
awk '{ print FILENAME, $0 }' "${#:--}"
When an arg is provided to the script, "$#" contains "file" so that is the arg that awk is called with:
$ ./tst.sh file
file foo
file bar
When no arg is provided to the script, "$#" is empty so awk is called with - (meaning read from stdin) as it's arg:
$ cat file | ./tst.sh
- foo
- bar
You almost always want to use "${#:--}" rather than "${*:--}" in this context, see https://unix.stackexchange.com/questions/41571/what-is-the-difference-between-and for more info on "$#" vs "$*".
${param:-default} expands to the value of $param if $param is set and not empty, otherwise it expands to default.
$* is all the command-line arguments to the script.
In ${*:--}, param is * and default is -. If $* is not an empty string, it expands to the script arguments. If it's empty, it expands to the default value -.
This can be used in a script to implement the common behavior that a program reads from the files listed in its arguments, and reads from standard input if no filename arguments are given. Many commands treat an input filename argument - as standing for the standard input.
NOTE: addressing OP's original, pre-edited post ...
See shell parameter expansion for a brief review of different options.
While the other answers reference the use of ${*:--} (and ${#:--}) as a alternate means of reading from stdin, OP's sample script is a bit simpler ... if the variable $* (ie, script's command line args) is empty then replace with the literal string -.
We can see this with a few examples:
$ third_party_bash_script
-
$ third_party_bash_script a b c
a b c
$ echo 'a b c' | third_party_bash_script
-
If we replace ${*:--} with ${*:-REPLACEMENT}:
$ third_party_bash_script
REPLACEMENT
$ third_party_bash_script a b c
a b c
$ echo 'a b c' | third_party_bash_script
REPLACEMENT
I'm guessing in OP's actual script there's more going on with the $files variable so in order to know for sure how the ${*:--} is being processed we'd need to see the actual script and how it's referencing the $files variable.
As for OP's set|files=|echo code snippets:
$ set one="1" two="2" three="3"
$ files="${*:--}"
$ echo $files
one=1 two=2 three=3
We can see the same behavior from the script with:
$ third_party_bash_script one="1" two="2" three="3"
one=1 two=2 three=3
I'm trying to create a script that creates an other script that uses $1 and $#, the problem is that those variables are being interpreted by the first script, so they are empty. Here's my problem, the first script creates the script /tmp/test.sh
#!/bin/bash
cat << EOF > /tmp/test.sh
#!/bin/bash
echo $1
echo $#
EOF
The result in /tmp/test.sh:
#!/bin/bash
echo
echo 0
Does anyone know how to avoid this and get in /tmp/test.sh $1 and $#?
I would like to have in /tmp/test.sh:
#!/bin/bash
echo $1
echo $#
Thanks in advance.
Quote the here-document delimiter so that the contents of the here document are treated as literal text (i.e., as if occurring in a single-quoted string).
cat << 'EOF' > /tmp/test.sh
#!/bin/bash
echo $1
echo $#
EOF
Any quoting will work, not just single quotes. The only important thing is that at least one character be escaped.
cat << \EOF
cat << "EOF"
cat << E"O"F
etc
I've got a program that I want to call by passing parameters from a shell variable. Throughout this question, I am going to assume that it is given by
#!/bin/sh
echo $#
i.e. that it prints out the number of arguments that are passed to it. Let's call it count-args.
I call my program like this:
X="arg1 arg2"
count-args $X
This works quite well. But now one of my arguments has a whitespace in it and I can't find a way to escape it, e.g. the following things do not work:
X="Hello\ World"
X="Hello\\ World"
X="'Hello World'"
In all of the cases, my program count-args prints out 2. I want to find a way so I can pass the string Hello World and that it returns 1 instead. How?
Just for clarification: I do not want to pass all parameters as a single string, e.g.
X="Hello World"
count-args $X
should print out 2. I want a way to pass parameters that contain whitespaces.
Use an array to store multiple, space-containing arguments.
$ args=("first one" "second one")
$ count-args "${args[#]}"
2
This can be solved with xargs. By replacing
count-args $X
with
echo $X | xargs count-args
I can use backslashes to escape whitespaces in $X, e.g.
X="Hello\\ World"
echo $X | xargs count-args
prints out 1 and
X="Hello World"
echo $X | xargs count-args
prints out 2.
count-args "$X"
The quotes ensure in bash, that the whole content of variable X is passed as a single parameter.
Your Counting script:
$ cat ./params.sh
#!/bin/sh
echo $#
For completeness here is what happens with various arguments:
$ ./params.sh
0
$ ./params.sh 1 2
2
$ ./params.sh
0
$ ./params.sh 1
1
$ ./params.sh 1 2
2
$ ./params.sh "1 2"
1
And here is what you get with variables:
$ XYZ="1 2" sh -c './params.sh $XYZ'
2
$ XYZ="1 2" sh -c './params.sh "$XYZ"'
1
Taking this a bit further:
$ cat params-printer.sh
#!/bin/sh
echo "Count: $#"
echo "1 : '$1'"
echo "2 : '$2'"
We get:
$ XYZ="1 2" sh -c './params-printer.sh "$XYZ"'
Count: 1
1 : '1 2'
2 : ''
This looks like what you wanted to do.
Now: If you have a script you cannot control and neither can you control the way the script is invoked. Then there is very little you can do to prevent a variable with spaces turning into multiple arguments.
There are quite a few questions around this on StackOverflow which indicate that you need the ability to control how the command is invoked else there is little you can do.
Passing arguments with spaces between (bash) script
Passing a string with spaces as a function argument in bash
Passing arguments to a command in Bash script with spaces
And wow! this has been asked so many times before:
How to pass argument with spaces to a shell script function
I have this (test) script:
#!/bin/bash
my_cmd_bad_ ( ) {
cmd="$#"
$cmd
}
my_cmd_good_ ( ) {
"$#"
}
my_cmd_bad_ ls -l "file with space"
my_cmd_good_ ls -l "file with space"
The output is (the file does not exist, which is not the point of this question):
ยป ~/test.sh
ls: cannot access file: No such file or directory
ls: cannot access with: No such file or directory
ls: cannot access space: No such file or directory
ls: cannot access file with space: No such file or directory
I am surprised that the first version does not work as expected: the parameter is not quoted, and instead of processing one file, it processes three. Why?
How can I save the command that I want to execute, properly quoted? I need to execute it later, where I do not have "$#" anymore.
A simple rework of this test script would be appreciated.
See similar question: How to pass command line parameters with quotes stored in single variable?
Use those utility functions ho save a command to a string for later execution:
bash_escape() {
# backtick indirection strictly necessary here: we use it to strip the
# trailing newline from sed's output, which Solaris/BSD sed *always* output
# (unlike GNU sed, which outputs "test": printf %s test | sed -e s/dummy//)
out=`echo "$1" | sed -e s/\\'/\\''\\\\'\\'\\'/g`
printf \'%s\' "$out"
}
append_bash_escape() {
printf "%s " "$1"
bash_escape "$2"
}
your_cmd_fixed_ ( ) {
cmd="$#"
while [ $# -gt 0 ] ; do
cmd=`append_bash_escape "$cmd" "$1"` ; shift
done
$cmd
}
You can quote any single parameter and evaluate it later:
my_cmd_bad_ ( ) {
j=0
for i in "$#"; do
cmd["$j"]=\"$"$i"\"
j=$(( $j + 1 ))
done;
eval ${cmd[*]}
}
You are combining three space-delimited strings "ls", "-l", and "file with space" into a single space-delimited string cmd. There's no way to know which spaces were originally quoted (in "file with space") and which spaces were introduced during the assignment to cmd.
Typically, it is not a good idea to try to build up command lines into a single string. Use functions, or isolate the actual command and leave the arguments in $#.
Rewrite the command like this:
my_cmd_bad_ () {
cmd=$1; shift
$cmd "$#"
}
See http://mywiki.wooledge.org/BashFAQ/050
Note that your second version is greatly preferred most of the time. The only exceptions are if you need to do something special. For example, you can't bundle an assignment or redirect or compound command into a parameter list.
The correct way to handle the quoting issue requires non-standard features. Semi-realistic example involving a template:
function myWrapper {
typeset x IFS=$' \t\n'
{ eval "$(</dev/fd/0)"; } <<-EOF
for x in $(printf '%q ' "$#"); do
echo "\$x"
done
EOF
}
myWrapper 'foo bar' $'baz\nbork'
Make sure you understand exactly what's going on here and that you really have a good reason for doing this. It requires ensuring side-effects can't affect the arguments. This specific example doesn't demonstrate a very good use case because everything is hard-coded so you're able to correctly escape things in advance and expand the arguments quoted if you wanted.
I have a bash script
fooA
#!/bin/bash
script_name=$1;
script_params=$( echo $# | awk '{ $1=""; print $0 }' );
bash /path/to/scripts/$script_name $script_params > /dev/stdout;
and another script fooB in the .../scripts/ directory.
#!/bin/bash
echo 1. $1
echo 2. $2
My plan is simple:
fooA fooB "some sentence 1" "some sentence 2"
should produce:
some sentence 1
some sentence 2
Using my current script, I would get
some
sentence
Because the double quotes are not preserved when calling fooB from fooA.
Keeping in mind that there many other scripts in the .../scripts directory, how would I change the script_params=$(...) line in fooA file to preserve variables when calling other scripts.
#jm666's answer will work fine if there are no additional constraints. For completeness, though, I'll give a version that doesn't mess with the first script's argument list:
#/bin/bash
script_name="$1"
script_params=( "${#:2}" )
bash /path/to/scripts/"$script_name" "${script_params[#]}" > /dev/stdout
Or you can skip the variables entirely:
#/bin/bash
bash /path/to/scripts/"$1" "${#:2}" > /dev/stdout
#!/bin/bash
name="$1"
shift
"/path/to/script/$name" "$#"