One of my bash shells is giving me different behavior than the others. I don't know what setting has changed, but check this out:
hersh#huey$ for f in $(echo a b c); do echo foo $f; done
foo a
foo b
foo c
hersh#huey$ logout
Connection to huey closed.
hersh#spf$ for f in $(echo a b c); do echo foo $f; done
foo a b c
On host huey, the "for" loop over did what I expected: it split on word boundaries. On host spf, the "for" loop did not split on word boundaries, it treated the whole thing as a single item. Similarly if I use $(ls) it will treat the whole big multi-line thing as a single item and print "foo" just in front of the first line.
On both hosts I'm running bash. In other bash shells on host spf it also works normally (splits on word boundaries). What could change that? Both are running the same version of bash (which is "4.2.25(1)-release (x86_64-pc-linux-gnu)"). The characters are the same, since I copy-pasted the for-loop command line with the mouse from the same source both times.
Thanks!
Check your $IFS
mogul#linuxine:~$ for f in $(echo a b c); do echo foo $f; done
foo a
foo b
foo c
mogul#linuxine:~$ export IFS=SOMETHINGBAD
mogul#linuxine:~$ for f in $(echo a b c); do echo foo $f; done
foo a b c
mogul#linuxine:~$
and have a look at the manual over here bash Word Splitting
Related
I am trying to understand what a parameter expansion does inside a bash script.
third_party_bash_script
#!/bin/sh
files="${*:--}"
# For my understanding I tried to print the contents of files
echo $files
pkill bb_stream
if [ "x$VERBOSE" != "" ]; then
ARGS=-v1
fi
while [ 1 ]; do cat $files; done | bb_stream $ARGS
When I run ./third_party_bash_script, all it prints is a hyphen - and nothing else. Since it did not make sense to me, I also tried to experiment with it in the terminal
$ set one="1" two="2" three="3"
$ files="${*:--}"
$ echo $files
one="1" two="2" three="3"
$ set four="4"
$ files="${*:--}"
four="4"
I can't seem to understand what it's doing. Could someone kindly help me with the interpretation of ${*:--} by the sh?
"$#" is an array of the arguments passed to your script, "$*" is a string of all of those arguments concatenated with blanks in between.
"${*:--}" is the string of arguments if any were provided (:-), or - otherwise which means "take input from stdin" otherwise.
"${#:--}" is the array of arguments if any were provided (:-), or - otherwise which means "take input from stdin" otherwise.
$ cat file
foo
bar
$ cat tst.sh
#!/usr/bin/env bash
awk '{ print FILENAME, $0 }' "${#:--}"
When an arg is provided to the script, "$#" contains "file" so that is the arg that awk is called with:
$ ./tst.sh file
file foo
file bar
When no arg is provided to the script, "$#" is empty so awk is called with - (meaning read from stdin) as it's arg:
$ cat file | ./tst.sh
- foo
- bar
You almost always want to use "${#:--}" rather than "${*:--}" in this context, see https://unix.stackexchange.com/questions/41571/what-is-the-difference-between-and for more info on "$#" vs "$*".
${param:-default} expands to the value of $param if $param is set and not empty, otherwise it expands to default.
$* is all the command-line arguments to the script.
In ${*:--}, param is * and default is -. If $* is not an empty string, it expands to the script arguments. If it's empty, it expands to the default value -.
This can be used in a script to implement the common behavior that a program reads from the files listed in its arguments, and reads from standard input if no filename arguments are given. Many commands treat an input filename argument - as standing for the standard input.
NOTE: addressing OP's original, pre-edited post ...
See shell parameter expansion for a brief review of different options.
While the other answers reference the use of ${*:--} (and ${#:--}) as a alternate means of reading from stdin, OP's sample script is a bit simpler ... if the variable $* (ie, script's command line args) is empty then replace with the literal string -.
We can see this with a few examples:
$ third_party_bash_script
-
$ third_party_bash_script a b c
a b c
$ echo 'a b c' | third_party_bash_script
-
If we replace ${*:--} with ${*:-REPLACEMENT}:
$ third_party_bash_script
REPLACEMENT
$ third_party_bash_script a b c
a b c
$ echo 'a b c' | third_party_bash_script
REPLACEMENT
I'm guessing in OP's actual script there's more going on with the $files variable so in order to know for sure how the ${*:--} is being processed we'd need to see the actual script and how it's referencing the $files variable.
As for OP's set|files=|echo code snippets:
$ set one="1" two="2" three="3"
$ files="${*:--}"
$ echo $files
one=1 two=2 three=3
We can see the same behavior from the script with:
$ third_party_bash_script one="1" two="2" three="3"
one=1 two=2 three=3
I have a shell script that executes a command with a bunch of arguments that are, themselves, constructed by the script.
Eg. the script contains a line like this
do $opts x y z $more_opts a b c "bit in quotes" $stuff
I'd like to print out what this line actually is.
If I try :
echo 'do $opts x y z $more_opts a b c "bit in quotes" $stuff'
then I get the line with $opts and $more_opts being quoted literally rather than expanded into their values.
If I try backticks :
echo `do $opts x y z $more_opts a b c "bit in quotes" $stuff`
then it executes the line itself.
How can I generate and print a version of this string, with the $opts expanded into their values, but unexecuted?
The most accurate is:
printf '%q ' $opts x y z $more_opts a b c "bit in quotes" $stuff ; echo
since it will do the same word-splitting, filename-globbing, etc., as your actual command. (For example, if $stuff is *, this version will correctly list all the files in your directory, since that's what your real command will do.)
(Hat-tip to Gordon Davisson for suggesting printf + %q, so the final resulting arguments get re-quoted, instead of just printing them as-is.)
A better approach, though, might be what Jonathan Leffler suggests above:
(set -x; cmd $opts x y z $more_opts a b c "bit in quotes" $stuff)
which will tell Bash to debug-print the command before running it. That way you don't need to update your echo command every time you update your actual command.
While the solutions outlined here work, I would strongly advice against doing it, because a statement of the form
echo myprog $foo $bar $baz
myprog $foo $bar $baz
creates a maintenance problem: Whenever you modify the myprog .... line, you have to update the echo .... line in the same way.
A better solution would be to define a function
function echorun {
printf "EXECUTING:" # You can leave this out, if you want to
printf " %q" "$#" # Thank you, Gordon Davisson
echo # Supplying final newline
"$#"
}
and run the commands which you are wanting to log, with
echorun do $opts x y z $more_opts a b c "bit in quotes" $stuff
I am having some troubles and misunderstanding about how are variable assignment and parenthesis handled in the different existing shells.
What is currently puzzling me is the following:
Always using the following command
./script.sh a b c d
when running the following code
#!/bin/zsh
bar=$#
for foo in $bar
do
echo $foo
done
the output is
a b c d
and with
#!/bin/zsh
bar=($#)
for foo in $bar
do
echo $foo
done
it is (what I initially wanted)
a
b
c
d
but using bash or sh
#!/bin/bash
bar=$#
for foo in $bar
do
echo $foo
done
gives
a
b
c
d
and
#!/bin/bash
bar=($#)
for foo in $bar
do
echo $foo
done
it is just
a
What is going on there ?
Joint operations
For both of the shells involved, the examples given will assume an explicitly set argv list:
# this sets $1 to "first entry", $2 to "second entry", etc
$ set -- "first entry" "second entry" "third entry"
In both shells, declare -p can be used to emit the value of a variable name in unambiguous form, though how they represent that form can vary.
In bash
Expansion rules in bash are generally compatible with ksh and, where applicable, POSIX sh semantics. Being compatible with these shells requires that unquoted expansion perform string-splitting and glob expansion (replacing * with a list of files in the current directory, for instance).
Using parenthesis in a variable assignment makes it an array. Compare these three assignments:
# this sets arr_str="first entry second entry third entry"
$ arr_str=$#
$ declare -p arr_str
declare -- arr="first entry second entry third entry"
# this sets arr=( first entry second entry third entry )
$ arr=( $# )
declare -a arr='([0]="first" [1]="entry" [2]="second" [3]="entry" [4]="third" [5]="entry")'
# this sets arr=( "first entry" "second entry" "third entry" )
$ arr=( "$#" )
$ declare -p arr
declare -a arr='([0]="first entry" [1]="second entry" [2]="third entry")'
Similarly, on expansion, quotes and sigils matter:
# quoted expansion, first item only
$ printf '%s\n' "$arr"
first entry
# unquoted expansion, first item only: that item is string-split into two separate args
$ printf '%s\n' $arr
first
entry
# unquoted expansion, all items: each word expanded into its own argument
$ printf '%s\n' ${arr[#]}
first
entry
second
entry
third
entry
# quoted expansion, all items: original arguments all preserved
$ printf '%s\n' "${arr[#]}"
first entry
second entry
third entry
In zsh
zsh does a great deal of magic to try to do what the user means, rather than what's compatible with historical shells (ksh, POSIX sh, etc). However, even there, doing the wrong thing can have results other than what you want:
# Assigning an array to a string still flattens it in zsh
$ arr_str=$#
$ declare -p arr_str
typeset arr_str='first entry second entry third entry'
# ...but quotes aren't needed to keep arguments together on array assignments.
$ arr=( $# )
$ declare -p arr
typeset -a arr
arr=('first entry' 'second entry' 'third entry')
# in zsh, expanding an array always expands to all entries
$ printf '%s\n' $arr
first entry
second entry
third entry
# ...and unquoted string expansion doesn't do string-splitting by default:
$ printf '%s\n' $arr_str
first entry second entry third entry
When you do this:
bar=($#)
You're actually creating a bash shell array. To iterate a bash array use:
bar=( "$#" ) # safer way to create array
for foo in "${bar[#]}"
do
echo "$foo"
done
I have a variable that looks sort of like this:
msg="newton apple tree"
I want to assign each of these words into separate variables. This is easy to do in bash:
read a b c <<< $msg
Is there a compact, readable way to do this in POSIX shell?
A here string is just syntactic sugar for a single-line here document:
$ msg="foo * bar"
$ read a b c <<EOF
> $msg
> EOF
$ echo "$a"
foo
$ echo "$b"
*
$ echo "$c"
bar
To write idiomatic scripts, you can't just look at each individual syntax element and try to find a POSIX equivalent. That's like translating text by replacing each individual word with its entry in the dictionary.
The POSIX way of splitting a string known to have three words into three arguments, similar but not identical to read is:
var="newton apple tree"
set -f
set -- $var
set +f
a=$1 b=$2 c=$3
echo "$a was hit by an $b under a $c"
It's not pretty, but as a general-purpose solution, you can work around this with a named pipe.
From BashFAQ #24:
mkfifo mypipe
printf '%s\n' "$msg" >mypipe &
read -r a b c <mypipe
printf is more reliable / better-specified than echo; echo behavior varies between implementations if you have a message containing only, say, -E or -n.
That said, for what you're doing here, you could just use parameter expansion:
a=${msg%% *}; msg=${msg#* }
b=${msg%% *}; msg=${msg#* }
c=${msg%% *}; msg=${msg#* }
This is my first time using Bash, so bear with me. I'm using Git Bash in Windows for a college project, trying to rewrite some C code that provides an alternate way of multiplying two numbers "a" and "b" to produce "c". This is what I came up with:
#!/bin/bash
declare -i a
declare -i b
declare -i c=0
declare -i i=0 # not sure if i have to initialize this as 0?
echo "Please enter a number: "
read a
echo "Please enter a number: "
read b
for i in {1..b}
do
let "c += a"
done
echo "$a x $b = $c"
I think part of the problem is in the for loop, that it only executes once. This is my first time using Bash, and if anyone could find the fault in my knowledge, that would be all I need.
There are problems with your loop:
You can't use {1..b}. Even if you had {1..$b} it wouldn't work because you would need an eval. It's easiest to use the seq command instead.
Your let syntax is incorrect.
Try this:
for i in $(seq 1 $b)
do
let c+=$a
done
Also, it's not necessary to declare or initialise i.
for i in {1..b}
won't work, because 'b' isn't interpreted as a variable but a character to iterate to.
For instance {a..c} expands to a b c.
To make the brace expansion work:
for i in $(eval echo "{1..$b}")
The let "c += a" won't work either.
let c+=$a might work, but I like ((c+=a)) better.
Another way is this:
for ((i = 1; i <= b; i++))
do
((c += a))
done
(might need to put #!/bin/bash at the top of your script, because sh does less than bash.)
Of course, bash already has multiplication, but I guess you knew that ...
If the absence of "seq" is your issue, you can replace it with something more portable, like
c=0
# Print an endless sequence of lines
yes |
# Only take the first $b lines
head -n "$b" |
# Add line number as prefix for each line
nl |
# Read the numbers into i, and the rest of the line into a dummy variable
while read i dummy; do
# Update the value of c: add line number
c=`expr "$c" + "$i"`
echo "$c"
done |
# Read the last number from the while loop
tail -n 1
This should be portable to any Bourne-compatible shell. The while ... echo ... done | tail -n 1 trick is necessary only if the value of c is not exported outside the while loop, as is the case in some, but not all, Bource shells.
You can implement a seq replacement with a Perl one-liner, but then you might as well write all of this in Perl (or awk, or Python, or what have you).