Being new to shell scripting, I am not clear about the Quoting and splitting concepts in bash. In the below snippet:
array1=("france-country":"Italy-country":"singapore-country")
echo ${#array1[#]}
IFS=":-"
for i in ${array1[#]}
do
echo "$i"
done
unset IFS
with IFS being :-, I thought the result would be:
france-country
Italy-country
belgium-country
as I had quoted them("france-country"). I think it should not get split on "-". But the results were:
france
country
Italy
country
belgium
country
It would be great if someone can point me out my mistake in understanding.
For your problem you can simply change the field separator as : i.e. IFS=: because each country name is separated by : not :- in your example
array1=("france-country":"Italy-country":"singapore-country")
echo ${#array1[#]}
IFS=":"
for i in ${array1[#]}
do
echo "$i"
done
unset IFS
FYI, array elements in bash are separated by space so the whole string "france-country":"Italy-country":"singapore-country" is a single element of the array therefore echo ${#array1[#]} will always be 1 . So I do not see any use of an array in this example. Simple variable would have been suffice.
This script shows how to split a colon-separated string into a Bash array.
#!/usr/bin/env bash
words="france-country:Italy-country:singapore-country:Australia-country"
IFS=':' array1=($words)
numwords="${#array1[#]}"
for((i=0; i<numwords; i++));do
echo "$i: ${array1[i]}"
done
output
0: france-country
1: Italy-country
2: singapore-country
3: Australia-country
Note that in
array1=($words)
we don't put quotes around $words, as that would prevent word splitting.
We don't actually need the quotes in
words="france-country:Italy-country:singapore-country:Australia-country"
but quotes (either single or double) would be needed if there were any spaces in that string. Similarly, the quotes are superfluous in numwords="${#array1[#]}", and many Bash programmers would omit them because we know the result of ${#array1[#]} will never have spaces in it.
It's a Good Idea to always use quotes in Bash unless you're sure you don't want them, eg in the case of array1=($words) above, or when performing tests with the [[ ... ]] syntax.
I suggest you bookmark BashGuide and browse through their FAQ. But the only way to really learn the arcane ways of quoting and splitting in Bash is to to write lots of scripts. :)
You may also find ShellCheck helpful.
Related
Let's suppose that you have a variable which is subject to word splitting, globing and pattern matching:
var='*
.?'
While I'm pretty sure that everyone agrees that "$var" is the best way to expand the variable as a string literal, I've identified a few cases where you don't need to use the double quotes:
Simple assignment: x=$var
Case statement: case $var in ...
Leftmost part of bash test construct: [[ $var .... ]]
UPDATE1: Bash here-string: <<< $var which works starting from bash-4.4 (thank you #GordonDavidson)
UPDATE2: Exported assignment (in bash): export x=$var
Is it correct? Is there any other shell/bash statement where the variable isn't subject to glob expansion or word splitting without using double-quotes? where expanding a variable with or without double quotes is 100% equivalent?
The reason why I ask this question is that when reading foreign code, knowing the above mentioned border-cases might help.
For example, one bug that I found in a script that I was debugging is something like:
out_exists="-f a.out"
[[ $out_exists ]] && mv a.out prog.exe
mv: cannot stat ‘a.out’: No such file or directory
This question is a duplicate of What are the contexts where Bash doesn't perform word splitting and globbing?, but that was closed before it was answered.
For a thorough answer to the question see the answer by Stéphane Chazelas to What are the contexts where Bash doesn't perform word splitting and globbing? - Unix & Linux Stack Exchange. Another good answer is in the "Where you can omit the double quotes" section in the answer by Gilles to When is double-quoting necessary? - Unix & Linux Stack Exchange.
There seem to be a small number of cases that aren't covered by the links above:
With the for (( expr1 ; expr2 ; expr3 )) ... loop, variable expansions in any of the expressions inside the (( ... )) don't need to be quoted.
Several of the expansions described in the Shell Parameter Expansion section of the Bash Reference Manual are described with a word argument that isn't subject to word splitting or pathname expansion (globbing). Examples include ${parameter:-word}, ${parameter#word}, and ${parameter%word}.
Great question! If you need to word split a variable, the quotes should be left off.
If I think of other cases, I'll add to this.
var='abc xyz'
set "$var"
echo $1
abc xyz
set $var
echo $1
abc
I have these file names that literally have double quotes in the path to deal with special characters issues, I want to loop through and echo the file paths while preserving the quotes, this seems to remove them:
for value in temp/sample."sample.id1".genotypes.txt temp/sample."sample.id2".genotypes.txt; do echo $value; done
I tried this but no luck:
for value in temp/sample."sample.id1".genotypes.txt temp/sample."sample.id2".genotypes.txt; do echo '${value}'; done
How do I do this?
You need to quote the strings to preserve the double quotes:
for value in 'temp/sample."sample.id1".genotypes.txt' 'temp/sample."sample.id2".genotypes.txt'; do
echo "$value"
done
Otherwise, writing some."thing" is identical to some.thing because the shell interprets the quotes.
You can also escape it :
for value in temp/sample.\"sample.id1\".genotypes.txt temp/sample.\"sample.id2\".genotypes.txt; do echo $value; done
for things like this, I like to use a slightly different approach that looks like a better design to me:
# make an array with the data
mapfile -t ary <<"EOF"
temp/sample."sample.id1".genotypes.txt
temp/sample."sample.id2".genotypes.txt
EOF
# use the data from the array
for f in "${ary[#]}"; do
printf '%s\n' "$f"
done
It will make your life a bit easier if your data grows, and you can then very easily transfer it to another file. Of course, if it's only for a one-time use (e.g., you made an error when naming your files and you only want to rename them), then learn how to have Bash properly parse the quotes as shown in the other answers (escaping them or using single quotes).
I am trying to parse a series of output lines that contain a mix of values and strings.
I thought that the set command would be a straightforward way to do it.
An initial test seemed promising. Here's a sample command line and its output:
$ (set "one two" three; echo $1; echo $2; echo $3)
one two
three
Obviously I get two variables echoed and nothing for the third.
However, when I put it inside my script, where I'm using read to capture the output lines, I get a different kind of parsing:
echo \"one two\" three |
while read Line
do
echo $Line
set $Line
echo $1
echo $2
echo $3
done
Here's the output:
"one two" three
"one
two"
three
The echo $Line command shows that the quotes are there but the set command does not use them to delimit a parameter. Why not?
In researching the use of read and while read I came across the while IFS= read idiom, so I tried that, but it made no difference at all.
I've read through dozens of questions about quoting, but haven't found anything that clarifies this for me. Obviously I've got my levels of quoting confused, but where? And what might I do to get what I want, which is to get the same kind of parsing in my script as I got from the command line?
Thanks.
read does not interpret the quotes, it just reads "one as one token, and two" as another. (Think of all the ways in which things could go wrong if the shell would evaluate input from random places. The lessons from Python 2 and its flawed input() are also an excellent illustration.)
If you really want to evaluate things, eval does that; but it comes with a boatload of caveats, and too often leads to security problems if done carelessly.
Depending on what you want to accomplish, maybe provide the inputs on separate lines? Or if these are user-supplied arguments, just keep them in "$#". Notice also how you can pass a subset of them into a function, which gets its own local "$#" if you want to mess with it.
(Tangentially, you are confusing yourself by not quoting the argument to echo. See When to wrap quotes around a shell variable.)
Why not?
read splits the input on each character that's in IFS. With unset or default IFS, that's space or tab or newline. Any other characters are not special in any way and quotes are not anyhow special.
Obviously I've got my levels of quoting confused, but where?
You wrongly assumed read is smart enough to interpret quotes. It isn't. Moreover, read ignores \ sequences. Read how to read a stream line by line and bash manual word splitting.
what might I do to get what I want, which is to get the same kind of parsing in my script as I got from the command line?
To get the same parsing as you got from the command line you may use eval. eval is evil. Eval command and security issues.
echo \"one two\" three |
while IFS= read -r line; do
eval "set $line" # SUPER VERY UNSAFE DO NOT USE
printf "%s\n" "$#"
done
When using eval a malicious user may echo '"$(rm -rf *)"' | ... remove your files in an instant. The simplest solution in the shell is to use xargs, which (mostly confusingly) includes quotes parsing when parsing input.
echo \"one two\" three | xargs -n1 echo
I'm writing a script in bash and I want it to execute a command and to handle each line separately. for example:
LINES=$(df)
echo $LINES
it will return all the output converting new lines with spaces.
example:
if the output was supposed to be:
1
2
3
then I would get
1 2 3
how can I place the output of a command into a variable allowing new lines to still be new lines so when I print the variable i will get proper output?
Generally in bash $v is asking for trouble in most cases. Almost always what you really mean is "$v" in double quotes:
LINES="$(df)"
echo "$LINES"
No, it will not. The $(something) only strips trailing newlines.
The expansion in argument to echo splits on whitespace and than echo concatenates separate arguments with space. To preserve the whitespace, you need to quote again:
echo "$LINES"
Note, that the assignment does not need to be quoted; result of expansion is not word-split in assignment to variable and in argument to case. But it can be quoted and it's easier to just learn to just always put the quotes in.
Hi I need to go over characters in string in bash including spaces. How can I do it?
Bash does support substrings directly (If that's what the OP wants):
$ A='Hello World!'
$ echo "${A:3:5}"
lo Wo
$ echo "${A:5:3}"
Wo
$ echo "${A:7:3}"
orl
The expansion used is generalized as:
${PARAMETER:OFFSET:LENGTH}
PARAMETER is your variable name. OFFSET and LENGTH are numeric expressions as used by `let'. See the bash info page on shell parameter expansion for more information, since there are a few important details on this.
Therefore, if you want to e.g. print all the characters in the contents of a variable each on its own line you could do something like this:
$ for ((i=0; i<${#A}; i++)); do echo ${A:i:1}; done
The advantage of this method is that you don't have to store the string elsewhere, mangle its contents or use external utilities with process substitution.
Not sure what you really mean, but in almost all cases, problems with strings including spaces can be solved by quoting them.
So, if you've got a nice day, try "a nice day" or 'a nice day'.
You use some external tool for it. The bash shell is really meant to be used to glue other programs together in usually simple combinations.
Depending on what you need, you might use cut, awk, sed or even perl.
Try this
#/bin/bash
str="so long and thanks for all the fish"
while [ -n "$str" ]
do
printf "%c\n" "$str"
str=${str#?}
done