Bash Shell parameter expansion ${ } - bash

I am not quite sure how to understand this-
$ var=' '
$ echo "|${var}|"
| |
$ echo "|${var// /}|"
||
Vs.
$ set -- '' '' ''
$ echo "|${*}|"
| |
$ echo "|${*// /}|"
| |
However, when I add this after the above
$ IFS=
$echo "|${*// /}|"
||
What is going wrong in the second set of commands? Is this the expected outcome?

Example 1
$ var=' '
$ echo "|${var}|"
| |
$ echo "|${var// /}|"
||
Here you have a simple string consisting of two spaces. When you expand it between two pipe characters, you see two spaces between the pipes. When you use pattern substitution to remove all the spaces from the expansion of the variable, you see the empty string between two pipes.
Example 2
$ set -- '' '' ''
First, you've set each of the first three positional parameters to the empty string. You can observe this by comparing the results of the ${1-foo} with {$4-foo} (which displays the parameter if set, but 'foo' if it is unset).
$ echo ${1-foo}
$ echo ${4-foo}
foo
So we can see that $1 is set, but null, while $4 is unset.
$ echo "|${*}|"
| |
Next, we see the result of expanding the special parameter $* inside quotation marks, which is a single string consisting of the positional parameters that are set, separated by the first character of the IFS parameter. IFS by default has a space as its first parameter, so what we see is a string that consists of 3 empty strings, each separated by a space, which is just a single string of 2 spaces.
$ echo "|${*// /}|"
| |
When you apply pattern substitution to $*, the substitution is applied to each positional parameter separately before the resulting parameters are joined using IFS. Since the positional parameters are already empty, removing spaces from them leaves them unchanged. So you get the same result as when you just expanded $* by itself.
Example 3
$ IFS=
$ echo "|${*// /}|"
||
The procedure here is the same as in example 2, with the important difference that now IFS is the null string, rather than its default of ''. Once again, the pattern substitution doesn't really do anything, because there are no spaces to remove from any of the positional parameters. But now, expanding $* results in a string consisting of the positional parameters with no intervening characters. Instead of $1 $2 $3, you get $1$2$3. Since all three are themselves empty strings, the result is the empty string.

Related

bash string split by delimiter breaks on empty value

I have a variable like LINE=foo,bar,,baz
I tried to split this with delimiter , using 2 different techniques:
array=(${LINE//,/ })
array=($(echo "$LINE" | tr ',' '\n'))
echo ${array[2]} should return an empty value but it returns baz
(Both treat baz as the 3rd value when it should be the 4th instead.)
You can do this with read -a and an alternate delimiter:
IFS=, read -a array <<<"$LINE"
Note that since the assignment to IFS is a prefix to the read command, it only applies to that one command, and you don't have to set it back to normal afterward. Also, unlike the ones that depend on word-splitting of unquoted variables, it won't try to "expand" any entries that look like filename wildcards into lists of matching files.
Demo:
$ LINE=foo,bar,,baz
$ IFS=, read -a array <<<"$LINE"
$ declare -p array
declare -a array='([0]="foo" [1]="bar" [2]="" [3]="baz")'
You are relying on a sequence of whitespace between tokens as the separator, but of course, that will lose any empty strings.
As a somewhat crude workaround, temporarily override IFS:
oldIFS=$IFS
IFS=','
array=($LINE)
IFS=$oldIFS
Demo: https://ideone.com/Dd1gUV
By default array treat all 'blank' characters as delimiters. If you need to preserve empty values you have to take care of them first.
line='foo, ,bar,,baz'
line=${line//,,/,null,}
line=${line//,[[:blank:]],/,null,}
array=(${line//,/ })
$ echo "${array[#]}"
foo null bar null baz
$ echo "${array[#]//null/ }"
foo bar baz
You could use mapfile (or readarray, same thing):
$ LINE=foo,bar,,baz
$ declare -a PARTS
$ mapfile -t -d, PARTS <<<"$LINE"
$ declare -p PARTS
declare -a PARTS=([0]="foo" [1]="bar" [2]="" [3]=$'baz\n')
There's an extraneous newline at the end of the 3rd element, hence the $'baz\n' value, so you'd have to handle that (see discussion in comments, though). Not sure where it comes from.

How can I increment an infix variable in Bash?

I have a string foo-0 that I want to convert to bar1baz, i.e., parse the trailing index and add a prefix/suffix. The part before the trailing index (in this case foo- can also contain numeric characters, but those should not be changed.
I tried the following:
echo foo-0 | cut -d'-' -f 2 | sed 's/.*/bar&baz/'
but that gives me only a partial solution (bar0baz). How can I increment the infix variable?
EDIT: the solutions below only work partially for what I am trying to achieve. This is my fault because I simplified the example above too much for the sake of clarity.
The final goal is to set an environmental variable (let's call it MY_ENV) to the output value using bash with the following syntax:
/bin/sh -c "echo $var | ... (some bash magic to replace the trailing index) | ... (some bash magic to set MY_ENV=the output of the pipe)"
Side note: The reason I am using /bin/sh -c "..." is because I want to use the command in a Kubernetes YAML.
Partial solution (using awk)
This works:
echo foo-0 | awk -F- '{print "bar" $2+1 "baz"}'
This doesn't (output is 1baz):
/bin/sh -c "echo foo-0 | awk -F- '{print \"bar\" $2+1 \"baz\"}'
Partial solution (using arithmetic context and parameter expansion)
$ var=foo-0
$ echo "bar$((${var//[![:digit:]]}+1))baz"
This does not work if var contains other numeric characters before the trailing index (e.g. for var foo=r2a-foo-0.
You may use awk:
awk -F- '{print "bar" $2+1 "baz"}' <<< 'foo-0'
bar1baz
You could use an arithmetic context and parameter expansion:
$ var=foo-0
$ echo "bar$((${var//[![:digit:]]}+1))baz"
bar1baz
Unrolled, from the inside:
${var//[![:digit:]]} removes all non-digits from var:
$ echo "${var//[![:digit:]]}"
0
$((blah+1)) adds 1 to the variable blah:
$ blah=0
$ echo "$((blah+1))"
1
or, instead of blah, we can use the result of the inner substitution:
$ echo "$(( ${var//[![:digit:]]} + 1 ))"
1
and finally, putting this between bar and baz, you get bar1baz.
Amending for the other case brought up: assuming there might be other digits and we want to increment only the trailing ones, e.g.,
var=2a-foo-21
To do this, we can use nested parameter expansion with extended globs (shopt -s extglob) and the +(pattern) pattern, which matches one or more of pattern. Observe:
$ echo "${var#"${var%%+([[:digit:]])}"}"
21
The outer expansion is ${var#pattern}, which removes the shortest match of pattern from the beginning of $var. For pattern, we use
"${var%%+([[:digit:]])}"
which is "remove the longest match of +([[:digit:]]) (one or more digits) from the end of $var". This leaves us with just the trailing digits, and incrementing them and adding string before and after looks something like this:
$ echo "bar$((${var#"${var%%+([[:digit:]])}"}+1))baz"
bar22baz
This is so unreadable that I'd suggest using regex instead:
$ re='([[:digit:]]+)$'
$ [[ $var =~ $re ]]
$ echo "bar$((${BASH_REMATCH[1]}+1))baz"
bar22baz

how to cut the pattern from a multiple line string and store to the new variable?

I have a string variable
$var="-- this is a --
-- comment --
hellow
world"
i need to cut the string string into two parts such that there are two new string variable
$var1="-- this is a --
-- comment --"
$var2="hellow
world"
i need the new line as it is in the $var1 $var2
what i have tried
var2=$(echo $var | sed 's/^--.*--//g')
i am not able to store the pattern in $var1 that it is deleting .
Note : hyphen is important and new line is important to maintain .
have tried bash string manipulation but it doesn't worked
Simply use grep
For var1
grep '\-\-.*\-\-'
For var2
grep -v '\-\-.*\-\-'
As suggest by #123, we don't need escaping, if we use -- in grep command.
So you can use this as well
For var1
grep -- '--.*--'
For var2
grep -v '--.*--'
Your $var as presented in your question seems to contain new lines,making your $var to consist of actually 4 different lines. In that case you can split the lines like:
$ var1=$(sed -n '1,2p' <<<"$var")
$ var2=$(sed -n '3,4p' <<<"$var")
$var1 gets the lines 1 and 2 of $var (or from a file if you like)
$var2 gets the lines 3 and 4 of $var (or from a file if you like)

Adding comma between command line parameter

I want to call a procedure through unix script, it will be generic script so parameters can very. Calling statement will be some something like
<scriptname> <procedure name> <param1> <param2> <param3> <param4>.. so on
What I need is from 2nd command line paramater to last paramter I want all values comma separeted something like this
<param1>,<param2>,<param3>,<param4>
I can do this using a loop, that is from 2nd command line parameter I will itereate each parameter and add comma in it. My question is can I do this with single command?
Note :- Space should be handled properly if present is command line parameter, after last parameter there should not be any comma
"${*:2}" expands to the list of arguments starting at $2, separated by the first character of IFS:
saveIFS=$IFS
IFS=","
args="${*:2}"
IFS=$saveIFS
echo "$args"
Note that this properly preserves spaces within arguments, rather than converting them to commas.
All parameters is $#. You can use sed to replace spaces with commas and then(or from the begining, cut the first field)
echo $# | sed s/" "/,/g | cut -d "," -f2-
a step forward, you can assign it to a variable:
comma_separated_params=`echo $# | sed s/" "/,/g | cut -d "," -f2-`
This technique below, performing the echo in a subshell, allows you to set IFS and then let the changes disappear with the subshell
$ set -- "a b c" "d e f" "g h i"
$ with_comma=$(IFS=,; echo "$*")
$ echo "$with_comma"
a b c,d e f,g h i

File content into unix variable with newlines

I have a text file test.txt with the following content:
text1
text2
And I want to assign the content of the file to a UNIX variable, but when I do this:
testvar=$(cat test.txt)
echo $testvar
the result is:
text1 text2
instead of
text1
text2
Can someone suggest me a solution for this?
The assignment does not remove the newline characters, it's actually the echo doing this. You need simply put quotes around the string to maintain those newlines:
echo "$testvar"
This will give the result you want. See the following transcript for a demo:
pax> cat num1.txt ; x=$(cat num1.txt)
line 1
line 2
pax> echo $x ; echo '===' ; echo "$x"
line 1 line 2
===
line 1
line 2
The reason why newlines are replaced with spaces is not entirely to do with the echo command, rather it's a combination of things.
When given a command line, bash splits it into words according to the documentation for the IFS variable:
IFS: The Internal Field Separator that is used for word splitting after expansion ... the default value is <space><tab><newline>.
That specifies that, by default, any of those three characters can be used to split your command into individual words. After that, the word separators are gone, all you have left is a list of words.
Combine that with the echo documentation (a bash internal command), and you'll see why the spaces are output:
echo [-neE] [arg ...]: Output the args, separated by spaces, followed by a newline.
When you use echo "$x", it forces the entire x variable to be a single word according to bash, hence it's not split. You can see that with:
pax> function count {
...> echo $#
...> }
pax> count 1 2 3
3
pax> count a b c d
4
pax> count $x
4
pax> count "$x"
1
Here, the count function simply prints out the number of arguments given. The 1 2 3 and a b c d variants show it in action.
Then we try it with the two variations on the x variable. The one without quotes shows that there are four words, "test", "1", "test" and "2". Adding the quotes makes it one single word "test 1\ntest 2".
This is due to IFS (Internal Field Separator) variable which contains newline.
$ cat xx1
1
2
$ A=`cat xx1`
$ echo $A
1 2
$ echo "|$IFS|"
|
|
A workaround is to reset IFS to not contain the newline, temporarily:
$ IFSBAK=$IFS
$ IFS=" "
$ A=`cat xx1` # Can use $() as well
$ echo $A
1
2
$ IFS=$IFSBAK
To REVERT this horrible change for IFS:
IFS=$IFSBAK
Bash -ge 4 has the mapfile builtin to read lines from the standard input into an array variable.
help mapfile
mapfile < file.txt lines
printf "%s" "${lines[#]}"
mapfile -t < file.txt lines # strip trailing newlines
printf "%s\n" "${lines[#]}"
See also:
http://bash-hackers.org/wiki/doku.php/commands/builtin/mapfile
Your variable is set correctly by testvar=$(cat test.txt). To display this variable which consist new line characters, simply add double quotes, e.g.
echo "$testvar"
Here is the full example:
$ printf "test1\ntest2" > test.txt
$ testvar=$(<test.txt)
$ grep testvar <(set)
testvar=$'test1\ntest2'
$ echo "$testvar"
text1
text2
$ printf "%b" "$testvar"
text1
text2
Just if someone is interested in another option:
content=( $(cat test.txt) )
a=0
while [ $a -le ${#content[#]} ]
do
echo ${content[$a]}
a=$[a+1]
done
The envdir utility provides an easy way to do this. envdir uses files to represent environment variables, with file names mapping to env var names, and file contents mapping to env var values. If the file contents contain newlines, so will the env var.
See https://pypi.python.org/pypi/envdir

Resources