I can use herestrings to pass a string to a command, e.g.
cat <<< "This is a string"
How can I use herestrings to pass two strings to a command? How can I do something like
### not working
diff <<< "string1" "string2"
### working but overkill
echo "string1" > file1
echo "string2" > file2
diff file1 file2
You can't use two herestrings as input to the same command. In effect, the latest one will replace all others. Demonstration:
cat <<< "string 1" <<< "string 2" <<< "string 3"
# only shows "string 3"
On the other hand, if what you want is really diff two immediate inputs, you can do it this way:
diff <(echo "string 1") <(echo "string 2")
You can simply concatenate the two strings:
cat <<< "string1""string2"
(not the lack of space between the two). The here string now consists of a single word whose contents are the contents of the two strings.
Related
My intent is to write a shell script to extract a pattern ,using regular expressions, from a file and fill an array with all the ocurrences of the pattern in order to foreach it.
What is the best way to achieve this?
I am trying to do it using sed. And a problem I am facing is that the patterns can have newlines and these newlines must be considered, eg:
File content:
"My name
is XXX"
"My name is YYY"
"Today
is
the "
When I extract all patterns between double quotes, including the double quotes, the output of the first ocurrence must be:
"My name
is XXX"
fill an array with all the ocurrences of the pattern
First convert your file to have meaningful delimiter, ex. null byte, with ex. GNU sed with -z switch:
sed -z 's/"\([^"]*\)"[^"]*/\1\00/g'
I've added the [^"]* on the end, so that characters not between " are removed.
After it it becomes more trivial to parse it.
You can get the first element with:
head -z -n1
Or sort and count the occurrences:
sort -z | uniq -z -c
Or load to an array with bash's maparray:
maparray -d '' -t arr < <(<input sed -z 's/"\([^"]*\)"[^"]*/\1\00/'g))
Alternatively you can use ex. $'\01' as the separator, as long as it's unique, it becomes simple to parse such data in bash.
Handling such streams is a bit hard in bash. You can't set variable value in shell with embedded null byte. Also expect sometimes warnings on command substitutions. Usually when handling data with arbitrary bytes, I convert it with xxd -p to plain ascii and back with xxd -r -p. With that, it becomes easier.
The following script:
cat <<'EOF' >input
"My name
is XXX"
"My name is YYY"
"Today
is
the "
EOF
sed -z 's/"\([^"]*\)"[^"]*/\1\x00/g' input > input_parsed
echo "##First element is:"
printf '"'
<input_parsed head -z -n1
printf '"\n'
echo "##Elemets count are:"
<input_parsed sort -z | uniq -z -c
echo
echo "##The array is:"
mapfile -d '' -t arr <input_parsed
declare -p arr
will output (the formatting is a bit off, because of the non-newline delimetered output from uniq):
##First element is:
"My name
is XXX"
##Elemets count are:
1 My name
is XXX 1 My name is YYY 1 Today
is
the
##The array is:
declare -a arr=([0]=$'My name\nis XXX' [1]="My name is YYY" [2]=$'Today\nis\nthe ')
Tested on repl.it.
This may be what you're looking for, depending on the answers to the questions I posted in a comment:
$ readarray -d '' -t arr < <(grep -zo '"[^"]*"' file)
$ printf '%s\n' "${arr[0]}"
"My name
is XXX"
$ declare -p arr
declare -a arr=([0]=$'"My name \nis XXX"' [1]="\"My name is YYY\"" [2]=$'"Today\nis\nthe "')
It uses GNU grep for -z.
Sed can extract your desired pattern with or without newlines.
But if you want to store the multiple results into a bash array,
it may be easier to make use of bash regex.
Then please try the following:
lines=$(< "file") # slurp all lines
re='"[^"]+"' # regex to match substring between double quotes
while [[ $lines =~ ($re)(.*) ]]; do
array+=("${BASH_REMATCH[1]}") # push the matched pattern to the array
lines=${BASH_REMATCH[2]} # update $lines with the remaining part
done
# report the result
for (( i=0; i<${#array[#]}; i++ )); do
echo "$i: ${array[$i]}"
done
Output:
0: "My name
is XXX"
1: "My name is YYY"
2: "Today
is
the "
This is how my input looks like:
string="a 1,a 2,a 3"
This is how I generate list out of the input:
sed -e 's/[^,]*/"&"/g' <<< ${string}
Above command gives me the desired output as:
"a 1","a 2","a 3"
How do I trim each element so that if the input is " a 1, a 2, a 3", my output still comes back as "a 1","a 2","a 3"?
I think it is important to understand that in bash, the double quotes have a special meaning.
string="a 1,a 2,a 3" represents the string a 1,a 2,a 3 (no quotes)
sed -e 's/[^,]*/"&"/g' <<< ${string} is equivalent to the variable out='"a 1","a 2","a 3"'
To accomplish what you want, you can do:
$ string=" a 1, a 2, a 3 "
$ echo "\"$(echo ${string//*( ),*( )/\",\"})\""
"a 1","a 2","a 3"
This is only using bash builtin operations.
replace all combinations of multiple spaces and commas by the quoted comma ${string//*( ),*( )/\",\"}
use word splitting to remove all leading and trailing blanks $(echo ...) (note: this is a bit ugly and will fail on cases like a 1 , a 2 as it will remove the double space between a and 1)
print two extra double-quotes at the beginning and end of the string.
A better way is to use a double substitution:
$ string=" a 1, a 2, a 3 "
$ foobar="\"${string//,/\",\"}\""
$ echo "${foobar//*( )\"*( )/\"}"
"a 1","a 2","a 3"
note: here we make use of KSH-globs which can be enabled with the extglob setting (shopt -s extglob)
Here an answer which extends your sed command with some basic preprocessing that removes unwanted spaces:
sed -E -e 's/ *(^|,) */\1/g;s/[^,]*/"&"/g' <<< ${string}
The -E option enables extended regular expression which saves some \.
EDIT: Since OP told to wrap output in " so adding it now.
echo "$string" | sed -E 's/^ +/"/;s/$/"/;s/, +/"\,"/g'
Output will be as follows.
echo "$string" | sed -E 's/^ +/"/;s/$/"/;s/, +/"\,"/g'
"a 1","a 2","a 3"
Could you please try following and let me know if this helps you.
awk '{sub(/^\" +/,"\"");gsub(/, +/,"\",\"")} 1' Input_file
In case you want to save output into same Input_file itself append > temp_file && mv temp_file Input_file.
Solution 2nd: Using sed.
sed -E 's/^" +/"/;s/, +/"\,"/g' Input_file
Instead of complex sed pattern, you can use grep -ow option.
[nooka#lori ~]$ string1="a 1,a 2,a 3"
[nooka#lori ~]$ string2=" a 1, a 2, a 3, a 4"
[nooka#lori ~]$
nooka#lori ~]$ echo $(echo $string1 | grep -ow "[a-zA-Z] [0-9]"|sed "s/^/\"/;s/$/\"/")|sed "s/\" /\",/g"
"a 1","a 2","a 3"
[nooka#lori ~]$ echo $(echo $string2 | grep -ow "[a-zA-Z] [0-9]"|sed "s/^/\"/;s/$/\"/")|sed "s/\" /\",/g"
"a 1","a 2","a 3","a 4"
1) use grep -ow to get only those words as per the pattern defined above. You can tweak the pattern per your needs (for ex: [a-zA-Z] [0-9][0-9]* etc) for more patterns cases.
2) Then you wrap the output (a 1 or a 2 etc) with a " using the first sed cmd.
3) Then you just put , between 2 " and you get what you wanted. This assumes you pattern always follows a single space between string and number value.
Ok. Finally got it working.
string=" a 1 b 2 c 3 , something else , yet another one with spaces , totally working "
trimmed_string=$(awk '{gsub(/[[:space:]]*,[[:space:]]*/,",")}1' <<< $string)
echo ${trimmed_string}
a 1 b 2 c 3,something else,yet another one with spaces,totally working
string_as_list=$(sed -e 's/[^,]*/"&"/g' <<< ${trimmed_string})
echo ${string_as_list}
"a 1 b 2 c 3","something else","yet another one with spaces","totally working"
This whole thing had to be done because terraform expects list
variables to be passed like that. They must be surrounded by double quotes (" "),
delimited by comma( , ) inside square brackets([ ]).
for a script
script.sh < a b c a
I want to put the inputs in one string, like that
string=" a b c a"
ie, preserve the spaces
how can i do that?
thanks
Consider using here documents and here strings.
script.sh:
#!/bin/bash
readarray -t INPUT
printf "input: %s\n" "${INPUT[#]}"
example:
bash script.sh <<< "a b c"
Redirection to a file is very usefull to append a string as a new line to a file, like
echo "foo" >> file.txt
echo "bar" >> file.txt
Result:
foo
bar
But is it also possible to redirect a string to the same line in the file ?
Example:
echo "foo" <redirection-command-for-same-line> file.txt
echo "bar" <redirection-command-for-same-line> file.txt
Result:
foobar
The newline is added by echo, not by the redirection. Just pass the -n switch to echo to suppress it:
echo -n "foo" >> file.txt
echo -n "bar" >> file.txt
-n do not output the trailing newline
An alternate way to echo results to one line would be to simply assign the results to variables. Example:
j=$(echo foo)
i=$(echo bar)
echo $j$i
foobar
echo $i $j
bar foo
This is particularly useful when you have more complex functions, maybe a complex 'awk' statement to pull out a particular cell in a row, then pair it with another set.
I have a text file test.txt with the following content:
text1
text2
And I want to assign the content of the file to a UNIX variable, but when I do this:
testvar=$(cat test.txt)
echo $testvar
the result is:
text1 text2
instead of
text1
text2
Can someone suggest me a solution for this?
The assignment does not remove the newline characters, it's actually the echo doing this. You need simply put quotes around the string to maintain those newlines:
echo "$testvar"
This will give the result you want. See the following transcript for a demo:
pax> cat num1.txt ; x=$(cat num1.txt)
line 1
line 2
pax> echo $x ; echo '===' ; echo "$x"
line 1 line 2
===
line 1
line 2
The reason why newlines are replaced with spaces is not entirely to do with the echo command, rather it's a combination of things.
When given a command line, bash splits it into words according to the documentation for the IFS variable:
IFS: The Internal Field Separator that is used for word splitting after expansion ... the default value is <space><tab><newline>.
That specifies that, by default, any of those three characters can be used to split your command into individual words. After that, the word separators are gone, all you have left is a list of words.
Combine that with the echo documentation (a bash internal command), and you'll see why the spaces are output:
echo [-neE] [arg ...]: Output the args, separated by spaces, followed by a newline.
When you use echo "$x", it forces the entire x variable to be a single word according to bash, hence it's not split. You can see that with:
pax> function count {
...> echo $#
...> }
pax> count 1 2 3
3
pax> count a b c d
4
pax> count $x
4
pax> count "$x"
1
Here, the count function simply prints out the number of arguments given. The 1 2 3 and a b c d variants show it in action.
Then we try it with the two variations on the x variable. The one without quotes shows that there are four words, "test", "1", "test" and "2". Adding the quotes makes it one single word "test 1\ntest 2".
This is due to IFS (Internal Field Separator) variable which contains newline.
$ cat xx1
1
2
$ A=`cat xx1`
$ echo $A
1 2
$ echo "|$IFS|"
|
|
A workaround is to reset IFS to not contain the newline, temporarily:
$ IFSBAK=$IFS
$ IFS=" "
$ A=`cat xx1` # Can use $() as well
$ echo $A
1
2
$ IFS=$IFSBAK
To REVERT this horrible change for IFS:
IFS=$IFSBAK
Bash -ge 4 has the mapfile builtin to read lines from the standard input into an array variable.
help mapfile
mapfile < file.txt lines
printf "%s" "${lines[#]}"
mapfile -t < file.txt lines # strip trailing newlines
printf "%s\n" "${lines[#]}"
See also:
http://bash-hackers.org/wiki/doku.php/commands/builtin/mapfile
Your variable is set correctly by testvar=$(cat test.txt). To display this variable which consist new line characters, simply add double quotes, e.g.
echo "$testvar"
Here is the full example:
$ printf "test1\ntest2" > test.txt
$ testvar=$(<test.txt)
$ grep testvar <(set)
testvar=$'test1\ntest2'
$ echo "$testvar"
text1
text2
$ printf "%b" "$testvar"
text1
text2
Just if someone is interested in another option:
content=( $(cat test.txt) )
a=0
while [ $a -le ${#content[#]} ]
do
echo ${content[$a]}
a=$[a+1]
done
The envdir utility provides an easy way to do this. envdir uses files to represent environment variables, with file names mapping to env var names, and file contents mapping to env var values. If the file contents contain newlines, so will the env var.
See https://pypi.python.org/pypi/envdir