How to only do quote removal in a shell? - bash

Given a string from an untrusted source, e.g.
MALICIOUS_INPUT="$(awk -F = '/^VERSION=/ {print $2}' /path/to/compromised/etc/os-release | head -n 1)"
is it possible to just apply pure shell quote removal (see Shell Command Language (IEEE Std 1003.1-2017) and Bash manual) to that string i.e. without doing variable expansions, arithmetic expansions, command substitution and similar?
This is needed, for example to parse strings from os-release files without source-ing the files.
Input
Expected result
'\"'
\"
"\""
"
'$foo${foo}$(pwd)$((1+2))'
$foo${foo}$(pwd)$((1+2))
"$foo${foo}$(pwd)$((1+2))"
$foo${foo}$(pwd)$((1+2))

Comparing applicability of the preexisting answers on Reading quoted/escaped arguments correctly from a string to this question:
parse_with_xargs() {
xargs printf '%s\0' <<<"$*"
}
parse_with_python() {
python -c '
import shlex, sys
for item in shlex.split(sys.stdin.read()):
sys.stdout.write(item + "\0")
' <<<"$*"
}
readarray -t example_lines <<'EOF'
'\"'
"\""
'$foo${foo}$(pwd)$((1+2))'
"$foo${foo}$(pwd)$((1+2))"
EOF
for line in "${example_lines[#]}"; do
printf 'Input line: %s\n' "$line"
printf 'Parsed with xargs: '; parse_with_xargs "$line" 2>&1; echo
printf 'Parsed with python: '; parse_with_python "$line" 2>&1; echo
echo
done
Output:
Input line: '\"'
Parsed with xargs: \"
Parsed with python: \"
Input line: "\""
Parsed with xargs: xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
Parsed with python: "
Input line: '$foo${foo}$(pwd)$((1+2))'
Parsed with xargs: $foo${foo}$(pwd)$((1+2))
Parsed with python: $foo${foo}$(pwd)$((1+2))
Input line: "$foo${foo}$(pwd)$((1+2))"
Parsed with xargs: $foo${foo}$(pwd)$((1+2))
Parsed with python: $foo${foo}$(pwd)$((1+2))

Related

printf command within Bash throwing error

I am using the following command in bash to subtract 2 numbers and print the result. Using bc tool as well
printf '\n"runtime": %s' "$(($a - $b) | bc -l)"
But getting an error
1517359690.775414500: command not found
How should i rewrite my printf command?
If your shell is bash, then this could be:
printf '\n"runtime": %s' "$(bc -l <<<"($a - $b)")"
If instead your shell is sh, then this could be:
printf '\n"runtime": %s' "$(echo "($a - $b)" | bc -l)"
Note that we're invoking a separate command -- echo -- whose output is piped into bc, rather than trying to run the numbers as a command themselves.
However, you shouldn't be using printf to create JSON documents in the first place.
Instead, use jq:
start=5.5; stop=6.10
other_value='this is an example string
it crosses multiple lines, and has "literal quotes" within it'
jq -nc \
--argjson start "$start" \
--argjson stop "$stop" \
--arg other_value "$other_value" \
'{"runtime": ($stop - $start), "other key": $other_value}'
You'll note that the string here is correctly escaped to be included in JSON: " is changed to \", the literal newline is changed to \n, and so forth.

I want to re-arrange a file in an order in shell

I have a file test.txt like below spaces in between each record
service[1.1],parttion, service[1.2],parttion, service[1.3],parttion, service[2.1],parttion, service2[2.2],parttion,
Now I want to rearrange it as below into a output.txt
COMPOSITES=parttion/service/1.1,parttion/service/1.2,parttion/service/1.3,parttion/service/2.1,parttion/service/2.2
I've tried:
final_str=''
COMPOSITES=''
# Re-arranging the composites and preparing the composite property file
while read line; do
partition_val="$(echo $line | cut -d ',' -f 2)"
composite_temp1_val="$(echo $line | cut -d ',' -f 1)"
composite_val="$(echo $composite_temp1_val | cut -d '[' -f 1)"
version_temp1_val="$(echo $composite_temp1_val | cut -d '[' -f 2)"
version_val="$(echo $version_temp1_val | cut -d ']' -f 1)"
final_str="$partition_val/$composite_val/$version_val,"
COMPOSITES=$COMPOSITES$final_str
done <./temp/test.txt
We start with the file:
$ cat test.txt
service[1.1],parttion, service[1.2],parttion, service[1.3],parttion, service[2.1],parttion, service2[2.2],parttion,
We can rearrange that file as follows:
$ awk -F, -v RS=" " 'BEGIN{printf "COMPOSITES=";} {gsub(/[[]/, "/"); gsub(/[]]/, ""); if (NF>1) printf "%s%s/%s",NR==1?"":",",$2,$1;}' test.txt
COMPOSITES=parttion/service/1.1,parttion/service/1.2,parttion/service/1.3,parttion/service/2.1,parttion/service2/2.2
The same command split over multiple lines is:
awk -F, -v RS=" " '
BEGIN{
printf "COMPOSITES=";
}
{
gsub(/[[]/, "/")
gsub(/[]]/, "")
if (NF>1) printf "%s%s/%s",NR==1?"":",",$2,$1
}
' test.txt
Here's what I came up with.
awk -F '[],[]' -v RS=" " 'BEGIN{printf("COMPOSITES=")}/../{printf("%s/%s/%s,",$4,$1,$2);}' test.txt
Broken out for easier reading:
awk -F '[],[]' -v RS=" " '
BEGIN {
printf("COMPOSITES=");
}
/../ {
printf("%s/%s/%s,",$4,$1,$2);
}' test.txt
More detailed explanation of the script:
-F '[],[]' - use commas or square brackets as field separators
-v RS=" " - use just the space as a record separator
'BEGIN{printf("COMPOSITES=")} - starts your line
/../ - run the following code on any line that has at least two characters. This avoids the empty field at the end of a line terminating with a space.
printf("%s/%s/%s,",$4,$1,$2); - print the elements using a printf() format string that matches the output you specified.
As concise as this is, the format string does leave a trailing comma at the end of the line. If this is a problem, it can be avoided with a bit of extra code.
You could also do this in sed, if you like writing code in line noise.
sed -e 's:\([^[]*\).\([^]]*\).,\([^,]*\), :\3/\1/\2,:g;s/^/COMPOSITES=/;s/,$//' test.txt
Finally, if you want to avoid external tools like sed and awk, you can do this in bash alone:
a=($(<test.txt))
echo -n "COMPOSITES="
for i in "${a[#]}"; do
i="${i%,}"
t="${i%]*}"
printf "%s/%s/%s," "${i#*,}" "${i%[*}" "${t#*[}"
done
echo ""
This slurps the contents of test.txt into an array, which means your input data must be separated by whitespace, per your example. It then adds the prefix, then steps through the array, using Parameter Expansion to massage the data into the fields you need. The last line (echo "") is helpful for testing; you may want to eliminate it in practice.

How can I print a newline as \n in Bash?

Basically, I want to achieve something like the inverse of echo -e.
I have a variable which stores a command output, but I want to print newlines as \n.
Here's my solution:
sed 's/$/\\n/' | tr -d '\n'
If your input is already in a (Bash) shell variable, say $varWithNewlines:
echo "${varWithNewlines//$'\n'/\\n}"
It simply uses Bash parameter expansion to replace all newline ($'\n') instances with literal '\n' each.
If your input comes from a file, use AWK:
awk -v ORS='\\n' 1
In action, with sample input:
# Sample input with actual newlines created with ANSI C quoting ($'...'),
# which turns `\n` literals into actual newlines.
varWithNewlines=$'line 1\nline 2\nline 3'
# Translate newlines to '\n' literals.
# Note the use of `printf %s` to avoid adding an additional newline.
# By contrast, a here-string - <<<"$varWithNewlines" _always appends a newline_.
printf %s "$varWithNewlines" | awk -v ORS='\\n' 1
awk reads input line by line
by setting ORS- the output record separator to literal '\n' (escaped with an additional \ so that awk doesn't interpret it as an escape sequence), the input lines are output with that separator
1 is just shorthand for {print}, i.e., all input lines are printed, terminated by ORS.
Note: The output will always end in literal '\n', even if your input does not end in a newline.
This is because AWK terminates every output line with ORS, whether the input line ended with a newline (separator specified in FS) or not.
Here's how to unconditionally strip the terminating literal '\n' from your output.
# Translate newlines to '\n' literals and capture in variable.
varEncoded=$(printf %s "$varWithNewlines" | awk -v ORS='\\n' 1)
# Strip terminating '\n' literal from the variable value
# using Bash parameter expansion.
echo "${varEncoded%\\n}"
By contrast, more work is needed if you want to make the presence of a terminating literal '\n' dependent on whether the input ends with a newline or not.
# Translate newlines to '\n' literals and capture in variable.
varEncoded=$(printf %s "$varWithNewlines" | awk -v ORS='\\n' 1)
# If the input does not end with a newline, strip the terminating '\n' literal.
if [[ $varWithNewlines != *$'\n' ]]; then
# Strip terminating '\n' literal from the variable value
# using Bash parameter expansion.
echo "${varEncoded%\\n}"
else
echo "$varEncoded"
fi
You can use printf "%q":
eol=$'\n'
printf "%q\n" "$eol"
$'\n'
A Bash solution
x=$'abcd\ne fg\nghi'
printf "%s\n" "$x"
abcd
e fg
ghi
y=$(IFS=$'\n'; set -f; printf '%s\\n' $x)
y=${y%??}
printf "%s\n" "$y"
abcd\ne fg\nghi

How to get output of grep in single line in shell script?

Here is a script which reads words from the file replaced.txt and displays the output each word in each line, But I want to display all the outputs in a single line.
#!/bin/sh
echo
echo "Enter the word to be translated"
read a
IFS=" " # Set the field separator
set $a # Breaks the string into $1, $2, ...
for a # a for loop by default loop through $1, $2, ...
do
{
b= grep "$a" replaced.txt | cut -f 2 -d" "
}
done
Content of "replaced.txt" file is given below:
hllo HELLO
m AM
rshbh RISHABH
jn JAIN
hw HOW
ws WAS
ur YOUR
dy DAY
This question can't be appropriate to what I asked, I just need the help to put output of the script in a single line.
Your entire script can be replaced by:
#!/bin/bash
echo
read -r -p "Enter the words to be translated: " a
echo $(printf "%s\n" $a | grep -Ff - replaced.txt | cut -f 2 -d ' ')
No need for a loop.
The echo with an unquoted argument removes embedded newlines and replaces each sequence of multiple spaces and/or tabs with one space.
One hackish-but-simple way to remove trailing newlines from the output of a command is to wrap it in printf %s "$(...) ". That is, you can change this:
b= grep "$a" replaced.txt | cut -f 2 -d" "
to this:
printf %s "$(grep "$a" replaced.txt | cut -f 2 -d" ") "
and add an echo command after the loop completes.
The $(...) notation sets up a "command substitution": the command grep "$a" replaced.txt | cut -f 2 -d" " is run in a subshell, and its output, minus any trailing newlines, is substituted into the argument-list. So, for example, if the command outputs DAY, then the above is equivalent to this:
printf %s "DAY "
(The printf %s ... notation is equivalent to echo -n ... — it outputs a string without adding a trailing newline — except that its behavior is more portably consistent, and it won't misbehave if the string you want to print happens to start with -n or -e or whatnot.)
You can also use
awk 'BEGIN { OFS=": "; ORS=" "; } NF >= 2 { print $2; }'
in a pipe after the cut.

Printf example in bash does not create a newline

Working with printf in a bash script, adding no spaces after "\n" does not create a newline, whereas adding a space creates a newline, e. g.:
No space after "\n"
NewLine=`printf "\n"`
echo -e "Firstline${NewLine}Lastline"
Result:
FirstlineLastline
Space after "\n "
NewLine=`printf "\n "`
echo -e "Firstline${NewLine}Lastline"
Result:
Firstline
Lastline
Question: Why doesn't 1. create the following result:
Firstline
Lastline
I know that this specific issue could have been worked around using other techniques, but I want to focus on why 1. does not work.
Edited:
When using echo instead of printf, I get the expected result, but why does printf work differently?
NewLine=`echo "\n"`
echo -e "Firstline${NewLine}Lastline"
Result:
Firstline
Lastline
The backtick operator removes trailing new lines. See 3.4.5. Command substitution at http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html
Note on edited question
Compare:
[alvaro#localhost ~]$ printf "\n"
[alvaro#localhost ~]$ echo "\n"
\n
[alvaro#localhost ~]$ echo -e "\n"
[alvaro#localhost ~]$
The echo command doesn't treat \n as a newline unless you tell him to do so:
NAME
echo - display a line of text
[...]
-e enable interpretation of backslash escapes
POSIX 7 specifies this behaviour here:
[...] with the standard output of the command, removing sequences of one or more characters at the end of the substitution
Maybe people will come here with the same problem I had:
echoing \n inside a code wrapped in backsticks. A little tip:
printf "astring\n"
# and
printf "%s\n" "astring"
# both have the same effect.
# So... I prefer the less typing one
The short answer is:
# Escape \n correctly !
# Using just: printf "$myvar\n" causes this effect inside the backsticks:
printf "banana
"
# So... you must try \\n that will give you the desired
printf "banana\n"
# Or even \\\\n if this string is being send to another place
# before echoing,
buffer="${buffer}\\\\n printf \"$othervar\\\\n\""
One common problem is that if you do inside the code:
echo 'Tomato is nice'
when surrounded with backsticks will produce the error
command Tomato not found.
The workaround is to add another echo -e or printf
printed=0
function mecho(){
#First time you need an "echo" in order bash relaxes.
if [[ $printed == 0 ]]; then
printf "echo -e $1\\\\n"
printed=1
else
echo -e "\r\n\r$1\\\\n"
fi
}
Now you can debug your code doing in prompt just:
(prompt)$ `mySuperFunction "arg1" "etc"`
The output will be nicely
mydebug: a value
otherdebug: whathever appended using myecho
a third string
and debuging internally with
mecho "a string to be hacktyped"
$ printf -v NewLine "\n"
$ echo -e "Firstline${NewLine}Lastline"
Firstline
Lastline
$ echo "Firstline${NewLine}Lastline"
Firstline
Lastline
It looks like BASH is removing trailing newlines.
e.g.
NewLine=`printf " \n\n\n"`
echo -e "Firstline${NewLine}Lastline"
Firstline Lastline
NewLine=`printf " \n\n\n "`
echo -e "Firstline${NewLine}Lastline"
Firstline
Lastline
Your edited echo version is putting a literal backslash-n into the variable $NewLine which then gets interpreted by your echo -e. If you did this instead:
NewLine=$(echo -e "\n")
echo -e "Firstline${NewLine}Lastline"
your result would be the same as in case #1. To make that one work that way, you'd have to escape the backslash and put the whole thing in single quotes:
NewLine=$(printf '\\n')
echo -e "Firstline${NewLine}Lastline"
or double escape it:
NewLine=$(printf "\\\n")
Of course, you could just use printf directly or you can set your NewLine value like this:
printf "Firstline\nLastline\n"
or
NewLine=$'\n'
echo "Firstline${NewLine}Lastline" # no need for -e
For people coming here wondering how to use newlines in arguments to printf, use %b instead of %s:
$> printf "a%sa" "\n"
a\na
$> printf "a%ba" "\n"
a
a
From the manual:
%b expand backslash escape sequences in the corresponding argument
We do not need "echo" or "printf" for creating the NewLine variable:
NewLine="
"
printf "%q\n" "${NewLine}"
echo "Firstline${NewLine}Lastline"
Bash delete all trailing newlines in commands substitution.
To save trailing newlines, assign printf output to the variable with printf -v VAR
instead of
NewLine=`printf "\n"`
echo -e "Firstline${NewLine}Lastline"
#FirstlineLastline
use
printf -v NewLine '\n'
echo -e "Firstline${NewLine}Lastline"
#Firstline
#Lastline
Explanation
According to bash man
3.5.4 Command Substitution
$(command)
or
`command`
Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
So, after adding any trailing newlines, bash will delete them.
var=$(printf '%s\n%s\n\n\n' 'foo' 'bar')
echo "$var"
output:
foo
bar
According to help printf
printf [-v var] format [arguments]
If the -v option is supplied, the output is placed into the value of the shell variable VAR rather than being sent to the standard output.
In this case, for safe copying of formatted text to the variable, use the [-v var] option:
printf -v var '%s\n%s\n\n\n' 'foo' 'bar'
echo "$var"
output:
foo
bar
Works ok if you add "\r"
$ nl=`printf "\n\r"` && echo "1${nl}2"
1
2

Resources