Related
Below is the snippet of a shell script from a larger script. It removes the quotes from the string that is held by a variable. I am doing it using sed, but is it efficient? If not, then what is the efficient way?
#!/bin/sh
opt="\"html\\test\\\""
temp=`echo $opt | sed 's/.\(.*\)/\1/' | sed 's/\(.*\)./\1/'`
echo $temp
Use tr to delete ":
echo "$opt" | tr -d '"'
NOTE: This does not fully answer the question, removes all double quotes, not just leading and trailing. See other answers below.
There's a simpler and more efficient way, using the native shell prefix/suffix removal feature:
temp="${opt%\"}"
temp="${temp#\"}"
echo "$temp"
${opt%\"} will remove the suffix " (escaped with a backslash to prevent shell interpretation).
${temp#\"} will remove the prefix " (escaped with a backslash to prevent shell interpretation).
Another advantage is that it will remove surrounding quotes only if there are surrounding quotes.
BTW, your solution always removes the first and last character, whatever they may be (of course, I'm sure you know your data, but it's always better to be sure of what you're removing).
Using sed:
echo "$opt" | sed -e 's/^"//' -e 's/"$//'
(Improved version, as indicated by jfgagne, getting rid of echo)
sed -e 's/^"//' -e 's/"$//' <<<"$opt"
So it replaces a leading " with nothing, and a trailing " with nothing too. In the same invocation (there isn't any need to pipe and start another sed. Using -e you can have multiple text processing).
If you're using jq and trying to remove the quotes from the result, the other answers will work, but there's a better way. By using the -r option, you can output the result with no quotes.
$ echo '{"foo": "bar"}' | jq '.foo'
"bar"
$ echo '{"foo": "bar"}' | jq -r '.foo'
bar
There is a straightforward way using xargs:
> echo '"quoted"' | xargs
quoted
xargs uses echo as the default command if no command is provided and strips quotes from the input, see e.g. here. Note, however, that this will work only if the string does not contain additional quotes. In that case it will either fail (uneven number of quotes) or remove all of them.
If you came here for aws cli --query, try this. --output text
You can do it with only one call to sed:
$ echo "\"html\\test\\\"" | sed 's/^"\(.*\)"$/\1/'
html\test\
The shortest way around - try:
echo $opt | sed "s/\"//g"
It actually removes all "s (double quotes) from opt (are there really going to be any more double quotes other than in the beginning and the end though? So it's actually the same thing, and much more brief ;-))
The easiest solution in Bash:
$ s='"abc"'
$ echo $s
"abc"
$ echo "${s:1:-1}"
abc
This is called substring expansion (see Gnu Bash Manual and search for ${parameter:offset:length}). In this example it takes the substring from s starting at position 1 and ending at the second last position. This is due to the fact that if length is a negative value it is interpreted as a backwards running offset from the end of parameter.
Update
A simple and elegant answer from Stripping single and double quotes in a string using bash / standard Linux commands only:
BAR=$(eval echo $BAR) strips quotes from BAR.
=============================================================
Based on hueybois's answer, I came up with this function after much trial and error:
function stripStartAndEndQuotes {
cmd="temp=\${$1%\\\"}"
eval echo $cmd
temp="${temp#\"}"
eval echo "$1=$temp"
}
If you don't want anything printed out, you can pipe the evals to /dev/null 2>&1.
Usage:
$ BAR="FOO BAR"
$ echo BAR
"FOO BAR"
$ stripStartAndEndQuotes "BAR"
$ echo BAR
FOO BAR
This is the most discrete way without using sed:
x='"fish"'
printf " quotes: %s\nno quotes: %s\n" "$x" "${x//\"/}"
Or
echo $x
echo ${x//\"/}
Output:
quotes: "fish"
no quotes: fish
I got this from a source.
Linux=`cat /etc/os-release | grep "ID" | head -1 | awk -F= '{ print $2 }'`
echo $Linux
Output:
"amzn"
Simplest ways to remove double quotes from variables are
Linux=`echo "$Linux" | tr -d '"'`
Linux=$(eval echo $Linux)
Linux=`echo ${Linux//\"/}`
Linux=`echo $Linux | xargs`
All provides the Output without double quotes:
echo $Linux
amzn
I know this is a very old question, but here is another sed variation, which may be useful to someone. Unlike some of the others, it only replaces double quotes at the start or end...
echo "$opt" | sed -r 's/^"|"$//g'
If you need to match single or double quotes, and only strings that are properly quoted. You can use this slightly more complex regex...
echo $opt | sed -E "s|^(['\"])(.*)\1$|\2|g"
This uses backrefences to ensure the quote at the end is the same as at the start.
In Bash, you could use the following one-liner:
[[ "${var}" == \"*\" || "${var}" == \'*\' ]] && var="${var:1:-1}"
This will remove surrounding quotes (both single and double) from the string stored in var while keeping quote characters inside the string intact. Also, this won't do anything if there's only a single leading quote or only a single trailing quote or if there are mixed quote characters at start/end.
Wrapped in a function:
#!/usr/bin/env bash
# Strip surrounding quotes from string [$1: variable name]
function strip_quotes() {
local -n var="$1"
[[ "${var}" == \"*\" || "${var}" == \'*\' ]] && var="${var:1:-1}"
}
str="'hello world'"
echo "Before: ${str}"
strip_quotes str
echo "After: ${str}"
My version
strip_quotes() {
while [[ $# -gt 0 ]]; do
local value=${!1}
local len=${#value}
[[ ${value:0:1} == \" && ${value:$len-1:1} == \" ]] && declare -g $1="${value:1:$len-2}"
shift
done
}
The function accepts variable name(s) and strips quotes in place. It only strips a matching pair of leading and trailing quotes. It doesn't check if the trailing quote is escaped (preceded by \ which is not itself escaped).
In my experience, general-purpose string utility functions like this (I have a library of them) are most efficient when manipulating the strings directly, not using any pattern matching and especially not creating any sub-shells, or calling any external tools such as sed, awk or grep.
var1="\"test \\ \" end \""
var2=test
var3=\"test
var4=test\"
echo before:
for i in var{1,2,3,4}; do
echo $i="${!i}"
done
strip_quotes var{1,2,3,4}
echo
echo after:
for i in var{1,2,3,4}; do
echo $i="${!i}"
done
I use this regular expression, which avoids removing quotes from strings that are not properly quoted, here the different outputs are shown depending on the inputs, only one with begin-end quote was affected:
echo '"only first' | sed 's/^"\(.*\)"$/\1/'
Output: >"only first<
echo 'only last"' | sed 's/^"\(.*\)"$/\1/'
Output: >"only last"<
echo '"both"' | sed 's/^"\(.*\)"$/\1/'
Output: >both<
echo '"space after" ' | sed 's/^"\(.*\)"$/\1/'
Output: >"space after" <
echo ' "space before"' | sed 's/^"\(.*\)"$/\1/'
Output: > "space before"<
STR='"0.0.0"' ## OR STR="\"0.0.0\""
echo "${STR//\"/}"
## Output: 0.0.0
There is another way to do it. Like:
echo ${opt:1:-1}
If you try to remove quotes because the Makefile keeps them, try this:
$(subst $\",,$(YOUR_VARIABLE))
Based on another answer: https://stackoverflow.com/a/10430975/10452175
How do I echo a variable inside single quotes?
echo 'test text "here_is_some_test_text_$counter" "output"' >> ${FILE}
Variables are expanded in double quoted strings, but not in single quoted strings:
$ name=World
$ echo "Hello $name"
Hello World
$ echo 'Hello $name'
Hello $name
If you can simply switch quotes, do so.
If you prefer sticking with single quotes to avoid the additional escaping, you can instead mix and match quotes in the same argument:
$ echo 'single quoted. '"Double quoted. "'Single quoted again.'
single quoted. Double quoted. Single quoted again.
$ echo '"$name" has the value '"$name"
"$name" has the value World
Applied to your case:
echo 'test text "here_is_some_test_text_'"$counter"'" "output"' >> "$FILE"
use printf:
printf 'test text "here_is_some_test_text_%s" "output"\n' "$counter" >> ${FILE}
Use a heredoc:
cat << EOF >> ${FILE}
test text "here_is_some_test_text_$counter" "output"
EOF
The most readable, functional way uses curly braces inside double quotes.
'test text "here_is_some_test_text_'"${counter}"'" "output"' >> "${FILE}"
You can do it this way:
$ counter=1 eval echo `echo 'test text \
"here_is_some_test_text_$counter" "output"' | \
sed -s 's/\"/\\\\"/g'` > file
cat file
test text "here_is_some_test_text_1" "output"
Explanation:
Eval command will process a string as command, so after the correct amount of escaping it will produce the desired result.
It says execute the following string as command:
'echo test text \"here_is_some_test_text_$counter\" \"output\"'
Command again in one line:
counter=1 eval echo `echo 'test text "here_is_some_test_text_$counter" "output"' | sed -s 's/\"/\\\\"/g'` > file
Output a variable wrapped with single quotes:
printf "'"'Hello %s'"'" world
Adding another pair of single quotes arround the variable solved my problem.
For your case:
echo 'test text "here_is_some_test_text_'$counter'" "output"' >> ${FILE}
with a subshell:
var='hello' echo 'blah_'`echo $var`' blah blah';
How do I echo a variable inside single quotes?
echo 'test text "here_is_some_test_text_$counter" "output"' >> ${FILE}
Variables are expanded in double quoted strings, but not in single quoted strings:
$ name=World
$ echo "Hello $name"
Hello World
$ echo 'Hello $name'
Hello $name
If you can simply switch quotes, do so.
If you prefer sticking with single quotes to avoid the additional escaping, you can instead mix and match quotes in the same argument:
$ echo 'single quoted. '"Double quoted. "'Single quoted again.'
single quoted. Double quoted. Single quoted again.
$ echo '"$name" has the value '"$name"
"$name" has the value World
Applied to your case:
echo 'test text "here_is_some_test_text_'"$counter"'" "output"' >> "$FILE"
use printf:
printf 'test text "here_is_some_test_text_%s" "output"\n' "$counter" >> ${FILE}
Use a heredoc:
cat << EOF >> ${FILE}
test text "here_is_some_test_text_$counter" "output"
EOF
The most readable, functional way uses curly braces inside double quotes.
'test text "here_is_some_test_text_'"${counter}"'" "output"' >> "${FILE}"
You can do it this way:
$ counter=1 eval echo `echo 'test text \
"here_is_some_test_text_$counter" "output"' | \
sed -s 's/\"/\\\\"/g'` > file
cat file
test text "here_is_some_test_text_1" "output"
Explanation:
Eval command will process a string as command, so after the correct amount of escaping it will produce the desired result.
It says execute the following string as command:
'echo test text \"here_is_some_test_text_$counter\" \"output\"'
Command again in one line:
counter=1 eval echo `echo 'test text "here_is_some_test_text_$counter" "output"' | sed -s 's/\"/\\\\"/g'` > file
Output a variable wrapped with single quotes:
printf "'"'Hello %s'"'" world
Adding another pair of single quotes arround the variable solved my problem.
For your case:
echo 'test text "here_is_some_test_text_'$counter'" "output"' >> ${FILE}
with a subshell:
var='hello' echo 'blah_'`echo $var`' blah blah';
Below is the snippet of a shell script from a larger script. It removes the quotes from the string that is held by a variable. I am doing it using sed, but is it efficient? If not, then what is the efficient way?
#!/bin/sh
opt="\"html\\test\\\""
temp=`echo $opt | sed 's/.\(.*\)/\1/' | sed 's/\(.*\)./\1/'`
echo $temp
Use tr to delete ":
echo "$opt" | tr -d '"'
NOTE: This does not fully answer the question, removes all double quotes, not just leading and trailing. See other answers below.
There's a simpler and more efficient way, using the native shell prefix/suffix removal feature:
temp="${opt%\"}"
temp="${temp#\"}"
echo "$temp"
${opt%\"} will remove the suffix " (escaped with a backslash to prevent shell interpretation).
${temp#\"} will remove the prefix " (escaped with a backslash to prevent shell interpretation).
Another advantage is that it will remove surrounding quotes only if there are surrounding quotes.
BTW, your solution always removes the first and last character, whatever they may be (of course, I'm sure you know your data, but it's always better to be sure of what you're removing).
Using sed:
echo "$opt" | sed -e 's/^"//' -e 's/"$//'
(Improved version, as indicated by jfgagne, getting rid of echo)
sed -e 's/^"//' -e 's/"$//' <<<"$opt"
So it replaces a leading " with nothing, and a trailing " with nothing too. In the same invocation (there isn't any need to pipe and start another sed. Using -e you can have multiple text processing).
If you're using jq and trying to remove the quotes from the result, the other answers will work, but there's a better way. By using the -r option, you can output the result with no quotes.
$ echo '{"foo": "bar"}' | jq '.foo'
"bar"
$ echo '{"foo": "bar"}' | jq -r '.foo'
bar
There is a straightforward way using xargs:
> echo '"quoted"' | xargs
quoted
xargs uses echo as the default command if no command is provided and strips quotes from the input, see e.g. here. Note, however, that this will work only if the string does not contain additional quotes. In that case it will either fail (uneven number of quotes) or remove all of them.
If you came here for aws cli --query, try this. --output text
You can do it with only one call to sed:
$ echo "\"html\\test\\\"" | sed 's/^"\(.*\)"$/\1/'
html\test\
The shortest way around - try:
echo $opt | sed "s/\"//g"
It actually removes all "s (double quotes) from opt (are there really going to be any more double quotes other than in the beginning and the end though? So it's actually the same thing, and much more brief ;-))
The easiest solution in Bash:
$ s='"abc"'
$ echo $s
"abc"
$ echo "${s:1:-1}"
abc
This is called substring expansion (see Gnu Bash Manual and search for ${parameter:offset:length}). In this example it takes the substring from s starting at position 1 and ending at the second last position. This is due to the fact that if length is a negative value it is interpreted as a backwards running offset from the end of parameter.
Update
A simple and elegant answer from Stripping single and double quotes in a string using bash / standard Linux commands only:
BAR=$(eval echo $BAR) strips quotes from BAR.
=============================================================
Based on hueybois's answer, I came up with this function after much trial and error:
function stripStartAndEndQuotes {
cmd="temp=\${$1%\\\"}"
eval echo $cmd
temp="${temp#\"}"
eval echo "$1=$temp"
}
If you don't want anything printed out, you can pipe the evals to /dev/null 2>&1.
Usage:
$ BAR="FOO BAR"
$ echo BAR
"FOO BAR"
$ stripStartAndEndQuotes "BAR"
$ echo BAR
FOO BAR
This is the most discrete way without using sed:
x='"fish"'
printf " quotes: %s\nno quotes: %s\n" "$x" "${x//\"/}"
Or
echo $x
echo ${x//\"/}
Output:
quotes: "fish"
no quotes: fish
I got this from a source.
Linux=`cat /etc/os-release | grep "ID" | head -1 | awk -F= '{ print $2 }'`
echo $Linux
Output:
"amzn"
Simplest ways to remove double quotes from variables are
Linux=`echo "$Linux" | tr -d '"'`
Linux=$(eval echo $Linux)
Linux=`echo ${Linux//\"/}`
Linux=`echo $Linux | xargs`
All provides the Output without double quotes:
echo $Linux
amzn
I know this is a very old question, but here is another sed variation, which may be useful to someone. Unlike some of the others, it only replaces double quotes at the start or end...
echo "$opt" | sed -r 's/^"|"$//g'
If you need to match single or double quotes, and only strings that are properly quoted. You can use this slightly more complex regex...
echo $opt | sed -E "s|^(['\"])(.*)\1$|\2|g"
This uses backrefences to ensure the quote at the end is the same as at the start.
In Bash, you could use the following one-liner:
[[ "${var}" == \"*\" || "${var}" == \'*\' ]] && var="${var:1:-1}"
This will remove surrounding quotes (both single and double) from the string stored in var while keeping quote characters inside the string intact. Also, this won't do anything if there's only a single leading quote or only a single trailing quote or if there are mixed quote characters at start/end.
Wrapped in a function:
#!/usr/bin/env bash
# Strip surrounding quotes from string [$1: variable name]
function strip_quotes() {
local -n var="$1"
[[ "${var}" == \"*\" || "${var}" == \'*\' ]] && var="${var:1:-1}"
}
str="'hello world'"
echo "Before: ${str}"
strip_quotes str
echo "After: ${str}"
My version
strip_quotes() {
while [[ $# -gt 0 ]]; do
local value=${!1}
local len=${#value}
[[ ${value:0:1} == \" && ${value:$len-1:1} == \" ]] && declare -g $1="${value:1:$len-2}"
shift
done
}
The function accepts variable name(s) and strips quotes in place. It only strips a matching pair of leading and trailing quotes. It doesn't check if the trailing quote is escaped (preceded by \ which is not itself escaped).
In my experience, general-purpose string utility functions like this (I have a library of them) are most efficient when manipulating the strings directly, not using any pattern matching and especially not creating any sub-shells, or calling any external tools such as sed, awk or grep.
var1="\"test \\ \" end \""
var2=test
var3=\"test
var4=test\"
echo before:
for i in var{1,2,3,4}; do
echo $i="${!i}"
done
strip_quotes var{1,2,3,4}
echo
echo after:
for i in var{1,2,3,4}; do
echo $i="${!i}"
done
I use this regular expression, which avoids removing quotes from strings that are not properly quoted, here the different outputs are shown depending on the inputs, only one with begin-end quote was affected:
echo '"only first' | sed 's/^"\(.*\)"$/\1/'
Output: >"only first<
echo 'only last"' | sed 's/^"\(.*\)"$/\1/'
Output: >"only last"<
echo '"both"' | sed 's/^"\(.*\)"$/\1/'
Output: >both<
echo '"space after" ' | sed 's/^"\(.*\)"$/\1/'
Output: >"space after" <
echo ' "space before"' | sed 's/^"\(.*\)"$/\1/'
Output: > "space before"<
STR='"0.0.0"' ## OR STR="\"0.0.0\""
echo "${STR//\"/}"
## Output: 0.0.0
There is another way to do it. Like:
echo ${opt:1:-1}
If you try to remove quotes because the Makefile keeps them, try this:
$(subst $\",,$(YOUR_VARIABLE))
Based on another answer: https://stackoverflow.com/a/10430975/10452175
Working with printf in a bash script, adding no spaces after "\n" does not create a newline, whereas adding a space creates a newline, e. g.:
No space after "\n"
NewLine=`printf "\n"`
echo -e "Firstline${NewLine}Lastline"
Result:
FirstlineLastline
Space after "\n "
NewLine=`printf "\n "`
echo -e "Firstline${NewLine}Lastline"
Result:
Firstline
Lastline
Question: Why doesn't 1. create the following result:
Firstline
Lastline
I know that this specific issue could have been worked around using other techniques, but I want to focus on why 1. does not work.
Edited:
When using echo instead of printf, I get the expected result, but why does printf work differently?
NewLine=`echo "\n"`
echo -e "Firstline${NewLine}Lastline"
Result:
Firstline
Lastline
The backtick operator removes trailing new lines. See 3.4.5. Command substitution at http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html
Note on edited question
Compare:
[alvaro#localhost ~]$ printf "\n"
[alvaro#localhost ~]$ echo "\n"
\n
[alvaro#localhost ~]$ echo -e "\n"
[alvaro#localhost ~]$
The echo command doesn't treat \n as a newline unless you tell him to do so:
NAME
echo - display a line of text
[...]
-e enable interpretation of backslash escapes
POSIX 7 specifies this behaviour here:
[...] with the standard output of the command, removing sequences of one or more characters at the end of the substitution
Maybe people will come here with the same problem I had:
echoing \n inside a code wrapped in backsticks. A little tip:
printf "astring\n"
# and
printf "%s\n" "astring"
# both have the same effect.
# So... I prefer the less typing one
The short answer is:
# Escape \n correctly !
# Using just: printf "$myvar\n" causes this effect inside the backsticks:
printf "banana
"
# So... you must try \\n that will give you the desired
printf "banana\n"
# Or even \\\\n if this string is being send to another place
# before echoing,
buffer="${buffer}\\\\n printf \"$othervar\\\\n\""
One common problem is that if you do inside the code:
echo 'Tomato is nice'
when surrounded with backsticks will produce the error
command Tomato not found.
The workaround is to add another echo -e or printf
printed=0
function mecho(){
#First time you need an "echo" in order bash relaxes.
if [[ $printed == 0 ]]; then
printf "echo -e $1\\\\n"
printed=1
else
echo -e "\r\n\r$1\\\\n"
fi
}
Now you can debug your code doing in prompt just:
(prompt)$ `mySuperFunction "arg1" "etc"`
The output will be nicely
mydebug: a value
otherdebug: whathever appended using myecho
a third string
and debuging internally with
mecho "a string to be hacktyped"
$ printf -v NewLine "\n"
$ echo -e "Firstline${NewLine}Lastline"
Firstline
Lastline
$ echo "Firstline${NewLine}Lastline"
Firstline
Lastline
It looks like BASH is removing trailing newlines.
e.g.
NewLine=`printf " \n\n\n"`
echo -e "Firstline${NewLine}Lastline"
Firstline Lastline
NewLine=`printf " \n\n\n "`
echo -e "Firstline${NewLine}Lastline"
Firstline
Lastline
Your edited echo version is putting a literal backslash-n into the variable $NewLine which then gets interpreted by your echo -e. If you did this instead:
NewLine=$(echo -e "\n")
echo -e "Firstline${NewLine}Lastline"
your result would be the same as in case #1. To make that one work that way, you'd have to escape the backslash and put the whole thing in single quotes:
NewLine=$(printf '\\n')
echo -e "Firstline${NewLine}Lastline"
or double escape it:
NewLine=$(printf "\\\n")
Of course, you could just use printf directly or you can set your NewLine value like this:
printf "Firstline\nLastline\n"
or
NewLine=$'\n'
echo "Firstline${NewLine}Lastline" # no need for -e
For people coming here wondering how to use newlines in arguments to printf, use %b instead of %s:
$> printf "a%sa" "\n"
a\na
$> printf "a%ba" "\n"
a
a
From the manual:
%b expand backslash escape sequences in the corresponding argument
We do not need "echo" or "printf" for creating the NewLine variable:
NewLine="
"
printf "%q\n" "${NewLine}"
echo "Firstline${NewLine}Lastline"
Bash delete all trailing newlines in commands substitution.
To save trailing newlines, assign printf output to the variable with printf -v VAR
instead of
NewLine=`printf "\n"`
echo -e "Firstline${NewLine}Lastline"
#FirstlineLastline
use
printf -v NewLine '\n'
echo -e "Firstline${NewLine}Lastline"
#Firstline
#Lastline
Explanation
According to bash man
3.5.4 Command Substitution
$(command)
or
`command`
Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
So, after adding any trailing newlines, bash will delete them.
var=$(printf '%s\n%s\n\n\n' 'foo' 'bar')
echo "$var"
output:
foo
bar
According to help printf
printf [-v var] format [arguments]
If the -v option is supplied, the output is placed into the value of the shell variable VAR rather than being sent to the standard output.
In this case, for safe copying of formatted text to the variable, use the [-v var] option:
printf -v var '%s\n%s\n\n\n' 'foo' 'bar'
echo "$var"
output:
foo
bar
Works ok if you add "\r"
$ nl=`printf "\n\r"` && echo "1${nl}2"
1
2