Related
Below is the snippet of a shell script from a larger script. It removes the quotes from the string that is held by a variable. I am doing it using sed, but is it efficient? If not, then what is the efficient way?
#!/bin/sh
opt="\"html\\test\\\""
temp=`echo $opt | sed 's/.\(.*\)/\1/' | sed 's/\(.*\)./\1/'`
echo $temp
Use tr to delete ":
echo "$opt" | tr -d '"'
NOTE: This does not fully answer the question, removes all double quotes, not just leading and trailing. See other answers below.
There's a simpler and more efficient way, using the native shell prefix/suffix removal feature:
temp="${opt%\"}"
temp="${temp#\"}"
echo "$temp"
${opt%\"} will remove the suffix " (escaped with a backslash to prevent shell interpretation).
${temp#\"} will remove the prefix " (escaped with a backslash to prevent shell interpretation).
Another advantage is that it will remove surrounding quotes only if there are surrounding quotes.
BTW, your solution always removes the first and last character, whatever they may be (of course, I'm sure you know your data, but it's always better to be sure of what you're removing).
Using sed:
echo "$opt" | sed -e 's/^"//' -e 's/"$//'
(Improved version, as indicated by jfgagne, getting rid of echo)
sed -e 's/^"//' -e 's/"$//' <<<"$opt"
So it replaces a leading " with nothing, and a trailing " with nothing too. In the same invocation (there isn't any need to pipe and start another sed. Using -e you can have multiple text processing).
If you're using jq and trying to remove the quotes from the result, the other answers will work, but there's a better way. By using the -r option, you can output the result with no quotes.
$ echo '{"foo": "bar"}' | jq '.foo'
"bar"
$ echo '{"foo": "bar"}' | jq -r '.foo'
bar
There is a straightforward way using xargs:
> echo '"quoted"' | xargs
quoted
xargs uses echo as the default command if no command is provided and strips quotes from the input, see e.g. here. Note, however, that this will work only if the string does not contain additional quotes. In that case it will either fail (uneven number of quotes) or remove all of them.
If you came here for aws cli --query, try this. --output text
You can do it with only one call to sed:
$ echo "\"html\\test\\\"" | sed 's/^"\(.*\)"$/\1/'
html\test\
The shortest way around - try:
echo $opt | sed "s/\"//g"
It actually removes all "s (double quotes) from opt (are there really going to be any more double quotes other than in the beginning and the end though? So it's actually the same thing, and much more brief ;-))
The easiest solution in Bash:
$ s='"abc"'
$ echo $s
"abc"
$ echo "${s:1:-1}"
abc
This is called substring expansion (see Gnu Bash Manual and search for ${parameter:offset:length}). In this example it takes the substring from s starting at position 1 and ending at the second last position. This is due to the fact that if length is a negative value it is interpreted as a backwards running offset from the end of parameter.
Update
A simple and elegant answer from Stripping single and double quotes in a string using bash / standard Linux commands only:
BAR=$(eval echo $BAR) strips quotes from BAR.
=============================================================
Based on hueybois's answer, I came up with this function after much trial and error:
function stripStartAndEndQuotes {
cmd="temp=\${$1%\\\"}"
eval echo $cmd
temp="${temp#\"}"
eval echo "$1=$temp"
}
If you don't want anything printed out, you can pipe the evals to /dev/null 2>&1.
Usage:
$ BAR="FOO BAR"
$ echo BAR
"FOO BAR"
$ stripStartAndEndQuotes "BAR"
$ echo BAR
FOO BAR
This is the most discrete way without using sed:
x='"fish"'
printf " quotes: %s\nno quotes: %s\n" "$x" "${x//\"/}"
Or
echo $x
echo ${x//\"/}
Output:
quotes: "fish"
no quotes: fish
I got this from a source.
Linux=`cat /etc/os-release | grep "ID" | head -1 | awk -F= '{ print $2 }'`
echo $Linux
Output:
"amzn"
Simplest ways to remove double quotes from variables are
Linux=`echo "$Linux" | tr -d '"'`
Linux=$(eval echo $Linux)
Linux=`echo ${Linux//\"/}`
Linux=`echo $Linux | xargs`
All provides the Output without double quotes:
echo $Linux
amzn
I know this is a very old question, but here is another sed variation, which may be useful to someone. Unlike some of the others, it only replaces double quotes at the start or end...
echo "$opt" | sed -r 's/^"|"$//g'
If you need to match single or double quotes, and only strings that are properly quoted. You can use this slightly more complex regex...
echo $opt | sed -E "s|^(['\"])(.*)\1$|\2|g"
This uses backrefences to ensure the quote at the end is the same as at the start.
In Bash, you could use the following one-liner:
[[ "${var}" == \"*\" || "${var}" == \'*\' ]] && var="${var:1:-1}"
This will remove surrounding quotes (both single and double) from the string stored in var while keeping quote characters inside the string intact. Also, this won't do anything if there's only a single leading quote or only a single trailing quote or if there are mixed quote characters at start/end.
Wrapped in a function:
#!/usr/bin/env bash
# Strip surrounding quotes from string [$1: variable name]
function strip_quotes() {
local -n var="$1"
[[ "${var}" == \"*\" || "${var}" == \'*\' ]] && var="${var:1:-1}"
}
str="'hello world'"
echo "Before: ${str}"
strip_quotes str
echo "After: ${str}"
My version
strip_quotes() {
while [[ $# -gt 0 ]]; do
local value=${!1}
local len=${#value}
[[ ${value:0:1} == \" && ${value:$len-1:1} == \" ]] && declare -g $1="${value:1:$len-2}"
shift
done
}
The function accepts variable name(s) and strips quotes in place. It only strips a matching pair of leading and trailing quotes. It doesn't check if the trailing quote is escaped (preceded by \ which is not itself escaped).
In my experience, general-purpose string utility functions like this (I have a library of them) are most efficient when manipulating the strings directly, not using any pattern matching and especially not creating any sub-shells, or calling any external tools such as sed, awk or grep.
var1="\"test \\ \" end \""
var2=test
var3=\"test
var4=test\"
echo before:
for i in var{1,2,3,4}; do
echo $i="${!i}"
done
strip_quotes var{1,2,3,4}
echo
echo after:
for i in var{1,2,3,4}; do
echo $i="${!i}"
done
I use this regular expression, which avoids removing quotes from strings that are not properly quoted, here the different outputs are shown depending on the inputs, only one with begin-end quote was affected:
echo '"only first' | sed 's/^"\(.*\)"$/\1/'
Output: >"only first<
echo 'only last"' | sed 's/^"\(.*\)"$/\1/'
Output: >"only last"<
echo '"both"' | sed 's/^"\(.*\)"$/\1/'
Output: >both<
echo '"space after" ' | sed 's/^"\(.*\)"$/\1/'
Output: >"space after" <
echo ' "space before"' | sed 's/^"\(.*\)"$/\1/'
Output: > "space before"<
STR='"0.0.0"' ## OR STR="\"0.0.0\""
echo "${STR//\"/}"
## Output: 0.0.0
There is another way to do it. Like:
echo ${opt:1:-1}
If you try to remove quotes because the Makefile keeps them, try this:
$(subst $\",,$(YOUR_VARIABLE))
Based on another answer: https://stackoverflow.com/a/10430975/10452175
Bear with me at first, thank you. Suppose I have
$ echo $'foo\nbar'
foo
bar
Now when I assign the string to a Bash variable, Bash does not give the same vertical output anymore:
$ str='foo\nbar'
$
$ echo $str
foo\nbar
$
$ echo $'str'
str
Try printf:
$ printf "$str\n"
foo
bar
Those examples are for illustration purposes because I am looking for a way to expand the newline(s) inside the $str variable such that I can substitute the $str variable on sed replacement (insertion) side.
# this does not work:
sed -i.bak $'/<!-- insert here -->/i\\\n'$'str'$'\\\n' index.html
# this works as expected though:
sed -i.bak $'/<!-- insert here -->/i\\\n'foo$'\\\n'bar$'\\\n' index.html
I did several ways to hack this but none worked; here is one example:
# this does not work:
sed -i.bak $'/<!-- insert here -->/i\\\n'`printf 'foo\\x0Abar'`$'\\\n' index.html
Further tests, I realized that as long as the variable does not contain newlines, things work as expected:
# This works as long as str2 does not contain any newline.
str2='foo_bar'
sed -i.bak $'/<!-- insert here -->/i\\\n'$str2$'\\\n' index.html
The expected result is that sed will insert 2 liners in place before <!-- insert here --> of the index.html file.
foo
bar
<!-- insert here -->
I try to achieve this as one liner. I know I can break sed into the vertical, multi-line form, which will be easier for me; however, I want to explore if there is a one liner style.
Is this doable or not?
My system is macOS High Sierra 10.13.6
Bash version: 3.2.57(1)-release
BSD sed was last updated on May 10, 2005
Your examples have a few subtle error, so here are a few examples regarding quoting and newlines in strings in bash and sed.
How quoting works in general:
# bash converts escape-sequence '\n' to real newline (0x0a) before passing it to echo
$ echo $'foo\nbar'
foo
bar
# bash passes literal 8 characters 'foo\nbar' to echo and echo simply prints them
$ echo 'foo\nbar'
foo\nbar
# bash passes literal 8 characters 'foo\nbar' to echo and echo converts escape-sequence
$ echo -e 'foo\nbar'
foo
bar
# bash passes literal string 'foo\nbar' to echo (twice)
# then echo recombines both arguments using a single space
$ str='foo\nbar'
$ echo $str "$str"
foo\nbar foo\nbar
# bash interprets escape-sequences and stores result 'foo<0x0a>bar' in str,
# then passes two arguments 'foo' and 'bar' to echo, due to "word splitting"
# then echo recombines both arguments using a single space
$ str=$'foo\nbar'
$ echo $str
foo bar
# bash interprets escape-sequences and stores result 'foo<0x0a>bar' in str,
# then passes it as a single argument to echo, without "word splitting"
$ str=$'foo\nbar'
$ echo "$str"
foo
bar
How to apply shell quoting, when dealing with newlines in sed
# replace a character with newline, using newline's escape-sequence
# sed will convert '\n' to a literal newline (0x0a)
$ sed 's/-/foo\nbar/' <<< 'blah-blah'
# replace a character with newline, using newline's escape-sequence in a variable
# sed will convert '\n' to a literal newline (0x0a)
$ str='foo\nbar' # str contains the escape-sequence '\n' and not a literal newline
$ sed 's/-/'"$str"'/' <<< 'blah-blah'
# replace a character with newline, using a literal newline.
# note the line-continuation-mark \ after 'foo' before the literal newline,
# which is part of the sed script, since everything in-between '' is literal
$ sed 's/-/foo\
bar/' <<< 'blah-blah' # end-of-command
# replace a character with newline, using a newline in shell-escape-mode
# note the same line-continuation-mark \ before $'\n', which is part of the sed script
# note: the sed script is a single string composed of three parts '…\', $'\n' and '…',
$ sed 's/-/foo\'$'\n''bar/' <<< 'blah-blah'
# the same as above, but with a single shell-escape-mode string instead of 3 parts.
# note the required quoting of the line-continuation-mark with an additional \ escape
# i.e. after shell-escaping the sed script contains a single \ and a literal newline
$ sed $'s/-/foo\\\nbar/' <<< 'blah-blah'
# replace a character with newline, using a shell-escaped string in a variable
$ str=$'\n' # str contains a literal newline (0x0a) due to shell escaping
$ sed 's/-/foo\'"$str"'bar/' <<< 'blah-blah'
# same as above with the required (quoted) line-continuation inside the variable
# note, how the single \ from '…foo\' (previous example) became \\ inside $'\\…'
$ str=$'\\\n' # str contains \ and a literal newline (0x0a) due to shell escaping
$ sed 's/-/foo'"$str"'bar/' <<< 'blah-blah'
All the sed examples will print the same:
blahfoo
barblah
So, a newline in sed's replacement string must either be
(1) newline's escape-sequence (i.e. '\n'), so sed can replace it with a literal newline, or
(2) a literal newline preceded by a line-continuation-mark (i.e. $'\\\n' or '\'$'\n', which is NOT the same as '\\\n' or '\\n' or $'\\n').
This means you need to replace each literal newline <0x0a> with newline's escape-sequence \n or insert a line-continuation-mark before each literal newline inside your replacement string before double-quote-expanding it into sed's substitute replacement string.
Since there are many more caveats regarding escaping in sed, I recommend you use awk's gsub function instead passing your replacement string as a variable via -v, e.g.
$ str=$'foo\nbar'
$ awk -v REP="$str" -- '{gsub(/-/, REP); print}' <<< 'blah-blah'
blahfoo
barblah
PS: I don't know, if this answer is entirely true in your case, because your operating system uses an outdated version of bash.
echo -e $str
where -e is
enable interpretation of backslash escapes
Use sed command r to insert arbitrary text
str="abc\ndef"
tmp=$(mktemp)
(
echo
printf -- "$str"
echo
) > "$tmp"
sed -i.bak '/<!-- insert here -->/r '"$tmp" index.html
rm -r "$tmp"
sed interprets newline as command delimiter. The ; doesn't really is a seds command delimeter, only newline is. Don't append/suffix ; or } or spaces in the w command - it will be interpreted as part of the filename (yes, spaces also). sed commands like w or r are escaped by a newline.
If you want more flexibility, rather move to awk.
I have a shell script with this code:
var=`hg st -R "$path"`
if [ -n "$var" ]; then
echo $var
fi
But the conditional code always executes, because hg st always prints at least one newline character.
Is there a simple way to strip whitespace from $var (like trim() in PHP)?
or
Is there a standard way of dealing with this issue?
I could use sed or AWK, but I'd like to think there is a more elegant solution to this problem.
A simple answer is:
echo " lol " | xargs
Xargs will do the trimming for you. It's one command/program, no parameters, returns the trimmed string, easy as that!
Note: this doesn't remove all internal spaces so "foo bar" stays the same; it does NOT become "foobar". However, multiple spaces will be condensed to single spaces, so "foo bar" will become "foo bar". In addition it doesn't remove end of lines characters.
Let's define a variable containing leading, trailing, and intermediate whitespace:
FOO=' test test test '
echo -e "FOO='${FOO}'"
# > FOO=' test test test '
echo -e "length(FOO)==${#FOO}"
# > length(FOO)==16
How to remove all whitespace (denoted by [:space:] in tr):
FOO=' test test test '
FOO_NO_WHITESPACE="$(echo -e "${FOO}" | tr -d '[:space:]')"
echo -e "FOO_NO_WHITESPACE='${FOO_NO_WHITESPACE}'"
# > FOO_NO_WHITESPACE='testtesttest'
echo -e "length(FOO_NO_WHITESPACE)==${#FOO_NO_WHITESPACE}"
# > length(FOO_NO_WHITESPACE)==12
How to remove leading whitespace only:
FOO=' test test test '
FOO_NO_LEAD_SPACE="$(echo -e "${FOO}" | sed -e 's/^[[:space:]]*//')"
echo -e "FOO_NO_LEAD_SPACE='${FOO_NO_LEAD_SPACE}'"
# > FOO_NO_LEAD_SPACE='test test test '
echo -e "length(FOO_NO_LEAD_SPACE)==${#FOO_NO_LEAD_SPACE}"
# > length(FOO_NO_LEAD_SPACE)==15
How to remove trailing whitespace only:
FOO=' test test test '
FOO_NO_TRAIL_SPACE="$(echo -e "${FOO}" | sed -e 's/[[:space:]]*$//')"
echo -e "FOO_NO_TRAIL_SPACE='${FOO_NO_TRAIL_SPACE}'"
# > FOO_NO_TRAIL_SPACE=' test test test'
echo -e "length(FOO_NO_TRAIL_SPACE)==${#FOO_NO_TRAIL_SPACE}"
# > length(FOO_NO_TRAIL_SPACE)==15
How to remove both leading and trailing spaces--chain the seds:
FOO=' test test test '
FOO_NO_EXTERNAL_SPACE="$(echo -e "${FOO}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
echo -e "FOO_NO_EXTERNAL_SPACE='${FOO_NO_EXTERNAL_SPACE}'"
# > FOO_NO_EXTERNAL_SPACE='test test test'
echo -e "length(FOO_NO_EXTERNAL_SPACE)==${#FOO_NO_EXTERNAL_SPACE}"
# > length(FOO_NO_EXTERNAL_SPACE)==14
Alternatively, if your bash supports it, you can replace echo -e "${FOO}" | sed ... with sed ... <<<${FOO}, like so (for trailing whitespace):
FOO_NO_TRAIL_SPACE="$(sed -e 's/[[:space:]]*$//' <<<${FOO})"
There is a solution which only uses Bash built-ins called wildcards:
var=" abc "
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
printf '%s' "===$var==="
Here's the same wrapped in a function:
trim() {
local var="$*"
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
printf '%s' "$var"
}
You pass the string to be trimmed in quoted form, e.g.:
trim " abc "
A nice thing about this solution is that it will work with any POSIX-compliant shell.
Reference
Remove leading & trailing whitespace from a Bash variable (original source)
In order to remove all the spaces from the beginning and the end of a string (including end of line characters):
echo $variable | xargs echo -n
This will remove duplicate spaces also:
echo " this string has a lot of spaces " | xargs echo -n
Produces: 'this string has a lot of spaces'
Bash has a feature called parameter expansion, which, among other things, allows string replacement based on so-called patterns (patterns resemble regular expressions, but there are fundamental differences and limitations).
[flussence's original line: Bash has regular expressions, but they're well-hidden:]
The following demonstrates how to remove all white space (even from the interior) from a variable value.
$ var='abc def'
$ echo "$var"
abc def
# Note: flussence's original expression was "${var/ /}", which only replaced the *first* space char., wherever it appeared.
$ echo -n "${var//[[:space:]]/}"
abcdef
Strip one leading and one trailing space
trim()
{
local trimmed="$1"
# Strip leading space.
trimmed="${trimmed## }"
# Strip trailing space.
trimmed="${trimmed%% }"
echo "$trimmed"
}
For example:
test1="$(trim " one leading")"
test2="$(trim "one trailing ")"
test3="$(trim " one leading and one trailing ")"
echo "'$test1', '$test2', '$test3'"
Output:
'one leading', 'one trailing', 'one leading and one trailing'
Strip all leading and trailing spaces
trim()
{
local trimmed="$1"
# Strip leading spaces.
while [[ $trimmed == ' '* ]]; do
trimmed="${trimmed## }"
done
# Strip trailing spaces.
while [[ $trimmed == *' ' ]]; do
trimmed="${trimmed%% }"
done
echo "$trimmed"
}
For example:
test4="$(trim " two leading")"
test5="$(trim "two trailing ")"
test6="$(trim " two leading and two trailing ")"
echo "'$test4', '$test5', '$test6'"
Output:
'two leading', 'two trailing', 'two leading and two trailing'
From Bash Guide section on globbing
To use an extglob in a parameter expansion
#Turn on extended globbing
shopt -s extglob
#Trim leading and trailing whitespace from a variable
x=${x##+([[:space:]])}; x=${x%%+([[:space:]])}
#Turn off extended globbing
shopt -u extglob
Here's the same functionality wrapped in a function (NOTE: Need to quote input string passed to function):
trim() {
# Determine if 'extglob' is currently on.
local extglobWasOff=1
shopt extglob >/dev/null && extglobWasOff=0
(( extglobWasOff )) && shopt -s extglob # Turn 'extglob' on, if currently turned off.
# Trim leading and trailing whitespace
local var=$1
var=${var##+([[:space:]])}
var=${var%%+([[:space:]])}
(( extglobWasOff )) && shopt -u extglob # If 'extglob' was off before, turn it back off.
echo -n "$var" # Output trimmed string.
}
Usage:
string=" abc def ghi ";
#need to quote input-string to preserve internal white-space if any
trimmed=$(trim "$string");
echo "$trimmed";
If we alter the function to execute in a subshell, we don't have to worry about examining the current shell option for extglob, we can just set it without affecting the current shell. This simplifies the function tremendously. I also update the positional parameters "in place" so I don't even need a local variable
trim() {
shopt -s extglob
set -- "${1##+([[:space:]])}"
printf "%s" "${1%%+([[:space:]])}"
}
so:
$ s=$'\t\n \r\tfoo '
$ shopt -u extglob
$ shopt extglob
extglob off
$ printf ">%q<\n" "$s" "$(trim "$s")"
>$'\t\n \r\tfoo '<
>foo<
$ shopt extglob
extglob off
You can trim simply with echo:
foo=" qsdqsd qsdqs q qs "
# Not trimmed
echo \'$foo\'
# Trim
foo=`echo $foo`
# Trimmed
echo \'$foo\'
I've always done it with sed
var=`hg st -R "$path" | sed -e 's/ *$//'`
If there is a more elegant solution, I hope somebody posts it.
With Bash's extended pattern matching features enabled (shopt -s extglob), you can use this:
{trimmed##*( )}
to remove an arbitrary amount of leading spaces.
You can delete newlines with tr:
var=`hg st -R "$path" | tr -d '\n'`
if [ -n $var ]; then
echo $var
done
# Trim whitespace from both ends of specified parameter
trim () {
read -rd '' $1 <<<"${!1}"
}
# Unit test for trim()
test_trim () {
local foo="$1"
trim foo
test "$foo" = "$2"
}
test_trim hey hey &&
test_trim ' hey' hey &&
test_trim 'ho ' ho &&
test_trim 'hey ho' 'hey ho' &&
test_trim ' hey ho ' 'hey ho' &&
test_trim $'\n\n\t hey\n\t ho \t\n' $'hey\n\t ho' &&
test_trim $'\n' '' &&
test_trim '\n' '\n' &&
echo passed
There are a lot of answers, but I still believe my just-written script is worth being mentioned because:
it was successfully tested in the shells bash/dash/busybox shell
it is extremely small
it doesn't depend on external commands and doesn't need to fork (->fast and low resource usage)
it works as expected:
it strips all spaces and tabs from beginning and end, but not more
important: it doesn't remove anything from the middle of the string (many other answers do), even newlines will remain
special: the "$*" joins multiple arguments using one space. if you want to trim & output only the first argument, use "$1" instead
if doesn't have any problems with matching file name patterns etc
The script:
trim() {
local s2 s="$*"
until s2="${s#[[:space:]]}"; [ "$s2" = "$s" ]; do s="$s2"; done
until s2="${s%[[:space:]]}"; [ "$s2" = "$s" ]; do s="$s2"; done
echo "$s"
}
Usage:
mystring=" here is
something "
mystring=$(trim "$mystring")
echo ">$mystring<"
Output:
>here is
something<
This is what I did and worked out perfect and so simple:
the_string=" test"
the_string=`echo $the_string`
echo "$the_string"
Output:
test
If you have shopt -s extglob enabled, then the following is a neat solution.
This worked for me:
text=" trim my edges "
trimmed=$text
trimmed=${trimmed##+( )} #Remove longest matching series of spaces from the front
trimmed=${trimmed%%+( )} #Remove longest matching series of spaces from the back
echo "<$trimmed>" #Adding angle braces just to make it easier to confirm that all spaces are removed
#Result
<trim my edges>
To put that on fewer lines for the same result:
text=" trim my edges "
trimmed=${${text##+( )}%%+( )}
# Strip leading and trailing white space (new line inclusive).
trim(){
[[ "$1" =~ [^[:space:]](.*[^[:space:]])? ]]
printf "%s" "$BASH_REMATCH"
}
OR
# Strip leading white space (new line inclusive).
ltrim(){
[[ "$1" =~ [^[:space:]].* ]]
printf "%s" "$BASH_REMATCH"
}
# Strip trailing white space (new line inclusive).
rtrim(){
[[ "$1" =~ .*[^[:space:]] ]]
printf "%s" "$BASH_REMATCH"
}
# Strip leading and trailing white space (new line inclusive).
trim(){
printf "%s" "$(rtrim "$(ltrim "$1")")"
}
OR
# Strip leading and trailing specified characters. ex: str=$(trim "$str" $'\n a')
trim(){
if [ "$2" ]; then
trim_chrs="$2"
else
trim_chrs="[:space:]"
fi
[[ "$1" =~ ^["$trim_chrs"]*(.*[^"$trim_chrs"])["$trim_chrs"]*$ ]]
printf "%s" "${BASH_REMATCH[1]}"
}
OR
# Strip leading specified characters. ex: str=$(ltrim "$str" $'\n a')
ltrim(){
if [ "$2" ]; then
trim_chrs="$2"
else
trim_chrs="[:space:]"
fi
[[ "$1" =~ ^["$trim_chrs"]*(.*[^"$trim_chrs"]) ]]
printf "%s" "${BASH_REMATCH[1]}"
}
# Strip trailing specified characters. ex: str=$(rtrim "$str" $'\n a')
rtrim(){
if [ "$2" ]; then
trim_chrs="$2"
else
trim_chrs="[:space:]"
fi
[[ "$1" =~ ^(.*[^"$trim_chrs"])["$trim_chrs"]*$ ]]
printf "%s" "${BASH_REMATCH[1]}"
}
# Strip leading and trailing specified characters. ex: str=$(trim "$str" $'\n a')
trim(){
printf "%s" "$(rtrim "$(ltrim "$1" "$2")" "$2")"
}
OR
Building upon moskit's expr soulution...
# Strip leading and trailing white space (new line inclusive).
trim(){
printf "%s" "`expr "$1" : "^[[:space:]]*\(.*[^[:space:]]\)[[:space:]]*$"`"
}
OR
# Strip leading white space (new line inclusive).
ltrim(){
printf "%s" "`expr "$1" : "^[[:space:]]*\(.*[^[:space:]]\)"`"
}
# Strip trailing white space (new line inclusive).
rtrim(){
printf "%s" "`expr "$1" : "^\(.*[^[:space:]]\)[[:space:]]*$"`"
}
# Strip leading and trailing white space (new line inclusive).
trim(){
printf "%s" "$(rtrim "$(ltrim "$1")")"
}
Use AWK:
echo $var | awk '{gsub(/^ +| +$/,"")}1'
You can use old-school tr. For example, this returns the number of modified files in a git repository, whitespaces stripped.
MYVAR=`git ls-files -m|wc -l|tr -d ' '`
This will remove all the whitespaces from your String,
VAR2="${VAR2//[[:space:]]/}"
/ replaces the first occurrence and // all occurrences of whitespaces in the string. I.e. all white spaces get replaced by – nothing
I would simply use sed:
function trim
{
echo "$1" | sed -n '1h;1!H;${;g;s/^[ \t]*//g;s/[ \t]*$//g;p;}'
}
a) Example of usage on single-line string
string=' wordA wordB wordC wordD '
trimmed=$( trim "$string" )
echo "GIVEN STRING: |$string|"
echo "TRIMMED STRING: |$trimmed|"
Output:
GIVEN STRING: | wordA wordB wordC wordD |
TRIMMED STRING: |wordA wordB wordC wordD|
b) Example of usage on multi-line string
string=' wordA
>wordB<
wordC '
trimmed=$( trim "$string" )
echo -e "GIVEN STRING: |$string|\n"
echo "TRIMMED STRING: |$trimmed|"
Output:
GIVEN STRING: | wordAA
>wordB<
wordC |
TRIMMED STRING: |wordAA
>wordB<
wordC|
c) Final note:
If you don't like to use a function, for single-line string you can simply use a "easier to remember" command like:
echo "$string" | sed -e 's/^[ \t]*//' | sed -e 's/[ \t]*$//'
Example:
echo " wordA wordB wordC " | sed -e 's/^[ \t]*//' | sed -e 's/[ \t]*$//'
Output:
wordA wordB wordC
Using the above on multi-line strings will work as well, but please note that it will cut any trailing/leading internal multiple space as well, as GuruM noticed in the comments
string=' wordAA
>four spaces before<
>one space before< '
echo "$string" | sed -e 's/^[ \t]*//' | sed -e 's/[ \t]*$//'
Output:
wordAA
>four spaces before<
>one space before<
So if you do mind to keep those spaces, please use the function at the beginning of my answer!
d) EXPLANATION of the sed syntax "find and replace" on multi-line strings used inside the function trim:
sed -n '
# If the first line, copy the pattern to the hold buffer
1h
# If not the first line, then append the pattern to the hold buffer
1!H
# If the last line then ...
$ {
# Copy from the hold to the pattern buffer
g
# Do the search and replace
s/^[ \t]*//g
s/[ \t]*$//g
# print
p
}'
There are a few different options purely in BASH:
line=${line##+([[:space:]])} # strip leading whitespace; no quote expansion!
line=${line%%+([[:space:]])} # strip trailing whitespace; no quote expansion!
line=${line//[[:space:]]/} # strip all whitespace
line=${line//[[:space:]]/} # strip all whitespace
line=${line//[[:blank:]]/} # strip all blank space
The former two require extglob be set/enabled a priori:
shopt -s extglob # bash only
NOTE: variable expansion inside quotation marks breaks the top two examples!
The pattern matching behaviour of POSIX bracket expressions are detailed here. If you are using a more modern/hackable shell such as Fish, there are built-in functions for string trimming.
I've seen scripts just use variable assignment to do the job:
$ xyz=`echo -e 'foo \n bar'`
$ echo $xyz
foo bar
Whitespace is automatically coalesced and trimmed. One has to be careful of shell metacharacters (potential injection risk).
I would also recommend always double-quoting variable substitutions in shell conditionals:
if [ -n "$var" ]; then
since something like a -o or other content in the variable could amend your test arguments.
Here's a trim() function that trims and normalizes whitespace
#!/bin/bash
function trim {
echo $*
}
echo "'$(trim " one two three ")'"
# 'one two three'
And another variant that uses regular expressions.
#!/bin/bash
function trim {
local trimmed="$#"
if [[ "$trimmed" =~ " *([^ ].*[^ ]) *" ]]
then
trimmed=${BASH_REMATCH[1]}
fi
echo "$trimmed"
}
echo "'$(trim " one two three ")'"
# 'one two three'
This does not have the problem with unwanted globbing, also, interior white-space is unmodified (assuming that $IFS is set to the default, which is ' \t\n').
It reads up to the first newline (and doesn't include it) or the end of string, whichever comes first, and strips away any mix of leading and trailing space and \t characters. If you want to preserve multiple lines (and also strip leading and trailing newlines), use read -r -d '' var << eof instead; note, however, that if your input happens to contain \neof, it will be cut off just before. (Other forms of white space, namely \r, \f, and \v, are not stripped, even if you add them to $IFS.)
read -r var << eof
$var
eof
To remove spaces and tabs from left to first word, enter:
echo " This is a test" | sed "s/^[ \t]*//"
cyberciti.biz/tips/delete-leading-spaces-from-front-of-each-word.html
var=' a b c '
trimmed=$(echo $var)
This is the simplest method I've seen. It only uses Bash, it's only a few lines, the regexp is simple, and it matches all forms of whitespace:
if [[ "$test" =~ ^[[:space:]]*([^[:space:]].*[^[:space:]])[[:space:]]*$ ]]
then
test=${BASH_REMATCH[1]}
fi
Here is a sample script to test it with:
test=$(echo -e "\n \t Spaces and tabs and newlines be gone! \t \n ")
echo "Let's see if this works:"
echo
echo "----------"
echo -e "Testing:${test} :Tested" # Ugh!
echo "----------"
echo
echo "Ugh! Let's fix that..."
if [[ "$test" =~ ^[[:space:]]*([^[:space:]].*[^[:space:]])[[:space:]]*$ ]]
then
test=${BASH_REMATCH[1]}
fi
echo
echo "----------"
echo -e "Testing:${test}:Tested" # "Testing:Spaces and tabs and newlines be gone!"
echo "----------"
echo
echo "Ah, much better."
Removing spaces to one space:
(text) | fmt -su
Assignments ignore leading and trailing whitespace and as such can be used to trim:
$ var=`echo ' hello'`; echo $var
hello
Python has a function strip() that works identically to PHP's trim(), so we can just do a little inline Python to make an easily understandable utility for this:
alias trim='python -c "import sys; sys.stdout.write(sys.stdin.read().strip())"'
This will trim leading and trailing whitespace (including newlines).
$ x=`echo -e "\n\t \n" | trim`
$ if [ -z "$x" ]; then echo hi; fi
hi
Below is the snippet of a shell script from a larger script. It removes the quotes from the string that is held by a variable. I am doing it using sed, but is it efficient? If not, then what is the efficient way?
#!/bin/sh
opt="\"html\\test\\\""
temp=`echo $opt | sed 's/.\(.*\)/\1/' | sed 's/\(.*\)./\1/'`
echo $temp
Use tr to delete ":
echo "$opt" | tr -d '"'
NOTE: This does not fully answer the question, removes all double quotes, not just leading and trailing. See other answers below.
There's a simpler and more efficient way, using the native shell prefix/suffix removal feature:
temp="${opt%\"}"
temp="${temp#\"}"
echo "$temp"
${opt%\"} will remove the suffix " (escaped with a backslash to prevent shell interpretation).
${temp#\"} will remove the prefix " (escaped with a backslash to prevent shell interpretation).
Another advantage is that it will remove surrounding quotes only if there are surrounding quotes.
BTW, your solution always removes the first and last character, whatever they may be (of course, I'm sure you know your data, but it's always better to be sure of what you're removing).
Using sed:
echo "$opt" | sed -e 's/^"//' -e 's/"$//'
(Improved version, as indicated by jfgagne, getting rid of echo)
sed -e 's/^"//' -e 's/"$//' <<<"$opt"
So it replaces a leading " with nothing, and a trailing " with nothing too. In the same invocation (there isn't any need to pipe and start another sed. Using -e you can have multiple text processing).
If you're using jq and trying to remove the quotes from the result, the other answers will work, but there's a better way. By using the -r option, you can output the result with no quotes.
$ echo '{"foo": "bar"}' | jq '.foo'
"bar"
$ echo '{"foo": "bar"}' | jq -r '.foo'
bar
There is a straightforward way using xargs:
> echo '"quoted"' | xargs
quoted
xargs uses echo as the default command if no command is provided and strips quotes from the input, see e.g. here. Note, however, that this will work only if the string does not contain additional quotes. In that case it will either fail (uneven number of quotes) or remove all of them.
If you came here for aws cli --query, try this. --output text
You can do it with only one call to sed:
$ echo "\"html\\test\\\"" | sed 's/^"\(.*\)"$/\1/'
html\test\
The shortest way around - try:
echo $opt | sed "s/\"//g"
It actually removes all "s (double quotes) from opt (are there really going to be any more double quotes other than in the beginning and the end though? So it's actually the same thing, and much more brief ;-))
The easiest solution in Bash:
$ s='"abc"'
$ echo $s
"abc"
$ echo "${s:1:-1}"
abc
This is called substring expansion (see Gnu Bash Manual and search for ${parameter:offset:length}). In this example it takes the substring from s starting at position 1 and ending at the second last position. This is due to the fact that if length is a negative value it is interpreted as a backwards running offset from the end of parameter.
Update
A simple and elegant answer from Stripping single and double quotes in a string using bash / standard Linux commands only:
BAR=$(eval echo $BAR) strips quotes from BAR.
=============================================================
Based on hueybois's answer, I came up with this function after much trial and error:
function stripStartAndEndQuotes {
cmd="temp=\${$1%\\\"}"
eval echo $cmd
temp="${temp#\"}"
eval echo "$1=$temp"
}
If you don't want anything printed out, you can pipe the evals to /dev/null 2>&1.
Usage:
$ BAR="FOO BAR"
$ echo BAR
"FOO BAR"
$ stripStartAndEndQuotes "BAR"
$ echo BAR
FOO BAR
This is the most discrete way without using sed:
x='"fish"'
printf " quotes: %s\nno quotes: %s\n" "$x" "${x//\"/}"
Or
echo $x
echo ${x//\"/}
Output:
quotes: "fish"
no quotes: fish
I got this from a source.
Linux=`cat /etc/os-release | grep "ID" | head -1 | awk -F= '{ print $2 }'`
echo $Linux
Output:
"amzn"
Simplest ways to remove double quotes from variables are
Linux=`echo "$Linux" | tr -d '"'`
Linux=$(eval echo $Linux)
Linux=`echo ${Linux//\"/}`
Linux=`echo $Linux | xargs`
All provides the Output without double quotes:
echo $Linux
amzn
I know this is a very old question, but here is another sed variation, which may be useful to someone. Unlike some of the others, it only replaces double quotes at the start or end...
echo "$opt" | sed -r 's/^"|"$//g'
If you need to match single or double quotes, and only strings that are properly quoted. You can use this slightly more complex regex...
echo $opt | sed -E "s|^(['\"])(.*)\1$|\2|g"
This uses backrefences to ensure the quote at the end is the same as at the start.
In Bash, you could use the following one-liner:
[[ "${var}" == \"*\" || "${var}" == \'*\' ]] && var="${var:1:-1}"
This will remove surrounding quotes (both single and double) from the string stored in var while keeping quote characters inside the string intact. Also, this won't do anything if there's only a single leading quote or only a single trailing quote or if there are mixed quote characters at start/end.
Wrapped in a function:
#!/usr/bin/env bash
# Strip surrounding quotes from string [$1: variable name]
function strip_quotes() {
local -n var="$1"
[[ "${var}" == \"*\" || "${var}" == \'*\' ]] && var="${var:1:-1}"
}
str="'hello world'"
echo "Before: ${str}"
strip_quotes str
echo "After: ${str}"
My version
strip_quotes() {
while [[ $# -gt 0 ]]; do
local value=${!1}
local len=${#value}
[[ ${value:0:1} == \" && ${value:$len-1:1} == \" ]] && declare -g $1="${value:1:$len-2}"
shift
done
}
The function accepts variable name(s) and strips quotes in place. It only strips a matching pair of leading and trailing quotes. It doesn't check if the trailing quote is escaped (preceded by \ which is not itself escaped).
In my experience, general-purpose string utility functions like this (I have a library of them) are most efficient when manipulating the strings directly, not using any pattern matching and especially not creating any sub-shells, or calling any external tools such as sed, awk or grep.
var1="\"test \\ \" end \""
var2=test
var3=\"test
var4=test\"
echo before:
for i in var{1,2,3,4}; do
echo $i="${!i}"
done
strip_quotes var{1,2,3,4}
echo
echo after:
for i in var{1,2,3,4}; do
echo $i="${!i}"
done
I use this regular expression, which avoids removing quotes from strings that are not properly quoted, here the different outputs are shown depending on the inputs, only one with begin-end quote was affected:
echo '"only first' | sed 's/^"\(.*\)"$/\1/'
Output: >"only first<
echo 'only last"' | sed 's/^"\(.*\)"$/\1/'
Output: >"only last"<
echo '"both"' | sed 's/^"\(.*\)"$/\1/'
Output: >both<
echo '"space after" ' | sed 's/^"\(.*\)"$/\1/'
Output: >"space after" <
echo ' "space before"' | sed 's/^"\(.*\)"$/\1/'
Output: > "space before"<
STR='"0.0.0"' ## OR STR="\"0.0.0\""
echo "${STR//\"/}"
## Output: 0.0.0
There is another way to do it. Like:
echo ${opt:1:-1}
If you try to remove quotes because the Makefile keeps them, try this:
$(subst $\",,$(YOUR_VARIABLE))
Based on another answer: https://stackoverflow.com/a/10430975/10452175
Working with printf in a bash script, adding no spaces after "\n" does not create a newline, whereas adding a space creates a newline, e. g.:
No space after "\n"
NewLine=`printf "\n"`
echo -e "Firstline${NewLine}Lastline"
Result:
FirstlineLastline
Space after "\n "
NewLine=`printf "\n "`
echo -e "Firstline${NewLine}Lastline"
Result:
Firstline
Lastline
Question: Why doesn't 1. create the following result:
Firstline
Lastline
I know that this specific issue could have been worked around using other techniques, but I want to focus on why 1. does not work.
Edited:
When using echo instead of printf, I get the expected result, but why does printf work differently?
NewLine=`echo "\n"`
echo -e "Firstline${NewLine}Lastline"
Result:
Firstline
Lastline
The backtick operator removes trailing new lines. See 3.4.5. Command substitution at http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html
Note on edited question
Compare:
[alvaro#localhost ~]$ printf "\n"
[alvaro#localhost ~]$ echo "\n"
\n
[alvaro#localhost ~]$ echo -e "\n"
[alvaro#localhost ~]$
The echo command doesn't treat \n as a newline unless you tell him to do so:
NAME
echo - display a line of text
[...]
-e enable interpretation of backslash escapes
POSIX 7 specifies this behaviour here:
[...] with the standard output of the command, removing sequences of one or more characters at the end of the substitution
Maybe people will come here with the same problem I had:
echoing \n inside a code wrapped in backsticks. A little tip:
printf "astring\n"
# and
printf "%s\n" "astring"
# both have the same effect.
# So... I prefer the less typing one
The short answer is:
# Escape \n correctly !
# Using just: printf "$myvar\n" causes this effect inside the backsticks:
printf "banana
"
# So... you must try \\n that will give you the desired
printf "banana\n"
# Or even \\\\n if this string is being send to another place
# before echoing,
buffer="${buffer}\\\\n printf \"$othervar\\\\n\""
One common problem is that if you do inside the code:
echo 'Tomato is nice'
when surrounded with backsticks will produce the error
command Tomato not found.
The workaround is to add another echo -e or printf
printed=0
function mecho(){
#First time you need an "echo" in order bash relaxes.
if [[ $printed == 0 ]]; then
printf "echo -e $1\\\\n"
printed=1
else
echo -e "\r\n\r$1\\\\n"
fi
}
Now you can debug your code doing in prompt just:
(prompt)$ `mySuperFunction "arg1" "etc"`
The output will be nicely
mydebug: a value
otherdebug: whathever appended using myecho
a third string
and debuging internally with
mecho "a string to be hacktyped"
$ printf -v NewLine "\n"
$ echo -e "Firstline${NewLine}Lastline"
Firstline
Lastline
$ echo "Firstline${NewLine}Lastline"
Firstline
Lastline
It looks like BASH is removing trailing newlines.
e.g.
NewLine=`printf " \n\n\n"`
echo -e "Firstline${NewLine}Lastline"
Firstline Lastline
NewLine=`printf " \n\n\n "`
echo -e "Firstline${NewLine}Lastline"
Firstline
Lastline
Your edited echo version is putting a literal backslash-n into the variable $NewLine which then gets interpreted by your echo -e. If you did this instead:
NewLine=$(echo -e "\n")
echo -e "Firstline${NewLine}Lastline"
your result would be the same as in case #1. To make that one work that way, you'd have to escape the backslash and put the whole thing in single quotes:
NewLine=$(printf '\\n')
echo -e "Firstline${NewLine}Lastline"
or double escape it:
NewLine=$(printf "\\\n")
Of course, you could just use printf directly or you can set your NewLine value like this:
printf "Firstline\nLastline\n"
or
NewLine=$'\n'
echo "Firstline${NewLine}Lastline" # no need for -e
For people coming here wondering how to use newlines in arguments to printf, use %b instead of %s:
$> printf "a%sa" "\n"
a\na
$> printf "a%ba" "\n"
a
a
From the manual:
%b expand backslash escape sequences in the corresponding argument
We do not need "echo" or "printf" for creating the NewLine variable:
NewLine="
"
printf "%q\n" "${NewLine}"
echo "Firstline${NewLine}Lastline"
Bash delete all trailing newlines in commands substitution.
To save trailing newlines, assign printf output to the variable with printf -v VAR
instead of
NewLine=`printf "\n"`
echo -e "Firstline${NewLine}Lastline"
#FirstlineLastline
use
printf -v NewLine '\n'
echo -e "Firstline${NewLine}Lastline"
#Firstline
#Lastline
Explanation
According to bash man
3.5.4 Command Substitution
$(command)
or
`command`
Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
So, after adding any trailing newlines, bash will delete them.
var=$(printf '%s\n%s\n\n\n' 'foo' 'bar')
echo "$var"
output:
foo
bar
According to help printf
printf [-v var] format [arguments]
If the -v option is supplied, the output is placed into the value of the shell variable VAR rather than being sent to the standard output.
In this case, for safe copying of formatted text to the variable, use the [-v var] option:
printf -v var '%s\n%s\n\n\n' 'foo' 'bar'
echo "$var"
output:
foo
bar
Works ok if you add "\r"
$ nl=`printf "\n\r"` && echo "1${nl}2"
1
2