I want to remove white spaces (\t, \n, \r, space) form the beginning and the end of a string if they exist
How to do that?
Is it possibe to that only with expressions like ${str#*}?
If you're using bash (which your idea of ${str#} seems to suggest), then you can use this:
echo "${str##[[:space:]]}" # trim all initial whitespace characters
echo "${str%%[[:space:]]}" # trim all trailing whitespace characters
You can say
sed -e 's/^[ \t\r\n]*//' -e 's/[ \t\r\n]*$//' <<< "string"
# ^^^^^^^^^^^ ^^^^^^^^^^
# beginning end of string
Or use \s to match tab and space if it is supported by your sed version.
If you can use sed then:
echo "${str}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
Related
#/!bin/sh
# infile is "person's clothes"
infile=$1
outfile=$2
sed -z "s|'|\\\\'|g;" <$infile >temp.txt
txt=`cat temp.txt`
echo $txt
# displays person\'s clothes
sed -e "s|paste_area|$txt|" blank.html >"$outfile"
# the relevant part of $outfile is "person's clothes" expecting "person\'s clothes"
"please add some context to explain the code sections" please read the code it has comments included
You need to double all the backslashes in $txt to make them literal when used in the sed replacement string. Otherwise they'll be treated as the escape prefix.
sed -e "s|paste_area|${txt/\\/\\\\}|" blank.html >"$outfile"
I have the below bash script:
STR1="US-1234 US-7685 TKT-008953"
#STR2= "${STR1// /,}"
STR2=`echo $STR1 | sed 's/ /,/g'`
echo $STR2
Current output: US-1234,US-7685,TKT-008953
Expected output: 'US-1234','US-9754','TKT-007643'
With bash and its parameter expansion:
STR1="US-1234 US-7685 TKT-008953"
STR1="${STR1// /\',\'}"
STR1="${STR1/#/\'}"
echo "${STR1/%/\'}"
Output:
'US-1234','US-7685','TKT-008953'
You may use
STR2="'$(echo "$STR1" | sed "s/ /','/g")'"
See online demo
All spaces are replaced with ',' using sed "s/ /','/g", and the initial and trailing single quotes are added inside a double quoted string.
$ echo 'US-1234 US-7685 TKT-008953' | sed -E "s/^|$/'/g; s/ /','/g"
'US-1234','US-7685','TKT-008953'
$ # can also use \x27 and continue using single quotes for the expression
$ echo 'US-1234 US-7685 TKT-008953' | sed -E 's/^|$/\x27/g; s/ /\x27,\x27/g'
'US-1234','US-7685','TKT-008953'
s/^|$/'/g will add single quote at start/end of line
s/ /','/g will replace space with ','
Use bash's global variable replacement to replace space with ',' and add quotes around it:
$ str2=\'${str1// /\',\'}\'
$ echo $str2
'US-1234','US-7685','TKT-008953'
Bear with me at first, thank you. Suppose I have
$ echo $'foo\nbar'
foo
bar
Now when I assign the string to a Bash variable, Bash does not give the same vertical output anymore:
$ str='foo\nbar'
$
$ echo $str
foo\nbar
$
$ echo $'str'
str
Try printf:
$ printf "$str\n"
foo
bar
Those examples are for illustration purposes because I am looking for a way to expand the newline(s) inside the $str variable such that I can substitute the $str variable on sed replacement (insertion) side.
# this does not work:
sed -i.bak $'/<!-- insert here -->/i\\\n'$'str'$'\\\n' index.html
# this works as expected though:
sed -i.bak $'/<!-- insert here -->/i\\\n'foo$'\\\n'bar$'\\\n' index.html
I did several ways to hack this but none worked; here is one example:
# this does not work:
sed -i.bak $'/<!-- insert here -->/i\\\n'`printf 'foo\\x0Abar'`$'\\\n' index.html
Further tests, I realized that as long as the variable does not contain newlines, things work as expected:
# This works as long as str2 does not contain any newline.
str2='foo_bar'
sed -i.bak $'/<!-- insert here -->/i\\\n'$str2$'\\\n' index.html
The expected result is that sed will insert 2 liners in place before <!-- insert here --> of the index.html file.
foo
bar
<!-- insert here -->
I try to achieve this as one liner. I know I can break sed into the vertical, multi-line form, which will be easier for me; however, I want to explore if there is a one liner style.
Is this doable or not?
My system is macOS High Sierra 10.13.6
Bash version: 3.2.57(1)-release
BSD sed was last updated on May 10, 2005
Your examples have a few subtle error, so here are a few examples regarding quoting and newlines in strings in bash and sed.
How quoting works in general:
# bash converts escape-sequence '\n' to real newline (0x0a) before passing it to echo
$ echo $'foo\nbar'
foo
bar
# bash passes literal 8 characters 'foo\nbar' to echo and echo simply prints them
$ echo 'foo\nbar'
foo\nbar
# bash passes literal 8 characters 'foo\nbar' to echo and echo converts escape-sequence
$ echo -e 'foo\nbar'
foo
bar
# bash passes literal string 'foo\nbar' to echo (twice)
# then echo recombines both arguments using a single space
$ str='foo\nbar'
$ echo $str "$str"
foo\nbar foo\nbar
# bash interprets escape-sequences and stores result 'foo<0x0a>bar' in str,
# then passes two arguments 'foo' and 'bar' to echo, due to "word splitting"
# then echo recombines both arguments using a single space
$ str=$'foo\nbar'
$ echo $str
foo bar
# bash interprets escape-sequences and stores result 'foo<0x0a>bar' in str,
# then passes it as a single argument to echo, without "word splitting"
$ str=$'foo\nbar'
$ echo "$str"
foo
bar
How to apply shell quoting, when dealing with newlines in sed
# replace a character with newline, using newline's escape-sequence
# sed will convert '\n' to a literal newline (0x0a)
$ sed 's/-/foo\nbar/' <<< 'blah-blah'
# replace a character with newline, using newline's escape-sequence in a variable
# sed will convert '\n' to a literal newline (0x0a)
$ str='foo\nbar' # str contains the escape-sequence '\n' and not a literal newline
$ sed 's/-/'"$str"'/' <<< 'blah-blah'
# replace a character with newline, using a literal newline.
# note the line-continuation-mark \ after 'foo' before the literal newline,
# which is part of the sed script, since everything in-between '' is literal
$ sed 's/-/foo\
bar/' <<< 'blah-blah' # end-of-command
# replace a character with newline, using a newline in shell-escape-mode
# note the same line-continuation-mark \ before $'\n', which is part of the sed script
# note: the sed script is a single string composed of three parts '…\', $'\n' and '…',
$ sed 's/-/foo\'$'\n''bar/' <<< 'blah-blah'
# the same as above, but with a single shell-escape-mode string instead of 3 parts.
# note the required quoting of the line-continuation-mark with an additional \ escape
# i.e. after shell-escaping the sed script contains a single \ and a literal newline
$ sed $'s/-/foo\\\nbar/' <<< 'blah-blah'
# replace a character with newline, using a shell-escaped string in a variable
$ str=$'\n' # str contains a literal newline (0x0a) due to shell escaping
$ sed 's/-/foo\'"$str"'bar/' <<< 'blah-blah'
# same as above with the required (quoted) line-continuation inside the variable
# note, how the single \ from '…foo\' (previous example) became \\ inside $'\\…'
$ str=$'\\\n' # str contains \ and a literal newline (0x0a) due to shell escaping
$ sed 's/-/foo'"$str"'bar/' <<< 'blah-blah'
All the sed examples will print the same:
blahfoo
barblah
So, a newline in sed's replacement string must either be
(1) newline's escape-sequence (i.e. '\n'), so sed can replace it with a literal newline, or
(2) a literal newline preceded by a line-continuation-mark (i.e. $'\\\n' or '\'$'\n', which is NOT the same as '\\\n' or '\\n' or $'\\n').
This means you need to replace each literal newline <0x0a> with newline's escape-sequence \n or insert a line-continuation-mark before each literal newline inside your replacement string before double-quote-expanding it into sed's substitute replacement string.
Since there are many more caveats regarding escaping in sed, I recommend you use awk's gsub function instead passing your replacement string as a variable via -v, e.g.
$ str=$'foo\nbar'
$ awk -v REP="$str" -- '{gsub(/-/, REP); print}' <<< 'blah-blah'
blahfoo
barblah
PS: I don't know, if this answer is entirely true in your case, because your operating system uses an outdated version of bash.
echo -e $str
where -e is
enable interpretation of backslash escapes
Use sed command r to insert arbitrary text
str="abc\ndef"
tmp=$(mktemp)
(
echo
printf -- "$str"
echo
) > "$tmp"
sed -i.bak '/<!-- insert here -->/r '"$tmp" index.html
rm -r "$tmp"
sed interprets newline as command delimiter. The ; doesn't really is a seds command delimeter, only newline is. Don't append/suffix ; or } or spaces in the w command - it will be interpreted as part of the filename (yes, spaces also). sed commands like w or r are escaped by a newline.
If you want more flexibility, rather move to awk.
I'm using GNU bash, version 4.4.19 to match to a line in a multi-line string (which I'm reading from a file).
File in.txt:
abc/def/
bar/foo/x
foobar/foo/y
foobar/quux/
In this file, with a pattern like ^bar/foo/.*$, I'm trying to match bar/foo/ (and not foobar/foo/y).
But since it's a multiline string, ^ and $ will not match each line but rather all of the string. Therefore I'm trying to use \b (word boundary) in my regexp.
This is what I'm trying but it's not working.:
in="$(cat in.txt)"
re=\\bbar/foo/.*\\b
[[ "$in" =~ $re ]] && echo OK
Other patterns I tried and didn't work:
re=\bbar/foo/.*\n
re=\\bbar/foo/.*\\n
re=\\bbar\/foo\/.*\\b
re=\\bbar\/foo\/.*\\n
re=\\bbar\/foo\/\(.*\)\\n
bash uses POSIX.2 regular expressions for =~ (see man 7 regex) as described here
... a '\' followed by
one of the characters "^.[$()|*+?{\" (matching that character taken as
an ordinary character), a '\' followed by any other character(!)
(matching that character taken as an ordinary character, as if the '\'
had not been present(!)), or a single character with no other signifi‐
cance (matching that character).
and they don't support \b as word boundary delimiter.
Your best option is to match line by line
ii=( $(< in.txt) )
for l in "${ii[#]}"
do
[[ "$l" =~ $re ]] && echo "match: $l"
done
Grep's P option supports word boundaries \b. So with GNU's grep you could do:
in="$(cat in.txt)"
re="\bbar/foo/[^ ]*"
grep -oP "$re" <<< $in
The output is:
bar/foo/x
You can use awk to match the wanted line:
awk '/^bar\/foo\/*/{print $0}' input_file
Hope this helps.
How to remove extra spaces in variable HEAD?
HEAD=" how to remove extra spaces "
Result:
how to remove extra spaces
Try this:
echo "$HEAD" | tr -s " "
or maybe you want to save it in a variable:
NEWHEAD=$(echo "$HEAD" | tr -s " ")
Update
To remove leading and trailing whitespaces, do this:
NEWHEAD=$(echo "$HEAD" | tr -s " ")
NEWHEAD=${NEWHEAD%% }
NEWHEAD=${NEWHEAD## }
Using awk:
$ echo "$HEAD" | awk '$1=$1'
how to remove extra spaces
Take advantage of the word-splitting effects of not quoting your variable
$ HEAD=" how to remove extra spaces "
$ set -- $HEAD
$ HEAD=$*
$ echo ">>>$HEAD<<<"
>>>how to remove extra spaces<<<
If you don't want to use the positional paramaters, use an array
ary=($HEAD)
HEAD=${ary[#]}
echo "$HEAD"
One dangerous side-effect of not quoting is that filename expansion will be in play. So turn it off first, and re-enable it after:
$ set -f
$ set -- $HEAD
$ set +f
This horse isn't quite dead yet: Let's keep beating it!*
Read into array
Other people have mentioned read, but since using unquoted expansion may cause undesirable expansions all answers using it can be regarded as more or less the same. You could do
set -f
read HEAD <<< $HEAD
set +f
or you could do
read -rd '' -a HEAD <<< "$HEAD" # Assuming the default IFS
HEAD="${HEAD[*]}"
Extended Globbing with Parameter Expansion
$ shopt -s extglob
$ HEAD="${HEAD//+( )/ }" HEAD="${HEAD# }" HEAD="${HEAD% }"
$ printf '"%s"\n' "$HEAD"
"how to remove extra spaces"
*No horses were actually harmed – this was merely a metaphor for getting six+ diverse answers to a simple question.
Here's how I would do it with sed:
string=' how to remove extra spaces '
echo "$string" | sed -e 's/ */ /g' -e 's/^ *\(.*\) *$/\1/'
=> how to remove extra spaces # (no spaces at beginning or end)
The first sed expression replaces any groups of more than 1 space with a single space, and the second expression removes any trailing or leading spaces.
echo -e " abc \t def "|column -t|tr -s " "
column -t will:
remove the spaces at the beginning and at the end of the line
convert tabs to spaces
tr -s " " will squeeze multiple spaces to single space
BTW, to see the whole output you can use cat - -A: shows you all spacial characters including tabs and EOL:
echo -e " abc \t def "|cat - -A
output: abc ^I def $
echo -e " abc \t def "|column -t|tr -s " "|cat - -A
output:
abc def$
Whitespace can take the form of both spaces and tabs. Although they are non-printing characters and unseen to us, sed and other tools see them as different forms of whitespace and only operate on what you ask for. ie, if you tell sed to delete x number of spaces, it will do this, but the expression will not match tabs. The inverse is true- supply a tab to sed and it will not match spaces, even if the number of them is equal to those in a tab.
A more extensible solution that will work for removing either/both additional space in the form of spaces and tabs (I've tested mixing both in your specimen variable) is:
echo $HEAD | sed 's/^[[:blank:]]*//g'
or we can tighten-up #Frontear 's excellent suggestion of using xargs without the tr:
echo $HEAD | xargs
However, note that xargs would also remove newlines. So if you were to cat a file and pipe it to xargs, all the extra space- including newlines- are removed and everything put on the same line ;-).
Both of the foregoing achieved your desired result in my testing.
Try this one:
echo ' how to remove extra spaces ' | sed 's/^ *//g' | sed 's/$ *//g' | sed 's/ */ /g'
or
HEAD=" how to remove extra spaces "
HEAD=$(echo "$HEAD" | sed 's/^ *//g' | sed 's/$ *//g' | sed 's/ */ /g')
I would make use of tr to remove the extra spaces, and xargs to trim the back and front.
TEXT=" This is some text "
echo $(echo $TEXT | tr -s " " | xargs)
# [...]$ This is some text
echo variable without quotes does what you want:
HEAD=" how to remove extra spaces "
echo $HEAD
# or assign to new variable
NEW_HEAD=$(echo $HEAD)
echo $NEW_HEAD
output: how to remove extra spaces