How to remove escape char from string in bash using sed? - bash

I need to remove escape character from the string in bash. I get a data structure, which contains url paths with / escaped so I receive the regular link:
http://stackoverflow.com/questions/ask
as one with escaped /:
http:\/\/stackoverflow.com\/questions\/ask
Now I need to remove \ from the second link. For this purpose I tried using sed
`echo '"'${paths[$index]}'"' | sed "s#\\##g"`
But I get an error:
sed: -e expression #1, char 6: unterminated `s' command
If I replace \\ with ie. _ it works like a charm and removes all occurrences of _ in a string. How do I get rid of escape characters in a string using sed?

try this:
.......|sed 's#\\##g'
or:
.......|sed "s#\\\\##g"
EDIT add a test output:
kent$ echo "http:\/\/stackoverflow.com\/questions\/ask"|sed "s#\\\\##g"
http://stackoverflow.com/questions/ask
kent$ echo "http:\/\/stackoverflow.com\/questions\/ask"|sed 's#\\##g'
http://stackoverflow.com/questions/ask

Your question isn't clear about which way round you want so here is both ways:
$ sed 's#/#\\/#g' <<< "http://stackoverflow.com/questions/ask"
http:\/\/stackoverflow.com\/questions\/ask
$ sed 's#\\/#/#g' <<< "http:\/\/stackoverflow.com\/questions\/ask"
http://stackoverflow.com/questions/ask

You don't need to use sed.
paths[index]=${paths[index]//\\/}
or simply
echo ${paths[index]//\\/}
to see the result without modifying the value in-place.

You can use this :
sed 's#\\##g'
But the problem is when you encounter a backslash that you actually want in the string, but is escaped. In that case :
sed 's/\\\\/\x1/' |sed 's/[\]//g' | sed 's/\x1/\\/g'
Replaces the double backslash with with a temp character[SOH], replaces all other backslashes and then restores the backslash that is needed.

Related

Insert the contents of the variable in SED command [duplicate]

If I run these commands from a script:
#my.sh
PWD=bla
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
xxx
bla
it is fine.
But, if I run:
#my.sh
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
$ sed: -e expression #1, char 8: Unknown option to `s'
I read in tutorials that to substitute environment variables from shell you need to stop, and 'out quote' the $varname part so that it is not substituted directly, which is what I did, and which works only if the variable is defined immediately before.
How can I get sed to recognize a $var as an environment variable as it is defined in the shell?
Your two examples look identical, which makes problems hard to diagnose. Potential problems:
You may need double quotes, as in sed 's/xxx/'"$PWD"'/'
$PWD may contain a slash, in which case you need to find a character not contained in $PWD to use as a delimiter.
To nail both issues at once, perhaps
sed 's#xxx#'"$PWD"'#'
In addition to Norman Ramsey's answer, I'd like to add that you can double-quote the entire string (which may make the statement more readable and less error prone).
So if you want to search for 'foo' and replace it with the content of $BAR, you can enclose the sed command in double-quotes.
sed 's/foo/$BAR/g'
sed "s/foo/$BAR/g"
In the first, $BAR will not expand correctly while in the second $BAR will expand correctly.
Another easy alternative:
Since $PWD will usually contain a slash /, use | instead of / for the sed statement:
sed -e "s|xxx|$PWD|"
You can use other characters besides "/" in substitution:
sed "s#$1#$2#g" -i FILE
一. bad way: change delimiter
sed 's/xxx/'"$PWD"'/'
sed 's:xxx:'"$PWD"':'
sed 's#xxx#'"$PWD"'#'
maybe those not the final answer,
you can not known what character will occur in $PWD, / : OR #.
if delimiter char in $PWD, they will break the expression
the good way is replace(escape) the special character in $PWD.
二. good way: escape delimiter
for example:
try to replace URL as $url (has : / in content)
x.com:80/aa/bb/aa.js
in string $tmp
URL
A. use / as delimiter
escape / as \/ in var (before use in sed expression)
## step 1: try escape
echo ${url//\//\\/}
x.com:80\/aa\/bb\/aa.js #escape fine
echo ${url//\//\/}
x.com:80/aa/bb/aa.js #escape not success
echo "${url//\//\/}"
x.com:80\/aa\/bb\/aa.js #escape fine, notice `"`
## step 2: do sed
echo $tmp | sed "s/URL/${url//\//\\/}/"
URL
echo $tmp | sed "s/URL/${url//\//\/}/"
URL
OR
B. use : as delimiter (more readable than /)
escape : as \: in var (before use in sed expression)
## step 1: try escape
echo ${url//:/\:}
x.com:80/aa/bb/aa.js #escape not success
echo "${url//:/\:}"
x.com\:80/aa/bb/aa.js #escape fine, notice `"`
## step 2: do sed
echo $tmp | sed "s:URL:${url//:/\:}:g"
x.com:80/aa/bb/aa.js
With your question edit, I see your problem. Let's say the current directory is /home/yourname ... in this case, your command below:
sed 's/xxx/'$PWD'/'
will be expanded to
sed `s/xxx//home/yourname//
which is not valid. You need to put a \ character in front of each / in your $PWD if you want to do this.
Actually, the simplest thing (in GNU sed, at least) is to use a different separator for the sed substitution (s) command. So, instead of s/pattern/'$mypath'/ being expanded to s/pattern//my/path/, which will of course confuse the s command, use s!pattern!'$mypath'!, which will be expanded to s!pattern!/my/path!. I’ve used the bang (!) character (or use anything you like) which avoids the usual, but-by-no-means-your-only-choice forward slash as the separator.
Dealing with VARIABLES within sed
[root#gislab00207 ldom]# echo domainname: None > /tmp/1.txt
[root#gislab00207 ldom]# cat /tmp/1.txt
domainname: None
[root#gislab00207 ldom]# echo ${DOMAIN_NAME}
dcsw-79-98vm.us.oracle.com
[root#gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: ${DOMAIN_NAME}/g'
--- Below is the result -- very funny.
domainname: ${DOMAIN_NAME}
--- You need to single quote your variable like this ...
[root#gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: '${DOMAIN_NAME}'/g'
--- The right result is below
domainname: dcsw-79-98vm.us.oracle.com
VAR=8675309
echo "abcde:jhdfj$jhbsfiy/.hghi$jh:12345:dgve::" |\
sed 's/:[0-9]*:/:'$VAR':/1'
where VAR contains what you want to replace the field with
I had similar problem, I had a list and I have to build a SQL script based on template (that contained #INPUT# as element to replace):
for i in LIST
do
awk "sub(/\#INPUT\#/,\"${i}\");" template.sql >> output
done
If your replacement string may contain other sed control characters, then a two-step substitution (first escaping the replacement string) may be what you want:
PWD='/a\1&b$_' # these are problematic for sed
PWD_ESC=$(printf '%s\n' "$PWD" | sed -e 's/[\/&]/\\&/g')
echo 'xxx' | sed "s/xxx/$PWD_ESC/" # now this works as expected
for me to replace some text against the value of an environment variable in a file with sed works only with quota as the following:
sed -i 's/original_value/'"$MY_ENVIRNONMENT_VARIABLE"'/g' myfile.txt
BUT when the value of MY_ENVIRONMENT_VARIABLE contains a URL (ie https://andreas.gr) then the above was not working.
THEN use different delimiter:
sed -i "s|original_value|$MY_ENVIRNONMENT_VARIABLE|g" myfile.txt

Bash: special characters in sed

I want to replace with sed in some bash script something like:
s:44:\"STRING\"
To:
s:NEWSTRING:\"NEWSTRING2\"
I tried many ways with escaping special characters, but I got always error
sed: -e expression #1, char 32: unterminateds' command`
or someting like that.
Can you please tell me the correct sed -i (sed -i "s/xxx/xxx/g" file) command for that?
You have to escape the backslashes properly:
sed 's/s:44:\\"STRING\\"/s:NEWSTRING:\\"NEWSTRING2\\"/'
Example:
$ echo 's:44:\"STRING\"' | sed 's/s:44:\\"STRING\\"/s:NEWSTRING:\\"NEWSTRING2\\"/g'
s:NEWSTRING:\"NEWSTRING2\"
You are missing the final delimiter. In your case it seems to be : therefore, you need to add a final : after your substitution content. It does not matter if you are using modification instruction like g

Unix Remove all occurences of character and save

How to remove all occurences of string "???" of a file and save it?
My approach so far:
cat file.txt | sed -ie '/s/???//' file.txt
However I get the following error:
sed: -e expression #1, char 4: unknown command: `?'
You can use this sed command:
sed -i 's/???//g' file.txt
There is no reason to use cat here as sed can directly operate on a file and save it in-line.
Also note that unlike other regex flavors BRE (Basic Regular Expressions) which is default regex engine of sed doesn't treat ? as a special regex meta character hence there is no need to escape ? here.

How to pass special characters through sed

I want to pass this command in my script:
sed -n -e "/Next</a></p>/,/Next</a></p>/ p" file.txt
This command (should) extract all text between the two matched patterns, which are both Next</a></p> in my case. However when I run my script I keep getting errors. I've tried:
sed -n -e "/Next\<\/a\>\<\/p\>/,/Next<\/a\>\<\/p>/ p" file.txt with no luck.
I believe the generic pattern for this command is this:
sed -n -e "/pattern1/,/pattern2/ p" file.txt
I can't get it working for Next</a></p> though and I'm guessing it has something to do with the special characters I am encasing. Is there any way to pass Next</a></p> in the sed command? Thanks in advance guys! This community is awesome!
You don't need to use / as a regular expression delimiter. Using a different character will make quoting issues slightly easier. The syntax is
\cregexc
where c can be any character (other than \) that you don't use in the regex. In this case, : might be a good choice:
sed -n -e '\:Next</a></p>:,\:Next</a></p>: p' file.txt
Note that I changed " to ' because inside double quotes, \ will be interpreted by bash as an escape character, whereas inside single quotes \ is just treated as a regular character. Consequently, you could have written the version with escaped slashes like this:
sed -n -e '/Next<\/a><\/p>/,/Next<\/a><\/p>/ p' file.txt
but I think the version with colons is (slightly) easier to read.
You need to escape the forward slashes inside the regular expressions with a \, since the forward slashes serve as delimiters for the regexes
sed -n -e '/Next<\/a><\/p>/,/Next<\/a><\/p>/p' file.txt

Using variable with sed

First of all i apologise in case this has been answered before but i couldn't solve my problem.
I need to search a pattern and then replace it with a line of text comprising of both text and variable.Btw i am using bash..
say
$var = "stacko.ver/rulz=" **Note: $var contain double quotes & = & a dot and /**
i want to so the follow
1.Search for ;te.xt = Note: The value to be search contain ; & = and a dot
2.Replace it with
textnum=$var
Of course $var should be replaced with its actual value
My attempts
sed -i "s/;te.xt =/textnum=$var/" file
sed -i "s/;te.xt =/textnum="$var"/" file
sed -i "s/";te.xt ="/"textnum=$var"/" file
None of these actually worked , either sed giving me an error or the value of $var not shown in file
Thanks for the help
Regards
Quoting doesn't help since this is a sed issue, not a bash issue. Just pick a sed s-expression delimiter that doesn't appear in your text:
sed -i "s|;te.xt =|textnum=$var|" file
You can pick any delimiter for s that doesn't appear in your input. sed -e 'streetlight' is a perfectly valid sed command.
I can see the error:
$ var="stacko.ver/rulz="
$ data="foo ;te.xt = bar"
$ sed "s/;te.xt =/textnum=$var/" <<< "$data"
sed: -e expression #1, char 31: unknown option to `s'
The problem is that $var contains a slash, so sed's s/// command is breaking. You need to pick a character that does not appear in $var
$ sed "s#;te.xt =#textnum=$var#" <<< "$data"
foo textnum=stacko.ver/rulz= bar
This can be hard -- what if slash and hash are in $var? Using bash, you can use ANSI-C quoting to use a control character that is unlikely to appear in your data, e.g.
$ sed $'s\037;te.xt =\037textnum=$var\037' <<< "$data"
foo textnum=stacko.ver/rulz= bar

Resources