This question already has answers here:
Is it possible to escape regex metacharacters reliably with sed
(4 answers)
Closed 5 years ago.
I have replaced urls in the past using sed with no problem before. However, this url imparticular is giving me trouble. It has quite a few ampersands and I need to replace them. How would I go about doing that?
sed -i.bak "s#<string>https://www.url1toreplace.com?blah=1234585474738743874386328764287364238746283764287346872364fN&blah=Y&blah=%2Fwebapp%2Fwcs%2Fblahblah%2Fblah%2Fen%2Fblahahah%3Fblah%3e212e123152%26cm_mmc%3DBLAH-_-BLAH-_-Null-_-Null</string>#<string>https://www.urltoreplace.com/blah/blah/blah/blah/en/blah?blah=129i312093132&cm_mmc=BLAH-_-BLAH-_-Null-_-Null</string>#g" path/to/xml/file
My problem is that it's not fully replacing the url. How do I escape the ampersands so I can successfully replace www.url1toreplace.com with www.urltoreplace.com and everything that follows?
In the replacement text, you need to escape &.
For example, without the escape, the whole of the original match is substituted in for each &:
$ echo '&' | sed 's#&#a & b & c#'
a & b & c
With the escape, \&, & is treated as an ordinary character:
$ echo '&' | sed 's#&#a \& b \& c#'
a & b & c
Your example
Let's take this test file:
$ cat file
<string>https://www.url1toreplace.com?blah=1234585474738743874386328764287364238746283764287346872364fN&blah=Y&blah=%2Fwebapp%2Fwcs%2Fblahblah%2Fblah%2Fen%2Fblahahah%3Fblah%3e212e123152%26cm_mmc%3DBLAH-_-BLAH-_-Null-_-Null</string>
And run the original command:
$ sed "s#<string>https://www.url1toreplace.com?blah=1234585474738743874386328764287364238746283764287346872364fN&blah=Y&blah=%2Fwebapp%2Fwcs%2Fblahblah%2Fblah%2Fen%2Fblahahah%3Fblah%3e212e123152%26cm_mmc%3DBLAH-_-BLAH-_-Null-_-Null</string>#<string>https://www.urltoreplace.com/blah/blah/blah/blah/en/blah?blah=129i312093132\&cm_mmc=BLAH-_-BLAH-_-Null-_-Null</string>#g" file
<string>https://www.urltoreplace.com/blah/blah/blah/blah/en/blah?blah=129i312093132&cm_mmc=BLAH-_-BLAH-_-Null-_-Null</string>
The above command fails. If we escape the &, however, we get:
$ sed 's#<string>https://www.url1toreplace.com?blah=1234585474738743874386328764287364238746283764287346872364fN&blah=Y&blah=%2Fwebapp%2Fwcs%2Fblahblah%2Fblah%2Fen%2Fblahahah%3Fblah%3e212e123152%26cm_mmc%3DBLAH-_-BLAH-_-Null-_-Null</string>#<string>https://www.urltoreplace.com/blah/blah/blah/blah/en/blah?blah=129i312093132\&cm_mmc=BLAH-_-BLAH-_-Null-_-Null</string>#g' file
<string>https://www.urltoreplace.com/blah/blah/blah/blah/en/blah?blah=129i312093132&cm_mmc=BLAH-_-BLAH-_-Null-_-Null</string>
This succeeds: the & in the replacement string successfully appears in the output.
Sample data file:
$ cat xfile
<string>https://www.old.home.com?x=123&y=abc&z=ABC_mmc%3D</string>
Desired output:
<string>https://www.new.home.biz?A=XYZ&B=123&C=987_jjj%2XD</string>
As John1024's already pointed out, if a sed replacement string contains &'s, the &'s have to be escaped (\&) (because & has a special meaning to sed).
Hmmmm, but that could be a major pain in the keister if ya gotta go through and (manually?) change all sed replacement patterns from & to \&. But this replacement can be automated with a few minor assumptions ...
Assumptions:
search and replace patterns can be stored in variables before and after, respectively (actually, only the after variable is needed for this idea to work, but for this example I'll use before and after variables)
before and after contain normal strings w/out any special escapes
your version of bash supports character replacement via the ${var// /} construct
Apply escapes to the after variable on the fly:
$ before='old.home.com?x=123&y=abc&z=ABC_mmc%3D'
$ after='new.home.biz?A=XYZ&B=123&C=987_jjj%2XD'
$ sed "s#${before}#${after//\&/\\\&}#g" xfile
<string>https://www.new.home.biz?A=XYZ&B=123&C=987_jjj%2XD</string>
${after//\&/\\\&} : in the after variable, replace all occurrences of & with \&
This eliminates the need to go through and manually escape all occurrences of & in the replacement string.
Related
I have the following string variable defined in bash as version
Uploaded registry/version1.3.1
I need to escape the / twice so that the string looks like:
registry\\/version1.3.1
Is there a way to use sed to find and replace all / to \\/?
One backslash is used to change an argument to text or vice versa.
echo 'Uploaded registry/version1.3.1' | sed 's/\//\\\\\//g'
Uploaded registry\\/version1.3.1
In this case you need \\ for one backslash and \/for a slash.
Using sed
You need two backslashes, \\, in the RHS (right-hand side) of the sed command to replaces with a single backslash.
Also, it is a good idea to use a character other than / as a regex delimiter to avoid overescaping.
You can thus use
sed 's,/,\\\\/,g'
See this online demo:
echo 'Uploaded registry/version1.3.1' | sed 's,/,\\\\/,g'
# => Uploaded registry\\/version1.3.1
Using string manipulation
You may also just use string manipulation in Bash:
s='Uploaded registry/version1.3.1'
echo "${s//\//\\\\\/}"
# => Uploaded registry\\/version1.3.1
Here, the backslash needs doubling again, and a single \ is added to escape /s. Note that "${s//a/b} means: replace all occurrences of a with b in s variable.
This question already has answers here:
How to escape single quote in sed?
(9 answers)
Closed 2 years ago.
I want to replace a set of characters with ' using sed.
This post suggest:
With single quotes around the argument (sed 's/…/…/'), use '\'' to put a single quote in the replacement text.
So, I tried following:
echo 'abcd' | sed 's/[abcd]/\'/g'
But it simply ends up expecting more input:
anir#DESKTOP-4856511:~$ echo 'abcd' | sed 's/[abcd]/\'/g'
>
>
>
> ^C
When I copy pasted echo 'abcd' | sed 's/[abcd]/\'/g' in .sh file and ran, it gave me following error:
anir#DESKTOP-4856511:~/Mahesha999/delete$ ./trysed.sh
./trysed.sh: line 1: unexpected EOF while looking for matching `''
./trysed.sh: line 2: syntax error: unexpected end of file
What the right way to do this? Is it impossible to escape single quote inside single quoted string (and I have to use double quotes only as explained here)?
As the post says:
With single quotes around the argument (sed 's/…/…/'), use '\'' to put a single quote in the replacement text.
So, using your example, you would do:
echo 'abcd' | sed 's/[abcd]/'\''/g'
If you want to replace with just one single quote:
echo 'abcd' | sed 's/[abcd][abcd]*/'\''/g'
The shell does not allow single quotes inside a single quoted string. What the code above does is create three strings (which are not separated by anything):
single-quoted string: 's/[abcd]/'
unquoted string containing just an escaped single-quote: \'
single-quoted string: '/g'
The shell then expands them and because they are not separated, they effectively become joined into a single string.
sed sees: s/[abcd]/'/g
I have to variables in a bash script:
$string = "The cat is green.\n"
$line = "Sunny day today.\n"
each of those variables contain "\n" character, how can I use sed to search and replace:
sed 's/$string/$line/g' file.txt
This doesn't seem to work, if I erase the "\n" from the strings sed works properly.
If I had only the text I could escape "\n" by adding a backslash:
sed 's/"The cat is green.\\n"/"Sunny day today.\\n"/g' file.txt
How can I manage to do search/replace when variables contain "\n" in them.
Thank you for the help.
It looks like you are trying to match the two-character sequence \n, as opposed to the single newline character that together they represent in some contexts. There is a tremendous difference between these.
As part of your example, you presented
sed 's/$string/$line/g' file.txt
, but that won't work at all, because variable references are not expanded within single-quoted strings. That has nothing whatever to do with the values of shell variables string and line.
But let's consider those values:
$string="The cat is green.\n"
$line="Sunny day today.\n"
[Extra spaces removed.]
Of course, the problem you're focusing on is that sed recognizes \n as a code for a newline character, but you also have the problem that in a regular expression, the . character matches any character, so if you want it to be treated as a literal then it, too, needs to be escaped (in the pattern, but not in the replacement). If you're trying to support search and replace for arbitrary text, then there are other characters you'll need to escape, too.
Answering the question as posed (escaping only \n sequences) you might do this:
sed "s/${string//\\n/\\\\n}/${line//\\n/\\\\n}/g"
The ${foo//pat/repl} form of parameter expansion performs pattern substitution on the expanded value, but note well that the pattern (pat) is interpreted according to shell globbing rules, not as a regular expression. That specific form replaces every appearance of the pattern; read the bash manual for alternatives that match only the first appearance and/or that match only at the beginning or the end of the parameter's value. Note, too, the extra doubling of the \ characters in the pattern substitution -- they need to be escaped for the shell, too.
Given your variable definitions, that command would be equivalent to this:
sed 's/The cat is green.\\n/Sunny day today.\\n/g'
In other words, exactly what you wanted. Again, however, be warned: that is not a general solution for arbitrary search & replace. If you want that, then you'll want to study the sed manual to determine which characters need to be escaped in the regex, and which need to be escaped in the replacement. Moreover, I don't see a way to do it with just one pattern substitution for each variable.
I want to replace some strings with other strings in a folder containing .ly files (.ly files are files used to write music, which can then be converted to svg files). One problem is that the files contain backslashes (\).
Anyway, what I want to do is to replace the following strings:
\f with \fffff
\p with \ppppp
mordent with reverseturn
r4 with \mordent
Each string I want to replace (and each string it is replaced for) has a space in the end. This is because I don't want strings like
\pt to be replaced with \pppppt
Similarly, because backslashes matter, strings like
\ff should stay the same (only f with a slash before and a space after should be replaced).
The code that I am using to achieve this is:
for f in *.ly
do
filename="${f}"
sed -i -- "s#\p #\ppppp #g; s#\f #\fffff #g; s#mordent #reverseturn #g; s#r4 #\maxima #g" $filename;
done
I am using delimited # instead of backslash exactly to achieve this.
However, the program is giving some totally weird behavior. In particular:
\f stays the same (wrong behavior, the right behavior is to get transformed to \fffff)
\p is converted to \ppppp (correct behavior)
\stemUp is converted to \stemUppppp (wrong behavior, the right behavior is to remain as it was)
This is perplexing me and giving me a headache. In particular, why it works correctly for \p but not for \f considering that it is the exact same code.
For reference, I attached a file before and after applying the sed commands.
Thanks in advance for any help!
NB: I am using Ubuntu 16.04 and sed (GNU 4.2.2).
$ echo '12p3 p \p foo' | sed 's/\p /\ppppp /g'
12p3 ppppp \ppppp foo
$ echo '12p3 p \p foo' | sed 's/\\p /\\ppppp /g'
12p3 p \ppppp foo
\ is a meta-character, to match it literally, it needs to be escaped, i.e \\
\ is special within double quotes too(because of bash, nothing to do with sed) - so don't use double quotes unless needed(for ex: sed substitution with bash variables). See also Difference between single and double quotes in Bash
This question already has answers here:
Escape a string for a sed replace pattern
(17 answers)
sed search and replace strings containing / [duplicate]
(2 answers)
Closed 3 years ago.
With Bash and SED I'm trying to replace two strings in a js file with URL's.
The two urls that should be inserted is input params when I run the .sh script.
./deploy.sh https://hostname.com/a/index.html https://hostname2.com/test
However to make this usable in my sed command I have to escape all forward slashes with: \\ ?
./deploy.sh https:\\/\\/hostname.com\\/a\\/index.html https:\\/\\/hostname2.com\\/test
If they are escaped this SED command works on Mac OSX Sierra
APP_URL=$1
API_URL=$2
sed "s/tempAppUrl/$APP_URL/g;s/tempApiUrl/$API_URL/g" index.src.js > index.js
Now I don't want to insert escaped urls as params, I want the script it self to escape the forward slashes.
This is what I've tried:
APP_URL=$1
API_URL=$2
ESC_APP_URL=(${APP_URL//\//'\\/'})
ESC_API_URL=(${API_URL//\//'\\/'})
echo 'Escaped URLS'
echo $ESC_APP_URL
#Echos result: https:\\/\\/hostname.com\\/a\\/index.html
echo $ESC_API_URL
#Echos result: https:\\/\\/hostname2.com\\/test
echo "Inserting app-URL and api-URL before dist"
sed "s/tempAppUrl/$ESC_APP_URL/g;s/tempApiUrl/$ESC_API_URL/g" index.src.js > index.js
The params looks the same but in this case the SED throws a error
sed: 1: "s/tempAppUrl/https:\\/\ ...": bad flag in substitute command: '\'
Could anyone tell me the difference here? The Strings looks the same but gives different results.
I suggest to replace
sed "s/regex/replace/" file
with
sed "s|regex|replace|" file
if your sed supports it. Then it is no longer necessary to escape the slashes.
The character directly after the s determines which character is the separator, which must appear three times in the s command.