This question already has answers here:
Escape a string for a sed replace pattern
(17 answers)
Closed 1 year ago.
I have a script that passes a variable into a sed command like this:
sed "s-\t-&${SUBDIRECTORY}/-"
But if the variable contains - (dash), then the sed command throws an error.
So this script:
VARIABLE="test-variable"
sed "s-\t-&${VARIABLE}/-"
Results in this error:
sed: 1: "s-\t-&test-variable/-": bad flag in substitute command: 'v'
I have not been able to find any answers to this issue; it works fine without the -.
How can I fix this?
Use a shell parameter expansion that escapes each instance of -:
sed "s-\t-&${VARIABLE//-/\\-}/-"
In the Bash manual, under Shell Parameter Expansion:
${parameter/pattern/string}
The pattern is expanded to produce a pattern just as in filename expansion. Parameter is expanded and the longest match of pattern against its value is replaced with string. [...] If pattern begins with /, all matches of pattern are replaced with string. Normally only the first match is replaced. [...]
Proper escaping is a fairly difficult problem in the shell, but you could do something like:
$ variable="test-variable"
$ printf '\t\n' | v="$variable" perl -pe 's-\t-$ENV{v}-'
test-variable
I'm learning bash and I'm trying to parse a webpage(https://chromium-i18n.appspot.com/ssl-address) and extract the href o
f interest using sed. The pattern I'm using is:
/<a\shref=\'\/ssl-address\/data\/([^\"]*)\'>/siU
However, I cant get the expression to work with sed. When i run:
data=$(wget ${serviceUrl} -q -O -)
parsedData=$(sed '/<a\shref=\'\''\/ssl-address\/data\/([^\"]*)\'\''>/siU/' <<< ${data})
echo ${parsedData}
I get the following error:
sed: 1: "/<a\shref=\'\/ssl-addre ...": unterminated substitute pattern
What am I doing wrong?
Is this what you're trying to do?
$ wget 'https://chromium-i18n.appspot.com/ssl-address' -q -O - |
sed -n 's:.*/ssl-address/data/\([^'\'']*\).*:\1:p'
AC
AD
AD/Canillo
AD/Encamp
I see you're getting some answers using double quotes instead of single around your sed script so you can do "...'..." instead of '...'\''...' - though tempting and it'd function OK for this particular current example, don't do it. To avoid any surprises now or if/when your requirements change later, in all shell programming always enclose strings and scripts in single quotes unless you need to expose them to the shell for interpretation and then use double quotes unless you need the shell to do globbing and file name expansion on them and then use no quotes.
All right, you are trying to parse an entire webpage.
This situation require to delete all the lines you don't need.
As #Ed Morton said, you can use something else than sed.
Your webpage is this as you told us in a comment, so you first need do download it.
Note that the changing how you download the source of the page, you can change some thing (E.G. copy pasting it from the console of Firefox you will have href=", using wget you will have href=').
That said, let's use wget as you are currently doing in your question.
# This will create the ssl-address file
wget "https://chromium-i18n.appspot.com/ssl-address"
# This will give you a list of all of the links in a href.
sed -e "/<a href='.*/! d" -e "s/<a href='\/ssl-address\/data\/\(.*\)'.*/\1/" ssl-address
EDIT:
Reading your comments I saw you would like to filter some of the output (E.G. deleting all the examples link)
This can be done adding a piece of sed in order to delete lines you don't need.
In your case you just need to add -e "/<a href='\/ssl-address\/examples.*/d" so the whole line of code should be as follow:
sed -e "/<a href='.*/! d" -e "/<a href='\/ssl-address\/examples.*/d" -e "s/<a href='\/ssl-address\/data\/\(.*\)'.*/\1/" ssl-address
You probably want something like this, based on that input data:
sed -e "s/.*href='\([^']*\)'.*/\1/"
It says, "match anything .* followed by the literal characters href=' followed by anything other than the ' character [^']* (we capture using the \( ... \) notation) followed by the ' character followed by anything".
Note I used the " to enclose the sed expression, to avoid you having to quote the '.
I have a a some SNMP dump:
1.3.6.1.2.1.1.2.0|5|1.3.6.1.4.1.9.1.1178
1.3.6.1.2.1.1.3.0|7|1881685367
1.3.6.1.2.1.1.4.0|6|""
1.3.6.1.2.1.1.5.0|6|"hgfdhg-4365.gfhfg.dfg.com"
1.3.6.1.2.1.1.6.0|6|""
1.3.6.1.2.1.1.7.0|2|6
1.3.6.1.2.1.1.8.0|7|0
1.3.6.1.2.1.1.9.1.2.1|5|1.3.6.1.4.1.9.7.129
1.3.6.1.2.1.1.9.1.2.2|5|1.3.6.1.4.1.9.7.115
And need to grep all data in first string after 1.3.6.1.2.1.1.2.0|5|, but not include this start of the string in grep itself. So, I must receive 1.3.6.1.4.1.9.1.1178 in grep. I've tried to use regex:
\b1.3.6.1.2.1.1.2.0\|5\|\s*([^\n\r]*)
But without any success. If a regular expression, or grep, is in fact the right tool, can you help me find the right regex? Otherwise, what tools should I consider instead?
With GNU grep +PCRE support, you can use Perl's \K flag to discard part of the matched string :
grep -Po "1\.3\.6\.1\.2\.1\.1\.2\.0\|5\|\K.*"
-P enables Perl's regex mode and -o switches output to matched parts rather than whole lines.
I had to escape the characters that have special meaning in Perl regexs, but this can be avoided as 123 suggests, by enclosing the characters to interpret literally between \Q and \E :
grep -Po "\Q1.3.6.1.2.1.1.2.0|5|\E\K.*"
I would usually solve this with sed as follows :
sed -n 's/1\.3\.6\.1\.2\.1\.1\.2\.0|5|\(.*\)/\1/p'
The -n flag disables implicit output and the search and replace command will remove the searched prefix from the line, leaving the relevant part to be printed.
The characters that have special meaning in GNU Basic Regular Expressions (BRE) must be escaped, which in this case is only .. Also note that the grouping tokens are \( and \) rather than the usual ( and ).
An alternate way to do this is in native shell, without any regexes at all. Consider:
prefix='1.3.6.1.2.1.1.2.0|5|'
while read -r line; do
[[ $line = "$prefix"* ]] && printf '%s\n' "${line#$prefix}"
done
If your original string is piped into the while read loop, the output is precisely 1.3.6.1.4.1.9.1.1178.
I have a file that is taking in a path as an argument:
./<filename> /path/to/file...
What I want to do is replace the /path/to/... part with /another/file/...
I was trying to sed the argument in the following manner:
CUR_PATH=$1
OLD_PATH="\/path\/to\/"
NEW_PATH="\/another\/file\/"
sed "s/$OLD_PATH/$NEW_PATH/" $CUR_PATH
But this isn't working because of the fact that sed is trying to actually modify the file at CUR_PATH and not the actual statement of CUR_PATH. How do I fix this? Thanks.
Another possibility is to use a here string:
CUR_PATH=$1
OLD_PATH="/path/to/"
NEW_PATH="/another/file/"
sed "s|$OLD_PATH|$NEW_PATH|" <<< $CUR_PATH
Also note that you can vary the delimiters for the substitution in sed, so that you don't have to escape the slashes in your path variables.
You don't need sed. bash a built-in substitution for variables. You can use:
NEW_PATH=${OLD_PATH/\/path\/to\//\/another\/file\/}
Note the backslashing of the /, because the expression is ${variable/old/new}.
You can use bash's substitution as Diego suggests, but for this particular case it is probably cleaner to do:
NEW_PATH="/another/file/${OLD_PATH##*/}"
which will replace the entire leading path of OLD_PATH with the string "/another/file/". Note that the double quotes are only necessary if OLD_PATH may contain whitespace.
If you do want to use sed, you can simply echo OLD_PATH into a pipe. And, when using sed for manipulating filenames, it is convenient to use a different separator. For example:
NEW_PATH=$( echo $OLD_PATH | sed s#/path/to/my#/another/file# )
I'm trying to escape a user-provided search string that can contain any arbitrary character and give it to sed, but can't figure out how to make it safe for sed to use. In sed, we do s/search/replace/, and I want to search for exactly the characters in the search string without sed interpreting them (e.g., the '/' in 'my/path' would not close the sed expression).
I read this related question concerning how to escape the replace term. I would have thought you'd do the same thing to the search, but apparently not because sed complains.
Here's a sample program that creates a file called "my_searches". Then it reads each line of that file and performs a search and replace using sed.
#!/bin/bash
# The contents of this heredoc will be the lines of our file.
read -d '' SAMPLES << 'EOF'
/usr/include
P#$$W0RD$?
"I didn't", said Jane O'Brien.
`ls -l`
~!##$%^&*()_+-=:'}{[]/.,`"\|
EOF
echo "$SAMPLES" > my_searches
# Now for each line in the file, do some search and replace
while read line
do
echo "------===[ BEGIN $line ]===------"
# Escape every character in $line (e.g., ab/c becomes \a\b\/\c). I got
# this solution from the accepted answer in the linked SO question.
ES=$(echo "$line" | awk '{gsub(".", "\\\\&");print}')
# Search for the line we read from the file and replace it with
# the text "replaced"
sed 's/'"$ES"'/replaced/' < my_searches # Does not work
# Search for the text "Jane" and replace it with the line we read.
sed 's/Jane/'"$ES"'/' < my_searches # Works
# Search for the line we read and replace it with itself.
sed 's/'"$ES"'/'"$ES"'/' < my_searches # Does not work
echo "------===[ END ]===------"
echo
done < my_searches
When you run the program, you get sed: xregcomp: Invalid content of \{\} for the last line of the file when it's used as the 'search' term, but not the 'replace' term. I've marked the lines that give this error with # Does not work above.
------===[ BEGIN ~!##$%^&*()_+-=:'}{[]/.,`"| ]===------
sed: xregcomp: Invalid content of \{\}
------===[ END ]===------
If you don't escape the characters in $line (i.e., sed 's/'"$line"'/replaced/' < my_searches), you get this error instead because sed tries to interpret various characters:
------===[ BEGIN ~!##$%^&*()_+-=:'}{[]/.,`"| ]===------
sed: bad format in substitution expression
sed: No previous regexp.
------===[ END ]===------
So how do I escape the search term for sed so that the user can provide any arbitrary text to search for? Or more precisely, what can I replace the ES= line in my code with so that the sed command works for arbitrary text from a file?
I'm using sed because I'm limited to a subset of utilities included in busybox. Although I can use another method (like a C program), it'd be nice to know for sure whether or not there's a solution to this problem.
This is a relatively famous problem—given a string, produce a pattern that matches only that string. It is easier in some languages than others, and sed is one of the annoying ones. My advice would be to avoid sed and to write a custom program in some other language.
You could write a custom C program, using the standard library function strstr. If this is not fast enough, you could use any of the Boyer-Moore string matchers you can find with Google—they will make search extremely fast (sublinear time).
You could write this easily enough in Lua:
local function quote(s) return (s:gsub('%W', '%%%1')) end
local function replace(first, second, s)
return (s:gsub(quote(first), second))
end
for l in io.lines() do io.write(replace(arg[1], arg[2], l), '\n') end
If not fast enough, speed things up by applying quote to arg[1] only once, and inline frunciton replace.
As ghostdog mentioned, awk '{gsub(".", "\\\\&");print}' is incorrect because it escapes out non-special characters. What you really want to do is perhaps something like:
awk 'gsub(/[^[:alpha:]]/, "\\\\&")'
This will escape out non-alpha characters. For some reason I have yet to determine, I still cant replace "I didn't", said Jane O'Brien. even though my code above correctly escapes it to
\"I\ didn\'t\"\,\ said\ Jane\ O\'Brien\.
It's quite odd because this works perfectly fine
$ echo "\"I didn't\", said Jane O'Brien." | sed s/\"I\ didn\'t\"\,\ said\ Jane\ O\'Brien\./replaced/
replaced`
this : echo "$line" | awk '{gsub(".", "\\\\&");print}' escapes every character in $line, which is wrong!. do an echo $ES after that and $ES appears to be \/\u\s\r\/\i\n\c\l\u\d\e. Then when you pass to the next sed, (below)
sed 's/'"$ES"'/replaced/' my_searches
, it will not work because there is no line that has pattern \/\u\s\r\/\i\n\c\l\u\d\e. The correct way is something like:
$ sed 's|\([#$#^&*!~+-={}/]\)|\\\1|g' file
\/usr\/include
P\#\$\$W0RD\$?
"I didn't", said Jane O'Brien.
\`ls -l\`
\~\!\#\#\$%\^\&\*()_\+-\=:'\}\{[]\/.,\`"\|
you put all the characters you want escaped inside [], and choose a suitable delimiter for sed that is not in your character class, eg i chose "|". Then use the "g" (global) flag.
tell us what you are actually trying to do, ie an actual problem you are trying to solve.
This seems to work for FreeBSD sed:
# using FreeBSD & Mac OS X sed
ES="$(printf "%q" "${line}")"
ES="${ES//+/\\+}"
sed -E s$'\777'"${ES}"$'\777'replaced$'\777' < my_searches
sed -E s$'\777'Jane$'\777'"${line}"$'\777' < my_searches
sed -E s$'\777'"${ES}"$'\777'"${line}"$'\777' < my_searches
The -E option of FreeBSD sed is used to turn on extended regular expressions.
The same is available for GNU sed via the -r or --regexp-extended options respectively.
For the differences between basic and extended regular expressions see, for example:
http://www.gnu.org/software/sed/manual/sed.html#Extended-regexps
Maybe you can use FreeBSD-compatible minised instead of GNU sed?
# example using FreeBSD-compatible minised,
# http://www.exactcode.de/site/open_source/minised/
# escape some punctuation characters with printf
help printf
printf "%s\n" '!"#$%&'"'"'()*+,-./:;<=>?#[\]^_`{|}~'
printf "%q\n" '!"#$%&'"'"'()*+,-./:;<=>?#[\]^_`{|}~'
# example line
line='!"#$%&'"'"'()*+,-./:;<=>?#[\]^_`{|}~ ... and Jane ...'
# escapes in regular expression
ES="$(printf "%q" "${line}")" # escape some punctuation characters
ES="${ES//./\\.}" # . -> \.
ES="${ES//\\\\(/(}" # \( -> (
ES="${ES//\\\\)/)}" # \) -> )
# escapes in replacement string
lineEscaped="${line//&/\&}" # & -> \&
minised s$'\777'"${ES}"$'\777'REPLACED$'\777' <<< "${line}"
minised s$'\777'Jane$'\777'"${lineEscaped}"$'\777' <<< "${line}"
minised s$'\777'"${ES}"$'\777'"${lineEscaped}"$'\777' <<< "${line}"
To avoid potential backslash confusion, we could (or rather should) use a backslash variable like so:
backSlash='\\'
ES="${ES//${backSlash}(/(}" # \( -> (
ES="${ES//${backSlash})/)}" # \) -> )
(By the way using variables in such a way seems like a good approach for tackling parameter expansion issues ...)
... or to complete the backslash confusion ...
backSlash='\\'
lineEscaped="${line//${backSlash}/${backSlash}}" # double backslashes
lineEscaped="${lineEscaped//&/\&}" # & -> \&
If you have bash, and you're just doing a pattern replacement, just do it natively in bash. The ${parameter/pattern/string} expansion in Bash will work very well for you, since you can just use a variable in place of the "pattern" and replacement "string" and the variable's contents will be safe from word expansion. And it's that word expansion which makes piping to sed such a hassle. :)
It'll be faster than forking a child process and piping to sed anyway. You already know how to do the whole while read line thing, so creatively applying the capabilities in Bash's existing parameter expansion documentation can help you reproduce pretty much anything you can do with sed. Check out the bash man page to start...