I have a bunch of text files (named install*) in which I need to replace the expression curl -L with the expression curl -k -L. I am on OS X 10, Yosemite.
The following attempts don't work:
sed -e "s/'curl -L'/'curl -k -L'/g" install*
sed -i '' -e "s/'curl -L'/'curl -k -L'/g" install*
The contents of the files are shown (as if I had typed cat), but replacement isn't performed.
What am I doing wrong?
Your problem is that you're nesting quoting mechanisms (delimiters) when you shouldn't.
Generally, you should use single quotes to enclose an sed script as a whole:
sed -i '' -e 's/curl -L/curl -k -L/g' install*
By using single quotes, the script is protected from accidental interpretation by the shell.
If you do want the shell to interpret parts of a sed script beforehand, splice those parts in as double-quoted strings.
To sed's s function, it is / that acts as the delimiter - no additional quoting of the strings between / instances should be used, and your use of ' simply causes sed to consider these single quotes part of the regex to match / the replacement string. (As an aside: you're free to choose a delimiter other than /).
That said, if you do need to force interpretation of certain characters as literals, use \-escaping, such as using \. in the regex to represent a literal .
This is the only quoting mechanism that sed itself supports.
However, in your particular case, neither the string used in the regex nor the string used as the replacement string need escaping.
For a generic way to escape strings for use as literals in these situations, see
https://stackoverflow.com/a/29613573/45375.
You need -i.
GNU sed uses -i without arguments to mean "replace in-place".
BSD sed needs an empty argument, so you'll have to use sed -i '' -e '...' install*
There's no portable way to do it with sed...
Related
I have 2 bash script variables defined:
THELINENUMBER="14" # an arbitrary line number, comes from a separate command
NEWLINE="a line/ with# special! characters<" # arbitrary line with special characters, comes from separate command
I need to use the line number ${THELINENUMBER} to replace a line in a file called after.txt with ${NEWLINE}.
How do I do that?
These are some examples I have tried:
sed -i '${THELINENUMBER}s#.*#"/"${NEWLINE}"/"' after.txt
sed -i "${THELINENUMBER}s#.*#"/"${NEWLINE}"/"" after.txt
sed -i "${THELINENUMBER}s/.*/'${NEWLINE}'" after.txt
sed -i '${THELINENUMBER}s,.*,${NEWLINE}' after.txt
I am told that the delimitter is usually a /, but those are present in my line replacement variable, so I can't use those. I tried with # and , but the desired behavior did not change. I am also told that " and ' are supposed to be used to turn off escaping in text (use literal string), but I have not been able to get that to work either. How do I pass in a string parameter into sed that has special characters? I am wondering if I should pass the variable ${NEWLINE} into another built-in function call to add escape characters or something before passing it into sed. Is sed the right tool for the job? I did not find much helpful information looking at the CLI manpages. I use Ubuntu 18.04.
I have referred to these sources in my internet search:
https://stackoverflow.com/questions/11145270/how-to-replace-an-entire-line-in-a-text-file-by-line-number
https://askubuntu.com/questions/76808/how-do-i-use-variables-in-a-sed-command
https://stackoverflow.com/questions/37372047/find-a-line-with-a-string-and-replace-entire-line-with-another-line
Use the c (change) command.
By the way, the naming convention for regular shell variables is NOT ALLCAPS, as that may result in accidental collisions with special variables like PATH.
sed "$linenumber c\\
$newline" file
Try
sed -i "${THELINENUMBER}s#.*#${NEWLINE}#" after.txt
this works because:
You require " enclosing the entire sed command instead of backtick so that the variables are expanded
No other quotes or backticks are needed to escape " in the variables as there aren't any: there are no literal (escaped) quotes inside the variables
An alternate separator (such as #) is required due to the / inside the NEWLINE variable.
I want to replace <lexicon uri="file://C:/image/png/grammars/custom/image-custom.lex?SWI.type=backup"/><lexicon uri="file://C:/image/jpg/grammars/custom/image-dot-custom.lex?SWI.type=backup"/> with null in multiple files.
The code is given below.
sed -i s|<lexicon uri="file://C:/image/png/grammars/custom/image-custom.lex?SWI.type=backup"/><lexicon uri="file://C:/image/jpg/grammars/custom/image-dot-custom.lex?SWI.type=backup"/>||g *
Here I am getting this error:
< was unexpected at this time.
Please clarify for me what is not working here.
Could you please try following and let me know if this helps you. By using # as sed's separator you need not to escape / in it only need to escape ., ? not to take their special meaning
sed -E 's#<lexicon uri="file://C:/image/png/grammars/custom/image-custom\.lex\?SWI\.type=backup"/><lexicon uri="file://C:/image/jpg/grammars/custom/image-dot-custom\.lex\?SWI\.type=backup"/>##' Input_file
Tested it with:
sed --version
GNU sed version 4.2.1
works with #
sed -i -e 's#<lexicon uri="file://C:/image/png/grammars/custom/image-custom.lex?SWI.type=backup"/><lexicon uri="file://C:/image/jpg/grammars/custom/image-dot-custom.lex?SWI.type=backup"/>##g' test.txt
The pattern contains shell metacharacters, which need to be quoted or escaped. Usually, in Bash, you should use single quotes around strings, unless you need the shell to interpolate variables and command substitutions and interpret backslash sequences (in which case use double quotes) or to also perform whitespace tokenization and wildcard expansion (in which case use no quotes). See also When to wrap quotes around a shell variable?
sed -i 's|<lexicon uri="file://C:/image/png/grammars/custom/image-custom.lex?SWI.type=backup"/><lexicon uri="file://C:/image/jpg/grammars/custom/image-dot-custom.lex?SWI.type=backup"/>||' *
I also took out the g flag, which only makes sense if you expect multiple matches within a single line. (Perhaps you do after all, in which case obviously put it back.)
I have a a some SNMP dump:
1.3.6.1.2.1.1.2.0|5|1.3.6.1.4.1.9.1.1178
1.3.6.1.2.1.1.3.0|7|1881685367
1.3.6.1.2.1.1.4.0|6|""
1.3.6.1.2.1.1.5.0|6|"hgfdhg-4365.gfhfg.dfg.com"
1.3.6.1.2.1.1.6.0|6|""
1.3.6.1.2.1.1.7.0|2|6
1.3.6.1.2.1.1.8.0|7|0
1.3.6.1.2.1.1.9.1.2.1|5|1.3.6.1.4.1.9.7.129
1.3.6.1.2.1.1.9.1.2.2|5|1.3.6.1.4.1.9.7.115
And need to grep all data in first string after 1.3.6.1.2.1.1.2.0|5|, but not include this start of the string in grep itself. So, I must receive 1.3.6.1.4.1.9.1.1178 in grep. I've tried to use regex:
\b1.3.6.1.2.1.1.2.0\|5\|\s*([^\n\r]*)
But without any success. If a regular expression, or grep, is in fact the right tool, can you help me find the right regex? Otherwise, what tools should I consider instead?
With GNU grep +PCRE support, you can use Perl's \K flag to discard part of the matched string :
grep -Po "1\.3\.6\.1\.2\.1\.1\.2\.0\|5\|\K.*"
-P enables Perl's regex mode and -o switches output to matched parts rather than whole lines.
I had to escape the characters that have special meaning in Perl regexs, but this can be avoided as 123 suggests, by enclosing the characters to interpret literally between \Q and \E :
grep -Po "\Q1.3.6.1.2.1.1.2.0|5|\E\K.*"
I would usually solve this with sed as follows :
sed -n 's/1\.3\.6\.1\.2\.1\.1\.2\.0|5|\(.*\)/\1/p'
The -n flag disables implicit output and the search and replace command will remove the searched prefix from the line, leaving the relevant part to be printed.
The characters that have special meaning in GNU Basic Regular Expressions (BRE) must be escaped, which in this case is only .. Also note that the grouping tokens are \( and \) rather than the usual ( and ).
An alternate way to do this is in native shell, without any regexes at all. Consider:
prefix='1.3.6.1.2.1.1.2.0|5|'
while read -r line; do
[[ $line = "$prefix"* ]] && printf '%s\n' "${line#$prefix}"
done
If your original string is piped into the while read loop, the output is precisely 1.3.6.1.4.1.9.1.1178.
I have a file with a lot of lines, two of them are:
videoId: 'S2Rgr6yuuXQ'
var vid_seq=1;
in a shell script, I have two variables,
for id, the value is always 11 characters/numbers
id='fsafsferii2'
id_seq=80
I want to modify these two lines with id and id_seq
videoId: 'fsafsferii2'
var vid_seq=80;
I used
sed -i 's/\(videoId: \).*\\1'${id}'/\2' file
but there are errors, what is wrong with my script?
thanks
The grep command won't "replace" text, it is for "global regular expression print". But sed will.
sed -i'' '/^videoId: /s/: .*/: '"$id"'/;/^var vid_seq=/s/=.*/='"$id_seq"';/'
I'm not a big fan of inserting variables into sed scripts this way, but sed is simple, and provides no mechanism for actually using actual variables on its own. If you're going to do this, include some format checking for the two variables to make sure they contain the data you want them to contain, before you run this sed script. An accidental / in a variable would cause the sed script to fail.
UPDATE per comments:
Here's a successful test:
$ id=fsafsferii2
$ id_seq=80
$ cat inp686
videoId: 'S2Rgr6yuuXQ'
var vid_seq=1;
$ sed '/^videoId: /s/: .*/: '"$id"'/;/^var vid_seq=/s/=.*/='"$id_seq"';/' < inp686
videoId: fsafsferii2
var vid_seq=80;
$
Of course, you'll need to do some quote magic to get the single quotes into your videoId, but I'm sure you can figure that out yourself.
UPDATE 2
According to sed's man page, the substitute command is in the form:
[2addr]s/regular expression/replacement/flags
The [2addr] means you can specify up to two "addresses", which can be line numbers or regular expressions to match. So the s (substitute) command can take a line, a range, a match, or a span between matches. In our case, we're just using a single match to identify what lines we want to execute the substitution on.
The script above is made up of two sed commands, separated by a semicolon.
/^videoId: / -- Match lines that start with the word videoId:...
s/: .*/: '"$id"'/; -- Substitute all text from the colon to the end of the line with whatever is in the $id environment variable.
/^var vid_seq=/ -- Match lines that ... meh, as above.
s/=.*/='"$id_seq"';/ -- Substitute all text from the equals sign on with $id_seq.
Note that the '"$id"' construct means that we are exiting the single quotes, then immediately entering double quotes for the expansion of the variable ... then exiting the double quotes and going back into a new set of single quotes. Sed scripts are safest inside single quotes because of the frequent use of characters that might be interpreted by a shell.
Note also that because sed's substitute command uses a forward slash as a delimiter, the $id and $id_seq variables may not contain a slash. If they might, you can switch to a different delimiter.
What is wrong with:
sed -i 's/\(videoId: \).*\\1'${id}'/\2' file
Missing the third delimiter (/). Valid syntax is s/regex/replace/
Incorrect regex pattern (let's assume ${id} has been substituted)
\(videoId: \).*\\1fsafsferii2
is telling it to match a string that looks like this:
videoId: anything\1fsafsferii2
(\\ in regex matches literal backslash, so \\1 would match a literal backslash followed by 1 instead of 1st sub-expression)
Replace the matched string with \2
But since there is only one set of parentheses, \2 is actually empty.
Also, since the regex pattern in 2. doesn't match anything, nothing is replaced.
This should work (GNU sed)
sed -i 's/\(videoId: \).*/\1 \x27'${id}'\x27/
s/\(var vid_seq=\).*/\1'${id_seq}'\;/' file
Note:
\x27 is the hexadecimal representation of single quote (to prevent clashing with the other single quote)
\; for literal semicolon. If ; is not escaped, it's interpreted to terminate the s command in sed.
I need to egrep a string that isn't known before runtime and that I'll get via shell variable (shell is bash, if that matters). Problem is, that string will contain special characters like braces, spaces, dots, slashes, and so on.
If I know the string I can escape the special characters one at a time, but how can I do that for the whole string?
Running the string through a sed script to prefix each special character with \ could be an idea, I still need to rtfm how such a script should be written. I don't know if there are other, better, options.
I did read re_format(7) but it seems there is no such thing like "take the whole next string as literal"...
EDIT: to avoid false positives, I should also add newline detection to the pattern, eg. egrep '^myunknownstring'
If you need to embed the string into a larger expression, sed is how I would do it.
s_esc="$(echo "$s" | sed 's/[^-A-Za-z0-9_]/\\&/g')" # backslash special characters
inv_ent="$(egrep "^item [0-9]+ desc $s_esc loc .+$" inventory_list)"
Use the -F flag to make the PATTERN a fixed literal string
$ var="(.*+[a-z]){3}"
$ echo 'foo bar (.*+[a-z]){3} baz' | grep -F "$var" -o
(.*+[a-z]){3}
Are you trying to protect the string from being incorrectly interpreted as bash syntax or are you trying to protect parts of the string from being interpreted as regular expression syntax?
For bash protection:
grep supports the -f switch:
-f FILE, --file=FILE
Obtain patterns from FILE, one per line. The empty file contains zero patterns, and therefore matches nothing.
No escaping is necessary inside the file. Just make it a file containing a single line (and thus one pattern) which can be produced from your shell variable if that's what you need to do.
# example trivial regex
var='^r[^{]*$'
pattern=/tmp/pattern.$$
rm -f "$pattern"
echo "$var" > "$pattern"
egrep -f "$pattern" /etc/password
rm -f "$pattern"
Just to illustrate the point.
Try it with -F instead as another poster suggested for regex protection.