This question already has answers here:
shell scripting using sed
(3 answers)
Closed 9 years ago.
So, I want to read file from stdin, delete all '/' in line that contain exactly 3 '/', and write the output to stdout. So a file contain:
/a1/b/c
/a/b2
///
/a
will have output:
a1bc
/a/b2
/a
I am thinking something like this:
sed -r 's/\/[^\/]*\/[^\/]*\/.*/"I not sure what do I need to put in here"/g'
however, I am not really sure what do I need to put in the replace session.
A sed solution:
sed '/.*\/.*\/.*\//{s#/##g}' file
If Perl is ok for you:
perl -F/ -ape '$_=#F>3?join"",#F:join "/",#F;' file
sed -e '/^[^\/]*\/[^\/]*\/[^\/]*\/[^\/]*$/ s%/%%g'
The gruesome pattern looks for start of line, a sequence of zero or more non-slashes followed by a slash, more non-slashes and a second slash, more non-slashes and a third slash, more non-slashes and the end of line. On any line that matches that, substitute the slashes by nothing globally.
There are other ways to write the regex, but they aren't substantially clearer. This will work in pretty much any version of sed. So will this:
sed -e '/^\([^\/]*\/\)\{3\}[^\/]*$/ s%/%%g'
It looks for start of line, 3 units of (zero or more non-slashes followed by a slash), zero or more non-slashes and end of line.
If your sed has extended regular expressions (GNU sed, for example), then you can gain some notational convenience.
sed -r -e '/^([^\/]*\/){3}[^\/]*$/ s%/%%g'
sed -r -e 's%^([^/]*)/([^/]*)/([^/]*)/([^/]*)$%\1\2\3\4%'
The latter captures the four sets of 'zero or more non-slashes' and pastes them together to make the replacement. You could write that with the non-extended regular expressions, but it would be even more laden with backslashes than before.
This is much simpler in awk:
awk -F/ 'NF==4 { gsub("/","") } {print}' tmp.txt
Related
This question already has answers here:
Using different delimiters in sed commands and range addresses
(3 answers)
Closed 1 year ago.
I have a Visual Studio project, which is developed locally. Code files have to be deployed to a remote server. The only problem is the URLs they contain, which are hard-coded.
The project contains URLs such as ?page=one. For the link to be valid on the server, it must be /page/one .
I've decided to replace all URLs in my code files with sed before deployment, but I'm stuck on slashes.
I know this is not a pretty solution, but it's simple and would save me a lot of time. The total number of strings I have to replace is fewer than 10. A total number of files which have to be checked is ~30.
An example describing my situation is below:
The command I'm using:
sed -f replace.txt < a.txt > b.txt
replace.txt which contains all the strings:
s/?page=one&/pageone/g
s/?page=two&/pagetwo/g
s/?page=three&/pagethree/g
a.txt:
?page=one&
?page=two&
?page=three&
Content of b.txt after I run my sed command:
pageone
pagetwo
pagethree
What I want b.txt to contain:
/page/one
/page/two
/page/three
The easiest way would be to use a different delimiter in your search/replace lines, e.g.:
s:?page=one&:pageone:g
You can use any character as a delimiter that's not part of either string. Or, you could escape it with a backslash:
s/\//foo/
Which would replace / with foo. You'd want to use the escaped backslash in cases where you don't know what characters might occur in the replacement strings (if they are shell variables, for example).
The s command can use any character as a delimiter; whatever character comes after the s is used. I was brought up to use a #. Like so:
s#?page=one&#/page/one#g
A very useful but lesser-known fact about sed is that the familiar s/foo/bar/ command can use any punctuation, not only slashes. A common alternative is s#foo#bar#, from which it becomes obvious how to solve your problem.
add \ before special characters:
s/\?page=one&/page\/one\//g
etc.
In a system I am developing, the string to be replaced by sed is input text from a user which is stored in a variable and passed to sed.
As noted earlier on this post, if the string contained within the sed command block contains the actual delimiter used by sed - then sed terminates on syntax error. Consider the following example:
This works:
$ VALUE=12345
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345
This breaks:
$ VALUE=12345/6
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
sed: -e expression #1, char 21: unknown option to `s'
Replacing the default delimiter is not a robust solution in my case as I did not want to limit the user from entering specific characters used by sed as the delimiter (e.g. "/").
However, escaping any occurrences of the delimiter in the input string would solve the problem.
Consider the below solution of systematically escaping the delimiter character in the input string before having it parsed by sed.
Such escaping can be implemented as a replacement using sed itself, this replacement is safe even if the input string contains the delimiter - this is since the input string is not part of the sed command block:
$ VALUE=$(echo ${VALUE} | sed -e "s#/#\\\/#g")
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345/6
I have converted this to a function to be used by various scripts:
escapeForwardSlashes() {
# Validate parameters
if [ -z "$1" ]
then
echo -e "Error - no parameter specified!"
return 1
fi
# Perform replacement
echo ${1} | sed -e "s#/#\\\/#g"
return 0
}
this line should work for your 3 examples:
sed -r 's#\?(page)=([^&]*)&#/\1/\2#g' a.txt
I used -r to save some escaping .
the line should be generic for your one, two three case. you don't have to do the sub 3 times
test with your example (a.txt):
kent$ echo "?page=one&
?page=two&
?page=three&"|sed -r 's#\?(page)=([^&]*)&#/\1/\2#g'
/page/one
/page/two
/page/three
replace.txt should be
s/?page=/\/page\//g
s/&//g
please see this article
http://netjunky.net/sed-replace-path-with-slash-separators/
Just using | instead of /
Great answer from Anonymous. \ solved my problem when I tried to escape quotes in HTML strings.
So if you use sed to return some HTML templates (on a server), use double backslash instead of single:
var htmlTemplate = "<div style=\\"color:green;\\"></div>";
A simplier alternative is using AWK as on this answer:
awk '$0="prefix"$0' file > new_file
You may use an alternative regex delimiter as a search pattern by backs lashing it:
sed '\,{some_path},d'
For the s command:
sed 's,{some_path},{other_path},'
This question already has answers here:
Insert line after match using sed
(8 answers)
Closed 8 years ago.
I have a file ".gitignore" contains various source file name as
src/abc
src/line
src/another
I like to add another line "src/line.cpp" after a match found "src/line"
result would look like as
src/abc
src/line
src/line.cpp
src/another
I am using sed as
set -- "$File" // $File contains src/line
IFS="/"; declare -a Array=($*)
echo "${Array[0]}" // This prints src
echo "${Array[1]}" // This prints line
sed -i '/$Array[0]\/$Array[1]/a $Array[0]\/$Array[1].cpp' $File
The sed command is not working.
I have a feeling that slashes are not properly handled. If I hard code as
sed -i '/src\/line/a src\/line.cpp' $File
then it works.
Any solutions? Thanks in advance!
You properly use double-quotes in echo "${Array[1]}", and the braces {} needed for the array index. But you neglect to do so in your sed line. So try:
sed -i "/${Array[0]}\/${Array[1]} *$/a ${Array[0]}\/${Array[1]}.cpp" .gitignore
I've added an extra $ here for the end of the line, so:
Repeating the command leaves the file unchanged
Reduces risk of inadvertent matches
The space-star-dollar *$ handles the case where a line has blank spaces at the end. You may also want to add a ^ to match the start of line.
Also, your sed command operates on $File; I think you want .gitignore.
But rather than fiddle around with IFS and arrays, it might be cleaner to use bash's parameter substitution. Replace ${Array[0]} and ${Array[1]} with:
${File%/*}
${File#*/}
Finally, you don't even need to do that, if you replace the sed slash / with for example # (but underscore _ would be bad, as it is commonly used in filenames).
So the penultimate command line is:
sed -i "\#$File#a $File.cpp" .gitignore
or, to handle blanks at the end of a line and to avoid spurious matches:
sed -i "\#$File *\$#a $File.cpp" .gitignore
Note only the first # is escaped.
With GNU sed, you can do it like:
sed -i.BAK 's_src/line_\0\n\0.txt_'
As it allows to use other separators than slashes.
You can use it this way:
s='src/line'; r='src/line.cpp'; sed "s~$s~&\n$r~" file
src/abc
src/line
src/line.cpp
src/another
This question already has answers here:
sed whole word search and replace
(5 answers)
Closed 8 years ago.
Please help me in solving the below issue. I have a file:
mat rat
mat dog
mat matress
I need to display
rat
dog
matress
I have coded with sed command to display the output: sed "s/$up//g"
($up will contain mat) . But using this command, I am getting the output as
rat
dog
ress
What do I do to resolve this?.
Please help.
The /g flag tries to apply the substitution command multiple times for each line. First two lines are fine because the word only appears once, but for the third line it will remove both.
You can solve it being more specific using zero-width assertions, like ^, or the GNU extension \b, like:
sed "s/^$up//g"
or
sed "s/$up\b//g"
Although the easier could be to remove the flag, like:
sed "s/$up//"
In all three cases the result is the same, at least for this kind of simple examples.
Using awk
awk '{print $NF}' inputFile
Test:
$ cat text
mat rat
mat dog
mat matress
$ awk '{print $NF}' text
rat
dog
matress
Your current command will remove all instances of $up anywhere, including multiple occurrences in a line and occurrences in the middle of a line.
If you want to match only $up at the very beginning of a line, and only when it is a whole (whitespace-delimited) word, try the following command:
sed "s/^$up\>//"
In GNU sed, the assertion ^ matches to the beginning of a line, and \> matches the end of a word (the zero-width "character" between a non-whitespace character and whitespace character).
If there might be whitespace before $up, you can use
sed "s/\(\s*\)$up\>/\1/"
This will remove just the $up and preserve all whitespace.
If you don't want to keep the whitespace between $up and the text after it, you can replace \> with \s\+, which matches to one or more (\+) whitespace characters (\s); i.e.,
sed "s/^$up\s\+//"
sed "s/\(\s*\)$up\s\+/\1/"
sed 's/^mat //' /path/to/file should do the trick. Note that there is no g; it's s/foo/bar; not s/foo/bar/g. Also, the ^ pegs the replacement to the beginning of each line.
If you are indeed assigning a variable such as $up, you can use sed "s/^$up//" /path/to/file.
For some reason I can't seem to find a straightforward answer to this and I'm on a bit of a time crunch at the moment. How would I go about inserting a choice line of text after the first line matching a specific string using the sed command. I have ...
CLIENTSCRIPT="foo"
CLIENTFILE="bar"
And I want insert a line after the CLIENTSCRIPT= line resulting in ...
CLIENTSCRIPT="foo"
CLIENTSCRIPT2="hello"
CLIENTFILE="bar"
Try doing this using GNU sed:
sed '/CLIENTSCRIPT="foo"/a CLIENTSCRIPT2="hello"' file
if you want to substitute in-place, use
sed -i '/CLIENTSCRIPT="foo"/a CLIENTSCRIPT2="hello"' file
Output
CLIENTSCRIPT="foo"
CLIENTSCRIPT2="hello"
CLIENTFILE="bar"
Doc
see sed doc and search \a (append)
Note the standard sed syntax (as in POSIX, so supported by all conforming sed implementations around (GNU, OS/X, BSD, Solaris...)):
sed '/CLIENTSCRIPT=/a\
CLIENTSCRIPT2="hello"' file
Or on one line:
sed -e '/CLIENTSCRIPT=/a\' -e 'CLIENTSCRIPT2="hello"' file
(-expressions (and the contents of -files) are joined with newlines to make up the sed script sed interprets).
The -i option for in-place editing is also a GNU extension, some other implementations (like FreeBSD's) support -i '' for that.
Alternatively, for portability, you can use perl instead:
perl -pi -e '$_ .= qq(CLIENTSCRIPT2="hello"\n) if /CLIENTSCRIPT=/' file
Or you could use ed or ex:
printf '%s\n' /CLIENTSCRIPT=/a 'CLIENTSCRIPT2="hello"' . w q | ex -s file
Sed command that works on MacOS (at least, OS 10) and Unix alike (ie. doesn't require gnu sed like Gilles' (currently accepted) one does):
sed -e '/CLIENTSCRIPT="foo"/a\'$'\n''CLIENTSCRIPT2="hello"' file
This works in bash and maybe other shells too that know the $'\n' evaluation quote style. Everything can be on one line and work in
older/POSIX sed commands. If there might be multiple lines matching the CLIENTSCRIPT="foo" (or your equivalent) and you wish to only add the extra line the first time, you can rework it as follows:
sed -e '/^ *CLIENTSCRIPT="foo"/b ins' -e b -e ':ins' -e 'a\'$'\n''CLIENTSCRIPT2="hello"' -e ': done' -e 'n;b done' file
(this creates a loop after the line insertion code that just cycles through the rest of the file, never getting back to the first sed command again).
You might notice I added a '^ *' to the matching pattern in case that line shows up in a comment, say, or is indented. Its not 100% perfect but covers some other situations likely to be common. Adjust as required...
These two solutions also get round the problem (for the generic solution to adding a line) that if your new inserted line contains unescaped backslashes or ampersands they will be interpreted by sed and likely not come out the same, just like the \n is - eg. \0 would be the first line matched. Especially handy if you're adding a line that comes from a variable where you'd otherwise have to escape everything first using ${var//} before, or another sed statement etc.
This solution is a little less messy in scripts (that quoting and \n is not easy to read though), when you don't want to put the replacement text for the a command at the start of a line if say, in a function with indented lines. I've taken advantage that $'\n' is evaluated to a newline by the shell, its not in regular '\n' single-quoted values.
Its getting long enough though that I think perl/even awk might win due to being more readable.
A POSIX compliant one using the s command:
sed '/CLIENTSCRIPT="foo"/s/.*/&\
CLIENTSCRIPT2="hello"/' file
Maybe a bit late to post an answer for this, but I found some of the above solutions a bit cumbersome.
I tried simple string replacement in sed and it worked:
sed 's/CLIENTSCRIPT="foo"/&\nCLIENTSCRIPT2="hello"/' file
& sign reflects the matched string, and then you add \n and the new line.
As mentioned, if you want to do it in-place:
sed -i 's/CLIENTSCRIPT="foo"/&\nCLIENTSCRIPT2="hello"/' file
Another thing. You can match using an expression:
sed -i 's/CLIENTSCRIPT=.*/&\nCLIENTSCRIPT2="hello"/' file
Hope this helps someone
The awk variant :
awk '1;/CLIENTSCRIPT=/{print "CLIENTSCRIPT2=\"hello\""}' file
I had a similar task, and was not able to get the above perl solution to work.
Here is my solution:
perl -i -pe "BEGIN{undef $/;} s/^\[mysqld\]$/[mysqld]\n\ncollation-server = utf8_unicode_ci\n/sgm" /etc/mysql/my.cnf
Explanation:
Uses a regular expression to search for a line in my /etc/mysql/my.cnf file that contained only [mysqld] and replaced it with
[mysqld]
collation-server = utf8_unicode_ci
effectively adding the collation-server = utf8_unicode_ci line after the line containing [mysqld].
I had to do this recently as well for both Mac and Linux OS's and after browsing through many posts and trying many things out, in my particular opinion I never got to where I wanted to which is: a simple enough to understand solution using well known and standard commands with simple patterns, one liner, portable, expandable to add in more constraints. Then I tried to looked at it with a different perspective, that's when I realized i could do without the "one liner" option if a "2-liner" met the rest of my criteria. At the end I came up with this solution I like that works in both Ubuntu and Mac which i wanted to share with everyone:
insertLine=$(( $(grep -n "foo" sample.txt | cut -f1 -d: | head -1) + 1 ))
sed -i -e "$insertLine"' i\'$'\n''bar'$'\n' sample.txt
In first command, grep looks for line numbers containing "foo", cut/head selects 1st occurrence, and the arithmetic op increments that first occurrence line number by 1 since I want to insert after the occurrence.
In second command, it's an in-place file edit, "i" for inserting: an ansi-c quoting new line, "bar", then another new line. The result is adding a new line containing "bar" after the "foo" line. Each of these 2 commands can be expanded to more complex operations and matching.
I'm running sed as a part of a shell script to clean up bind logs for insertion into a database.
One of the sed commands is the following:
sed -i 's/-/:/g' $DPath/named.query.log
This turns out to be problematic as it disrupts any resource requests that also include a dash (I'm using : as a delimiter for an awk statement further down).
My question is how do I limit the sed command above to only the first ten characters of the line? I haven't seen a specific switch that does this, and I'm nowhere near good enough with RegEx to even start on developing one that works. I can't just use regex to match the preceding numbers because it's possible that the pattern could be part of a resource request. Heck, I can't even use pattern matching for ####-##-## because, again, it could be part of the resource.
Any ideas are much appreciated.
It's [almost always] simpler with awk:
awk '{target=substr($0,1,10); gsub(/-/,":",target); print target substr($0,11)}' file
I think the shortest solution, and perhaps the simplest, is provided by sed itself, rather than awk[ward]:
sed "h;s/-/:/g;G;s/\(..........\).*\n........../\1/"
Explanation:
(h) copy everything to the hold space
(s) do the substitution (to the entire pattern space)
(G) append the hold space, with a \n separator
(s) delete the characters up to the tenth after the \n, but keep the first ten.
Some test code:
echo "--------------------------------" > foo
sed -i "h;s/-/:/g;G;s/\(..........\).*\n........../\1/" foo
cat foo
::::::::::----------------------
I'm not sure how make sed do it per se, however, I do know that you can feed sed the first 10 characters then paste the rest back in, like so:
paste -d"\0" <(cut -c1-10 $DPath/named.query.log | sed 's/\-/:/g') <(cut -c11- $DPath/named.query.log)
You can do the following:
cut -c 1-10 $DPath/named.query.log | sed -i 's/-/:/g'
The cut statemnt takes only the first 10 chars of each line in that file. The output of that should be piped in a file. As of now it will just output to your terminal