Our log pattern is in the following format dd/Mon/year:time(22/Feb/2018:13).
Goal is we want to find logs between 2 different times. We used sed to get the log between 2 times.
sed -n '/22\/Feb\/2018:13:/,/22\/Feb\/2018:16/p' /var/log/apache2/domlogs/access.log
The above command is working manually. We created a two variables called LAST and NOW in the script and assigned the date variables as mentioned below.
NOW="22/Feb/2018:16"
LAST="22/Feb/2018:13"
We have used the following sed commands to print the same output however it doesn't help us to print the same output.
sed -n '/'"$LAST"'/'"$NOW"'/p' /var/log/apache2/domlogs/access.log
The command gives the below error
sed: -e expression #1, char 5: unknown command: `F'
If we use normal string for LAST and NOW then above command works fine. Only problem is if the variable contains / in the input
You can freely change your delimiter of sed's regular expression by preceding it with a backslash, e.g. \!. Following command should work:
sed -n '\!'"$LAST"'!,\!'"$NOW"'!p' /var/log/apache2/domlogs/access.log
If ! is expected to show up in your date time format, you can use your judgment to choose another one.
You should read some good sed tutorial before working with sed, e.g.: http://www.grymoire.com/Unix/sed.html
I think you originally want to implement a range search in sed while you might miss the syntax of that. I fixed it above and it's tested.
Related
This question already has answers here:
Using different delimiters in sed commands and range addresses
(3 answers)
Closed 1 year ago.
I have a Visual Studio project, which is developed locally. Code files have to be deployed to a remote server. The only problem is the URLs they contain, which are hard-coded.
The project contains URLs such as ?page=one. For the link to be valid on the server, it must be /page/one .
I've decided to replace all URLs in my code files with sed before deployment, but I'm stuck on slashes.
I know this is not a pretty solution, but it's simple and would save me a lot of time. The total number of strings I have to replace is fewer than 10. A total number of files which have to be checked is ~30.
An example describing my situation is below:
The command I'm using:
sed -f replace.txt < a.txt > b.txt
replace.txt which contains all the strings:
s/?page=one&/pageone/g
s/?page=two&/pagetwo/g
s/?page=three&/pagethree/g
a.txt:
?page=one&
?page=two&
?page=three&
Content of b.txt after I run my sed command:
pageone
pagetwo
pagethree
What I want b.txt to contain:
/page/one
/page/two
/page/three
The easiest way would be to use a different delimiter in your search/replace lines, e.g.:
s:?page=one&:pageone:g
You can use any character as a delimiter that's not part of either string. Or, you could escape it with a backslash:
s/\//foo/
Which would replace / with foo. You'd want to use the escaped backslash in cases where you don't know what characters might occur in the replacement strings (if they are shell variables, for example).
The s command can use any character as a delimiter; whatever character comes after the s is used. I was brought up to use a #. Like so:
s#?page=one&#/page/one#g
A very useful but lesser-known fact about sed is that the familiar s/foo/bar/ command can use any punctuation, not only slashes. A common alternative is s#foo#bar#, from which it becomes obvious how to solve your problem.
add \ before special characters:
s/\?page=one&/page\/one\//g
etc.
In a system I am developing, the string to be replaced by sed is input text from a user which is stored in a variable and passed to sed.
As noted earlier on this post, if the string contained within the sed command block contains the actual delimiter used by sed - then sed terminates on syntax error. Consider the following example:
This works:
$ VALUE=12345
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345
This breaks:
$ VALUE=12345/6
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
sed: -e expression #1, char 21: unknown option to `s'
Replacing the default delimiter is not a robust solution in my case as I did not want to limit the user from entering specific characters used by sed as the delimiter (e.g. "/").
However, escaping any occurrences of the delimiter in the input string would solve the problem.
Consider the below solution of systematically escaping the delimiter character in the input string before having it parsed by sed.
Such escaping can be implemented as a replacement using sed itself, this replacement is safe even if the input string contains the delimiter - this is since the input string is not part of the sed command block:
$ VALUE=$(echo ${VALUE} | sed -e "s#/#\\\/#g")
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345/6
I have converted this to a function to be used by various scripts:
escapeForwardSlashes() {
# Validate parameters
if [ -z "$1" ]
then
echo -e "Error - no parameter specified!"
return 1
fi
# Perform replacement
echo ${1} | sed -e "s#/#\\\/#g"
return 0
}
this line should work for your 3 examples:
sed -r 's#\?(page)=([^&]*)&#/\1/\2#g' a.txt
I used -r to save some escaping .
the line should be generic for your one, two three case. you don't have to do the sub 3 times
test with your example (a.txt):
kent$ echo "?page=one&
?page=two&
?page=three&"|sed -r 's#\?(page)=([^&]*)&#/\1/\2#g'
/page/one
/page/two
/page/three
replace.txt should be
s/?page=/\/page\//g
s/&//g
please see this article
http://netjunky.net/sed-replace-path-with-slash-separators/
Just using | instead of /
Great answer from Anonymous. \ solved my problem when I tried to escape quotes in HTML strings.
So if you use sed to return some HTML templates (on a server), use double backslash instead of single:
var htmlTemplate = "<div style=\\"color:green;\\"></div>";
A simplier alternative is using AWK as on this answer:
awk '$0="prefix"$0' file > new_file
You may use an alternative regex delimiter as a search pattern by backs lashing it:
sed '\,{some_path},d'
For the s command:
sed 's,{some_path},{other_path},'
I am trying to write a script that will be doing some log highlighting for some human readability. Issue I am running into is trying to match a wildcard while also trying to print the wildcard variable. In my file I am searching for "psuslot-*" (can be either 1 or 2). The goal is to highlight the string "psuslot-1" or "psuslot-2" in the line of the log. My current command I am testing with is:
cat file.txt | sed "s|\(psuslot-.\)|$(printf "\033[93m"\1"\033[0m")|"
Highlights this in the output: 1
I know the groups are relatively correct as this command retains the correct group:
cat file.txt | sed "s|\(psuslot-.\)|\1|"
Retained in the output: psuslot-1 and psuslot-2
Using capture groups does not look to translate well into the shell execution. Is there a way to translate the wildcard into the execution?
You could use
sed 's|\(psuslot-.\)|\o33[93m\1\o33[0m|'
or less readable
sed 's|\(psuslot-.\)|'$(printf '\033[93m')'\1'$(printf '\033[0m')'|'
I have taken the syntax for the octal value from the GNU sed manual: Escape Sequences - specifying special characters
\oxxx
Produces or matches a character whose octal ASCII value is xxx.
And \o033 produced the same result as \o33 (same as hexadecimal \x1b), so I took the shorter one.
I'm trying to change #!/usr/bin/python to #!/usr/bin/python2.6
I've tried the following command line:
sed -i -e 's/.#!/usr/bin/python.*/#!/usr/bin/python2.6/' /usr/bin/yum
...which returns the following error:
sed: -e expression #1, char 11: unknown option to `s'
I can't find a good answer anywhere on Google.
Thanks in advance.
You're getting this error because sed interprets s/.#!/usr/b as your first search & replace command, and b isn't a valid flag for the command.
Indeed, the syntax of these commands is s<delimiter><search pattern><delimiter><replace><delimiter><flags>, where the most widely used <delimiter> is /.
You could escape the / in your search pattern and replace string so they aren't interpreted as the delimiter, however since you've got a lot of them I would suggest using another delimiter.
For example, using + as a delimiter your sed command would become sed -i -e 's+.#!/usr/bin/python.*+#!/usr/bin/python2.6+'.
As a side-note, the first . in your command is probably a mistake as the shebang should be written directly at the beginning of the files it appears in.
I assume that your line is first line from your file. With GNU sed:
sed '1s/python$/&2.6/' file
If you want to edit your file "in place" use sed's option -i.
I am trying to print out a specific section of a file which I have determined using line numbers, but the line numbers will vary from day to day so I need to be able to get the line numbers, store them as variables, then use sed to cut the lines from the stored file.
Here's what I have so far:
start-loader is a file that contains the lines I want to print, but also contains a lot of junk.
I can use sed -n '93,109p' start-loader to print out what I need, but what I want to do is this:
sed -n '$FL,$LFp' start-loader
where the variables are the line numbers I've stored.
I know that the above is not proper syntax, from a lot of research on the matter, but everything I've used either returns an error or does not work. I've tried double quotes, single then double for variables, braces for variables, and a few other things along with numerous different syntax styles. Would anyone happen to know how I can properly do this?
You need to separate the p command from your last-line variable somehow. Either of the following should work:
$ sed -n "$FL,${LF}p" start-loader
$ sed -n "$FL,$LF p" start-loader
Without the separation, the shell would try to expand the variable LFp, which does not exist, resulting in an empty string being passed to sed and causing a syntax error.
You also need to use double-quotes, not single-quotes, to allow the variables to be expanded before sed sees them.
I hope the example below answered your question:
kent$ s=3
kent$ e=8
kent$ seq 20 |sed -n "$s,$e p"
3
4
5
6
7
8
For some reason I can't seem to find a straightforward answer to this and I'm on a bit of a time crunch at the moment. How would I go about inserting a choice line of text after the first line matching a specific string using the sed command. I have ...
CLIENTSCRIPT="foo"
CLIENTFILE="bar"
And I want insert a line after the CLIENTSCRIPT= line resulting in ...
CLIENTSCRIPT="foo"
CLIENTSCRIPT2="hello"
CLIENTFILE="bar"
Try doing this using GNU sed:
sed '/CLIENTSCRIPT="foo"/a CLIENTSCRIPT2="hello"' file
if you want to substitute in-place, use
sed -i '/CLIENTSCRIPT="foo"/a CLIENTSCRIPT2="hello"' file
Output
CLIENTSCRIPT="foo"
CLIENTSCRIPT2="hello"
CLIENTFILE="bar"
Doc
see sed doc and search \a (append)
Note the standard sed syntax (as in POSIX, so supported by all conforming sed implementations around (GNU, OS/X, BSD, Solaris...)):
sed '/CLIENTSCRIPT=/a\
CLIENTSCRIPT2="hello"' file
Or on one line:
sed -e '/CLIENTSCRIPT=/a\' -e 'CLIENTSCRIPT2="hello"' file
(-expressions (and the contents of -files) are joined with newlines to make up the sed script sed interprets).
The -i option for in-place editing is also a GNU extension, some other implementations (like FreeBSD's) support -i '' for that.
Alternatively, for portability, you can use perl instead:
perl -pi -e '$_ .= qq(CLIENTSCRIPT2="hello"\n) if /CLIENTSCRIPT=/' file
Or you could use ed or ex:
printf '%s\n' /CLIENTSCRIPT=/a 'CLIENTSCRIPT2="hello"' . w q | ex -s file
Sed command that works on MacOS (at least, OS 10) and Unix alike (ie. doesn't require gnu sed like Gilles' (currently accepted) one does):
sed -e '/CLIENTSCRIPT="foo"/a\'$'\n''CLIENTSCRIPT2="hello"' file
This works in bash and maybe other shells too that know the $'\n' evaluation quote style. Everything can be on one line and work in
older/POSIX sed commands. If there might be multiple lines matching the CLIENTSCRIPT="foo" (or your equivalent) and you wish to only add the extra line the first time, you can rework it as follows:
sed -e '/^ *CLIENTSCRIPT="foo"/b ins' -e b -e ':ins' -e 'a\'$'\n''CLIENTSCRIPT2="hello"' -e ': done' -e 'n;b done' file
(this creates a loop after the line insertion code that just cycles through the rest of the file, never getting back to the first sed command again).
You might notice I added a '^ *' to the matching pattern in case that line shows up in a comment, say, or is indented. Its not 100% perfect but covers some other situations likely to be common. Adjust as required...
These two solutions also get round the problem (for the generic solution to adding a line) that if your new inserted line contains unescaped backslashes or ampersands they will be interpreted by sed and likely not come out the same, just like the \n is - eg. \0 would be the first line matched. Especially handy if you're adding a line that comes from a variable where you'd otherwise have to escape everything first using ${var//} before, or another sed statement etc.
This solution is a little less messy in scripts (that quoting and \n is not easy to read though), when you don't want to put the replacement text for the a command at the start of a line if say, in a function with indented lines. I've taken advantage that $'\n' is evaluated to a newline by the shell, its not in regular '\n' single-quoted values.
Its getting long enough though that I think perl/even awk might win due to being more readable.
A POSIX compliant one using the s command:
sed '/CLIENTSCRIPT="foo"/s/.*/&\
CLIENTSCRIPT2="hello"/' file
Maybe a bit late to post an answer for this, but I found some of the above solutions a bit cumbersome.
I tried simple string replacement in sed and it worked:
sed 's/CLIENTSCRIPT="foo"/&\nCLIENTSCRIPT2="hello"/' file
& sign reflects the matched string, and then you add \n and the new line.
As mentioned, if you want to do it in-place:
sed -i 's/CLIENTSCRIPT="foo"/&\nCLIENTSCRIPT2="hello"/' file
Another thing. You can match using an expression:
sed -i 's/CLIENTSCRIPT=.*/&\nCLIENTSCRIPT2="hello"/' file
Hope this helps someone
The awk variant :
awk '1;/CLIENTSCRIPT=/{print "CLIENTSCRIPT2=\"hello\""}' file
I had a similar task, and was not able to get the above perl solution to work.
Here is my solution:
perl -i -pe "BEGIN{undef $/;} s/^\[mysqld\]$/[mysqld]\n\ncollation-server = utf8_unicode_ci\n/sgm" /etc/mysql/my.cnf
Explanation:
Uses a regular expression to search for a line in my /etc/mysql/my.cnf file that contained only [mysqld] and replaced it with
[mysqld]
collation-server = utf8_unicode_ci
effectively adding the collation-server = utf8_unicode_ci line after the line containing [mysqld].
I had to do this recently as well for both Mac and Linux OS's and after browsing through many posts and trying many things out, in my particular opinion I never got to where I wanted to which is: a simple enough to understand solution using well known and standard commands with simple patterns, one liner, portable, expandable to add in more constraints. Then I tried to looked at it with a different perspective, that's when I realized i could do without the "one liner" option if a "2-liner" met the rest of my criteria. At the end I came up with this solution I like that works in both Ubuntu and Mac which i wanted to share with everyone:
insertLine=$(( $(grep -n "foo" sample.txt | cut -f1 -d: | head -1) + 1 ))
sed -i -e "$insertLine"' i\'$'\n''bar'$'\n' sample.txt
In first command, grep looks for line numbers containing "foo", cut/head selects 1st occurrence, and the arithmetic op increments that first occurrence line number by 1 since I want to insert after the occurrence.
In second command, it's an in-place file edit, "i" for inserting: an ansi-c quoting new line, "bar", then another new line. The result is adding a new line containing "bar" after the "foo" line. Each of these 2 commands can be expanded to more complex operations and matching.