Use sed to edit crontab - bash

I am writing a sed command that should uncomment an entry in crontab. Is there a better way to do this? The first option that comes to mind is sed.
Example:
crontab -l
# 5 * * * 3 bash test.sh
The sed command should uncomment this entry. This is what I have now.
sed "s#\# 5 * * * 3 bash test.sh#5 * * * 3 bash test.sh#"
Obviously this sed command doesn't do the task.
ps:the sed command will eventually make its way into a script.

Sed is great for matching specific regular expressions and manipulating text in certain ways, but this doesn't seem to me to be one of them. While you can use sed for this task, the result is perhaps overly complex and fragile.
Your initial attempt was:
sed "s#\# 5 * * * 3 bash test.sh#5 * * * 3 bash test.sh#"
This fails because the * character is a special character within your regular expression, and is translated as "zero or more of the previous atom" (in this case, a space). Strictly speaking, you may get this sed script to work by escaping the asterisks in your regex (not required in your replacement pattern).
But this only helps for this specific pattern. What if one of your co-workers decides to run this script at 6 minutes after the hour instead of 5, in order to avoid conflict with another script? Or there's a space after the comment character? Suddenly your sed substitution fails.
To uncomment out every commented occurrence of the script in question, you might use:
crontab -l | sed '/# *\([^ ][^ ]* *\)\{5\}[^ ]*test\.sh/s/^# *//' | crontab -
If you're using a more modern sed, you could replace this BRE with a slightly shorter ERE:
crontab -l | sed -E '/# *([^ ]+ *){5}[^ ]*test\.sh/s/^# *//' | crontab -
This takes the output of crontab -l, which is obviously your complete crontab, manipulates it with sed, and then writes a new crontab based on its output using crontab -. The sed script matches searches for lines matching what looks like a valid crontab (to avoid actual comments that simply mention your script), then does a simple substitution to remove only the comment character at the start. The matched pattern breaks out like this:
# * - Matches the comment character followed by zero or more spaces
([^ ]+ +){5} - five non-space strings, followed by spaces
[^ ]* - Any number of non-space characters, which lead up to:
test\.sh - your script.
Note, however, that this doesn't match all valid crontab times, which might include tags like #reboot, #weekly, #midnight, etc. Check out man 5 crontab for details.
A non-sed alternative like awk might be in order. The following awk solution makes more sense to me:
crontab -l | awk -v script="test.sh" '
{ field=6 }
/^# / { field++ }
index($field,script) { sub(/^#/,"") }
1' \
| crontab -
While it's just a little longer, I find it easier to read and understand.

Whether using sed or other tool whould not make much a difference.
I'd use sed also.
But to properly modify what cron is using, please mind to use crontab command.
Do NOT try just editing the data file (e.g. /etc/crontab on some systems). Cron won't pick up the changes!!!.
You might use a pipe:
crontab -l | sed -e "s#\# 5 \* \* \* 3 bash test.sh#5 * * * 3 bash test.sh#"| crontab
to perform the change.
Nevertheless, would it not just be simpler to add the functionality of enabling/disabling into the script being run?

Related

How to truncate extraneous output using shell script? [duplicate]

This question already has answers here:
How to insert strings containing slashes with sed? [duplicate]
(11 answers)
Why it's not possible to use regex to parse HTML/XML: a formal explanation in layman's terms
(10 answers)
Closed 3 years ago.
I am trying to eliminate everything before and after the JSON contained in a specific part of a webpage so I can send that to a PHP script. I've tried a number of ways to get rid of the container content but all of them so far have failed, including one method that has worked in the exact same syntax for related purposes:
The characters that are between the two asterisks (**) at the beginning and end I need removed:
**var songs = [**{"timestamp":1555176393000,"title":"Enter Sandman","trackId":"ba_5cbb546d-5c1c-490e-9908-761b89dd5166","artist":"Metallica","artistId":"52_65f4f0c5-ef9e-490c-aee3-909e7ae6b2ab","album":"Metallica","albumId":"d0_6e729716-c0eb-3f50-a740-96ac173be50d","npe_id":"3cc5fe24d0ffcbb9152d861f27ae801660"},{"timestamp":1555176702000,"title":"Start Me Up","trackId":"76_d0b86399-11e5-4d11-b4fe-ce4b3f9a4736","artist":"The Rolling Stones","artistId":"1b_b071f9fa-14b0-4217-8e97-eb41da73f598","album":"Tattoo You","albumId":"d1_778b345b-e8a1-4054-b5ba-c611d3fda421","npe_id":"f0dc0ab12ef99a6e0087cad12886509b7b"},{"timestamp":1555176909000,"title":"Fame","trackId":"4e_cdef4b88-7314-431a-9cdd-d457296a65b7","artist":"David Bowie","artistId":"ab_5441c29d-3602-4898-b1a1-b77fa23b8e50","album":"Best of Bowie","albumId":"21_3709ee5a-d087-370f-afb4-f730092c7a94","npe_id":"2b8b3a170baa77125891d72a0474d3343a"},{"timestamp":1555177158000,"title":"Rocket","trackId":"34_aa5b9053-849e-4788-972f-7941303175b6","artist":"Def Leppard","artistId":"c1_7249b899-8db8-43e7-9e6e-22f1e736024e","album":"Hysteria","albumId":"06_de5cf055-d875-41f8-9261-89b11b7ff145","npe_id":"0d87b580f140a85feaebc7d77f75db2a3d"},{"timestamp":1555177826000,"title":"Mama, I'm Coming Home","trackId":"cb_e5b09171-9527-4d24-8ab6-1e922fdd66d3","artist":"Ozzy Osbourne","artistId":"4b_8aa5b65a-5b3c-4029-92bf-47a544356934","album":"No More Tears","albumId":"66_8f3d5a65-036c-3260-b9bb-36f1d0d80c11","npe_id":"6b766464fe945f275bf478192dcd33cfdc"},{"timestamp":1555178076000,"title":"Gold Dust Woman","trackId":"a4_ef8c1eca-f344-4bfb-82ea-763aa8aeaad9","artist":"Fleetwood Mac","artistId":"66_bd13909f-1c29-4c27-a874-d4aaf27c5b1a","album":"2010-01-08: The Rock Boat X, Lido Deck, Carnival Inspiration","albumId":"80_4f229af0-2afc-431d-87ff-f7f6af66268e","npe_id":"f6417d98fd1fefcca227d82a8ac9b84197"},{"timestamp":1555178363000,"title":"With or Without You","trackId":"79_6b9a509f-6907-4a6e-9345-2f12da09ba4b","artist":"U2","artistId":"26_a3cb23fc-acd3-4ce0-8f36-1e5aa6a18432","album":"The Joshua Tree","albumId":"0c_d287c703-5c25-3181-85d4-4d8c1a7d8ecd","npe_id":"23b19420196b28e2156ecda87c11b882e0"},{"timestamp":1555178654000,"title":"Who Are You","trackId":"7d_431b9746-c6ec-489d-9199-c83676171ae8","artist":"The Who","artistId":"22_f2fa2f0c-b6d7-4d09-be35-910c110bb342","album":"Who Are You","albumId":"40_b255da2c-6583-35f9-95e3-ef5f9c14e868","npe_id":"e01896f74f24968bb7727eaafbf6250b8f"},{"timestamp":1555179031000,"title":"Authority Song","trackId":"31_f5ff19f7-95f3-4a22-8996-3788c264e0b8","artist":"John Mellencamp","artistId":"4d_0aad6b52-fd93-4ea4-9c5d-1f66e1bc9f0a","album":"Words & Music: John Mellencamp's Greatest Hits","albumId":"9e_1240c510-7015-4484-baac-ce17f5277ea1","npe_id":"244785e3b1d75effb9fdecbb6df76b009f"},{"timestamp":1555179256000,"title":"Touch Me","trackId":"9d_1dd1f86c-2120-45f3-ac9f-3c87257fe414","artist":"The Doors","artistId":"13_9efff43b-3b29-4082-824e-bc82f646f93d","album":"The Soft Parade","albumId":"db_c29d7552-b5df-42b8-aae7-03d1e250cb3a","npe_id":"1b5d155eb2eeee6fc1fdb50a94b100669c"}]**; <ol class="songs tracks"></ol>**
Here is the shell script which produces the above at present:
#!/bin/sh
curl -v --silent http://player.listenlive.co/41851/en/songhistory >/var/tmp/wklh$1.a.txt
pta=`cat /var/tmp/wklh$1.a.txt | grep songs > /var/tmp/wklh$1.b.txt`
ptb=`cat /var/tmp/wklh$1.b.txt | sed -n -e '/var songs = /,/; <span title/ p' > /var/tmp/wklh$1.c.txt`
ptc=`cat /var/tmp/wklh$1.c.txt | grep songs > /var/tmp/wklh$1.d.txt`
#ptd=`cat /var/tmp/wklh$1.d.txt | sed -i 's/var songs = [//g' /var/tmp/wklh$1.d.txt`
#ptd=`cat /var/tmp/wklh$1.d.txt | sed -i 's/}]; <ol class="songs tracks"></ol>//g' /var/tmp/wklh$1.d.txt`
json=`cat /var/tmp/wklh$1.d.txt`
echo $json
metadata=`php /etc/asterisk/scripts/music/wklh.php $json`
echo $metadata
The commented out lines are what I was trying to use to remove the extraneous content, since it is predictable every time. However, when uncommented, I get the following errors:
sed: -e expression #1, char 18: unterminated `s' command
sed: -e expression #1, char 38: unknown option to `s'
I've examined my sed statement, but I can't find any discrepancies between how I use it here and in other working shell scripts.
Is there actually a syntax error here (or unallowed characters)? Or is there a better way I can do this?
Your shell script has serious issues.
The syntax
variable=`commands`
takes the output of commands and assigns it to variable. But in every case, you are redirecting all output to a file; so the variable will always be empty.
Unless you need the temporary files for reasons which are not revealed in your question (such as maybe being able to check how many bytes of output you got in each temporary file for a monitoring report, or something like that), a pipeline would be much superior.
#!/bin/sh
curl -v --silent http://player.listenlive.co/41851/en/songhistory |
grep songs |
sed -n -e '/var songs = /,/; <span title/ p' |
grep songs |
php /etc/asterisk/scripts/music/wklh.php
This also does away with the useless uses of cat and the useless uses of echo and so also coincidentally removes the quoting errors. The grep x | sed -n 's/y/z/p' is a useless use of grep which can easily be refactored to sed -n '/x/s/y/z/p'
Square brackets are special to sed. Simply escape them.
s/var songs = \[//g
If you use slash / as the regex delimiter, it becomes special. Either escape it or use a different delimiter.
s/}]; <ol class="songs tracks"><\/ol>//g
s|}]; <ol class="songs tracks"></ol>||g
if your data in 'd' file, try gnu sed,
sed -Ez 's/^\*\*[^\*]+\*\*(.+)]\*\*[^\*]+\*\*\s*$/\1/' d
remove last ] too, to correctly balance the Json

Replacing multiple lines of text using sed

I have a file input.txt containing the following text
Total users:
abc
xyz
pqrs
The number of users is subject to change but there will be atleast one user all the time. I want to replace the user names with wild card '*' using sed
Total users:
*
Is there a way by which I can look for 'Total users:' string and replace everything after that with a *. Some thing like
sed s/Total users: \n[Till EOF]/Toal users: \n*/g
One way:
$ sed '3,$s/\S*$/*/' file
Total users:
*
*
*
This does the substitution s/\S*$/*/ from the third line in the file till the last line 3,$ where the substitution replaces any none whitespace characters till the end of line with a single *. Modified the substitution command as appropriate for your actual file as this will fail if you allow spaces in usernames. A more robust replacement might be:
$ sed -r '3,$s/(\s+).*/\1*/' file
Total users:
*
*
*
This will replace after the initial whitespace with a single *. Use the -i option if you want to store the changes back to the file:
$ sed -ri '3,$s/(\s+).*/\1*/' file
Edit:
To replace all users with a single *:
$ sed -r '3{s/(\s+).*/\1*/;q}' file
Total users:
*
Although creating this file would have been much quicker than asking a question.
This might be what you need
sed -e "/Total users:/b; s|^\([[:blank:]]*\)[^[:blank:]]\+\(.*\)|\1*\2|" input.txt
This will do it:
sed -e '/^Total users:/ s/.*/&\n\n */; q' input.txt
When it finds a line starting with Total users:, it replaces with itself, appends two line breaks and asterisk, and exits without processing any further lines.
If you are using a more limited version of sed where you cannot use ; to separate multiple commands, you can write like this to work around:
sed -e '/^Total users:/ s/.*/&\n\n */' -e '/^Total users:/ q' input.txt
Which is more verbose, but more portable.

using sed to write a shell script [duplicate]

This question already has answers here:
shell scripting using sed
(3 answers)
Closed 9 years ago.
So, I want to read file from stdin, delete all '/' in line that contain exactly 3 '/', and write the output to stdout. So a file contain:
/a1/b/c
/a/b2
///
/a
will have output:
a1bc
/a/b2
/a
I am thinking something like this:
sed -r 's/\/[^\/]*\/[^\/]*\/.*/"I not sure what do I need to put in here"/g'
however, I am not really sure what do I need to put in the replace session.
A sed solution:
sed '/.*\/.*\/.*\//{s#/##g}' file
If Perl is ok for you:
perl -F/ -ape '$_=#F>3?join"",#F:join "/",#F;' file
sed -e '/^[^\/]*\/[^\/]*\/[^\/]*\/[^\/]*$/ s%/%%g'
The gruesome pattern looks for start of line, a sequence of zero or more non-slashes followed by a slash, more non-slashes and a second slash, more non-slashes and a third slash, more non-slashes and the end of line. On any line that matches that, substitute the slashes by nothing globally.
There are other ways to write the regex, but they aren't substantially clearer. This will work in pretty much any version of sed. So will this:
sed -e '/^\([^\/]*\/\)\{3\}[^\/]*$/ s%/%%g'
It looks for start of line, 3 units of (zero or more non-slashes followed by a slash), zero or more non-slashes and end of line.
If your sed has extended regular expressions (GNU sed, for example), then you can gain some notational convenience.
sed -r -e '/^([^\/]*\/){3}[^\/]*$/ s%/%%g'
sed -r -e 's%^([^/]*)/([^/]*)/([^/]*)/([^/]*)$%\1\2\3\4%'
The latter captures the four sets of 'zero or more non-slashes' and pastes them together to make the replacement. You could write that with the non-extended regular expressions, but it would be even more laden with backslashes than before.
This is much simpler in awk:
awk -F/ 'NF==4 { gsub("/","") } {print}' tmp.txt

Insert line after match using sed

For some reason I can't seem to find a straightforward answer to this and I'm on a bit of a time crunch at the moment. How would I go about inserting a choice line of text after the first line matching a specific string using the sed command. I have ...
CLIENTSCRIPT="foo"
CLIENTFILE="bar"
And I want insert a line after the CLIENTSCRIPT= line resulting in ...
CLIENTSCRIPT="foo"
CLIENTSCRIPT2="hello"
CLIENTFILE="bar"
Try doing this using GNU sed:
sed '/CLIENTSCRIPT="foo"/a CLIENTSCRIPT2="hello"' file
if you want to substitute in-place, use
sed -i '/CLIENTSCRIPT="foo"/a CLIENTSCRIPT2="hello"' file
Output
CLIENTSCRIPT="foo"
CLIENTSCRIPT2="hello"
CLIENTFILE="bar"
Doc
see sed doc and search \a (append)
Note the standard sed syntax (as in POSIX, so supported by all conforming sed implementations around (GNU, OS/X, BSD, Solaris...)):
sed '/CLIENTSCRIPT=/a\
CLIENTSCRIPT2="hello"' file
Or on one line:
sed -e '/CLIENTSCRIPT=/a\' -e 'CLIENTSCRIPT2="hello"' file
(-expressions (and the contents of -files) are joined with newlines to make up the sed script sed interprets).
The -i option for in-place editing is also a GNU extension, some other implementations (like FreeBSD's) support -i '' for that.
Alternatively, for portability, you can use perl instead:
perl -pi -e '$_ .= qq(CLIENTSCRIPT2="hello"\n) if /CLIENTSCRIPT=/' file
Or you could use ed or ex:
printf '%s\n' /CLIENTSCRIPT=/a 'CLIENTSCRIPT2="hello"' . w q | ex -s file
Sed command that works on MacOS (at least, OS 10) and Unix alike (ie. doesn't require gnu sed like Gilles' (currently accepted) one does):
sed -e '/CLIENTSCRIPT="foo"/a\'$'\n''CLIENTSCRIPT2="hello"' file
This works in bash and maybe other shells too that know the $'\n' evaluation quote style. Everything can be on one line and work in
older/POSIX sed commands. If there might be multiple lines matching the CLIENTSCRIPT="foo" (or your equivalent) and you wish to only add the extra line the first time, you can rework it as follows:
sed -e '/^ *CLIENTSCRIPT="foo"/b ins' -e b -e ':ins' -e 'a\'$'\n''CLIENTSCRIPT2="hello"' -e ': done' -e 'n;b done' file
(this creates a loop after the line insertion code that just cycles through the rest of the file, never getting back to the first sed command again).
You might notice I added a '^ *' to the matching pattern in case that line shows up in a comment, say, or is indented. Its not 100% perfect but covers some other situations likely to be common. Adjust as required...
These two solutions also get round the problem (for the generic solution to adding a line) that if your new inserted line contains unescaped backslashes or ampersands they will be interpreted by sed and likely not come out the same, just like the \n is - eg. \0 would be the first line matched. Especially handy if you're adding a line that comes from a variable where you'd otherwise have to escape everything first using ${var//} before, or another sed statement etc.
This solution is a little less messy in scripts (that quoting and \n is not easy to read though), when you don't want to put the replacement text for the a command at the start of a line if say, in a function with indented lines. I've taken advantage that $'\n' is evaluated to a newline by the shell, its not in regular '\n' single-quoted values.
Its getting long enough though that I think perl/even awk might win due to being more readable.
A POSIX compliant one using the s command:
sed '/CLIENTSCRIPT="foo"/s/.*/&\
CLIENTSCRIPT2="hello"/' file
Maybe a bit late to post an answer for this, but I found some of the above solutions a bit cumbersome.
I tried simple string replacement in sed and it worked:
sed 's/CLIENTSCRIPT="foo"/&\nCLIENTSCRIPT2="hello"/' file
& sign reflects the matched string, and then you add \n and the new line.
As mentioned, if you want to do it in-place:
sed -i 's/CLIENTSCRIPT="foo"/&\nCLIENTSCRIPT2="hello"/' file
Another thing. You can match using an expression:
sed -i 's/CLIENTSCRIPT=.*/&\nCLIENTSCRIPT2="hello"/' file
Hope this helps someone
The awk variant :
awk '1;/CLIENTSCRIPT=/{print "CLIENTSCRIPT2=\"hello\""}' file
I had a similar task, and was not able to get the above perl solution to work.
Here is my solution:
perl -i -pe "BEGIN{undef $/;} s/^\[mysqld\]$/[mysqld]\n\ncollation-server = utf8_unicode_ci\n/sgm" /etc/mysql/my.cnf
Explanation:
Uses a regular expression to search for a line in my /etc/mysql/my.cnf file that contained only [mysqld] and replaced it with
[mysqld]
collation-server = utf8_unicode_ci
effectively adding the collation-server = utf8_unicode_ci line after the line containing [mysqld].
I had to do this recently as well for both Mac and Linux OS's and after browsing through many posts and trying many things out, in my particular opinion I never got to where I wanted to which is: a simple enough to understand solution using well known and standard commands with simple patterns, one liner, portable, expandable to add in more constraints. Then I tried to looked at it with a different perspective, that's when I realized i could do without the "one liner" option if a "2-liner" met the rest of my criteria. At the end I came up with this solution I like that works in both Ubuntu and Mac which i wanted to share with everyone:
insertLine=$(( $(grep -n "foo" sample.txt | cut -f1 -d: | head -1) + 1 ))
sed -i -e "$insertLine"' i\'$'\n''bar'$'\n' sample.txt
In first command, grep looks for line numbers containing "foo", cut/head selects 1st occurrence, and the arithmetic op increments that first occurrence line number by 1 since I want to insert after the occurrence.
In second command, it's an in-place file edit, "i" for inserting: an ansi-c quoting new line, "bar", then another new line. The result is adding a new line containing "bar" after the "foo" line. Each of these 2 commands can be expanded to more complex operations and matching.

Sed syntax - break down

I'm wondering if someone can help me to understand the following command. The purpose of this command is to purge the old kernels however I would like to understand the syntax:
dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge
This is what I have so far:
dpkg -l 'linux-*' - list packages contain "linux-*" pattern
sed '/^ii/!d;/'... - find the lines starting with ii but don't contain d;
"$(uname -r | sed "s/(.*)-([^0-9]+)/\1/")"'/d - command substitution to list the current kernel and extract only the number from the version, sed "s/(.*)... - search for any number of characters, ...([^0-9]... - starting with numbers 0-9, I don't understand this bit: ...+)/\1/...
I'm completely lost with this one:
s/^[^ ]* [^ ]* ([^ ])./\1/;/[0-9]/!d' - is it looking for empty characters starting the string?
Regards
So, we have one long sed command here to analyse:
'/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d'
It's fed a list of packages matching 'linux-*' in dpkg syntax, which is a bit cryptic to start with, but the start of the lines indicates information about the package state.
The sed command itself is quite long (though I'm guilty of worse!), but fairly straightforward, with no looping or complications. sed, as the Fine Manual would tell you, accepts programs as a list of commands separated by semi-colons. So, the given sed program has four commands.
Firstly, '/^ii/!d'. We delete lines which don't start with 'ii'. Simple enough. We're looking for installed packages to remove.
Secondly, '/'"$(uname -r | sed "s/(.*)-([^0-9]+)/\1/")"'/d'. This is a command-line within a command-line (using bash's $() syntax), so we'll work from the inside out. The aim is clearly to filter out from the list of packages our currently running kernel so we don't remove it. The output from uname -r doesn't quite match the debian package names, so it's filtered from something like "3.0.0-generic" to just "3.0.0". The inner sed command, "s/(.*)-([^0-9]+)/\1/")", is just a simple regular expression search-and-replace that cuts off the end of the input after any hyphen. After $() substitution, the outer sed command looks like this '/3.0.0/d' (depending on kernel you're running). It just deletes lines that contain your current kernel version number.
Thirdly and fourthly, 's/^[^ ]* [^ ]* ([^ ])./\1/' and '/[0-9]/!d' are the final filters. The first one again is a basic search-and-replace (it would be much clearer done in awk) that extracts the third field of the line, which will be the package name, and after that we delete lines which don't contain a digit (or else we'd remove the meta-packages that pull in the kernel upgrades).
Finally, xargs grabs its input line-by-line and runs the command you pass it on each line, with the line appended to its arguments. So, we remove each package we've found.

Resources