I have an ubuntu server where I schedule crontab processes like the following.
59 2 * * * : Backup Settings; ~/backup_settings.sh
At the conclusion of the process, I will get an email with the subject line "Backup Settings ...". Essentially the noop function (:) does nothing with the words "Backup Settings". I would like to add today's date to the email subject. Naturally, I tried
59 2 * * * : $(date +%Y%m%d) Backup Settings; ~/backup_settings.sh
but that doesn't result in the desired email subject, i.e. "20180519 Backup Settings". The $(...) code gets unevaluated. I don't want to run another script with email functionality that will then call backup_settings.sh. Is there a way of doing it using just Bash commands in crontab?
The character % is special in the crontab and has to be escaped as \%:
59 2 * * * : $(date +\%Y\%m\%d) Backup Settings; "$HOME/backup_settings.sh"
From man 5 crontab on an Ubuntu system:
The entire command portion of the line, up to a newline or % character, will be executed by
/bin/sh or by the shell specified in the SHELL variable of the crontab file. Percent-signs (%) in the command, unless escaped with backslash (\), will be changed into
newline characters, and all data after the first % will be sent to the command as standard input.
Note, though, that cron will put the verbatim command of the cronjob in as the subject in any email that it sends, not the expanded command line.
To send an email with your own title, use mail explicitly:
59 2 * * * "$HOME/backup_settings.sh" | mail -s "$(date +\%Y\%m\%d) Backup Settings" myname
(where myname is the address you'd like to send the email to).
Related
I'm working on a ksh script to retrieve a file every hour via sftp that will be put on a job scheduler to be run hourly. The script must navigate to a folder with yesterday's date (remote/path/yyyymmdd/). The filename also has yesterday's date and a timestamp (filename_yyyymmdd_hhmmss.dat). Since the job will be scheduled, my script has to include the previous hour - ex. if the job runs at 11:02, the file to retrieve would be filename_yyyymmdd_10mmss.dat. The minutes and seconds will always be the same - ex 4949. There will be multiple files in the remote directory and I only want to retrieve the latest one so that there are not multiple input files being processed by our jobs. The remote directory will also have other files being created regularly, so I can't retrieve just the last modified files.
I have variables to return yesterday's date and the previous hour, but the sftp command isn't returning the full filename and isn't retrieving the file. I've tried concatenating the variables, using brackets & quotes & parenthesis, assigning multiple variables to a single variable, and exporting the variables.
vdate=$(TZ=bb24 date '+%Y%m%d')
vhour=$(date '+%H')
prevhour=$((vhour - 1))
sftp user#host << EOF
lcd /my/dir/
cd /remote/path/$vdate/
get filename_$vdate_$prevhour*.dat
bye
EOF
exit
When running the script, the file cannot be found and the full filename isn't
returned:
File "/remote/path/20190411/filename_20190411" not found.
instead of
File "/remote/path/20190411/filename_20190411_10*.dat" not found.
Every combination of variables that I try returns the same not found - ending after filename_$vdate.
I've tried some other combinations but always get the same not found:
newvar=${vdate}_${prevhour}
get filename_$newvar*.dat
and
newvar=${vdate}\\_${prevhour}
get filename_$newvar*.dat
File "/remote/path/20190411/filename_20190411" not found.
You have a problem in your script at prevhour=$((vhour - 1))
this way a text 02 after you make subtraction, it will be 1 and not 01 and it will match to undesired files, or even none as 00 - 1 is -1
[edvin]$ vdate=$(TZ=bb24 date '+%Y%m%d')
[edvin]$ vhour=$(date '+%H')
[edvin]$ prevhour=$((vhour - 1))
[edvin]$ echo $vhour
03
[edvin]$ echo $prevhour
2
[edvin]$ prevhour=$(date -d '1 hour ago' '+%H')
[edvin]$ echo $prevhour
02
date's -d option not avaliable on some system.
I believe in that in your attempt the shell is considered the * as part of the variable prevhour as you did not put it into {} that separate variables from sorrunding text.
This is my working solution based by your attempt:
#!/bin/ksh
r_host='server2'
r_user='edvin'
l_dir='./content'
r_dir='./Test_folder'
# this still not cover the case of midnight
# it put 00 to 23 but day have to be yesterday as well
##vdate=$(TZ=bb24 date '+%Y%m%d')
##vhour=$(date '+%H') # not used
##prevhour=$(date -d '1 hour ago' '+%H')
# vtime = YYYYmmdd_HH -1 H
vtime=$(TZ=bb24 date -d '1 hour ago' '+%Y%m%d_%H')
sftp ${r_user}#${r_host} << EOF
lcd ${l_dir}
cd ${r_dir}
get filename_${vtime}*.dat
bye
EOF
exit
Output:
[edvin]$ ./script.ksh
Connected to server2.
sftp> lcd ./content
sftp> cd ./Test_folder
sftp> get filename_20190415_02*.dat
Fetching /home/edvin/Test_folder/filename_20190415_020000.dat to filename_20190415_020000.dat
Fetching /home/edvin/Test_folder/filename_20190415_020100.dat to filename_20190415_020100.dat
Fetching /home/edvin/Test_folder/filename_20190415_020200.dat to filename_20190415_020200.dat
Fetching /home/edvin/Test_folder/filename_20190415_020300.dat to filename_20190415_020300.dat
Fetching /home/edvin/Test_folder/filename_20190415_020400.dat to filename_20190415_020400.dat
Fetching /home/edvin/Test_folder/filename_20190415_020500.dat to filename_20190415_020500.dat
Fetching /home/edvin/Test_folder/filename_20190415_020600.dat to filename_20190415_020600.dat
Fetching /home/edvin/Test_folder/filename_20190415_020700.dat to filename_20190415_020700.dat
Fetching /home/edvin/Test_folder/filename_20190415_020800.dat to filename_20190415_020800.dat
Fetching /home/edvin/Test_folder/filename_20190415_020900.dat to filename_20190415_020900.dat
Fetching /home/edvin/Test_folder/filename_20190415_021000.dat to filename_20190415_021000.dat
sftp> bye
There is many thing can go wrong still in this solution,
like if remote directory not exist, not accessible, script will still go on with the rest of the command, same for the local directory and for the files as well. The connection also can run various problems you might want to handle. You like to schedule it so might a solution needed to avoid script spawn over and over again if one already run.
scp would be more preferred way to do this, as you use password less authentication.
If scp is not an option for some reason, with expect this can be handled quite well.
I use the following script to retrieve information about mounted file-systems on several hundred Solaris (v9,10,11) and Red Hat Enterprise Linux (v5,6,7) servers for analysis.
# retrieves for all mounted file-systems: server, device, allocated, used, available, percent_used, mount_directory, permissions, owner_name, and group_name
server=$(uname -n)
df -h | awk '
NF == 6 { print ($0); }
NF == 1 { device = $1; }
NF == 5 { print (device, " ", $0); }
' | while read device allocated used available percent mount
do
ls -ld "${mount}" | read permissions links owner_name group_name size month day time directory
echo "${server} ${device} ${allocated} ${used} ${available} ${percent} ${mount} ${permissions} ${owner_name} ${group_name}"
done
I perform this operation from Windoze using PuTTY "plink" utility.
plink -m filesys.script server_name >>filesys.txt
All worked as expected until my default shell was changed from ksh to bash on all servers. Now, the second read command that obtains ls output for permissions, owner_name, and group_name is not functioning and does not produce any error messages either. Therefore the result is that only seven tokens are in output (server through mount) and there is nothing for permissions, owner_name, or group_name.
I have confirmed that if I upload the script to the Unix server with a shebang (#!/bin/ksh) at the top line the script works as expected. However, I do not want to push this script to hundreds of servers and maintain the script in a distributed mechanism. I would like to retain the script on central Windoze workstation and call with -m parameter of plink. Placing a shabang at top of the file does not execute ksh using plink -m option.
The Bash shell versions that are in play are 3.2 and 4.1. I have also made certain that the Windoze script file has carriage returns removed. The awk utility is used to handle situations where the device name is too long and df breaks the output over two lines.
Again, the first read (from df/awk) is working fine but the second (ls output) is not. I confirmed by placing a 'set' following the second read and those environment varriables were not in the environment.
The read (as a pipe element) happens in a subshell, so even though it actually does execute perfectly, once that pipeline exits its results aren't available to the echo running on a separate line (as part of the parent process that originally spawned the pipeline). This is fully allowed by POSIX; which component of a pipeline, if any, is performed by the shell spawning that pipeline is unspecified by the standard and thus implementation-defined.
You can address the issue by putting the echo inside of the same pipeline element as the read:
server=$(uname -n)
df -h | awk '
NF == 6 { print ($0); }
NF == 1 { device = $1; }
NF == 5 { print (device, " ", $0); }
' | while read device allocated used available percent mount
do
# NOTE: parsing output from "ls" is unreliable
ls -ld "${mount}" | {
read permissions links owner_name group_name size month day time directory
echo "${server} ${device} ${allocated} ${used} ${available} ${percent} ${mount} ${permissions} ${owner_name} ${group_name}"
}
done
References:
BashFAQ #24 (I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?)
ParsingLs (Why you shouldn't parse the output of ls(1))
If you have GNU stat or find, either of which allows you to provide a format string to control metadata output, I would strongly suggest using them in place of ls -l for parsing metadata. Even perl is somewhat better for the purpose, having only a single universally available implementation with uniform stat behavior between releases.
I have setup a cron job as shown below but it won't run. When I run the script manually, I don't see any errors.
#_____WPR Jobs
00 9 * * * mon-sat /var/spool/ftpexts/bin/exe_get_x_wpr.sh >> /var/spool/ftpexts/outboundlogs/exe_get_x_wpr.log
00 9 * * * mon-sat /var/spool/ftpexts/bin/exe_get_y_wpr.sh >> /var/spool/ftpexts/outboundlogs/exe_get_y_wpr.log
00 9 * * * mon-sat /var/spool/ftpexts/bin/exe_get_z_wpr.sh >> /var/spool/ftpexts/outboundlogs/exe_get_z_wpr.log
When I execute the script manually as shown below, it runs smoothly with log records too.
/var/spool/ftpexts/bin/exe_get_x_wpr.sh >> /var/spool/ftpexts/outboundlogs/exe_get_x_wpr.log
crontab is trying to execute mon-sat as a command.
The day of the week is specified as the 5th field of a crontab entry. You have *, which means it runs on any day of the week. Delete that 5th field, making mon-sat the 5th field. (Interesting, I didn't know until now that crontab would recognize names.)
UPDATE: The crontab(5) man page (type man 5 crontab to read it on your system) says:
Names can also be used for the "month" and "day of week" fields. Use
the first three letters of the particular day or month (case doesn't
matter). Ranges or lists of names are not allowed.
You say that mon-sat worked for you. A quick experiment indicates that ranges of names actually do work, but since the documentation says they're not allowed, I suggest not depending on that. Write 1-6 rather than mon-sat if you want the job to run Monday through Saturday.
I need some help with displaying how many times two strings are found on the same line! Lets say I want to search the file 'test.txt', this file contains names and IP's, I want to enter a name as a parameter when running the script, the script will search the file for that name, and check if there's an IP-address there also. I have tried using the 'grep' command, but I don't know how I can display the results in a good way, I want it like this:
Name: John Doe IP: xxx.xxx.xx.x count: 3
The count is how many times this line was found, this is how my grep script looks like right now:
#!/bin/bash
echo "Searching $1 for the Name '$2'"
result=$(grep "$2" $1 | grep -E "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)")
echo $result
I will run the script like 'sh search test.txt John'.
I'm having trouble displaying the information I get from the grep command, maybe there's a better way to do this?
EDIT:
Okey, I will try to explain a little better, let's say I want to search a .log file, I want a script to search that file for a string the user enters as a parameter. i.e if the user enters 'sh search test.log logged in' the script will search for the string "logged in" within the file 'test.log'. If the script finds this line on the same line as a IP-address the IP address is printed, along with how many times this line was found.
And I simply don't know how to do it, I'm new to shell scripting, and was hoping I could use grep along with regular expressions for this! I will keep on trying, and update this question with an answer if I figure it out.
I don't have said file on my computer, but it looks something like this:
Apr 25 11:33:21 Admin CRON[2792]: pam_unix(cron:session): session opened for user 192.168.1.2 by (uid=0)
Apr 25 12:39:01 Admin CRON[2792]: pam_unix(cron:session): session closed for user 192.168.1.2
Apr 27 07:42:07 John CRON[2792]: pam_unix(cron:session): session opened for user 192.168.2.22 by (uid=0)
Apr 27 14:23:11 John CRON[2792]: pam_unix(cron:session): session closed for user 192.168.2.22
Apr 29 10:20:18 Admin CRON[2792]: pam_unix(cron:session): session opened for user 192.168.1.2 by (uid=0)
Apr 29 12:15:04 Admin CRON[2792]: pam_unix(cron:session): session closed for user 192.168.1.2
Here is a simple Awk script which does what you request, based on the log snippet you posted.
awk -v user="$2" '$4 == user { i[$11]++ }
END { for (a in i) printf ("Name: %s IP: %s count: %i\n", user, a, i[a]) }' "$1"
If the fourth whitespace-separated field in the log file matches the requested user name (which was passed to the shell script as its second parameter), add one to the count for the IP address (from field 11).
At the end, loop through all non-zero IP addresses, and print a summary for each. (The user name is obviously whatever was passed in, but matches your expected output.)
This is a very basic Awk script; if you think you want to learn more, I urge you to consult a simple introduction, rather than follow up here.
If you want a simpler grep-only solution, something like this provides the information in a different format:
grep "$2" "$1" |
grep -o -E '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)' |
sort | uniq -c | sort -rn
The trick here is the -o option to the second grep, which extracts just the IP address from the matching line. It is however less precise than the Awk script; for example, a user named "sess" would match every input line in the log. You can improve on that slightly by using grep -w in the first grep -- that still won't help against users named "pam" --, but Awk really gives you a lot more control.
My original answer is below this line, partly becaus it's tangentially useful, partially because it is required in order to understand the pesky comment thread below.
The following
result=$(command)
echo $result
is wrong. You need the second line to be
echo "$result"
but in addition, the detour over echo is superfluous; the simple way to write that is simply
command
I have a shell script for getting data from some servers, in ~/.bin/shellScript :
OLD="$(curl --silent http://someServer:12345/stats.json | json someKey)"
NEW="$(curl --silent http://otherServer:12345/stats.json | json someKey")
echo "OLD $OLD - NEW $NEW"
I want to echo the results for running it interactively, but I've been wanting to log the results collected too.
So crontab -e, and add */5 * * * * /home/user/.bin/shellScript >> /media/dump/scriptDump.
Running the script interactively works fine - I get OLD 123 - NEW 456, but when I look at what's been running from cron, I get OLD - NEW with no data being grabbed with curl.
As discovered in the comments, you need to add the full path of json when you are calling it. This is because crontab's limited environment.
So instead of
OLD="$(curl --silent http://someServer:12345/stats.json | json someKey)"
NEW="$(curl --silent http://otherServer:12345/stats.json | json someKey")
it has to be
OLD="$(curl --silent http://someServer:12345/stats.json | /path/to/json someKey)"
NEW="$(curl --silent http://otherServer:12345/stats.json | /path/to/json someKey)"
^^^^^^^^^^^^^
[[Note your second line had ") instead of )"]]
Otherwise, you can also add the json path into crontab, as indicated on How to get CRON to call in the correct paths:
PATH=/usr/local/sbin: ... :/path/of/json