Being a relative beginner, I can't figure this out. I have a script that is started via cron. Within this script is an if/fi where I check to see if a (yearly archive) directory does not exist. It it does not, I create the directory, and ATTEMPT to echo that to the cron's log file that is created for each run. The directory is created, but the echo does not appear in the log file.
Here is a snippet of the code in question.
035: yyyy=`date +%Y`
036: today=`date +%m/%d/%Y`
037: time=`date +%r` #+%l:%M:%S%P`
038: dayofweek=`date +%A`
039: numDayOfWeek=`date +%u`
040:
041: echo "Run Date/Time: $today $time"
042:
043: WFADIR="/data/ssa1/home1/NEI/GAP-EFT-FLAT/$yyyy"
044: if [ ! -d $WFADIR ] ; then
045: mkdir /data/ssa1/home1/NEI/GAP-EFT-FLAT/$yyyy
046: chmod 777 /data/ssa1/home1/NEI/GAP-EFT-FLAT/$yyyy
047: echo ""
048: echo "New folder $yyyy created in GAP-EFT-FLAT"
049: fi
050:
051: #display test variables for output
052: echo ""
053: echo "HOSTNAME..........: ${HOSTNAME^^}"
054: echo ""
055:
And here is the FULL log file.
Run Date/Time: 01/03/2023 08:00:01 AM
HOSTNAME..........: BASYSPROD
EFT contribution file found...
Calling expect script to transmit contribution file...
spawn sftp -P 22 -i privatekey.pem username#domain.com:/inbound/NATIO080_ACH_3
Connected to domain.com.
Changing to: /inbound/NATIO080_ACH_3
sftp> put B06737_CON_20230103
Uploading B06737_CON_20230103 to /inbound/NATIO080_ACH_3/B06737_CON_20230103
B06737_CON_20230103 0% 0 0.0KB/s --:-- ETA
B06737_CON_20230103 100% 2470 70.0KB/s 00:00
sftp> Returned from contribution expect script...
Archiving sent contribution file...
Sending email confirmation...
Process completed...
EFT 401K file found...
Calling expect script to transmit 401K file...
spawn sftp -P 22 -i privatekey.pem username#domain.com:/inbound/NATIO080_ACH_4
Connected to domain.com.
Changing to: /inbound/NATIO080_ACH_4
sftp> put B06736_401K_20230103
Uploading B06736_401K_20230103 to /inbound/NATIO080_ACH_4/B06736_401K_20230103
B06736_401K_20230103 0% 0 0.0KB/s --:-- ETA
B06736_401K_20230103 100% 7980 216.4KB/s 00:00
sftp> Returned from 401K expect script...
Archiving sent 401K file...
Sending email confirmation...
As you can see, the echo from line 41 is in the log file. Then, as this was the first run for 2023, the 2023 directory did not yet exist. It WAS created and the permissions were changed as well, with lines 45 and 46, respectively.
drwxrwxrwx. 2 neiauto staff 61 Jan 3 08:00 2023
So why do lines 47 and 48 appear not to execute, and the next echo in the log file is from line 52, 53 and 54, with the hostname display, surrounded by blank lines?
I was expecting a blank line, and "New folder 2023 created in GAP-EFT-FLAT" to be echoed after the Run date/time (first) line of the log file, and before the host name display.
Very likely your directory already existed. Add an else echo $WFADIR already exists to your code to have your answer next year :-). My guess would be that the same code was run twice (on the same, or another host if shared disk-space was used).
Related
I have written a Bash script to sync all files from a local folder to s3 bucket. And all the logs are sent to email when the script is run through the cron job. The email output looks like this:
-----------------------Started at: Wed 3 Mar 10:56:01 +04 2021---------------------------
Start uploading to s3 bucket mohsin7007
Completed 1.8 KiB/15.5 KiB (5.2 KiB/s) with 3 file(s) remaining
upload: ../home/mohsin/Desktop/data2s3/README.md to s3://mohsin7007/README.md
Completed 1.8 KiB/15.5 KiB (5.2 KiB/s) with 2 file(s) remaining
Completed 12.9 KiB/15.5 KiB (15.0 KiB/s) with 2 file(s) remaining
upload: ../home/mohsin/Desktop/data2s3/LICENSE to s3://mohsin7007/LICENSE
Completed 12.9 KiB/15.5 KiB (15.0 KiB/s) with 1 file(s) remaining
Completed 15.5 KiB/15.5 KiB (18.0 KiB/s) with 1 file(s) remaining
------------------------Completed at: Wed 3 Mar 10:56:01 +04 2021---------------------------
I would like to bold out the "Started at" and "completed at" lines. So it will be more readable when looking for the logs from a certain date, like this:
-------------------------Started at: Wed 3 Mar 10:56:01 +04 2021---------------------------
I have used Tput utility to make these lines bold, however, the issue is when I run the script in the terminal, the outcome is as expected. However, when I open the log file or view the logs in email, The results are not bold lines.
Could you guys please help me that how can I get the above-mentioned lines in bold letters
I am pasting here my script as well.
#!/bin/sh
DEST=mytestingbucket8719
SOURCE=/home/sham/Desktop/data2s3
Date=`date`
bold=$(tput bold)
normal=$(tput sgr0)
echo " "
echo "$bold-----------------------Started at: $Date---------------------------$normal"
echo "Start uploading to s3 bucket" $BUCKET
aws s3 sync $SOURCE s3://$DEST
echo "complete uploading to s3 bucket" $BUCKET
echo "$bold------------------------Completed at: $Date---------------------------$normal"
echo " "
I have a bash script running on Ubuntu 18.04. I scheduled it using SYSTEMD timer.
#!/bin/bash
backupdb(){
/usr/bin/mysqldump -u backupuser -pbackuppassword --add-locks --extended-insert --hex-blob $1 > /opt/mysqlbackup/$1.sql
/bin/gzip -c /opt/mysqlbackup/$1.sql > /opt/mysqlbackup/$1-$(date +%A).sql.gz
rm -rf /opt/mysqlbackup/$1.sql
echo `date "+%h %d %H:%M:%S"`": " $1 "- Size:" `/usr/bin/stat -c%s "${1}-$(date +%A).sql.gz"` >> /opt/mysqlbackup/backupsql.log
}
# List of databases to backup
backupdb cardb
backupdb bikedb
When I run this script interactively, the backup log get 2 entries:
Jun 16 20:15:03: cardb - Size: 200345
Jun 16 20:15:12: bikedb - Size: 150123
However, when this is run as a SYSTEMD timer service, the log still gets 2 entries but no file size is given in the log file. Not 0, it's simply blank. The backup file, cardb.sql.gz is created and is non-zero. I can unzip it and it does contain a valid SQL file.
I can't figure out why this is happening.
You need to specify the absolute path of your file
Without specifying the absolute path you are making the assumption that the systemd timer is running your script from the same directory you tested it from. To remedy this, you can either use the absolute path or change directories before accessing your file.
echo `date "+%h %d %H:%M:%S"`": " $1 "- Size:" `/usr/bin/stat -c%s "/opt/mysqlbackup/${1}-$(date +%A).sql.gz"` >> /opt/mysqlbackup/backupsql.log
I'm working on a ksh script to retrieve a file every hour via sftp that will be put on a job scheduler to be run hourly. The script must navigate to a folder with yesterday's date (remote/path/yyyymmdd/). The filename also has yesterday's date and a timestamp (filename_yyyymmdd_hhmmss.dat). Since the job will be scheduled, my script has to include the previous hour - ex. if the job runs at 11:02, the file to retrieve would be filename_yyyymmdd_10mmss.dat. The minutes and seconds will always be the same - ex 4949. There will be multiple files in the remote directory and I only want to retrieve the latest one so that there are not multiple input files being processed by our jobs. The remote directory will also have other files being created regularly, so I can't retrieve just the last modified files.
I have variables to return yesterday's date and the previous hour, but the sftp command isn't returning the full filename and isn't retrieving the file. I've tried concatenating the variables, using brackets & quotes & parenthesis, assigning multiple variables to a single variable, and exporting the variables.
vdate=$(TZ=bb24 date '+%Y%m%d')
vhour=$(date '+%H')
prevhour=$((vhour - 1))
sftp user#host << EOF
lcd /my/dir/
cd /remote/path/$vdate/
get filename_$vdate_$prevhour*.dat
bye
EOF
exit
When running the script, the file cannot be found and the full filename isn't
returned:
File "/remote/path/20190411/filename_20190411" not found.
instead of
File "/remote/path/20190411/filename_20190411_10*.dat" not found.
Every combination of variables that I try returns the same not found - ending after filename_$vdate.
I've tried some other combinations but always get the same not found:
newvar=${vdate}_${prevhour}
get filename_$newvar*.dat
and
newvar=${vdate}\\_${prevhour}
get filename_$newvar*.dat
File "/remote/path/20190411/filename_20190411" not found.
You have a problem in your script at prevhour=$((vhour - 1))
this way a text 02 after you make subtraction, it will be 1 and not 01 and it will match to undesired files, or even none as 00 - 1 is -1
[edvin]$ vdate=$(TZ=bb24 date '+%Y%m%d')
[edvin]$ vhour=$(date '+%H')
[edvin]$ prevhour=$((vhour - 1))
[edvin]$ echo $vhour
03
[edvin]$ echo $prevhour
2
[edvin]$ prevhour=$(date -d '1 hour ago' '+%H')
[edvin]$ echo $prevhour
02
date's -d option not avaliable on some system.
I believe in that in your attempt the shell is considered the * as part of the variable prevhour as you did not put it into {} that separate variables from sorrunding text.
This is my working solution based by your attempt:
#!/bin/ksh
r_host='server2'
r_user='edvin'
l_dir='./content'
r_dir='./Test_folder'
# this still not cover the case of midnight
# it put 00 to 23 but day have to be yesterday as well
##vdate=$(TZ=bb24 date '+%Y%m%d')
##vhour=$(date '+%H') # not used
##prevhour=$(date -d '1 hour ago' '+%H')
# vtime = YYYYmmdd_HH -1 H
vtime=$(TZ=bb24 date -d '1 hour ago' '+%Y%m%d_%H')
sftp ${r_user}#${r_host} << EOF
lcd ${l_dir}
cd ${r_dir}
get filename_${vtime}*.dat
bye
EOF
exit
Output:
[edvin]$ ./script.ksh
Connected to server2.
sftp> lcd ./content
sftp> cd ./Test_folder
sftp> get filename_20190415_02*.dat
Fetching /home/edvin/Test_folder/filename_20190415_020000.dat to filename_20190415_020000.dat
Fetching /home/edvin/Test_folder/filename_20190415_020100.dat to filename_20190415_020100.dat
Fetching /home/edvin/Test_folder/filename_20190415_020200.dat to filename_20190415_020200.dat
Fetching /home/edvin/Test_folder/filename_20190415_020300.dat to filename_20190415_020300.dat
Fetching /home/edvin/Test_folder/filename_20190415_020400.dat to filename_20190415_020400.dat
Fetching /home/edvin/Test_folder/filename_20190415_020500.dat to filename_20190415_020500.dat
Fetching /home/edvin/Test_folder/filename_20190415_020600.dat to filename_20190415_020600.dat
Fetching /home/edvin/Test_folder/filename_20190415_020700.dat to filename_20190415_020700.dat
Fetching /home/edvin/Test_folder/filename_20190415_020800.dat to filename_20190415_020800.dat
Fetching /home/edvin/Test_folder/filename_20190415_020900.dat to filename_20190415_020900.dat
Fetching /home/edvin/Test_folder/filename_20190415_021000.dat to filename_20190415_021000.dat
sftp> bye
There is many thing can go wrong still in this solution,
like if remote directory not exist, not accessible, script will still go on with the rest of the command, same for the local directory and for the files as well. The connection also can run various problems you might want to handle. You like to schedule it so might a solution needed to avoid script spawn over and over again if one already run.
scp would be more preferred way to do this, as you use password less authentication.
If scp is not an option for some reason, with expect this can be handled quite well.
I'm currently debugging a shell script, which acts as a master-script in a data pipeline. In order to run the pipeline, you feed a bunch of arguments into the shell script. From there, the shell script sequentially calls 6 different scripts [4 in R, 2 in Python], writes out stuff to log files, and so on. Basically, my idea is to use this script to automate a data pipeline that takes a long time to run.
Right now, if any of the individual R or Python scripts break within the shell script, it just jumps to the next script that it's supposed to call. However, running script 03.py requires the data input to scripts 01.R and 02.R to be fully run and processed, otherwise 03 will produce erroneous output data which will then be written out and further processed in later scripts.
What I want to do is,
1. Break the overall shell script if there's an error in any of the R scripts
2. Output a message telling me where this error happened [line of individual R / python script]
Here's a sample of the master.sh shell script which calls the individual scripts.
#############
# STEP 2 : RUNNING SCRIPTS
#############
# A - 01.R
#################################################################
# log_file - this needs to be reassigned for every individual script
log_file=01.log
current_time=$(date)
echo "Current time: $current_time"
echo "Now running script 01. Log file output being written to $log_file_dir$log_file."
Rscript 01.R -f $input_file -s $sql_db > $log_file_dir$log_file
# current time/date
current_time=$(date)
echo "Current time: $current_time"
# B - 02.R
#################################################################
log_file=02.log
current_time=$(date)
echo "Current time: $current_time"
echo "Now running script 02. Log file output being written to $log_file_dir$log_file"
Rscript 02.R -f $input_file -s $sql_db > $log_file_dir$log_file
# PRINT OUT TIMINGS
current_time=$(date)
echo "Current time: $current_time"
This sequence is repeated throughout the master.sh script until script 06.R, after which it collates some data retrieved from output files and log files, and prints them to stout.
Here's some sample output that gets printed by my current master.sh, which shows how the script just keeps moving even though 01.R has produced an error.
file: test-data/minisample.txt
There are a total of 101 elements in file.
Using the main database.
Writing log-files to this directory: log_files/minisample/.
Writing output-csv with classifications to output/minisample.csv.
Current time: Wed Nov 14 18:19:53 UTC 2018
Now running script 01. Log file output being written to log_files/minisample/01.log.
Loading required package: stringi
Loading required package: dplyr
Attaching package: ‘dplyr’
The following objects are masked from ‘package:stats’:
filter, lag
The following objects are masked from ‘package:base’:
intersect, setdiff, setequal, union
Loading required package: RMySQL
Loading required package: DBI
Loading required package: methods
Loading required package: hms
Error: The following 2 arguments need to be provided:
-f <input file>.csv
-s <MySQL db name>
Execution halted
Current time: Wed Nov 14 18:19:54 UTC 2018
./master.sh: line 95: -1: substring expression < 0
./master.sh: line 100: -1: substring expression < 0
./master.sh: line 104: -1: substring expression < 0
Total time taken to run script 01.R:
Average time taken per user to run script 01.R:
Total time taken to run pipeline so far [01/06]:
Average time taken per user to run pipeline so far [01/06]:
Current time: Wed Nov 14 18:19:54 UTC 2018
Now running script 02. Log file output being written to log_files/minisample/02.log
Seeing as the R script 01.R produces an error, I want the script master.sh to stop. But how?
Any help would be greatly appreciated, thanks in advance!
As another user mentioned, simply running set -e will make your script terminate on first error. However, if you want more control, you can also check the exit status with ${?} or simply $? assuming your program gives an exit code of 0 on success, and non-zero otherwise.
#!/bin/bash
url=https://nosuchaddress1234.com/nosuchpage.html
error_file=errorFile.txt
wget ${url} 2> ${error_file}
exit_status=${?}
if [ ${exit_status} -ne 0 ]; then
echo -n "wget ${url} "
if [ ${exit_status} -eq 4 ]; then
echo "- Network failure."
elif [ ${exit_status} -eq 8 ]; then
echo "- Server issued an error response."
else
echo "- Other error"
fi
echo "See ${error_file} for more details"
exit ${exit_status};
fi
I like to put some boilerplate at the top of most scripts like this -
trap 'echo >&2 "ERROR in $0 at line $LINENO, Aborting"; exit $LINENO;' ERR
set -u
While coding at debugging, I usually add
set -x
And a lot of trace "comments" with colons -
: this will parse its args but only show under set -x
Then the trick is to make sure any errors you know are ok are handled.
Conditionals consume the errors, so those are safe.
if grep foo nonexistantfile
then : do the success stuff
else : if you *want* a failout here, just call false
false here will abort # args don't matter :)
fi
By the same token, if you just want to catch and ignore a known possible error -
ls $mightNotExist ||: # || says "do on fail"; : is an alias for "true"
Just always check your likely errors. Then the only thing that will crash your script is a fail.
Found the Solution !!!!
After a gob of Googling, I found this in a forum from a person asking " How to: Add or display today’s date from a shell script"
This is what I did
I added the following to the beginning of my ftp script
#!/bin/bash
TODAY=$(date)
HOST=$(hostname)
echo "--------------------------------------------"
echo "This script was run: $TODAY ON HOST:$HOST "
echo "--------------------------------------------"
# below is original code minus the #!/bin/sh
#
cd /folder where csv files are/
ftp -v -i -n 111.222.333.444 <<EOF
user mainuser dbuser
mput phas*.csv
bye
EOF
Now my log, on each cron event of the ftp'ing, show:
This script was run: Tue Nov 12 11:16:02 EST 2013 ON MyServer's HostName>
On the crontab, I changed the entry for logging to include 2 >> so the log is appended and not re-written:
16 11 * * * /srv/phonedialer_tmp/ftp-date.sh &>> /srv/phonedialer_tmp/ftp-date.log
I found a way to create a log file of daily ftp's by searching here:
./ftp_csv.sh 2>&1 > ftp_csv.log
I works great in that is records each time the cronjob runs. However, what I cannot find is a way to insert the date/time of each event. As you can see below, it records the transferring of the files.
is there a way I can somehow add the date/timestamp to the beginning or end of each recorded event within the log file?
[stevek#localhost phonedialer_tmp]$ cat ftp_csv.log
Connected to 1.2.3.4 (1.2.3.4).
220 Microsoft FTP Service
331 Password required for mainuser.
230 User mainuser logged in.
221
Connected to 1.2.3.4 (1.2.3.4).
220 Microsoft FTP Service
331 Password required for mainuser.
230 User mainuser logged in.
221
Connected to 1.2.3.4 (1.2.3.4).
220 Microsoft FTP Service
331 Password required for mainuser.
230 User mainuser logged in.
221 ETC
Thanks so much for any information