I have a .log file which restarts at the beginning of each month, each message beginning with the following timestamp format: 01-07-2016 00:00:00:868|
There are thousands of messages per day and I'd like to create a short script which can figure out when the date increments and output each date to a new file with just that day's data. I'm not proficient in bash but I'd like to use sed or awk, as it's very useful for automating processes at my job and creating reports.
Below script will split the input log file into multiple files with the date added as a suffix to the input file name:
split_logfile_by_date
#!/bin/bash
exec < $1
while read line
do
date=$(echo $line|cut -d" " -f 1)
echo $line >> $1.$date
done
Example:
$ ls
log
$ split_logfile_by_date log
$ ls
log log.01-07-2016 log.02-07-2016 log.03-07-2016
awk '{log = FILENAME "." $1; print > log}' logfile
This will write all the 01-07-2016 records to the file logfile.01-07-2016
Related
I'm using a Raspberry Pi as a backup server. I use cron to run each backup job nightly and log the output to a file specific to each job. So each morning I have a bunch of log files (job1.log .. jobN.log). The log files are overwritten each time the job runs. I have another cron job (that runs after all the backup jobs) that sends me an email showing the last line of each log file. This all works as expected.
I'd like to be able to get a status in the subject of the email based on the last lines of the log files. When a backup job is successfully completed, the last line of the log file has some info followed by the word "completed" (which isn't included if the job fails). In my script that sends the email, I use "tail -1 >> summary.txt" for each log file, so summary.txt is a collection containing the last line of each logfile (and is included in the body of the email sent to me).
What I'd like to do is to check the last word of each line in summary.txt to see if all jobs completed successfully, and set the subject of the email appropriately (a simple "backup succeeded" or "backup failed" would be sufficient).
What would be the best way to do this? I know one possibility would be to use awk '{print $NF}' to get the last word of each line, but I'm not sure how to use that.
EDIT: As requested, here is the simplified code I'm currently using to send the "status" email to myself:
#!/bin/sh
tail -1 job1.log > summary.txt
tail -1 job2.log >> summary.txt
tail -1 job3.log >> summary.txt
mail -s "PI Backup Report" myemail#myhost < summary.txt
I know I could create an additional file with just the last lines by adding
awk '{print $NF}' summary.txt > results.txt
to the above script before the "mail" line, but then I still need to parse the results.txt file. How would I determine the status based on that file? Thanks again!
Measure total vs success lines in summary.txt.
xargs echo to trim results of excess whitespace
grep with regex specifying the line should end in "completed"
wc -l for line count
Set the title using an if statement
TOTAL=$(wc -l < summary.txt | xargs echo)
SUCCESS=$(grep -e 'completed$' summary.txt | wc -l)
title=$(if [ $TOTAL = $SUCCESS ]; then echo 'All Succeeded'; else echo "$SUCCESS/$TOTAL succeeded"; fi)
echo $title # or pass into mail command as subject
First of all, sorry as the title might not be a good one. Actually can't really think of a good title but I'll try to explain as much as I can here, so here goes,
I have a file called timeInfo which contains Date Time string in the following format,
2018-06-05 00:35:51 Controller shutdown process initiated
2018-06-05 05:32:22 Controller startup process initiated
...
...
Now what I'm trying to do is, I need to get this time and convert it into EPOCH and store it into a temporary file tempFile, what I've tried so far is ...
//$file points to **timeInfo** file
echo `grep 'Controller startup' $file | date -d "`awk '{ print $1,$2 }'`" >> $TEMP_FILE`
On using this I get the following error,
command substitution: line 73: unexpected EOF while looking for matching `"'
Then I tried a different approach and used the following code,
echo `grep "Controller startup" $file | awk '{print $1,$2}' >> $TEMP_FILE`
With this I get a file tempFile with the following info,
2018-06-06 00:35:31
2018-06-06 00:51:32
Which seems to be much better but I need to have it in EPOCH! Is there a way I can change the above script to save date time string in EPOCH format inside tempFile.
Hoping to hear your suggestion! Thank you
this may be what you want, needs gawk
$ awk '{t=$1 FS $2; gsub(/[-:]/," ",t); print mktime(t)}' file
1528173351
1528191142
or perhaps this
$ awk '/Controller (startup|shutdown)/{t=$1 FS $2;
gsub(/[-:]/," ",t);
print mktime(t)}' file
cat logfile|awk '{print($1,$2)}' |xargs -n 1 -I_ date +'%s' --date=_
Cat the file. Then awk the first two fields. and the using "xargs -n 1" passing one one data at a time to the date command and the using %s to get epoch.
I have files that has following format in directory:
SLS20160112.001 (20160112 stands for YYYYMMDD)
I wish to archive all previous month files, for example:
SLS20160201.001
SLS20150201.001
SLS20160107.001
SLS20160130.001
For the above files listed, i will archive SLS20160107.001 and SLS20160130.001 because from the filename it stamps January.
For the SLS20160201.001 it still remains as i only want to archive previous month file. I can only extract date from the filename, not the mdate or adate.
My current logic is to loop through all files, then get previous month files and then pipe out the filename and tar it. But not sure how to do that part.
for file in SLS*; do
f="${file%.*}"
GET PREVIOUS MONTH FILES AND THEN ECHO
done | tar cvzf SlSBackup_<PREVIOUS_MONTH>.TAR.GZ -T-
It looks like you want to solve the problem by using a shell script. I do a lot of work on Mac so I use csh/tcsh (default shell for OSX), and my answer will be in csh/tcsh script. You can either translate it to bash (your shell) or you can easily spawn a new shell by just typing$ tcsh.
You can write a small shell script which can filter filelist for your desired month.
#!/bin/csh -f
set mon_wanted = '01'
foreach file (`ls -1 SLS*.*`)
set mon = `echo $file | awk '{print substr($0, 8, 2)}'`
if ($mon != $mon_wanted) continue
echo -n $file ' '
end
Let's say the filename is foo.csh. Make it executable by
$ chmod 755 foo.csh
Then,
$ tar cvzf `foo.csh` > out.tar.gz
I'm looking for a programmatic way to add a file's date to the filename. I'm on a Mac, Yosemite (10.10).
Using Bash, I have put a fair amount of effort into this, but just too new to get there so far. Here's what I have so far:
#!/bin/bash
#!/bin/bash
(IFS='
'
for x in `ls -l | awk '{print$9" "$7"-"$6"-"$9}'`
do
currentfilename=$(expr "$x" : '\($substring\)')
filenamewithdate=$(expr "$x" : '.*\($substring\)')
echo "$currentfilename"
echo "$filenamewithdate"
done)
The idea here is to capture detailed ls output, use awk to capture the strings for the columns with the filename ($9), and also date fields ($7 and $6), then loop that output to capture the previous filename and new filename with the date to mv the file from old filename to new. The awk statement adds a space to separate current filename from new. The echo statement is there now to test if I am able to parse the awk ouput. I just don't know what to add for $substring to get the parts of the string that are needed.
I have much more to learn about Bash scripting, so I hope you'll bear with me as I learn. Any guidance?
Thanks.
Looking at the stat man page, you'd want:
for file in *; do
filedate=$(stat -t '%Y-%m-%dT%H:%M:%S' -f '%m' "$file")
newfile="$file-$filedate"
echo "current: $file -> new: $newfile"
done
Adjust your preferred datetime format to your taste.
You could save a line with
for file in *; do
newfile=$(stat -t '%Y-%m-%dT%H:%M:%S' -f '%N-%m' "$file")
I have a text file with table like
367612510.243586 717.860170
367612512.493918 722.249134
367668441.429983 692.407935
367668479.810461 692.407935
367668482.618858 727.953771
367668515.150386 727.953771
where the first column is UNIX time, and the second is some value. I want to display this list with human readable date and time in BASH. How to do it?
It is possible to do it with additional script file and xargs, but I'm looking for way to do it in one line.
try
awk '{print strftime("%c",$1)}' input.txt
another approach using a while-loop in bash
while read d _; do date -d #$d; done < input.txt
update:
bash solution to print the second field as well
while read f1 f2; do echo $(date -d #$f1) $f2; done < input.txt