I am trying to print the time of all the files using the following shell script. But I see that not always bytes 42 to 46 is the time, as it changes due to more/less bytes in username and other details. Is there another way to fetch the time?
#!/bin/sh
for file in `ls `
do
#echo `ls -l $file`
echo `ls -l $file | cut -b 42-46`
done
Use awk.
Try ls -l | awk '{ print $6 $7 $8}'
This will print the 6th, 7th and 8th fields of ls -l split by whitespace
If the fields are different for you change the numbers to adjust which fields.
The output from ls varies depending on the age of the files. For files less than about 6 months old, it is the month, day, time (in hours and minutes); for files more than about 6 months old, it prints the month, day, year.
The stat command can be used to get more accurate times.
For example, to print the time of the last modification and file name of some text files, try:
stat -c '%y %n' *.txt
From the manual:
%x Time of last access
%X Time of last access as seconds since Epoch
%y Time of last modification
%Y Time of last modification as seconds since Epoch
%z Time of last change
%Z Time of last change as seconds since Epoch
man stat
Related
I have a directory of several files, and each file contains a timestamp at the end of its name, in the format of .%Y-%m-%dT%H-%M-%S, like following
filename.2021-02-12T10-29-59
filename.2021-02-11T05-04-30
filename.2021-02-10T00-12-30
filename.2021-02-10T20-30-30
...
I'm writing a script that whose timestamp is 3 days older than the current date.
For example, if current date is 2021-02-10 (from date '+%Y-%m-%d' command), it should delete all files older than 2021-02-07.
You can do with GNU awk and xargs:
awk -F\. '{ newdat=gensub("[T-]"," ","g",$2);if (mktime(newdat)<( strftime("%s")-259200)) { print $0 } }' <(for i in filename*;do echo $i;done) | xargs rm
Loop on the files in the directory with the name *.filename and redirect the output back into awk. Take the second "." delimited field and replace all "T" and "-" characters, with a space, reading the result into a variable newdat. This is then used for the mktime function to compare the date in epoch format to the current epoch format date (attained with strftime) minus 259200 (seconds in 3 days). If the difference is greater than 3 days, print the filename and pipe the output through to xargs rm to remove the file(s)
Use xargs echo as opposed to rm to first verify that the files are listed as expected.
I know how to tail a text file with a specific number of lines,
tail -n50 /this/is/my.log
However, how do I make that line count a variable?
Let's say I have a large log file which is appended to daily by some program, all lines in the log file start with a datetime in this format:
Day Mon YY HH:MM:SS
Every day I want to output the tail of the log file but only for the previous days records. Let's say this output runs just after midnight, I'm not worried about the tail spilling over into the next day.
I just want to be able to work out how many rows to tail, based on the first occurrence of yesterdays date...
Is that possible?
Answering the question of the title, for anyone who comes here that way, head and tail can both accept a code for how much of the file to exclude.
For tail, use -n +num for the line number num to start at
For head, use -n -num for the number of lines not to print
This is relevant to the actual question if you have remembered the number of lines from the previous time you did the command, and then used that number for tail -n +$prevlines to get the next portion of the partial log, regardless of how often the log is checked.
Answering the actual question, one way to print everything after a certain line that you can grep is to use the -A option with a ridiculous count. This may be more useful than the other answers here as you can get a number of days of results. So to get everything from yesterday and so-far today:
grep "^`date -d yesterday '+%d %b %y'`" -A1000000 log_file.txt
You can combine 2 greps to print between 2 date ranges.
Note that this relies on the date actually occurring in the log file. It has the weakness that if no events were logged on a particular day used as the range marker, then it will fail to find anything.
To resolve that you could inject dummy records for the start and end dates and sort the file before grepping. This is probably overkill, though, and the sort may be expensive, so I won't example it.
I don't think tail has any functionality like this.
You could work out the beginning and ending line numbers using awk, but if you just want to exact those lines from the log file, the simplest way is probably to use grep combined with date to do it. Matching yesterday's date at beginning of line should work:
grep "^`date -d yesterday '+%d %b %y'`" < log_file.txt
You may need to adjust the date format to match exactly what you've got in the log file.
You can do it without tail, just grep rows with previous date:
cat my.log | grep "$( date -d "yesterday 13:00" '+%d %m %Y')"
And if you need line count you can add
| wc -l
I worked this out through trial and error by getting the line numbers for the first line containing the date and the total lines, as follows:
lines=$(wc -l < myfile.log)
start=$(cat myfile.log | grep -no $datestring | head -n1 | cut -f1 -d:)
n=$((lines-start))
and then a tail, based on that:
tail -n$n myfile.log
Ive seen a lot of questions regarding removing files that are older than x number of hours. I have not seen any pertaining to removing lines in a file older than x number of hours.
Here is an example of the log I am dealing with. For the sake of the example, assume current time is 2016-12-06 06:08:48,594
2016-12-05 00:44:48,194 INFO this line should be deleted
2016-12-05 01:02:10,220 INFO this line should be deleted
2016-12-05 05:59:10,540 INFO this line should be deleted
2016-12-05 06:08:10,220 INFO this line should be deleted
2016-12-05 16:05:30,521 INFO do not delete this line
2016-12-05 22:23:08,623 INFO do not delete this line
2016-12-06 01:06:28,323 INFO do not delete this line
2016-12-06 05:49:55,619 INFO do not delete this line
2016-12-06 06:06:55,419 INFO do not delete this line
I realize that it might be easier to do this in python or Perl but this needs to be done in bash. That being said, please post any and all relevant answers.
So far Ive tried using sed, awk, etc to convert the timestamps to seconds.
#! /bin/bash
TODAY=$(date +%Y-%m-%d)
# one day ago
YESTERDAY=$(date -d #$(( $(date +"%s") - 86400)) +%Y-%m-%d)
REPORT_LOG=report_log-$TODAY.log
# current date in seconds
NOW=$(date +%s)
# oldest date in the log trimmed by timestamp
OLDEST_DATE=$(head -1 $REPORT_LOG | awk '{print $1" "$2}')
# oldest date converted to seconds
CONVERT_OLDEST_DATE=$(date -d "$OLDEST_DATE" +%s)
TIME_DIFF=$(($NOW-$CONVERT_OLDEST_DATE))
# if difference is less than 24 hours, then...
if [ $TIME_DIFF -ge 86400 ]; then
LATEST_LOG_TIME=$(tail -1 $REPORT_LOG | awk '{print $2}'| cut -c 1-8)
RESULTS=$(awk "/${YESTERDAY} ${LATEST_LOG_TIME}/{i++}i" $REPORT_LOG)
if [ -z $RESULTS]; then
awk "/${YESTERDAY} ${LATEST_LOG_TIME}/{i++}i" $REPORT_LOG > $REPORT_LOG.tmp && mv $REPORT_LOG.tmp $REPORT_LOG
else
echo "Out of ideas at this point"
fi
else
echo "All times newer than date"
fi
The problem with my above snippet is that it relies on a date to repeat itself for the awk to work, which is not always the case. There are hour long gaps in the log files so it is possible for the last line's date (ex. 2016-12-06 06:06:55) to be the only time that date appears. If the timestamp has not previously appeared, my script will delete all results before the matched timestamp.
Any and all help is appreciated.
awk to the rescue!
$ awk -v d="2016-12-05 06:08:48,594" '($1 " " $2) > d' file
will print the newer entries. Obviously, you want to create the date dynamically.
Ignoring the milliseconds part to simplify, you can use
$ awk -v d="$(date --date="yesterday" "+%Y-%m-%d %H:%m:%S,999")" ...
Note that lexical comparison works only for your hierarchial formatted date (why don't everybody use this?), for any other format you are better off converting to seconds from epoch and do numerical comparison on integers
Do the dates in times since the Unix epoch, using the format string +%s. For instance:
yesterday=$(date --date="yesterday" +%s)
Then interpret the dates you've extracted with awk or similar like:
dateInUnixEpoch=$(date --date="$whateverDate" +%s)
Then just compare the dates:
if [ "$yesterday" -ge "$dateInUnixEpoch" ];
then do whatever to delete the lines
fi
I'm running a background script that involves reading a log file every 5 minutes formatted like this
21:25:57 [INFO] event from 5 minutes ago
21:26:54 [INFO] potentially relevant event
21:28:26 [INFO] some event
21:30:06 [INFO] another event
except I only want to look at a the lines that were printed within the last 5 minutes. Essentially I need to find the line where date -s "$logdate" +%s <= date -d "5 min ago" +%s closest to the end of the file and ignore all the lines before it.
Unfortunately there are no dates in the timestamps, so it makes it tricky in that the timestamps will recur every day unless I restart the server and reset the log (I'd prefer not to). Using tac instead of cat might be more effective since I only care about the lines starting from the end of the file.
If you have GNU date:
$ date +%T
04:56:32
$ date -d "5 min ago" +%T
04:51:45
And you can do:
tac logfile | awk -v start=$(date -d "5 min ago" +%T) '$1 < start {exit} 1' | tac
I've written a bash shell script and part of it is checks if a file is over 30 days old using the find command, sadly when I uploaded it to my host it did not work as the find command is locked down.
Does anybody know how to check if a file is over 30 days old WITHOUT the find command?
I'm thinking I need to do an "ls -a $filename" then parse the date string out, convert it to a unix date and then compare it with todays date but I'm very rusty at unix.
Thanks
stat -c %Z filename gives the last change time in unixtime
stat -c %Y filename if you want the last modification time.
You can use if [ $file -ot $timestamp_file ] (-ot meaning "older than"). You would have to construct the appropriate file, for example with touch (which takes timestamp options). If this is a periodic task you can also create the timestamp file on each run to use next time.
Let ls output a unix timestamp, like this:
ls -l --time-style='+%s'
Then get the timestamp from the output, and simlpy calculate whether there are 30 × 24 × 60 × 60 seconds between the current time and the timestamp from ls.
You can get the current timestamp using
date '+%s';
That doesn't work in Linux: no -v option in date
t=`date +%Y%m%d%H%M`
let u=$t-1000000
touch -t "$u" $DATEFILE
if [ $SERVERFILE -ot $DATEFILE ]; then
rm $SERVERFILE
fi
OK this is how I did it in the end...
I touched a file with a date of today - 1 month.
touch -t "$(date -v -1m +%Y%m%d%H%M)" ${DATEFILE}
Did an older than test on the 2 files.
if [ ${SERVERFILE} -ot ${DATEFILE} ]
Thanks everybody.