Log File :-
LOCATION CPIC (TCP/IP) on local host with Unicode
ERROR Error Message
TIME Mon May 4 11:37:17 2020
RELEASE 721
COMPONENT CPIC (TCP/IP) with Unicode
VERSION 3
RC 473
LINE 9261
COUNTER 15
Changing trace level: handle 38331088 / destination (null) / level 0
Tried with below command to automatically get the last 10 minutes log.
sed -n "/ $(date +\%R -d "-10 min")/,$"p logfile.log | grep "ERROR"
no output displayed
Expected Output : Error message in Last 10 mins.
Any Solution ?
The challenge in this case is that the 10 minute test requires comparison of timestamp. Using pattern matching on the exact time 10 minutes ago might fail, if there was no error in the exact minutes.
As an alternative, consider the following 'AWK' based solution. It will skip lines until it see a line matching the following:
* First token is TIME
* The day of the month matching today
* Time in last 10 minutes
awk -v DD=$(date +%-d -d '-10 min') -v WHEN="$(date +\%R -d '-10 min')" '
$1 == "TIME" && $4 == +DD && $5 >= WHEN { p=1 }
p { print }
' logfile
The solution can be improved to be more generic. Current implementation will not work well during the first 10 minutes after midnight. This can be addressed (if needed) by OP.
I am implementing the top command. I need to calculate the Time+ field just like the top command.
As of right now, I am getting the utime and stime then I put the system to sleep and then getting it these values again. I add the 4 quantities and divide the total by 100 which gives me a number. Here is the code for reference:
oldutime=$(awk '{print $14}' /proc/$word/stat )
oldstime=$(awk '{print $15}' /proc/$word/stat )
newutime=$(awk '{print $14}' /proc/$word/stat )
newstime=$(awk '{print $15}' /proc/$word/stat )
total_time=`expr $oldutime + $oldstime + $newutime + $newstime`
timee=$((total_time / 100))
After this I need to format this number so that it looks like the output of TIME+ field (minutes: seconds.hunderehts) in the top program and I need help doing that. I was looking at the date command but couldn't figure it out.
EDIT:
Desired for format: 0:00.24 (minutes:seconds.hunderdths)
Output: 360
The following scriptlet can be used to format time in milliseconds:
Code assume total_time is calculated - either per OP question, or modified for comments (using $((...))) instead of expr.
# Calculate total time in tick
total_time=...
ticks_per_sec=$(getconf CLK_TCK)
# Seconds times 100
s100=$((total_time*100/ticks_per_sec))
printf '%d:%02d.%02d\n' $((s100/100/60)) $((s100/100%60)) $((s100%100))
I have a file that looks like this:
user1,135.4,MATLAB,server1,14:53:59,15:54:28
user2,3432,Solver_HF+,server1,14:52:01,14:54:28
user3,3432,Solver_HF+,server1,14:52:01,15:54:14
user4,3432,Solver_HF+,server1,14:52:01,14:54:36
I want to run a comparison between the last two columns and if the difference is greater than an hour(such as lines 1 and 3) it will trigger something like this:
echo "individual line from file" | mail -s "subject" email#site.com
I was trying to come up with a possible solution using awk, but I'm still fairly new to linux and couldn't quite figure out something that worked.
the following awk scripts maybe is your want
awk 'BEGIN{FS=","}
{a="2019 01 01 " gensub(":"," ","g",$5);
b="2019 01 01 " gensub(":"," ","g",$6);
c = int((mktime(b)-mktime(a))/60)}
{if (c >= 60){system("echo "individual line from file" | mail -s "subject" email#site.com")}}' your_filename
then put the scritps into crontab or other trigger
for example
*/5 * * * * awk_scripts.sh
if you just want check new line . use tail -n filename may be more useful than cat
Here you go: (using gnu awk due to mktime)
awk -F, '{
split($(NF-1),t1,":");
split($NF,t2,":");
d1=mktime("0 0 0 "t1[1]" "t1[2]" "t1[3]" 0");
d2=mktime("0 0 0 "t2[1]" "t2[2]" "t2[3]" 0");
if (d2-d1>3600) print $0}' file
user1,135.4,MATLAB,server1,14:53:59,15:54:28
user3,3432,Solver_HF+,server1,14:52:01,15:54:14
Using field separator as comma to get the second last and last field.
The split the two field inn to array t1 and t2 to get hour min sec
mktime converts this to seconds.
do the math and print only lines with more than 3600 seconds
This can then be piped to other commands.
See how time function are used int gnu awk: https://www.gnu.org/software/gawk/manual/html_node/Time-Functions.html
How do I compare current timestamp and a field of a file and print the matched and unmatched data. I have 2 columns in a file (see below)
oac.bat 09:09
klm.txt 9:00
I want to compare the timestamp(2nd column) with current time say suppose(10:00) and print the output as follows.
At 10:00
greater.txt
xyz.txt 10:32
mnp.csv 23:54
Lesser.txt
oac.bat 09:09
klm.txt 9:00
Could anyone help me on this please ?
I used awk $0 > "10:00", which gives me only 2nd column details but I want both the column details and I am taking timestamp from system directly from system with a variable like
d=`date +%H:%M`
With GNU awk you can just use it's builtin time functions:
awk 'BEGIN{now = strftime("%H:%M")} {
split($NF,t,/:/)
cur=sprintf("%02d:%02d",t[1],t[2])
print > ((cur > now ? "greater" : "lesser") ".txt")
}' file
With other awks just set now using -v and date up front, e.g.:
awk -v now="$(date +"%H:%M")" '{
split($NF,t,/:/)
cur = sprintf("%02d:%02d",t[1],t[2])
print > ((cur > now ? "greater" : "lesser") ".txt")
}' file
The above is untested since you didn't provide input/output we could test against.
Pure Bash
The script can be implemented in pure Bash with the help of date command:
# Current Unix timestamp
let cmp_seconds=$(date +%s)
# Read file line by line
while IFS= read -r line; do
let line_seconds=$(date -d "${line##* }" +%s) || continue
(( line_seconds <= cmp_seconds )) && \
outfile=lesser || outfile=greater
# Append the line to the file chosen above
printf "%s\n" "$line" >> "${outfile}.txt"
done < file
In this script, ${line##* } removes the longest match of '* ' (any character followed by a space) pattern from the front of $line thus fetching the last column (the time). The time column is supposed to be in one of the following formats: HH:MM, or H:MM. Actually, date's -d option argument
can be in almost any common format. It can contain month names, time zones, ‘am’ and ‘pm’, ‘yesterday’, etc.
We use the flexibility of this option to convert the time (HH:MM, or H:MM) to Unix timestamp.
The let builtin allows arithmetic to be performed on shell variables. If the last let expression fails, or evaluates to zero, let returns 1 (error code), otherwise 0 (success). Thus, if for some reason the time column is in invalid format, the iteration for such line will be skipped with the help of continue.
Perl
Here is a Perl version I have written just for fun. You may use it instead of the Bash version, if you like.
# For current date
#cmp_seconds=$(date +%s)
# For specific hours and minutes
cmp_seconds=$(date -d '10:05' +%s)
perl -e '
my #t = localtime('$cmp_seconds');
my $minutes = $t[2] * 60 + $t[1];
while (<>) {
/ (\d?\d):(\d\d)$/ or next;
my $fh = ($1 * 60 + $2) > $minutes ? STDOUT : STDERR;
printf $fh "%s", $_;
}' < file >greater.txt 2>lesser.txt
The script computes the number of minutes in the following way:
HH:MM = HH * 60 + MM minutes
If the number of minutes from the file are greater then the number of minutes for the current time, it prints the next line to the standard output, otherwise to standard error. Finally, the standard output is redirected to greater.txt, and the standard error is redirected to lesser.txt.
I have written this script for demonstration of another approach (algorithm), which can be implemented in different languages, including Bash.
I have a Very large (13 GiB) csv file (3856321 rows and 1698) where as expected some of the dates are differently formated. The file looks like ::
2013/01/08 2:11:30 AM,abdc,good time ...
2015/12/28 8:19:30 PM,abdc,good time ...
2/15/2016 10:46:30 AM,kdafh,almost as good ...
12/13/2014 10:46:00 PM,asjhdk,not that good ...
02-Jan-2014,bad time,good time ...
1/1/2015,nomiss time,boy ...
10/15/2016 17:08:30,bad,boy ...
I want to convert it to a same time format and required output is ::
1/8/2013 2:11:30,abdc,good time
12/28/2015 20:19:30,abdc,good time
2/15/2016 10:46:30,kdafh,almost as good
12/13/2014 22:46:00,asjhdk,not that good
1/2/2014 00:00:00,bad time,good time
1/1/2015 00:00:00,nomiss time,boy
10/15/2016 17:08:30,bad,boy
I managed to format the time using the following scripts
awk -F ',' 'BEGIN{FS=OFS=","}{split($1,a," ");
if(a[3]=="PM")
{ split(a[2],b,":");
b[1]=b[1]+12
a[2]=b[1]":"b[2]":"b[3]
};
if(a[2]=="")
{
a[2]="00:00:00"
}
tmp=a[1];
# tmp2=system("date -d `tmp` +%m/%d/%Y");
# print tmp2
$1=tmp" "a[2]
}1' time_input.csv
I borrowed the idea of formatting dates from question https://unix.stackexchange.com/questions/177888/how-to-convert-date-format-in-file
which is commented out in the second last line. However, this does not work in my case. I get an error
date: invalid date ‘+%m/%d/%Y’
Is there an easier and better way to do this? Thanks in advance
With Python, using the dateutils and csv modules:
import dateutil.parser as parser
import csv
with open('time_input.csv', 'rb') as inputfile, open('time_output.csv', 'w') as outputfile:
reader = csv.reader(inputfile, delimiter=',')
writer = csv.writer(outputfile)
for row in reader:
row[0] = parser.parse(row[0]).strftime('%m/%d/%Y %H:%M:%S')
writer.writerow(row)
The result is output to time_output.csv file.
Awk sure is one great way of doing it but since it's really early morning here I don't like to think about all those ifs so here is one in php, since it's got a really nice strtotime function:
$ cat program.php
<?php
$handle = fopen("file", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
// process the line read.
$arr = explode(",", $line, 2);
echo date("m/d/Y H:i:s", strtotime($arr[0])), ",", $arr[1];
}
fclose($handle);
} else {
// error opening the file.
}
Run it:
$ php -f program.php
01/08/2013 02:11:30,abdc,good time
12/28/2015 20:19:30,abdc,good time
02/15/2016 10:46:30,kdafh,almost as good
12/13/2014 22:46:00,asjhdk,not that good
01/02/2014 00:00:00,bad time,good time
01/01/2015 00:00:00,nomiss time,boy
10/15/2016 17:08:30,bad,boy
The read line by line loop comes from here: How to read a file line by line in php. I only added lines with explode and strtotime.
The explode splits the line to pieces by the first , and stores them to array $arr. strtotime function is applied to the first element $arr[0]. $arr[1] is later outputed as-is.
You can try below awk command -
Input
vipin#kali:~$ cat kk.txt
2013/01/08 2:11:30 AM,abdc,good time
2015/12/28 8:19:30 PM,abdc,good time
2/15/2016 10:46:30 AM,kdafh,almost as good
12/13/2014 10:46:00 PM,asjhdk,not that good
02-Jan-2014,bad time,good time
1/1/2015,nomiss time,boy
10/15/2016 17:08:30,bad,boy
filtering -
vipin#kali:~$ awk -F"," '{split($1,a," "); printf ("%s,%s,%s",$2,$3,",");system("date -d \""a[1]" "a[2]"\" +\"%m/%d/%Y %H:%M:%S\"")}' kk.txt
abdc,good time,,01/08/2013 02:11:30
abdc,good time,,12/28/2015 08:19:30
kdafh,almost as good,,02/15/2016 10:46:30
asjhdk,not that good,,12/13/2014 10:46:00
bad time,good time,,01/02/2014 00:00:00
nomiss time,boy,,01/01/2015 00:00:00
bad,boy,,10/15/2016 17:08:30
Move the filtered output to file kk.txt2
vipin#kali:~$ awk -F"," '{split($1,a," "); printf ("%s,%s,%s",$2,$3,",");system("date -d \""a[1]" "a[2]"\" +\"%m/%d/%Y %H:%M:%S\"")}' kk.txt > kk.txt2
Output
vipin#kali:~$ awk -F"," '{print $NF,$1,$2}' OFS="," kk.txt2
01/08/2013 02:11:30,abdc,good time
12/28/2015 08:19:30,abdc,good time
02/15/2016 10:46:30,kdafh,almost as good
12/13/2014 10:46:00,asjhdk,not that good
01/02/2014 00:00:00,bad time,good time
01/01/2015 00:00:00,nomiss time,boy
10/15/2016 17:08:30,bad,boy
Explanation -
Use Split function on column 1 and put it in a and then use system function of awk to format the date as per our need.
I can print the output in order but it was printing a leading zero so i am printing formatted date in last column that is why i am moving the data in another file.
and finally you can print the column in your order.