This question already has answers here:
Filter log file entries based on date range
(5 answers)
Closed 4 years ago.
I have an application that appends to the same log file. As this file is rather large (around 8 gb), I'd like to extract portions based on the timestamp at the beginning of the line.
-bash-3.2$ cat application.log | egrep --color "Starting Application|Exception"
08:46:01.328 [main] INFO Starting Application...
09:14:53.670 [Thread-1] ERROR Resolver - Caught exception -> com.jgoodie.AuthzException: Authorization failed
Caused by: com.jgoodie.AuthzException: Authorization failed
09:56:15.739 [main] INFO Starting Application...
10:17:08.932 [Thread-1] ERROR Resolver - Caught exception -> com.jgoodie.AuthzException: Authorization failed
Caused by: com.jgoodie.AuthzException: Authorization failed
In the above example, I'd like to extract the logs for the first run of the application (between 08:46:01.328 and 09:56:15.739). Is there any simple way (preferably a one liner) to do this?
Thanks
sed -n '/08:46:01.328/,/09:56:15.739/p' application.log
perl -lne 'if(/^08:46:01.328/.../^09:56:15.739/){print}' your_file
Related
How can I separate the logs from one pod to another?
Below is what I worked on:
CrashLoopBackOff=`for i in $(kubectl get po -n namespace | grep CrashLoopBackOff | awk '{print $1}'); do echo $i; done`
for y in $CrashLoopBackOff; do
k8s_logs=`kubectl logs $y -n namespace | tail -10` arr2+="$k8s_logs\n"
done
But the output is all together for 2 crashed pods or more pods and I cannot differentiate the logs from each pod. Any idea how can I put an echo or something between each pod log?
2021-03-12 07:30:11.622 [ERROR] [gstp_server_app.cc:4007] Failed to start subscription from SEL
2021-03-12 07:30:11.622 [ERROR] [gstp_server_app.cc:4010] Exception: 0 In order to do catch up, static entity: must have history enabled.
2021-03-12 07:30:11.622 [FATAL] [gstp_server_app.cc:1441] JMS Catchup initialization error.
2021-03-12 07:30:11.622 [FATAL] [gstp_server_app.cc:695] Failed to connect to JMS for data
2021-03-12 07:30:13.188 [INFO ] [jms_server.cc:495] End JMSServer.run
Ending JMSServer::Run
GSTP JMS Server Application Shutting down.
255
2021-03-12 07:31:51.828 [ERROR] [rcltocvll.cc:325] Zero Curve does not exist for DEPOT asof 02/06/12.
2021-03-12 07:31:51.831 [ERROR] [gbdopspec.cc:297] Error Creating Curve List Out Of Risk Class
2021-03-12 07:31:51.831 [ERROR] [sectheo.cc:705] Error retrieving security curves.
2021-03-12 07:31:51.833 [ERROR] [sectheo.cc:999] Error computing theoretical price for Security
2021-03-12 07:31:51.833 [ERROR] [rcltocvll.cc:325] Zero Curve does not exist for DEPOT asof 02/06/12.
2021-03-12 07:31:51.833 [ERROR] [gbdopspec.cc:297] Error Creating Curve List Out Of Risk Class List
2021-03-12 07:31:51.833 [ERROR] [sectheo.cc:705] Error retrieving security curves.
2021-03-12 07:31:51.833 [ERROR] [sectheo.cc:999] Error computing theoretical price for Security
bash: line 1: 6 Killed gstp_server_jms -N -LOGGER INFO -CFG
Please don't forget to give part of kubectl get po -n namespace's output for people to know what kind of data you are working on.
Also, the backquote (`) is used in the old-style command substitution, e.g. foo=`command` . The foo=$(command) syntax is recommended instead. Backslash handling inside $() is less surprising, and $() is easier to nest. See http://mywiki.wooledge.org/BashFAQ/082
See also https://mywiki.wooledge.org/BashFAQ/001 to learn how to read lines of input
The rest of your code is not even working, so here is working code based on my assumptions on what you tried to achieve:
arr2=()
while read -r crashing_pod _; do
while IFS= read -r line; do
arr2+=("$crashing_pod: $line")
# arr2+=("$line")
done < <(kubectl logs "$crashing_pod" -n namespace | tail -10)
# arr2+=('' "end of $crashing_pod 's logs" '')
done < <(kubectl get po -n namespace | grep CrashLoopBackOff)
printf %s\\n "${arr2[#]}"
You'll see I have prepended each line with the pod name, but you can use the two commented outlines instead if you'd rather have a separator like requested.
I want to pipe the output from bzip2 and use it as an input to fill a TDB database using tbdloader2 from apache-jena-3.9.0.
I already found
Generating TDB Dataset from archive containing N-TRIPLES files
but the proposed solution there did not work for me.
bzip2 -dc test.ttl.bz2 | tdbloader2 --loc=/pathto/TDBdatabase_test -- -
produces
20:08:01 INFO -- TDB Bulk Loader Start
20:08:01 INFO Data Load Phase
20:08:01 INFO Got 1 data files to load
20:08:01 INFO Data file 1: /home/user/-
File does not exist: /home/user/-
20:08:01 ERROR Failed during data phase
Similar results I got with with (inspired by https://unix.stackexchange.com/questions/16990/using-data-read-from-a-pipe-instead-than-from-a-file-in-command-options)
bzip2 -dc test.ttl.bz2 | tdbloader2 --loc=/pathto/TDBdatabase_test /dev/stdin
20:34:45 INFO -- TDB Bulk Loader Start
20:34:45 INFO Data Load Phase
20:34:45 INFO Got 1 data files to load
20:34:45 INFO Data file 1: /proc/16256/fd/pipe:[92062]
File does not exist: /proc/16256/fd/pipe:[92062]
20:34:45 ERROR Failed during data phase
and
bzip2 -dc test.ttl.bz2 | tdbloader2 --loc=/pathto/TDBdatabase_test /dev/fd/0
20:34:52 INFO -- TDB Bulk Loader Start
20:34:52 INFO Data Load Phase
20:34:52 INFO Got 1 data files to load
20:34:52 INFO Data file 1: /proc/16312/fd/pipe:[97432]
File does not exist: /proc/16312/fd/pipe:[97432]
20:34:52 ERROR Failed during data phase
unpacking the bz2 file manually and then adding it works fine:
bzip2 -d test.ttl.bz2
tdbloader2 --loc=/pathto/TDBdatabase_test test.ttl
Would be great if someone could point me in the right direction.
tdbloader2 accepts bz2 compressed files on the command line:
tdbloader2 --loc=/pathto/TDBdatabase_test test.ttl.bz2
It doesn't accept input from a pipe - and if it did, then it would not know the syntax is Turtle which it gets from the file extension.
This question already has answers here:
How to get the part of a file after the first line that matches a regular expression
(12 answers)
Closed 4 years ago.
5G_Fixed_Wireless_Dashboard_TestScedule||||||||||||||||^M
Report Run Date||08/07/2018|||||||||||||||||||||^M
Requesting User Company||NEW|||||||||||||||||||||^M
Report Criteria|||||||||||||||||||||||^M
" Service Job Updated from Date:
Service Job Updated to Date:
Service Job Created from Date: 08/06/2018
Service Job Created to Date:
Service Job Status:
Resolution Code:"|||||||||||||||||||||||^M
Created Date|Job Status|Schedule Date|Job
Number|Service Job Type|Verizon Customer Order
Number|Verizon Location Code|Service|Installation
Duration|Part Number
I want to print starting from Created Date. The result
file should be something like below.
Created Date|Job Status|Schedule Date|Job
Number|Service Job Type|Verizon Customer Order
Number|Verizon Location Code|Service|Installation
Duration|Part Number
I have tried the following lines after you people linked me to some other questions. But my requirement is to print the result to the same file.
FILELIST=find $MFROUTDIR -maxdepth 1 -name "XXXXXX_5G_Order_*.txt"
for nextFile in $FILELIST;do
cat $nextFile | sed -n -e '/Created Date/,$p'
done
By writing above lines of code, output is printed on console. Could you please suggest some way to print it in same file.
This can be easily done with a simple awk command:
awk '/^Created Date/{p=1} p' file
Created Date|Job Status|Schedule Date|Job
Number|Service Job Type|Verizon Customer Order
Number|Verizon Location Code|Service|Installation
Duration|Part Number
We set a flag p to 1 when we encounter a line that starts with Created Date. Later we use awk default action to print each line when p==1.
References:
Effective AWK Programming
Awk Tutorial
This question already has answers here:
Can awk patterns match multiple lines?
(6 answers)
How to parse multi line records (with awk?)
(2 answers)
Closed 5 years ago.
In below test.log file I want to search only today's date with Message "Upload_Msg_testListeningFn"; if both the conditions match then redirect the output.
cat /tmp/test.log
===========================================================================
system Version [45.2.18] (FATAL ERROR INFORMATION)
File: [test003.cxx]
Line: [240]
Date: [19-01-2018 21:09:40]
User ID: [SYSTEM]
Message : [batman error: Upload_Msg_testListeningFn]
Error Code : [TEST0000057]
Outside FGH
Session Id: 6768678898:00:F
=======================================
Output should be:
Date: [19-01-2018 21:09:40]
Message : [batman error: Upload_Msg_testListeningFn]
I used awk but somewhere I am missing something
i used the below command for finding lines after particular date.
awk '/^Date/{c=3} c&&c--' /tmp/test.log
Background: I'd like to assert that no exceptions are written to a log for 30 seconds. Basically it's a smoke test to see if my application has come up and we haven't introduced any serious bugs.
Requirements: I'd like to do this using a bash script, preferably using common shell utilities. An exception is basically a single line that starts with !. There are a lot of other log lines written that are not exceptions.
Questions: How can I do this?
Here's one possible solution:
timeout 30 tail -F my.log -n 0|grep --line-buffered '^!'|head -n 1
. I can then check whether the exit code is 124 or 143 (timed out, don't know why it varies) or 0 (line found). This is my best bet so far. However, the solution doesn't seem to exit very quickly upon exception. I'd love to hear other solutions!
Assuming log file will only be updated by program on an event of exception.
You can use the following command:
stat log_file_name
And you'll get output similar to below. You can run stat after 30 sec or so, compare results of current and previous stat if you don't see any change in the timestamp then the file has not been modified or otherwise.
Access: 2015-03-27 15:22:17.000000000 +0530
Modify: 2015-03-27 15:22:16.000000000 +0530
Change: 2015-03-27 15:22:16.000000000 +0530