I have many log files, like that
log file 1
2016-09-11_19:40:15.537#15437 [INFO] A1
2016-09-11_19:40:15.537#15437 [WARN] A2
2016-09-11_19:40:15.542#15437 [INFO] A3
log file 2
2016-09-11_19:40:15.537#437 [INFO] B1
2016-09-11_19:40:15.540#437 [INFO] B2
I wish I can merge them by script or other method like that
sort by time
2016-09-11_19:40:15.537#15437 [INFO] A1
2016-09-11_19:40:15.537#15437 [WARN] A2
2016-09-11_19:40:15.537#437 [INFO] B1
2016-09-11_19:40:15.540#437 [INFO] B2
2016-09-11_19:40:15.542#15437 [INFO] A3
How do I to merge the files with efficient way ?
thanks !
Ref: Merging multiple log files by date including multilines
As mentioned in the above question, if you are certain that all the log lines start with timestamp, you can do:
cat logA.log logB.log | sort -n
This would not work when there are other lines such as stack trace which do not start with timestamp.
I think you can check out the above question and answers if your considering a similar scenario.
Try rust-based tool Super Speedy Syslog Searcher
(assuming you have rust installed)
cargo install super_speedy_syslog_searcher
then
s4 --color=never /var/log > logs-merged.log
Super Speedy Syslog Searcher will sort varying log messages by datetime.
Insert all of your files to an empty Notepad++ document.
Edit → Line Operations → Sort Lines Lexicographically Ascending.
Related
I'm using a .mrc script to add a data output to irc currently I'm outputting like this -----> [iNFO] - 12 / 125M/S. What I would like to do is output like this [iNFO] - 12F / 125M/S, so what I need is to add the "F" after the 12.
Here's the code snippet I use
`scon 19 msg #test [iNFO] - $remove($strip($4),$chr(44)) F / $remove($strip($6),$chr(44))`
As you can see that outputs like this [iNFO] - 12 F / 125M/S so I need to remove the space between the "12" and the "F".
Any help would be greatly appreciated.
I want a simple way to add 2 numbers taken from a text file. Details below:
Daily, I run clamscan against my /home/ folder, which generates a simple log along the lines of this:
Scanning 851M in /home/.
----------- SCAN SUMMARY -----------
Infected files: 0
Time: 0.000 sec (0 m 0 s)
Start Date: 2021:11:27 06:25:02
End Date: 2021:11:27 06:25:02
Weekly, I scan both my /home/ folder and an external drive, so I get twice as much in the log:
Scanning 851M in /home/.
----------- SCAN SUMMARY -----------
Infected files: 0
Time: 0.000 sec (0 m 0 s)
Start Date: 2021:11:28 06:25:02
End Date: 2021:11:28 06:25:02
Scanning 2.8T in /mnt/ext/.
----------- SCAN SUMMARY -----------
Infected files: 0
Time: 0.005 sec (0 m 0 s)
Start Date: 2021:11:28 06:26:30
End Date: 2021:11:28 06:26:30
I don't email the log to myself, I just have a bash script that sends an email that (for the daily scan) reads the number that comes after "Infected files:" and says either "No infected files found" or "Infected files found, check log." (And, to be honest, once I'm 100% comfortable that it all works the way I want it to, I'll skip the "No infected files found" email.) The problem is, I don't know how to make that work for the weekly scan of multiple folders, because the summary I get doesn't combine those numbers.
I'd like the script to find both lines that start "Infected files:", get the numbers that follow, and add them. I guess the ideal solution use a loop in case I ever need to scan more than two folders. I've taken a couple of stabs at it with grep and cut, but I'm just not experienced enough a coder to make it all work.
Thanks!
This bash script will print out the sum of infected files:
#!/bin/bash
n=$(sed -n 's/^Infected files://p' logfile)
echo $((${n//$'\n'/+}))
or a one-liner:
echo $(( $(sed -n 's/^Infected files: \(.*\)/+\1/p' logfile) ))
I have a Markdown file for logging. Each section has a heading named by date in chronological order. Part of the file is the following.
## 20200628
- Traceback
- [x] code debug
- [x] read Section 2
## 20200629
- Homepage rebuilding
## 20200630
- 13:00: Meeting Tom
I would like to sort the logs in a reverse way, after which it will show like the following.
## 20200630
- 13:00: Meeting Tom
## 20200629
- Homepage rebuilding
## 20200628
- Traceback
- [x] code debug
- [x] read Section 2
I was expecting to use VIM but I didn't figure out what is a proper way to achieve this. Any suggestions would be appreciated a lot. Thanks in advance.
We capture each block using a regex that matches its start, then
we delete it and paste at the beginning of the file.
:$put _ | g/^## 202006[2-3][890]/normal! dapggP
The :$put _ part was used to add a blank line at the end, so each block has a blank line after it.
: .................... vim command mode
$ .................... at end of the file
_ .................... a blank line
| ................... another vim command
g ................... a global command
/ ................... start of a search
^ ................... every start of the line
## .................. two literal ##
202006 ............... literal numbers
[2-3] ............... followed by 2 until 3
[890] ............... followed by 8 9 or zero
/ .................... end of the search
normal! .............. a normal command
dap .................. delete a paragraph
gg ................... jump to the beginning of the file
P .................... put deleted text before the cursor
I'm trying to find a solution to fastly merge 2 log files coming from 2 application servers.
The log files is like this:
00:00:00,028 DEBUG [com.acme.productionservice...
I would like something that based on the time stamp print one line of the log file or another for example:
if file one have 2 lines:
00:00:00,028 DEBUG [com.acme.productionservice...
00:00:00,128 DEBUG [com.acme.productionservice...
and file two have this 3 lines:
00:00:00,045 DEBUG [com.acme.productionservice...
00:00:00,100 DEBUG [com.acme.productionservice...
00:00:00,150 DEBUG [com.acme.productionservice...
the output should be
00:00:00,028 DEBUG [com.acme.productionservice... (file 1)
00:00:00,045 DEBUG [com.acme.productionservice... (file 2)
00:00:00,100 DEBUG [com.acme.productionservice... (file 2)
00:00:00,128 DEBUG [com.acme.productionservice... (file 1)
00:00:00,150 DEBUG [com.acme.productionservice... (file 2)
the only way I currently know is using
cat file1 file | sort
but this is very slow for gb of logs
I need something like reading the 2 files and compare the timestamps and decide what to print.
I ended up by using
sort -m
I also used a trick to understand from which log file the log comes from with
for a in *.log ; do
awk '$0=FILENAME" "$0' $a > $a.log
do
sort -m -k 2 *.log.log
Try Super Speedy Syslog Searcher
(assuming you have rust installed)
cargo install super_speedy_syslog_searcher
then
s4 log1 log2
However, Super Speedy Syslog Searcher expects to find a datetime stamp. If you can change the logging format from timestamp to datetimestamp then s4 can sort and merge the lines.
As part of a script I'm writing, I'm trying to grab the latest revision number within a given branch in SVN.
I had tried the following code (where ${sourcebranch} is the SVN URL for my branch):
svn info ${sourcebranch} | awk '/Revision/ { print $2; }'
However, this seems to give me the latest revision number for the entire repository, not just the branch.
I really just want the branch... any ideas?
Use log instead:
svn log --limit 1 ${sourcebranch}
This will return the last commit to the branch, similar to this output:
------------------------------------------------------------------------
r14159 | author_name | 2014-04-25 18:54:49 -0400 (Fri, 25 Apr 2014) | 5 lines
log message
------------------------------------------------------------------------
From there, just parse the r#### field.
"Revision" value is applied to whole repo, you need "Last changed rev".