Calculate Line Number in Log with Logstash Pipeline - ruby

Imagine that have some logs like this
Jun2012|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|
Jun2012|17:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|
Jun2013|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|
Jun2012|18:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|
Jun2012|19:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|
.....
I can see the line number in an editor but I want to calculate line number of each line and make field of it.In out put inside a json like
{...,"line_num" : 23, ...}
How can i make this ? Can someone give an example ?

Ruby has a special variable $. that has the line number.
File.foreach(logfile).each do |line|
puts "#{$.}: #{line}"
end

Try the below with readlines and with_index
result = []
File.readlines('test.log').each.with_index(1) do |line, index|
result << { content: line, line_num: index }
end
Output:
2.6.3 :042 > pp result
[{:content=>
"Jun2012|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|\n",
:line_num=>1},
{:content=>
"Jun2012|17:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|\n",
:line_num=>2},
{:content=>
"Jun2013|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|\n",
:line_num=>3},
{:content=>
"Jun2012|18:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|\n",
:line_num=>4},
{:content=>
"Jun2012|19:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|",
:line_num=>5}]

Logstash has no information about the line number of the event in a file, and it only tracks the byte offset when you use the file input.
If you are using filebeat to ship your logs to logstash you also have the information about the byte offset, but there is no tracking of the actual line number.
If you want to just order your events you can use this offset, if you need the actual line number you will need to implement a way to add this information in your event before sending it to logstash.

I have solved that issue, with creating your own input codec plugin you can calculate line numbers via iterating over events

Related

How to make animated gif in Gnuplot 5

Basically, I have solved the heat equation for (x,y,t) and I want to show the variation of the temperature function with time.The program was written in Fortran 90 and the solution data was stored in a file diffeqn3D_file.txt.
This is the program:
Program diffeqn3D
Implicit none
Integer:: b,c,d,l,i,j,k,x,y,t
Real:: a,r,s,h,t1,k1,u,v,tt,p
Real,Dimension(0:500,0:500,0:500):: f1 !f=f(x,t)
!t1=time step and h=position step along x and
!k=position step along y and a=conductivity
open(7, file='diffeqn3D_file.txt', status='unknown')
a=0.024
t1=0.1
h=0.1
k1=0.1
r=(h**2)/(k1**2)
s=(h**2)/(a*t1)
l=10
tt=80.5
b=100
c=100
d=100
!The temperature is TT at x=0 and 0 at x=l.
!The rod is heated along the line x=0.
!Initial conditions to be changed as per problem..
Do x=0,b
Do y=0,c
Do t=0,d
If(x==0) Then
f1(x,y,t)=tt
Else If((x.ne.0).and.t==0) Then
f1(x,y,t)=0
End If
End Do
End Do
End Do
print *,f1(9,7,5)
print *,r
print *,a,h,t1,h**2,a*t1,(h**2)/(a*t1)
print *,f1(0,1,1)
print *,f1(3,1,1)
!num_soln_of_eqnwrite(7,*)
Do t=1,d
Do y=1,c-1
Do x=1,b-1
p=f1(x-1,y,t-1)+f1(x+1,y,t-1)+r*f1(x,y-1,t-1)+r*f1(x,y+1,t-1)-(2+2*r-s)*f1(x,y,t-1)
f1(x,y,t)=p/s
!f1(x,t)=0.5*(f1(x-1,t-1)+f1(x+1,t-1))
!print *,f1(x,t),b
End Do
End Do
End Do
Do i=0,d
Do k=0,b
Do j=0,c
u=k*h
v=j*k1
write(7,*) u,v,f1(k,j,i)
End Do
End Do
write(7,*) " "
write(7,*) " "
End Do
close(7)
End Program diffeqn3D
And after compilation and run, I enter the following code in gnuplot but it does not run, rather it hangs up or creates a gif picture, not animation.
set terminal gif animate delay 1
set output 'diffeqn3D.gif'
stats 'diffeqn3D_file.txt' nooutput
do for [i=1:int(STATS_blocks)] {
splot 'diffeqn3D_file.txt'
}
Sometimes it also puts up a warning message, citing no z-values for autoscale range.
What is wrong with my code and how should I proceed?
First, try to add some print commands for "debug" information:
set terminal gif animate delay 1
set output 'diffeqn3D.gif'
stats 'diffeqn3D_file.txt' nooutput
print int(STATS_blocks)
do for [i=1:int(STATS_blocks)] {
print i
splot 'diffeqn3D_file.txt'
}
Second, what happens?
The splot command does not have an index specifier, try to use:
splot 'diffeqn3D_file.txt' index i
Without the index i gnuplots always plots the whole file which has two consequences:
The data file is quite large. Plotting takes quite a long time and it seems that gnuplot hangs.
Gnuplot plots always the same data, there are no changes which show up in an animation.
Now gnuplot runs much faster and we will fix the autoscale error. Again, there are two points:
The index specifies a data set within the data file. The stats command counts those sets which "are separated by pairs of blank records" (from gnuplot documentation). Your data file ends with a pair of blank records - this starts a new data set in gnuplot. But this data set is empty which finally leads to the error. There are only STATS_blocks-1 data sets.
The index is zero based. The loop should start with 0 and end at STATS_blocks-2.
So we arrive at this plot command:
do for [i=0:int(STATS_blocks)-2] {
print i
splot 'diffeqn3D_file.txt' index i
}

combine pipe delimited lines of text in multiple lines with vbscript?

I've got a text file of output that looks essentially like this:
SMITHERSON, SMITH|00012345|15-Jan-1999|000885340
619649339|29-Sep-2015 00:09:30|Black|JOHNERSON, JOHN
00067890|02-Dec-1996|000490365|620094551
29-Sep-2015 23:06:01|Green|DAVISON, DAVE|00086543|06-Jun-2001|000938585
226438332|28-Sep-2015 00:12:12|Yellow
Seven pieces of data, they are always in the correct order but unfortunately they run together and onto different lines. There are carriage return + line feeds at the end of each line and there aren't pipe delimiters. The individual pieces of data are never split over multiple lines - I'm having a hard time explaining so here's another example:
DATA 1|DATA 2|DATA 3
DATA 4
DATA 5|DATA 6|DATA 7
DATA 1|DATA 2|DATA 3|DATA 4
DATA 5|DATA 6|DATA 7
etc...
They will have spaces between them but each piece of data will always stay on it's own line.
And I'm trying to turn it into this:
SMITHERSON, SMITH|00012345|15-Jan-1999|000885340|619649339|29-Sep-2015 00:09:30|Black
JOHNERSON, JOHN|00067890|02-Dec-1996|000490365|620094551|29-Sep-2015 23:06:01|Green
DAVISON, DAVE|00086543|06-Jun-2001|000938585|226438332|28-Sep-2015 00:12:12|Yellow
DATA 1|DATA 2|DATA 3|DATA 4|DATA 5|DATA 6|DATA 7
DATA 1|DATA 2|DATA 3|DATA 4|DATA 5|DATA 6|DATA 7
etc.
Seven pieces of data each on their own line, but still seperated by the '|' for another piece of software to read correctly.
I am spending about one hour every day correcting the text files by hand, so I've been trying to find an example I can work from to do this for a while but have not had any luck wrapping my head around this.
This code is ok. I only tested your sample text, not big files.
It will replace line feeds with the delimiter, then convert the entire file into one big array:
Set fso = CreateObject("Scripting.FileSystemObject")
Set input = fso.OpenTextFile("input.txt", 1)
Set output = fso.OpenTextFile("output.txt", 2, True)
Dim data: data = input.ReadAll
input.Close()
data = Replace(data, vbCrlf, "|")
data = Split(data, "|")
For i=0 To UBound(data) Step 7
output.WriteLine data(i) & "|" & data(i+1) & "|" & data(i+2) & "|" & data(i+3) & "|" & data(i+4) & "|" & data(i+5) & "|" & data(i+6)
Next
output.Close()
Untested, but something like this might do it. (Essentially it copies input to output as a stream, but newlines in the input are converted to pipe characters and every seventh pipe in the output is converted to a newline)
Set fs = CreateObject("Scripting.FileSystemObject")
Set f = fs.OpenTextFile("D:\data\thefile.txt", 1)
Set o = fs.OpenTextFile("D:\data\combined.txt", 2, True)
pipecount = 0
Do While f.AtEndOfFile <> True
If f.AtEndOfLine = True Then
c = f.Read(2) ' Skip the CR+LF
c = "|" ' and pretend we got a pipe character
Else
c = f.Read(1)
End If
If c = "|" Then
pipecount = pipecount + 1
If pipecount = 7 Then
pipecount = 0
o.WriteLine()
Else
o.Write("|")
End If
Else
o.Write(c)
End If
End While
o.Close()

how to replace #VARIABLE in text file with value of Emails in xml file

How i can read this xml file
<Subs>
<Sub Report="BusinessSummarySubs" EMails="lalla#yahoo.com; haha#yahoo.com">
<Sub Report="PlayerSubs" EMails="hehe#hotmail.com">
</Subs>
and replace #VARIABLE in BusinesSummarySubs.txt with EMails value in
Here is the content(part of the content) from BusinessSumarySubs.txt
CType(extensionParams(0),ParameterValue).Name = "TO"
CType(extensionParams(0),ParameterValue).Label = ""
CType(extensionParams(0),ParameterValue).Value = "#VARIABLE"
If you look here, you'll see how to search for and to access attributes. Follow the link chain to 'the same for text' and do a mental diff, if you want to get a skeleton for a minimal XML processing script to use for your next task.
Single placeholder substitution in VBScript is easy: just use Replace:
>> attr = "lalla#yahoo.com; haha#yahoo.com"
>> content = "... .Value = ""#VARIABLE"" ..."
>> ph = "#VARIABLE"
>> WScript.Echo Replace(content, ph, attr)
>>
... .Value = "lalla#yahoo.com; haha#yahoo.com" ...
>>
Something like this i suposed
set xmlDoc=CreateObject("Microsoft.XMLDOM")
xmlDoc.async="false"
xmlDoc.load("note.xml")
for each Emails in xmlDoc.documentElement.childNodes
document.write(Emails .nodename)
document.write(": ")
document.write(Emails .text)
next

Heading image or string?

I just created a page which the user can choose the mode to show, such as icon (image) or just temperature (text). Those are my code:
<?lc
#tell the browser that we will send binary data (png)
put header("Content-type:image/png")
#get ip address
if $_GET["ip"] = "" then
put $_SERVER["REMOTE_ADDR"] into ipAddress
else
put $_GET["ip"] into ipAddress
end if
#get the mode
put $_GET["mode"] into mode
#get longtitude and latitude data
put url ("http://vo.afteroffice.com/r/v/public/pub/geoloc?ip=" & ipAddress) into geoInfo
split geoInfo by cr and ":"
#input lon and lat data into url
put url ("http://api.yr.no/weatherapi/locationforecast/1.8/?lat=" & geoInfo["latitude"] & ";lon=" & geoInfo["longtitude"]) into weatherDetail
#create tree from XML data
put revCreateXMLTree(weatherDetail, false, true, false) into tLocation
#get data about temperature and clouds
put revXMLAttribute(tLocation,"weatherdata/product/time/location/temperature", "value") into temperature
#create the output
if mode = "temperature" then
put "The temperature is " & temperature
else if mode = "icon" then
put URL("binfile:icon/hot01.png")
else if mode = "" then
put "Oke"
end if
?>
My image & text problems are:
If I don't put put header("Content-type:image/png") in my code and I call mode=icon, the result will be "‰PNG IHDRPPŽò­tEXtSoftwareAdobe ImageReadyqÉe
If I put put header("Content-type:image/png") in my code and I call mode=temperature, the result will be "Te image cannot displayed, cause it contains error"
Could you help me? Thank you.
And how about adding something like
if mode = "temperature" then
# do not put header
else if mode = "icon" then
put header("Content-type:image/png")
end if
at the position where you have your content type header now?
You just have to add the definition of mode at the beginning instead of the middle of your script

Recovering hex data from a large log-file using Ruby and RegEx

I'm trying to filter/append lines of hex data from a large log-file, using Ruby and RegEx.
The lines of the log-file that I need look like this:
Data: 10 55 61 (+ lots more hex data)
I want to add all of the hex data, for further processing later. The regex /^\sData:(.+)/ should do the trick.
My Ruby-program looks like this:
puts "Start"
fileIn = File.read("inputfile.txt")
fileOut = File.new("outputfile.txt", "w+")
fileOut.puts "Start of regex data\n"
fileIn.each_line do
dataLine = fileIn.match(/^\sData:(.+)/).captures
fileOut.write dataLine
end
fileOut.puts "\nEOF"
fileOut.close
puts "End"
It works - sort of - but the lines in the output file are all the same, just repeating the result of the first regex match.
What am I doing wrong?
You are iterating over the same entire file. You need to iterate over the line.
fileIn.each_line do |line|
dataLine = line.match(/^\sData:(.+)/).captures
fileOut.write dataLine
end

Resources