Faster Way To Examine Text in AutoIt? - text-files

Question

I'm attempting to use AutoIt to examine a text file and output select lines into a CSV. The problem I keep running into is that it takes forever. The current method examines a single line at a time. It can burn through 5-10 lines per second, but I'm looking for something much faster within the AutoIt framework.
Code:
#include <File.au3>
$xnConfirm = False
$xnConfirmMsg = 0
while $xnConfirm = False
$xnFile = FileOpenDialog("File to Examine...","%userprofile%","All (*.*)") ;InputBox("File???", "Which file do you want to review?","C:\")
If FileExists($xnFile) = True Then
$xnConfirm = True
Else
$xnConfirmMsg = msgbox(1,"File Not Found...",$xnFile & " does not exist." & #crlf & "Please select another file.")
EndIf
WEnd
$xnConfirm = False
$xnConfirmMsg = 0
while $xnConfirm = False
$xnTargetFile = FileOpenDialog("Location to Save to...",$xnFile & " - output.csv","All (*.*)");"%userprofile%\Documents\output.csv"
;FileSaveDialog("Location to Save to...","%userprofile%","All (*.*)",16,"output - " & $xnFile & " - output.csv") ;
Consolewrite("Outputting to " & $xnTargetFile & #crlf)
if fileexists($xnTargetFile) then
$xnConfirmMsg = msgbox(4,"Overwrite?","Are you sure you want to overwrite " & #crlf & $xnTargetFile)
if $xnConfirmMsg = 6 Then
$xnConfirm = True
filedelete($xnTargetFile)
EndIf
Else
$xnConfirm = True
EndIf
WEnd
progresson("Line count","Verifying the number of lines in " & $xnFile)
$xnFileLine = _FileCountLines($xnFile) ;InputBox("Number of lines","How many lines are in this document?",10000)
consolewrite("Loading "& $xnFile & " with " & $xnFileLine & " total lines." & #crlf)
progressoff()
local $hfl = FileOpen($xnFile,0)
FileWrite($xnTargetFile,"")
FileOpen($xnTargetFile, 1)
$i = 1
ProgressOn("Creating CSV","Extracting matching data.","",0,0,16)
$xnTargetLine = 1
FileWriteLine($xnTargetFile,"Timestamp,Message,Category,Priority,EventId,Severity,Title,Machine,App Domain,ProcessID,Process Name,Thread Name,Win32 ThreadId")
While $i < $xnFileLine
;$xnCurrentLine = FileReadLine($xnFile,$i) ;Old Settings
$xnCurrentLine = FileReadLine($hfl,$i)
;MsgBox(1,"",$xnCurrentLine)
Select
Case stringinstr($xnCurrentLine,"Timestamp:")
$xnTargetLine = stringmid($xnCurrentLine,12,stringlen($xnCurrentLine) - 12 + 1) & ","
Case stringinstr($xnCurrentLine,"Message:")
$xnTargetLine = $xnTargetLine & stringmid($xnCurrentLine,10,stringlen($xnCurrentLine) - 10 + 1) & ","
Case stringinstr($xnCurrentLine,"Category:")
$xnTargetLine = $xnTargetLine & stringmid($xnCurrentLine,11,stringlen($xnCurrentLine) - 11 + 1) & ","
Case stringinstr($xnCurrentLine,"Win32 ThreadId:")
$xnTargetLine = $xnTargetLine & stringmid($xnCurrentLine,16,stringlen($xnCurrentLine) - 16 + 1) & #crlf
FileWriteLine($xnTargetFile,$xnTargetLine)
case Else
consolewrite("Nothing on line " & $i & #crlf)
EndSelect
$i = $i + 1
ProgressSet(round($i/$xnFileLine * 100,1),$i & " of " & $xnFileLine & " lines examined." & #cr & "Thank you for your patience.")
WEnd
ProgressOff()
To address the question of what this is doing, I'm reading a log file similar to a trace log. I want the events to output to a CSV so I can examine trends. The format in the log file looks like this:
Timestamp: 9/26/2013 3:33:23 AM
Message: Log Event Received
Category: Transaction
Win32 ThreadId:2872
I know that's the code format, but I hope it's easier to read.

Answer 1

(I wanted to add a comment asking for a sampling of the data being read in, however I don't have enough points yet...)
Depending on the size of the input file I recommend reading the entire file into an array in one swoop using _FileReadToArray() and then looping through the array in memory (instead of keeping access to the file open during the entire process). In addition, I wouldn't write to the output file each time either - I'd write to a string and then save the string when completed.
Something like:
$outputFileData = ""
$inputFileData = _FileReadToArray($xnFile)
For $Counter = 1 to $inputFileData[0]
$tmpLine = $inputFileData[$Counter]
Select
Case stringinstr($tmpLine,"Timestamp:")
$outputFileData = stringmid($tmpLine,12,stringlen($tmpLine) - 12 + 1) & ","
Case stringinstr($tmpLine,"Message:")
$outputFileData &= stringmid($tmpLine,10,stringlen($tmpLine) - 10 + 1) & ","
Case stringinstr($xnCurrentLine,"Category:")
$outputFileData &= stringmid($tmpLine,11,stringlen($tmpLine) - 11 + 1) & ","
Case stringinstr($xnCurrentLine,"Win32 ThreadId:")
$outputFileData &= stringmid($tmpLine,16,stringlen($tmpLine) - 16 + 1) & #CRLF
case Else
ConsoleWrite("Nothing on line " & $i & #crlf)
EndSelect
Next
FileWriteLine($xnTargetFile, $outputFileData)
(Please note I didn't include any error checking nor did I check it for errors :)

Answer 2

Im not sure if it would be really faster but, you could use Regexp.
If you could tell me a little bit more what the rules are here:
Case stringinstr($xnCurrentLine,"Timestamp:")
$xnTargetLine = stringmid($xnCurrentLine,12,stringlen($xnCurrentLine) - 12 + 1) & ","
Case stringinstr($xnCurrentLine,"Message:")
$xnTargetLine = $xnTargetLine & stringmid($xnCurrentLine,10,stringlen($xnCurrentLine) - 10 + 1) & ","
Case stringinstr($xnCurrentLine,"Category:")
$xnTargetLine = $xnTargetLine & stringmid($xnCurrentLine,11,stringlen($xnCurrentLine) - 11 + 1) & ","
Case stringinstr($xnCurrentLine,"Win32 ThreadId:")
$xnTargetLine = $xnTargetLine & stringmid($xnCurrentLine,16,stringlen($xnCurrentLine) - 16 + 1) & #crlf
FileWriteLine($xnTargetFile,$xnTargetLine)
case Else
consolewrite("Nothing on line " & $i & #crlf)
and if you could give me 2 or 3 example lines i could try to make you a, Regexp function wich i think will be much faster.
Edit:
I Made an example Script.
If the Input File looks something like this:
Timestamp: 9/26/2013 3:33:23 AM
Message: Log Event Received
Category: Transaction
Win32 ThreadId:2872
Then This Script works just fine
#include <Array.au3>
Local $file = FileOpen("InputFile.txt", 0)
$sText = FileRead($file)
$aSnippets = StringRegExp($sText,"(?:Timestamp:|Message:|Category:|Win32 ThreadId:)(?: )?(.+)",3)
_ArrayDisplay($aSnippets)
The result is an array Containing the Following things:
[0] = 9/26/2013 3:33:23 AM
[1] = Log Event Received
[2] = Transaction
[3] = 2872
etc.
If you want to combine these 4 lines in one, try to use a for loop (if you want to, i can make you one)
For 100 Lines he Needs 0.490570878768441 Miliseconds to store every Value in one array.

Answer 3

There is an other possible idea.
You could copy the Input File, rename it, and then Delete every useles Data out of it.
That would be very easy with RegularExpressions and probably even faster.
If you show me a example of the Input File and how the Output file should look like i could try :)