Import data from binary file is very slow - performance

I have a folder full of .txt files that contain test data. I wrote a macro to sift through the .txt files to extract the information I want based on some search criteria and then write those search results into a binary file.
So now that I have a binary file that contains a reduced data set, I wrote another macro to search binary file for what I actually want.
For some reason, my macro is very slow to read in data from the binary file.
As a comparison, I wrote a macro that looked through all txt files for a specific search, wrote that into binary file, and then just read that back into Excel. That only took 60 seconds.
Here's a snippet. I'm wondering if the if-else statement to check if it matches my search criteria [LC(a) and EID(e)] is slowing things down or size of binary file (only 200 MB).
Type MyBinaryRecordInfo
MyBinaryRecordInfo1(1 To 12) As String ' variable length
End Type
i = 1
Open currentpath & "\" & bin_fname & ".DAT" For Binary As #f
' read records from the binary file
For a = 1 To totalLC
For e = 1 To totalElm
Do While Loc(f) < LOF(f)
Call ReadBinRecord(MyRecord, f, ElmType)
Sheets(ElmType).Select
If MyRecord.MyBinaryRecordInfo1(1) = LC(a) Then
If MyRecord.MyBinaryRecordInfo1(2) = EID(e) Then
For j = 1 To totalbinrec
With MyRecord
Cells(i + 3, j) = .MyBinaryRecordInfo1(j)
End With
Next j
i = i + 1
Exit Do
End If
End If
Loop
Next e
Next a
Close #f ' close the file
Sub ReadBinRecord(MyRecord As MyBinaryRecordInfo, f As Integer, ElmType As String)
' reads the next record from an open binary file
Dim intSize As Integer
For j = 1 To totalbinrec
With MyRecord
Get f, , intSize ' read the size of the ID field
.MyBinaryRecordInfo1(j) = String(intSize, " ") ' set the variable length
Get f, , .MyBinaryRecordInfo1(j) ' read the variable string field
End With
Next j

Possibly the slow part is writing out to Excel. In particular, you change the sheet every time you read the row
Assuming you know the number of rows in the binary file in advance, you can read into memory and then write out once at the end. The code would look something like the following:
Option Explicit
Type MyBinaryRecordInfo
MyBinaryRecordInfo1(1 To 12) As String ' variable length
End Type
Sub x()
num_rows = 5000
Open currentpath & "\" & bin_fname & ".DAT" For Binary As #f
Dim V(1 To totalElm) As Variant
Dim V2(1 To num_rows, 1 To 12) As Variant
For e = 1 To totalElm
V(e) = V2
Next e
' read records from the binary file
For a = 1 To totalLC
For e = 1 To totalElm
Do While Loc(f) < LOF(f)
Call ReadBinRecord(MyRecord, f, ElmType)
If MyRecord.MyBinaryRecordInfo1(1) = LC(a) Then
If MyRecord.MyBinaryRecordInfo1(2) = EID(e) Then
For j = 1 To totalbinrec
With MyRecord
V(e)(i, j) = .MyBinaryRecordInfo1(j)
End With
Next j
i = i + 1
Exit Do
End If
End If
Loop
Next e
Next a
Close #f ' close the file
' write out
For e = 1 To totalElm
Sheets(ElmType).Select
Cells(3, 1).Resize(num_rows, 12).Value = V(e)
Next e
End Sub
Sub ReadBinRecord(MyRecord As MyBinaryRecordInfo, f As Integer, ElmType As String)
' reads the next record from an open binary file
Dim intSize As Integer
For j = 1 To totalbinrec
With MyRecord
Get f, , intSize ' read the size of the ID field
.MyBinaryRecordInfo1(j) = String(intSize, " ") ' set the variable length
Get f, , .MyBinaryRecordInfo1(j) ' read the variable string field
End With
Next j
End Sub
If you don't know the number of rows then you just Redim Preserve the inner variants once every 500 rows or so.
As this writes everything out at the end, you might find it helpful to use Application.Statusbar = "my string" to write a progress message

Related

error mismatch VBA

enter image description hereI'm getting hard time figuring what is going on in my code. It keeps saying "run-time error:'13' ". How can I handle this error?
Dim ws As Worksheet
Dim rearr(), wrarr()
Dim fName As String
Dim rowno As Long, colno As Long, rec As Long
Dim cnt As Long, cnt2 As Long
Dim delim As String
'specify output sheet
Set ws = Worksheets("1.Locations")
'specify text file to read (.csv in this example)
fName = "C:\Users\HP\Desktop\names.csv"
'set text file delimiter
delim = "," 'for Tab delimiter use delim = Chr(9)
ifnum = FreeFile
'set start row/col for text data to be placed ("A1" in this example)
rowno = 3 'row 1
colno = 2 'col A
With ws
Open fName For Input Access Read As #ifnum
rec = 0
Do While Not EOF(ifnum)
Line Input #ifnum, tmpvar
rec = rec + 1
'Put whole record into array
ReDim Preserve rearr(1 To rec)
rearr(rec) = tmpvar
'Split fields into a second array
wrarr = Split(rearr(rec), delim)
cnt2 = UBound(wrarr)
'Write fields out to specified ws range, one row per record
For cnt = 0 To cnt2
ws.Cells(rowno, colno + cnt) = wrarr(cnt)
Next cnt
rowno = rowno + 1
Loop
Close #ifnum
End With
I have reproduced your error.
In your declarations, change
Dim wrarr()
to
Dim wrarr

VBscript - select file based on priority

I have a folder that I will be looping through to process files differently based on their filenames. Doing good on my script (first one!), until I realized there will be filenames that have also have numbers representing priority. For example in the folder there may be:
'NV_CX67_mainx.dxf'
'NV_CX67_mainx1.dxf'
'NV_CX67_mainx2.dxf '
'NV_CX67_mainxroad.dxf'
'NV_CX67_motx.dxf'
'NV_CX67_resxroad.dxf'
The mainx, mainx1 and mainx2 are the same file type but mainx2 has priority and should be the only one processed. Currently, my statement is:
If Instr(1,FileRef, "mainx",1) then
How might I add a 2nd filter to process only the file with the highest number before moving onto the next file?
You are going to have run through the following process
Sort your input files
Loop through each file one by one
Compare the current file to the previous one you looked at minus the numbers to see if it greater.
Only process an item you have scanned all the similar items to ensure this one has the largest number
I wrote up an example below. Notice only NV_CX67_mainx4.dxf, and NV_CX67_mainxroad.dxf get processed:
Option Explicit
Dim i, sBaseFileName, sPrevFileName, prevBaseFile
sPrevFileName = "~"
prevBaseFile = "~"
Dim arr(5)
'Initialize test array. This will need to be sorted for this code to work properly
arr(0) = "NV_CX67_mainx.dxf"
arr(1) = "NV_CX67_mainx4.dxf"
arr(2) = "NV_CX67_mainx2.dxf"
arr(3) = "NV_CX67_mainxroad.dxf"
arr(4) = "NV_CX67_motx.dxf"
arr(5) = "NV_CX67_resxroad.dxf"
'Loop through the array
For i = LBound(arr) to UBound(arr)
If Instr(1, arr(i), "mainx",1) Then 'Check prev qualifier
sBaseFileName = getsBaseFileName(arr(i))
'First Case
If prevBaseFile = "~" Then
prevBaseFile = sBaseFileName
sPrevFileName = arr(i)
'Tie - Figure out which one to keep based on number at end of file name
ElseIf prevBaseFile = sBaseFileName Then
sPrevFileName = GetMaxFile(sPrevFileName, arr(i))
prevBaseFile = getsBaseFileName(sPrevFileName)
'New Case - Process prev case
Else
'Process File
MsgBox ("Processing " + sPrevFileName)
'Capture new current file for future processing
sPrevFileName = arr(i)
prevBaseFile = getsBaseFileName(sPrevFileName)
End If
End If
Next
'If last file was valid process it
If sPrevFileName <> "~" Then
MsgBox ("Processing " + sPrevFileName)
End If
'Return the larger of the two files based on numbers at end.
'Note "file9.txt" > "file10.txt" in this code
Function GetMaxFile(sFile1, sFile2)
GetMaxFile = sFile1
If sFile2 > sFile1 Then
GetMaxFile = sFile2
End If
End Function
'Return the file without extension and trailing numbers
'getsBaseFileName("hello123.txt") returns "hello"
Function getsBaseFileName(sFile)
Dim sFileRev
Dim iPos
getsBaseFileName = sFile
sFileRev = StrReverse(sFile)
'Get rid of the extension
iPos = Instr(1, sFileRev, ".",1)
If iPos < 1 Then
Exit Function
End If
sFileRev = Right(sFileRev, Len(sFileRev)-iPos)
'Get rid of trailing numbers
Do
If InStr(1, "1234567890", Left(sFileRev, 1), 1) Then
sFileRev = Right(sFileRev, Len(sFileRev)-1)
Else
Exit Do
End If
Loop While(Len(sFileRev) > 0)
getsBaseFileName = StrReverse(sFileRev)
End Function

vba loop performance too much time

I have 250.000 rows and I wanted to erase all rows that have a 0 in col AR. This takes too much time using a filter and deleting only visible cells, so I wrote a code. But still takes 1 minute for 1000 lines. So I will have to take 250 minutes!!! Besides after the first 6 minutes (6k lines) the number showed in AS3 (see code below) freezes, so I don't know if it's still running.
Is there a way to do this more efficiently (using less time)?
My code is:
Sub delrow()
Application.Calculation=xlCalculationManual
With Sheets("bners")
LR3 = Range("A" & Rows.Count).End(xlUp).Row
For i3 = 3 To LR3
range("AS2")=i3
a = Sheets("bners").Range("AR" & i3).Value
If a = 0 Then
Rows(i3).Delete
Else
End If
Next i3
End With
Application.calculate
End Sub
thanks!
Yes, definitely Step -1. But does that alone make it fast?
This batches the deletes 10 at a time (if needed now).
Option Explicit
Dim ws as Range
Sub delrow1()
Dim LR3&, i3&, a&
Set ws = Sheets("bners")
LR3 = ws.Range("A" & Rows.Count).End(xlUp).Row
For i3 = LR3 To 3 Step -1
a = ws.Cells(i3, "AR").Value
If a = 0 Then
Call delrow2(i3)
End If
Next i3
Call delrow2(0) ' flush
End Sub
Sub delrow2(delRow&) ' deletes 10 rows at a time
Static a1&(10), na1&
Dim i1&, zRange As Range
If delRow = 0 Then ' finish;end;flush
For i1 = 1 To na1
ws.Rows(a1(i1)).Delete
Next i1
na1 = 0
Else ' store row in array a1
na1 = na1 + 1
a1(na1) = delRow
If na1 = 10 Then ' del 10 rows
Set zRange = Union( _
Rows(a1(1)), Rows(a1(2)), Rows(a1(3)), Rows(a1(4)), Rows(a1(5)), _
Rows(a1(6)), Rows(a1(7)), Rows(a1(8)), Rows(a1(9)), Rows(a1(10)))
ws.Range(zRange).Rows.Delete
na1 = 0
End If
End If
I liked this method I found a couple weeks ago but didn't remember until last night http://goo.gl/NYtY9R that could easily be adapted for yours
Sub RowKiller()
Dim F As Range, rKill As Range
Set F = Range("A2:A250000")
Set rKill = Nothing
For Each r In F
v = r.Text
If InStr(1, v, "0") = 1 Then
If rKill Is Nothing Then
Set rKill = r
Else
Set rKill = Union(r, rKill)
End If
End If
Next r
If Not rKill Is Nothing Then
rKill.EntireRow.Delete
End If
End Sub
To me very efficient in that it builds up into the Union and then deletes all at once instead of deleting one at a time.
in the example , you with sheets() is totally useless, as you forgot every dot "." before the words cells or range or rows.
I'll try an other approach, by using two VBA arrays (not tested, and might memory overflow).
first array is original data before the macro.
second array is data after the macro
I won't delete rows, i just write my second array from the good lines of the 1rst array,
and then paste it over the sheet
Sub RowKill()
'Declaring Variables :
Dim MaxRows as long 'number of lines in the First Array
Dim NewRows as Long 'number of lines in the Second Array
Dim q as long 'simple loop counter
Dim i as long 'simple loop counter , for the purpose of copying line
Dim Rg As Range 'Range of the original Data (number of lines = MaxRows-2, because the Original example code starts at 3, not 1)
Dim Sh as Worksheet
Dim Array1() as variant 'First VBA Array
Dim Array2() as variant 'Second VBA Array
with Application
.enableevents=false
.screnupdating=false
.Calculation=xlCalculationManual
end with
set Sh=thisworkbook.Sheets("bners")
with Sh
MaxRows = .Range( .Rows.Count , 44).End(xlUp).Row ' note the .rows, and i read on cloumn 44 and not 1
Set Rg = .Range( .cells(3,44) , .cells ( MaxRows,44) ) '44 is the column of .range("AR")
'The Range Rg is important , later we delete the whole thing ^^
Redim Array1 ( 1 to MaxRows, 1 to 44) 'Only if "AR" is your last column
Array1 = Rg.value2 'if you work with dates or time format in your cells, please replace by : Array1 = Rg.value
for q= 3 to MaxRows
if Array1 (q , 44) <> 0 Then 'wasn't sure, because empty cells will trigger too, in wich case: <>"" would be better, or: If not IsEmpty( Array1 (q,44)) .....
call CopyRowToSecondArray ( q , NewRows , Array2)
End If
next q
End With 'Sh
'Rg.delete 'old version
With Sh
.range ( .cells(1,1) , .cells (44 , NewRows).Value2 = Array2 ' again use .value, if you have date or time formating inside the data cells
if NewRows<MaxRows then .range ( .cells(1,NewRows+1) , .cells (44 , MaxRows).Value2 = ""
End with
with Application
.enableevents= True
.screnupdating= True
.Calculation=xlCalculationAutomatic
end with
Set Rg = Nothing
Ser Sh = Nothing
Erase Array1, Array 2
End Sub
Sub CopyRowToSecondArray ( byval q as long , byref NewRows as long , byref Array2 as variant)
Dim i as long
NewRows=NewRows+1
Redim Preserve Array2 (1 to NewRows, 1 to 44)
for i = 1 to 44 'this entire for i loop, might be faster with unknown vba array function (i'm new), please share with me
Array2 ( NewRows , i) = Array1 ( q , i )
next i
end sub
Maybe there is a better way to simply copy a whole line from one array to an other, i don't know...
The code is untested, and , I assumed 44 is the last column (change only in loops and Rg if needed), so copy your work before testing my code.
Hope this helps, and is faster.

Excel Sort By Numbers Macro

I use a macro to delete rows which doesnt containing numbers for my report.
This macro find critical path numbers and split them. In a1 column it delete the numbers which doesnt in the list.
This macro works fine. Beside that i want to sort a1 column by critical path number orders.
In this link I added what i want and my report file. There is a critical path text at the bottom in report file. When i click Düzenle macro delete rows but not sort by critical path number orders.
Thanks for your helps!
I do not like performing complex changes and deleting rows at the same time. If anything goes wrong, you have to restore the worksheet. I have introduced a new worksheet "Critical Path" and have copied to it everything required from worksheet "Revit KBK Sonuç" in the desired sequence.
I have described what I am doing and why within the macro. I hope it is all clear but ask if necessary.
Option Explicit
Sub ertert()
' I avoid literals within the code if I think those literals may change
' over time and/or if I think a name would make the code clearer.
Const ColLast As Long = 10
Const ColShtHdrLast As Long = 2
Const TableHdr1 As String = "Total Pressure Loss Calculations by Sections"
Dim ColCrnt As Long
Dim Section() As String
Dim CriticalPath As String
Dim InxSect As Long
Dim Rng As Range
Dim RowDestNext As Long
Dim RowSrcLast As Long
Dim RowTableHdr1 As Long
Dim wshtDest As Worksheet
Dim wshtSrc As Worksheet
Set wshtSrc = Worksheets("Revit KBK Sonuç")
Set wshtDest = Worksheets("Critical Path")
With wshtDest
.Cells.EntireRow.Delete
End With
' I only work on the ActiveWorksheet if the user is to select the
' target worksheet in this way. Code is easier to understand if
' With statements are used.
With wshtSrc
' Copy column widths
For ColCrnt = 1 To ColLast
wshtDest.Columns(ColCrnt).ColumnWidth = .Columns(ColCrnt).ColumnWidth
Next
' I avoid stringing commands together. The resultant code may be
' marginally faster but it takes longer to write and much longer
' to decipher when you return to the macro in 12 months.
' Extract critial path string and convert to array of Section numbers
RowSrcLast = .Cells(Rows.Count, "A").End(xlUp).Row
CriticalPath = .Cells(RowSrcLast, "A").Value
' Extract text before trailing total pressure loss
CriticalPath = Split(CriticalPath, ";")(0)
' Discard introductory text and trim spaces
CriticalPath = Trim(Split(CriticalPath, ":")(1))
Section = Split(CriticalPath, "-")
Set Rng = .Cells.Find(What:=TableHdr1)
If Rng Is Nothing Then
Call MsgBox("I am unable to find the row containing """ & _
TableHdr1 & """", vbOKOnly)
Exit Sub
End If
RowTableHdr1 = Rng.Row
' Copy header section of worksheet without buttons
.Range(.Cells(1, 1), .Cells(RowTableHdr1 - 1, ColShtHdrLast)).Copy _
Destination:=wshtDest.Cells(1, 1)
' Copy table header
.Range(.Cells(RowTableHdr1, 1), .Cells(RowTableHdr1 + 1, ColLast)).Copy _
Destination:=wshtDest.Cells(RowTableHdr1, 1)
RowDestNext = RowTableHdr1 + 2
' Copy rows for each section in critical path to destination worksheet
For InxSect = 0 To UBound(Section)
Set Rng = .Columns("A:A").Find(What:=Section(InxSect), LookAt:=xlWhole)
If Rng Is Nothing Then
Call MsgBox("I am unable to find the row(s) for Section" & _
Section(InxSect), vbOKOnly)
Else
Set Rng = Rng.MergeArea ' Expand to include all rows for section
' Copy all rows for section
Rng.EntireRow.Copy Destination:=wshtDest.Cells(RowDestNext, 1)
' Step output row number
RowDestNext = RowDestNext + Rng.Rows.Count
End If
Next
' Copy critical path row
.Rows(RowSrcLast).EntireRow.Copy Destination:=wshtDest.Cells(RowDestNext, 1)
RowDestNext = RowDestNext + 1
End With
' Add border at bottom of output table
With wshtDest
With .Range(.Cells(RowDestNext, 1), _
.Cells(RowDestNext, ColLast)).Borders(xlEdgeTop)
.LineStyle = xlContinuous
.Weight = xlMedium
.ColorIndex = 16
End With
End With
End Sub
New version of macro in response to request
Because the sections have different numbers of rows, no in situ sort is possible.
Version 1 solved this problem by copying required rows to a different worksheet. Version 2 solves this problem by copying them to a workarea below the original table but within the same worksheet. That is, a new table is built beneath the old.
Once the new table is complete, the old table is deleted to move the new table into the correct position.
Sub ertert()
Const ColLast As Long = 10
Const ColShtHdrLast As Long = 2
Const TableHdr1 As String = "Total Pressure Loss Calculations by Sections"
Dim ColCrnt As Long
Dim Section() As String
Dim CriticalPath As String
Dim InxSect As Long
Dim Rng As Range
Dim RowDestNext As Long
Dim RowDestStart As Long
Dim RowSrcLast As Long
Dim RowTableHdr1 As Long
Dim wsht As Worksheet
Set wsht = ActiveSheet
With wsht
' Extract critial path string and convert to array of Section numbers
RowSrcLast = .Cells(Rows.Count, "A").End(xlUp).Row
CriticalPath = .Cells(RowSrcLast, "A").Value
' Extract text before trailing total pressure loss
CriticalPath = Split(CriticalPath, ";")(0)
' Discard introductory text and trim spaces
CriticalPath = Trim(Split(CriticalPath, ":")(1))
Section = Split(CriticalPath, "-")
Set Rng = .Cells.Find(What:=TableHdr1)
If Rng Is Nothing Then
Call MsgBox("I am unable to find the row containing """ & _
TableHdr1 & """", vbOKOnly)
Exit Sub
End If
RowTableHdr1 = Rng.Row
' Because there is no fixed number of rows per section no in-situ sort is
' practical. Instead copy required rows in required section to destination
' area below existing area.
RowDestStart = RowSrcLast + 2
RowDestNext = RowDestStart
' Copy rows for each section in critical path to destination area
For InxSect = 0 To UBound(Section)
Set Rng = .Columns("A:A").Find(What:=Section(InxSect), LookAt:=xlWhole)
If Rng Is Nothing Then
Call MsgBox("I am unable to find the row(s) for Section" & _
Section(InxSect), vbOKOnly)
Else
Set Rng = Rng.MergeArea ' Expand to include all rows for section
' Copy all rows for section
Rng.EntireRow.Copy Destination:=.Cells(RowDestNext, 1)
' Step output row number
RowDestNext = RowDestNext + Rng.Rows.Count
End If
Next
' Copy critical path row
.Rows(RowSrcLast).EntireRow.Copy Destination:=.Cells(RowDestNext, 1)
RowDestNext = RowDestNext + 1
' Add border at bottom of output table
With .Range(.Cells(RowDestNext, 1), _
.Cells(RowDestNext, ColLast)).Borders(xlEdgeTop)
.LineStyle = xlContinuous
.Weight = xlMedium
.ColorIndex = 16
End With
' Now have new table on rows RowDestStart to RowDestNext-1.
' Delete rows RowTableHdr1+2 to RowDestStart-1 (old table) to
' move new table into desired position.
.Rows(RowTableHdr1 + 2 & ":" & RowDestStart - 1).EntireRow.Delete
End With
End Sub

Load ASCII code of "alphanumeric chars" in first n bytes of binary file in vbscript (make the code faster)

I want to load the ascii code of all letters and digits in first n bytes (100000 for example) of a binary file into an array. I wrote this code:
Option Explicit
Dim i, lCharCount, lFileByte, lFileArray(99999)
Dim oFSO, oStream, sInFileName
'Validate input command line
If WScript.Arguments.Count < 1 Then
MsgBox "No input file has been specified!", vbExclamation, "My Script"
WScript.Quit
End If
sInFileName = WScript.Arguments(0)
Set oFSO = CreateObject("Scripting.FileSystemObject")
Set oStream = oFSO.OpenTextFile(sInFileName, 1)
Do While Not oStream.AtEndOfStream
lFileByte = Asc(oStream.Read(1))
If (lFileByte > 47 And lFileByte < 58) Or (lFileByte > 64 And lFileByte < 91) Or (lFileByte > 96 And lFileByte < 123) Then
lFileArray(lCharCount) = lFileByte
lCharCount = lCharCount + 1
If lCharCount = 100000 Then Exit Do
End If
Loop
oStream.Close: Set oStream = Nothing
But I need it to run faster. I'd rather not use ADODB but, I'm open to all suggestions if it can't be sped up using FSO. Any ideas?
Try something like this:
cnt = 100000
data = oFSO.OpenTextFile(sInFileName).Read(cnt)
ReDim lFileArray(Len(data)-1)
For i = 1 To Len(data)
lFileArray(i-1) = Asc(Mid(data, i, 1))
Next
Try calling ReadAll on your file instead char by char. This will read the entire file and return it as a string. Then use the same loop, but this time on the returned string, using string scan methods.

Resources