Seeking efficient algorithm to analyze data similar to Pivot Table in VBA - algorithm

Question

Introduction
I already have a working solution to the problem this question will describe. Being new to programming, I think my solution is fairly ugly, but: 1) I am not sure that’s true, and 2) I don’t see a more elegant solution after a solid week reading around the internet studying. Since I can easily solve the problem on an ad hoc basis using Pivot Tables and the subject spreadsheet is common report style, I am thinking that many others have already solved this problem (but I haven’t found it or know the keywords to search for). I will provide the raw data, the desired output, the code I currently have working, and some alternatives that I’ve looked at, but didn’t use and why. I am asking you for a better programming approach -- if you see one -- or at least validation that my method was reasonably as efficient as it could be done. Thank you.
The Raw Data
Below is an example of a report which contains data to be summarized:
Report #| Assignee| Type of Report| Department| Status
1 | Shannon| Ad hoc| Accounting| Declined
2 | Shannon| Ad hoc| Accounting| Completed
3 | Shannon| Change| Accounting| New
4 | Shannon| Change| Shipping| In Progress
5 | Shannon| Training| Shipping| Declined
6 | Shannon| Training| CustService| Completed
7 | Shannon| Training| CustService| New
8 | Jason| Ad hoc| CustService| In Progress
9 | Jason| Change| CustService| Declined
10 | Jason| Ad hoc| Accounting| Completed
11 | Jason| Training| Accounting| New
12 | Thomas| Training| Accounting| In Progress
13 | Jason| Change| Shipping| Declined
14 | Jason| Ad hoc| Shipping| Completed
15 | Thomas| Ad hoc| CustService| New
16 | Jason| New| CustService| In Progress
17 | Thomas| New| CustService| Declined
18 | Thomas| Change| CustService| Completed
19 | Thomas| Ad hoc| Shipping| New
20 | Thomas| Change| Shipping| In Progress
-Continues in similar fashion until:-
545 | Phyllis| New| Accounting| Declined
Desired Output
The data needs to be summarized in a manner similar to what is below. That is, it is filtered by Assignee name, and a count of the subcategories is the output for each parent category. (Note: This output could be easily obtained using a Pivot Table ad hoc, but I want to put it into a running table that builds over time programmatically for trend analysis.)
Shannon: Type of Report Department Status
Ad hoc= 25 Accounting= 45 Declined = 12
Change= 13 CustService= 2 In Progress= 24
Training= 3 Shipping= 75 New= 56
New= 81 Completed= 30
Jason: Type of Report Department Status
Ad hoc= 12 Accounting= 21 Declined = 0
Change= 3 CustService= 23 In Progress= 12
Training= 20 Shipping= 4 New= 12
New= 13 Completed= 24
-Continues for each "Assignee"-
The summarizing I am doing is very similar to what you get from a PivotTable, and that's currently how the manual process gets the data. I need to get the data, however, and put into a daily tracking table which retains the historical counts and is used for trend analysis on each assignee, so a PivotTable by itself isn’t a solution.
Currently Working Code
This is the entire piece of working code, including two subs that are called which are appended to the end:
Sub CollateData()
Dim HdrNm As New Collection 'Collection used to read and reference column indices.
'Variables used for referencing the "Assigned To" column
Dim Assignee As New Scripting.Dictionary
Dim nmAssignee As New Scripting.Dictionary
Dim Asgn As String
Dim a As Integer
Dim aKey As Variant
'Variables used for referencing the "Type of Report" column
Dim TypRep As New Scripting.Dictionary
Dim nmTypRep As New Scripting.Dictionary
Dim arrTypRep() As Integer
Dim Typ As String
Dim t As Integer
'Variables used for referencing the "Department" column
Dim Dept As New Scripting.Dictionary
Dim nmDept As New Scripting.Dictionary
Dim arrDept() As Integer
Dim Bus As String
Dim b As Integer
'Variables used for referencing the "Task Status" column
Dim TskStatus As New Scripting.Dictionary
Dim nmTskStatus As New Scripting.Dictionary
Dim arrTskStatus() As Integer
Dim Tsk As String
Dim s As Integer
'Other variables
Dim DataWS As Worksheet
Dim ScratchWS As Worksheet
Dim lastrow As Integer, x As Integer
Set DataWS = ThisWorkbook.Worksheets("SheetWithRawData")
lastrow = DataWS.Cells(Rows.Count, 11).End(xlUp).Row
Call ReadHeaderRow(DataWS, HdrNm) 'Fills the HdrNm collection with column index using column headers for keys
'Initialize variables for the loop that follows
a = 1
t = 1
b = 1
s = 1
'This next seciont/first loop goes through the report to identify a unique list of assignees and category lists _
' which need to be summed. These lists will be used to ReDim the 2-dimensional arrays to appropriate _
' size, as well as reference the elements of the 2D array(s).
'
' NOTE: I am using the seemingly duplicative Dictionaries (e.g. TypRep & nmTypeRep) in order to have _
' access to the category as both a string and as an integer/index.
For x = 2 To lastrow
If Not Assignee.Exists(DataWS.Cells(x, HdrNm("Assigned to")).Value) Then
Assignee.Add DataWS.Cells(x, HdrNm("Assigned to")).Value, a
nmAssignee.Add a, DataWS.Cells(x, HdrNm("Assigned to")).Value
a = a + 1
End If
If Not TypRep.Exists(DataWS.Cells(x, HdrNm("Type of Report")).Value) Then
TypRep.Add DataWS.Cells(x, HdrNm("Type of Report")).Value, t
nmTypRep.Add t, DataWS.Cells(x, HdrNm("Type of Report")).Value
t = t + 1
End If
If Not Dept.Exists(DataWS.Cells(x, HdrNm("Department")).Value) Then
Dept.Add DataWS.Cells(x, HdrNm("Department")).Value, b
nmDept.Add b, DataWS.Cells(x, HdrNm("Department")).Value
b = b + 1
End If
If Not TskStatus.Exists(DataWS.Cells(x, HdrNm("Task Status")).Value) Then
TskStatus.Add DataWS.Cells(x, HdrNm("Task Status")).Value, s
nmTskStatus.Add s, DataWS.Cells(x, HdrNm("Task Status")).Value
s = s + 1
End If
Next x
'Assign the appropriate dimensions to the following 2D arrays
ReDim arrTypRep(1 To Assignee.Count, 1 To TypRep.Count)
ReDim arrDept(1 To Assignee.Count, 1 To Dept.Count)
ReDim arrTskStatus(1 To Assignee.Count, 1 To TskStatus.Count)
'The following, second loop now goes through and sums up the count of each category element for each _
' Assignee. Using this technique, I only go through the list/report once (or twice, if you consider _
' the previous loop to dimension the arrays) in order to tabulate the desired data.
For x = 2 To lastrow
Asgn = DataWS.Cells(x, HdrNm("Assigned to")).Value
Typ = DataWS.Cells(x, HdrNm("Type of Report")).Value
Bus = DataWS.Cells(x, HdrNm("Department")).Value
Tsk = DataWS.Cells(x, HdrNm("Task Status")).Value
arrTypRep(Assignee.item(Asgn), TypRep.item(Typ)) = arrTypRep(Assignee.item(Asgn), TypRep.item(Typ)) + 1
arrDept(Assignee.item(Asgn), Dept.item(Bus)) = arrDept(Assignee.item(Asgn), Dept.item(Bus)) + 1
arrTskStatus(Assignee.item(Asgn), TskStatus.item(Tsk)) = arrTskStatus(Assignee.item(Asgn), TskStatus.item(Tsk)) + 1
Next x
'Now to generate the output of the data we collected:
On Error Resume Next
Application.DisplayAlerts = False
With ThisWorkbook
.Worksheets("DesiredOutput").Delete
.Worksheets.Add after:=.Worksheets(1)
End With
Application.DisplayAlerts = True
On Error GoTo 0
Set ScratchWS = ThisWorkbook.ActiveSheet
ScratchWS.Name = "DesiredOutput"
x = 1
'Loop through each Assignee and dump out the collected counts
For Each aKey In Assignee
Call OutputData("Type of Report", Assignee, nmAssignee, aKey, TypRep, nmTypRep, arrTypRep, x)
Call OutputData("Department", Assignee, nmAssignee, aKey, Dept, nmDept, arrDept, x)
Call OutputData("Task Status", Assignee, nmAssignee, aKey, TskStatus, nmTskStatus, arrTskStatus, x)
Next aKey
Range("B1").ColumnWidth = 3
Range("A1, C1").EntireColumn.AutoFit
End Sub
****************************************************************************
****************************************************************************
Sub OutputData(Title As String, Assignee As Scripting.Dictionary, nmAssignee As Scripting.Dictionary, _
aKey As Variant, ReportCategory As Scripting.Dictionary, nmReportCategory As Scripting.Dictionary, _
arrCategory() As Integer, x As Integer)
Dim CatKey As Variant
With Cells(x, 2)
.Value = Title
.Font.Bold = True
End With
x = x + 1
For Each CatKey In ReportCategory
Cells(x, 1).Value = nmAssignee.item(Assignee.item(aKey))
Cells(x, 3).Value = nmReportCategory.item(ReportCategory.item(CatKey))
Cells(x, 4).Value = arrCategory(Assignee.item(aKey), ReportCategory.item(CatKey))
x = x + 1
Next CatKey
x = x + 1
End Sub
**************************************************************************
**************************************************************************
Private Sub ReadHeaderRow(TargetWS As Worksheet, HdrNm As Collection)
Dim lastcolumn As Integer
Dim x As Integer
lastcolumn = TargetWS.Cells(1, Columns.Count).End(xlToLeft).Column
For x = 1 To lastcolumn
HdrNm.Add TargetWS.Cells(1, x).Column, TargetWS.Cells(1, x).Value
Next x
End Sub
Other Possible Solutions Considered/Discarded
Looping through list for each Assignee
I thought about generating a dictionary/collection of assignees and then looping through the report gathering the data for each assignee on each pass, but the number of assignees could change (go up), and the report list could go up, and thus many extraneous passes through the list.
Jagged Arrays/Collections/Dictionaries
I was excited when I first learned that I could do a dictionary of dictionaries (array of arrays, etc), but as I understand it, I can't use the first dictionary for assignees (1st Dimension) and a second dictionary for all of one category (e.g. Type of Report) as the second dimension. I would actually need to create a separate dictionary for each assignee and category. In other words, if I had just one category (Type of Report) and 15 assignees, I would actually need to create 16 dictionaries: 1st dictionary would be "Assignee" --the assignee's name as the key, and 2nd through 16th dictionaries (TypRep1 through TypRep15) as the items corresponding to the keys in dictionary Assignee. Plus, I cannot create the dictionaries dynamically as the number of Assignees might change, so this method was out for me, unless I've misunderstood something important (always possible). My knowledge of jagged datatypes comes from here: http://bytecomb.com/collections-of-collections-in-vba/
Custom Data Type
I haven't tried this because I just came across it, and I don't know much about it, but maybe this problem would be solved as a custom data type. I'm going to go read about them more, but perhaps that's a better solution in a way I do not yet understand.
Concluding Statement
I know this was a lot to read, sorry. Thanks for sticking this far. I would greatly appreciate any suggestions on how to achieve what my code above does achieve. I feel confident that the reason I haven't found a better way is that this problem's solution is obvious to everyone but me, and it seems like it would be something anyone coding in VBA/Excel would come across with some frequency. Thank you for your helpful input.

Answer 1

We use objects in programming. Most computers have Excel, so you could get excel to do it for you.
Windows come with an object, which I think of as a datatype. A created in memory, disconnected recordset.
So you would do
rs.filter = "Assignee='Shannon' AND Status='Cancelled'"
then
msgbox rs.recordcount
Gives you the number.
Or you can sort and enumerate.
This cuts lines from the top or bottom of a file.
cscript scriptname.vbs "" t x 5 <infile.txt >outfile.txt
Set Arg = WScript.Arguments
set WshShell = createObject("Wscript.Shell")
Set Inp = WScript.Stdin
Set Outp = Wscript.Stdout
Set rs = CreateObject("ADODB.Recordset")
With rs
.Fields.Append "LineNumber", 4
.Fields.Append "Txt", 201, 5000
.Open
LineCount = 0
Do Until Inp.AtEndOfStream
LineCount = LineCount + 1
.AddNew
.Fields("LineNumber").value = LineCount
.Fields("Txt").value = Inp.readline
.UpDate
Loop
.Sort = "LineNumber ASC"
If LCase(Arg(1)) = "t" then
If LCase(Arg(2)) = "i" then
.filter = "LineNumber < " & LCase(Arg(3)) + 1
ElseIf LCase(Arg(2)) = "x" then
.filter = "LineNumber > " & LCase(Arg(3))
End If
ElseIf LCase(Arg(1)) = "b" then
If LCase(Arg(2)) = "i" then
.filter = "LineNumber > " & LineCount - LCase(Arg(3))
ElseIf LCase(Arg(2)) = "x" then
.filter = "LineNumber < " & LineCount - LCase(Arg(3)) + 1
End If
End If
Do While not .EOF
Outp.writeline .Fields("Txt").Value
.MoveNext
Loop
End With
One approach for you would be to put your 4 criteria into 4 arrays- Assignee, Status, Dept, theotherone.
For each a in assignee()
For each b in Status()
For each c in Dept()
For each d in other()
.filter = "assinnee=" & a & "AND Status=" & b & "And dept=" & c
msgbox .recordcount
Next
Next
Next
Next