Yammer API get/messages.json returns incomplete results - yammer

Question

When I compare the data returned by get/messages.json to the data retrieved using the export in yammer. get/messages.json returns 6,300 records. The data export returns 10,469 records. Similar to this post I too am using the older_than parameter. There was a comment in that post that suggested a rate limit issue. I can assure you I am not exceeding the rate limit as I pause for 15 seconds after every 10 requests.
To get my incomplete 6,300 rows I...
Use the export API to get a list of all of the groups
Loop through that list of groups and download the messages for each group using https://www.yammer.com/api/v1/messages/in_group/:group_id.json
Then I use https://www.yammer.com/api/v1/messages.json with the older_than parameter to get all of the messages in the All Company feed.
The issue appears to be with step 3.
Here is the code related to step 3 outlined above:
Sub GetAllCompanyMessages()
Try
Console.WriteLine("Getting All Company Messages")
If File.Exists(allCompanyPath) Then
'delete previous
If Directory.Exists(allCompanyPath) Then
For Each oldFile As String In Directory.GetFiles(allCompanyPath)
File.Delete(oldFile)
Next
Directory.Delete(allCompanyPath)
End If
'create dir
Directory.CreateDirectory(allCompanyPath)
Else
'create dir
Directory.CreateDirectory(allCompanyPath)
'Throw New Exception("Yammer Data Export Zip Download Failed")
End If
'get first group of messages
Console.WriteLine("Getting All Company Batch 1")
Dim client As New WebClient()
client.Headers.Add("Authorization", "Bearer " & accessToken)
client.DownloadFile(allCompanyMessagesURL, allCompanyPath & "1.json")
'getOlderThanID
Dim olderThanID As Int32 = getOlderThanID(allCompanyPath & "1.json")
'get remaining messages in batches of 20
Dim i As Int32 = 2
'Dim prevOlderThanID As Int32 = 0
Dim nextOlderThanID As Int32 = olderThanID
Do Until i = 0
Console.WriteLine("Getting All Company Batch " & i & " olderthanID " & nextOlderThanID)
client = Nothing
client = New WebClient()
client.Headers.Add("Authorization", "Bearer " & accessToken)
client.DownloadFile(allCompanyMessagesURL & "?older_than=" & nextOlderThanID, allCompanyPath & i & ".json")
'prevOlderThanID = nextOlderThanID
nextOlderThanID = getOlderThanID(allCompanyPath & i & ".json")
i = i + 1
If nextOlderThanID = 0 Then
'exit loop
i = 0
End If
' HANDLES 10 REQUESTS IN 10 SECONDS LIMIT
If i >= 10 Then
If i Mod 10 = 0 Then
' CAUSES APP TO WAIT 15 SECONDS AFTER EVERY 10th REQUEST
Console.WriteLine("Sleeping for 15 seconds")
System.Threading.Thread.Sleep(15000)
End If
End If
Loop
Console.WriteLine("Concatenating All Company Batches")
Dim masterJobject As New JObject
masterJobject = JObject.Parse("{""messages"":[]}")
For Each path As String In Directory.GetFiles(allCompanyPath, "*.json")
Console.WriteLine("Concatenating All Company Batch: " & path)
'open each json get messages object and append
Dim jObj As JObject = JObject.Parse(File.ReadAllText(path))
Dim jms As New JsonMergeSettings
'beh 5.24.17 jms.MergeArrayHandling = MergeArrayHandling.Union
jms.MergeArrayHandling = MergeArrayHandling.Concat
masterJobject.Merge(jObj, jms)
'File.Delete(path)
Next
Console.WriteLine("Building Yammer-All-Company-Messages.json")
File.WriteAllText(outputJSONpath & "Yammer-All-Company-Messages.json", "{ ""messages"":" & masterJobject("messages").ToString() & "}")
Catch ex As Exception
ErrorHandler("ERROR GetAllCompanyMessages: " & ex.Message)
End Try
End Sub
Function getOlderThanID(ByVal jsonPath As String) As Int32
Dim result As Int32 = 0
Try
Dim jObj As New JObject
jObj = JObject.Parse(File.ReadAllText(jsonPath))
If CBool(jObj("meta")("older_available")) = True Then
If Not IsNothing(jObj("messages").Last()("id")) Then
result = jObj("messages").Last()("id")
End If
End If
Catch ex As Exception
ErrorHandler("ERROR getOlderThanID: " & ex.Message)
End Try
Return result
End Function
I would appreciate any insight on what the issue might be with the get/messages.json API endpoint, and how I might modify my code to resolve this issue.

Answer 1

There are technical limits on the number of items that will be returned from the REST API for messages. These were designed for client applications that need recent data. The best option would be use individual calls to the message endpoint from the REST API (api_url field) to fill in any gaps in your archive. Make sure everything is stored in a persistent fashion.