FB Messenger API - Receiving double requests - ruby

I have a working FB Bot built with Ruby which allows players to play a scavenger hunt.
Sometimes though, when I have multiple players in a team, FB is sending me a players 'Answer' webhook twice. I have looked into it and at first thought it was to do with the 20 second timeout if FB gets no 200 OK response (Docs here). After checking the logs though, I am receiving the second webhook from FB only 14 seconds later. See below:
# Webhook #1
{"object"=>"page", "entry"=>[{"id"=>"252445748474312", "time"=>1532153642358, "messaging"=>[{"sender"=>{"id"=>"1709242109154907"}, "recipient"=>{"id"=>"252445748474312"}, "timestamp"=>1532153641935, "message"=>{"mid"=>"0FeOChulGjuPgg3YJqEgajNsY8kMfNRt_bpIdeegEeE54h-KB8szcd-EQ-UHUT3850RwHgH4TxVYFkoFwxqhtg", "seq"=>402953, "text"=>"Larrikins"}}]}]}
# Webhook #2 (14 seconds later)
{"object"=>"page", "entry"=>[{"id"=>"252445748474312", "time"=>1532153656901, "messaging"=>[{"sender"=>{"id"=>"1709242109154907"}, "recipient"=>{"id"=>"252445748474312"}, "timestamp"=>1532153641935, "message"=>{"mid"=>"0FeOChulGjuPgg3YJqEgajNsY8kMfNRt_bpIdeegEeE54h-KB8szcd-EQ-UHUT3850RwHgH4TxVYFkoFwxqhtg", "seq"=>402953, "text"=>"Larrikins"}}]}]}
Notice both are exactly the same apart from the first "time" attribute (14 secs later).
Due to a number of methods and calls that I process after receiving the first webhook, the 200 OK response is only being sent back to FB once I have finished sending my messages in response (hence the 14 second delay).
So I have two questions:
Is the 14 second delay too long and that is why FB is resending? If so, how can I send a 200OK response straight away (head :ok)?
Is it another issue entirely?

You also ensure that "Echo" is disabled.
Go to Settings>Webhooks, edit events.
Asyncronous language like NodeJS is recomended, in my case y work with AWS SQS, I have workers that process the requests witout blocking (dont wait), I return 200,"ok" to FB to avoid that FB send again the message to my webhook.
Anothe apporach maybe store the mid in database, and check in each request if the mid exists, if exists the dont process the message. I was use Dynamo DB (AWS) with TTL enabled, thus with TTL my database autoclean every hour erasing old request.

I think it is the 15 second wait before replying, was also happening to me as Facebook auto retries when you don't reply fast enough. Te EEe Te's idea is solid, write some mechanism to cache mids and check if it is a duplicate before processing

Related

While using Instaloader via command line, how can I force 429 errors to cause requests to be retried after a longer period of time?

I am using Instaloader via command line on Windows 11, with the following command:
.\instaloader --login=MYUSERNAME :saved --dirname-pattern="Saved_Posts\{profile}" --filename-pattern="{profile}-{shortcode}" --no-resume --no-metadata-json --slide 1 --no-captions --no-video-thumbnails --no-iphone
This attempts to download approximately 12,000 saved posts from a profile. Instaloader behaves as expected for several thousand posts, occasionally giving the following error:
Too many queries in the last time. Need to wait 15 seconds, until 13:19.
The process then resumes successfully for several hundred more posts. Eventually, however, I start encountering 429 errors:
JSON Query to graphql/query: 429 Too Many Requests [retrying; skip with ^C]
Number of requests within last 10/11/20/22/30/60 minutes grouped by type:
d6f4427fbe92d846298cf93df0b937d3: 0 0 0 0 0 0
f883d95537fbcd400f466f63d42bd8a1: 0 0 0 1 1 11
* 2b0673e0dc4580674a88d426fe00ea90: 59 64 121 134 191 709
Instagram responded with HTTP error "429 - Too Many Requests". Please
do not run multiple instances of Instaloader in parallel or within
short sequence. Also, do not use any Instagram App while Instaloader
is running.
The request will be retried in 7 seconds, at 14:01.
This error then repeats over and over again, I believe until the default maximum connection attempts limit is reached and it moves onto the next post — which also receives the same error. Importantly, this error does not go away after several hours of these 'slower' requests being made; it seems to persist as long as Instaloader stays open. I have seen these 429 errors with very few requests in the last 60 minutes (i.e. <100), which makes me think I am hitting quite a long shadowban.
I have tried setting the maximum connection attempts to 0 (i.e. retry indefinitely), but this time limit appears to be capped at 666 seconds, or 11 minutes. The error does not seem to clear even leaving Instaloader to send requests every 11 minutes in this way; it is as though each individual request 'resets' the ban or something.
I am looking for a way of resolving this issue, which could include:
Adding a command to force 429 errors to be retried after subsequently longer periods of time (instead of the number of seconds being capped at 666 seconds)
Adding a command that 'preserves' wait times after each 429 error. e.g. if downloading Post 456 fails and retries after 5, then 10, then 15 seconds before successfully downloading, and then downloading Post 457 immediately fails... start the wait for a retry on Post 457 at at LEAST 15 seconds, rather than going back to 5!
Avoiding the 429 errors in the first place, if there appears to be an issue with my command line prompt
Breaking down the request into 'batches' and running one of those prompts every few days. e.g. is there a way to download Saved Posts 1-500, then 500-1000, and so on? (The Saved Posts are not necessarily in chronological order of the post date, which is what I've tried so far)
I have looked at several other posts on 429 errors but the general theme seems to be either:
Wait some time for the issue to clear — have tried this for up to 48 hours, but running the command again starts from post #1 and never makes it to the latter half of posts
Disable iPhone API requests — already done, which helps but does not solve the issue
The 429 errors simply should not be encountered during normal behaviour – well, they are!

ChromeCast - Stream Calls Failing when Stalled for long time

I'm attempting to play a live stream on ChromeCast. The stream is thrown fine and starts playback appropriately.
However when I play the stream longer: somewhere between 2-15 minutes, the player stops playing and I get MediaStatus.IDLE_REASON_ERROR in my RemoteMediaClient.Callback.
When looking at the console logs from ChromeCast I see that 3-4 calls are failed. Here are the logs:
14:50:26.931 GET https://... 0 ()
14:50:27.624 GET https://... 0 ()
14:50:28.201 GET https://... 0 ()
14:50:29.351 GET https://... 0 ()
14:50:29.947 media_player.js:64 [1381.837s] [cast.player.api.Host] error: cast.player.api.ErrorCode.NETWORK/3126000
Looking at Cast MediaPlayer.ErrorCode Error 312.* is
Failed to retrieve the media (bitrated) playlist m3u8 file with three retries.
Developers need to validate that their playlists are indeed available. It could be the case that a user that cannot reach the playlist as well.
I checked, the playlist was available. So I thought perhaps the server wasn't responding. So I looked at the network calls response logs.
Successful Request
Stalled Request
Note that the stall time far exceeds the usual stall time.
ChromeCast isn't making these calls at all, the requests are simply stalled for a long time until they are cancelled. All the successful requests are stalled for less than 14ms (mostly under 2ms).
The Network Analysis Timing Breakdown provides three reasons for stalling
There are higher priority requests.
There are already six TCP connections open for this origin, which is the limit. Applies to HTTP/1.0 and HTTP/1.1 only.
The browser is briefly allocating space in the disk cache
While I do believe the first one should not be the case, the later two can be. However in both cases I believe the fault lies with cast.player.
Am I doing something wrong?
Has anyone else faced the same issue? Is there any way to either fix it or come up with a work-around.

Gatling: define message rate in minutes (JMS scenario)

I'm performing a load tests against IBM MQ and would like to have 10 msgs./users being submitted in 10 minutes (just a as a proof of concept).
I'm injecting the respective load like this:
scn_message_ZIP_DP102.inject(rampUsers(10) over(10 minutes)).protocols(jmsConfigMQ1)
But when checking the logs I'm seeing the applicaiton is being flooded with messages. I'd expect to have just 10 messages being submitted in a timeframe of 10 minutes.
Well we have an answer - in 10 minutes you start 10 users and every of them is sending message after message in a 48 hour loop, so instead of 10 messages you probably have hundreds of millions of them. Remove during loop and it should be fine fe.:
val scnMessageID14 = scenario("Load testing InboundQueue on MQ-HOST-1 with MessageID14")
.exec(
jms("F&F testing with MessageID 14")
.send
.queue("MESSAGES.QUEUE")
.textMessage(message14)
)

Redis subscriber is not notified by EXPIRE key 0

I've got a Redis client subscribed to __keyevent#0__:expired notifications. It works perfectly, either when the key expires by itself (ttl reached) or when I expire them manually with a number of seconds greater than 0, like so:
EXPIRE myKey 1
The subscriber sees the expired event and can therefore take some actions.
However, if I want to manually delete the key and have the subscriber notified, I use EXPIRE with 0 as the number of seconds:
EXPIRE myKey 0
The key gets deleted, but the subscriber doesn't receive anything.
I can't see anything related to this in the doc. Can anyone explain this behavior?
From reviewing the source code (expire.c, ~252), setting an expiry value of <=0 (or using EXPIREAT with a time in the past) results in a deletion of the key rather than an expiry (and accordingly a DEL notification rather than an EXPIRED event).
This behavior is indeed undocumented and it would be good if you could submit a PR that fixes that to the documentation repo (https://github.com/antirez/redis-doc).

Yammer JSON Feed returning only 20 messages

I am trying to get all the messages from a particular group. I am getting the json feed back. The only problem is, its returning only 20 messages. Is this set as default or something. Is there any way by by which while doing the request, I can specify whether I want all the messages, by default just 20 or even messages posted between the start and the end date?
My RestApi call is:
https://www.yammer.com/api/v1/messages/in_group/[id].json
From Yammer Developer Documentation
<
Autocomplete: 10 requests in 10 seconds.
Messages: 10 requests in 30 seconds.
Notifications: 10 requests in 30 seconds.
All Other Resources: 10 requests in 10 seconds.
These limits are independent e.g. in the same 30 seconds period, you could make 10 message calls and 10 notification calls. The specific rate limits are subject to change but following the guidelines below will ensure that your app is not blocked.>>
I have tried using limit as the parameter to change the number of message more than 20. But it doesnt seem to be working?
Is this problem because of Rate Limit. If not, what's the problem?
Official documentation from Yammers Developer documentation
Messages - Viewing Messages
Endpoints:
1) All public messages in the user’s (whose access token is being used to make the API call henceforth referred to as current user) Yammer network. Corresponds to “All” conversations in the Yammer web interface.
GET https://www.yammer.com/api/v1/messages.json
2) The user’s feed, based on the selection they have made between “Following” and “Top” conversations.
GET https://www.yammer.com/api/v1/messages/my_feed.json
3) The algorithmic feed for the user that corresponds to “Top” conversations, which is what the vast majority of users will see in the Yammer web interface.
GET https://www.yammer.com/api/v1/messages/algo.json
4) The “Following” feed which is conversations involving people, groups and topics that the user is following.
GET https://www.yammer.com/api/v1/messages/following.json
5) All messages sent by the user. Alias for /api/v1/messages/from_user/logged-in_user_id.format.
GET https://www.yammer.com/api/v1/messages/sent.json
6) Private messages received by the user.
GET https://www.yammer.com/api/v1/messages/private.json
7) All messages received by the user.
GET https://www.yammer.com/api/v1/messages/received.json
Parameters:
The messages API endpoints return a similar structure and support the following query parameters:
older_than - Returns messages older than the message ID specified as a numeric string. This is useful for paginating messages. For example, if you’re currently viewing 20 messages and the oldest is number 2912, you could append “?older_than=2912″ to your request to get the 20 messages prior to those you’re seeing.
newer_than - Returns messages newer than the message ID specified as a numeric string. This should be used when polling for new messages. If you’re looking at messages, and the most recent message returned is 3516, you can make a request with the parameter “?newer_than=3516″ to ensure that you do not get duplicate copies of messages already on your page.
threaded=[true | extended] - threaded=true will only return the first message in each thread. This parameter is intended for apps which display message threads collapsed. threaded=extended will return the thread starter messages in order of most recently active as well as the two most recent messages, as they are viewed in the default view on the Yammer web interface.
limit - Return only the specified number of messages. Works for threaded=true and threaded=extended.
Noted the limit parameter that you can set on your GET request - so based on this documentation if it is correct (I'm not a Yammer Developer but I do use it) you should be able to do
https://www.yammer.com/api/v1/messages.json?limit=50
That is in theory but reading through the documentation there is a section on Search that has
page - Only 20 results of each type will be returned for each page, but a total count is returned with each query. page=1 (the default) will return items 1-20, page=2 will return items 21-30, etc.
Which says to me they are limited to 20 results returned.
UPDATE
After testing this with https://www.yammer.com/api/v1/messages.json?limit=50 and it not returning 50 messages but doing https://www.yammer.com/api/v1/messages.json?limit=5 will return only 5 messages I would say that Yammer restrict the number of messages to 20 Also after reading through the documents a bit more I read
For example, if you’re currently viewing 20 messages and the oldest is number 2912, you could append “?older_than=2912″ to your request to get the 20 messages prior to those you’re seeing"
This says to me that they will only return a max of 20. So I think you are stuck with 20 messages at a time.
Hope this helps.
You need to use Parameters:
The messages API endpoints return a similar structure and support the following query parameters:
older_than - Returns messages older than the message ID specified as a numeric string. This is useful for paginating messages. For example, if you’re currently viewing 20 messages and the oldest is number 2912, you could append “?older_than=2912″ to your request to get the 20 messages prior to those you’re seeing.
newer_than - Returns messages newer than the message ID specified as a numeric string. This should be used when polling for new messages. If you’re looking at messages, and the most recent message returned is 3516, you can make a request with the parameter “?newer_than=3516″ to ensure that you do not get duplicate copies of messages already on your page.
threaded=[true | extended] - threaded=true will only return the first message in each thread. This parameter is intended for apps which display message threads collapsed. threaded=extended will return the thread starter messages in order of most recently active as well as the two most recent messages, as they are viewed in the default view on the Yammer web interface.
limit - Return only the specified number of messages. Works for threaded=true and threaded=extended.
Example : GET https://www.yammer.com/api/v1/messages.json?older_than=2912
while older can be ID of message number 20 and so on you can get 20 by 20
I solved by requesting subsequent pages in a recursive manner.
You can simply increase the page parameter until the response is empty, or update the older_than parameter until the property meta.older_available is false.

Resources