I am consuming a game API that updates all active player statistics in real time in the game.
I'm trying to make a way for my code to listen to this API (outside of a loop) and when there are changes in your json response, my code will print on the console. I'm currently trying with Ruby Events, but I didn't get anything other than out of a loop (while true).
old_data = ""
while true
sleep 1
data = Utils.req("GET", "https://127.0.0.1:2999/liveclientdata/playerscores?summonerName=Yoruku", {}, false)
#{"assists":0,"creepScore":50,"deaths":0,"kills":5,"wardScore":0}
next if data.eql? old_data
old_data = data
p "NEW DATA: #{data}"
end
Your code seems to be doing exactly what you want it to do.
You used a technique called polling. It has its issues, like performance and rate limits which you need to consider. But you can't really not use a loop in this case. Because that what polling essentially is.
You could maybe use some async scheduler (like sidekiq) and after each http request you could schedule another one in the future. Or you could use something like sidekiq-cron gem. In that way you can avoid using a loop.
If you want to avoid making requests even when nothing changed on the server you'll need to use some websockets or so called long polling. But idk if the api you to talk to supports it.
Alternatively the api could create a webhook and the api would call you when there is a change.
Related
I understand that delaying or yielding in the ESPAsyncWebServer library callbacks are a no-no. However, my callback function needs to query another device via the Serial port. This process is slow and will crash the ESP32 as a result.
Here is an example:
void getDeviceConfig(AsyncWebServerRequest *request) {
AsyncResponseStream *response =
request->beginResponseStream("application/json");
StaticJsonDocument<1024> doc;
JsonArray array = doc.createNestedArray("get");
for (size_t i = 0; i < request->params(); i++)
array.add(request->getParam(i)->value());
serializeJson(doc, Serial);
/* At this point, the remote device determines what is being asked for
and builds a response. This can take fair bit of time depending on
what is being asked (>1sec) */
response->print(Serial.readStringUntil('\n'));
request->send(response);
}
I looked into building a response callback. However, I would need to know ahead of time how much data the remote device will generate. There's no way for me to know this.
I also looked into using a chunked response. In this case, the library will continuously call my callback function until I return 0 (which indicates that there is no more data). This is a good start - but doesn't quite fit. I can't inform of the caller that there is definitely more data coming, I just haven't received a single byte yet. All I can do here is return 0 which will stop the caller.
Is there an alternative approach I could use here?
The easiest way to do this without major changes to your code is to separate the request and the response and poll periodically for the results.
Your initial request as you have it written would initiate the work. The callback handler would set global boolean variable indicating there was work to be done, and if there were any parameters for the work, would save them in globals. Then it would return and the client would see the HTTP request complete but wouldn't have an answer.
In loop() you'd look for the boolean that there was work to be done, do the work, store any results in global variables, set a different global boolean indicating that the work was done, and set the original boolean that indicated work needed to be done to false.
You'd write a second HTTP request that checked to see if the work was complete, and issue that request periodically until you got an answer. The callback handler for the second request would check the "work was done" boolean and return either the results or an indication that the results weren't available yet.
Doing it this way would likely be considered hostile on a shared server or public API, but you have 100% of the ESP32 at your disposal so while it's wasteful it doesn't matter that it's wasteful.
It would also have problems if you ever issued a new request to do work before the first one was complete. If that is a possibility you'd need to move to a queueing system where each request created a queue entry for work, returned an ID for the request, and then the polling request to ask if work was complete would send the ID. That's much more complicated and a lot more work.
An alternate solution would be to use websockets. ESPAsyncWebServer supports async websockets. A websocket connection stays open indefinitely.
The server could listen for a websocket connection and then instead of performing a new HTTP request for each query, the client would send an indication over the websocket that it wanted to the server to do the work. The websocket callback would work much the same way as the regular HTTP server callback I wrote about above. But when the work was complete, the code doing it would just write the result back to the client over the websocket.
Like the polling approach this would get a lot more complicated if you could ever have two or more overlapping requests.
I'm making a search agreggator and I've been wondering how could I improve the performance of the search.
Given that I'm getting results from different websites, currently I need to wait to receive the results for each provider but this is done one after another so the whole request takes a while to respond.
The easiest solution would be to just make a request from the client for each provider, but this would end up with a ton of request per search, (but if this is the proper way I'll just do it.)
Why I've been wondering is if there's way to return results everytime a provider responds, so if we have providers A, B and C and B already returned results then send it back to the client. In order for this to work all the searchs would need to run in parallel of course.
Do you know a way of doing this?
I'm trying to build a search experience similar to SkyScanner, that loads results but then you can see it still keeps getting more records and it sorts them on the fly (on client side as far as I can see).
Caching is the key here. Best practices for external API (or scraping) is to be as little of a 'taker' as possible. So in your Laravel setup, get your results, but cache the results for as long as makes sense for your app. Although the odds in a skyscanner situation is low that two users will make the exact same request, the odds are much higher that a user will make the same request multiple times, or may share the link, etc.
https://laravel.com/docs/8.x/cache
cache(['key' => 'value'], now()->addMinutes(10));
$value = cache('key');
To actually scrape the content, you could use this:
https://github.com/softonic/laravel-intelligent-scraper
Or to use an API which is the nicer route:
https://docs.guzzlephp.org/en/stable/
On the client side, you could just make a few calls to your own service in separate requests and that would give you your asynchronous feel you're looking for.
I am developing an API using Sinatra on the server-side. I would like to have an HTTP request which is made, but continues to hang/wait and keep alive until a later event (another event) causes it to complete with a certain response value at a later time.
In other words:
get '/api/foo/:request_identifier' do
# some code here
wait_until_finished params[:request_identifier]
end
# When this URL is visited, the hanging request with the matching
# request identifier will complete, sending "foo response text" to the
# client.
get '/api/bar/:request_identifier' do
make_it_finish params[:request_identifier] "foo response text"
"bar response text"
end
How could I implement this, or something to this effect?
I have also considered having the client constantly making requests to the server polling for completed requests, but the high number of requests could result in an expensive internet bill.
I'd be careful with hanging requests as it is not a great user experience. That being said, if you need to have one thing finish before another, here are some options:
Use an event emitter
Use an async library
Without full context of your problem it is hard to recommend one over the other, however, based on what you've described it sounds like a "Promise" would solve your issue here, which is recommendation #2. It basically allows you to wait for one thing to finish before doing thing 2.
We are using aiohttp to post data into elastic search server. Elastic on such insertions generates response for each inserted line, which results in massive unwanted traffic coming back to client application. We wanted to get around this problem using following code
response = await http_session.request("POST", url, data = data, params = params)
first_n_bytes = (await response.content.read(n_bytes)).decode("utf-8")
response.release()
# response.close()
First we tried release method, but from documentation and from bandwidth measurements it seems to also download the whole content. Then we tried response.close() but we are quite unsure whether this is safe thing to do while maintaining the same http_session for other requests.
The question is whether response.close() is safe and whether it would even solve our problem, or alternatively whether there is some other way of doing it asynchronously.
Yes, calling resp.close() is safe.
It closes opened connection to server without reading the response tail.
Obviously keep-alives are not supported with explicit connection closing, what's why resp.release() is recommended for default usage.
But in you case resp.close() should work pretty well.
I have a Sinatra app that basically takes some input values and then finds data matching those values from external services like Flickr, Twitter, etc.
For example:
input:"Chattanooga Choo Choo"
Would go out and find images at Flickr on the Chattanooga Choo Choo and tweets from Twitter, etc.
Right now I have something like:
#images = Flickr::...find...images..
#tweets = Twitter::...find...tweets...
#results << #images
#results << #tweets
So my question is, is there an efficient way in Ruby to run those requests concurrently? Instead of waiting for the images to finish before the tweets finish.
Threads would work, but it's a crude tool. You could try something like this:
flickr_thread = Thread.start do
#flickr_result = ... # make the Flickr request
end
twitter_thread = Thread.start do
#twitter_result = ... # make the Twitter request
end
# this makes the main thread wait for the other two threads
# before continuing with its execution
flickr_thread.join
twitter_thread.join
# now both #flickr_result and #twitter_result have
# their values (unless an error occurred)
You'd have to tinker a bit with the code though, and add proper error detection. I can't remember right now if instance variables work when declared inside the thread block, local variables wouldn't unless they were explicitly declared outside.
I wouldn't call this an elegant solution, but I think it works, and it's not too complex. In this case there is luckily no need for locking or synchronizations apart from the joins, so the code reads quite well.
Perhaps a tool like EventMachine (in particular the em-http-request subproject) might help you, if you do a lot of things like this. It could probably make it easier to code at a higher level. Threads are hard to get right.
You might consider making a client side change to use asynchronous Ajax requests to get each type (image, twitter) independently. The problem with server threads (one of them anyway) is that if one service hangs, the entire request hangs waiting for that thread to finish. With Ajax, you can load an images section, a twitter section, etc, and if one hangs the others will still show their results; eventually you can timeout the requests and show a fail whale or something in that section only.
Yes why not threads?
As i understood. As soon as the user submit a form, you want to process all request in parallel right? You can have one multithread controller (Ruby threads support works really well.) where you receive one request, then you execute in parallel the external queries services and then you answer back in one response or in the client side you send one ajax post for each service and process it (maybe each external service has your own controller/actions?)
http://github.com/pauldix/typhoeus
parallel/concurrent http requests
Consider using YQL for this. It supports subqueries, so that you can pull everything you need with a single (client-side, even) call that just spits out JSON of what you need to render. There are tons of tutorials out there already.