I'm trying to troubleshoot a script and I'm 99% positive that its the CFHTTP function I'm running because when I remove it, the page seems to be much faster, but it's inconsistent. That said, even when the page runs long, the tickcount that I'm outputting around the cfhttp request is short. I'm not sure that CFHTTP is running inside the bounds of tickcount or not since its sending out the request for processing remotely. When I isolate the two things, the tickcounts don't match up.
Does anyone know a way to get the time it takes to execute a CFHTTP function without using tickcount so I can test this properly? I'm using CF9 (not 901 for certain reasons out of my control) so we don't have http logging either.
Thanks a bunch.
John
Related
I'm using Shodan's API through the Anaconda Terminal on Windows 10 to get data against the query below, but after a few seconds of running, the ETA timer freezes, and my network activity drops to zero. Hitting Control+C restarts it when this happens and gets it moving for a few seconds again, but it stops soon after.
shodan download --limit 3100000 data state:"wa"
Also, when it is running- the download speeds seem pretty slow; and I wanted to inquire if there was any way I can speed it up? My Universities internet is capable of upwards of 300 Mbps, but the download seems to cap at 5 Mbps.
I don't know how to solve either of these issues; my device has enough space and my internet isn't disconnecting. We've tried running the Anaconda Terminal as an Administrator, but that hasn't helped either.
I am not familiar with the specific website, but in general seeing limited speed or stopped downloads are not caused by things 'on your side' like the university connection, or even your download script.
Odds are that the website wants to protect itself, and that you need to use the api differently (for example with a different account). Or that you have some usage limits in place based on your account, that you hit.
The best course of action may be to contact the website and ask them how to do this.
I heard back from Shodan support; cross-posting some of their reply here-
The API is not designed for large, bulk export of data. As a result,
you're encountering a few problems/ limits:
There is a hard limit of 1 million results per search query. This means that it isn't possible to download all results for the search query "state:wa".
The search API performs best on the first few pages and progressively responds slower the deeper into the results you get.
This means that the first few pages return instantly whereas the 100th
page will take potentially 10+ seconds.
You can only send 1 request per second so you can't multiplex/ parallelize the search requests.
A lot of high-level analysis can be performed using search facets.
There's documentation on facets in the shodan.pdf booklet floating around their site for returning summary information from their API.
I am helping to develop a fairly complex web application with Django. However some pages are taking 10+ seconds to render. I can't seem to get to the bottom of why this is so slow. The Django Debug Toolbar shows that the bottleneck is clearly CPU, rather than the database or anything else, but it doesn't go into any further detail (as far as I can see). I have tried running a profiler on the system, but can't make head or tail of what is going on from this either. It only shows deep internals as taking up the vast majority of the time, especially <method 'poll' of 'select.poll' objects>, and other such functions as builtins.hasattr, getmodule, ismodule, etc.
I've also tried stepping through the code manually, or pausing execution at random points to try to catch what's going on. I can see that it takes quite a long time to get through a ton of import statements, but then it spends a huge amount of time inside render() functions, taking especially long loading a large number of model fields--something in the order of 100 fields.
None of what is happening looks wrong to me, except for the insane amount of time everything takes to run. It's as though Python is just taking forever to load and parse each .py file before doing any processing. However, this is all running on a brand new Digital Ocean server with an SSD, with Gunicorn, PostgreSQL and Nginx. Does anyone have any hints how I can get to the bottom of this?
In my automated test I have an area that occasionally shows up (and needs to be clicked on when it does show up). This is the perfect place to use an OptionalStep prefix, to prevent the step from failing if the optional area never shows up.
Thing is, I would like the OptionalStep to only wait a second or two before moving on to the rest of the test. Just as I can have object.Exist(2) only wait for 2 seconds, is there a way to have OptionalStep wait for only a couple of seconds?
Some other caveats:
I'd like to keep this as one small line. I know I could create a
multi-line logic test that uses object.Exist(2) inside an If/Then
statement, but I'd rather have the code be small and trim.
I don't want to change the global 20 second timeout just for this one
step.
Since this optional step only shows up in one specific area, it seems
like Recovery Scenarios would not be a good choice to have running
throughout the entire test.
Vitaly's comment would be a good solution as you are possibly unnecessarily over complicating your test.
Also having such a long global timeout is not recommended and should be as low as possible. I usually have it set at around 3 seconds and deal with the synchronisation in the code.
Anything that takes a long period of time should be known about upfront and dealt with in the code. Having a global timeout for everything will cause your test to run unnecessarily slow when most object cannot be found errors occur.
No, this is NOT the "my page doesn't have any traffic and it has to be reloaded" issue.
We have 4 dynos for an alpha application. The reason we do, is because each page takes over 2 seconds to load. Even little things like rendering a text string (no layouts, erb or anything).
If I watch our logs, for our longer queries, they report response times in the 300-700ms range--which is far shorter than 2 seconds.
The DNS is cached, and the collective time to load given that isn't a slow DNS issue. And, that shouldn't affect subsequent page loads, right?
Any thoughts on how to get to the bottom of this would be appreciated.
Here are two screenshots to show what I mean.
http://dl.dropbox.com/u/7175041/Screenshots/qo.png
http://dl.dropbox.com/u/7175041/Screenshots/qq.png
Thanks!
First thing I'd do is to switch on NewRelic Basic - it's a free performance monitor integrated with Heroku. That'll help you get a bearing on the basics of where the trouble is coming from.
I take it that you don't see similar results locally? If you don't, then skip this step, but if you do, you can also run NewRelic locally and interrogate all of your queries for response times.
I'd stay away from using things like the Benchmark library - that was my first thought in troubleshooting a speed issue, but Benchmark is necessarily going to ignore elements of your app that are outside the pure Ruby layer, and if that's where you're slow then NewRelic catches that anyway.
Finally, if all else fails, a support ticket with Heroku's team has always been extremely helpful to me. Just make sure you check the box that lets them clone your app, it makes things a lot easier for them.
Let us know what you find out - I'm curious to see what the particular gremlin is!
I put a stopwatch on it. The first time the app loads (no settings file exists) it takes about 190ms to fail to load four settings. The app runs, three bools and a short string are written as settings, and the next time the app loads, it takes 400ms to read the first setting from the IsolatedStorageSettings.ApplicationSettings collection and about 1ms to get the remainder.
Is there anything I can do to ameliorate this load time?
Ues a better Serialization method ;)
XMLSerialization is okay for more complex graphs, but for simple settings, binary serialization would be much better. Also, when you say fail to load, I assume you're doing a check to see if the files exist? If not, I think there may be exceptions being thrown internally which would slow down execution as well.