Joining concurrent/identical HTTP GET requests? - ajax
Do browsers join concurrent identical HTTP GET requests? At least, for static or cache-able content?
That is, if something like this happens:
| AJAX/HTTP-GET(resourceX)
| [start download]------------------------------------------->[finish download]
|
| AJAX/HTTP-GET(resourceX)
| [start download]--------->etc...
|
+------------------------------------------------------------------> Time
Will the browser figure out "Hey you're already trying to download resourceX! Don't try downloading it twice, it won't do anything!"?
**Update:
Now of course, I can go to some site and try downloading a big file (e.g., "BigFile"), and click the link twice; this will (duplicately) download both BigFile and BigFile(1). Granted, it's an error on the user's part, but still...
For cache-able resources (e.g., downloading some javascript file), it seems pretty inefficient if browsers couldn't figure out these duplicates...
The browser won't notice. It acts just like regular HTTP traffic. It might cache the request once the first one is finished (if the proper cache-control fields are set), but concurrently, no.
Related
ChromeCast - Stream Calls Failing when Stalled for long time
I'm attempting to play a live stream on ChromeCast. The stream is thrown fine and starts playback appropriately. However when I play the stream longer: somewhere between 2-15 minutes, the player stops playing and I get MediaStatus.IDLE_REASON_ERROR in my RemoteMediaClient.Callback. When looking at the console logs from ChromeCast I see that 3-4 calls are failed. Here are the logs: 14:50:26.931 GET https://... 0 () 14:50:27.624 GET https://... 0 () 14:50:28.201 GET https://... 0 () 14:50:29.351 GET https://... 0 () 14:50:29.947 media_player.js:64 [1381.837s] [cast.player.api.Host] error: cast.player.api.ErrorCode.NETWORK/3126000 Looking at Cast MediaPlayer.ErrorCode Error 312.* is Failed to retrieve the media (bitrated) playlist m3u8 file with three retries. Developers need to validate that their playlists are indeed available. It could be the case that a user that cannot reach the playlist as well. I checked, the playlist was available. So I thought perhaps the server wasn't responding. So I looked at the network calls response logs. Successful Request Stalled Request Note that the stall time far exceeds the usual stall time. ChromeCast isn't making these calls at all, the requests are simply stalled for a long time until they are cancelled. All the successful requests are stalled for less than 14ms (mostly under 2ms). The Network Analysis Timing Breakdown provides three reasons for stalling There are higher priority requests. There are already six TCP connections open for this origin, which is the limit. Applies to HTTP/1.0 and HTTP/1.1 only. The browser is briefly allocating space in the disk cache While I do believe the first one should not be the case, the later two can be. However in both cases I believe the fault lies with cast.player. Am I doing something wrong? Has anyone else faced the same issue? Is there any way to either fix it or come up with a work-around.
HOWTO split response time into dns name lookup, wait, transfer time in JMeter
I would like to know if I can get a breakdown of response times in JMeter load tests. E.g. when I use curl I can get the breakdown of each response time by specifying curl format like so,
\n
time_namelookup: %{time_namelookup}\n
time_connect: %{time_connect}\n
time_appconnect: %{time_appconnect}\n
time_pretransfer: %{time_pretransfer}\n
time_redirect: %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
----------\n
time_total: %{time_total}\n
\n
and then making the actual curl call like so,
curl -w "#curl-format.txt" "http://some.api/call"
As you can see this gives me the breakdown in terms of time spent doing a DNS Name resolution, connecting with the server, transferring response form server to the client etc.
Is it possible to get something similar in JMeter?
So I have at least found a way to get what I want, partially. In Jmeter I can collect the Connect time, which is a combination of DNS lookup, handshake & connection. If someone has a better answer, would be happy to know it.
AJAX query weird delay between DNS lookup and initial connection on Chrome but not FF, what is it?
I have an AJAX query on my client that passes two parameters to a server:
var url = window.location.origin + "/instanceStats"
$.getJSON(url, { 'unit' : unit, "stat" : stat }, function(data) {
instanceData[key] = data;
var count = showInstanceStats(targetElement, unit, stat, limiter);
});
The server itself is a very simple Python Flask application. On that particular URL, it grabs the "unit" and "stat" parameters from the query to determine the name of a CSV file and line within that file, grabs the line, and sends the data back to the client formatted as JSON (roughly 1KB).
Here is the funny thing: When I measure the time it takes for the data to come back, I observe that some queries are fast (between 20 and 40 ms), and some queries are slow (between 320 and 350 ms). Varying the "stat" parameter (i.e. selecting a different line in the CSV) doesn't seem to have any impact. The fast and slow queries usually switch back and forth (i.e. all even queries are fast, all odd ones are slow). The Python server itself reports roughly the same time for each query.
AJAX itself doesn't seem to have any impact either, as I can take the url that is constructed in the JS and paste it into the browser myself and get the same behavior. Here are some measurements from two subsequent queries:
Fast: http://i.imgur.com/VQ7qopd.png
Slow: http://i.imgur.com/YuG0ROM.png
This seems to be Chrome-specific, as I've tried it on Firefox and the same experiment yields roughly the same query time everytime (between 30 and 50 ms). This is unfortunate, as I want to deploy on both Chrome and Firefox.
What's causing this behavior, and how can I fix it?
I've run into this also. It only seems to happen when using localhost. If you use 127.0.0.1 (or even the computer name), it will not have the extra delay.
I'm having it too, and it's exactly the same: my Node.js application serves Ajax requests and no matter which /url I request it's either 30ms or 300ms and it switches back and forth: odd requests are long, even requests are short. The thing I see in Chrome Web Inspector (aka Chrome DevTools) is that there is a long gap between "DNS lookup" and "Initial Connection". They say it's OCSP related here: http://www.webpagetest.org/forums/showthread.php?tid=12357 OCSP is some kind of certificate validation protocol: https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol Moving from localhost to 127.0.0.1 seems to fix it: response times are 30ms now.
Scraping Real Time Visitors from Google Analytics
I have a lot of sites and want to build a dashboard showing the number of real time visitors on each of them on a single page. (would anyone else want this?) Right now the only way to view this information is to open a new tab for each site.
Google doesn't have a real-time API, so I'm wondering if it is possible to scrape this data. Eduardo Cereto found out that Google transfers the real-time data over the realtime/bind network request. Anyone more savvy have an idea of how I should start? Here's what I'm thinking:
Figure out how to authenticate programmatically
Inspect all of the realtime/bind requests to see how they change. Does each request have a unique key? Where does that come from? Below is my breakdown of the request:
https://www.google.com/analytics/realtime/bind?VER=8
&key= [What is this? Where does it come from? 21 character lowercase alphanumeric, stays the same each request]
&ds= [What is this? Where does it come from? 21 character lowercase alphanumeric, stays the same each request]
&pageId=rt-standard%2Frt-overview
&q=t%3A0%7C%3A1%3A0%3A%2Ct%3A11%7C%3A1%3A5%3A%2Cot%3A0%3A0%3A4%2Cot%3A0%3A0%3A3%2Ct%3A7%7C%3A1%3A10%3A6%3D%3DREFERRAL%3B%2Ct%3A10%7C%3A1%3A10%3A%2Ct%3A18%7C%3A1%3A10%3A%2Ct%3A4%7C5%7C2%7C%3A1%3A10%3A2!%3Dzz%3B%2C&f
The q variable URI decodes to this (what the?):
t:0|:1:0:,t:11|:1:5:,ot:0:0:4,ot:0:0:3,t:7|:1:10:6==REFERRAL;,t:10|:1:10:,t:18|:1:10:,t:4|5|2|:1:10:2!=zz;,&f
&RID=rpc
&SID= [What is this? Where does it come from? 16 character uppercase alphanumeric, stays the same each request]
&CI=0
&AID= [What is this? Where does it come from? integer, starts at 1, increments weirdly to 150 and then 298]
&TYPE=xmlhttp
&zx= [What is this? Where does it come from? 12 character lowercase alphanumeric, changes each request]
&t=1
Inspect all of the realtime/bind responses to see how they change. How does the data come in? It looks like some altered JSON. How many times do I need to connect to get the data? Where is the active visitors on site number in there? Here is a dump of sample data:
19
[[151,["noop"]
]
]
388
[[152,["rt",[{"ot:0:0:4":{"timeUnit":"MINUTES","overTimeData":[{"values":[49,53,52,40,42,55,49,41,51,52,47,42,62,82,76,71,81,66,81,86,71,66,65,65,55,51,53,73,71,81],"name":"Total"}]},"ot:0:0:3":{"timeUnit":"SECONDS","overTimeData":[{"values":[0,1,1,1,1,0,1,0,1,1,1,0,2,0,2,2,1,0,0,0,0,0,2,1,1,2,1,2,0,5,1,0,2,1,1,1,2,0,2,1,0,5,1,1,2,0,0,0,0,0,0,0,0,0,1,1,0,3,2,0],"name":"Total"}]}}]]]
]
388
[[153,["rt",[{"ot:0:0:4":{"timeUnit":"MINUTES","overTimeData":[{"values":[52,53,52,40,42,55,49,41,51,52,47,42,62,82,76,71,81,66,81,86,71,66,65,65,55,51,53,73,71,81],"name":"Total"}]},"ot:0:0:3":{"timeUnit":"SECONDS","overTimeData":[{"values":[2,1,1,1,1,1,0,1,0,1,1,1,0,2,0,2,2,1,0,0,0,0,0,2,1,1,2,1,2,0,5,1,0,2,1,1,1,2,0,2,1,0,5,1,1,2,0,0,0,0,0,0,0,0,0,1,1,0,3,2],"name":"Total"}]}}]]]
]
388
[[154,["rt",[{"ot:0:0:4":{"timeUnit":"MINUTES","overTimeData":[{"values":[53,53,52,40,42,55,49,41,51,52,47,42,62,82,76,71,81,66,81,86,71,66,65,65,55,51,53,73,71,81],"name":"Total"}]},"ot:0:0:3":{"timeUnit":"SECONDS","overTimeData":[{"values":[0,3,1,1,1,1,1,0,1,0,1,1,1,0,2,0,2,2,1,0,0,0,0,0,2,1,1,2,1,2,0,5,1,0,2,1,1,1,2,0,2,1,0,5,1,1,2,0,0,0,0,0,0,0,0,0,1,1,0,3],"name":"Total"}]}}]]]
]
Let me know if you can help with any of the items above!
To get the same, Google has launched new Real Time API. With this API you can easily retrieve real time online visitors as well as several Google Analytics with following dimensions and metrics. https://developers.google.com/analytics/devguides/reporting/realtime/dimsmets/ This is quite similar to Google Analytics API. To start development on this, https://developers.google.com/analytics/devguides/reporting/realtime/v3/devguide
With Google Chrome I can see the data on the Network Panel. The request endpoint is https://www.google.com/analytics/realtime/bind Seems like the connection stays open for 2.5 minutes, and during this time it just keeps getting more and more data. After about 2.5 minutes the connection is closed and a new one is open. On the Network panel you can only see the data for the connections that are terminated. So leave it open for 5 minutes or so and you can start to see the data. I hope that can give you a place to start.
Having google in the loop seems pretty redundant. Suggest you use a common element delivered on demand from the dashboard server and include this item by absolute URL on all pages to be monitored for a given site. The script outputting the item can read the IP of the browser asking and these can all be logged into a database and filtered for uniqueness giving a real time head count.
<?php
$user_ip = $_SERVER["REMOTE_ADDR"];
/// Some MySQL to insert $user_ip to the database table for website XXX goes here
$file = 'tracking_image.gif';
$type = 'image/gif';
header('Content-Type:'.$type);
header('Content-Length: ' . filesize($file));
readfile($file);
?>
Ammendum:
A database can also add a timestamp to every row of data it stores. This can be used to further filter results and provide the number of visitors in the last hour or minute.
Client side Javascript with AJAX for fine tuning or overkill
The onblur and onfocus javascript commands can be used to tell if the the page is visible, pass the data back to the dashboard server via Ajax. http://www.thefutureoftheweb.com/demo/2007-05-16-detect-browser-window-focus/
When a visitor closes a page this can also be detected by the javascript onunload function in the body tag and Ajax can be used to send data back to the server one last time before the browser finally closes the page.
As you may also wish to collect some information about the visitor like Google analytics does this page https://panopticlick.eff.org/ has a lot of javascript that can be examined and adapted.
I needed/wanted realtime data for personal use so I reverse-engineered their system a little bit.
Instead of binding to /bind I get data from /getData (no pun intended).
At /getData the minimum request is apparently: https://www.google.com/analytics/realtime/realtime/getData?pageId&key={{propertyID}}&q=t:0|:1
Here's a short explanation of the possible query parameters and syntax, please remember that these are all guesses and I don't know all of them:
Query Syntax: pageId&key=propertyID&q=dataType:dimensions|:page|:limit:filters
Values:
pageID: Required but seems to only be used for internal analytics.
propertyID: a{{accountID}}w{{webPropertyID}}p{{profileID}}, as specified at the Documentation link below. You can also find this in the URL of all analytics pages in the UI.
dataType:
t: Current data
ot: Overtime/Past
c: Unknown, returns only a "count" value
dimensions (| separated or alone), most values are only applicable for t:
1: Country
2: City
3: Location code?
4: Latitude
5: Longitude
6: Traffic source type (Social, Referral, etc.)
7: Source
8: ?? Returns (not set)
9: Another location code? longer.
10: Page URL
11: Visitor Type (new/returning)
12: ?? Returns (not set)
13: ?? Returns (not set)
14: Medium
15: ?? Returns "1"
page:
At first this seems to work for pagination but after further analysis it looks like it's also used to specify which of the 6 pages (Overview, Locations, Traffic Sources, Content, Events and Conversions) to return data for.
For some reason 0 returns an impossibly high metrictotal
limit: Result limit per page, maximum of 50
filters:
Syntax is as specified at the Documentation 2 link below except the OR is specified using | instead of a comma.6==CUSTOM;1==United%20States
You can also combine multiple queries in one request by comma separating them (i.e. q=t:1|2|:1|:10,t:6|:1|:10).
Following the above "documentation", if you wanted to build a query that requests the page URL and city of the top 10 active visitors with a traffic source type of CUSTOM located in the US you would use this URL: https://www.google.com/analytics/realtime/realtime/getData?key={{propertyID}}&pageId&q=t:10|2|:1|:10:6==CUSTOM;1==United%20States
Documentation
Documentation 2
I hope that my answer is readable and (although it's a little late) sufficiently answers your question and helps others in the future.
Cucumber & test data management for non-Ruby apps
I'm testing an app that's basically a message-handling application - a message comes in, it's content is analysed, then it's sent somewhere else for processing. The app isn't built in Ruby. As you might imagine, the main testing approah consists of generating a whole bunch of different types of (quite complex) messages, loading them into the app, waiting a few seconds then ensuring that they get sent to the correct place. Functionally, the testing's going well, but I've currently got all of the test messages & desired message destinations defined in Ruby code - I'd like to move them to either a YAML file, or (second choice) a database. I'd prefer to use a YAML file over a database because it's easier to version control, and for non-technical testers to edit the message content directly. Is there a "recommended" way to implement this sort of data management in Cucumber? It sort of smells like a fixtures approach makes sense, but fixtures to me have always involved populating a database from a file and then using the DB for testing, and I'm not 100% sure this is the best/simplest fit for this particular problem.
I believe what you will be most happy with is a Scenario Outline. You could perhaps create a yaml file an load it from a step, but that would not make a very useful test output. What you (I think) would really like is to see each message and its destination sorted by weather it passed or failed. The example below is for Failed Logins, but it gets the point accross. Scenario Outline: Failed Login Given I am not authenticated When I go to "/login" And I fill in "login" with "<mail>" And I fill in "password" with "<password>" And I press "Log In" Then the login request should fail Then I should see an error message Examples: | mail | password | | not_an_address | nil | | not#not | 123455 | | 123#abc.com | wrong_paasword | Each Example will turn green, red or yellow depending on whether it worked, failed or was pending.