I would like to write a little component with XPCOM that can sniff all HTTP responses received by the browser. Right now the only examples that I can find (like the one appended below) only allow me to retrieve the response for a request that I fire myself:
var req = new XMLHttpRequest();
req.open('GET', 'http://www.mozilla.org/', true);
req.onreadystatechange = function (aEvt) {
if (req.readyState == 4) {
if(req.status == 200)
dump(req.responseText);
else
dump("Error loading page\n");
}
};
What I want is for any HTTP response that the browser receives get the HTTP headers of the corresponding request.
Thanks
You can also use the http-on-modify-request and http-on-examine-response notifications. See https://developer.mozilla.org/en/XUL_School/Intercepting_Page_Loads#HTTP_Observers
You can sniff all http traffic via the nsIHttpActivityObserver, an example cribbed from the Firefox web console:
const Cc = Components.classes;
Cu.import("resource://gre/modules/XPCOMUtils.jsm");
XPCOMUtils.defineLazyServiceGetter(this, "activityDistributor",
"#mozilla.org/network/http-activity-distributor;1",
"nsIHttpActivityDistributor");
let httpTrafficObserver = {
/**
* Begin observing HTTP traffic that we care about,
* namely traffic that originates inside any context that a Heads Up Display
* is active for.
*/
startHTTPObservation: function httpObserverFactory()
{
// creates an observer for http traffic
var self = this;
var httpObserver = {
observeActivity :
function observeActivity(aChannel,
aActivityType,
aActivitySubtype,
aTimestamp,
aExtraSizeData,
aExtraStringData)
{
if (aActivityType ==
activityDistributor.ACTIVITY_TYPE_HTTP_TRANSACTION ||
aActivityType ==
activityDistributor.ACTIVITY_TYPE_SOCKET_TRANSPORT) {
aChannel = aChannel.QueryInterface(Ci.nsIHttpChannel);
let transCodes = this.httpTransactionCodes;
if (aActivitySubtype ==
activityDistributor.ACTIVITY_SUBTYPE_REQUEST_HEADER ) {
let httpActivity = {
url: aChannel.URI.spec,
method: aChannel.requestMethod,
channel: aChannel
};
}
}
},
httpTransactionCodes: {
0x5001: "REQUEST_HEADER",
0x5002: "REQUEST_BODY_SENT",
0x5003: "RESPONSE_START",
0x5004: "RESPONSE_HEADER",
0x5005: "RESPONSE_COMPLETE",
0x5006: "TRANSACTION_CLOSE",
0x804b0003: "STATUS_RESOLVING",
0x804b0007: "STATUS_CONNECTING_TO",
0x804b0004: "STATUS_CONNECTED_TO",
0x804b0005: "STATUS_SENDING_TO",
0x804b000a: "STATUS_WAITING_FOR",
0x804b0006: "STATUS_RECEIVING_FROM"
}
};
this.httpObserver = httpObserver;
activityDistributor.addObserver(httpObserver);
}
};
and http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsIHttpActivityObserver.idl
Related
I created a script in Google Sheets, which is working well but after a while I'm getting the following error:
Exception: Service invoked too many times for one day: urlfetch
I think I called the function like 200-300 times in the day, for what I checked it should be below the limit.
I read we can use cache to avoid this issue but not sure how to use it in my code.
function scrapercache(url) {
var result = [];
var description;
var options = {
'muteHttpExceptions': true,
'followRedirects': false,
};
var cache = CacheService.getScriptCache();
var properties = PropertiesService.getScriptProperties();
try {
let res = cache.get(url);
if (!res) {
// trim url to prevent (rare) errors
url.toString().trim();
var r = UrlFetchApp.fetch(url, options);
var c = r.getResponseCode();
// check for meta refresh if 200 ok
if (c == 200) {
var html = r.getContentText();
cache.put(url, "cached", 21600);
properties.setProperty(url, html);
var $ = Cheerio.load(html); // make sure this lib is added to your project!
// meta description
if ($('meta[name=description]').attr("content")) {
description = $('meta[name=description]').attr("content").trim();
}
}
result.push([description]);
}
}
catch (error) {
result.push(error.toString());
}
finally {
return result;
}
}
how can I use cache like this to enhance my script please?
var cache = CacheService.getScriptCache();
var result = cache.get(url);
if(!result) {
var response = UrlFetchApp.fetch(url);
result = response.getContentText();
cache.put(url, result, 21600);
Answer:
You can implement CacheService and PropertiesService together and only retrieve the URL again after a specified amount of time.
Code Change:
Be aware that additional calls to retrieving the cache and properties will slow your function down, especially if you are doing this a few hundred times.
As the values of the cache can be a maximum of 100 KB, we will use CacheService to keep track of which URLs are to be retrieved, but PropertiesService to store the data.
You can edit your try block as so:
var cache = CacheService.getScriptCache();
var properties = PropertiesService.getScriptProperties();
try {
let res = cache.get(url);
if (!res) {
// trim url to prevent (rare) errors
url.toString().trim();
var r = UrlFetchApp.fetch(url, options);
var c = r.getResponseCode();
// check for meta refresh if 200 ok
if (c == 200) {
var html = r.getContentText();
cache.put(url, "cached", 21600);
properties.setProperty(url, html);
var $ = Cheerio.load(html); // make sure this lib is added to your project!
// meta description
if ($('meta[name=description]').attr("content")) {
description = $('meta[name=description]').attr("content").trim();
}
}
result.push([description]);
}
}
catch (error) {
result.push(error.toString());
}
finally {
return result;
}
References:
Class CacheService | Apps Script | Google Developers
Class Cache | Apps Script | Google Developers
Class PropertiesService | Apps Script | Google Developers
Related Questions:
Service invoked too many times for one day: urlfetch
My custom v3 CAF receiver app is successfully playing the first few live & vod assets. After that, it gets into a state were media commands are being queued because "Load is in progress". It is still (successfully) fetching manifests, but MEDIA_STATUS remains "buffering". The log then shows:
[ 4.537s] [cast.receiver.MediaManager] Load is in progress, media command is being queued.
[ 5.893s] [cast.receiver.MediaManager] Buffering state changed, isPlayerBuffering: true old time: 0 current time: 0
[ 5.897s] [cast.receiver.MediaManager] Sending broadcast status message
CastContext Core event: {"type":"MEDIA_STATUS","mediaStatus":{"mediaSessionId":1,"playbackRate":1,"playerState":"BUFFERING","currentTime":0,"supportedMediaCommands":12303,"volume":{"level":1,"muted":false},"currentItemId":1,"repeatMode":"REPEAT_OFF","liveSeekableRange":{"start":0,"end":20.000999927520752,"isMovingWindow":true,"isLiveDone":false}}}
CastContext MEDIA_STATUS event: {"type":"MEDIA_STATUS","mediaStatus":{"mediaSessionId":1,"playbackRate":1,"playerState":"BUFFERING","currentTime":0,"supportedMediaCommands":12303,"volume":{"level":1,"muted":false},"currentItemId":1,"repeatMode":"REPEAT_OFF","liveSeekableRange":{"start":0,"end":20.000999927520752,"isMovingWindow":true,"isLiveDone":false}}}
Fetch finished loading: GET "(manifest url)".
No errors are shown.
Even after closing and restarting the cast session, the issue remains. The cast device itself has to be rebooted to resolve it. It looks like data is kept between sessions.
It could be important to note that the cast receiver app is not published yet. It is hosted on a local network.
My questions are:
What could be the cause of this stuck behavior?
Is there any session data kept between session?
How to fully reset the cast receiver app, without the necessity to restart the cast device.
The receiver app itself is very basic. Other than license wrapping it resembles the vanilla example app:
const { cast } = window;
const TAG = "CastContext";
class CastStore {
static instance = null;
error = observable.box();
framerate = observable.box();
static getInstance() {
if (!CastStore.instance) {
CastStore.instance = new CastStore();
}
return CastStore.instance;
}
get debugLog() {
return this.framerate.get();
}
get errorLog() {
return this.error.get();
}
init() {
const context = cast.framework.CastReceiverContext.getInstance();
const playerManager = context.getPlayerManager();
playerManager.addEventListener(
cast.framework.events.category.CORE,
event => {
console.log(TAG, "Core event: " + JSON.stringify(event));
}
);
playerManager.addEventListener(
cast.framework.events.EventType.MEDIA_STATUS,
event => {
console.log(TAG, "MEDIA_STATUS event: " + JSON.stringify(event));
}
);
playerManager.addEventListener(
cast.framework.events.EventType.BITRATE_CHANGED,
event => {
console.log(TAG, "BITRATE_CHANGED event: " + JSON.stringify(event));
runInAction(() => {
this.framerate.set(`bitrate: ${event.totalBitrate}`);
});
}
);
playerManager.addEventListener(
cast.framework.events.EventType.ERROR,
event => {
console.log(TAG, "ERROR event: " + JSON.stringify(event));
runInAction(() => {
this.error.set(`Error detailedErrorCode: ${event.detailedErrorCode}`);
});
}
);
// intercept the LOAD request to be able to read in a contentId and get data.
this.loadHandler = new LoadHandler();
playerManager.setMessageInterceptor(
cast.framework.messages.MessageType.LOAD,
loadRequestData => {
this.framerate.set(null);
this.error.set(null);
console.log(TAG, "LOAD message: " + JSON.stringify(loadRequestData));
if (!loadRequestData.media) {
const error = new cast.framework.messages.ErrorData(
cast.framework.messages.ErrorType.LOAD_CANCELLED
);
error.reason = cast.framework.messages.ErrorReason.INVALID_PARAM;
return error;
}
if (!loadRequestData.media.entity) {
// Copy the value from contentId for legacy reasons if needed
loadRequestData.media.entity = loadRequestData.media.contentId;
}
// notify loadMedia
this.loadHandler.onLoadMedia(loadRequestData, playerManager);
return loadRequestData;
}
);
const playbackConfig = new cast.framework.PlaybackConfig();
// intercept license requests & responses
playbackConfig.licenseRequestHandler = requestInfo => {
const challenge = requestInfo.content;
const { castToken } = this.loadHandler;
const wrappedRequest = DrmLicenseHelper.wrapLicenseRequest(
challenge,
castToken
);
requestInfo.content = wrappedRequest;
return requestInfo;
};
playbackConfig.licenseHandler = license => {
const unwrappedLicense = DrmLicenseHelper.unwrapLicenseResponse(license);
return unwrappedLicense;
};
// Duration of buffered media in seconds to start/resume playback after auto-paused due to buffering; default is 10.
playbackConfig.autoResumeDuration = 4;
// Minimum number of buffered segments to start/resume playback.
playbackConfig.initialBandwidth = 1200000;
context.start({
touchScreenOptimizedApp: true,
playbackConfig: playbackConfig,
supportedCommands: cast.framework.messages.Command.ALL_BASIC_MEDIA
});
}
}
The LoadHandler optionally adds a proxy (I'm using a cors-anywhere proxy to remove the origin header), and stores the castToken for licenseRequests:
class LoadHandler {
CORS_USE_PROXY = true;
CORS_PROXY = "http://192.168.0.127:8003";
castToken = null;
onLoadMedia(loadRequestData, playerManager) {
if (!loadRequestData) {
return;
}
const { media } = loadRequestData;
// disable cors for local testing
if (this.CORS_USE_PROXY) {
media.contentId = `${this.CORS_PROXY}/${media.contentId}`;
}
const { customData } = media;
if (customData) {
const { licenseUrl, castToken } = customData;
// install cast token
this.castToken = castToken;
// handle license URL
if (licenseUrl) {
const playbackConfig = playerManager.getPlaybackConfig();
playbackConfig.licenseUrl = licenseUrl;
const { contentType } = loadRequestData.media;
// Dash: "application/dash+xml"
playbackConfig.protectionSystem = cast.framework.ContentProtection.WIDEVINE;
// disable cors for local testing
if (this.CORS_USE_PROXY) {
playbackConfig.licenseUrl = `${this.CORS_PROXY}/${licenseUrl}`;
}
}
}
}
}
The DrmHelper wraps the license request to add the castToken and base64-encodes the whole. The license response is base64-decoded and unwrapped lateron:
export default class DrmLicenseHelper {
static wrapLicenseRequest(challenge, castToken) {
const wrapped = {};
wrapped.AuthToken = castToken;
wrapped.Payload = fromByteArray(new Uint8Array(challenge));
const wrappedJson = JSON.stringify(wrapped);
const wrappedLicenseRequest = fromByteArray(
new TextEncoder().encode(wrappedJson)
);
return wrappedLicenseRequest;
}
static unwrapLicenseResponse(license) {
try {
const responseString = String.fromCharCode.apply(String, license);
const responseJson = JSON.parse(responseString);
const rawLicenseBase64 = responseJson.license;
const decodedLicense = toByteArray(rawLicenseBase64);
return decodedLicense;
} catch (e) {
return license;
}
}
}
The handler for cast.framework.messages.MessageType.LOAD should always return:
the (possibly modified) loadRequestData, or
a promise for the (possibly modified) loadRequestData
null to discard the load request (I'm not 100% sure this works for load requests)
If you do not do this, the load request stays in the queue and any new request is queued after the initial one.
In your handler, you return an error if !loadRequestData.media, which will get you into that state. Another possibility is an exception in the load request handler, which will also get you in that state.
I guess we have a different approach and send everything possible through sendMessage, when we loading stuff we create a new cast.framework.messages.LoadRequestData() which we load with playerManager.load(loadRequest).
But I guess that you might be testing this on an integrated Chromecast, we see this problems as well!?
I suggest that you do one or more
Enable gzip compression on all responses!!!
Stop playback playerManager.stop() (maybe in the interseptor?)
Change how the licenseUrl is set
How we set licenseUrl
playerManager.setMediaPlaybackInfoHandler((loadRequestData, playbackConfig) => {
playbackConfig.licenseUrl = loadRequestData.customData.licenseUrl;
return playbackConfig;
}
);
i am using ajax and jquery to send a synchronous http request.
I have to use synchronous request because i want to return some value from ajax function after evaluating result of the server side script.
i know synchronous request will freeze the browser but due to my requirement i will have to do this request.
i also knows that in synchronous request there is no use of onreadystatechange function we should evaluate our result after sending the request or below the send function.By doing this my code is in working state.
But, my problem is that when i used onreadystatechange function it is working in firefox >=4 version but not working in below firefox 4 version.
Please. help me in finding out whether problem is with the code or the browser.
i am not able to find out the bug now i am helpless...plz help
here is my code
function test(txt_obj) {
var keywords = txt_obj.value;
var SHttpRequestObject = false;
var url = "/speller/server-scripts/ifmisspelled_words.html" + '?keywords=' + keywords);
var speller_res = 0;
if (window.XMLHttpRequest){
SHttpRequestObject = new XMLHttpRequest();
} else if (window.ActiveXObject){
SHttpRequestObject = new ActiveXObject("Microsoft.XMLHTTP");
}
SHttpRequestObject.open("POST", url, false);
if (SHttpRequestObject){
SHttpRequestObject.onreadystatechange = function() {
if (SHttpRequestObject.readyState == 4 && SHttpRequestObject.status == 200)
{
var result = eval("(" + SHttpRequestObject.responseText + ")");
if(result.error) {
speller_res = 1;
} else if(result.word_exist) {
speller_res = 1;
}
else if(result.word_not_exist) {
speller_res = 0;
}
}
};
}
SHttpRequestObject.send(null);
return speller_res;
}
From MDN:
onreadystatechange
Warning: This must not be used from native code. You should also not
use this with synchronous requests.
I am using an observer on "http-on-modify-request" to analyze HTTP requests (and responses with the corresponding other observers).
Is it possible to determine whether the HTTP request / response is the main frame loading (the actual page DOM)? As opposed to another resource (image, css, sub_frame, etc.).
The docs have most of the answer you're looking for here and I've modified it below for use with the addon-sdk.
You can watch for an IFRAME by comparing the location with the top.document location.
I don't think there's an easy way to detect loading of images, etc so you'll probably want to just watch for the first hit that's not an IFRAME and regard everything else as css/image/script content loading.
var chrome = require("chrome");
var httpmods = {
observe : function(aSubject, aTopic, aData) {
console.log("observer", aSubject, aTopic, aData);
aSubject.QueryInterface(chrome.Ci.nsIHttpChannel);
var url = aSubject.URI.spec;
var dom = this.getBrowserFromChannel(aSubject);
if (dom) {
if (dom.top.document && dom.location === dom.top.document.location) {
console.log("ISN'T IFRAME");
} else {
console.log("IS IFRAME");
}
}
},
getBrowserFromChannel: function (aChannel) {
try {
var notificationCallbacks =
aChannel.notificationCallbacks ? aChannel.notificationCallbacks : aChannel.loadGroup.notificationCallbacks;
if (!notificationCallbacks)
return null;
var domWin = notificationCallbacks.getInterface(chrome.Ci.nsIDOMWindow);
return domWin;
}
catch (e) {
dump(e + "\n");
return null;
}
}
}
require("observer-service").add("http-on-modify-request", httpmods.observe, httpmods);
for (var i = 0; i < 5; ++i) {
var xhr;
if (window.XMLHttpRequest) {
xhr = new XMLHttpRequest();
} else if (window.ActiveXObject) {
xhr = new ActiveXObject("Msxml2.XMLHTTP");
}
xhr.open('GET', '/Test/LongOperation?p=' + new Date());
xhr.send('');
}
This is only a demo (not live code) but it illustrates the core problem.
LongOperation is a method that returns a result after 10 seconds.
Questions:
Why does IE8 (and maybe other IEs) hang when the user tries to navigate away from page right after the above code snippet has been executed? FireFox/Safari cancel these requests and allow navigation to another page. If you replace 'i < 5' with 'i < 4' then IE would not hang.
How to work around this ugly IE behavior? Users are very upset when their browser suddenly hangs.
Most browsers have an inbuilt limit of 4 connections to any given server. One way to work around this "problem" might be to use a different hostname for out of band XML requests - your user requests will go to the main hosts, while the AJAX requests can go to the second server.
My answer to my question. I abort all not completed xhr objects in window.onbeforeunload. At least this solution works for me. I slightly override $.ajax() method behavior:
;(function($) {
var rq = [];
var ajax = $.ajax;
$.ajax = function(settings) {
// override complete() operation
var complete = settings.complete;
settings.complete = function(xhr) {
if (xhr) {
// xhr may be undefined, for example when downloading JavaScript
for (var i = 0, len = rq.length; i < len; ++i) {
if (rq[i] == xhr) {
// drop completed xhr from list
rq.splice(i, 1);
break;
}
}
}
// execute base
if (complete) {
complete.apply(this, arguments)
}
}
var r = ajax.apply(this, arguments);
if (r) {
// r may be undefined, for example when downloading JavaScript
rq.push(r);
}
return r;
};
// 'kill' all pending xhrs
$(window).bind('beforeunload', function() {
$.each(rq, function(i, xhr) {
try {
xhr.abort();
} catch(e) {
$debug.fail('failed to abort xhr');
}
});
rq = [];
});
})(jQuery);
$debug - my utility class
Try running them asynchronously and then triggering the next http request when the each completes. I suspect that the xmlhttp request is blocking the UI thread of IE whereas the implementations of that on other browsers is a little more graceful.
Hopefully that will give you a work-around for question 2 but I can only guess at the true reason for question 1, it could just be a bug.