Writing an equivalent to Chrome's onBeforeRequest in a Safari extension - ajax

Chrome extensions have the ability to intercept all web requests to specified URLs using chrome.webRequest.onBeforeRequest. This includes not only static asset requests, but requests for AJAX, PJAX, favicons, and everything in between.
Apple provides a few close approximations to this functionality, such as the beforeLoad (handles images, CSS, and JS) and beforeNavigate (handles full page loads) event handlers, but neither catch AJAX requests. I've tried overloading XMLHttpRequest in an attempt to catch AJAX loads to no avail (I might be doing something wrong). Here's a brief example of how I'm doing this:
var originalOpen = window.XMLHttpRequest.prototype.open;
window.XMLHttpRequest.prototype.open = function(method, url, async, username, password) {
console.log("overriden");
return originalOpen.apply(this, arguments);
}
How can I catch all web requests (AJAX, CSS, JS, etc.) in a Safari extension?

Update: You can check entire code flow on my first Safari Extension I've wrote for TimeCamp tracker: https://github.com/qdevro/timecamp.safariextz
I have succeeded to intercept all AJAX calls (actually the responses were interesting for me, because there all the magic happens), but unfortunately I couldn't find (yet) a solution to send it back to my injected script (I still work on this) now fully working - getting the xhr to the injected script:
I've done it like this:
1) on the injected START script, I've added into the DOM another script (the one which does the interception):
$(document).ready(function(){
var script = document.createElement('script');
script.type = 'text/javascript';
script.src = safari.extension.baseURI + 'path/to/your/script/bellow.js';
document.getElementsByTagName('head')[0].appendChild(script);
})
2) the interception code uses this repository as override of the XMLHttpRequest object, that I've tweaked a little bit as well in order to attach the method, url and sent data to it in order to be easily available when the response get's back.
Basically, I've overriden the open() method of the XMLHttpsRequest to attach those values that I might need in my script, and added the sentData in the send() method as well:
var RealXHROpen = XMLHttpRequest.prototype.open;
...
// Override open method of all XHR requests (inside wire() method
XMLHttpRequest.prototype.open = function(method, url, async, user, pass) {
this.method = method;
this.url = url;
RealXHROpen.apply(this, arguments);
}
...
// Override send method of all XHR requests
XMLHttpRequest.prototype.send = function(sentData) {
...
this.sentData = sentData;
...
}
Then, I've added a callback on the response, which get's a modified XMLHttpRequest object WHEN the data comes back, and cotains everything: url, method, sentData and responseText with the retrieved data:
AjaxInterceptor.addResponseCallback(function(xhr) {
console.debug("response",xhr);
// xhr.method - contains the method used by the call
// xhr.url - contains the URL was sent to
// xhr.sentData - contains all the sent data (if any)
// xhr.responseText - contains the data sent back to the request
// Send XHR object back to injected script using DOM events
var event = new CustomEvent("ajaxResponse", {
detail: xhr
});
window.dispatchEvent(event);
});
AjaxInterceptor.wire();
For sending the XHR object from the intercept script back to the injected script, I just had to use DOM events like #Xan has suggested (thanks for that):
window.addEventListener("ajaxResponse", function(evt) {
console.debug('xhr response', evt.detail);
// do whatever you want with the XHR details ...
}, false);
Some extra hints / (workflow) optimisations that I've used in my project:
I've cleaned the GET url's and moved all the parameters (? &) into the dataSent property;
I've merged this dataSent property if there's the case (in send(data) method)
I've added an identifier on request send (timestamp) in order to match it later (see point bellow and get the idea);
I've sent a custom event to the script called "ajaxRequest" in order to prepare / optimise load times (I had to request some other data to some external API using CORS - by passing the call / response back and forth to the global.html which is capable of handling CORS), so I didn't had to wait for the original request to come back before sending my API call, but just matching the responses based on timestamp above;

Related

Using HtmlUnit, is there a way to pause execution of Javascript, then resume?

In HtmlUnit for testing, I'm coming across a case where, on page load, it'd be useful to NOT execute the Javascript automatically, and instead wait for me to initiate and tell the Javascript to start executing?
My specific use-case is testing something which the Javascript does some tests, and then does a location replace to send the user on to another page. I want to check some headers which I'm returning for testing/validation, and then let the JS execute as usual.
My current thought is to have a flag I pass to the page when testing which will cause the JS to not automatically run, and wait until I call a JS function from within the Java code via webClient.getJavaScriptEngine().execute().
While not specifically being able to pause JavaScript before invoking, it may be worthwhile to use the WebConnectionWrapper class to inspect/modify the response data or outgoing requests, effectively giving you a chance to execute your own code before the JavaScript is invoked.
An example usage of this is as follows:
try (final WebClient webClient = new WebClient()) {
webClient.getOptions().setThrowExceptionOnScriptError(false);
// set more options
// create a WebConnectionWrapper with an (subclassed) getResponse() impl
new WebConnectionWrapper(webClient) {
public WebResponse getResponse(WebRequest request) throws IOException {
WebResponse response = super.getResponse(request);
if (request.getUrl().toExternalForm().contains("my_url")) {
String content = response.getContentAsString();
// intercept and/or change content
WebResponseData data = new WebResponseData(content.getBytes(),
response.getStatusCode(), response.getStatusMessage(), response.getResponseHeaders());
response = new WebResponse(data, request, response.getLoadTime());
}
return response;
}
};
// use the client as usual
HtmlPage page = webClient.getPage(uri);
}
The above code is from the official documentation here:
How to modify the outgoing request or incoming response?
The getResponse() method that you would override is called before each request is made and also allows you to modify the WebResponse object that is passed back to WebClient for its continued processing.
Sorry but at the moment (version 2.43.0) we have no such option. Feel free to open a issue on github for this.
I guess other test tools might also benefit from this function.

webRequest API redirect uses http method of parent request

I'm using the webRequest-API in a WebExtension to monitor requests made by the client. This works quite well, however the API doesn't behave as expected in case of a redirect:
In this case a POST is issued which is answered with a 302 FOUND and a new location. My browser (Firefox 57, other versions and other browsers - e.g. Chrome - act the same way) follows this redirect and now issues a GET to the new location.
Unfortunately the webRequest-API behaves differently: It traces the first POST (which is correct) but than handles the second request as a POST, too, whereas it should be a GET. This is a severe problem since the API traces something my browser supposedly did, which it actually did in another way...
This scenario (the browser-part) can be reproduced by following this link to surfnet.nl and choosing an IDP form the list (e.g. Academisch Medisch Centrum).
So, long story short: Why does the webRequest-API behave different form the way browsers behave? And is there a way to let it exactly trace the browser's actions?
Interestingly the webRequest-API might do it correctly, regarding the documentation:
Even if the specification requires the method, and the body, not to be altered when the redirection is performed, not all user-agents conform here [browsers obviously change the method!], and you can still find buggy software out there. It is therefore recommended to set the 302 code only as a response for GET or HEAD methods and to use 307 Temporary Redirect instead, as the method change is explicitly prohibited in that case.
In the cases where you want the method used to be changed to GET, use 303 See Other instead.
EDIT:
It seems as if browser change the method on a 302 due to historical reasons, even tho it contradicts RFC 2616...
https://stackoverflow.com/a/8139246/594832
https://trac.ietf.org/trac/httpbis/ticket/160
https://stackoverflow.com/a/8138447/594832
Anyways... the question remains: how can I induce the webRequest-API to act the same way?
For anyone interested, I ended up with the following:
The docs let me to this (redirectUrl):
[...] Redirects initiated by a redirect action use the original request method for the redirect, with one exception: If the redirect is initiated at the onHeadersReceived stage, then the redirect will be issued using the GET method. [...]
Although the statement above gave me some hope, the method of that redirected request was still labeled as a POST...
So I updated the code to something roughly like this:
// Keep track of all requests issued so far and their responses
var httpRequests = [];
// Redirected requests come in with their originating parents' ID
browser.webRequest.onBeforeRequest.addListener(request => {
// Check if the request is a redirect
var isRedirected = function(requestId) {
var parentRequest = httpRequests.find(r => r.req.requestId === requestId);
if (parentRequest != null && parentRequest.res != null && parentRequest.res.statusCode === 302) {
return true;
}
return false;
};
// The webRequest-API seems to keep the HTTP verbs which is correct in resepct to RFC 2616 but
// differs from a typical browser behaviour which will usually change the POST to a GET. So do we here...
if (request.method === 'POST' && isRedirected(request.requestId)) {
console.log(`Redirected 302-request '${request.requestId}' is a POST but is here changed to a GET to conform to browser behaviour...`);
request.method = 'GET';
}
// Store the request
var entry = {
id: id,
req: request
};
httpRequests.push(entry);
});
browser.webRequest.onHeadersReceived.addListener(response => {
// Store the response alongside the request
var entry = httpRequests.find(req => req.id === response.requestId);
entry.res = response;
// If it turns out that a request should be redirected...
if (response.statusCode === 302) {
var location = response.responseHeaders.find(header => header.name.toLowerCase() === "location");
console.log(`Redirecting request '${id}' to new location '${location.value}'...`);
// The new location is set as redirectUrl, which results in a new invocation of onBeforeRequest with
// the current requestId
return {
redirectUrl: location.value
};
}
});
What happens?
Say onBeforeRequest receives a new request 2640 which is a POST.
onHeadersReceived gets the response which says that this request should be redirected to a new location (302).
The code above does so by setting the redirectUrl to that new location.
Then onBeforeRequest is triggered again. The webRequest-API passes the same requestId to it (2640).
The code checks if there's a parent for this request, and in this case it's true.
The method (which is still POST) is then modified to GET.
For the request lifecycle have a look at the illustration in the docs.

Compression response filter fails on breeze.js Metadata call

I have an http module where I'm adding a response filter below for compression. This works for all API calls except for 1, the call to MetaData. If I remove the [BreezeController] decoration it works fine. I think it has to do with action filter attribute that converts the string return type into an HttpResponse return type with string content.
The error I'm getting is " Exception message: The stream state of the underlying compression routine is inconsistent."
I've done some testing where a method thats defined to return an HttpResponse works fine. So I think its the scenario where the method is defined to return string, and then the action filter changes it to HttpResponse at runtime.
Any ideas how I can get this to work?
Here's the response filter being added in BeginRequest:
HttpApplication app = (HttpApplication)sender;
// Check the header to see if it can accept compressed output
string encodings = app.Request.Headers.Get("Accept-Encoding");
if (encodings == null)
return;
Stream s = app.Response.Filter;
encodings = encodings.ToLower();
if (encodings.Contains("gzip"))
{
app.Response.Filter = new GZipStream(s, CompressionMode.Compress);
app.Response.AppendHeader("Content-Encoding", "gzip");
}
Don't know the specifics of what you're doing but I know that the [BreezeController] attribute strips out filters and adds back just the ones that breeze wants.
One approach might be to define a separate controller (ModelMetadataController) that only serves the metadata. This controller doesn't have the [BreezeController] attribute; it's a plain old Web API controller.
Then you create a "Breeze controller" (ModelController) with all of the usual methods except the Metadata method.
You call the metadata controller from the client during app launch via MetadataStore.fetchMetadata just to get metadata.
Once you have populated a metadataStore in this fashion, you use it in your EntityManager which sends query and save requests to the "real" Web API data controller.
The client code might look something like this:
var ds = new breeze.DataService({
serviceName: 'breeze/Model' // the breeze query & save controller
});
var ms = new MetadataStore({
namingConvention: breeze.NamingConvention.camelCase, // assuming that's what you want
});
ms.addDataService(ds); // associate the metadata-to-come with the "real" dataService
var manager = new breeze.EntityManager({
dataService: ds,
metadataStore: ms
});
// the fun bit: fetch the metadata from a different controller
var promise = ms.fetchMetadata('breeze/ModelMetadata') // the metadata-only controller!
return promise; // wait on it appropriately

GWT FormPanel method replaced by input parameter

We've discovered a strange new bug in a GWT application I'm maintaining, and I'm not sure when it became an issue. Possibly with a new Firefox version.
We're sending a POST request to the server using a FormPanel, essentially like many examples I've seen online. But since we actually want a PUT request, one of the hidden input parameters is named "method" and has a value of "put".
Now, when I look at the request in Fiddler coming from Firefox, it is being transformed into a GET request with all the parameters in the QueryString. In IE and Chrome, the parameters are in the body of a POST request.
I've displayed the value of FormPanel.getMethod() in an alert, and in IE and Chrome the string "post" is displayed, whereas in firefox it is showing "object HTMLInputElement". Unfortunately, hosted mode debugging does not work with this project.
It obviously looks like the FormPanel's getMethod() function is returning the hidden input parameter named method instead of the actual form's method in Firefox.
Technically I should avoid changing the servlet as this is from an OpenSource project that we use, though I've found I can fix the issue by changing the hidden input parameter's name to "_method" on both ends.
Has anyone ever seen anything like this? I can't find anything in Google.
UPDATE: We're using GWT 2.3 in case that helps
Some insight can be found here Are the PUT, DELETE, HEAD, etc methods available in most web browsers?
I would also suggest using XMLHttpRequest. In this case you [most probably] don't have to change anything on the server side.
In case if you use Submit button, you can write in its clickHandler function:
submitMyForm(yourTextBox.getText(), self);
// self - is the instance of main class (named UploadForm here), needs to be passed here for future reference
and then some more (you can adapt this for your needs):
private native void submitMyForm(String text, UploadForm handler)/*-{
var fd = new FormData();
fd.append("textValue", text);
var xhr = new XMLHttpRequest();
var upload = xhr.upload;
readyStateChangeHandler = function () {
if (xhr.readyState == 4) {
if (xhr.status == 200) {
var serverResponse = eval(xhr.responseText); // optional
handler.#com.project.UploadForm::onUploadIsDone(Lcom/google/gwt/core/client/JavaScriptObject;)(serverResponse);
} else {
handler.#com.project.UploadForm::onUploadFailed(I)(status);
}
}
};
xhr.onreadystatechange = readyStateChangeHandler;
xhr.open("PUT", yourActionUrlHere);
xhr.send(formData);
}-*/;

can't seem to get progress events from node-formidable to send to the correct client over socket.io

So I'm building a multipart form uploader over ajax on node.js, and sending progress events back to the client over socket.io to show the status of their upload. Everything works just fine until I have multiple clients trying to upload at the same time. Originally what would happen is while one upload is going, when a second one starts up it begins receiving progress events from both of the forms being parsed. The original form does not get affected and it only receives progress updates for itself. I tried creating a new formidable form object and storing it in an array along with the socket's session id to try to fix this, but now the first form stops receiving events while the second form gets processed. Here is my server code:
var http = require('http'),
formidable = require('formidable'),
fs = require('fs'),
io = require('socket.io'),
mime = require('mime'),
forms = {};
var server = http.createServer(function (req, res) {
if (req.url.split("?")[0] == "/upload") {
console.log("hit upload");
if (req.method.toLowerCase() === 'post') {
socket_id = req.url.split("sid=")[1];
forms[socket_id] = new formidable.IncomingForm();
form = forms[socket_id];
form.addListener('progress', function (bytesReceived, bytesExpected) {
progress = (bytesReceived / bytesExpected * 100).toFixed(0);
socket.sockets.socket(socket_id).send(progress);
});
form.parse(req, function (err, fields, files) {
file_name = escape(files.upload.name);
fs.writeFile(file_name, files.upload, 'utf8', function (err) {
if (err) throw err;
console.log(file_name);
})
});
}
}
});
var socket = io.listen(server);
server.listen(8000);
If anyone could be any help on this I would greatly appreciate it. I've been banging my head against my desk for a few days trying to figure this one out, and would really just like to get this solved so that I can move on. Thank you so much in advance!
Can you try putting console.log(socket_id);
after form = forms[socket_id]; and
after progress = (bytesReceived / bytesExpected * 100).toFixed(0);, please?
I get the feeling that you might have to wrap that socket_id in a closure, like this:
form.addListener(
'progress',
(function(socket_id) {
return function (bytesReceived, bytesExpected) {
progress = (bytesReceived / bytesExpected * 100).toFixed(0);
socket.sockets.socket(socket_id).send(progress);
};
})(socket_id)
);
The problem is that you aren't declaring socket_id and form with var, so they're actually global.socket_id and global.form rather than local variables of your request handler. Consequently, separate requests step over each other since the callbacks are referring to the globals rather than being proper closures.
rdrey's solution works because it bypasses that problem (though only for socket_id; if you were to change the code in such a way that one of the callbacks referenced form you'd get in trouble). Normally you only need to use his technique if the variable in question is something that changes in the course of executing the outer function (e.g. if you're creating closures within a loop).

Resources