According to the Crossrider docs, there is currently a webRequest object with the onRequest event, which allows for accessing the request URI, etc., but I could not find any way to inspect the response data. I was wondering if there is a way to actually inspect the response of that request, such as the response headers and possibly the response data. Thanks for the help.
I'm going out on a limb here but I think your after an AJAX request, in which case you are after the appAPI.request API where you can indeed obtain response and header data. The appAPI.webRequest API is like Chrome's webRequest API used "to observe and analyze traffic and to intercept, block, or modify requests in-flight".
If this is not the case, please can you clarify your scenario.
[RESPONSE TO COMMENT]
Let me first clarify that the webRequest API runs before the requests are made (i.e. the browser is requesting the page but has not yet made the actual HTTP request) and hence it's not relevant to talk of response data.
However, since the URL is provided in the details of the webRequest, you can get the response data and headers yourself using the aforementioned appAPI.request API, as follows:
appAPI.ready(function() {
appAPI.webRequest.monitor.onBeforeNavigate.addListener({
callback: function(details) {
appAPI.request.get({
url: details.requestUrl,
onSuccess: function(response, additionalInfo) {
appAPI.db.async.set(details.requestUrl, {
response: response,
headers: additionalInfo.headers
});
}
});
}
});
});
NOTE: For this example I have used the new webRequest.monitor API designed to be lightweight specifically for monitoring. It's not currently documented (docs should be completed within the next week or so) but you can start using it already for Chrome, Firefox, and Internet Explorer.
[Disclosure: I am a Crossrider employee]
Related
How can network requests be monitored and evaluated for their request URL, parameters, request, and response data?
Desired solution
I want to be notified or being given a custom exception, if specific content occurs in request or response.
Example
Assume a web application with many dynamic Ajax requests. A request or response might contain a broken value, e.g. undefined.
Request URL:
http://localhost:8080/app/?undefined=1
Response JSON data:
{"undefined":"1"}
Attempts
Filtering for request content in Dev Tools is not possible
PostMan tests seem not viable (e.g. no user interactions)
Not tried/found yet
Guesses of what might work ...
Software to intercepts requests and log/alert details
Proxying any URLs on an OS via some standalone application
If importing axios, you could leverage custom interceptors:
// Add a response interceptor
axios.interceptors.response.use(function (response) {
// Any status code that lie within the range of 2xx cause this function to trigger
// Do something with response data
return response;
}, function (error) {
// Any status codes that falls outside the range of 2xx cause this function to trigger
// Do something with response error
return Promise.reject(error);
});
// Similar functionality can be achieved for request
axios.interceptors.request.use(function (config) {
// Do something before request is sent
return config;
}, function (error) {
// Do something with request error
return Promise.reject(error);
});
Proxy software
For this case, any proxy that comes with logging ability, or can be extended:
https://mitmproxy.org
https://github.com/http-party/node-http-proxy
Exemplary for many variants under NodeJS
Tunneling
Expose localhost to a domain for testing:
https://ngrok.com
Use the logging capability to get request details:
https://ngrok.com/docs/ngrok-agent/api#list-requests
Localhost vs. 127.0.0.1
localhost should better become 127.0.0.1 or a custom dev domain. On MacOS, localhost did not work with HTTP interceptor browser plugins.
Question Overview:
I am accessing a list of files stored in my AWS S3 bucket through a CORS request of presigned files. This basically works fine. However, the objects have some custom METADATA attached to them, which I can't access. I understood, that I can access this metadata only when I add the header key (e.g. "x-amz-meta-1234", where 1234 is the key of my metadata) to the Expose-Headers of the target-bucket's CORS config. While this works so far for me, I can't set the expose-header with a wildcard (e.g. "x-amz-meta-*"), which would solve my problem, but AWS doesn't support wildcards for the expose-header entries.
However, when I look in the NETWORK tab of my Chrome Dev Tools, all desired metadata is showing up in the headers during the GET/HEAD request (note the entries on the lower part, x-amz-meta-4021 and -template_id):
This is my HEAD call:
$.ajax({
url: url,
dataType: 'json',
crossDomain: true,
type: 'HEAD',
success: function(data, status, jqXHR) {
console.log('got some response ..?');
console.log(data);
console.log(jqXHR);
console.log('responseHeader template_id: ' + jqXHR.getResponseHeader('x-amz-meta-template_id'));
console.log('responseHeader meta-4021: ' + jqXHR.getResponseHeader('x-amz-meta-4021'));
console.log(jqXHR.getAllResponseHeaders());
},
error: function(error, xhr, data) {
console.log('in error..');
console.log(error);
console.log(xhr);
console.log(data);
}
});
});
And this is the console output:
Object {readyState: 4, getResponseHeader: function, getAllResponseHeaders:
function, setRequestHeader: function, overrideMimeType: function…}
responseHeader template_id: 813
responseHeader meta-4021: null
x-amz-meta-template_id: 813
Last-Modified: Fri, 09 Jun 2017 13:05:33 GMT
Content-Type: video/mp4
I set expose-header for the metadata-entry 'template_id' explicitly and therefore the header-data is returned correctly for this entry. However, for the entry '4021' I didn't set the expose-header. The problem is, that this metadata (and the keys) are produced by our (android/ios) apps, and I can't really control the keys of that metadata that easily.
Whats puzzling me: why am I able to see the whole response in the chrome network tab, but can't access this data from a client-side script? There are many possible workarounds and solutions, but I basically want to understand, why my browser can display me data, which can't be accessed by jQuery.
PS: in case you want to see the CORS config or the full script, please let me know. I tried to be as precise as possible. Thanks in advance!
I basically want to understand, why my browser can display me data, which can't be accessed by jQuery.
To understand this, you need to understand the purpose of CORS.
CORS isn't really about access control, and CORS isn't really working on your site's behalf. CORS is working on behalf of the user and the browser, to prevent the browser from becoming a confused deputy and doing something the user would not have wanted. This usually coincides with something the site would also not have wanted, but that's secondary.
The browser's default behavior is to assume that programmatic access to cross-origin requests is bad, which is why they are denied when no Access-Control-Allow-Origin header is present. Your bank would not want internetbadguys.com to make ajax requests to the bank web site, and if that site tried, the browser would block it unless the bank's web server foolishly allowed it with a CORS response.
CORS is a mechanism for your site to tell the brower, "yes, the cross-origin request you are making is not unexpected, it's allowed... and from this response, the browser is allowed to engage in certain behaviors, such as exposing the following response headers to the code making the request."
In that light, the behavior you observe is correct. Exposing headers (or not) doesn't mean include them in the HTTP response (or not) -- exposing headers gives the browser permission to expose what it knows to the ajax caller. If the cross-origin origin wants them exposed, it has to be explicit.
I've been trying to fix this all day.
I got a Lambda function I'm trying to run when I make a POST request to the endpoint (configured with AWS API GATEWAY).
Everything runs fine, my options gets a 200 okay, my post gets a 200 okay and comes back with the data I want.
However when I get it back, it "thinks" it failed. I use JQuery for this project, and it comes back as the "fail" cb.
Looking at the console log the problem is quite clearly the following error:
XMLHttpRequest cannot load
https://xxxxxxxxx.execute-api.us-west-2.amazonaws.com/prod/createCustomer.
No 'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'http://localhost:3000' is therefore not allowed
access.
a standard CORS issue. But nothing I do seems to work. I enabled CORS on the API gateway (before that the OPTIONS would fail). the 'Access-Control-Allow-Origin': is set to: '*' which should allow access from any origin. I of course deployed the new API.
I also tried not using CORS and instead enabling the AWS proxy thing, and having my response like this { statusCode, headers, body } but when doing that I get no response whatsoever.
I have absolutely no clue where to start attempting to fix this. I even tried a dirty jsonp method. But still same problem. :(
The only bit of data I can find on my request that doesn't seem okay, is this: x-cache:Miss from cloudfront
Here's the full response headers from the POST request:
content-length:2312
content-type:application/json
date:Thu, 23 Mar 2017 22:15:08 GMT
status:200
via:1.1 95a477af435073615179b256d8101334.cloudfront.net (CloudFront)
x-amz-cf-id:Hc6POYFO0HKB1xriSg2iH7O1po7ah926a4dQkgfSNBUZ460RoHRNuw==
x-amzn-requestid:2b5ed745-1016-11e7-b497-cb0a77cd1479
x-amzn-trace-id:Root=1-58d448ea-56717776eaa3f5389083e9ca
x-cache:Miss from cloudfront
Sadly as is quite obvious, the Access control headers aren't there... I assume that's the problem I'm desperately trying to fix. But I have no idea why since I set it to ' * ' during the Enable CORS step of this process.
I was able to get semi-close to a solution by manually creating a POST method (as opposed to "ANY"), then enabling CORS again, then turning on the PROXY on integration response, and set my response to: { statusCode, headers, body } This will come back as a successfull event and run the right callback function in my ajax call. However, using this method I get no data back from Lambda... despite having it in "body", all I receive is an empty object.
I don't really have a preference of using CORS or manually adding headers, that's fine I won't be updating this much. I just really really need to get it functioning :( Any help would be greatly appreciated.
If you got x-cache:Miss from cloudfront from your API, it is fine because API Gateway doesn't enable the edge cache on the cloudfront side. If you want CORS to work with Lambda proxy integration, you can return the status code and the headers, like,
{
"statusCode": 200,
"headers": { "Access-Control-Allow-Origin": "<domains you need>" }
}
Then, it will be like an empty response from the client side.
I have now idea why, but I was able to get it working. Using the following steps:
Enable CORS,
Deploy API, (can probably be skipped...)
Manually add POST method,
Enable Proxy on Integration response,
Deploy API again
change callback response to format: { statusCode: 200, headers: {}, body: {} }
use JSON.parse() to parse your payload.
This is definitely not a perfect answer. For some reason turning on the Proxy modified how my Lambda function received data, so I had to Stringify it to not cause an error. I can't offer an explanation why this happened.
The headers I used were:
"Access-Control-Allow-Methods": "DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT",
"Access-Control-Allow-Origin": "*"
A better solution to this would also be a way to do this just using CORS. As you don't have to manually insert headers in all your responses. But I couldn't get that working. So please, if someone knows a better solution to this post it!
I've an API endpoint hosted (built via Django Rest Framework), for eg:- domain.com/api/fetch_all?start=0&end=50. This fetches all the results from the database in a pagination manner.
Now I'm representing this information on a webpage. Its more or less like an open forum where everyone can read the data, but only some can write. I'm viewing this data onto the webpage via an AJAX request hitting the above endpoint. For eg:-
$.ajax({
type:'get',
contentType: 'application/json',
url:'domain.com/api/fetch_all?start=0&end=50',
cache : true,
dataType:'json',
success:function(data)
{
// presenting the information when the page loads.
}
});
So, my questing is how can I secure my APIs, so that no robots can access the data that I'm presenting on my forum. For eg:- if any code/script tries to access my APIs, it should throw 403 Forbidden error.
import requests
# this should return 403 error
response = requests.get('domain.com/api/fetch_all?start=0&end=50')
However, if I try to get this data via the browser AJAX request, it should return the data. How can I make sure whether the request is coming from a browser(man-handled) or a robot?
PS: I cannot add OAuth functionality over here, since I dont have a login form.
It's not possible to restrict requesters in this way, because a robot could always add headers to spoof being a browser. Anything you do on your client can be copied by an attacker. Without requiring auth, the best you can do is rate limiting - track requests on a per-client basis, and only allow a certain number of requests per time unit.
A partially-functional solution would be to look at the User-Agent header. That should include browser information, and might let you knock out some robots, but not all or even most of them.
We are trying to create a RESTful API that will be hosted on server x.foo.com. Client html applications (built in jquery) will be hosted on y.foo.com.
I am dealing with cross-domain issues by setting the Access-Control-Allow-Origin header as described here http://www.w3.org/TR/cors/.
So far so good, and I can now successfully make AJAX calls from host y to host x.
However, I ran into a gotcha with POST requests. The typical response to a post request is a redirect. However, the XMLHttpRequest object will not follow cross domain redirects, thus resulting in a failed call.
// Hosted on y.foo.com
$.ajax({
type: "POST",
url : http://x.foo.com/myapp/",
success: function(data) {
alert("success!");
}
});
// Return status: 302
// (Which errors out in firebug)
Anyone know of any techniques to handle the redirect (to a resource on server x) that I get from this post for a client hosted on y?
How about the client sends a special header for AJAX requests, and depending on whether it's an AJAX request or not, you can change the response instead of doing a redirect.