I've an API endpoint hosted (built via Django Rest Framework), for eg:- domain.com/api/fetch_all?start=0&end=50. This fetches all the results from the database in a pagination manner.
Now I'm representing this information on a webpage. Its more or less like an open forum where everyone can read the data, but only some can write. I'm viewing this data onto the webpage via an AJAX request hitting the above endpoint. For eg:-
$.ajax({
type:'get',
contentType: 'application/json',
url:'domain.com/api/fetch_all?start=0&end=50',
cache : true,
dataType:'json',
success:function(data)
{
// presenting the information when the page loads.
}
});
So, my questing is how can I secure my APIs, so that no robots can access the data that I'm presenting on my forum. For eg:- if any code/script tries to access my APIs, it should throw 403 Forbidden error.
import requests
# this should return 403 error
response = requests.get('domain.com/api/fetch_all?start=0&end=50')
However, if I try to get this data via the browser AJAX request, it should return the data. How can I make sure whether the request is coming from a browser(man-handled) or a robot?
PS: I cannot add OAuth functionality over here, since I dont have a login form.
It's not possible to restrict requesters in this way, because a robot could always add headers to spoof being a browser. Anything you do on your client can be copied by an attacker. Without requiring auth, the best you can do is rate limiting - track requests on a per-client basis, and only allow a certain number of requests per time unit.
A partially-functional solution would be to look at the User-Agent header. That should include browser information, and might let you knock out some robots, but not all or even most of them.
Related
I want to login to a website and follow redirection whit ajax or XMLHttpRequest or any thing else exept php.
Actually whene i try to do it, i have error "302 Moved Temporarily" but the webpage is the right page so i don't know why i get this error.
The website is an external website (not on my server).
This is my code :
$.ajax({
type: "POST",
contentType: "application/x-www-form-urlencoded",
url: "http://website/index.php",
data: { username: "myuser", password: "123456" },
success: function(data) {
console.log("success ", data.response);
},
error: function(data) {
console.log("error ", data.error);
},
dataType: "html"
});
If you try use ajax outside your domain, you will probably get this error message:
XMLHttpRequest cannot load http://www.example.com/path/filename. Origin
null is not allowed by Access-Control-Allow-Origin.
The reason you get this error message is because of the Same-origin policy. The policy permits scripts running on pages originating from the same site to access each other's data with no specific restrictions, but prevents scripts access to data that is stored on a different domain.
This could be a problem if you are trying to access publicly hosted data, but there are ways around it.
Here is the list of methods:
Implement CORS (Cross-Origin Resource Sharing)
Use JSONP (JSON Padding)
Use postMessage method
Setting up a local proxy
CORS (Cross-Origin Resource Sharing)
CORS is a mechanism that allows resources on a web page to be requested from another domain outside the domain the resource originated from. In particular, JavaScript's AJAX calls can use the XMLHttpRequest mechanism. Such "cross-domain" requests would otherwise be forbidden by web browsers, per the same origin security policy. CORS defines a way in which the browser and the server can interact to determine whether or not to allow the cross-origin request. It is more useful than only allowing same-origin requests, but it is more secure than simply allowing all such cross-origin requests.
JSONP (JSON Padding)
JSONP or "JSON with padding" is a communication technique used in JavaScript programs running in web browsers to request data from a server in a different domain, something prohibited by typical web browsers because of the same-origin policy. JSONP takes advantage of the fact that browsers do not enforce the same-origin policy on <script> tags.
Because of the same origin policy, we can not make cross domain AJAX requests, but we can have <script> tags that load javascript files from other domains. JSONP uses this exception in order to make cross domain requests by dynamically creating a <script> tag with necessary URL.
postMessage method
window.postMessage method is part of HTML5 introductions. It allows communication between window frames without being subject to same origin policy. Using postMessage() one can trigger a message event with attached data on another window, even if the window has different domain, port or a protocol. The frame where the event is triggered must add an event listener in order to be able to respond.
Let's see an example. Assume, we are on http://example.com (1) website and would like to make a request to http://example2.net (2) domain. We first must obtain a reference to (2) window. This can be either iframe.contentWindow, window.open, or window.frames[]. For our case it's best to create a hidden iframe element and send messages to it.
Setup local proxy
This method overcomes same origin policy by proxying content on another domain through itself. Thus making cross-domain issue irrelevant. To use this method you will either a) setup your server as a reverse proxy to fetch content from another server or b) write a script that would do that.
This cross domain querying solution works because you actually loading content from your own domain. You request the URL and the proxy script on your server loads the content and passes it over to you.
http://www.ajax-cross-origin.com/how.html You can visit this link if you want to learn about these methods in details. There is also a jquery plugin named ajax cross origin to tackle similar issues.
Question Overview:
I am accessing a list of files stored in my AWS S3 bucket through a CORS request of presigned files. This basically works fine. However, the objects have some custom METADATA attached to them, which I can't access. I understood, that I can access this metadata only when I add the header key (e.g. "x-amz-meta-1234", where 1234 is the key of my metadata) to the Expose-Headers of the target-bucket's CORS config. While this works so far for me, I can't set the expose-header with a wildcard (e.g. "x-amz-meta-*"), which would solve my problem, but AWS doesn't support wildcards for the expose-header entries.
However, when I look in the NETWORK tab of my Chrome Dev Tools, all desired metadata is showing up in the headers during the GET/HEAD request (note the entries on the lower part, x-amz-meta-4021 and -template_id):
This is my HEAD call:
$.ajax({
url: url,
dataType: 'json',
crossDomain: true,
type: 'HEAD',
success: function(data, status, jqXHR) {
console.log('got some response ..?');
console.log(data);
console.log(jqXHR);
console.log('responseHeader template_id: ' + jqXHR.getResponseHeader('x-amz-meta-template_id'));
console.log('responseHeader meta-4021: ' + jqXHR.getResponseHeader('x-amz-meta-4021'));
console.log(jqXHR.getAllResponseHeaders());
},
error: function(error, xhr, data) {
console.log('in error..');
console.log(error);
console.log(xhr);
console.log(data);
}
});
});
And this is the console output:
Object {readyState: 4, getResponseHeader: function, getAllResponseHeaders:
function, setRequestHeader: function, overrideMimeType: function…}
responseHeader template_id: 813
responseHeader meta-4021: null
x-amz-meta-template_id: 813
Last-Modified: Fri, 09 Jun 2017 13:05:33 GMT
Content-Type: video/mp4
I set expose-header for the metadata-entry 'template_id' explicitly and therefore the header-data is returned correctly for this entry. However, for the entry '4021' I didn't set the expose-header. The problem is, that this metadata (and the keys) are produced by our (android/ios) apps, and I can't really control the keys of that metadata that easily.
Whats puzzling me: why am I able to see the whole response in the chrome network tab, but can't access this data from a client-side script? There are many possible workarounds and solutions, but I basically want to understand, why my browser can display me data, which can't be accessed by jQuery.
PS: in case you want to see the CORS config or the full script, please let me know. I tried to be as precise as possible. Thanks in advance!
I basically want to understand, why my browser can display me data, which can't be accessed by jQuery.
To understand this, you need to understand the purpose of CORS.
CORS isn't really about access control, and CORS isn't really working on your site's behalf. CORS is working on behalf of the user and the browser, to prevent the browser from becoming a confused deputy and doing something the user would not have wanted. This usually coincides with something the site would also not have wanted, but that's secondary.
The browser's default behavior is to assume that programmatic access to cross-origin requests is bad, which is why they are denied when no Access-Control-Allow-Origin header is present. Your bank would not want internetbadguys.com to make ajax requests to the bank web site, and if that site tried, the browser would block it unless the bank's web server foolishly allowed it with a CORS response.
CORS is a mechanism for your site to tell the brower, "yes, the cross-origin request you are making is not unexpected, it's allowed... and from this response, the browser is allowed to engage in certain behaviors, such as exposing the following response headers to the code making the request."
In that light, the behavior you observe is correct. Exposing headers (or not) doesn't mean include them in the HTTP response (or not) -- exposing headers gives the browser permission to expose what it knows to the ajax caller. If the cross-origin origin wants them exposed, it has to be explicit.
We've built a very lightweight simple Chrome extension for our customers, its private and won't be made publicly available but due to restrictions by Google now needs to at least exist privately on their store.
The idea of the extension is to automatically track cashback opportunities for our clients computers. It does this by checking each new URL they visit just once against our API - if its not a shopping site that exists (eg Facebook or Google) then storage makes sure that URL is never checked again, if it is, then its checked only once in a 24 hour period and the URL returned by the API is visited with an AJAX GET Request.
The get request loads the URL via the cashback site so it sets the correct cookies for the user automatically, in the background, without disrupting the users browsing session.
The extension though whilst accepted by Apple and Mozilla for Firefox and Safari is being rejected by Google and it appears to be due to our XHR request - though they sent the same generic rejection request and it appears its being rejected via an automatic check - it appears whats being flagged up is that the GET request could be request external javascript (eg malicious stuff) from 3rd parties and of course all code (and quite rightly) needs to be within the extension itself.
They provide examples of using cross-origin calls on their site here https://developer.chrome.com/extensions/xhr
As we only need to set the cookies from the URL we visit, would there be anyway to filter the get request to abide by their security rules and instead of just downloading everything from the URL we'd block downloading external javascript libraries etc?
This is well beyond my realms of coding here so i'm more interested if anyone could think of a way we COULD do this that would pass Google's checks - ultimately the AJAX request we do just loads the website they are already on, but as its going via a tracking company to set the cookie it will of course call on a couple of simple redirects first usually to set the session ID. If its possible for us to still use this data whilst making it pass Google checks, i'm not sure.
Here is a snippet of current XHR related code
kango.xhr.send(details, function(data) {
if (data.status == 200 && data.response != null) {
var text = data.response;
console.log(window.location.href);
kango.console.log("THE RESPONSE URL: " + text);
var affilDetails = {
method: 'GET',
url: text,
async: true,
contentType: 'text'
};
and
kango.xhr.send(affilDetails, function (afilData) {
console.log(thisDomain + " expire updated to " + new Date(expireshort));
if (afilData.status == 200 && afilData.response != null) {
kango.storage.setItem(thisDomain,expireshort);
console.log("RESPONSE URL HAS BEEN VISITED VIA AJAX.");
Recently I got to the point to host a static webpage with a subscription option on aws s3 while website development is undergoing. My static web page makes an ajax call to another RESTful service with an email of a subscriber as a parameter. When subscription is done I need to notify a subscriber. Here it seams an issue with the callback.
$.ajax({
type: 'GET',
url: 'http://www.my-domain.com/api/Subscribe?email=' + email
}).success(function (data) {
if (data) {
alert('Thank you for registering!');
}
});
After subscription is done ".success(" doesn't fire up. Response on the request is:
Reload the page to get source for: http://www.my-domain.com/api/Subscribe?email=john.smith#simplyemail.com
Does anyone know if it's an s3 feature or something else?
As per your description this seems to be related with CORS policy.
Look to "why CORS" as Amazon defines it:
In order to keep your content safe, your web browser implements something called the same origin policy.
The default policy ensures that scripts and other active content
loaded from one site or domain cannot interfere or interact with
content from another location without an explicit indication that this
is the desired behavior.
In certain cases, the developer of the original page might have
legitimate reasons to write code that interacts with content or
services at other locations. CORS provides the mechanism to allow the
developer to tell the browser to allow this interaction.
I understood that:
[...] ajax call to another RESTful service[...]
Means call to another server, and this may be blocked by Browser because of CORS.
References:
Mozilla
W3C
Following on from the thread
Is Safari on iOS 6 caching $.ajax results?
If io6 safari is caching the results from non unique ajax calls then it must also be caching the call itself. Would this then mean it is caching usernames and passwords in a login situation thereby posing a security risk?
Short version: If you're sending usernames and passwords over the wire in plaintext, you've already opened a huge security hole.
Long version: Browsers will cache based on URI, so if you're sending user/pass as GET variables, then yes it will cache and yes it is a security risk. However, even if the browser didn't cache this, you're still doing something wrong. A third party need only look at the HTTP header to see what the user/pass is.
If you are sending this as POST, it is a bit harder to find the username/password. The browser will not cache the request as the URL is always the same. However, it is still possible to read the content of the request and find the user/pass.
To be the most secure, use HTTPS and pass the values via POST. The entire HTTP request is encrypted, including the headers. However, the browser will still cache the URL, so using GET variables is still a bad idea.
Example from the jQuery documentation on using POST with ajax:
$.ajax({
type: "POST",
url: "some.php",
data: { name: "John", location: "Boston" }
}).done(function( msg ) {
alert( "Data Saved: " + msg );
});