I am trying using facebook from Scrapy.
I can log in successfully. However, I can't get the posts on facebook because the posts comes from ajax calls.
I tried using firebug and check the XHR. I got this:
I tried to check the response of all those request but none of them contains the actual data of the posts.
What is the ajax request that get the posts on facebook?
Thanks in advance
Don't use the AJAX calls from your browser. Those are encrypted since Facebook uses HTTPS connections. Focus on using their API to make the calls you need to get the data you want. That's the entire purpose of their Graph API.
Related
I am trying to make a scraper for a page of a supermarket. I noticed that this supermarket make API HTTP calls via AJAX, and if I enter to Inspect > Network then I can see the request headers of the calls I need.
Inside the requests headers there is an X-Token. If I use this X-Token with the URL via Postman I can get all the info in JSON format, which is better than scraping the web.
The problem is that this X-Token expires (I think, it still works). Is there any possibility to make a call to the page and "intercept" this API call in order to retrieve this X-Token and use it for the next custom requests?
I'am using Ruby on Rails :)
I have been trying this for quite some time and with no avail. Would someone point me to the right direction.
I have a simple suitelet
the suitelet dumps a html in the response by reading a file
The html has a button. On ajax call of that button, a restlet gets called and a request is sent.
This is working great with Login and internal calls. However, I want the suitelet to be available without login and also the restlet.
When I do that, I get CORS error and using jsonp when I relsolve that, I get 401 Unauthorized
Tried Outh and NLAuth, no avail.
Will really appreciate any help or pointers.
If the info isn't sensitive you can call the suitelet from the domain of your website. If you have enabled https on the site then that'll just work
e.g. https://forms.na3.netsuite.com/app/site... becomes http://example.com/app/site...
If you are using SCA then you can also just do this by calling a .ss service function. The nice thing about that is that you can set the role on the .ss file and have better control of your resources without just making the customer center role capable of seeing everything. Same issue with https though.
If the info is sensitive you can get around the issue on most browsers by embedding an iframe at the https:// forms etc and using the onMessage operation to transfer info from the non-secure to the secure. (see https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage)
It's certainly possible to call a RESTlet from a Suitelet but it sounds like your issue is more how to call the Suitelet securely.
I followed the example on http://docs.brightcove.com/en/video-cloud/media/references/reference.html#Video. The response gets shown in an iframe. However, I would like to be able to get the JSON response in order to store the data (i.e., Brightcove video ID) in my own database. I tried using AJAX post but Brightcove doesn't seem to accept post requests from a different origin. Is there a way to simply get the response data without displaying it in an iframe or in a separate window?
You can't get the response data in javascript since the API response does not include CORS headers. You'd need to do this in server-side code, which also has the advantage of not exposing your API token to the browser.
I know how to use ajax for submitting a form and all. What I am concerned about is, what is actually happening in the background when a form is submitted via ajax.
How are the values transferred? Encrypted or not? And what is the
need of specifying submission type, I mean get or post, if the URL is
not showing the form fields?
Edit: Found this on w3schools:
GET requests can be cached
GET requests remain in the browser history
GET requests can be bookmarked
GET requests should never be used when dealing with sensitive data
GET requests have length restrictions
GET requests should be used only to retrieve data
POST requests are never cached
POST requests do not remain in the browser history
POST requests cannot be bookmarked
POST requests have no restrictions on data length
How do these apply to ajax form submission?
Basically, when you Ajax-submit a form, it is doing exact same thing as what would happen when you as a user GET or POST submit a form - except that it is done in an asynchronous thread by the browser - i.e. called XMLHttpRequest.
If you submit form as a GET request, all of the form values are stitched together as parameter strings and appended to the URL (form's ACTION URL) - prefixed by a ?. This means anyone who can intercept that communication can read the submitted form data even if request is sent to a HTTPS URL. The POST method sends form data as a separate block (from the URL) and if URL is HTTPS then form data gets encrypted.
It looks like you are just starting out in the world of web development - welcome to the world of programming. I would recommend reading up on some good web development/programming books (I don't want to promote any particular book here). Amazon may help suggest few good ones under "Web Development" kind of search terms.
Also, I suggest that you read up a little on GET vs. POST by googling for it (I can only include one or two links - google will show you hundreds).
For the clear understanding & behind the scene things please refer the links given below.
http://www.jabet.com/
How does AJAX work?
Actually ajax request is same as the normal requests at the server end.
GET or POST has their own use cases. for example: GET has a limit of data transfer depending on the browsers from 1KB to 10 KB. where POST has no such limits.
For a server both AJAX & normal request both are same. so it depends on server code which method you wish to support.
ajax requests are NOT encrypted.
http://www.w3schools.com/tags/ref_httpmethods.asp
It looks like you want a very detailed answer so you can find it yourself:
Google it and read thoroughly the pages (wikipedia for example)
Read http://www.w3.org/TR/XMLHttpRequest/
Inspect the packets between your browser and the server
I am trying to use ajax in my spring mvc application. When I try a url (post/get) which is secured and needs authentication, the response is the html of login page as it is redirected behind the scenes.
What is the best approach to overcome this issue?
First, I would avoid displaying Ajax links to URLs needing authentication if the user is not authenticated, if possible.
If not always possible, your login page could be returned with a specific HTTP response code, (or any other way to distinguish it from a normal response) and your JavaScript callback could replace the entire body of the current page with the HTML received if this response code is received. Most AJAX libraries come with a way to define a handler to all the AJAX requests. Such a global handler could be used here.
The login page could also be adapted to only return a status code in case of an AJAX request, and the JavaScript code would then redirect to the login page (without using AJAX) if this status code is received.
I may not have explained the issue well. So I did not get the right response. However the response from JB Nizet contained some other points. So thank you.
I could solve the issue after coming back to this issue after some time, so
I posted about this on my blog.
I hope it is useful.