I'm writing a crawler to get the content from a website which uses AJAX.
There is a "show more" button at the bottom of the page, and my origin approach is to use Selenium.PhantomJS to pretend a web browser but it works in some website and some don't.
I'm wondering if there is some way i can directly get the underly JSON file of the AJAX action. Please give me some details, thanks.
By the way, I'm using Python.
I understand this is less of a python than a scraping problem in general (and I understand you meant "scraping" instead of "crawling" as a scraper reads/parses/processes one page whereas a crawler processes multiple pages and they're relation to each other).
You can get the JSON file immediately given you know it's URL. If you don't (for example because the URL changes from time to time), you might need to search through javascript files on the page manually to find out how the URL is generated.
Once you know the JSON file's URL, it's quite simple. As you already seem to know how to get the HTML of the "main" page, you can use your existing code to get the JSON file.
I'm not familiar with PhantomJS, but I reckon it's easier to get the JSON file immediately instead of simulating an AJAX request (if that's even possible with Phantom).
Using a TWebBrowser in Delphi, I'm showing a website that uses JSON to update its content. To get the source code of the newly updated content and load it into a Memo, I believe I have to get the URLs of the GET requests. Unfortunately, these are always different and generated with an encrypted Javascript. Is there any way to list the URLs the GET requests go to in a similar way like FireBug does in its console view?
Thanks a bunch!!!
I was trying to parse some HTML content from a site. Nokogiri works perfectly for content loaded the first time.
Now the issue is how to fetch that content which is loaded using AJAX. For example, there is a "see more" link and more items are fetched using AJAX, or consider a case for AJAX-based tabs.
How can I fetch that content?
You won't be able to parse anything that requires a JavaScript runtime to produce that content using Nokogiri. Nokogiri is a HTML/XML parser, not a web browser.
PhantomJS on the other hand is a web browser, albeit a special kind of browser ;) Take a look at that and have a play.
It isn't completely clear what you want to do, but if you are trying to get access to additional HTML that is loaded by AJAX, then you will need to study the code, figure out what URL is being used for the AJAX request, whether any session IDs or cookies have been set, then create a new URL that reproduces what AJAX is using. Request that, and you should get the new content back.
That can be difficult to do though. As #Nuby said, Mechanize could be good help, as it is designed to manage cookies and sessions for you in the background. Mechanize uses Nokogiri internally so if you request a page from Mechanize, you can use Nokogiri searches against it to drill down and extract any particular JavaScript strings. They'll be present as text, so then you can use regex or substring matches to get at the particular parameters you need, then construct the new URL and ask Mechanize to get it.
My website uses ajax.
I've got a user list page which list users in an ajax table (with paging and more information stuff...).
The url of this page is :
/user-list
User list is created by ajax. When the user click on one user, he is redirected to a page which url is : /member/memberName
So we can see here that ajax is used to generate content and not to manage navigation (with the # character).
I want to detect bot to index all pages.
So, in ajax I want to display an ajax table with paging and cool ajax effetcs (more info...) and when I detect a bot I want to display all users (without paging) with a link to the member page like this :
JohnBob...
Do you think I can be black listed with this technique ? If you think so, could you please provide an alternative solution by keeping these clean urls and without redeveloping the user-list (without ajax) ?
Google support a specification to make AJAX crawlable:
http://code.google.com/web/ajaxcrawling/docs/specification.html
I did an experiment and it works:
http://seo-website-designer.com/SEO-Ajax-Google-Solution
As this is a Google specification, you won't get penalised (unless you abuse it).
Saying that, only Google support it at the moment (AFAIK).
Also, I believe following the concept of Progressive Enhancement is a better approach. That is, create a working html website then make the JavaScript enhance it
Maybe use the urls with an onclick to trigger your AJAX scripting? Like
Some URL
I don't think Google would punish you for this, you primarily use JScript, but you do provide a fall back for their bot, so your site doesn't get any less accessible.
EDIT
Ok, I misunderstood. Then my guess would be you basically have two options:
1. Write a different part of your site where bots end up, or,
2. Rewrite your current site to for example always give a 'full' page, with an option to only get, say, the content div. Then you can get only the content with JavaScript, but bots will always get a nice page.
I'm writing an AJAX app, but as the user moves through the app, I'd like the URL in the address bar to update despite the lack of page reloads. Basically, I'd like for them to be able to bookmark at any point and thereby return to the current state.
How are people handling maintaining RESTfulness in AJAX apps?
The way to do this is to manipulate location.hash when AJAX updates result in a state change that you'd like to have a discrete URL. For example, if your page's url is:
http://example.com/
If a client side function executed this code:
// AJAX code to display the "foo" state goes here.
location.hash = 'foo';
Then, the URL displayed in the browser would be updated to:
http://example.com/#foo
This allows users to bookmark the "foo" state of the page, and use the browser history to navigate between states.
With this mechanism in place, you'll then need to parse out the hash portion of the URL on the client side using JavaScript to create and display the appropriate initial state, as fragment identifiers (the part after the #) are not sent to the server.
Ben Alman's hashchange plugin makes the latter a breeze if you're using jQuery.
Look at sites like book.cakephp.org. This site changes the URL without using the hash and use AJAX. I'm not sure how it does it exactly but I've been trying to figure it out. If anyone knows, let me know.
Also github.com when looking at a navigating within a certain project.
It is unlikely the writer wants to reload or redirect his visitor when using Ajax.
But why not use HTML5's pushState/replaceState?
You'll be able to modify the addressbar as much as you like. Get natural looking urls, with AJAX.
Check out the code on my latest project:
http://iesus.se/
This is similar to what Kevin said. You can have your client state as some javascript object, and when you want to save the state, you serialize the object (using JSON and base64 encoding). You can then set the fragment of the href to this string.
var encodedState = base64(json(state));
var newLocation = oldLocationWithoutFragment + "#" + encodedState;
document.location = newLocation; // adds new entry in browser history
document.location.replace(newLocation); // replaces current entry in browser history
The first way will treat the new state as a new location (so the back button will take them to the previous location). The latter does not.
SWFAddress works in Flash & Javascript projects and lets you create bookmarkable URLs (using the hash method mentioned above) as well as giving you back-button support.
http://www.asual.com/swfaddress/
The window.location.hash method is the preferred way of doing things. For an explanation of how to do it,
Ajax Patterns - Unique URLs.
YUI has an implementation of this pattern as a module, which includes IE specific work arounds for getting the back button working along with re-writing the address using the hash. YUI Browser History Manager.
Other frameworks have similar implementations as well. The important point is if you want the history to work along with the re-writing the address, the different browsers need different ways of handling it. (This is detailed in the first link article.)
IE needs an iframe based hack, where Firefox will produce double history using the same method.
If OP or others are still looking for a way to do modify browser history to enable state, using pushState and replaceState, as suggested by IESUS, is the 'right' way to do it now. It's main advantage over location.hash seems to be that it creates actual URLs, not just hashes. If browser history using hashes is saved, and then revisited with JavaScript disabled, the app won't work, since the hashes aren't sent to the server. However, if pushState has been used, the entire route will be sent to the server, which you can then build to respond appropriately to the routes. I saw an example where the same mustache templates were used on both the server and the client side. If the client had JavaScript enabled, he would get snappy responses by avoiding the roundtrip to the server, but the app would work perfectly fine without the JavaScript. Thus, the app can gracefully degrade in the absence of JavaScript.
Also, I believe there is some framework out there, with a name like history.js. For browsers that support HTML5, it uses pushState, but if the browser doesn't support that, it automatically falls back to using hashes.
Check if user is 'in' the page, when you click on the URL bar, JavaScript says you are out of page.
If you change the URL bar and press 'ENTER' with the symbol '#' within it then you go into the page again, without click on the page manually with mouse cursor, then a keyboard event command (document.onkeypress) from JavaScript will be able to check if it's enter and active the JavaScript for redirection.
You can check if user is IN the page with window.onfocus and check if he's out with window.onblur.
Yeah, it's possible.
;)