The data on the webpage is displayed dynamically and it seems that checking for every change in the html and extracting the data is a very daunting task and also needs me to use very unreliable XPaths. So I would want to be able to extract the data from the XHR packets.
I hope to be able to extract information from XHR packets as well as generate 'XHR' packets to be sent to the server.
The extracting information part is more important for me because the sending of information can be handled easily by automatically triggering html elements using casperjs.
I'm attaching a screenshot of what I mean.
The text in the response tab is the data I need to process afterwards. (This XHR response has been received from the server.)
This is not easily possible, because the resource.received event handler only provides meta data like url, headers or status, but not the actual data. The underlying phantomjs event handler acts the same way.
Stateless AJAX Request
If the ajax call is stateless, you may repeat the request
casper.on("resource.received", function(resource){
// somehow identify this request, here: if it contains ".json"
// it also also only does something when the stage is "end" otherwise this would be executed two times
if (resource.url.indexOf(".json") != -1 && resource.stage == "end") {
var data = casper.evaluate(function(url){
// synchronous GET request
return __utils__.sendAJAX(url, "GET");
}, resource.url);
// do something with data, you might need to JSON.parse(data)
}
});
casper.start(url); // your script
You may want to add the event listener to resource.requested. That way you don't need to way for the call to complete.
You can also do this right inside of the control flow like this (source: A: CasperJS waitForResource: how to get the resource i've waited for):
casper.start(url);
var res, resData;
casper.waitForResource(function check(resource){
res = resource;
return resource.url.indexOf(".json") != -1;
}, function then(){
resData = casper.evaluate(function(url){
// synchronous GET request
return __utils__.sendAJAX(url, "GET");
}, res.url);
// do something with the data here or in a later step
});
casper.run();
Stateful AJAX Request
If it is not stateless, you would need to replace the implementation of XMLHttpRequest. You will need to inject your own implementation of the onreadystatechange handler, collect the information in the page window object and later collect it in another evaluate call.
You may want to look at the XHR faker in sinon.js or use the following complete proxy for XMLHttpRequest (I modeled it after method 3 from How can I create a XMLHttpRequest wrapper/proxy?):
function replaceXHR(){
(function(window, debug){
function args(a){
var s = "";
for(var i = 0; i < a.length; i++) {
s += "\t\n[" + i + "] => " + a[i];
}
return s;
}
var _XMLHttpRequest = window.XMLHttpRequest;
window.XMLHttpRequest = function() {
this.xhr = new _XMLHttpRequest();
}
// proxy ALL methods/properties
var methods = [
"open",
"abort",
"setRequestHeader",
"send",
"addEventListener",
"removeEventListener",
"getResponseHeader",
"getAllResponseHeaders",
"dispatchEvent",
"overrideMimeType"
];
methods.forEach(function(method){
window.XMLHttpRequest.prototype[method] = function() {
if (debug) console.log("ARGUMENTS", method, args(arguments));
if (method == "open") {
this._url = arguments[1];
}
return this.xhr[method].apply(this.xhr, arguments);
}
});
// proxy change event handler
Object.defineProperty(window.XMLHttpRequest.prototype, "onreadystatechange", {
get: function(){
// this will probably never called
return this.xhr.onreadystatechange;
},
set: function(onreadystatechange){
var that = this.xhr;
var realThis = this;
that.onreadystatechange = function(){
// request is fully loaded
if (that.readyState == 4) {
if (debug) console.log("RESPONSE RECEIVED:", typeof that.responseText == "string" ? that.responseText.length : "none");
// there is a response and filter execution based on url
if (that.responseText && realThis._url.indexOf("whatever") != -1) {
window.myAwesomeResponse = that.responseText;
}
}
onreadystatechange.call(that);
};
}
});
var otherscalars = [
"onabort",
"onerror",
"onload",
"onloadstart",
"onloadend",
"onprogress",
"readyState",
"responseText",
"responseType",
"responseXML",
"status",
"statusText",
"upload",
"withCredentials",
"DONE",
"UNSENT",
"HEADERS_RECEIVED",
"LOADING",
"OPENED"
];
otherscalars.forEach(function(scalar){
Object.defineProperty(window.XMLHttpRequest.prototype, scalar, {
get: function(){
return this.xhr[scalar];
},
set: function(obj){
this.xhr[scalar] = obj;
}
});
});
})(window, false);
}
If you want to capture the AJAX calls from the very beginning, you need to add this to one of the first event handlers
casper.on("page.initialized", function(resource){
this.evaluate(replaceXHR);
});
or evaluate(replaceXHR) when you need it.
The control flow would look like this:
function replaceXHR(){ /* from above*/ }
casper.start(yourUrl, function(){
this.evaluate(replaceXHR);
});
function getAwesomeResponse(){
return this.evaluate(function(){
return window.myAwesomeResponse;
});
}
// stops waiting if window.myAwesomeResponse is something that evaluates to true
casper.waitFor(getAwesomeResponse, function then(){
var data = JSON.parse(getAwesomeResponse());
// Do something with data
});
casper.run();
As described above, I create a proxy for XMLHttpRequest so that every time it is used on the page, I can do something with it. The page that you scrape uses the xhr.onreadystatechange callback to receive data. The proxying is done by defining a specific setter function which writes the received data to window.myAwesomeResponse in the page context. The only thing you need to do is retrieving this text.
JSONP Request
Writing a proxy for JSONP is even easier, if you know the prefix (the function to call with the loaded JSON e.g. insert({"data":["Some", "JSON", "here"],"id":"asdasda")). You can overwrite insert in the page context
after the page is loaded
casper.start(url).then(function(){
this.evaluate(function(){
var oldInsert = insert;
insert = function(json){
window.myAwesomeResponse = json;
oldInsert.apply(window, arguments);
};
});
}).waitFor(getAwesomeResponse, function then(){
var data = JSON.parse(getAwesomeResponse());
// Do something with data
}).run();
or before the request is received (if the function is registered just before the request is invoked)
casper.on("resource.requested", function(resource){
// filter on the correct call
if (resource.url.indexOf(".jsonp") != -1) {
this.evaluate(function(){
var oldInsert = insert;
insert = function(json){
window.myAwesomeResponse = json;
oldInsert.apply(window, arguments);
};
});
}
}).run();
casper.start(url).waitFor(getAwesomeResponse, function then(){
var data = JSON.parse(getAwesomeResponse());
// Do something with data
}).run();
I may be late into the party, but the answer may help someone like me who would fall into this problem later in future.
I had to start with PhantomJS, then moved to CasperJS but finally settled with SlimerJS. Slimer is based on Phantom, is compatible with Casper, and can send you back the response body using the same onResponseReceived method, in "response.body" part.
Reference: https://docs.slimerjs.org/current/api/webpage.html#webpage-onresourcereceived
#Artjom's answer's doesn't work for me in the recent Chrome and CasperJS versions.
Based on #Artjom's answer and based on gilly3's answer on how to replace XMLHttpRequest, I have composed a new solution that should work in most/all versions of the different browsers. Works for me.
SlimerJS cannot work on newer version of FireFox, therefore no good for me.
Here is the the generic code to add a listner to load of XHR (not dependent on CasperJS):
var addXHRListener = function (XHROnStateChange) {
var XHROnLoad = function () {
if (this.readyState == 4) {
XHROnStateChange(this)
}
}
var open_original = XMLHttpRequest.prototype.open;
XMLHttpRequest.prototype.open = function (method, url, async, unk1, unk2) {
this.requestUrl = url
open_original.apply(this, arguments);
};
var xhrSend = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.send = function () {
var xhr = this;
if (xhr.addEventListener) {
xhr.removeEventListener("readystatechange", XHROnLoad);
xhr.addEventListener("readystatechange", XHROnLoad, false);
} else {
function readyStateChange() {
if (handler) {
if (handler.handleEvent) {
handler.handleEvent.apply(xhr, arguments);
} else {
handler.apply(xhr, arguments);
}
}
XHROnLoad.apply(xhr, arguments);
setReadyStateChange();
}
function setReadyStateChange() {
setTimeout(function () {
if (xhr.onreadystatechange != readyStateChange) {
handler = xhr.onreadystatechange;
xhr.onreadystatechange = readyStateChange;
}
}, 1);
}
var handler;
setReadyStateChange();
}
xhrSend.apply(xhr, arguments);
};
}
Here is CasperJS code to emit a custom event on load of XHR:
casper.on("page.initialized", function (resource) {
var emitXHRLoad = function (xhr) {
window.callPhantom({eventName: 'xhr.load', eventData: xhr})
}
this.evaluate(addXHRListener, emitXHRLoad);
});
casper.on('remote.callback', function (data) {
casper.emit(data.eventName, data.eventData)
});
Here is a code to listen to "xhr.load" event and get the XHR response body:
casper.on('xhr.load', function (xhr) {
console.log('xhr load', xhr.requestUrl)
console.log('xhr load', xhr.responseText)
});
Additionally, you can also directly download the content and manipulate it later.
Here is the example of the script I am using to retrieve a JSON and save it locally :
var casper = require('casper').create({
pageSettings: {
webSecurityEnabled: false
}
});
var url = 'https://twitter.com/users/username_available?username=whatever';
casper.start('about:blank', function() {
this.download(url, "hop.json");
});
casper.run(function() {
this.echo('Done.').exit();
});
I'm using ajax to pull in content to create a light box type content page. It also has a next and prev button once loaded so hence the use of ajax.
I wanted to use Ajax for the page navigation too. But if someone clicks a page link and then tries to use the light box feature both the jquery and ajax requests no longer work within the loaded area.
I've read a lot about bind and delegate but not sure how to use them in this context
Here's my main pieces of code:
// This gets called on document ready
function clicky() {
$link.click(function(e) {
e.preventDefault();
var linkPage = $(this).attr('href');
if ($(this).hasClass('pages')){
// PAGE specific code
if ($('body').scrollTop() != 0) {
$('body').animate({ scrollTop: 0 }, 500, function(){
pageLoad(linkPage);
});
} else { pageLoad(linkPage); }
console.log('page');
} else {
// PRODUCT specific code
if ($('body').scrollTop() != 0) {
$('body').animate({ scrollTop: 0 }, 500, function(){
productLoad(linkPage);
});
} else {
productLoad(linkPage);
}
}
});
}
Here's my ajax for the two different areas:
// Ajax stuff going on for pages
function pageLoad(linkPage) {
// Page stuff fades out
history.pushState(null, null, linkPage);
$("#page-content").load(linkPage + " #guts", function(){
// Loads in page content
});
}
// Ajax stuff going on for Products
function productLoad(linkPage) {
// Page stuff fades out
history.pushState(null, null, linkPage);
$("#product-content").load(linkPage + " #guts", function(){
// Shows an overlay/lightbox and loads in content
});
}
Edit: This worked for me
$(document).on('click', '.link' , function(){
console.log('this worked');
return false;
});
This worked:
$(document).on('click', '.link' , function(){
console.log('this worked');
return false;
});
I'v got two questions. First. How can I reduce this code?
$('#m').click(function() {
var href = $(this).attr('href');
$('#con').hide().load('inc/main.php').fadeIn('normal');
return false;
});
$('#b').click(function() {
var href = $(this).attr('href');
$('#con').hide().load('inc/blog.php').fadeIn('normal');
return false;
});
$('#p').click(function() {
var href = $(this).attr('href');
$('#con').hide().load('inc/portfolio.php').fadeIn('normal');
return false;
});
$('#l').click(function() {
var href = $(this).attr('href');
$('#con').hide().load('inc/lebenslauf.php').fadeIn('normal');
return false;
});
$('#k').click(function() {
var href = $(this).attr('href');
$('#con').hide().load('inc/kontakt.php').fadeIn('normal');
return false;
});
I'm using a lib called perfect scrollbar. It is included this way:
$(document).ready(function(a){a("#scrollbox").perfectScrollbar({wheelSpeed:20,wheelPropagation:!1})});
When main.php is loaded in with this script, the scrollbar is not there like it should be. It's because the document doesn't refresh like usual. What to I need to write to get it working when loaded in?
Write a function & pass each selector & filepath to this function
$('#m').click(some_function()
{
helperfunction($(this), 'inc/main.php');
});
function helperfunction(selector, phpfilepath) {
var href = selector.attr('href');
$('#con').hide().load(phpfilepath).fadeIn('normal');
return false;
}
I'm working on a site, http://teneo.telegraphbranding.com/, and I am hoping to load the pages via ajax so that the sidebar and its animation remain consistent.
When the 'About' link is clicked I need it to load about2.php via a jquery ajax function. But I'm not having any luck. When I can get the page to load via ajax it kills all the jquery functionality on the page. From what I've read I think I need to call the jquery upon successful completion of the ajax call.
I've tried everything it seems and can't get it work. What I need is about2.php to load when the link is clicked and the jquery to dynamically size the divs, like it does on the homepage.
I tried this, but it won't even load the page:
//Dropdown
$(document).ready(function () {
var dropDown = $('.dropdown');
$('h3.about').on('click', function() {
dropDown.slideDown('fast', function() {
var url = 'about2.php'
$.get(url, function(data) {
//anything in this block runs after the ajax call
var missionWrap = $('#mission-wrap');
var w = $(window);
w.on('load resize',function() {
missionWrap.css({ width:w.width(), height:w.height()});
});
var missionContent = $('#mission-content');
var w = $(window);
w.on('load resize',function() {
missionContent.css({ width:w.width() - 205 });
});
});
});
});
});
And then this loads the page, but kills all the jQuery associated with it:
var dropDown = $('.dropdown');
$('h3.about').on('click', function() {
dropDown.slideDown('fast', function() {
$('#index-wrap').load('/about2.php');
});
});
Thank you very much.
I also tried this and it just broke everything:
$(document).ready(function () {
var dropDown = $('.dropdown');
$('h3.about').on('click', function () {
dropDown.slideDown('fast', function () {
$.ajax({
type: 'GET',
url: 'about2.php',
success: function () {
var missionWrap = $('#mission-wrap');
var w = $(window);
w.on('load resize', function () {
missionWrap.css({
width: w.width(),
height: w.height()
});
});
var missionContent = $('#mission-content');
var w = $(window);
w.on('load resize', function () {
missionContent.css({
width: w.width() - 205
});
});
};
});
});
});
});
This should work.
function resize_me () {
var w = $(window);
var missionWrap = $('#mission-wrap');
var missionContent = $('#mission-content');
missionWrap.css({ width:w.width(), height:w.height()});
missionContent.css({ width:w.width() - 205 });
}
//Dropdown
$(document).ready(function () {
$(window).on('load resize', function () {
resize_me();
});
var dropDown = $('.dropdown');
$('h3.about').on('click', function() {
dropDown.slideDown('fast', function() {
var url = 'about2.php'
$.get(url, function(data) {
resize_me();
});
});
});
}
I believe that the problem was that the load event that you were attaching to the window object was not being triggered by the successful load of the $.get.
This is a rough outline of how to structure your application to handle this correctly.
Assuming your HTML looks something like this:
<html>
...
<div id="index-wrap">
<!-- this is the reloadable part -->
...
</div>
...
</html>
You need to refactor all the jQuery enhancements to the elements inside #index-wrap to be in a function you can call after a reload:
function enhance(root) {
$('#some-button-or-whatever', root).on('click', ...);
}
(I.e. look up all the elements under the root element that was loaded using AJAX.)
You need to call this function when the page is first loaded, as well as after the AJAX call in the completion callback:
$('#index-wrap').load('/foo.php', function() {
enhance(this);
});
You can also potentially get rid of some of this using delegated ("live") events, but it's best to hit the jQuery documentation on how those work.
As for events bound to window or DOM elements which aren't loaded dynamically, you shouldn't need to rebind them at all based on which subpage is loaded, just check which of your elements loaded is in a handler that you set up once:
$(window).on('load resize', function() {
var $missionContent = $('#missionContent');
if ($missionContent.length) {
$missionContent.css(...);
}
});
I have a wordpress theme in which i have to create a page that has a movie that goes loop.
it has 3 menu points which change the div text without reloading the page.
So far no problem.
Click here – put it in content box 2
But when im on a different page and i click for example the second link, it should go to the video page and change the text to the second one.
How can i do this?
Is there a way to do that with?
url/wordpressname/#1
Found a Solution: i found a solution here: http://www.deluxeblogtips.com/2010/05/how-to-ajaxify-wordpress-theme.html
which i changed to fit my needs:
jQuery(document).ready(function($) {
var $mainContent = $("#text"),
siteUrl = "http://" + top.location.host.toString(),
url = '';
$(document).delegate("a[href^='"+siteUrl+"']:not([href*=/wp-admin/]):not([href*=/wp-login.php]):not([href$=/feed/])", "click", function() {
//location.hash = this.pathname;
//return false;
});
$("#searchform").submit(function(e) {
location.hash = '?s=' + $("#s").val();
e.preventDefault();
});
$(window).bind('hashchange', function(){
url = window.location.hash.substring(1);
if (!url) {
return;
}
if (url=="1") {
$mainContent.html('<p>Text1</>');
}
if (url=="2") {
$mainContent.html('<p>Text2</>');
}
if (url=="3") {
$mainContent.html('<p>Text3</>');
}
if (url=="4") {
$mainContent.html('<p>Text4</>');
}
// url = url + "#content";
//$mainContent.animate({opacity: "0.1"}).html('<p>Please wait...</>').load(url, function() {
//$mainContent.animate({opacity: "1"});
//});
});
$(window).trigger('hashchange');
});