I need to crawl a site and get all the links from it, the problem is - this site uses ajax, and Go's standart http.Get(..) will return something like:
<body>
//javascript here
<div class="content"></div>
//javascript here
</body>
Div is empty. Is there some solution with golang?
http.Get(Url) just get the response of the Url.
resp.Content is like:
<body>
//javascript here
<div class="content"></div>
//javascript here
</body>
if you want to get the content in the div, you need to analysis the javascript and know how the ajax to get data. Then you can simulate the processes the get what you want.
You must use one of the headless webbrowsers.
Without checking i'm saying that there is no Go binding for the three remaining browser engines (Webkit/Firefox,IE).
Switch to python and javascript.
Related
I want to load a webpage in several parts, maybe using jquery ajax or just simple javascript ajax functions so the user can start interacting with the webpage withouth having to wait for the whole page to load. Just like gmail does.
How can I achieve such a thing?
Generally you need to dedicate some div containers for inserting dynamically loaded contents.
<div id="header"></div>
<div id="content"></div>
<div id="sidebar"></div>
<div id="footer"></div>
Then you can load other pages on startup.
$(function(){
$('#content').load('content.php');
$('#content').load('sidebar.php');
});
Of course this is only a start. You have to check for errors on loading and retry. Also using hash value you can dynamically load content & sidebar. etc..
I've got a simple AJAX POST form set up in a jQTouch application. We're talking out-of-the-box simple here:
<form id="contact" class="topPage" method="post" action="/process/mobile-submit.cfm">
<!-- Various form guts go here -->
</form>
And this works just great. My users punch in their info, my server-side script does its job and gobbles up the lead data and spits back an out-of-the-box simple response.
<div>
<div class="toolbar">
Back
</div>
<div class="info">
<strong>Thank You For Your Submission</strong><br />
We have received your inquiry, and blah blah blah jibber jabber.
</div>
</div>
Everyone's happy... except those of us who are trying to track the conversion in Google Analytics. Now, I've got virtual pageviews set up on each panel in this application using the pageAnimationEnd event, which is easy as pie when you know what selectors those are going to be attached to in advance, but when jQTouch creates a new segment from the form return, it has a generic serialized ID like #page-N.
I've tried adding a loose script block into the form return. That works fine for Firefox on my desktop, not so much for Safari on my phone.
Since I've allowed jQTouch to handle the AJAX particulars for me in this instance, is there a straightforward way to attach a success handler to it? Or am I better off trying to bind a pageAnimationEnd handler on $('[id^=page-]') and hope the business doesn't want me to do anything else with ad hoc form returns until we replace this app with one written in jQuery Mobile?
Worked it out.
The return fragment can declare its own ID, naturally, and jQTouch will then treat it as though it were an original part of the document. I had previously assumed jQTouch didn't give a toss about what attributes I gave the fragment; I was wrong.
This means that you could goTo it like any other portion of the document. It also means that you can bind a pageAnimationEnd handler on the return fragment either by ID or by class name and it will behave as expected. Thus:
<div class="formResult">
<div class="toolbar">
Back
</div>
<div class="info">
<strong>Thank You For Your Submission</strong><br />
We have received your inquiry, and blah blah blah jibber jabber.
</div>
And:
$('.formReturn').live('pageAnimationEnd', function(evt, info) {
if (info.direction == 'in') {
// Goal completion code
} else {
$(this).remove();
}
});
I haven't explained this well.
But what i mean is, if I have an ajax script that loads the content of a page in a DIV element, through the function 'loadpage('whatever.php');, instead of going around manually doing this to all links, is there a way of having a script that automatically makes all regular links load through that ajax function?
Like on Facebook, your profile loads through ajax, yet if you look at their code, they just have a regular link to the profile.
Cheers!
Sure, you can do it with jQuery.
This script goes through the document, finds every anchor element and binds an event handler to the click event of each. When the anchor element is clicked, the event handler finds the href attribute and loads that page into #targetDiv (you can call this div whatever you want, of course).
<script type="text/javascript"
src="http://ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.min.js">
</script>
<script type="text/javascript">
$(document).ready(function() {
$("a").click(function() {
$("#targetDiv").load(($(this).attr("href") + " body");
return false;
});
});
</script>
...
<!-- In your document body, this is the div you'd load the pages into. -->
<div id="targetDiv"></div>
You can use JQuery for this (if I understand your question right).
First you can make the function loadpage() as follows:
function loadpage(divId, url) {
$('#' + divId).load(url);
return false;
}
.load() isn't supported by all browsers though. If you want to do this without .load() then you can check out .get(). For more info on .load(), take a look at http://docs.jquery.com/Ajax/load
I'm assuming it would go something like this:
$(document).ready(function(){
$("a").click(function(){
$("body").load($(this).attr("href") + " body");
return false;
});
});
This would make all <a> tags on the page call a function that downloads a HTML document from the href attribute of the tag, strip out it's body tag, and replace the contents of the current page's body tag with the contents of the new body tag. This way, it's easier to work this with no JavaScript, as well as integrate it into an existing site.
To use it, you place this into a <script> tag in the head of your main page, or in an external JS file.
Please note, however, that this code only updates the contents of the <body> tag, the head (including the title tag) remains untouched. You may need to add extra code to update things like this.
Simple and Nice. Check this out:
Bjax
Usage:
<script src="bjax.min.js" type="text/javascript"></script>
<link href="bjax.min.css" rel="stylesheet" type="text/css" />
Finally, include this in the HEAD of your html:
$('a').bjax();
For more settings, checkout demo here:
Bjax Demo
I have a web page that displays comments, and users can click a 'Flag' link to report an inappropriate comment.
When they click the Flag link, I use AJAX and innerHTML to display a dropdown box underneath the comment with a reason code, e.g. Spam, Offensive, Unrelated to topic, etc., as well as a Cancel button.
If the user clicks Submit, I want to use another AJAX request to send their response to a PHP file, where the database is updated, and they receive a "Thank you" on their end (without reloading the page). I essentially want the DIV that displays the dropdown box to be replaced with "Thank you" using another AJAX request.
That's where the problem is. It seems that I cannot execute an AJAX request from within the HTML response from the first AJAX request. The JavaScript functions fail -- even a simple Alert('hello world') doesn't work. I tried placing the JavaScript functions in the main page that calls the first AJAX request, as well placing it in the PHP file that displays as the HTML response from the first AJAX request, but I did not have any luck -- the functions just do not run when they are called.
Everything works fine if I load the PHP file externally, so I know the JavaScript is correct. It just doesn't work when I load the PHP file into the HTML response DIV and then call the JavaScript from there.
So to sum everything up, how do you execute JavaScript functions from the HTML response of an AJAX request?
EDIT: here is a sample of what I want to do:
This is the AJAX part that populates the DIV when the person clicks the Flag link:
xmlhttp.onreadystatechange=function(){
if (xmlhttp.readyState==4)
{
document.getElementById(whichdiv).innerHTML=xmlhttp.responseText;
}
};
The value of xmlhttp.responseText comes from this external file:
<input type="hidden"/>
<script type="text/javascript" language="javascript">
function displayalert()
{
alert ('Hello World!');
}
</script>
<form name="myform" id="myform">
<input type="text" name="myfield" value="teststring"/><br/>
<input type="button" name="button" value="Submit"
onclick="displayalert();"/>
</form>
Note: the <input type="hidden"/> above comes from a suggestion I found off of http://msdn.microsoft.com/en-us/library/ms533897%28VS.85%29.aspx.
When the user clicks the button, the javascript displayalert() function doesn't run. The alert box never pops up. If I load the file externally instead of calling it with innerHTML, the script works fine.
Can the xmlhttp.responseText contain JavaScript code?
depends on the browser:
IE doesnt support scriptEval on html that is loaded with ajax, which means that if you have script blocks in your html, they wont be called.
Firefox supports script eval.
What i usually do is shove some json into an input, then check if the browser supports scriptEval, if it doesnt, pull the json, eval it, and call some method passing json.
if the browser supports scriptEval, i also include a script block that contains a call to the same method with the json.
you may also want to read this:
http://webreflection.blogspot.com/2007/08/global-scope-evaluation-and-dom.html
I have been reading up on this, and it seems that if you use ajax you can only bring in content that resides on the same domain whereas with an iframe you can bring in content from any domain. Is that the case? What other differences are there?
Bear in mind they're two completely separate technologies.
A (i)frame really loads a complete HTML page in area into the browser. Whether the page is on the same or another domain, for pure viewing, doesn't matter.
Ajax only describes a system to facilitate JavaScript to talk with (and with current security restriction across browser, only with) the server from which you document within which you generated the JavaScript call from.
The (i)frame technology loads and renders a complete HTML page from any URL given. Certain security restrictions accessing other documents from other domains with JavaScript still apply.
With Ajax, it's only meant to use purely JavaScript to talk to the originating server (send some data) and usually get some data back. In JavaScript. What this data is and what you do with it, is up to you. Whether you insert it into the DOM (Document Object Model), exchange parts or load a new page is up to you.
To a certain degree you have all freedom you want. You can have an (i)frame on a page, still make a Ajax call and decide to load another URL into the (i)frame. Or use the Ajax return value to generate new HTML dynamically inside the (i)frame. Or outside, in another document.
The security restrictions applying in this case is called "same origin policy".
Quite simply, an iframe is like a regular frame, but it doesn't split the browser window up into sections, it sits right inside a page and is affected by the scrollbar.
Ajax, on the other hand, uses javascript to do partial loads of a page, allowing small amounts of data to be loaded from the server without needing to do a complete postback. For example, Youtube uses Ajax when you post comments, vote, queue videos to play, etc. They do this so that your video isn't interrupted and restarted by a complete page postback.
Besides these differences mentioned by others, there are others as well.
iframe loads an entire html/php page, whether it is from the own server or other external server. Usually, it has a fresh <html>, <head> and <body> tag as well. Ajax only loads part of the html/php page.
Besides, Ajax pulls the CSS (and maybe, even javascript codes) from the parent file, but in case of Iframe, it cannot pull the same.
E.g this is the master file coding.
<!doctype html>
<html>
<head>
<style>
.gappu {background-color:black;color:red;}
</style>
<meta charset="utf-8">
<script src="../AllJqueries/jquery-1.11.3.min.js"></script> <!-- Use your own jQuery file -->
<script>
<!--
$(document).ready(function(){
$.ajax({url:"slave1.php?bare=true", success:function(data){
$(".myDomain").html(data);
}});
}); /* End of Main Jquery */
//-->
</script>
<title>Ajax vs Iframe</title>
</head>
<body>
<div class="myDomain"></div>
<div>Iframe below</div>
<iframe width="100%" height="500px" src="slave1.php"></iframe>
</body>
</html>
Now, we also have another file, named as slave1.php
<?php
if(isset($_GET['bare'])) $bare = $_GET['bare'];
else $bare = false;
if(!$bare):
?>
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<style>
.gappu {background-color:blue;color:yellow;}
</style>
<!-- You can remove the above style later, and see the difference. The parent style will not apply for iframe -->
<title>Inside the Iframe</title>
</head>
<body>
<?php endif; ?>
<div class="gappu">Hi, welcome to this demo</div>
<?php if(!$bare): ?>
</body>
</html>
<?php endif;
In case of Ajax call, the line Hi, welcome to this demo will be in black background and red color, since it is borrowing the css from the parent. But in iframe, it will be in blue background and white color, which is defined in slave1.php. You can remove the style from slave1.php, and you will find plain text printed in iframe format.
Hope this helps. Cheers.
Vijay Srinivas