I am trying to extract the pricing for an item on the following page: https://www.lowesforpros.com/pd/DEWALT-20-Volt-Max-1-2-in-Cordless-Brushless-Drill/1000135807
In the following code nothing is returned:
response.xpath("//*[#id='main']/div[6]/section[1]/div[3]/div[2]/div[2]/div/span[1]/text()").extract()]
I have looked at the source and do not see any indication of JS in use to pull the pricing.
What about simple:
response.xpath('//span[#itemprop="price"]/#content').extract_first()
The price section is not included in the basic HTML of the page. It is loaded by javascript after the completion of page loading. Consequently, the path is not specified anything. You have to use javascript renderer engines such as Splash or web drivers like Selenium
Related
I am trying to scrape the video links on the web page, https://www.tokopedia.com/chocoapple/ready-stock-bnib-iphone-128gb-7-plus-jet-black-garansi-apple-1-tahun-10?src=topads
There are links, which are getting generated through "webyclip" service which loads the data after the page is loaded. I want the updated HTML source of the page after all the JavaScripts and AJAX are loaded (Similar when we do "Inspect element" on a browser). How to get it done through the chromedp package (https://github.com/knq/chromedp). It is a headless browser for GoLang. Please help. I am a newbie in web scraping.
EDIT: It is not similar to the another question mentioned in the link. As this is specific to chromedp package. The one in the comments ask for how to/ what to use to scrape dynamic contents.
After many attempts, Finally, I found the way and solved my query.
You can check my GitHub repository for this query.
Thank you.
I'm using jet-pack carousel in my gallery website and i want to add image rating capabilities. I have the "wp-postratings" plugin running on the rest of my site.
I'm assuming that i could add this to individual images as wp treats each image as a post, if so where would i find the template i should add this code to?
All the plugin requires is that i add one line of php to the page/image but i can't figure out where it shoud go as the code for carousel seems to be entirely js and php generated.
demo: http://so.devilmaycode.it/adding-rating-feature-to-jetpack-carousel/gallery/
in a js file jetPack_meet_wpPostRating.js copy this code (place the file in the js folder of your theme):
https://gist.github.com/aSeptik/9739808
in the functions.php file of your theme put this code:
https://gist.github.com/aSeptik/9739808
NOTE
after installing the above scripts make sure to delete the browser cache before test.
tested with latest wordpress, jetpack and wp-postrating versions.
this will also work with multiple gallery in the same page.
does not need to put any extra [ratings] shortcode anyware.
the bounty for this answer is too low... ;)
FIX
Fixed bugs added some check and debug to consol.log...
You can use hook into the Jetpack Carousel on the server-side
(like to output your data, for example).
Then on the client-side, get your Javascript rating code to
load after the Jetpack+Carousel JS, and hook into them
(hook on the bindings, etc).
JetPack provides 2 JavaScript hooks:
1) jp_carousel.afterOpen
2)jp_carousel.beforeClose.
Action hook code example
Here's my issue, I'm trying to load a FB comments module via an AJAX div. Now, it works if I go directly to the php page, but if I load it from within the div, it doesn't. I've looked at the other posts (FB javascript SDK after jquery LOAD (ajax)), but still can't seem to get it to work, looking for clarification.
If you want to see what I'm talking about, check http://azconceptphoto.com/lindsey and check out testimonials.
Ideally, on the normal part of my site, I can just load the HTML5 code and it works fine, though for some reason it doesn't remotely work via the AJAX div. And considering the posts from before are out of date (2011), I was hoping to get more feedback on this.
Based on the code from your link you are not using any of the PrototypeJS Ajax methods to get the backend script. Ajax.Updater is the method you need to use so....
Here is my suggestion
change the definition of loadXMLDoc() to
function loadXMLDoc(div, location)
{
new Ajax.Updater(div,location,{'evalScripts':true});
}
this will update the contents of div with the contents of location. Also if the response has <script> tags in it then the javascript will be parsed and evaluated (thats the option evalScripts)
I am using hash-bang URLs in my AJAX application and I am implementing the server-side for:
handle ?_escaped_fragment_=key1=value1%26key2=value2
So when I look at Google's FAQ, it says that this URL has an equivalent snapshot
It is easy to see that the snapshot content is not the same as corresponding hash-bang url. This Google example does not help and therefore my question:
My HTML page has three components/panels/sections that are being updated by AJAX. I use the onclick event on the hash-bang URLs to fetch the content from server and then update relevant section of the HTML page. My panels are updated independent of each other and each panel has its own hash-bang URL .
My question is:
Should the HTML snapshot contain the entire page with all 3 sections or only the updated section?
If I am to return the entire page, it is almost impossible to get the state of the other 2 sections correctly, so would the Googlebot reject my site if the other 2 sections are returned in their default state ?
this is a good question, sadly no answer for this one :( im looking for the same. My problem is that EVERYTHING are news loaded with ajax, so each news is actually a little peace of text so im asking myself if my snapshots should be only the current new or a full page with all the info that i have in my home plus current new's content
Do you have news about that topic ?
I'm working on a website which has been developed in plone. Now I'm facing an issue, I would like to load certain content from a template via an ajax call on normal Plone page(on some event trigger).
Do I need to create any python script??If yes where has it to be placed? and moreover how do I integrate it with TAL(I guess that would be needed) but I'm not sure how.
Could anyone guide me on this with necessary pointers/docs that I should look into?It would of great help to come over my issue and get things rolling.
Thanks,
Avinash
In the "Plone Developer Documentation" there's a section for Javascripting in Plone that perfectly fits your needs
Your question is a bit vague:
From your question, it seems that you just want your ajax call to return html to populate data on the page somewhere then?
Also, it sounds like you want to do the development TTW in the ZMI? Most developers would would use an add-on product and return your ajax response.
However, you can do it TTW with page templates just fine.
Create the new page template
populate it with the template code that gives you the desired output when called within the context of content on a site. For example, http://mysite.com/plone/page/my-template
in your javascript, use a url that in the ajax call: $.ajax({url: 'http://mysite.com/plone/page/my-template', success: function(data){ $('#content').append(data);}})
It's not really anything special to do ajax within plone--just use the tools available and piece it together.