I'm trying to select the ingredients in an ingredients list, but there are also tooltips scattered amongst them (on the BBC Good Food site).
As a stripped-down example:
<li class="ingredients-list__item" itemprop="ingredients">
400g
<a href="/glossary/new-potatoes" class="ingredients-list__glossary-link tooltip-processed">
new potato
<div id="gf-tooltip-0" class="gf-tooltip" role="tooltip">
<div class="gf-tooltip__content">
<div class="gf-tooltip__text">
<p>unwanted tooltip</p>
</div>
</div>
</div>
</a>, halved if large
<span class="ingredients-list__glossary-element" id="ingredients-glossary"></span>
</li>
I'm trying to end up with '400g new potato, halved if large', or equally good, ['400g', 'new potato', ', halved if large'].
Amongst other things I've tried:
s.xpath("//li[#class='ingredients-list__item'][not(div[#class='gf-tooltip'])]//text()").extract()
But this still returns the text in the tooltip div.
One possible way would be excluding text nodes where any of the ancestor is a tooltip div (broken into 2 lines for readability) :
//li[#class='ingredients-list__item']
//text()[not(ancestor::div[#class='gf-tooltip'])]
Related
I would like to show an example.
This how the page looks:
<a class="aclass">
<div class="divclass"></div>
<div id="innerclass">
<span class="spanclass">Hello</span>
</div>
</a>
<a class="aclass">
<div class="divclass"></div>
<div id="innerclass">
<span class="spanclass">Pick Delivery Location</span>
</div>
</a>
I want to select anchor tags that have a child (direct or non-direct) span that has the text 'Hello'.
Right now, I do something like this:
//a[#class='aclass'][div/span[text() = 'Hello']]
I want to be able to select without having to select direct children (div in this case), like this:
//a[#class='aclass'][//span[text() = 'Hello']]
However, the second one finds all the anchor tags with the class 'aclass' rather than the one with the span with 'Hello' text.
I hope I worded my question clearly. Please feel free to edit if necessary.
In your attempt, // goes back to the root of the document - effectively you are saying "Give me the as for which there is a span anywhere in the document", which is why you get them all.
What you need is the descendant axis :
//a[#class='aclass' and descendant::span[text() = 'Hello']]
Note I have joined the conditions with and, but two separate conditions would also work.
I have the following HTML
<span class="medium bold day-time-clock">
09:00
<div class="tooltip-box first-free-tip ">
<div class="tooltip-box-inner">
<span class="fa fa-clock-o"></span>
Some more text
</div>
</div>
</span>
I want an XPath that only gets the text 09:00, not Some more text NOT using text()[1] because that causes other problems. My current XPath looks like this
("//span[1][contains(#class, 'day-time-clock')]/text()")
I want one that ignores this whole part of the HTML
<div class="tooltip-box first-free-tip ">
<div class="tooltip-box-inner">
<span class="fa fa-clock-o"></span>
Some more text
</div>
</div>
You can limit the level of descendant:: nodes with position().
So the following expression does work:
span/descendant::node()[2 > position()]
Adjust the number in the predicate to your needs, 2 is only an example. A disadvantage of this approach is that the counting of the descendants is only accurate for the first child in the descending tree.
Another approach is limiting the both: the ancestors and the descendants:
span/descendant::node()[3 > count(ancestor::*) and 1 > count(descendant::*)]
Here, too, you have to adjust the numbers in the predicates to get any useful results.
Use normalize-space() for select all non-whitespace nodes of the document:
//span[contains(#class, 'day-time-clock')]/text()[normalize-space()]
I think (if I understand you correctly) that
"..//div[contains(#class, 'tooltip-box')]/parent::span"
gets you there.
I'm trying to get the xpath of an element with a similar xpath to others but has a "neighbor" element that's different . Please see example below.
<div>
<div id='a'> </div>
<span> Text here </span> #this is what i'm trying to get
</div>
<div>
<div id='b'> </div>
<span> Text here </span>
</div>
I tried using //div//span, but this gives me the 2 spans. So i tried using //div//child::div[#id='a']//ancestor::div//child::span, but it doesn't look pleasant and looks repetitive. Does this have a better implementation?
try
//div[div[#id='a']]/span
it says get the span child node of all div nodes with child node div (with an #id equal to 'a').
In the code sample below I'm looking to extract, using Xpath inside of Scrapy, first from list 1 and then from list 2. Some items may be linked out while others are just items in the list. What I need is two strings (or lists) one for List 1 and one for List 2
<div class="row">
<div class="col-xs-12 no-padding-xs">
<h3 class="text-primary gutter-xs">List 1</h3>
<div class="well well-sm">
Miniature, Mustang, Paint Pony, Pinto, Pony, POA, Quarter Pony, Shetland Pony, Spanish Mustang
</div>
<h3 class="text-primary gutter-xs">List 2</h3>
<div class="well well-sm">
All Around, Driving, Halter, Lesson, Natural Horsemanship, Show, Trail Riding, Western Pleasure, Western Riding, Youth, Champion Trainer, POA Ponies for Sale, Newaygo County, Horse Boarding, Equestrian Coaching, Michigan, Riding Lessons, Horse Lea
</div>
</div>
</div>
Not sure that I understood you properly, but you can try:
from w3lib.html import remove_tags
for list_text in ['List 1', 'List 2']:
div_data = response.xpath('//h3[text()="{}"]/following-sibling::div[1]'.format(list_text)).get()
if not div_data:
continue
print [remove_tags(i).strip() for i in div_data.split(',')]
Or if you want just strings:
for list_text in ['List 1', 'List 2']:
div_data = response.xpath('//h3[text()="{}"]/following-sibling::div[1]'.format(list_text)).get()
if not div_data:
continue
print remove_tags(div_data)
There’s a document structured as follows:
<div class="document">
<div class="title">
<AAA/>
</div class="title">
<div class="lead">
<BBB/>
</div class="lead">
<div class="photo">
<CCC/>
</div class="photo">
<div class="text">
<!-- tags in text sections can vary. they can be `div` or `p` or anything. -->
<DDD>
<EEE/>
<DDD/>
<CCC/>
<FFF/>
<FFF>
<GGG/>
</FFF>
</DDD>
</div class="text">
<div class="more_text">
<DDD>
<EEE/>
<DDD/>
<CCC/>
<FFF/>
<FFF>
<GGG/>
</FFF>
</DDD>
</div class="more_text">
<div class="other_stuff">
<DDD/>
</div class="other_stuff">
</div class="document">
The task is to grab all the elements between <div class="lead"> and <div class="other_stuff"> except the <div class="photo"> element.
The Kayessian method for node-set intersection $ns1[count(.|$ns2) = count($ns2)] works perfectly. After substituting $ns1 with //*[#class="lead"]/following::* and $ns2 with //*[#class="other_stuff"]/preceding::*,
the working code looks like this:
//*[#class="lead"]/following::*[count(. | //*[#class="other_stuff"]/preceding::*)
= count(//*[#class="other_stuff"]/preceding::*)]/text()
It selects everything between <div class="lead"> and <div class="other_stuff"> including the <div class="photo"> element. I tried several ways to insert not() selector in the formula itself
//*[#class="lead" and not(#class="photo ")]/following::*
//*[#class="lead"]/following::*[not(#class="photo ")]
//*[#class="lead"]/following::*[not(self::class="photo ")]
(the same things with /preceding::* part) but they don't work. It looks like this not() method is ignored – the <div class="photo"> element remains in the selection.
Question 1: How to exclude the unnecessary element from this intersection?
It’s not an option to select from <div class="photo"> element excluding it automatically because in other documents it can appear in any position or doesn't appear at all.
Question 2 (additional): Is it OK to use * after following:: and preceding:: in this case?
It initially selects everything up to the end and to the beginning of the whole document. Could it be better to specify the exact end point for the following:: and preceding:: ways? I tried //*[#class="lead"]/following::[#class="other_stuff"] but it doesn’t seem to work.
Question 1: How to exclude the unnecessary element from this intersection?
Adding another predicate, [not(self::div[#class='photo'])] in this case, to your working XPath should do. For this particular case, the entire XPath would look like this (formatted for readability) :
//*[#class="lead"]
/following::*[
count(. | //*[#class="other_stuff"]/preceding::*)
=
count(//*[#class="other_stuff"]/preceding::*)
][not(self::div[#class='photo'])]
/text()
Question 2 (additional): Is it OK to use * after following:: and preceding:: in this case?
I'm not sure if it would be 'better', what I can tell is following::[#class="other_stuff"] is invalid expression. You need to mention the element to which the predicate will be applied, for example, 'any element' following::*[#class="other_stuff"], or just 'div' following::div[#class="other_stuff"].