"<i>" interrupting correct node selection - xpath

Question

I'm trying to select a table field with the following structure:
<td class='postac'>proszek do sporz. roztworu do wlewu <I>i.v.</I>
1,5 g
1 fiol. typu Monovial
</td>
After using xpath expression sel.xpath("//table[#class='table-postaci']/tbody/tr/td[2]/text()").extract() I get two values instead of one:
u'proszek do sporz. roztworu do wlewu ',
u'\r\n 1,5 g\r\n 1 fiol. typu Monovial\r\n '
Is it some clean xpath method to get this "td" field as a single value? I know I could get the field with //table[#class='table-postaci']/tbody/tr/td[2] and then strip the tags in the scrapy pipeline. However, I'm looking for some simplier solution. Thank you

Answer 1

You can loop over each table row tr and for each row join all text node descendants of the 2nd td cell:
In [13]: from scrapy.selector import Selector
In [14]: selector = Selector(text="""<table class='table-postaci'>
....: <thead><th>Nazwa preparatu</th><th>Postać i dawka</th><th>Producent</th><th>Cena 100%</th>
....: <th>Odpłatność po refundacji</th>
....: </thead>
....: <tbody>
....:
....: <tr>
....: <td class='postac'>Zinacef </td>
....: <td class='postac'>proszek do sporz. roztworu do wlewu <I>i.v.</I>
....: 1,5 g
....: 1 fiol. typu Monovial
....: </td>
....: <td>GlaxoSmithKline – Wielka Brytania</td>
....: <td class='cena'> b/d </td>
....: <td>
....: </td>
....: </tr>
....: <tr>
....: <td class='postac'>Zinacef </td>
....: <td class='postac'>proszek do sporz. roztworu do wlewu <I>i.v.</I>
....: 750 mg
....: 1 fiol. typu Monovial
....: </td>
....: <td>GlaxoSmithKline – Wielka Brytania</td>
....: <td class='cena'> b/d </td>
....: <td>
....: </td>
....: </tr>
....: </tbody>
....: </table""")
In [15]: selector.xpath('//table/tr')
Out[15]: []
In [16]: selector.xpath('//table//tr')
Out[16]:
[<Selector xpath='//table//tr' data=u'<tr><td class="postac">Zinacef </td>\n\t\t<'>,
<Selector xpath='//table//tr' data=u'<tr><td class="postac">Zinacef </td>\n\t\t<'>]
In [17]: for row in selector.xpath('//table//tr'):
....: print row.xpath('td[2]//text()').extract()
....:
[u'proszek do sporz. roztworu do wlewu ', u'i.v.', u'\n 1,5 g\n 1 fiol. typu Monovial\n ']
[u'proszek do sporz. roztworu do wlewu ', u'i.v.', u'\n 750 mg\n 1 fiol. typu Monovial\n ']
In [18]: [u''.join(row.xpath('td[2]//text()').extract()) for row in selector.xpath('//table//tr')]
Out[18]:
[u'proszek do sporz. roztworu do wlewu i.v.\n 1,5 g\n 1 fiol. typu Monovial\n ',
u'proszek do sporz. roztworu do wlewu i.v.\n 750 mg\n 1 fiol. typu Monovial\n ']
In [19]:

Answer 2

You should avoid /text() for exactly this reason. Usually you don't want the individual text nodes, you want the string value of the element, which you can get with the string() function. It's not clear what programming language you are calling XPath from, or whether it's XPath 1.0 or 2.0 - that will affect the detail, e.g. whether to get the string value of the element in the XPath expression or in the host language.

Answer 3

The td node in your question has three child nodes – first a text node with the contents:
proszek do sporz. roztworu do wlewu
second an I element node that has its own child text node, and last another text node with the contents:
\n 1,5 g\n 1 fiol. typu Monovial\n
Your query, the end of which looks like td[2]/text(), only selects the immediate text node children of the td element, so it doesn’t select the I element node or its text node child. The result is the two text nodes that you are seeing.
You could select all text node decedents of the td element using td[2]//text() (note the double slash //). This will return three text nodes in the result – the two as above and a third containing i.v. in between them. You could then join them outside XPath (I’m not familiar with scrapy so I can’t tell you how to that in this case).
As far as I know you can’t join the three nodes directly using XPath 1.0, but it might be possible with XPath 2.0.