Get siblings from header until next header in an unsemantic table - xpath

Question

Using Scrapy I'd like to parse a webpage containing a very unsemantic table. What I'm looking for is a "print every following-sibling until you meet the following element"-XPath-query.
<table>
<tr>
<th>Title</th>
<th>Name</th>
<th>Comment</th>
<th>Note</th>
</tr>
<tr style="background-color:#CCDDEF;">
<td colspan="4"> <b>HEADER1</b></td>
</tr>
<tr>
<td>Title1.1</td>
<td>-</td>
<td>Info1.1</td>
<td></td>
</tr>
<tr style="background-color:#CCDDEF;">
<td colspan="4"> <b>HEADER2</b></td>
</tr>
<tr>
<td>Title2.1</td>
<td>Name2.1</td>
<td></td>
<td></td>
</tr>
<tr>
<td>Title2.2</td>
<td>Name2.2</td>
<td>Info2.2</td>
<td></td>
</tr>
<tr style="background-color:#CCDDEF;">
<td colspan="4"> <b>HEADER3</b></td>
</tr>
<tr>
<td>Title3.1</td>
<td>Name3.1</td>
<td></td>
<td></td>
</tr>
</table>
I'd like to group every Title, Name, Comment and Note under each header. I have tried with various XPaths (with variations of following-sibling, preceding-sibling and count) but I either get nothing, everything or every tr which is not a header.
I'm currently getting the headers with //tr[#style] or //tr[td[#colspan="4"]].
The following is the parse-function in my Scrapy-spider (which prints the header and all of the tr's which is not a header):
def parse(self, response):
hxs = HtmlXPathSelector(response)
sites = hxs.select('//*[#id="content-text"]//tr[td[#colspan="4"]]')
for site in sites:
print site.select('./td/b/text()').extract()
print site.select('./following-sibling::tr[not(td[#colspan])]')

Answer 1

This XPath expression:
/*/tr[#style or td[#colspan='4']][1]/following-sibling::tr
[count(. | /*/tr[#style or td[#colspan='4']][2]/preceding-sibling::tr)
=
count(/*/tr[#style or td[#colspan='4']][2]/preceding-sibling::tr)
]
selects all tr elements that are between the 1st and 2nd headers:
<tr>
<td>Title1.1</td>
<td>-</td>
<td>Info1.1</td>
<td/>
</tr>
To select all tr elements that are between the Kth and (K+1)th headers, simply replace in the above expression 1 with K (the number) and 2 with K+1 (the number).
XSLT - based verification:
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output omit-xml-declaration="yes" indent="yes"/>
<xsl:strip-space elements="*"/>
<xsl:template match="/">
<xsl:copy-of select=
"/*/tr[#style or td[#colspan='4']][1]/following-sibling::tr
[count(. | /*/tr[#style or td[#colspan='4']][2]/preceding-sibling::tr)
=
count(/*/tr[#style or td[#colspan='4']][2]/preceding-sibling::tr)
]
"/>
</xsl:template>
</xsl:stylesheet>
When this transformation is applied on the provided XML document:
<table>
<tr>
<th>Title</th>
<th>Name</th>
<th>Comment</th>
<th>Note</th>
</tr>
<tr style="background-color:#CCDDEF;">
<td colspan="4">
<b>HEADER1</b>
</td>
</tr>
<tr>
<td>Title1.1</td>
<td>-</td>
<td>Info1.1</td>
<td></td>
</tr>
<tr style="background-color:#CCDDEF;">
<td colspan="4">
<b>HEADER2</b>
</td>
</tr>
<tr>
<td>Title2.1</td>
<td>Name2.1</td>
<td></td>
<td></td>
</tr>
<tr>
<td>Title2.2</td>
<td>Name2.2</td>
<td>Info2.2</td>
<td></td>
</tr>
<tr style="background-color:#CCDDEF;">
<td colspan="4">
<b>HEADER3</b>
</td>
</tr>
<tr>
<td>Title3.1</td>
<td>Name3.1</td>
<td></td>
<td></td>
</tr>
</table>
the Xpath expression is evaluated and the selected nodes are copied to the output:
<tr>
<td>Title1.1</td>
<td>-</td>
<td>Info1.1</td>
<td/>
</tr>
Explanation:
This is a simple application of the Kayessian (after Dr. Michael Kay) formula for node-set intersection:
$ns1[count(.|$ns2) = count($ns2)]
In this particulat case we substitute $ns1 with:
/*/tr[#style or td[#colspan='4']][1]/following-sibling::tr
and we substitute $ns2 with:
/*/tr[#style or td[#colspan='4']][2]/preceding-sibling::tr