Selenium performance in converting an HTML table into a .NET DataTable - performance

Using Selenium with VB.NET using the ChromeDriver. I am using XPath to get a collection of TR nodes and then iterating through each row to get the TD data.
Source looks like this
<table width="100%" border="0" cellpadding="2" cellspacing="1">
<tbody>
<tr bgcolor="#E7E7BD">
<td class="content">RUSSIA</td>
<td class="content">ABCD</td>
<td class="content">13-APR-18</td>
<td class="content">26-APR-18</td>
<td class="content"> 01234567</td>
<td class="content"/>
<td class="content"/>
</tr>
<tr bgcolor="#E7E7BD">
<td class="content">ZURICH</td>
<td class="content">XYZS</td>
<td class="content">09-NOV-18</td>
<td class="content"/>
<td class="content"> 98765432</td>
<td class="content">Z</td>
<td class="content">DAA</td>
</tr>
</tbody>
</table>
I use this to get the rows which works well and returns some 9500 rows:
Dim trNodes As IReadOnlyCollection(Of IWebElement) = driver.FindElements(By.XPath("//tbody/tr[preceding-sibling::tr/td/a/#name='MAIN' and following-sibling::tr/td/a/#name='APPX']"))
Then I iterate through each HTML row and add it to a .NET DataTable
For Each elem As IWebElement In trNodes
'get the cells in the row (ie the TD elements in the TR element)
Dim webCells As IReadOnlyCollection(Of IWebElement) = elem.FindElements(By.XPath("td"))
'create a new DataRow and add data
Dim row As DataRow = dt.NewRow
'now cycle through all the fields and add them to the DataRow
For i As Integer = 0 To webCells.Count - 1
row(i + 1) = webCells(i).Text
Next i
'add the DataRow to the DataTable
dt.Rows.Add(row)
Next elem
Currently it is working as designed BUT it takes a very long time to process. 9500 rows with 7 fields is taking about an hour and a half to insert into a DataTable. Is this expected performance? Is there something I can do to speed up performance?

From what I have gathered, Selenium is great at automation but parsing HTML is not a strong suit. So for the node parsing I am using HTML Agility Pack and that does it in under 5 minutes.

Related

Fetch parent of a specific row in a table without iteration

Consider the below table structure contains many rows with multiple column values. I need to identify the parent of specific row, which has to be identified using the cell .
<table class = 'grid'>
<thead id = 'header'>
</thead>
<tbody>
<tr>
<td>
<span class="group">
<span class="group__link"><a class="disabledlink"">copy</a>
</span>
</span>
</td>
<td class="COLUMNNAME">ACE</td>
<td class="COLUMNLONGNAME">Adverse Childhood Experiences</td>
<li>Family Medicine</li>
<li>General Practice</li>
</td>
<td class="COLUMNSEXFILTER">Both</td>
<td class="COLUMNAGEFILTERMIN">Any</td>
<td class="COLUMNTYPE">Score Only</td>
</tr>
<tr>
<td class="nowrap" showactionitem="2">
<span class="group">
<span class="group__link"><a onclick="Check()" href="#">copy</a>
</span>
</span>
</td>
<td class="COLUMNNAME">AM-PAC</td>
<td class="COLUMNLONGNAME">AM-PAC Generic Outpatient Basic Mobility Short Form</td>
<td class="COLUMNNOTE"></td>
<td class="COLUMNRESTRICTEDYN">No</td>
<td class="COLUMNSPECIALTYID"></td>
<td class="COLUMNSEXFILTER">Both</td>
<td class="COLUMNAGEFILTERMIN">Any</td>
<td class="COLUMNTYPE">Score Only</td>
</tr>
<tr></tr>
<tr></tr>
</tbody></thead>
</table>
Likewise this table contains around 100 rows. I did the same using iteration and it is working fine.
Is it possible to find the parent of specific row without iteration?
You can use the parent method to find the parent of an element. Assuming that you have located a table cell, let's call it cell, you can get its row using parent and then the parent of the row with another call to parent:
cell.parent
#=> a <tr> element
cell.parent.parent
#=> the parent of the specific row - a <tbody> element in this case
Chaining multiple parent calls can become tedious and difficult to maintain. For example, you would have to call parent 4 times to get the table cell of the "copy" link. If you are after an ancestor (ie not immediate parent), you are better off using XPath:
cell.table(xpath: './ancestor::table')
#=> the <table> element containing the cell
browser.link(text: 'copy').tr(xpath: './ancestor::tr')
#=> the <tr> element containing a copy link
Hopefully Issue 451 will be implemented soon, which will remove the need for XPath. You would be able to call:
cell.parent(tag_name: 'table') # equivalent to `cell.table(xpath: './ancestor::table')`
There's no need for anything fancy, Watir has an Element#parent method.
You can use this one:
parent::node()
The below example will selects the parent node of the input tag of Id='email'.
Ex: //input[#id='email']/parent::*
the above can also be re-written as
//input[#id='email']/..
XPath tutorial for Selenium

PDF Layout with MigraDoc

I am trying to achieve following matrix kind of layout:
TABLE1,1 TABLE1,2
CHART2,1 TABLE2,2
TABLE3 --> occupies whole row
CHART4 --> ocupies whole row
CHART5,1 CHART5,2
................. List goes on...
These components may span over multiple pages. What is the best way to have them side by side and still be able to view them in MigraDoc.
CHART5,1 could be a combination of 4 charts in one cell.
In HTML view I can use following analogy:
<TABLE>
<TR>
<TD>TABLE1,1</TD> <TD>TABLE1,2 </TD>
</TR>
<TR>
<TD>CHART2,1</TD> <TD>TABLE2,2 </TD>
</TR>
<TR>
<TD>TABLE3</TD colspan =2>
</TR>
<TR>
<TD>CHART4</TD colspan =2>
</TR>
<TR>
<TD>CHART5,1</TD> <TD>CHART5,2 </TD>
</TR>
</TABLE>
The MigraDoc equivalent for colspan=2 is MergeRight=1. This is a property of the Cell class.

Scraping page with correct xpath using Mechanize and nokogiri

I am trying to access data contained in a table that is itself contained in a table with class ='L1'.
So basically my html structure is like this:
<table class="L1">
<table>
<tr></tr>
<tr>
<td></td>
<td>data</td>
</tr>
<tr>
<td></td>
<td>data</td>
</tr>
...ect...ect
</table>
</table>
I need to catch the data contained in a all <a> </a> that are in the second contained in <tr> </tr> but only starting with the second <tr> of the table.
So far I came up with that:
html_body = Nokogiri::HTML(body)
links = html_body.css('.L1').xpath("//table/tbody/tr/td[2]/a[1]")
But seems to me that this doesn't express the fact that I want to start only after the second <tr> (second <tr> included?
What would be the right code to do this ?
You can use position() to select the later elements that you want.
html_body = Nokogiri::HTML(body)
links = html_body.css('.L1').xpath("//table/tbody/tr[position()>1]/td[2]/a[1]")
As the comments on that SO answer say, remember XPath counts from 1, so >1 skips the first tr.

xpath: searching a node in a html table row (multiple conditions)

Looking for a xpath node whose table row must fulfill several conditions
Searching for those node "col_functions" whose table row values is "John Wayne" from the table #class="table_list".
("col_functions", "col_firstname" and "col_lastname are sibling nodes and childs from the table)
<table class="table_list">
<tbody>
<tr>
<td class="col_firstname">John</td>
<td class="col_lastname">Lennon</td>
<td class="col_functions"></td>
</tr>
<tr>
<td class="col_firstname">John</td>
<td class="col_lastname">Wayne</td>
<td class="col_functions"></td> <=== looking for this node!!
</tr>
<tr>
<td class="col_firstname">Wayne</td>
<td class="col_lastname">John</td>
<td class="col_functions"></td>
</tr>
</tbody>
<table>
One option would be to check for class names all over the place:
//table[#class="table_list"]//tr[td[#class="col_firstname"] = "John" and td[#class="col_lastname"] = "Wayne"]/td[#class="col_functions"]/text()
Here we are basically checking all rows inside the table for cells with first name John and last name Wayne, getting the cell with col_functions as an output.
Using siblings it will be like that:
//table[#class='table_list']//td[#class='col_firstname'][text()='John']/following-sibling::td[#class='col_lastname'][text=()'Wayne']/following-sibling::td[#class='col_functions']

Import data from HTML page using feeds importer in drupal

I'm trying to import some data from a HTML page with feeds importer. The context is this:
<table class="tabela">
<tr valign="TOP">
<td class="formulario-legenda">Nome:</td>
<td nowrap="nowrap">
<b>Raul Fernando de Almeida Moreira Vidal</b>
</td>
</tr>
<tr valign="TOP">
<td class="formulario-legenda">Sigla:</td>
<td>
<b>RMV</b>
</td>
</tr>
<tr valign="TOP">
<td class="formulario-legenda">Código:</td>
<td>206415</td>
</tr>
<tr valign="TOP">
<td class="formulario-legenda">Estado:</td>
<td>Ativo</td>
</tr>
</table>
<table>
<tr>
<td class="topo">
<table>
<tr>
<td class="formulario-legenda">Categoria:</td>
<td>Professor Associado</td>
</tr>
<tr>
<td class="formulario-legenda">Carreira:</td>
<td>Pessoal Docente de Universidades</td>
</tr>
<tr>
<td class="formulario-legenda">Grupo profissional:</td>
<td>Docente</td>
</tr>
<tr valign="TOP">
<td class="formulario-legenda">Departamento:</td>
<td>
<a href="uni_geral.unidade_view?pv_unidade=151"
title="Departamento de Engenharia Informática">Departamento de Engenharia Informática</a>
</td>
</tr>
</table>
</td>
</tr>
</table>
I tried with this:
/html/body/div/div/div/div/div/div/div/table/tbody/tr/td/table/tbody/tr[1]/td[2]
but nothing appears. Can someone help me with the right syntax to obtain "Grupo Profissional"?
Quick answer that might work
Considering just the HTML sample you provided (which only has two tables) you can select the text you want using this expression, based on the table's position:
//table[2]//tr[3]/td[1]/text()
This will work in the HTML you pasted above. But it might not work in your actual scenario, since you might have other tables, the table you want to select has no ID and you didn't suggest some invariant text in your code which could be used to anchor the context for the expression. Assuming the initial part of your XPath expression (the div sequence) is correct, you might be able to use:
/html/body/div/div/div/div/div/div/div/table[2]//tr[3]/td[1]/text()
But it's wuite a fragile expression and vulnerable to any changes in the document.
A (possibly) better solution
A better alternative is to look for some identifier you could use. I can only guess, since I don't know your code. In your sample code, I would guess that Codigo and the number following it 206415 might be some identifier. If it is, you could use it to anchor your context. First you select it:
//table[.//td[text()='Código:']/following-sibling::td='206415']
The expression above will select the table which contains a td with the exact text Código: followed by a td containing the exact text 206415. This will create a unique context (considering that the number is an unique identifier). From that context, you can now select the text you want, which is inside the next table (following-sibling::table[1]). This is the context of the second table:
//table[.//td[text()='Código:']/following-sibling::td='206415']/following-sibling::table[1]
And this should select the text you want (Grupo profissional:) which is in the third row tr[3] and first cell/column td[1] of that table:
//table[.//td[text()='Código:']/following-sibling::td='206415']/following-sibling::table[1]//tr[3]/td[1]/text()

Resources