How to iterate through web tables using selenium and ruby - ruby

I want to iterate through an html table with n number rows and columns as follows:
<table class='table'>
<tbody>
<tr>
<td>Spratly Islands</td>
<td>Vietnam</td>
<td>Azerbaijan</td>
<td>Georgia</td>
</tr>
<tr>
<td>Sri Lanka</td>
<td>Israel</td>
<td>Cyprus</td>
<td>Yemen</td>
</tr>
<tr>
<td>Maldives</td>
<td>Kuwait</td>
<td>West Malaysia</td>
<td>Nepal</td>
</tr>
...
</tbody>
</table>
I want to get the column names for each row using xpath and print it. How to do this in ruby?
Thanks,
RV

To Iterate the table in ruby, Use the following code
I assume the first row is in index 1.
driver.find_elements(xpath: "//table[#class='table']//tr").each.with_index(1) do |_,index|
driver.find_elements(xpath: "//table[#class='table']//tr[#{index}]/td").each do |cell|
puts cell.text
end
puts '*****'
end
And I suggest you to move WATIR which is very nice wrapper for Ruby Selenium-Binding which actually has the syntax for table iteration,
In WATIR, you could do,
b.table(class: 'table').rows.each do |row|
row.cells.each do |cell|
puts cell.text
end
puts '*****'
end

Related

How to use Conditional in Nokogiri

Is there a way to put No Url Foud in a blank or missing anchor tag.
The reason of asking this is that the textnode output 50 textnode but the url only output 47 as some of the anchor is missin or not availble, causing the next list to colaps and completely ruin the list
see the screenshots td tag|Td list
I could get the textNode and the attributes the only problem here is some of the td list has a missing anchor causing the other list to collapse
<table>
<tr>
<td>TextNode</td>
</tr>
<tr>
<td>TextNode</td>
</tr>
<tr>
<td>TextNode</td>
</tr>
<tr>
<td>TextNode With No Anchor</td>
</tr> <tr>
<td>TextNode</td>
</tr>
<tr>
<td>TextNode With No Anchor</td>
</tr>
</table>
company_name = page.css("td:nth-child(2)")
company_name.each do |line|
c_name = line.text.strip
# this will output 50 titles
puts c_name
end
directory_url = page.css("td:nth-child(1) a")
directory_url.each do |line|
dir_url = line["href"]
# this will output 47 Urls since some list has no anchor tag.
puts dir_url
end
You can't find things that aren't there. You have to find things that are there, and then search within them for elements that may or may not be present.
Like:
directory = page.css("td:nth-child(1)")
directory.each do |e|
anchor = e.css('a')
puts anchor.any? ? anchor[0]['href'] : '(No URL)'
end

How to iterate over rows and print out CSV::Table values in Sinatra

I'm making a simple Sinatra app and have the following 'numbers.csv' file:
Name,Title,Phone
Ally Smith,Manager,888-552-444
Rick Jones,Director,888-552-447
Hayley Bowman,Accountant,888-552-424
The task is to print the name and phone numbers in an erb file.
I originally solved this problem with nested arrays:
get '/' do
# turns CSV into array with nested arrays
#names = CSV.read('names.csv')
# delete headers
#names.shift
erb :index
end
and an each loop:
<tr>
<th>Name</th>
<th>Number</th>
</tr>
<% #names.each do |row| %>
<tr>
<td> <%= row[0] %> </td>
<td> <%= row[2] %> </td>
<tr>
<% end %>
Now I am trying to overcome my fear of working with objects and hashes, so I am approaching the problem like this:
get '/' do
# turns CSV into object with key value pairs
#numbers = CSV.read('numbers.csv', headers:true)
erb :index
end
I know that the benefit of using a CSV::Table object with headers means that I can access the values like so:
#names[i]["Name"]
#names[i]["Phone"]
where i is the counter for each row, followed by the key names.
I tried writing a while loop in the erb file – fail. Also a foreach loop that I understand is as CSV method and modified my code accordingly – fail.
I'm having trouble iterating over the rows and calling the specific keys.
I'm not sure if I am using the right loops.
Any advice would be greatly appreciated.
I have been working on this for hours :(
The code is fine. As i show below:
> numbers = CSV.read('numbers.csv', headers:true)
> numbers.each do |row|
> puts "#{row['Name']} - #{row['Phone']}"
> end
Ally Smith - 888-552-444
Rick Jones - 888-552-447
Hayley Bowman - 888-552-424
I think your problem is your variables. you should use #numbers instead of #names in your second example:
#names[i]["Name"] # should be #numbers[i]['Name']

'html-table' gem and issues with HTML::Table::Head

I'm creating two HTML tables. The first one is perfect, and the second one has the same HEAD as the first table no matter what I do.
Here's the problematic code:
require 'html/table'
include HTML
title1 = [1,2,3]
data1 = [1,2,3]
table1 = HTML::Table.new
table1.push Table::Head.create{ |row| row.content = title1 }
data1.each { |entry| table1.push Table::Row.new{|row| row.content = entry}}
title2 = [1,2]
data2 = [1,2]
table2 = HTML::Table.new
table2.push Table::Head.create{ |row| row.content = title2}
data2.each { |entry| table2.push Table::Row.new{ |row| row.content = entry}
}
This is the result from puts table1.html:
<table>
<thead>
<tr>
<td>1</td>
<td>2</td>
<td>3</td>
</tr>
</thead>
<tr>
<td>1</td>
</tr>
<tr>
<td>2</td>
</tr>
<tr>
<td>3</td>
</tr>
</table>
This is the result from puts table2.html:
<table>
<thead>
<tr>
<td>1</td>
<td>2</td>
<td>3</td>
</tr>
</thead>
<tr>
<td>1</td>
</tr>
<tr>
<td>2</td>
</tr>
</table>
There are no issues with the content but HEAD looks the same in both tables. Why?
EDIT:
I've simplified the initial code a bit:
`require 'html/table'
include HTML
s= Table::Head.create{ |row| row.content = 1 }
m= Table::Head.create{ |row| row.content = 2 }
puts s
<td>1</td>
puts m
<td>1</td>`
puts .inspect shows that both variables store same instance object>
puts s.inspect
puts m.inspect
[[#<HTML::Table::Row::Data:0x007ff52b096e38 #html_begin="<td", #html_body="1", #html_end="</td>">]]
[[#<HTML::Table::Row::Data:0x007ff52b096e38 #html_begin="<td", #html_body="1", #html_end="</td>">]]
Until now I was not aware of this gem and I don't understand the reason why someone would want to add this to his codebase. What value does it add? What do you accomplish WITH the gem that you can not accomplish WITHOUT?
The reason you see the same output is because it is the same object that is returned from create:
require 'html/table'
include HTML
s = Table::Head.create{ |row| row.content = 1 }
m = Table::Head.create{ |row| row.content = 2 }
puts s.object_id == m.object_id # => true
If you look at the source code (https://github.com/djberg96/html-table/blob/master/lib/html/head.rb#L18) then it is clear that this is intended behavior:
# This is our constructor for Head objects because it is a singleton
# class. Optionally, a block may be provided. If an argument is
# provided it is treated as content.
#
def self.create(arg=nil, &block)
##head = new(arg, &block) unless ##head
##head
end
According to this code and the comment a <thead> is a singleton and should only exist once.
Without looking any further into the library and how it works: IMHO treating <thead> as a singleton is plain wrong and a reason to stop using this library right away. You could contact the author if you are curious.
As a rule of thumb: when there is class level variables (##) then there is trouble.
So what can you do?
You need to create a HTML table outside of a web framework like Rails? You could:
Use ERB: http://ruby-doc.org/stdlib-2.3.1/libdoc/erb/rdoc/ERB.html
HTML is "just XML (tm)" so you can use REXML: http://ruby-doc.org/stdlib-1.9.3/libdoc/rexml/rdoc/REXML.html
These are both "built in" solutions. Available in your Ruby right away. But you could also use a different templating solution (haml, slim, ...) or, because REXML interface is not the most straightforward i think, another XML generator (ox, oga, nokogiri) or builder/xml.

match table row id's with a common prefix

This might be merely a syntax question.
I am unclear how to match only table rows whose id begins with rowId_
agent = Mechanize.new
pageC1 = agent.get("/customStrategyScreener!list.action")
The table has class=tableCellDT.
pageC1.search('table.tableCellDT tr[#id=rowId_]') # parses OK but returns 0 rows since rowId_ is not matched exactly.
pageC1.search('table.tableCellDT tr[#id=rowId_*]') # Throws an error since * is not treated like a wildcard string match
EXAMPLE HTML:
<table id="row" cellpadding="5" class="tableCellDT" cellspacing="1">
<thead>
<tr>
<th class="tableHeaderDT">#</th>
<th class="tableHeaderDT sortable">
Screener</th>
<th class="tableHeaderDT sortable">
Strategy</th>
<th class="tableHeaderDT"> </th></tr></thead>
<tbody>
<tr id="rowId_BullPut" class="odd">
<td> 1 </td>
<td> Bull</td>
<td></td>
<td>Edit
Delete
View
</td></tr>
NOTE
pageC1 is a Mechanize::Page object, not a Nokogiri anything. Sorry that wasn't clear at first.
Mechanize::Page doesn't have #css or #xpath methods, but a Nokogiri doc can be extracted from it (used internally anyway).
To get the tr elements that have an id starting with "rowId_":
pageC1.search('//tr[starts-with(#id, "rowId_")]')
You want either the CSS3 attribute starts-with selector:
pageC1.css('table.tableCellDT tr[id^="rowId_"]')
or the XPath starts-with() function:
pageC1.xpath('.//table[#class="tableCellDT"]//tr[starts-with(#id,"rowId_")]')
Although the Nokogiri Node#search method will intelligently pick between CSS or XPath selector syntax based on what you wrote, that does not mean that you can mix both CSS and XPath selector syntax in the same query.
In action:
>> require 'nokogiri'
#=> true
>> doc = Nokogiri.HTML <<ENDHTML; true #hide output from IRB
">> <table class="foo"><tr id="rowId_nonono"><td>Nope</td></tr></table>
">> <table class="tableCellDT">
">> <tr id="rowId_yesyes"><td>Yes1</td></tr>
">> <tr id="rowId_andme2"><td>Yes2</td></tr>
">> <tr id="rowIdNONONO"><td>Needs underscore</td></tr>
">> </table>
">> ENDHTML
#=> true
>> doc.css('table.tableCellDT tr[id^="rowId_"]').map(&:text)
#=> ["Yes1", "Yes2"]
>> doc.xpath('.//table[#class="tableCellDT"]//tr[starts-with(#id,"rowId_")]').map(&:text)
#=> ["Yes1", "Yes2"]
Thanks to
http://nokogiri.org/Nokogiri/XML/Node.html#method-i-css
and the answers above, here is the final code that solves my problem of getting just the rows I need, and then reading only certain information from each one:
pageC1.search('//tr[starts-with(#id, "rowId_")]').each do |row|
# Read the string after _ in rowId_, part of the "id" in <tr>
rid = row.attribute("id").text.split("_")[1] # => "BullPut"
# Get the URL of the 3rd <a> link in <td> cell 4
link = row.css("td[4] a[3]")[0].attributes["href"].text # => "link3?model.itemId=2262&amp;model.source=list"
end

Ruby - traverse through nokogiri element

I have an html like this:
...
<table>
<tbody>
...
<tr>
<th> head </th>
<td> td1 text<td>
<td> td2 text<td>
...
</tr>
</tbody>
<tfoot>
</tfoot>
</table>
...
I'm using Nokogiri with ruby. I want traverse through each row and get the text of th and corresponding td into an hash.
require "nokogiri"
#Parses your HTML input
html_data = "...stripped HTML markup code..."
html_doc = Nokogiri::HTML html_data
#Iterates over each row in your table
#Note that you may need to clarify the CSS selector below
result = html_doc.css("table tr").inject({}) do |all, row|
#Modify if you need to collect only the first td, for example
all[row.css("th").text] = row.css("td").text
end
I didn't run this code, so I'm not absolutely sure but the overall idea should be right:
html_doc = Nokogiri::HTML("<html> ... </html>")
result = []
html_doc.xpath("//tr").each do |tr|
hash = {}
tr.children.each do |node|
hash[node.node_name] = node.content
end
result << hash
end
puts result.inspect
See the docs for more info: http://nokogiri.org/Nokogiri/XML/Node.html

Resources