I'm using selenium-webdriver and i'm looking for something really simple but I didn't find it in the documentation.
This is a part of my code :
browser = Selenium::WebDriver.for :firefox, :profile => profile
browser.navigate.to 'an_url'
# I find ancestor element
browser.find_element(:id, "displaylinks").find_element(:id, "link0").find_element(:class, "link-center")
# but I want to click the first link child of this last element
Any ideas ?
There are a couple of different selectors that you could use:
ancestor_element = browser.find_element(:id, "displaylinks").find_element(:id, "link0").find_element(:class, "link-center")
#Using tag_name
ancestor_element.find_element(:tag_name, 'a').click
#Using css-selector
ancestor_element.find_element(:css, 'a').click
#Using xpath-selector (direct child)
ancestor_element.find_element(:xpath, './a').click
#Using xpath-selector (anywhere in ancestor)child)
ancestor_element.find_element(:xpath, './/a').click
Related
I'm new to Ruby automation test using cucumber and selenium-webdriver.
I received the source code automation from another guy. The way he find element on page some thing looks like:
element(:error_message) { browser.elements(class: 'input-invalidate') }
Now I need to access the element inside a ShadowRoot, did some researches but could not get answer for the Ruby code.
Below picture is an example, I'd like to get the div tag with id="maincontainer" inside that shadowroot, anyone can help please?
Thanks you
This is now supported in Selenium 4.0, here's a working example:
driver.get('http://watir.com/examples/shadow_dom.html')
shadow_host = driver.find_element(id: 'shadow_host')
shadow_root = shadow_host.shadow_root
shadow_content = shadow_root.find_element(id: 'shadow_content')
Also verify that you have selenium devtools gem installed, and at least chrome v96 with newest chromedriver
This will work like Titus pointed out when locating element with Selenium
browser = Watir::Browser.new
browser.goto "http://watir.com/examples/shadow_dom.html"
shadow_host = browser.driver.find_element(id: 'shadow_host')
shadow_root = shadow_host.shadow_root
shadow_content = shadow_root.find_element(id: 'shadow_content')
You can also locate element with Watir and then call shadow_root on underlying Selenium element
shadow_host = browser.div(id: 'shadow_host') #Watir::Div
shadow_root = shadow_host.wd.shadow_root
edit: this should also work in theory - converting ShadowRoot to Watir element, but it breaks afterwards.
browser.goto "http://watir.com/examples/shadow_dom.html"
shadow_host = browser.div(id: 'shadow_host') #Watir::Div
shadow_root = shadow_host.wd.shadow_root #Selenium::WebDriver::ShadowRoot
watir_shadow = browser.div(element: shadow_root) #Watir::Div
watir_shadow.divs.count #undefined method `keys' for nil:NilClass
I might be doing something wrong :) best to ask #titusfortner
/watirs/watir-7.1.0/lib/watir/locators/element/selector_builder.rb:73:in `merge_scope?'
/watirs/watir-7.1.0/lib/watir/locators/element/selector_builder.rb:50:in `normalize_selector'
/watirs/watir-7.1.0/lib/watir/locators/element/selector_builder.rb:28:in `build'
/watirs/watir-7.1.0/lib/watir/element_collection.rb:47:in `build'
/watirs/watir-7.1.0/lib/watir/element_collection.rb:18:in `initialize'
/watirs/watir-7.1.0/lib/watir/container.rb:28:in `new'
/watirs/watir-7.1.0/lib/watir/container.rb:28:in `elements'
I want the text "HEATMAPS" to be clicked after the webpage is opened. I have tried number of click methods, including recognizing as hyperlink, as text, as using xpath etc. None of them worked. I feel, I am either misunderstanding the links, as to be a hyperlink or choosing a wrong xpath.
Link of the web page
PFB the code below
require 'watir-webdriver'
require 'watir-ng'
WatirNg.patch!
WatirNg.register(:ng_scope).patch!
browser = Watir::Browser.new
browser.goto 'http://app.vwo.com/#/campaign/108/summary? token=eyJhY2NvdW50X2lkIjoxNTA3MzQsImV4cGVyaW1lbnRfaWQiOjEwOCwiY3JlYXRlZF9vbiI6MTQ0NDgxMjQ4MSwidHlwZSI6ImNhbXBhaWduIiwidmVyc2lvbiI6MSwiaGFzaCI6IjJmZjk3OTVjZTgwNmFmZjJiOTI5NDczMTc5YTBlODQxIn0='
lin = browser.link :text=> 'HEATMAPS'
lin.exist?
lin.click
Can someone please guide me on this, as to how I can make that link with the text "HEATMAPS" in the page get clicked.
The error i get:
`This code has slept for the duration of the default timeout waiting for an Element to exist. If the test is still passing, consider using Element#exists? instead of rescuing UnknownObjectException
C:/Ruby22/lib/ruby/gems/2.2.0/gems/watir-6.1.0/lib/watir/elements/element.rb:507:in `rescue in wait_for_exists': timed out after 30 seconds, waiting for {:text=>"HEATMAPS", :tag_name=>"a"} to be located (Watir::Exception::UnknownObjectException)
from C:/Ruby22/lib/ruby/gems/2.2.0/gems/watir-6.1.0/lib/watir/elements/element.rb:497:in `wait_for_exists'
from C:/Ruby22/lib/ruby/gems/2.2.0/gems/watir-6.1.0/lib/watir/elements/element.rb:515:in `wait_for_present'
from C:/Ruby22/lib/ruby/gems/2.2.0/gems/watir-6.1.0/lib/watir/elements/element.rb:533:in `wait_for_enabled'
from C:/Ruby22/lib/ruby/gems/2.2.0/gems/watir-6.1.0/lib/watir/elements/element.rb:656:in `element_call'
from C:/Ruby22/lib/ruby/gems/2.2.0/gems/watir-6.1.0/lib/watir/elements/element.rb:114:in `click'
from C:/Users/Mrityunjeyan/Documents/GitHub/Simpleprograms/webautomation.rb:10:in `<main>'`
This would display me the inner_html text but still wouldnt click
lin = browser.span(:class => 'ng-scope').inner_html
puts lin
The problem is that the link's text is not "HEATMAPS". It is actually "Heatmaps". The text locator is case-sensitive, which means you need:
lin = browser.link :text=> 'Heatmaps'
You can see this if you inspect the HTML:
<a ui-sref="campaign.heatmap-clickmap" href="#/analyze/analysis/108/heatmaps?token=eyJhY2NvdW50X2lkIjoxNTA3MzQsImV4cGVyaW1lbnRfaWQiOjEwOCwiY3JlYXRlZF9vbiI6MTQ0NDgxMjQ4MSwidHlwZSI6ImNhbXBhaWduIiwidmVyc2lvbiI6MSwiaGFzaCI6IjJmZjk3OTVjZTgwNmFmZjJiOTI5NDczMTc5YTBlODQxIn0%3D">
<!-- boIf: isAnalyticsCampaign -->
<span bo-if="isAnalyticsCampaign" class="ng-scope">Heatmaps</span>
<!-- boIf: !isAnalyticsCampaign -->
</a>
It only looks like "HEATMAPS" due to the styling. One of the styles includes a text-transform: uppercase; which visually capitalizes the text. Watir does not interpret the styles, so only knows that the text node is "Heatmaps".
Once you have identified the link, clicking still has a problem:
lin.click
#=> Selenium::WebDriver::Error::UnknownError:
#=> unknown error: Element <a ui-sref="analyze.heatmaps" ng-class="{selected: (locationContains('analyze', 'heatmap') && !locationContains('analyze', '/analysis') && state.current.name !== 'campaign.heatmap-clickmap') || (isAnalyzeHeatmapEnabled && (isAnalyzeDeprecatedHeatmapView || locationContains('analyze', '/analysis', 'heatmaps')) )}" data-qa="nav-main-analyze-heatmap" href="#/analyze/heatmap">...</a>
#=> is not clickable at point (90, 121).
#=> Other element would receive the click: <div ng-show="!isCROSetupView" class="">...</div>
The link being located is actually the one in the left menu, which is disabled, rather than the top menu. You need to scope the link locator to just the top menu. Using the parent ul element appears to be sufficient:
lin = browser.ul(class: 'page-nav').link(text: 'Heatmaps')
lin.click
To see the click complete, you might want to tell Chrome not to close at the end of the script. This is done by opening the browser using the following option:
caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {'detach' => true})
browser = Watir::Browser.new :chrome, desired_capabilities: caps
Your target link doesn't have any text. I confirmed with nokogiri that the text of the <a> tag includes the text inside the child <span> tag, and it turns out Watir works the same way.
If you use a Chrome browser, you can select:
View > Developer > Developer tools
Then you can select an element on the page, and the corresponding section in the html will be highlighted. Here is what the html looks like:
<a ui-sref="campaign.heatmap-clickmap"
href="#/analyze/analysis/108/heatmaps?token=eyJhY2NvdW50X2lkIjoxNTA3MzQsImV4cGVyaW1lbnRfaWQiOjEwOCwiY3JlYXRlZF9vbiI6MTQ0NDgxMjQ4MSwidHlwZSI6ImNhbXBhaWduIiwidmVyc2lvbiI6MSwiaGFzaCI6IjJmZjk3OTVjZTgwNmFmZjJiOTI5NDczMTc5YTBlODQxIn0%3D">
<!-- boIf: isAnalyticsCampaign -->
<span bo-if="isAnalyticsCampaign" class="ng-scope">Heatmaps</span>
<!-- boIf: !isAnalyticsCampaign --> </a>
The basic structure is:
<a><span>Heatmaps</span></a>
Chrome even has a feature where you can right click on an element and get its xpath, which you can use with Watir.
However, when I execute my program to go to that page, I see Chrome launch, and then I'm presented with a login page rather than the page that was presented to me at your link.
Arghhh...what a colossal waste of time. I thought Watir must be broken. With the proper url, I can get the link using several different techniques:
1)
require 'watir'
require 'watir-ng' #<=NOTE THIS
WatirNg.register(:'bo_if').patch! #<= NOTE THIS
br = Watir::Browser.new :chrome
br.goto 'http://app.vwo.com/#/analyze/analysis/108/summary?token=eyJhY2NvdW50X2lkIjoxNTA3MzQsImV4cGVyaW1lbnRfaWQiOjEwOCwiY3JlYXRlZF9vbiI6MTQ0NDgxMjQ4MSwidHlwZSI6ImNhbXBhaWduIiwidmVyc2lvbiI6MSwiaGFzaCI6IjJmZjk3OTVjZTgwNmFmZjJiOTI5NDczMTc5YTBlODQxIn0%3D'
target_link = br.span(
class: 'ng-scope',
'bo_if': 'isAnalyticsCampaign',
).parent #<= NOTE THIS
puts target_link.text #HEATMAPS
puts target_link.href #http://app.vwo.com/#/analyze/analysis/1.....
2)
require 'watir'
require 'watir-ng' #<=NOTE THIS
WatirNg.register(:ui_sref).patch! #<=NOTE THIS
br = Watir::Browser.new :chrome
br.goto 'http://app.vwo.com/#/analyze/analysis/108/summary?token=eyJhY2NvdW50X2lkIjoxNTA3MzQsImV4cGVyaW1lbnRfaWQiOjEwOCwiY3JlYXRlZF9vbiI6MTQ0NDgxMjQ4MSwidHlwZSI6ImNhbXBhaWduIiwidmVyc2lvbiI6MSwiaGFzaCI6IjJmZjk3OTVjZTgwNmFmZjJiOTI5NDczMTc5YTBlODQxIn0%3D'
target_link = br.link(
ui_sref: 'campaign.heatmap-clickmap',
href: '#/analyze/analysis/108/heatmaps?token=eyJhY2NvdW50X2lkIjoxNTA3MzQsImV4cGVyaW1lbnRfaWQiOjEwOCwiY3JlYXRlZF9vbiI6MTQ0NDgxMjQ4MSwidHlwZSI6ImNhbXBhaWduIiwidmVyc2lvbiI6MSwiaGFzaCI6IjJmZjk3OTVjZTgwNmFmZjJiOTI5NDczMTc5YTBlODQxIn0%3D'
)
puts target_link.text #HEATMAPS
puts target_link.href #http://app.vwo.com/#/analyze/analysis/108...
3) Getting the xpath by right clicking on the link in Chrome:
require 'watir'
br = Watir::Browser.new :chrome
br.goto 'http://app.vwo.com/#/analyze/analysis/108/summary?token=eyJhY2NvdW50X2lkIjoxNTA3MzQsImV4cGVyaW1lbnRfaWQiOjEwOCwiY3JlYXRlZF9vbiI6MTQ0NDgxMjQ4MSwidHlwZSI6ImNhbXBhaWduIiwidmVyc2lvbiI6MSwiaGFzaCI6IjJmZjk3OTVjZTgwNmFmZjJiOTI5NDczMTc5YTBlODQxIn0%3D'
target_link = br.link(
xpath: '//*[#id="main-container"]/ul/li[3]/a'
)
puts target_link.text #HEATMAPS
puts target_link.href #http://app.vwo.com/#/analyze/analysis/108....
xpath can handle any tag or attribute name, unlike Watir, so it seems like a good tool in that regard.
4) A less brittle xpath:
require 'watir'
br = Watir::Browser.new :chrome
br.goto 'http://app.vwo.com/#/analyze/analysis/108/summary?token=eyJhY2NvdW50X2lkIjoxNTA3MzQsImV4cGVyaW1lbnRfaWQiOjEwOCwiY3JlYXRlZF9vbiI6MTQ0NDgxMjQ4MSwidHlwZSI6ImNhbXBhaWduIiwidmVyc2lvbiI6MSwiaGFzaCI6IjJmZjk3OTVjZTgwNmFmZjJiOTI5NDczMTc5YTBlODQxIn0%3D'
a_href = "#/analyze/analysis/108/heatmaps?token=eyJhY2NvdW50X2lkIjoxNTA3MzQsImV4cGVyaW1lbnRfaWQiOjEwOCwiY3JlYXRlZF9vbiI6MTQ0NDgxMjQ4MSwidHlwZSI6ImNhbXBhaWduIiwidmVyc2lvbiI6MSwiaGFzaCI6IjJmZjk3OTVjZTgwNmFmZjJiOTI5NDczMTc5YTBlODQxIn0%3D"
target_link = br.link(
xpath: %Q{ //a[#href="#{a_href}"] } +
'[#ui-sref="campaign.heatmap-clickmap"]' +
'[child::span[#class="ng-scope"][#bo-if="isAnalyticsCampaign"][text()="Heatmaps"]]'
)
The xpath looks for an <a> tag with two attributes:
//a[#href="blah"][#ui-sref="bleh"]
which has a child <span> (i.e. a direct child) with three attributes:
[child::span[#class="blih"][#bo-if="bloh"][text()="Heatmaps"]]
After I programmatically click the link:
target_link.click
Watir goes to the next page.
I'm learning my first coding language (Ruby) and I'm having trouble crawling a link. I'm trying to grab image-URLs and eventually going to save to a CSV. I've looked at many tutorials and a lot of question on here, but none seem to solve my problem.
The problem appears to be in the final line (Line 19).
Error Message: 19:in `[]': no implicit conversion of String into Integer
Any help would be appreciated!
require 'rubygems'
require 'nokogiri'
require 'open-uri'
PAGE_URL = "http://www.bestbuy.com/site/samsung-showcase-27-8-cu-ft-french-door-refrigerator-with-thru-the-door-ice-and-water-stainless-steel/5236091.p?id=1219116001631&skuId=5236091"
page = Nokogiri::HTML(open(PAGE_URL))
link_extract = page.css('div#pdp-content div.image-gallery-main-slide a img[data-index="1"]')
puts link_extract[0]['src']
It looks like a large part of this page is built with javascript.
If you look at the raw html you get you won't see the content you're looking for since youre essentially doing a curl to get the data.
Soo how should you proceed? I'd say you have two options:
The easiest way to get the data you want, is to get the json they're using at #pdp-model-data
Use some kind of web driver that works well with javascript. I tried phantomjs but it looks like that page contains some invalid syntax and it's only pulling the first image.
e.g.
require 'capybara/poltergeist'
url = "http://www.bestbuy.com/site/samsung-showcase-27-8-cu-ft-french-door-refrigerator-with-thru-the-door-ice-and-water-stainless-steel/5236091.p?id=1219116001631&skuId=5236091"
session = Capybara::Session.new(:poltergeist)
session.visit(url)
page = Nokogiri::HTML::DocumentFragment.parse(session.html)(session.html)
page.search("#pdp-content div.image-gallery-main-slide a img")
# => [#<Nokogiri::XML::Element:0x3ffe438d4100 name="img" attributes=[#<Nokogiri::XML::Attr:0x3ffe438d409c name="src" value="http://pisces.bbystatic.com/image2/BestBuy_US/images/products/5236/5236091_sa.jpg;canvasHeight=500;canvasWidth=500">, #<Nokogiri::XML::Attr:0x3ffe438d4088 name="alt" value="Samsung - Showcase 27.8 Cu. Ft. French Door Refrigerator with Thru-the-Door Ice and Water - Stainless-Steel - Larger Front">, #<Nokogiri::XML::Attr:0x3ffe438d4074 name="width" value="500">, #<Nokogiri::XML::Attr:0x3ffe438d4060 name="height" value="500">, #<Nokogiri::XML::Attr:0x3ffe438d404c name="data-index" value="0">, #<Nokogiri::XML::Attr:0x3ffe438d4038 name="style" value="display: block; ">]>]
edit #1:
require 'nokogiri'
require 'open-uri'
require 'json'
url = "http://www.bestbuy.com/site/samsung-showcase-27-8-cu-ft-french-door-refrigerator-with-thru-the-door-ice-and-water-stainless-steel/5236091.p?id=1219116001631&skuId=5236091"
page = Nokogiri::HTML(open(url))
json = page.css('#pdp-model-data').attribute('data-gallery-images').value
gallery = JSON.parse(json, symbolize_names: true)
puts gallery[1][:url]
#=> "http://img.bbystatic.com/BestBuy_US/images/products/5236/5236091cv1a.jpg"
I'm trying to write a script that will automatically click a button when entering a site.
The HTML of the site is as follows:
<span id="zolaDisclaimerButton" class="dijitReset dijitStretch dijitButtonContents" waistate="labelledby-zolaDisclaimerButton_label" wairole="button" dojoattachpoint="titleNode,focusNode" role="button" aria-labelledby="zolaDisclaimerButton_label" tabindex="0" title="I acknowledge this disclaimer... Let ZoLa begin!" style="-moz-user-select: none;">
I have tried the following and they don't work:
browser.span(:id, "zolaDisclaimerButton").click
browser.button(:id, "zolaDisclaimerButton").click
How do I go about clicking those types of button? The URL in question is:
http://gis.nyc.gov/doitt/nycitymap/template?applicationName=ZOLA
EDIT: this is the code I use:
require "watir"
browser = Watir::Browser.new
browser.goto "http://gis.nyc.gov/doitt/nycitymap/template?applicationName=ZOLA"
browser.span(:id, "zolaDisclaimerButton").click
puts "fin"
It navigates to the page, doesn't click anything, then prints 'fin' (to let me know it's done). No exception is thrown.
If the page is not loading in time, you can add waits - see http://watirwebdriver.com/waiting/. These wait methods will wait until either the condition is met or a time limit is exceeded. It is better than using sleep since it only waits as long as needed (rather than waiting 2 seconds for something that loads in 1 second).
In this case, use the when_present method on the span before clicking it:
require "watir"
browser = Watir::Browser.new
browser.goto "http://gis.nyc.gov/doitt/nycitymap/template?applicationName=ZOLA"
browser.span(:id, "zolaDisclaimerButton").when_present.click
puts "fin"
I'm new to Ruby and for my first scripting assignment, I've been asked to write a web scraping script to grab elements of our DNS listings from GoDaddy.
Having issues with scraping the links and then I need to follow the links. I need to get the link from the "GoToSecondaryDNS" js element below. I'm using Mechanize and Nokogiri:
<td class="listCellBorder" align="left" style="width:170px;">
<div style="padding-left:4px;">
<div id="gvZones21divDynamicDNS"></div>
<div id="gvZones21divMasterSlave" cicode="41022" onclick="GoToSecondaryDNS('iwanttoscrapethislink.com',0)" class="listFeatureButton secondaryDNSNoPremium" onmouseover="ShowSecondaryDNSAd(this, event);" onmouseout="HideAdInList(event);"></div>
<div id="gvZones21divDNSSec" cicode="41023" class="listFeatureButton DNSSECButtonNoPremium" onmouseover="ShowDNSSecAd(this, event);" onmouseout="HideAdInList(event);" onclick="UpgradeLinkActionByID('gvZones21divDNSSec'); return false;" useClick="true" clickObj="aDNSSecUpgradeClicker"></div>
<div id="gvZones21divVanityNS" onclick="GoToVanityNS('iwanttoscrapethislink.com',0)" class="listFeatureButton vanityNameserversNoPremium" onmouseover="ShowVanityNSAd(this, event);" onmouseout="HideAdInList(event);"></div>
<div style="clear:both;"></div>
</div>
</td>
How can I scrape the link 'iwanttoscrapethislink.com' and then interact with the onclick to follow the link and scrape content on the following page with Ruby?
So far, I have a simple start to the code:
require 'rubygems'
require 'mechanize'
require 'open-uri'
def get_godaddy_data(url)
web_agent = Mechanize.new
result = nil
### login to GoDaddy admin
page = web_agent.get('https://dns.godaddy.com/Default.aspx?sa=')
## there is only one form and it is the first form on thepage
form = page.forms.first
form.username = 'blank'
form.password = 'blank'
## form.submit
web_agent.submit(form, form.buttons.first)
site_name = page.css('div.gvZones21divMasterSlave onclick td')
### export dns zone data
page = web_agent.get('https://dns.godaddy.com/ZoneFile.aspx?zone=' + site_name + '&zoneType=0&refer=dcc')
form = page.forms[3]
web_agent.submit(form, form.buttons.first).save(uri.host + 'scrape.txt')
## end
end
### read export file
##return File.open(uri.host + 'scrape.txt', 'rb') { |file| file.read }
end
def scrape_dns(url)
site_name = page.css('div.gvZones21divMasterSlave onclick td')
LIST_URL = "https://dns.godaddy.com/ZoneFile.aspx?zone=" + site_name + '&zoneType=0&refer=dcc"
page = Nokogiri::HTML(open(LIST_URL))
#not sure how to scrape onclick urls and then how to click through to continue scraping on the second page for each individual DNS
end
You can't interact with "onclick" because Nokogiri isn't a JavaScript engine.
You can extract the contents and then use that as the URL for a subsequent web request. Assuming doc contains the parsed HTML:
doc.at('div[onclick^="GoToSecondaryDNS"]')['onclick']
will give you the value for the onclick parameter. ^= means "find the word starting with", so that lets us rule out other <div> tags with onclick parameters and returns:
"GoToSecondaryDNS('iwanttoscrapethislink.com',0)"
Using a simple regex [/'(.+)'/,1] will get you the hostname:
doc.at('div[onclick^="GoToSecondaryDNS"]')['onclick'][/'(.+)'/,1]
=> "iwanttoscrapethislink.com"
The rest, such as how to get access to Mechanize's internal Nokogiri document, and how to create the new URL, are left for you to figure out.