I'm trying to use the DOM inspector to find the ID of the "More" button at the bottom of this page that reveals more results.
I'm trying to do something like this example:
require 'watir-webdriver'
b = Watir::Browser.new
b.goto 'svpply.com/editors_pick'
#count products
puts b.elements(:xpath => '//li[#data-class="Product"]').count
#=> 30
#Now click button
show_all = b.button(:id => "btn_all")
show_all.click
sleep 4
#count products again
puts b.elements(:xpath => '//li[#data-class="Product"]').count
#=>60
However, I'm unclear on how to search for that particular id within the DOM structure. Can someone also explain the difference between an attribute, element, id, and node?
To use the DOM Inspector for the More button:
Open the DOM Inspector (Ctrl+Shift+I)
Click the [More] button that you want to inspect
On the right side of the DOM Inspector bar, click the [HTML] button
This should show the HTML for the page, which will include the details of the [More] control. You'll notice that the element is actually a DIV not a button. As well that the ID is in the form "_more".
--> This should show the HTML for the page, which will show the details of the [More] control. You'll notice that the element is actually a DIV not a button. As well that the ID is in the form "_more".
So to do your example with the Quora page, you would do something like:
require 'watir-webdriver'
class QuoraPage
def initialize(browser)
#browser = browser
end
def goto()
#browser.goto 'http://www.quora.com/Startups/best_questions'
wait_questions_loaded
end
def click_more()
#browser.div(:id, /_more/).click
wait_questions_loaded
end
def questions_count()
#browser.links(:class, 'question_link').count{ |x| x.visible? }
end
def wait_questions_loaded()
begin
questions_start_count = questions_count()
sleep(2)
end while questions_start_count != questions_count()
end
end
page = QuoraPage.new(Watir::Browser.new :chrome)
page.goto
puts page.questions_count
page.click_more
puts page.questions_count
Note that I had to put the sleeps in otherwise webdriver hangs like anonygoose mentioned. I tried different wait_untils, but did not manage to find something that worked (other than sleep which is not very robust).
Regarding your question about nodes, elements, etc. I think you are best to look at http://www.w3schools.com/dom/default.asp.
To press the button on svpply you can use simply
b.button(:text => "Show All").click
Counting all the products that appear on the page could potentially be done with
b.lis(:class => "grab large").count
This is all for the svpply site. I can't get quora to automate at all, it just stalls my watir-webdriver indefinitely.
You'll also want to wait before you have watir count the products. This can be done with:
b.wait_until{b.lis(:class => "grab large").count > 30}
Related
I have a Ruby application using Selenium Webdriver and Nokogiri. I want to choose a class, and then for each div corresponding to that class, I want to perform an action based on the contents of the div.
For example, I'm parsing the following page:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=puppies
It's a page of search results, and I'm looking for the first result with the word "Adoption" in the description. So the bot should look for divs with className: "result", for each one check if its .description div contains the word "adoption", and if it does, click on the .link div. In other words, if the .description does not include that word, then the bot moves on to the next .result.
This is what I have so far, which just clicks on the first result:
require "selenium-webdriver"
require "nokogiri"
driver = Selenium::WebDriver.for :chrome
driver.navigate.to "https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=puppies"
driver.find_element(:class, "link").click
You can get list of elements that contains "adopt" and "Adopt" by XPath using contains() then use union operator (|) to union results from "adopt" and "Adopt". See code below:
driver = Selenium::WebDriver.for :chrome
driver.navigate.to "https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=puppies"
sleep 5
items = driver.find_elements(:xpath,"//div[#class='g']/div[contains(.,'Adopt')]/h3/a|//div[#class='g']/div[contains(.,'adopt')]/h3/a")
for element in items
linkText = element.text
print linkText
element.click
end
The pattern to handle each iteration will be determined by the type of action executed on each item. If the action is a click, then you can't list all the links to click on each of them since the first click will load a new page, making the elements list obsolete.
So If you wish to click on each link, then one way is to use an XPath containing the position of the link for each iteration:
# iteration 1
driver.find_element(:xpath, "(//h3[#class='r']/a)[1]").click # click first link
# iteration 2
driver.find_element(:xpath, "(//h3[#class='r']/a)[2]").click # click second link
Here is an example that clicks on each link from a result page:
require 'selenium-webdriver'
driver = Selenium::WebDriver.for :chrome
wait = Selenium::WebDriver::Wait.new(timeout: 10000)
driver.navigate.to "https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=puppies"
# define the xpath
search_word = "Puppies"
xpath = ("(//h3[#class='r']/a[contains(.,'%s')]" % search_word) + ")[%s]"
# iterate each result by inserting the position in the XPath
i = 0
while true do
# wait for the results to be loaded
wait.until {driver.find_elements(:xpath, "(//h3[#class='r']/a)[1]").any?}
# get the next link
link = driver.find_elements(:xpath, xpath % [i+=1]).first
break if !link
# click the link
link.click
# wait for a new page
wait.until {driver.find_elements(:xpath, "(//h3[#class='r']/a)[1]").empty?}
# handle the new page
puts "Page #{i}: " + driver.title
# return to the main page
driver.navigate.back
end
puts "The end!"
I don't code in ruby, but one way you could do it in python is:
driver.find_elements
notice how elements is plural, I would grab all the links and put them into an array like.
href = driver.find_elements_by_xpath("//div[#class='rc]/h3/a").getAttribute("href");
Then get all of the descriptions the same way. Do a for loop for every element of description, if the description has the word "Adoption" in it navigate to that website.
for example:
if description[6] has the word adoption find the string href[6] and navigate to href[6].
I hope that makes sense!
Can you show me the way to use selenium in ruby to scroll down to bottom of page? I read this
element = driver.find_element(:xpath, "//div[#class='footer small']")
element.location_once_scrolled_into_view
but in this link, i can't find any element. Can you show me the way without element like that found. Thank you!
When I looked at the page, I did not see a div with class footer. This might be why you cannot find the element.
For me, the last visible element appears to be the wallpapers - ie div with class pic. You can get the last picture and scroll to it using the following. Note that we find all of the pictures and then take the last one in the collection.
last_picture = driver.find_elements(:css, 'div.pic').last
last_picture.location_once_scrolled_into_view
After you scroll to the last wallpaper, you will want to wait for the page to finish loading. For example, the following will wait until the image count increases:
require 'selenium-webdriver'
driver = Selenium::WebDriver.for :firefox
driver.navigate.to 'http://www.mobileswall.com/'
# Check how many elements are there initially
puts driver.find_elements(:css, 'div.pic').length
#=> 30
# Scroll to the last image
driver.find_elements(:css, 'div.pic').last.location_once_scrolled_into_view
# Wait for the additional images to load
current_count = driver.find_elements(:css, 'div.pic').length
until current_count < driver.find_elements(:css, 'div.pic').length
sleep(1)
end
# Check how many elements are there now
puts driver.find_elements(:css, 'div.pic').length
#=> 59
I am trying to automate an online survey on a website but I get this error each time:
Selenium::WebDriver::Error::UnknownError: unknown error: Element is not clickable at
point (561, 864). Other element would receive the click: a id="habla_oplink_a"
class="habla_oplink_a_normal hbl_pal_header_font_size hbl_pal_title_fg "
What I need to understand is how I can scroll to a certain point of the page so that my script can resume filling out the survey on the page.
This is my code that manages to fill out a portion of the survey but fails when it reaches a row which is not in view inside the browser (a row that requires the user to scroll down to):
buttons = browser.elements(:class => "assessment-choice")
buttons.each do |button|
button.click
end
I would also like to be able to change my code so that it only selects a specific option but the HTML on the page is not very friendly.
This is the webpage I am looking at: https://staging2.clearfit.com/assessment/assessment/95867fb272df436352a0bd5fbdd
The HTML of one of the options on the survey:
<a id="answers_79_0" class="assessment-choice" onmouseover="answerOver(this)" onmouseout="answerOut(this)" onclick="setAssessmentAnswer(this, 3, '0', '79', '#answers_49839163')">Strongly<br>Agree</a>
Using execute_script
To scroll to an element, you will need to execute javascript:
browser.execute_script('arguments[0].scrollIntoView();', button)
This can be seen to be working in the following script. Without the line to scroll, a chat tab overlays one of the buttons causing an exception.
require 'watir-webdriver'
browser = Watir::Browser.new :chrome
browser.goto 'https://staging2.clearfit.com/assessment/assessment/95867fb272df436352a0bd5fbdd'
buttons = browser.elements(:class => "assessment-choice")
buttons.each do |button|
browser.execute_script('arguments[0].scrollIntoView();', button)
button.click
end
Using the watir-scroll gem
Note that you can install the watir-scroll gem to make the scrolling line nicer. The gem allows the line to simply be:
browser.scroll.to button
The script would then look like:
require 'watir-webdriver'
require 'watir-scroll'
browser = Watir::Browser.new :chrome
browser.goto 'https://staging2.clearfit.com/assessment/assessment/95867fb272df436352a0bd5fbdd'
buttons = browser.elements(:class => "assessment-choice")
buttons.each do |button|
browser.scroll.to button
button.click
end
Firstly, this should be unnecessary. According to the spec, all element interactions require implicit scrolling to the element. If something does prevent this from happening, though, you can use this Selenium method instead of a javascript implementation:
buttons = browser.elements(:class => "assessment-choice")
buttons.each do |button|
button.wd.location_once_scrolled_into_view
button.click
end
public
def scroll_to(param)
args = case param
when :top, :start
'window.scrollTo(0, 0);'
when :center
'window.scrollTo(window.outerWidth / 2, window.outerHeight / 2);'
when :bottom, :end
'window.scrollTo(0, document.body.scrollHeight);'
when Array
['window.scrollTo(arguments[0], arguments[1]);', Integer(param[0]), Integer(param[1])]
else
raise ArgumentError, "Don't know how to scroll to: #{param}!"
end
#browser.execute_script(*args)
end
public
# This method pulls the object on the page you want to interact with, then it 'jumps to it'.
def jump_to(param)
# Leveraging the scroll_to(param) logic, this grabs the cooridnates,
# and then makes them an array that is able to be located and moved to.
# This is helpful when pages are getting too long and you need to click a button
# or interact with the browser, but the page 'Cannot locate element'.
location = param.wd.location
location = location.to_a
$helper.scroll_to(location)
end
Then you just call jump_to(element) and it "Jumps" to it.
This is how I got around it- not sure if that is a normal way. The problem is it goes to point (0,0); working on a version that moves to it to center screen.
I am using cucumber and watir. The question is in reference to the code below:
When(/^I click on all 'Show more'$/) do
#browser.links(:class, "more-matches").each do |d|
if d.text == "Show more"
d.click
end
end
end
Now, when the test case reaches this step-definition, the test case is shown as passed without clicking on all the links captured using #browser.links(:class, "more-matches").
The particular code does not get implemented may be because the ajax call has not been completed yet and the array holds zero elements and does not loop through. The code works if I introduce a "sleep 2" at the beginning of this step definition. Can anyone tell me how to handle this case by adding a code so that the ajax call has completed and the array holds all the elements and loops successfully. I have also tried adding the code:
if #browser.execute_script('return jQuery.active').to_i == 0
but it did not work as well.
Kindly suggest a way that the step definition gets executed and does not pass because of empty array.
Using Element#wait_until_present
Usually, you would know how many links should be present. Therefore, you could wait until the expected number of links are present.
When(/^I click on all 'Show more'$/) do
# Wait for the expected number of links to appear
# (note that :index is zero-based, hence the minus 1)
expected_number = 5
#browser.link(:class => "more-matches",
:index => (expected_number-1)).wait_until_present
# Click the links
#browser.links(:class, "more-matches").each do |d|
if d.text == "Show more"
d.click
end
end
end
If you do not know how many links are expected, it makes it a bit more difficult to ensure consistency. However, you might be able to get away with just checking that at least one link is present. Hopefully if one is present, all of the others are present.
When(/^I click on all 'Show more'$/) do
# Wait until at least one link appears
#browser.link(:class => "more-matches").wait_until_present
# Click the links
#browser.links(:class, "more-matches").each do |d|
if d.text == "Show more"
d.click
end
end
end
Using Browser#wait_until
An alternative approach is to use wait_until. The waiting for at least 5 links can be re-written as:
When(/^I click on all 'Show more'$/) do
# Wait for the expected number of links to appear
expected_number = 5
#browser.wait_until do
#browser.links(:class => "more-matches").length >= expected_number
end
# Click the links
#browser.links(:class, "more-matches").each do |d|
if d.text == "Show more"
d.click
end
end
end
I use Mechanize to loop through a table, which is paginated.
I have a problem with a form that holds multiple submit inputs. The input tags are used as pagination and they are generated dynamically. When I loop through the pages I need to scrape, I need to be able to pick the right input, since only one of them will take me to the “next page”. The right tag can be identified by different attributes such as name, class, value etc. My problem is though, that I can’t find out how to tell mechanize which one to use.
I tried this:
require 'mechanize'
require 'yaml'
url = "http://www.somewhere.com"
agent = Mechanize.new
page = agent.get(url)
loop do
puts "some content from site using nokogiri"
if next_page = page.form_with(:action => /.*/)
page = next_page.submit(page.form_with(:action => /.*/).submits[3])
else
break
end
end
From this post, http://rubyforge.org/pipermail/mechanize-users/2008-November/000314.html, but as told the number of tags are changing so just picking a hardcoded number of the submits is not too good an idea.
What I would like to know is if there is a way like this:
loop do
puts "some content from site using nokogiri"
if next_page = page.form_with(:action => /.*/)
page = next_page.submit(:name => /the_right_submit_button/)
else
break
end
end
or something like that, maybe with a css or xpath selector.
I usually use form.button_with to select the right button to click:
form = results_page.forms[0]
results_page = form.submit(form.button_with(:name=>'ctl00$ContentBody$ResultsPager$NextButton'))