Experiencing a problem when running watir-web driver to test Ajax responses from our site. The site is designed to contain a list of items. When a new item is added the site will submit the entry via Ajax and wait for a response to tell the site to display this new entry. We are finding these Ajax responses are coming back blank whilst running the page via watir in both chrome and Firefox. However when trying the scenario out side of watir the Ajax responses contain data.
My question is are we over looking something when setting up our browsers?
Additional info using latest versions of gems but testing on Ubuntu 12.04.
here is the code to set-up the two browsers:
download_directory = "#{Dir.pwd}/Results" #download location (FF only)
profileFF = Selenium::WebDriver::Firefox::Profile.new
profileFF['browser.download.folderList'] = 2 # custom location
profileFF['browser.download.dir'] = download_directory
profileFF['browser.helperApps.neverAsk.saveToDisk'] = "application/msword" #auto save rulw
c = Watir::Browser.new :chrome
f = Watir::Browser.new :firefox ,:profile => profileFF
Related
Consider two webpages one and two. Site number two is easy to scrape using nokogiri because it doesn't use JS. Site number one however cannot be scraped using just nokogiri. I googled and searched far and wide and found that if I loaded the page with an automated web browser I could scrape the the rendered HTML. I have the following code right below:
# creates an instance
driver = Selenium::WebDriver.for :chrome
# opens an existing webpage
driver.get 'http://www.bigstub.com/search.aspx'
# wait is used to let the webpage load up and let the JS render
wait = Selenium::WebDriver::Wait.new(:timeout => 5)
My question is that I am trying to let the page load up an close immediately once I get my desired class. An example is that if I adjust the time out to 10 seconds until I can find the class .title-holder how would I write this code?
Pusedo code:
rendered_source_page will time out if .include?("title-holder"). I just don't know how to write it.
UPDATE:
In regards to the headless question, selenium has an option or configuration in where you can add in a headless option. This is done by the code below:
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--headless')
driver = Selenium::WebDriver.for :chrome, options: options
For my next question in order for the site to fully scrape the JS rendered HTML I set my timeout variable to 5 seconds:
wait = Selenium::WebDriver::Wait.new(:timeout => 5)
wait.until { /title-holder/.match(driver.page_source) }
wait.until pretty much means wait 5 seconds until I find a title-holder class inside of the page_source or rendered HTML. This pretty much solved all my questions.
I am assuming you are running selenium on a server. So first install Xvfb
sudo apt-get install xvfb
Install firefox
sudo apt-get install firefox
Add the following two gems to your gemfile. You will need headless because you want to run the selenium webdriver on your server. Headless will start and stop Xvfb for you.
#gemfile
gem 'selenium-webdriver'
gem 'headless'
Code for scraping
headless = Headless.new
headless.start
driver = Selenium::WebDriver.for :firefox
driver.navigate.to example.com
wait = Selenium::WebDriver::Wait.new(:timeout => 30)
#scraping code comes here
Housekeeping so that you don't run out of memory.
driver.quit
headless.destroy
Hope this helps.
In regards to the headless question, selenium has an option or configuration in where you can add in a headless option. This is done by the code below:
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--headless')
driver = Selenium::WebDriver.for :chrome, options: options
For my next question in order for the site to fully scrape the JS rendered HTML I set my timeout variable to 5 seconds:
wait = Selenium::WebDriver::Wait.new(:timeout => 5)
wait.until { /title-holder/.match(driver.page_source) }
wait.until pretty much means wait 5 seconds until I find a title-holder class inside of the page_source or rendered HTML. This pretty much solved all my questions.
I have a ruby script that launches a chrome browser and loads the AdBlock Plus extension. Unfortunately I can't figure out how to load a custom filter that I've added through the options panel for the extension (I added a filter to block particular div id's). When I load the extension it treats like it's the first time it's been loaded, so I'm wondering how to reload the state I left it in after creating the custom filter.
Here's a snippet of how I'm launching it:
Selenium::WebDriver::Chrome.path = '/opt/google/chrome/chrome'
browser = Watir::Browser.new(:chrome, :switches => %w[--load extension=/home/someuser/.config/google-chrome/Default/Extensions/cfhdojbkjhnklbpkdaibdccddilifddb/1.12.1_0 --no-first-run])
I found a solution to my problem. Adding the user-data-dir switch, it now loads any settings made to the extension.
browser = Watir::Browser.new(:chrome, :switches => %w[--user-data-dir=/home/someuser/.config/google-chrome --load-extension=/home/someuser/.config/google-chrome/Default/Extensions/cfhdojbkjhnklbpkdaibdccddilifddb/1.12.1_0 --no-first-run])
I'm trying to (ab)use the capybara web testing framework to automate some tasks on github that are not accessible via the github API and which require me to be logged in and click on buttons to send AJAX requests.
Since capybara/selenium is a testing framework it helpfully creates a temporary session which has no cookies in it. I'd like to either stop it from doing that, or else I'd like to know how to load my cookie store into the browser session that it creates.
All I'm trying to do is this:
#!/usr/bin/env ruby
require 'selenium-webdriver'
driver = Selenium::WebDriver.for :chrome
driver.navigate.to "https://github.com"
Or this:
#!/usr/bin/env ruby
require 'capybara'
Capybara.register_driver :selenium do |app|
Capybara::Selenium::Driver.new(app, :browser => :chrome)
end
session = Capybara::Session.new(:selenium)
session.visit "https://www.github.com"
In both cases I get the github.com landing page you'd see as a logged-out user or incognito mode in the browser. I'd like to get my logged-in landing page like I just fired up a web browser myself and navigated to that URL.
Since I have 2FA setup on github that makes automating the login process from the github landing page somewhat annoying, so I'd like to avoid automating logging into github. The tasks that I want to automate do not require re-authenticating via 2FA.
ANSWER:
For MacOSX+Ruby+Selenium this works:
#!/usr/bin/env ruby
require 'selenium-webdriver'
caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {"debuggerAddress" => "127.0.0.1:20480"}, detach: false)
driver = Selenium::WebDriver.for :chrome, :desired_capabilities => caps
driver.navigate.to "https://github.com"
Then fire up chrome with this:
% /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=/Users/lamont/Library/Application\ Support/Google/Chrome --profile-directory=Default --remote-debugging-port=20480
Obviously the paths will need to be adjusted because they're OSX-centric and have my homedir in them.
There is also a bug in the selenium-webdriver gem for ruby where it inserts a 'detach' option which gets into a fight with 'debuggerAddress':
/Users/lamont/.rvm/gems/ruby-2.2.4/gems/selenium-webdriver-2.53.0/lib/selenium/webdriver/remote/response.rb:70:in `assert_ok': unknown error: cannot parse capability: chromeOptions (Selenium::WebDriver::Error::UnknownError)
from unknown error: unrecognized chrome option: detach
The lib/selenium/webdriver/chrome/bridge.rb file can be edited to take that out as a quick hack:
chrome_options['binary'] = Chrome.path if Chrome.path
chrome_options['nativeEvents'] = true if native_events
chrome_options['verbose'] = true if verbose
#chrome_options['detach'] = detach.nil? || !!detach
chrome_options['noWebsiteTestingDefaults'] = true if no_website_testing_defaults
chrome_options['prefs'] = prefs if prefs
To implement something similar in Ruby, check out this page that goes over that. Thanks to lamont for letting me know in the comments.
You can start chrome using a specific Chrome profile. I am not sure what the ruby implementation would look like, but in python it looks something like:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
options = ChromeOptions()
# more on this line here later.
options.add_experimental_option('debuggerAddress', '127.0.0.1:7878')
driver = webdriver.Chrome(chrome_options=otpions)
In order for this to work you need to do a few things.
manually start chrome from terminal/command prompt with these command line arguments
--user-data-dir=/path/to/any/custom/directory/home/user/Desktop/Chromedir --profile-directory="Profile 1" --remote-debugging-port=7878
make sure "Profile 1" is already existing in the same --user-data-dir (make sure user Profile 1 has necessary chrome://components/
to run any apps that require those components)
you can use any free port in place of 7878
verify that http://localhost:7878 is running and returns value.
This should manually launch chrome with the "Profile 1" profile, and so long as it has logged into the site in question, it will stay logged in like a normal user so long as you follow these instructions to run the tests.
I used this to write a quick netflix bot that clicks the "continue playing" button when it pops up, and it's the only way to get DRM content to play as far as I have found. But it retains the cookies for the login, and also launches chrome with whatever components the profile is set up to have.
I have tried launching chrome with specific profiles before using different methodologies, but this was the only way to really force it to work how I wanted it to.
Edit: There are methods for saving cookie info as well although I don't know how well they work. Check out this link for more info, as my solution is probably not the best solution even if it works.
The show_me_the_cookies gem provides cross-driver cookie manipulation and can let you add new cookies. The one thing to be aware of when using selenium is that you need to visit the domain before you can create cookie for it, so you'll need to do something like
visit "https://www.github.com"
create_cookie(...)
visit "https://www.github.com"
for it to work - first visit just puts the browser/driver in a state where you can create the cookie, second visit actually goes to the page with the cookies set.
I had to tweak the OP's answer (from within her question) to get this going with Ruby in 2022.
Prerequisites
Chromedriver installed and allowed to run even though it's not signed:
> brew install chromedriver
> xattr -d com.apple.quarantine /usr/local/bin/chromedriver
Chrome launched and accepting commands on a specific port:
> /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=~/Library/Application\ Support/Google/Chrome --profile-directory=Default --remote-debugging-port=20480
This created a new profile in Chrome so I signed in to my account and got the browser set up, ready to start interacting with the (legacy EdTech) site I'm trying to automate.
Actual use
require 'selenium-webdriver'
caps = Selenium::WebDriver::Remote::Capabilities.chrome("goog:chromeOptions" => {"debuggerAddress" => "127.0.0.1:20480"})
driver = Selenium::WebDriver.for :chrome, capabilities: caps
driver.navigate.to "https://www.google.com"
Until recently, I was using mechanize to get a web page, then do some parsing with nokogiri. But because some content was loaded with Ajax after I have started using Watir instead. My code looks like this:
def get_page(url)
browser = Watir::Browser.start url
sleep 1
page = Nokogiri::HTML.parse(browser.html)
browser.close
return page
end
It works fine but since I am getting a lot of pages the browser.start will open a tons of windows. I found the close as you see, but is there a way to just not show the browser window at all?
I am new in using watir but I can only imagine this to be a bug:
require "watir"
Watir::Browser.default = 'safari'
b = Watir::Browser.new
b.goto()
=> nil
b.link(:title, "Start").click
Leads me to the next page as expected but on the following page no link works even they are there
b.link(:title, "Do something").exist?
=> true
When I then enter
b.link(:title, "Do something").click
nothing happens, even the href attribute of course links to the next page (to be more precise, it is the same page but different request parameters)
Identifying the link with :xpath looks similar.
The same thing works fine with "firefox" as browser.
I use Safari 5.0.1, ruby 1.8.7 and installed beside others safariwatir-0.3.8
In fact controlling safari partially works but I have no glue why it hangs on the second page. Here is an example using a scenario in the internet:
this works fine:
browser.goto("http://reiseauskunft.bahn.de/bin/query.exe/dn?S=Frankfurt(Main)Hbf&Z=Nrnberg%20Hbf&date=%2B30&time=1500&start=1&")
=> nil
this also works fine - the link does exist:
browser.link(:title, /Mit Tickets zum Normalpreis/).exist?
=> true
this does not work - the browser window stays unchanged:
browser.link(:title, /Mit Tickets zum Normalpreis/).click
=> nil
I would like to try fire_event or fireEvent but irb says invalid method. I guess the hints I found about firing an event are outdated?
As far as I know, the only way to drive Safari is to use safariwatir gem (works only on Mac OS X). Try this (from SafariWatir wiki page at Watir wiki):
require 'rubygems'
require 'safariwatir'
browser = Watir::Safari.new
browser.goto("http://google.com")