I'm using watir-webdriver for screenshots, and now it is only giving me full site screenshots. How can I see just what is above the fold?
Current code:
driver = Webdriver::UserAgent.driver(:browser => :firefox, :agent => :iphone, :orientation => ;landscape)
browser = Watir::Browser.new driver
browser.goto "http://forefathersgroup.com"
browser.driver.save_screenshot "forefathers.png
There is no way to get only visible part of the page using watir-webdriver, because selenium, which is used in this library does not provide such functionality.
So one of possible solutions is crop a screenshot after its creation
Related
Consider two webpages one and two. Site number two is easy to scrape using nokogiri because it doesn't use JS. Site number one however cannot be scraped using just nokogiri. I googled and searched far and wide and found that if I loaded the page with an automated web browser I could scrape the the rendered HTML. I have the following code right below:
# creates an instance
driver = Selenium::WebDriver.for :chrome
# opens an existing webpage
driver.get 'http://www.bigstub.com/search.aspx'
# wait is used to let the webpage load up and let the JS render
wait = Selenium::WebDriver::Wait.new(:timeout => 5)
My question is that I am trying to let the page load up an close immediately once I get my desired class. An example is that if I adjust the time out to 10 seconds until I can find the class .title-holder how would I write this code?
Pusedo code:
rendered_source_page will time out if .include?("title-holder"). I just don't know how to write it.
UPDATE:
In regards to the headless question, selenium has an option or configuration in where you can add in a headless option. This is done by the code below:
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--headless')
driver = Selenium::WebDriver.for :chrome, options: options
For my next question in order for the site to fully scrape the JS rendered HTML I set my timeout variable to 5 seconds:
wait = Selenium::WebDriver::Wait.new(:timeout => 5)
wait.until { /title-holder/.match(driver.page_source) }
wait.until pretty much means wait 5 seconds until I find a title-holder class inside of the page_source or rendered HTML. This pretty much solved all my questions.
I am assuming you are running selenium on a server. So first install Xvfb
sudo apt-get install xvfb
Install firefox
sudo apt-get install firefox
Add the following two gems to your gemfile. You will need headless because you want to run the selenium webdriver on your server. Headless will start and stop Xvfb for you.
#gemfile
gem 'selenium-webdriver'
gem 'headless'
Code for scraping
headless = Headless.new
headless.start
driver = Selenium::WebDriver.for :firefox
driver.navigate.to example.com
wait = Selenium::WebDriver::Wait.new(:timeout => 30)
#scraping code comes here
Housekeeping so that you don't run out of memory.
driver.quit
headless.destroy
Hope this helps.
In regards to the headless question, selenium has an option or configuration in where you can add in a headless option. This is done by the code below:
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--headless')
driver = Selenium::WebDriver.for :chrome, options: options
For my next question in order for the site to fully scrape the JS rendered HTML I set my timeout variable to 5 seconds:
wait = Selenium::WebDriver::Wait.new(:timeout => 5)
wait.until { /title-holder/.match(driver.page_source) }
wait.until pretty much means wait 5 seconds until I find a title-holder class inside of the page_source or rendered HTML. This pretty much solved all my questions.
I am using chrome 56, chrome driver 2.27(latest release) with selenium web driver 3.1.0. Referring to the issue(https://github.com/seleniumhq/selenium-google-code-issue-archive/issues/1811) where chrome closes all the instances once the program finishes and it does not give me a chance to debug. I just want to know if this is fixed then why it is still happening ? or i am missing something ?
I am using the following code. Any help is appreciated.
require "uri"
require "net/http"
require 'watir-webdriver'
require 'selenium-webdriver'
#b = Watir::Browser.new :chrome
#b.goto 'http://www.google.com'
Firstly, watir-webdriver gem is deprecated. The updated code is in the watir gem. Also, you shouldn't need to require any of those other gems directly.
The chromedriver service is stopped when the ruby process exits. If you do not want the browsers that were started by chromedriver to close as well, you need to use the detach parameter. Currently this is done like so:
require 'watir'
caps = Selenium::WebDriver::Remote::Capabilities.chrome
caps[:chrome_options] = {detach: true}
#b = Watir::Browser.new :chrome, desired_capabilities: caps
Declare these
caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {'detach' => true})
browser = Watir::Browser.new :chrome, desired_capabilities: caps
On a side note! this might give a problem when you are running multiple scenario tests, chromedriver will actively refuse connection in case an other test initiates in the same chrome session. Ensure you have browser.close whenever required.
I can't seem to find anything about this, and I'm starting to get upset, because this code used to work:
browser = Watir::Browser.new :firefox, :profile => "Anna"
However, now that code, while it will launch firefox successfully if I use :marionette => true, if I use :profile => "Anna" or even if I do this:
caps = Selenium::WebDriver::Remote::Capabilities.firefox({:profile => profile,:marionette=> true})
browser = Watir::Browser.new :firefox, :desired_capabilities => caps
Watir will still launch firefox and work properly, but it is still using its own profile. I have custom x509 certificates I need to use in that profile (and the site is throwing "This site is not secure" screens at me because it's still in development), so how can I use it? I know there's a way to do it, but I've been Googling for an hour now and have yet to find it. Removing :marionette => true will let the profile load properly, but since Firefox 48 doesn't support the old Firefox driver, I can't make it work.
I'm working on web-application automation using Ruby 1.9.3p484 and Watir-webdriver (0.8.0) with Firefox 41.0 for Ubuntu.
I want the browser to not load any images. To do that, I try to change the 'permissions.default.image' firefox parameter to 2.
I have tried the following code:
profile = Selenium::WebDriver::Firefox::Profile.new
profile['permissions.default.image'] = 2
browser = Watir::Browser.new(:firefox, :profile => profile)
browser.goto url
But, the browser keep loading the images. In the about:config page, the 'permissions.default.image' is still set to 1.
Any ideas on what could be wrong?
I don't know why, but if I use from_name method in Selenium class init, then you code is working perfectly:
profile = Selenium::WebDriver::Firefox::Profile.from_name "default"
I've been playing with Watir and Tor browser and I can't get them to work.
The correct browser opens, however, I'm unable to open a website.
My code:
Selenium::WebDriver::Firefox.path = '\path\Tor Browser\Start Tor Browser.exe'
driver = Selenium::WebDriver.for :firefox
browser = Watir::Browser.new :firefox, :driver => driver
This results in a message box popping up, which says:
Your Firefox profile cannot be loaded. It may be missing or inaccessible.
The correct browser also opens up, however, the website doesn't.
I've also tried:
Selenium::WebDriver::Firefox.path = '\path\Tor Browser\Browser\firefox.exe'
driver = Selenium::WebDriver.for :firefox
browser = Watir::Browser.new :firefox, :driver => driver
With this, the message box doesn't appear, but still, the website is not loaded.
I'm not sure, if the problem is in my code or in the browser.
Thank you very much for any help!
Start Tor then run this, assuming your socks port is 9050
require 'watir-webdriver'
profile = Selenium::WebDriver::Firefox::Profile.new
profile['network.proxy.socks'] = "127.0.0.1"
profile['network.proxy.socks_port'] = 9050
profile['network.proxy.type'] = 1
$browser = Watir::Browser.new :firefox, :profile => profile
$browser.goto "whatsmyip.org"
Firefox and Tor are not the same browser. Watir uses webdriver API to control the browser. You can't use Firefox webdriver API to control another browser.
In your second example, browser don't open the site, but you don't tell him to. You should add :
browser.goto('http://my.example.com')
And don't forget to put a browser.quit at the end of your code to close the browser.