I'm attempting to do some web scraping using headless chrome via selinium-webdriver on Heroku but most recently ran into trouble. Visiting, https://music.youtube.com I get a page with the following:
<div class="message">
Sorry, YouTube Music is not optimized for your browser. Check for updates or try Google Chrome.
</div>
I can confirm that my chrome driver is up to date and nothing on my end has changed to cause this break in functionality. Using any other automated scraping gem similar to selinium gives me a similar message in telling me my browser is deprecated. Note that performing these actions is no problem when I do it on my local machine. To give some background, I originally followed this answer to get everything running correctly on heroku prior to the breakage. On top of this here's my setup:
gem 'selenium-webdriver'
gem 'webdrivers', '~> 4.0', require: false
require 'webdrivers'
require 'selenium-webdriver'
chrome_bin_path = ENV.fetch('GOOGLE_CHROME_SHIM', nil)
options = Selenium::WebDriver::Chrome::Options.new
options.binary = chrome_bin_path if chrome_bin_path
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
$driver = Selenium::WebDriver.for :chrome, options: options
What would be the most logical answer to why I'm getting this message on a site like youtube?
Related
I am very new at BDD testing and I am trying to figure out how to open a chrome browser in a specific URL using these technologies. I already installed bddfire gem and already ran it.
Now I have a feature:
#openingChrome
Scenario: Opening Chrome on facebook page
Given I open chrome and write "url"
bddfir_steps.rb
Given(/^I open chrome and write "([^"]*)"$/) do |arg1|
$session.visit("https://www.facebook.com.br")
end
In the hooks file I wrote
Before do
$session = Capybara::Session.new(:selenium, browser: :chrome)
end
And in the env.rb file that came with bddfire gem there is this
Capybara.register_driver :chrome do |app|
Capybara::Selenium::Driver.new(app, browser: :chrome)
end
So chrome is already added and I already installed chrome webdriver. Unfortunately this line of code is not working: $session = Capybara::Session.new(:selenium, browser: :chrome)
it throws this error: The second parameter to Session::new should be a rack app if passed. (TypeError)
does anyone know why?
Session#new takes the name of the driver and optionally a rack app instance to start. Since you don't seem to be starting an app and you're registering your driver with a name of :chrome you want
Before do
$session = Capybara::Session.new(:chrome)
end
I am using chrome 56, chrome driver 2.27(latest release) with selenium web driver 3.1.0. Referring to the issue(https://github.com/seleniumhq/selenium-google-code-issue-archive/issues/1811) where chrome closes all the instances once the program finishes and it does not give me a chance to debug. I just want to know if this is fixed then why it is still happening ? or i am missing something ?
I am using the following code. Any help is appreciated.
require "uri"
require "net/http"
require 'watir-webdriver'
require 'selenium-webdriver'
#b = Watir::Browser.new :chrome
#b.goto 'http://www.google.com'
Firstly, watir-webdriver gem is deprecated. The updated code is in the watir gem. Also, you shouldn't need to require any of those other gems directly.
The chromedriver service is stopped when the ruby process exits. If you do not want the browsers that were started by chromedriver to close as well, you need to use the detach parameter. Currently this is done like so:
require 'watir'
caps = Selenium::WebDriver::Remote::Capabilities.chrome
caps[:chrome_options] = {detach: true}
#b = Watir::Browser.new :chrome, desired_capabilities: caps
Declare these
caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {'detach' => true})
browser = Watir::Browser.new :chrome, desired_capabilities: caps
On a side note! this might give a problem when you are running multiple scenario tests, chromedriver will actively refuse connection in case an other test initiates in the same chrome session. Ensure you have browser.close whenever required.
I am currently working on an automation project where I need to use Ruby/Selenium to discover specific http headers returned to the user after authentication to a web app. I am able to automate the web app just fine; however, when I try to use a Chrome extension the browser returns the following error:
The webpage at chrome-extension://[extension address] might be temporarily down or it may have moved permanently to a new web address.
After looking into this, it appears that the Selenium web driver is using a different Chrome profile than my regular Chrome profile. As such, I was wondering if someone knew if there is a way to to tell Selenium to use my regular Chrome profile with the extension loaded or build a new profile and install the extension during runtime.
So far, most of the answers I have found were centralized around Python and Java. Please let me know if I can provide more information.
To launch Chrome with the default profile on Windows:
require 'selenium-webdriver'
switches = ['user-data-dir='+ENV['LOCALAPPDATA']+'\\Google\\Chrome\\User Data']
driver = Selenium::WebDriver.for :chrome, :switches => switches
driver.navigate.to "https://www.google.co.uk"
Or to add an extension to the created profile:
require 'selenium-webdriver'
driver = Selenium::WebDriver.for :chrome,
:desired_capabilities => Selenium::WebDriver::Remote::Capabilities.chrome({
'chromeOptions' => {
'extensions' => [
Base64.strict_encode64(File.open('C:\\App\\extension.crx', 'rb').read)
]
}
})
driver.navigate.to "https://www.google.co.uk"
I'm trying to (ab)use the capybara web testing framework to automate some tasks on github that are not accessible via the github API and which require me to be logged in and click on buttons to send AJAX requests.
Since capybara/selenium is a testing framework it helpfully creates a temporary session which has no cookies in it. I'd like to either stop it from doing that, or else I'd like to know how to load my cookie store into the browser session that it creates.
All I'm trying to do is this:
#!/usr/bin/env ruby
require 'selenium-webdriver'
driver = Selenium::WebDriver.for :chrome
driver.navigate.to "https://github.com"
Or this:
#!/usr/bin/env ruby
require 'capybara'
Capybara.register_driver :selenium do |app|
Capybara::Selenium::Driver.new(app, :browser => :chrome)
end
session = Capybara::Session.new(:selenium)
session.visit "https://www.github.com"
In both cases I get the github.com landing page you'd see as a logged-out user or incognito mode in the browser. I'd like to get my logged-in landing page like I just fired up a web browser myself and navigated to that URL.
Since I have 2FA setup on github that makes automating the login process from the github landing page somewhat annoying, so I'd like to avoid automating logging into github. The tasks that I want to automate do not require re-authenticating via 2FA.
ANSWER:
For MacOSX+Ruby+Selenium this works:
#!/usr/bin/env ruby
require 'selenium-webdriver'
caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {"debuggerAddress" => "127.0.0.1:20480"}, detach: false)
driver = Selenium::WebDriver.for :chrome, :desired_capabilities => caps
driver.navigate.to "https://github.com"
Then fire up chrome with this:
% /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=/Users/lamont/Library/Application\ Support/Google/Chrome --profile-directory=Default --remote-debugging-port=20480
Obviously the paths will need to be adjusted because they're OSX-centric and have my homedir in them.
There is also a bug in the selenium-webdriver gem for ruby where it inserts a 'detach' option which gets into a fight with 'debuggerAddress':
/Users/lamont/.rvm/gems/ruby-2.2.4/gems/selenium-webdriver-2.53.0/lib/selenium/webdriver/remote/response.rb:70:in `assert_ok': unknown error: cannot parse capability: chromeOptions (Selenium::WebDriver::Error::UnknownError)
from unknown error: unrecognized chrome option: detach
The lib/selenium/webdriver/chrome/bridge.rb file can be edited to take that out as a quick hack:
chrome_options['binary'] = Chrome.path if Chrome.path
chrome_options['nativeEvents'] = true if native_events
chrome_options['verbose'] = true if verbose
#chrome_options['detach'] = detach.nil? || !!detach
chrome_options['noWebsiteTestingDefaults'] = true if no_website_testing_defaults
chrome_options['prefs'] = prefs if prefs
To implement something similar in Ruby, check out this page that goes over that. Thanks to lamont for letting me know in the comments.
You can start chrome using a specific Chrome profile. I am not sure what the ruby implementation would look like, but in python it looks something like:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
options = ChromeOptions()
# more on this line here later.
options.add_experimental_option('debuggerAddress', '127.0.0.1:7878')
driver = webdriver.Chrome(chrome_options=otpions)
In order for this to work you need to do a few things.
manually start chrome from terminal/command prompt with these command line arguments
--user-data-dir=/path/to/any/custom/directory/home/user/Desktop/Chromedir --profile-directory="Profile 1" --remote-debugging-port=7878
make sure "Profile 1" is already existing in the same --user-data-dir (make sure user Profile 1 has necessary chrome://components/
to run any apps that require those components)
you can use any free port in place of 7878
verify that http://localhost:7878 is running and returns value.
This should manually launch chrome with the "Profile 1" profile, and so long as it has logged into the site in question, it will stay logged in like a normal user so long as you follow these instructions to run the tests.
I used this to write a quick netflix bot that clicks the "continue playing" button when it pops up, and it's the only way to get DRM content to play as far as I have found. But it retains the cookies for the login, and also launches chrome with whatever components the profile is set up to have.
I have tried launching chrome with specific profiles before using different methodologies, but this was the only way to really force it to work how I wanted it to.
Edit: There are methods for saving cookie info as well although I don't know how well they work. Check out this link for more info, as my solution is probably not the best solution even if it works.
The show_me_the_cookies gem provides cross-driver cookie manipulation and can let you add new cookies. The one thing to be aware of when using selenium is that you need to visit the domain before you can create cookie for it, so you'll need to do something like
visit "https://www.github.com"
create_cookie(...)
visit "https://www.github.com"
for it to work - first visit just puts the browser/driver in a state where you can create the cookie, second visit actually goes to the page with the cookies set.
I had to tweak the OP's answer (from within her question) to get this going with Ruby in 2022.
Prerequisites
Chromedriver installed and allowed to run even though it's not signed:
> brew install chromedriver
> xattr -d com.apple.quarantine /usr/local/bin/chromedriver
Chrome launched and accepting commands on a specific port:
> /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=~/Library/Application\ Support/Google/Chrome --profile-directory=Default --remote-debugging-port=20480
This created a new profile in Chrome so I signed in to my account and got the browser set up, ready to start interacting with the (legacy EdTech) site I'm trying to automate.
Actual use
require 'selenium-webdriver'
caps = Selenium::WebDriver::Remote::Capabilities.chrome("goog:chromeOptions" => {"debuggerAddress" => "127.0.0.1:20480"})
driver = Selenium::WebDriver.for :chrome, capabilities: caps
driver.navigate.to "https://www.google.com"
Using Ruby 2.0.0 p481 in RubyMine and chromedriver 2.10
When Chrome starts it displays a message in a yellow popup bar: "You are using an unsupported command-line flag: --ignore-certificate-errors. Stability and security will suffer." This simple example reproduces the problem.
require "selenium-webdriver"
driver = Selenium::WebDriver.for :chrome
driver.navigate.to login_url
This question has been answered for Java and Python. I have looked everywhere for a Ruby analog. Does anyone have a suggestion or know how to translate the Python answer (Unsupported command-line flag: --ignore-certificate-errors) to Ruby? Thank you!
The Ruby selenium-webdriver API doesn't expose a separate Chrome options object like Java/Python but you can set the options via "Capabilities".
The Capabilities web page provides a Ruby example and the table of recognized capabilities that you can inject. Plugging those together with excludeSwitches:
caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {"excludeSwitches" => [ "--ignore-certificate-errors" ]})
driver = Selenium::WebDriver.for :chrome, desired_capabilities: caps
Take a look at Watir too, it's a front end for WebDriver.
Their examples show how you can send a :switches array which is passed straight through to the web driver so you can do the same. That makes adding other switches a bit easier rather than going through capabilities.
There is a chromedriver issue on the topic as well. There are posts detailing that you can add a --test-type argument to work around the certificate issue and ruby code examples like above.
I adjusted:
driver = Selenium::WebDriver.for :chrome
to read:
driver = Selenium::WebDriver.for :chrome, :switches => %w[--test-type]
...and the script ran successfully without the yellow flag. Clearly, other command-line switches could be added.
Thank you to Nguyen Vu Hoang and mtm.
I donot know ruby, however my approach is set mode "test-type" to ChromeDriver capabilities
Here's my sample code in Java
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities = DesiredCapabilities.chrome();
ChromeOptions options = new ChromeOptions();
options.addArguments("test-type", "start-maximized",
"no-default-browser-check");
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
With Capybara:
Capybara.register_driver :chrome do |app|
Capybara::Selenium::Driver.new(app, browser: :chrome, switches: ['--test-type'])
end
This error caused my rspec tests to fail. I think it was causing Chrome to slow down so the above fixes did remove the error messages but did not resolve my problem of my tests failing.
If you are using chromedriver-helper then this should fix the problem. Run this from the command line
chromedriver-update
This is described more in chromedriver-helper, "This might be necessary on platforms on which Chrome auto-updates, which has been known to introduce incompatibilities with older versions of chromedrive"
Once I ran this I was able to remove the fixes described elsewhere and Chrome ran with out warnings.