I'm looking to use Selenium with a username/password authenticated proxy in Ruby. I realize that most people use ProxyChain when doing this in Chrome, but I'd like to use a solution without any additional gems since it doesn't play well on Heroku, plus I'm using Firefox so there seems to be a possible other option judging by THIS question though it's written in Python.
I used the selenium docs to translate that code to Ruby, but Selenium is still not using my proxy when navigating to a webpage. Oddly enough when I refresh the page manually it prompts me for the username/password but it doesn't do that on the initial page load.
profile = Selenium::WebDriver::Firefox::Profile.new
profile["network.proxy.type"] = 1
# proxy ip and port are fake for this example
profile["network.proxy.http"] = "182.192.157.60"
profile["network.proxy.http_port"] = 12345
# set the username and password
profile["network.proxy.socks_username"] = "my_username"
profile["network.proxy.socks_password"] = "my_password"
options = Selenium::WebDriver::Firefox::Options.new(profile: profile)
driver = Selenium::WebDriver.for :firefox, options: options
If anyone has any ideas I would certainly appreciate the help. Thank you.
Related
I try to open internet explorer via watir-webdriver with my code:
require "watir-webdriver"
profile = Selenium::WebDriver::Firefox::Profile.new
profile.proxy = Selenium::WebDriver::Proxy.new :http => 'http://username:password#xx.xxx.xxx.xx:xxxxx', :ssl => 'http://username:password#xx.xxx.xxx.xx:xxxxx'
And I get this error stating something about the I'm guessing?
This is what it reads:
C:/Ruby/lib/ruby/gems/2.2.0/gems/selenium-webdriver-2.53.0/lib/selenium/webdriver/firefox/profile.rb:205:in `Integer': invalid value for Integer(): "//username:password#xx.xxx.xxx.xx:xxxxx" (ArgumentError)
from C:/Ruby/lib/ruby/gems/2.2.0/gems/selenium-webdriver-2.53.0/lib/selenium/webdriver/firefox/profile.rb:205:in `set_manual_proxy_preference'
from C:/Ruby/lib/ruby/gems/2.2.0/gems/selenium-webdriver-2.53.0/lib/selenium/webdriver/firefox/profile.rb:176:in `proxy='
from test1.rb:6:in `<main>'
I can explain the error at least. You can't pass in basic authentication to a proxy url because the code expects that a colon in the string is distinguishing a host from a port (which needs to be an integer.
I do not have much experience with using proxies with Selenium. Poking around on the internet for the problem give suggestions such as setting signon.autologin.proxy to true in your designated Firefox profile, or adding the Firefox autoauth extension. Essentially you can save your password once manually in your profile, and then automated tests won't have to input it.
I am currently working on an automation project where I need to use Ruby/Selenium to discover specific http headers returned to the user after authentication to a web app. I am able to automate the web app just fine; however, when I try to use a Chrome extension the browser returns the following error:
The webpage at chrome-extension://[extension address] might be temporarily down or it may have moved permanently to a new web address.
After looking into this, it appears that the Selenium web driver is using a different Chrome profile than my regular Chrome profile. As such, I was wondering if someone knew if there is a way to to tell Selenium to use my regular Chrome profile with the extension loaded or build a new profile and install the extension during runtime.
So far, most of the answers I have found were centralized around Python and Java. Please let me know if I can provide more information.
To launch Chrome with the default profile on Windows:
require 'selenium-webdriver'
switches = ['user-data-dir='+ENV['LOCALAPPDATA']+'\\Google\\Chrome\\User Data']
driver = Selenium::WebDriver.for :chrome, :switches => switches
driver.navigate.to "https://www.google.co.uk"
Or to add an extension to the created profile:
require 'selenium-webdriver'
driver = Selenium::WebDriver.for :chrome,
:desired_capabilities => Selenium::WebDriver::Remote::Capabilities.chrome({
'chromeOptions' => {
'extensions' => [
Base64.strict_encode64(File.open('C:\\App\\extension.crx', 'rb').read)
]
}
})
driver.navigate.to "https://www.google.co.uk"
I'm trying to (ab)use the capybara web testing framework to automate some tasks on github that are not accessible via the github API and which require me to be logged in and click on buttons to send AJAX requests.
Since capybara/selenium is a testing framework it helpfully creates a temporary session which has no cookies in it. I'd like to either stop it from doing that, or else I'd like to know how to load my cookie store into the browser session that it creates.
All I'm trying to do is this:
#!/usr/bin/env ruby
require 'selenium-webdriver'
driver = Selenium::WebDriver.for :chrome
driver.navigate.to "https://github.com"
Or this:
#!/usr/bin/env ruby
require 'capybara'
Capybara.register_driver :selenium do |app|
Capybara::Selenium::Driver.new(app, :browser => :chrome)
end
session = Capybara::Session.new(:selenium)
session.visit "https://www.github.com"
In both cases I get the github.com landing page you'd see as a logged-out user or incognito mode in the browser. I'd like to get my logged-in landing page like I just fired up a web browser myself and navigated to that URL.
Since I have 2FA setup on github that makes automating the login process from the github landing page somewhat annoying, so I'd like to avoid automating logging into github. The tasks that I want to automate do not require re-authenticating via 2FA.
ANSWER:
For MacOSX+Ruby+Selenium this works:
#!/usr/bin/env ruby
require 'selenium-webdriver'
caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {"debuggerAddress" => "127.0.0.1:20480"}, detach: false)
driver = Selenium::WebDriver.for :chrome, :desired_capabilities => caps
driver.navigate.to "https://github.com"
Then fire up chrome with this:
% /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=/Users/lamont/Library/Application\ Support/Google/Chrome --profile-directory=Default --remote-debugging-port=20480
Obviously the paths will need to be adjusted because they're OSX-centric and have my homedir in them.
There is also a bug in the selenium-webdriver gem for ruby where it inserts a 'detach' option which gets into a fight with 'debuggerAddress':
/Users/lamont/.rvm/gems/ruby-2.2.4/gems/selenium-webdriver-2.53.0/lib/selenium/webdriver/remote/response.rb:70:in `assert_ok': unknown error: cannot parse capability: chromeOptions (Selenium::WebDriver::Error::UnknownError)
from unknown error: unrecognized chrome option: detach
The lib/selenium/webdriver/chrome/bridge.rb file can be edited to take that out as a quick hack:
chrome_options['binary'] = Chrome.path if Chrome.path
chrome_options['nativeEvents'] = true if native_events
chrome_options['verbose'] = true if verbose
#chrome_options['detach'] = detach.nil? || !!detach
chrome_options['noWebsiteTestingDefaults'] = true if no_website_testing_defaults
chrome_options['prefs'] = prefs if prefs
To implement something similar in Ruby, check out this page that goes over that. Thanks to lamont for letting me know in the comments.
You can start chrome using a specific Chrome profile. I am not sure what the ruby implementation would look like, but in python it looks something like:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
options = ChromeOptions()
# more on this line here later.
options.add_experimental_option('debuggerAddress', '127.0.0.1:7878')
driver = webdriver.Chrome(chrome_options=otpions)
In order for this to work you need to do a few things.
manually start chrome from terminal/command prompt with these command line arguments
--user-data-dir=/path/to/any/custom/directory/home/user/Desktop/Chromedir --profile-directory="Profile 1" --remote-debugging-port=7878
make sure "Profile 1" is already existing in the same --user-data-dir (make sure user Profile 1 has necessary chrome://components/
to run any apps that require those components)
you can use any free port in place of 7878
verify that http://localhost:7878 is running and returns value.
This should manually launch chrome with the "Profile 1" profile, and so long as it has logged into the site in question, it will stay logged in like a normal user so long as you follow these instructions to run the tests.
I used this to write a quick netflix bot that clicks the "continue playing" button when it pops up, and it's the only way to get DRM content to play as far as I have found. But it retains the cookies for the login, and also launches chrome with whatever components the profile is set up to have.
I have tried launching chrome with specific profiles before using different methodologies, but this was the only way to really force it to work how I wanted it to.
Edit: There are methods for saving cookie info as well although I don't know how well they work. Check out this link for more info, as my solution is probably not the best solution even if it works.
The show_me_the_cookies gem provides cross-driver cookie manipulation and can let you add new cookies. The one thing to be aware of when using selenium is that you need to visit the domain before you can create cookie for it, so you'll need to do something like
visit "https://www.github.com"
create_cookie(...)
visit "https://www.github.com"
for it to work - first visit just puts the browser/driver in a state where you can create the cookie, second visit actually goes to the page with the cookies set.
I had to tweak the OP's answer (from within her question) to get this going with Ruby in 2022.
Prerequisites
Chromedriver installed and allowed to run even though it's not signed:
> brew install chromedriver
> xattr -d com.apple.quarantine /usr/local/bin/chromedriver
Chrome launched and accepting commands on a specific port:
> /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=~/Library/Application\ Support/Google/Chrome --profile-directory=Default --remote-debugging-port=20480
This created a new profile in Chrome so I signed in to my account and got the browser set up, ready to start interacting with the (legacy EdTech) site I'm trying to automate.
Actual use
require 'selenium-webdriver'
caps = Selenium::WebDriver::Remote::Capabilities.chrome("goog:chromeOptions" => {"debuggerAddress" => "127.0.0.1:20480"})
driver = Selenium::WebDriver.for :chrome, capabilities: caps
driver.navigate.to "https://www.google.com"
Is there any way to dynamically change the proxy being used by Firefox when using selenium webdriver?
Currently I have proxy support using a proxy profile but is there a way to change the proxy when the browser is alive and running?
My current code:
proxy = Proxy({
'proxyType': 'MANUAL',
'httpProxy': proxy_ip,
'ftpProxy': proxy_ip,
'sslProxy': proxy_ip,
'noProxy': '' # set this value as desired
})
browser = webdriver.Firefox(proxy=proxy)
Thanks in advance.
This is a slightly old question.
But it is actually possible to change the proxies dynamically thru a "hacky way"
I am going to use Selenium JS with Firefox but you can follow thru in the language you want.
Step 1: Visiting "about:config"
driver.get("about:config");
Step 2 : Run script that changes proxy
var setupScript=`var prefs = Components.classes["#mozilla.org/preferences-service;1"]
.getService(Components.interfaces.nsIPrefBranch);
prefs.setIntPref("network.proxy.type", 1);
prefs.setCharPref("network.proxy.http", "${proxyUsed.host}");
prefs.setIntPref("network.proxy.http_port", "${proxyUsed.port}");
prefs.setCharPref("network.proxy.ssl", "${proxyUsed.host}");
prefs.setIntPref("network.proxy.ssl_port", "${proxyUsed.port}");
prefs.setCharPref("network.proxy.ftp", "${proxyUsed.host}");
prefs.setIntPref("network.proxy.ftp_port", "${proxyUsed.port}");
`;
//running script below
driver.executeScript(setupScript);
//sleep for 1 sec
driver.sleep(1000);
Where use ${abcd} is where you put your variables, in the above example I am using ES6 which handles concatenation as shown, you can use other concatenation methods of your choice , depending on your language.
Step 3: : Visit your site
driver.get("http://whatismyip.com");
Explanation:the above code takes advantage of Firefox's API to change the preferences using JavaScript code.
As far as I know there are only two ways to change the proxy setting, one via a profile (which you are using) and the other using the capabilities of a driver when you instantiate it as per here. Sadly neither of these methods do what you want as they both happen before as you create your driver.
I have to ask, why is it you want to change your proxy settings? The only solution I can easily think of is to point firefox to a proxy that you can change at runtime. I am not sure but that might be possible with browsermob-proxy.
One possible solution is to close the webdriver instance and create it again after each operation by passing a new configuration in the browser profile
Have a try selenium-wire, It can even override header field
from seleniumwire import webdriver
options = {
'proxy': {
"http": "http://" + IP_PORT,
"https": "http://" + IP_PORT,
'custom_authorization':AUTH
},
'connection_keep_alive': True,
'connection_timeout': 30,
'verify_ssl': False
}
# Create a new instance of the Firefox driver
driver = webdriver.Firefox(seleniumwire_options=options)
driver.header_overrides = {
'Proxy-Authorization': AUTH
}
# Go to the Google home page
driver.get("http://whatismyip.com")
driver.close()
For whatever reason, watir changes my Firefox browser settings without my instruction to do this.
If I open Firefox manually (at the office), it normally is set to 'No Proxy' and works just fine.
However, if watir launches a new FF instance, it is set to 'Manual Proxy', and a a browser error states that it's trying to use a proxy server which is refusing connection.
We have no proxy server at work.
How do I get watir to launch a Firefox window with No Proxy? I've searched the web and have not found a single example of this.
(BTW, the FF settings from a watir-launched session are independent of normal FF settings; in other words, manually changing FF settings doesn't correct the problem).
Based on the information here, there are 5 different proxy configurations for Firefox:
0
Direct connection, no proxy. (Default in Windows and Mac previous to 1.9.2.4 /Firefox 3.6.4)
[edit]
1
Manual proxy configuration.
[edit]
2
Proxy auto-configuration (PAC).
[edit]
4
Auto-detect proxy settings.
[edit]
5
Use system proxy settings. (Default in Linux; default for all platforms, starting in 1.9.2.4 /Firefox 3.6.4)
It looks like "0" is the one you need. We set that as described on the Watir-Webdriver help page for Firefox:
profile = Selenium::WebDriver::Firefox::Profile.new
profile["network.proxy.type"] = 0
browser = Watir::Browser.new :firefox, :profile => profile
All of the profile["lorem ipsum"] type options are listed in the about:config menu URL in Firefox, and are accessed/changed in a similar fashion.