Very odd behavior from Firefox (geckodriver) with Selenium webdriver. Given the exact same parameters as Chrome (chromedriver) with Selenium, geckodriver seems to ignore proxy settings and connect directly to the Internet. Any reason why geckodriver operates this way or how to work around it?
def selenium_chrome_full(proxy, url)
options = Selenium::WebDriver::Chrome::Options.new(args: ["start-maximized", "--proxy-server=%s" % proxy])
driver = Selenium::WebDriver.for(:chrome, capabilities: options)
driver.get(url)
end
def selenium_firefox_full(proxy, url)
options = Selenium::WebDriver::Firefox::Options.new(args: ["start-maximized", "--proxy-server=%s" % proxy])
driver = Selenium::WebDriver.for(:firefox, capabilities: options)
driver.get(url)
end
proxy = "0.0.0.0:8080"
url = "https://www.google.com/search?client=firefox-b-1-d&q=whatsmyip"
selenium_firefox_full(proxy, url)
selenium_chrome_full(proxy, url)
For a bit of additional context, I also ran a debugger to see the value of options after initialization they are as follows for Firefox and Chrome respectively:
#<Selenium::WebDriver::Firefox::Options:0x000000010814b390 #debugger_address=nil, #options={:args=>["start-maximized", "--proxy-server=0.0.0.0:8080"], :browser_name=>"firefox", :prefs=>{}}, #profile=nil>
#<Selenium::WebDriver::Chrome::Options:0x0000000102362150 #options={:args=>["start-maximized", "--proxy-server=0.0.0.0:8080"], :prefs=>{}, :emulation=>{}, :local_state=>{}, :exclude_switches=>[], :perf_logging_prefs=>{}, :window_types=>[], :browser_name=>"chrome"}, #profile=nil, #logging_prefs={}, #encoded_extensions=[], #extensions=[]>
Related
I`m trying to write some tools with chrome Chrome DevTools Protocol https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-enable.
I want get page ResponseBody and I don't know where can I find RequestId. So here is my simple Ruby code:
chrome = ChromeRemote.client
# Enable events
chrome.send_cmd("Network.enable")
chrome.send_cmd("Page.enable")
puts chrome.send_cmd "Network.getCookies"
# for this command I need RequestId ->
puts chrome.send_cmd "Network.getResponseBody"
For now I have empty result for puts chrome.send_cmd "Network.getResponseBody".
You need to listen to the requestWillBeSent event, which would give you both URL and requestId
How can I turn off incognito mode when I start Chrome with Ferrum?
This is my code, I was trying to use user-data-dir, but it didn`t help me
require "ferrum"
BROWSER_PATH="C:/Program Files/Google/Chrome/Application/chrome.exe"
# run bot
def akira
chrome = Ferrum::Browser.new(
browser_path:BROWSER_PATH,
headless:false,
browser_options:{
'no-sandbox':nil,
'--incognito':false,
'user-data-dir':'E:/ruby/akira/session',
}
)
chrome.go_to("https://google.com")
chrome.screenshot(path: "google.png")
chrome.quit
end
akira
chrome = Ferrum::Browser.new(
browser_path:BROWSER_PATH,
headless:false,
browser_options:{
'no-sandbox':nil,
'incognito':nil
}
user_data_dir: 'E:/ruby/akira/session'
)
I am currently using selenium and crawling a website.
I have tested if I could set a proxy server on Selenium.
But now, I want to set a paid rental proxy server and I got a trial IP address whose the format looks like this IP:PORT:USER:PASS.
And I don't know how to set USER:PASS.
The provider didn't know how to set in Selenium.
So I don't know what I can do now.
With random proxy this worked fine.
proxy_host = '185.186.61.44'
proxy_port = '11334'
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument("--proxy-server=http://#{proxy_host}:#{proxy_port}")
So I wanted to set something like this.
proxy_host = '185.186.61.44'
proxy_port = '12323'
proxy_user = "7a2345129"
proxy_pass = "easdga341d4"
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument("--proxy-server=http://#{proxy_host}:#{proxy_port}:#{proxy_user}:#{proxy_pass}")
but I found that it was not that easy as I read some solution that uses puppeteer.
I wonder if there are any solution for my case.
If anybody has any clues I would love you to tell me.
Thank you.
Selenium 4 added support for basic auth, which at the time of writing is Chrome specific.
See here for more details.
To specify basic auth creds:
driver.devtools.new
driver.register(username: 'username', password: 'password')
Example using scraperapi.com as proxy
require 'selenium-webdriver'
proxy = Selenium::WebDriver::Proxy.new(
http: 'proxy-server.scraperapi.com:8001',
ssl: 'proxy-server.scraperapi.com:8001'
)
cap = Selenium::WebDriver::Remote::Capabilities.chrome(proxy: proxy)
options = Selenium::WebDriver::Chrome::Options.new(
args: [
'--no-sandbox',
'--headless',
'--disable-dev-shm-usage',
'--single-process',
'--ignore-certificate-errors'
]
)
driver = Selenium::WebDriver.for(:chrome, capabilities: [options,cap])
driver.devtools.new
driver.register(username: 'scraperapi', password: 'xxxx')
driver.navigate.to("http://httpbin.org/ip")
puts "content: #{driver.page_source}"
The Chrome::Options above are specific to my usecase, expect for the ignore-certificate-errors option which is needed to handle https traffic using scraperapi's proxies.
gemfile had:
gem 'selenium-devtools', '~> 0.91.0'
gem 'selenium-webdriver', '~> 4.1'
The format of an URL is most like
proto://USER:PASS#host.domain.tld:port
So your code have to look like :
proxy_host = '185.186.61.44'
proxy_port = '12323'
proxy_user = "7a2345129"
proxy_pass = "easdga341d4"
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument("--proxy-server=http://{proxy_user}:#{proxy_pass}##{proxy_host}:#{proxy_port}:#")
I'm trying to download .csv file using selenium tool in python but after clicked on save file link in web page, the browser is showing me dialog box.I tried "alert" to handle this but it gives me error like, alert is not present....
I'm using python as a scripting language and selenium as a tool.
here is the my code:
fp = webdriver.FirefoxProfile()
fp.set_preference('browser.download.folderList', 2) # custom location
fp.set_preference('browser.download.manager.showWhenStarting', False)
fp.set_preference('browser.download.dir', 'C:\Temp\')
fp.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/csv,text/csv,text/comma-separated-values, application/octet-stream')
driver = webdriver.Firefox(profile)
driver.window_handles
driver.swich_to_window(driver.window_handles[0])
url=''
driver.get(url)
pass=driver.find_element_by_id('pswd')
pass.send_keys('xyz123')
driver.find_element_by_id('btnLogin').click()
driver.implicitly_wait(30)
driver.find_element_by_link_text('FileName').click()
driver.switch_to_alert().accept()
Finally I am done with Save file dialog box. I have handled it with some firefox profile preference settings. Since I was not unable to find out the exact MIME type I used all possible.
Below is my code to handle the Save file Dialog box :
fp = webdriver.FirefoxProfile()
fp.set_preference('browser.download.folderList', 2)
fp.set_preference('browser.download.manager.showWhenStarting', False)
fp.set_preference('browser.download.dir', r'C:\Temp\')
fp.set_preference('browser.helperApps.neverAsk.openFile', 'text/csv,application/x-msexcel,application/excel,application/x-excel,application/vnd.ms-excel,image/png,image/jpeg,text/html,text/plain,application/msword,application/xml')
fp.set_preference('browser.helperApps.neverAsk.saveToDisk', 'text/csv,application/x-msexcel,application/excel,application/x-excel,application/vnd.ms-excel,image/png,image/jpeg,text/html,text/plain,application/msword,application/xml')
fp.set_preference('browser.helperApps.alwaysAsk.force', False)
fp.set_preference('browser.download.manager.alertOnEXEOpen', False)
fp.set_preference('browser.download.manager.focusWhenStarting', False)
fp.set_preference('browser.download.manager.useWindow', False)
fp.set_preference('browser.download.manager.showAlertOnComplete', False)
fp.set_preference('browser.download.manager.closeWhenDone', False)
driver = webdriver.Firefox(fp)
driver.window_handles
driver.switch_to_window(driver.window_handles[0])
url=''
driver.get(url)
pass=driver.find_element_by_id('pswd')
pass.send_keys('xyz123')
driver.find_element_by_id('btnLogin').click()
driver.implicitly_wait(30)
driver.find_element_by_link_text('FileName').click()
time.sleep(5)
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
import time
import pyautogui
try :
driver = webdriver.Firefox()
driver.implicitly_wait(30)
driver.maximize_window()
driver.get("https://www.citysdk.eu/wp-content/uploads/2013/09/DELIVERABLE_WP4_TA_SRS_0.21.pdf")
# WebDriverWait(driver, 10).until(lambda d: d.execute_script('return document.readyState') == 'complete')
# Click the OK button and close
time.sleep(5)
webelem = driver.find_element_by_id('download')
webelem.click()
time.sleep(5)
print('press enter')
pyautogui.press('enter')
except Exception as err:
print('ERROR: %sn' % str(err))
driver.quit()
How to disable this "first run" page once and for all for FF?
When FF driver is created, it opens tab with -
https://www.mozilla.org/en-US/firefox/42.0/firstrun/learnmore/
and additional tab with target page.
To turn off this annoying start page:
in C# with Selenium 2.48 I found the following solution:
FirefoxProfile prof = new FirefoxProfile();
prof.SetPreference("browser.startup.homepage_override.mstone", "ignore");
prof.SetPreference("startup.homepage_welcome_url.additional", "about:blank");
Driver = new FirefoxDriver(prof);
...and it will never bother you again.
Note: One of these settings alone will also work. I use them together to make it bullet-proof.
Ifound a solution, works fine
FirefoxProfile fp = new FirefoxProfile();
fp.setPreference("browser.startup.homepage", "about:blank");
fp.setPreference("startup.homepage_welcome_url", "about:blank");
fp.setPreference("startup.homepage_welcome_url.additional", "about:blank");
I have faced the same problem. I have just changed the Selenium version to 2.48 and the problem solved.
This is caused by incompatibility between Selenium and Firefox versions, but not by any one specific version number.
You should be 1-2 Firefox versions behind the newest, if your WebDriver is on the latest version. Otherwise, roll the Firefox version back even further if your WebDriver is older, or upgrade Webdriver.
To get an older Firefox, try https://ftp.mozilla.org/pub/firefox/releases/ or http://www.oldapps.com/
or on Linux, depending on your distro
yum list --showduplicates firefox
sudo yum install firefox-<version>
or
apt-cache show firefox | grep Version
sudo apt-get install firefox=<version>
C# solution, upgraded Selenium WebDriver to 2.49.0 solved the issue for me.
The above solutions do not work, I've tried them.
What did work for me, and probably will for you (if using firefox 43 or less) is:
prof.setPreference("xpinstall.signatures.required", false);
prof.setPreference("toolkit.telemetry.reportingpolicy.firstRun", false);
The problems with 43 and selenium are twofold, the default signed extensions setting (to true) and the first run page. These lines solve both. These must be set programatically. If you try to set them in about:config (or directly in prefs.js) it will not affect the new browsers you open with selenium. It should be noted that they say firefox 44 will not allow you to set the signed extensions variable (so this will not work on 44).
I include the codeblock from my now working code showing the correct use:
FirefoxProfile prof = new FirefoxProfile();
//FirefoxProfile prof = profile.getProfile("default");
//prof.setPreference("browser.startup.homepage", proteinPageUrl);
//prof.setPreference("startup.homepage_welcome_url", proteinPageUrl);
//prof.setPreference("startup.homepage_welcome_url.additional", proteinPageUrl);
prof.setPreference("xpinstall.signatures.required", false);
prof.setPreference("toolkit.telemetry.reportingpolicy.firstRun", false);
//Object socketLock = new Object();
//synchronized(socketLock){
//driver = new FirefoxDriver();
driver = new FirefoxDriver(prof);
//driver = forceInit();
//driver.open();
//}//end synch block
//get protein page
boolean done = true;
do{
driver.get(proteinPageUrl);
final Wait<WebDriver> waitDriver = new FluentWait<WebDriver>(driver)
.withTimeout(30, java.util.concurrent.TimeUnit.SECONDS)
.pollingEvery(5, java.util.concurrent.TimeUnit.SECONDS);
try{
inputTextFeildElement = waitDriver.until(new Function<WebDriver,WebElement>(){
public WebElement apply(WebDriver diver){
return driver.findElement(By.name("term"));
}});
}
catch(NoSuchElementException nsee){
//if not find by name try find by id
if(driver.findElements(By.id("term")).size() != 0){
try{
inputTextFeildElement = driver.findElement(By.id("term"));
done = true;
} catch(NoSuchElementException nsee2){
synchronized(threadLogFile){
try {
threadLogWriter = new PrintWriter(new FileWriter(threadLogFile.getAbsoluteFile(), true));
} catch (IOException ioe) {
System.out.println("error opening file for append: " + ioe.getMessage());
ioe.printStackTrace();
}//catch
threadLogWriter.println("Thread Id: " + threadId + " with thread name: " + threadName + " fails to find input element by name or id to put accession: " + accession);
threadLogWriter.flush();
threadLogWriter.close();
}//synchronized
done = false;
}//catch nsee2
}//catch nsee
}
catch(ElementNotVisibleException enve){
done = false;
}
}while(!done);
If you using selenium webdriver from Capybara/Cucumber, then you can change the default url when you register your driver using startup.homepage_welcome_url.additional:
Capybara.register_driver :firefox do |app|
profile = Selenium::WebDriver::Firefox::Profile.new
profile['browser.startup.homepage_override.mstone'] = 'ignore'
profile['startup.homepage_welcome_url.additional'] = 'about:blank'
Capybara::Selenium::Driver.new(app, :browser => :firefox, :profile => profile)
end
I faced the same problem.
My solution:
I downgraded Firefox to 36.0.
It worked fine with Selenium 2.53.1.
I hope this help. :)
The problem here for me was, I was using firefox 46.0.1 for testing with latest version of selenium driver which of course worked fine with modern version of the firefox but acted weird with firefox 46.0.1.
To get it working, I had to downgrade the Selenium version to 2.53.0 from 4.4.0 and the respective dependent references should be brought down too.