I have code with the Chrome browser and this works, but I never worked with Firefox, but an Ubuntu server normally works only with Firefox, and now I have a question: How can I work with a proxy on the Firefox browser using the proxy_chain_rb gem?
I think my code for the Chrome browser will work in Firefox if you tell me how I can make Firefox options. My problem - I don’t know how I can use Firefox options and manuals are old. How can I replace my code for Google in Firefox?
Code
require 'watir'
require 'proxy_chain_rb'
require 'selenium-webdriver'
time2 = Time.now
file = File.new("report.json", "a:UTF-8")
myuseragent = File.readlines("user_agents.txt").sample
options = Selenium::WebDriver::Chrome::Options.new
options.add_emulation(user_agent: (myuseragent))
options.add_argument('--headless')
puts "Work started: " + time2.inspect
u_proxy = File.readlines("proxy.txt").sample
real_proxy = u_proxy
server = ProxyChainRb::Server.new
generated_proxy = server.start(real_proxy)
proxy = {
http: generated_proxy,
ssl: generated_proxy
}
caps = Selenium::WebDriver::Remote::Capabilities.chrome(:proxy => proxy)
driver = Selenium::WebDriver.for :chrome, :desired_capabilities => caps, options: options
driver.execute_script('return navigator.userAgent')
driver.get "https://raskruty.ru/"
Related
I have a problem setting the preferred (accepted language) within headless Chrome using Selenium Webdriver and Ruby. I use the following WebDriver settings:
Selenium::WebDriver::Chrome.driver_path = #config[<path to the Chrome Driver>]
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--headless')
options.add_argument('--disable-translate')
options.add_argument("--lang=de")
The driver is then initialized with:
#selenium_driver = Selenium::WebDriver.for :chrome, options: options
Everything works fine but at some pages Chrome returns English content even when I navigate to the German page URL (e.g. page.de). In these cases the Chrome driver returns the English content due to an internal forwarding to page.de/en. I do not specify the en path in my queried URL.
I have tried to set the language using the Webdriver preference:
options.add_preference('accept_languages', 'de')
instead of the add_argument but it doesn't change anything of the behavior.
Does anyone have an idea how to force a headless Chrome controlled by Selenium Webdriver within Ruby to request page content in a defined language or - not optimal but it might help as a workaround - to stop the forwarding?
Any help greatly appreciated
Best
Krid
I found a solution that works for me. As in many cases the problem was sitting in front of the screen and simply doesn't work precisely enough ;-)
Instead of using
options.add_argument("--lang=de")
you have to use
options.add_argument("--lang=de-DE")
When I use an IETF language tag the code I initially posted works as expected.
I'am using this in my test_helper.rb Works fine for me.
Capybara.register_driver :selenium do |app|
Chromedriver.set_version "2.36"
desired_capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
'chromeOptions' => {
'prefs' => {
'intl.accept_languages' => 'en-US'
},
args: ['disable-gpu', 'headless']
}
)
Capybara::Selenium::Driver.new(app, { browser: :chrome, desired_capabilities: desired_capabilities })
end
Capybara.javascript_driver = :chrome
Capybara.default_driver = :selenium
This prefs hash inside an options hash did the trick for me. It's at the end of the driven_by :selenium line.
(Inside test/application_syste_test_case.rb)
# frozen_string_literal: true
require 'test_helper'
require 'capybara/rails'
class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
driven_by :selenium, using: :chrome, screen_size: [1400, 1400], options: { prefs: { 'intl.accept_languages' => 'de,de-DE;q=0.9,en;q=0.1' } }
# ...
2021-06-14 UPDATE:
The previous example produces this deprecation warning:
WARN Selenium [DEPRECATION] :prefs is deprecated. Use Selenium::WebDriver::Chrome::Options#add_preference instead.
IMO, the solution below is uglier, but I'm posting it for when it's fully deprecated and the original stops working.
class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
driven_by(:selenium,
using: :chrome,
screen_size: [1400, 1400],
options: {
options: Selenium::WebDriver::Chrome::Options.new(
prefs: { 'intl.accept_languages' => 'de,de-DE;q=0.9,en;q=0.1' }
)
},
)
You should be able to solve your problem by adding an experimental option:
options.add_option('prefs', {'intl.accept_languages': 'en,en_US'})
I'm sure it works with Python, but I've not tried with Ruby: this approach is the correct one, not sure about the implementation.
You can find in this repository the code which handles your problem in Python code, and in this Q&A how to implement experimental_options in Ruby
For me works:
options = Selenium::WebDriver::Firefox::Options.new
options.add_preference("intl.accept_languages", 'de-DE')
Capybara::Selenium::Driver.new(app, browser: :firefox, options: options)
The below java code works fine in my machine to disable automation extension.
How I can write a replica of this code in ruby
Java
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("useAutomationExtension", false);
In Watir Ruby
this code doesn't work
Watir::Browser.new(:chrome, :switches => %w[--disable-popup-blocking --disable-extensions --disable-infobars] )
Maybe try passing it as a raw option:
b = Watir::Browser.new :chrome, options: {options: {'useAutomationExtension' => false}}
You should be able to pass them along with args: See the update of Watir 6.6: http://watir.com/watir-6-6/
In short:
Watir::Browser.new :chrome, options: {args: %w[--disable-popup-blocking --disable-extensions --disable-infobars]}
This is how I like to start the browser keeping all options variable.
browser_name = ENV['BROWSER'] || 'chrome'
settings = {}
if ENV['DEVICE'] == 'remote'
settings[:url] = 'http://selenium__standalone-chrome:4444/wd/hub/'
end
if browser_name == 'chrome'
args = ['--ignore-certificate-errors', '--disable-popup-blocking', '--disable-translate', '--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream']
settings[:options] = {args: args}
end
Watir::Browser.new browser_name.to_sym, settings
I found the root cause which is not allowing the script to run.
I navigated to below path in "regedit" and deleted the "ExtensionInstallBlacklist" folder but this is a temporary solution the registry will be auto created in some time.
Path:
“HKEY_CURRENT_USER\Software\Policies\Google\Chrome\ExtensionInstallBlacklist”
The code is given Below , where it is launching three firefox browser
, all browser has different proxy settings. Using watir how launch all three browser same time using tread in watir???
require 'selenium-webdriver'
require 'rubygems'
require 'watir'
require 'rautomation'
require './CLReport.class'
require 'win32ole'
# TO INITIATE FIRST FIREFOX BROWSER
# THE PROXY DATA CAN BE parameterized from Excel sheet
profile = Selenium::WebDriver::Firefox::Profile.new
profile.proxy = Selenium::WebDriver::Proxy.new :http => 'myproxy.com:8080', :ssl => 'myproxy.com:8080'
$b1 = Watir::Browser.new :firefox, :profile => profile
$b1.goto("https://google.com")
# TO INITIATE SECOND FIREFOX BROWSER
# THE PROXY DATA CAN BE parameterized from Excel sheet
profile = Selenium::WebDriver::Firefox::Profile.new
profile.proxy = Selenium::WebDriver::Proxy.new :http => 'myproxy.com:8081', :ssl => 'myproxy.com:8081'
$b2 = Watir::Browser.new :firefox, :profile => profile
$b2.goto("https://google.com")
# TO INITIATE THORD FIREFOX BROWSER
# THE PROXY DATA CAN BE parameterized from Excel sheet
profile = Selenium::WebDriver::Firefox::Profile.new
profile.proxy = Selenium::WebDriver::Proxy.new :http => 'myproxy.com:8082', :ssl => 'myproxy.com:8082'
$b3 = Watir::Browser.new :firefox, :profile => profile
$b3.goto("https://google.com")
Now my question is how to join $b1,$b2,$b3 in a single browser using thread so that
only $browser.link(:text, "form application")click should work for all three browser parallely insted of writing
$b1.link(:text, "form application").click
$b2.link(:text, "form application").click
$b3.link(:text, "form application").click
i.e single line of code work work in three firefox browser same time parallely.
It is not possible because $b1,$b2,$b3 are instances of different browser,You can not make them equal.What are you doing is right. Or You can do some thing like that.
array = [$b1,$b2,$b3]
array.each { |browser|
browser.link(:text, "form application").click
}
I have a ruby web crawler that is currently coded to run in firefox. How do I switch it over to Chrome instead?
def open_browser()
tweaked_profile = Selenium::WebDriver::Firefox::Profile.new
tweaked_profile['nglayout.initialpaint.delay'] = 0
tweaked_profile.assume_untrusted_certificate_issuer=false
tweaked_profile['permissions.default.image'] = 2
tweaked_profile['network.proxy.type'] = 1
tweaked_profile['network.proxy.http'] = 'ec2proxy.csnzoo.com'
tweaked_profile['network.proxy.http_port'] = 8080
driver = Selenium::WebDriver.for :firefox, :profile => tweaked_profile
$browser = Watir::Browser.new(driver)
end
Should I just ditch watir and go with chromedriver or will watir work for this?
Check out http://watirwebdriver.com/chrome/, which has this example:
profile = Selenium::WebDriver::Chrome::Profile.new
...
b = Watir::Browser.new :chrome, :profile => profile
Also, these SO questions provide alternatives for crawling sites: Web crawler in ruby and What are some good Ruby-based web crawlers?
I learned how to use Firefox 4 with watir and webdriver (on Win7 x64), setting profile items. Example:
profile = Selenium::WebDriver::Firefox::Profile.new
profile["browser.download.useDownloadDir"] = true
profile["browser.download.dir"] = 'D:\\FirefoxDownloads'
profile["browser.helperApps.neverAsk.saveToDisk"] = "application/csv"
driver = Selenium::WebDriver.for :firefox, :profile => profile
browser = Watir::Browser.new(driver)
What I try to do with the example below, is setting CSV files to be always downloaded to a specific directory, never opened.
The code above succeeds in setting all the files automatically downloaded to the specified directory, but setting browser.helperApps.neverAsk.saveToDisk has no effect: I still get the open/save question.
After the script runs, the Firefox window is still open, and I enter the URL about:config.
I can see that browser.helperApps.neverAsk.saveToDisk was correctly set to application.csv , but in firefox/options/options/applications I don't see the entry for CSV files.
It seems that the menu setting, that is really effective, is not really bound with the about:config setting.
What am I doing wrong?
I've done some testing of this for you, unfortunately there doesn't seem to be a standard content-type for CSV files. You can try passing a comma separated list of content-types, hopefully one of those work for you. For me it was application/octet-stream that did the trick...
require 'watir-webdriver'
require 'selenium-webdriver'
profile = Selenium::WebDriver::Firefox::Profile.new
profile["browser.download.useDownloadDir"] = true
profile["browser.download.dir"] = '/tmp'
profile["browser.helperApps.neverAsk.saveToDisk"] = "text/plain, application/vnd.ms-excel, text/csv, text/comma-separated-values, application/octet-stream"
driver = Selenium::WebDriver.for :firefox, :profile => profile
browser = Watir::Browser.new(driver)
browser.goto "http://altentee.com/test/test.csv"
In Firefox 6+, I couldn't get this to work without specifically setting the 'browser.download.folderList' value:
profile = Selenium::WebDriver::Firefox::Profile.new
profile['browser.download.folderList'] = 2 #custom location
profile['browser.download.dir'] = download_directory
profile['browser.helperApps.neverAsk.saveToDisk'] = "text/csv, application/csv"
b = Watir::Browser.new :firefox, :profile => profile
See: http://watirwebdriver.com/browser-downloads/