I have a few web-scraping scripts that I've been using for a while now that have been working without issue. However because of an update of something somewhere (I think chrome+chromedriver), the the browsers are not loading with the preferences/options I specify.
Current code:
preferences = {
:download => {
:prompt_for_download => false,
:directory_upgrade => true,
:default_directory => 'C:/DownloadFolder/',
}
}
args = ['--disable-infobars']
browser = Watir::Browser.new :chrome, :chrome_options => {:detach => true, :prefs => preferences, :args => args}
The problems I'm noticing are that the '--disable-infobars' and download folder location are not being applied.
ruby version: 2.3.3p222
watir version: 6.16.5
selenium webdriver version: 3.142.3
chrome version: 75.0.3770.100
chromedriver version : 75.0.3770.90
Taken from help I got elsewhere:
options = Selenium::WebDriver::Chrome::Options.new.tap do |o|
o.add_argument('--disable-infobars')
o.add_preference(:download, directory_upgrade: true,
prompt_for_download: false,
default_directory: 'C:\\DownloadFolder\\')
o.add_option(:detach, true)
end
browser = Watir::Browser.new :chrome, options: options
Two things solved the problem. 1 is specifying the options through selenium rather that Watir. The other is no longer being able to use single forward slashes in folder paths.
Related
I was trying to test with Selenium, but can not download a pdf, pdfs keep opening.
See my other post: RUBY: Selenium webdriver, setup to download pdf files instead of opening them
It was advised to try Watir, so I did, and I get the same result. Here is my Watir setup. Please advise on how to fix this issue.
require 'watir'
require 'pry'
prefs = {
download: {
prompt_for_download: false,
default_directory: '/Users/ar/pdf_downloads'
}
}
browser = Watir::Browser.new :chrome, options: {prefs: prefs}
# Goto Login Page (file)
url="file:///Users/ar/info.html"
browser.goto url
browser.button(id: 'formsubmit').click
sleep 5
# Goto info
info_url = 'https://webapp.domain.com/info'
browser.goto info_url
sleep 5
elements = browser.elements(css: "#ar-pdfreport a")
link = elements.first.attribute("href")
browser.goto link
There is a bug in Selenium-WebDriver v3.142.7 where using symbols for the prefs generates the wrong result - eg does not set the download directory. See https://github.com/SeleniumHQ/selenium/issues/7917 for more details.
Switch the symbols to Strings:
prefs = {
download: {
'prompt_for_download' => false,
'default_directory' => '/Users/ar/pdf_downloads'
},
plugins: {
'always_open_pdf_externally' => true
}
}
I try to download a file using headless chrome and the file doesn't seems to be getting downloaded anywhere. I could see that it's actually a security feature to restrict file download in headless but, is there a workaround for the same in Ruby? Tried the below code but no luck.
download_path = "#{Pathname.pwd}/test-data/downloaded"
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument("--disable-dev-shm-usage");
options.add_argument('--headless') #Declaring the browser to run in headless mode
options.add_preference(
:download, directory_upgrade: true,
prompt_for_download: false,
default_directory: download_path
)
options.add_preference(:browser, set_download_behavior: { behavior: 'allow' })
#driver = Selenium::WebDriver.for :chrome, options: options #Browser object initialization
set_screen_resolution(1400, 900)
$driver = #driver
bridge = #driver.send(:bridge)
path = '/session/:session_id/chromium/send_command'
path[':session_id'] = bridge.session_id
bridge.http.call(:post, path, cmd: 'Page.setDownloadBehavior',
params: {
behavior: 'allow',
downloadPath: download_path
})
I expect the file to be downloaded using headless chrome but it's not happening.
When you click on a download link and if it opens in a separate tab before the file starts downloading, then you need to apply your above mentioned script to the newly opened tab too, because you've set the session id only for the current tab and not for the newly opened tab. So, try apply this script to that newly opened tab before trying to download a file. I'm sure it'll work.
def download_file(label, download_path)
ele = Locator.new(:xpath, "//ul[#class='cnt']//div[text()='#{label}']/..//a")
download_url = get_attribute(ele.get_how, ele.get_what, "href")
#driver.execute_script("window.open()")
#driver.switch_to.window(#driver.window_handles.last)
bridge = #driver.send(:bridge)
path = '/session/:session_id/chromium/send_command'
path[':session_id'] = bridge.session_id
bridge.http.call(:post, path, {
"cmd" => 'Page.setDownloadBehavior',
"params" => {
"behavior" => 'allow',
"downloadPath" => download_path
}
})
#driver.get(download_url)
#driver.close
#driver.switch_to.window(#driver.window_handles.last)
end
I have a Flash-based app that I need to test using Cucumber. As flash is not enabled by default I need to enable it before each test and whitelist the url I believe. If I pause the test in it's background phase I can manually set these options.
How can I automate this approach though, I have looked into adding options and preferences, but still cannot seem to get to work.
So this is my standard setup in an env.rb file
Capybara.register_driver :chrome do |app|
chrome_binary = '/Applications/Google Chrome.app'
capabilities = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => { "binary" => chrome_binary + '/Contents/MacOS/Google Chrome' })
Capybara::Selenium::Driver.new(app, :browser => :chrome, :desired_capabilities => capabilities, :options => options)
end
Further reading has highlighted options such as
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('arg-here')
prefs = {"enable flash here ? "}
options.add_experimental_option("prefs", prefs)
The add_experimental_option throws undefined method add_experimental_option for #<Selenium::WebDriver::Chrome::Options:0x007fca30c10988>
Has anyone automated this process?
To enable Flash before each test and WhiteList the url you can use the following code block to configure the WebDriver instance to allow Flash:
ChromeOptions options = new ChromeOptions();
Map<String, Object> prefs = new HashMap<String, Object>();
prefs.put("profile.default_content_setting_values.plugins", 1);
prefs.put("profile.content_settings.plugin_whitelist.adobe-flash-player", 1);
prefs.put("profile.content_settings.exceptions.plugins.*,*.per_resource.adobe-flash-player", 1);
prefs.put("PluginsAllowedForUrls", "https://your_url.com");
options.setExperimentalOption("prefs", prefs);
WebDriver driver = new ChromeDriver(options);
Here you can find a detailed discussion on Manage Flash in Chrome and on PluginsAllowedForUrls
Update :
You haven't mentioned in your comment through which client you are not able to find setExperimentalOption or set_experimental_option. Here is the snapshot from my IDE which have no errors/warnings :
And here is the JavaDoc :
This is what worked for me in the end
Capybara.register_driver :chrome do |app|
chrome_binary = '/Applications/Google Chrome.app'
prefs = {"profile.default_content_setting_values.plugins" => 1,
"profile.content_settings.plugin_whitelist.adobe-flash-player" => 1,
"profile.content_settings.exceptions.plugins.*,*.per_resource.adobe-flash-player" => 1,
"PluginsAllowedForUrls" => "hendricks-as3.localhost.bbc.co.uk"
}
capabilities = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => { "binary" => chrome_binary + '/Contents/MacOS/Google Chrome', "prefs" => prefs })
Capybara::Selenium::Driver.new(app, :browser => :chrome, :desired_capabilities => capabilities)
end
Reading this documentation helped me realise what to do. (also DebanjanB answer helped me with the specific profile options but as in Java no good to my specific needs, thanks though :-) )
https://sites.google.com/a/chromium.org/chromedriver/capabilities
I'm trying to debug why when running remote webdriver tests on a headless linux host download dialogs are presented in chrome. I believe the chrome version is 45.
Couple of Env Details
Selenium 2.53 (gem)
Selenium 2.53 Server Jar
Chrome Driver 2.21
The framework/Tests are written in Ruby utilizing Capybara for driving web tests. Here is a brief snippet of how the remote driver is initialized.
prefernces = {
:download => {
:prompt_for_download => false,
:default_directory => '/home/john.doe/Downloads/'
}
}
caps = Selenium::WebDriver::Remote::Capabilities.chrome()
caps['chromeOptions'] = {'prefs' => prefernces}
http_client = Selenium::WebDriver::Remote::Http::Default.new
http_client.timeout = 240
options = {
browser: :remote,
url: "http://<server_url>:4444/wd/hub",
desired_capabilities: caps,
http_client: http_client
}
# Returns Remote Driver
Capybara::Selenium::Driver.new(app, options)
I have verified via the hub that the chromeOptions are set, but when a file is downloaded, we're presented with a file dialog prompt.
I've burned the candle looking for a solution to this problem. Thanks for the help and consideration!
try removing the / from the end of the default_directory and also setting directory_upgrade: true. Other than that make sure the browser has permission to write to the selected directory (note also the correct spelling of preferences)
preferences = {
:download => {
:default_directory => '/home/john.doe/Downloads',
:directory_upgrade => true,
:prompt_for_download => false,
}
}
caps = Selenium::WebDriver::Remote::Capabilities.chrome(
'chromeOptions' => {'prefs' => preferences}
)
Here is an example to download a file with Capybara / Selenium / Chrome :
require 'capybara'
require 'selenium-webdriver'
Capybara.register_driver :chrome do |app|
Capybara::Selenium::Driver.new(app,
:url => "http://localhost:4444/wd/hub",
:browser => :chrome,
:desired_capabilities => Selenium::WebDriver::Remote::Capabilities.chrome(
'chromeOptions' => {
'prefs' => {
'download.default_directory' => File.expand_path("C:\\Download"),
'download.directory_upgrade' => true,
'download.prompt_for_download' => false,
'plugins.plugins_disabled' => ["Chrome PDF Viewer"]
}
}
)
)
end
session = Capybara::Session.new(:chrome)
session.visit "https://www.mozilla.org/en-US/foundation/documents"
session.click_link "IRS Form 872-C"
sleep 20
I'm trying crawl along some pages with watir-webdriver and chromium. I haven't got any success by googling arround so here is my question:
I don't need any images so, to speed up things, I try to disable image loading.
Using firefox as my browser is relatively straigtforward
profile = Selenium::WebDriver::Firefox::Profile.new
profile['network.image.imageBehavior'] = 2
browser = Watir::Browser.new :firefox, :profile => profile
But I haven't had any success with chromium. As far as I've learned, you can set preferences and pass commandline options this way (an example):
prefs = {
:download => {
:prompt_for_download => false,
:default_directory => '/tmp'
},
}
args = ['--start-maximized', '--incognito']
browser = Watir::Browser.new :chrome, :prefs => prefs, :args => args,
That works ok. The problem is that, AFAIK, there are no commandline options nor preferences to block images in chromium.
Any idea?
My setup:
LinuxMint 16 (32.0.1700.107-0ubuntu0.13.10.1~20140204.972.1)
ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]
watir-webdriver v0.6.8
chromedriver v2.9.248304
chromium-browser 32.0.1700.107 Built on Ubuntu 13.10
I've managed to disable image loading and many more things through a proxy, being any the choosen browser driver.
I used a local proxy called privoxy in my case.
prefs = {
:profile => {
:managed_default_content_settings => {
:images => 2
}
}
}
b = Watir::Browser.new :chrome, :prefs => prefs