Basically I'm trying out Selenium webdriver (using FireFox) and right now I am trying to sign up to a Google account.
However, the strange thing is that whenever I run Selenium and let it use the (empty I assume?) Selenium FireFox profile Google seems to detect it and block me (asking for phone vertification).
This is even the case when I load up the selenium profile and manually sign up.
When I sign up manually (and don't use the selenium profile) I can sign up just fine.
Is the Selenium FireFox profile some how special which enables the servers to detect it?
EDIT: I'm trying to startup selenium with my default FF profile (however it keeps starting up in an empty profile) - here's the code:
OpenQA.Selenium.Proxy proxySetting = new OpenQA.Selenium.Proxy();
proxySetting.HttpProxy = proxy;
proxySetting.FtpProxy = proxy;
proxySetting.SslProxy = proxy;
FirefoxProfile profile = new FirefoxProfile("default");
profile.SetProxyPreferences(proxySetting);
profile.SetPreference("browser.privatebrowsing.autostart", true);
_driver = new FirefoxDriver(profile);
EDIT:
I managed to open the default firefox profile but now it doesn't use my proxy settings. How can I use the normal profile and still customize the profile proxies?
This post talks about an HtmlDriver tag being added to the HTML in the FirefoxDriver which would be a dead giveaway
Google is a strong supporter of Open Source, and even Selenium itself, however I don't think Google would particularly condone a Selenium script creating a bunch of spam accounts that probably would never be used, and just take space.
That being said, I believe that it would be possible potentially.
The only way that Google would be able to know you are using Selenium, is based on the Request Headers. It's possible either the User-Agent has something to do with Selenium, or one of the other Headers.
My solution would be to use something like Fiddler to listen to the requests that Firefox is sending, and then edit your Selenium scripts to account for, and change those requests so Google does not know that you are using Selenium.
This most likely goes against their terms of use, so exercise caution, and use this answer for educational purposes only.
Is there a chance, if you were using the complete path to your firefox profile directory? (e.g. C:\Users\???\AppData\Roaming\Mozilla\Firefox\Profiles\your_profile.default)
Related
I know that there are many threads talking about this but i've tried many of the solutions suggested but nothing seems to work. Im gonna be very specific so you guys could please help me!
Im trying to do web scraping to a website using Selenium in Python 3 on Windows 10. This website blocks me after a certain number of requests so what I've red is that if I use Tor as the Selenium web driver I can just ask Tor for a new identity (which means a different IP) every specific number of requests.
The following code lets me do the scraping I want using Tor firefox profile in the Tor Browser folder. The only thing missing with these code is that I've havent been able to request a new identity (new IP).
profiler = webdriver.FirefoxProfile(r"C:\Users\Samir\Desktop\Tor
Browser\Browser\TorBrowser\Data\Browser\profile.default")
profiler.set_preference("network.proxy.type", 1)
profiler.set_preference("network.proxy.socks",'127.0.0.1')
profiler.set_preference("network.proxy.socks_port",9050)
driver = webdriver.Firefox(firefox_profile=profiler)
driver.implicitly_wait(15)
driver.get("The URL I want to scrape")
#Extract whatever information i want from the URL
I tried to use the Stem library to get new identity but this does not seem to work with the Tor firefox profile of the previous code.However this works fine if I just open the browser double clicking on the Tor Browser shortcut icon that is created when I install Tor.
#This is the code in stem that gets new identity using Stem. As I said,
#this does not work with the selenium firefox profile for Tor.
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)
Okay so to wrap up, is there a way to get a new IP with the previous code I showed? Or what can I do to achieve what I want using python 3, selenium and tor on windows 10 plus anyother library or whatever thats necessary.
If you have questions or need more information to help me just let me know.
Thaks a lot!!
I am a QA automation analyst responsible for testing a multi-platform online banking application. For our automation testing we use RubyMine suite with Gherkin/Cucumber, Ruby-scripted steps and Selenium-webdriver, Watir and page-object gems/libraries.
I have a number of scripts that I cannot automate completely without manual intervention, and these include blocking certain network calls through Telerik Fiddler to produce warning pages or error messages, etc. The future of our automation would be to do this through RubyMine instead of Fiddler for network blocking. I know there is a way to do this in Chrome using Inspect Element and the Network menu using enable request blocking. However, I cannot figure out a way to force Chrome through Ruby/Selenium to block a given request. The only way is do manually do it myself, and therefore I can't actually automate these as wanted.
So, my question -- is this a possibility to automate request-blocking with Selenium-webdriver? And, if so, where should I begin to look for help with this?
Thanks.
To block URLs from loading with Selenium with the DevTool API:
def send_cmd(driver, cmd, params={})
bridge = driver.send(:bridge)
resource = "session/#{bridge.session_id}/chromium/send_command_and_get_result"
response = bridge.http.call(:post, resource, {'cmd':cmd, 'params': params})
raise response[:value] if response[:status]
return response[:value]
end
send_cmd(driver, "Network.setBlockedURLs", {'urls': ["*"]})
send_cmd(driver, "Network.enable")
It's not very well documented, but you can also implement request blocking by passing the host-resolver-rules option to chrome and mapping the domain to localhost or an invalid IP. Something like this should work for you:
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--host-resolver-rules=MAP www.google-analytics.com 127.0.0.1')
driver = Selenium::WebDriver.for :chrome, options: options
For those wanting to know, python version is just:
driver.execute_cdp_cmd('Network.setBlockedURLs', {"urls": ["www.baidu.com"]})
driver.execute_cdp_cmd('Network.enable', {})
Try https://github.com/lightbody/browsermob-proxy
I dont know if it can satisfy your requirement as I am no way a network expert, I do use browsermob-proxy extensively to capture network request along with selenium and there is a method to blacklist certain request
https://browsermob-proxy-py.readthedocs.io/en/stable/client.html#browsermobproxy.Client.blacklist
How to disable loading external urls on seleniumlibrary/robotframework
I know that with watir-WebDriver, I can make use of RubyBindings to have the browser load specific profiles or Firefox add-ons when I create a new browser instance. However, can I use Watir to actually use the add-on(s) I open?
The reason I ask is that I am trying to implement a web scraper to navigate to websites and record HTTP interactions. However, since Tamper Data already does the HTTP request/response logging I require, I'd rather use its functionality instead of having to redo it myself.
If this is not possible, I'm wondering if anyone knows a unit tester that will allow me to:
Open a Firefox browser & load Tamper Data
Navigate to specified pages
Click a button on Tamper Data's UI
You can't interact with extensions using bare watir/webdriver as far as i know, need to find a workaround ... Try something like rautomation - https://github.com/jarmo/RAutomation or autoit - http://www.autoitscript.com/site/
This works for me to launch firebug:
Win 7 & XP:
require 'watir-webdriver'
default = Watir::WebDriver::Firefox::Profile.new
If you are admin on your machine it will be the following... otherwise search and provide path:
default.add_extension("/Users/Administrator/AppData/Roaming/Mozilla/Firefox/Profiles/krqve9uc.firebug/extensions/firebug#software.joehewitt.com.xpi")
b = Watir::Browser.new(:firefox, :profile => default)
The problem: toggle javascript support without restarting firefox (nor resorting to different driver) during cucumber test run.
If Firefox's prefutils were exposed to javascript in a web page, that would make it possible. But it is not the case.
So, is there a plugin that does it? Or is there another way to solve the problem? Or is there a good tutorial (that highlights the exposing bit) on how to make such a plugin?
Edit
On a second thought, how would javascript be of any help once it is disabled? Probably the whole idea is a bit screwed.
I assume that your tests run with normal web content privileges. In that case, they aren't going to be able to affect browser settings such as whether JavaScript is enabled (I assume that's what you mean by "toggle JavaScript support").
I'd implement a simple XPCOM component with a method to turn JS support on and off (by setting the appropriate pref). You can expose it as a JavaScript global property so that your tests can access it. See Expose an XPCOM component to javascript in a web page for more details. Package your component in an extension and make sure it is installed in the Firefox instance where your tests are running.
If you want to access the preferences API directly from your content script, you can add the following prefs to Firefox, either in about:config or by adding the following lines to prefs.js in your profile directory:
user_pref("capability.principal.codebase.p1.granted", "UniversalXPConnect UniversalBrowserRead UniversalBrowserWrite UniversalPreferencesRead UniversalPreferencesWrite UniversalFileRead");
user_pref("capability.principal.codebase.p1.id", "http://www.example.com");
user_pref("capability.principal.codebase.p1.subjectName", "");`
user_pref("signed.applets.codebase_principal_support", true);
Replace www.example.com with the domain that you want to grant the privileges to. Also add this line to your JS code before you call the preferences API:
netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect');
A local file (something loaded from file:///) is allowed to request additional privileges. Normally you would get a prompt asking whether you want to allow access - you can "auto-accept" the prompt by adding the following lines to prefs.js in the Firefox profile:
user_pref("capability.principal.codebase.p0.granted", "UniversalXPConnect");
user_pref("capability.principal.codebase.p0.id", "file://");
user_pref("capability.principal.codebase.p0.subjectName", "");
You page can then do:
netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect");
var branch = Components.classes["#mozilla.org/preferences-service;1"]
.getService(Components.interfaces.nsIPrefBranch);
branch.setBoolPref("javascript.enabled", false);
This will definitely work if your page is a local file. Judging by the error message however, you are currently running code from about:blank. It might be that changing capability.principal.codebase.p0.id into about:blank or into moz-safe-about:blank will allow that page to get extended privileges as well but I am not sure.
However, none of this will really help if JavaScript is already disabled and you need to enable it. This can only be solved by writing an extension and adding it to the test profile. JavaScript in Firefox extensions works regardless of this setting.
That means you need Javascript to toggle enabling or disabling Javascript.
function setJavascriptPref(bool) {
prefs = Components.classes["#mozilla.org/preferences-service;1"]
.getService(Components.interfaces.nsIPrefBranch);
prefs.setBoolPref("javascript.enabled", bool);
}
I'm developing an addon using Firefox Add-on SDK. I am using cfx to run my addons when testing.
The app posts some text when highlighted to twitter.
The problem is each time I modify the code I have to log back into twitter. Is there any way to ensure cookies are saved whilst testing/debugging? Similar to what happens when you open and close Firefox normally?
What are your parameters for calling cfx? By default, cxf executes in a new profile.
You need to use -p PROFILEDIR or --profiledir=PROFILEDIR param to use a specific profile, and associated settings.
From notes # https://jetpack.mozillalabs.com/sdk/1.0b5/docs/dev-guide/addon-development/cfx-tool.html