I need to write some scripts that access some websites. A script from the command line would get some pages, post some forms, screen-scrape some information, etc.
It cannot really be a library "browser" like libwww-perl, because some steps might require user interactions (CAPTCHAs, Ajax-only forms, any interaction surprises, etc.).
The most practical way I can think of would be remotely opening a tab in Firefox, and injecting JavaScript code into it, something a bit like what Greasemonkey and Selenium do. It doesn't necessarily have to be for Firefox and can be a different browser if that's easier.
So what would be the best way to do that?
Have you considered Selenium Remote Control? I've automated browser interaction using the tool before and it works very well, providing a lot of flexibility
Depending on your exact needs, you might be able to leverage the Selenium IDE which is an easy to use Firefox plugin that allows easy scripting.
You can use XPCOM to extend Firefox in every way imaginable. You could write some kind of interface that connects with another process maybe.
I'm not sure what the "best" way to do it would be, but one possibility would be to use AppleScript for the job. Firefox, however, doesn't have extensive scripting capabilities—if you are willing to use Safari, there is an AppleScript command available to inject JavaScript code into a document (the do JavaScript command—look it up in Safari's scripting dictionary, available from within Script Editor).
Also, in order to run AppleScripts from the command line, use osascript:
osascript path/to/script.scpt
To write srcripts on OS X there are two ways I would recommend, and both of them are in ruby. The first is Watir which is an automated testing framework that will control both firefox and safari on Mac os x.
Another, prehaps better way for screen scraping would be to use hpricot which is a html parser that is really easy to use.
In the background Watir uses JSSh - a TCP/IP JavaScript Shell Server for Firefox to do this is. JSSH allows you you control the browser from a telnet session.
Whichever way you go, if ther eare catchpa's they will stop you though. It's sort of the whole point of them :-)
Related
I'm currently working on writing a suite of test scripts using watir webdriver. Is there something out there that would make script generation easier than looking directly at the HTTP and manually putting the script together? Maybe something captures user interactions with the browser elements and then writes that to a script.
I could just write them manually, but I may as well ask and see if there is a better way.
There are a couple record and playback tools that are available for Selenium (like IDE), and several non-open source solutions as well. Most of the Selenium and Watir development communities actively discourage their usage for writing test suites as they create very brittle tests that are difficult to maintain over time.
Watir does allow you to locate elements based on text or regular expressions, which can make it easier to find many elements without looking at the html. In general, though, you the tester have a better idea of the structure of your website, what id elements are there, and what css elements are unique on a page, or unlikely to change with future site updates, etc.
For testing purpose, I'm looking for a tool that simulates browsing activity. I'm not looking for just HTTP(S) traffic generator, I need to define some browsing scenarios. For example, to browse [x] links deep, or randomly jump from page to page, or to randomly fill and submit forms, maybe even generate some erroneous requests. It's important to support all HTTP verbs (PUT, HEAD, DELETE, ...etc.), and hopefully, but not necessarily, command line. It'd be a very big plus to have randomizable fields (IP address, User-Agent, ...etc.)
If no such tool exist, what are the recommended packages to script such in ruby?
This is called "end to end" web testing (e2e)
You may want to look at selenium, a technology that is able to take control of a browser and automate user browsing scenarios.
Selenium is usually used through some kind of control tool. Since you use ruby, you may want to look at selenium-webdriver
If you want random interactions, I heard of a tool called gremlins
I suggest you look into capybara https://github.com/jnicklas/capybara
You can use capybara with the most common ruby test frameworks, rspec, cucumber, test::unit...
It supports selenium by default but you can also make it headless (not opening a browser window) if you use another driver such as capybara-webkit.
Check you the README, you'll find everything you need.
Is there a way to detect whether someone is conducting web testing using watir-webdriver on your site? I have read somewhere that it is fairly easy to detect watir/selenium, but I never managed to get more details about it.
I have tried UserAgent detection, but that's not something very useful as far as it's easy to change it.
Right, I will make my comments into an answer as requested.
I doubt if it's possible. The idea of Selenium is to automate browsers by simulating actions like real users. You can't possibly detect it from server side, unless Selenium fails to simulate (e.g. click really fast, but if the Selenium code is written deliberately to simulate a real user in a slow fashion, then I'd say it will be difficult to detect).
On the other hand, User Agent approach won't work if someone runs it using common browsers with default UA.
The title says it all. I'm trying to script keyboard clicks in the browser (preferably Chrome). It's simple, from the current page I want to tab through 5 elements, then click enter, then repeat.
Right now I'm using a trial of the mac App "Fake" but it seems overly complicated for what I'm trying to do. I'm new to Macs and having trouble figuring out the simplest way to accomplish this. Any suggestions?
I'd take a look at Sikuli. It has a lot of things in its favor:
Comes with a nice starter IDE (or, you can write scripts in your own editor)
You can write Sikuli scripts in Jython, which is nice if you already know (or want to learn) Python
There's a Java API too, should you want to use pure Java
Sikuli's ability to do fuzzy matching of screenshots can be surprisingly robust
Scripts can be run from the command line once you grow out of the IDE (or want to plug the scripts into a continuous integration system)
I want some personally developed JavaScript code to execute whenever I load a page in Safari. Seems like addblock for Safari does this. Anyone know how to do this?
Safari is not extensible. There's no addon framework for it. But yet there's adblock and verious other addons available for it, although Apple's Webkit and Safari developers discourage users from using them, calling them 'binary hacks'. Seems though some of these addons use InputManager, which isn't documented at all anywhere, at least for not for how people are using it to load scripts in Safari. I guess I'm going to have to backwards engineer to see how addblock does it, but before I do, I thought I'd ask around here. Anyone know?
Input managers are a commonly (ab)used way of injecting arbitrary code into another application's runtime. Once you are there, you have to reverse-engineer enough of the application itself to figure out how to get the behavior you want; usually that involves method swizzling to replace parts of the application you are hacking. It's not documented because there's no API to document, but you can learn about the individual pieces (how to write an input manager in general, how method swizzling in Objective C works, how to use tools like class-dump) and then put it all together.
What you are describing sounds like Greasemonkey though, and there are least one or two hacks already out there to enable Greasemonkey-like behavior in Safari. I'd suggest seeing if one of them meets your needs first.