Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I'm looking to automate some web interactions, namely periodic download of files from a secure website. This basically involves entering my username/password and navigating to the appropriate URL.
I tried simple scripting in Python, followed by more sophisticated scripting, only to discover this particular website is using some obnoxious javascript and flash based mechanism for login, rendering my methods useless.
I then tried HTMLUnit, but that doesn't seem to want to work either. I suspect use of Flash is the issue.
I don't really want to think about it any more, so I'm leaning towards scripting an actual browser to log in and grab the file I need.
Requirements are:
Run on linux server (ie. no X running). If I really need to have X I can make that happen, but I won't be happy.
Be reliable. I want to start this thing and never think about it again.
Be scriptable. Nothing too sophisticated, but I should be able to tell the browser the various steps to take and pages to visit.
Are there any good toolkits for a headless, X-less scriptable browser? Have you tried something like this and if so do you have any words of wisdom?
What about phantomjs?
I did related task with IE embedded browser (although it was gui application with hidden browser component panel). Actually you can take any layout engine and cut output logic. Navigation is should be done via firing script-like events.
You can use Crowbar. It is headless version of firefox (Gecko engine). It turns browser into RESTful server that can accept requests ("fetch url"). So it parse html, represent it as DOM, wait defined delay for all script performed.
It works on linux. I suppose you can easily extend it for your goal using JS and rich XULrunner abilities.
Have you tried Selenium? It will allow you to record a usage scenario, using an extension for Firefox, which can later be played back using a number of different methods.
Edit: I just realized this was a very late response. :)
Have a look at WebKitDriver. The project includes headless implementation of WebKit.
I don't know how to do flash interactions (and am also interested), but for html/javascript you can use Chickenfoot.
And to get a headless + scriptable browser working on Linux you can use the Qt webkit library. Here is an example use.
To accomplish this, I just write Chrome extensions that post to CouchDBs (example and its Futon). Add the Couch to the permissions in the manifest to allow cross-domain XHRs.
(I arrived at this thread in search of a headless alternative to what I've been doing; having found this thread, I'm going to try Crowbar at some point.)
Also, considering the bizarre characteristics of this website, I can't help wondering whether you can exploit some security hole to get around the Flash and Javascript.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
Original Question rephrased:
I know a good amount PHP, Js, CSS, OOP and have recently honed my regex skills by using the vim editor's netrw and elinks plugins to download a series of web pages (about a million lines) that were parsed and made ready for uploading into my website. I work on a linux/ubuntu system, a localhost setup and this particular project is implementing the Concrete5 cms - which is written in PHP.
Seeing the benefits of scraping and parsing information, I would like to have my site dynamically perform this function, though on a much smaller scale; such as, enabling my new user to transfer their personal information from another website into mine - which will typically be under a secure connection (though not always) and password.
Question: What is the best tool (scripting language) to use for this? I do not know either Perl or Ruby but I believe either one of those would be a good choice. I have also heard AWK and SED. I'm sure I can figure out HOW to do it once I begin studying the language. I would really appreciate some experienced input on which language would be the best to begin investing my time into learning it.
Thanks for your help.
I would strongly recommend Ruby and Capybara for web scraping. (See the non-test related examples toward the bottom of the capybara page). Reasons:
Simple, short scraping syntax, cookie support, js support.
Ruby has many other uses, a friendly syntax, and an active job market.
Capybara has multiple supported drivers. You can run a real browser (visibly), a real browser headlessly (invisibly) so javascript sites work. With the same code, you can toggle the driver to run http requests with no js (mechanize) for speed. This helps you overcome many hurdles (like needing to run JS/Ajax), needing to see the interaction, etc. with a change to a single line of code (Capybara.current_driver = :some_driver).
Drivers: Capybara-Webkit, Capybara-Mechanize
Ability to use CS, or Xpath selectors, whatever you're comfortable with.
Active development, and an ecosystem growing rapidly around the underlying technologies.
Perl has two very nice ready-to-use tools for scraping that I know of: Web::Scraper and Scrappy. Both are able to work with CSS3 and XPath selectors for identifying elements; Scrappy builds on Web::Scraper and adds integrated scraping and crawling, with a nice URL-matching system to select the links to follow to gather more information, (while Web::Scraper works with a single document). It moves between pages using the well-established and robust WWW::Mechanize library, which is smart, reliable, and aware of authentication and cookies.
If you want to get into the lower level yourself, there are a lot of good tools to build on, including the aforementioned WWW::Mechanize, HTML::TreeBuilder, HTML::TreeBuilder::XPath, HTML::TableExtractor and more.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I'm looking for a tool for automated testing. I will use scripting for testing, rather than just automating, which means that it can't just send some key-presses. It needs to examine the app's window to see if it was updated correctly. The cheaper the better, with no downside limit on cost. ;-)
I need a real language, with functions, variables, loops, branching, etc., and the apps are developed in Delphi.
I need to be able to focus controls*, wait for windows to appear, search for controls/windows, send keyboard input and check that various windows/controls have changed state (e.g a button is now disabled, a text box contains the correct text, a dialog box opens, etc.).
I used to use AutoIt a few years ago and am now coming back to scripting. I wondered if there has been any new development while I was gone.
Any suggestions?
* I do not want to move the cursor to absolute coordinates in case the app's layout changes. This quote from Wikipedia explains why:
Regression testing becomes a problem with GUIs as well. This is because the GUI may change significantly across versions of the application, even though the underlying application may not. A test designed to follow a certain path through the GUI may not be able to follow that path since a button, menu item, or dialog may have changed location or appearance.
I believe you just described TestComplete by SmartBear Software.
Autoit does everything you listed. I use it for the same porpoises all the time. Also, Delphi is good in the sense that uses standard windows controls which makes it very easy to hook into.
I've had some success with sikuli, which has the advantage of being python (jython) code. It is built on top of OpenCV, and uses a full image processing engine to pattern match parts of the screen. It also contains editors and functions for screenshots, waiting on changes, as well as keystroke and mouse injection.
Something different from my other answer: PyWinAuto
It has full scripting (because it's Python), it uses control names instead of X, Y coordinates, it waits for windows and you can send keypresses. And it's free.
The downsides should be obvious :)
To automate a delphi application main problem i found was that the any testing tools (QTP with delphi addon , Test Complete , Testanywhere, autoit, sikuli) they all dont detect any Tlabels and Tlist and and other components, which for an application build in delphi forms a prominent part of the UI. Even to do func based testing we need to ultimately examine a component in the front end to see whether a func is giving right out. Even to right a script to automate we need to first be able to detect and add the object to the repository.
We have an application in which we have heavy dependency on a lot of third party components. I have tried to check feasibility with almost all the tools available in market.
The top two tools in market were of no use.
QTP with Delphi addon is of no use even in there website they have openly mentioned it wont support many classes build in delphi.
TestComplete its better than QTP where in it detects many components menu components and if i remember correctly grid etc.
Then after few googling i found
TestAnywhere which was certainly better in detecting the thirdparty build components. Only problem i found with this was that it was not detecting Tlabel classes and some components dont remember which.
When i speak of detecting components it may be partial or fully ie some tools were able to detect that there is a component at XY co-ordinates but failed to get the component classes or details (like components property name etc) without which it was difficult in automating and writing a script.
Rest all was more or less combination or subset of these above there.
Still Searching !!!!!!!!!!!
With TestComplete, if you compile your app as an open app - i.e. include their libs, it makes your app super inspectable. Great for testing, probably not best for security. Test complete was the best, it is just TOOOO expensive for my budget.
The scripting in it is great, the understanding of Delphi is great. The pricetag is just not great, especially if you are trying to get traction for GUI testing within an org, like it sounds like you are.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 11 months ago.
Improve this question
I'm working on UI automation.
We are using the following tools.
Bewildr
Snoop
Our WPF application uses a custom framework developed by the company. Many of the buttons are generated dynamically. For example, the controls that have ID guids, get new ID guids every time I run the program. Many controls don't have names.
Are there any other tools which might be worth a look?
Is this commercial or personal - ie do you have a budget? That'll affect whether you might consider the Mercury or HP suites, or just go straight to opensource ;)
http://en.wikipedia.org/wiki/List_of_GUI_testing_tools provides a good list of GUI testing tools. AutoIT is nice and easy to learn and use, especially if you're a coder anyway. Phantom AL and IcuTest are both useful for WPF applications.
If you have a budget, there's not much better than the Mercury/HP toolsets - QTP (QuickTest Pro) and WinRunner - the former uses VBScript while the later uses a custom Test Script Language - very clever for quickly writing tests.
I won't provide links to them all as the Wiki article already has that, but I hope that helps.
As for targeting the names, hypothetically you could work out the order in which they're being loaded and tab through them that way, ignoring names and guids. Alternatively you could send clicks to targetted coordinates on the app if you know where the buttons are going to be.
Mark,
There's nothing you mention that bewildr can't already do. Even if you don't know the name, id or even the type of object, you can always get elements dynamically using the .children method... See this for a brief intro: http://www.natontesting.com/2010/11/27/bewildr-0-1-7/
...and here for code examples:
https://github.com/natritmeyer/bewildr/blob/82cd1e907484583be26bc22024ca6a8f34c0d6a4/features/step_definitions/hierarchy_steps.rb
#Jon Abaca
As my knowledge, It depends on which interfaces you are going to test(mobile/web) and you are going to test those applications with code knowledge staff or not.
with less knowledge of coding, mobile testing, cross browsing and ci/cd. you can go with Katalon-studio, yes it's free
or else better to go with Selenium.
https://github.com/last-hit-aab/last-hit is a UI automation testing tool for chrome developer to test their web site without change test script
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I've got an idea for an idiotically simple application, one that converts HAML and SASS into HTML & CSS files for the user by watching directory changes (like Compass). Almost all the components are already available in the community, I just need to figure out what to use for the front-end.
The catch:
It must be:
a standalone app (i.e. users must NOT be required to install Ruby or HAML),
that looks good,
and is available in several platforms (linux, mac, windows).
So far I know very little about:
RubyScript2Exe: which packages ruby applications for you
Adobe AIR: desktop-style web-development...but is it easy to integrate with Ruby?
Adobe Flex: Is this only for web-based development?
Java /jRudy: (I get scared just thinking about it)
FXRuby: a ruby GUI toolkit which is unfortunately too old-fashioned (read 'ugly') to attract the audience I'm looking to target (designers and HTML developers....no, I'm not planning to charge for it, just want to make an attractive app)
Shoes: Another ruby-based GUI toolkit that may or may not suffice...is there a GUI
builder for this?
Of course, other options are more than welcome.
If you provide an answer, please be kind enough to also leave a link to a good starter tutorial that integrates Ruby and your technology of choice?
I recently had to decide on a Windowed front end for a simple app. I looked into FXRuby, TKRuby, Shoes and WXRuby.
Shoes was the only one that helped me make my app. The rest were (probably) more powerful but the cost in complexity (compared to Shoes) seemed vast. I had never had to sit down and work with a big ugly window API before and didn't want to learn one just to achieve my simple report generator. It wasn't clear how to take code for these API's and reliably generate an executable. Shoes' built in packager works nicely for me.
The one problem I had with Shoes was the trouble getting documentation. I eventually learned that running shoes -m launches a shoes app which acts as a very useful manual. The official tutorial is a worthwhile (and short) read. That's located here.
Shoes served me well and will be my first port of call on any simple utility i choose to make in the future.
have you had a look at titanium desktop? might be what your looking for
Oh, hotness flows from my pores about this question. I believe the future of the internet lies over thisaway Cappuccino. I know it sounds like a plug but I swear, I'm just impressed as hell by 280slides and Atlas. A web framework that's built using Cocoa's interface builder and can be compiled for both Cocoa natively as well as a kickass web page by a simple drop-down box? Hot hot hot. Boiling maybe?
Limelight is another alternative. It's JRuby based and available as a binary install for Windows and OSX, or as gem for any platform. There is a tutorial and screencast linked on the Limelight homepage.
I haven't used it, but thought it was worth a mention (I did download for Windows, but couldn't get it to launch - I suspect my work proxy is causing problems).
My vote would be for Shoes as well.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
I'm writing a GUI application that will let users interact with command-line programs. The programs are crystallography programs, in this case. They take a long time to run.
There's a certain common workflow using the command-line programs. The output from one program is typically processed and then is used by other programs. The user needs to be able to fill in various text boxes and select options that are sent to the command-line programs.
As I'm lazy and don't want to do more work than I need to, what tools are out there that will help me in doing this? The software needs to work initially on Linux, but also running on Windows at some point would be neat.
Would also be neat if there was some sort of DSL for non-programmers to be able to extend/modify the GUI application (to add new programs and change the options and so on).
Take a look at GTK Server it is easily scriptable from any language and since it runs as a service (bit like a browser) it can be used across systems. That is, the system running the command line stuff can be on a powerful system and the GTK Server running on the user's desktop PC. The examples are easy to follow and the matrix of versions and languages is bound to include yours. If you are unlucky, just pick the closest combination for examples.
I'd suggest having a look at the Galaxy web portal. It is basically a web based GUI to command line bioinformatics tools (but works equally well for any command line application), with workflow capabilities built in, and thus should have exactly what you're looking for.
It should be fairly easy to add your own wrapper, by following Galaxy's tool configuration format. See this page for how to do this.
TCL/TK was the traditional way.
Today I would use python with either wxPython (more complete) or FLTK(simpler).
If you don't need to readback results from the commandline program it's much easier -
eg os.system("command options")
You can use WShell and redirect stdin stdout.
object console = WShell.Exec(cmdLine);
if (console.StdOut.AtEndOfStream)
s = Pipe.StdOut.ReadLine();
Downside that you still have to figure out how to hode console window. But it is easy one. You need to find this window by window title and set its properties to ivisible. Or use freeware some utility from internet which can do it automatically for you.