Is there a stand alone version of the Microsoft Test Runner (The tool, that is started, when I run manual tests from the Microsoft Test Manager)? I want to assign Tests to Testers, but they should not have to install the whole Test Manager.
Yes and no. Testers need a Test Pro license, at least, to run manual tests. In that sense they need to be able to install MTM. However, you can run manual tests from Web Access, which doesn't need MTM to be installed. Be aware that you cannot use data collectors if you are using the Web Access test runner.
Related
I have been doing some hands on labs in Appium for a couple of weeks now, using tutorials on YouTube, Udemy and other sources. I am pretty much comfortable in running those tests using sample APKs in these tutorials.
Now, I would like to also understand details on how Appium tests would be run on an actual Mobile App Dev project using JUnit or TestNG where we do not work on the APK, but rather that the automated tests be triggered during build using IntelliJ and Gradle. Running the automated tests manually does not make sense, because these tutorials do that only rather than the tests being kicked off during build. Any of you'll with live experience with Bitrise - if you can also give your inputs, it would really help since in my project Bitrise would be used as well.
Any inputs on this would be greatly appreciated!
Thanks in advance.
PS - I am newbie tester :)
i use Bitrise, but only use them as a continuous integration (CI), I don't use mobile devices there. I use aws device farm to run automated tests.
When you said
Running the automated tests manually does not make sense
it depends: project size, test suite size, how many new features are released on each new pull request (PR), how many new features are released in each new version, execution time of everything, Costs
The value of having automated tests that fire on every push/commit , PR or Release should be evaluated.
For example, if you use the testing pyramid concept:
Unit Tests (owner is same developer), check each build
Service Tests (owner: backend and automatic QA) check each build
User Interface Tests (QC and automatic QA) verify each new version or Release
Is there a way to automatically create Azure Dev Ops test cases from a test assembly or from a solution with test scripts? With TFS 2015, I know that you can automatically create cases using the tcm.exe command line tool. But we are migrating to Azure Dev Ops, and we need to migrate our test plans and test cases as well. As of now, we are told that we'll need to manually associate the test scripts with test cases one-by-one. Is there a way to replicate tcm.exe's /import functionality with Azure Dev Ops?
I agree with Matt. TCM.exe primary use case was to associate test automation to Test case work items so that they could be executed as part of the automated workflow.
However, with the new Pipelines capabilities, you no longer require the test automation to be associated with Test Case workitems. You can run the test automation directly within Pipelines.
See for more info:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/vstest?view=azure-devops#arguments
https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops
I am using tools like Puppet/Chef/Ansible to set up and config development environments and production servers.
Whenever I update the configuration, I run the tool against my development environment and log in to check manually if things works as expected.
But this is tedious to do, and I can't test everything every time, so is there any way I can automate the testing?
There are Infrastructure Testing Frameworks for this:
ServerSpec / InSpec - ruby-based. Famous, big community, nice looking and best in his class.
BATS - Bash Automated Testing System, which is a bit easier.
TestInfra - Python-based infra testing framework. Still pretty young, very small community. Intro.
Goss - Fast (written in Go), small tool for validating server/infra configuration. Test scenarios are written in yaml.
Automation:
There is interesting Molecule project - some automation for testing Ansible roles, designed by Cisco. Never tried it yet.
Step further would be using TestKitchen which handles automation to spin up Vagrant or Docker or even AWS instance and test Puppet/Chef/Ansible with Rspec/BATS against just spinned up machines.
So what you need - pick up framework, write tests and run your playbooks/recipes & tests against mock VMs.
Ideally is to keep your "infra as code" in vcs and configure ci like TravisCI to run your tests for every PR once you bring new changes in your repository.
You can even follow tdd here: write tests first, make them fail, then write actual implementation in your favorite configuration management tool and see if that change makes tests green/passed.
MOAR Infrastructure Testing & Automation!
If you can let us know, what you want to test. We can help better.
But,
Did you check the dry-run mode? I think, Puppet and Ansible supports it, you can have a cron or some automated script which runs all the puppet/ansible modules against a single(test) node.
More info:
1. http://docs.ansible.com/ansible/playbooks_checkmode.html
2. Check the noop mode in https://docs.puppet.com/puppet/latest/reference/man/agent.html
I'm new to Coded UI testing and TFS. I've written a coded UI test for my web application and looking for different options to automatically run it. Apparently the first option is to run it from visual studio. I have also found an article explaining how to associate a test method with a test case however I don't know whether I can run my test method (s) from TFS or not? How about running them with my build? I know the unit tests can be run with the build however I'm not sure about coded UI tests.
Yes, it is possible for you to run Coded UI test in the TFS build process.
Firstly, you need to configure the Team Build Service Host run as an interactive process. See this link.
Secondly, add the Coded UI test project into TFS Version Control.
Thirdly, create a build definition to build the Coded UI test project, and also specify tests to run.
Please have a check of this blog for the detailed steps: http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx
I've just installed Hudson and it is running beautifully. It builds, runs JUnit-tests and also CheckStyle analysis.
Next step for us would be to create an installation, install it and then run automated tests on the actual installation. I would then like to fail the build if the tests fail or at least publish the results somehow. I think we would set it up so that part runs periodically or manually triggered.
We use InstallAnywhere for installation and IBM Rational Functional Tester for automated tests.
So questions are: anyone created a similar setup? are there any plugins, tutorials or other resource that could help me along. Or do you have any tips or advice in general.
The command line reference for Rational Functional Tester:
http://publib.boulder.ibm.com/infocenter/rfthelp/v8r0m0/index.jsp?topic=/com.ibm.rational.test.ft.doc/topics/RobotJCommandLine.html
Sample command for running a test:
java -classpath "C:\IBM\RFT\FunctionalTester\bin\rational_ft.jar"
com.rational.test.ft.rational_ft -datastore \\My_project\AUser\RobotJProjects -user admin -project
\\My_project\AUser\TestManagerProjects\Test.rsp -build "Build 1" -logfolder "Default" -log
"Al_SimpleClassicsA#1" -rt.log_format "TestManager" -rt.bring_up_logviewer true -playback
basetests.SimpleClassicsA_01
An additional note, you'll want to configure windows properly on your agent machine which will be running the tests. This is not advice specific to Hudson or RFT, but rather all GUI automation tools on Windows. RFT will require an interactive desktop environment for it to be able to click buttons, etc. If you have your Hudson agent running as a Windows service, there will be no desktop. See the following: Silverlight tests not working unless RDP connection open
We have run a fairly complicated distributed build on Hudson, it is a process that basically follows:
Test on Windows.
Test on OSX, run code coverage & push results to server.
Test on OSX Tiger.
Package for OSX Leopard & push build to server.
Package for Windows & push build to server.
Update product website.
We don't use InstallAnywhere or Rational Functional Tester, but have similar sorts of mechanisms in their place. The key we found to making it all sing in Hudson was being able run our various steps from the command line. Maven and appropriate plugins made short work of this task. So my advice would be just that, using whatever build tool you are using (ant, maven, ?) configure them so that you can run your rational functional tester and install anywhere from the command line with a simple goal passed to your build tool (i.e. mvn test or mvn assembly:assembly).
After that, make sure whatever machine Hudson is running on has everything installed (i.e. Rational Functional Tester) and configured, so that you can open up the command line and type in the goal and have your tests correctly execute.
Hooking it up in Hudson from that point on is fairly simple - just pass in the goal when you configure the build.
I believe the best answer is that integrating RFT with Hudson/Jenkins is a useless endeavor.
As this IBM FAQ says, to make RFT work you must:
be logged in the machine;
the screen can't be locked;
if you are remotely connected, you can't minimize the connection screen.
So you can't run Jenkins/Hudson as a service, making it not very useful. You must run it from your logged account. If you are in a corporate computer (very probable if you are using RFT), you probably must use a hack to prevent the screen saver to start. If the screen is locked, your tests will always fails.
It isn't very difficult to configure your tests to run from the command line, you just have to take care of the return codes when the tests fail and succeed.
Jenkins/Hudson would also give you some advantages, like integrating the tests with your version control, probably automatically running the tests when a commit is made. It would also help sending emails when the tests fail.
But you still would have to integrate the RFT logs with some kind of JUnit plugin to have a nice report. You also would have to have script to run the tests using the command line.
I think it is not worth the trouble to use an continuous integration server with RFT. Better just have your tests running every day in Windows Task Scheduler. It is a simpler solution with less failure points.
Or use my final solution: quit RFT and use the free Selenium with a headless web driver.
I have some general advice on this because I have not yet implemented this myself.
I am assuming you want to have Hudson run the RFT scripts automatically for you via a build or Hudson process?
I want to implement something similar in my organisation as well.
I have not yet been able to implement this because of organisational constraints but here is what I have thought out/done so far:
Downloaded Windows process viewer, got the command for running the tests.
Made shell Script out of it, separated out the variables etc
The future plan is to setup a Windows Slave machine which would have all the tools in it that would be required once the Tests are kicked off, for eg. the correct versions of browsers, and environment variables, and other tools that are required.
Hudson would kick off a process which runs the shell scripts created which runs all the RFT Scripts and performs necessary operations on the slave machine.