I am trying to create a custom test report using Maven as my build tool, JUnit as my framework, along with selenium test cases. I was using maven's surefire report plugin but I need to include more information in my report. Can anyone direct me to a good tutorial on how to create a custom maven reporting tool?
We have a much better test automation dashboard based on just few api calls, ARES dashboard (built under Testastra and owned by ZenQ) is a much better option to try and it's absolutely free.
ARES, is an acronym for Test Automation Results dashboard. It's a TestAutomation framework/tool agonistic solution, that simplifies the collection of Test automation results and their analysis via live dashboard, daily/weekly trends, frequent failures etc. Website: http://www.testastra.com/#ares
Below repo has some code samples, documentation and usage of ARES test automation dashboard:https://github.com/testastra/ARES
Give it a try.
Related
In our company, we use a framework for test automation created with Jmeter.
All the test cases are loaded by .csv files in Jmeter and then executed: The results are written in a separated .csv file.
After this, I have to import the results file into excel so I can create statistics and reports.
I'm looking for a tool that can import the csv files and create statistics automatically (better if web-based). A tool that integrates with Jira/Zephyr would be great - something that directly integrates JMeter output into Jira/Zephyr and links the results to test cases would be perfect.
Are there any tools that can integrate JMeter with Jira/Zephyr, and if there are, what are their advantages and disadvantages?
Jira has buch of REST APIs you can use to create a task after your performance test and attach what ever you want (i.e jmeter results) to the task.
I'm looking for a tool that can import the csv files and create statistics automatically (better if web-based)
Since JMeter 3.0 it's possible to generate HTML Reporting Dashboard out of .jtl/.csv result file
There is BM.Sense online JMeter (and more) results analysis service.
Are there any tools that can integrate JMeter with Jira/Zephyr, and if there are, what are their advantages and disadvantages?
Jira has web APIs so you should be able to perform any operation you need by sending simple HTTP Requests to the JIRA API endpoints.
There is Runscope Zapier which can integrate with a lot of external services including but not limited to Jira, Bitbucket, Github, Trello, etc.
I have a general question about Jmeter. I am using Jmeter and I want to create Continuous integration solution using Jmeter. The problem is that there are several QA, that each one of them have test plans according his development. the problem is that if using Jmeter all test plan need to run as one test plan, and for each of test plan to create report, and than manually over each report and check for failed scenarios. So all the handling is much harder than I thought, Jmeter is wonderful solution but how to use it with several test plans, and if I want it to run nightly? and in the mornng to get single report with all test plans that created? Can someone please advise if Jmeter can be solution for CI nightly? is it possible to merge all reports to single report?, each test plan is for unique development, and we will have multiple testplans over time
Did you have a look at those plugins:
https://github.com/jmeter-maven-plugin/jmeter-maven-plugin which allows you to fail a build based on presence of errors in load test results which you could mix with https://jmeter-plugins.org/wiki/AutoStop/
https://plugins.jenkins.io/performance
JMeter per se is just a load testing tool, you need a continuous integration server. If you don't have one in your company there are several free and open source solutions you can consider like:
Jenkins
Buildbot
Cruise Control
Check out Jenkins vs. Other Open Source Continuous Integration Servers article to learn more about aforementioned tools, see sample build dashboards and commands to kick off a JMeter test.
For functional automation we use to create a framework which is reusable for automating application. Is there any way to create Performance testing framework in jmeter. So that we can use same framework for Performance testing of different applications.
Please help if any one knows and provide more information regarding it.
You can consider JMeter as a "framework" which already comes with test elements to build requests via different protocols/transports, applying assertions, generating reports, etc.
It is highly unlikely you will be able to re-use existing script for another application as JMeter acts on protocol level therefore there will be different requests for different applications.
There is a mechanism in JMeter allowing to re-use pieces of test plan as modules so you won't have to duplicate your code, check out Test Fragments and Module Controller, however it is more applicable for a single application.
The only "framework-like" approach I can think of is adding your JMeter tests into continuous integration process so you will have a build step which will execute performance tests and publish reports, basically you will be able to re-use the same test setup and reporting routine and the only thing which will change from application to application will be .jmx test script(s). See JMeter Maven Plugin and/or JMeter Ant Task for more details.
You must first ask yourself, how dynamic is my conversation that I am attempting to replicate. If you have a very stable services API where the exposed external interface is static, but the code to handle it on the back end is changing, then you have a good shot at building something which has a long life.
But, if you are like the majority of web sites in the universe then you are dealing with developers who are always changing something, adding a resource, adding of deleting form values (hidden or not), headers, etc.... In this case you should consider that your scripts are perishable, with a limited life, and you will need to rebuild them at some point.
Having noted the limited lifetime of a piece of code to test a piece of code with a limited lifetime, are there some techniques you can use to insulate yourself? Yes. Rule of thumb is the higher up the stack you go to build your test scripts the more insulated you are from changes under the covers ( assuming the layer you build to is stable ). The trade off is with more of the intelligence under the covers of your test interface, the higher the resource cost for any individual virtual user which then dictates more hosts for test execution and more skew from client side code which can distort the view of what is coming from the server. An example, run a selenium script instead of a base jmeter script. A browser is invoked, you have the benefit of all of the local javascript processing to handle the dynamic changes and your script has a longer life.
I am working on a project where the core application needs a harness to send messages which will process the message and feed to a portal web application. I am writing the automated testing for the portal application using cucumber/ruby/capybara.
I have never used Jmeter but through a colleague i just understand the basics of the Jmeter GUI tool. I have been asked me to use Jmeter as harness tool so that I can later extend the functionality to do performance testing as well. How can I do this ? How can i use Jmeter in my automated scripts using cucumber/ruby/selenium or capybara to do my testing ?
I dont want to use GUI as the whole test case should be automated end to end, I have to figure out a way to integrate Jmeter functionality to do test data set up and then run my browser test to check whether the data exists.
i hope my question is clear, ask me if you are not. Hoping to get a solution as i dont think it is a uncommon question.
Note: i have seen the ruby-jmeter gem, i think it does partly what i need but it is actually a DSL to use jmeter in ruby.the problem with that is I have to stick the syntax of gem and have to figure all the options in gem. I have also read a blog that this can be done using Jmeter recording proxy and running the browser tests at the proxy which I am not sure how to do that. here is the link where this is mentioned:
https://groups.google.com/forum/#!topic/cukes/4ZGkf3a234Y
As per my understanding of the problem, please go through the below links
for recording the script using Jmeter:
http://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.pdf
for integrating selenium in Jmeter:
There are actually 2 ways of achieving this:
exporting all your selenium scripts as *.jar and copying it to /junit folder under /lib folder of Jmeter.
Use JUnit Sampler to execute your test cases:
http://intensetesting.wordpress.com/2013/09/25/integrating-jmeter-with-selenium-code/
use WebDeriver plugin for Jmeter and write your selenium test cases in Jmeter:
http://jmeter-plugins.org/wiki/WebDriverTutorial/
Also you can run Jmeter via command line.
We use webpagetest to monitor our websites and see how it is performing. I know we can integrate Yslow with Phantom js and then Jenkins.
Is there any way we can integrate webpagetest.org with CI jenkins.
Thanks
You might try webpagetest-api which is a CLI tool and NodeJs module. You could fetch your test results and check for whatever metric you're after then output pass/no-pass.
Unfortunately CI integration does not come for free, you can either build your own integration with webpagetest-api node module or do some bash script magic with the CLI tool. Keep in mind webpagetest-api test (runTest) command is asynchronous, i.e.: it only schedule a test returning the test id so you can later check the results. There are some ways to integrate both test scheduling and fetching results, check it out my post on last performance calendar.