We are using TFS/VS 2010 to run Selenium tests which are scheduled in the TFS controller. After the build and tests are finished I would like to run the failed tests from that build.
Currently I am doing this by using a Windows Scheduled Task and executing a batch file which calls a powershell script which gets the latest build version (and failed tests) and then executes them (using mstest) and finally publishes the results back to build.
I just want this to happen without a windows scheduled task, it is too fickle. I believe I need to edit ProcessTemplate.xaml and add an event (InvokeProcess) to achieve this, I just can't find much on it.
Thanks in advance!
Related
I have a gitlab runner installed on one of my test servers.
I want to build and deploy my app on every commit.
The server is a windows server and the app is a .net core 1.1 app.
my build script works fine but eventually it runs dotnet MyApp.dll which obviously makes the pipeline stop wait until it finishes (but, of course, my app won't finish, I want it to run..)
I tried running start dotnet MyApp.dll but that still doesn't work as gitlab's runner won't stop running until all of it's child processes exit.
I am certain I'm using gitlab's CI in a non idiomatic way but fail to understand how to deploy locally correctly.
Any suggestions?
Windows doesn't offer any easy way to disown a process and you probably don't want to task yourself with stopping the process on your next deploy. What you should do is use SRVANY.EXE to create service out of your application and then use the Gitlab CI to stop the service, replace the files and run it again. It's been a while since I used Windows so I'm sorry but I can't provide the exact commands to run.
We are running selenium tests using TestNG in TeamCity.
Is there a way to know the progress of the current run beyond the progress bar and build log?
Information of interest is:
Which tests were run and passed
Which tests were run and failed
...
Thx
As I know testng plugin for teamcity does not have such option. You can try real-time reporter e.g. extentreports. It could be easily connected via testng listener and pass results to report server.
Here is a link to another question which I answered recently which addresses something similar to what you're looking for with ExtentReports.
ITestListener - ExtentReport
TeamCity can report the tests in a build and you should really get that in order to use TeamCity at full power. It can list the test failures in a build with details, display test history, you can assign investigations of a test to your team members or mute the test failures...
If you use TeamCity's Ant or Maven build steps, TestNG tasks should be recognized automatically and build status should turn into something like "Tests failed: 1 (1 new), passed: 301".
Otherwise try to generate XML report in one of supported format and use XML Report build feature in TeamCity.
Yet another alternative is to report the tests in a completely custom way through TeamCity service messages.
All these approaches should update TeamCity build status and details as soon as the test is reported by the build tool. For service messages this is as soon as a test finish is reported, for Ant this is after each test run, for Maven this is after finishing tests of a module and for the XML test report build feature, this is while the XML is being saved on disk, even in incomplete form.
If you are rather looking for a specific test progress, that can be seen in the build log or you can notify TeamCity about stages in the build from within the test/script.
I am using Selenium for automation with Eclipse and TestNG.
Whenever required I execute script from testng.xml.
Why do I need a Build tool loke Maven or Ant because Compiling and execution work taken care by Eclipse.
Only possible reason that I can think of are:
1. Scheduling using Windows 'Task Schedular'
2. Ant allows creating a batch file for execution, so execution gets simplified.
2. Craeting ReportNG reports that are triggered through ANT
Is there any other good reason?
Also why do I need Jenkins?
Jenkins can be really helpful when you want Continuous Integration
builds to be Tested frequently.
In the environment where continuous development and Continuous
Integration goes hand in hand we can use Jenkins to test the build
regularly.
With Selenium Automation we can build/develop the automation script
for the developed builds. Once the Automation is developed we can
create a bat file and use Jenkins to run the bat file.
With Jenkins we can also set the time duration when to execute the
test builds like every day at 12:00 pm IST. Also we can get easily
notified if any test build execution fails because Jenkins also
provide features to send email notification on failure.
Also in some odd cases where we need to test the AUT on Linux/Unix
environment we can easily deploy the Automation on the such Test
system and use ANT to run the Selenium Automation from command
prompt.
We've recently found that an acceptance test project fails occasionally on our build server- due to our web drivers Type is not resolved error. I'm trying an experiment to see if its a question of timing of build steps. To this end I've tried to create a separate build step which launches the webdriver executable separately and then proceed onto the unit tests - the issue I have is when I launch the process it blocks the next step after it has successfully started.
eg. Type is not resolved for member 'OpenQA.Selenium.WebDriverException,WebDriver, Version=2.41.0.0, Culture=neutral, PublicKeyToken=null'..
Is there a way I can progress to the next build step after I call an exe?
Thanks
Do you have multiple build agents? Have you considered splitting this into separate build configurations and having triggers on previous build configurations?
You could have configurations:
Compile,
Webdriver, triggered on successful compile completion
Unit Tests, triggered on successful compile completion.
It's not nice, but it should give you the non-blocking behavior you're after. I can't see any other way of making it happen, build configurations are supposed to run as a linear sequence.
We are currently running a Jenkins (Hudson) CI server to build and package our .net web projects and database projects. Everything is working great but I want to start writing unit tests and then only passing the build if the unit tests pass. We are using the built in msbuild task to build the web project. With the following arguments ...
MsBuild Version .NET 4.0
MsBuild Build File ./WebProjectFolder/WebProject.csproj
Command Line Arguments ./target:Rebuild /p:Configuration=Release;DeployOnBuild=True;PackageLocation=".\obj\Release\WebProject.zip";PackageAsSingleFile=True
We need to run automated tests over our code that run automatically when we build on our machines (post build event possibly) but also run when Jenkins does a build for that project.
If you run it like this it doesn't build the unit tests project because the web project doesn't reference the test project. The test project would reference the web project but I'm pretty sure that would be butchering our automated builds as they exist primarily to build and package our deployments. Running these tests should be a step in that automated build and package process.
Options ...
Create two Jenkins jobs. one to run the tests ... if the tests pass another build is triggered which builds and packages the web project. Put the post build event on the test project.
Build the solution instead of the project (make sure the solution contains the required tests) and put post build events on any test projects that would run the nunit console to run the tests. Then use the command line to copy all the required files from each of the bin and content directories into a package.
Just build the test project in jenkins instead of the web project in jenkins. The test project would reference the web project (depending on what you're testing) and build it.
Problems ...
There's two jobs and not one. Two things to debug not one. One to see if the tests passed and one to build and compile the web project. The tests could pass but the build could fail if its something that isn't used by what you're testing ...
This requires us to know exactly what goes into the build. Right now msbuild does it all for us. If you have multiple teams working on a project everytime an extra folder is created you have to worry about the possibly brittle command line statements.
This seems like a corruption of our main purpose here. The tests should be a step in this process not the overriding most important thing in this process. I'm also not 100% sure that a triggered build is the same as a normal build does it do all the same things as a normal build. Move all the correct files in the same way move them all into the same directories etc.
Initial problem.
We want to run our tests whenever our main project is built. But adding a post build event to the web project that runs against the test project doesn't work because the web project doesn't reference the test project and won't trigger a build of this project. I could go on ... but that's enough ...
We've spent about a week trying to make this work nicely but haven't succeeded. Feel free to edit this if you feel you can get a better response ...
In Jenkins/Hudson, it's quite ok to have many jobs. some for doing compilation triggered version control changes, some for running (unit) tests triggered by successful builds, some for doing more tests (integeration) trigered by successful earlier tests, some for deploying, triggered by successfully passing all tests.
Look at plugins like join, build pipeline, parametrized trigger and more to help out with this.
This will also allow things to happen in parallel, by using multiple nodes. Trying to cram everything in one job is not the way to go.