I have a standard build.gradle file which suppose to run testNg tests:
test{
useTestNG(){
spiListenersToSkip = "some value" - causes "option unknown" exception
...
useDefaultListeners=true
...
}
}
I want to pass a testNg runner command line argument (spiListenersToSkip) from the build file.
The problem is that gradle testNg plugin seem to be able to work only with arguments defined in it's TestNgOptions class. One thing to mention is that the list of options in the class is much shorter than actual list of available testNg CL arguments.
Would really appreciate any thoughts on how to pass command line parameter to testNg runner.
I think the right way is to path during execution
gradlew test -Psuite1
in case you want to choose one and only one of multiple suites
def suite1 = project.hasProperty("suite1")
def suite2 = project.hasProperty("suite2")
test {
useTestNG() {
dependsOn cleanTest
useDefaultListeners = true
if(suite1) {
suites "src/test/resources/simpleSuite.xml"
}
if(suite2) {
suites "src/test/resources/advancedSuite.xml"
}
}
}
Then you can easily choose by passing with -P
gradlew test -Psuite2
or even 2 of three
gradlew test -Psuite2 -Psuite1
I try to create jenkinsfile for parallel execution command mvn test with different arguments. On the first stage of jenkinsfile I create *.csv file where are will be future arguments for mvn test command. Also I don't know the quantity of parallel stages (it depends on first stage where I get data from DB). So, summarize it again. Logic:
First stage for getting data from DB over command mvn test (with args). On this test I save data into csv file.
In loop of jenkinsfile I read every string, parse it and get args for arallel execution mvn test (with args based on the parsed data).
Now it looks like this (only necessary fragments of jenkinsfile):
def buildProject = { a, b, c ->
node {
stage(a) {
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh "mvn -Dtest=test2 test -Darg1=${b} -Darg2=${c}"
}
}
}
}
stages {
stage('Preparation of file.csv') {
steps {
sh 'mvn -Dtest=test1 test'
}
}
stage('Parallel stage') {
steps {
script {
file = readFile "file.csv"
lines = file.readLines()
def branches = [:]
for(i = 0; i < lines.size(); i++) {
values = lines[i].split(';')
branches["${values[0]}"] = { buildProject(values[0], values[1], values[2]) }
}
parallel branches
}
}
}
}
So, which problems do I face now with?
I see in log following error:
[ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/Data/jenkins/workspace//#2)
I look at workspaces of Jenkins and see that there were created several empty(!!!) directories (quantity equals to quantity of parallel stages). And therefore mvn command could be executed because of absence of pom.xml and other files.
In branches the same data are saved on every iteration of loop and in 'stage(a)' I see the same title (but every iteration of loop has unique 'values[0]').
Can you help me with this issue?
Thank you in advance!
So, regarding this jenkins issue issues.jenkins.io/browse/JENKINS-50307 and workaround which could be found there, task could be closed!
I have created a complex pipeline. In each stage I have called a job. I want to see the console output for each job in a stage in Jenkins. How to get it?
The object returned from a build step can be used to query the log like this:
pipeline {
agent any
stages {
stage('test') {
steps {
echo 'Building anotherJob and getting the log'
script {
def bRun = build 'anotherJob'
echo 'last 100 lines of BuildB'
for(String line : bRun.getRawBuild().getLog(100)){
echo line
}
}
}
}
}
}
The object returned from the build step is a RunWrapper class object. The getRawBuild() call is returning a Run object - there may be other options than reading the log line-by-line from the looks of this class. For this to work you need to either disable the pipeline sandbox or get script approvals for these methods:
method hudson.model.Run getLog int
method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild
If you are doing this for many builds, it would be worth putting some code in a pipeline shared library to do what you need or define a function in the pipeline.
I would like to get timing information on how long it took a whole *.java class to run AND timing information on each test as well in the gradle output. Is there a way to do that with gradle?
Currently, I just have
beforeTest{ descr ->
logger.warn("Starting Test ${descr.className} : ${descr.name}")
}
It depends on your intent. For debugging purposes, I usually run gradle with --profile flag, which generates the full report of task execution times. See Gradle Command Line.
If you wish to do something ad-hoc with times, you'd need to code the desired behavior. For example, this will print execution time for each test:
test {
afterTest { descriptor, result ->
def totalTime = result.endTime - result.startTime
println "Total time of $descriptor.name was $totalTime"
}
}
See also:
Testing
TestResult
A variation of the accepted answer with rounding millis to seconds:
test {
afterSuite { descriptor, result ->
def duration = java.util.concurrent.TimeUnit.MILLISECONDS
.toSeconds(result.endTime - result.startTime)
println "Total duration of $descriptor: $duration seconds"
}
}
Is it possible to have Xcode run your unit tests multiple times?
I had an issue in several unit tests that caused intermittent failures. Now that I think I've fixed it, my only option appears to mash ⌘ + U until I'm 95% confident the bug is gone.
I know other unit testing frameworks make it quite easy to run a single test, test case, or test suite multiple times. Do we have this luxury in XCTest yet?
For me it works in swift
override func invokeTest() {
for time in 0...15 {
print("this test is being invoked: \(time) times")
super.invokeTest()
}
}
Try overriding invoke test: https://developer.apple.com/documentation/xctest/xctestcase/1496282-invoketest?language=objc
- (void)invokeTest
{
for (int i=0; i<100; i++) {
[super invokeTest];
}
}
It might help you to use
func testMultiple() {
self.measureBlock() {
...
XCTAssert(errMessage == nil, "no error expected")
}
}
This runs the code inside self.measureBlock() multiple times to measure the average time.
It is work to change the code, but you might want to know the execution time anyways.
This answer might be close enough to what you want and it is easy to do.
One alternative is to do this via the command line. You can run a single test using the -only-testing argument, and avoid building using test-without-building i.e. (new lines added for clarity)
for i in {1..10}; \
do xcodebuild \
test-without-building \
-workspace MyApp.xcworkspace \
-scheme Debug \
-destination 'platform=iOS Simulator,OS=11.2,name=iPhone 8' \
-only-testing:MyApp.Tests/TestFile/myTest;
done
Try using a for loop:
func testMultiple() {
for _ in 0...100 {
...
XCTAssert(errMessage == nil, "no error expected")
}
}
Note this doesn't work within a self.measureBlock(). You'll get an NSInternalConsistencyException: Cannot measure metrics while already measuring metrics
However, you can CALL this within a measureBlock():
func testMultiple() {
for _ in 0...100 {
...
XCTAssert(errMessage == nil, "no error expected")
}
}
func testPerformance() {
self.measureBlock() {
self.testMultiple()
}
}
Xcode 8 runs the measureBlock code 10 times.
I had used the invokeTest() override in the past (Xcode 10) with great success. But now in Xcode 11 its not working (for me at least). What I ended up doing was:
func test99Loop() {
for i in 0..<LOOP_COUNT {
if i > 0 { tearDown(); sleep(1); setUp() }
test3NineUrls()
do { tearDown(); sleep(1) }
setUp()
test6NineCombine()
print("Finished Loop \(i)")
}
}
I obviously use setup/teardown, and this is the proper way to do those multiple times (since the first and last are called by Xcode).
You can now do this in Xcode
Edit the Test Plan. In the Test Navigator, at the top you should see "Test Plan: MyAppName (Default)". Tap on this and select "Edit Test Plan".
In the editor, select Configurations, Configuration 1
Under Test Execution you can set Test Repetition Mode and Maximum Test Repetitions.
Run the tests from the menu Product > Test
Xcode has a built-in way you can do this:
Right-click the test and select Run <test> Repeatedly...
That menu has a few configurations, one of them is Stop After where you can select Failure which will stop the execution if it failed during one of the repetitions.
You can also run a test class or the whole test suite repeatedly.