I would like to get timing information on how long it took a whole *.java class to run AND timing information on each test as well in the gradle output. Is there a way to do that with gradle?
Currently, I just have
beforeTest{ descr ->
logger.warn("Starting Test ${descr.className} : ${descr.name}")
}
It depends on your intent. For debugging purposes, I usually run gradle with --profile flag, which generates the full report of task execution times. See Gradle Command Line.
If you wish to do something ad-hoc with times, you'd need to code the desired behavior. For example, this will print execution time for each test:
test {
afterTest { descriptor, result ->
def totalTime = result.endTime - result.startTime
println "Total time of $descriptor.name was $totalTime"
}
}
See also:
Testing
TestResult
A variation of the accepted answer with rounding millis to seconds:
test {
afterSuite { descriptor, result ->
def duration = java.util.concurrent.TimeUnit.MILLISECONDS
.toSeconds(result.endTime - result.startTime)
println "Total duration of $descriptor: $duration seconds"
}
}
Related
I try to create jenkinsfile for parallel execution command mvn test with different arguments. On the first stage of jenkinsfile I create *.csv file where are will be future arguments for mvn test command. Also I don't know the quantity of parallel stages (it depends on first stage where I get data from DB). So, summarize it again. Logic:
First stage for getting data from DB over command mvn test (with args). On this test I save data into csv file.
In loop of jenkinsfile I read every string, parse it and get args for arallel execution mvn test (with args based on the parsed data).
Now it looks like this (only necessary fragments of jenkinsfile):
def buildProject = { a, b, c ->
node {
stage(a) {
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh "mvn -Dtest=test2 test -Darg1=${b} -Darg2=${c}"
}
}
}
}
stages {
stage('Preparation of file.csv') {
steps {
sh 'mvn -Dtest=test1 test'
}
}
stage('Parallel stage') {
steps {
script {
file = readFile "file.csv"
lines = file.readLines()
def branches = [:]
for(i = 0; i < lines.size(); i++) {
values = lines[i].split(';')
branches["${values[0]}"] = { buildProject(values[0], values[1], values[2]) }
}
parallel branches
}
}
}
}
So, which problems do I face now with?
I see in log following error:
[ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/Data/jenkins/workspace//#2)
I look at workspaces of Jenkins and see that there were created several empty(!!!) directories (quantity equals to quantity of parallel stages). And therefore mvn command could be executed because of absence of pom.xml and other files.
In branches the same data are saved on every iteration of loop and in 'stage(a)' I see the same title (but every iteration of loop has unique 'values[0]').
Can you help me with this issue?
Thank you in advance!
So, regarding this jenkins issue issues.jenkins.io/browse/JENKINS-50307 and workaround which could be found there, task could be closed!
I have created a complex pipeline. In each stage I have called a job. I want to see the console output for each job in a stage in Jenkins. How to get it?
The object returned from a build step can be used to query the log like this:
pipeline {
agent any
stages {
stage('test') {
steps {
echo 'Building anotherJob and getting the log'
script {
def bRun = build 'anotherJob'
echo 'last 100 lines of BuildB'
for(String line : bRun.getRawBuild().getLog(100)){
echo line
}
}
}
}
}
}
The object returned from the build step is a RunWrapper class object. The getRawBuild() call is returning a Run object - there may be other options than reading the log line-by-line from the looks of this class. For this to work you need to either disable the pipeline sandbox or get script approvals for these methods:
method hudson.model.Run getLog int
method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild
If you are doing this for many builds, it would be worth putting some code in a pipeline shared library to do what you need or define a function in the pipeline.
I'm going to put a fat bounty on this as soon as the system permits.
What I'm specifically having trouble with is getting coverage and getting integration tests working. For these I see the uninformative error:
Resource not found: com.something.somethingelse.SomeITCase
Uninformative because there's nothing to relate it to, nor does it mean much to someone who has none of the context.
Here's other weirdness I'm seeing. In cases where there's no integration tests for a subproject, I see this:
JaCoCoItSensor: JaCoCo IT report not found: /dev/build/dilithium/target/jacoco-it.exec
Why would I see target? This isn't a Maven project. A global search shows there's no target directory mentioned anywhere in the code base.
Then, there's this section of the documentation:
sonarqube {
properties {
properties["sonar.sources"] += sourceSets.custom.allSource.srcDirs
properties["sonar.tests"] += sourceSets.integTest.allSource.srcDirs
}
}
Near as I can tell sourceSets.integTest.allSource.srcDirs returns Files, not Strings. Also it should be:
sonarqube {
properties {
property "sonar.tests", "comma,separated,file,paths"
}
Note that you get an error if you have a directory in there that doesn't exist. Of course there's apparently no standard for what directory to put integration tests in and for some sub-projects they may not even exist. The Gradle standard would be to simply ignore non-existent directories. Your code ends up looking like:
sonarqube {
StringBuilder builder = new StringBuilder()
sourceSets.integrationTest.allSource.srcDirs.each { File dir ->
if ( dir.exists() ) {
builder.append(dir.getAbsolutePath())
builder.append(",")
}
}
if (builder.size() > 1) builder.deleteCharAt(builder.size() -1 )
if (builder.size() > 1 )
properties["sonar.tests"] += builder.toString()
properties["sonar.jacoco.reportPath"] +=
"$project.buildDir/jacoco/test.exec,$project.buildDir/jacoco/integrationTest.exec"
}
Sonar is reporting no coverage at all. If I search for the *.exec files, I see what I would expect. That being:
./build/jacoco/test.exec
./build/jacoco/integrationTest.exec
...but weirdly, I also see this:
./build/sonar/com.proj_name_component_management-component_proj-recordstate/jacoco-overall.exec
What is that? Why is it in such a non-standard location?
OK, I've added this code:
properties {
println "Before: " + properties.get("sonar.tests")
println "Before: " + properties.get("sonar.jacoco.reportPath")
property "sonar.tests", builder.toString()
property "sonar.jacoco.reportPath", "$project.buildDir/jacoco/test.exec,$project.buildDir/jacoco/integrationTest.exec"
println "After: " + properties.get("sonar.tests")
println "After: " + properties.get("sonar.jacoco.reportPath")
}
...which results in:
[still running]
I don't want any bounty or any points.
Just a suggestion.
Can you get ANY Jacoco reports at all?
Personally I would separate the 2: namely Jacoco report generation and Sonar.
I would first try to simply generate Jacoco THEN I would look at why Sonar can not get a hold of them.
Can a gradle task depend on the failure of another task?
For example, I have an auxillary task that opens the test report in a browser. I want the report to only appear when the task "test" fails, not when all tests pass as it does now.
task viewTestReport(dependsOn: 'test') << {
def testReport = project.testReportDir.toString() + "/index.html"
"powershell ii \"$testReport\"".execute()
}
You can try to set task's finilizedBy property, like:
task taskX << {
throw new GradleException('This task fails!');
}
task taskY << {
if (taskX.state.failure != null) {
//here is what shoud be executed if taskX fails
println 'taskX was failed!'
}
}
taskX.finalizedBy taskY
You can find the explanation gradle's user guide in chapter 14.11 "Finalizer tasks". Shortly, due to docs:
Finalizer tasks will be executed even if the finalized task fails.
So, you just have to check the state of the finilized task with TaskState and if it was failed, do what you wanted.
Update:
Unfortunately, because configuration is always executed for all tasks, seems not possible to create some custom task to set the flag to show report within script. On execution phase it is not possible too, because the task will not be called if previewsly runned task has failed. But you can do, what you wanted, providing the build script arguments, like:
task viewTestReport << {
if (project.hasProperty("showReport") && test.state.failure != null) {
//here is what shoud be executed on taskX fail
println 'test was failed!'
}
}
test.finalizedBy(viewTestReport)
In that case, you have to provide -PshowReport arguments, while you call any gradle task, if ou want to get the report in test task fail. For example, if you call:
gradle test -PshowReport
then report will be shown if test task fails, but if you call it:
gradle test
no report will be shown in any case.
Is it possible to have Xcode run your unit tests multiple times?
I had an issue in several unit tests that caused intermittent failures. Now that I think I've fixed it, my only option appears to mash ⌘ + U until I'm 95% confident the bug is gone.
I know other unit testing frameworks make it quite easy to run a single test, test case, or test suite multiple times. Do we have this luxury in XCTest yet?
For me it works in swift
override func invokeTest() {
for time in 0...15 {
print("this test is being invoked: \(time) times")
super.invokeTest()
}
}
Try overriding invoke test: https://developer.apple.com/documentation/xctest/xctestcase/1496282-invoketest?language=objc
- (void)invokeTest
{
for (int i=0; i<100; i++) {
[super invokeTest];
}
}
It might help you to use
func testMultiple() {
self.measureBlock() {
...
XCTAssert(errMessage == nil, "no error expected")
}
}
This runs the code inside self.measureBlock() multiple times to measure the average time.
It is work to change the code, but you might want to know the execution time anyways.
This answer might be close enough to what you want and it is easy to do.
One alternative is to do this via the command line. You can run a single test using the -only-testing argument, and avoid building using test-without-building i.e. (new lines added for clarity)
for i in {1..10}; \
do xcodebuild \
test-without-building \
-workspace MyApp.xcworkspace \
-scheme Debug \
-destination 'platform=iOS Simulator,OS=11.2,name=iPhone 8' \
-only-testing:MyApp.Tests/TestFile/myTest;
done
Try using a for loop:
func testMultiple() {
for _ in 0...100 {
...
XCTAssert(errMessage == nil, "no error expected")
}
}
Note this doesn't work within a self.measureBlock(). You'll get an NSInternalConsistencyException: Cannot measure metrics while already measuring metrics
However, you can CALL this within a measureBlock():
func testMultiple() {
for _ in 0...100 {
...
XCTAssert(errMessage == nil, "no error expected")
}
}
func testPerformance() {
self.measureBlock() {
self.testMultiple()
}
}
Xcode 8 runs the measureBlock code 10 times.
I had used the invokeTest() override in the past (Xcode 10) with great success. But now in Xcode 11 its not working (for me at least). What I ended up doing was:
func test99Loop() {
for i in 0..<LOOP_COUNT {
if i > 0 { tearDown(); sleep(1); setUp() }
test3NineUrls()
do { tearDown(); sleep(1) }
setUp()
test6NineCombine()
print("Finished Loop \(i)")
}
}
I obviously use setup/teardown, and this is the proper way to do those multiple times (since the first and last are called by Xcode).
You can now do this in Xcode
Edit the Test Plan. In the Test Navigator, at the top you should see "Test Plan: MyAppName (Default)". Tap on this and select "Edit Test Plan".
In the editor, select Configurations, Configuration 1
Under Test Execution you can set Test Repetition Mode and Maximum Test Repetitions.
Run the tests from the menu Product > Test
Xcode has a built-in way you can do this:
Right-click the test and select Run <test> Repeatedly...
That menu has a few configurations, one of them is Stop After where you can select Failure which will stop the execution if it failed during one of the repetitions.
You can also run a test class or the whole test suite repeatedly.