Gitlab job is successful despite assertion failure - maven

I am running SoapUI assertions using maven image in gitlab. Even though the assertion fails the build is successful in gitlab. I have tried using mvn integration-tests -ff and as well as -fae but no luck. Also used allow_failure: false. This did not work either. Please advise as to how to fail the gitlab pipeline job if there is a failure in assertions.
Here is my yml file
T001-0011:
extends: .ETE -stage
image: adoptopenjdk/maven-openjdk11
variables:
MAVEN_CLI_OPTS: "--fail-fast"
script:
- 'mvn -f ./TV001/pom.xml $MAVEN_CLI_OPTS integration-test'
allow_failure: false
when: always
Here is the gitlab log
1 error
09:53:48,937 ERROR [SoapUITestCaseRunner] JDBC_Request failed, exporting to [/builds/gitlab/data/test-team-automation-scripts/./SV321/Warnings/target/surefire-reports/TestSuite_1-AC1-JDBC_Request-0-FAILED.txt]
09:53:48,938 INFO [SoapUITestCaseRunner] Finished running SoapUI testcase [AC1], time taken: 904ms, status: FAILED
09:53:48,953 INFO [SoapUITestCaseRunner] Running SoapUI testcase [AC2]
09:53:48,963 INFO [SoapUITestCaseRunner] running step [IDN220001-Request2]
09:53:48,966 DEBUG [HttpClientSupport$SoapUIHttpClient] Stale connection check
09:53:48,968 DEBUG [HttpClientSupport$SoapUIHttpClient] Attempt 1 to execute request
09:53:48,968 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Sending request: GET /apikey/v1/warnings/waning/IDN22000 HTTP/1.1
09:53:48,974 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Receiving response: HTTP/1.1 404 Not Found
09:53:48,975 DEBUG [HttpClientSupport$SoapUIHttpClient] Connection can be kept alive indefinitely
09:53:49,018 INFO [log] HTTP status code: 404
09:53:49,019 INFO [SoapUITestCaseRunner] Assertion [Valid HTTP Status Codes] has status UNKNOWN
09:53:49,019 INFO [SoapUITestCaseRunner] Assertion [Script Assertion] has status VALID
09:53:49,019 INFO [SoapUITestCaseRunner] Finished running SoapUI testcase [AC2], time taken: 8ms, status: FINISHED
09:53:49,021 INFO [SoapUITestCaseRunner] Project [DPD-3396] finished with status [FAILED] in 2591ms
SoapUI 5.3.0 TestCaseRunner Summary
-----------------------------
Time Taken: 2599ms
Total TestSuites: 1
Total TestCases: 2 (1 failed)
Total TestSteps: 3
Total Request Assertions: 5
Total Failed Assertions: 1
Total Exported Results: 3
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:03 min
[INFO] Finished at: 2022-06-15T09:53:54+10:00
[INFO] ------------------------------------------------------------------------
Cleaning up file based variables
00:01
Job succeeded

The mvn tests fail but the exit code that the command itself returns, probably is zero.
It is saying, I managed to run the tests.
But for you this doesn't help since you want to check the result of the tests.
Gitlab in order to fail a job checks the exit code of the commands used. You could force mvn to return an erroneous exit code when the tests fail
You could add the following flag
-Dmaven.test.failure.ignore=false

Related

SonarQube - Failed to get CE Task status - HTTP code 502

I am trying to run sonarqube (hosted remotely) via SonarScanner command through local machine for magento application (for PHP), but getting below error every time. I tried to find solution for this, but didnt find much related to my issue
anyone has any idea about this error?
13:22:38.820 INFO: ------------- Check Quality Gate status
13:22:38.820 INFO: Waiting for the analysis report to be processed (max 600s)
13:22:38.827 DEBUG: GET 200 https://example.com/api/ce/task?id=AYE-URS3o9NiO9ce0vrw | time=7ms
13:22:43.845 DEBUG: GET 200 https://example.com/api/ce/task?id=AYE-URS3o9NiO9ce0vrw | time=11ms
13:22:48.854 DEBUG: GET 200 https://example.com/api/ce/task?id=AYE-URS3o9NiO9ce0vrw | time=9ms
13:22:53.866 DEBUG: GET 200 https://example.com/api/ce/task?id=AYE-URS3o9NiO9ce0vrw | time=12ms
13:22:58.871 DEBUG: GET 502 https://example.com/api/ce/task?id=AYE-URS3o9NiO9ce0vrw | time=5ms
13:22:58.899 DEBUG: eslint-bridge server will shutdown
13:23:04.549 DEBUG: stylelint-bridge server will shutdown
13:23:09.571 INFO: ------------------------------------------------------------------------
13:23:09.571 INFO: EXECUTION FAILURE
13:23:09.571 INFO: ------------------------------------------------------------------------
13:23:09.571 INFO: Total time: 12:09.000s
13:23:09.688 INFO: Final Memory: 14M/50M
13:23:09.688 INFO: ------------------------------------------------------------------------
13:23:09.689 ERROR: Error during SonarScanner execution
Failed to get CE Task status - HTTP code 502: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
</body>
</html>

messageExchange.getTimestamp() maven overwriting value

when using messageExchange.getTimestamp() directely friom soapui .
im getting the right values back. so the response time is around 50ms.
However, if i use maven to trigger the same project. then the time which is log for getTimestamp becomes 500ms.
so question is, is the "messageExchange.getTimestamp()" calcuiting the actual response time or the run time to trigger the request and get response ?
also, if the project contains more than one file. so 3 requests to be sent.
the first one takes 500ms
second one takes 50ms
and third takes 45ms
so the times for the second the third request are right. not the first.
how to fix this ?
logs below showing the time difference. keeping in mind its the same request. if this is run directly from soapui, issues isnt there. issue only comes up when we run this throguh maven.
T E S T S
-------------------------------------------------------
Running com.test.app.AppTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- soapui-maven-plugin:5.2.1:test (soapUI4) # projectNAme ---
SoapUI 5.2.1 Maven2 TestCase Runner
2017-02-27 10:44:57,239 [BuilderThread 0] INFO com.eviware.soapui.tools.SoapUITestCaseRunner - Setting global property [enviroment] to [Enviroment]
2017-02-27 10:44:57,239 [BuilderThread 0] INFO com.eviware.soapui.tools.SoapUITestCaseRunner - Setting global property [projectname] to [projectNAme]
10:44:57,939 INFO [DefaultSoapUICore] initialized soapui-settings from [C:\Users\test\soapui-settings.xml]
10:45:02,071 INFO [PluginManager] 0 plugins loaded in 2 ms
10:45:02,071 INFO [DefaultSoapUICore] All plugins loaded
10:45:03,427 INFO [WsdlProject] Loaded project from [file:/C:/automation/soapui-project.xml]
10:45:03,437 INFO [SoapUITestCaseRunner] Running SoapUI tests in project [#resposnetimeAutomation]
10:45:03,439 INFO [SoapUITestCaseRunner] Running Project [#resposnetimeAutomation], runType = SEQUENTIAL
10:45:05,637 INFO [log] =============START=============
10:45:05,669 INFO [SoapUITestCaseRunner] Running SoapUI testcase [Projecttest]
10:45:05,677 INFO [SoapUITestCaseRunner] running step [setProperties]
10:45:05,743 INFO [SoapUITestCaseRunner] running step [GetRequests]
10:45:05,829 INFO [SoapUITestCaseRunner] running step [assignPropertyValues]
10:45:06,078 INFO [log] Assign prop about to send file name: tc_01_tst.xml
10:45:06,202 INFO [SoapUITestCaseRunner] running step [TriggerRequest]
10:45:06,818 DEBUG [HttpClientSupport$SoapUIHttpClient] Attempt 1 to execute request
10:45:06,818 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Sending request: POST /endpoint/test HTTP/1.0
10:45:06,851 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Receiving response: HTTP/1.1 200 OK
10:45:06,868 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Connection shut down
10:45:06,912 INFO [SoapUITestCaseRunner] running step [verificationStep]
10:45:08,222 INFO [log] response Time in ms: ==================== 445
10:45:08,222 INFO [log] responsegetTimestamp Time in ms: ==================== 1488152706869
10:45:08,222 INFO [log] ==================================================================================
10:45:08,222 INFO [log] HTTP status code: 200
10:45:08,223 INFO [log] ==================================================================================
10:45:09,262 INFO [SoapUITestCaseRunner] running step [triggerrequest]
10:45:09,285 DEBUG [HttpClientSupport$SoapUIHttpClient] Attempt 1 to execute request
10:45:09,285 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Sending request: POST /endpoint/test HTTP/1.0
10:45:09,312 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Receiving response: HTTP/1.1 200 OK
10:45:09,312 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Connection shut down
10:45:09,322 INFO [SoapUITestCaseRunner] running step [verificationStep]
10:45:09,323 INFO [log] response Time in ms: ==================== 31
10:45:09,324 INFO [log] responsegetTimestamp Time in ms: ==================== 1488152709313
10:45:09,324 INFO [log] ==================================================================================
10:45:09,324 INFO [log] HTTP status code: 200
10:45:09,324 INFO [log] ==================================================================================
10:45:09,933 INFO [SoapUITestCaseRunner] running step [assignPropertyValues]
10:45:09,933 INFO [log] Assign prop about to send file name: tc2dass.xml
10:45:09,978 INFO [SoapUITestCaseRunner] running step [triggerrequest]
10:45:09,987 DEBUG [HttpClientSupport$SoapUIHttpClient] Attempt 1 to execute request
10:45:09,987 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Sending request: POST /endpoint/test HTTP/1.0
10:45:10,008 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Receiving response: HTTP/1.1 200 OK
10:45:10,009 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Connection shut down
10:45:10,017 INFO [SoapUITestCaseRunner] running step [verificationStep]
10:45:10,019 INFO [log] response Time in ms: ==================== 25
10:45:10,020 INFO [log] responsegetTimestamp Time in ms: ==================== 1488152710009
10:45:10,020 INFO [log] ==================================================================================
10:45:10,020 INFO [log] HTTP status code: 200
10:45:10,020 INFO [log] ==================================================================================
10:45:10,713 INFO [SoapUITestCaseRunner] Finished running SoapUI testcase [Projecttest], time taken: 4908ms, status: FAILED
10:45:10,723 INFO [SoapUITestCaseRunner] Project [#resposnetimeAutomation] finished with status [FAILED] in 7278ms

SOAPUI maven report not showing full log in the HTML report

I was able to generate a html report using maven with the "maven-surefire-report-plugin" but the HTML report is plain simple and does not have logs for each test.
This is what it looks like:
Is it possible to have all the test steps under these test cases? The messages appears in the output log file, so something like this:
2016-09-30 22:43:50,822 INFO [SoapUITestCaseRunner] Assertion [SOAP Response] has status VALID
2016-09-30 22:43:50,822 INFO [SoapUITestCaseRunner] Assertion [Assert Success Message] has status VALID
2016-09-30 22:43:50,822 INFO [SoapUITestCaseRunner] Assertion [Valid HTTP Status Code 200] has status VALID
2016-09-30 22:43:50,822 INFO [SoapUITestCaseRunner] Finished running SoapUI

Read timed out on /batch/project (SonarQube 4.5)

I am struggling with the following problem.
Log of the executed when sonar-runner in Jenkins.
SonarQube Runner 2.4
Java 1.7.0_67 Oracle Corporation (64-bit)
Windows 7 6.1 amd64
SONAR_RUNNER_OPTS=-Xms256m -Xmx512m
INFO: Runner configuration file: D:\CI_BUILD\sonar-runner-2.4\conf\sonar-runner.properties
INFO: Project configuration file: NONE
INFO: Default locale: "ko_KR", source code encoding: "UTF-8"
INFO: Work directory: D:\CI_BUILD\jenkins_home\jobs\CAFE-CLIENT\workspace\.sonar
INFO: SonarQube Server 4.5
05:17:52.119 INFO - Load global referentials...
05:17:53.382 INFO - Load global referentials done: 1263 ms
05:17:53.414 INFO - User cache: C:\Users\pcms_build\.sonar\cache
05:17:53.445 INFO - Install plugins
05:17:54.225 INFO - Install JDBC driver
05:17:54.303 INFO - Create JDBC datasource for jdbc:jtds:sqlserver://10.240.232.145/sonardb;SelectMethod=Cursor
05:17:56.222 INFO - Initializing Hibernate
05:17:59.217 INFO - Load project referentials...
05:18:19.232 INFO - Load project referentials done: 20015 ms
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
Total time: 34.757s
Final Memory: 21M/291M
INFO: ------------------------------------------------------------------------
ERROR: Error during Sonar runner execution
ERROR: Unable to execute Sonar
ERROR: Caused by: Unable to request: /batch/project?key=CAFE-CLIENT&preview=false
ERROR: Caused by: Read timed out
ERROR:
ERROR: To see the full stack trace of the errors, re-run SonarQube Runner with the -e switch.
ERROR: Re-run SonarQube Runner using the -X switch to enable full debug logging.
Build step 'Invoke Standalone Sonar Analysis' marked build as failure
05:18:19.232 INFO - Load project referentials done: 20015 ms
When "Load project referentials done" value is larger than 20,000 ms,
Always "Read timed out" occurs.
Loading of project referentials is not expected to take more than a few seconds. It is very likely you have a performance issue on your SQ server (or with your DB). Here are some criteria that may increase duration of this web service:
you have installed a lot of language plugins or you have enabled a lot of rules in quality profiles
your project is a multi-module project with a lot of modules
Compare for example with:
http://nemo.sonarqube.org/batch/project?key=org.codehaus.sonar%3Asonar&preview=true
that returns more than 1000 rules and 33 modules. It takes less than 2 seconds to return data.
EDIT: it seems there was some SQL issues depending on your DB vendor. See http://jira.codehaus.org/browse/SONAR-5849 and http://jira.codehaus.org/browse/SONAR-6063. Should be fixed in upcoming 4.5.3

sonar-runner fails every time

While running sonar-runner (with debug flags, and stack trace) I'm getting an error.
My sonar server is bound to mysql which runs on the same machine. It was running ok for some time, but lately it have started ending with following error.
10:25:23.030 DEBUG - 178223 measures to dump
10:25:23.629 DEBUG - Updating semaphore batch-test:test1
10:25:33.739 DEBUG - Updating semaphore batch-test:test1
10:25:43.968 DEBUG - Updating semaphore batch-test:test1
10:25:49.551 INFO - Store results in database
10:25:49.551 DEBUG - Execute org.sonar.batch.index.ComponentDataPersister
10:25:54.151 DEBUG - Updating semaphore batch-test:test1
10:26:02.159 DEBUG - Execute org.sonar.batch.issue.IssuePersister
10:26:02.196 DEBUG - ooo Using Connection [jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8, UserName=sonar#localhost, MySQL Connector Java]
10:26:02.196 DEBUG - ==> Preparing: select * from projects p WHERE p.kee=?
10:26:02.196 DEBUG - ==> Parameters: test:test1:com.test.framework.gba.GbaInfoProvider(String)
10:26:02.198 DEBUG - Release semaphore on project : org.sonar.api.resources.Project#45de530a[id=2644,key=test:test1,qualifier=TRK], with key batch-test:test1
10:26:02.281 DEBUG - To prevent a memory leak, the JDBC Driver [com.mysql.jdbc.Driver] has been forcibly deregistered
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
Total time: 27:30.454s
Final Memory: 27M/872M
Any ideas why it fails?

Resources