I need to configure the following params:
environment, trend, history, executors, retries,
etc. I need these params for cucumber to work with ruby. I searched in a lot of places and I did not find much. I would appreciate if you can provide these params.
History
Allure stores history information to allure-report/history folder during report generation. So you need to copy such folder from previous launch into your allure-results before you generate the report.
History features are handled out of the box by Allure CI plugins
Executor info
To add an information about your test executor create a file executor.json in your allure-results:
{
"name": "Jenkins",
"type": "jenkins",
"url": "http://example.org",
"buildOrder": 13,
"buildName": "allure-report_deploy#13",
"buildUrl": "http://example.org/build#13",
"reportUrl": "http://example.org/build#13/AllureReport",
"reportName": "Demo allure report"
}
In the report such information will be displayed like this:
Allure CI plugins will add executor information for you
Categories
You can also add custom problem categories in the report. Create file categories.json and place it into your results folder:
[
{
"name": "Ignored tests",
"messageRegex": ".*ignored.*",
"matchedStatuses": [
"skipped"
]
},
{
"name": "Infrastructure problems",
"messageRegex": ".*RuntimeException.*",
"matchedStatuses": [
"broken"
]
},
{
"name": "Outdated tests",
"messageRegex": ".*FileNotFound.*",
"matchedStatuses": [
"broken"
]
},
{
"name": "Regression",
"messageRegex": ".*\\sException:.*",
"matchedStatuses": [
"broken"
]
}
]
Retries
Retries are handled by default. For each test Allure generates historyId (usually it is md5(fullName + parameterValues)). If report contains few results with the same historyId Allure only displays latest, other are marked as retries and available from Retries section on test result page.
For example, if you run your test 3 times (first 2 are broken, latest is passed, Allure will display such test as passed)
Related
When accessing my Spring Actuator /info endpoint I receive the following information:
{
"git": {
"branch": "8743b52063cd84097a65d1633f5c74f5",
"commit": {
"id": "b3j2924",
"time": "05.07.2021 # 10:00:00 UTC"
}
},
"build": {
"encoding": {
"source": "UTF-8"
},
"version": "1.0",
"artifact": "my-artifact",
"name": "my-app",
"time": 0451570333.122000000,
"group": "my.group"
}
}
My project does not maintain a META-INF/build-info.properties file.
I now wanted to write a unit-test for that exact output but get the following error:
java.lang.AssertionError:
Expecting actual:
"{"git":{"branch":"8743b52063cd84097a65d1633f5c74f5","commit":{"id":"b3j2924","time":"05.07.2021 # 10:00:00 UTC"}}}"
to contain:
"build"
The whole build block is missing in the output.
My questions are the following:
What needs to be done to access the build information during a local unit-test run without providing a META-INF/build-info.properties.
From where does Spring Actuator retrieve the actual build information when my project does not has a META-INF/build-info.properties file so it gives me the output from above?
The build-info.properties file is typically generated at build time by Spring Boot's Maven or Gradle plugins.
I am unable to associate test methods to test cases in the Test Explorer (the "Associate to Test Case" option is greyed out), or via MTM, or the VSTS website. I simply cannot find a way to associate tests to the test cases for automated testing.
I am using .NET core app 1.1, ASP.NET using MSTest (though I have tried Xunit as well, to no avail). Even a basic test case that always passes cannot be associated.
In MTM, I have created a test plan that has the Automation Status set to "Plan".
In VSTS, I have a project using a Git repository, in which I have a CI build created that successfully discovers, runs and reports the tests.
I cannot find where to associate these specific tests to test runs. How may I accomplish this?
Refer to these steps to associate test method to test case:
Install MTM 2017 (Run VS 2017 install app (vs_Enterprise.exe)>Modify)
Open VS 2017>Tools>Options>Work Items>General> Select Visual Studio (Compatibility mode) for Open work items in:
Open your test project in VS 2017 and build
Open Team explorer and connect to your team project
Type test case id in Search Work items box> Press enter to open test case
Select Associated Automation tab and click …
Select a test method > OK
Save test case work item
Another way is that you can associate the test method with test case through Update a field REST API.
For example:
PATCH https://[account].visualstudio.com/DefaultCollection/_apis/wit/workitems/[testcaseid]?api-version=1.0
Content-Type: application/json-patch+json
Body:
[
{
"op": "add",
"path": "/fields/Microsoft.VSTS.TCM.AutomatedTestName",
"value": "[namespace.classname.methodname (e.g. UnitTestProject1.UnitTest1.TestMethod2)]"
},
{
"op": "add",
"path": "/fields/Microsoft.VSTS.TCM.AutomatedTestStorage",
"value": "[assembly name(e.g. unittestproject1.dll)"
},
{
"op": "add",
"path": "/fields/Microsoft.VSTS.TCM.AutomatedTestId",
"value": "[guid id]"
},
{
"op": "add",
"path": "/fields/Microsoft.VSTS.TCM.AutomatedTestType",
"value": "Unit Test"
},
{
"op": "add",
"path": "/fields/Microsoft.VSTS.TCM.AutomationStatus",
"value": "Automated"
}
]
The AutomatedTestId is a Guid value, so you can generate a new Guid by using this C# code:
Guid g = Guid.NewGuid();
string s = g.ToString();
Check this extension that I have created https://github.com/JanuszNowak/janono.ado.testcase.associate.cli . It allows associating in automatic manner. So if yours selenium test are using xunit, nunit or mstest you can associate them automatically.
Code sample:
namespace ExampleTestProject
{
[TestClass]
[janono.ado.testcase.associate.Organization("janono-pub")]
public class UnitTest1
{
[TestMethod]
[janono.ado.testcase.associate.TestCase(5)] //<---
public void TestMethod1()
{
//yours test method content
//...
//
}
}
}
I am deploying a package that contains a deploy.ps1 file. As you already know Octopus is running this script on deploying by default, I want to prevent it happening and run a custom script instead.
If you have a requirement like this, then it's better to move the powershell that starts the services to a separate build step and then tag the tentacles you want that script to run on.
In your deployment step for the service, set the start mode to "Manual"
Then have a step that starts the service, and scope that script to the environments / servers that you want to auto start
The code for the step template I use here is
{
"Id": "ActionTemplates-1",
"Name": "Enable and start service",
"Description": null,
"ActionType": "Octopus.Script",
"Version": 8,
"Properties": {
"Octopus.Action.Package.NuGetFeedId": "feeds-builtin",
"Octopus.Action.Script.Syntax": "PowerShell",
"Octopus.Action.Script.ScriptSource": "Inline",
"Octopus.Action.RunOnServer": "false",
"Octopus.Action.Script.ScriptBody": "$serviceName = $OctopusParameters[\"ServiceName\"]\n\nwrite-host \"the service is: \" $serviceName\n\n& \"sc.exe\" config $serviceName start= delayed-auto\n& \"sc.exe\" start $serviceName\n\n"
},
"Parameters": [
{
"Name": "ServiceName",
"Label": "Service Name",
"HelpText": null,
"DefaultValue": null,
"DisplaySettings": {
"Octopus.ControlType": "SingleLineText"
}
}
],
"$Meta": {
"ExportedAt": "2016-10-10T10:21:21.980Z",
"OctopusVersion": "3.3.2",
"Type": "ActionTemplate"
}
}
You may want to modify the step template as it will set the service to "Automatic - Delayed" and then start the service.
Are you able to move the script to a sub folder?
These scripts must be located in the root of your package
http://docs.octopusdeploy.com/display/OD/Custom+scripts
Alternatively - don't include your deploy.ps1 script in the deployment package if it should never be deployed.
I am running into some performance issues trying to run the chutzpah.console.exe with my chutzpah.json file. For this project, I am using TypeScript and pre-compiling the TypeScript files into .js files before running the tests. Each test file that I have generally references one other TypeScript file that it depends on - this is so Visual Studio is happy and so that the Chutzpah extensions work. For example, my drawerBusiness.spec.ts file contains this reference:
/// <reference path="../../../../app/business/documents/drawerBusiness.ts"/>
My chutzpah.json file is as follows:
{
"Compile": {
"Mode": "External",
"Extensions": [ ".ts" ],
"ExtensionsWithNoOutput": [ ".d.ts" ]
},
"Framework": "jasmine",
"TestHarnessReferenceMode": "Normal",
"TestHarnessLocationMode": "SettingsFileAdjacent",
"References": [
{ "Path": "./lib/jquery/jquery.js" },
{ "Path": "./lib/angular/angular.js" },
{ "Path": "./lib/angular-mocks/angular-mocks.js" },
{ "Path": "./lib/jasmine-jquery/jasmine-jquery.js" },
{ "Path": "./lib/underscore/underscore.js" },
{ "Path": "./lib/angular-bootstrap/ui-bootstrap-tpls.js" },
{ "Path": "./tests/dependencies.js" }
],
"Tests": [
{ "Include": "*/tests/specs/*.ts", "ExpandReferenceComments": "true" }
]
}
As you can see from above, I need to load some external libraries. I presume these get loaded once and then are used for each of the tests?
When I was using pure JavaScript, this configuration was blazing fast. It completed quite quickly in Visual Studio and through running the tests from the chutzpah.console.exe. However, since I have switched over to Chutzpah, the tests are much slower:
Tests complete: 263
=== 263 total, 0 failed, took 74.64 seconds ===
These results are even worse on our build machine, and it's timing out. It seems like there's a huge delay between each spec file that is run, and I think that is causing the overall slowness - once the spec file starts running, it completes in a few one-hundredths of a second. Does anyone have any ideas about what could possibly be configured incorrectly?
Here are a few things you can try to help get to the bottom of this issue:
Can you try enableding "EnableTestFileBatching": true in your chutzpah.json file to see if it helps? There can be slowness if you have MANY test files and this setting tells chutzpah to batching them into one harness.
Can you provide the output of the /trace option so I can see all the timings?
Can you provide a repro that I can run locally?
I'm trying to create some tasks in Visual Studio Code to run all the tests in my go project.
I usually execute the tests on the command line using:
go test ./...
In Visual Studio Code my tasks.json looks like this:
{
"version": "0.1.0",
"command": "go",
"tasks": [
{
"taskName": "build",
"isBuildCommand": true
},
{
"taskName": "test",
"isTestCommand": true,
"args": ["./..."]
}
]
}
So Build works fine (CTRL + SHIFT + B)
But when I try to run the tests (CTRL + SHIFT + T) the following error occurs:
go: unknown subcommand "./..."
It seems to be omitting the "test" param, but when I comment out the args it runs go test fine.
Any ideas?
THIS MAY BE A BUG
VSCode Reverse Args and Task as of v0.8.0
This may be a bug that still persists in the newer versions. As of v0.9.1 I have not had a chance to test. Prior to 0.9.1 at least one hack worked by reversing the task and it's arg as in the following example:
{
"version": "0.1.0",
"command": "go",
"tasks": [
{
"taskName": "build",
"isBuildCommand": true
},
{
"taskName": "./...",
"isTestCommand": true,
"args": ["test"]
}
]
}
It's hard to believe that this has still persisted until v0.8.0 so there may be a preferred solution that I have not discovered.
Here is a link to a prior post that deals with a similar issue:
Define multiple tasks in VSCode
Scroll down to my answer for more explanation.