"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\TCM.exe" run /create "/title:Overnight Build Run" /planid:8 /collection:http://hahaha:8080/tfs/DevelopmentCollection /teamproject:SQL"/querytext:SELECT * FROM TestSuite WHERE Planid='8'" "/settingsname:SoakTest" /owner:test\test /builddir:\\wxxx\Builds\SQL\Live "/testenvironment:Soak" /include
It runs all Tests in Test Plan I only want to Run Tests from Test Suites inside this Test Plan, what should be the query?
Anyone have any idea?
You can run Test Cases which belong to a particular Test Suite using parameter /suiteid calling tcm.exe:
TCM.exe run /create /title:<Title> /planid:<PlanId> /suiteid:<SuiteId> /collection:http://hahaha:8080/tfs/DevelopmentCollection /teamproject:SQL /settingsname:SoakTest /owner:test\test /builddir:\\wxxx\Builds\SQL\Live /testenvironment:Soak /include
Use
tcm.exe suites /list /planid:<your plan id> /collection:<CollectionURL> /teamproject:<Team Project name>
to list all Test Suites belonging to the particular Test Plan.
That's how to find out Test Suite ID No
cd C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE
TCM suites /list /planid:8 /collection:your peoject TFS URL /teamproject:Project Name
Related
I have two unit test projects, one for basic processing tests that only interact with mocks and another that needs to call an external loader before the tests can the called. The loader is used to invoke Spark via its Java runtime.
I can run my tests on the command line with something like:
spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner (more options) dotnet test mytest.dll
This successfully runs the tests and dotnet outputs a line like this:
Passed! - Failed: 0, Passed: 2, Skipped: 0, Total: 2, Duration: 10 ms - mytests.dll (netcoreapp3.1)
but the Visual Studio's Test Explorer doesn't know it ran and passed, so I get no green ticks. If I run a similar command line:
spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner (more options) debug
and run the tests from Visual Studio, everything works great and I get my green ticks. But I have to do it manually.
So the questions are:
how to I configure Visual Studio so that if I run my tests, it will run them via the loader and show the results.
how do I do it just for this one project and not the other unit test project? *.runsettings files seem to be solution wide.
I have a Xamarin project that I regularly push to Appcenter, where it's build. That works fine, but now I added NUnit 3 tests to my solution and while they execute fine locally, they seem not to be executed during Appcenter build.
How do I configure my solution so that my test project is executed on Appcenter? Seems like one has to combine it with Xamarin.UITest, but I don't really understand what steps are necessary for that. Note that my NUnit tests are not UI tests, they are normal unit tests.
Update:
Quote from https://devblogs.microsoft.com/appcenter/faster-android-tests-and-nunit-3/
Now you can update your NUnit package along with UITest to the latest versions and run tests both locally and in App Center,
So I would expect NUnit tests to be run during App Center build without an additional script. Ca anyone shed some light on this? Adding a bounty now.
The easiest way I found to have AppCenter run a project's NUnit tests was by adding a post-build script for each app and install NunitXml.TestLogger Nuget package into your NUnit project, which will output an Xml file of your test results.
To create a post-build script it needs to be in the root directory of the Android/iOS .csproj and named appcenter-post-build.sh. Then your script should look something like this:
#Android post build script
#Make sure the directly to the NUnit csproj is correct
ProjectPath="$APPCENTER_SOURCE_DIRECTORY\YourProject.NUnit\YourProject.NUnit.csproj"
echo "$ProjectPath"
#To generate the xml file it requires nuget NunitXml.TestLogger
dotnet test "$APPCENTER_SOURCE_DIRECTORY" --logger:"nunit;LogFilePath=TestResults.xml"
echo "NUnit tests result:"
pathOfTestResults=$(find $APPCENTER_SOURCE_DIRECTORY -name 'TestResults.xml')
cat $pathOfTestResults
echo
#Looks for a failing test and causes the build to fail if found
grep -q 'result="Failed"' $pathOfTestResults
if [[ $? -eq 0 ]]
then
echo "A test Failed"
exit 1
else
echo "All tests passed"
fi
The last part should cause your AppCenter build to fail if a test does. Also, you may need to try building it twice in AppCenter before it picks up that a post build script has been added to your repo.
My understanding of the blog is that the UITest used to based on nunit 2.x. So, previously if you accidentally updated to nunit3.x your UITests wouldn't work.
They have now updated it so that you can use nunit 3.x to run your UI tests.
I believe #Nick Peppers give the correct approach. Sample post build script here:
https://github.com/microsoft/appcenter/blob/master/sample-build-scripts/xamarin/nunit-test/appcenter-post-build.sh
I recently added xxx.runsettings file to my solution to pass parameters from TFS variables to my solution(url). Now I'm not able to generate .trx result file after running TFS build in Test summary/results page, only code coverage is generated.
[.runsettings code taken from here-https://msdn.microsoft.com/en-us/library/jj635153.aspx
Can anyone here tell me how to edit the runsettings file in order to show .trx result file in my test summary instead of code coverage?
TFS 2015 Update 3,TFS ms build
Can see this window once build is completed.
Run functional test Logs
Publish results log -
Logs
2017-06-21T17:20:49.9138829Z Executing the powershell script: C:\agent\tasks\PublishTestResults\1.0.22\PublishTestResults.ps1
2017-06-21T17:20:50.0628925Z ##[warning]No test result files were found using search pattern 'C:\agent_work\2\s**\TestResults\xyz*.trx'.
Just remove <ResultsDirectory>.\TestResults</ResultsDirectory> section from the runsettings file should fix your issue.
By the way, Publish Test Result task does not work in your scenario since it can only publish the test result files on the build agent while the trx file is usually generated on the test agent with Run Functional Test task.
Since you are running the VStest. Which will not generate the .trx result file.
VStest step is actually using VSTest.Console.exe command, it will use the /logger:TfsPublisher which not generate the .trx file. So if you use the built-in tasks such as Visual Studio Test or Run Functional Tests to run tests, results are automatically published and you do not need a separate publish test results task.
To log results into a Visual Studio Test Results File (TRX) use /Logger:trx. More details please refer this command. To generate a .trx file is not related to runsettings file.
For TFS , the test result is automatically published, you could click test run for more details.
I've created NUnit Runner task, but it failes all the time.
As an executable I'm using nunit3-console.exe
My tests in .net 4.6.1 test project use NUnit3
This is how the job is configured:
nunit job configuration
After plan run I've got error:
Could not find test result reports in the C:\Users\dw\bamboo-home\xml- data\build-dir\DWT-BW-JOB1 directory.
Invalid argument: -xml=BW\14\TestResult.xml
I've found that is because NUnit runner task on Bamboo is designed to work with NUnit2. The "-xml" argument is not supported in NUnit3 console runner.
I've found somebody found a solution, but it does not work for me: http://vijayshinva.github.io/devops/bamboo/nunit/tdd/ci/2016/02/25/configure-atlassian-bamboo-nunit-runner-to-execute-nunit-3-tests.html
In my case I've got another error:
Could not find test result reports in the C:\Users\dw\bamboo-home\xml- data\build-dir\DWT-BW-JOB1 directory.
'nunit3-console.exe' is not recognized as an internal or external command, operable program or batch file.
Please, help resolve this issue. Thank you
I'm trying to figure out which is the command line arguments used by Visual Studio when you run the MsTest tests, I guess it starts with:
MSTest.exe /testmetadata:%SolutionName%.vsmdi /testlist:
But I couldn't figure out how to fill the testlist parameter, because both the test list name and id get the following error:
The test list path 8c43105b-9dc1-4917-a39f-aa66a61bf5b6 cannot be found.
An error occurred while executing the /testlist switch.
I'm trying to figure out which is the command line arguments used by
Visual Studio when you run the MsTest tests
It depends on how do you run your tests from Visual Studio. See the following examples:
You are selecting some tests from the Test View window and run them
MSTest.exe /testcontainer:TestProject.dll /test:TestMethod1 /test:TestMethod2 ...
Your are running all the tests from the Test View window
MSTest.exe /testcontainer:TestProject.dll
You have filtered your tests by a category through Test View window and run this category
MSTest.exe /testcontainer:TestProject.dll /category:CategoryName
You have opened the *.vsmdi file and selected some TestLists to run
MSTest.exe /testmetadata:*.vsmdi /testlist:TestList1 /testlist:TestList2 ...
You are running Load or Ordered tests
MSTest.exe /testcontainer:LoadTest1.loadtest /testcontainer:OrderedTest1.orderedtest
You can combine the above examples (arguments) to create the MSTest command that suits on your case. The only restriction you have is that you cannot use the /testmetada and /testcontainer arguments together.
As for the TestList argument you just need to give as parameter the name of the list. If it is not found then your test list does not exist or it does not belong to the *.vsmdi you have defined on the /testmetadata argument.
I am sure that you have already done it, but you can check the following link:
MSTest.exe Command-Line Options
See following link. Even though this post is about msbuild. It uses exec task for calling mstest.
If you use /testlist you need to give metadata file.
You can use /testcontainer and give dll for your test project. It will run all of your tests.
/testcontainer:[file name] Load a file that contains tests. You can
Specify this option more than once to
load multiple test files.
Examples:
/testcontainer:mytestproject.dll
/testcontainer:loadtest1.loadtest