How to create Unit Tests for Xamarin / Bitrise - xamarin

I'm setting up a new Hybrid Mobile Application with Xamarin and I want to use Bitrise for the Continuous Integration. But I don't know how to properly create tests.
Right now, I've setup a Workflow where I build the project, in order to run tests I read that I need to use the NUnit Runner step (which I added) but I don't know where I should put my tests in my projects for this step to execute them.
I tried to add a new NUnit Project with Visual Studio in my solution
My test look like this :
using NUnit.Framework;
namespace Tests
{
public class Tests
{
[SetUp]
public void Setup()
{
}
[Test]
public void Test1()
{
Assert.Pass();
}
}
}
So it should execute withput any problem right ? But when my Workflow is executed I end up with an error like this one: Failed to read test result, error: test result not exist at: /Users/vagrant/deploy/TestResult.xml
I've got all the default settings for the NUnit step in my Workflow.
What am I doing wrong ?

There are multiple unit testing frameworks, NUnit, XUnit & Microsoft Testing Framework.
According to some documents, Bitrise is able to support two of them: NUnit & XUnit.
So when I tried to Add a new Unit Test project to my Xamarin solution, I chose NUnit and in the Toolbar clicked Build->Rebuild UnitTests.
Then I selected View->Debug Pads-> Unit Tests and then Run All.
You ll see that the default test will run & a test results box will appear saying all the tests pass.

Related

Rerun C# failed tests with xUnit framework in TeamCity

I'm wondering if anyone has a way of rerunning failed tests automatically on Team City? I'm using C#, XUNIT and Selenium for a suite of automated UI tests. They can be flakey and a simple rerun will pass the test on the second try. I can't seem to find a solution. I've used other frameworks before that allow you to pass a console param for reruns.
This is something you have to do at the level of your test runner, i.e. xUnit. You can consider xUnitRetry plugin (the link has installation documentation):
[Retry(5)] //will try 5 times
public void TryAFewTimes()
{
tried++;
Assert.True(tried >= 5);
}

Queries on Unit Testing in Xamarin

I'm developing cross platform app for Android, Windows Desktop 8.1, WinPhone 8.1, iOS on Xamarin and now added Test projects. I went through multiple resources for UITests but had following queries. It could be helpful if anyone can correct me.
Projects of type Android test, Windows Phone test, iOS test exists. When these are run, these executes as an app in device/emulator. So, how would these test original app running on same device?
If I have two different android project, how can I specify in above test projects which one to test?
I also added UITest project, which is class library project. In this I can mention apk file using following code:
[SetUp]
public void BeforeEachTest()
{
app = ConfigureApp.Android.ApkFile(#"Path\to\apk\com.namespace.apk")
.DeviceSerial("emulator-1234")
.StartApp();
}
How to reference a project to test rather then .apk file, so I could modify project every time I run tests?
How can I combine UITest project and native test projects? Like common portable library is referenced by native projects.
Thanks in advance
Regarding point 4.
At present, running console UnitTests against Xamarin.Forms, i.e testing the ViewModel, Commands, and Navigation in isolation of the UI takes a fair bit of setup. Testing the ViewModels in total isolation of the Xamarin.Forms framework disallows testing across multiple ViewModels.
For instance, testing a login viewmodel in complete isolation where:
user enters username/password
click login button
redirected to the authenticated screen with their data populated
will not work, as that requires a hook on the viewmodel navigation, so the test can authenticate on the Login ViewModel, then navigate and initialise the Authenticated ViewModel.
So I put out a nuget library that allows to simply achieve this, and also provide Bdd using specflow. The test run as nunit test on vs test shell, or any nunit console runners (such as in Build Servers such as Team City).
Example:
public class GeneralSteps : TestStepBase
{
public GeneralSteps(ScenarioContext scenarioContext)
: base(scenarioContext)
{
// you need to instantiate your steps by passing the scenarioContext to the base
}
[Given(#"I am on the main view")]
public void GivenIAmOnTheMainView()
{
Resolver.Instance.Resolve<INavigationService>().PushAsync<MainViewMode>();
Resolver.Instance.Resolve<INavigationService>().CurrentViewModelType.ShouldEqualType<MainViewModel>();
}
[When(#"I click on the button")]
public void WhenIClickOnTheButton()
{
GetCurrentViewModel<MainViewModel>().GetTextCommand.Execute(null);
}
[Then(#"I can see a Label with text ""(.*)""")]
public void ThenICanSeeALabelWithText(string text)
{
GetCurrentViewModel<MainViewModel>().Text.ShouldEqual(text);
}
}
Step by step instructions here
Nuget package here
Not sure I understand this question.
UITest will deploy and run the app in the device/emulator and will then run the tests against that app. For iOS apps being tested it will start the emulator for you. For Android apps being tested you will have to start the emulator yourself. If they are different Android apps then presumably the UITests would be different for each Android app so keeping the UITests as separate project seems reasonable, especially if you are uploading the tests to Xamarin Test Cloud.
Also note that UITest does not support Windows Phone projects currently, so it can only be used to test your Android and iOS project.
With two different Android projects you would either create a UITest project for each Android project or you would need to specify the path to the .apk file in the test, using ConfigureApp.Android.ApkFile () as you have already done in the code example.
If you are using Xamarin Studio you can run the UITests directly in the IDE without having to specify the full path to the .apk file or the iOS .app.
What you would need to do is have the UITest project reference the Android project. However it needs a special type of reference so you will need to use Xamarin Studio to do this. You add this special type of reference from the Unit Tests window, by finding the Test Apps item, selecting Add App Project and then selecting the Android and/or iOS project.
More information is available in the Adding UITest to a solution on Xamarin's developer guide.
Note that you can have a single UITest project test both the iOS and Android apps. Assuming that they are both the same application. However you may need to create Android and iOS specific UI test queries to find UI elements on the screen.
First you should ask yourself if you really need to combine UITests with native test projects. For example the UITests may take a long time to run, so if you have some fast unit tests you may not want to run them all together.
Combining them is possible if the native test projects are essentially NUnit projects. A UITest project is basically a NUnit project. It has a reference to NUnit and is a class library. If your native tests can be run from a NUnit project then you could combine them. However you would not be able to run the native tests in Xamarin's Test Cloud.
If the native test projects do not use NUnit then you cannot combine them with the UITest project.

Emulating test lists on a build agent using VS2013/TFS2013

How do you replace the 'old' test lists (as in a VS2008 vsmdi file) on a VS2013/TFS2013 build agent? We have VS2013 Professional and Team Foundation Server, and it seems that test lists are no longer supported, and the 'new' test playlists are not supported either for the TFS build agent. MSBUILD no longer seems to support the concept of lists.
Can someone clarify how you should execute selected lists of tests using MSBUILD on a TFS2013 build agent?
Instead of using MSBuild command, you need to choose to use MSTest command. However, by default the TfvcTemplate.12.xaml build template does not support the MSTest.exe option when you select the Test Runner in the build definition. Good news is there is a simple way for you to make it work:
Instead of using TFS2013 build process template, you can choose to use TFS2012 build process template which enables MSTest. You can get it here. Then select to use the TFS2012 build template, click to Automated Tests part, select MSTest Test Metadata File as test runner.
An alternative is to use Test Filters. Tag your unit tests in source control using attributes:
[Category("ListX")]
[TestMethod]
public void Test(){ /* ... */ }
Then you can add a Test Filter to your build definition (sorry for the typo in the screenshot, should read TestCategory=Critical, or ListX in the example above:
See also:
https://jessehouwing.net/xaml-build-staged-execution-of-unit-tests/

Run Automation unit test scripts developed using specflow-selenium from MTM 2013

I want to make sure that I can run my unit test developed using specflow-c# from MTM on my local machine.
Background info:
- Developed the unit test methods using SpecFlow-C#
- Checked in my project to TFS
Problem:
- Now can anyone guide me step by step, how to achieve the integration of this unit methods.
I have gone through various articles of the MSDN, but somehow i get lost in there.
I have learnt that i need to create a build defination (But how to?), setup test controller and test agent (Again how?).
Please guide me in this.
Thanks in advance.
MTM needs three things to execute tests:
A build with which to associate the test run. It can only read from TFS. If you're not using TFS for build/deployment you can run the TFSBuild.exe tool to create a dummy build in TFS to point MTM to.
A test lab in which to run the test. Install both the controller and agent on your local machine. Open the agent configuration tool and register it to the controller. Open the controller configuration tool and register it with TFS Team Project Collection. Once this is done, you should be able to see the controller when setting up the lab environment in MTM.
A test case in TFS which is associated with a unit test. As far as I know, this association must be made in Visual Studio. In Team Explorer, open up the work items. Find your test case and click on the Associated Automation tab. In the Test Name field you select a unit test method to tie to the testcase. This is the test that gets run when the testcase is executed in MTM.

How do I run many unit tests in Build-Deploy-Test build in TFS 2010?

I have been using the Build-Deploy-Test build workflow for TFS 2010 (see here http://msdn.microsoft.com/en-us/vstudio/gg131922.aspx) and have come up against a very annoying limitation.
Here is what I have done:
Setup a build to be queued
Configured it to restore to a test-ready snapshot
Added some deployment steps; in this case starting mongo db
I then came to the step where you configure which tests to run and hit an issue. Firstly you need configure a test plan with some test suites; a test suite consisting of test cases. The problem is that each test case can only be associated with a single unit test.
With having to create a test case per unit test, it means that every time a unit test is added a new test case must be created. Is there any way you can associate many unit tests with a test suite that can be ran in the Build-Deploy-Test build workflow.
One way around this problem is to update your test suite using the tcm.exe testcase /import /syncsuite: command. This command has to be run before test execution begins and a logical place to add it would be into the build template.
I suggest you to use Test List Editor in Test Tab on Visual Studio, create your differents lists of tests, organize your tests based on functionnal after that you can execute on your build list of tests by entering name of list.
link : http://msdn.microsoft.com/en-us/library/ms182463(v=vs.100).aspx
For configuring Build Definition, i suggets you to edit & select Automated Test Tab and insert list.
link : http://support.smartbear.com/articles/testcomplete/testcomplete-and-team-build/

Resources