It seems really bizarre that Visual Studio for Mac (the recently-renamed Xamarin Studio) doesn't support automated unit testing for their mobile device (Android/iOS) applications. Actually, I guess it isn't that bizarre, they REALLY want you to pay for their cloud testing service.
How can I run my own automated unit tests, on my own machine? Here's what I mean by that - I want to create an application that:
Contains C# files containing unit tests that require NO user interface
Can be invoked from my Jenkins or Hudson automated build server
Can run on either a physical iDevice attached to the build server, or the iPad simulator
Runs all unit tests, reports their pass/fail status back to the build server, and then exits, all without any human intervention.
It seems like I could do this on the simulator using
xcrun simctl spawn
and then either
Have my test app writes its output to its storage and figure out where on my Mac's hard disk the simulator storage lives, or
Have my test app send its output to the simulator's pasteboard and grab it using
xcrun simctl pbpaste
But these both require that I find a way to write a C# app that runs on iOS, runs my tests, stores the output where the "real world" can find it, and then shuts down.
And I have to figure out how to copy all of the necessary EXEs and DLLs over to the simulated iPad? Or have Visual Studio package it all into an actual Mac app?
To run unit tests on an actual device you can use NUnit.Xamarin. This is a library that creates a Xamarin.Forms app that you can run on the real device, and this app will look for unit tests and run them:
https://github.com/nunit/nunit.xamarin
You have to run this manually as an app and look at the results. If you want to automate that part then you can create a Xamarin UITest automated test to launch your app, let the tests run then download the results.
https://developer.xamarin.com/guides/testcloud/uitest/
This is something we do using Jenkins as part of our standard mobile app pull requests. Jenkins runs a script that runs the UITest unit tests to launch the unit test app, wait for a result then grab the output and either pass or fail the UI test depending on the result of the screen output.
We just look for 'Test passed' or 'Test Failed', and if the test fails we manually run them to look for what went wrong, but you could expand on the UI test to download the results.
Related
I am trying to prepare my Windows application for deployment and I used the Windows App certification kit tool to run tests on it. I get the error in the image attached, the documentation in the link is useless. I am using Xamarin to build my application, any ideas on how I can fix this?
If you are running the local WACK test, you could try to upload it to the store and take the result of the online WACK test as standard. Because there might be some unexpected errors in the local tests due to different reasons for the local device. The online WACK test will run no matter you run the local test or not. The online WACK test is more accurate and the store only checks the online test result.
We are running Automation scripts for Mobile devices with Android, and IOS.
We are trying to run two feature files for two Android devices(One Plus and Samsung S10).
Initially scripts are ran on both the devices but when the first Feature file execution is stopped then automatically the execution process on Second device also stopped.
Appium version - 1.16.0
We tried changing the order of the devices also like Running Feature file of Samsung s10 first and One Plus later and changed by initiating the execution of feature file of OnePlus device first and Samsung S10 next.
In both the cases the same happened when execution of first initiated is ended the other process is stopped suddenly without any error
We are retrieving the capabilities of the devices from database.
Running these scripts on Mac machine
I suspect that switching devices in the middle of a run is probably not a good idea
So to start I'd suggest creating a project folder for each device you are trying to test. Then run cucumber in one folder to test the first device and then a second folder to test the second.
Once you have that working then you can think about how to run both things at the same time, without having multiple project folders on your machine. There are lots of ways to do this
setup a dedicated CI machine
use a CI service like CircleCI
...
The key idea here is to parallize using multiple Cucumber instances rather than a single cucumber instance.
We have a testing framework made in AutoIt for our Windows apps (older legacy apps that we will continue supporting). These have never been run on a schedule or part of CI (always been run manually). I tried to get some kind of auto run (or even just status reporting) out of the tests, with minimal luck.
I have a VM where the tests can run. I experimented with my own web app, which works okay locally for running and status reporting. But when set up on the server, AutoIt reports it cannot open the application. Same thing happens if I try to run the tests from a .bat file.
My current solution is to have AutoIt call my web app to report status (working okay locally, untested on the server), or to see if I can get AutoIt to report results back to TeamCity. I have the agent installed but when I run the build from TeamCity, AutoIt reports it can't launch the application. I tried this while logged into the VM, logged out, with RDS open; no luck.
Is it possible to run the tests manually from the VM and send results back to TeamCity? When I run them from TeamCity it reads the AutoIt output (which is in the expected format), but I need to let TeamCity know to update the results (so we can use TeamCity rather than my web app to show the results).
I may need to find a way to let TeamCity know a build has been started, which might then let it know to keep an eye on the process' output, but I'm not positive. Any ideas?
I solved this so it could be done more traditionally.
If anyone is confused by what "running the agent from the console" meant, it just means installing the agent without selecting the "as a service" checkbox, and then manually starting the agent by cd'ing to the BuildAgent/bin directory and running the command agent start. I also created a batch file that will do this automatically (but you must run it as admin).
Further, I found AutoIt couldn't do anything if the test doesn't run from the right directory, so I had to devise a solution to this.
The only issue now is that I have to have an old laptop always connected to the virtual machine the tests and app run on (since the AutoIt tests won't work without the VM desktop being interactive).
I am trying to run unit tests for Mac application in xcode 5.0 .
The unit tests run fine if I have the sandbox disabled but as soon as I enabled the sandbox the tests are unable to run, failing with error loading bundle (IDEBundleInjection.c: Error loading bundle).
I've seen a few other reports about this error and looks like the general assumption is that it won't work. The suggested workaround is to create a new configuration in xcode with sandboxing disabled and running the tests with that configuration.
This works, of course, but it seems like a poor workaround since it will be running the tests in a different environment than the real application normally would and potentially missing on a whole category of bugs resulting from different permissions, directories etc.
I can hardly believe that Apple intended developers to disable sandboxing for testing, seems very short sighted. Has anyone figured out a better solution for running unit tests for an application that is intended to run in sandbox mode?
Visual Studio 2010 has the ability to run test scripts using different browser profiles. For a specific scenario with two scripts (script1 and script2) you can configure the Browser Mix to run 50% of your scripts as IE7 users and 50% of your scripts as mobile users.
The problem I am facing is that the mobile and desktop versions of a web site are completely different. So the same script that is recorded for the desktop will not run for the mobile. I cannot see any configuration or way to force script1 to only run as IE7 and script2 to only run as safari for iphone.
This is possible with many other performance tools but I fear given the simplicity of MSVS Load testing it may not be possible. Does anyone know if this is possible, the only way I can think that it might be is if somehow in each script I can override the browser mix setting?
So I found the answer. I was trying to add a desktop browser only test and a mobile only test to the same scenario. The key here is to create a separate scenario within the existing "load test" and have one just for desktop browser tests and one just for mobile tests, as demonstrated in the following picture, where the "Web Test" scenario will be split between IE7 and Firefox3, while the "Mobile" scenario will just run on a smartphone.