I am trying to run unit tests for Mac application in xcode 5.0 .
The unit tests run fine if I have the sandbox disabled but as soon as I enabled the sandbox the tests are unable to run, failing with error loading bundle (IDEBundleInjection.c: Error loading bundle).
I've seen a few other reports about this error and looks like the general assumption is that it won't work. The suggested workaround is to create a new configuration in xcode with sandboxing disabled and running the tests with that configuration.
This works, of course, but it seems like a poor workaround since it will be running the tests in a different environment than the real application normally would and potentially missing on a whole category of bugs resulting from different permissions, directories etc.
I can hardly believe that Apple intended developers to disable sandboxing for testing, seems very short sighted. Has anyone figured out a better solution for running unit tests for an application that is intended to run in sandbox mode?
Related
I have a cross-platform desktop application written in Xamarin.Forms that runs both in Windows and MacOS. I want to do some UI automation on top of that application.
After some research it seems like the most cross-platform friendly option is to use Something like Sikuli. As the default stack on our team is centered on the .NET stack we want to use SikuliSharp or Sikuli4Net to perform the automated UI tests.
However, despite of the fact that we've been able to run Sikuli4Net successfully on Windows, automating several flows so far. We have a dire situation on MacOS. Our team (myself included) doesn't have a lot (or maybe any) knowledge of Java applications.
I've installed the JDK 8, but was unable to run the tests in the same way we did on Windows. The code builds, but it seems like something in the environment is lacking.
With Sikuli4Sharp when starting the APILauncher like this:
launch = new APILauncher(true);
launch.Start();
I get the following error:
With SikuliSharp when trying to run a simple demo application on our software I have this error:
I have tried to set up the SIKULI_HOME environment variable using this answer as a reference, but still the same problem (and I did restart the console and IDE, even my machine).
When I run echo $SIKULI_HOME on the terminal I do get the directory that cointains the .jar files:
So I'm kind of lost about were to go from here. These problems made me unsure about being possible to run Sikuli4Net or SikuliSharp on MacOS environments. Is this the case? If not, what am I doing wrong?
as mentioned in the error message: sikuli-script.jar is missing.
You have to check on what version of Sikuli/SikuliX your SikuliSharp or Sikuli4Net are depending on.
In doubt you have to dive into the sources of those Net packages.
I am trying to prepare my Windows application for deployment and I used the Windows App certification kit tool to run tests on it. I get the error in the image attached, the documentation in the link is useless. I am using Xamarin to build my application, any ideas on how I can fix this?
If you are running the local WACK test, you could try to upload it to the store and take the result of the online WACK test as standard. Because there might be some unexpected errors in the local tests due to different reasons for the local device. The online WACK test will run no matter you run the local test or not. The online WACK test is more accurate and the store only checks the online test result.
It seems really bizarre that Visual Studio for Mac (the recently-renamed Xamarin Studio) doesn't support automated unit testing for their mobile device (Android/iOS) applications. Actually, I guess it isn't that bizarre, they REALLY want you to pay for their cloud testing service.
How can I run my own automated unit tests, on my own machine? Here's what I mean by that - I want to create an application that:
Contains C# files containing unit tests that require NO user interface
Can be invoked from my Jenkins or Hudson automated build server
Can run on either a physical iDevice attached to the build server, or the iPad simulator
Runs all unit tests, reports their pass/fail status back to the build server, and then exits, all without any human intervention.
It seems like I could do this on the simulator using
xcrun simctl spawn
and then either
Have my test app writes its output to its storage and figure out where on my Mac's hard disk the simulator storage lives, or
Have my test app send its output to the simulator's pasteboard and grab it using
xcrun simctl pbpaste
But these both require that I find a way to write a C# app that runs on iOS, runs my tests, stores the output where the "real world" can find it, and then shuts down.
And I have to figure out how to copy all of the necessary EXEs and DLLs over to the simulated iPad? Or have Visual Studio package it all into an actual Mac app?
To run unit tests on an actual device you can use NUnit.Xamarin. This is a library that creates a Xamarin.Forms app that you can run on the real device, and this app will look for unit tests and run them:
https://github.com/nunit/nunit.xamarin
You have to run this manually as an app and look at the results. If you want to automate that part then you can create a Xamarin UITest automated test to launch your app, let the tests run then download the results.
https://developer.xamarin.com/guides/testcloud/uitest/
This is something we do using Jenkins as part of our standard mobile app pull requests. Jenkins runs a script that runs the UITest unit tests to launch the unit test app, wait for a result then grab the output and either pass or fail the UI test depending on the result of the screen output.
We just look for 'Test passed' or 'Test Failed', and if the test fails we manually run them to look for what went wrong, but you could expand on the UI test to download the results.
I have a few automated UI tests implemented with CodedUI. I have a virtual machine where the tests are run by Test Manager (via test controller and test agent). The benefits of this set up is that I can do other work while the tests are run. If I use my desktop to run the tests.
Recently, these tests are failing. The test is not able to step over from a input field to input password field. So, every test fail due to the tests can't log in to the system under test.
When I run these tests on my desktop then the tests are run without any issue.
Since the tests have been failing there were no updates on either system. Only, restarts.
Both system is Windows 7. Both has the same IE10 version. I requested a new virtual machine and the issue is the same. Tests are failing.
I'm in the situation where I can't decide which way of working is the proper. I want my tests running on the virtual env.
Have anybody experienced like this? Do you know any method to identify what could be the problem?
Another part of the situation is that, if I set up the tests to run on Chrome, then they are running, but it seems CodedUI doesn't like working together with other browsers than IE. I got some "method is not implemented" exceptions. I feel this would be another hell.
First of all, Coded UI only works with IE, unless you use Selenium components.
If you can, add the error the test gives after failing.
In any case make sure your test machine has access to the system you're logging into. To run a Coded UI test on a remote machine you also need to have a user logged in.
The root of my problem is really simple, however annoying and this is not the first I'm looking for when I experience issue. The browsers I use for test have different Browser Mode and Document Mode settings.
That one where the tests failed had "Browser Mode: IE 10 Compatibility Mode" and "Document Mode: IE 7 Standards".
The another one where the tests have passed had "Browser Mode: IE 10" and "Document Mode: quirks".
I modified the browser set up values and the tests are running properly.
I work on automated tests for DirectX based applications that run on various automated servers. The servers are setup to run as a service, which means when the tests execute they are executed under session 0.
However, the tests started failing after installing the windows platform update kb2670838. When initializing the application, a DXGI_ERROR_NOT_CURRENTLY_AVAILABLE error occurs after a call to CreateSwapChain.
I have found that some API calls do not work under session 0. (See http://msdn.microsoft.com/en-us/library/windows/desktop/hh404562(v=vs.85).aspx).
But last week this worked. I posted this in various Microsoft forums, but it seems no one else is having this problem. So my question is has anyone else experienced this? And how did you work around it?