I have reasonable good experiences with Xcode9 server, developing for the Mac. It seems to work, even my unit-test suite is running. However, I struggle with UI tests. My test run out of the IDE, but not in the server, giving me the message:
No destinations were specified with the -destination flag which were
valid for the specified scheme 'UI Tests'.
I tried to add -destination 'platform=OS X,arch=x86_64' to the "arguments passed to xcodebuild" section of the bot, but that seems to be ignored.
I believe that this is somehow related to the fact that UI tests have a defined target. But who knows how to fix it?
Related
I'm running tests in Visual Studio using "Test Explorer" with NUnit and a .runsettings file (specified by choosing the option in the GUI "Select Settings File")
My settings file (called mytests.runsettings) is:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<RunConfiguration>
<DisableAppDomain>True</DisableAppDomain>
</RunConfiguration>
<ForceListContent>true</ForceListContent>
<NUnit>
<DomainUsage>None</DomainUsage>
</NUnit>
</RunSettings>
I have verified that it is loading this file (verified by adding the framework node and setting it to a fake version, which then results in an error).
But no matter what I do, it doesn't run without an AppDomain!
Running from the command line does work:
nunit3-console.exe --domain=None --inprocess MyTests.dll
What do i need to do to get it to use that setting in NUnit?
I don't believe you can do that.
There is no recognized DomainUsage element under NUnit in the runsettings file. DomainUsage is an internally used property, which will be honored if set. But you can't set it that way. Your DomainUsage element is simply being ignored.
If the adapter received your DisableAppDomain setting, then it would set DomainUsage to None. However, I don't believe it is actually receiving it.
Point 2 requires some explanation. Note that I haven't worked on the adapter for a few years and I'm going from memory but here it is...
The DisableAppDomain setting was added to allow Visual Studio to force NUnit to try to run without using an AppDomain. The Test Explorer is supposed to set things up so that it's possible to run that way, i.e. by making sure everything is already available in the current domain.
In order to prevent misuse of the feature, I believe that Test Explorer always overrides any user-provided setting. Again, this is from memory of work that was done a few years ago, but it seems as if the results you are seeing validate it.
The rationale for this past decision was that Test Explorer is completely responsible for setting up the Process and AppDomain used to run the tests. The user has no way to impact that and neither does NUnit. Of course, when using the console runner, that's not the case - control is in the hands of the user.
Something else to investigate is why you feel the need to run without a test AppDomain being created. But that's probably another question. :-)
I'll ask some other folks who may have a better memory than I to look at this as well.
UPDATE:
#Terje, who maintains the adapter now, replied and confirmed that there is no way to set DomainUsage in the runsettings file or any other way we know of when running under the test adapter. The docs have been corrected to avoid the implicit suggestion that it's possible.
We believe, but have not confirmed experimentally, that TestExplorer creates it's own AppDomain whenever it uses this setting to suppress its creation by the test adapter.
This answer is a follow up to the answer from #charlie above, just need some more space here.
I've checked on the domains that are being created, based on whether the DisableAppDomain is set or not.
When the disable app domain is not set, the appdomain is created by NUnit with applicationbase and friendly name as shown below:
The NUnitCheckDomain is the test dll.
When the disable app domain is set, NUnit no longer sets its own domain, and you then see it runs under the testhost appdomain, which is the process that runs all tests:
So this seems to work the way it should and can.
Do you need it to run under some other app domain than one of these?
BTW: If I run the NUnitCheckDomain test assembly using the NUnit3 console, then it a) works when run directly b) crashes when used with the parameters you give above (domain=None and --inprocess) with not able to load NUnit.Framework. #charlie - Any reason this should not work the same using NUnit3-Console ?
We do have troubles getting our CI to work with Xcode. We encounter the issue Could not determine bundle identifier for xxxTest TEST_HOST: some path that does not exist.
What we have done so far:
On the CI Machine we are able to run all tests on the device, but if we trigger it with a commit the CI does end in that error.
We had that issue on other machines too, when we tried to run the tests of the application. We fixed that by cleaning the machine and reboot the devices.(Had do fix some scheme settings) Not we can run all tests but we run into the issue sometimes without any reason.
We also double checked the schemes. The Host Application is set in every Tests scheme to the App itself.
Our project has the following targets:
App
App Unit Tests
App UI Tests
AppeareanceKit (Module)
AppeareanceKit Unit Tests
Models (Module)
Models Unit Test
Found solution:
Diabling the Tests except the App UITest
Run the tests once
Enable all Tests
Run the tests
Question:
What is happening there? How can we prevent from having that issue over and over again?
Known questions:
Xcode 8 Beta - Convert to Current Swift Syntax Failed: Could not find test host
Tests stop working under xcode 8 TEST_HOST error
So after ages of having this issue, we could find the right checkbox to fix this issue.
It somehow sounds weird but the issue was the configurations for the command-line builds inside of the Project settings.
I assume this happens because we always test against the debug configuration and it was set to release. Therefore it did the wrong build for the test host.
This does set the defaultConfigurationName inside of the {projectname}.xcodeproject.
Please comment if you have any more insights of this.
In my app I found a solution.
The problem was of course in the test target.
I hope that this solution will work for you too. Go to your test target build settings.
Search for test host.
In my case this testHost was looking like
build/{buildConfigurationName}-iphoneos/{schemeName}.app/{appName}
Which is wrong as the actual path must be
build/{buildConfigurationName}-iphoneos/{schemeName}.app/{schemeName}
so if you have a build configuration called Alpha, in your app named MyApp, with a scheme called MyApp-Alpha the test host path should be
build/Alpha-iphoneos/MyApp-Alpha.app/MyApp-Alpha
it was
build/Alpha-iphoneos/MyApp-Alpha.app/MyApp
I hope this can help... unfortunately this issue appears in many cases.
One reason to have this error is that your info.plist of your target is not in its path. So if you moved it or removed the reference to it, you may get this error.
So if you moved it, put it back into its place or path. if you removed the reference to it, add it again, close and reopen the project.
The test explorer (MS-Test) kind of stopped working. It won’t discover any of my tests or sometimes only some of it. When I right click on my test and run it, the output window says “Run test finished: 1 run (time)”, but the test explorer would only display “Last Test Run Failed” (see screenshot), without information about which test or assert failed. I’ve tried restarting the VS, which I thought solved the issue once, but not currently. I’ve also tried build, rebuild, but I just can’t get it working again? Any idea what might help? All of this was working fine until early morning today...
Test Explorer has problems understanding test projects if they are build for x64 or x86, and solution is targeting a different platform.
Also, under Test > Test Settings > Default Test Architecture select appropriate value and then clean and rebuild.
I am quite curious to know if there is a way to achieve the same that you get, when in Xcode you press cmd-u (or select Test from the menu).
I am writing some UI tests for an application, and would love to automate the whole thing.
So far, any attempt to automate via applescript has been quite unsuccessful (I told myself how hard it is to simulate 2 button pressed...); so I thought that maybe there is a proper way to just run the command line commands and have the simulator to pop and run the tests, like it does in Xcode.
Altho if this function was known, I would find something online, but it seems that there is no way to simulate the test command via console.
Am I missing something?
Have you checked out the xcodebuild command?
The man page says:
buildaction ...
Specify a build action (or actions) to perform on the target. Available build actions are:
test Test a scheme from the build root (SYMROOT). This requires specifying a scheme.
So, perhaps that will do what you want.
Has anyone seen this very strange behaviour before?
I've got a solution whith 70 unit tests. All of them pass on my dev machine.
Whenever I commit my changes, our continuous integration process kicks in and the build box will eventually run the same 70 unit tests.
There is only ONE test in the build box that fails all the time.
The error is in one line that only gets a record from our unit test db. (I know it sucks having unit test to rely on data but please don't focus on this as it's not relevant now)
The most weird thing is when I logon myself to the build box, open up the same visual studio solution and manually kick off the unit tests. Result: ALL PASS!
Has anyone ever had this weird situation? I'm guessing there is some weird thing going on with Cruise Control.NET and MSTest?
Surely your unit test runner produces a good log that shows the exact exception message or error? It's kinda pointless to guess at it but an "access denied" kind of error would be an obvious candidate. Setup whatever dbase engine you use (you forgot to mention that too) to give the user account that runs the tests on the build grunt access to the tables.
As said in another answer, it doesn't make too much sense to guess about it when there are detailed logs around...
But because I had this situation several times, here's a guess anyway:
The account, which is used by the CI server to run the tests, may not have appropriate permissions in the database. This would also explain, why the same test succeeds when you run it manually (then with your user account)...
HTH!
Thomas
thanks for your inputs but it wasn't anything related to credentials at all.
I've found out that other tests that were running before that particular one were leaving my unit test database in an inconsistent state, therefore causing errors to the test in question.
It's not a good practice to have your unit tests relying on data, so unless you are extremely bound to it like myself, this is what a recommend to everyone: DO NOT RELY ON DATA TO DO YOUR UNIT TESTING !!!! Make sure you have all the good stuff in place, specifically a good IOC/dependency injector container so your classes are loosely coupled and you can mock up any interface you may want to unit test easily!
If you have system tests that you want to run on your build server or in general, want to be able to run correctly on any machine, including your own, then you must make sure that their states are independent.
In your case, you should have each test init prepare the DB it uses (either by copying a file-based DB or by emptying/filling a service-based DB). Each test should also attempt to undo its changes (delete file or empty DB) but not assume that other tests have done so successfully.