I would like to post the results of tests for my iPhone app to my TestLink using XML-RPC.
I use Kiwi in my project, and now I want to get the results of tests. Can I know whether the conditions on my testcase passed of failed?
The results of Kiwi tests are recorded in basically the same way as the results of typical OCUnit tests, thus there shouldn't be anything special about exporting the results of Kiwi tests compared to other Xcode test frameworks. With Xcode 4, a log file is generated at:
~/Library/Developer/Xcode/DerivedData/(product_identifier)/TestResults/(timestamp).xctestresults/results.plist
This plist file has a simple format, with an array of dicts for each Kiwi spec. The "Test Identifier" and "Test Name" values are generated by Kiwi by concatenating strings in the (possibly nested) context and it declarations for each spec, and the "Test Result" value will either be Succeeded or Failed.
You might want to refer to some other questions (3 different links) on Stack Overflow that discuss exporting Xcode test results or automating/scripting testing.
If you don't mind running the Kiwi tests from the command line then you could use xctool. This allows you to output the contents of the tests that were run (and their pass/fail statuses) to a JSON file which is friendly(er) for importing into a third party ticketing system like Test Link compared to the raw XC.
After installing xctool run your tests like this:
xctool test \
-project ProjectName.xcodeproj/ \
-scheme SchemeName \
-reporter phabricator > ~/Desktop/test_results.json
That will output a JSON file which can be imported directly into Phabricator, but you could run another script after to put the JSON into in a format Test Link can integrate.
Related
I'm running the following command to run my .net Core tests:
dotnet test
This runs fine. I want to now generate code coverage stats, so after following this article, I ran this:
dotnet test AI.Core.Tests.csproj
/p:CollectCoverage=true
/p:CoverletOutputFormat=cobertura
/p:CoverletOutput=TestResults\Coverage
I get the following output from this command:
C:\Users\sp4_rm\.nuget\packages\coverlet.msbuild\2.2.1\build\netstandard2.0\coverlet.msbuild.targets(23,5): error :
Index was out of range. Must be non-negative and less than the size of the collection.
[C:\Users\sp4_rm\Desktop\EVO\AI.Core\src\Tests\AI.Core.Tests.csproj]
C:\Users\sp4_rm\.nuget\packages\coverlet.msbuild\2.2.1\build\netstandard2.0\coverlet.msbuild.targets(23,5): error :
Parameter name: index
[C:\Users\sp4_rm\Desktop\EVO\AI.Core\src\Tests\AI.Core.Tests.csproj]
See screen shot below:
Has anyone got this command running? What am I doing wrong?
Ok so this was due to a school boy error in not actually have any code to test (or test case to test it) in my sample project!! Adding a couple of classes into the main projects and a couple of tests in the test project does away with this problem! (just in case anyway does the same thing as me!)
We have a static library we've been distributing for some time, and now want to distribute it as a framework. The static library is built to contain both phone and simulator slices by performing two xcodebuild commands and lipo'ing the results together. This has been working fine.
But when I initially attempted to apply the same approach to a framework, the App Store validation process griped that the resulting product wasn't built for full bitcode. Having looked into it, people online are saying I need to use archive instead of build for the device slice. It results in an xcodebuild command like this:
xcodebuild archive -target OurProduct -scheme OurProduct -archivePath ./Archive/OurProduct.xcarchive
Now at this point I appear to have a valid xcarchive. Then I'm running:
xcodebuild -exportArchive -archivePath "./Archive/OurProduct.xcarchive" -exportPath "./Archive/OurProduct.framework" -exportOptionsPlist "./exportOptions.plist"
But I get the error:
error: exportArchive: exportOptionsPlist error for key 'method':
expected one of {}, but found app-store
"app-store" is the value of key "method" in my export options plist file. I need some help trying to understand what could be the problem here, or if there's a more proper way to do this.
setting skip_install to YES fixed my issue.
When using stack in a project with multiple cabal files, there seems to be a difference in verbosity when using a stack command on a single target vs multiple targets.
This seems to happen for all sorts of stack commands, so I'll just give a few examples.
With stack build sub-project-1, stack will show progress on each of the modules it compiles. But stack build sub-project-1 sub-project-2 or stack build will suddenly show only which target it is working on and not show any of the modules it is working on.
Another example I have found recently is in stack test. I wanted to get a full list of all the Tasty tests I have, so I ran stack test --test-arguments="-l". But all it printed out was:
sub-project-1-0.0.0.1: test (suite: run-tests, args: -l)
sub-project-2-0.0.0.1: test (suite: run-tests, args: -l)
Completed 2 action(s).
Log files have been written to: /projdir/.stack-work/logs/
Even if I manually specify the targets I want eg:stack test sub-project-1 sub-project-2 --test-arguments="-l", it gives me the same unhelpful output.
I have to run stack with exactly one target: stack test sub-project-1 --test-arguments="-l" to get any of the output I am looking for:
sub-project-1-0.0.0.1: test (suite: run-tests, args: -l)
all tests/tasty tests/this is a test
all tests/tasty tests/this is another test
Is there anything I can do to get stack to not squelch the output when running against more than one package? Stack verbosity levels don't seem to have anything to do with this. They seem to consider -v to mean "print out debugging statements"
Thanks to sjakobi for the suggestion.
Recent versions of stack have an option: --dump-logs
This option will show the output from each action stack takes on each target.
I have created an release app with rebar3 (beta-4).
Added some eunit tests and wrote some code.
For now I have to debug one test case to see what I have to add to make the implementation to work properly.
I found some articles about using dbg from Erlang console and I found how to write debug info from Eunit. But I need to get info from code that I have to test (the actual implementation(logic)).
Is there a way to debug Erlang code (actual source code, not the test one) when rebar3 is used with eunit argument?
I'm using tracing in terminal like there: https://aloiroberto.wordpress.com/2009/02/23/tracing-erlang-functions/
One way to do this is use rebar3 to run a shell under the test profile, then start the debugger and set up your breakpoints and such:
$ rebar3 as test shell
...
1> debugger:start().
{ok, <0.67.0>}
This will pop up the debugger GUI. Once the debugger is set up and ready, run your test under eunit:
2> eunit:test(your_test_module,[verbose]).
======================== EUnit ========================
your_test_module: xyz_test_ (module 'your_test_module')...
Assuming you set up a suitable breakpoint in the debugger, this will hit it, but you'll likely run into a problem with this approach: by default, eunit tests time out after 5 seconds, which doesn't give you much time for debugging. You need to specify a longer timeout for your test, which is why the example above shows that what's running is a test fixture named xyz_test_, which wraps the actual test with a long timeout. Such a fixture is pretty simple:
-include_lib("eunit/include/eunit.hrl").
xyz_test_() ->
{timeout,3600,
[fun() -> ?assertMatch(expected_value, my_module:my_fun()), ok end]}.
Here, the actual test is the anonymous function, which matches the return value from my_module:my_fun/0, which for this example represents the business logic under test. This example fixture sets the test timeout to one hour; you can of course set it as needed for your application.
I found a command-line tool called "xctest" that apparently can run the unit tests in your project. This executable lives here:
/Applications/Xcode.app/Contents/Developer/usr/bin/xctest
When I try to run this executable on my xctest bundle, I'm using:
$ ./xctest /Users/myusername/Library/Developer/Xcode/DerivedData/MyApp-abcdefghijklmnop/Build/Products/Debug/MyAppTests.xctest
However, I get the following output:
Test Suite '(null)' started at 2013-11-14 21:16:45 +0000
Test Suite '(null)' finished at 2013-11-14 21:16:45 +0000.
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
There's no man page for xctest, as far as I can tell, but entering just ./xctest at the command-line yields:
Usage: xctest [--test Self | All | None | <TestCaseClassName/testMethodName>] <path of unit to be tested>
In particular, I'd like to be able to test just a particular method in a test class, which is why I'd like to use this xctest command.
I do see that there is a way to run all the tests from the command line like:
$ xcodebuild test -scheme MyApp
This runs all the unit tests and works properly (I see my unit test results, unlike when using xctest). But I'm interested in being able to run a single test method from the command-line, such as:
$ ./xctest --test MyAppTests/testExample /Users/myusername/Library/Developer/Xcode/DerivedData/MyApp-abcdefghijklmnop/Build/Products/Debug/MyAppTests.xctest
Despite what the usage message says -XCTest is the argument you need:
xctest -XCTest MyAppTests/testExample testbundle.xctest
For a direct invocation of xctest to work you may also need to set DYLD_FRAMEWORK_PATH and DYLD_LIBRARY_PATH to your built products directory. In general you need to use the same arguments and environment as Xcode does, you can see this by putting a breakpoint in one of your tests, running them through Xcode, then printing out the values of arguments and environment for [NSProcessInfo processInfo].
To avoid messing with all that note you can also modify the scheme in Xcode to run only specific tests. Under Product > Scheme > Edit Scheme select the Test action and expand the test bundle. You can use the check boxes to select the tests to run and xcodebuild's test action will then run only these tests.