I'm trying to add automated tests to my macOS app on Travis CI, but can't quite figure out code signing.
My (private) GitHub repository is set up to trigger Travis build jobs when I push to master.
For iOS projects, Xcode builds/runs/tests the project for the Simulator platform, so no code signing is required for testing (signing with a distribution identity is necessary to deploy a build, of course. But I just want to run unit tests).
But for macOS apps there is no "Simulator": the code is built and run on the development machine itself.
This article explains how to add distribution code signing artifacts to Travis' machine, so it can build/sign a distribution binary for iOS.
I have modified the steps explained there to use macOS development artifacts instead of iOS Distribution ones. The scripts that decrypt my artifacts and install them on the Travis machine seem to work with no problem.
The Problem
However, unlike for distribution, development provisioning profiles contain a specific list of devices on which builds are allowed to run; in my case, my profile obviously only contains de device ID of my local machine. Obviously there is no way I can get the device ID of the mac that Travis uses, and even if I could, the build obviously runs on a different machine each time.
How Can I Build and Unit-Test macOS Apps on Travis CI?
TL; DR:
I solved the code-signing problem by disabling code-signing altogether (it isn't needed for running unit tests), as explained in this brilliant answer by #robmayoff.
The Details
I still need code-signing for testing my app locally (it uses entitlements to read from/write to user-selected files and folders). So I created a new configuration in Xcode by cloning Debug, named UnitTest.
I disabled code-signing and set Development Team to "none" for this configuration.
Next, I created a dedicated scheme (shared, of course) that uses the UnitTest configuration for build and test. I had to do this in both the App target and the Tests target (I gave up on the UI tests because they require Accessibility enabled to run, and the remote machine does not have those permissions). I had to do this because I couldn't get the xcbuild tool to use the UnitTests configuration.
I haven't quite got my Travis build to successfully complete yet, but I've overcome most obstacles (code-signing included).
Hope this helps someone!
Update
I finally got Travis CI to successfully build and test my app. Off-topic, but my code had the following issues:
Deployment target of macOS 10.15 (latest available image on Travis is 10.14)
The project was linking against CryptoKit.framework (available only since 10.15!); had to replace it with similar functionality in CommonCrypto.
Somehow the issues above didn't prevent building the app, only running the tests. I was getting an 'image not found' error for CryptoKit.framework.
Related
I just got to know about Travis CI and went through some of their docs. It seems to be a nice solution for open source projects.
With my reading so far through Travis docs, I am doubtful whether I will be able to connect it to my personal hardware in some manner.
I am working on some IoT related project written in C/C++ hosted on github. Building and publishing images on artifactory, on Travis CI should not be a problem. But when it comes to testing, definitely it cannot be tested on their (Travis's) hardware. The binaries need to be put on my development board (raspberry pi) and then test cases should be executed. Once test suit finishes, Travis CI should be notified of the results.
Is Travis allows such functionality? If not, then it would be a great limitation.
But when it comes to testing, definitely it cannot be tested on their
(Travis's) hardware
By this statement you answered your own question. If you want to use Travis for showing test status publicly, your best chance is running the tests on premises using Jenkins, GitLab CI or any other CI server and then offering an API to Travis (e.g. a file with the test results). All Travis would then do is fetching and showing the test results.
If you want to control things from within Travis, you could trigger builds from Travis using a HTTP call and then wait for the remote build to finish before showing its results. Both Jenkins and GitLab CI offer remote triggering. However, this requires your CI servers to be publicly accessible, which might be a security risk (e.g. people triggering non-stop builds).
TravisCI is primarily useful for testing libraries and projects that can be run and tested on common systems (linux, etc.) I don't believe there is a way to run TravisCI locally however, MinionCI seems to offer a solution for running a CI server locally following a style similar to TravisCI, check it out here.
I am an intern now, new to automation test.My goal here is to help my company set up CI for client side.
Right now I have a maven project contains several tests using Appium java-client lib, under Eclipse IDE, which could run the UI tests locally. My goal next step is to hook my tests with the gitlab repo(which is already there, created by the android developers), but I am stuck here. Could somebody help me out?
Please try to be specific:
how should I set up the .gitlab.yaml?
can we just have the script in yaml to download Appium and maven?
or we could just download Appium, but import all the Appium java-client jars to libs in main?
If either of above is true, how? if neither, what and how should I
do?
Where should I put my test in gitlab in that repo? Or I don't have to
put my tests in the existing repo. Instead, I could have another one
and tell yaml where to reach? Again, how?
It will be helpful if you could help me go through the workflow.
Like, when I developers check in code, gitlab read the yaml, then
build, then find my test suits in where(Q3), then execute etc.
Many thanks in advance!
Since finally someone is also interested in this question, let me share my solution to this.
So, if you are looking at this question, I assume you already have your test suite and you could test it locally in your machine, either have your app installed in a simulator or a real device. Now you need to read more about gitlab pipeline and gitlab CI :
pipeline: https://docs.gitlab.com/ee/ci/pipelines.html
gitlab CI: https://docs.gitlab.com/ee/ci/quick_start/
And you should have noticed that, one of the advantages of Appium is that you don't need to change a thing about the App you are testing, you are testing exactly the same App which is going into production. To learn more about Apppium:
http://appium.io/docs/en/about-appium/intro/
Now, to run the automation test, you need your test suite, the app, and Appium server. What we need to do is adding another stage in .gitlab-ci.yml, tell it to
take the newly compiled App, compile your test suite
install the App in simulator/real device
compile your test suite and run it.
To make things easier to understand, we start with question 4, workflow:
So when the code is checked in to gitlab, the gitlab runner runs the jobs of each stage in your .gitlab-ci.yml, and when it runs to your stage, it does the automation test, and note that it is running on your server, so it means you need to have Appium installed on your server and have it up and running when try to run your automation test suite. Now the problem is that, is your server capable to do so? If you wanna do the automation test in your server, you need to install Appium on it, simulator probably(and which might need your server to equip with GPU), etc, these are the concerns of maintaining server. The alternative would be using the third-party service ,which is what I did. Turns out our(when I was in that company) server isn't capable of running automation UI test, so we turned to AWS-ADF(Amazon Device Farm), there are many other service providers you could choose, see the link for references:
https://adtmag.com/blogs/dev-watch/2017/05/device-clouds.aspx
So I basically have a python script in my functional test stage, and it will grab the newly complied App, the automation test suite, upload them to AWS ADF, and then schedule a run, yields result when the run is finished.
so, to answer question 1:
we need to create one more stage for our functional test in .gitlab.yaml, in my case, I have a stage functionalTest_project stage after the stage which compiles the Android App. And then you script the necessary cmd in your stage, or if its too lengthy, your script in another file(put it in your repo) and then execute it. In my case, I put my script in python_ci.py, and then I execute it in my stage use “python python_ci.py” .(here you need a docker with these requirement, see below too)
You don’t download Appium, you set up Appium on your or if you use a cloud service, that service should set up Appium for you.
What I did it is that I use maven built and package the test suite locally and then push it to gitlab repo, which now I believe the better way would be compile and package it in the your functionalTest stage in .gitlab.yml. now it comes back to first point of question 1, how to get maven, my understanding is that its a dependency of the server, like python, so they could both be obtained by telling gitlab to execute your script with a docker that has python and maven dependency.
answer to question 3:
put it in the same repo, but out of the Android project(i.e. they will under the same directory).
how to tell yml to reach the test suite? remember they are in the same server, so you could the relative path in your yml script to tell yml where to get your test suite.
Hope this helps!
I've got a phonegap project that is getting built via Jenkins via the Xcode plugin. After the project is built, I'm using the TestFlight plugin to upload it there. Everything works great.
Now, I need to add two more versions of this for our different environments so I'll have a Dev and Test version (and also Prod, but not to TestFlight).
In order to do this, I need to change the name of the application so both Dev and Test show up in TestFlight, MyProgram-Dev, and MyProgram-Test. Then the production version would just be MyProgram.
In the xcode section on Jenkins, there's a "custom xcodebuild arguments" field that can be filled out, which I'm assuming is where I need to make my changes. I want to do this at build time because I don't want the program name to change when merging Dev into Test, etc.
I've done alot of googling and can't find any example command line parameters for the xcode build process. Is it the CFBundle that I need to be changing? What is the name of the parameter to change?
Edit: So I was able to change the production name via 'PRODUCT_NAME=MyProject-Test', but that produced the following error:
The following build commands failed:
Ld /Users/buildaccount/Library/Developer/Xcode/DerivedData/MyProject-cpvmaiwadviebmbhkdggxpzejddf/Build/Intermediates/MyProject.build/Release-iphoneos/MyProject.build/Objects-normal/armv7/MyProject-Test normal armv7
Wow, that took alot of googling and testing.
Here's what was the eventual fix for my scenario. It was a combination of this post/answer, as well as this one.
my team is currently using buildbot to automate overnight and continuous-integration builds and regression tests. For builds and unit tests, the builder just invokes a script which syncs the sources from p4 and then runs xcodebuild. The regression tests are also launched by a shell script, and are themselves combinations of shell scripts and AppleScripts. The builds are all performed on one system but then buildbot triggers tests to happen on multiple Macs, with different architecture and OS combinations. The things I like about buildbot are the automatic triggers (so the tests run only if and when the build succeeds), and the reporting including the waterfall view to see the overall status.
However, I see a number of problems too. The "master" process which coordinates the builds is either leaky or just has a huge working set which means that it consumes a couple of gigabytes of memory. Occasionally network problems mean that a slave will be lost; rather than retrying later it will just fail the build. In fact if the slave is supposed to be triggered for a dependent build, it will fail the first build after it has otherwise successfully completed.
So what are the rest of you using to automate your Xcode builds and unit tests? How do those solutions work for you? Anything you would recommend?
I use TeamCity in a windows environment but I believe it will work on Mac and has xcode build agents.
Another similar question here.
Our team use Hudson with a script which converts the Unit Test output from XCode to NUnit format.
I know this is an older question, but for those using TFS to hold their Xcode projects, I created a custom build activity to make automating Xcode builds via TFS easier. The code is hosted on Codeplex here: http://tfsxcodebuild.codeplex.com/.
Hope someone finds it useful!
We worked out what was using all the memory - log files - older buildbot keeps all logs forever and keeps them in memory (possibly until a restart).
Newer buildbot can be configured to keep a fixed amount of history.
Buildbot build log files should be limited in size to avoid the problem.
I use the XcodeBuilder that's part of CruiseControl. Of course it helps that I wrote it. :)
But I did use it on a real multiple person project for an iPhone app (Surf) that's for sale in the app store.
Now that it's three years later I have my own answer to provide to this question. I'm using Jenkins, mainly for the plug-ins that it provides. It has a plug-in for building targets in Xcode projects (or schemes in workspaces). There's a plug-in to run the Clang static analyzer. It interfaces with my bug-tracker system, it'll automatically push builds to Testflight too.
In using our TeamCity Continuous Integration server we have uncovered some issues that we are unsure as to the best way to handle. Namely how to reference external applications that our application requires on the CI server.
This was initially uncovered with a dependency on Crystal Reports, so we went and installed Crystal Reports on the Server fixing the immediate problem. However as we move more applications over to the CI server we are finding more dependencies.
What is the best strategy here? Is it to continue installing the required applications on the Server?
Thanks
Where possible make the external dependencies part of your build system.
For instance check the installer in to your version control system and have a step that checks it out and runs it in silent mode (many installers support a mode with no user action sometimes using the commandline /s).
This way if you need to set up another build machine for a branch or just for new hardware everything is repeatable.
If your builds require the actual application to complete the build, then you should probably continue to install the application on your build server.
If you just need references to dlls or assemblies from the application, then what we've done at my company is to create installable 'SDKs' of the references required for a particular applicatoin and install them on our development and build machines in well-known library directories that our solutions reference.
On the build machine, our pre-build steps install the correct version of the dependencies and then clean them up when we are finished.
Recently, we've moved to using virtual machines for our build machines that our build process activates. These VMs get the SDKs installed on them as a pre-build, and then are restored to their snap-shot state after the build. We had some dependencies that were almost impossible to uninstall, so this made for a clean starting point each time.
If you use Maven to build, you can define your dependencies in the pom.xml file. They will then be automatically downloaded if necessary.
I am not sure if I followed correctly...
I am assuming your application is dependent on this external app, while building? In that case it should be on the machine doing CI...