Recommended Go build system for a CI server? - go

So I have a go project with go-gettable dependencies, tests, etc.
I want to integrate it into Jenkins. Is there an automated build system for go that anyone recommends for this use case, other than writing makefiles?
I need:
automatic installation of go-get deps (they can be in a spec file of course)
recursive build.
running tests.
GOPATH/GOROOT management (to isolate SDKs and paths)
I've used godag in the past for this sort of job but it seems a bit unmaintained.
EDIT: For the time being I'm living with the following script entered directly into Jenkins as a build step:
#this gets the dependencies but doesn't install them, avoiding permission problems
go get -d
#build the packages, -x outputs the compiler command line
go build -x
#this was tricky - ./... means "for each sub-package recursively"
go test ./...

You can do it with teamcity as well.
Here is an example for building terraform.
Teamcity agent setup:
Install Go
Add Go to path
Don't forget to restart agent as path was changed
Teamcity build setup:
Use agent side checkout (if we want to include the .git folder, which a lot of build scripts make use of)
Use checkout rule (we want to use the Go convention):
+:. => src/github.com/mitchellh/terraform
Build steps:
echo cd %system.teamcity.build.checkoutDir%
cd "%system.teamcity.build.checkoutDir%"
path=C:\Program Files (x86)\Git\bin;%env.Path%;%system.teamcity.build.checkoutDir%\bin;
echo path=C:\Program Files (x86)\Git\bin;%env.Path%;%system.teamcity.build.checkoutDir%\bin;
set GOPATH=%system.teamcity.build.checkoutDir%
echo set GOPATH=%system.teamcity.build.checkoutDir%
echo "Getting dependancies"
go get golang.org/x/tools/cmd/vet
go get golang.org/x/tools/cmd/cover
go get golang.org/x/tools/cmd/stringer
go get github.com/mitchellh/gox
echo cd %system.teamcity.build.checkoutDir%\src\github.com\mitchellh\terraform
cd "%system.teamcity.build.checkoutDir%\src\github.com\mitchellh\terraform"
echo "Update resources"
go get ./...
echo "Run static code analysis"
go tool vet -asmdecl -atomic -bool -buildtags -copylocks -methods -nilfunc -printf -rangeloops -shift -structtags -unsafeptr .
echo "Build"
cd scripts
sh build.sh
echo "Run unit tests"
go test -timeout=30s -parallel=4
echo "Run code coverage"
go test -coverprofile=coverage.out
go tool cover -html=coverage.out
rm coverage.out

I am using a Team City build server on a Mac that runs a rake file, in the rake file I do all the commands to get dependancies, (go gets), tests and builds to the correct environment.
let me know if you want any pointers in writing the Rake files,
As a side note, i have been creating functional tests for REST Api's using this frame work. This has saved my code many times over. http://github.com/DigitalInnovation/cucumber_rest_api

Since August 2019, TeamCity 2019.1 now supports Go directly.
See "Building Go programs in TeamCity"
To enable Go support in TeamCity,
go to Build Configuration Settings | Build Features,
click Add build feature, and
select Golang from the list.
Support for Go is provided by TeamCity out of the box, there are no external plugins required.
TeamCity parses results of go test command execution. The results are persisted and it is possible to review the figures in a historical perspective.
Consequently, all the TeamCity features that are related to test reporting are now available for Go developers.

Related

go generate on build stage

In some build systems like gradle or blaze I can generate code (ent or proto) on a build stage and don't add in to a repository.
Is it possible to do the same for the go build command?
Yes, if you add "go generate" as your pre-build step in CI script.
$ go generate
$ go build
$ go test
But I would recommend more practical approach: to store your generated code in your repo and check it on CI - run go generate and assert that there is no changes.
Links
https://go.dev/blog/generate

Can't run test function in GoLand: cannot find package "."

I have used VS Code as my code editor for building service using Go for almost one year. Then, I tried to switch to GoLand. But, when I tried run a test function there is an error: cannot find package "." What is the problem?
Note: I use go module as go dependency management tool. When I use dep (in another project), there is no error when running a test function. My project is in GOPATH.
Please make sure there's a valid go.mod file defined at the ROOT of your project i.e. in $GOPATH/src/<Project-name>.
If you don't have, you can create one using go mod init command using the shell. More information on the same - https://github.com/golang/go/wiki/Modules
After that please try running the test from the shell. First cd into the directory where the test file is present. Then, use go test . -test "<TestName>" -v to run your test. If the issue goes away, you can run the test from IDE and it should work fine.

Making a CI server build all XCode targets as .ipa's

I'm looking into TeamCity and Jenkins, for a CI server.
My goal is this: every time someone commits a change to our repo, the CI server builds all targets in the project as .ipa's - ready for downloading/installing on a device.
I got Teamcity and Jenkins up and running, using a Mac mini as a build slave. That part of it is working fine.
Using Jenkins XCode plugin, I succeeded in building all targets as .ipa's.
I havent had such luck with Teamcity. The XCode plugin doesnt allow building all targets. Rather, you have to specify which targets you want to build, in each build configuration.
I approached the makers of Teamcity, and they gave me some convoluted method involving meta runners and a lot of duplication, in order to accomplish my goal.
Instead of relying on plugins, I'd rather build the .ipa's using shell scripting. However, as I'm not a script ninja, I can't figure out how to go about this.
I can figure out how to build one target via scripting, but it illudes me how make it build them all. Everytime I create a new target in the project, I don't want to have to add it at the CI server. The server should be able to automatically build all targets in the project.
...Maybe someone has a better solution? Any help is much appreciated.
What you should do (codes are bash script snippets, ready to run on OS X, you don't need to install anything except Xcode's CLI/Command Line Tools):
if you want to do this for every Xcode project file you have in your repository you should first search for these (if you have a specific Xcode project you can skip this)
for path in $(find . -type d -name '*.xcodeproj' -or -name '*.xcworkspace')
do
after this you can query all the shared (!) schemes through Xcode's command line tool
if [[ "$project" == *".xcodeproj" ]]; then
xcodebuild_output=($(xcodebuild -list -project "$project"))
else
xcodebuild_output=($(xcodebuild -list -workspace "$project"))
fi
now you have all the schemes so you can simply xcodebuild them one-by-one
Here's a bash script we developed to search for every Xcode project and every scheme configuration in a repository: https://github.com/concretebuilder/steps-cocapods-and-repository-validator/blob/master/find_schemes.sh
Note: you need to mark you schemes as shared to get xcodebuild (the command line interface of Xcode) list them.

Jenkins + Windows + CMake + multiple build types (Debug, Release)

How can I make Jenkins do the following?
Checkout trunk/ from SVN, then build configurations Debug and Release using CMake, without having duplicate jobs for the configurations.
Took me a while to figure this out. Here's how I managed to do it.
Create a free-style job "Checkout". This job is going to do all the stuff that doesn't depend on the configuration type (Debug/Release).
Under "Source Code Management" select Subversion
Fill in the Repository URL. Probably a good idea to make it point to /trunk.
Set Local module dir to "." (no quotes)
As Check-out Strategy "Emulate clean" is nice
Build trigger Poll SCM, set Schedule to "5 * * * *" to check every 5 minutes.
Now under Advanced Project Options check 'Use custom workspace' and set the dir to e.g. "c:/src". We don't want Jenkins to use its internal workspace, because we want other jobs to be able to access the source.
Under Build add the following Windows batch command, which is used to clean the build dir. For some reason, CMake doesn't provide a way to do this.
cd c:\
rmdir /S /Q build
mkdir build
cd build
cmake --version
rem optionally: svn info c:\src
cmake -G "Visual Studio 10" c:\src
Create another job "Build", this time make it a "multi-configuration" job. This job is going to run for each configuration (Debug/Release).
First, set the Build Triggers to build after job "Checkout"
Now under Configuration Matrix add an axis "configuration" with values "Debug Release" (whitespace = separator). Unfortunately, the CMake builder plugin for Jenkins doesn't work with multi-configuration jobs. We can't even use cmake --build, because it always builds the Debug configuration. To build, we have to use another batch script:
cd c:\build
call "%ProgramFiles(x86)%\Microsoft Visual Studio 10.0\VC\vcvarsall.bat"
msbuild ALL_BUILD.vcxproj /verbosity:minimal /maxcpucount:1 /property:Configuration=%configuration%
If you want to build the entire solution, specify the .sln file instead of ALL_BUILD.vcxproj. If you only want to build a specific project, use
msbuild <solution>.sln /target:<project>
Use Jenkins Matrix job. Define one of the axes as build_mode with values Debug and Release. You then run CMake that will create both configurations for the compilation tool you'll be using (XCode, gcc, VisualStudio, etc.). You can then use build_mode as if it were an environment variable and pass it to build steps that do actual compilation.
When using the Visual Studio generator you can pass the configuration to build to the cmake --build-command:
cmake --build . --config Release
cmake --build . --config Debug
See also the CMake docs.
After using Jenkins for a while now, I found that you should use as few jobs as possible if you want to reuse the source directory.
The default setup in Jenkins is that each build uses a different directory as its workspace. Which implies that you do a complete SVN checkout every build. Which takes forever.
If you want to use the same source directory for every build, you have to worry about synchronization: Only one build at a time. As far as I know, Jenkins has no built-in means of synchronization. The only way is to only use one executor. Even then you can't control the way the executor chooses its next job.
Let's say job "SVN update" triggers job "Build". Someone starts "SVN update #33", which is supposed to trigger "Build #33". If, however, Jenkins' "Poll SCM" feature schedules "SVN update" #34 in the meantime, I haven't found a way to tell it that "Build #33" must run before "SVN update #34". So you might end up with "SVN update #34" running before "Build #33", and everything fails. Unless you manually disable the polling job. And remind yourself to re-enable it afterwards, of course.
Anyways. After using Jenkins for two years, I change my answer to: Never use multiple jobs that share resources (like the source dir), and bake all the logic into shell scripts (for loop over configurations).

How to set up a CI environment using jenkins, rvm and cucumber

I am new to CI and would like your thoughts and input on how to go about my problem. I would like to first start off that I have been wrestling with this for 2 days(and I don't have that much background in sys ad) so please play nice?(I am mainly a front-end web dev) :)
Basically my plan was to install jenkins then make a CI env with these steps:
poll for any changes to github
if there are, run the build script:
a. migrate the development and test dbs?(does that mean i have to put the config/database.yml in my repo?)
b. run cucumber
c. if all tests pass go to 3, else fail
run any rake setup stuff
run the server(deploy)
I have done some of the stuff by cheating:
in my local, i switch my rvm to the correct one i need(rvm use 1.8.7-p174#mygemset)
run jenkins(java -jar jenkins.war) so that it gets the RVM ruby as default
run spork in a separate terminal(because for some reason my cucumbers don't run without spork - that's another problem)
build the project manually by clicking Build
so basically, I want to automate these stuff. Maybe what I need is a set of steps to follow(general or specific, depending on your taste) so I can setup my CI up and running.
Keep in mind that my "cheats" won't do as I want to test different projects with different setups and the startup cheat just won't do. Currently, my project build was successful because all I did was to run cucumber(and all my cukes pass). I want it to be able to deploy after it passes so maybe some help there also? Thanks
Okay I will try and help you as best I can.
poll for any changes to github
This can be easily done with the Github Plugin located here
if there are, run the build script: a. migrate the development and test dbs?(does that mean i have to put the config/database.yml in my
repo?) b. run cucumber c. if all tests pass go to 3, else fail
Then all you would is run the build script you have configure in the in the build from
Select "Add Build Step" -> "Execute shell".
You can either do that which is probably what I would do because when you create build you want them to be portable so you can start up in new jenkins instances, so you dont have to setup your build machine, with build specific files.
Then you run your tests, if they fail the build should fail regardless here is some information on running ruby on rails tests. if you need to manually fail a build in a script based on a result usually exiting a script with non-zero will fail the build. If not continue and run your rake and deployment scripts.
Just a few notes on Jenkins it wont do everything for you but if you can do it manaually Jenkins can automate it. So anything you have setup running manually with a little bit of effort you can get up and running automated with Jenkins
Here is another answer you might find helpful in your general setup and ideology behind Jenkins.
Goodluck!

Resources