I have a Go project that I want to build with Travis-CI and deploy it to a specific provider.
I familiar with Gimme project that will use a cross-compilation to do so.
But because Travis already support linux and osx I only need this feature for Windows build.
The big motivation is, of course, to avoid cross-compilation run time error as there are plenty of it.
My question is how can I create, in the same .travis.yml file, a different build flow:
Native linux/os build (with "os" section).
Windows compilation using Gimmme
The .travis.yml file for the first option will look something like:
language: go
go:
- 1.5.1
branches:
only:
- master
os:
- osx
- linux
before_script:
- go get -d -v ./...
script:
- go build -v ./...
# - go test -v ./...
before_deploy:
- chmod +x ./before_deploy.sh
- ./before_deploy.sh
The .travis.yml file for the second option will look something like:
language: go
go:
- 1.5.1
branches:
only:
- master
env:
- GIMME_OS=windows GIMME_ARCH=amd64
before_script:
- go get -d -v ./...
script:
- go build -v ./...
# - go test -v ./...
before_deploy:
- chmod +x ./before_deploy.sh
- ./before_deploy.sh
Is there a nice clean way to combine these two (with Environment variables or any other crazy idea)?
It might be simple, but matrix environement can not be done for a specific OS ...
Then just select with local environement variable:
language: go
go:
- 1.5.1
branches:
only:
- master
os:
- osx
- linux
install:
- if [ "$TRAVIS_OS_NAME" == "linux" ]; then
export GIMME_OS=windows;
export GIMME_ARCH=amd64;
fi
before_script:
- go get -d -v ./...
script:
- go build -v ./...
after_script:
- go test -v ./...
before_deploy:
- ./before_deploy.sh
An other way:
language: go
go:
- 1.5.1
branches:
only:
- master
matrix:
include:
- os: linux
env: GIMME_OS=windows; GIMME_ARCH=amd64;
- os: osx
before_script:
- go get -d -v ./...
script:
- go build -v ./...
after_script:
- go test -v ./...
before_deploy:
- ./before_deploy.sh
Note: the commande: - chmod +x ./before_deploy.sh can be directly done in your repository and commited on it ...
Note: The environament variable can be accessibe: http://docs.travis-ci.com/user/environment-variables/#Default-Environment-Variables or calling \printenv`
Related
I've released MooseX::Extended to the CPAN (github repository here).
I'm trying to set up github actions and the linux tests run just fine. However, (Windows is failing with this error:
Configuring true-v1.0.2 ... OK
==> Found dependencies: Function::Parameters
--> Working on Function::Parameters
Fetching http://www.cpan.org/authors/id/M/MA/MAUKE/Function-Parameters-2.001003.tar.gz ... OK
Configuring Function-Parameters-2.001003 ... OK
Building Function-Parameters-2.001003 ... OK
Successfully installed Function-Parameters-2.001003
! Installing true failed. See C:\Users\RUNNER~1\.cpanm\work\1653412748.5640\build.log for details. Retry with --force to force install it.
Building true-v1.0.2 ... FAIL
Of course, I can't see that C:\Users\RUNNER~1\.cpanm\work\1653412748.5640\build.log to understand what happened.
The true module passes its CPAN testers tests on Windows, so I don't know why it's failing in Github Actions.
My workflow looks like this:
# Hacked from https://github.com/skaji/perl-github-actions-sample/blob/master/.github/workflows/windows.yml
# See also: https://perlmaven.com/github-actions-running-on-3-operating-systems
name: windows
on:
push:
branches:
- '*'
tags-ignore:
- '*'
pull_request:
jobs:
perl:
runs-on: windows-latest
strategy:
fail-fast: false
matrix:
perl-version:
- '5.20'
- '5.22'
- '5.24'
- '5.26'
- '5.28'
- '5.30'
- '5.32'
- '5.34'
- 'latest'
steps:
- uses: actions/checkout#v2
- name: Set up Perl
run: |
choco install strawberryperl
echo "C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin" >> $GITHUB_PATH
- name: perl -V
run: perl -V
- name: Install Dependencies
run: curl -sL https://git.io/cpm | perl - install -g --show-build-log-on-failure Dist::Zilla
- name: Run Tests
run: |
dzil authordeps --missing | cpanm --notest
dzil listdeps --author --missing | cpanm --notest
dzil test --author --release
This is the PR to which the actions are attached.
I don't have access to a Windows box. Does anyone know what I missed?
Since GitHub Actions/Workflows uses a container for Windows that already has a version of Strawberry Perl pre-installed, it will not allow you to install any other version. You cannot remove the version of Perl that's pre-installed, and removing/installing a new one via Chocolatey is also next to impossible. If you re-install the version from Chocolatey that's already on the container, it seems to allow this, but it's basically a NOOP for you as a test setup.
The container also has MinGW installed; this can be bad for us as well. Having MinGW installed separately prevents XS modules from building (whether they be a dependency or if your own module is an XS module). Granted, this only happens if MinGW appears in the PATH ahead of your Perl install, but when you remove one Perl and add another, you're going to hit this problem.
To get around this, the best course of action is to remove the currently installed version of Perl from the PATH environment variable, along with their currently installed version of MinGW. Once both are safely out of the PATH, you can install a Portable[1] Strawberry Perl, put that Perl's paths in your PATH and begin testing with a fresh install of Strawberry Perl. GitHub recently broke our ability to do this directly in an Action YAML file.
That all sounds like a big headache, but it's really not. There's an Action available to us for this very purpose: actions-setup-perl. With this action you can easily test using any version of Perl you like. So, if you're hearing someone report a bug on Perl v5.26 on Windows, you can now add that to your matrix and test easily without the need for any back-and-forth from the user:
name: windows
on:
push:
branches:
- '*'
tags-ignore:
- '*'
pull_request:
jobs:
perl:
runs-on: windows-latest
strategy:
fail-fast: true
matrix:
perl-version:
- '5.30'
# - '5.28'
# - '5.26'
# - '5.24'
# - '5.22'
# - '5.20'
# - '5.18'
# - '5.16'
- '5.14'
steps:
- name: Setup perl
uses: shogo82148/actions-setup-perl#v1
with:
perl-version: ${{ matrix.perl-version }}
distribution: strawberry
- name: Set git to use LF
run: |
git config --global core.autocrlf false
git config --global core.eol lf
- uses: actions/checkout#v2
- name: perl -V
run: perl -V
- name: Ensure we have a working toolchain
run: cpanm ExtUtils::Manifest App::cpanminus
- name: Install Dependencies
run: cpanm -n --installdeps .
- name: Run Tests
run: cpanm --test-only -v .
[1] Portable versions of Strawberry Perl are zipped up, already compiled versions of Perl that do not require you to run an installer on Windows. This means that no heightened privileges are required, etc. You just unzip the archive in the directory you want to run Perl from, then add the relevant paths to Perl in your $env:PATH variable. It takes away any annoyances of build irregularities, etc. I've found it to be the most sane way to test on Windows.
I am trying to make a Linux executable file of my Go project. I have the following configuration in my .config-ci.yml in my gitlab project.
demo_job_1:
tags:
- cpf
- cpf-test
- testing
- unit-testing
script:
- go run test/main.go
- GOOS=linux GOARCH=amd64 go build
- go env
- cd test
- ./test
- echo Successfully run and Built
After running this pipeline, I still get the GOOS=windows when I check in env file. How can I build my project so that the output after building is of Linux executable file. Right now, I am getting .exe file which runs on Windows only.
You can see the project details in the following gitlab:
https://gitlab.com/smilekison/test
This is the error that is shown by Pipeline Job:
$ go run test/main.go
Hello world
$ GOOS=linux GOARCH=amd64 go build
GOOS=linux : The term 'GOOS=linux' is not recognized as the name of a cmdlet, function, script file, or operable
program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\WINDOWS\TEMP\build_script487326907\script.ps1:207 char:1
+ GOOS=linux GOARCH=amd64 go build
+ ~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (GOOS=linux:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
First to address your actual error: it seems you are on a windows based runner. That means you have to use windows CMD commands. It does not know ENV, etc.
You can do go env -w GOOS="linux" instead. Same with GOARCH. Then just run go build ..
You can also use a variables section to overwrite go env with environment variables:
variables:
GOOS: "linux"
GOARCH: "amd64"
It goes at the top of the gitlab file somewhere.
Here my typical build pipeline for Go projects using docker containers:
build_App:
image: golang:1.15.3
stage: build
allow_failure: false
tags:
- unix
script:
- go mod download
- mkdir $CI_PROJECT_DIR/release
- cd cmd/app
- GOOS=linux GOARCH=amd64 go build -o $CI_PROJECT_DIR/release/app .
artifacts:
paths:
- $CI_PROJECT_DIR/release
And test pipeline
go_test:
image: golang:1.15.3
stage: verify
allow_failure: false
tags:
- unix
script:
- go mod download
- go test -race -cover ./...
This is based on a runner that uses docker images to build in.
I needed to write go env -w GOOS="linux" GOARCH="amd64" to make the executable file for linux and if I want to make executable for windows I have to just rename linux to windows and I can make go language install by using image : golang:1.15.7 here. This way my .gitlab-ci.yml file can get GO Lang install and can run any go command.
demo_job_1:
stages:
-build
build:
stage: build
image : golang:1.15.7
tags:
- cpf
- cpf-test
- testing
- unit-testing
script:
- go run test/main.go
- go env -w GOOS=linux GOARCH=amd64
- go env
- cd test
- ./test
- echo Successfully run and Built
I want to build a docker image with a fixed version of micro and go dependencies. I plan to do it with dep:
git checkout git#github.com:micro/micro.git
dep ensure
git add Gopkg.toml
git add Gopkg.lock
# Build micro
CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o micro ./main.go
# Build docker image
...
So, my question is does it the best solution to build consistent micro docker image?
An example of a Dockerfile can be:
FROM golang:1.9-alpine3.6 as builder
# Install package manager
RUN apk add --no-cache --virtual .go-dependencies git curl \
&& curl https://glide.sh/get | sh
# Copy files from context
WORKDIR /go/src/github.com/foo/bar
COPY . .
# Install project dependencies, test and build
RUN glide install \
&& go test ./... \
&& CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o ./entry ./main.go ./plugins.go
# Build final image with binary
FROM alpine:3.6
RUN apk add --update ca-certificates && \
rm -rf /var/cache/apk/* /tmp/*
WORKDIR /
COPY --from=builder /go/src/github.com/foo/bar/entry .
ENTRYPOINT [ "/entry" ]
And the glide.yaml would look like this:
package: .
import:
- package: github.com/micro/go-micro
version: ^0.3.0
subpackages:
- client
- server
- package: github.com/micro/go-plugins
version: ^0.6.1
subpackages:
- wrapper/trace/opentracing
- broker/nats
- transport/nats
- package: github.com/opentracing/opentracing-go
version: ^1
- package: github.com/openzipkin/zipkin-go-opentracing
version: ^0.3
testImport:
- package: github.com/golang/mock
subpackages:
- gomock
- package: github.com/smartystreets/goconvey
subpackages:
- convey
In my case, dep looks great and fast enough, moreover, it's official dependency manager in go so I think it's a right choice.
I'm currently using GitLab in combination with CI runners to run unit tests of my project, to speed up the process of bootstrapping the tests I'm using the built-in cache functionality, however this doesn't seem to work.
Each time someone commits to master, my runner does a git fetch and proceeds to remove all cached files, which means I have to stare at my screen for around 10 minutes to wait for a test to complete while the runner re-downloads all dependencies (NPM and PIP being the biggest time killers).
Output of the CI runner:
Fetching changes...
Removing bower_modules/jquery/ --+-- Shouldn't happen!
Removing bower_modules/tether/ |
Removing node_modules/ |
Removing vendor/ --'
HEAD is now at 7c513dd Update .gitlab-ci.yml
Currently my .gitlab-ci.yml
image: python:latest
services:
- redis:latest
- node:latest
cache:
key: "$CI_BUILD_REF_NAME"
untracked: true
paths:
- ~/.cache/pip/
- vendor/
- node_modules/
- bower_components/
before_script:
- python -V
# Still gets executed even though node is listed as a service??
- '(which nodejs && which npm) || (apt-get update -q && apt-get -o dir::cache::archives="vendor/apt/" install nodejs npm -yqq)'
- npm install -g bower gulp
# Following statements ignore cache!
- pip install -r requirements.txt
- npm install --only=dev
- bower install --allow-root
- gulp build
test:
variables:
DEBUG: "1"
script:
- python -m unittest myproject
I've tried reading the following articles for help however none of them seem to fix my problem:
http://docs.gitlab.com/ce/ci/yaml/README.html#cache
https://fleschenberg.net/gitlab-pip-cache/
https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/336
Turns out that I was doing some things wrong:
Your script can't cache files outside of your project scope, creating a virtual environment instead and caching that allows you to cache your pip modules.
Most important of all: Your test must succeed in order for it to cache the files.
After using the following config I got a -3 minute time difference:
Currently my configuration looks like follows and works for me.
# Official framework image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python
image: python:latest
# Pick zero or more services to be used on all builds.
# Only needed when using a docker container to run your tests in.
# Check out: http://docs.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-service
services:
- mysql:latest
- redis:latest
cache:
untracked: true
key: "$CI_BUILD_REF_NAME"
paths:
- venv/
- node_modules/
- bower_components/
# This is a basic example for a gem or script which doesn't use
# services such as redis or postgres
before_script:
# Check python installation
- python -V
# Install NodeJS (Gulp & Bower)
# Default repository is outdated, this is the latest version
- 'curl -sL https://deb.nodesource.com/setup_8.x | bash -'
- apt-get install -y nodejs
- npm install -g bower gulp
# Install dependencie
- pip install -U pip setuptools
- pip install virtualenv
test:
# Indicate to the framework that it's being unit tested
variables:
DEBUG: "1"
# Test script
script:
# Set up virtual environment
- virtualenv venv -ppython3
- source venv/bin/activate
- pip install coverage
- pip install -r requirements.txt
# Install NodeJS & Bower + Compile JS
- npm install --only=dev
- bower install --allow-root
- gulp build
# Run all unit tests
- coverage run -m unittest project.tests
- coverage report -m project/**/*.py
Which resulted in the following output:
Fetching changes...
Removing .coverage --+-- Don't worry about this
Removing bower_components/ |
Removing node_modules/ |
Removing venv/ --`
HEAD is now at 24e7618 Fix for issue #16
From https://git.example.com/repo
85f2f9b..42ba753 master -> origin/master
Checking out 42ba7537 as master...
Skipping Git submodules setup
Checking cache for master... --+-- The files are back now :)
Successfully extracted cache --`
...
project/module/script.py 157 9 94% 182, 231-244
---------------------------------------------------------------------------
TOTAL 1084 328 70%
Creating cache master...
Created cache
Uploading artifacts...
venv/: found 9859 matching files
node_modules/: found 7070 matching files
bower_components/: found 982 matching files
Trying to load /builds/repo.tmp/CI_SERVER_TLS_CA_FILE ...
Dialing: tcp git.example.com:443 ...
Uploading artifacts to coordinator... ok id=127 responseStatus=201 Created token=XXXXXX
Job succeeded
For the coverage report, I used the following regular expression:
^TOTAL\s+(?:\d+\s+){2}(\d{1,3}%)$
I'm using casper.js & backstop.js in Travis CI to run tests with phantom.js. But I would prefer to use slimer.js instead of phantom.js.
Is it possible to do? I tried install it with:
npm install -g slimerjs
and with:
env:
- SLIMERJSLAUNCHER=$(which firefox) DISPLAY=:99.0 PATH=$TRAVIS_BUILD_DIR/slimerjs:$PATH
addons:
firefox: "42.0"
before_script:
- "sh -e /etc/init.d/xvfb start"
- "echo 'Installing Slimer'"
- "wget http://download.slimerjs.org/v0.9/0.9.6/slimerjs-0.9.6.zip"
- "unzip slimerjs-0.9.6.zip"
- "mv slimerjs-0.9.6 ./slimerjs"
both not working and I get an error:
Gecko error: it seems /usr/local/bin/firefox is not compatible with SlimerJS. See Gecko version compatibility.
I tried different versions of FF specified in application.ini but without any success.
I checked the project: https://github.com/JulianBirch/cljs-ajax (referred in: https://github.com/travis-ci/travis-ci/issues/1290) and went over the git history in the .travis.yml file and it seems there is a way to have a green build with slimer 0.9.6.
Copy/pasting the .travis.yml of the last build with slimerjs included (build status is green: https://travis-ci.org/JulianBirch/cljs-ajax/jobs/104345408):
language: clojure
lein: lein2
env:
- SLIMERJSLAUNCHER=$(which firefox) DISPLAY=:99.0 PATH=$TRAVIS_BUILD_DIR/slimerjs:$PATH
addons:
firefox: "24.0"
before_script:
- "sh -e /etc/init.d/xvfb start"
- "curl https://slimerjs.org/slimerjs-pubkey.gpg | gpg --import"
- "wget http://download.slimerjs.org/releases/0.9.6/slimerjs-0.9.6-linux-x86_64.tar.bz2"
- "wget http://download.slimerjs.org/releases/0.9.6/slimerjs-0.9.6-linux-x86_64.tar.bz2.asc"
- "gpg --verify-files *.asc"
- "tar jxfv slimerjs-0.9.6-linux-x86_64.tar.bz2"
- "mv slimerjs-0.9.6 ./slimerjs"
- "yes | sudo lein2 upgrade 2.5.2"
sudo: required
Well, it might also depend on the VM type you use, but it should be a good starting point.
Anyway, I feel like heading the same direction, so it would be cool if you could share the config working for you.