How to freeze micro version with dependencies? - go

I want to build a docker image with a fixed version of micro and go dependencies. I plan to do it with dep:
git checkout git#github.com:micro/micro.git
dep ensure
git add Gopkg.toml
git add Gopkg.lock
# Build micro
CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o micro ./main.go
# Build docker image
...
So, my question is does it the best solution to build consistent micro docker image?

An example of a Dockerfile can be:
FROM golang:1.9-alpine3.6 as builder
# Install package manager
RUN apk add --no-cache --virtual .go-dependencies git curl \
&& curl https://glide.sh/get | sh
# Copy files from context
WORKDIR /go/src/github.com/foo/bar
COPY . .
# Install project dependencies, test and build
RUN glide install \
&& go test ./... \
&& CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o ./entry ./main.go ./plugins.go
# Build final image with binary
FROM alpine:3.6
RUN apk add --update ca-certificates && \
rm -rf /var/cache/apk/* /tmp/*
WORKDIR /
COPY --from=builder /go/src/github.com/foo/bar/entry .
ENTRYPOINT [ "/entry" ]
And the glide.yaml would look like this:
package: .
import:
- package: github.com/micro/go-micro
version: ^0.3.0
subpackages:
- client
- server
- package: github.com/micro/go-plugins
version: ^0.6.1
subpackages:
- wrapper/trace/opentracing
- broker/nats
- transport/nats
- package: github.com/opentracing/opentracing-go
version: ^1
- package: github.com/openzipkin/zipkin-go-opentracing
version: ^0.3
testImport:
- package: github.com/golang/mock
subpackages:
- gomock
- package: github.com/smartystreets/goconvey
subpackages:
- convey

In my case, dep looks great and fast enough, moreover, it's official dependency manager in go so I think it's a right choice.

Related

Is it possible to install gopkg.in packages in Docker?

I am trying to run golang application which use goracle library with such Dockerfile:
FROM golang:1.12
RUN go get github.com/gorilla/mux && \
go get github.com/gorilla/handlers && \
go get github.com/lib/pq && \
go get github.com/joho/godotenv && \
go get github.com/jinzhu/gorm && \
go get gopkg.in/goracle.v2
ADD ./ /go/src/application
WORKDIR /go/src/application
RUN go build -o /bin application
ENV PORT=8000
CMD ["/bin"]
Unfortunatly it raise error when I try to create image:
package gopkg.in/goracle.v2: unrecognized import path "gopkg.in/goracle.v2" (https fetch: Get https://gopkg.in/goracle.v2?go-get=1: proxyconnect tcp: tls: first record does not look like a TLS handshake)
The command '/bin/sh -c go get github.com/gorilla/mux && go get github.com/gorilla/handlers && go get github.com/lib/pq && go get github.com/joho/godotenv && go get github.com/jinzhu/gorm && go get gopkg.in/goracle.v2' returned a non-zero code: 1
Why I can't install goracle library in Docker? How to fix this problem?
In my case the CentOS server where was located Docker has proxy. For thats why I couldn't download the gopkg.in/goracle.v2 package.
SOLUTION:
1) Create vender folder inside your project.
2) Remove source code of gopkg.in/goracle.v2 package which you has in go/src folder to vender folder.
3) Run you Dockerfile.
In my case this instruction removed problem with importing of gopkg.in/goracle.v2
package.
I hope this post will helpful for somebody!

AWS CodeBuild Unable to "go get" a package

I have an AWS CodeBuild job, defined with the following buildspec file (apologies if indentation isn't reproduced correctly):
version: 0.2
env:
variables:
PACKAGE: "github.com/user/package"
phases:
install:
commands:
- mkdir -p ${GOPATH}/src/${PACKAGE}
- cp -a ${CODEBUILD_SRC_DIR}/. ${GOPATH}/src/${PACKAGE}
- cd ${GOPATH}/src/${PACKAGE} && go get ./...
build:
commands:
- cd ${GOPATH}/src/${PACKAGE} && go build -o ${CODEBUILD_SRC_DIR}/application
post_build:
commands:
- aws cloudformation package --template-file template.yml --output-template-file serverless.yml --s3-bucket some-bucket
artifacts:
files:
- serverless.yml
This fails in the install phase.
The go application I'm trying to build has several sub-packages and external dependencies. When running "go get ./..." I'm getting
cannot find package github.com/user/package/sub-package in any of:
/usr/local/go/src/github.com/user/package/sub-package(from $GOROOT) /go/src/github.com/user/package/sub-package(from $GOPATH) /codebuild/output/src708017258/src/github.com/user/package/sub-package
When "debugging" (by putting in some echos and listing the contents of the newley created folders) it appears that everything is in the right place and everything should just work.

GitLab CI - Cache not working

I'm currently using GitLab in combination with CI runners to run unit tests of my project, to speed up the process of bootstrapping the tests I'm using the built-in cache functionality, however this doesn't seem to work.
Each time someone commits to master, my runner does a git fetch and proceeds to remove all cached files, which means I have to stare at my screen for around 10 minutes to wait for a test to complete while the runner re-downloads all dependencies (NPM and PIP being the biggest time killers).
Output of the CI runner:
Fetching changes...
Removing bower_modules/jquery/ --+-- Shouldn't happen!
Removing bower_modules/tether/ |
Removing node_modules/ |
Removing vendor/ --'
HEAD is now at 7c513dd Update .gitlab-ci.yml
Currently my .gitlab-ci.yml
image: python:latest
services:
- redis:latest
- node:latest
cache:
key: "$CI_BUILD_REF_NAME"
untracked: true
paths:
- ~/.cache/pip/
- vendor/
- node_modules/
- bower_components/
before_script:
- python -V
# Still gets executed even though node is listed as a service??
- '(which nodejs && which npm) || (apt-get update -q && apt-get -o dir::cache::archives="vendor/apt/" install nodejs npm -yqq)'
- npm install -g bower gulp
# Following statements ignore cache!
- pip install -r requirements.txt
- npm install --only=dev
- bower install --allow-root
- gulp build
test:
variables:
DEBUG: "1"
script:
- python -m unittest myproject
I've tried reading the following articles for help however none of them seem to fix my problem:
http://docs.gitlab.com/ce/ci/yaml/README.html#cache
https://fleschenberg.net/gitlab-pip-cache/
https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/336
Turns out that I was doing some things wrong:
Your script can't cache files outside of your project scope, creating a virtual environment instead and caching that allows you to cache your pip modules.
Most important of all: Your test must succeed in order for it to cache the files.
After using the following config I got a -3 minute time difference:
Currently my configuration looks like follows and works for me.
# Official framework image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python
image: python:latest
# Pick zero or more services to be used on all builds.
# Only needed when using a docker container to run your tests in.
# Check out: http://docs.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-service
services:
- mysql:latest
- redis:latest
cache:
untracked: true
key: "$CI_BUILD_REF_NAME"
paths:
- venv/
- node_modules/
- bower_components/
# This is a basic example for a gem or script which doesn't use
# services such as redis or postgres
before_script:
# Check python installation
- python -V
# Install NodeJS (Gulp & Bower)
# Default repository is outdated, this is the latest version
- 'curl -sL https://deb.nodesource.com/setup_8.x | bash -'
- apt-get install -y nodejs
- npm install -g bower gulp
# Install dependencie
- pip install -U pip setuptools
- pip install virtualenv
test:
# Indicate to the framework that it's being unit tested
variables:
DEBUG: "1"
# Test script
script:
# Set up virtual environment
- virtualenv venv -ppython3
- source venv/bin/activate
- pip install coverage
- pip install -r requirements.txt
# Install NodeJS & Bower + Compile JS
- npm install --only=dev
- bower install --allow-root
- gulp build
# Run all unit tests
- coverage run -m unittest project.tests
- coverage report -m project/**/*.py
Which resulted in the following output:
Fetching changes...
Removing .coverage --+-- Don't worry about this
Removing bower_components/ |
Removing node_modules/ |
Removing venv/ --`
HEAD is now at 24e7618 Fix for issue #16
From https://git.example.com/repo
85f2f9b..42ba753 master -> origin/master
Checking out 42ba7537 as master...
Skipping Git submodules setup
Checking cache for master... --+-- The files are back now :)
Successfully extracted cache --`
...
project/module/script.py 157 9 94% 182, 231-244
---------------------------------------------------------------------------
TOTAL 1084 328 70%
Creating cache master...
Created cache
Uploading artifacts...
venv/: found 9859 matching files
node_modules/: found 7070 matching files
bower_components/: found 982 matching files
Trying to load /builds/repo.tmp/CI_SERVER_TLS_CA_FILE ...
Dialing: tcp git.example.com:443 ...
Uploading artifacts to coordinator... ok id=127 responseStatus=201 Created token=XXXXXX
Job succeeded
For the coverage report, I used the following regular expression:
^TOTAL\s+(?:\d+\s+){2}(\d{1,3}%)$

Golang Dockerfile Failing

I have a Golang web application that I am looking to run in Docker container. I am able to run it fine outside of the container, so I know it works, but when I build it from Dockerfile and run it, it gives me an error.
The Makefile looks like the following
GOCMD = go
GOBUILD = $(GOCMD) build
GOGET = $(GOCMD) get -v
GOCLEAN = $(GOCMD) clean
GOINSTALL = $(GOCMD) install
GOTEST = $(GOCMD) test
.PHONY: all
all: build
test:
$(GOTEST) -v -cover ./...
build:
$(GOGET); $(GOBUILD) -v -o engine
clean:
$(GOCLEAN) -n -i -x
rm -f $(GOPATH)/bin/engine
rm -rf bin/engine
install:
$(GOINSTALL)
And the Dockerfile looks like the following
FROM golang
ADD engine /go/bin/engine
EXPOSE 7777
ENTRYPOINT /go/bin/engine
I am building the image and running it using the following
docker build -t engine .
docker run -d --name engine -p 7777:7777 engine
and its giving me the following error
/go/bin/engine: 1: /go/bin/engine: Syntax error: "(" unexpected
When you build a binary, go build assumes that you are trying to build for your current computer, it chooses values for GOOS and GOARCH (described here) for you.
If you are not building on a linux machine then you will need to cross compile the binary for linux, as this is what the OS inside the docker container will be running. Explanation here
You need something like:
GOOS=linux
build:
$(GOGET); GOOS=$(GOOS) $(GOBUILD) -v -o engine

Travis CI + Go: creating a specific build flow for different OS

I have a Go project that I want to build with Travis-CI and deploy it to a specific provider.
I familiar with Gimme project that will use a cross-compilation to do so.
But because Travis already support linux and osx I only need this feature for Windows build.
The big motivation is, of course, to avoid cross-compilation run time error as there are plenty of it.
My question is how can I create, in the same .travis.yml file, a different build flow:
Native linux/os build (with "os" section).
Windows compilation using Gimmme
The .travis.yml file for the first option will look something like:
language: go
go:
- 1.5.1
branches:
only:
- master
os:
- osx
- linux
before_script:
- go get -d -v ./...
script:
- go build -v ./...
# - go test -v ./...
before_deploy:
- chmod +x ./before_deploy.sh
- ./before_deploy.sh
The .travis.yml file for the second option will look something like:
language: go
go:
- 1.5.1
branches:
only:
- master
env:
- GIMME_OS=windows GIMME_ARCH=amd64
before_script:
- go get -d -v ./...
script:
- go build -v ./...
# - go test -v ./...
before_deploy:
- chmod +x ./before_deploy.sh
- ./before_deploy.sh
Is there a nice clean way to combine these two (with Environment variables or any other crazy idea)?
It might be simple, but matrix environement can not be done for a specific OS ...
Then just select with local environement variable:
language: go
go:
- 1.5.1
branches:
only:
- master
os:
- osx
- linux
install:
- if [ "$TRAVIS_OS_NAME" == "linux" ]; then
export GIMME_OS=windows;
export GIMME_ARCH=amd64;
fi
before_script:
- go get -d -v ./...
script:
- go build -v ./...
after_script:
- go test -v ./...
before_deploy:
- ./before_deploy.sh
An other way:
language: go
go:
- 1.5.1
branches:
only:
- master
matrix:
include:
- os: linux
env: GIMME_OS=windows; GIMME_ARCH=amd64;
- os: osx
before_script:
- go get -d -v ./...
script:
- go build -v ./...
after_script:
- go test -v ./...
before_deploy:
- ./before_deploy.sh
Note: the commande: - chmod +x ./before_deploy.sh can be directly done in your repository and commited on it ...
Note: The environament variable can be accessibe: http://docs.travis-ci.com/user/environment-variables/#Default-Environment-Variables or calling \printenv`

Resources