I have an AWS CodeBuild job, defined with the following buildspec file (apologies if indentation isn't reproduced correctly):
version: 0.2
env:
variables:
PACKAGE: "github.com/user/package"
phases:
install:
commands:
- mkdir -p ${GOPATH}/src/${PACKAGE}
- cp -a ${CODEBUILD_SRC_DIR}/. ${GOPATH}/src/${PACKAGE}
- cd ${GOPATH}/src/${PACKAGE} && go get ./...
build:
commands:
- cd ${GOPATH}/src/${PACKAGE} && go build -o ${CODEBUILD_SRC_DIR}/application
post_build:
commands:
- aws cloudformation package --template-file template.yml --output-template-file serverless.yml --s3-bucket some-bucket
artifacts:
files:
- serverless.yml
This fails in the install phase.
The go application I'm trying to build has several sub-packages and external dependencies. When running "go get ./..." I'm getting
cannot find package github.com/user/package/sub-package in any of:
/usr/local/go/src/github.com/user/package/sub-package(from $GOROOT) /go/src/github.com/user/package/sub-package(from $GOPATH) /codebuild/output/src708017258/src/github.com/user/package/sub-package
When "debugging" (by putting in some echos and listing the contents of the newley created folders) it appears that everything is in the right place and everything should just work.
Related
I have a CodePipline that grabs code out of CodeCommit bundles it up in CodeBuild and then publishes it via CloudFormation.
I want to use the Python package gspread and because it's not part of the standard AWS Linux image I need to install it.
Currently when the code is run I get the error:
[ERROR] Runtime.ImportModuleError: Unable to import module 'index': No module named 'gspread'
Code structure
- buildspec.yml
- template.yml
package/
- gspread/
- gspread-3.6.0.dist-info/
- (37 other python packages)
source/
- index.py
buildspec.yml -- EDITED
version: 0.2
phases:
install:
commands:
# Use Install phase to install packages or any pre-reqs you may need throughout the build (e.g. dev deps, security checks, etc.)
- echo "[Install phase]"
- pip install --upgrade pip
- pip install --upgrade aws-sam-cli
- sam --version
- cd source
- ls
- pip install --target . gspread oauth2client
# consider using pipenv to install everything in the environement and then copy the files installed into the /source folder
- ls
runtime-versions:
python: 3.8
pre_build:
commands:
# Use Pre-Build phase to run tests, install any code deps or any other customization before build
# - echo "[Pre-Build phase]"
build:
commands:
- cd ..
- sam build
post_build:
commands:
# Use Post Build for notifications, git tags and any further customization after build
- echo "[Post-Build phase]"
- export BUCKET=property-tax-invoice-publisher-deployment
- sam package --template-file template.yml --s3-bucket $BUCKET --output-template-file outputtemplate.yml
- echo "SAM packaging completed on `date`"
##################################
# Build Artifacts to be uploaded #
##################################
artifacts:
files:
- outputtemplate.yml
discard-paths: yes
cache:
paths:
# List of path that CodeBuild will upload to S3 Bucket and use in subsequent runs to speed up Builds
- '/root/.cache/pip'
The index.py file has more in it than this. But to show the offending line.
-- index.py --
import os
import boto3
import io
import sys
import csv
import json
import smtplib
import gspread #**<--- Right here**
def lambda_handler(event, context):
print("In lambda_handler")
What I've tried
Creating the /package folder and committing the gspread and other packages
Running "pip install gspread" in the CodeBuild builds: commands:
At the moment, I'm installing it everywhere and seeing what sticks. (nothing is currently sticking)
Version: Python 3.8
I think you may need to do the following steps :
Use virtual env to install the packages locally.
Create requirements.txt to let code build know of the package requirement.
In CodeBuild buildspec.xml , include commands to install virutal env and then supply requirements.txt.
pre_build:
commands:
pip install virtualenv
virtualenv env
. env/bin/activate
pip install -r requirements.txt
Detailed steps here for reference :
https://adrian.tengamnuay.me/programming/2018/07/01/continuous-deployment-with-aws-lambda/
I am trying to experiment a bit with imports and now stuck on the station where simple import of package is failing in CircleCI while it is buildable locally successfully. Could anybody please share what is wrong or what I am doing wrong that is not letting code to be built in CircleCI?
Repo structure:
- project
--- main.go
--- graphic
----- graphics.go
main.go import definition:
package main
import (
"bufio"
"flag"
"fmt"
"io/ioutil"
"log"
"os"
"strings"
graphic "../project/graphic"
)
graphics.go import definition:
package graphic
import (
"fmt"
"io/ioutil"
"log"
"github.com/fogleman/gg"
)
CircleCI config.yml:
version: 2
jobs:
build:
docker:
- image: circleci/golang:1.9
working_directory: /go/src/github.com/<my-account>/project
steps:
- checkout
# specify any bash command here prefixed with `run: `
# - run: go get -v -t -d ./...
- run:
name: run filesystem path configuration
command: |
pwd
pwd -P
echo $GOROOT
echo $GOPATH
export GOBIN=$GOPATH/bin
echo $GOBIN
export PATH=$PATH:$GOBIN
ls -latr $GOROOT
ls -latr $GOPATH
ls -latr $GOBIN
go env
- run:
name: run dependecy managament
command: |
go get -v -u github.com/golang/dep/cmd/dep
go get -v -u github.com/fogleman/gg
go get -v -u github.com/aws/aws-sdk-go/aws
go get -v -u github.com/aws/aws-sdk-go/aws/session
go get -v -u github.com/aws/aws-sdk-go/service/cloudwatch
- run:
name: run documentation
command: |
godoc -v -index -timestamps main
- run:
name: run tests
command: |
go vet -v ./...
- run:
name: run build and deploy
command: |
dep init
mkdir build
dep ensure
go build -v -o build/main.exe main.go
- run:
name: run qa
command: go test -v main.go
- store_artifacts:
path: ./build
Failing error message:
#!/bin/bash -eo pipefail
dep init
mkdir build
dep ensure
go build -v -o build/main.exe main.go
Using ^1.3.0 as constraint for direct dep github.com/fogleman/gg
Locking in v1.3.0 (0403632) for direct dep github.com/fogleman/gg
Locking in master (cff245a) for transitive dep golang.org/x/image
Using ^1.23.3 as constraint for direct dep github.com/aws/aws-sdk-go
Locking in v1.23.3 (fbdf1bd) for direct dep github.com/aws/aws-sdk-go
Locking in master (e2365df) for transitive dep github.com/golang/freetype
Locking in (c2b33e84) for transitive dep github.com/jmespath/go-jmespath
graphic/graphics.go:8:2: cannot find package "_/go/src/github.com/<my-account>/project/vendor/github.com/fogleman/gg" in any of:
/usr/local/go/src/_/go/src/github.com/<my-account>/project/vendor/github.com/fogleman/gg (from $GOROOT)
/go/src/_/go/src/github.com/<my-account>/project/vendor/github.com/fogleman/gg (from $GOPATH)
Exited with code 1
i think for your case will solve if you use package manager, like DEP or other.
I am trying to run golang application which use goracle library with such Dockerfile:
FROM golang:1.12
RUN go get github.com/gorilla/mux && \
go get github.com/gorilla/handlers && \
go get github.com/lib/pq && \
go get github.com/joho/godotenv && \
go get github.com/jinzhu/gorm && \
go get gopkg.in/goracle.v2
ADD ./ /go/src/application
WORKDIR /go/src/application
RUN go build -o /bin application
ENV PORT=8000
CMD ["/bin"]
Unfortunatly it raise error when I try to create image:
package gopkg.in/goracle.v2: unrecognized import path "gopkg.in/goracle.v2" (https fetch: Get https://gopkg.in/goracle.v2?go-get=1: proxyconnect tcp: tls: first record does not look like a TLS handshake)
The command '/bin/sh -c go get github.com/gorilla/mux && go get github.com/gorilla/handlers && go get github.com/lib/pq && go get github.com/joho/godotenv && go get github.com/jinzhu/gorm && go get gopkg.in/goracle.v2' returned a non-zero code: 1
Why I can't install goracle library in Docker? How to fix this problem?
In my case the CentOS server where was located Docker has proxy. For thats why I couldn't download the gopkg.in/goracle.v2 package.
SOLUTION:
1) Create vender folder inside your project.
2) Remove source code of gopkg.in/goracle.v2 package which you has in go/src folder to vender folder.
3) Run you Dockerfile.
In my case this instruction removed problem with importing of gopkg.in/goracle.v2
package.
I hope this post will helpful for somebody!
I want to build a docker image with a fixed version of micro and go dependencies. I plan to do it with dep:
git checkout git#github.com:micro/micro.git
dep ensure
git add Gopkg.toml
git add Gopkg.lock
# Build micro
CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o micro ./main.go
# Build docker image
...
So, my question is does it the best solution to build consistent micro docker image?
An example of a Dockerfile can be:
FROM golang:1.9-alpine3.6 as builder
# Install package manager
RUN apk add --no-cache --virtual .go-dependencies git curl \
&& curl https://glide.sh/get | sh
# Copy files from context
WORKDIR /go/src/github.com/foo/bar
COPY . .
# Install project dependencies, test and build
RUN glide install \
&& go test ./... \
&& CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o ./entry ./main.go ./plugins.go
# Build final image with binary
FROM alpine:3.6
RUN apk add --update ca-certificates && \
rm -rf /var/cache/apk/* /tmp/*
WORKDIR /
COPY --from=builder /go/src/github.com/foo/bar/entry .
ENTRYPOINT [ "/entry" ]
And the glide.yaml would look like this:
package: .
import:
- package: github.com/micro/go-micro
version: ^0.3.0
subpackages:
- client
- server
- package: github.com/micro/go-plugins
version: ^0.6.1
subpackages:
- wrapper/trace/opentracing
- broker/nats
- transport/nats
- package: github.com/opentracing/opentracing-go
version: ^1
- package: github.com/openzipkin/zipkin-go-opentracing
version: ^0.3
testImport:
- package: github.com/golang/mock
subpackages:
- gomock
- package: github.com/smartystreets/goconvey
subpackages:
- convey
In my case, dep looks great and fast enough, moreover, it's official dependency manager in go so I think it's a right choice.
I have a Go project that I want to build with Travis-CI and deploy it to a specific provider.
I familiar with Gimme project that will use a cross-compilation to do so.
But because Travis already support linux and osx I only need this feature for Windows build.
The big motivation is, of course, to avoid cross-compilation run time error as there are plenty of it.
My question is how can I create, in the same .travis.yml file, a different build flow:
Native linux/os build (with "os" section).
Windows compilation using Gimmme
The .travis.yml file for the first option will look something like:
language: go
go:
- 1.5.1
branches:
only:
- master
os:
- osx
- linux
before_script:
- go get -d -v ./...
script:
- go build -v ./...
# - go test -v ./...
before_deploy:
- chmod +x ./before_deploy.sh
- ./before_deploy.sh
The .travis.yml file for the second option will look something like:
language: go
go:
- 1.5.1
branches:
only:
- master
env:
- GIMME_OS=windows GIMME_ARCH=amd64
before_script:
- go get -d -v ./...
script:
- go build -v ./...
# - go test -v ./...
before_deploy:
- chmod +x ./before_deploy.sh
- ./before_deploy.sh
Is there a nice clean way to combine these two (with Environment variables or any other crazy idea)?
It might be simple, but matrix environement can not be done for a specific OS ...
Then just select with local environement variable:
language: go
go:
- 1.5.1
branches:
only:
- master
os:
- osx
- linux
install:
- if [ "$TRAVIS_OS_NAME" == "linux" ]; then
export GIMME_OS=windows;
export GIMME_ARCH=amd64;
fi
before_script:
- go get -d -v ./...
script:
- go build -v ./...
after_script:
- go test -v ./...
before_deploy:
- ./before_deploy.sh
An other way:
language: go
go:
- 1.5.1
branches:
only:
- master
matrix:
include:
- os: linux
env: GIMME_OS=windows; GIMME_ARCH=amd64;
- os: osx
before_script:
- go get -d -v ./...
script:
- go build -v ./...
after_script:
- go test -v ./...
before_deploy:
- ./before_deploy.sh
Note: the commande: - chmod +x ./before_deploy.sh can be directly done in your repository and commited on it ...
Note: The environament variable can be accessibe: http://docs.travis-ci.com/user/environment-variables/#Default-Environment-Variables or calling \printenv`