I'm having an issue with using custom substitutions in my cloudbuild.yaml.
substitutions:
_CUSTOM_SUBSTITUTION: this-is-a-path
availableSecrets:
secretManager:
- versionName: projects/$_CUSTOM_SUBSTITUTION/secrets/client_id/versions/1
env: CLIENT_ID
- versionName: projects/$_CUSTOM_SUBSTITUTION/secrets/client_secret/versions/1
env: CLIENT_SECRET
From what I can tell from trial and error, using something like $PROJECT_ID in the place of $_CUSTOM_SUBSTITUTION will run the build, but if I use a custom substitution like above, the trigger does not run a build at all when a commit is pushed.
I've also tested with various other base substitutions, like $BRANCH_NAME to the same effect. I'm getting the feeling that it's just not possible to do this in cloudbuild at the moment?
It ended up being a combination of need curly braces ${_CUSTOM_SUBSTITUTION} and some syntax fixing in the cloudbuild.yaml. I didn't have enough experience with cloudbuild to find that.
The offending part was something this:
AUTH_TOKEN=$$(cat /workspace/token.txt). Originally I had just 1 $ there, which was also working code pulled from another project.
For anyone running into this in the future, using gloud builds submit can let you run it directly for troubleshooting.
I'm trying to optimize our poetry build times by storing a tar.gz of the venv on the end of the build in an azure storage blob (an S3) with key generated by doing a md5sum of the poetry.lock and the pyproject.toml. (this works fine)
So, in the beginning of another build, I would hash those files and try to find if that blob exists in storage. If yes, download it and extract its contents to the .venv/ dir of the project.
This worked fine at the beginning but after a while I started getting builds that were throwing this error when trying to run a command through poetry: (any command)
+ poetry run reorder-python-imports --diff-only <SOME_FILES>
FileNotFoundError
[Errno 2] No such file or directory
at /usr/local/lib/python3.8/os.py:591 in _execvpe
587│ argrest = (args,)
588│ env = environ
589│
590│ if path.dirname(file):
→ 591│ exec_func(file, *argrest)
592│ return
593│ saved_exc = None
594│ path_list = get_exec_path(env)
595│ if name != 'nt':
I have confirmed that there is no .venv folder already in the project dir and that the contents are exactly the same as if it were going to install it from the network.
If I just run poetry install on top of the cached venv, it says it has no more dependencies to install, but it throws the error above.
If I delete the venv and install everything again, commands work fine.
I have no more ideas on how to debug and solve this issue. Help would be very appreciated! :)
Ok, I think I've understood that virtualenvs have some hardcoded paths that are not easy to move around.
For what it says in this post Can I move a virtualenv? we should not move venvs as a good practice.
And there it goes my way of speeding up builds...
Using CircleCI - version: 2.1 - for continuous deployment where caching installed dependencies. Based on save_cache documentation:
Generates and stores a cache of a file or directory of files such as dependencies or source code in our object storage. Later jobs can restore this cache.
Current scenario:
See the simplified caching step below in .circleci/config.yml file:
steps:
- node/with-cache:
steps:
- checkout
- run: npm install
- save_cache:
key: dependencies
paths: node_modules
The problem is coming once adding new package to the project thus package.json file is changing. In the same time CircleCI shows the message for Saving Cache step:
Skipping cache generation, cache already exists for key: dependenciesFound one created at 2020-05-23 19:29:29 +0000 UTC
Then once restoring the cache obviously does not find the newly added package in the build step:
./src/index.tsxCannot find module: 'package-name'. Make sure this package is installed.
Questions:
Is there any way to check package.json changes in the pipeline? Ideally I would install the dependencies only in those cases, so the cache can be purged and updated.
Maybe I did not see something in the documentation. Any help is appreciated, thank you!
The problem is the cache key you used is "dependencies", a plain string. This key never changes, so you will always use the same exact cache.
You need to use a cache key that changes, preferably based on package.lock. Please read the section of cache keys in the CircleCI Docs for more information: https://circleci.com/docs/2.0/caching/#using-keys-and-templates
First of all: I am somewhat new to cloud build. Compared to previously used methods, I find it a wrenching, unripe and fairly annoying framework. Endless time is spend getting builders to work that supposedly work out of the box (like the helm builder for example), and it's limitations are astonishing and frustrating. Perhaps the following problem is a good example:
I would like to build and push a docker image. According to the documentation, the images to be pushed to the docker repository at the end (I'm using GCR for this) reside in the following configuration section in my cloudbuild.yaml file:
images:
- 'eu.gcr.io/$PROJECT_ID/my-project:${_TAG}'
- 'eu.gcr.io/$PROJECT_ID/my-project:latest'
I can set the _TAG substitution manually by using the section:
substitutions:
_TAG: x.y.z
but that means I have to manually fix the version number in this file every time. Worse still: if I branch out, I need to maintain the version number all the time. I have a python project in this case and it uses setuptools, the version is naturally contained in the setup.py file and I can parse it out with no problem. Attempts to parse the number into a specific file and use $(cat VERSION) in the images section fail, because the system claims it can't substitute the $(cat VERSION) part. So how can I overwrite the _TAG variable inside of another build step such, that it appears correct in the 'images' section?
If you are using triggered builds from Cloud Source Repositories, GitHub, or Bitbucket you can tag your commit and use the $TAG_NAME default substitution variable.
images:
- 'eu.gcr.io/$PROJECT_ID/my-project:$TAG_NAME'
- 'eu.gcr.io/$PROJECT_ID/my-project:latest'
On the other hand if you are using the Cloud SDK to submit the Cloud Build build you can provide values with the --substitutions argument:
gcloud builds submit [SOURCE] --config config.yaml --substitutions _TAG=x.y.z
Also I believe you would find this GitOps-style continuous delivery with Cloud Build tutorial very helpful. It explains how to create a continuous integration and delivery (CI/CD) pipeline on Google Cloud Platform using Cloud Build.
You can tag your image with several tags using cloudbuild.yaml and define the Docker build step with:
steps:
- name: gcr.io/cloud-builders/docker
args:
- build
- '--no-cache'
- '-t'
- '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
- '-t'
- '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:latest'
- .
- '-f'
- Dockerfile.prod
id: Build
And resulting images with:
images:
- '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
- '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:latest'
Suppose you have a repository at github.com/someone/repo and you fork it to github.com/you/repo. You want to use your fork instead of the main repo, so you do a
go get github.com/you/repo
Now all the import paths in this repo will be "broken", meaning, if there are multiple packages in the repository that reference each other via absolute URLs, they will reference the source, not the fork.
Is there a better way as cloning it manually into the right path?
git clone git#github.com:you/repo.git $GOPATH/src/github.com/someone/repo
If you are using go modules. You could use replace directive
The replace directive allows you to supply another import path that might
be another module located in VCS (GitHub or elsewhere), or on your
local filesystem with a relative or absolute file path. The new import
path from the replace directive is used without needing to update the
import paths in the actual source code.
So you could do below in your go.mod file
module some-project
go 1.12
require (
github.com/someone/repo v1.20.0
)
replace github.com/someone/repo => github.com/you/repo v3.2.1
where v3.2.1 is tag on your repo. Also can be done through CLI
go mod edit -replace="github.com/someone/repo#v0.0.0=github.com/you/repo#v1.1.1"
To handle pull requests
fork a repository github.com/someone/repo to github.com/you/repo
download original code: go get github.com/someone/repo
be there: cd "$(go env GOPATH)/src"/github.com/someone/repo
enable uploading to your fork: git remote add myfork https://github.com/you/repo.git
upload your changes to your repo: git push myfork
http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html
To use a package in your project
https://github.com/golang/go/wiki/PackageManagementTools
One way to solve it is that suggested by Ivan Rave and http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html -- the way of forking.
Another one is to workaround the golang behavior. When you go get, golang lays out your directories under same name as in the repository URI, and this is where the trouble begins.
If, instead, you issue your own git clone, you can clone your repository onto your filesystem on a path named after the original repository.
Assuming original repository is in github.com/awsome-org/tool and you fork it onto github.com/awesome-you/tool, you can:
cd $GOPATH
mkdir -p {src,bin,pkg}
mkdir -p src/github.com/awesome-org/
cd src/github.com/awesome-org/
git clone git#github.com:awesome-you/tool.git # OR: git clone https://github.com/awesome-you/tool.git
cd tool/
go get ./...
golang is perfectly happy to continue with this repository and doesn't actually care some upper directory has the name awesome-org while the git remote is awesome-you. All import for awesome-org are resovled via the directory you have just created, which is your local working set.
In more length, please see my blog post: Forking Golang repositories on GitHub and managing the import path
edit: fixed directory path
If your fork is only temporary (ie you intend that it be merged) then just do your development in situ, eg in $GOPATH/src/launchpad.net/goamz.
You then use the features of the version control system (eg git remote) to make the upstream repository your repository rather than the original one.
It makes it harder for other people to use your repository with go get but much easier for it to be integrated upstream.
In fact I have a repository for goamz at lp:~nick-craig-wood/goamz/goamz which I develop for in exactly that way. Maybe the author will merge it one day!
Here's a way to that works for everyone:
Use github to fork to "my/repo" (just an example):
go get github.com/my/repo
cd ~/go/src/github.com/my/repo
git branch enhancement
rm -rf .
go get github.com/golang/tools/cmd/gomvpkg/…
gomvpkg <<oldrepo>> ~/go/src/github.com/my/repo
git commit
Repeat each time when you make the code better:
git commit
git checkout enhancement
git cherry-pick <<commit_id>>
git checkout master
Why? This lets you have your repo that any go get works with. It also lets you maintain & enhance a branch that's good for a pull request. It doesn't bloat git with "vendor", it preserves history, and build tools can make sense of it.
Instead of cloning to a specific location, you can clone wherever you want.
Then, you can run a command like this, to have Go refer to the local version:
go mod edit -replace github.com/owner/repo=../repo
https://golang.org/cmd/go#hdr-Module_maintenance
The answer to this is that if you fork a repo with multiple packages you will need to rename all the relevant import paths. This is largely a good thing since you've forked all of those packages and the import paths should reflect this.
Use vendoring and submodules together
Fork the lib on github (go-mssqldb in this case)
Add a submodule which clones your fork into your vendor folder but has the path of the upstream repo
Update your import statements in your source code to point to the vendor folder, (not including the vendor/ prefix). E.g. vendor/bob/lib => import "bob/lib"
E.g.
cd ~/go/src/github.com/myproj
mygithubuser=timabell
upstreamgithubuser=denisenkom
librepo=go-mssqldb
git submodule add "git#github.com:$mygithubuser/$librepo" "vendor/$upstreamgithubuser/$librepo"
Why
This solves all the problems I've heard about and come across while trying to figure this out myself.
Internal package refs in the lib now work because the path is unchanged from upstream
A fresh checkout of your project works because the submodule system gets it from your fork at the right commit but in the upstream folder path
You don't have to know to manually hack the paths or mess with the go tooling.
More info
https://git-scm.com/book/en/v2/Git-Tools-Submodules
How do I fix the error message "use of an internal package not allowed" when go getting a golang package?
https://github.com/denisenkom/go-mssqldb/issues/406
https://github.com/golang/go/wiki/PackageManagementTools#go15vendorexperiment
The modern answer (go 1.15 and higher, at least).
go mod init github.com/theirs/repo
Make an explicit init arg that is the ORIGINAL package names. If you don't include the repo name, it will assume the one in gopath. But when you use go modules, they no longer care where they are on disk, or where git actually pulls dependencies from.
To automate this process, I wrote a small script. You can find more details on my blog to add a command like "gofork" to your bash.
function gofork() {
if [ $# -ne 2 ] || [ -z "$1" ] || [ -z "$2" ]; then
echo 'Usage: gofork yourFork originalModule'
echo 'Example: gofork github.com/YourName/go-contrib github.com/heirko/go-contrib'
return
fi
echo "Go get fork $1 and replace $2 in GOPATH: $GOPATH"
go get $1
go get $2
currentDir=$PWD
cd $GOPATH/src/$1
remote1=$(git config --get remote.origin.url)
cd $GOPATH/src/$2
remote2=$(git config --get remote.origin.url)
cd $currentDir
rm -rf $GOPATH/src/$2
mv $GOPATH/src/$1 $GOPATH/src/$2
cd $GOPATH/src/$2
git remote add their $remote2
echo Now in $GOPATH/src/$2 origin remote is $remote1
echo And in $GOPATH/src/$2 their remote is $remote2
cd $currentDir
}
export -f gofork
You can use command go get -f to get you a forked repo
in your Gopkg.toml file add these block below
[[constraint]]
name = "github.com/globalsign/mgo"
branch = "master"
source = "github.com/myfork/project2"
So it will use the forked project2 in place of github.com/globalsign/mgo