Our CI pipeline publishes wheels for branches that have a version of
<base version>-dev<timestamp>+<branch name>.p<pipeline id>
So if I am working on cool-stuff on in the xyzzy branch, it might upload a wheel for version 1.2.3.dev202211221111+xyzzy.p1234
Somebody else, working in the foobar branch, might cause 1.2.3.dev202211221115+foobar.p1235 to be created.
How can I get pip to install the latest version from the xyzzy branch? I tried pip install cool-stuff>1.2.3.dev*+xyzzy but it complained that it could not find a matching version (even though the available versions that it listed included a +xyzzy tag.)
pip install cool-stuff==1.2.3.dev202211221111+xyzzy.p1234 did work, but I would prefer not have have to update the time stamp and pipeline number each time. I am hoping to put cool-stuff >= <magic> in my config file and just run pip install -e . whenever I need new dependencies.
What format do I need to use here?
As far as I know, it is not possible. I can not think of a workable solution that would be based on the version string only. The "local" part of the version string (in other words the part after the plus sign +) can not be used to differentiate between two releases as you intend to.
If I were in your situation, I think I would investigate a solution where the CI/CD pipelines generate distributions with a name customized according to the git branch. For example in your case the pipelines should generate wheels for Library-foobar or Library-xyzzy, depending on what branch is currently being worked on (while still keeping the same top-level import names of course). This assumes that you can customize your pipelines and processes deeply enough to support such a workflow.
Related
I'm running a project that uses pip and a requirements.txt file to install and keep track of some dependencies. I want to write some sh scripts to run, build and test the application. For starters I would like a way to check if the current folder is in a pyenv and, if not, create one to enclose the application and not mess around other people's dependencies. Also, I would like an opinion of the best way to keep track of this kind of dependencies, if the requirements.txt is a good approach and if there's a way to keep track of installed versions just like happens with node packages.
Use Pipenv. It's a better way of tracking your depencies than requirements.txt and it uses Pyenv to automatically install your project's required Python version.
From the website:
The problems that Pipenv seeks to solve are multi-faceted:
You no longer need to use pip and virtualenv separately. They work together.
Managing a requirements.txt file can be problematic, so Pipenv uses Pipfile and Pipfile.lock to separate abstract dependency
declarations from the last tested combination.
Hashes are used everywhere, always. Security. Automatically expose security vulnerabilities.
Strongly encourage the use of the latest versions of dependencies to minimize security risks arising from outdated components.
Give you insight into your dependency graph (e.g. $ pipenv graph).
Streamline development workflow by loading .env files.
[...]
Pipenv Features
Enables truly deterministic builds, while easily specifying only what you want.
Generates and checks file hashes for locked dependencies.
Automatically install required Pythons, if pyenv is available.
Automatically finds your project home, recursively, by looking for a Pipfile.
Automatically generates a Pipfile, if one doesn’t exist.
Automatically creates a virtualenv in a standard location.
Automatically adds/removes packages to a Pipfile when they are installed or uninstalled.
Automatically loads .env files, if they exist.
I am trying to install the libguestfs package for golang but couldn't find the way to do so. I went through this but it didn't work for me and I believe it's empty. I see only test files here
Following is the content of the README
$GOPATH
-------
http://golang.org/doc/code.html
Note that $GOPATH is set (by the ./run script) to the
libguestfs/golang directory so that "installs" happen in the local
build tree.
From this I could figure out that I need to build the package but again for that, I couldn't find some help or proper document.
Is there any way to do the installation? I am using RHEL 7.7
Libguestfs (the C library) is packaged already in RHEL 7, so I would suggest first of all installing that using yum install libguestfs. That's the version that Red Hat supports.
We don't ship the golang bindings, not even in RHEL 8. Furthermore the bindings in all languages are generated from a mechanical description of the API and thus not stored directly inside our git repository. That's why you could only find test files in git.
However we do distribute the generated files in the tarballs, so an easy way is to grab the latest tarball from https://download.libguestfs.org/1.42-stable/ and look in the golang/src/libguestfs.org/guestfs subdirectory of the tarball.
The harder way is to generate them from git. These steps worked for me on RHEL 7.7, YMMV:
sudo yum-builddep libguestfs
git clone https://github.com/libguestfs/libguestfs
cd libguestfs
git submodule init
git submodule update
./bootstrap
./autogen.sh SUPERMIN=/usr/bin/supermin5
make
and again look in the golang/ subdirectory.
I'm reasonably sure that golang bindings from one version should work with another version of the C library. We did test this at one point in time, so if it's no longer true then it may have regressed as these things sometimes do.
First I have read this answer: Vendoring in Go 1.6, then I use it as my example.
My gopath is GOPATH="/Users/thinkerou/xyz/", and the follow like:
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou$ pwd
/Users/baidu/xyz/src/ou
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou$ ls
main.go vendor
Now, I use go get, then becomes this:
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou$ ls
main.go vendor
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou$ cd vendor/
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou/vendor$ ls
vendor.json
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/ou/vendor$ cd ../..
thinkerou#MacBook-Pro-thinkerou:~/xyz/src$ ls
github.com ou
thinkerou#MacBook-Pro-thinkerou:~/xyz/src$ cd github.com/
thinkerou#MacBook-Pro-thinkerou:~/xyz/src/github.com$ ls
zenazn
vendor.json is this:
{
"comment": "",
"package": [
{
"path": "github.com/zenazn/goji"
}
]
}
then, I should use what commands? why have no use vendor? My go version is 1.6.2.
With Go1.6, vendoring is built in as you read. What does this mean? Only one thing to keep in mind:
When using the go tools such as go build or go run, they first check to see if the dependencies are located in ./vendor/. If so, use it. If not, revert to the $GOPATH/src/ directory.
The actual "lookup paths" in Go 1.6 are, in order:
./vendor/github.com/zenazn/goji
$GOPATH/src/github.com/zenazn/goji
$GOROOT/src/github.com/zenazn/goji
With that said, go get will continue to install into you $GOPATH/src; and, go install will install into $GOPATH/bin for binaries or $GOPATH/pkg for package caching.
So, how do I use ./vendor?!?!
Hehe, armed with the knowledge above, it's pretty simple:
mkdir -p $GOPATH/src/ou/vendor/github.com/zenazn/goji
cp -r $GOPATH/src/github.com/zenazn/goji/ $GOPATH/src/ou/vendor/github.com/zenazn/goji
In short, to use vendoring, you copy the files using the same github.com/zenazn/goji full path, into your vendor director.
Now, the go build/install/run tooling will see and use your vendor folder.
An easier way instead of copying everything manually
Instead of finding and copying all 25+ vendor items, managing their versions, updating other projects etc... It would be better to use a dependency management tool. There are many out there and a little googling will point to you several.
Let me mention two that works with the vendor folder and doesn't fight you:
godep
govendor
In short, these tools will inspect your ou code, find the remote dependencies, and copy them from your $GOPATH/src to your $GOPATH/src/ou/vendor directory (actually, whatever current directory you are in when you run them).
For example, say you have all of your dependencies installed and working normally in your $GOPATH/src/ou/ project using the normal GOPATH/src/github installation of your dependencies. Your project runs and your tests validate everything is working with the exact version of the repos you have. With Godep as an example, you'd run this from your project root folder $GOPATH/src/ou/:
godep save ./...
This would copy all dependencies your project uses into your ./vendor folder.
Godep is by far and large the most popular. They have their own Slack channel on the Gopher Slack group. And, it's the one I use on my teams.
Govendor is another alternative I read has a nice sync feature. I haven't used it though.
Over Usage of Dependency Management Tool
This is purely opinion, and I'm sure haters will downvote... But as I need to finish my blog post on the subject, let me mention here that most people worry too much about depdency management in Go.
Yes, there is a need to lock in a repo to a version you depend on so you can ensure your system builds in production. Yes there is a need to ensure no breaking changes to a way a dependency is interrupting something.
Use dependency management for those, absolutely.
But, there is overuse of simple projects that lock in huge amounts of dependencies when in reality...
You may only need to lock in only 1 dependencies; otherwise, you want the latest version of MySQL drivers and test assertion frameworks for bug fixes.
This is where using the ./vendor/ folder apart from dependency managrment tools can really shine: you'd only need to copy that repo that need you lock in.
You selectively pick the one misbehaving repo and put it into your ./vendor/ folder. By doing this, you are telling your consumers:
Hey, this one repo needs to be held back at this revision. All others are fine and use the latest of those and update often with go get -u ./...; but, this one failed with newer versions so don't upgrade this one repo.
But if blanketly saving all your dependencies with a dependency management tool, you are basically telling your consumers:
There may or may not be a problem with one or more repos out of the 20 in the vendor folder. You may or may not be able to update them. You may or may not be able to get the latest MySQL driver. We simply don't know which may or may not be causing problems and just locked in something that worked at the time that I ran godep save. So yeah, upgrade at your own risk.
Personally, I have ran into this several times. A dependency was updated with a breaking change, and we have dozens of repos dependent on it. Vendoring just that one repo in /vendor allows us to use that one version of dependency, while go get ./... continues to run normally for all other repos to get the latest. We run with the latest bug fixes in PSQL and MySQL and others (there are constant fixes for these!) and so on.
How can I put my Go binary into a Debian package? Since Go is statically linked, I just have a single executable--I don't need a lot of complicated project metadata information. Is there a simple way to package the executable and resource files without going through the trauma of debuild?
I've looked all over for existing questions; however, all of my research turns up questions/answers about a .deb file containing the golang development environment (i.e., what you would get if you do sudo apt-get install golang-go).
Well. I think the only "trauma" of debuild is that it runs lintian after building the package, and it's lintian who tries to spot problems with your package.
So there are two ways to combat the situation:
Do not use debuild: this tool merely calls dpkg-buildpackage which really does the necessary powerlifting. The usual call to build a binary package is dpkg-buildpackage -us -uc -b. You still might call debuild for other purposes, like debuild clean for instance.
Add the so-called "lintian override" which can be used to make lintian turn a blind eye to selected problems with your package which, you insist, are not problems.
Both approaches imply that you do not attempt to build your application by the packaging tools but rather treat it as a blob which is just wrapped to a package. This would require slightly abstraining from the normal way debian/rules work (to not attempt to build anything).
Another solution which might be possible (and is really way more Debian-ish) is to try to use gcc-go (plus gold for linking): since it's a GCC front-end, this tool produces a dynamically-linked application (which links against libgo or something like this). I, personally, have no experience with it yet, and would only consider using it if you intend to try to push your package into the Debian proper.
Regarding the general question of packaging Go programs for Debian, you might find the following resources useful:
This thread started on go-nuts by one of Go for Debian packagers.
In particular, the first post in that thread links to this discussion on debian-devel.
The second thread on debian-devel regarding that same problem (it's a logical continuation of the former thread).
Update on 2015-10-15.
(Since this post appears to still be searched and found and studied by people I've decided to update it to better reflec the current state of affairs.)
Since then the situation with packaging Go apps and packages got improved dramatically, and it's possible to build a Debian package using "classic" Go (the so-called gc suite originating from Google) rather than gcc-go.
And there exist a good infrastructure for packages as well.
The key tool to use when debianizing a Go program now is dh-golang described here.
I've just been looking into this myself, and I'm basically there.
Synopsis
By 'borrowing' from the 'package' branch from one of Canonical's existing Go projects, you can build your package with dpkg-buildpackage.
install dependencies and grab a 'package' branch from another repo.
# I think this list of packages is enough. May need dpkg-dev aswell.
sudo apt-get install bzr debhelper build-essential golang-go
bzr branch lp:~niemeyer/cobzr/package mypackage-build
cd mypackage-build
Edit the metadata.
edit debian/control file (name, version, source). You may need to change the golang-stable dependency to golang-go.
The debian/control file is the manifest. Note the 'build dependencies' (Build-Depends: debhelper (>= 7.0.50~), golang-stable) and the 3 architectures. Using Ubuntu (without the gophers ppa), I had to change golang-stable to golang-go.
edit debian/rules file (put your package name in place of cobzr).
The debian/rules file is basically a 'make' file, and it shows how the package is built. In this case they are relying heavily on debhelper. Here they set up GOPATH, and invoke 'go install'.
Here's the magic 'go install' line:
cd $(GOPATH)/src && find * -name '*.go' -exec dirname {} \; | xargs -n1 go install
Also update the copyright file, readme, licence, etc.
Put your source inside the src folder. e.g.
git clone https://github.com/yourgithubusername/yourpackagename src/github.com/yourgithubusername/yourpackagename
or e.g.2
cp .../yourpackage/ src/
build the package
# -us -uc skips package signing.
dpkg-buildpackage -us -uc
This should produce a binary .deb file for your architecture, plus the 'source deb' (.tgz) and the source deb description file (.dsc).
More details
So, I realised that Canonical (the Ubuntu people) are using Go, and building .deb packages for some of their Go projects. Ubuntu is based on Debian, so for the most part the same approach should apply to both distributions (dependency names may vary slightly).
You'll find a few Go-based packages in Ubuntu's Launchpad repositories. So far I've found cobzr (git-style branching for bzr) and juju-core (a devops project, being ported from Python).
Both of these projects have both a 'trunk' and a 'package' branch, and you can see the debian/ folder inside the package branch. The 2 most important files here are debian/control and debian/rules - I have linked to 'browse source'.
Finally
Something I haven't covered is cross-compiling your package (to the other 2 architectures of the 3, 386/arm/amd64). Cross-compiling isn't too tricky in go (you need to build the toolchain for each target platform, and then set some ENV vars during 'go build'), and I've been working on a cross-compiler utility myself. Eventually I'll hopefully add .deb support into my utility, but first I need to crystallize this task.
Good luck. If you make any progress then please update my answer or add a comment. Thanks
Building deb or rpm packages from Go Applications is also very easy with fpm.
Grab it from rubygems:
gem install fpm
After building you binary, e.g. foobar, you can package it like this:
fpm -s dir -t deb -n foobar -v 0.0.1 foobar=/usr/bin/
fpm supports all sorts of advanced packaging options.
There is an official Debian policy document describing the packaging procedure for Go: https://go-team.pages.debian.net/packaging.html
For libraries: Use dh-make-golang to create a package skeleton. Name your package with a name derived from import path, with a -dev suffix, e.g. golang-github-lib-pq-dev. Specify the dependencies ont Depends: line. (These are source dependencies for building, not binary dependencies for running, since Go statically links all source.)
Installing the library package will install its source code to /usr/share/golang/src (possibly, the compiled libraries could go into .../pkg). Building depending Go packages will use the artifacts from those system-wide locations.
For executables: Use dh-golang to create the package. Specify dependencies in Build-Depends: line (see above regarding packaging the dependencies).
I recently discovered https://packager.io/ - I'm quite happy with what they're doing. Maybe open up one of the packages to see what they're doing?
I've installed the Go release version as root.
Go1 removed all exp/ code.
Is there smart method to get exp/* back after Go1?
(I mean how to install in my local GOPATH?)
[My Solution]
# pull from go repository to $HOME/repo/go
cd $HOME/repo
hg clone https://go.googlecode.com/hg/go
# make symbolic link to your GOPATH(eg. $HOME/go)
cd $HOME/go/src
ln -s $HOME/repo/go/src/pkg/exp .
The exp/html library was incomplete which is why it was removed for Go1.
However if you really want to use it then
go get code.google.com/p/go/src/pkg/exp/html
may install it back for you. If you want a slightly more complete html parser then you might checkout http://code.google.com/p/go-html-transform/ as well it has an html5 parser as well as a css selector based scraping and transformation library.
EDIT: Apparently trying to go get the package that way doesn't really work. It appears the only way to install this is to checkout the go source code and then install from source. This is actually a really quick an painless process if you want to go that route.
Building from source is the way to do this. When you do the hg update step though, note that since the exp tree is not tagged go1, that hg update release won't get it for you. Instead hg update weekly will get it, and is probably what you want.
Edit: Weekly releases were discontinued after Go 1, so hg update weekly will access increasingly stale code. A better strategy is hg update tip, then copy the exp directory or directories of interest somewhere and recompile it with whatever Go version you are using, Go 1.0.1, for example.
Note: with go 1.4 (Q4, 2014), the url for that exp package will change (again):
code.google.com/p/go.exp => golang.org/x/exp
That means now:
go get golang.org/x/exp
See "Go 1.4 subrepo renaming".
Regarding the html package, it is in net/html, so this will become (as commented by andybalholm):
go get golang.org/x/net/html
The exp packages have been moved to different repositories now, to make them easier to install. Now you can install the former exp/html with go get "golang.org/x/net/html".
This answer is outdated.
This is covered in the golang wiki:
https://code.google.com/p/go-wiki/wiki/InstallingExp
% cd $GOPATH/src
% hg clone https://code.google.com/p/go go-exp
requesting all changes
adding changesets
adding manifests
adding file changes
added 13323 changesets with 50185 changes to 7251 files (+5 heads)
updating to branch default
3464 files updated, 0 files merged, 0 files removed, 0 files unresolved
% mv go-exp/src/pkg/exp .
% rm -rf go-exp
% go install exp/...
Then, to use it:
import "exp/proxy"
I tried this a few months ago and it worked pretty well. Also, when I ran go install ... I limited it to only the package I was interested in: go install exp/html (if I recall, correctly).