Previously, when defining how to build a Python package, you can include folders using a line in Manifest.in:
recursive-include my_package/assets *
However, I would like to poetry and the associated pyproject.toml, instead of Manifest.in.
Can pyproject.toml replace Manifest.in? Alternatively, how can poetry be used with Manifest.in?
Yes, poetry aims to completely obsolete the use of Manifest.in files. The assumption is that every file that is part of your package should also be part of the distributable. So, as long as you follow a more or less standard project structure, Things Should Just Work™.
And if your package happens to be structured in a strange way, or you want to package files that for some reason aren't in the source folder, you can use the include and exclude sections for that.
Related
I've just come across the https://github.com/FriendsOfPHP/security-advisories tool which looks a great way to automatically scan for the vulnerabilities that are in that community-contributed database.
It scans a composer.lock file for packages with vulnerabilities. However, it's made me realise that my understanding of Composer is not what it should be!
I have a project that has a composer.json file that requires a single demo/package. That demo package also has requirements, like demo/dep.
The result of running composer install --no-dev is that I have a composer.lock file which includes:
demo/package version 1.0
demo/dep version 1.2
All good so far, and running symfony security:check /path/to/my/project/composer.lock gives me a green light, no vulnerabilities.
However on close inspection of the files now in my vendor dir, I can see there's a vendor/demo/package/composer.lock file, which contains references to demo/dep at version 1.1 - which has a security vulnerability against it.
As I understand, I have the safer 1.2 version installed - so says my project's composer.lock file, but why is a composer.lock file included with the vendor's package?
Does that mean that the dodgy code is installed somewhere, too? Or can I just simply ignore the composer.lock files if there's a composer.lock file in a dir above it or such? composer show does not list the versions in the nested lock file. Or maybe I should ignore composer.lock files if there's no sibling ./vendor/ dir?
Why not simply inspect your folders to find a vulnerable version? If there was any, you should find a vendor folder within that package, that's where that package could have installed stuff from it's own composer.lock
Usually, only the composer.json of a package is evaluated to install dependencies. If there is a lock file within one package's folder, you should ask the maintainer of that package why this is the case, but for installing dependencies on your system, this does not matter.
Side note: writing "usually" refers to the standard model of installations. I've seen some crude stuff where Composer plugins put other rules in place, but this cannot be said for your project without knowing more about the structure.
After I install the package transformer by pip install transformer, I find it under three locations.
/home/jinggu/anaconda3/bin/transformers
/home/jinggu/anaconda3/lib/python3.7/site-packages/transformers-2.1.1.dist-info/*
/home/jinggu/anaconda3/lib/python3.7/site-packages/transformers/*
What is the difference between these three?
The two in site-packages are for the metadata about the package (transformers-2.1.1.dist-info/) and the actual source code (transformers/). The file in the bin/ folder is called an entry point and represents a commandline interface provided by the package. In this case, you can see this defined in the setup.py file and that it points to running the main() function.
The anaconda3/bin folder contains executables installed w/ the package. If packages would like to provide a command line interface they will come with binaries to be executed from the command line. Not sure what transformers is, but if you navigate to anaconda3/bin/ you should be able to ./transformers -flags or cla's.
The python3.7/site-packages/ folders contain the python source code that can be imported to be used in your projects. This is generally how anaconda packages are used - and how you use the packaged libraries through conda's environment.
In your case, the transformers package came with binaries as well as the source code (to be imported for use in your projects).
When I clone a project written in golang, it is normal that a lot of imports like
'github.com/XXXX' are missing. Is there any way to get these imports in batches by a command? or I am suppose to get them one by one.
You should use go get to get "remote" packages. Quoting from Command go: Download and install packages and dependencies
Get downloads the packages named by the import paths, along with their dependencies. It then installs the named packages, like 'go install'.
You may use the -v flag in all of the following commands, and so you will see what go get is doing under the hood.
You may use the -d flag if you just want to download the packages but you do not want to install them.
The examples use the example remote package github.com/somebody/somepackage, but obviously it works for other packages hosted outside of github.com.
For more information, see the official doc: Command go, or type go help get.
To get a single package with all the dependencies of that package and install them, use
go get github.com/somebody/somepackage
To get a package with all its dependencies, and all other packages rooted at that path (along with their dependencies), and install all of them, use:
go get github.com/somebody/somepackage/...
Quoting from Command go:
An import path is a pattern if it includes one or more "..." wildcards, each of which can match any string, including the empty string and strings containing slashes. Such a pattern expands to all package directories found in the GOPATH trees with names matching the patterns.
To get a package with all its dependencies (and "subpackages") including dependencies of tests, and install all of them, use:
go get -t github.com/somebody/somepackage/...
To update a package you already have, use:
go get -u github.com/somebody/somepackage/...
To fetch dependencies of a package you already have (which is not necessarily from a remote location):
go get path/to/package/name/...
Or go to its folder and then you may use a relative path:
go get ./...
A lot of golang projects now use dependency management so you should look for that first. e.g a Glide.lock (glide) or Gopkg.lock (dep - the way people are moving now) file present in the root of the project.
https://github.com/golang/dep
https://golang.github.io/dep
if dep is used and you have it installed then dep ensure will set the dependencies up for you and make sure you get the versions the author intended
if a project is not using dependency management you can just get the packages with go get ./... but I don't think you will be guaranteed the correct versions (e.g if the author was pinned to a version tag for a dep)
If you run dep init it sets up dep on a project and will attempt to resolve the correct versions, however this doesnt always work if the stars dont align (e.g I have seen issues with dependencies using gopkg.in)
try using go get ./... in root of your project
NOTE: As of Sep. 4th, 2015, I can no longer reproduce this problem, using current versions of python and setuptools/distutils, in either python 2 or 3.
NOTE: I've answered this question myself and will be accepting the answer as soon as the system lets me, but this would appear to be the only instance of an English-language reference to the problem on the internet (according to google), so I thought I'd post both in hopes this can be useful.
I ran into a problem that I found impossible to debug when installing a python package of my own. It's a very simple package, with a flat structure (no subdirectories), and a few data files I need copied over at installation. With a few deletions for brevity, I have this:
spherical_functions
├── MANIFEST.in
├── __init__.py
├── spherical_functions.py
├── ladder_operator_coefficients.npy
└── setup.py
When installing from my source directory, I have no problems. But when I either switched to setuptools or installed with pip (which is needed for distribution), I get the error
can't copy 'ANIFEST.in': doesn't exist or not a regular file
Note the missing M at the beginning of that file name. I hacked into distutils a bit to verify that it was actually trying to copy ANIFEST.in, rather than just misreporting the name. Unfortunately, the trail ends there, because the rest of the setuptools code is hidden in some egg I don't understand.
A similar problem arises without a MANIFEST.in file, using just the package_data option to the setup function, where it misses the first letter of the data file, as in
error: can't copy 'adder_operator_coefficients.npy': doesn't exist or not a regular file
With a hint from this thread, I realized that I was making the mistake; I didn't really understand all the options to the setup function in setup.py. I had
package_dir={'spherical_functions': ''},
when I should have had
package_dir={'spherical_functions': '.'},
(Note the extra dot.) After including this, it seems to work just fine.
I might suggest that the resulting behavior is wrong in any case, suggesting a bug in setuptools and/or distutils. Or at least there could be a check to ensure that the user didn't do something stupid like I did...
Note that you can just use the py_modules keyword to setup(), which should simply be a list of module (file) names to install, if you wish. This is the simplest kind of python distribution.
setup(...
py_modules=['spherical_functions'],
...)
In this manner, after installing your distribution, one could simply run from spherical_functions import <a_member_inside_that_file>.
How can I put my Go binary into a Debian package? Since Go is statically linked, I just have a single executable--I don't need a lot of complicated project metadata information. Is there a simple way to package the executable and resource files without going through the trauma of debuild?
I've looked all over for existing questions; however, all of my research turns up questions/answers about a .deb file containing the golang development environment (i.e., what you would get if you do sudo apt-get install golang-go).
Well. I think the only "trauma" of debuild is that it runs lintian after building the package, and it's lintian who tries to spot problems with your package.
So there are two ways to combat the situation:
Do not use debuild: this tool merely calls dpkg-buildpackage which really does the necessary powerlifting. The usual call to build a binary package is dpkg-buildpackage -us -uc -b. You still might call debuild for other purposes, like debuild clean for instance.
Add the so-called "lintian override" which can be used to make lintian turn a blind eye to selected problems with your package which, you insist, are not problems.
Both approaches imply that you do not attempt to build your application by the packaging tools but rather treat it as a blob which is just wrapped to a package. This would require slightly abstraining from the normal way debian/rules work (to not attempt to build anything).
Another solution which might be possible (and is really way more Debian-ish) is to try to use gcc-go (plus gold for linking): since it's a GCC front-end, this tool produces a dynamically-linked application (which links against libgo or something like this). I, personally, have no experience with it yet, and would only consider using it if you intend to try to push your package into the Debian proper.
Regarding the general question of packaging Go programs for Debian, you might find the following resources useful:
This thread started on go-nuts by one of Go for Debian packagers.
In particular, the first post in that thread links to this discussion on debian-devel.
The second thread on debian-devel regarding that same problem (it's a logical continuation of the former thread).
Update on 2015-10-15.
(Since this post appears to still be searched and found and studied by people I've decided to update it to better reflec the current state of affairs.)
Since then the situation with packaging Go apps and packages got improved dramatically, and it's possible to build a Debian package using "classic" Go (the so-called gc suite originating from Google) rather than gcc-go.
And there exist a good infrastructure for packages as well.
The key tool to use when debianizing a Go program now is dh-golang described here.
I've just been looking into this myself, and I'm basically there.
Synopsis
By 'borrowing' from the 'package' branch from one of Canonical's existing Go projects, you can build your package with dpkg-buildpackage.
install dependencies and grab a 'package' branch from another repo.
# I think this list of packages is enough. May need dpkg-dev aswell.
sudo apt-get install bzr debhelper build-essential golang-go
bzr branch lp:~niemeyer/cobzr/package mypackage-build
cd mypackage-build
Edit the metadata.
edit debian/control file (name, version, source). You may need to change the golang-stable dependency to golang-go.
The debian/control file is the manifest. Note the 'build dependencies' (Build-Depends: debhelper (>= 7.0.50~), golang-stable) and the 3 architectures. Using Ubuntu (without the gophers ppa), I had to change golang-stable to golang-go.
edit debian/rules file (put your package name in place of cobzr).
The debian/rules file is basically a 'make' file, and it shows how the package is built. In this case they are relying heavily on debhelper. Here they set up GOPATH, and invoke 'go install'.
Here's the magic 'go install' line:
cd $(GOPATH)/src && find * -name '*.go' -exec dirname {} \; | xargs -n1 go install
Also update the copyright file, readme, licence, etc.
Put your source inside the src folder. e.g.
git clone https://github.com/yourgithubusername/yourpackagename src/github.com/yourgithubusername/yourpackagename
or e.g.2
cp .../yourpackage/ src/
build the package
# -us -uc skips package signing.
dpkg-buildpackage -us -uc
This should produce a binary .deb file for your architecture, plus the 'source deb' (.tgz) and the source deb description file (.dsc).
More details
So, I realised that Canonical (the Ubuntu people) are using Go, and building .deb packages for some of their Go projects. Ubuntu is based on Debian, so for the most part the same approach should apply to both distributions (dependency names may vary slightly).
You'll find a few Go-based packages in Ubuntu's Launchpad repositories. So far I've found cobzr (git-style branching for bzr) and juju-core (a devops project, being ported from Python).
Both of these projects have both a 'trunk' and a 'package' branch, and you can see the debian/ folder inside the package branch. The 2 most important files here are debian/control and debian/rules - I have linked to 'browse source'.
Finally
Something I haven't covered is cross-compiling your package (to the other 2 architectures of the 3, 386/arm/amd64). Cross-compiling isn't too tricky in go (you need to build the toolchain for each target platform, and then set some ENV vars during 'go build'), and I've been working on a cross-compiler utility myself. Eventually I'll hopefully add .deb support into my utility, but first I need to crystallize this task.
Good luck. If you make any progress then please update my answer or add a comment. Thanks
Building deb or rpm packages from Go Applications is also very easy with fpm.
Grab it from rubygems:
gem install fpm
After building you binary, e.g. foobar, you can package it like this:
fpm -s dir -t deb -n foobar -v 0.0.1 foobar=/usr/bin/
fpm supports all sorts of advanced packaging options.
There is an official Debian policy document describing the packaging procedure for Go: https://go-team.pages.debian.net/packaging.html
For libraries: Use dh-make-golang to create a package skeleton. Name your package with a name derived from import path, with a -dev suffix, e.g. golang-github-lib-pq-dev. Specify the dependencies ont Depends: line. (These are source dependencies for building, not binary dependencies for running, since Go statically links all source.)
Installing the library package will install its source code to /usr/share/golang/src (possibly, the compiled libraries could go into .../pkg). Building depending Go packages will use the artifacts from those system-wide locations.
For executables: Use dh-golang to create the package. Specify dependencies in Build-Depends: line (see above regarding packaging the dependencies).
I recently discovered https://packager.io/ - I'm quite happy with what they're doing. Maybe open up one of the packages to see what they're doing?