How do I specify the repository for all python dependencies? - pip

I use pip/poetry/Pipenv to install dependencies depending on what is less broken and painful in a given month. I have a private pypi compatible repo and have many dependencies on public pypi.
I can see that when a dependency is missing from the private repo or if the private repo is misconfigured, pip will try to get the dependency from public pypi, which at best is the wrong package at worst is malicious code as describe here.
Is there any way to tell pip, poetry and/or Pipenv which repository it should use on a per package basis without fallback behaviors?
It looks like registering all my private package names on public pypi is considered name squatting and is against pypi's rules.
This is different from the various questions on how to install from a private repository. The question is how to force the package manager to only install a package from the specified repository when there are unavoidably two or more repositories.

You can specify whatever package index you like in pipenv. This is an example from the advanced usage docs from pipenv. More usage and separate use cases can be found here
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[[source]]
url = "http://pypi.home.kennethreitz.org/simple"
verify_ssl = false
name = "home"
[dev-packages]
[packages]
requests = {version="*", index="home"}
maya = {version="*", index="pypi"}
records = "*"

Related

How to update a go dependency with different module name than src path?

The location: github.com/elastic/beats
The mod file: github.com/elastic/beats/go.mod
The module name: github.com/elastic/beats/v7
The tag: v7.10.2
What LoTR incantation of go get to I have to run to get a little dependency update action?
This will update to latest minor.patch version of v7:
go get github.com/elastic/beats/v7
or if you want a specific version to update/downgrade to:
go get github.com/elastic/beats/v7#v7.10.2
Adding the -u flag will additionally update the dependencies of github.com/elastic/beats/v7:
go get -u github.com/elastic/beats/v7
The argument list passed to go get should generally be a list of package paths or patterns, not just a module path.
For example, you might invoke:
go get -d github.com/elastic/beats/v7/libbeat/beat#latest
in order to obtain the latest version of package …/libbeat/beat and also download any transitive dependencies needed for that package.
(You can pass just a module path, and that should also update the version of the dependency module overall, but it will not download source code or module checksums for transitive dependencies that may be needed in order to build the updated package. go get does not in general know which transitive dependencies will be relevant to the commands that you plan to invoke after it, and it does not do extra work to speculatively identify relevant dependencies.)

error while using google.golang.org/grpc from a vendor directory

I am having issues using google.golang.org/grpc from vendor directory and I get the below error
cannot use &metadata.HeaderMD (type *"google.golang.org/grpc/metadata".MD) as type *"project1/vendor/google.golang.org/grpc/metadata".MD in argument to grpc.Header
I get the error though I am using the necessary version of the package which I copied from my gopath. But, when I delete the golang.google.org/grpc folder from vendor my project fetches the dependency from gopath and it works fine though the one gopath is a copy of when I have in vendor directory and every other library in vendor directory works fine except grpc.
When you created project1/vendor/google.golang.org/grpc, it means that for packages under project1/..., an import of google.golang.org/gprc/... will be transparently remapped to the vendor version.
Any packages outside of project1 will continue to import the non-vendored google.golang.org/grpc/... packages. While the vendored package might be a copy of the upstream, Go treats them as independent packages. So the types they contain are not equivalent.
What has most likely happened is that one of your non-vendored dependencies imports the grpc package and uses its types in its public API. When you make use of that API from project1, you get the upstream type which can't be assigned to variables using the vendored types.
There's two possible solutions to this problem:
Vendor all of your dependencies that make use of what you've already vendored.
If you're using Go >= 1.11, switch to the newer Go module build system. This will let you continue to control when you upgrade your dependencies without having the project1/vendor/... tree to confuse the type system.

How to find dependency causing "Sirupsen/logrus" vs. "sirupsen/logrus" unexpected module path error?

I am trying to convert https://github.com/appscode/voyager from glide to go mod.
I am getting an error like below:
go: github.com/Sirupsen/logrus#v1.4.1: parsing go.mod: unexpected module path "github.com/sirupsen/logrus"
go: error loading module requirements
How do I find out the source of this old Sirupsen module?
How do I find out the source of this old Sirupsen module?
Use the Go 1.13 beta (go get golang.org/dl/go1.13beta1 && go1.13beta1 download) or even better, try the latest Go on tip / master (go get golang.org/dl/gotip && gotip download).
Go 1.13 has improved error messages in general. It should help in your case, including most likely showing the import chain leading up to the error.
For example:
$ gotip build .
go: example.com/temp/mod imports
github.com/docker/libcompose/docker imports
github.com/Sirupsen/logrus: github.com/Sirupsen/logrus#v1.4.2: parsing go.mod:
module declares its path as: github.com/Sirupsen/logrus
but was required as: github.com/sirupsen/logrus
In that example, you can see that docker/libcompose/docker is importing the old and now incorrect uppercase version of Sirupsen/logrus.
The most common reason people see a Sirupsen/logrus vs. sirupsen/logrus case mismatch is when importing github.com/docker/docker or one of the other docker repos. Importing docker repos is a bit confusing with modules, including because:
The docker/docker repo does not follow semver.
There is a very old v1.13.1 semver tag on the docker/docker repo.
Even though it is a couple years old, it is still the "latest" semver tag on that repo, and hence that old version is picked by default by the go command if you don't ask for a more specific version.
That old docker/docker version imports the old and not incorrect uppercase Sirupsen/logrus, which can then trigger the error reported in the question above.
The docker client package has had breaking changes after v1.13.1.
There is generally confusion about docker/docker vs. docker/engine repositories, and about what import paths to use.
The docker repos do not have go.mod files.
For the docker/docker repo, the import path remains github.com/docker/docker, but it needs to come from github.com/docker/engine, so the recommended approach is often for a docker importer to do import "github.com/docker/docker" and edit their go.mod to something like this:
require (
github.com/docker/docker v1.13.1
)
replace github.com/docker/docker => github.com/docker/engine <tag-or-commit-hash>
Docker issue #39302 tracks trying to document how to import docker repos when using modules.

How to add multiple repositories with the same prefix to composer?

Suppose you have
ssh://git#example.com/project/foo
ssh://git#example.com/project/bar
ssh://git#example.com/project/baz
And you want to add project/foo, project/bar, project/baz as dependencies. Configuring each with composer config repositories.project/foo git ssh://git#example.com/project/foo doesn't seem very efficient.
There is no shortcut for packages with the same prefix - each repository must be defined separately, and in this case one repository can contain only one package.
If you have many private packages, you should probably switch to Satis or Private Packagist.

How to provide additional class for a package on OSGi system classpath for J2SE 1.8 profile

Inside ee.j2se-1.8 the package org.w3c.dom is included.
org.osgi.framework.system.packages = \
...
org.w3c.dom,\
...
org.osgi.framework.bootdelegation = \
...
org.w3c.*,\
org.xml.*,\
....
The bundle xml-apis_1.4.1.20140905-131237.jar contains the same package org.w3c.dom, but with one important class more ... ElementTraversal.class more.
So here my questions...
How can I provide the missing class inside my Equinox OSGi runtime?
You are suffering from the fact that people do not take packages seriously. We have a similar problem in the jta API. The version delivered by the VM is not identical to the version that is delivered as a special package.
The solution is to include the JAR with the extra packages on the class path where the framework resides. It looks like you're using bnd (good!) so this would look like:
-runpath: xml-apis__xml-apis;version="[1.4.1,2.0.0)"
If this had been a bundle with proper exports then you would have automatically gotten its exports as system packages. Looking at the packages in this bundle it seems all packages are already in the standard exports of the VM. However, if you have packages int his JAR that are not exported by the VM then you can add them as follows:
-runsystempackages: javax.xml.foo

Resources