Can pip.conf specify two index-url at the same time? - pip

I have tried using pip with index-url in pip.conf. However, i can not make sure that we can get all the necessary python library. So, i want to know if pip support specify more than one index-url in [global] section in pip.conf.

In your pip.conf, you will also have to add both of the index hosts as trusted, so would look something like this:
[global]
index-url = http://download.zope.org/simple
trusted-host = download.zope.org
pypi.org
secondary.extra.host
extra-index-url= http://pypi.org/simple
http://secondary.extra.host/simple
In this example, you have a primary index and two extra index urls and all hosts are trusted.
If you don't specify the host as trusted, you will get the following error:
The repository located at secondary.extra.host is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host secondary.extra.host'.
Cheers!

If you want more than one package index you have to use the --extra-index-url
From the pip man page:
-i,--index-url <url>
Base URL of Python Package Index (default https://pypi.python.org/simple/).
--extra-index-url <url>
Extra URLs of package indexes to use in addition to --index-url.
In pip.conf the name of settings must be put without --. From the documentation:
The names of the settings are derived from the long command line option, e.g. if you want to use a different package index (--index-url) and set the HTTP timeout (--default-timeout) to 60 seconds your config file would look like this:
[global]
timeout = 60
index-url = http://download.zope.org/ppix
So you can add in your pip.conf
extra-index-url = http://myserver.com/pip

updating radtek 's answer with the new URL to pypi.
It changed to https://pypi.org
So for your pip to be able to fall back to the original pypi server you'll need to add "https://pypi.org/simple" as an extra-index-url while keeping your local server as index-url.
Don't forget to add both to your "trusted-host" list
This update is based on the comment of onelaview: "Official PyPI now supports HTTPS so you can specify https://pypi.org/simple/ for extra-index-URL and avoid specifying pypi.org in trusted-host."
So your pip.conf needs to be containing the following:
[global]
index-url = https://somedomain.org/simple
trusted-host = somedomain.org
pypi.org
secondary.extra.host
extra-index-url= http://pypi.org/simple <= either one of these is fine
https://pypi.org/simple <= either one of these is fine
http://secondary.extra.host/simple

You can also do this by setting an environment variable:
export PIP_EXTRA_INDEX_URL=http://localhost:8080/simple/
which is equivalent to
[global]
extra-index-url = http://localhost:8080/simple/
but does not require a pip.conf file

I'd add to #Tomasz Bartkowiak answer. You can pass multiple URLs to a PIP_TRUSTED_HOST,PIP_EXTRA_INDEX_URL using spaces:
export PIP_TRUSTED_HOST="somedomain.org pypi.org secondary.extra.host"
export PIP_EXTRA_INDEX_URL="http://pypi.org/simple https://pypi.org/simple http://secondary.extra.host/simple"

Related

Poetry: changing to private pypi repository on the command line

Locally, we use a private source URL for our poetry configurations, and the credentials will be found in our pip.ini. However, in our CI/CD pipeline we obtain a secret which changes the PIP_EXTRA_INDEX_URL. This is not enough, because we need to change the pyproject.toml (see below) with this URL.
[[tool.poetry.source]]
name = "private"
url = "https://someplace.pkgs.visualstudio.com/_packaging/somewhere/pypi/simple/"
secondary = true
This is not changed with poetry config repositories.private "$PIP_EXTRA_INDEX_URL". How should we change this then?
You can just the link without the credentials and specify the login like this:
poetry config repositories.someplace https://someplace.pkgs.visualstudio.com/_packaging/somewhere/pypi/
poetry config http-basic.someplace someuser $(personal-access-token)

private repo - go 1.13 - `go mod ..` failed: ping "sum.golang.org/lookup" .. verifying package .. 410 gone

I am using golang 1.13 .
I have a project that depends on a private gitlab project.
I have the ssh keys for the same.
When I try to retrieve the dependencies for a newly created module, I am getting the following error:
$ go version
go version go1.13 linux/amd64
$ go mod why
go: downloading gitlab.com/mycompany/myproject v0.0.145
verifying gitlab.com/mycompany/myproject#v0.0.145: gitlab.com/mycompany/myproject#v0.0.145: reading https://sum.golang.org/lookup/gitlab.com/mycompany/myproject#v0.0.145: 410 Gone
I have no idea why it is trying to ping sum.golang.org/lookup since it is a private gitlab project.
My ~/.gitconfig contains the following (based on my looking up in google search for similar errors)
# Enforce SSH
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
[url "git#gitlab.com:"]
insteadOf = https://gitlab.com/
The error still persists.
I would expect the package to be downloaded from my private gitlab project repository to the current project.
Is there anything I need to do in my private gitlab project repository to make it ready for 'go get' ?
The private gitlab project repository already contains the go.sum and go.mod for the project as well.
Anything that I am missing ?
edit: 1) The private repo name and the company name contains no asterisks or any other special characters. only alphabets and not even numeric characters.
Answering my own question after looking up,
Setting the GOPRIVATE variable seems to help.
GOPRIVATE=gitlab.com/mycompany/* go mod why
"
The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database.
" from https://golang.org/doc/go1.13
Aliter:
Setting the env variable GONOSUMDB also seems to work.
Specifically, invoking the following command seems to help.
GONOSUMDB=gitlab.com/mycompany/* go mod why
The above env variable prevents the ping to sum.golang.org/lookup for a checksum match. It also prevents leaking the names of private repos to a public checksum db. [ Source - https://docs.gomods.io/configuration/sumdb/ ]
Also - here at
* GONOSUMDB=prefix1,prefix2,prefix3 sets a list of module path prefixes, again possibly containing globs, that should not be looked up using the database.
source: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md
Related Issues:
https://github.com/golang/go/issues/32291
https://github.com/golang/go/issues/33985
["Go 1.13 has been released, and this issue was filed well after the freeze window. The proposed changes will not happen in 1.13, but don't assume they will necessarily happen in 1.14 either." from issue 33985 above. ]
Basically it failed to verify private repository. However I don't like turning off checksum, but you can easily set GOSUMDB to off before trying to get module. something like this:
GOSUMDB=off go get github.com/mycompany/myproject
ref: https://github.com/golang/go/issues/35164#issuecomment-546503518
A second and better solution is to set GOPRIVATE environment variable that controls which modules the go command considers to be private (not available publicly) and should therefore NOT use the proxy or checksum database. The variable is a comma-separated list of glob patterns (same syntax of Go's path.Match) of module path prefixes. For example,
export GOPRIVATE=*.corp.example.com,rsc.io/private
Or
go env -w GOPRIVATE=github.com/mycompany/*
Last solution you can try is to turn off such checks for all private repositories that you don't want to go public or being verified through sum.golang.org/lookup/github.com/mycompany/...
GONOSUMDB=gitlab.com/mycompany/* go mod why
Note that:
If you have issues fetching modules or repos over https, you may want to add the following to your ~/.gitconfig to make go get/fetch repositories using ssh instead of https
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
Change following go variable's setting and then upgrade your package,
$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOSUMDB=off
$ go get -u <your dependency package>
I have this scenario too and this works for me.
edit your .git/config and add two lines in it.( I have this in a global .gitconfig in home dir)
[url "ssh://youprivate.com"]
insteadOf = https://yourprivate.com
export GOSUMDB=off
Then everything will OK.

How to install ruby locally via rbenv/ruby-build?

I require installing ruby without internet access. As ruby-build docs suggest I can change the mirror URL via specifying the environment variable RUBY_BUILD_MIRROR_URL. I did this and although it looks at my local repo for ruby it still attempts to connect to online repo to install yaml.
env RUBY_BUILD_MIRROR_URL=http://10.10.161.39/platforms/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3 /opt/rbenv/bin/rbenv install 2.0.0-p247
Downloading yaml-0.1.6.tar.gz...
-> http://10.152.161.39/platforms/proteus/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3/7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749
-> http://pyyaml.org/download/libyaml/yaml-0.1.6.tar.gz
error: failed to download yaml-0.1.6.tar.gz
BUILD FAILED (RedHatEnterpriseServer 5.10 using ruby-build 20150928)
I tried placing the yaml-0.1.6.tar.gz file in my local repo however that makes no difference besides it will fail since the sha2 checksum provided in the URL is for ruby-2.0.0-p247.tar.gz file.
How can install ruby offline with rbenv?
Update 1
I discovered that you can modify the lookup config file to point to a local mirror instead. i.e: /opt/rbenv/plugins/ruby-build/share/ruby-build/2.0.0-p247
install_package "yaml-0.1.6" "http://10.10.161.39/platforms/common/yaml-0.1.6.tar.gz#7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749" --if needs_yaml
install_package "openssl-1.0.1p" "ttp://10.10.161.39/platforms/common/openssl-1.0.1p.tar.gz#bd5ee6803165c0fb60bbecbacacf244f1f90d2aa0d71353af610c29121e9b2f1" mac_openssl --if has_broken_mac_openssl
install_package "ruby-2.0.0-p247" "http://10.10.161.39/platforms/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3"
Is there a better way or is this is the best way forward?
So here's how I got it to work:
Update the contents of the download file in /opt/rbenv/plugins/ruby-build/share/ruby-build/<ruby-version> to point to your local repo.
You will also notice how each file has a long hash valued after the '#' symbol in the URL. For Example:
install_package "yaml-0.1.6" "http://10.10.161.39/platforms/common/yaml-0.1.6.tar.gz#7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749" --if needs_yaml
This hash value is the sha256sum the file which rbenv will use to validate if it is the expected file.
So you will need to generate the value by running sha256sum <filename> and appending to each file in the URL path.
Complete example below:
install_package "yaml-0.1.6" "http://10.10.161.39/platforms/common/yaml-0.1.6.tar.gz#7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749" --if needs_yaml
install_package "openssl-1.0.1p" "ttp://10.10.161.39/platforms/common/openssl-1.0.1p.tar.gz#bd5ee6803165c0fb60bbecbacacf244f1f90d2aa0d71353af610c29121e9b2f1" mac_openssl --if has_broken_mac_openssl
install_package "ruby-2.0.0-p247" "http://10.10.161.39/platforms/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3"
In the example above we have a dedicated repository server at http://10.10.161.39/platforms/common. If your packages are locally available, you will need to point to the local path and verify if it works.

Configure ini file for HHVM 3.0 via socket with nginx

I'm able to start the HipHop VM to use a unix socket. I can accomplish this via:
/usr/bin/hhvm --config /etc/hhvm/server.ini --mode daemon -vPidFile=/var/run/hhvm/pid -vServer.Type=fastcgi -vServer.FileSocket=/var/run/hhvm/hhvm.sock
However, I can't find a reference anywhere with how to set this in the ini file I'm specifying for my config. To use a TCP port the line in server.ini is:
hhvm.server.port = 9000
I've tried both
hhvm.server.filesocket=/var/run/hhvm/hhvm.sock
hhvm.server.socket=/var/run/hhvm/hhvm.sock
Both fail. Anyone know the file setting or where a reference for these settings can be found?
Although I can't find any documentation--they haven't yet written the updated version for the ini file format (as of 2014-05-01): https://github.com/hhvm/hack-hhvm-docs/issues/156
Regardless I figured it out and they confirmed it should be:
hhvm.server.file_socket=/var/run/hhvm/hhvm.sock
It looks like you take the camel case command line argument -vServer.FileSocket and drop the v, lowercase it, split it with underscores instead of camel case.
If y ou follow the above rewrite rules you can convert the old format to the new.

Unable to add a port to my local repo in MacPorts

I am trying to add a port to my local repo in MacPorts.
I know the guide.
I run at /Users/Masi/bin/MacPorts/ports/Git unsuccessfully
portindex
Creating software index in /Users/Masi/bin/MacPorts/ports/Git
Total number of ports parsed: 0
Ports successfully parsed: 0
Ports failed: 0
My port-file is the following
PortSystem 1.0
name git-svn
version 1.0
categories git
maintainers sl
description svn for Git
long_description Git-svn is a tool which allows Git to use svn
homepage http://www.kernel.org/pub/software/scm/git/docs/git-svn.html
platforms darwin
master_sites http://git-scm.com/
checksums md5
depends_lib
port:syfi-dev\
port:syfi-doc\
port:python-syfi0\
port:libcln5\
port:libsyfi0\
port:libginac1.4\
port:libsyfi0-dev\
port:syfi-bin\
#I do not know what these are: I leave them as they are by default
configure.args --enable-perl-site-install \
--mandir=${prefix}/share/man
My sources.conf
# MacPorts system wide sources configuration file
# $Id: sources.conf 42662 2008-11-28 23:18:50Z raimue#macports.org $
# To setup a local ports repository, insert a "file://" entry following
# the example below that points to your local ports directory:
# Example: file:///Users/landonf/misc/MacPorts/ports
file:///Users/Masi/bin/MacPorts/ports
rsync://rsync.macports.org/release/ports
# The default MacPorts repository should always be tagged [default]
# for proper functionality of various resources (port groups, mirror
# sites, etc). If you switch it from the rsync:// URL, be sure to keep
# it tagged [default].
# To prevent a source from synchronizing when `port sync` is used,
# append [nosync] at the end as shown in this example:
# Example: file:///Users/landonf/misc/MacPorts/ports [nosync]
# NOTE: The port command parses source URLs in order and installs the
# first occurrance when a port appears in multiple repositories.
# So keep "file://" URLs above other URL types.
# To get the ports tree from the master MacPorts server in California, USA use:
# rsync://rsync.macports.org/release/ports/
# To get it from the mirror in Trondheim, Norway use:
# rsync://trd.no.rsync.macports.org/release/ports/
# A current list of mirrors is available at http://trac.macports.org/wiki/Mirrors
rsync://rsync.macports.org/release/ports/ [default]
How can you add a port to your local MacPorts successfully?
Portfiles must be organized in the following fashion:
$LOCAL_PORT_DIR/{category}/{portname}/
All files, including Portfile, go under that directory.
In your case, your local port dir is /Users/Masi/bin/MacPorts/ports, and your port, git-svn, is in the category git, so your directory structure should look like this:
/Users/Masi/bin/MacPorts/ports/git/git-svn/Portfile
(In the special case of git-svn, though, you can always install the port git-core with the variant +svn, like so: $ sudo port install git-core +svn.)
(Also, as another side note, Git-related ports are typically organized under the category "devel", not "git".)
I'm not sure but I think you need to change your directory structure to this format: portcategory/portname/Portfile. In your case that would be /Users/Masi/bin/MacPorts/ports/git/git-svn/Portfile
Then try to run portindex from the root of you local repo (/Users/Masi/bin/MacPorts/ports)

Resources