Locally, we use a private source URL for our poetry configurations, and the credentials will be found in our pip.ini. However, in our CI/CD pipeline we obtain a secret which changes the PIP_EXTRA_INDEX_URL. This is not enough, because we need to change the pyproject.toml (see below) with this URL.
[[tool.poetry.source]]
name = "private"
url = "https://someplace.pkgs.visualstudio.com/_packaging/somewhere/pypi/simple/"
secondary = true
This is not changed with poetry config repositories.private "$PIP_EXTRA_INDEX_URL". How should we change this then?
You can just the link without the credentials and specify the login like this:
poetry config repositories.someplace https://someplace.pkgs.visualstudio.com/_packaging/somewhere/pypi/
poetry config http-basic.someplace someuser $(personal-access-token)
Related
I'm trying to use a private theme/module with a personal access token. I can get this working by adding the following to my global git config.
git config --global url."https://{USER}:{TOKEN}#github.com".insteadOf "https://github.com"
Then running hugo mod get -u it will pull changes as expected.
I don't want this set in my global config and if I set it locally I get an error, because Go doesn't seem to be using the local config.
Set my configurations locally within the root of the site/repository:
git config --local url."https://{USER}:{TOKEN}#github.com".insteadOf "https://github.com"
Then running hugo mod get -u I get the following error:
go get: module github.com/USER/REPOSITORY: git ls-remote -q origin in /var/folders/26/gqnv01_55p964v8yz39d51fw0000gn/T/hugo_cache/modules/filecache/modules/pkg/mod/cache/vcs/b410fc7b91fbc1121b5f6ec2bb2711c27cd172b4084c213e1430a33cde552597: exit status 128:
remote: Repository not found.
fatal: repository 'https://github.com/USER/REPOSITORY/' not found
How can I get Go/Hugo to use my local git config rather than the global?
From the hugo mod source code, hugo will look for a go.mod in your project:
filepath.Walk(dirname, func(path string, info os.FileInfo, err error) error {
if info.IsDir() {
return nil
}
if info.Name() == "go.mod" {
// Found a module.
dir := filepath.Dir(path)
fmt.Println("Update module in", dir)
Check where your go.mod is, and do (in that go.mod parent folder):
git config -l --show-origin --show-scope
That will tell you if your expected local config is actually there or not.
Look for any .git folder which would indicate a nested git repository/submodule, which would ignore your initial git config --local command
An issue like 34513 seems to suggests though that go mod won't take into account the local repository:
The git configuration only affects operations on the underlying git repo.
The error that you're seeing is coming from before that, when the go command is attempting to resolve the repo for the requested package path.
The official documentation only references the global config .gitconfig.
I solved this by adding a directory replacement mapping to the site’s config, instead of modifying the git url. This points to my locally cloned theme and updates the served site whenever I modify the theme.
module:
imports:
path: 'github.com/[USER]/[REPO-NAME]'
replacements: 'github.com/[USER]/[REPO-NAME] -> ../../[REPO-NAME]/'
I am using golang 1.13 .
I have a project that depends on a private gitlab project.
I have the ssh keys for the same.
When I try to retrieve the dependencies for a newly created module, I am getting the following error:
$ go version
go version go1.13 linux/amd64
$ go mod why
go: downloading gitlab.com/mycompany/myproject v0.0.145
verifying gitlab.com/mycompany/myproject#v0.0.145: gitlab.com/mycompany/myproject#v0.0.145: reading https://sum.golang.org/lookup/gitlab.com/mycompany/myproject#v0.0.145: 410 Gone
I have no idea why it is trying to ping sum.golang.org/lookup since it is a private gitlab project.
My ~/.gitconfig contains the following (based on my looking up in google search for similar errors)
# Enforce SSH
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
[url "git#gitlab.com:"]
insteadOf = https://gitlab.com/
The error still persists.
I would expect the package to be downloaded from my private gitlab project repository to the current project.
Is there anything I need to do in my private gitlab project repository to make it ready for 'go get' ?
The private gitlab project repository already contains the go.sum and go.mod for the project as well.
Anything that I am missing ?
edit: 1) The private repo name and the company name contains no asterisks or any other special characters. only alphabets and not even numeric characters.
Answering my own question after looking up,
Setting the GOPRIVATE variable seems to help.
GOPRIVATE=gitlab.com/mycompany/* go mod why
"
The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database.
" from https://golang.org/doc/go1.13
Aliter:
Setting the env variable GONOSUMDB also seems to work.
Specifically, invoking the following command seems to help.
GONOSUMDB=gitlab.com/mycompany/* go mod why
The above env variable prevents the ping to sum.golang.org/lookup for a checksum match. It also prevents leaking the names of private repos to a public checksum db. [ Source - https://docs.gomods.io/configuration/sumdb/ ]
Also - here at
* GONOSUMDB=prefix1,prefix2,prefix3 sets a list of module path prefixes, again possibly containing globs, that should not be looked up using the database.
source: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md
Related Issues:
https://github.com/golang/go/issues/32291
https://github.com/golang/go/issues/33985
["Go 1.13 has been released, and this issue was filed well after the freeze window. The proposed changes will not happen in 1.13, but don't assume they will necessarily happen in 1.14 either." from issue 33985 above. ]
Basically it failed to verify private repository. However I don't like turning off checksum, but you can easily set GOSUMDB to off before trying to get module. something like this:
GOSUMDB=off go get github.com/mycompany/myproject
ref: https://github.com/golang/go/issues/35164#issuecomment-546503518
A second and better solution is to set GOPRIVATE environment variable that controls which modules the go command considers to be private (not available publicly) and should therefore NOT use the proxy or checksum database. The variable is a comma-separated list of glob patterns (same syntax of Go's path.Match) of module path prefixes. For example,
export GOPRIVATE=*.corp.example.com,rsc.io/private
Or
go env -w GOPRIVATE=github.com/mycompany/*
Last solution you can try is to turn off such checks for all private repositories that you don't want to go public or being verified through sum.golang.org/lookup/github.com/mycompany/...
GONOSUMDB=gitlab.com/mycompany/* go mod why
Note that:
If you have issues fetching modules or repos over https, you may want to add the following to your ~/.gitconfig to make go get/fetch repositories using ssh instead of https
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
Change following go variable's setting and then upgrade your package,
$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOSUMDB=off
$ go get -u <your dependency package>
I have this scenario too and this works for me.
edit your .git/config and add two lines in it.( I have this in a global .gitconfig in home dir)
[url "ssh://youprivate.com"]
insteadOf = https://yourprivate.com
export GOSUMDB=off
Then everything will OK.
I would like to store my credentials in ~/.aws/credentials and not in environmental variables, but I am struggling.
To load the credentials I use (from here)
credentials = Aws::SharedCredentials.new({region: 'myregion', profile_name: 'myprofile'})
My ~/.aws/credentials is
[myprofile]
AWS_ACCESS_KEY = XXXXXXXXXXXXXXXXXXX
AWS_SECRET_KEY = YYYYYYYYYYYYYYYYYYYYYYYYYYY
My ~/.aws/config is
[myprofile]
output = json
region = myregion
I then define a resource with
aws = Aws::EC2::Resource.new(region: 'eu....', credentials: credentials)
but if I try for example
aws.instances.first
I get the error Error: #<Aws::Errors::MissingCredentialsError: unable to sign request without credentials set>
Everything works if I hard code the keys
According to the source code aws loads credentials automatically only from ENV.
You can create credentials with custom attributes.
credentials = Aws::Credentials.new(AWS_ACCESS_KEY, AWS_SECRET_KEY)
aws = Aws::EC2::Resource.new(region: 'eu-central-1', credentials: credentials)
In your specific case, it seems there is no way to pass custom credentials to SharedCredentials.
If you just do
credentials = Aws::SharedCredentials.new()
it loads the default profile. You should be able to load myprofile by passing in :profile_name as an option.
I don't know if you can also override the region though. You might want to try to loose that option, see how it works.
To install logstash with rpm (https://www.elastic.co/guide/en/logstash/current/installing-logstash.html) I need to manually specify the baseurl and the pgpkey
I can't see these options in the chef docs (https://docs.chef.io/resource_rpm_package.html)
Is there a way to do this with chef?
You can try with "yum_repository" resource. For, example:-
Github Source:- https://github.com/chef-cookbooks/yum
# add the Zenoss repository
yum_repository 'zenoss' do
description "Zenoss Stable repo"
baseurl "http://dev.zenoss.com/yum/stable/"
gpgkey 'http://dev.zenoss.com/yum/RPM-GPG-KEY-zenoss'
action :create
end
I have tried using pip with index-url in pip.conf. However, i can not make sure that we can get all the necessary python library. So, i want to know if pip support specify more than one index-url in [global] section in pip.conf.
In your pip.conf, you will also have to add both of the index hosts as trusted, so would look something like this:
[global]
index-url = http://download.zope.org/simple
trusted-host = download.zope.org
pypi.org
secondary.extra.host
extra-index-url= http://pypi.org/simple
http://secondary.extra.host/simple
In this example, you have a primary index and two extra index urls and all hosts are trusted.
If you don't specify the host as trusted, you will get the following error:
The repository located at secondary.extra.host is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host secondary.extra.host'.
Cheers!
If you want more than one package index you have to use the --extra-index-url
From the pip man page:
-i,--index-url <url>
Base URL of Python Package Index (default https://pypi.python.org/simple/).
--extra-index-url <url>
Extra URLs of package indexes to use in addition to --index-url.
In pip.conf the name of settings must be put without --. From the documentation:
The names of the settings are derived from the long command line option, e.g. if you want to use a different package index (--index-url) and set the HTTP timeout (--default-timeout) to 60 seconds your config file would look like this:
[global]
timeout = 60
index-url = http://download.zope.org/ppix
So you can add in your pip.conf
extra-index-url = http://myserver.com/pip
updating radtek 's answer with the new URL to pypi.
It changed to https://pypi.org
So for your pip to be able to fall back to the original pypi server you'll need to add "https://pypi.org/simple" as an extra-index-url while keeping your local server as index-url.
Don't forget to add both to your "trusted-host" list
This update is based on the comment of onelaview: "Official PyPI now supports HTTPS so you can specify https://pypi.org/simple/ for extra-index-URL and avoid specifying pypi.org in trusted-host."
So your pip.conf needs to be containing the following:
[global]
index-url = https://somedomain.org/simple
trusted-host = somedomain.org
pypi.org
secondary.extra.host
extra-index-url= http://pypi.org/simple <= either one of these is fine
https://pypi.org/simple <= either one of these is fine
http://secondary.extra.host/simple
You can also do this by setting an environment variable:
export PIP_EXTRA_INDEX_URL=http://localhost:8080/simple/
which is equivalent to
[global]
extra-index-url = http://localhost:8080/simple/
but does not require a pip.conf file
I'd add to #Tomasz Bartkowiak answer. You can pass multiple URLs to a PIP_TRUSTED_HOST,PIP_EXTRA_INDEX_URL using spaces:
export PIP_TRUSTED_HOST="somedomain.org pypi.org secondary.extra.host"
export PIP_EXTRA_INDEX_URL="http://pypi.org/simple https://pypi.org/simple http://secondary.extra.host/simple"