We use npm behind a company firewall and thus use proxy and https-proxy settings in the npm configuration. This works fine as long as all npm modules are loaded from an external registry. But as a matter of fact we also have internal modules on an internal github. The access to these modules must not use the proxy of course. My question is: Can I specify a list of domains in the npm configuration, for which the proxy should not be used? Something like the no_proxy environment variable in the unix shell?
Assuming your environment looks like this:
Build server with internet access only over proxy: your.proxy.host:3128
Local Nexus Registry: https://your.local.nexus.registry/nexus/content/groups/npm/
NPM must use local Nexus Registry. Configuration file: .npmrc
registry = https://your.local.nexus.registry/nexus/content/groups/npm/
You can tell npm to use a proxy by setting the environment variables
http_proxy=http://your.proxy.host:3128
https_proxy=http://your.proxy.host:3128
but then npm will also try to reach your (local) Nexus Registry using the proxy.
You need to have one of the latest npm Versions (npm 2.14.7 works fine) and set an additional environment variable to exclude your Nexus Registry from the proxy:
no_proxy=your.local.nexus.registry
Since NPM 6.4.1, released on 2018-08-22, you can use the noproxy option, even with a custom registry configured.
Example :
npm config set registry "http://<my-npm-registry-host>:<registry-port>"
npm config set proxy "http://<my-proxy-host>:<proxy-port>"
npm config set https-proxy "http://<my-proxy-host>:<proxy-port>"
npm config set noproxy "my-proxy-host" (accepts pattern like *.domain)
Check config :
npm config list
References :
NPM doc for noproxy
NPM changelog
NPM PR #46
to exclude certain domains , you can add below lines in .npmrc
noproxy[]=.local.nexus
noproxy[]=.internal.npm.repo
https://docs.npmjs.com/misc/config#noproxy
As adiesner indicated, you cannot set a noproxy/no_proxy in the npm config, so the only way to do this is via the environment variables:
npm config set registry "your.local.nexus.registry/nexus/content/groups/npm/"
export http_proxy="http://your.proxy.host:3128"
export https_proxy=$http_proxy
export no_proxy="your.local.nexus.registry"
If you are using cntlm you should configure the NoProxy option in the configuration file "/etc/cntlm.conf". For instance,
Username your_user
Domain UCI.CU
Password your_password
Proxy 10.0.0.1:8080
Listen 3128
NoProxy uci.cu, edu.cu, nexus.prod.uci.cu, localhost, 127.0.0.*, 10.*, 192.168.*
Remember that you must restart cntlm: sudo systemctl restart cntlm
For PowerShell, the commands are:
$Env:http_proxy = "http://your.proxy.host:3128"
$Env:https_proxy = "http://your.proxy.host:3128"
$Env:no_proxy = "..."
Related
I am trying to install packages from within the corporate network along with some packages that exist within the corporate repository. I was able to configure the following proxy settings for yarn
yarn config set proxy "${PROXY}"
yarn config set https-proxy "${PROXY}"
But since some packages exist within the corporate network installing them fails. I couldnt find a way to set no proxy for internal corporate urls. There is a workaround posted here - https://github.com/yarnpkg/yarn/issues/5048#issuecomment-604181595. But I would like to know if there is a better way
yarn doesn't seem to to have a concept of no proxy. It either tries to proxy everything or nothing...very naive. Fortunately, npm does so we use npm for all of our projects.
I'm trying to connect to my private npm feed from Mac. I generated credentials from Connect to feed menu and they looked like that:
; begin auth token
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:username=ANYTHING-BUT-EMPTY
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:_password=BASE64-ENCODED-PAT-GOES-HERE
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:email=npm requires email to be set but doesn't use the value
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/:username=ANYTHING-BUT-EMPTY
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/:_password=BASE64-ENCODED-PAT-GOES-HERE
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/:email=npm requires email to be set but doesn't use the value
; end auth token
I placed that in .npmrc file in my project and it didn't work. When im trying to do npm install I get this error:
code E401
npm ERR! Unable to authenticate, need: Bearer authorization_uri=https://login.windows.net/...,
Basic realm="https://pkgsprodsu3weu.app.pkgs.visualstudio.com/", TFS-Federated
I also placed these credentials in $HOME directory which also didn't solve the issue. What am I doing wrong? In which .npmrc file should they be? Should I run additional commands to use them?
How to connect NPM to azure artifacts feed on Mac?
The .npmrc file which including the credentials should set int the $home directory.
Check the document Use npm to store JavaScript packages in Azure DevOps Services or TFS:
On your development machine, you also have an .npmrc file in $home for
Linux or Mac systems, or $env.HOME for Windows systems. This .npmrc
file should contain credentials for all of the registries that you
need to connect to. The npm client will look at your project's .npmrc
file, discover the registry, and fetch matching credentials from
$home/.npmrc or $env.HOME/.npmrc. The next section will discuss
credential acquisition.
Since it still not work for you, you could check if your npmrc token has expired. In your .npmrc, I found you are using BASE64-ENCODED-PAT-GOES-HERE, it seems you are using the PAT, but in the 90-day token type. The .npmrc file should like:
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:username=ANYTHING-BUT-EMPTY
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:_password=BASE64-ENCODED-PAT-GOES-HERE
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:email=YOUREMAIL#EXAMPLE.COM
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:always-auth=true
Check Create a token that lasts longer than 90 days.
If you still have 401 error, please check if your PAT has expired and convert to Base64String or do not have enough permission.
Hope this helps.
In my case for some reason I needed to surround the base 64 encoded token with double quotes and square bracket to make it work.
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:_password="[BASE64-ENCODED-PAT-GOES-HERE]"
Note the : "[ and at the end ]".
after adding that, all worked just fine.
I'm running MacOS Big Sur 11.6
To be able to connect to the ADO npm feed without saving the credentials, you can get an access token in ADO, and pass that as a parameter to the following script:
setup-npmrc-feed-auth.bash
#!/bin/bash
DecodedPat=$1
NPMAuthIdent=$(echo -ne "$DecodedPat" | base64);
pnpm config set registry https://pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/registry/ --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/registry/:username {orgName} --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/registry/:_password $NPMAuthIdent --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/registry/:email some#email.com --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/:username {orgName} --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/:_password $NPMAuthIdent --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/:email some#email.com --location=global
in your workspace, have the following:
.npmrc
registry=https://pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/registry/
auto-install-peers=true
strict-peer-dependencies=false
always-auth=true
In my package.json I have a depedency to antoher git repository:
"my-dependeny":"git+https://somehost.com/my-dependency.git#<commit-hash>"
After running npm install this works fine while developing because git is using my personal stored credentials.
But executing npm install in a jenkins-pipline will fail, because there are not stored credentials. So I first tried to generate a deploy token in gitlab and edit the package.json:
"my-dependeny":"git+https://<deploy-token>:<password>#somehost.com/my-dependency.git#<commit-hash>"
Now it will work on jenkins!
But the problem with this approach is, that Windows autoamtically updates the credential for the host (somehost.com) in the Credentials Manager:
So when any developer is running npm install and is afterwards trying to access another project within the same host it will fail, because it will use the deploy-token for the autenthication.
Of course you could add new user to gitlab which has access to all relevant projects. Then you could store this user on jenkins, but this doesn't seem to be an optimal or even good solution.
Maybe there could also be a way to stop the Credential Manager from updating git credentials. But this also doesn't sound well, because every developer would have to prevent this, which is not very intuitive.
My Question
Is there a good way to access npm git-depencies from jenkins without requiring extra steps that need to be done by each developer?
I was finally able to find a solution by myself with the help of git-credential-store.
The following steps apply for linux systems, because in my case the Jenkins is running on Linux. But there should be a similiar solution for Windows systems.
Create empty ~/.git-credentials file. If you use a Dockerfile to copy the .git-credentials file you most likly have to use /root/git-credentials as path (e.g. COPY .git-credentials /root/)
Insert your credentials like the following to your ~/.git-credentials file:
<protocol>://<deploy-token>:<password>#<hostname>/<project>
Example:
https://gitlab+deploy-10:mYSecReTPaSSWord#somehost.com/my-dependency.git
or multiple credentials (follow 4. step below to enable this feature):
https://gitlab+deploy-10:mYSecReTPaSSWord#somehost.de/some-path/some-project.git
https://gitlab+deploy-11:otHerPaSSWord#somehost.de/some-path/some-other-project.git
NOTE:
The line-endings in the ~/.git-credentials have to be LF. If
they are CR;LF unexpected errors can occur.
git config --global credential.helper store
This will tell git that you wanna use your .git-credentials file for authentication.
(~/.git-credentials is the default location)
optional: git config --global credential.useHttpPath true
This is needed if you want to use mutliple Deploy Tokens for multiple projects within the same hostname. Without credential.useHttpPath true in the example above this would mean, that gitlab+deploy-10:mYSecReTPaSSWord would be used in every request to somehost.de. The HTTP path would be ignored completly.
The following is how I solved this in my jenkins file:
withCredentials([usernamePassword(credentialsId: 'my-git-username-and-api-token', passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
sh '''
git config --global credential.username ${GIT_USERNAME}
git config --global credential.helper "!echo password=${GIT_PASSWORD}; echo"
npm install
npm run build
'''
}
As you can see it uses GitHub username and API token from the credentials store. It also runs git config every time. For me this is necessary as I'm running my builds in a docker node. This solves the problem we encountered where git+https dependencies in npm were not being resolved in our jenkins pipeline.
To set a proxy when using npm I use :
npm config set proxy http://proxy.pro.com:8080
npm config set https-proxy http://proxy.pro.com:8080
I set this with each use of npm in mac terminal. Where should this command be added in order to permanently set the proxy so is available to all instances of terminal ?
I could run a startup script on mac boot but how to ensure the proxy config is available to all instances of terminal ?
You can create a .npmrc file and place the config in there
proxy http://proxy.pro.com:8080
https-proxy http://proxy.pro.com:8080
npm config set https-proxy https://username:pwd#proxy.company.com:8080
npm config set proxy https://username:pwd#proxy.company.com:8080
npm config set sslVerify false
npm config set strict-ssl false
set HTTP_PROXY=http://username:pwd#proxy.company.com:8080
set HTTPS_PROXY=https://username:pwd#proxy.company.com:8080
Permanently set proxy:
Add HTTP_PROXY and HTTPS_PROXY to System environment variables
You could put it in your bash profile, or other shell startup script. So, for example $HOME/.profile.
If you mean, "all instances of Terminal for all users", then you would need to put it in /etc/bashrc.
I believe my work proxy is preventing me from being able to add themes and packages to Atom. From the preferences menu, I get:
Fetching featured packages and themes failed. Hide output…
tunneling socket could not be established, cause=140499728967552:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:766:
Is it possible to make it use my $https_proxy variable? Is there some way to configure it to not use https?
You can configure your proxy settings in ~/.atom/.apmrc (or Atom\resources\app\apm\node_modules\atom-package-manager\.apmrc in Windows). Per the apm README:
If you are using a proxy you can configure apm to use it by setting the https-proxy config in your ~/.atom/.apmrc file like so:
https-proxy = https://9.0.2.1:0
It seems that as of Atom 1.0, there are some components that respect the http-proxy and https-proxy variables, and others that don't. For example, the initial check for the version of Atom works, but the check for packages doesn't seem to respect the http-proxy or https-proxy settings.
I was able to get Atom working with Fiddler as my proxy (on 127.0.0.1:8888) by running the following commands (on Windows):
apm config set proxy http://127.0.0.1:8888
apm config set strict-ssl false
I did not need to set http-proxy or https-proxy. I don't know if these settings have been deprecated or not, but they don't seem to work reliably in 1.0. The setting proxy works (and upgrades itself to TLS 1.2 automatically).
The entirety of my %USERPROFILE%\.atom\.apmrc file is:
strict-ssl=false
proxy=http://127.0.0.1:8888/
I was getting this error on Windows 7 fro Atom 1.0, when trying to look up packages.
In my case the issue was resolved by setting https-proxy variable to use http protocole instead of https. so both of the following parameters have exact same value.
here are the values from ~.atom.apmrc file
proxy=http://[host]:[port]/
https-proxy=http://[host]:[port]/
the answer by #NYCdotNet below that suggested to use
strict-ssl=false
didn't work as i was able to lookup some packages but installation failed with timeout error from GIT.
Atom will use your shell's proxy variables (like $https_proxy) if you start Atom from a shell that has these variables set. You need to have the Atom command line tools installed for that to work.
From a shell, you can simply type atom to open the editor for the current directory. It will use all environment variables from this shell, including the proxy variables.
I find this a lot easier than setting the variables in the config file.
I edited the ~/.atom/.apmrc file to set my proxy as mentioned by #AlexMooney, but still got errors.
The solution was to write
proxy = http://host:port
strict-ssl = false
in that ~/.atom/.apmrc file.
For Windows you can easily configure the https-proxy via command line:
amd config set https-proxy https://9.0.2.1:0
It should be stored under C:\Users\...\.apm in file .apmrc
See userconfig with command
amd config list
To config Proxy for Atom to install new pakage, just open CMD and run these commands:
apm config set strict-ssl false
apm config set proxy your_proxy
apm config set your_proxy
I am working behind a proxy server and spent about half a day on this issue, setting https_proxy as well as http_proxy didn't make a difference.
What did it for me was setting the proxy from the cmd line like so:
apm config set proxy http://myproxyaddress:port
I still can't install packages through Atom's gui, but doing it through the cmd line works fine. That I'll take.
I later realised I could've switched to the wifi and got it to work immediately...