How to access npm git-dependency from jenkins? - windows

In my package.json I have a depedency to antoher git repository:
"my-dependeny":"git+https://somehost.com/my-dependency.git#<commit-hash>"
After running npm install this works fine while developing because git is using my personal stored credentials.
But executing npm install in a jenkins-pipline will fail, because there are not stored credentials. So I first tried to generate a deploy token in gitlab and edit the package.json:
"my-dependeny":"git+https://<deploy-token>:<password>#somehost.com/my-dependency.git#<commit-hash>"
Now it will work on jenkins!
But the problem with this approach is, that Windows autoamtically updates the credential for the host (somehost.com) in the Credentials Manager:
So when any developer is running npm install and is afterwards trying to access another project within the same host it will fail, because it will use the deploy-token for the autenthication.
Of course you could add new user to gitlab which has access to all relevant projects. Then you could store this user on jenkins, but this doesn't seem to be an optimal or even good solution.
Maybe there could also be a way to stop the Credential Manager from updating git credentials. But this also doesn't sound well, because every developer would have to prevent this, which is not very intuitive.
My Question
Is there a good way to access npm git-depencies from jenkins without requiring extra steps that need to be done by each developer?

I was finally able to find a solution by myself with the help of git-credential-store.
The following steps apply for linux systems, because in my case the Jenkins is running on Linux. But there should be a similiar solution for Windows systems.
Create empty ~/.git-credentials file. If you use a Dockerfile to copy the .git-credentials file you most likly have to use /root/git-credentials as path (e.g. COPY .git-credentials /root/)
Insert your credentials like the following to your ~/.git-credentials file:
<protocol>://<deploy-token>:<password>#<hostname>/<project>
Example:
https://gitlab+deploy-10:mYSecReTPaSSWord#somehost.com/my-dependency.git
or multiple credentials (follow 4. step below to enable this feature):
https://gitlab+deploy-10:mYSecReTPaSSWord#somehost.de/some-path/some-project.git
https://gitlab+deploy-11:otHerPaSSWord#somehost.de/some-path/some-other-project.git
NOTE:
The line-endings in the ~/.git-credentials have to be LF. If
they are CR;LF unexpected errors can occur.
git config --global credential.helper store
This will tell git that you wanna use your .git-credentials file for authentication.
(~/.git-credentials is the default location)
optional: git config --global credential.useHttpPath true
This is needed if you want to use mutliple Deploy Tokens for multiple projects within the same hostname. Without credential.useHttpPath true in the example above this would mean, that gitlab+deploy-10:mYSecReTPaSSWord would be used in every request to somehost.de. The HTTP path would be ignored completly.

The following is how I solved this in my jenkins file:
withCredentials([usernamePassword(credentialsId: 'my-git-username-and-api-token', passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
sh '''
git config --global credential.username ${GIT_USERNAME}
git config --global credential.helper "!echo password=${GIT_PASSWORD}; echo"
npm install
npm run build
'''
}
As you can see it uses GitHub username and API token from the credentials store. It also runs git config every time. For me this is necessary as I'm running my builds in a docker node. This solves the problem we encountered where git+https dependencies in npm were not being resolved in our jenkins pipeline.

Related

Using external git-lfs server in gitlab-ci on a windows gitlab runner?

I am trying to use an external git-lfs server for a job running on a windows gitlab runner (which is under my control). The git-lfs server used offers only HTTPS authentication (SSH is not an option). For security reasons, I don't want to commit my credentials into the file .lfsconfig.
For Linux, I found this answer and adapted it for my purpose:
git config --global credential.mydomain.helper '!f() { echo "username=${GIT_LFS_USER}"; echo "password=${GIT_LFS_PASSWORD}"; }; f'
I tried to configure this on the windows gitlab-runner used (pre_clone_script option in config.toml), however I didn't manage to get it to work (I am getting authorization denied from the lfs server) and I am not sure this can work, because as far as I understand on Windows git uses Windows Credential Manager per default.
How can I provide the credentials of an external git-lfs server to run a gitlab-ci job on a Windows gitlab-runner?
I managed to make it work using this syntax in the configuration file config.toml of the windows gitlab runner (replace myusername, myurl, and mypassword with real strings):
pre_clone_script="""git config --global credential.https://myurl.com.username "myusername"
git config --global "credential.https://myurl.com.helper" '!f() { echo password="mypassword"; }; f'
"""
To make it even cleaner, it's possible to use pre_clone_script="Invoke-Command $CI_PRE_CLONE_SCRIPT" and to define CI_PRE_CLONE_SCRIPT in each gitlab's project configuration.

How to connect NPM to azure artifacts feed on Mac?

I'm trying to connect to my private npm feed from Mac. I generated credentials from Connect to feed menu and they looked like that:
; begin auth token
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:username=ANYTHING-BUT-EMPTY
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:_password=BASE64-ENCODED-PAT-GOES-HERE
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:email=npm requires email to be set but doesn't use the value
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/:username=ANYTHING-BUT-EMPTY
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/:_password=BASE64-ENCODED-PAT-GOES-HERE
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/:email=npm requires email to be set but doesn't use the value
; end auth token
I placed that in .npmrc file in my project and it didn't work. When im trying to do npm install I get this error:
code E401
npm ERR! Unable to authenticate, need: Bearer authorization_uri=https://login.windows.net/...,
Basic realm="https://pkgsprodsu3weu.app.pkgs.visualstudio.com/", TFS-Federated
I also placed these credentials in $HOME directory which also didn't solve the issue. What am I doing wrong? In which .npmrc file should they be? Should I run additional commands to use them?
How to connect NPM to azure artifacts feed on Mac?
The .npmrc file which including the credentials should set int the $home directory.
Check the document Use npm to store JavaScript packages in Azure DevOps Services or TFS:
On your development machine, you also have an .npmrc file in $home for
Linux or Mac systems, or $env.HOME for Windows systems. This .npmrc
file should contain credentials for all of the registries that you
need to connect to. The npm client will look at your project's .npmrc
file, discover the registry, and fetch matching credentials from
$home/.npmrc or $env.HOME/.npmrc. The next section will discuss
credential acquisition.
Since it still not work for you, you could check if your npmrc token has expired. In your .npmrc, I found you are using BASE64-ENCODED-PAT-GOES-HERE, it seems you are using the PAT, but in the 90-day token type. The .npmrc file should like:
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:username=ANYTHING-BUT-EMPTY
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:_password=BASE64-ENCODED-PAT-GOES-HERE
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:email=YOUREMAIL#EXAMPLE.COM
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:always-auth=true
Check Create a token that lasts longer than 90 days.
If you still have 401 error, please check if your PAT has expired and convert to Base64String or do not have enough permission.
Hope this helps.
In my case for some reason I needed to surround the base 64 encoded token with double quotes and square bracket to make it work.
//pkgs.dev.azure.com/<yourorganization>/_packaging/<yourfeed>/npm/registry/:_password="[BASE64-ENCODED-PAT-GOES-HERE]"
Note the : "[ and at the end ]".
after adding that, all worked just fine.
I'm running MacOS Big Sur 11.6
To be able to connect to the ADO npm feed without saving the credentials, you can get an access token in ADO, and pass that as a parameter to the following script:
setup-npmrc-feed-auth.bash
#!/bin/bash
DecodedPat=$1
NPMAuthIdent=$(echo -ne "$DecodedPat" | base64);
pnpm config set registry https://pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/registry/ --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/registry/:username {orgName} --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/registry/:_password $NPMAuthIdent --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/registry/:email some#email.com --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/:username {orgName} --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/:_password $NPMAuthIdent --location=global
pnpm config set //pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/:email some#email.com --location=global
in your workspace, have the following:
.npmrc
registry=https://pkgs.dev.azure.com/{orgName}/{projectName}/_packaging/{feedName}/npm/registry/
auto-install-peers=true
strict-peer-dependencies=false
always-auth=true

AWX Failing to install requirements from private repo

I have an AWX installation with a project. The project is fetched perfectly from the private git repo.
The project has a requirements file which is executed by AWX. The requirements file contains private repositories like this:
- src: git+http://gitlab.test.com/ansible/test.git
version: master
name: test
This however fails with the following error:
test was NOT installed successfully: - command git clone failed in directory.
How can I configure AWX to use the git credentials that are in place already in awx?
It seems like AWX works 'smarter' than I thought it would.
In this case, AWX used the same credentials that it used to fetch the project in the first place. I had two options to resolve this:
Change the links to work via git
Update the same git credential and also add the username and password of that git user.

Jenkins Error cloning remote repo 'origin', slave node

I need some help here, It's been a week I'm with this problem, can't figure out what's going on. I'm not able to clone a git repo from a slave node (Jenkins). I added the ssh key, host and slave (I've already tried generating a single key and one for each virtual and host)).
On Jenkins:
url:git#github.com: < repo>
Credentials: Here I tried with username/password, username with ssh file, username with ssh key directly, and -none-.
It doesn't seems like there is an authentication problem since I can clone the repo manually from console (both, slave and host). I can also connect with
ssh -T git#github.com
so the ssh key is fine, but when I build, this appear on console:
Building remotely on IE10Win7 in workspace C:\Users\IEUser\Desktop\<
folder >
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository git#github.com:< repo>.git
git init C:\Users\IEUser\Desktop\< folder> # timeout=10
ERROR: Error cloning remote repo 'origin'
ERROR: Error cloning remote repo 'origin'
Performing Post build task...
Does anyone have an idea? Hope someone can give me a clue, Thanks!
I fixed this problem by setting the slave node tool path, selecting git and setting its value to
C:\Program Files (x86)\Git\bin\git.exe
Location: Configure Node - Tool Locations
I recently updated several jenkins plugins and had this problem after the updates. Rolling back the git plugin didn't help, but I did a few other things to get it working. I listed all three here, but it was probably (2) that fixed the problem. Apparently the git executable was reset to default. So, configuring the git executable within the specific project was probably all that was needed. However the other items might come in handy too.
(1) The default git on a jenkins linux install geenrally points to /usr/lib... You need to specify a separate GitForWindows that points to the windows version:
Manage Jenkins
Configure System
Under Git - Git Installations
Add Git -> Git
Give it a name to be referenced in projects
(mine is WindowsGit)
Set Path to Git Executable
(mine is "C:\Program Files (x86)\Git\bin\git.exe")
(for recent git the path is "C:\Program Files\Git\bin\git.exe")
(2) Configure git on the specific project:
Select the project
Select Configure
Under Source Code Management - Git
Select Git Executable as configured in 1)
Set credentials or add new (ssh keys, etc)
(3) Updating the jenkins slave service to run as a specific user:
Go to Windows Services on the slave -- StartMenu, type "services"
Select the Jenkins Slave service in the list on the right
Right-click and select "Properties" of the Jenkins Slave service
Select the "Log On" tab
Update the username and password used in manual tests
Domain login can be specificied with <DOMAIN>\<USERNAME>
Local logins just use <USERNAME>
OK to save and exit
Right-click again and select "Restart" to make the changes active.
I found a decent workaround in my case. The git clone command always inherits its process owner, which may make a difference, even if the two owners of Jenkins (SYSTEM) and cmd (USER) seem to have the same rights on your system. All other configurations were identical (keys, knownhosts, Git client version).
So as far as I can see, calling git clone from cmd will succeed because it calls the remote as USER, whereas git clone called from Jenkins may be rejected because it calls the remote as SYSTEM. In Services, where you would typically start Jenkins through the GUI, you can configure the service to run as a different user (right click on service --> Properties --> Log On). I had to put it like USER#DOMAIN, e.g. Jenkins#company.local or so. I'm not sure about how a cmd parameter would look, but I'd expect there to be one.
Also, I don't quite know what difference this workaround makes in the end, because on my Jenkins, SYSTEM and USER are configured to have the same rights across the system and they are of course both recognized as "Jenkins" by remote. Still, it does the trick for me. Deeper insights welcome.
I was facing similar problem and found that I need to add git to my PATH environment variable for a Windows based slave. I think that #dhj suggestion 2 might work as well in this case.
I found this workaround on Jenkins Jira.
In my case, I started getting this exact error after updating Git on some of my build machines (via Chocolatey, using the "git.install" package) from 1.9.4 to 2.5.0. The old 1.9.4 install was a 32-bit package but the new one is a 64-bit one, so the default install location switched from C:\Program Files (x86)\Git to C:\Program Files\Git. I had the 64-bit path configured on the Jenkins master (since it had the newer Git version), but some slaves still had the older 32-bit version installed, so the slaves were attempting to use an incorrect path. I could have overridden the Git path for individual slaves but the cleaner solution for me was simply to upgrade all the slaves to the newer 64-bit version.
I tried most of the above:
Specify git location.
Set service user.
Run as Administrator.
None of it worked. Eventually decided to uninstall git64 and install git32... changed the git path to the new location (in x86 Program Files). And everything worked.
I ran into this issue recently.
We had some items in our PATH EV which we had added when trying to connect Winium and Selenium to our Jenkins instance.
We removed these items, but still Jenkins seemed to be holding on to the values. After a bit of troubleshooting: restarting Jenkins; restarting the Jenkins server; setting the EVs at the node level; etc., we restarted the Jenkins JNLP service on the Windows slave.
And they lived happily ever after.
I also got the same error and fixed by adding a git path in the "Manage Jenkins -> Global Tool Configuration".
In my env, The windows Jenkins master run a job on the ubuntu slave machine, the master machine has used the default git ('C:\Program Files\Git\git.exe') path and the slave machine has used the git ('/usr/bin/git') path, therefore, I configured slave git path into the master configuration '/usr/bin/git' and its start working for me

Jenkins configuration of git plugin

I get this error when configuring git repo in a new jenkins project:
Failed to connect to repository : Error performing command: ls-remote -h file:///c:\Dev\git\rx HEAD
I have a local git repo on my windows machine and cloning the same path from command prompt works on the same machine. I don't know what could be the problem (git.exe is on system path)
Windows installs Jenkins as a service. But it runs the Jenkins service under different user permissions that don't have access to GIT. You can follow this link to find the user that Jenkins runs under. http://antagonisticpleiotropy.blogspot.com/2012/08/running-jenkins-in-windows-with-regular.html
For example. My local user name on my windows is "nathandrewsire", exclude the quotes. Open windows services manager (either cmd and c:\services.msc, OR use windows key + R). Enable Log On > This account:. Then type in .\username (e.g. .\nathandrewsire). Then restart the service and you will have access to GIT from Jenkins builds.
I also found a snippet for windows shell command that will output the user to verify it.
echo %PATH%
echo %USERDOMAIN%\%USERNAME%
http://antagonisticpleiotropy.blogspot.com/2012/08/running-jenkins-in-windows-with-regular.html
Linux
If you're using Linux, this error can also be caused from not enabling share on your Git repo. Linux jenkins user won't be able to access Git report under another user unless....
git --bare init --shared=group
Also, your jenkins user and Git repo user must belong to the same group for file permission access. There's other alternatives to that like messing with umasks and ACL's but setting up a linux group for your two users is the easiest way.
Make sure NOT TO OVERWRITE your Git repo with the bare init command. If you already have a Git repo, you'll need to setup/move to a new one with the shared option. I didn't see a way to convert existing repos to shared because I am guessing the files would all need to be updated?
NOTE***
I used VMWare Workstation shared folders to build on a windows guest. Jenkins needs the following format when entering the Git repo.
//vmware-host/Shared Folders/vm-share/ridcyDevelopment.git/
Unable to get Jenkins and Git to play nice

Resources