Jenkins Git pligin slow checkout - jenkins-pipeline

I doing clone outside of Jenkins into some shared folder using LSF workload management platform by IBM
Or let’s say I doing clone manually from command line
Then I want to pull newer changes (commits) and checkout branch using GIT plugin
It works, but takes same time as full clone. Means takes 5-7 minutes instead of 20 seconds
Can you explain this behavior, please ? How I can handle it ?
My local repository created from command line is completely functional. And I want to make checkout in 20 sec
Thank you
Code:
stage('checkout') {
steps {
script {
node('linux4') {
ws("/comm/data_10/some_folder") {
// Git plugin will not pull submodules with --force option,
// if there are local changes it fails the CI (JENKINS-27625, JENKINS-55875).
sh "git submodule foreach --recursive git reset --hard || true"
checkout([
$class: 'GitSCM',
branches: [[name: "develop"]],
doGenerateSubmoduleConfigurations: false,
extensions: [[$class: 'CheckoutOption', timeout: 25 ],
[$class: 'SubmoduleOption',
disableSubmodules: false,
parentCredentials: false,
recursiveSubmodules: true,
reference: '',
trackingSubmodules: true]],
userRemoteConfigs: [[credentialsId: env.CRED, url: env.REPO]]
])
}}}}}
Log (I added debug variable as hudson.plugins.git.GitSCM.verbose=true). Checkout took 6 min:
10:40:29 Using checkout strategy: Default
10:40:29 Last Built Revision: Revision 7588941db5123456975ff1234567ef8733f20b0b (origin/develop)
10:40:29 The recommended git tool is: NONE
10:40:29 using credential f76306ee-b3d1-4169-9582-123467
10:40:29 Fetching changes from the remote Git repository
10:40:28 > git rev-parse --is-inside-work-tree # timeout=10
10:40:28 > git config remote.origin.url <removed_remote_URL> # timeout=10
10:40:28 Fetching upstream changes from <removed_remote_URL>
10:40:28 > git --version # timeout=10
10:40:28 > git --version # 'git version 2.28.1'
10:40:28 using GIT_ASKPASS to set credentials SIRC DevOps
10:40:28 > git fetch --tags --progress -- git#gitlab-srv:CDE-Users/HW_ALG.git +refs/heads/*:refs/remotes/origin/* # timeout=10
10:40:31 getCandidateRevisions(false,develop,,,hudson.plugins.git.util.BuildData#18b394fa[scmName=<null>,remoteUrls=[<removed_remote_URL>],buildsByBranchName={origin/develop=Build #31 of Revision 7588941db57d8597975ff98bb3d7ef8733f20b0b (origin/develop)},lastBuild=Build #31 of Revision 7588941db57d8597975ff98bb3d7ef8733f20b0b (origin/develop)]) considering branches to build
10:40:31 Qualifying develop as a branch in repository origin -> origin/develop
10:40:31 rev-parse origin/develop -> AnyObjectId[7588941db5123456975ff1234567ef8733f20b0b]
10:40:31 Found a new commit AnyObjectId[7588941db5123456975ff1234567ef8733f20b0b] to be built on origin/develop
10:40:31 Checking out Revision 7588941db5123456975ff1234567ef8733f20b0b (origin/develop)
10:40:30 > git rev-parse origin/develop^{commit} # timeout=10
10:40:30 > git config core.sparsecheckout # timeout=10
10:40:30 > git checkout -f 7588941db5123456975ff1234567ef8733f20b0b # timeout=25
10:46:37 Commit message: "Merge branch '__branch_name_removed__' into 'develop'"
10:46:36 > git rev-list --no-walk 7588941db5123456975ff1234567ef8733f20b0b # timeout=10
Environment: OS Linux. Git is running as Docker container
Tested on versions:
Jenkins 2.164.1 git plugin 3.9.3 and
Jenkins 2.263.1 git plugin 4.5.1

Related

go get and AzureDevOps authentication not working

I am desperately trying to figure out why a go get command from a package residing inside a Git repo on AzureDevOps does not work.
Both golang projects (the project where the package resides as well as the consumer trying to import it) are located inside the same AzureDevOps project.
I am attempting to import a package from the one into the other but alas with no avail.
I followed the instructions found here: https://learn.microsoft.com/en-us/azure/devops/repos/git/go-get?view=azure-devops
I have created a PAT and tested that I am able to clone the repo from where I want to import the package from, and that works fine.
git clone https://me:78rt5u89dg635ef4fb30e4b0229e4329756986416673b4e331ag#dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
So I have in my mind (maybe incorrectly so) made the assumption that the PAT is able to authenticate correctly. That said, I unfortunately have to use cntlm proxy to get to the outside, but this works well in general.
In my golang project inside the terminal I have set : export GOPRIVATE=dev.azure.com
In my GitConfig file located at location ~/.gitconfig I have set the following values:
# This is Git's per-user configuration file.
[user]
email = me#my-comp.com
name = Morné Kruger
[core]
editor = vi
[url https://me:78rt5u89dg635ef4fb30e4b0229e4329756986416673b4e331a#dev.azure.com/my-comp/my-proj/_git/k8s-team-operator]
insteadOf = https://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
I then run the go get command with the verbose flag, and then get the following less positive response:
me#focalfossa:~/dev/my-comp/repos/k8s-teamapi-functions$ go get -x dev.azure.com/my-comp/my-proj/_git/k8s-team-operator.git
# get https://dev.azure.com/my-comp/my-proj/_git?go-get=1
# get https://dev.azure.com/?go-get=1
# get https://dev.azure.com/my-comp?go-get=1
cd .
git ls-remote https://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
# get https://dev.azure.com/my-comp/my-proj?go-get=1
cd .
git ls-remote git+ssh://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
# get https://dev.azure.com/?go-get=1: 403 Forbidden (0.274s)
# get https://dev.azure.com/my-comp/my-proj?go-get=1: 203 Non-Authoritative Information (0.307s)
# get https://dev.azure.com/my-comp?go-get=1: 203 Non-Authoritative Information (0.308s)
# get https://dev.azure.com/my-comp/my-proj/_git?go-get=1: 203 Non-Authoritative Information (0.311s)
cd .
git ls-remote ssh://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
mkdir -p /home/me/go/pkg/mod/cache/vcs # git3 git://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
# lock /home/me/go/pkg/mod/cache/vcs/540338d35ef4fb30e4b0229e4329756986416673b4e3318516900fca1efcee17.lock# /home/me/go/pkg/mod/cache/vcs/540338d35ef4fb485763tg30417801e630216673b4e3318516900fca1efcee17 for git3 git://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
cd /home/me/go/pkg/mod/cache/vcs/540338d35ef4fb30e4b02906874yh25630216673b4e3318516900fca1efcee17; git ls-remote -q origin
2.083s # cd /home/me/go/pkg/mod/cache/vcs/540338d35ef49059y87e329e417801e630216673b4e3318516900fca1efcee17; git ls-remote -q origin
go get: module dev.azure.com/my-comp/my-proj/_git/k8s-team-operator.git: git ls-remote -q origin in /home/me/go/pkg/mod/cache/vcs/540338d35ef4fb30e4b022125f7y879430216673b4e3318516900fca1efcee17: exit status 128:
fatal: unable to connect to dev.azure.com:
dev.azure.com[0: 13.107.42.20]: errno=Connection refused
dev.azure.com[1: 2620:1ec:21::20]: errno=Network is unreachable
Versions:
go1.16.5 linux/amd64
Ubuntu 20.04
GoLand 2021.1.2
git version 2.25.1
I would greatly appreciate any tips or troubleshooting ideas that could point out the error in my ways!

Github Action - Error: Process completed with exit code 1

I am trying to setup github actions to deploy my application.
My bash-script to start the deployment process looks like this:
#!/bin/sh
set -e
vendor/bin/phpunit
(git push) || true
git checkout production
git merge master
git push origin production
git checkout master
When executing the above script I get:
My workflows/main.yml file looks like the following:
name: CD
on:
push:
branches: [ production ]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
token: ${{ secrets.PUSH_TOKEN }}
# - name: Set up Node
# uses: actions/setup-node#v1
# with:
# node-version: '12.x'
# - run: npm install
# - run: npm run production
- name: Commit built assets
run: |
git config --local user.email "action#github.com"
git config --local user.name "GitHub Action"
git checkout -B deploy
git add -f public/
git commit -m "Build front-end assets"
git push -f origin deploy
- name: Deploy to production
uses: appleboy/ssh-action#master
with:
username: root
host: xx.xx.xxx.xxx
password: ${{ secrets.SSH_PASSWORD }}
script: 'cd /var/www/myProject && ./server_deploy.sh'
When running the script I get the following raw log:
2021-03-14T16:04:06.7573174Z ##[section]Starting: Request a runner to run this job
2021-03-14T16:04:07.1344029Z Can't find any online and idle self-hosted runner in current repository that matches the required labels: 'ubuntu-latest'
2021-03-14T16:04:07.1344097Z Can't find any online and idle self-hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
2021-03-14T16:04:07.1344652Z Found online and idle hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
2021-03-14T16:04:07.2661405Z ##[section]Finishing: Request a runner to run this job
2021-03-14T16:04:17.0975920Z Current runner version: '2.277.1'
2021-03-14T16:04:17.9082004Z ##[group]Operating System
2021-03-14T16:04:17.9083391Z Ubuntu
2021-03-14T16:04:17.9084015Z 20.04.2
2021-03-14T16:04:17.9084509Z LTS
2021-03-14T16:04:17.9085154Z ##[endgroup]
2021-03-14T16:04:17.9085790Z ##[group]Virtual Environment
2021-03-14T16:04:17.9086576Z Environment: ubuntu-20.04
2021-03-14T16:04:17.9087226Z Version: 20210302.0
2021-03-14T16:04:17.9088413Z Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20210302.0/images/linux/Ubuntu2004-README.md
2021-03-14T16:04:17.9090008Z Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F
2021-03-14T16:04:17.9091225Z ##[endgroup]
2021-03-14T16:04:17.9093732Z ##[group]GITHUB_TOKEN Permissions
2021-03-14T16:04:17.9095423Z Actions: write
2021-03-14T16:04:17.9096121Z Checks: write
2021-03-14T16:04:17.9096752Z Contents: write
2021-03-14T16:04:17.9097367Z Deployments: write
2021-03-14T16:04:17.9098205Z Issues: write
2021-03-14T16:04:17.9098869Z Metadata: read
2021-03-14T16:04:17.9099572Z OrganizationPackages: write
2021-03-14T16:04:17.9100326Z Packages: write
2021-03-14T16:04:17.9100966Z PullRequests: write
2021-03-14T16:04:17.9101747Z RepositoryProjects: write
2021-03-14T16:04:17.9102476Z SecurityEvents: write
2021-03-14T16:04:17.9103267Z Statuses: write
2021-03-14T16:04:17.9103974Z ##[endgroup]
2021-03-14T16:04:17.9108171Z Prepare workflow directory
2021-03-14T16:04:18.3861957Z Prepare all required actions
2021-03-14T16:04:18.3875377Z Getting action download info
2021-03-14T16:04:18.7415606Z Download action repository 'actions/checkout#v2'
2021-03-14T16:04:19.4196162Z Download action repository 'appleboy/ssh-action#master'
2021-03-14T16:04:20.1340765Z ##[group]Build container for action use: '/home/runner/work/_actions/appleboy/ssh-action/master/Dockerfile'.
2021-03-14T16:04:20.1449431Z ##[command]/usr/bin/docker build -t 5588e4:46e0c05d2f1c4fbf93b7f69ca6a2f1b5 -f "/home/runner/work/_actions/appleboy/ssh-action/master/Dockerfile" "/home/runner/work/_actions/appleboy/ssh-action/master"
2021-03-14T16:04:21.6962547Z Sending build context to Docker daemon 293.9kB
2021-03-14T16:04:21.6963066Z
2021-03-14T16:04:21.6963632Z Step 1/4 : FROM appleboy/drone-ssh:1.6.3-linux-amd64
2021-03-14T16:04:21.6964381Z 1.6.3-linux-amd64: Pulling from appleboy/drone-ssh
2021-03-14T16:04:21.6965036Z 188c0c94c7c5: Pulling fs layer
2021-03-14T16:04:21.6965576Z 66fca843054b: Pulling fs layer
2021-03-14T16:04:21.6966101Z f62e9573bab1: Pulling fs layer
2021-03-14T16:04:21.6966632Z 06a0f34100cd: Pulling fs layer
2021-03-14T16:04:21.6967110Z 06a0f34100cd: Waiting
2021-03-14T16:04:21.6967633Z f62e9573bab1: Verifying Checksum
2021-03-14T16:04:21.6968295Z f62e9573bab1: Download complete
2021-03-14T16:04:21.6968871Z 66fca843054b: Verifying Checksum
2021-03-14T16:04:21.6969544Z 66fca843054b: Download complete
2021-03-14T16:04:21.6970079Z 188c0c94c7c5: Verifying Checksum
2021-03-14T16:04:21.6970589Z 188c0c94c7c5: Download complete
2021-03-14T16:04:21.6971347Z 06a0f34100cd: Verifying Checksum
2021-03-14T16:04:21.6971878Z 06a0f34100cd: Download complete
2021-03-14T16:04:22.0484952Z 188c0c94c7c5: Pull complete
2021-03-14T16:04:22.2702074Z 66fca843054b: Pull complete
2021-03-14T16:04:22.3773889Z f62e9573bab1: Pull complete
2021-03-14T16:04:22.6271473Z 06a0f34100cd: Pull complete
2021-03-14T16:04:22.6379226Z Digest: sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaa
2021-03-14T16:04:22.6402600Z Status: Downloaded newer image for appleboy/drone-ssh:1.6.3-linux-amd64
2021-03-14T16:04:22.6418359Z ---> 47f94d42e479
2021-03-14T16:04:22.6422631Z Step 2/4 : COPY entrypoint.sh /entrypoint.sh
2021-03-14T16:04:23.2992272Z ---> 894dff86e37f
2021-03-14T16:04:23.3137154Z Step 3/4 : RUN chmod +x /entrypoint.sh
2021-03-14T16:04:23.3492501Z ---> Running in 8b348a5718bd
2021-03-14T16:04:25.0149456Z Removing intermediate container 8b348a5718bd
2021-03-14T16:04:25.0167277Z ---> 2bc40f78951d
2021-03-14T16:04:25.0168110Z Step 4/4 : ENTRYPOINT ["/entrypoint.sh"]
2021-03-14T16:04:25.0374209Z ---> Running in be0a69eed1dc
2021-03-14T16:04:26.0155734Z Removing intermediate container be0a69eed1dc
2021-03-14T16:04:26.0157218Z ---> e08d3462082b
2021-03-14T16:04:26.0218123Z Successfully built e08d3462082b
2021-03-14T16:04:26.0355661Z Successfully tagged 5588e4:46e0c05d2f1c4fbf93b7f69ca6a2f1b5
2021-03-14T16:04:26.0368493Z ##[endgroup]
2021-03-14T16:04:26.0761189Z ##[group]Run actions/checkout#v2
2021-03-14T16:04:26.0761944Z with:
2021-03-14T16:04:26.0763619Z token: ***
2021-03-14T16:04:26.0764261Z repository: myUser/myProject
2021-03-14T16:04:26.0764899Z ssh-strict: true
2021-03-14T16:04:26.0765450Z persist-credentials: true
2021-03-14T16:04:26.0765987Z clean: true
2021-03-14T16:04:26.0766410Z fetch-depth: 1
2021-03-14T16:04:26.0766837Z lfs: false
2021-03-14T16:04:26.0767254Z submodules: false
2021-03-14T16:04:26.0767708Z ##[endgroup]
2021-03-14T16:04:26.5724570Z Syncing repository: myUser/myProject
2021-03-14T16:04:26.5725598Z ##[group]Getting Git version info
2021-03-14T16:04:26.5726878Z Working directory is '/home/runner/work/myProject/myProject'
2021-03-14T16:04:26.5728059Z [command]/usr/bin/git version
2021-03-14T16:04:26.5892765Z git version 2.30.1
2021-03-14T16:04:26.5894282Z ##[endgroup]
2021-03-14T16:04:26.5903587Z Deleting the contents of '/home/runner/work/myProject/myProject'
2021-03-14T16:04:26.5905407Z ##[group]Initializing the repository
2021-03-14T16:04:26.5909187Z [command]/usr/bin/git init /home/runner/work/myProject/myProject
2021-03-14T16:04:26.6029092Z hint: Using 'master' as the name for the initial branch. This default branch name
2021-03-14T16:04:26.6030401Z hint: is subject to change. To configure the initial branch name to use in all
2021-03-14T16:04:26.6031533Z hint: of your new repositories, which will suppress this warning, call:
2021-03-14T16:04:26.6032386Z hint:
2021-03-14T16:04:26.6033943Z hint: git config --global init.defaultBranch <name>
2021-03-14T16:04:26.6034638Z hint:
2021-03-14T16:04:26.6035442Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2021-03-14T16:04:26.6036651Z hint: 'development'. The just-created branch can be renamed via this command:
2021-03-14T16:04:26.6037336Z hint:
2021-03-14T16:04:26.6037909Z hint: git branch -m <name>
2021-03-14T16:04:26.6041711Z Initialized empty Git repository in /home/runner/work/myProject/myProject/.git/
2021-03-14T16:04:26.6053565Z [command]/usr/bin/git remote add origin https://github.com/myUser/myProject
2021-03-14T16:04:26.6112480Z ##[endgroup]
2021-03-14T16:04:26.6113335Z ##[group]Disabling automatic garbage collection
2021-03-14T16:04:26.6121163Z [command]/usr/bin/git config --local gc.auto 0
2021-03-14T16:04:26.6165369Z ##[endgroup]
2021-03-14T16:04:26.6176967Z ##[group]Setting up auth
2021-03-14T16:04:26.6187245Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2021-03-14T16:04:26.6246287Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2021-03-14T16:04:26.6900209Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2021-03-14T16:04:26.6940390Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2021-03-14T16:04:26.7216178Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2021-03-14T16:04:26.7266677Z ##[endgroup]
2021-03-14T16:04:26.7268826Z ##[group]Fetching the repository
2021-03-14T16:04:26.7277912Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +ea9bc17ea94e3e6c966f65eadaab92e81b90346c:refs/remotes/origin/production
2021-03-14T16:04:27.4292763Z remote: Enumerating objects: 2050, done.
2021-03-14T16:04:27.5051531Z remote: Counting objects: 100% (2050/2050), done.
2021-03-14T16:04:27.6195514Z remote: Compressing objects: 100% (1621/1621), done.
2021-03-14T16:04:28.5762484Z Receiving objects: 100% (2050/2050), 2.44 MiB | 4.84 MiB/s
2021-03-14T16:04:28.5763176Z Receiving objects: 100% (2050/2050), 13.82 MiB | 14.48 MiB/s, done.
2021-03-14T16:04:28.6012285Z Resolving deltas: 100% (469/469)
2021-03-14T16:04:28.6012872Z Resolving deltas: 100% (469/469), done.
2021-03-14T16:04:28.8206899Z From https://github.com/myUser/myProject
2021-03-14T16:04:28.8208438Z * [new ref] ea9bc17ea94e3e6c966f65eadaab92e81b90346c -> origin/production
2021-03-14T16:04:28.8229099Z ##[endgroup]
2021-03-14T16:04:28.8229852Z ##[group]Determining the checkout info
2021-03-14T16:04:28.8232088Z ##[endgroup]
2021-03-14T16:04:28.8232766Z ##[group]Checking out the ref
2021-03-14T16:04:28.8236775Z [command]/usr/bin/git checkout --progress --force -B production refs/remotes/origin/production
2021-03-14T16:04:28.9951267Z Switched to a new branch 'production'
2021-03-14T16:04:28.9952452Z Branch 'production' set up to track remote branch 'production' from 'origin'.
2021-03-14T16:04:28.9961000Z ##[endgroup]
2021-03-14T16:04:29.0014527Z [command]/usr/bin/git log -1 --format='%H'
2021-03-14T16:04:29.0046451Z 'ea9bc17ea94e3e6c966f65eadaab92e81b90346c'
2021-03-14T16:04:29.0162817Z ##[group]Run git config --local user.email "action#github.com"
2021-03-14T16:04:29.0163849Z [36;1mgit config --local user.email "action#github.com"[0m
2021-03-14T16:04:29.0164730Z [36;1mgit config --local user.name "GitHub Action"[0m
2021-03-14T16:04:29.0165375Z [36;1mgit checkout -B deploy[0m
2021-03-14T16:04:29.0165987Z [36;1mgit add -f public/[0m
2021-03-14T16:04:29.0166580Z [36;1mgit commit -m "Build front-end assets"[0m
2021-03-14T16:04:29.0167255Z [36;1mgit push -f origin deploy[0m
2021-03-14T16:04:29.0215326Z shell: /usr/bin/bash -e {0}
2021-03-14T16:04:29.0215807Z ##[endgroup]
2021-03-14T16:04:29.3153383Z Switched to a new branch 'deploy'
2021-03-14T16:04:29.3313550Z On branch deploy
2021-03-14T16:04:29.3316648Z nothing to commit, working tree clean
2021-03-14T16:04:29.3381472Z ##[error]Process completed with exit code 1.
2021-03-14T16:04:29.3653154Z Post job cleanup.
2021-03-14T16:04:29.4723616Z [command]/usr/bin/git version
2021-03-14T16:04:29.4777037Z git version 2.30.1
2021-03-14T16:04:29.4815499Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2021-03-14T16:04:29.4854852Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2021-03-14T16:04:29.5117676Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2021-03-14T16:04:29.5149269Z http.https://github.com/.extraheader
2021-03-14T16:04:29.5160054Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2021-03-14T16:04:29.5197186Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2021-03-14T16:04:29.5497217Z Cleaning up orphan processes
Currently nothing gets committed on my production branch, as the production action does not start.
Any suggestions what I am doing wrong?
Why do I get this error?
I really appreciate your replies!
Why do I get this error?
"This error", to be clear, is:
2021-03-14T16:04:29.3381472Z ##[error]Process completed with exit code 1.
The process that completed is the shell:
2021-03-14T16:04:29.0215326Z shell: /usr/bin/bash -e {0}
which ran some commands in a script; obviously one of those commands must have exited with status code 1 ("generic failure"), which led bash -e to also exit with status code 1.
We can see each command's output, so we can correlate each command to its output based on knowing what commands print what stuff. Here is the script:
git config --local user.email "action#github.com"
git config --local user.name "GitHub Action"
git checkout -B deploy
git add -f public/
git commit -m "Build front-end assets"
git push -f origin deploy
(I took out the color-changing ESC [ ... m sequences since they just get in the way, and stripped the timestamps since they're not really relevant.) You should recognize these commands: they are the ones you supplied in your YAML file.
Here is the output from each of those commands:
Switched to a new branch 'deploy'
On branch deploy
nothing to commit, working tree clean
Now, git config normally prints nothing, so the two git configs probably succeeded because we saw something printed (the first line). What we saw printed was Switched to a new branch 'deploy'.
This is exactly what the third command, git checkout -B deploy, would print when it succeeds. So we must have reached the third command, which succeeded. That would let us move on to the fourth command, git add -f public/.
The git add command normally prints nothing. So we can't really be sure if it worked or not at this point, but we do see further output, beginning with On branch .... That definitely doesn't come from git add, so presumably git add succeeded, which would let us move on to the fifth command, git commit -m [message]. This command does print things, on both success and failure: on success we get a message showing the new commit's abbreviated hash ID, and on failure, we get git status output:
$ git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
$ git commit -m test
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
This is, in fact, a failure of sorts: there is nothing to commit, so Git does not make a new commit. Why is there nothing to commit? Because Git's index matches the current (HEAD) commit. That is, git add -f public/ updated no files. Either public/ is empty, or the files inside public/ match those in the HEAD commit.
If we check the actual exit status from a git commit that fails in this way, it is 1:
$ echo $?
1
So the git commit command is the one that failed, and it failed because there was nothing new to commit.
You have multiple choices for how to proceed:
Figure out why there was nothing to commit, when you clearly expected there to be something to commit.
Check whether there is anything to commit, and if not, commit nothing.
Use git commit --allow-empty to instruct Git to make a new commit even if there's no need for a new commit. (This has the advantage of being extremely simple.)
Forcibly assume that the git commit succeeds even when it fails, by using the (cmd || true) construct, just as you did here:
(git push) || true
I put these in approximate order of increasing badness, though some of this is a matter of opinion / judgment.
Note that when diagnosing some script's failure, there are several approaches that can help if it's not immediately obvious which command within the script failed. For instance, here, it was clear that git commit failed, because only git commit winds up acting like git status, and it does that precisely when it's going to fail with a "nothing to commit" message. But what if it were some other, more-ambiguous failure? One can narrow down these failures by adding echo commands, so that we know which part(s) of the script are running. Or, if it doesn't interfere too much with other things, we can set -x in a bash script: this causes the shell to print out, to standard error output, a + line for each command it actually runs, just before running it. This would immediately pin the exit status to the command that produced it.

NPM install keeps failing in the same place in when building Docker container on Ubuntu EC2

I've been trying to create some build automation scripts to be ran by a job on my Jenkins server on an Ubuntu EC2 instance. I want the script to remove any loose containers and images, then build the new version of my image and replace the running container. Here's my build script:
# Granting superuser
sudo -i
# Initializing Variables
_imagetag=nurenui-v1
# Initializing functions
buildImage()
{
docker build -t "$_imagetag" .
}
removeOldImage()
{
docker rmi $_imagetag
}
pruneImagesAndContainers()
{
docker container prune --force
docker image prune --force
}
pruneImagesAndContainers
if [ "$(docker images $_imagetag)" == "" ]; then
buildImage
else
removeOldImage
buildImage
fi
There were a few jobs that I ran that worked very well. But for some reason, today, the build keeps failing on NPM install in my Dockerfile:
FROM node as build-stage
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build
FROM nginx:stable-alpine as production-stage
COPY --from=build-stage /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
NPM install works fine on my machine, but I can't test docker on it because I am still on Win8. If i try to build the docker image inside or outside of jenkins, I always get this:
17:30:22 Started by GitHub push by ObsidianTech
17:30:22 Building in workspace /var/lib/jenkins/workspace/UI/NuRenUI
17:30:22 [WS-CLEANUP] Deleting project workspace...
17:30:22 [WS-CLEANUP] Deferred wipeout is used...
17:30:22 [WS-CLEANUP] Done
17:30:22 using credential ab9c5d12-0a8d-4edb-9be4-83813cf29068
17:30:22 Cloning the remote Git repository
17:30:22 Cloning repository https://github.com/ObsidianTech/NuRenPivot.git
17:30:22 > git init /var/lib/jenkins/workspace/UI/NuRenUI # timeout=10
17:30:22 Fetching upstream changes from https://github.com/ObsidianTech/NuRenPivot.git
17:30:22 > git --version # timeout=10
17:30:22 using GIT_ASKPASS to set credentials
17:30:22 > git fetch --tags --progress https://github.com/ObsidianTech/NuRenPivot.git +refs/heads/*:refs/remotes/origin/*
17:30:23 > git config remote.origin.url https://github.com/ObsidianTech/NuRenPivot.git # timeout=10
17:30:23 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
17:30:23 > git config remote.origin.url https://github.com/ObsidianTech/NuRenPivot.git # timeout=10
17:30:23 Fetching upstream changes from https://github.com/ObsidianTech/NuRenPivot.git
17:30:23 using GIT_ASKPASS to set credentials
17:30:23 > git fetch --tags --progress https://github.com/ObsidianTech/NuRenPivot.git +refs/heads/*:refs/remotes/origin/*
17:30:24 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
17:30:24 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
17:30:24 Checking out Revision 53e55914222b3f53f173e313d2dc8c7c10746083 (refs/remotes/origin/master)
17:30:24 > git config core.sparsecheckout # timeout=10
17:30:24 > git checkout -f 53e55914222b3f53f173e313d2dc8c7c10746083
17:30:24 Commit message: "trying to fix npm issue"
17:30:24 > git rev-list --no-walk b01ea4d30b391f700a0c597a20bbb5ff6e6f552d # timeout=10
17:30:24 [NuRenUI] $ /bin/sh -xe /tmp/jenkins6764800963704352013.sh
17:30:24 + ls
17:30:24 Dockerfile
17:30:24 README.md
17:30:24 babel.config.js
17:30:24 package-lock.json
17:30:24 package.json
17:30:24 public
17:30:24 qaBuild.sh
17:30:24 qaDeploy.sh
17:30:24 src
17:30:24 + chmod 777 ./qaBuild.sh
17:30:24 + chmod 777 ./qaDeploy.sh
17:30:24 + ./qaBuild.sh
17:30:28 Deleted Containers:
17:30:28 9cd52dadac2f379dffc630d01f0ade2069b0a5683a0dce2c2261490529f75db6
17:30:28
17:30:28 Total reclaimed space: 138.6MB
17:30:28 Deleted Images:
17:30:28 deleted: sha256:0fbd6c6b36a785ecebe8f9ecc971e7645ef209a7a62de4996b773a37bee6b4a4
17:30:28 deleted: sha256:ba7d8f555b9eb7cf15dbda40569efe6171bddeeeaa0deced3816217690943f21
17:30:28 deleted: sha256:0ce9c7023bc64f5798954340ae9b44d9afe231b3e724351059dbd275e9e98c87
17:30:28 deleted: sha256:8b626fa0dcd67eab716ed8999e0fdbfe063c630685f7a77dfb592f15df93198c
17:30:28
17:30:28 Total reclaimed space: 421.3kB
17:30:28 ./qaBuild.sh: 23: [: REPOSITORY TAG IMAGE ID CREATED SIZE
17:30:28 nurenui-v1 latest c6fd05d82c12 13 hours ago 43.7MB: unexpected operator
17:30:28 Error response from daemon: conflict: unable to remove repository reference "nurenui-v1" (must force) - container e300a467c2e0 is using its referenced image c6fd05d82c12
17:30:29 Sending build context to Docker daemon 67.87MB
17:30:29 Step 1/10 : FROM node:latest as build-stage
17:30:29 ---> 502d06d3bfdf
17:30:29 Step 2/10 : WORKDIR /app
17:30:29 ---> Running in d12805e29d2e
17:30:30 Removing intermediate container d12805e29d2e
17:30:30 ---> 4f573ab1b889
17:30:30 Step 3/10 : COPY package*.json ./
17:30:30 ---> 9fc9cb7ed731
17:30:30 Step 4/10 : RUN npm install
17:30:30 ---> Running in 70fe3d18ecc4
17:31:03
17:31:03 > yorkie#2.0.0 install /app/node_modules/yorkie
17:31:03 > node bin/install.js
17:31:03
17:31:03 [91m
17:31:03 [0mThe command '/bin/sh -c npm install' returned a non-zero code: 1
17:31:05 Build step 'Execute shell' marked build as failure
17:31:05 Finished: FAILURE
I keep getting some kind of error about yorkie before it fails. I have no idea why this is happening or how to fix it.
Edit: package.json
{
"name": "nurenpivot",
"version": "0.1.0",
"private": true,
"scripts": {
"serve": "vue-cli-service serve",
"build": "vue-cli-service build",
"lint": "vue-cli-service lint"
},
"dependencies": {
"core-js": "^2.6.5",
"vue": "^2.6.6",
"vue-router": "^3.0.4",
"vuex": "^3.1.0"
},
"devDependencies": {
"#vue/cli-plugin-babel": "^3.5.0",
"#vue/cli-plugin-eslint": "^3.5.0",
"#vue/cli-service": "^3.5.0",
"babel-eslint": "^10.0.1",
"eslint": "^5.8.0",
"eslint-plugin-vue": "^5.0.0",
"vue-template-compiler": "^2.5.21"
},
"eslintConfig": {
"root": true,
"env": {
"node": true
},
"extends": [
"plugin:vue/essential",
"eslint:recommended"
],
"rules": {},
"parserOptions": {
"parser": "babel-eslint"
}
},
"postcss": {
"plugins": {
"autoprefixer": {}
}
},
"browserslist": [
"> 1%",
"last 2 versions",
"not ie <= 8"
]
}
Looking at your logs (before you deleted them from the original question), I could see that you have several errors in your build script way before the step in the Dockerfile where you do the npm install. For example, sudo was failing, the test with [ was also failing, to mention some.
This build script is actually very badly written, as it never checks for errors. Any problem would be simply ignored. When an important command can fail, you have to check the return code ($?) to be equal to zero, before you proceed. Another option you have is to stop on any error using set -e at the very beginning of the script.
As for the other errors, there's not enough information to help you. Please update the question with the complete logs, as they were before (just remove the sensitive parts) and with the contents of your package.json. It's also needed to check the npm and nodejs version running on Jenkins and on your machine, to check if they match.
Edit after logs were re-added to the question:
Looking at these lines of your logs:
17:30:28 ./qaBuild.sh: 23: [: REPOSITORY TAG IMAGE ID CREATED SIZE
17:30:28 nurenui-v1 latest c6fd05d82c12 13 hours ago 43.7MB: unexpected operator
17:30:28 Error response from daemon: conflict: unable to remove repository reference "nurenui-v1" (must force) - container e300a467c2e0 is using its referenced image c6fd05d82c12
It became clear that you are using the raw output from the command docker images $_imagetag, which is definitely not what you want. If you try this command, you'll see that the output contains a lot of other information, like a header, size, tag, etc. Even if the image doesn't exist, it outputs the header line. It's literally what the logs are telling you, something like this:
REPOSITORY TAG IMAGE ID CREATED SIZE
nurenui-v1 latest c6fd05d82c12 13 hours ago 43.7MB
To do that check, you have to add -q to the command. This way it won't output anything if the image is not found. Change that line to look like this:
if [ -n "$(docker images -q $_imagetag)" ]; then
removeOldImage
fi
buildImage
... but this is not the only problem you have. I can also see a container using that image... and I didn't even check the package.json.

Git interactive commands fail to complete with message "The system cannot find the path specified" on MinGw64

I'm trying to perform git add --patch -- <file> and the command never completes. The MINGW64 shell just displays "The system cannot find the path specified."
When narrowing down the cause by setting GIT_TRACE=1 and trying to run some of the component commands, I found that the error is caused by the --interactive option and the same effect is seen no matter what git command is being run.
Setting GIT_TRACE=1 on the following interactive command gives the output shown:
$ GIT_TRACE=1 git add --interactive -- .gitignore
19:16:47.185964 git.c:350 trace: built-in: git 'add' '--interactive' '--' '.gitignore'
19:16:47.185964 run-command.c:336 trace: run_command: 'add--interactive' '--' '.gitignore'
19:16:47.215964 git.c:564 trace: exec: 'git-add--interactive' '--' '.gitignore'
19:16:47.215964 run-command.c:336 trace: run_command: 'git-add--interactive' '--' '.gitignore'
19:16:47.345964 git.c:350 trace: built-in: git 'rev-parse' '--git-dir'
19:16:47.375964 git.c:350 trace: built-in: git 'rev-parse' '--show-prefix'
19:16:47.405964 git.c:350 trace: built-in: git 'config' '--get-colorbool' 'color.interactive' 'false'
19:16:47.436965 git.c:350 trace: built-in: git 'config' '--get-colorbool' 'color.diff' 'false'
19:16:47.466965 git.c:350 trace: built-in: git 'config' '--get-color' 'reset'
19:16:47.496965 git.c:350 trace: built-in: git 'config' '--get' 'diff.algorithm'
19:16:47.546965 git.c:350 trace: built-in: git 'config' '--bool' '--get' 'diff.compactionheuristic'
19:16:47.576965 git.c:350 trace: built-in: git 'config' '--get' 'interactive.difffilter'
19:16:47.606965 git.c:350 trace: built-in: git 'config' '--bool' '--get' 'interactive.singlekey'
19:16:47.636965 git.c:350 trace: built-in: git 'rev-parse' '--git-dir'
19:16:47.666965 git.c:350 trace: built-in: git 'update-index' '--refresh'
19:16:47.793965 git.c:350 trace: built-in: git 'ls-files' '--' '.gitignore'
The system cannot find the path specified.
19:16:47.843965 git.c:350 trace: built-in: git 'diff-index' '--cached' '--numstat' '--summary' '4b825dc642cb6eb9a060e54bf8d69288fbee4904' '--' '.gitignore'
19:16:47.873966 git.c:350 trace: built-in: git 'diff-files' '--numstat' '--summary' '--raw' '--' '.gitignore'
Trying to add the file without the interactive option works as expected. The operation only fails when there is an interactive (or patch) option added.
Note that I initially had the following issue when trying to run the command:
Can't locate Git.pm in #INC (you may need to install the Git module) (#INC contains: /mingw64/share/perl5/site_perl D:/Apps/StrawberryPerl/perl/site/lib D:/Apps/StrawberryPerl/perl/vendor/lib D:/Apps/StrawberryPerl/perl/lib) at C:\Program Files\Git\mingw64/libexec/git-core\git-add--interactive line 7.
BEGIN failed--compilation aborted at C:\Program Files\Git\mingw64/libexec/git-core\git-add--interactive line 7.
However after copying the Git.pm and Error.pm modules to the perl include folder, the problem changed to the one described at the top of this question.
Finally, as a further attempt at figuring this out I ran the command in a Windows command prompt (cmd.exe) and had a totally different error:
c:\code\proj>git add --patch .gitignore
error: wrong number of arguments
usage: git config [<options>]
Config file location
--global use global config file
--system use system config file
--local use repository config file
-f, --file <file> use given config file
--blob <blob-id> read config from given blob object
Action
--get get value: name [value-regex]
--get-all get all values: key [value-regex]
--get-regexp get values for regexp: name-regex [value-regex]
--get-urlmatch get value specific for the URL: section[.var] URL
--replace-all replace all matching variables: name value
[value_regex]
--add add a new variable: name value
--unset remove a variable: name [value-regex]
--unset-all remove all matches: name [value-regex]
--rename-section rename section: old-name new-name
--remove-section remove a section: name
-l, --list list all
-e, --edit open an editor
--get-color find the color configured: slot [default]
--get-colorbool find the color setting: slot [stdout-is-tty]
Type
--bool value is "true" or "false"
--int value is decimal number
--bool-or-int value is --bool or --int
--path value is a path (file or directory name)
Other
-z, --null terminate values with NUL byte
--name-only show variable names only
--includes respect include directives on lookup
--show-origin show origin of config (file, standard input, blob, command line)
config --get-color color.interactive.prompt bold blue: command returned error: 129
I checked my git config files and I don't set this option so I'm guessing it's a default that's set somewhere.
Apart from the obvious aspect of the question (i.e. getting the --interactive or --patch options to work as expected), I'd also appreciate any advice or tips on debugging this further. The various TRACE options I used didn't really highlight where the problem was, apart from showing me that it was the --interactive option that was the issue.
In short - the answer here was to upgrade the version of Git for Windows used. My version was 2.10.1.windows.1 which is a couple of years old. I upgraded to 2.19.0.windows.1 and all is now working as it should.
So the issue might be because there was an error in that version of git, in which case hopefully this will give anyone else using that version a heads-up. Alternatively, there could have been an issue with my installation somewhere along the line, which was automatically corrected by the new git installation. In that case, consider this an idea of things to try if you are having similar problems.
Not a particularly fulfilling answer - but an answer nonetheless.

Git remote fetch origin is already added in .git/config file, still need to set "remote.origin.fetch" every time in git bash before doing "git fetch"

My .git/config file in my git repo already contains below setting:
[remote "origin"]
url = //URL to remote repo
fetch = +refs/heads/*:refs/remote/origin/*
Still when I try git fetch in my git bash, it returns nothing.
But if I run git config remote.origin.fetch "+refs/heads/*:refs/remote/origin/*" in my git bash before doing git fetch, it list all the required remote branches details.
Why do I have to set it every time before doing git fetch? Is there any way I don't have to do that every time I do a git fetch?
I just tested it on a local clone of the Git repo (https://github.com/git/git/)
The first fetch does return nothing
vonc#voncvb C:\Users\vonc\prog\git\git
> git fetch
The config is already set to the default value:
vonc#voncvb C:\Users\vonc\prog\git\git
> git config remote.origin.fetch
+refs/heads/*:refs/remotes/origin/*
If I set again the config, and do a git fetch, I do see indeed additional information:
vonc#voncvb C:\Users\vonc\prog\git\git
> git config remote.origin.fetch "+refs/heads/*:refs/remote/origin/*"
vonc#voncvb C:\Users\vonc\prog\git\git
> git fetch
From https://github.com/git/git
* [new branch] master -> refs/remote/origin/master
* [new branch] maint -> refs/remote/origin/maint
* [new branch] next -> refs/remote/origin/next
* [new branch] pu -> refs/remote/origin/pu
* [new branch] todo -> refs/remote/origin/todo
But if I repeat the git config command again (git config remote.origin.fetch "+refs/heads/*:refs/remote/origin/*") and do additional git fetch, they won't return any new information (empty output)
That is because the first one completed the FETCH_HEAD:
vonc#voncvb C:\Users\vonc\prog\git\git
> more .git\FETCH_HEAD
e3331758f12da22f4103eec7efe1b5304a9be5e9 branch 'master' of https://github.com/git/git
fc54c1af3ec09bab8b8ea09768c2da4069b7f53e not-for-merge branch 'maint' of https://github.com/git/git
fac676dfb9e0fea308fef5113ed7db2362fcdc9e not-for-merge branch 'next' of https://github.com/git/git
5fc3b55b9a8ca42699202482765d9768020ee313 not-for-merge branch 'pu' of https://github.com/git/git
be70464a91c93a321dc16707dec820870df970d2 not-for-merge branch 'todo' of https://github.com/git/git
In other words, it does not fetch any data, it only updates the references already fetched.
See "What does FETCH_HEAD in Git mean?"

Resources