Background
I need to close multiple Git branches (hundreds) that have been left open on a remote repository.
I wanted to sort them by last-commit and put them in a text-file so others could confirm they were no longer needed.
I found a method that allowed me to dump them in a way that I could distribute (and open in Excel to sort by date)
git for-each-ref --format='%(committerdate:short) %09 %(authorname) %09 %(refname)' | sort -k5n -k2M -k3n -k4n >> branches.txt
And then I need to read the updated text-file back in and delete the remote branches:
#!/bin/bash
#prefix of the branches if they are all remote
prefix="refs/remotes/origin/"
prefix1="refs/remotes/"
# path to branches, compiled with:
# git for-each-ref --format='%(committerdate:short) %09 %(authorname) %09 %(refname)' | sort -k5n -k2M -k3n -k4n >> branches.txt
# read through entire input
input="branches.txt"
while IFS= read -r line
do
# echo "$line"]
IFS=' ' read -r -a array <<< "$line"
# search the array for the prefix of the branch
for index in "${!array[#]}"
do
if [[ ${array[$index]} == *"refs/remotes/origin"* ]]; then
#echo -e " \e[34m ${array[$index]} \e[39m Last commit: \e[34m" ${array[0]} ${array[1]} ${array[2]} ${array[3]} ${array[4]}
# echo ${array[$index]#"$prefix"}
#remove the prefix if they are not pulled locally
branch=${array[$index]#"$prefix"}
echo $(git push origin --delete $branch)
fi
done
done < "$input"
Issue
However, keep getting an error that the refspec does not exist:
fatal: invalid refspec ':PracticeBranch?'
I don't have the ' or ? in the name of the branch variable - so this is probably my ignorance of using echo with `bash but I don't know where it is coming from.
A command like:
git push origin --delete PracticeBranch
When calling directly from the shell?
Related
I am using a CI/CD system to automate the building of Docker Images from a git repository. The Image Tag of the image corresponds to the short (i.e. 8-characters) hash of the corresponding git commit, e.g. myimage:123456ab.
The repository contains source code that gets packaged in the Docker Image and stuff like documentation and deployment configuration that is excluded using a .dockerignore file (similar to .gitignore).
While the process works in general, it leads to rebuilding and redeploying Docker Images that are absolute identical, because the only changes were made to files that did not become part of the Image (e.g. the repositories README).
Using only the shell (bash in this case), git and standard *nix tools, is there a way to get the short hash of the latest commit that changed a file which is not ignored by the .dockerignore file? This should as well cover removing a non-ignored file.
You can do this through a combination of git log and git show.
The following script will go backwards through the revision history and find the first commit to have a change that would not be ignored by .dockerignore
for commit in $(git log --pretty=%H)
do
# Get the changed file names for the commit.
# Use `sed 1d` to remove the first line, which is the commit description
files=$(git show $commit --oneline --name-only | sed 1d)
if docker-check-ignore $files
then
echo $commit
exit 0
fi
done
exit 1
And then you could define docker-check-ignore as a script like the following:
#!/bin/sh
DIR=$(mktemp -d)
pushd $DIR
# Set up a temporary git repository so we can use
# git check-ignore with .dockerignore
git init
popd
cp .dockerignore $DIR/.gitignore
pushd $DIR
git check-ignore $#
# Store the error code
ERROR=$?
popd
rm -rf $DIR
exit $ERROR
I will leave reducing the number of file system operations rather than creating/removing a directory for each commit.
#!/usr/bin/env bash
declare -a ign_table=()
# Populates ign_table with patterns from .dockerignore
while IFS= read -r line || [[ ${line} ]]; do
ign_table+=("${line}")
done < <(sed '/^#/d;/^$/d' .dockerignore)
is_docker_ignored() {
locale -i ignore=1 # false, default not ignored
for ign_patt in "${ign_table[#]}"; do
# If pattern starts with ! it is an exception rule
# when filename match !pattern, do not ignore it
# shellcheck disable=SC2053 # $ign_patt must not use quotes to match wildcards
if [[ ${ign_patt} =~ ^\!(.*) ]] && [[ ${1} == ${BASH_REMATCH[1]} ]]; then
return 1 # false: no need to check further patterns, file not ignored
fi
# Normal exclusion pattern, if file match,
# shellcheck disable=SC2053 # $ign_patt must not use quotes to match wildcards
if [[ ${1} == $ign_patt ]]; then
ignore=0 # true: it match an ignore pattern, file may not be ignored if it later matches an exception pattern
fi
done
return "${ignore}"
}
while IFS= read -r file
do
is_docker_ignored "${file}" && continue # File is in .dockerignore
commit_hash="$(git rev-list --all -1 "${file}")"
printf '%s\n' "${commit_hash:0:8}"
done < <(git ls-files)
I am currently doing this:
while read l
do
echo git add $l/
git add $l/
# sed -i -e '1,1d' data/commit-folders.csv
# echo git commit -am "'Autocommit'"
# git commit -uno -am "'Autocommit'"
# echo git push origin master
# git push origin master
done < data/commit-folders.csv
Essentially just git add <folder> for a list of folders in a CSV file. I would like for this to be more robust, in that every time it restarts it restarts from where it left off. So I added that commented out line which does an in-place delete sed -i -e '1,1d' data/commit-folders.csv. However, with while read line, it messes up with the current line if they are being deleted. So I'm wondering how to do this properly.
How to iterate through a CSV file with <path> on each line, and delete the path once it is successfully git added. It seems like you need to have a loop that selects the first line from a file, and then deletes it from the file afterwards, rather than using while read line.
Here a solution without sed.
#!/bin/bash
csv="data/commit-folders.csv"
done="$(mktemp)"
# autoremove tempfile at exit
trap 'rm "$done"' EXIT
# loop over all lines in csv
while read -r file; do
printf "git add %s\n" "$file"
git add "$file"
# write processed files in tempfile
printf "%s\n" "$file" >> "$done"
#...
done < "$csv"
# create tempfile for merge result
newfile="$(mktemp)"
# sort: merge and sort $cvs with $done
# uniq -u: write only unique files into tempfile
sort "$csv" "$done" | uniq -u > "$newfile"
# override $csv with tempfile
mv "$newfile" "$csv"
you can use sed -i "/${l}/d" , it will find the exact line and delete it. This assumes that there would be no duplicate lines.
while read l
do
echo git add $l/
git add $l/
# sed -i -e '1,1d' data/commit-folders.csv
sed -i "/${l}/d" data/commit-folders.csv
# echo git commit -am "'Autocommit'"
# git commit -uno -am "'Autocommit'"
# echo git push origin master
# git push origin master
done < data/commit-folders.csv
Good day to all:
I am writing a bash script to pull the data from git repos. I have an array to store the repository names:
declare -a arr=(
https://"$USER"#stash/repo1.git
https://"$USER"#stash/rep2.git
)
I process it as follows:
for i in "${arr[#]}"
do
git clone "$i"
......
git install
done
This works. Now I want to be able to optionally specify a git branch:
declare -a arr=(
-b branch1 --single-branch https://"$USER"#stash/repo1.git
https://"$USER"#stash/rep2.git
)
The array processing script fails with multiple ugly errors:
- switch `b' requires a value
- not valid identifier, etc
What is the right and simple approach to make it work?
c.f. printf "%s\n" "${arr[#]}". Just because those args are on one line doesn't make them one element. :)
You are trying to do two things with one array.
Try this -
declare -a arr=(
"https://$USER#stash/repo1.git#branch1"
"https://$USER#stash/rep2.git"
)
for i in "${arr[#]}"
do IFS='#' read url branch <<< "$i"
if [[ -n "$branch" ]]
then git clone -b "$branch" --single-branch "$url"
else git clone "$url"
: ......
git install
done
Warning - no chance to test this. Caveat Scriptor - and of course this means # can't be in your url anywhere.
Maybe it would work if you remove quotes around
git clone "$i"
So it would look like this:
git clone $i
I think that with quotes shell should treat all your line as one single argument instead of many:
git clone "-b branch1 --single-branch https://"$USER"#stash/repo1.git"
This looks wrong.
EDIT:
#CharlesDuffy pointed to another problem I missed: you should also have quotes around the entire line in the array definition (but this won't work if you really need internal quotes around $USER, see example below):
declare -a arr=(
"-b branch1 --single-branch https://$USER#stash/repo1.git"
https://"$USER"#stash/rep2.git
)
I checked this on my local machine, and it seems to work.
$ ls
one/ test.sh*
$ ( cd one/ ; git branch )
* another_branch
master
$ cat test.sh
#!/bin/bash
declare -a arr=(
"-b another_branch --single-branch one two"
)
for i in "${arr[#]}"
do
git clone $i
done
$ ls
one/ test.sh*
$ ./test.sh
Cloning into 'two'...
done.
$ ls
one/ test.sh* two/
$ ( cd two/ ; git branch )
* another_branch
$
EDIT2:
This will only work if you can safely omit internal quotes around $USER. If you need them you should use eval inside the for loop and also quote the internal quotes in array declaration.
$ cat ./test.sh
#!/bin/bash
USER="username containing spaces" # just for example!
git () {
echo "$5"
}
declare -a arr=(
"-b branch1 --single-branch https://\"$USER\"#stash/repo1.git"
)
for i in "${arr[#]}"
do
printf "without eval:\t"
git clone $i
printf "with eval:\t"
eval "git clone $i"
done
$ ./test.sh
without eval: https://"username
with eval: https://username containing spaces#stash/repo1.git
This is yet another mistake that #CharlesDuffy found in my answer. Thanks Charles, I learn a lot from digging deeper into the problems you pointed!
I have one pre-commit script/hook working just fine to search for specific string pattern in the files and reject the commit. I'm not sure how to read the incoming files in the pre-receive script to search for string pattern.
My pre-commit scripts looks like this:
#!/usr/bin/env bash
if git rev-parse --verify HEAD >/dev/null 2>&1
then
against=HEAD
else
# Initial commit: diff against an empty tree object
EMPTY_TREE=$(git hash-object -t tree /dev/null)
against=$EMPTY_TREE
fi
FILES=$(git diff --cached --name-only $against)
if [ -n "$FILES" ]; then
string1 = $(grep -rE --line-number 'access_key' $FILES)
if [ -n "$string1" ] then
echo "string1 there so reject it"
while true; do
exit 1;
done
fi
fi
I'm not sure how to convert this to a pre-receive hook script on git server side.
I've been trying this for hours with no luck. Can someone please help me out here?
I am looking for somehelp to write a pre-commit hook on windows to check for Jira issue key while commiting.Commit should not be allowed if Jira key is not present.I couldnt find any way.I am new to scripting.Any help would be highly appreciated.
I assume you are talking about a hooks in a Git repository.
Navigate to your local Git repository and go into the folder .git\hooks
Create a file named commit-msg
Insert the following content (no idea how to format it correctly)
#!/bin/bash
# The script below adds the branch name automatically to
# every one of your commit messages. The regular expression
# below searches for JIRA issue key's. The issue key will
# be extracted out of your branch name
REGEX_ISSUE_ID="[a-zA-Z0-9,\.\_\-]+-[0-9]+"
# Find current branch name
BRANCH_NAME=$(git symbolic-ref --short HEAD)
if [[ -z "$BRANCH_NAME" ]]; then
echo "No branch name... "; exit 1
fi
# Extract issue id from branch name
ISSUE_ID=$(echo "$BRANCH_NAME" | grep -o -E "$REGEX_ISSUE_ID")
echo "$ISSUE_ID"': '$(cat "$1") > "$1"
If you have now a branch named like feature/MYKEY-1234-That-a-branch-name
and add as commit message "Add a new feature"
Your final commit message will look like
MYKEY-1234: Add a new feature
You can put the hook globally when using Git 2.9.
Please find here further useful information:
https://andy-carter.com/blog/automating-git-commit-messages-with-git-hooks
Git hooks : applying `git config core.hooksPath`
You have to put the following script in your local Git repository at .git/hooks/prepare-commit-msg. This will be run whenever you add a new commit.
#!/bin/bash
# get current branch
branchName=`git rev-parse --abbrev-ref HEAD`
# search jira issue id in pattern
jiraId=$(echo $branchName | sed -nr 's,[a-z]*\/*([A-Z]+-[0-9]+)-.+,\1,p')
# only prepare commit message if pattern matched and jiraId was found
if [[ ! -z $jiraId ]]; then
# $1 is the name of the file containing the commit message
sed -i.bak -e "1s/^/\n\n$jiraId: /" $1
fi
First, we get the branch name, for example feature/JIRA-2393-add-max-character-limit.
Next, we extract the key, removing the prefix feature.
The resulting commit message will be prefixed by "JIRA-2393: "
The script also works when there is no prefix, e.g. without feature/, bugfix/, etc.
You can use git server-side pre-receive hook.
https://git-scm.com/docs/git-receive-pack
In the code below for a successful push, you must specify the Jira issue key in the comment for commit.
#!/bin/bash
#
# check commit messages for JIRA issue numbers
# This file must be named pre-receive, and be saved in the hook directory in a bare git repository.
# Run "chmod +x pre-receive" to make it executable.
#
# Don't forget to change
# - Jira id regex
jiraIdRegex="\[JIRA\-[0-9]*\]"
error_msg="[POLICY] The commit doesn't reference a JIRA issue"
while read oldrev newrev refname
do
for sha1Commit in $(git rev-list $oldrev..$newrev);
do
echo "sha1 : $sha1Commit";
commitMessage=$(git log --format=%B -n 1 $sha1Commit)
jiraIds=$(echo $commitMessage | grep -Pqo $jiraIdRegex)
if ! jiraIds; then
echo "$error_msg: $commitMessage" >&2
exit 1
fi
done
done
exit 0