Background
I would like to make a shell function that takes .gitmodules and iterates over each module executing certain commands based off of each submodules properties (e.g. <PATH> or <URL> or <BRANCH>).
➡️ The default format of .gitmodules:
[submodule "PATH"]
path = <PATH>
url = <URL>
[submodule "PATH"]
path = <PATH>
url = <URL>
branch = <BRANCH>
➡️ Pseudocode:
def install_modules() {
modules = new list
fill each index of the modules list with each submodule & its properties
iteratate over modules
if module # 'path' contains a specified 'branch':
git submodule add -b 'branch' 'url' 'path'
else:
git submodule add 'url' 'path'
}
⚠️ Current install_modules()
# currently works for grabbing the first line of the file
# doesn't work for each line after.
install_modules() {
declare -A regex
regex["module"]='\[submodule "(.*)"\]'
regex["url"]='url = "(.*)"'
regex["branch"]='branch = "(.*)"'
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
cat < ".gitmodules" | while read -r LINE; do
if [[ $LINE =~ ${regex[module]} ]]; then
PATH=${BASH_REMATCH[1]}
echo "$PATH"
fi
done
}
.gitmodules is a .gitconfig-like file so you can use git config to read it. For example, read all values from a .gitmodules, split values by = (key=value), and split keys by .:
git config -f .gitmodules -l | awk '{split($0, a, /=/); split(a[1], b, /\./); print b[1], b[2], b[3], a[2]}'
git config -f .gitmodules -l prints something like
submodule.native/inotify_simple.path=native/inotify_simple
submodule.native/inotify_simple.url=https://github.com/chrisjbillington/inotify_simple
and awk output would be
submodule native/inotify_simple path native/inotify_simple
submodule native/inotify_simple url https://github.com/chrisjbillington/inotify_simple
With a little help from #phd and Restore git submodules from .gitmodules (which #phd pointed me towards), I was able to construct the function that I needed.
install_submodules()
⚠️ Note: Assume $REPO_PATH is declared & initialized.
⚠️ My answer is an adaptation from https://stackoverflow.com/a/53269641/5290011.
install_submodules() {
git -C "${REPO_PATH}" config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
while read -r KEY MODULE_PATH
do
# If the module's path exists, remove it.
# This is done b/c the module's path is currently
# not a valid git repo and adding the submodule will cause an error.
[ -d "${MODULE_PATH}" ] && sudo rm -rf "${MODULE_PATH}"
NAME="$(echo "${KEY}" | sed 's/^submodule\.\(.*\)\.path$/\1/')"
url_key="$(echo "${KEY}" | sed 's/\.path$/.url/')"
branch_key="$(echo "${KEY}" | sed 's/\.path$/.branch/')"
URL="$(git config -f .gitmodules --get "${url_key}")"
BRANCH="$(git config -f .gitmodules --get "${branch_key}" || echo "master")"
git -C "${REPO_PATH}" submodule add --force -b "${BRANCH}" --name "${NAME}" "${URL}" "${MODULE_PATH}" || continue
done
git -C "${REPO_PATH}" submodule update --init --recursive
}
Related
I am writing a Jenkins pipeline. I am trying to capture last part of the git url without the git extension. For instance: https://github.hhhh.com/aaaaaa-dddd/xxxx-yyyy.git. I want only xxxx-yyyy to be returned. Below is my code:
String getProjectName() {
echo "inside getProjectName +++++++"
# projectName = sh(
# script: "git config --get remote.origin.url",
# returnStdout: true
# ).trim()
def projectName= sh returnStdout:true, script: '''
#!/bin/bash
GIT_LOG = $(env -i git config --get remote.origin.url)
echo $GIT_LOG
basename -s .git "$GIT_LOG"; '''
echo "projectName: ${projectName}"
return projectName
}
PS: Please ignore the commented lines of code.
There is basic Bourne shell functionality that achieves that:
# strip everything up to the last /
projectName=${GIT_LOG##*/}
# strip trailing .git
projectName=${projectName%.git}
This leaves just the requested name in projectName.
No space before and after =:
x='https://github.hhhh.com/aaaaaa-dddd/xxxx-yyyy.git'
basename "$x" .git
Output:
xxxx-yyyy
Background
I have successfully configured Bash completion for various Git aliases. For example:
$ git config alias.subject
!git --no-pager show --quiet --pretty='%s'
$ function _git_subject() { _git_show; }
$ git subject my<TAB>
$ git subject my-branch
Challenge
However, I have a Git alias that I don't know how to set up Bash completion for. The problem is that I want the alias to complete as if for the top-level Git command itself. The alias is this:
$ git config alias.alias
alias = !"f() { if [[ \"$#\" != 1 ]]; then >&2 echo \"Usage: git alias COMMAND\"; return 1; fi; git config alias.\"$1\"; }; f"
# Example
$ git alias s
status
I have tried using _git, __git_main, and __git_wrap__git_main, but none of them work (I think it leads to an infinite loop since it never returns after I press tab).
Is there a way to add completion for a Git alias that completes as if it was the top-level Git command? Or specifically how to have completion for this alias?
Tried but doesn't work
function _git_alias() { _git; }
function _git_alias() { __git_main; }
function _git_alias() { __git_wrap__git_main; }
Desired behavior
$ git alias su<TAB>
subject submodule
$ git alias sub
Alternatively, if there's an easy way to complete for only aliases that would be cool, too. I would like to know how to complete as if for the top-level Git command just for curiosity as well, though.
I was finally able to create a working solution with a bit of hackery around the "magic" Bash completion variables. I changed these variables to "pretend" we were completing the given command as given to git itself.
If anybody has any suggestions to simplify this I would totally be open to suggestions.
# This is complex because we want to delegate to the completion for Git
# itself without ending up with an infinite loop (which happens if you try
# to just delegate to _git).
_git_alias() {
if [[ "$COMP_CWORD" -lt 2 ]]; then
return
fi
local old_comp_line_length new_comp_line_length
COMP_WORDS=(git "${COMP_WORDS[#]:2}")
((COMP_CWORD -= 1))
old_comp_line_length=${#COMP_LINE}
if [[ "$COMP_LINE" =~ ^[^[:blank:]]+[[:blank:]]+[^[:blank:]]+[[:blank:]]+(.*)$ ]]; then
COMP_LINE="git ${BASH_REMATCH[1]}"
fi
new_comp_line_length=${#COMP_LINE}
(( COMP_POINT += new_comp_line_length - old_comp_line_length ))
_git "$#"
# git alias blah
# ^
# 01234567890123
# 0 1
# point: 11
# length: 13
#
# git blah
# ^
# 01234567
# point: 5
# length: 7
#
# point = point - (old length) + (new length)
# point = 11 - 13 + 7
# point = -2 + 7
# point = 5
}
Background:
I'm using an AWS CodeBuild buildspec.yml to iterate through directories from a GitHub repo. Before looping through the directory path $TF_ROOT_DIR, I'm using a bash if statement to check if the GitHub branch name $BRANCH_NAME is within an env variable $LIVE_BRANCHES. As you can see in the error screenshot below, the bash if statement outputs the error: syntax error: bad substitution. When I reproduce the if statement within a local bash script, the if statement works as it's supposed to.
Here's the env variables defined in the CodeBuild project:
Here's a relevant snippet from the buildspec.yml:
version: 0.2
env:
shell: bash
phases:
build:
commands:
- |
if [[ " ${LIVE_BRANCHES[*]} " == *"$BRANCH_NAME"* ]]; then
# Iterate only through BRANCH_NAME directory
TF_ROOT_DIR=${TF_ROOT_DIR}/*/${BRANCH_NAME}/
else
# Iterate through both dev and prod directories
TF_ROOT_DIR=${TF_ROOT_DIR}/*/
fi
- echo $TF_ROOT_DIR
Here's the build log that shows the syntax error:
Here's the AWS CodeBuild project JSON to reproduce the CodeBuild project:
{
"projects": [
{
"name": "terraform_validate_plan",
"arn": "arn:aws:codebuild:us-west-2:xxxxx:project/terraform_validate_plan",
"description": "Perform terraform plan and terraform validator",
"source": {
"type": "GITHUB",
"location": "https://github.com/marshall7m/sparkify_end_to_end.git",
"gitCloneDepth": 1,
"gitSubmodulesConfig": {
"fetchSubmodules": false
},
"buildspec": "deployment/CI/dev/cfg/buildspec_terraform_validate_plan.yml",
"reportBuildStatus": false,
"insecureSsl": false
},
"secondarySources": [],
"secondarySourceVersions": [],
"artifacts": {
"type": "NO_ARTIFACTS",
"overrideArtifactName": false
},
"cache": {
"type": "NO_CACHE"
},
"environment": {
"type": "LINUX_CONTAINER",
"image": "hashicorp/terraform:0.12.28",
"computeType": "BUILD_GENERAL1_SMALL",
"environmentVariables": [
{
"name": "TF_ROOT_DIR",
"value": "deployment",
"type": "PLAINTEXT"
},
{
"name": "LIVE_BRANCHES",
"value": "(dev, prod)",
"type": "PLAINTEXT"
}
Here's the associated buildspec file content: (buildspec_terraform_validate_plan.yml)
version: 0.2
env:
shell: bash
parameter-store:
AWS_ACCESS_KEY_ID_PARAM: TF_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY_PARAM: TF_AWS_SECRET_ACCESS_KEY_ID
phases:
install:
commands:
# install/incorporate terraform validator?
pre_build:
commands:
# CodeBuild environment variables
# BRANCH_NAME -- GitHub branch that triggered the CodeBuild project
# TF_ROOT_DIR -- Directory within branch ($BRANCH_NAME) that will be iterated through for terraform planning and testing
# LIVE_BRANCHES -- Branches that represent a live cloud environment
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_PARAM
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_PARAM
- bash -version || echo "${BASH_VERSION}" || bash --version
- |
if [[ -z "${BRANCH_NAME}" ]]; then
# extract branch from github webhook
BRANCH_NAME=$(echo $CODEBUILD_WEBHOOK_HEAD_REF | cut -d'/' -f 3)
fi
- "echo Triggered Branch: $BRANCH_NAME"
- |
if [[ " ${LIVE_BRANCHES[*]} " == *"$BRANCH_NAME"* ]]; then
# Iterate only through BRANCH_NAME directory
TF_ROOT_DIR=${TF_ROOT_DIR}/*/${BRANCH_NAME}/
else
# Iterate through both dev and prod directories
TF_ROOT_DIR=${TF_ROOT_DIR}/*/
fi
- "echo Terraform root directory: $TF_ROOT_DIR"
build:
commands:
- |
for dir in $TF_ROOT_DIR; do
#get list of non-hidden directories within $dir/
service_dir_list=$(find "${dir}" -type d | grep -v '/\.')
for sub_dir in $service_dir_list; do
#if $sub_dir contains .tf or .tfvars files
if (ls ${sub_dir}/*.tf) > /dev/null 2>&1 || (ls ${sub_dir}/*.tfvars) > /dev/null 2>&1; then
cd $sub_dir
echo ""
echo "*************** terraform init ******************"
echo "******* At directory: ${sub_dir} ********"
echo "*************************************************"
terraform init
echo ""
echo "*************** terraform plan ******************"
echo "******* At directory: ${sub_dir} ********"
echo "*************************************************"
terraform plan
cd - > /dev/null
fi
done
done
Given this is just a side project, all files that could be relevant to this problem are within a public repo here.
UPDATES
Tried adding #!/bin/bash shebang line but resulted in the CodeBuild error:
Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: #!/bin/bash
version: 0.2
env:
shell: bash
phases:
build:
commands:
- |
#!/bin/bash
if [[ " ${LIVE_BRANCHES[*]} " == *"$BRANCH_NAME"* ]]; then
# Iterate only through BRANCH_NAME directory
TF_ROOT_DIR=${TF_ROOT_DIR}/*/${BRANCH_NAME}/
else
# Iterate through both dev and prod directories
TF_ROOT_DIR=${TF_ROOT_DIR}/*/
fi
- echo $TF_ROOT_DIR
Solution
As mentioned by #Marcin, I used an AWS managed image within Codebuild (aws/codebuild/standard:4.0) and downloaded Terraform within the install phase.
phases:
install:
commands:
- wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip -q
- unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip && mv terraform /usr/local/bin/
I tried to reproduce your issue, but it all works fine for me.
The only thing I've noticed is that you are using $BRANCH_NAME but its not defined anywhere. But even with missing $BRANCH_NAME the buildspec.yml you've posted runs fine.
Update using hashicorp/terraform:0.12.28 image
Point of my task : gather info about repo and place it to file while update hook and commit it ( perfectly with new commit).
Problems : when I'm doing commit -> it lockes origin repository and after this push is failing. my code looks like this :
#!/bin/bash
# --- Command line
refname="$1"
oldrev="$2"
newrev="$3"
export GIT_WORK_TREE=$PWD
# --- Safety check
if [ -z "$GIT_DIR" ]; then
echo "Don't run this script from the command line." >&2
echo " (if you want, you could supply GIT_DIR then run" >&2
echo " $0 <ref> <oldrev> <newrev>)" >&2
exit 1
fi
version="${refname##*_}"
branchName="${refname##*/}"
filePath="_componentVersion/BranchVersion.ps1"
if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then
echo "usage: $0 <ref> <oldrev> <newrev>" >&2
exit 1
fi
# --- Check types
# if $newrev is 0000...0000, it's a commit to delete a ref.
zero="0000000000000000000000000000000000000000"
if [ "$newrev" = "$zero" ]; then
newrev_type=delete
else
newrev_type=$(git cat-file -t $newrev)
fi
case "$refname","$newrev_type" in
refs/heads/*,commit)
if [ "$oldrev " != "$zero" ]; then
version="${refname##*_}"
branchName="${refname##*/}"
filePath="_componentVersion/BranchVersion.ps1"
countOfCommits=$(git rev-list --count START_$version..$newrev)
countOfPushes=$(git log --pretty=oneline START_$version..$newrev | grep 'Issue nr: HOOK_$version' | wc -l)
countOfPushes=$(($countOfPushes+1))
echo git log --pretty=oneline START_$version..$newrev
message="
# -----------------------
# Brancht Version Info
# -----------------------
\$branch = '$version'
\$countOfCommits = $countOfCommits
\$countOfPushes = $countOfPushes # push
\$commitHash = '$newrev'
"
# credits go to https://stackoverflow.com/questions/9670302/commit-directly-to-a-bare-repository
# branch commit - here we will do the magic about count of commits and about count of pushes
# here we create file for info
# Empty the index, not sure if this step is necessary
git read-tree --empty
# Load the current tree. A commit ref is fine, it'll figure it out.
git read-tree "${newrev}"
# create blob from stdin
BLOB_ID=$(echo "$message" | git hash-object -w --stdin)
# update indexes in git
git update-index --add --cacheinfo 100644 "$BLOB_ID" "$filePath"
# Create a tree from your new index
TREE_ID=$(git write-tree)
# Commit it.
NEW_COMMIT=$(echo "Issue nr: HOOK_$version $message" | git commit-tree "$TREE_ID" -p "$oldrev")
# Update the branch
git update-ref "$refname" "$NEW_COMMIT" "$oldrev"
fi
# Done
exit 0
;;
*)
# Other actions except commit to branch / for now - we won't check it
exit 0
;;
esac
# --- Finished
exit 0
I'm working with bare repo. And example of commit taken from here
Execte problem is
remote: error: cannot lock ref 'refs/heads/REL_7.0.0': ref refs/heads/REL_7.0.0 is at 54f2454ddab36eda001e27946733a7b0e981f097 but expected 89a3032e0bfb999273205e32b7f6d57173c4bd7e
You can create commits.
You cannot update references that are locked, which includes the one that the update hook is being called for.
Since git push can push multiple reference names, there may be additional locked references. In general it's not a good idea to update anything that anyone might be git pushing inside a hook invoked by git push. In other words, don't try to update any branch or tag name. If you want to create new objects, attach them to some name outside these two name-spaces.
(Aside: the git read-tree --empty is not necessary, but it's a good idea to use a temporary index file anyway, rather than using the main index.)
I would like to write a script that reads a text file that has all the nodes listed in there:
node1
node2
node3
.
.
.
It creates a .conf file for each node in the
/etc/icinga2/zones.d/master/hosts/new/ directory
Copies the content of the file name windows-template into each
new conf file.
Then finds the phrase "hostname.hostdomain.com" in each conf file
and replaces that with the filename minus the .conf. So for example,
for node1, I will have node1.conf in which there is a phrase
"hostname.hostdomain.com" which needs to be replaced with node1
Then pings the hostname which is technically the filename minus
".conf" and replaces the 10.20.20.1 with the correct hostname.
I tried wrirting the script and part 1 and 2 work, part 3 works too but it replaces the hostname.hostdomain.com with "$f" which is not right. And I have no clue how to do number 4.
Can you please help?
Thank you
This is my windows-template.conf file:
object Host "hostname.hostdomain.com" {
import "production-host"
check_command = "hostalive"
address = "10.20.20.1"
vars.client_endpoint = name
vars.disks["disk C:"] = {
disk_partition = "C:"
}
vars.os = "Windows"
}
object Zone "hostname.hostdomain.com" {
endpoints = [ "hostname.hostdomain.com" ];
parent = "master";
}
object Endpoint "hostname.hostdomain.com" {
host = "10.20.20.1"
}
And this is my script:
#!/bin/bash
cd /etc/icinga2/zones.d/master/hosts/new
while read f; do
cp -v "$f" /etc/icinga2/zones.d/master/hosts/new/"$f.conf"
cp windows-template.conf "$f.conf"
chown icinga:icinga "$f.conf"
sed -i 's/hostname.hostdomain.com/$f/g' "$f.conf"
# git add "$f.conf"
# git commit -m "Add $f"
done < windows-list.txt
Thank you
You need double quotes for the shell to expand your variable. Try
sed -i "s/hostname.hostdomain.com/$f/g" "$f.conf"
Does this work for you?
#!/bin/bash
cd /etc/icinga2/zones.d/master/hosts/new
while read f; do
cp -v "$f" /etc/icinga2/zones.d/master/hosts/new/"$f.conf"
cp windows-template.conf "$f.conf"
chown icinga:icinga "$f.conf"
sed -i "s/hostname.hostdomain.com/$f/g" "$f.conf"
hostname=$( ssh -o StrictHostKeyChecking=no "username#$f" -n "hostname" )
mv "$f.conf" "${hostname}.conf"
# git add "$f.conf"
# git commit -m "Add $f"
done < windows-list.txt
Where username is your username, and I assume you copy your pub key to the hosts.