Processing an array of git commands inside bash script - bash

Good day to all:
I am writing a bash script to pull the data from git repos. I have an array to store the repository names:
declare -a arr=(
https://"$USER"#stash/repo1.git
https://"$USER"#stash/rep2.git
)
I process it as follows:
for i in "${arr[#]}"
do
git clone "$i"
......
git install
done
This works. Now I want to be able to optionally specify a git branch:
declare -a arr=(
-b branch1 --single-branch https://"$USER"#stash/repo1.git
https://"$USER"#stash/rep2.git
)
The array processing script fails with multiple ugly errors:
- switch `b' requires a value
- not valid identifier, etc
What is the right and simple approach to make it work?

c.f. printf "%s\n" "${arr[#]}". Just because those args are on one line doesn't make them one element. :)
You are trying to do two things with one array.
Try this -
declare -a arr=(
"https://$USER#stash/repo1.git#branch1"
"https://$USER#stash/rep2.git"
)
for i in "${arr[#]}"
do IFS='#' read url branch <<< "$i"
if [[ -n "$branch" ]]
then git clone -b "$branch" --single-branch "$url"
else git clone "$url"
: ......
git install
done
Warning - no chance to test this. Caveat Scriptor - and of course this means # can't be in your url anywhere.

Maybe it would work if you remove quotes around
git clone "$i"
So it would look like this:
git clone $i
I think that with quotes shell should treat all your line as one single argument instead of many:
git clone "-b branch1 --single-branch https://"$USER"#stash/repo1.git"
This looks wrong.
EDIT:
#CharlesDuffy pointed to another problem I missed: you should also have quotes around the entire line in the array definition (but this won't work if you really need internal quotes around $USER, see example below):
declare -a arr=(
"-b branch1 --single-branch https://$USER#stash/repo1.git"
https://"$USER"#stash/rep2.git
)
I checked this on my local machine, and it seems to work.
$ ls
one/ test.sh*
$ ( cd one/ ; git branch )
* another_branch
master
$ cat test.sh
#!/bin/bash
declare -a arr=(
"-b another_branch --single-branch one two"
)
for i in "${arr[#]}"
do
git clone $i
done
$ ls
one/ test.sh*
$ ./test.sh
Cloning into 'two'...
done.
$ ls
one/ test.sh* two/
$ ( cd two/ ; git branch )
* another_branch
$
EDIT2:
This will only work if you can safely omit internal quotes around $USER. If you need them you should use eval inside the for loop and also quote the internal quotes in array declaration.
$ cat ./test.sh
#!/bin/bash
USER="username containing spaces" # just for example!
git () {
echo "$5"
}
declare -a arr=(
"-b branch1 --single-branch https://\"$USER\"#stash/repo1.git"
)
for i in "${arr[#]}"
do
printf "without eval:\t"
git clone $i
printf "with eval:\t"
eval "git clone $i"
done
$ ./test.sh
without eval: https://"username
with eval: https://username containing spaces#stash/repo1.git
This is yet another mistake that #CharlesDuffy found in my answer. Thanks Charles, I learn a lot from digging deeper into the problems you pointed!

Related

bash, script to git add one file and commit

I'm trying to create a script to do this:
git add "file"
git commit -m "Comment"
My idea is to run:
gac "file" "Comment"
I know I can do something similar but for all files, with:
echo 'alias gac="/path/to/gitaddcommit.sh"' >> ~/.bash_profile
And the .sh would be:
!/bin/bash
git add .
echo “Enter commit message: “
git commit -am “$commitMessage”
Well you need two things :
A bin folder where you can put every sh script you want to use everywhere.
More knowledge about shell scripting and how you can get argv (in your ex: 'file' 'Comment')
So first go to your /home/<username> then mkdir bin && cd bin && pwd
then copy the pwd and add it into your PATH env variable inside your .bashrc
path example: PATH='/bin/:/sbin/:/home//bin
Then source ~/.bashrc you can now use every sh script inside you bin folder everywhere.
Cool so first problem done !
you don't have to do echo alias gac="/path/to/gitaddcommit.sh"' >> ~/.bash_profile anymore.
Now second problem here a post that can help you post
And let me show you for your example :
cd ~/bin && vi gac.sh
Now the script :
#!/bin/sh
if [ "$#" -ne 2 ]; then
echo "Usage: ./gac FILENAME COMMIT_MESSAGE" >&2
exit 1
fi
git add "$1"
git commit -am "$2"
First we check the number or arg then git add and commit.
Simple and fast maybe checking if arg one is a file might be a good idea too.
PS: i'm going to re write my post ahah
Here's what I have in my .bashrc:
ga ()
{
if test "$1" != "-f" && git rev-parse HEAD > /dev/null 2>&1 && ! git diff-index --quiet HEAD; then
echo 'Repo is dirty. -f to force' 1>&2;
return 1;
fi;
git add "$#";
list=$(git diff --name-only --cached | tr \\n \ );
git commit -m "Add $list"
}
The commit message is autogenerated, but you could easily modify it to prompt the user or take it from somewhere else.

Bash script to close multiple remote branches from text-file

Background
I need to close multiple Git branches (hundreds) that have been left open on a remote repository.
I wanted to sort them by last-commit and put them in a text-file so others could confirm they were no longer needed.
I found a method that allowed me to dump them in a way that I could distribute (and open in Excel to sort by date)
git for-each-ref --format='%(committerdate:short) %09 %(authorname) %09 %(refname)' | sort -k5n -k2M -k3n -k4n >> branches.txt
And then I need to read the updated text-file back in and delete the remote branches:
#!/bin/bash
#prefix of the branches if they are all remote
prefix="refs/remotes/origin/"
prefix1="refs/remotes/"
# path to branches, compiled with:
# git for-each-ref --format='%(committerdate:short) %09 %(authorname) %09 %(refname)' | sort -k5n -k2M -k3n -k4n >> branches.txt
# read through entire input
input="branches.txt"
while IFS= read -r line
do
# echo "$line"]
IFS=' ' read -r -a array <<< "$line"
# search the array for the prefix of the branch
for index in "${!array[#]}"
do
if [[ ${array[$index]} == *"refs/remotes/origin"* ]]; then
#echo -e " \e[34m ${array[$index]} \e[39m Last commit: \e[34m" ${array[0]} ${array[1]} ${array[2]} ${array[3]} ${array[4]}
# echo ${array[$index]#"$prefix"}
#remove the prefix if they are not pulled locally
branch=${array[$index]#"$prefix"}
echo $(git push origin --delete $branch)
fi
done
done < "$input"
Issue
However, keep getting an error that the refspec does not exist:
fatal: invalid refspec ':PracticeBranch?'
I don't have the ' or ? in the name of the branch variable - so this is probably my ignorance of using echo with `bash but I don't know where it is coming from.
A command like:
git push origin --delete PracticeBranch
When calling directly from the shell?

bash string incorrectly split

I have this script
#!/bin/bash
function clone {
url=$(cli-tool "$1" that finds url)
echo $url
$(git clone ${url})
}
echo prints the correct url in the format
"https://gitprovider.com/Example/_git/Repo%20Name" (not a real url but that mimics the real url)
But git clone outputs
fatal: could not create work tree dir 'Repo%20Name"': Invalid argument
If I execute
git clone "https://gitprovider.com/Example/_git/Repo%20Name"
the correct repo will be cloned.
So why isn't
$(git clone ${url})
Working?
Command substitution is only needed when you want to use the output of a command as an argument to another command. In your case, the output of git clone is then parsed as sequence of words used to build a command line. You don't want to do that; you just want git clone ... to run and have its output displayed on the terminal.
Compare
$ echo $(echo foo)
foo
$ $(echo foo)
bash: foo: command not found
You just want git clone "$url", not $(git clone "$url").
Instead of using $(git clone ${url}), just use git clone "${url}", i.e., drop the $( ) thing.

Perform command from command line inside directories from glob in bash shell script

In a bash shell script do-for.sh I want to perform a command inside all directories named in a glob using bash. This has been answered oodles of times, but I want to provide the command itself on the command line. In other words assuming I have the directories:
foo
bar
I want to enter
do-for * pwd
and have bash print the working directory inside foo and then inside bar.
From reading the umpteen answers on the web, I thought I could do this:
for dir in $1; do
pushd ${dir}
$2 $3 $4 $5 $6 $7 $8 $9
popd
done
Apparently though the glob * gets expanded into the other command line arguments variable! So the first time through the loop, for $2 $3 $4 $5 $6 $7 $8 $9 I expected foo pwd but instead it appears I get foo bar!
How can I keep the glob on the command line from being expanded into the other parameters? Or is there a better way to approach this?
To make this clearer, here is how I want to use the batch file. (This works fine on the Windows batch file version, by the way.)
./do-for.sh repo-* git commit -a -m "Added new files."
I will assume you are open to your users having to provide some kind of separator, like so
./do-for.sh repo-* -- git commit -a -m "Added new files."
Your script could do something like (this is just to explain the concept, I have not tested the actual code) :
CURRENT_DIR="$PWD"
declare -a FILES=()
for ARG in "$#"
do
[[ "$ARG" != "--" ]] || break
FILES+=("$ARG")
shift
done
if
[[ "${1-}" = "--" ]]
then
shift
else
echo "You must terminate the file list with -- to separate it from the command"
(return, exit, whatever you prefer to stop the script/function)
fi
At this point, you have all the target files in an array, and "$#" contains only the command to execute. All that is left to do is :
for FILE in "${FILES[#]-}"
do
cd "$FILE"
"$#"
cd "$CURRENT_DIR"
done
Please note that this solution has the advantage that if your user forgets the "--" separator, she will be notified (as opposed to a failure due to quoting).
In this case the problem is not the expansion of metacharacter, is just that your script has an undefined number of arguments of which the last one is the command to execute for all previous arguments.
#!/bin/bash
CMND=$(eval echo "\${$#}") # get the command as last argument without arguments or
while [[ $# -gt 1 ]]; do # execute loop for each argument except last one
( cd "$1" && eval "$CMND" ) # switch to each directory received and execute the command
shift # throw away 1st arg and move to the next one in line
done
Usage: ./script.sh * pwd or ./script.sh * "ls -l"
To have the command followed by arguments (ex. ./script.sh * ls -l) the script has to be longer because each argument has to be tested if it's a directory until the command is identified (or backwards until a dir is identified).
Here is an alternative script that would accept the syntax: ./script.sh <dirs...> <command> <arguments...>
For example: ./script.sh * ls -la
# Move all dirs from args to DIRS array
typeset -i COUNT=0
while [[ $# -gt 1 ]]; do
[[ -d "$1" ]] && DIRS[COUNT++]="$1" && shift || break
done
# Validate that the command received is valid
which "$1" >/dev/null 2>&1 || { echo "invalid command: $1"; exit 1; }
# Execute the command + it's arguments for each dir from array
for D in "${DIRS[#]}"; do
( cd "$D" && eval "$#" )
done
Here is how I would do it:
#!/bin/bash
# Read directory arguments into dirs array
for arg in "$#"; do
if [[ -d $arg ]]; then
dirs+=("$arg")
else
break
fi
done
# Remove directories from arguments
shift ${#dirs[#]}
cur_dir=$PWD
# Loop through directories and execute command
for dir in "${dirs[#]}"; do
cd "$dir"
"$#"
cd "$cur_dir"
done
This loops over the arguments as seen after expansion, and as long as they are directories, they are added to the dirs array. As soon as the first non-directory argument is encountered, we assume that now the command starts.
The directories are then removed from the arguments with shift, and we store the current directory in cur_dir.
The last loop visits each directory and executes the command consisting of the rest of the arguments.
This works for your
./do-for.sh repo-* git commit -a -m "Added new files."
example – but if repo-* expands to anything other than directories, the script breaks because it will try to execute the filename as part of the command.
It could be made more stable if, for example, the glob and the command were separated by an indicator such as --, but if you know that the glob will always be just directories, this should work.
I will begin with the Windows batch file that you mentioned twice as working. The big difference is that on Windows, the shell doesn’t make any globbing, leaving it to the various commands (and each of them does it differently), while on Linux/Unix the globbing is usually done by the shell, and can be prevented by quoting or escaping. Both the Windows approach and the Linux approach have their merits, and they compare differently in different use cases.
For regular bash users, quoting
./do-for.sh repo-'*' git commit -a -m "Added new files."
or escaping
./do-for.sh repo-\* git commit -a -m "Added new files."
are the simplest solution, because they are what they consistently use on a daily basis. If your users need a different syntax, you have all the solutions proposed so far, that I will classify into four categories before proposing my own (note that in each example below do-for.sh stands for a different script adopting the respective solution, which can be found in one of the other answers.)
Disable shell globbing. This is clumsy, because, even if you remember which shell option does it, you have to remember to reset it to default to have the shell working normally afterwards.
Use a separator:
./do-for.sh repo-* -- git commit -a -m "Added new files."
This works, is similar to the solution adopted in similar situations with other shell commands, and fails only if your expansion of directory names includes a directory name exactly equal to the separator (an unlikely event, which wouldn’t happen in the above example, but in general might happen.)
Have the command as the last argument, all the rest are directories:
./do-for.sh repo-* 'git commit -a -m "Added new files."'
This works, but again, it involves quoting, possibly even nested, and there is no point in preferring it to the more usual quoting of globbing characters.
Try to be smart:
./do-for.sh repo-* git commit -a -m "Added new files."
and consider to be dealing with directories till you hit a name which is not a directory. This would work in many cases, but might fail in obscure ways (e.g. when you have a directory named like the command).
My solution doesn’t belong to any of the mentioned categories. In fact, what I propose is not to use * as a globbing character in the first argument of your script. (This is similar to the syntax used by the split command where you provide a non-globbed prefix argument for the files to be generated.) I have two versions (code below). With the first version, you would do the following:
# repo- is a prefix: the command will be excuted in all
# subdirectories whose name starts with it
./do-for.sh repo- git commit -a -m "Added new files."
# The command will be excuted in all subdirectories
# of the current one
./do-for.sh . git commit -a -m "Added new files."
# If you want the command to be executed in exactly
# one subdirectory with no globbing at all,
# '/' can be used as a 'stop character'. But why
# use do-for.sh in this case?
./do-for.sh repo/ git commit -a -m "Added new files."
# Use '.' to disable the stop character.
# The command will be excuted in all subdirectories of the
# given one (paths have to be always relative, though)
./do-for.sh repos/. git commit -a -m "Added new files."
The second version involves using a globbing character the shell knows nothing about, such as SQL’s % character
# the command will be excuted in all subdirectories
# matching the SQL glob
./do-for.sh repo-% git commit -a -m "Added new files."
./do-for.sh user-%-repo git commit -a -m "Added new files."
./do-for.sh % git commit -a -m "Added new files."
The second version is more flexible, as it allows non-final globs, but is less standard for the bash world.
Here is the code:
#!/bin/bash
if [ "$#" -lt 2 ]; then
echo "Usage: ${0##*/} PREFIX command..." >&2
exit 1
fi
pathPrefix="$1"
shift
### For second version, comment out the following five lines
case "$pathPrefix" in
(*/) pathPrefix="${pathPrefix%/}" ;; # Stop character, remove it
(*.) pathPrefix="${pathPrefix%.}*" ;; # Replace final dot with glob
(*) pathPrefix+=\* ;; # Add a final glob
esac
### For second version, uncomment the following line
# pathPrefix="${pathPrefix//%/*}" # Add a final glob
tmp=${pathPrefix//[^\/]} # Count how many levels down we have to go
maxDepth=$((1+${#tmp}))
# Please note that this won’t work if matched directory names
# contain newline characters (comment added for those bash freaks who
# care about extreme cases)
declare -a directories=()
while read d; do
directories+=("$d")
done < <(find . -maxdepth "$maxDepth" -path ./"$pathPrefix" -type d -print)
curDir="$(pwd)"
for d in "${directories[#]}"; do
cd "$d";
"$#"
cd "$curDir"
done
As in Windows, you would still need to use quotes if the prefix contains spaces
./do-for.sh 'repository for project' git commit -a -m "Added new files."
(but if the prefix does not contain spaces, you can avoid quoting it and it will correctly deal with any space-containing directory names beginning with that prefix; with obvious changes, the same is true for %-patterns in the second version.)
Please note the other relevant differences between a Windows and a Linux environment, such as case sensitivity in pathnames, differences in which characters are considered special, and so on.
In bash you may execute "set -o noglob" which will inhibit the shell to expand path names (globs). But this has to be set on the running shell before you execute the script, otherwise you should quote any meta character which you provide in the arguments.
find-while-read combination is one of the safest combination to parse file names. Do something like below
#!/bin/bash
myfunc(){
cd "$2"
eval "$1" # Execute the command parsed as an argument
}
cur_dir=$(pwd) # storing the current directory
find . -type d -print0 | while read -rd '' dname
do
myfunc "pwd" "$dname"
cd "$cur_dir" #Remember myfunc changes the current working dir, so you need this
done
Why not keep it simple and create a shell function that uses find but eases the burden for your users of typing out its commands, for example:
do_for() { find . -type d \( ! -name . \) -not -path '*/\.*' -name $1 -exec bash -c "cd '{}' && "${#:2}" " \; }
So they can type something like do_for repo-* git commit -a -m "Added new files."
Note, if you want to use the * by itself, you'll have to escape it:
do_for \* pwd
Wildcards are evaluated by the shell before being passed to any program or script. There is nothing you can do about that.
But if you accept quoting the globbing expression then this script should to do the trick
#!/usr/bin/env bash
for dir in $1; do (
cd "$dir"
"${#:2}"
) done
I tried it out with two test directories and it seems to be working. Use it like this:
mkdir test_dir1 test_dir2
./do-for.sh "test_dir*" git init
./do-for.sh "test_dir*" touch test_file
./do-for.sh "test_dir*" git add .
./do-for.sh "test_dir*" git status
./do-for.sh "test_dir*" git commit -m "Added new files."
Nobody proposing a solution using find ? Why not try something like this:
find . -type d \( -wholename 'YOURPATTERN' \) -print0 | xargs -0 YOURCOMMAND
Look at man find for more options.

How to inhibit line breaks when calling echo several times in a row

I have created an alias for Git:
alias gtbs='echo on Branch; git rev-parse --abbrev-ref HEAD; echo last Modified; git status -s;'
but this prints output each on newline
like
on Branch:
develop
Added/Modified:
M scripts/myscript
while instead i just want in one line
on Branch: develop, Added/Modified: M scripts/myscript
The echo command adds an implicit newline, which you're observing.
Either pass the -n command-line option to echo (but this is not POSIX-compliant, though many shells implement it, as well as /bin/echo from the coreutils project, typically found on GNU/Linux systems) or use the printf command like this:
printf 'on branch: %s, last modified: %s' \
$(git rev-parse --abbrev-ref HEAD) $(git status -s)
(The \\ is here for formatting purposes.)
You can also use backticks `...` instead of $(...).
Also note that git status itself might produce output spanning several lines, so I'd say the most sensible thing would be to ask printf to insert the newline right before the output of git status, so the whole encantation would become:
printf 'printf 'on branch: %s, last modified:\n' \
$(git rev-parse --abbrev-ref HEAD) && git-status -s
If, for some reason you do mean to have git status -s output a single line, you might want to delete any newlines from it:
printf '...last modified: %s' $(git status -s | paste -sd ,)
…and if you want multiple-character separators, the last encantation would become a bit more involved:
$(IFS=\n git-status -s | while read line; do printf '%s; ' "$line"; done)

Resources