How does process substitution work with while loops? - bash

I'm reading/editing a bash git integration script
This snippet is supposed to print ${SYMBOL_GIT_PUSH} or ${SYMBOL_GIT_PULL} alongside how many commits i am behind and/or ahead by.
local marks
while IFS= read -r line; do
if [[ $line =~ ^## ]]; then
[[ $line =~ ahead\ ([0-9]+) ]] && marks+=" ${BASH_REMATCH[1]}${SYMBOL_GIT_PUSH}"
[[ $line =~ behind\ ([0-9]+) ]] && marks+=" ${BASH_REMATCH[1]}${SYMBOL_GIT_PULL}"
else
marks="${SYMBOL_GIT_MODIFIED}${marks}"
break
fi
done < <(git status --porcelain --branch 2>/dev/null)
printf '%s' "$marks"
Example:
4↑ 10↓
It is working, but i am trying to understand it.
Why is there some IFS and how does it work with process substitution?
I've heard process isn't defined in sh. Is there a way to do this the /bin/sh way or at least more efficiently?
I was provided with a link that should explain what IFS does.
I switched mixed up things and managed to remove the process substitution:
local marks
git status --porcelain --branch 2>/dev/null |
while IFS= read -r line; do
if [[ $line =~ ^## ]]; then
[[ $line =~ ahead\ ([0-9]+) ]] && marks+=" ${BASH_REMATCH[1]}${SYMBOL_GIT_PUSH}"
[[ $line =~ behind\ ([0-9]+) ]] && marks+=" ${BASH_REMATCH[1]}${SYMBOL_GIT_PULL}"
else
marks="${SYMBOL_GIT_MODIFIED}${marks}"
break
fi
done
printf '%s\n' "$marks"
But now, the value of $marks isn't saved and it prints nothing.
I was provided with another link that explains why.
Will return and update on what i've found.
I used the command grouping workaround and wrapped the loop and the print statement inside curly braces:
Also, i made the /bin/sh version almost functional (the exception - show how much commits i'm ahead or behind, not hard, i'm sure i'll do something with awk or cut).
I took advantage of fact that grep returns non-0 when nothing matches.
git status --porcelain --branch 2>/dev/null | {
SYMBOL_GIT_PUSH='↑'
SYMBOL_GIT_PULL='↓'
while IFS= read -r line
do
if echo "$line" | egrep -q '^##'
then
echo "$line" | egrep -q 'ahead' && marks="$marks $SYMBOL_GIT_PUSH"
echo "$line" | egrep -q 'behind' && marks="$marks $SYMBOL_GIT_PULL"
else
marks="*$marks"
break
fi
done
printf ' %s' "$marks"
}
This was a fun learning experience! Thanks to everyone who helped. When i find the 100% solution i'll update this.

Here's the bashism-less git info function.
__git() {
git_eng="env LANG=C git"
ref="$($git_eng symbolic-ref --short HEAD 2>/dev/null)"
[ -n "$ref" ] && ref="$SYMBOL_GIT_BRANCH$ref" || ref="$($git_eng describe --tags --always 2>/dev/null)"
[ -n "$ref" ] || return;
git status --porcelain --branch 2>/dev/null | {
SYMBOL_GIT_PUSH='↑'
SYMBOL_GIT_PULL='↓'
while IFS= read -r line
do
if echo "$line" | grep -E -q '^##'
then
echo "$line" | grep -E -q 'ahead' &&
marks="$marks $SYMBOL_GIT_PUSH$(echo "$line" | sed 's/.*\[ahead //g' | sed 's/\].*//g')"
echo "$line" | grep -E -q 'behind' &&
marks="$marks $SYMBOL_GIT_PULL$(echo "$line" | sed 's/.*\[behind //g' | sed 's/\].*//g')"
else
marks="$SYMBOL_GIT_MODIFIED$marks"
break
fi
done
printf ' %s%s' "$ref" "$marks"
}
}
sed searches for [ahead and deletes it, as well as everything before it, then it pipes it into another sed which deletes everything past ]. This way only the number remains.

Related

Convert folder and file names to camel case

I have a list of folders and files whose names contain spaces. How can I change the names into camel case?
for oldname in *
do
newname=`echo $oldname | sed -e 's/ /_/g'`
if [ "$newname" = "$oldname" ]
then
continue
fi
if [ -e "$newname" ]
then
echo Skipping "$oldname", because "$newname" exists
else
mv "$oldname" "$newname"
fi
done
I have found this but it changes the spaces into underscores.
Try this Shellcheck-clean Bash code:
#! /bin/bash -p
lowers=abcdefghijklmnopqrstuvwxyz
uppers=ABCDEFGHIJKLMNOPQRSTUVWXYZ
for oldname in *; do
[[ $oldname == *[[:space:]]* ]] || continue
read -r -d '' -a parts <<<"$oldname"
newname=''
for p in "${parts[#]}"; do
char1=${p:0:1}
if [[ $lowers == *"$char1"* ]]; then
tmp=${lowers%"$char1"*}
uchar1=${uppers:${#tmp}:1}
newname+=${uchar1}${p:1}
else
newname+=$p
fi
done
if [[ -e $newname ]]; then
printf "Skipping '%s', because '%s' exists\\n" "$oldname" "$newname" >&2
else
echo mv -v -- "$oldname" "$newname"
fi
done
The code is intended to work with (the now ancient) Bash 3 because my understanding is that that is still the current version of the standard Bash on macOS. The code for uppercasing the first letter of filename parts is much more complicated than it would be with later versions of Bash (which have built-in mechanisms for case conversion). See How to convert a string to lower case in Bash? for information about changing case in various ways in various versions of Bash.
The code just prints the mv command that would be run. Remove the echo to make it actually do the mv.
See the accepted, and excellent, answer to Why is printf better than echo? for an explanation of why I replaced echo with printf for the "Skipping" message.
For comparison, this is Bash 4+ code:
#! /bin/bash -p
for oldname in *; do
[[ $oldname == *[[:space:]]* ]] || continue
read -r -d '' -a parts <<<"$oldname"
newname=''
for p in "${parts[#]}"; do
newname+=${p^}
done
if [[ -e $newname ]]; then
printf "Skipping '%s', because '%s' exists\\n" "$oldname" "$newname" >&2
else
echo mv -v -- "$oldname" "$newname"
fi
done
You can use the regular expression aptitude to deal with upper and lower case translations, regarding your current local collation (LC_ALL, check with the locale command).
If your filename's "words" are separated with a space and are all in lower case, you can use a simple shell script like this :
#!/bin/sh
while read -r FILENAME ; do
NEWNAME="`echo \"${FILENAME}\" | sed 's/ *\([^ ]\)/\u\1/g'`"
if [ ! "${NEWNAME}" ] ; then
NEWNAME="${FILENAME}";
fi
if [ "${FILENAME}" = "${NEWNAME}" ]; then
printf "No change : %s\\n" "${FILENAME}" >&2;
else
if [ -e "${NEWNAME}" ] ; then
printf "Already changed : %s => %s\\n" "${FILENAME}" "${NEWNAME}" >&2;
else
echo "mv \"${FILENAME}\" \"${NEWNAME}\"";
fi
fi
done
Remove the echo on echo "mv \"${FILENAME}\" \"${NEWNAME}\""; to do the mv.
Note that it should work fine with accented letters or any unicode letter having lower and upper code.
The script takes the file list to operate from stdin, so to use it "as is", you can use something like the following examples :
find . -type 'f' | theScript.sh
For a whole tree of files.
For folders, you'll have to operate them separately. List them and sort them in a descending order.
ls -1 | theScript.sh
For files in the current folder.
If your files may have all or partial upper cases at start and you look to force them entirely to camel case, you can change the line :
NEWNAME="`echo \"${FILENAME}\" | sed 's/ *\([^ ]\)/\u\1/g'`"
With:
NEWNAME="\`echo \"${FILENAME}\" | sed 's/\(.*\)/\l\1/;s/ *\([^ ]\)/\u\1/g'\`"
If you have rename installed, then all you need to do is :
rename 's/ /_/g' *

Intermittent piping failure in bash

I have a code snippet that looks like this
while grep "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#"; do
grep -n "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#" | head -n1 | while read -r line ; do
lineno=$(echo $line | cut -d':' -f1)
spaces=$(sed "${lineno}!d" /tmp/kubernetes/$basefile | awk -F'[^ \t]' '{print length($1)}')
spaces=$((spaces-1))
# Delete line that had {{SECRETS}}
sed -i -e "${lineno}d" /tmp/kubernetes/$basefile
while IFS='' read -r secretline || [[ -n "$secretline" ]]; do
newline=$(printf "%*s%s" $spaces "" "$secretline")
sed -i "${lineno}i\ ${newline}" /tmp/kubernetes/$basefile
lineno=$((lineno+1))
done < "/tmp/secrets.yaml"
done
done
in /tmp/kubernetes/$basefile, the string {{SECRETS}} appears twice 100% of the time.
Almost every single time, this completes fine. However, very infrequently, the script errors on its second loop through the file. like so, according to set -x
...
IFS=
+ read -r secretline
+ [[ -n '' ]]
+ read -r line
exit code 1
When it works, the set -x looks like this, and continues processesing the file correctly.
...
+ IFS=
+ read -r secretline
+ [[ -n '' ]]
+ read -r line
+ grep '{{SECRETS}}' /tmp/kubernetes/deployment.yaml
+ grep -v '#'
I have no answer for how this can only happen occasionally, so I think there's something about bash piping's parallelism I don't understand. Is there something in grep -n "{{SECRETS}}" /tmp/kubernetes/$basefile | grep -v "#" | head -n1 | while read -r line ; do that could lead to out-of-order execution somehow? Based on the error, it seems like it's trying to read a line, but can't because previous commands didn't work. But there's no indication of that in the set -x output.
A likely cause of the problem is that the pipeline containing the inner loop both reads and writes the "basefile" at the same time. See How to make reading and writing the same file in the same pipeline always “fail”?.
One way to fix the problem is do a full read of the file before trying to update it. Try:
basepath=/tmp/kubernetes/$basefile
secretspath=/tmp/secrets.yaml
while
line=$(grep -n "{{SECRETS}}" "$basepath" | grep -v "#" | head -n1)
[[ -n $line ]]
do
lineno=$(echo "$line" | cut -d':' -f1)
spaces=$(sed "${lineno}!d" "$basepath" \
| awk -F'[^ \t]' '{print length($1)}')
spaces=$((spaces-1))
# Delete line that had {{SECRETS}}
sed -i -e "${lineno}d" "$basepath"
while IFS='' read -r secretline || [[ -n "$secretline" ]]; do
newline=$(printf "%*s%s" $spaces "" "$secretline")
sed -i "${lineno}i\ ${newline}" "$basepath"
lineno=$((lineno+1))
done < "$secretspath"
done
(I introduced the variables basepath and secretspath to make the code easier to test.)
As an aside, it's also possible to do this with pure Bash code:
basepath=/tmp/kubernetes/$basefile
secretspath=/tmp/secrets.yaml
updated_lines=()
is_updated=0
while IFS= read -r line || [[ -n $line ]] ; do
if [[ $line == *'{{SECRETS}}'* && $line != *'#'* ]] ; then
spaces=${line%%[^[:space:]]*}
while IFS= read -r secretline || [[ -n $secretline ]]; do
updated_lines+=( "${spaces}${secretline}" )
done < "$secretspath"
is_updated=1
else
updated_lines+=( "$line" )
fi
done <"$basepath"
(( is_updated )) && printf '%s\n' "${updated_lines[#]}" >"$basepath"
The whole updated file is stored in memory (in the update_lines array) but that shouldn't be a problem because any file that's too big to store in memory will almost certainly be too big to process line-by-line with Bash. Bash is generally extremely slow.
In this code spaces holds the actual space characters for indentation, not the number of them.

Using grep -q in shell one-liners

I've written a script to list commits in a repo that contain a specific file. It's working perfectly, but I don't understand why I had to write this:
for c in $(git rev-list "$rev_list"); do
git ls-tree --name-only -r "$c" | grep -q "$file"
if [ $? -eq 0 ]; then
echo "Saw $file in $c"
fi
done
Whereas I normally write the same thing like this:
[[ $(git ls-tree --name-only -r "$c" | grep -q "$file") ]] && echo "Saw $file in $c"
# or
[[ ! $(git ls-tree --name-only -r "$c" | grep -q "$file") ]] || echo "Saw $file in $c"
Neither of the short versions work: they don't output anything. When I write it so that it shows all commits that don't contain the file, I do get output:
[[ $(git ls-tree --name-only -r "$c" | grep -q "$file") ]] || echo "Did not see $file in $c"
However, if I then take a commit hash from the output and run
git ls-tree -r <the hash> | grep file
I notice the file is in the tree for some commits, leading me to believe it's just listing all the commits the script processes. Either way, I'm probably missing something, but I can't exactly work out what it is
You don't need to wrap the command in a conditional statement ([[ $(command) ]]). In fact, that will never work with grep -q, because you're actually testing whether the command prints anything. You can just do this:
git ls-tree --name-only -r "$c" | grep -q "$file" && echo "Saw $file in $c"
In general, any code block like
foreground_command
if [ $? -eq 0 ]
then
bar
fi
can be replaced with either
if foreground_command
then
bar
fi
or even
foreground_command && bar
Which of the three alternatives you should use depends on whether foreground_command, bar, or both are multi-line commands.
awk to the rescue:
git ls-tree --name-only -r "$c" | awk "/$file/{printf '%s in %s\n', '$file', '$c'}"

Bash ping output in csv format

My aim is to transform the output (the last 2 lines) of the ping command in a CSV style.
Here are some examples:
In case there is a packet loss lower than 100% <
URL, PacketLoss, Min, Average, Max, Deviation
In case there is packet loss equal to 100%
URL, 100, -1, -1, -1, -1
My script is below, but when the packet loss is 100% the output is:
URL, 100,
So the problem is at the if statement, as it does not enter in elif, I use the same syntax as checking if the address is full or not (with "www." or not).
Can you please have a look because I tried multiple things and it did not work.
My script:
#!/bin/bash
declare site=''
declare result='';
if [[ "$1" == "www."* ]]; then
site="$1";
else
site="www.$1";
fi
result="$site";
pingOutput=$(ping $site -c10 -i0.2 -q| tail -n2);
fl=true;
while IFS= read -r line
do
# !!! The problem is here, the if statement is not working properly and I do not know why !!!
if [ "$fl" == "true" ]; then
result="$result $(echo "$line" | cut -d',' -f3 | cut -d" " -f2 | sed -r 's/%//g')";
fl=false;
elif [[ "$line" == "ms"* ]]; then
result="$result $(echo "$line" | cut -d' ' -f4 | sed -r 's/\// /g')";
else
result="$result -1 -1 -1 -1";
fi
done <<< "$pingOutput"
echo "$result";
This is a pretty old question but I've just stumbled upon it today. Below I paste a slight modified version of the above script that fixes the if issue and works on Mac OS.
P.S. You can uncomment the # prctg=100.0% line to see the if working.
#!/bin/bash
declare site=''
declare result=''
declare prctg=''
[[ "$1" == "www."* ]] && site="$1" || site="www.$1"
result="$site"
pingOutput=$(ping $site -c10 -i0.2 -q | tail -n2)
fl=true
while IFS= read -r line
do
#echo $line
if [ "$fl" == "true" ]
then
prctg=$(echo "$line" | grep -Eo "[.[:digit:]]{1,10}%")
result="$result,$prctg"
fl=false
# prctg=100.0%
else
if [ "$prctg" == "100.0%" ]
then
result="$result,-1,-1,-1,-1"
else
result="$result,$(echo "$line" | cut -d' ' -f4 | sed -E 's/\//,/g')"
fi
fi
done <<< "$pingOutput"
echo "$result"
I hope it helps someone from the future! :)
Since the second line of the pingOutput was never processed (the loop ended before) the action of adding the -1 to the output was never performed.
Due to this problem I decided to capture the percentage of failure and act when no packets were returned (100%), I also simplified some expressions you used initially.
I investigated the script and came up with the following solution:
#!/bin/bash
declare site=''
declare result=''
declare prctg=''
[[ "$1" == "www."* ]] && site="$1" || site="www.$1"
result="$site"
pingOutput=$(ping $site -c10 -i0.2 -q| tail -n2)
fl=true
while IFS= read -r line
do
# !!! The problem is here, the if statement is not working properly and I do not know why !!!
echo $line
if [ "$fl" == "true" ]
then
prctg=$(echo "$line" | grep -Po "[0-9]{0,3}(?=%)")
result="$result $prctg"
fl=false
fi
if [ "$prctg" == "100" ]
then
result="$result -1 -1 -1 -1"
else
result="$result $(echo "$line" | cut -d' ' -f4 | sed -r 's/\// /g')"
fi
done <<< "$pingOutput"
echo "$result"

Bash string variable won't pass value

If the last pipe is removed, it seems the value will pass and things will work until the connection is no longer active. Then the value then goes empty or null with a double quote still there. The sed command can strip that but the pipe won't let the value afterwards be passed. I'm stuck.
iwgetid wlan0 | grep 'ESSID:' | cut -c 18-24 | wtf=$(echo "$1"
[[ -z "$1" ]] && echo -e "Wi-Fi Not Connected!" || echo -e "Connected"
Anything on the right-hand side of a pipeline is run in a subshell, meaning that assignments done there aren't visible anywhere else in your shell.
Also, where you get $1 from is unclear here -- the values from wtf aren't getting into the positional arguments by anything you're doing. Fixing that:
wtf=$(iwgetid wlan0 | grep 'ESSID:' | cut -c 18-24 | sed -e 's/^"//' -e 's/"$//')
[[ -z "$wtf" ]] && echo -e "Wi-Fi Not Connected!" || echo -e "Connected"
[[ ! -z "$wtf" ]] && echo -e "Connected" || echo -e "Wi-Fi Not Connected!"
...that said -- this is really awful code. Readers, please don't consider places where I'm quoting from the OP as condoning same. :)

Resources