Test if the input stream not empty without consuming data from it - bash

Consider next snippet.
generate_build_specs(){
echo ubuntu gcc9 debug
echo macos clang debug
echo macos gcc10 release
echo macos clang release
}
generate_build_specs | awk -v IGNORECASE=1 '/macos/ && /release/' |
if input_data_available ;then
echo "FIXME. Cannot build 'macos release' until it is fixed. Buildspec dropped: $(cat)"
fi
My idea is to echo message with contents of input stream only if data in stream available. Something like peek(char) in stream, not taking it from there.
Of course, I know a workaround. i.e
var="$(cat)"; if test -n "$var"; echo "blah:: $var" ; fi

You don't need to peel to the stream. Just store the output in a variable.
buildspec=$(generate_build_specs | awk -v IGNORECASE=1 '/macos/ && /release/')
if [[ -n "${buildspec}" ]]; then
printf "%s\n" "FIXME. Cannot build 'macos release' until it is fixed." \
"Buildspec dropped:" "${buildspec}"
fi
I used printf for getting newlines whre you might want them, you can also use echo 3 times.

Not exactly what you asked for, but for your case equivalent in functionality:
if generate_build_specs | awk -v IGNORECASE=1 '/macos/ && /release/' | grep -q .
then
echo No data produced
fi
If we are picky, we should point out that if the pipe produces data consisting of empty lines only, this would also cause a no data produced
If you really have exactly that awk command which you have posted (and did not show a simplified version here to explain your point), you could perhaps also have written
if generate_build_specs | grep -i macos | grep -iqv 'release'
then
..echo "No data produced which contains the strings 'macos' and 'release'"
fi
This means that the string macos must appear earlier in a line than the string release. Since I don't know how the expected output of generate_builds_specs looks like, I can't say whether this approach will work in your case.

During the discussion, I realized that
there is no way to check if a stream is ended without reading at least one char from it.
input_data_available could not be implemented in Bash. It is a conceptual question on streams.
So the current solution is to store input to variable, then work on it.
var="$(cat)"; if test -n "$var"; then echo "::ERROR:: $var" ; fi
A better (i.e. laconic) way to do the same is:
sed -zE 's,^.+$,::ERROR:: \0\n,'
It means prepend string and output only if input is not empty. Regexp ^.+$ checks if there is at least one symbol. But I think as all oneliners this one is harder to read.
Thank you everybody for participation.

Related

Delete duplicate commands of zsh_history keeping last occurence

I'm trying to write a shell script that deletes duplicate commands from my zsh_history file. Having no real shell script experience and given my C background I wrote this monstrosity that seems to work (only on Mac though), but takes a couple of lifetimes to end:
#!/bin/sh
history=./.zsh_history
currentLines=$(grep -c '^' $history)
wordToBeSearched=""
currentWord=""
contrastor=0
searchdex=""
echo "Currently handling a grand total of: $currentLines lines. Please stand by..."
while (( $currentLines - $contrastor > 0 ))
do
searchdex=1
wordToBeSearched=$(awk "NR==$currentLines - $contrastor" $history | cut -d ";" -f 2)
echo "$wordToBeSearched A BUSCAR"
while (( $currentLines - $contrastor - $searchdex > 0 ))
do
currentWord=$(awk "NR==$currentLines - $contrastor - $searchdex" $history | cut -d ";" -f 2)
echo $currentWord
if test "$currentWord" == "$wordToBeSearched"
then
sed -i .bak "$((currentLines - $contrastor - $searchdex)) d" $history
currentLines=$(grep -c '^' $history)
echo "Line deleted. New number of lines: $currentLines"
let "searchdex--"
fi
let "searchdex++"
done
let "contrastor++"
done
^THIS IS HORRIBLE CODE NOONE SHOULD USE^
I'm now looking for a less life-consuming approach using more shell-like conventions, mainly sed at this point. Thing is, zsh_history stores commands in a very specific way:
: 1652789298:0;man sed
Where the command itself is always preceded by ":0;".
I'd like to find a way to delete duplicate commands while keeping the last occurrence of each command intact and in order.
Currently I'm at a point where I have a functional line that will delete strange lines that find their way into the file (newlines and such):
#sed -i '/^:/!d' $history
But that's about it. Not really sure how get the expression to look for into a sed without falling back into everlasting whiles or how to delete the duplicates while keeping the last-occurring command.
The zsh option hist_ignore_all_dups should do what you want. Just add setopt hist_ignore_all_dups to your zshrc.
I wanted something similar, but I dont care about preserving the last one as you mentioned. This is just finding duplicates and removing them.
I used this command and then removed my .zsh_history and replacing it with the .zhistory that this command outputs
So from your home folder:
cat -n .zsh_history | sort -t ';' -uk2 | sort -nk1 | cut -f2- > .zhistory
This effectively will give you the file .zhistory containing the changed list, in my case it went from 9000 lines to 3000, you can check it with wc -l .zhistory to count the number of lines it has.
Please double check and make a backup of your zsh history before doing anything with it.
The sort command might be able to be modified to sort it by numerical value and somehow archieve what you want, but you will have to investigate further about that.
I found the script here, along with some commands to avoid saving duplicates in the future
I didn't want to rename the history file.
# dedupe_lines.zsh
if [ $# -eq 0 ]; then
echo "Error: No file specified" >&2
exit 1
fi
if [ ! -f $1 ]; then
echo "Error: File not found" >&2
exit 1
fi
sort $1 | uniq >temp.txt
mv temp.txt $1
Add dedupe_lines.zsh to your home directory, then make it executable.
chmod +x dedupe_lines.zsh
Run it.
./dedupe_lines.zsh .zsh_history

Bash Script IF statement not functioning

I am currently testing a simple dictionary attack using bash scripts. I have encoded my password "Snake" with sha256sum by simply typing the following command:
echo -n Snake | sha256sum
This produced the following:
aaa73ac7721342eac5212f15feb2d5f7631e28222d8b79ffa835def1b81ff620 *-
I then copy pasted the hashed string into the program, but the script is not doing what is intended to do. The script is (Note that I have created a test dictionary text file which only contains 6 lines):
echo "Enter:"
read value
cat dict.txt | while read line1
do
atax=$(echo -n "$line1" | sha256sum)
if [[ "$atax" == "$value" ]];
then
echo "Cracked: $line1"
exit 1
fi
echo "Trying: $line1"
done
Result:
Trying: Dog
Trying: Cat
Trying: Rabbit
Trying: Hamster
Trying: Goldfish
Trying: Snake
The code should display "Cracked: Snake" and terminate, when it compares the hashed string with the word "Snake". Where am I going wrong?
EDIT: The bug was indeed the DOS lines in my textfile. I made a unix file and the checksums matched. Thanks everyone.
One problem is that, as pakistanprogrammerclub points out, you're never initializing name (as opposed to line1).
Another problem is that sha256sum does not just print out the checksum, but also *- (meaning "I read the file from standard input in binary mode").
I'm not sure if there's a clean way to get just the checksum — probably there is, but I can't find it — but you can at least write something like this:
atax=$(echo -n "$name" | sha256sum | sed 's/ .*//')
(using sed to strip off everything from the space onwards).
couple issues - the variable name is not set anywhere - do you mean value? Also better form to use redirection instead of cat
while read ...; do ... done <dict.txt
Variables set by a while loop in a pipeline are not available in the parent shell not the other way around as I mistakenly said before - it's not an issue here though
Could be a cut n paste error - add an echo after the first read
echo "value \"$value\""
also after atax is set
echo "line1 \"$line1\" atax \"$atax\""

Performance with bash loop when renaming files

Sometimes I need to rename some amount of files, such as add a prefix or remove something.
At first I wrote a python script. It works well, and I want a shell version. Therefore I wrote something like that:
$1 - which directory to list,
$2 - what pattern will be replacement,
$3 - replacement.
echo "usage: dir pattern replacement"
for fname in `ls $1`
do
newName=$(echo $fname | sed "s/^$2/$3/")
echo 'mv' "$1/$fname" "$1/$newName&&"
mv "$1/$fname" "$1/$newName"
done
It works but very slowly, probably because it needs to create a process (here sed and mv) and destroy it and create same process again just to have a different argument. Is that true? If so, how to avoid it, how can I get a faster version?
I thought to offer all processed files a name (using sed to process them at once), but it still needs mv in the loop.
Please tell me, how you guys do it? Thanks. If you find my question hard to understand please be patient, my English is not very good, sorry.
--- update ---
I am sorry for my description. My core question is: "IF we should use some command in loop, will that lower performance?" Because in for i in {1..100000}; do ls 1>/dev/null; done creating and destroying a process will take most of the time. So what I want is "Is there any way to reduce that cost?".
Thanks to kev and S.R.I for giving me a rename solution to rename files.
Every time you call an external binary (ls, sed, mv), bash has to fork itself to exec the command and that takes a big performance hit.
You can do everything you want to do in pure bash 4.X and only need to call mv
pat_rename(){
if [[ ! -d "$1" ]]; then
echo "Error: '$1' is not a valid directory"
return
fi
shopt -s globstar
cd "$1"
for file in **; do
echo "mv $file ${file//$2/$3}"
done
}
Simplest first. What's wrong with rename?
mkdir tstbin
for i in `seq 1 20`
do
touch tstbin/filename$i.txt
done
rename .txt .html tstbin/*.txt
Or are you using an older *nix machine?
To avoid re-executing sed on each file, you could instead setup two name streams, one original, and one transformed, then sip from the ends:
exec 3< <(ls)
exec 4< <(ls | sed 's/from/to/')
IFS=`echo`
while read -u3 orig && read -u4 to; do
mv "${orig}" "${to}";
done;
I think you can store all of file names into a file or string, and use awk and sed do it once instead of one by one.

In a small script to monitor a folder for new files, the script seems to be finding the wrong files

I'm using this script to monitor the downloads folder for new .bin files being created. However, it doesn't seem to be working. If I remove the grep, I can make it copy any file created in the Downloads folder, but with the grep it's not working. I suspect the problem is how I'm trying to compare the two values, but I'm really not sure what to do.
#!/bin/sh
downloadDir="$HOME/Downloads/"
mbedDir="/media/mbed"
inotifywait -m --format %f -e create $downloadDir -q | \
while read line; do
if [ $(ls $downloadDir -a1 | grep '[^.].*bin' | head -1) == $line ]; then
cp "$downloadDir/$line" "$mbedDir/$line"
fi
done
The ls $downloadDir -a1 | grep '[^.].*bin' | head -1 is the wrong way to go about this. To see why, suppose you had files named a.txt and b.bin in the download directory, and then c.bin was added. inotifywait would print c.bin, ls would print a.txt\nb.bin\nc.bin (with actual newlines, not \n), grep would thin that to b.bin\nc.bin, head would remove all but the first line leaving b.bin, which would not match c.bin. You need to be checking $line to see if it ends in .bin, not scanning a directory listing. I'll give you three ways to do this:
First option, use grep to check $line, not the listing:
if echo "$line" | grep -q '[.]bin$'; then
Note that I'm using the -q option to supress grep's output, and instead simply letting the if command check its exit status (success if it found a match, failure if not). Also, the RE is anchored to the end of the line, and the period is in brackets so it'll only match an actual period (normally, . in a regular expression matches any single character). \.bin$ would also work here.
Second option, use the shell's ability to edit variable contents to see if $line ends in .bin:
if [ "${line%.bin}" != "$line" ]; then
the "${line%.bin}" part gives the value of $line with .bin trimmed from the end if it's there. If that's not the same as $line itself, then $line must've ended with .bin.
Third option, use bash's [[ ]] expression to do pattern matching directly:
if [[ "$line" == *.bin ]]; then
This is (IMHO) the simplest and clearest of the bunch, but it only works in bash (i.e. you must start the script with #!/bin/bash).
Other notes: to avoid some possible issues with whitespace and backslashes in filenames, use while IFS= read -r line; do and follow #shellter's recommendation about double-quotes religiously.
Also, I'm not very familiar with inotifywait, but AIUI its -e create option will notify you when the file is created, not when its contents are fully written out. Depending on the timing, you may wind up copying partially-written files.
Finally, you don't have any checking for duplicate filenames. What should happen if you download a file named foo.bin, it gets copied, you delete the original, then download a different file named foo.bin. As the script is now, it'll silently overwrite the first foo.bin. If this isn't what you want, you should add something like:
if [ ! -e "$mbedDir/$line" ]; then
cp "$downloadDir/$line" "$mbedDir/$line"
elif ! cmp -s "$downloadDir/$line" "$mbedDir/$line"; then
echo "Eeek, a duplicate filename!" >&2
# or possibly something more constructive than that...
fi

Match Substring from Command Output

There are some close candidates for this question having already been answered, and I've tried several methods of trying to solve the issue. Specifically, my scenario is this:
I have an array of utility names that may or may NOT be installed on a linux machine (e.g.: ssh, sudo, etc.), so I am trying to check if the utility exists or not based on the result of trying to invoke the utilities in turn. I'm trying to do this in bash. Bash version is 4.1.5(1) running on Ubuntu 10.10 but planned to deploy on BusyBox.
If the utility doesn't exist, then usually you get a message saying "not found" or it includes that exact string. Otherwise you get a usage message. I have tried some regex expressions for the grep I use, but it hasn't made any difference, which leads me to believe there is something more fundamentally flawed with my code.
I am fully aware there are utilities that do this, but with the environment I am working in I do not have access to things like dpkg to check utilities/packages. In short, the environment I plan to deploy this on has NO PACKAGE MANAGEMENT.
What I have roughly goes like this:
#!/bin/bash
TOOLS=( 'ssh' 'soodo' 'dhclient' 'iperf')
#list of tools is abridged for convenience and added 'soodo' as a sure miss
#add a ridiculous option flag so don't accidentally trip any real flags
if `echo ${TOOLS[0]} -222222 | grep -q "not found"`; then
echo "${TOOLS[0]} is not installed."
else echo `${TOOLS[0]} --version`
#I am aware that --version is not applicable for all utilities, but this is just
#for sake of example.
My problem is that the if never seems to be accurately picked up. If I tweark ` marks around it either creates false positives or false negatives on the if (e.g.: a program like soodo will be claimed to exist when it doesn't, and something like ssh will be reported as not installed even though it is).
If you guys need any further clarification on what I'm trying to do or the like, please ask. It's the least I can provide back in exchange for some insight by others.
#!/bin/bash
TOOLS=( 'ssh' 'soodo' 'dhclient' 'iperf')
#list of tools is abridged for convenience and added 'soodo' as a sure miss
for TOOL in ${TOOLS[#]}
do
which $TOOL > /dev/null
RESULT=$?
if [ $RESULT -eq 0 ]
then
echo $TOOL is available
else
echo $TOOL is not available
fi
done
For bash, type is the way to determine if a command is a program in your PATH, or a function or an alias.
TOOLS=( 'ssh' 'soodo' 'dhclient' 'iperf')
for tool in "${TOOLS[#]}"; do
if type -p "$tool" > /dev/null; then
echo "$tool is installed"
else
echo "$tool is not installed"
fi
done
The errors in what you're doing:
if `echo ${TOOLS[0]} -222222 | grep -q "not found"`; then
What's happening there:
first, echo ${TOOLS[#]} -222222 prints "ssh -222222" to stdout
that pipes into grep -q "not found" which prints nothing to stdout
the backticks substitute the output from the pipeline (a blank line, which is always the output from grep -q) into the if command, so you get if <a newline> ; then
You'll get the same result as if $(printf "\n"); then echo Y; else echo N; fi which is always true.
To do what you're attempting, you'd have to write:
if "${TOOLS[0]}" -222222 2>&1 | grep -q "not found"; then ...
That will execute the pipeline and then if will consider the exit status. Exit status zero is considered true, any other exit status is considered false.
However, don't do this to find out if a program exists.

Resources