I try to generate a csv file through a lot of functions like that :
function get_sudo_version {
sudo -V 2>/dev/null|grep -i "sudo version"
}
sudo_version=$(get_sudo_version)
Function above is a simple example but in some cases i cannot be sure the output is correct.
I would like to know what is the best way to validate the function return one text line only.
I thought about something like that
function validate_output {
output=$1;
echo $1|grep -q "\n";
echo $?;
}
mytest="val1
err2
err3"
But it's obviously not working because the variable does not keep the retrun line character:
echo $mytest
val1 err2 err3
So if someone has a good idea of how i could wirte a generic check function i would be glad.
Thanks
If you have GNU grep, you could simply ensure that grep doesn't produce more than one line of output in the first place via grep -m 1. Alternatively, use sed '/sudo version/!d;q' instead of grep.
A function that simply checks lines of input while passing them through might look like:
shopt -s lastpipe # if bash
# ksh or bash
validate_output() {
(( $(tee >(wc -l) >&2) == 1 ))
} 2>&1
# bash
validate_output2() {
local lines
tee /dev/fd/2 | mapfile -tc1 -C '((++lines < 2)) || return; :'
((lines))
} 2>&1
get_sudo_version | validate_output || echo failed
Many variations on that possible of course. IMO it's pretty pointless and shouldn't be used for something like this. Just design your get_sudo_version so that it guarantees the right results.
You can count lines with wc -l:
$ LINES="$(echo -n | wc -l)"; [[ "$LINES" -gt 0 ]] && echo 'at least one line'
$ LINES="$(echo -e 'a\nb\nc' | wc -l)"; [[ "$LINES" -gt 0 ]] && echo 'at least one line'
at least one line
Related
I have a bash script that produces some text from a pipe of commands. Based on a command line option I want to do some validation on the output. For a contrived example...
CHECK_OUTPUT=$1
...
check_output()
{
if [[ "$CHECK_OUTPUT" != "--check" ]]; then
# Don't check the output. Passthrough and return.
cat
return 0
fi
# Check each line exists in the fs root
while read line; do
if [[ ! -e "/$line" ]]; then
echo "Error: /$line does not exist"
return 1
fi
echo "$line"
done
return 0
}
ls /usr | grep '^b' | check_output
[EDIT] better example: https://stackoverflow.com/a/52539364/1888983
This is really useful, particularly if I have multiple functions that can becomes passthroughs. Yes, I could move the CHECK_OUTPUT conditional and create a pipe with or without check_output but I'd need to write lines for each combination for more functions. If there are better ways to dynamically build a pipe I'd like to know.
The problem is the "useless use of cat". Can this be avoided and make check_output like it wasn't in the pipe at all?
Yes, you can do this -- by making your function a wrapper that conditionally injects a pipeline element, instead of being an unconditional pipeline element itself. For example:
maybe_checked() {
if [[ $CHECK_OUTPUT != "--check" ]]; then
"$#" # just run our arguments as a command, as if we weren't here
else
# run our arguments in a process substitution, reading from stdout of same.
# ...some changes from the original code:
# IFS= stops leading or trailing whitespace from being stripped
# read -r prevents backslashes from being processed
local line # avoid modifying $line outside our function
while IFS= read -r line; do
[[ -e "/$line" ]] || { echo "Error: /$line does not exist" >&2; return 1; }
printf '%s\n' "$line" # see https://unix.stackexchange.com/questions/65803
done < <("$#")
fi
}
ls /usr | maybe_checked grep '^b'
Caveat of the above code: if the pipefail option is set, you'll want to check the exit status of the process substitution to have complete parity with the behavior that would otherwise be the case. In bash version 4.3 or later (IIRC), $? is modified by process substitutions to have the relevant PID, which can be waited for to retrieve exit status.
That said, this is also a use case wherein using cat is acceptable, and I'm saying this as a card-carying member of the UUOC crowd. :)
Adopting the examples from John Kugelman's answers on the linked question:
maybe_sort() {
if (( sort )); then
"$#" | sort
else
"$#"
fi
}
maybe_limit() {
if [[ -n $limit ]]; then
"$#" | head -n "$limit"
else
"$#"
fi
}
printf '%s\n' "${haikus[#]}" | maybe_limit maybe_sort sed -e 's/^[ \t]*//'
I am trying to execute a word count command on a log file and if the file has the "error" string, I want to take some action, but I can't seem to properly convert the grep to word count command to a real number so it compares properly to the greater than zero. So far with several variations, the conditional statement is always true.
if ((grep -Ei "error" myfile.log | wc -l)) > 0; then echo 1; else echo 0; fi
First of all, you can write conditions based on the exit code of programs.
If grep finds a matching line, it exits with success:
if grep -qEi "error" myfile.log; then echo 1; else echo 0; fi
I added the -q flag to not print the matching line, as you probably don't need it.
I strongly recommend to use the above solution, without wc.
But for the sake of completeness, here's some more explanation about different ways of comparing numbers.
One way to compare numbers is with -gt ("greater than") within [ ... ]:
if [ $(grep -Ei "error" myfile.log | wc -l) -gt 0 ]; then echo 1; else echo 0; fi
You can read about other operators within [ ... ] in help test.
Or using arithmetic context within ((...)):
if (($(grep -Ei "error" myfile.log | wc -l) > 0)); then echo 1; else echo 0; fi
Notice that in both of these examples I wrapped the grep ... | wc -l within a $(...) sub-shell to capture the output.
The syntax you wrote is incorrect.
I am learning bash. I would like to get the return value and matched line by grep at once.
if cat 'file' | grep 'match_word'; then
match_by_grep="$(cat 'file' | grep 'match_word')"
read a b <<< "${match_by_grep}"
fi
In the code above, I used grep twice. I cannot think of how to do it by grep once. I am not sure match_by_grep is always empty even when there is no matched words because cat may output error message.
match_by_grep="$(cat 'file' | grep 'match_word')"
if [[ -n ${match_by_grep} ]]; then
# match_by_grep may be an error message by cat.
# So following a and b may have wrong value.
read a b <<< "${match_by_grep}"
fi
Please tell me how to do it. Thank you very much.
You can avoid the double use of grep by storing the search output in a variable and seeing if it is not empty.
Your version of the script without double grep.
#!/bin/bash
grepOutput="$(grep 'match_word' file)"
if [ ! -z "$grepOutput" ]; then
read a b <<< "${grepOutput}"
fi
An optimization over the above script ( you can remove the temporary variable too)
#!/bin/bash
grepOutput="$(grep 'match_word' file)"
[[ ! -z "$grepOutput" ]] && (read a b <<< "${grepOutput}")
Using double-grep once for checking if-condition and once to parse the search result would be something like:-
#!/bin/bash
if grep -q 'match_word' file; then
grepOutput="$(grep 'match_word' file)"
read a b <<< "${grepOutput}"
fi
When assigning a variable with a string containing a command expansion, the return code is that of the (rightmost) command being expanded.
In other words, you can just use the assignment as the condition:
if grepOutput="$(cat 'file' | grep 'match_word')"
then
echo "There was a match"
read -r a b <<< "${grepOutput}"
(etc)
else
echo "No match"
fi
Is this what you want to achieve?
grep 'match_word' file ; echo $?
$? has a return value of the command run immediately before.
If you would like to keep track of the return value, it will be also useful to have PS1 set up with $?.
Ref: Bash Prompt with Last Exit Code
I have a series of commands chained together with pipes:
should_create_one_line | expects_one_line
The first command should_create_one_line should produce an output that only has one line, but under strange circumstances it is possible for the output to be multiline or empty.
I would like to add a step in between these two, validate_one_line:
should_create_one_line | validate_one_line | expects_one_line
If its input contains exactly 1 line then validate_one_line will simply output its input. If its input contains more than 1 line or is empty then validate_one_line should cause the whole sequence of steps to stop and return an error code.
What command can I use for validate_one_line?
Use read. Here's a shell function that meets your specs:
exactly_one_line() {
local line # Use to echo the line
read -r line || return # Guarantee at least one line is read
read && return 1 # Indicate failure if another line is successfully read
echo "$line"
}
Notes
"One line" assumes a single line followed by a newline. If your input could be like, a file with contents but no newlines, then this will fail.
Given a pipeline like a|b, a cannot prevent b from running. At a minimum, b needs to handle when a produces no output.
Demo:
$ wc -l empty oneline twolines
0 empty
1 oneline
2 twolines
3 total
$ exactly_one_line < empty; echo $?
1
$ exactly_one_line < oneline; echo $?
oneline
0
$ exactly_one_line < twolines; echo $?
1
First off, you should seriously consider adding the validation code to expects_one_line. According to this post, each process starts in its own subshell, meaning that even if validate_one_line fails, you will get an error in expects_one_line because it will try to run with no input (or a blank line). That being said, here is a bash one-liner that you can insert into your pipe to validate:
should_create_one_line.sh | ( var="$(cat)"; [ $(echo "$var" | wc -l) -ne 1 ] && exit 1 || echo "$var") | expects_one_line.sh
The problem here is that when the validation subshell returns in the exit 1 case, expects_one_line.sh will still get a single blank line. If this works for you, then great. If not, it would be better to just put the following into the beginning of expects_one_line.sh:
input="$(cat)"
[ $(echo "$var" | wc -l) -ne 1 ] && exit 1
This would guarantee that expects_one_line.sh fails properly when getting a single line without having to wonder about what the empty line that the validation outputs will do to the script.
You may find this post helpful: How to read mutliline input from stdin into variable and how to print one out in shell(sh,bash)?
You can use a bash script to check the incoming data and call the other command when the input is only 1 line
The following code starts cat when it is ONLY fet in 1 line
sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' cat
How this works
Try reading a line, if failed go to step 5
If variable $LINE is NOT empty, goto step 6
Save line inside variable $LINE
Goto step 1
If $LINE is NOT empty, goto step 7
Exit the program with status code 1
Call our program and pass our $line to it using printf
Example usage:
Printing out only if grep found 1 match:
grep .... | sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' cat
Example of the question poster:
should_create_one_line | sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' expects_one_line
I'm really struggling to see why this while-loop never ends, when the loop starts, my variable LOC is set to Testing/, which is a directory I created to test this program, it has the following layout:
I want the loop to end once all Directories have had the "count" function applied to them.
Here are the things I have tried;
I've checked my count function, and it doesn't produce an infinite loop
I've tried running through the algorithm by hand
PARSE=1
LOC=$LOC/
count
AVAILABLEDIR=$(ls $LOC -AFl | sed "1 d" | grep "/$" | awk '{ print $9 }')
while [ $PARSE = "1" ]
do
if [[ ${AVAILABLEDIR[#]} == '' ]]; then
PARSE=0
fi
DIRBASE=$LOC
for a in ${AVAILABLEDIR[#]}; do
LOC="${DIRBASE}${a}"
LOCLIST="$LOCLIST $LOC"
count
done
for a in ${LOCLIST[#]}; do
TMPAVAILABLEDIR=$(ls $a -AFl | sed "1 d" | grep "/$" | awk '{ print $9 }')
PREPEND=$a
if [[ ${TMPAVAILABLEDIR[#]} == '' ]]; then
continue
fi
for a in ${TMPAVAILABLEDIR[#]}; do
TMPAVAILABLEDIR2="$TMPAVAILABLEDIR2 ${PREPEND[#]}${a}"
done
NEWAVAILABLEDIR="$NEWAVAILABLEDIR $TMPAVAILABLEDIR2"
done
AVAILABLEDIR=$NEWAVAILABLEDIR
NEWAVAILABLEDIR=''
LOC=''
done
I am really struggling, and any input would be greatly appreciated, I've been trying to figure this out for the last couple of hours.
You should try to run the script with argument -x, or write it into the first line:
#!/bin/bash -x
Then it tells you everything it does.
In that case, you might notice two errors:
You never reset TMPAVAILABLEDIR2
You do ls on regular files as well.
If you really must avoid recursion, try this. It completely recursion-free:
#!/bin/bash
count() {
echo counting "$1"
}
todo=(Testing)
while test ${#todo[#]} != 0
do
doit=("${todo[#]}")
todo=()
for dir in "${doit[#]}"
do
for entry in "$dir"/* # If the directory is empty, this shows an entry named "*"
do
test -e "$entry" || continue # Skip the entry "*" of an empty directory
count "$entry"
test -d "$entry" || continue
todo+=("$entry")
done
done
done
You wrote you want to perform "count" on all directories.
Look at the options of find:
find $LOC -type d | while read dir; do
cd $LOC
cd ${dir}
count
done
Or shorter (when your function count accepts a directory as parameter 1):
find $LOC -type d | xargs count
I now see you do not want to use find or ls -R (recursive function). Then you should make your own recursive function, something like
function parseDir {
ls -d */ $1 | while read dir; do
count
parseDir $1/$dir
done
}
I have no idea if this will work, but it’s an interesting question I couldn't stop thinking about.
while true ; do
for word in "$(echo *)" ; do
if [[ -d "$word" ]] ; then
d[$((i++))]="$PWD"/"$word"
elif [[ -f "$word" ]] ;then
f[$((j++))]="$PWD"/"$word"
fi
done
[[ $k -gt $i ]] && cd ..
cd "$d[$((k++))]" || break
done