I'm trying to define a bash function, highlight, that I can use to highlight search terms in the output of a previous command. When I do this from the terminal, it works fine as follows:
# highlight all occurrences of bar in file foo
cat foo | egrep '(bar|$)'
Yes, catting is a simplified example, but it demonstrates how I can do this from the command line. I'd like to use this generically as: cat foo | hightlight bar
From what I've read, I can't simply pipe results to egrep like I hoped so I naively tried defining my bash function as:
highlight() {
while read line; do
pat="'("$1"|$)'"
echo \"$line\" | egrep $pat
done
}
However, this isn't working. Please advise.
Your quoting is just about entirely wrong.
pat="'("$1"|$)'"
you include literal single quotes in the pattern, and you're actually not quoting the function parameter.
echo \"$line\" | egrep $pat
You're including literal double quotes in the echo statement, and failing to quote both variables.
This is better:
highlight() {
while read -r line; do
pat="($1|$)"
echo "$line" | grep -E "$pat"
done
}
However, grep knows how to read from stdin, so simplify:
highlight() { grep -E "($1|$)"; }
I'm not sure what you read, but it is wrong. egrep stands for extended grep because it is using the extended POSIX regular expression syntax. It behaves like the standard grep -E
Read the man page:
egrep is the same as grep -E
It seems to me you should have changed this
pat="'("$1"|$)'"
To
pat="($1|\$)"
Also, I don't see the need of "$". In addition, I think it is better to move the initialization of "pat" out of the loop. Here is what I got (tested):
#!/bin/bash
function highlight {
pattern="$1"
while read line; do
echo "$line" | egrep --color "$pattern"
done
}
echo -e 'a\nb\nbar\nhibar' | highlight bar
Related
I am trying to make a custom function (hisgrep) to grep from history.
I had it working before, when the code was basically "history | grep $1", but I want the implementation to be able to grep multiple keywords. (e.g. "hisgrep docker client" would equal "history | grep docker | grep client").
My problem is that, when I try to do this I get this error: "-bash: history: |: numeric argument required."
I've tried changing how the command was called in the end from $cmd to just $cmd, but that did nothing.
Here's the code:
#!/bin/bash
function hisgrep() {
cmd='history'
for arg in "$#"; do
cmd="$cmd | grep $arg"
done
`$cmd`
}
Sadly, bash doesn't have something called "foldl" or similar function.
You can do it like this:
histgrep() {
local str;
# save the history into some list
# I already filter the first argument, so the initial list is shorter
str=$(history | grep -e "$1");
shift;
# for each argument
for i; do
# pass the string via grep
str=$(<<<"$str" grep "$i")
done
printf "%s\n" "$str"
}
Notes:
Doing cmd="$cmd | grep $arg" and then doing `$cmd` looks unsafe.
Remember to quote your variables.
Use https://www.shellcheck.net/ to check your scripts.
Backticks ` are deprecated. Use $() command substitution.
using both function and parenthesis function func() is not portable. Just do func().
As for the unsafe version, you need to pass it via eval (and eval is evil), which by smart using printf shortens to just:
histgrep() { eval "history $(printf "| grep -e '%s' " "$#")"; }
But I think we can do a lot safer by expanding the arguments after command substitution, inside the eval call:
histgrep() { eval "history $(printf '| grep -e "$%s" ' $(seq $#))"; }
The eval here will see history | grep -e "$1" | grep -e "$2" | ... which I think looks actually quite safe.
It does not work because | is interpreted as an argument to the history command.
Is there a way to conditionally pipe the output of a command through sed, in a bash script? Depending upon a script option, I either want to pipe the output of a long pipe through sed, or omit the pipe through sed. Currently I'm doing
if [ $pipeit ]; then
sed_args='/omit this line/d'
else
sed_args='/$^/d' # pass-thru (what's a better sed pass thru?)
fi
some_cmd | sed "$sed_args"
I would keep it as simple as:
if [ $pipeit ]; then
some_cmd | sed '/omit this line/d'
else
some_cmd
fi
Why should you call sed if you don't need it? Just for your information, a possible sed command that does not change the input would be sed -n p
Btw, if some_cmd is kind of a large beast and you want to avoid duplicating it, wrap it into a function.
By default sed prints all lines:
if [ $pipeit ]; then
sed_args='/omit this line/d'
else
sed_args="" # pass-thru
fi
some_cmd | sed "${sed_args}"
There is this other tested solution:
some_cmd | if [ $pipeit ]; then
sed "/omit this line/d"
else
sed ""
fi
cat could be used instead of the sed ""
Finally, a string can be built and executed using eval.
some_cmd='printf "foo\n\nbar\n"'
if [ $pipeit ]; then
conditional_pipe='| sed "/foo/d"'
else
conditional_pipe=""
fi
eval "${some_cmd}" "${conditional_pipe}"
If some_cmd is complex it migth be tricky to build a string that would behave as expected with eval.
----
First solution for history
Using an impossible match with sed would make it print all lines to stdout:
$ printf "foo\n\nbar\n" | sed "/./{/^$/d}"
foo
bar
/./ selects a line with at least one char.
/^$/ selects an empty line.
I used many times [``] to capture output of command to a variable. but with following code i am not getting right output.
#!/bin/bash
export XLINE='($ZWP_SCRIP_NAME),$ZWP_LT_RSI_TRIGGER)R),$ZWP_RTIMER'
echo 'Original XLINE'
echo $XLINE
echo '------------------'
echo 'Extract all word with $ZWP'
#works fine
echo $XLINE | sed -e 's/\$/\n/g' | sed -e 's/.*\(ZWP[_A-Z]*\).*/\1/g' | grep ZWP
echo '------------------'
echo 'Assign all word with $ZWP to XVAR'
#XVAR doesn't get all the values
export XVAR=`echo $XLINE | sed -e 's/\$/\n/g' | sed -e 's/.*\(ZWP[_A-Z]*\).*/\1/g' | grep ZWP` #fails
echo "$XVAR"
and i get:
Original XLINE
($ZWP_SCRIP_NAME),$ZWP_LT_RSI_TRIGGER)R),$ZWP_RTIMER
------------------
Extract all word with $ZWP
ZWP_SCRIP_NAME
ZWP_LT_RSI_TRIGGER
ZWP_RTIMER
------------------
Assign all word with $ZWP to XVAR
ZWP_RTIMER
why XVAR doesn't get all the values?
however if i use $() to capture the out instead of ``, it works fine. but why `` is not working?
Having GNU grep you can use this command:
XVAR=$(grep -oP '\$\KZWP[A-Z_]+' <<< "$XLINE")
If you pass -P grep is using Perl compatible regular expressions. The key here is the \K escape sequence. Basically the regex matches $ZWP followed by one or more uppercase characters or underscores. The \K after the $ removes the $ itself from the match, while its presence is still required to match the whole pattern. Call it poor man's lookbehind if you want, I like it! :)
Btw, grep -o outputs every match on a single line instead of just printing the lines which match the pattern.
If you don't have GNU grep or you care about portability you can use awk, like this:
XVAR=$(awk -F'$' '{sub(/[^A-Z_].*/, "", $2); print $2}' RS=',' <<< "$XLINE")
First, the smallest change that makes your code "work":
echo "$XLINE" | tr '$' '\n' | sed -e 's/.*\(ZWP[_A-Z]*\).*/\1/g' | grep ZWP_
The use of tr replaces a sed expression that didn't actually do what you thought it did -- try looking at its output to see.
One sane alternative would be to rely on GNU grep's -o option. If you can't do that...
zwpvars=( ) # create a shell array
zwp_assignment_re='[$](ZWP_[[:alnum:]_]+)(.*)' # ...and a regex
content="$XLINE"
while [[ $content =~ $zwp_assignment_re ]]; do
zwpvars+=( "${BASH_REMATCH[1]}" ) # found a reference
content=${BASH_REMATCH[2]} # stuff the remaining content aside
done
printf 'Found variable: %s\n' "${zwpvars[#]}"
I have written the following filter as a function in my ~/.bash_profile:
hilite() {
export REGEX_SED=$(echo $1 | sed "s/[|()]/\\\&/g")
while read line
do
echo $line | egrep "$1" | sed "s/$REGEX_SED/\x1b[7m&\x1b[0m/g"
done
exit 0
}
to find lines of anything piped into it matching a regular expression, and highlight matches using ANSI escape codes on a VT100-compatible terminal.
For example, the following finds and highlights the strings bin, U or 1 which are whole words in the last 10 lines of /etc/passwd:
tail /etc/passwd | hilite "\b(bin|[U1])\b"
However, the script runs very slowly as each line forks an echo, egrep and sed.
In this case, it would be more efficient to do egrep on the entire input, and then run sed on its output.
How can I modify my function to do this? I would prefer to not create any temporary files if possible.
P.S. Is there another way to find and highlight lines in a similar way?
sed can do a bit of grepping itself: if you give it the -n flag (or #n instruction in a script) it won't echo any output unless asked. So
while read line
do
echo $line | egrep "$1" | sed "s/$REGEX_SED/\x1b[7m&\x1b[0m/g"
done
could be simplified to
sed -n "s/$REGEX_SED/\x1b[7m&\x1b[0m/gp"
EDIT:
Here's the whole function:
hilite() {
REGEX_SED=$(echo $1 | sed "s/[|()]/\\\&/g");
sed -n "s/$REGEX_SED/\x1b[7m&\x1b[0m/gp"
}
That's all there is to it - no while loop, reading, grepping, etc.
If your egrep supports --color, just put this in .bash_profile:
hilite() { command egrep --color=auto "$#"; }
(Personally, I would name the function egrep; hence the usage of command).
I think you can replace the whole while loop with simply
sed -n "s/$REGEX_SED/\x1b[7m&\x1b[0m/gp"
because sed can read from stdin line-by-line so you don't need read
I'm not sure if running egrep and piping to sed is faster than using sed alone, but you can always compare using time.
Edit: added -n and p to sed to print only highlighted lines.
Well, you could simply do this:
egrep "$1" $line | sed "s/$REGEX_SED/\x1b[7m&\x1b[0m/g"
But I'm not sure that it'll be that much faster ; )
Just for the record, this is a method using a temporary file:
hilite() {
export REGEX_SED=$(echo $1 | sed "s/[|()]/\\\&/g")
export FILE=$2
if [ -z "$FILE" ]
then
export FILE=~/tmp
echo -n > $FILE
while read line
do
echo $line >> $FILE
done
fi
egrep "$1" $FILE | sed "s/$REGEX_SED/\x1b[7m&\x1b[0m/g"
return $?
}
which also takes a file/pathname as the second argument, for case like
cat /etc/passwd | hilite "\b(bin|[U1])\b"
my question seems to be general, but i can't find any answers.
In sed command, how can you replace the substitution pattern by a value returned by a simple bash function.
For instance, I created the following function :
function parseDates(){
#Some process here with $1 (the pattern found)
return "dateParsed;
}
and the folowing sed command :
myCatFile=`sed -e "s/[0-3][0-9]\/[0-1][0-9]\/[0-9][0-9]/& parseDates &\}/p" myfile`
I found that the caracter '&' represents the current pattern found, i'd like it to be passed to my bash function and the whole pattern to be substituted by the pattern found +dateParsed.
Does anybody have an idea ?
Thanks
you can use the "e" option in sed command like this:
cat t.sh
myecho() {
echo ">>hello,$1<<"
}
export -f myecho
sed -e "s/.*/myecho &/e" <<END
ni
END
you can see the result without "e":
cat t.sh
myecho() {
echo ">>hello,$1<<"
}
export -f myecho
sed -e "s/.*/myecho &/" <<END
ni
END
Agree with Glenn Jackman.
If you want to use bash function in sed, something like this :
sed -rn 's/^([[:digit:].]+)/`date -d #&`/p' file |
while read -r line; do
eval echo "$line"
done
My file here begins with a unix timestamp (e.g. 1362407133.936).
Bash function inside sed (maybe for other purposes):
multi_stdin(){ #Makes function accepet variable or stdin (via pipe)
[[ -n "$1" ]] && echo "$*" || cat -
}
sans_accent(){
multi_stdin "$#" | sed '
y/àáâãäåèéêëìíîïòóôõöùúûü/aaaaaaeeeeiiiiooooouuuu/
y/ÀÁÂÃÄÅÈÉÊËÌÍÎÏÒÓÔÕÖÙÚÛÜ/AAAAAAEEEEIIIIOOOOOUUUU/
y/çÇñÑߢÐð£Øø§µÝý¥¹²³ªº/cCnNBcDdLOoSuYyY123ao/
'
}
eval $(echo "Rogério Madureira" | sed -n 's#.*#echo & | sans_accent#p')
or
eval $(echo "Rogério Madureira" | sed -n 's#.*#sans_accent &#p')
Rogerio
And if you need to keep the output into a variable:
VAR=$( eval $(echo "Rogério Madureira" | sed -n 's#.*#echo & | desacentua#p') )
echo "$VAR"
do it step by step. (also you could use an alternate delimiter , such as "|" instead of "/"
function parseDates(){
#Some process here with $1 (the pattern found)
return "dateParsed;
}
value=$(parseDates)
sed -n "s|[0-3][0-9]/[0-1][0-9]/[0-9][0-9]|& $value &|p" myfile
Note the use of double quotes instead of single quotes, so that $value can be interpolated
I'd like to know if there's a way to do this too. However, for this particular problem you don't need it. If you surround the different components of the date with ()s, you can back reference them with \1 \2 etc and reformat however you want.
For instance, let's reverse 03/04/1973:
echo 03/04/1973 | sed -e 's/\([0-9][0-9]\)\/\([0-9][0-9]\)\/\([0-9][0-9][0-9][0-9]\)/\3\/\2\/\1/g'
sed -e 's#[0-3][0-9]/[0-1][0-9]/[0-9][0-9]#& $(parseDates &)#' myfile |
while read -r line; do
eval echo "$line"
done
You can glue together a sed-command by ending a single-quoted section, and reopening it again.
sed -n 's|[0-3][0-9]/[0-1][0-9]/[0-9][0-9]|& '$(parseDates)' &|p' datefile
However, in contrast to other examples, a function in bash can't return strings, only put them out:
function parseDates(){
# Some process here with $1 (the pattern found)
echo dateParsed
}