I am trying to make a custom function (hisgrep) to grep from history.
I had it working before, when the code was basically "history | grep $1", but I want the implementation to be able to grep multiple keywords. (e.g. "hisgrep docker client" would equal "history | grep docker | grep client").
My problem is that, when I try to do this I get this error: "-bash: history: |: numeric argument required."
I've tried changing how the command was called in the end from $cmd to just $cmd, but that did nothing.
Here's the code:
#!/bin/bash
function hisgrep() {
cmd='history'
for arg in "$#"; do
cmd="$cmd | grep $arg"
done
`$cmd`
}
Sadly, bash doesn't have something called "foldl" or similar function.
You can do it like this:
histgrep() {
local str;
# save the history into some list
# I already filter the first argument, so the initial list is shorter
str=$(history | grep -e "$1");
shift;
# for each argument
for i; do
# pass the string via grep
str=$(<<<"$str" grep "$i")
done
printf "%s\n" "$str"
}
Notes:
Doing cmd="$cmd | grep $arg" and then doing `$cmd` looks unsafe.
Remember to quote your variables.
Use https://www.shellcheck.net/ to check your scripts.
Backticks ` are deprecated. Use $() command substitution.
using both function and parenthesis function func() is not portable. Just do func().
As for the unsafe version, you need to pass it via eval (and eval is evil), which by smart using printf shortens to just:
histgrep() { eval "history $(printf "| grep -e '%s' " "$#")"; }
But I think we can do a lot safer by expanding the arguments after command substitution, inside the eval call:
histgrep() { eval "history $(printf '| grep -e "$%s" ' $(seq $#))"; }
The eval here will see history | grep -e "$1" | grep -e "$2" | ... which I think looks actually quite safe.
It does not work because | is interpreted as an argument to the history command.
Related
I am creating this function to make multiple grep's over every line of a file. I run it as following:
cat file.txt | agrep string1 string2 ... stringN
function agrep () {
for a in $#; do
cmd+=" | grep '$a'";
done ;
while read line ; do
eval "echo "\'"$line"\'" $cmd";
done;
}
The idea is to print every line that contains all the strings: string1, string2, ..., stringN. This already works but I want to avoid the usage of the for to construct the expression:
| grep string1 | grep string2 ... | stringN
And if it's possible, also the usage of eval. I tried to make some expansion as follows:
echo "| grep $"{1..3}
And I get:
| grep $1 | grep $2 | grep $3
This is almost what I want but the problem is that when I try:
echo "| grep $"{1..$#}
The expansion doesn't occur because bash cant expand {1..$#} due to the $#. It just works with numbers. I would like to construct some expansion that works in order to avoid the usage of the for in the agrep function.
agrep () {
if [ $# = 0 ]; then
cat
else
pattern="$1"
shift
grep -e "$pattern" | agrep "$#"
fi
}
Instead of running each multiple greps on each line, just get all the lines that match string1, then pipe that to grep for string2, etc. One way to do this is make agrep recursive.
agrep () {
if (( $# == 0 )); then
cat # With no arguments, just output everything
else
grep "$1" | agrep "${#:2}"
fi
}
It's not the most efficient solution, but it's simple.
(Be sure to note Rob Mayoff's answer, which is the POSIX-compliant version of this.)
awk to the rescue!
you can avoid multiple grep calls and constructing the command by switching to awk
awk -v pat='string1 string2 string3' 'BEGIN{n=split(pat,p)}
{for(i=1;i<=n;i++) if($0!~p[i]) next}1 ' file
enter your space delimited strings as in the example above.
Not building a string for the command is definitely better (see chepner's and Rob Mayoff's answers). However, just as an example, you can avoid the for by using printf:
agrep () {
cmd=$(printf ' | grep %q' "$#")
sh -c "cat $cmd"
}
Using printf also helps somewhat with special characters in the patterns. From help printf:
In addition to the standard format specifications described in printf(1),
printf interprets:
%b expand backslash escape sequences in the corresponding argument
%q quote the argument in a way that can be reused as shell input
%(fmt)T output the date-time string resulting from using FMT as a format
string for strftime(3)
Since the aim of %q is providing output suitable for shell input, this should be safe.
Also: You almost always want to use "$#" with the quotes, not just plain $#.
I'm trying to define a bash function, highlight, that I can use to highlight search terms in the output of a previous command. When I do this from the terminal, it works fine as follows:
# highlight all occurrences of bar in file foo
cat foo | egrep '(bar|$)'
Yes, catting is a simplified example, but it demonstrates how I can do this from the command line. I'd like to use this generically as: cat foo | hightlight bar
From what I've read, I can't simply pipe results to egrep like I hoped so I naively tried defining my bash function as:
highlight() {
while read line; do
pat="'("$1"|$)'"
echo \"$line\" | egrep $pat
done
}
However, this isn't working. Please advise.
Your quoting is just about entirely wrong.
pat="'("$1"|$)'"
you include literal single quotes in the pattern, and you're actually not quoting the function parameter.
echo \"$line\" | egrep $pat
You're including literal double quotes in the echo statement, and failing to quote both variables.
This is better:
highlight() {
while read -r line; do
pat="($1|$)"
echo "$line" | grep -E "$pat"
done
}
However, grep knows how to read from stdin, so simplify:
highlight() { grep -E "($1|$)"; }
I'm not sure what you read, but it is wrong. egrep stands for extended grep because it is using the extended POSIX regular expression syntax. It behaves like the standard grep -E
Read the man page:
egrep is the same as grep -E
It seems to me you should have changed this
pat="'("$1"|$)'"
To
pat="($1|\$)"
Also, I don't see the need of "$". In addition, I think it is better to move the initialization of "pat" out of the loop. Here is what I got (tested):
#!/bin/bash
function highlight {
pattern="$1"
while read line; do
echo "$line" | egrep --color "$pattern"
done
}
echo -e 'a\nb\nbar\nhibar' | highlight bar
I often find myself doing something like this a lot:
something | grep cat | grep bat | grep rat
when all I recall is that those three words must have occurred somewhere, in some order, in the output of something...Now, i could do something like this:
something | grep '.*cat.*bat.*rat.*'
but that implies ordering (bat appears after cat). As such, I was thinking of adding a bash function to my environment called mgrep which would turn:
mgrep cat bat rat
into
grep cat | grep bat | grep rat
but I'm not quite sure how to do it (or whether there is an alternative?). One idea would be to for loop over the parameters like so:
while (($#)); do
grep $1 some_thing > some_thing
shift
done
cat some_thing
where some_thing is possibly some fifo like when one does >(cmd) in bash but I'm not sure. How would one proceed?
I believe you could generate a pipeline one command at a time, by redirecting stdin at each step. But it's much simpler and cleaner to generate your pipeline as a string and execute it with eval, like this:
CMD="grep '$1' " # consume the first argument
shift
for arg in "$#" # Add the rest in a pipeline
do
CMD="$CMD | grep '$arg'"
done
eval $CMD
This will generate a pipeline of greps that always reads from standard input, as in your model. Note that it protects spaces in quoted arguments, so that it works correctly if you write:
mgrep 'the cat' 'the bat' 'the rat'
Thanks to Alexis, this is what I did:
function mgrep() #grep multiple keywords
{
CMD=''
while (($#)); do
CMD="$CMD grep \"$1\" | "
shift
done
eval ${CMD%| }
}
You can write a recursive function; I'm not happy with the base case, but I can't think of a better one. It seems a waste to need to call cat just to pass standard input to standard output, and the while loop is a bit inelegant:
mgrep () {
local e=$1;
# shift && grep "$e" | mgrep "$#" || while read -r; do echo "$REPLY"; done
shift && grep "$e" | mgrep "$#" || cat
# Maybe?
# shift && grep "$e" | mgrep "$#" || echo "$(</dev/stdin)"
}
my question seems to be general, but i can't find any answers.
In sed command, how can you replace the substitution pattern by a value returned by a simple bash function.
For instance, I created the following function :
function parseDates(){
#Some process here with $1 (the pattern found)
return "dateParsed;
}
and the folowing sed command :
myCatFile=`sed -e "s/[0-3][0-9]\/[0-1][0-9]\/[0-9][0-9]/& parseDates &\}/p" myfile`
I found that the caracter '&' represents the current pattern found, i'd like it to be passed to my bash function and the whole pattern to be substituted by the pattern found +dateParsed.
Does anybody have an idea ?
Thanks
you can use the "e" option in sed command like this:
cat t.sh
myecho() {
echo ">>hello,$1<<"
}
export -f myecho
sed -e "s/.*/myecho &/e" <<END
ni
END
you can see the result without "e":
cat t.sh
myecho() {
echo ">>hello,$1<<"
}
export -f myecho
sed -e "s/.*/myecho &/" <<END
ni
END
Agree with Glenn Jackman.
If you want to use bash function in sed, something like this :
sed -rn 's/^([[:digit:].]+)/`date -d #&`/p' file |
while read -r line; do
eval echo "$line"
done
My file here begins with a unix timestamp (e.g. 1362407133.936).
Bash function inside sed (maybe for other purposes):
multi_stdin(){ #Makes function accepet variable or stdin (via pipe)
[[ -n "$1" ]] && echo "$*" || cat -
}
sans_accent(){
multi_stdin "$#" | sed '
y/àáâãäåèéêëìíîïòóôõöùúûü/aaaaaaeeeeiiiiooooouuuu/
y/ÀÁÂÃÄÅÈÉÊËÌÍÎÏÒÓÔÕÖÙÚÛÜ/AAAAAAEEEEIIIIOOOOOUUUU/
y/çÇñÑߢÐð£Øø§µÝý¥¹²³ªº/cCnNBcDdLOoSuYyY123ao/
'
}
eval $(echo "Rogério Madureira" | sed -n 's#.*#echo & | sans_accent#p')
or
eval $(echo "Rogério Madureira" | sed -n 's#.*#sans_accent &#p')
Rogerio
And if you need to keep the output into a variable:
VAR=$( eval $(echo "Rogério Madureira" | sed -n 's#.*#echo & | desacentua#p') )
echo "$VAR"
do it step by step. (also you could use an alternate delimiter , such as "|" instead of "/"
function parseDates(){
#Some process here with $1 (the pattern found)
return "dateParsed;
}
value=$(parseDates)
sed -n "s|[0-3][0-9]/[0-1][0-9]/[0-9][0-9]|& $value &|p" myfile
Note the use of double quotes instead of single quotes, so that $value can be interpolated
I'd like to know if there's a way to do this too. However, for this particular problem you don't need it. If you surround the different components of the date with ()s, you can back reference them with \1 \2 etc and reformat however you want.
For instance, let's reverse 03/04/1973:
echo 03/04/1973 | sed -e 's/\([0-9][0-9]\)\/\([0-9][0-9]\)\/\([0-9][0-9][0-9][0-9]\)/\3\/\2\/\1/g'
sed -e 's#[0-3][0-9]/[0-1][0-9]/[0-9][0-9]#& $(parseDates &)#' myfile |
while read -r line; do
eval echo "$line"
done
You can glue together a sed-command by ending a single-quoted section, and reopening it again.
sed -n 's|[0-3][0-9]/[0-1][0-9]/[0-9][0-9]|& '$(parseDates)' &|p' datefile
However, in contrast to other examples, a function in bash can't return strings, only put them out:
function parseDates(){
# Some process here with $1 (the pattern found)
echo dateParsed
}
Most scripts that parse /proc/cmdline break it up into words and then filter out arguments with a case statement, example:
CMDLINE="quiet union=aufs wlan=FOO"
for x in $CMDLINE
do
»···case $x in
»···»···wlan=*)
»···»···echo "${x//wlan=}"
»···»···;;
»···esac
done
The problem is when the WLAN ESSID has spaces. Users expect to set wlan='FOO
BAR' (like a shell variable) and then get the unexpected result of 'FOO with the above code, since the for loop splits on spaces.
Is there a better way of parsing the /proc/cmdline from a shell script falling short of almost evaling it?
Or is there some quoting tricks? I was thinking I could perhaps ask users to entity quote spaces and decode like so: /bin/busybox httpd -d "FOO%20BAR". Or is that a bad solution?
There are some ways:
cat /proc/PID/cmdline | tr '\000' ' '
cat /proc/PID/cmdline | xargs -0 echo
These will work with most cases, but will fail when arguments have spaces in them. However I do think that there would be better approaches than using /proc/PID/cmdline.
set -- $(cat /proc/cmdline)
for x in "$#"; do
case "$x" in
wlan=*)
echo "${x#wlan=}"
;;
esac
done
Most commonly, \0ctal escape sequences are used when spaces are unacceptable.
In Bash, printf can be used to unescape them, e.g.
CMDLINE='quiet union=aufs wlan=FOO\040BAR'
for x in $CMDLINE; do
[[ $x = wlan=* ]] || continue
printf '%b\n' "${x#wlan=}"
done
Since you want the shell to parse the /proc/cmdline contents, it's hard to avoid eval'ing it.
#!/bin/bash
eval "kernel_args=( $(cat /proc/cmdline) )"
for arg in "${kernel_args[#]}" ; do
case "${arg}" in
wlan=*)
echo "${arg#wlan=}"
;;
esac
done
This is obviously dangerous though as it would blindly run anything that was specified on the kernel command-line like quiet union=aufs wlan=FOO ) ; touch EVIL ; q=( q.
Escaping spaces (\x20) sounds like the most straightforward and safe way.
A heavy alternative is to use some parser, which understand shell-like syntax.
In this case, you may not even need the shell anymore.
For example, with python:
$ cat /proc/cmdline
quiet union=aufs wlan='FOO BAR' key="val with space" ) ; touch EVIL ; q=( q
$ python -c 'import shlex; print shlex.split(None)' < /proc/cmdline
['quiet', 'union=aufs', 'wlan=FOO BAR', 'key=val with space', ')', ';', 'touch', 'EVIL', ';', 'q=(', 'q']
Use xargs -n1:
[centos#centos7 ~]$ CMDLINE="quiet union=aufs wlan='FOO BAR'"
[centos#centos7 ~]$ echo $CMDLINE
quiet union=aufs wlan='FOO BAR'
[centos#centos7 ~]$ echo $CMDLINE | xargs -n1
quiet
union=aufs
wlan=FOO BAR
[centos#centos7 ~]$ xargs -n1 -a /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.14.4.el7.x86_64
root=UUID=3260cdba-e07e-408f-93b3-c4e9ff55ab10
ro
consoleblank=0
crashkernel=auto
rhgb
quiet
LANG=en_US.UTF-8
You could do something like the following using bash, which would turn those arguments in to variables like $cmdline_union and $cmdline_wlan:
bash -c "for i in $(cat /proc/cmdline); do printf \"cmdline_%q\n\" \"\$i\"; done" | grep = > /tmp/cmdline.sh
. /tmp/cmdline.sh
Then you would quote and/or escape things just like you would in a normal shell.
In posh:
$ f() { echo $1 - $3 - $2 - $4
> }
$ a="quiet union=aufs wlan=FOO"
$ f $a
quiet - wlan=FOO - union=aufs -
You can define a function and give your $CMDLINE unquoted as an argument to the function. Then you'll invoke shell's parsing mechanisms. Note, that you should test this on the shell it will be working in -- zsh does some funny things with quoting ;-).
Then you can just tell the user to do quoting like in shell:
#!/bin/posh
CMDLINE="quiet union=aufs wlan=FOO"
f() {
while test x"$1" != x
do
case $1 in
union=*) echo ${1##union=}; shift;;
*) shift;;
esac
done
}
f $CMDLINE
(posh - Policy-compliant Ordinary SHell, a shell stripped of any features beyond standard POSIX)
Found here a nice way to do it with awk, unfortunately it will work only with doublequotes:
# Replace spaces outside double quotes with newlines
args=`cat /proc/cmdline | tr -d '\n' | awk 'BEGIN {RS="\"";ORS="\"" }{if (NR%2==1){gsub(/ /,"\n",$0);print $0} else {print $0}}'`
IFS='
'
for line in $args; do
key=${line%%=*}
value=${line#*=}
value=`echo $value | sed -e 's/^"//' -e 's/"$//'`
printf "%20s = %s\n" "$key" "$value"
done