I want to merge output of three logwatch outputs and pipe result through sendmail.
Example:
#!/bin/sh
LOG_DIR="/var/log/remote-hosts"
MAIL_TO="me#email.com"
sh -c "logwatch && find ${LOG_DIR} -type d -name \"ip*\" -print0 | xargs -0 -I{} sh -c 'logwatch --logdir {} --hostname $(basename {})'" |
sed '1!b;s/^/To: '${MAIL_TO}'\nSubject: Logwatch report\n\n/' | sendmail -t
first logwatch is executed on /var/log folder
and then I would like to traverse /var/log/remote-hosts subfolders (ip-10-0-0-38 and ip-10-0-0-39 ) with find and also do logwatch on them.
The merged output will be sent throught sentmail. However I would like to replace hostname with basename of /var/log/remote-hosts subfolder so instead of /var/log/remote-hosts/ip-10-0-0-38 I will have ip-10-0-0-38 only.
But unfortunatelly I don't how to do the basename part correctly. Any help? Thanks in advance.
Don't use sh -c for grouping statements, use (...):
(logwatch && find ${LOG_DIR} -type d -name "ip*" -print0 | xargs -0 -I{} sh -c 'logwatch --logdir {} --hostname $(basename {})') |
sed '1!b;s/^/To: '${MAIL_TO}'\nSubject: Logwatch report\n\n/' | sendmail -t
Related
Based on the top answer in Running multiple commands with xargs I'm trying to use find / xargs to work upon more files. Why the first file 1.txt is missing in for loop?
$ ls
1.txt 2.txt 3.txt
$ find . -name "*.txt" -print0 | xargs -0
./1.txt ./2.txt ./3.txt
$ find . -name "*.txt" -print0 | xargs -0 sh -c 'for arg do echo "$arg"; done'
./2.txt
./3.txt
Why do you insist on using xargs? You can do the following as well.
while read -r file; do
echo $file
done <<<$(find . -name "*.txt")
Because this is executed in the same shell, changing variables is possible in the loop. Otherwise you'll get a sub-shell in which that doesn't work.
When you use your for-loop in a script example.sh, the call example.sh var1 var2 var3 will put var1 in the first argument, not example.sh.
When you want to process one file for each command, use the xargs option -L:
find . -name "*.txt" -print0 | xargs -0 -L1 sh -c 'echo "$0"'
# or for a simple case
find . -name "*.txt" -print0 | xargs -0 -L1 echo
I ran across this while having the same issue. You need the extra _ at the end as place holder 0 for xargs
$ find . -name "*.txt" -print0 | xargs -0 sh -c 'for arg do echo "$arg"; done' _
I am writing a command using find, grep and sort to display a sorted list of all files that contain 'some-text'.
I was unable to figure out the command.
Here is my attempt:
$find . -type f |grep -l "some-text" | sort
but it didn't work.
You need to use something like XARGS so that the content of each file passed through the pipe | is made available for grep.
XARGS: converts input from standard input into arguments to a command
In my case, I have files1,2,3 and they contain the word test. This will do it.
za:tmp za$ find . -type f | xargs grep -l "test" | sort
./file1.txt
./file2.txt
./file3.txt
or
za:tmp za$ find . -type f | xargs grep -i "test" | sort
./file1.txt:some test string
./file2.txt:some test string
./file3.txt:some test string
You can use it in any unix:
find . -type f -exec sh -c 'grep "some text" {} /dev/null > /dev/null 2>&1' \; -a -print 2> /dev/null|sort
A more optimized solution that works only with GNU-grep:
find . -type f -exec grep -Hq "some-text" {} \; -a -print 2> /dev/null|sort
I found an unexpected to me behavior of "find -exec" bash command and I would appreciate some interpretation. The same job can be done with "for file_name in find ....; do...." loop, so the question is why it doesn't work with -exec option of find.
There are two folders (SRC/ and src/) with the same set of files. I want to compare the files in these folders:
find src/ -type f -exec sh -c "diff {} `echo {} | sed 's/src/SRC/'`" \;
this, however, doesn't compare the files... Due to some reason sed command doesn't make the the substitution. If there is only one file, e.g., "a", in each of this folders then a command
find src/ -type f -exec sh -c "echo {} `echo {} | sed 's/src/SRC/'`" \;
outputs
src/a src/a
if one does a similar thing in bash, all the following commands give the same result (SRC/a):
echo src/a | sed 's/src/SRC/'
echo `echo src/a | sed 's/src/SRC/'`
sh -c "echo src/a | sed 's/src/SRC/'"
sh -c "echo `echo src/a | sed 's/src/SRC/'`"
but if this commands are supplied to "find -exec ..." the outputs are different:
find src/ -type f -exec bash -c "echo {} | sed 's/src/SRC/'" \;
gives "SRC/a"
and
find src/ -type f -exec bash -c "echo `echo {} | sed 's/src/SRC/'`" \;
gives "src/a"
Is that the expected behavior?
Use single quotes for sh -c for the script is interpreted by your shell first. And Pass the filename as an argument for sh instead of using {} inside the quotes:
find src/ -type f -exec sh -c 'diff "$1" "$(printf "%s\n" "$1" | sed "s/src/SRC/")"' _ {} \;
Or with bash:
find src/ -type f -exec bash -c 'diff "$1" "${1/src/SRC}"' _ {} \;
I'm using command find to recursively browse through directory tree, counting files, sizes, etc...
Now I need to get directory depth of each file.
Is there any portable way for both FreeBSD and CentOS?
I know that find is able to prinf actual directory depth but sadly this works only on CentOS, not FreeBSD.
Additionaly - I need to keep standard find output OR put directory depth on the beginning of output and cut it from there.
You can count the / in path :
$ find . -type f -exec bash -c 'echo '{}' | grep -o / | wc -l' \;
Or with file names :
$ mkdir -p one/two/three four/five && touch file one/two/file one/two/three/file
$ find . -type f -exec bash -c 'echo -n '{}' :; echo '{}' | grep -o / | wc -l' \;
./file :1
./one/two/file :3
./one/two/three/file :4
Try this:
find . -type d -exec bash -c 'echo $(tr -cd / <<< "$1"|wc -c):$1' -- {} \; | sort -n | tail -n 1 | awk -F: '{print $1, $2}'
I want to find a bash command that will let me grep every file in a directory and write the output of that grep to a separate file. My guess would have been to do something like this
ls -1 | xargs -I{} "grep ABC '{}' > '{}'.out"
but, as far as I know, xargs doesn't like the double-quotes. If I remove the double-quotes, however, then the command redirects the output of the entire command to a single file called '{}'.out instead of to a series of individual files.
Does anyone know of a way to do this using xargs? I just used this grep scenario as an example to illustrate my problem with xargs so any solutions that don't use xargs aren't as applicable for me.
Do not make the mistake of doing this:
sh -c "grep ABC {} > {}.out"
This will break under a lot of conditions, including funky filenames and is impossible to quote right. Your {} must always be a single completely separate argument to the command to avoid code injection bugs. What you need to do, is this:
xargs -I{} sh -c 'grep ABC "$1" > "$1.out"' -- {}
Applies to xargs as well as find.
By the way, never use xargs without the -0 option (unless for very rare and controlled one-time interactive use where you aren't worried about destroying your data).
Also don't parse ls. Ever. Use globbing or find instead: http://mywiki.wooledge.org/ParsingLs
Use find for everything that needs recursion and a simple loop with a glob for everything else:
find /foo -exec sh -c 'grep "$1" > "$1.out"' -- {} \;
or non-recursive:
for file in *; do grep "$file" > "$file.out"; done
Notice the proper use of quotes.
A solution without xargs is the following:
find . -mindepth 1 -maxdepth 1 -type f -exec sh -c "grep ABC '{}' > '{}.out'" \;
...and the same can be done with xargs, it turns out:
ls -1 | xargs -I {} sh -c "grep ABC '{}' > '{}.out'"
Edit: single quotes added after remark by lhunath.
I assume your example is just an example and that you may need > for other things. GNU Parallel http://www.gnu.org/software/parallel/ may be your rescue. It does not need additional quoting as long as your filenames do not contain \n:
ls | parallel "grep ABC {} > {}.out"
If you have filenames with \n in it:
find . -print0 | parallel -0 "grep ABC {} > {}.out"
As an added bonus you get the jobs run in parallel.
Watch the intro videos to learn more: http://pi.dk/1
The 10 seconds installation will try to do a full installation; if that fails, a personal installation; if that fails, a minimal installation:
$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh
If you need to move it to a server, that does not have GNU Parallel installed, try parallel --embed.
Actually, most of the answers here do not work with all filenames (if they contain double and single quotes), including the answer by lhunath and Stephan202.
This solution works with filenames with single and double quotes:
find . -mindepth 1 -print0 | xargs -0 -I{} sh -c 'grep ABC "$1" > "$1.out"' -- {}
Here's a test with filename with both single and double quotes:
echo ABC > "I'm here.txt"
# lhunath solution (hangs waiting for input)
$ find . -exec sh -c 'grep "$1" > "$1.out"' -- {} \;
# Stephan202 solutions
$ find . -mindepth 1 -maxdepth 1 -type f -exec sh -c "grep ABC '{}' > '{}.out'" \;
grep: ./Im: No such file or directory
grep: here.txt > ./Im here.txt.out: No such file or directory
$ ls -1 | xargs -I {} sh -c "grep ABC '{}' > '{}.out'"
xargs: unterminated quote
# this solution
$ find . -mindepth 1 -print0 | xargs -0 -I{} sh -c 'grep ABC "$1" > "$1.out"' -- {}
$ ls -1
"I'm here.txt"
"I'm here.txt.out"