I want to search files that does not contain a specific string.
I used -lv but this was a huge mistake because it was returning all the files that contain any line not containing my string.
I knew what I need exactly is grep -L, however, Solaris grep does not implement this feature.
What is the alternative, if any?
You can exploit grep -c and do the following (thanks #Scrutinizer for the /dev/null hint):
grep -c foo /dev/null * 2>/dev/null | awk -F: 'NR>1&&!$2{print $1}'
This will unfortunately also print directories (if * expands to any) which might not be desired in which case a simple loop, albeit slower, might be your best bet:
for file in *; do
[ -f "${file}" ] || continue
grep -q foo "${file}" 2>/dev/null || echo "${file}"
done
However, if you have GNU awk 4 on your system you can do:
awk 'BEGINFILE{f=0} /foo/{f=1} ENDFILE{if(!f)print FILENAME}' *
after using grep -c u can use grep again to find your desired filenames:
grep -c 'pattern' * | grep ':0$'
and to see just filnames :
grep -c 'pattern' * | grep ':0$' | cut -d":" -f1
You can use awk like this:
awk '!/not this/' file
To do multiple not:
awk '!/jan|feb|mars/' file
Related
This is what my loop contains:
cat /$f/stat | awk '{print $1,$3,$4,$7,$17}' /$f/stat
cd $f
sudo ls fd | wc -l
cd ..
At first, it shows the output of:
cat /$f/stat | awk '{print $1,$3,$4,$7,$17}' /$f/stat
And it prints the output of this on a new line:
cd $f
sudo ls fd | wc -l
cd ..
How do I combine these so that it shows them on one line?
At the outset, use shellcheck to validate your script.
Looks like you want awk's output and wc -l's output to be on the same line. Use command substitution for this:
printf '%s %s\n' "$(awk '{print $1,$3,$4,$7,$17}' "$f/stat")" "$(sudo ls "$f/fd" | wc -l)"
no need for cat | awk which is a case of UUOC - awk is reading input from the file passed as an argument; also, it looks like you need "$f/stat" and not "/$f/stat"
enclose variables in double quotes to prevent word splitting and globbing
use full path $f/fd instead of having to do a cd $f and back
Since parsing ls output is considered a bad practice, you could do this instead, on Linux:
printf '%s %s\n' "$(awk '{print $1,$3,$4,$7,$17}' "$f/stat")" "$(sudo find "$f/fd" -maxdepth 1 -print0 | tr -cd '\0' | wc -c)"
find ... -print0 prints NUL terminated list of files
tr -cd '\0' - deletes all characters other than NUL
wc -c - counts the number of NULs, which is the number of file names in find output
I am extracting data from files and I'd like to apply these working (ugly) command lines to all the txt files from a given folder. Thus I would also need to append a string to the output file name to avoid overwriting during the loop... any suggestion is warmly welcome.
for file in ./TEST/;
do
awk '/R.Time/,/LC/' 070_WT3a.txt|awk '/R.Time/,/PDA/'|grep -v -E "PDA|LC"|grep -w -v "80,00000"|grep -w -v "80,00833"|grep -w -v "80,01667"|grep -w -v "80,01067"|grep -w -v "80,02133"|sed -n '1,9601p' > ./Output/Fluo.txt;
awk '/R.Time/,/LC/' 070_WT3a.txt|awk '/R.Time/,/PDA/'|grep -v -E "PDA|LC"|grep -w -v "80,00000"|grep -w -v "80,00833"|grep -w -v "80,01667"|grep -w -v "80,01067"|grep -w -v "80,02133"|sed -n '9603,19203p' > ./Output/RID.txt;
done
Inside the loop you can use the variable ${file}. A first improvement (with additional lines that you can add after a pipe) :
for file in ./TEST/;
do
filebasename=${file##*/}
awk '/R.Time/,/LC/' ${file}.txt |
awk '/R.Time/,/PDA/' |
grep -v -E "PDA|LC" |
grep -w -v "80,00000"|
grep -w -v "80,00833"|
grep -w -v "80,01667"|
grep -w -v "80,01067"|
grep -w -v "80,02133"|
sed -n '1,9601p' > ./Output/Fluo_${filebasename};
awk '/R.Time/,/LC/' 070_WT3a.txt |
awk '/R.Time/,/PDA/'|
grep -v -E "PDA|LC"|
grep -w -v "80,00000"|
grep -w -v "80,00833"|
grep -w -v "80,01667"|
grep -w -v "80,01067"|
grep -w -v "80,02133"|
sed -n '9603,19203p' > ./Output/RID_${filebasename};
done
The next thing you can do is improving the parsing.
Without example input/output it is hard to see/test a solution, I can not tell for sure that all files needs to be split on lines 9601/9603/19203, what seems to be working for 070_WT3a.txt.
I would like to start with skipping the 80* lines, but these lines might have the boundaries R.Time/LC inside, so that won't help.
You might want to test on 070_WT3a.txt with
awk '/R.Time/,/LC/' 070_WT3a.txt |awk '/R.Time/,/PDA/'|
grep -v -E "PDA|LC"|grep -Ewv "80,0(0000|0833|1667|1067|2133)"
You can try to combine the 2 awk's into one (or even get the grep's inside the awk, but that is becoming offtopic and difficult to test without clear requirements and examples.
EDIT:
After testing with an example input I found this simplified:
for file in TEST/*.txt; do
filebasename=${file##*/}
awk '/LC Chromatogram(Detector A-Ch1)/,/^80,/' "${file}" |
grep -E "^[0-7]" > Output/Fluo_${filebasename}
awk '/LC Chromatogram(Detector B-Ch1)/,/^80,/' "${file}" |
grep -E "^[0-7]" > Output/RID_${filebasename}
done
Inside the loop I use ${file}, that will have different filenames each loop.
The filenaam is also used for the name of the outputfiles. The filename will start with TEST/, that can be stripped with ${file##*/} (there are a lot different ways like using cut -d"/" and sed 's/.., this one is fast).
I want to process an old database where password are plain text (comma separated ; passwd is the 5th field in the csv file where the database has been exported) to crypt them for further use by dokuwiki. Here is my bash command (grep and sed are there to extract the crypted passwd from curl output) :
cat users.csv | awk 'FS="," { print $4 }' | xargs -l bash -c 'curl -s --data-binary "pass1=$0&pass2=$0" "https://sprhost.com/tools/SMD5.php" -o - ' | xargs | grep -o '<tt.*tt>' | sed -e 's/tt//g' | sed -e 's/<[^>]*>//g'
I get the following comment from xargs
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
And only the first line of the file is processed, and nothing appends then.
Using the -0 option, and playing around with quotes, doesn't solve anything. Where am I wrong in the command line ? May be a more advanced language will be more adequate to do this.
Thank for help, LM
In general, if you have such a long pipe of commands, it is better to split them if things go wrong. Going through your pipe:
cat users.csv |
Nothing unexpected there.
awk 'FS="," { print $4 }' |
You probably wanted to do awk 'BEGIN {FS=","} { print $4 }'. Try the first two commands in the pipe and see if they produce the correct answer.
xargs -l bash -c 'curl -s --data-binary "pass1=$0&pass2=$0" "https://sprhost.com/tools/SMD5.php" -o - ' |
Nothing wrong there, although there might be better ways to do an MD5 hash.
xargs |
What is this xargs doing in the pipe? It should be removed.
grep -o '<tt.*tt>' |
Note that this will produce two lines:
<tt>$1$17ab075e$0VQMuM3cr5CtElvMxrPcE0</tt>
<tt><your_docuwiki_root>/conf/users.auth.php</tt>
which is probably not what you expected.
sed -e 's/tt//g' |
sed -e 's/<[^>]*>//g'
which will remove the html-tags, though
sed 's/<tt>//;s/<.tt>//'
will do the same.
So I'd say a wrong awk and an xargs too many.
Output of awk '{print $4}' is
b05808aa-c6ad-4d30-a334-198ff5726f7c
59996d37-9008-4b3b-ab22-340955cb6019
2b41f358-ff6d-418c-a0d3-ac7151c03b78
7ac4995c-ff2c-4717-a2ac-e6870a5670f0
I need to grep file st.log by these records. Something like awk '{print $4}' |xargs -i grep -w "pattern from awk" st.log I dont know how to pass pattern correctly?
What about
awk '{print $4}' | grep -F -f - st.log
Credits to Eric Renouf, who noticed that -f - can be used for standard input instead -f <(cat), Note: -f /dev/stdin also works and avoids launching a new process.
or closer to the question to have the output ordered
awk '{print $4}' | xargs -i grep -F {} st.log
maybe -w was not the option OP needed but -F
grep --help
-F, --fixed-strings PATTERN is a set of newline-separated strings
-w, --word-regexp force PATTERN to match only whole words
-w will match only line that contain exactly pattern
examples
grep -w . <<<a # matches
grep -w . <<<ab # doesn't match
grep -F . <<<a # doesn't match
grep -F . <<<a.b # matches
May be something along these lines be helpful
How to process each line received as a result of grep command
awk '{print $4} | while read -r line; do
grep $line st.log
done
This is my bash command
grep -rl "System.out.print" Project1/ |
xargs -I{} grep -H -n "System.out.print" {} |
cut -f-2 -d: |
sed "s/\(.*\):\(.*\)/filename is \1 and line number is \2/
What I'm trying to do here is,I'm trying to iterate through sub folders and check what files contains "System.out.print" (using grep)
using 2nd grep trying to get file names and line numbers
using sed command I display those to console.
from here I want to remove "System.out.print" with "XXXXX" how I can pipe sed command to this?
pls help me
thanxx
GNU sed has an option to change files in place:
find Project1/ -type f | xargs sed -i 's/System\.out\.print/XXXXX/g'
Btw, your script could be written as:
grep -rsn 'root' /etc/ |
awk -F: '{ print "filename is", $1, "and line number is", $2 }'
I'm just building on hop's answer, which I found to be more useful than find -exec. I had search_text dispersed all over my computer, in logs, config files and so on, but I didn't want to search (or especially change) anything in /dev, /sys, /proc, and so on. One note, read man xargs; it doesn't like file names with spaces.
grep -HriIl --exclude-dir=dev --exclude-dir=proc --exclude-dir=sys search_text / | xargs sed -i 's/search_text/replace_text/g'