Output of awk '{print $4}' is
b05808aa-c6ad-4d30-a334-198ff5726f7c
59996d37-9008-4b3b-ab22-340955cb6019
2b41f358-ff6d-418c-a0d3-ac7151c03b78
7ac4995c-ff2c-4717-a2ac-e6870a5670f0
I need to grep file st.log by these records. Something like awk '{print $4}' |xargs -i grep -w "pattern from awk" st.log I dont know how to pass pattern correctly?
What about
awk '{print $4}' | grep -F -f - st.log
Credits to Eric Renouf, who noticed that -f - can be used for standard input instead -f <(cat), Note: -f /dev/stdin also works and avoids launching a new process.
or closer to the question to have the output ordered
awk '{print $4}' | xargs -i grep -F {} st.log
maybe -w was not the option OP needed but -F
grep --help
-F, --fixed-strings PATTERN is a set of newline-separated strings
-w, --word-regexp force PATTERN to match only whole words
-w will match only line that contain exactly pattern
examples
grep -w . <<<a # matches
grep -w . <<<ab # doesn't match
grep -F . <<<a # doesn't match
grep -F . <<<a.b # matches
May be something along these lines be helpful
How to process each line received as a result of grep command
awk '{print $4} | while read -r line; do
grep $line st.log
done
Related
I've been trying to print all the instances of a matching pattern from file.
Input file:
{"id":"prod123","a":1.3,"c":"xyz","q":2},
{"id":"prod456","a":1.3,"c":"xyz","q":1}]}
{"id":"prod789","a":1.3,"currency":"xyz","q":2},
{"id":"prod101112","a":1.3,"c":"xyz","q":1}]}
I'd want to print everything between "id":" and ",.
Expected output:
prod123
prod456
prod789
prod101112
I'm using the command
grep -Eo 'id\"\:\"[^"]+"\"\,*' | grep -Eo '^[^"]+'
Am I missing anything here?
What went wrong is the place of the comma in the first grep:
grep -Eo 'id.\:.[^"]+"\,"' inputfile
You need to do something extra for getting the desired substring.
grep -Eo 'id.\:.[^"]+"\,"' inputfile | cut -d: -f2 | grep -Eo '[^",]+'
I used cut, that would be easy for your example input.
cut -d'"' -f4 < inputfile
You have alternatives, like using jq, or
sed -r 's/\{"id":"([^"]*).*/\1/' inputfile
or using awk (solution now like cut but can be changed easy)
awk -F'"' '{print $4}' inputfile
How to feed xargs to a piped grep for a piped cat command.
Command 1:
(Generates a grep pattern with unique PIDs for a particular date time, read from runtime.log)
cat runtime.log | grep -e '2018/09/13 14:50' | awk -F'[ ]' '{print $4}' | awk -F'PID=' '{print $2}' | sort -u | xargs -I % echo '2018/09/13 14:50.*PID='%
The output of above command is (It's custom grep pattern):
2018/09/13 14:50.*PID=13109
2018/09/13 14:50.*PID=14575
2018/09/13 14:50.*PID=15741
Command 2:
(Reads runtime.log and fetch the appropriate lines based on the grep pattern (Ideally the grep pattern should comes from command 1))
cat runtime.log | grep '2018/09/13 14:50.*PID=13109'
The question is How to combine both Command 1 & Command 2
Below combined version of command doesn't gives the expected output (The produced output had lines having the date other than '2018/09/13 14:50')
cat runtime.log | grep -e '2018/09/13 14:50' | awk -F'[ ]' '{print $4}' | awk -F'PID=' '{print $2}' | sort -u | xargs -I % echo '2018/09/13 14:50.*PID='% | cat runtime.log xargs grep
grep has an option -f. From man grep:
-f FILE, --file=FILE
Obtain patterns from FILE, one per line. The empty file contains zero patterns, and therefore matches nothing. (-f is specified by POSIX .)
So you could use
cat runtime.log | grep -e '2018/09/13 14:50' | awk -F'[ ]' '{print $4}' | awk -F'PID=' '{print $2}' | sort -u | xargs -I % echo '2018/09/13 14:50.*PID='% > a_temp_file
cat runtime.log | grep -f a_temp_file
The shell has a syntax that avoids having to create the temporary file. <(). From man bash:
Process Substitution
Process substitution is supported on systems that support named pipes
(FIFOs) or the /dev/fd method of naming open files. It takes the form
of <(list) or >(list). The process list is run with its input or
output connected to a FIFO or some file in /dev/fd. The name of this
file is passed as an argument to the current command as the result of
the expansion. If the >(list) form is used, writing to the file will
provide input for list. If the <(list) form is used, the file passed
as an argument should be read to obtain the output of list.
So you can combine it to:
cat runtime.log | grep -f <(cat runtime.log | grep -e '2018/09/13 14:50' | awk -F'[ ]' '{print $4}' | awk -F'PID=' '{print $2}' | sort -u | xargs -I % echo '2018/09/13 14:50.*PID='%)
I am extracting data from files and I'd like to apply these working (ugly) command lines to all the txt files from a given folder. Thus I would also need to append a string to the output file name to avoid overwriting during the loop... any suggestion is warmly welcome.
for file in ./TEST/;
do
awk '/R.Time/,/LC/' 070_WT3a.txt|awk '/R.Time/,/PDA/'|grep -v -E "PDA|LC"|grep -w -v "80,00000"|grep -w -v "80,00833"|grep -w -v "80,01667"|grep -w -v "80,01067"|grep -w -v "80,02133"|sed -n '1,9601p' > ./Output/Fluo.txt;
awk '/R.Time/,/LC/' 070_WT3a.txt|awk '/R.Time/,/PDA/'|grep -v -E "PDA|LC"|grep -w -v "80,00000"|grep -w -v "80,00833"|grep -w -v "80,01667"|grep -w -v "80,01067"|grep -w -v "80,02133"|sed -n '9603,19203p' > ./Output/RID.txt;
done
Inside the loop you can use the variable ${file}. A first improvement (with additional lines that you can add after a pipe) :
for file in ./TEST/;
do
filebasename=${file##*/}
awk '/R.Time/,/LC/' ${file}.txt |
awk '/R.Time/,/PDA/' |
grep -v -E "PDA|LC" |
grep -w -v "80,00000"|
grep -w -v "80,00833"|
grep -w -v "80,01667"|
grep -w -v "80,01067"|
grep -w -v "80,02133"|
sed -n '1,9601p' > ./Output/Fluo_${filebasename};
awk '/R.Time/,/LC/' 070_WT3a.txt |
awk '/R.Time/,/PDA/'|
grep -v -E "PDA|LC"|
grep -w -v "80,00000"|
grep -w -v "80,00833"|
grep -w -v "80,01667"|
grep -w -v "80,01067"|
grep -w -v "80,02133"|
sed -n '9603,19203p' > ./Output/RID_${filebasename};
done
The next thing you can do is improving the parsing.
Without example input/output it is hard to see/test a solution, I can not tell for sure that all files needs to be split on lines 9601/9603/19203, what seems to be working for 070_WT3a.txt.
I would like to start with skipping the 80* lines, but these lines might have the boundaries R.Time/LC inside, so that won't help.
You might want to test on 070_WT3a.txt with
awk '/R.Time/,/LC/' 070_WT3a.txt |awk '/R.Time/,/PDA/'|
grep -v -E "PDA|LC"|grep -Ewv "80,0(0000|0833|1667|1067|2133)"
You can try to combine the 2 awk's into one (or even get the grep's inside the awk, but that is becoming offtopic and difficult to test without clear requirements and examples.
EDIT:
After testing with an example input I found this simplified:
for file in TEST/*.txt; do
filebasename=${file##*/}
awk '/LC Chromatogram(Detector A-Ch1)/,/^80,/' "${file}" |
grep -E "^[0-7]" > Output/Fluo_${filebasename}
awk '/LC Chromatogram(Detector B-Ch1)/,/^80,/' "${file}" |
grep -E "^[0-7]" > Output/RID_${filebasename}
done
Inside the loop I use ${file}, that will have different filenames each loop.
The filenaam is also used for the name of the outputfiles. The filename will start with TEST/, that can be stripped with ${file##*/} (there are a lot different ways like using cut -d"/" and sed 's/.., this one is fast).
here is a piece of log
21:36 b05808aa-c6ad-4d30-a334-198ff5726f7c new
22:21 59996d37-9008-4b3b-ab22-340955cb6019 new
21:12 2b41f358-ff6d-418c-a0d3-ac7151c03b78 new
12:36 7ac4995c-ff2c-4717-a2ac-e6870a5670f0 new
i print it by awk '{print $2}' st.log
so i got
b05808aa-c6ad-4d30-a334-198ff5726f7c
59996d37-9008-4b3b-ab22-340955cb6019
2b41f358-ff6d-418c-a0d3-ac7151c03b78
7ac4995c-ff2c-4717-a2ac-e6870a5670f0
now i need to pass it to grep, in this manner
awk '{print $2}' |xargs -i grep -w "pattern from awk" st.log
I need exactly how to pass each founded record from awk to grep. I do not need other solutions, because my task is more complicated, than this piece. Thank you.
With bash and grep:
grep -f <(awk '{print $2}' piece_of_log) st.log
No need for awk:
grep -Ff <(cut -d' ' -f2 log)
It seems you're looking for the replace string option:
-I replace-str
Replace occurrences of replace-str in the initial-arguments with
names read from standard input. Also, unquoted blanks do not
terminate input items; instead the separator is the newline
character. Implies -x and -L 1.
Like this:
awk '{print $2}' | xargs -I{} grep -w {} st.log
i want to svn blame lines of code which include "todo | fixme"
i have the general flow of the script but struggle to combine it into one
finding the lines with "todo"
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" .
blame the line of code
svn blame ${file} | cat -n |grep ${linenumber}
i could get $file and $linenumber from the first command with awk, but i dont know how to pipe the values i extract with awk into the second command.
i am missing the glue to combine these commands into one "script" (- :
You can build the command with awk and then pipe it to bash:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | grep \"%s\"\n", $1, $2}'
That prints one command per input line with the following format:
svn blame "${file}" | cat -n | grep "${linenumber}"
The varibales are replaces. When you execute the command as above they are only printed to the shell, that you can comfirm if everything is right. If yes add a last pipe to the in of the command that the ouput is redirected to bash. The complete command would look like this:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | grep \"%s\"\n", $1, $2}' | bash
A small notice: I think you want to print the line number extracterd in the first command, aren't you? But grep ${linenumber} just gives the line containing the string ${linenumber}. To print only the linenumber use that command: sed -n "2p" to print line number 2 for example. The complete command would then look like this:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | sed -n \"%sp\"\n", $1, $2}' | bash