I am attempting to filter a log file and am running into issues, what I have so far is the following, which does not work,
tail -f /var/log/squid/accesscustom.log | awk '/username/;/user-name/ {print $1; fflush("")}' | awk '!x[$0]++' > /var/log/squid/accesscustom-filtered.log
The goal is to take a file that contains
ipaddress1 username
ipaddress7
ipaddress2 user-name
ipaddress1 username
ipaddress5
ipaddress3 username
ipaddress4 user-name
and save to accesscustom-filtered.log
ipaddress1
ipaddress2
ipaddress3
ipaddress4
It works without the output to accesscustom-filtered.log but something in the > isn't working right and the file ends up empty.
Edit: Changed the original example to be correct
Use tee:
tail -f /var/log/squid/accesscustom.log | awk '/username/;/user-name/ {print $1}' | tee /var/log/squid/accesscustom-filtered.log
See also: Writing “tail -f” output to another file and Turn off buffering in pipe
Note: awk doesn't buffer like grep in the superuser example, so you shouldn't need to do anything special with your awk command. (more info)
Related
I am building a shell script.
The script gets git log such as:
"Updating 2f8b547d..eb94967a Fast-forward...."
but I want to get 2f8b547d..eb94967a snippet.
I am a new one for the shell. So, Thanks for you help.
Update:
For the more, I want use the snippet as a param. Because I will excute
git log 2f8b547d..eb94967a
You can pipe it to awk like so:
echo "Updating 2f8b547d..eb94967a Fast-forward...." | awk '{print $2}'
Your result will be 2f8b547d..eb94967a.
If it is a script, say, abc.sh that had such output, then you can run:
$> ./abc.sh | awk '{print $2}'
awk takes the output and splits the information by space. Updating is represented with $1. 2f8b547d..eb94967a is $2 and so on. In the above example, we ask awk to print out the 2nd item in the output.
As an alternative to awk (don't get me wrong, awk is super for this job as well), you can simply use cut with a space delimiter extract the second field, e.g.
cut -d' ' -f2 yourgit.log
You can also pipe output to cut or redirect the input file to it using < as well. It essentially does the same as the awk command, it just being a different choice of which utility to use.
Here another alternative:
echo "Updating 2f8b547d..eb94967a Fast-forward...." | read u hash rest
After this, the string you are looking for is stored on the variable hash:
echo $hash
I need to use awk to see what users are logged in the computer, create a file with their names and inside that file print the pid of the process they're running. I've used this, but it does not work:
who | awk '{for(i = 0; i < NR; i++)
system("ps -u " $1 "| tail +2 | awk '{print $1}' >" $1".log")
}'
Is there any way to do this?
Thanks a lot!
To achieve your goal of using awk to create those files, I would start with ps rather than with who. That way, ps does more of the work so that awk can do less. Here is an example that might work for you. (No guarantees, obviously!)
ps aux | awk 'NR>1 {system("echo " $2 " >> " $1 ".txt")}'
Discussion:
The command ps aux prints a table describing each active process, one line at a time. The first column of each line contains the name of the process's user, the second column its PID. The line also contains lots of other information, which you can play with as you improve your script. That's what you pipe into awk. (All this is true for Linux and the BSDs. In Cygwin, the format is different.)
Inside awk, the pattern NR>1 gets rid of the first line of the output, which contains the table headers. This line is useless for the files you want awk to generate.
For all other lines in the output of ps aux, awk adds the PID of the current process (ie, $2) to the file username.txt, using $1 for username. Because we append with >> rather than overwriting with >, all PIDs run by the user username end up being listed, one line at a time, in the file username.txt.
UPDATE (Alternative for when who is mandatory)
If using who is mandatory, as noted in a comment to the original post, I would use awk to strip needless lines and columns from the output of who and ps.
for user in $(who | awk 'NR>1 {print $1}')
do
ps -u "$user" | awk 'NR>1' > "$user".txt
done
For readers who wonder what the double-quotes around $user are about : Those serve to guard against globbing (if $user contains asterisks (*)) and word splitting (if $user contains whitespace).
I will leave my original answer stand for the benefit of any readers with more freedom to choose the tools for their job.
Is that what you had in mind?
I have a file which contains text as follows:
Directory /home/user/ "test_user"
bunch of code
another bunch of code
How can I get from this file only the /home/user/ part?
I've managed to use awk -F '"' 'NR==1{print $1}' file.txt to get rid of rest of the file and I'm gettig output like this:
Directory /home/user/
How can I change this command to get only /home/user/ part? I'd like to make it as simple as possible. Unfortunately, I can't modify this file to add/change the content.
this should work the fastest, noticeable if your file is large
awk '{print $2; exit}' file
it will print the second field of the first line and stop processing the rest of the file.
With awk it should be:
awk 'NR==1{print $2}' file.txt
Setting the field delimiter to " was wrong Since it splits the line into these fields:
$1 = 'Directory /home/user/'
$2 = 'test_user'
$3 = '' (empty)
The default record separator, which is [[:space:]]+, splits like this:
$1 = 'Directory'
$2 = '/home/user/'
$3 = '"test_user"'
As an alternate, you can use head and cut:
$ head -n 1 file | cut -d' ' -f2
Not sure why you are using the -F" as that changes the delimiter. If you remove that, then $2 will get you what you want.
awk 'NR==1{print $2}' file.txt
You can also use awk to execute the print when the line contains /home/user instead of counting records:
awk '/\home\/user\//{print $2}' file.txt
In this case, if the line were buried in the file, or if you had multiple instances, you would get the name for every occurrence wherever it was.
Adding some grep
grep Directory file.txt|awk '{print $2}'
I've got this problem, where I want to save an output of a command as a filename and stream output from a different command (within the same script) to that file. I wasn't able to find a solution online, so here goes. Below is the code I have:
zgrep --no-filename 'some-patter\|other-pattern' /var/something/something/$1/* | awk -F '\t' '{printf $8; printf "scriptLINEbreakerPARSE"; print $27}' | while read -r line ; do
awk -F 'scriptLINEbreakerPARSE' '{print $1}' -> save output of this as a filename
awk -F 'scriptLINEbreakerPARSE' '{print $2}' >> the_filename_from_above
done
So basically I want to use the first awk in the loop to save the output as a filename and then the second awk output will save to the file with that filename.
Any help would be appreciated guys.
You're doing too much work. Just output to the desired file in the first awk command:
zgrep --no-filename 'some-patter\|other-pattern' /var/something/something/$1/* |
awk -F '\t' '{printf $27 > $8}'
See https://www.gnu.org/software/gawk/manual/html_node/Redirection.html
I wanted to grep for strings "Manager" and "DBA" in all file at current directory and provide the output as
filename:Target string:count
I don't know if you were after a one-liner or not but I think this should provide the output you would like.
Starting with two files:
test_file.txt:
HEre is some random text
Manager
Some stuff
Manager
Testing
This Manager did DBA
and this manager did DBA too
but these guys did not
and test_file2.txt:
This is another file
which checks the Manager
DBA thing
I ran this:
grep -o "Manager\|DBA" *.txt | sort | uniq -c | awk -F "[: ]+" '{printf %s:%s:%s\n", $3, $4, $2}'
To get this output (note it is case sensitive):
test_file2.txt:DBA:1
test_file2.txt:Manager:1
test_file.txt:DBA:2
test_file.txt:Manager:3
Hope that's what you were after.