I am trying to save the output of a grep filter to a file.
I want to run tcpdump for a long time, and filter a certain IP to a file.
tcpdump -i eth0 -n -s 0 port 5060 -vvv | grep "A.B.C."
This works fine. It shows me IP's from my network.
But when I add >> file.dump at the end, the file is always empty.
My script:
tcpdump -i eth0 -n -s 0 port 5060 -vvv | grep "A.B.C." >> file.dump
And yes, it must be grep. I don't want to use tcpdump filters because it gives me millions of lines and with grep I get only one line per IP.
How can I redirect (append) the full output of the grep command to a file?
The output of tcpdump is probably going through stderr, not stdout. This means that grep won't catch it unless you convert it into stdout.
To do this you can use |&:
tcpdump -i eth0 -n -s 0 port 5060 -vvv |& grep "A.B.C."
Then, it may happen that the output is a continuous stream, so that you somehow have to tell grep to use line buffering. For this you have the option --line-buffered option.
All together, say:
tcpdump ... |& grep --line-buffered "A.B.C" >> file.dump
Related
Summary: I'm trying to create a Bash script that selects an IP address and it's associated open ports, then run each IP through Nmap and display/store the results.
The format of the ports file (portsList):
192.168.1.1 53 udp
192.168.1.1 80 tcp
192.168.1.1 1900 tcp
192.168.1.110 135 tcp
192.168.1.115 1080 tcp
The format of the IP file (ipList):
192.168.1.1
192.168.1.110
192.168.1.115
My Bash script:
#!/bin/bash
portsvar="$(cat formattedPorts)"
hostvar="$(cat oneHost)"
while read -r line;
do
echo "$line" > oneHost
grep -wf oneHost portsList | awk '{print $2}' | tr '\n' ',' > formattedPorts
nmap -n -sV -p"$portsvar" "$hostvar" >> scanResults
cat scanResults
done < ipList
I spent several hours, trying multiple variations of this code and tried to find a solution online. But to no avail. The most this script can do is scan and output for the first selected IP and ports.
Additional considerations would be; if a port is UDP, to append 'U:' on the ports option. Also, if there is a way to display the output of Nmap scanning, while it is being redirected '>>' to a file.
Bash is what I'm most familiar with, but I'm open to learning a solution in a different language.
Edit
I tried jhnc's solution below by replacing:
echo "$line" > oneHost
grep -wf oneHost portsList | awk '{print $2}' | tr '\n' ',' > formattedPorts
nmap -n -sV -p"$portsvar" "$hostvar" >> scanResults
cat scanResults
with jhnc's code in the comment below. the resulting output was:
nmap -n -sV -p T:135 192.168.1.110
nmap -n -sV -p T:5985 192.168.1.112
nmap -n -sV -p T:54112,T:60000,T:8009,T:8888 192.168.1.131
nmap -n -sV -p T:5040 192.168.1.132
nmap -n -sV -p T:1041,T:1900,T:20005,T:33344,T:49152,T:49153,T:80 192.168.1.1
This output was echo'ed onto the terminal 5 times. From this I'll try to find a solution to have the input of this group of IP's ran once.
(maybe it is the "tcpflow" problem)
I write a script to monitoring http traffic, and I install tcpflow, then grep
it works (and you should make a http request, for example curl www.163.com)
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: '
it outputs like this (continuously)
Host: config.getsync.com
Host: i.stack.imgur.com
Host: www.gravatar.com
Host: www.gravatar.com
but I can't continue to use pipe
does not work (nothing output)
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | cut -b 7-
does not work (nothing output)
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | grep H
When I replace sudo tcpflow with cat foo.txt, it works:
cat foo.txt | grep '^Host: ' | grep H
so what's wrong with pipe or grep or tcpflow ?
update:
This is my final script: https://github.com/zhengkai/config/blob/master/script/monitor_outgoing_http.sh
To grep a continuous stream use --line-buffered option:
sudo tcpflow -p -c -i eth0 port 80 2> /dev/null | grep --line-buffered '^Host'
--line-buffered
Use line buffering on output. This can cause a performance penalty.
Some reflections about buffered outputting(stdbuf tool is also mentioned):
Pipes, how do data flow in a pipeline?
I think the problem is because of stdio buffering, you need to use GNU stdbuf before calling grep,
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | stdbuf -o0 grep '^Host: '
With the -o0, it basically means the output (stdout) stream from tcpflow will be unbuffered. The default behavior will be to automatically buffer up data into 40961 byte chunks before sending to next command in pipeline, which is what overriden using stdbuf
1. Refer this nice detail into the subject.
I use the following command to send pinging IP's to a script:
sudo tcpdump -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo \
| cut -d " " -f 10 | xargs -L2 ./pong.sh
Unfortunately this gives me:
tcpdump: Unable to write output: Broken pipe
To dissect my commands:
Output the ping's from the traffic:
sudo tcpdump -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo
Output:
11:55:58.812177 IP xxxxxxx > 127.0.0.1: ICMP echo request, id 50776, seq 761, length 64
This will get the IP's from the tcpdump output:
cut -d " " -f 10 # output: 127.0.0.1
Get the output to the script:
xargs -L2 ./pong.sh
This will mimic the following example command:
./pong.sh 127.0.0.1
The strange thing is that the commands work seperate (on their own)...
I tried debugging it but I have no experience with debugging pipes. I checked the commands but they seem fine.
It would seem that's cut stdio buffering is interfering here, i.e. replace | xargs ... by | cat in your cmdline to find out.
Fwiw below cmdline wfm (pipe straight to xargs then use the shell itself to get the nth arg), note btw the extra tcpdump args : -c10 (just to limit to 10pkts, then show the 10/2 lines) and -Q in (only inbound pkts):
$ sudo tcpdump -c 10 -Q in -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo 2>/dev/null | \
xargs -L2 sh -c 'echo -n "$9: "; ping -nqc1 $9 | grep rtt'
192.168.100.132: rtt min/avg/max/mdev = 3.743/3.743/3.743/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 5.863/5.863/5.863/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 6.167/6.167/6.167/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 4.256/4.256/4.256/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 1.545/1.545/1.545/0.000 ms
$ _
For those coming across this (like me), tcpdump buffering is the issue.
From the man page:
-l Make stdout line buffered. Useful if you want to see the data
while capturing it. For example:
# tcpdump -l | tee dat
or
# tcpdump -l > dat & tail -f dat
I have got an ip list in a file called ips:
ip1
ip2
I want to tcpkill from this list. I cannot do it from tcpkill command options. I need a script which will write :
tcpkill -i eth0 -9 host ip1 or host ip2
the list can grow in time.
the only thing I know is that I will need a loop to read the list:
for IP in $(cat list) ; do
echo "tcpkill -i eth0 -9 host $IP " (for
the first element of the list)
echo "or host $IP" (for the rest of
the list).
So my questions are:
1. what would be the right syntax for this purpose?
2. Is there a completely different way of doing this?
Any idea folks ?!
thank you very much.
This single line command would append the IP list to tcpkill command to form the complete command:
tcpkill -i eth0 -9 host $(cat list | xargs | sed 's/ / or host /g')
You can put this command after echo to see how the command gets formed.
echo tcpkill -i eth0 -9 host $(cat list | xargs | sed 's/ / or host /g')
Problem description:
I want to print only the source and destination address from a tcpdump[1].
Have one working solution, but believe it could be improved a lot. An example that captures 5 packets, just as an example of what I'm looking for:
tcpdump -i eth1 -n -c 5 ip | \
cut -d" " -f3,5 | \
sed -e 's/^\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\)\..* \([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\).*$/\1 > \2/'
Question:
Can this be done in any easier way? Performance is also an issue here.
[1] A part of a test if the snort home_net is correctly defined, or if we see traffic not defined in the home_net.
Solution:
Ok, thanks to everyone who have replied to this one. There have been two concerns related to the answers, one is the compatibility across different linux-versions and the second one is speed.
Here is the results on the speed test I did. First the grep-version:
time tcpdump -l -r test.dmp -n ip 2>/dev/null | grep -P -o '([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*? > ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)' | grep -P -o '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | xargs -n 2 echo >/dev/null
real 0m5.625s
user 0m0.513s
sys 0m4.305s
Then the sed-version:
time tcpdump -n -r test.dmp ip | sed -une 's/^.* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..*$/\1 > \3/p' >/dev/null
reading from file test.dmp, link-type EN10MB (Ethernet)
real 0m0.491s
user 0m0.496s
sys 0m0.020s
And the fastest one, the awk-version:
time tcpdump -l -r test.dmp -n ip | awk '{ print gensub(/(.*)\..*/,"\\1","g",$3), $4, gensub(/(.*)\..*/,"\\1","g",$5) }' >/dev/null
reading from file test.dmp, link-type EN10MB (Ethernet)
real 0m0.093s
user 0m0.111s
sys 0m0.013s
Unfortunately I have not been able to test how compatible they are, but the awk needs gnu awk to work due to the gensub function. Anyway, all three solutions works on the two platforms I have tested them on. :)
Here's one way using GNU awk:
tcpdump -i eth1 -n -c 5 ip | awk '{ print gensub(/(.*)\..*/,"\\1","g",$3), $4, gensub(/(.*)\..*/,"\\1","g",$5) }'
Try this:
tcpdump -i eth1 -n -c 5 ip 2>/dev/null | sed -r 's/.* ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).* > ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*/\1 > \2/'
If running from a .sh script, remember to escape \1 & \2 as required.
Warning You have to use unbuffered ou line-buffered output to monitor the output of another command like tcpdump.
But you command seem correct.
To simplify, you could:
tcpdump -i eth1 -n -c 5 ip |
sed -une 's/^.* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..*$/\1 > \3/p'
Notice the u switch usefull without -c 5 at tcpdump
tcpdump -ni eth1 ip |
sed -une 's/^.* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..*$/\1 > \3/p'
& here is a grep only solution:
tcpdump -l -i eth1 -n -c 5 ip 2>/dev/null | grep -P -o '([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*? > ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)' | grep -P -o '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | xargs -n 2 echo
Note -l, in case you don't want to limit the number of packets using -c.