(maybe it is the "tcpflow" problem)
I write a script to monitoring http traffic, and I install tcpflow, then grep
it works (and you should make a http request, for example curl www.163.com)
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: '
it outputs like this (continuously)
Host: config.getsync.com
Host: i.stack.imgur.com
Host: www.gravatar.com
Host: www.gravatar.com
but I can't continue to use pipe
does not work (nothing output)
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | cut -b 7-
does not work (nothing output)
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | grep H
When I replace sudo tcpflow with cat foo.txt, it works:
cat foo.txt | grep '^Host: ' | grep H
so what's wrong with pipe or grep or tcpflow ?
update:
This is my final script: https://github.com/zhengkai/config/blob/master/script/monitor_outgoing_http.sh
To grep a continuous stream use --line-buffered option:
sudo tcpflow -p -c -i eth0 port 80 2> /dev/null | grep --line-buffered '^Host'
--line-buffered
Use line buffering on output. This can cause a performance penalty.
Some reflections about buffered outputting(stdbuf tool is also mentioned):
Pipes, how do data flow in a pipeline?
I think the problem is because of stdio buffering, you need to use GNU stdbuf before calling grep,
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | stdbuf -o0 grep '^Host: '
With the -o0, it basically means the output (stdout) stream from tcpflow will be unbuffered. The default behavior will be to automatically buffer up data into 40961 byte chunks before sending to next command in pipeline, which is what overriden using stdbuf
1. Refer this nice detail into the subject.
Related
I use the following command to send pinging IP's to a script:
sudo tcpdump -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo \
| cut -d " " -f 10 | xargs -L2 ./pong.sh
Unfortunately this gives me:
tcpdump: Unable to write output: Broken pipe
To dissect my commands:
Output the ping's from the traffic:
sudo tcpdump -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo
Output:
11:55:58.812177 IP xxxxxxx > 127.0.0.1: ICMP echo request, id 50776, seq 761, length 64
This will get the IP's from the tcpdump output:
cut -d " " -f 10 # output: 127.0.0.1
Get the output to the script:
xargs -L2 ./pong.sh
This will mimic the following example command:
./pong.sh 127.0.0.1
The strange thing is that the commands work seperate (on their own)...
I tried debugging it but I have no experience with debugging pipes. I checked the commands but they seem fine.
It would seem that's cut stdio buffering is interfering here, i.e. replace | xargs ... by | cat in your cmdline to find out.
Fwiw below cmdline wfm (pipe straight to xargs then use the shell itself to get the nth arg), note btw the extra tcpdump args : -c10 (just to limit to 10pkts, then show the 10/2 lines) and -Q in (only inbound pkts):
$ sudo tcpdump -c 10 -Q in -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo 2>/dev/null | \
xargs -L2 sh -c 'echo -n "$9: "; ping -nqc1 $9 | grep rtt'
192.168.100.132: rtt min/avg/max/mdev = 3.743/3.743/3.743/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 5.863/5.863/5.863/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 6.167/6.167/6.167/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 4.256/4.256/4.256/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 1.545/1.545/1.545/0.000 ms
$ _
For those coming across this (like me), tcpdump buffering is the issue.
From the man page:
-l Make stdout line buffered. Useful if you want to see the data
while capturing it. For example:
# tcpdump -l | tee dat
or
# tcpdump -l > dat & tail -f dat
I'm working on enabling Session Multiplexing between two servers.
I want to close all existing sockets for this server(or IP) before creating new one and close newly created one after finishing my task. This is what I've done so far:
remote_ip=192.168.20.2 #User inut
remote_port=222
Can create socket by:
SSHSOCKET=~/.ssh/remote_$remote_ip
ssh -M -f -N -o ControlPath=$SSHSOCKET $remote_ip -p $remote_port
Can search control path by:
ps x | grep $remote_ip | grep ssh | cut -d '=' -f 2
/root/.ssh/remote_192.168.20.2 192.168.20.2 -p 222
Can close socket by:
ssh -S /root/.ssh/remote_192.168.20.2 192.168.20.2 -p 64555 -O exit
Trying to close the socket by:
ps x | grep $remote_ip | grep ssh | cut -d '=' -f 2 | xargs ssh -S | xargs -i {} "-O exit"
But I get:
Pseudo-terminal will not be allocated because stdin is not a terminal.
Tried using -t and -tt:
ps x | grep $remote_ip | grep ssh | cut -d '=' -f 2 | xargs ssh -Stt | xargs -i {} "-O exit"
ssh: Could not resolve hostname /root/.ssh/remote_192.168.20.2: Name or service not known
xargs: ssh: exited with status 255; aborting
Can anyone please help me with this?
If you want to kill every connection from your machine to a given remote IP address and port, you can do so as follows (using fuser, a tool from the psmisc package included with all major Linux distros):
fuser -k -n tcp ",${remote_ip},${remote_port}"
I am trying to save the output of a grep filter to a file.
I want to run tcpdump for a long time, and filter a certain IP to a file.
tcpdump -i eth0 -n -s 0 port 5060 -vvv | grep "A.B.C."
This works fine. It shows me IP's from my network.
But when I add >> file.dump at the end, the file is always empty.
My script:
tcpdump -i eth0 -n -s 0 port 5060 -vvv | grep "A.B.C." >> file.dump
And yes, it must be grep. I don't want to use tcpdump filters because it gives me millions of lines and with grep I get only one line per IP.
How can I redirect (append) the full output of the grep command to a file?
The output of tcpdump is probably going through stderr, not stdout. This means that grep won't catch it unless you convert it into stdout.
To do this you can use |&:
tcpdump -i eth0 -n -s 0 port 5060 -vvv |& grep "A.B.C."
Then, it may happen that the output is a continuous stream, so that you somehow have to tell grep to use line buffering. For this you have the option --line-buffered option.
All together, say:
tcpdump ... |& grep --line-buffered "A.B.C" >> file.dump
Problem description:
I want to print only the source and destination address from a tcpdump[1].
Have one working solution, but believe it could be improved a lot. An example that captures 5 packets, just as an example of what I'm looking for:
tcpdump -i eth1 -n -c 5 ip | \
cut -d" " -f3,5 | \
sed -e 's/^\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\)\..* \([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\).*$/\1 > \2/'
Question:
Can this be done in any easier way? Performance is also an issue here.
[1] A part of a test if the snort home_net is correctly defined, or if we see traffic not defined in the home_net.
Solution:
Ok, thanks to everyone who have replied to this one. There have been two concerns related to the answers, one is the compatibility across different linux-versions and the second one is speed.
Here is the results on the speed test I did. First the grep-version:
time tcpdump -l -r test.dmp -n ip 2>/dev/null | grep -P -o '([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*? > ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)' | grep -P -o '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | xargs -n 2 echo >/dev/null
real 0m5.625s
user 0m0.513s
sys 0m4.305s
Then the sed-version:
time tcpdump -n -r test.dmp ip | sed -une 's/^.* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..*$/\1 > \3/p' >/dev/null
reading from file test.dmp, link-type EN10MB (Ethernet)
real 0m0.491s
user 0m0.496s
sys 0m0.020s
And the fastest one, the awk-version:
time tcpdump -l -r test.dmp -n ip | awk '{ print gensub(/(.*)\..*/,"\\1","g",$3), $4, gensub(/(.*)\..*/,"\\1","g",$5) }' >/dev/null
reading from file test.dmp, link-type EN10MB (Ethernet)
real 0m0.093s
user 0m0.111s
sys 0m0.013s
Unfortunately I have not been able to test how compatible they are, but the awk needs gnu awk to work due to the gensub function. Anyway, all three solutions works on the two platforms I have tested them on. :)
Here's one way using GNU awk:
tcpdump -i eth1 -n -c 5 ip | awk '{ print gensub(/(.*)\..*/,"\\1","g",$3), $4, gensub(/(.*)\..*/,"\\1","g",$5) }'
Try this:
tcpdump -i eth1 -n -c 5 ip 2>/dev/null | sed -r 's/.* ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).* > ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*/\1 > \2/'
If running from a .sh script, remember to escape \1 & \2 as required.
Warning You have to use unbuffered ou line-buffered output to monitor the output of another command like tcpdump.
But you command seem correct.
To simplify, you could:
tcpdump -i eth1 -n -c 5 ip |
sed -une 's/^.* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..*$/\1 > \3/p'
Notice the u switch usefull without -c 5 at tcpdump
tcpdump -ni eth1 ip |
sed -une 's/^.* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..* \(\([0-9]\{1,3\}\.\?\)\{4\}\)\..*$/\1 > \3/p'
& here is a grep only solution:
tcpdump -l -i eth1 -n -c 5 ip 2>/dev/null | grep -P -o '([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*? > ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)' | grep -P -o '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | xargs -n 2 echo
Note -l, in case you don't want to limit the number of packets using -c.
How to write a echo server bash script using tools like nc, echo, xargs, etc capable of simultaneously processing requests from multiple clients each with durable connection?
The best that I've came up so far is
nc -l -p 2000 -c 'xargs -n1 echo'
but it only allows a single connection.
If you use ncat instead of nc your command line works fine with multiple connections but (as you pointed out) without -p.
ncat -l 2000 -k -c 'xargs -n1 echo'
ncat is available at http://nmap.org/ncat/.
P.S. with the original the Hobbit's netcat (nc) the -c flag is not supported.
Update: -k (--keep-open) is now required to handle multiple connections.
Here are some examples. ncat simple services
TCP echo server
ncat -l 2000 --keep-open --exec "/bin/cat"
UDP echo server
ncat -l 2000 --keep-open --udp --exec "/bin/cat"
In case ncat is not an option, socat will also work:
socat TCP4-LISTEN:2000,fork EXEC:cat
The fork is necessary so multiple connections can be accepted. Adding reuseaddr to TCP4-LISTEN may be convenient.
netcat solution pre-installed in Ubunutu
The netcat pre-installed in Ubuntu 16.04 comes from netcat-openbsd, and has no -c option, but the manual gives a solution:
sudo mknod -m 777 fifo p
cat fifo | netcat -l -k localhost 8000 > fifo
Then client example:
echo abc | netcat localhost 8000
TODO: how to modify the input string value? The following does not return any reply:
cat fifo | tr 'a' 'b' | netcat -l -k localhost 8000 > fifo
The remote shell example however works:
cat fifo | /bin/sh -i 2>&1 | netcat -l -k localhost 8000 > fifo
I don't know how to deal with concurrent requests simply however.
what about...
#! /bin/sh
while :; do
/bin/nc.traditional -k -l -p 3342 -c 'xargs -n1 echo'
done