Hi everyone this is my first question on stackoverflow!
I'm using this software (it's a NIDS); one of its features is using socat to create a proxy that saves the traffic to a pcap.
That's the command it uses to do this: /usr/bin/socat -d OPENSSL-LISTEN:50010,cipher=HIGH,method=TLS1.2,reuseaddr,pf=ip4,fk,cert=/usr/local/owlh/src/owlhnode/conf/certs/ca.pem,verify=0 SYSTEM:"/usr/sbin/tcpdump -n -r - -s 0 -G 50 -W 100 -w /usr/local/owlh/pcaps/remote-test%d%m%Y%H%M%S.pcap not port 22"
That's what happens when using curl i try to make a request to google through the proxy:
╭─myasnik#tanuki ~/…/ossihr-poc/docker ‹master*›
╰─$ export https_proxy=https://0.0.0.0:50010/
╭─myasnik#tanuki ~/…/ossihr-poc/docker ‹master*›
╰─$ export http_proxy=https://0.0.0.0:50010/
╭─myasnik#tanuki ~/…/ossihr-poc/docker ‹master*›
╰─$ curl --proxy-insecure www.google.it
curl: (52) Empty reply from server
root#owlh-node:/# /usr/bin/socat -d OPENSSL-LISTEN:50010,cipher=HIGH,method=TLS1.2,reuseaddr,pf=ip4,fk,cert=/usr/local/owlh/src/owlhnode/conf/certs/ca.pem,verify=0 SYSTEM:"/usr/sbin/tcpdump -n -r - -s 0 -G 50 -W 100 -w /usr/local/owlh/pcaps/remote-test%d%m%Y%H%M%S.pcap not port 22"
tcpdump: unknown file format
2020/08/18 12:00:08 socat[1590] W system("/usr/sbin/tcpdump -n -r - -s 0 -G 50 -W 100 -w /usr/local/owlh/pcaps/remote-test%d%m%Y%H%M%S.pcap not port 22") returned with status 256
2020/08/18 12:00:08 socat[1590] W system(): No such file or directory
2020/08/18 12:00:08 socat[1589] E waitpid(): child 1590 exited with status 1
Thanks a lot for your help in advantage!
Here is the answer to the question, i think i misunderstood the way it was supposed to be done: https://github.com/OwlH-net/OwlH-Node/issues/47
(maybe it is the "tcpflow" problem)
I write a script to monitoring http traffic, and I install tcpflow, then grep
it works (and you should make a http request, for example curl www.163.com)
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: '
it outputs like this (continuously)
Host: config.getsync.com
Host: i.stack.imgur.com
Host: www.gravatar.com
Host: www.gravatar.com
but I can't continue to use pipe
does not work (nothing output)
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | cut -b 7-
does not work (nothing output)
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | grep H
When I replace sudo tcpflow with cat foo.txt, it works:
cat foo.txt | grep '^Host: ' | grep H
so what's wrong with pipe or grep or tcpflow ?
update:
This is my final script: https://github.com/zhengkai/config/blob/master/script/monitor_outgoing_http.sh
To grep a continuous stream use --line-buffered option:
sudo tcpflow -p -c -i eth0 port 80 2> /dev/null | grep --line-buffered '^Host'
--line-buffered
Use line buffering on output. This can cause a performance penalty.
Some reflections about buffered outputting(stdbuf tool is also mentioned):
Pipes, how do data flow in a pipeline?
I think the problem is because of stdio buffering, you need to use GNU stdbuf before calling grep,
sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | stdbuf -o0 grep '^Host: '
With the -o0, it basically means the output (stdout) stream from tcpflow will be unbuffered. The default behavior will be to automatically buffer up data into 40961 byte chunks before sending to next command in pipeline, which is what overriden using stdbuf
1. Refer this nice detail into the subject.
I use the following command to send pinging IP's to a script:
sudo tcpdump -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo \
| cut -d " " -f 10 | xargs -L2 ./pong.sh
Unfortunately this gives me:
tcpdump: Unable to write output: Broken pipe
To dissect my commands:
Output the ping's from the traffic:
sudo tcpdump -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo
Output:
11:55:58.812177 IP xxxxxxx > 127.0.0.1: ICMP echo request, id 50776, seq 761, length 64
This will get the IP's from the tcpdump output:
cut -d " " -f 10 # output: 127.0.0.1
Get the output to the script:
xargs -L2 ./pong.sh
This will mimic the following example command:
./pong.sh 127.0.0.1
The strange thing is that the commands work seperate (on their own)...
I tried debugging it but I have no experience with debugging pipes. I checked the commands but they seem fine.
It would seem that's cut stdio buffering is interfering here, i.e. replace | xargs ... by | cat in your cmdline to find out.
Fwiw below cmdline wfm (pipe straight to xargs then use the shell itself to get the nth arg), note btw the extra tcpdump args : -c10 (just to limit to 10pkts, then show the 10/2 lines) and -Q in (only inbound pkts):
$ sudo tcpdump -c 10 -Q in -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo 2>/dev/null | \
xargs -L2 sh -c 'echo -n "$9: "; ping -nqc1 $9 | grep rtt'
192.168.100.132: rtt min/avg/max/mdev = 3.743/3.743/3.743/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 5.863/5.863/5.863/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 6.167/6.167/6.167/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 4.256/4.256/4.256/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 1.545/1.545/1.545/0.000 ms
$ _
For those coming across this (like me), tcpdump buffering is the issue.
From the man page:
-l Make stdout line buffered. Useful if you want to see the data
while capturing it. For example:
# tcpdump -l | tee dat
or
# tcpdump -l > dat & tail -f dat
How to write a echo server bash script using tools like nc, echo, xargs, etc capable of simultaneously processing requests from multiple clients each with durable connection?
The best that I've came up so far is
nc -l -p 2000 -c 'xargs -n1 echo'
but it only allows a single connection.
If you use ncat instead of nc your command line works fine with multiple connections but (as you pointed out) without -p.
ncat -l 2000 -k -c 'xargs -n1 echo'
ncat is available at http://nmap.org/ncat/.
P.S. with the original the Hobbit's netcat (nc) the -c flag is not supported.
Update: -k (--keep-open) is now required to handle multiple connections.
Here are some examples. ncat simple services
TCP echo server
ncat -l 2000 --keep-open --exec "/bin/cat"
UDP echo server
ncat -l 2000 --keep-open --udp --exec "/bin/cat"
In case ncat is not an option, socat will also work:
socat TCP4-LISTEN:2000,fork EXEC:cat
The fork is necessary so multiple connections can be accepted. Adding reuseaddr to TCP4-LISTEN may be convenient.
netcat solution pre-installed in Ubunutu
The netcat pre-installed in Ubuntu 16.04 comes from netcat-openbsd, and has no -c option, but the manual gives a solution:
sudo mknod -m 777 fifo p
cat fifo | netcat -l -k localhost 8000 > fifo
Then client example:
echo abc | netcat localhost 8000
TODO: how to modify the input string value? The following does not return any reply:
cat fifo | tr 'a' 'b' | netcat -l -k localhost 8000 > fifo
The remote shell example however works:
cat fifo | /bin/sh -i 2>&1 | netcat -l -k localhost 8000 > fifo
I don't know how to deal with concurrent requests simply however.
what about...
#! /bin/sh
while :; do
/bin/nc.traditional -k -l -p 3342 -c 'xargs -n1 echo'
done
On Linux, I can use netstat -pntl | grep $PORT or fuser -n tcp $PORT to find out which process (PID) is listening on the specified TCP port. How do I get the same information on Mac OS X?
On macOS Big Sur and later, use this command:
sudo lsof -i -P | grep LISTEN | grep :$PORT
or to just see just IPv4:
sudo lsof -nP -i4TCP:$PORT | grep LISTEN
On older versions, use one of the following forms:
sudo lsof -nP -iTCP:$PORT | grep LISTEN
sudo lsof -nP -i:$PORT | grep LISTEN
Substitute $PORT with the port number or a comma-separated list of port numbers.
Prepend sudo (followed by a space) if you need information on ports below #1024.
The -n flag is for displaying IP addresses instead of host names. This makes the command execute much faster, because DNS lookups to get the host names can be slow (several seconds or a minute for many hosts).
The -P flag is for displaying raw port numbers instead of resolved names like http, ftp or more esoteric service names like dpserve, socalia.
See the comments for more options.
For completeness, because frequently used together:
To kill the PID:
sudo kill -9 <PID>
# kill -9 60401
Up to macOS 13 Ventura, every version of macOS supports this:
sudo lsof -iTCP -sTCP:LISTEN -n -P
Personally I've end up with this simple function in my ~/.bash_profile:
listening() {
if [ $# -eq 0 ]; then
sudo lsof -iTCP -sTCP:LISTEN -n -P
elif [ $# -eq 1 ]; then
sudo lsof -iTCP -sTCP:LISTEN -n -P | grep -i --color $1
else
echo "Usage: listening [pattern]"
fi
}
Then listening command gives you a listing of processes listening on some port and listening smth greps this for some pattern.
Having this, it's quite easy to ask about particular process, e.g. listening dropbox, or port, e.g. listening 22.
lsof command has some specialized options for asking about port, protocol, process etc. but personally I've found above function much more handy, since I don't need to remember all these low-level options. lsof is quite powerful tool, but unfortunately not so comfy to use.
You can also use:
sudo lsof -i -n -P | grep TCP
This works in Mavericks.
Update January 2016
Really surprised no-one has suggested:
lsof -i :PORT_NUMBER
to get the basic information required. For instance, checking on port 1337:
lsof -i :1337
Other variations, depending on circumstances:
sudo lsof -i :1337
lsof -i tcp:1337
You can easily build on this to extract the PID itself. For example:
lsof -t -i :1337
which is also equivalent (in result) to this command:
lsof -i :1337 | awk '{ print $2; }' | head -n 2 | grep -v PID
Quick illustration:
For completeness, because frequently used together:
To kill the PID:
kill -9 <PID>
# kill -9 60401
or as a one liner:
kill -9 $(lsof -t -i :1337)
For the LISTEN, ESTABLISHED and CLOSED ports
sudo lsof -n -i -P | grep TCP
For the LISTEN ports only
sudo lsof -n -i -P | grep LISTEN
For a specific LISTEN port, ex: port 80
sudo lsof -n -i -P | grep ':80 (LISTEN)'
Or if you just want a compact summary [no service/apps described], go by NETSTAT. The good side here is, no sudo needed
netstat -a -n | grep 'LISTEN '
Explaining the items used:
-n suppress the host name
-i for IPv4 and IPv6 protocols
-P omit port names
-a [over netstat] for all sockets
-n [over netstat] don't resolve names, show network addresses as numbers
Tested on High Sierra 10.13.3 and Mojave 10.14.3
the last syntax netstat works on linux too
on OS X you can use the -v option for netstat to give the associated pid.
type:
netstat -anv | grep [.]PORT
the output will look like this:
tcp46 0 0 *.8080 *.* LISTEN 131072 131072 3105 0
The PID is the number before the last column, 3105 for this case
On macOS, here's an easy way to get the process ID that's listening on a specific port with netstat. This example looks for a process serving content on port 80:
find server running on port 80
netstat -anv | egrep -w [.]80.*LISTEN
sample output
tcp4 0 0 *.80 *.* LISTEN 131072 131072 715 0
The 2nd from the last column is the PID. In above, it's 715.
options
-a - show all ports, including those used by servers
-n - show numbers, don't look up names. This makes the command a lot faster
-v - verbose output, to get the process IDs
-w - search words. Otherwise the command will return info for ports 8000 and 8001, not just "80"
LISTEN - give info only for ports in LISTEN mode, i.e. servers
On the latest macOS version you can use this command:
lsof -nP -i4TCP:$PORT | grep LISTEN
If you find it hard to remember then maybe you should create a bash function and export it with a friendlier name like so
vi ~/.bash_profile
and then add the following lines to that file and save it.
function listening_on() {
lsof -nP -i4TCP:"$1" | grep LISTEN
}
Now you can type listening_on 80 in your Terminal and see which process is listening on port 80.
On Snow Leopard (OS X 10.6.8), running 'man lsof' yields:
lsof -i 4 -a
(actual manual entry is 'lsof -i 4 -a -p 1234')
The previous answers didn't work on Snow Leopard, but I was trying to use 'netstat -nlp' until I saw the use of 'lsof' in the answer by pts.
I am a Linux guy. In Linux it is extremely easy with netstat -ltpn or any combination of those letters. But in Mac OS X netstat -an | grep LISTEN is the most humane. Others are very ugly and very difficult to remember when troubleshooting.
checkout this project/tool: procs
install on MacOs: brew install procs
This allows you control what to display with procs.
To see TCP/UDP Ports, add below to ~/.procs.toml after installing the tool.
[[columns]]
kind = "TcpPort"
style = "BrightYellow|Yellow"
numeric_search = true
nonnumeric_search = false
align = "Left"
[[columns]]
kind = "UdpPort"
style = "BrightGreen|Green"
numeric_search = false
nonnumeric_search = true
align = "Left"
Here is a sample output:
lsof -n -i | awk '{ print $1,$9; }' | sort -u
This displays who's doing what. Remove -n to see hostnames (a bit slower).
This did what I needed.
ps -eaf | grep `lsof -t -i:$PORT`
I made a small script to see not only who is listening where but also to display established connections and to which countries. Works on OSX Siera
#!/bin/bash
printf "\nchecking established connections\n\n"
for i in $(sudo lsof -i -n -P | grep TCP | grep ESTABLISHED | grep -v IPv6 |
grep -v 127.0.0.1 | cut -d ">" -f2 | cut -d " " -f1 | cut -d ":" -f1); do
printf "$i : " & curl freegeoip.net/xml/$i -s -S | grep CountryName |
cut -d ">" -f2 | cut -d"<" -f1
done
printf "\ndisplaying listening ports\n\n"
sudo lsof -i -n -P | grep TCP | grep LISTEN | cut -d " " -f 1,32-35
#EOF
Sample output
checking established connections
107.178.244.155 : United States
17.188.136.186 : United States
17.252.76.19 : United States
17.252.76.19 : United States
17.188.136.186 : United States
5.45.62.118 : Netherlands
40.101.42.66 : Ireland
151.101.1.69 : United States
173.194.69.188 : United States
104.25.170.11 : United States
5.45.62.49 : Netherlands
198.252.206.25 : United States
151.101.1.69 : United States
34.198.53.220 : United States
198.252.206.25 : United States
151.101.129.69 : United States
91.225.248.133 : Ireland
216.58.212.234 : United States
displaying listening ports
mysqld TCP *:3306 (LISTEN)
com.avast TCP 127.0.0.1:12080 (LISTEN)
com.avast TCP [::1]:12080 (LISTEN)
com.avast TCP 127.0.0.1:12110 (LISTEN)
com.avast TCP [::1]:12110 (LISTEN)
com.avast TCP 127.0.0.1:12143 (LISTEN)
com.avast TCP [::1]:12143 (LISTEN)
com.avast TCP 127.0.0.1:12995 (LISTEN)
com.avast [::1]:12995 (LISTEN)
com.avast 127.0.0.1:12993 (LISTEN)
com.avast [::1]:12993 (LISTEN)
Google TCP 127.0.0.1:34013 (LISTEN)
This may be useful to check if you are connected to north-korea! ;-)
This is a good way on macOS High Sierra:
netstat -an |grep -i listen
Inspired by user Brent Self:
lsof -i 4 -a | grep LISTEN
For macOS I use two commands together to show information about the processes listening on the machine and process connecting to remote servers. In other words, to check the listening ports and the current (TCP) connections on a host you could use the two following commands together
1. netstat -p tcp -p udp
2. lsof -n -i4TCP -i4UDP
Thought I would add my input, hopefully it can end up helping someone.
Just a slight improvement on Michał Kalinowski's answer (I don't have enough reputation to leave a comment there): if you are trying to find the process listening on a port numbered 255 and below, the grep command might print lines related to the IP address, and which do not correspond to the desired result. For a port with any number, the grep command might also erroneously match the device's MAC address or PID. To improve on this, I suggest changing the command to grep --color ":$1 "