I'm using netcat piped thorugh pv to measure network speed
Server A
netcat -l -q -1 -p 1234 | pv > /dev/null
Server B
dd if=/dev/zero bs=10M count=1 | nc -v -n 10.10.10.2 1234
Result is returned on server B as
(UNKNOWN) [10.10.10.2] 1234 (?) open
1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 5.24922 s, 2.0 MB/s
What I need to do is convert MB/s to Mbit/s and output that so in script I can do
echo $speed
16 Mbits/sec
Also I need netcat to always be listengin on Server A and not close after the transfer has completed. I've tries the -k and -q -1 switches but no luck.
Related
I want to use netcat to return the result of a command on a server. BUT here is the trick, I want the command to run when the connection is made. Not when I start the netcat.
Here is a simplified single shell example. I want the ls to run when I connect to 1234 (which I would normally do from a remote server, obviously this example is pointless I could do an ls locally)
max $ ls | nc -l 1234 &
[1] 36685
max $ touch file
max $ nc 0 1234
[1]+ Done ls | nc -l 1234
max $ ls | nc -l 1234 &
[1] 36699
max $ rm file
max $ nc 0 1234
file
[1]+ Done ls | nc -l 1234
You can see the ls runs when I start the listener, not when I connect to it. So in the first instance I had no file when I started it and created the file, then made the connection and it reported the state of the filesystem when the listen command started (empty), not the current state. And the second time around when file was already gone it showed it as still present.
Something similar to the way it works when you redirect from a file. eg:
max $ touch file
max $ nc -l 1234 < file &
[1] 36768
max $ echo content > file
max $ nc 0 1234
content
[1]+ Done nc -l 1234 < file
The remote connection gets the latest content of the file, not the content when the listen command started.
I tried using the "file redirect" style with a subshell and that doesn't work either.
max $ nc -l 1234 < <(cat file) &
[1] 36791
max $ echo bar > file
max $ nc 0 1234
content
[1]+ Done nc -l 1234 < <(cat file)
The only thing I can think of is adding my command |netcat to xinetd.conf/systemd... I was probably going to have to add it to systemd as a service anyway.
(Actual thing I want to do : provide the list of users of the VPN to a network port for a remote service to get a current user list. My command to generate the list looks like :
awk '/Bytes/,/ROUTING/' /etc/openvpn/openvpn-status.log | cut -f1 -d. | tail -n +2 | tac | tail -n +2 | sort -b | join -o 2.2 -j1 - <(sort -k1b,1 /etc/openvpn/generate-new-certs.sh)
)
I think you want something like this, which I actually did with socat:
# Listen on port 65500 and exec '/bin/date' when a connection is received
socat -d -d -d TCP-LISTEN:65500 EXEC:/bin/date
Then in another Terminal, a few seconds later:
echo Hi | nc localhost 65500
Note: For macOS users, you can install socat with homebrew using:
brew install socat
I use the following command to send pinging IP's to a script:
sudo tcpdump -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo \
| cut -d " " -f 10 | xargs -L2 ./pong.sh
Unfortunately this gives me:
tcpdump: Unable to write output: Broken pipe
To dissect my commands:
Output the ping's from the traffic:
sudo tcpdump -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo
Output:
11:55:58.812177 IP xxxxxxx > 127.0.0.1: ICMP echo request, id 50776, seq 761, length 64
This will get the IP's from the tcpdump output:
cut -d " " -f 10 # output: 127.0.0.1
Get the output to the script:
xargs -L2 ./pong.sh
This will mimic the following example command:
./pong.sh 127.0.0.1
The strange thing is that the commands work seperate (on their own)...
I tried debugging it but I have no experience with debugging pipes. I checked the commands but they seem fine.
It would seem that's cut stdio buffering is interfering here, i.e. replace | xargs ... by | cat in your cmdline to find out.
Fwiw below cmdline wfm (pipe straight to xargs then use the shell itself to get the nth arg), note btw the extra tcpdump args : -c10 (just to limit to 10pkts, then show the 10/2 lines) and -Q in (only inbound pkts):
$ sudo tcpdump -c 10 -Q in -ne -l -i eth0 icmp and icmp[icmptype]=icmp-echo 2>/dev/null | \
xargs -L2 sh -c 'echo -n "$9: "; ping -nqc1 $9 | grep rtt'
192.168.100.132: rtt min/avg/max/mdev = 3.743/3.743/3.743/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 5.863/5.863/5.863/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 6.167/6.167/6.167/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 4.256/4.256/4.256/0.000 ms
192.168.100.132: rtt min/avg/max/mdev = 1.545/1.545/1.545/0.000 ms
$ _
For those coming across this (like me), tcpdump buffering is the issue.
From the man page:
-l Make stdout line buffered. Useful if you want to see the data
while capturing it. For example:
# tcpdump -l | tee dat
or
# tcpdump -l > dat & tail -f dat
Oftentimes I want to post something to a github bug like
$ ping google.com
PING google.com (216.58.195.238): 56 data bytes
Request timeout for icmp_seq 0
64 bytes from 216.58.195.238: icmp_seq=0 ttl=53 time=1064.747 ms
Right now I run the command, use screen's C-a C-[ to highlight the area, enter to copy it to that buffer, paste it into vim, write it to a file and then cat that into pbcopy. There has to be a better way.
Is there a command I can run which will tee the command I type prefixed with a $ and all the output to pbcopy? Or anything close? I envision
$ demo ping google.com
PING google.com (216.58.195.238): 56 data bytes
Request timeout for icmp_seq 0
64 bytes from 216.58.195.238: icmp_seq=0 ttl=53 time=1064.747 ms
^C
$
and now the original thing I pasted is in my mac clipboard.
You can do
script log.txt
ping www.google.com
exit
And you'll have your command and output saved in log.txt
Edit
Based on your comment, what you want is
command="whatever command you want to run"
echo \$ $command > log.txt
$command >> log.txt
I don't think you'll find a single command that does exactly this.
For MacOS (Mavericks), I am making a shell script to gather transfer stats over time for command dd.
The manual page says:
If dd receives a SIGINFO (see the status argument for stty(1)) signal,
the current input and output block counts will be written to the
standard error output in the same format as the standard completion
message.
Therefore, just like in Linux, I tried:
kill -INFO <pid_of_dd>
The command completes successfully with status 0, however the terminal in which dd process connected to, there is no stats information in standard output/standard error.
So what is the correct way to get dd to print stats in its output?
You can also press Ctrl+T in the Terminal tab to get the same behavior:
MacBook-Pro:~ $ dd if=~/source_image.dmg of=/dev/disk1
load: 0.87 cmd: dd 7229 uninterruptible 0.21u 3.91s
265809+0 records in
265808+0 records out
136093696 bytes transferred in 131.170628 secs (1037532 bytes/sec)
load: 0.99 cmd: dd 7229 uninterruptible 0.32u 5.89s
415769+0 records in
415768+0 records out
212873216 bytes transferred in 203.357068 secs (1046795 bytes/sec)
It seems to work for me:
$ dd if=/dev/zero of=/dev/null bs=1k &
[1] 33990
$ kill -INFO 33990
4787784+0 records in
4787784+0 records out
4902690816 bytes transferred in 4.260769 secs (1150658706 bytes/sec)
$ kill -INFO 33990
8357846+0 records in
8357846+0 records out
8558434304 bytes transferred in 7.428820 secs (1152058392 bytes/sec)
$ kill 33990
$ ps
PID TTY TIME CMD
1342 ttys000 0:00.02 -bash
2290 ttys001 0:00.17 -bash
[1]+ Terminated: 15 dd if=/dev/zero of=/dev/null bs=1k
$
I also found via commandlinefu that you can also do:
killall -INFO dd
If you had to run sudo dd to start dd you might try:
sudo killall -INFO dd
Also, I started dd in the background and with nohup so when I ran sudo killall -INFO dd and got nothing back for output I had to remember to go and look at the nohup.out file because that is where the response was logged to.
Worked great on OS X Mavericks.
You can press Ctrl+T while the dd command is running or, to have a nice progress bar, you can install pv (pipe viewer) with homebrew:
brew install pv
and then place pv in between
dd if=diskimage.img | pv | dd of=/dev/disk2
example output 1
18MB 0:00:11 [1.70MiB/s] [ <=> ]
(with size of transferred data, elapsed time and speed)
Progress bar and ETA
you can also input the size of the image (16GB in this example), to have :
dd if=diskimage.img | pv -s 16G | dd of=/dev/disk2
example output 2 (with also progress bar and estimated time):
1.61GiB 0:12:19 [2.82MiB/s] [===> ] 10% ETA 1:50:25
Using curl from the shell, what is the best way to discard (or detect) files that are not completely downloaded because a timeout occurred? What I'm trying to do is:
curl -m 2 --compress -o "dest/#1" "http://url/{$list}"
When a timeout occurs, the log shows it, and the part of the file that was downloaded is saved to disk:
[4/13]: http://URL/123.jpg --> dest/123.jpg
99 97984 99 97189 0 0 45469 0 0:00:02 0:00:02 --:--:-- 62500
curl: (28) Operation timed out after 2000 milliseconds with 97189 bytes received
I'm trying to either get rid of the files that were not 100% downloaded, or have them listed to attempt a resume (-C flag), later.
The best solution I have found so far is to capture the stderr of the curl call, and parse it with a combination of perl and grep to get the output file names:
curl -m 2 -o "dest/#1" "http://url/{$list}" 2>curl.out
perl -pe 's/[\t\n ]+/ /g ; s/--> /\n/g' curl.out | grep -i "Curl: (28)" | perl -pe 's/ .*//g'