I'm trying to get diagnostics data from a device to which I connect over the network. It offers a nice tight proprietary protocol where you send a 3 or 4 character request and receive a <~30 character response. Things to take into account:
You must wait 20 ms between requests (I want to poll as fast as possible)
You must wait for the response before sending the next request
While the request must end in a Unix line end, the responses come with Mac line ends (I know)
I'm trying to do this with netcat. Basic operation is easy:
$ echo STA | netcat <IP> <Port>
123498754
The STA request will give you a status word that netcat writes to stdout. I can also do this:
$ cat | netcat <IP> <Port>
STA # <- typed input
12345678 # written by netcat
STA # <- typed input
12345678 # written by netcat
Here I can see that the first request takes considerably longer: multiple seconds versus no noticeable delay. I want to get rid of the delay. And so my quest begins: Keep netcat open!
I cannot just pipe a file into netcat, because I have to wait for the responses. So I'm trying to do this using FIFOs. I created two (/d/pc2dev and /d/dev2pc), both with rw permission. I could write a script based on that if I could get the basics working. This works:
#### SHELL 1 #############################
$ tail -f /d/pc2dev | netcat <IP> <Port> | tr '\r' '\n'
12345678654 # written by netcat with delay
12345678654 # written by netcat as soon as I send the request on shell 2
12345678654 # ditto
#### SHELL 2 #############################
$ echo STA > /d/pc2dev
$ echo STA > /d/pc2dev
$ echo STA > /d/pc2dev
(without the tr, responses will get written over one another.)
However, this does not:
#### SHELL 1 #############################
$ tail -f /d/pc2dev | netcat <IP> <Port> | tr '\r' '\n' > /d/dev2pc
#### SHELL 2 #############################
$ echo STA > /d/pc2dev
$ echo STA > /d/pc2dev
#### SHELL 3 #############################
$ cat /d/dev2pc
# expecting output here - none comes!
Writing to a normal file also doesn't work:
#### SHELL 1 #############################
$ tail -f /d/pc2dev | netcat <IP> <Port> | tr '\r' '\n' > log.txt
#### SHELL 2 #############################
$ echo STA > /d/pc2dev
$ echo STA > /d/pc2dev
$ cat log.txt
# nothing
It seems that the write operation to FIFO/file is buffered somewhere, while the write to stdout is done immediately. (However, even on ctrl+c'ing netcat et al., this hypothetical buffered output does not get written.)
Be it like that or otherwise - how can I get this to work?
try piping to tee instead of redirecting the output, man 1 tee, see what happens.
The following worked
$ tail -f /d/pc2dev | stdbuf -i0 -o0 netcat <IP> <Port> | stdbuf -i0 -o0 tr '\r' '\n' > /d/dev2pc
Related
mkfifo /tmp/f ; cat /tmp/f | /bin/bash -i 2>&1 | nc -l -p 1234 > /tmp/f
I am new to bash, I am trying to understand this piece of "code".
Why a while loop is not needed? How can this work? Is it itself a loop? Why? How?
Also, cat filePipe by itself ONLY PRINTS ONE LINE, and then exits (I just tested it), and to make cat not to exit I do: while cat pipeFile ; do : ; done. So how does that above work?
I don't get the order of execution... at the beginning /tmp/f is empty, so cat /tmp/f should "send" an empty stream to /bin/bash which just send it to nc which opens a connection and "sends" the interactive bash to whoever connects... and the response of the client is sent to /tmp/f ... and then? What? How can it can go back and do the same things again?
When bash parses the line mkfifo /tmp/f ; cat /tmp/f | /bin/bash -i 2>&1 | nc -l -p 1234 > /tmp/f, several things happen. First, the fifo is created. Then, in no particular order, 3 things happen: cat is started, bash is started and nc is started with its output stream connected to /tmp/f. cat is now going to block until some other process opens /tmp/f for writing; the nc is about to do that (or already did it, but we don't know if cat will start before nc or if nc starts before cat, nor do we know in which order they will open the fifo, but whoever does it first will block until the other completes the operation). Once all 3 processes start, they will just sit there waiting for some data. Eventually, some external process connects to port 1234 and sends some data into nc, which writes into /tmp/f. cat (eventually) reads that data and sends it downstream to bash, which processes the input and (probably) writes some data into nc, which sends it back across the socket connection.
If you have a test case in which cat /tmp/f only writes one line of data, that is simply because whatever process you used to write into /tmp/f only wrote a single line. Try: printf 'foo\nbar\nbaz\n' > /tmp/f & cat /tmp/f or while sleep 1; do date; done > /tmp/f & cat /tmp/f
/tmp/f is NOT empty, but a fifo, a bi-directional link.
Someone connects to port 1234, type something, which nc will forward to fifo which then feeds into bash.
bash runs the command and sends results back to nc.
.1 You misunderstand what happen when you echo "string" >/path/fifo
.a) When you just echo something >/path/to/somewhere, you
(test accessibility, then) open target somewhere for writting
write someting in openned file descriptor (fd)
close (relax) accessed file.
.b) A fifo (The First In is the First Out.) is not a file.
Try this:
# Window 1:
mkfifo /tmp/fifotest
cat /tmp/fifotest
# Window 2:
exec {fd2fifo}>/tmp/fifotest
echo >&$fd2fifo Foo bar
You will see cat not terminating.
echo >&$fd2fifo Baz
exec {fd2fifo}>&-
Now, cat will close
So there is no need of any loop!
.2 command cat /tmp/f | /bin/bash -i 2>&1 | nc -l -p 1234 > /tmp/f
could be written (avoid useless use of cat):
bash -i 2>&1 </tmp/f | nc -l -p 1234 > /tmp/f
but you could do same operation but from different point of vue:
nc -l -p 1234 </tmp/f | bash -i >/tmp/f 2>&1
The goal is
to drive bash's STDIN from nc's STDOUT and
connect back bash's STDOUT and STDERR to nc's STDIN.
.3 The more: bashism
Under bash, you could avoid creating fifo by using unnamed fifo:
coproc nc -l -p 1234; bash -i >&${COPROC[1]} 2>&1 <&${COPROC[0]}
or
exec {ncin}<> <(:); nc -l -p 1234 <&$ncin | bash -i >&$ncin 2>&1
readarray -t FILES < text.in
for ELEMENT in ${FILES[#]}
do
printf $ELEMENT
sleep 2
done | nc -q -1 localhost 5555
This code simulates a slow network client which sends a text line from text.in every 2 seconds to a key-value-store server with netcat.
How do I compare the output of netcat with a file test.out?
Example for text.in:
GET$5$hello
SET$5$hello$7$myworld
GET$5$hello
QUIT
Example for test.out:
ERR
OK
VALUE$7$myworld
Once the QUIT line is sent, the server closes the connection. After that I want to compare the standard output of netcat to test.out
I could do this: nc localhost 5555 < text.in | diff - test.out but then the entire file text.in is sent immediately which defeats the purpose of simulating a slow client.
I tried the following but it gets stuck and never returns anything:
readarray -t FILES < text.in
for ELEMENT in ${FILES[#]}
do
printf $ELEMENT
sleep 2
done | nc -q -1 localhost 5555 | diff - test.out
EDIT: ANSWER IN COMMENTS
A very simple use of netcat is below:
#!/bin/sh
echo "hello world" > original.txt
base64 original.txt > encoded.txt
cat encoded.txt | nc -l -p 1234 -q 0
#!/bin/sh
nc localhost 1234 -q 0 > received.txt
base64 -d received.txt > decoded.txt
rm received.txt encoded.txt
echo "Sent:"
md5sum original.txt
echo "Received:"
md5sum decoded.txt
rm decoded.txt original.txt
This creates a file, encodes it into base64, sends it over netcat, and then decodes it, finally comparing whether what was sent and what was received is identical using a checksum comparison.
On a Kali Linux machine I was using earlier, the netcat connection closes upon execution of the second script, but trying on Ubuntu at home, this is not the case. I need to manually close the connection with Ctrl+D.
How can I make it such that the connection closes after receiving this data?
I think including the -c flag for nc should do it. Also, are you sure you want to use Bourne shell, and not Bash? I'd suggest changing your shebang to #!/bin/bash
I am using ios-webkit-proxy-debug remote server which usually shuts down or just disconnects.
I want to restart server if last line of out contains "Disconnected" or command is not running at all.
The problem may be that the output is buffered. You may have luck with the util "stdbuf" to disable the buffer. (another tool is "unbuffer") You can fully disable all buffers with:
stdbuf -i0 -o0 -e0 [command] # 0 is unbuffered and L is line-buffered
Your command might look like this:
stdbuf -oL -eL ios_webkit_debug_proxy |& tee -a proxy.log
tail -f -n0 proxy.log | grep --line-buffered "Disconnected" | while read line ; do [restart server] ; done
I tested it with this:
# This is one terminal
cd /tmp
echo > log
# This in another terminal
cd /tmp
tail -f -n0 log | grep --line-buffered "disconnect" | while read line ; do echo "found disconnect" ; done
# Then, in the first terminal
echo "test" >> log # second terminal does nothing
echo "disconnect" >> log # second terminal echos "found disconnect"
The tail -n0 is because if the tail reads a disconnect in a log file that already exists, it will restart the server as soon as you run that command.
EDIT:
stdbuf is overridden by tee (see man tee). You may have more luck in a different format but some stuff to play around with:
stdbuf -oL -eL ios_webkit_debug_proxy 2>&1 >> proxy.log
# or
unbuffer ios_webkit_debug_proxy |& tee -a proxy.log | grep --line-buffered "Disconnected" | while read line ; do [restart server] ; done
I'm interested in following several remote files at the same time and simultanteously aggregating stats over the file. So far, I'm doing this as follows:
mkfifo mfifo
ssh -ft host1 'tail -f /path/to/log | grep something' > mfifo &
ssh -ft host2 'tail -f /path/to/log | grep something' > mfifo &
ssh -ft host3 'tail -f /path/to/log | grep something' > mfifo &
cat mfifo | awk '{x += $4; print $3} END {printf "total: %d", x}'
This pretty much works as expected, with an aggregation of the grepped logs streamed through awk. However, I'm not sure how to get the final total to be printed. I gather that I need to close the writers of the fifo, but I'm not sure how to do this. Any suggestions regarding how to do this without storing the whole stream as a file?
Killing FIFO Writers
You can use fuser to kill the processes writing to a file. For example:
fuser -TERM -k -w mfifo; sleep 5; fuser -k -w mfifo
Note that fuser defaults to sending SIGKILL, so the example given sends an explicit SIGTERM, then waits five seconds before forcefully terminating the process. This should allow your processes to clean up after themselves, but feel free to adjust the invocation to suit.
Also, note that we're passing the -w flag, so that fuser only kills processes with write access. Without this flag, you'll also be killing cat and awk.