I have two programs A and B (in my case a C program and a Java program) that are supposed communicate with each other. The invocation inside a bash script of those programs looks like this:
mkfifo fifo1
mkfifo fifo2
A < fifo1 > fifo2 &
java B < fifo2 > fifo1
I know that I could do it with one fifo, but I also want to be able to show the communication on the console. The following works fine:
mkfifo fifo1
mkfifo fifo2
A < fifo1 | tee fifo2 &
java B < fifo2 | tee fifo1
My question is: Why does the second script work while the first one just hangs?
Side question: While the second version works, as soon as I redirect the output of the script to a file, the communication is no longer interleaved but ordered by process. Is there a way to prevent this?
Why does the second script work while the first one just hangs?
man open:
When opening a FIFO with O_RDONLY or O_WRONLY set:
If O_NONBLOCK is set, an open() for reading-only
shall return without delay. An open() for writing-only shall
return an error if no process currently has the file open for
reading.
If O_NONBLOCK is clear, an open() for reading-only
shall block the calling thread until a thread opens the file for
writing. An open() for writing-only shall block the calling
thread until a thread opens the file for reading.
In the first script, A opens fifo1 and B opens fifo2, both with O_RDONLY; A blocks until B would open fifo1 for writing, while B blocks until A would open fifo2 for writing… a circular wait situation. (Actually, shells open the fifos, but the resulting circular waiting is the same.)
In the second script, A opens fifo1 and B opens fifo2, both with O_RDONLY - so far the same as above. But in parallel, the first tee opens fifo2 and the second tee opens fifo1 for writing, thus unblocking A and B.
While the second version works, as soon as I redirect the output of
the script to a file, the communication is no longer interleaved but
ordered by process. Is there a way to prevent this?
This may be due to stdout buffering; try … stdbuf -oL tee … or post your input and output.
Related
I am trying to write the output of top to a file every 5 seconds which I also got to work.
cmd = "top -s 5 -pid 123 >> top.txt"
p = IO.popen(cmd)
However, the problem I have is that I can't find a way of closing top
I have tried
Process.kill('KILL', p.pid)
but top keeps writing to the output file.
p.close hangs but if I ctrl + c it does seem to exit the top command as well. But this as this requires me to manually ctrl + c it is not a viable solution.
Any help would be appreciated!
The problem is the redirection. >> is a feature of a shell: it starts the process and connects its stdout to the given file. In order for Ruby to do this, it actually starts a shell, which starts top and sets up the redirection.
So p.pid is the PID of the shell, not top. When you kill it, it kills only the shell and top gets disowned, continuing to run under PID 1.
I recommend using Popen3 instead and running just top -s 5 -pid 123 without redirection. This gives you the subprocess as well as its stdout/stderr, so you can manage the output yourself (such as appending it to a file) while being able to kill it.
Alternatively, make a wrapper shell script that runs top with redirection and set it up to kill top when it exits: How do I terminate all the subshell processes? Then have Ruby run that wrapper script.
What's a good implementation of a multiprocess queue in Bash?
I was considering a FIFO, with each line representing an element in the queue:
mkfifo fifo
ls > fifo
In a different process:
read element < fifo
The expected result is that the reader process reads one line (i. e. one element) and stores it in the variable $element, leaving the rest of the queue untouched so that other reader processes can get elements (lines) as well.
Unfortunately this does not work. The read statement opens the FIFO, causing the writer (ls) to complete at once, closing the FIFO then seems to cause the remaining data to be dropped, other elements cannot be read by another process (in fact, the next read < fifo hangs until another writer appears and writes into the FIFO).
I also considered touching files in a special directory (as a writer) and moving the files away (as a reader), but this seems tedious and obviously is not feasible for millions of queue entries.
Can I get the FIFO variant to work somehow?
Is there a different way of implementing a shell queue, having several writers and several readers, all working on the same queue?
You just need to keep the PIPE open
$mkfifo PIPE
$cat > PIPE &
Pipe is now open indefinitely until you kill the cat.
$ls > PIPE &
$read Line < PIPE
$echo $Line
file1
You can now write and read to your hearts content.
I may have found an answer myself. I'm not using FIFOs but a minimalistic TCP server accepting input from one port and writing output line by line to another.
To set up the TCP server, I use this script:
nc -k -l 4444 | while read a
do echo "$a" | nc -l 4445
done
(Append & to run this in the background, of course.)
Then the writers can do sth like this:
for ((i=0; i<10000; i++))
do
printf "x%02d\n" "$i"
done >/dev/tcp/127.0.0.1/4444
and the readers can do sth like this:
while ! { read a < /dev/tcp/localhost/4445; } 2>/dev/null
do
sleep 2 # we poll; if there is nothing, we sleep between polls
done
echo "$a"
This script fetches one element (line) and processes it (echo "$a"). Do this in a loop if you want to drain the queue.
I'm not all too happy with the polling solution, but tests show that it works reliably with two writers and two readers (and I don't see why more readers and writers should pose a problem).
I have a script that creates a FIFO and launches a program that writes output to the FIFO. I then read and parse the output until the program exits.
MYFIFO=/tmp/myfifo.$$
mkfifo "$MYFIFO"
MYFD=3
eval "exec $MYFD<> $MYFIFO"
external_program >&"$MYFD" 2>&"$MYFD" &
EXT_PID=$!
while kill -0 "$EXT_PID" ; do
read -t 1 LINE <&"$MYFD"
# Do stuff with $LINE
done
This works fine reading input while the program is still running, but it looks like the timeout to read is ignored, and read call hangs after the external program exits.
I've used read with a timeout successfully in other scripts, and a simple test script that leaves out the external program times out correctly. What am I doing wrong here?
EDIT: It looks like read -t functions as expected when I run my script from the command line, but when I run it as part of an xcodebuild build process, the timeout does not function. What is different about these two environments?
I don't think -t will work with redirection.
From the man page here:
-t timeout
Cause read to time out and return failure if a complete line
of input is not read within timeout seconds. This option has no
effect if read is not reading input from the terminal or a pipe.
Is it possible to set a process to die in its start parameters?
What I'm trying to do is set a process to die either before it's started or when it's starting.
When I try to get a pid of an opened named pipe with cat > $pipe by adding an ampersand and $! after it, it spams the terminal with ">" symbols, so I was wondering if it were possible to start cat > $pipe with a parameter to die on a PID's SIGHUP.
How would I go about doing this? If I put anything after the cat > $pipe, it will not work correctly for me.
"get a pid of an opened named pipe"
A named pipe does not have a pid, only processes have pids (the clue is in the 'p')
However, that does not appear to be anything to do with the title of the question. By default, a process will die on a SIGHUP. However, a child process inherits the parent's signal mask, so if the parent ignored SIGHUP then that will be the case in the child (not true for handlers). So you can force a die with (for example):
trap 'exit 128' SIGHUP
But how does that part of the question relate to named pipes? Are you trying to find which processes have the pipe open? You can iterate through /proc for that.
EDIT after comments from the OP:
If you run cat > mypipe & then the cat will hang trying to access the keyboard - cat by default reads STDIN.
[1]+ Stopped cat > mypipe
So then you have to bring it into forground (fg) to enter data, normally terminated by <CTRL>+D. I am at a loss as to why you want to use cat in this way.
Anyway, if you run in background it is very easy to get a background job's pid:
assuming it is job number 1
set $(jobs -l %1)
pid=$2
Maybe you could further investigate why you can't run the job in background, and show an example (use the script command to get a copy of your session in a file called typescript)
Have you tried putting it in parentheses? (cat > $pipe) &$
I'm trying to figure out what a line means in a bash script file:
mkfifo mypipe
nc -l 12345 < mypipe | /home/myprogram > mypipe
Here's what I understand: nc -l part creates a server-side like behavior on port 12345, which takes in input from mypipe, which pipes that output to a program, which pipes the program output back into mypipe.
My question is firstly is my analysis correct? Second, what exactly is the mkfifo, like what kind of file is it? I also don't understand what nc -l outputs exactly in order to pipe into the myprogram.
Thanks for any help.
mkfifo creates a pipe file. Here, FIFO means "first-in, first-out". Whatever one process writes into the pipe, the second process can read. It is not a "real" file - the data never gets saved to the disk; but Linux abstracts a lot of its mechanisms as files, to simplify things.
nc -l 12345 will bind to socket 12345 and listen; when it catches an incoming connection, it will pass the standard input to the remote host, and the remote host's incoming data to the standard output.
Thus, the architecture here is:
remote host -> nc -> regular pipe -> myprogram
myprogram -> mypipe -> nc -> remote host
effectively letting myprogram and remote host talk, even though myprogram was designed to read from stdin and write to stdout.
Since the bash pipe (|) only handles one direction of communication, you need to make an explicit second pipe to do bidirectional inter-process connection.