How do you fork and communicate with a subprocess in D?
I think I'm pretty much looking for http://erdani.com/d/new-stdio/phobos-prerelease/std_process.html#pipeProcess but pipeProcess doesn't seem to be in D2.
I want to execute a child process and be able to read from its stdout and stderr, write to its stdin and receive notification of it exiting and its exit code.
What is the best way of doing this in D?
Thanks,
Chris.
Eventually this functionality will be in Phobos, but for now, here's a class I tossed together: Pastebin link
You use it like this:
auto stream = new ProcessStream("ls -a");
// read, write, etc. from stream
Currently, same way as you would in C. pipe, fork, exec.
Not pretty.
Related
With clojure.java.shell/sh it's possible to execute a shell command. After the invoked process is finished, the function returns a map containing it's exit code, std-out and std-err strings.
How can I capture stdout/-err of a spawned process from the moment it started? And: How can I terminate the process from within a clojure program/repl?
As far as I know it is not possible with clojure.java.shell/sh. You might take a look at Raynes/conch which provides features you ask for (getting output right after start etc.)
You can also DIY with java.lang.ProcessBuilder and java.lang.Process where you have full access to process's input stream or a method to terminate it.
There's a long-running Unix process which output I'd wish to capture and process with Clojure. A good example of one such process is a repl-y / nREPL session: its duration is indefinite, and output gets printed to stdout.
If I try (clojure.java.io/sh "lein" "repl"), evaluation will block until the underlying process finishes, and then I can observe the output.
This is not what I want - I want to get a stream immediately instead.
Can I achieve this using clojure.java.io, or similar, existing Clojure tools? Wouldn't mind resorting to Java otherwise.
Take a look at the me.raynes.conch library, it's a bit more versatile than clojure.java.shell. It's low-level API seems to be what you're looking for.
Not a detailed answer, but the source for Clojure's sh function is pretty short. If you reworked it slightly to remove the .waitFor (or added a higher-order function to consume the partial reads returned by the InputStreamReader as they arrived), you could probably get updated data as it's returned by the process. But be careful of deadlocks in case your subprocess expects input as well (as in your lein repl example).
I have created a bash script and it runs in the background. It has a PID which is stored in a file, and I can use KILL to pass predefined signals to the process.
From time to time however, I'd like to pass information to the process manually. Preferably what I would like to happen is to be able to pass a string or array of information, which is captured through TRAP, then the forever loop inside the bash file will process the information. Is there an easy way to pass information into a background process?
Thanks
You can create a fifo, have the main process write to it and have the child read from it.
mkfifo link
run_sub < link &
generate_output > link
Have it listen on a socket and implement a protocol to achieve your communication aims, probably a bit much for bash.
Or, have it try to read a particular file on receipt of a particular signal. For example, it is common for programs to re-read their configuration files on receipt of a HUP.
The systemu page says:
systemu can be used on any platform to return status, stdout, and stderr of any command. unlike other methods like open3/popen4 there is zero danger of full pipes or threading issues hanging your process or subprocess.
(https://github.com/ahoward/systemu)
Could anyone explain this a little bit?
Methods like popen and its various spinoffs are convenient and are part of the expected API for a full I/O library.
However, they must be used either casually or carefully because they are prone to deadlock. By casually, I mean, if you both write and read from the command, it's still OK as long as you either don't write a lot or don't read a lot. By carefully, I mean, you can move large amounts of data, but only if you keep the inner details of the operation in mind and deliberately engineer against deadlock.
Imagine writing lots of stuff to your popened command and then reading a result. If you write more than a pipe will buffer, then your process will sleep. That's OK in practice, most of the time, but what if the command has to write a lot of stuff? Now it may sleep and not finish reading input that you are sending. You won't finish sending input so you will never wake up and read results.
Deadlock!
I've written a Windows application using the native win32 API. My app will launch other processes and capture the output and highlight stderr output in red.
In order to accomplish this I create a separate pipe for stdout and stderr and use them in the STARTUPINFO structure when calling CreateProcess. I then launch a separate thread for each stdout/stderr handle that reads from the pipe and logs the output to a window.
This works fine in most cases. The problem I am having is that if the child process logs to stderr and stdout in quick succession, my app will sometimes display the output in the incorrect order. I'm assuming this is due to using two threads to read from each handle.
Is it possible to capture stdout and stderr in the original order they were written to, while being able to distinguish between the two?
I'm pretty sure it can't be done, short of writing the spawned program to write in packets and add a time-stamp to each. Without that, you can normally plan on buffering happening in the standard library of the child process, so by the time they're even being transmitted through the pipe to the parent, there's a good chance that they're already out of order.
In most implementations of stdout and stderr that I've seen, stdout is buffered and stderr is not. Basically what this means is that you aren't guaranteed they're going to be in order even when running the program on straight command line.
http://en.wikipedia.org/wiki/Stderr#Standard_error_.28stderr.29
The short answer: You cannot ensure that you read the lines in the same order that they appear on cmd.exe because the order they appear on cmd.exe is not guaranteed.
Not really, you would think so but std_out is at the control of the system designers - exactly how and when std_out gets written is subject to system scheduler, which by my testing is subordinated to issues that are not as documented.
I was writing some stuff one day and did some work on one of the devices on the system while I had the code open in the editor and discovered that the system was giving real-time priority to the driver, leaving my carefully-crafted c-code somewhere about one tenth as important as the proprietary code.
Re-inverting that so that you get sequential ordering of the writes is gonna be challenging to say the least.
You can redirect stderr to stdout:
command_name 2>&1
This is possible in C using pipes, as I recall.
UPDATE: Oh, sorry -- missed the part about being able to distinguish between the two. I know TextMate did it somehow using kinda user visible code... Haven't looked for a while, but I'll give it a peek. But after some further thought, could you use something like Open3 in Ruby? You'd have to watch both STDOUT and STDERR at the same time, but really no one should expect a certain ordering of output regarding these two.
UPDATE 2: Example of what I meant in Ruby:
require 'open3'
Open3.popen3('ruby print3.rb') do |stdin, stdout, stderr|
loop do
puts stdout.gets
puts stderr.gets
end
end
...where print3.rb is just:
loop do
$stdout.puts 'hello from stdout'
$stderr.puts 'hello from stderr'
end
Instead of throwing the output straight to puts, you could send a message to an observer which would print it out in your program. Sorry, I don't have Windows on this machine (or any immediately available), but I hope this illustrates the concept.
I'm pretty sure that even if you don't separate them at all, you're still not guaranteed that they'll interchange one another in the correct order.
Since the intent is to annotate the output os an existing program, any possible interleaving of the two streams must be correct. The original developer will have placed appropriate flush() calls to ensure any mandatory ordering is honoured.
As previously explained, record each fragment that is written with a time stamp, and use this to recover the sequence actually seen by the output devices.