Whenever I wish to run some outside process in Ruby I write something like this:
output = `outer_process`
This works well, and the output of the process is placed into "output". But sometimes the process takes a lot of time and gives a lot of output and I would like to see it on the screen even before it stopped running. Is there a way to do this?
Take a look at the open4 gem. There are some limitations, but assuming there is output to STDOUT from your process, you could do something like this:
Open4.open4( outer_process ) do | pid, pstdin, pstdout, pstderr |
pstdout.each { |line| puts line }
end
This is pretty similar, in terms of underlying mechanisms, to Anand's suggestion in comments.
Note this will not work immediately if the process you call is not flushing STDOUT. If you need to work around that limitation, you will need to provide a terminal for the child process, which is possible in Ruby, but more complicated - see answer to Continuously read from STDOUT of external process in Ruby
Related
I've spotted something that I find very puzzling about the behavior of System.cmd. I just wanted to ask if anyone might have thoughts on what I may be doing wrong or what may be going on.
I've been trying to wrap an Elixir script around the ack programmer's grep. I tried this:
{_message, errlevel} = System.cmd("ack",[])
And I get back the help text that ack displays on an empty command line; I won't bother to reproduce it here because it's not necessarily germane to the question.
Then I try this:
{_message, errlevel} = System.cmd("ack",[""])
And it looks like iex hangs. Now I realize in the first case the output may be going to stderr rather than stdout. But there's another reason I'm asking about this; I found something even more interesting to me. Because I'm not 100% committed to using ack I thought I'd try ripgrep on the thought that it might interact with stdout better.
So if I do this:
{_message, errlevel} = System.cmd("rg",[])
Same as ack with no arguments--shows me the help text. Again I'm guessing it's probably out to stderr. I could check to confirm my assumption but what's even more interesting to me is that when I do this:
{_message, errlevel} = System.cmd("rg",[""])
It hangs again!
I had always figured the issue is with how ack interacts with stdout but now I'm not so sure since I see the same behavior with ripgrep. This is Elixir 1.13.2 on MacOSX 13.1. I've seen this same behavior with older versions of MacOSX.
Any idea how I can get the ack and/or ripgrep process to terminate so I get a response back? I've seen this https://hexdocs.pm/elixir/main/Port.html#module-zombie-operating-system-processes and I can try it but I was hoping for something slightly less hacky, I guess. Any suggestions? Also if I use the :stderr_to_stdout option set to true, it doesn't seem to make any difference.
I've seen this Q & A but I'm not totally clear on how using Task.start_link would help in this case. I mean would one do a Task.start_link on System.cmd?
You are executing a command that expects input on STDIN, but with System.cmd/3, there is no mechanism to provide the input.
Elixir has no way to know the behaviour of the command you are executing, so waits for the process to terminate, which never happens. As José mentioned on the issue Roger Lipscombe raised, this is expected behaviour.
If you want to send the OS process input via STDIN, you need to use Ports. However, there are limitations there too, which I asked about here.
For ack specifically, it reads from STDIN if you don't provide a filename. So you can workaround the limitation by putting the data in a file, and providing the filename as an argument, rather than piping the data via OS streams.
Looks like a bug. I've filed https://github.com/elixir-lang/elixir/issues/12321.
There's a long-running Unix process which output I'd wish to capture and process with Clojure. A good example of one such process is a repl-y / nREPL session: its duration is indefinite, and output gets printed to stdout.
If I try (clojure.java.io/sh "lein" "repl"), evaluation will block until the underlying process finishes, and then I can observe the output.
This is not what I want - I want to get a stream immediately instead.
Can I achieve this using clojure.java.io, or similar, existing Clojure tools? Wouldn't mind resorting to Java otherwise.
Take a look at the me.raynes.conch library, it's a bit more versatile than clojure.java.shell. It's low-level API seems to be what you're looking for.
Not a detailed answer, but the source for Clojure's sh function is pretty short. If you reworked it slightly to remove the .waitFor (or added a higher-order function to consume the partial reads returned by the InputStreamReader as they arrived), you could probably get updated data as it's returned by the process. But be careful of deadlocks in case your subprocess expects input as well (as in your lein repl example).
I have a program that runs in the command line (i.e. $ run program starts up a prompt) that runs mathematical calculations. It has it's own prompt that takes in text input and responds back through standard-out/error (or creates a separate x-window if needed, but this can be disabled). Sometimes I would like to send it small input, and other times I send in a large text file filled with a series of input on each line. This program takes a lot of resources and also has a large startup time, so it would be best to only have one instance of it running at a time. I could keep open the program-prompt and supply the input this way, or I can send the process with an exit command (to leave prompt) which just prints the output. The problem with sending the request with an exit command is that the program must startup each time (slow ...). Furthermore, the output of this program is sometimes cryptic and it would be helpful to filter the output in some way (eg. simplify output, apply ANSI colors, etc).
This all makes me want to put some 2-way IO filter (or is that "pipe"? or "wrapper"?) around the program so that the program can run in the background as single process. I would then communicate with it without having to restart. I would also like to have this all while filtering the output to be more user friendly. I have been looking all over for ideas and I am stumped at how to accomplish this in some simple shell accessible manor.
Some things I have tried were redirecting stdin and stdout to files, but the program hangs (doesn't quit) and only reads the file once making me unable to continue communication. I think this was because the prompt is waiting for some user input after the EOF. I thought that this could be setup as a local server, but I am uncertain how to begin accomplishing that.
I would love to find some simple way to accomplish this. Additionally, if you can think of a way to perform this, do you think there is a way to also allow for attaching or detaching to the prompt by request? Any help and ideas would be greatly appreciated.
You could create two named pipes (man mkfifo) and redirect input and output:
myprog < fifoin > fifoout
Then you could open new terminal windows and do this in one:
cat > fifoin
And this in the other:
cat < fifoout
(Or use tee to save the input/output as well.)
To dump a large input file into the program, use:
cat myfile > fifoin
I would like to talk to a interactive bash process.
Here is an example, so you know what I want to archieve:
Program starts a new bash process.
User types "ls" into my program.
Program sends this command to the bash process.
Program reads all available output of the bash (including the prompt) and displays it back to the user.
GOTO 1
As you can guess, there is much room for nifty manipulations here and there... ;-)
It would be wonderful if this also worked for subprocesses (started by the bash process) and curses-based programs.
I would like to implement this functionality in Ruby, and already have experimented with IO.popen, but strange things happen. You are also welcome to do this in other languages.
Ok, I've found a solution. This work pretty nicely, you can even start vim inside it :-)
require "pty"
system("stty raw -echo")
PTY.spawn("bash -i") do |pin, pout|
Thread.new do
loop do
pout.print STDIN.getc.chr
end
end
loop do
print pin.sysread(512)
STDOUT.flush
end
end
This does the following:
enable character-wise input (limited to UNIXoids, I'm afraid)
create a pseudo-TTY, start a interactive bash session inside
forward each character from STDIN to the bash
print every output back to the user
Have you tried using the Session gem?
http://rubygems.org/gems/session
https://github.com/ahoward/session (Homepage with introduction.)
I don't have any experience with it, but the README seems to describe what you want. It's description says, "session kicks the ass", so it should be fun/productive to play with it in any case.
I've written a Windows application using the native win32 API. My app will launch other processes and capture the output and highlight stderr output in red.
In order to accomplish this I create a separate pipe for stdout and stderr and use them in the STARTUPINFO structure when calling CreateProcess. I then launch a separate thread for each stdout/stderr handle that reads from the pipe and logs the output to a window.
This works fine in most cases. The problem I am having is that if the child process logs to stderr and stdout in quick succession, my app will sometimes display the output in the incorrect order. I'm assuming this is due to using two threads to read from each handle.
Is it possible to capture stdout and stderr in the original order they were written to, while being able to distinguish between the two?
I'm pretty sure it can't be done, short of writing the spawned program to write in packets and add a time-stamp to each. Without that, you can normally plan on buffering happening in the standard library of the child process, so by the time they're even being transmitted through the pipe to the parent, there's a good chance that they're already out of order.
In most implementations of stdout and stderr that I've seen, stdout is buffered and stderr is not. Basically what this means is that you aren't guaranteed they're going to be in order even when running the program on straight command line.
http://en.wikipedia.org/wiki/Stderr#Standard_error_.28stderr.29
The short answer: You cannot ensure that you read the lines in the same order that they appear on cmd.exe because the order they appear on cmd.exe is not guaranteed.
Not really, you would think so but std_out is at the control of the system designers - exactly how and when std_out gets written is subject to system scheduler, which by my testing is subordinated to issues that are not as documented.
I was writing some stuff one day and did some work on one of the devices on the system while I had the code open in the editor and discovered that the system was giving real-time priority to the driver, leaving my carefully-crafted c-code somewhere about one tenth as important as the proprietary code.
Re-inverting that so that you get sequential ordering of the writes is gonna be challenging to say the least.
You can redirect stderr to stdout:
command_name 2>&1
This is possible in C using pipes, as I recall.
UPDATE: Oh, sorry -- missed the part about being able to distinguish between the two. I know TextMate did it somehow using kinda user visible code... Haven't looked for a while, but I'll give it a peek. But after some further thought, could you use something like Open3 in Ruby? You'd have to watch both STDOUT and STDERR at the same time, but really no one should expect a certain ordering of output regarding these two.
UPDATE 2: Example of what I meant in Ruby:
require 'open3'
Open3.popen3('ruby print3.rb') do |stdin, stdout, stderr|
loop do
puts stdout.gets
puts stderr.gets
end
end
...where print3.rb is just:
loop do
$stdout.puts 'hello from stdout'
$stderr.puts 'hello from stderr'
end
Instead of throwing the output straight to puts, you could send a message to an observer which would print it out in your program. Sorry, I don't have Windows on this machine (or any immediately available), but I hope this illustrates the concept.
I'm pretty sure that even if you don't separate them at all, you're still not guaranteed that they'll interchange one another in the correct order.
Since the intent is to annotate the output os an existing program, any possible interleaving of the two streams must be correct. The original developer will have placed appropriate flush() calls to ensure any mandatory ordering is honoured.
As previously explained, record each fragment that is written with a time stamp, and use this to recover the sequence actually seen by the output devices.