I'm writing a program that reads from an input stream, i.e.
erl -run p main -noshell -s erlang halt < input
The problem is that it takes a lot of time to read it (the input stream is huge) using this read function:
read_input(L) ->
case io:get_line("") of
eof ->
lists:reverse(L);
E0 ->
read_input([E0|L])
end.
I have been looking for more efficient alternatives, but I have found nothing. I have tried to read the file using
{ok, Binary} = file:read_file("input")
This is by far much more efficient. The problem is that I have to run this program in a platform where the name is unknown so I'd need some alternative to do so. additionally, I can't select the flags used when running, e.g. flag -noinput cannot be added to the command line.
Whatever help you can give will be welcomed.
You can use open_port/2 to open stdin and read binaries from it. For example:
-module(p).
-export([start/0]).
start() ->
process_flag(trap_exit, true),
P = open_port({fd,0,1}, [in, binary]),
Bin = read(P,<<>>),
io:format("received ~p\n", [Bin]),
halt(0).
read(P, Bin) ->
receive
{P, {data, Data}} ->
read(P, <<Bin/binary, Data/binary>>);
{'EXIT',P,_} ->
Bin
end.
The code has to trap exits so it knows to exit its reading loop when the port closes. This example reads everything into a single binary returned from the read/2 function and then prints it out and exits, but obviously you can perform further operations on the binary in your actual application.
You can run this like this:
erl -noinput -s p < input
Although Steve's solution is fastest known to me solution there can be used file module solution with quite good performance:
-module(p).
-export([start/0]).
-define(BLK_SIZE, 16384).
start() ->
do(),
halt().
do() ->
Bin = read(),
io:format("~p~n", [byte_size(Bin)]).
read() ->
ok = io:setopts(standard_io, [binary]),
read(<<>>).
read(Acc) ->
case file:read(standard_io, ?BLK_SIZE) of
{ok, Data} ->
read(<<Acc/bytes, Data/bytes>>);
eof ->
Acc
end.
It works with invocation like:
erl -noshell -s p < input
Note both approaches could be used for line-oriented input using {line, Max_Line_Size} option for port or file:read_line/1 for file module solution. Since version 17 (if I recall correctly) there is fixed performance bug in file:read_line/1 I found so it is good now. Anyway, you should not expect performance and comfort of Perl.
Related
What I'm trying to achieve:
From a Ruby process, spawning a subprocess
The subprocess should print as normal back to the terminal. By "normal", I mean the process shouldn't miss out color output, or ignore user input (STDIN).
For that subprocess, capturing STDOUT/STDERR (jointly) e.g. into a String variable that can be accessed after the subprocess is dead. Escape characters and all.
Capturing STDOUT/STDERR is possible by passing a different IO pipe, however the subprocess can then detect that it's not in a tty. For example git log will not print characters that influence text color, nor use it's pager.
Using a pty to launch the process essentially "tricks" the subprocess into thinking it's being launched by a user. As far as I can tell, this is exactly what I want, and the result of this essentially ticks all the boxes.
My general tests to test if a solution fits my needs is:
Does it run ls -al normally?
Does it run vim normally?
Does it run irb normally?
The following Ruby code is able to check all the above:
to_execute = "vim"
output = ""
require 'pty'
require 'io/console'
master, slave = PTY.open
slave.raw!
pid = ::Process.spawn(to_execute, :in => STDIN, [:out, :err] => slave)
slave.close
master.winsize = $stdout.winsize
Signal.trap(:WINCH) { master.winsize = $stdout.winsize }
Signal.trap(:SIGINT) { ::Process.kill("INT", pid) }
master.each_char do |char|
STDOUT.print char
output.concat(char)
end
::Process.wait(pid)
master.close
This works for the most part but it turns out it's not perfect. For some reason, certain applications seem to fail to switch into a raw state. Even though vim works perfectly fine, it turned out neovim did not. At first I thought it was a bug in neovim but I have since been able to reproduce the problem using the Termion crate for the Rust language.
By setting to raw manually (IO.console.raw!) before executing, applications like neovim behave as expected, but then applications like irb do not.
Oddly spawning another pty in Python, within this pty, allows the application to work as expected (using python -c 'import pty; pty.spawn("/usr/local/bin/nvim")'). This obviously isn't a real solution, but interesting nonetheless.
For my actual question I guess I'm looking towards any help to resolving the weird raw issue or, say if I've completely misunderstood tty/pty, any different direction to where/how I should look at the problem.
[edited: see the bottom for the amended update]
Figured it out :)
To really understand the problem I read up a lot on how a PTY works. I don't think I really understood it properly until I drew it out. Basically PTY could be used for a Terminal emulator, and that was the simplest way to think of the data flow for it:
keyboard -> OS -> terminal -> master pty -> termios -> slave pty -> shell
|
v
monitor <- OS <- terminal <- master pty <- termios
(note: this might not be 100% correct, I'm definitely no expert on the subject, just posting it incase it helps anybody else understand it)
So the important bit in the diagram that I hadn't really realised was that when you type, the only reason you see your input on screen is because it's passed back (left-wards) to the master.
So first thing's first - this ruby script should first set the tty to raw (IO.console.raw!), it can restore it after execution is finished (IO.console.cooked!). This'll make sure the keyboard inputs aren't printed by this parent Ruby script.
Second thing is the slave itself should not be raw, so the slave.raw! call is removed. To explain this, I originally added this because it removes extra return carriages from the output: running echo hello results in "hello\r\n". What I missed was that this return carriage is a key instruction to the terminal emulator (whoops).
Third thing, the process should only be talking to the slave. Passing STDIN felt convenient, but it upsets the flow shown in the diagram.
This brings up a new problem on how to pass user input through, so I tried this. So we basically pass STDIN to the master:
input_thread = Thread.new do
STDIN.each_char do |char|
master.putc(char) rescue nil
end
end
that kind of worked, but it has its own issues in terms of some interactive processes weren't receiving a key some of the time. Time will tell, but using IO.copy_stream instead appears to solve that issue (and reads much nicer of course).
input_thread = Thread.new { IO.copy_stream(STDIN, master) }
update 21st Aug:
So the above example mostly worked, but for some reason keys like CTRL+c still wouldn't behave correctly. I even looked up other people's approach to see what I could be doing wrong, and effectively it seemed the same approach - as IO.copy_stream(STDIN, master) was successfully sending 3 to the master. None of the following seemed to help at all:
master.putc 3
master.putc "\x03"
master.putc "\003"
Before I went and delved into trying to achieve this in a lower level language I tried out 1 more thing - the block syntax. Apparently the block syntax magically fixes this problem.
To prevent this answer getting a bit too verbose, the following appears to work:
require 'pty'
require 'io/console'
def run
output = ""
IO.console.raw!
input_thread = nil
PTY.spawn('bash') do |read, write, pid|
Signal.trap(:WINCH) { write.winsize = STDOUT.winsize }
input_thread = Thread.new { IO.copy_stream(STDIN, write) }
read.each_char do |char|
STDOUT.print char
output.concat(char)
end
Process.wait(pid)
end
input_thread.kill if input_thread
IO.console.cooked!
end
Bundler.send(:with_env, Bundler.clean_env) do
run
end
I have a Fortran program that starts with opening and reading data from a .txt file.
At the end of the program a new file is written, which replaces the old file (that was originally imported).
However it can occur that the file that needs to be opened does not exists, for that case the variables that should be imported from the .txt file should be 0.
I thought by doing this with the code below, however this does not work and the script is aborted when the file history.txt does not exists.
How can I let the script set default values to my variables when the history.txt file does not exists?
OPEN(UNIT=in_his,FILE="C:\temp\history.txt",ACTION="read")
if (stat .ne. 0) then !In case history.txt cannot be opened (iteration 1)
write(*,*) "history.txt cannot be opened"
KAPPAI=0
KAPPASH=0
go to 99
end if
read (in_his, *) a, b
KAPPAI=a
KAPPASH=b
write (*, *) "KAPPAI=", a, "KAPPASH=", b
99 close(in_his)
The file that is imported is pretty simple and looks like:
9.900000000000006E-003 3.960000000000003E-003
I would use IOSTAT as stated by #Fortranner. I would also set defaults before trying to open the file and I tend not to use goto's. As in:
program test
implicit none
integer :: in_his, stat
real :: KAPPAI, KAPPASH
in_his = 7
KAPPAI = 0
KAPPASH = 0
OPEN(UNIT=in_his, FILE="history.txt",ACTION='read',IOSTAT=stat,STATUS='OLD')
if (stat .ne. 0) then
write(*,*) "history.txt cannot be opened"
stop 1
end if
read (in_his, *) KAPPAI, KAPPASH
close(in_his)
write (*, *) "KAPPAI=", KAPPAI, "KAPPASH=", KAPPASH
end program test
Another way is to use an inquire statement and check for the existence of the file before you try to open it. This would set a logical variable that could be used in an IF statement to handle the two cases: 1) open file and read values, or 2) set default values w/o opening the file. Or set the default values first, then have the IF statement only handle the case of opening the file and reading the values.
Set iostat in the open statement and handle the case where it is nonzero.
There are two ways to do this. One is using IOSTAT specifier in the OPEN statement like Fortranner and Timothy Brown suggested. The other is to use the ERR specifier in the OPEN statement which lets you specify a label to which the program will transfer control in the even of an error:
OPEN(UNIT=in_his,FILE="C:\temp\history.txt",ACTION="read",STATUS='OLD',ERR=101)
...
101 CONTINUE
The label must be in the same scoping unit as the OPEN statement.
Good day!
It's a very simple question i think but i can't figure out how to handle it. So i ask you for advice or direction.
I make a system call to unix command and during its execution it asks me to input a string description. How can i do that?
Thank you!
Seems like my problem is solved half a way. To make it absolutely clear can anybody tell me why this code:
#/usr/local/bin/ruby19
#Process.daemon(true)
exec "/bin/cp src dst"
works fine but if # from the Process.daemon(true) is removed it does nothing?
You can use IO.pipe and spawn (in Ruby 1.9.3) to create a pipe to the other process for writing to it. For example,
r, w = IO.pipe
spawn("cat", :in => r)
r.close
# the write to the pipe, which `cat` will read from
w.write("hello\n")
This is a bit long, so bear with me!
I'm having a bit of trouble working with a Haskell program, that I have to use as part of a uni project. For reference, it's Casper.
So, you're supposed to execute a script, which is actually a Bash script to invoke Hugs interpreter like this:
exec $HUGSBIN/hugs $HUGSARGS +p"Casper> " $FILES
Where $FILES points to a Main.lhs file.
After this, I need to invoke a function "compile" with a path to a file, in the interpreter.
I need to perform the above in a scripted manner. I need this automated because I'm writing a program that will call on Casper in the background.
So I compiled the .lhs file. Now I want to execute the "compile" function but I have no idea how this is done. I try:
./Main compile <a path>
from the command line but it returns me an error about a file "test" not found. Upon investigation, I see these lines in the Main.lhs file:
>main :: String -> IO()
>main = compile "test"
>compile :: String -> IO()
>compile s = catch (compile0 False s) handler
[...snipped]
The 2nd line solves this question. Now my question is, how do I invoke the "compile" function and pass a path to it after I have compiled main.lhs? From the interpreter, I just type "compile " and it works, but I can't get the same to work after compiling the main.lhs and executing from the command line? Any ideas why? Is there any way I can script Hugs if all else fails?
Thank you for any assistance!
You may access the command-line arguments passed to a Haskell program via getArgs. For example, it sounds like you want a main function that does something like this:
>main = do
> args <- getArgs
> case args of
> [] -> putStrLn "What file did you want me to compile?"
> [filename] -> compile filename
> _ -> putStrLn "I only compile one file at a time."
Modify to taste.
Replace main with
main = getArgs >>= \(arg1:_) -> compile arg1
This will pass the first command line argument (arg1) to compile instead of "test", and ignore the rest (_). You may need to add
import System
or
import System.Environment
I can't remember what is needed in hugs for this.
How do I create a background process with Haskell on windows without a visible command window being created?
I wrote a Haskell program that runs backup processes periodically but every time I run it, a command window opens up to the top of all the windows. I would like to get rid of this window. What is the simplest way to do this?
You should really tell us how you are trying to do this currently, but on my system (using linux) the following snippet will run a command without opening a new terminal window. It should work the same way on windows.
module Main where
import System
import System.Process
import Control.Monad
main :: IO ()
main = do
putStrLn "Running command..."
pid <- runCommand "mplayer song.mp3" -- or whatever you want
replicateM_ 10 $ putStrLn "Doing other stuff"
waitForProcess pid >>= exitWith
Thanks for the responses so far, but I've found my own solution. I did try a lot of different things, from writing a vbs script as suggested to a standalone program called hstart. hstart worked...but it creates a separate process which I didn't like very much because then I can't kill it in the normal way. But I found a simpler solution that required simply Haskell code.
My code from before was a simple call to runCommand, which did popup the window. An alternative function you can use is runProcess which has more options. From peeking at the ghc source code file runProcess.c, I found that the CREATE_NO_WINDOW flag is set when you supply redirects for all of STDIN, STOUT, and STDERR. So that's what you need to do, supply redirects for those. My test program looks like:
import System.Process
import System.IO
main = do
inH <- openFile "in" ReadMode
outH <- openFile "out" WriteMode
runProcess "rsync.bat" [] Nothing Nothing (Just inH) (Just outH) (Just outH)
This worked! No command window again! A caveat is that you need an empty file for inH to read in as the STDIN eventhough in my situation it was not needed.
The simplest way I can think of is to run the rsync command from within a Windows Shell script (vbs or cmd).
I don't know anything about Haskell, but I had this problem in a C project a few months ago.
The best way to execute an external program without any windows popping up is to use the ShellExecuteEx() API function with the "open" verb. If ShellExecuteEx() is available to you in Haskell, then you should be able to achieve what you want.
The C code looks something like this:
SHELLEXECUTEINFO Info;
BOOL b;
// Execute it
memset (&Info, 0, sizeof (Info));
Info.cbSize = sizeof (Info);
Info.fMask = SEE_MASK_NOCLOSEPROCESS | SEE_MASK_FLAG_NO_UI;
Info.hwnd = NULL;
Info.lpVerb = "open";
Info.lpFile = "rsync.exe";
Info.lpParameters = "whatever parameters you like";
Info.lpDirectory = NULL;
Info.nShow = SW_HIDE;
b = ShellExecuteEx (&Info);
if (b)
{
// Looks good; if there is an instance, wait for it
if (Info.hProcess)
{
// Wait
WaitForSingleObject (Info.hProcess, INFINITE);
}
}