I'm using pexpect to automate ssh-ing into a remote server that prompts for a password. The process is very simple, and works great:
child = pexpect.spawn("ssh -Y remote.server")
child.expect(re.compile(b".*password.*"))
child.sendline(password)
child.interact()
This works great, however, I notice one quite annoying quirk I have not been able to figure out. When I use vim in this terminal, it seems to not resize correctly. When ssh-ing directly, and using a program such as vim, I can resize my terminal window (locally), and the remote program automatically/interactively fixes the columns and lines. My pexpect instance does not. There are a few other minor quirks that I can lvie with, but this one is quite annoying.
I'm hoping to find a way I can get my pexpect ssh session to behave the same way a native ssh session does, or at the very least understand the reason the two behave differently.
The SIGWINCH handles window size change. If you need it behave the same of native ssh, you should also set the pexpect initial window size:
import pexpect, struct, fcntl, termios, signal, sys
def get_terminal_size():
s = struct.pack("HHHH", 0, 0, 0, 0)
a = struct.unpack('hhhh', fcntl.ioctl(sys.stdout.fileno(), termios.TIOCGWINSZ, s))
return a[0], a[1]
def sigwinch_passthrough(sig, data):
global p
if not p.closed:
p.setwinsize(*get_terminal_size())
p = pexpect.spawn('/bin/bash')
# Set the window size the same of current terminal window size
p.setwinsize(*get_terminal_size())
# Hook the window change signal so that the pexpect window size change as well
signal.signal(signal.SIGWINCH, sigwinch_passthrough)
p.interact()
There's actually an example for this under the interact() function in pexpect's doc. Just like writing C code, it requires a SIGWINCH handler.
import pexpect, struct, fcntl, termios, signal, sys
def sigwinch_passthrough (sig, data):
s = struct.pack("HHHH", 0, 0, 0, 0)
a = struct.unpack('hhhh', fcntl.ioctl(sys.stdout.fileno(),
termios.TIOCGWINSZ , s) )
global p
p.setwinsize(a[0], a[1])
# Note this 'p' global and used in sigwinch_passthrough.
p = pexpect.spawn('/bin/bash')
signal.signal(signal.SIGWINCH, sigwinch_passthrough)
p.interact()
Related
I have a scripts which generates huge amount of data and dump it to stdout. Using less it is easy to scroll through it:
./myscript | less
If myscript is running and providing data continuously, the latest less can update the view automatically (if the stdout buffer is flushed). This simple Python script shows this case:
import time
import os
import sys
for i in range(1000):
stream = sys.stdout.buffer
msg = str(i) + "\n"
stream.write(msg.encode())
stream.flush()
if (i + 0) % 100 == 0:
time.sleep(5)
python3 pipe.py | less
What I want to achieve is that the script only provides the next page of data if the user reached the end of the scrolling in less (instead of automatic update which the script above does).
Is it possible to do this? (Note the Python script is just an example, the question is much more about the shell part.)
What I'm trying to achieve:
From a Ruby process, spawning a subprocess
The subprocess should print as normal back to the terminal. By "normal", I mean the process shouldn't miss out color output, or ignore user input (STDIN).
For that subprocess, capturing STDOUT/STDERR (jointly) e.g. into a String variable that can be accessed after the subprocess is dead. Escape characters and all.
Capturing STDOUT/STDERR is possible by passing a different IO pipe, however the subprocess can then detect that it's not in a tty. For example git log will not print characters that influence text color, nor use it's pager.
Using a pty to launch the process essentially "tricks" the subprocess into thinking it's being launched by a user. As far as I can tell, this is exactly what I want, and the result of this essentially ticks all the boxes.
My general tests to test if a solution fits my needs is:
Does it run ls -al normally?
Does it run vim normally?
Does it run irb normally?
The following Ruby code is able to check all the above:
to_execute = "vim"
output = ""
require 'pty'
require 'io/console'
master, slave = PTY.open
slave.raw!
pid = ::Process.spawn(to_execute, :in => STDIN, [:out, :err] => slave)
slave.close
master.winsize = $stdout.winsize
Signal.trap(:WINCH) { master.winsize = $stdout.winsize }
Signal.trap(:SIGINT) { ::Process.kill("INT", pid) }
master.each_char do |char|
STDOUT.print char
output.concat(char)
end
::Process.wait(pid)
master.close
This works for the most part but it turns out it's not perfect. For some reason, certain applications seem to fail to switch into a raw state. Even though vim works perfectly fine, it turned out neovim did not. At first I thought it was a bug in neovim but I have since been able to reproduce the problem using the Termion crate for the Rust language.
By setting to raw manually (IO.console.raw!) before executing, applications like neovim behave as expected, but then applications like irb do not.
Oddly spawning another pty in Python, within this pty, allows the application to work as expected (using python -c 'import pty; pty.spawn("/usr/local/bin/nvim")'). This obviously isn't a real solution, but interesting nonetheless.
For my actual question I guess I'm looking towards any help to resolving the weird raw issue or, say if I've completely misunderstood tty/pty, any different direction to where/how I should look at the problem.
[edited: see the bottom for the amended update]
Figured it out :)
To really understand the problem I read up a lot on how a PTY works. I don't think I really understood it properly until I drew it out. Basically PTY could be used for a Terminal emulator, and that was the simplest way to think of the data flow for it:
keyboard -> OS -> terminal -> master pty -> termios -> slave pty -> shell
|
v
monitor <- OS <- terminal <- master pty <- termios
(note: this might not be 100% correct, I'm definitely no expert on the subject, just posting it incase it helps anybody else understand it)
So the important bit in the diagram that I hadn't really realised was that when you type, the only reason you see your input on screen is because it's passed back (left-wards) to the master.
So first thing's first - this ruby script should first set the tty to raw (IO.console.raw!), it can restore it after execution is finished (IO.console.cooked!). This'll make sure the keyboard inputs aren't printed by this parent Ruby script.
Second thing is the slave itself should not be raw, so the slave.raw! call is removed. To explain this, I originally added this because it removes extra return carriages from the output: running echo hello results in "hello\r\n". What I missed was that this return carriage is a key instruction to the terminal emulator (whoops).
Third thing, the process should only be talking to the slave. Passing STDIN felt convenient, but it upsets the flow shown in the diagram.
This brings up a new problem on how to pass user input through, so I tried this. So we basically pass STDIN to the master:
input_thread = Thread.new do
STDIN.each_char do |char|
master.putc(char) rescue nil
end
end
that kind of worked, but it has its own issues in terms of some interactive processes weren't receiving a key some of the time. Time will tell, but using IO.copy_stream instead appears to solve that issue (and reads much nicer of course).
input_thread = Thread.new { IO.copy_stream(STDIN, master) }
update 21st Aug:
So the above example mostly worked, but for some reason keys like CTRL+c still wouldn't behave correctly. I even looked up other people's approach to see what I could be doing wrong, and effectively it seemed the same approach - as IO.copy_stream(STDIN, master) was successfully sending 3 to the master. None of the following seemed to help at all:
master.putc 3
master.putc "\x03"
master.putc "\003"
Before I went and delved into trying to achieve this in a lower level language I tried out 1 more thing - the block syntax. Apparently the block syntax magically fixes this problem.
To prevent this answer getting a bit too verbose, the following appears to work:
require 'pty'
require 'io/console'
def run
output = ""
IO.console.raw!
input_thread = nil
PTY.spawn('bash') do |read, write, pid|
Signal.trap(:WINCH) { write.winsize = STDOUT.winsize }
input_thread = Thread.new { IO.copy_stream(STDIN, write) }
read.each_char do |char|
STDOUT.print char
output.concat(char)
end
Process.wait(pid)
end
input_thread.kill if input_thread
IO.console.cooked!
end
Bundler.send(:with_env, Bundler.clean_env) do
run
end
import multiprocessing as mul
def f(x):
return x**2
pool = mul.Pool(5)
rel = pool.map(f,[1,2,3,4,5,6,7,8,9,10])
print(rel)
When I run the program above, the application is stuck in a loop and can't stop.
I am using python 3.5 in windows, is there something wrong?
This is what I see on my screen:
I am new to finance data analysis; and I am trying to find out a way to solve the big data problem with parallel computing.
Its not working because you are typing the commands in a shell; try saving the code in a file and running it directly.
Don't forget to copy the code correctly, you were missing a very important if statement (see the documentation).
Save this to a file, for example example.py on the desktop:
import multiprocessing as mul
def f(x):
return x**2
if __name__ == '__main__':
pool = mul.Pool(5)
rel = pool.map(f,[1,2,3,4,5,6,7,8,9,10])
print(rel)
Then, open a command prompt and type:
python %USERPROFILE%\Desktop\example.py
All code written and tested on python 3.4 windows 7.
I was designing a console app and had a need to use stdin from command-line (win os) to issue commands and to change the operating mode of the program. The program depends on multiprocessing to deal with cpu bound loads to spread to multiple processors.
I am using stdout to monitor that status and some basic return information and stdin to issue commands to load different sub-processes based on the returned console information.
This is where I found a problem. I could no get the multiprocessing module to accept stdin inputs but stdout was working just fine. I think found the following help on stack So I tested it and found that with the threading module this all works great, except for the fact that all output to stdout is paused until each time stdin is cycled due to GIL lock with stdin blocking.
I will say I have been successful with a work around implemented with msvcrt.kbhit(). However, I can't help but wonder if there is some sort of bug in the multiprocessing feature that is making stdin not read any data. I tried numerous ways and nothing worked when using multiprocessing. Even attempted to use Queues, but I did not try pools, or any other methods from multiprocessing.
I also did not try this on my linux machine since I was focusing on trying to get it to work.
Here is simplified test code that does not function as intended (reminder this was written in Python 3.4 - win7):
import sys
import time
from multiprocessing import Process
def function1():
while True:
print("Function 1")
time.sleep(1.33)
def function2():
while True:
print("Function 2")
c = sys.stdin.read(1) # Does not appear to be waiting for read before continuing loop.
sys.stdout.write(c) #nothing in 'c'
sys.stdout.write(".") #checking to see if it works at all.
print(str(c)) #trying something else, still nothing in 'c'
time.sleep(1.66)
if __name__ == "__main__":
p1 = Process(target=function1)
p2 = Process(target=function2)
p1.start()
p2.start()
Hopefully someone can shed light on whether this is intended functionality, if I didn't implement it correctly, or some other useful bit of information.
Thanks.
When you take a look at Pythons implementation of multiprocessing.Process._bootstrap() you will see this:
if sys.stdin is not None:
try:
sys.stdin.close()
sys.stdin = open(os.devnull)
except (OSError, ValueError):
pass
You can also confirm this by using:
>>> import sys
>>> import multiprocessing
>>> def func():
... print(sys.stdin)
...
>>> p = multiprocessing.Process(target=func)
>>> p.start()
>>> <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>
And reading from os.devnull immediately returns empty result:
>>> import os
>>> f = open(os.devnull)
>>> f.read(1)
''
You can work this around by using open(0):
file is either a string or bytes object giving the pathname (absolute or relative to the current working directory) of the file to be opened or an integer file descriptor of the file to be wrapped. (If a file descriptor is given, it is closed when the returned I/O object is closed, unless closefd is set to False.)
And "0 file descriptor":
File descriptors are small integers corresponding to a file that has been opened by the current process. For example, standard input is usually file descriptor 0, standard output is 1, and standard error is 2:
>>> def func():
... sys.stdin = open(0)
... print(sys.stdin)
... c = sys.stdin.read(1)
... print('Got', c)
...
>>> multiprocessing.Process(target=func).start()
>>> <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>
Got a
How do I create a background process with Haskell on windows without a visible command window being created?
I wrote a Haskell program that runs backup processes periodically but every time I run it, a command window opens up to the top of all the windows. I would like to get rid of this window. What is the simplest way to do this?
You should really tell us how you are trying to do this currently, but on my system (using linux) the following snippet will run a command without opening a new terminal window. It should work the same way on windows.
module Main where
import System
import System.Process
import Control.Monad
main :: IO ()
main = do
putStrLn "Running command..."
pid <- runCommand "mplayer song.mp3" -- or whatever you want
replicateM_ 10 $ putStrLn "Doing other stuff"
waitForProcess pid >>= exitWith
Thanks for the responses so far, but I've found my own solution. I did try a lot of different things, from writing a vbs script as suggested to a standalone program called hstart. hstart worked...but it creates a separate process which I didn't like very much because then I can't kill it in the normal way. But I found a simpler solution that required simply Haskell code.
My code from before was a simple call to runCommand, which did popup the window. An alternative function you can use is runProcess which has more options. From peeking at the ghc source code file runProcess.c, I found that the CREATE_NO_WINDOW flag is set when you supply redirects for all of STDIN, STOUT, and STDERR. So that's what you need to do, supply redirects for those. My test program looks like:
import System.Process
import System.IO
main = do
inH <- openFile "in" ReadMode
outH <- openFile "out" WriteMode
runProcess "rsync.bat" [] Nothing Nothing (Just inH) (Just outH) (Just outH)
This worked! No command window again! A caveat is that you need an empty file for inH to read in as the STDIN eventhough in my situation it was not needed.
The simplest way I can think of is to run the rsync command from within a Windows Shell script (vbs or cmd).
I don't know anything about Haskell, but I had this problem in a C project a few months ago.
The best way to execute an external program without any windows popping up is to use the ShellExecuteEx() API function with the "open" verb. If ShellExecuteEx() is available to you in Haskell, then you should be able to achieve what you want.
The C code looks something like this:
SHELLEXECUTEINFO Info;
BOOL b;
// Execute it
memset (&Info, 0, sizeof (Info));
Info.cbSize = sizeof (Info);
Info.fMask = SEE_MASK_NOCLOSEPROCESS | SEE_MASK_FLAG_NO_UI;
Info.hwnd = NULL;
Info.lpVerb = "open";
Info.lpFile = "rsync.exe";
Info.lpParameters = "whatever parameters you like";
Info.lpDirectory = NULL;
Info.nShow = SW_HIDE;
b = ShellExecuteEx (&Info);
if (b)
{
// Looks good; if there is an instance, wait for it
if (Info.hProcess)
{
// Wait
WaitForSingleObject (Info.hProcess, INFINITE);
}
}