Using expect() and interact() simultaneously in pexpect - python-2.x

The general problem is, that I want to use pexpect to call scripts that require sudo rights, but I don't always want to enter my password (only once).
My plan is to use pexpect to spawn a bash session with sudo rights and to call scripts from there. Basically I always want to keep the session busy, whenever one script stopped, I want to start another. But while the scripts are running, I want the user to have control. Meaning:
The scripts should be called after something like expect("root#"), so whenever the session is idle, it starts another script. While the scripts are running interact() is giving the user control of possible input he wants to give.
My idea was to use different threads to to solve this problem. My code (for the proof of concept) looks like this:
import pexpect
import threading
class BashInteractThread(threading.Thread):
def __init__(self, process):
threading.Thread.__init__(self)
self.pproc = process
def run(self):
self.pproc.interact()
s = pexpect.spawn("/bin/bash", ['-i', '-c', "sudo bash"])
it = BashInteractThread(s)
it.start()
s.expect("root#")
s.sendline("cd ~")
while(s.isalive()):
pass
s.close()
When I call this script, it does not give me any output, but the process seems to have been started. Still, I cannot CTRL-C or CTRL-D to kill the process - I have to kill the process separately. The behavior I would expect, would be to get prompted to enter a password and after that it should automatically change the directory to the home directory.
I don't exactly know why it does not work, but I guess the output only gets forwarded either to interact() or to expect().
Does anyone have an idea on how to solve this? Thanks in advance.

You can take advantage of interact(output_filter=func). I just wrote a simple example (no coding style!). What it does is spawn a Bash shell and repeatedly invoke Python for the user to interact with. To exit the trap, just input (or print) the magic words LET ME OUT.
expect() would not work anymore after interact(), so need to do the pattern matching work manually.
The code:
[STEP 101] # cat interact_with_filter.py
import pexpect, re
def output_filter(s):
global proc, bash_prompt, filter_buf, filter_buf_size, let_me_out
filter_buf += s
filter_buf = filter_buf[-filter_buf_size:]
if "LET ME OUT" in filter_buf:
let_me_out = True
if bash_prompt.search(filter_buf):
if let_me_out:
proc.sendline('exit')
proc.expect(pexpect.EOF)
proc.wait()
else:
proc.sendline('python')
return s
filter_buf = ''
filter_buf_size = 256
let_me_out = False
bash_prompt = re.compile('bash-[.0-9]+[$#] $')
proc = pexpect.spawn('bash --noprofile --norc')
proc.interact(output_filter=output_filter)
print "BYE"
[STEP 102] #
Let's try it:
[STEP 102] # python interact_with_filter.py
bash-4.4# python
Python 2.7.9 (default, Jun 29 2016, 13:08:31)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit() <---- user input
bash-4.4# python
Python 2.7.9 (default, Jun 29 2016, 13:08:31)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit() <---- user input
bash-4.4# python
Python 2.7.9 (default, Jun 29 2016, 13:08:31)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> LET ME OUT <---- user input
File "<stdin>", line 1
LET ME OUT
^
SyntaxError: invalid syntax
>>> exit() <---- user input
bash-4.4# BYE
[STEP 103] #

Related

Why does the value returned by python script gets modified by the shell?

My pythonscript looks like this
#!/usr/bin/python3
#File Name: pythonScript.py
from sys import exit
if '__name__'=='__main__':exit(402)
And this is the shell script
python3 pythonScript.py
echo $?
It prints 146. How does 402 get mapped to 146? Some other pairs such as this are (402, 146), (100,0), (0, 0), (56, 0) etc.
Can the python script return value to shell this way, and is the ? the correct variable to capture this?
My machine version is this, if this is important.
4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
in bash, the exit code is visible only the lowermost 8 bits, low 8 bits of 402 is 146:
>>> 402 & 0b11111111
146
It's not the shell. The system call interface only accommodates an unsigned 8-bit number; so the maximum value which can be communicated is 255.
The same thing would happen if you wrote a C program whose int main() tried to return(402); and you called it from another C program with exec, without any shell.

Writing to Windows Event Log using win32evtlog from pywin32 library

I have a simple python script that will be running on a windows server, I'd like to log specific events throughout the script to the windows event log. Does anyone have a simple and precise example of writing to the windows event log so I can view the event from the event viewer. I've read through the docs for the pywin32 library and I can't find any clear examples. I've tried building an event using:
win32evtlogutil.ReportEvent(ApplicationName, EventID, EventCategory,
EventType, Inserts, Data, SID)
I've had no success, could someone explain the ReportEvent a bit more in depth?
A simple example:
>>> import sys
>>> import win32evtlogutil
>>> import win32evtlog
>>> import time
>>>
>>>
>>> "Python {0:s} on {1:s}".format(sys.version, sys.platform)
'Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32'
>>>
>>> DUMMY_EVT_APP_NAME = "Dummy Application"
>>> DUMMY_EVT_ID = 7040 # Got this from another event
>>> DUMMY_EVT_CATEG = 9876
>>> DUMMY_EVT_STRS = ["Dummy event string {0:d}".format(item) for item in range(5)]
>>> DUMMY_EVT_DATA = b"Dummy event data"
>>>
>>> "Current time: {0:s}".format(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))
'Current time: 2018-07-18 20:03:08'
>>>
>>> win32evtlogutil.ReportEvent(
... DUMMY_EVT_APP_NAME, DUMMY_EVT_ID, eventCategory=DUMMY_EVT_CATEG,
... eventType=win32evtlog.EVENTLOG_WARNING_TYPE, strings=DUMMY_EVT_STRS,
... data=DUMMY_EVT_DATA)
>>>
Output:
You can see the correspondence between the values that I input from code, and the event fields in the (above) image of the Event Viewer (mmc) window.
win32evtlogutil.ReportEvent is part of [GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions, which is a Python wrapper over WINAPIs.
Everything you need to know is explained at [MS.Docs]: ReportEventW function, which is the WINAPI used to accomplish this task. Make sure to read it carefully (and some other URLs that it references) in order to get more familiar about the arguments, what their values could be, and other info.
Make sure not to abuse (tests included), or you might end up getting the event log polluted with lots of garbage data.

NumPy bitwise "and" fails for int32 & long, but not for long & int32; why?

Compare this:
>>> import numpy; numpy.int32(-1) & 0xFFFFFFFF00000000
TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs
could not be safely coerced to any supported types according to the casting rule ''safe''
With this:
>>> import numpy; 0xFFFFFFFF00000000 & numpy.int32(-1)
18446744069414584320L
Are both working as intended or is at least one of them a bug? Why does it occur?
The difference is in which object's __and__ or __rand__ method is being called. Normally, the left-hand expression has it's __and__ called first. If it returns NotImplemented, then the right hand expression will get a chance (and __rand__ will be called).
In this case, numpy.int32 has decided that it cannot be "anded" with a long -- At least not with a long whose value is above what can be represented by native types...
However, based on your experiments, python's long is happy to "and" with a numpy.int32 -- Or, possibly your version of numpy did not implement __rand__ symmetrically with __and__. This is possibly also python version dependent (e.g if your version of python decided to return a value rather than NotImplemented).
On my computer, neither work:
Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:43:17)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import numpy
>>> numpy.__version__
'1.11.2'
But we can see what is being called using the following script:
import sys
import numpy
class MyInt32(numpy.int32):
def __and__(self, other):
print('__and__')
return super(MyInt32, self).__and__(other)
def __rand__(self, other):
print('__rand__')
return super(MyInt32, self).__and__(other)
try:
print(MyInt32(-1) & 0xFFFFFFFF00000000) # Outputs `__and__` before the `TypeError`
except TypeError:
pass
try:
print(0xFFFFFFFF00000000 & MyInt32(-1)) # Outputs `__rand__` before the `TypeError`
except TypeError:
pass
sys.maxint & MyInt32(-1) # Outputs `__rand__`
print('great success')
(sys.maxint + 1) & MyInt32(-1) # Outputs `__rand__`
print('do not see this')

How do I read / understand ansible logs on target host (written by syslog)

When you execute ansible on some host, it will write to syslog on that host, something like this:
Dec 1 15:00:22 run-tools python: ansible-<stdin> Invoked with partial=False links=None copy_links=None perms=None owner=False rsync_path=None dest_port=22 _local_rsync_path=rsync group=False existing_only=False archive=True _substitute_controller=False verify_host=False dirs=False private_key=None dest= compress=True rsync_timeout=0 rsync_opts=None set_remote_user=True recursive=None src=/etc/ansible/repo/external/golive/ checksum=False times=None mode=push ssh_args=None delete=False
Dec 1 15:00:22 run-tools python: ansible-<stdin> Invoked with partial=False links=None copy_links=None perms=None owner=False rsync_path=None dest_port=22 _local_rsync_path=rsync group=False existing_only=False archive=True _substitute_controller=False verify_host=False dirs=False private_key=None dest= compress=True rsync_timeout=0 rsync_opts=None set_remote_user=True recursive=None src=/etc/ansible/repo/external/golive/ checksum=False times=None mode=push ssh_args=None delete=False
Dec 1 15:00:22 run-tools python: ansible-<stdin> Invoked with partial=False links=None copy_links=None perms=None owner=False rsync_path=None dest_port=22 _local_rsync_path=rsync group=False existing_only=False archive=True _substitute_controller=False verify_host=False dirs=False private_key=None dest= compress=True rsync_timeout=0 rsync_opts=None set_remote_user=True recursive=None src=/etc/ansible/repo/external/golive/ checksum=False times=None mode=push ssh_args=None delete=False
Dec 1 15:00:56 run-tools python: ansible-<stdin> Invoked with filter=* fact_path=/etc/ansible/facts.d
Dec 1 15:09:56 run-tools python: ansible-<stdin> Invoked with checksum_algorithm=sha1 mime=False get_checksum=True path=/usr/local/bin/check_open_files_generic.sh checksum_algo=sha1 follow=False get_md5=False
Dec 1 15:09:56 run-tools python: ansible-<stdin> Invoked with directory_mode=None force=False remote_src=None path=/usr/local/bin/check_open_files_generic.sh owner=root follow=False group=root state=None content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None dest=/usr/local/bin/check_open_files_generic.sh selevel=None original_basename=check_open_files_generic.sh regexp=None validate=None src=check_open_files_generic.sh seuser=None recurse=False delimiter=None mode=0755 backup=None
Dec 1 15:20:03 run-tools python: ansible-<stdin> Invoked with warn=True executable=None _uses_shell=False _raw_params=visudo -c removes=None creates=None chdir=None
Is there any documentation or explanation of these logs that would help me understand how to read them? Specifically I would like to be able to see what exactly ansible did, which files it touched etc. Is it possible to find it there? Or reconfigure ansible so that it writes this kind of information in there?
Is it possible to configure these logs at all? How?
I am not aware of documentation that explains the contents of syslog messages specifically. However, you can look at some of the logging code in AnsibleModule.log() to see what's going on. Basically, it's reporting module names and the parameters they were called with.
For configuring logs, there are some good suggestions in response to this related question. The summary is that you can get more information - including your request about what ansible did - by specifying a log path and running with the verbose -v flag. For more fine-grained control, you can attack the problem from two different angles:
From the playbook side, you can use the debug module or tailor your handling of changed/failed results to suit your needs. Both of those changes can add useful context to your log output.
Outside of playbooks, you can use Ansible callback plugins to control logging. Here is an example of a callback plugin which intercepts logs and outputs something more human readable.

system(): why do I not have the same permissions when using R in EMACS as I do in the bash terminal?

update: the error only occurs when logged into R from within emacs
what works:
When I ssh into a remote server and run
$ ./foo.rb
from the bash shell, it works. Furthermore, if I launch R and execute
$ R
system('./foo.rb')
I am in a group with permission to read/write/execute the file. File permissions are -rwxrwx---
what doesn't work:
Launch emacs and start an R session:
M-x R
ssh-myserver:.
system('./foo.rb')
I get the following error:
ruby: Permission denied -- foo.rb (LoadError)
why is this? Is there a way to work around this?
I can not find any information from ?system or ?system2
Here is the output from sessionInfo()
> sessionInfo()
R version 2.12.2 (2011-02-25)
Platform: x86_64-redhat-linux-gnu (64-bit)
locale:
[1] C
attached base packages:
[1] grid stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] PECAn_0.1.1 xtable_1.5-6 gridExtra_0.7 RMySQL_0.7-5
[5] DBI_0.2-5 ggplot2_0.8.9 proto_0.3-8 reshape_0.8.3
[9] plyr_1.6 rjags_2.2.0-2 coda_0.13-5 lattice_0.19-17
[13] randtoolbox_1.09 rngWELL_0.9 MASS_7.3-11 XML_3.2-0
loaded via a namespace (and not attached):
[1] digest_0.4.2
Warning message:
'DESCRIPTION' file has 'Encoding' field and re-encoding is not possible
output of 'id' and 'env' from ssh and emacs, per comment by #sarnold (changed user names, group names, and ip addresses)
1. server
1.1 'id'
uid=1668(dleb) gid=1668(dleb) groups=117(ebusers),159(lab_admin),166(lab),1340(pal_web),1668(dleb)
1.2 'env'
LC_PAPER=en_US.UTF-8
LC_ADDRESS=en_US.UTF-8
LC_MONETARY=en_US.UTF-8
SHELL=/usr/local/bin/system-specific
KDE_NO_IPV6=1
SSH_CLIENT=888.888.888.88 51857 22
NCARG_FONTCAPS=/usr/lib64/ncarg/fontcaps
LC_NUMERIC=en_US.UTF-8
USER=dleb
LS_COLORS=
LC_TELEPHONE=en_US.UTF-8
KDEDIR=/usr
NCARG_GRAPHCAPS=/usr/lib64/ncarg/graphcaps
MAIL=/var/mail/dleb
PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin
LC_IDENTIFICATION=en_US.UTF-8
LC_COLLATE=en_US.UTF-8
R_LIBS=/home/a-m/dleb/lib/R
PWD=/home/dleb
NCARG_ROOT=/usr
KDE_IS_PRELINKED=1
LANG=en_US.UTF-8
NCARG_DATABASE=/usr/lib64/ncarg/database
MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles
LOADEDMODULES=
LC_MEASUREMENT=en_US.UTF-8
NCARG_LIB=/usr/lib64/ncarg
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
NCARG_NCARG=/usr/share/ncarg
SHLVL=1
HOME=/home/a-m/dleb
LOGNAME=dleb
CVS_RSH=ssh
SSH_CONNECTION=888.888.888.88 51857 999.999.999.99 22
LC_CTYPE=en_US.UTF-8
MODULESHOME=/usr/share/Modules
LESSOPEN=|/usr/bin/lesspipe.sh %s
DISPLAY=localhost:15.0
LC_TIME=en_US.UTF-8
G_BROKEN_FILENAMES=1
LC_NAME=en_US.UTF-8
_=/bin/env
emacs/ess R session
2.1 system('id')
uid=1668(dleb) gid=1668(dleb) groups=117(ebusers),159(lab_admin),166(lab),1340(pal_web),1668(dleb)
2.2 system('env')
LN_S=ln -s
R_TEXI2DVICMD=/usr/bin/texi2dvi
LC_PAPER=en_US.UTF-8
SED=/bin/sed
LC_ADDRESS=en_US.UTF-8
R_PDFVIEWER=/usr/bin/xdg-open
LC_MONETARY=en_US.UTF-8
HOSTNAME=ebi-forecast
R_INCLUDE_DIR=/usr/include/R
R_PRINTCMD=lpr
SHELL=/usr/local/bin/system-specific
TERM=dumb
AWK=gawk
HISTSIZE=1
R_RD4DVI=ae
SSH_CLIENT=888.888.888.88 51159 22
KDE_NO_IPV6=1
R_RD4PDF=times,hyper
R_PAPERSIZE=a4
NCARG_FONTCAPS=/usr/lib64/ncarg/fontcaps
PERL=/usr/bin/perl
LC_NUMERIC=en_US.UTF-8
SSH_TTY=/dev/pts/14
LC_ALL=C
EMACS=t
USER=dleb
LC_TELEPHONE=en_US.UTF-8
LS_COLORS=
LD_LIBRARY_PATH=/usr/lib64/R/lib:/usr/local/lib64:/usr/lib/jvm/jre/lib/amd64/server:/usr/lib/jvm/jre/lib/amd64:/usr/lib/jvm/java/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
TAR=/bin/gtar
ENV=
R_ZIPCMD=/usr/bin/zip
KDEDIR=/usr
PAGER=/usr/bin/less
NCARG_GRAPHCAPS=/usr/lib64/ncarg/graphcaps
R_GZIPCMD=/usr/bin/gzip
PATH=/bin:/usr/bin:/usr/sbin:/usr/local/bin
LC_COLLATE=en_US.UTF-8
LC_IDENTIFICATION=en_US.UTF-8
EGREP=/bin/grep -E
PWD=/home/a-m/dleb/pecan
INPUTRC=/etc/inputrc
R_LIBS=/home/a-m/dleb/lib/R
NCARG_ROOT=/usr
R_SHARE_DIR=/usr/share/R
WHICH=/usr/bin/which
EDITOR=vi
LANG=en_US.UTF-8
KDE_IS_PRELINKED=1
R_LIBS_SITE=/usr/local/lib/R/site-library:/usr/local/lib/R/library:/usr/lib64/R/library:/usr/share/R/library
M ODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles
NCARG_DATABASE=/usr/lib64/ncarg/database
LC_MEASUREMENT=en_US.UTF-8
LOADEDMODULES=
PS3=
R_BROWSER=/usr/bin/xdg-open
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
NCARG_LIB=/usr/lib64/ncarg
HOME=/home/a-m/dleb
SHLVL=1
NCARG_NCARG=/usr/share/ncarg
R_ARCH=
TR=/usr/bin/tr
MAKE=make
R_UNZIPCMD=/usr/bin/unzip
LOGNAME=dleb
CVS_RSH=ssh
LC_CTYPE=en_US.UTF-8
SSH_CONNECTION=888.888.888.88 51159 999.999.999.99 22
R_BZIPCMD=/usr/bin/bzip2
MODULESHOME=/usr/share/Modules
LESSOPEN=|/usr/bin/lesspipe.sh %s
PROMPT_COMMAND=
R_HOME=/usr/lib64/R
DISPLAY=localhost:22.0
R_PLATFORM=x86_64-redhat-linux-gnu
INSIDE_EMACS=23.2.1,tramp:2.1.18-23.2
R_LIBS_USER=~/R/x86_64-redhat-linux-gnu-library/2.12
LC_TIME=en_US.UTF-8
R_DOC_DIR=/usr/share/doc/R-2.12.2
R_SESSION_TMPDIR=/tmp/RtmpqA6bpJ
HISTFILE=/home/a-m/dleb/.tramp_history
G_BROKEN_FILENAMES=1
LC_NAME=en_US.UTF-8
_=/bin/env
Assuming you started up R as the same user, you do. You error is not coming from a permissions problem for foo.rb, however, or else your shell would be giving the error. (i.e. sh: ./test.rb: Permission denied; see example below). Here, ruby itself is giving the error. Without knowing exactly what is in your foo.rb, I would suggest digging in there to see what it is trying to load/source, and checking the permissions on those.
#!/usr/bin/env ruby
puts 'Hello world'
Now in R....
> system('ls -l test.rb')
-rw-r--r-- 1 jcolby staff 40 Oct 21 08:23 test.rb
> system('./test.rb')
sh: ./test.rb: Permission denied
> system('chmod a+x test.rb')
> system('./test.rb')
Hello world
I presume the M ODULEPATH in the Emacs-derived output is simply a copy and paste typo.
The differences between the two env outputs is much greater than I expected; I've selected the ones that look slightly suspicious to me:
$ diff -u works fails
--- works 2011-10-24 15:04:02.000000000 -0700
+++ fails 2011-10-24 15:12:36.000000000 -0700
...
+LD_LIBRARY_PATH=/usr/lib64/R/lib:/usr/local/lib64:/usr/lib/jvm/jre/lib/amd64/server:/usr/lib/jvm/jre/lib/amd64:/usr/lib/jvm/java/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
...
-PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin
-PWD=/home/dleb
...
+PATH=/bin:/usr/bin:/usr/sbin:/usr/local/bin
...
+PWD=/home/a-m/dleb/pecan
...
In the emacs-derived session, your LD_LIBRARY_PATH environment variable may be changing specifics of which dynamically linked libraries are being used when executing ruby. If you ssh in to your server and execute your foo.rb with the changed LD_LIBRARY_PATH, does it work or fail?
LD_LIBRARY_PATH=/usr/lib64/R/lib:/usr/local/lib64:/usr/lib/jvm/jre/lib/amd64/server:/usr/lib/jvm/jre/lib/amd64:/usr/lib/jvm/java/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib ./foo.rb
The PATH environment variable between the two sessions is different; perhaps you have permission to execute /usr/local/bin/ruby (or the libraries in /usr/local/lib/ruby/) but not /usr/bin/ruby (or the libraries in /usr/lib/ruby/). Does your script use #!env ruby or does it use #!/usr/bin/ruby (or some other fixed path)?
Your pwd in one instance is /home/dleb, the other /home/a-m/dleb/pecan -- but HOME is set to /home/a-m/dleb on both systems. Is /home/dleb a symbolic link or does it actually exist separate from /home/a-m/dleb? (This really is grasping at straws -- I don't think this is it, but this problem is baffling.)
One last thing to consider: is your server confined with a tool such as AppArmor, SELinux, TOMOYO, or SMACK? Any of these mandatory access control tools can prevent an application from writing in specific locations, perhaps they aren't yet configured for your site. Check dmesg(1) output to see if there are any rejection messages, most or all these tools log to dmesg(1) if auditd(8) isn't running.

Resources