I'm using Pysftp to sftp put a large number of files, and would love to get some progress outputs as it runs so I know how it's doing. Is there a verbose option (or equivalent) I can use to get that ouput?
Thanks.
I could be incorrectly importing paramiko here as I believe its in pysftp anyway but you should get the jist.
import logging
import paramiko
import pysftp
paramiko.util.log_to_file('log_debug.txt', level = 'DEBUG')
# As far as I can see you can use DEBUG, INFO, WARNING or ERROR here but there's possibly more
with pysftp.Connection([whatever you're doing here normally]) as sftp:
sftp.put([or whatever you're doing here instead])
with open('log_debug.txt') as logfile:
log = logfile.read()
print (log)
Obviously you don't have to print it, do what you like...
I've only got this from trying to do something similar, I'm not an expert!
Related
I want to run pywikibot from inside Docker container, so I could run some cron jobs with it from the cloud (maybe Azure).
I added code of my bot and user-config.py file to my Docker container, but when it tries to update some page, it uses getpass to read password from input:
Password for user BunykBot on wikipedia:uk (no characters will be shown): ^CWARNING: /usr/local/lib/python3.7/getpass.py:91: GetPassWarning: Can not control echo on the terminal.
passwd = fallback_getpass(prompt, stream)
Is there any way to give it password from some variable? I see that login.py script that creates .lwp file uses site.login() which uses api.LoginManager, but not gives it password anywhere, so it obtains it from input. Is there any way I could monkeypatch this with not much effort? Or do I need some updated fork of pywikibot?
So, I decided to go with monkey patching, and just created file that needs to be imported before pywikibot:
import os
def patch_wiki():
import pywikibot
original = pywikibot.input
def new_input(question, password=False, default='', force=False):
if password:
return os.getenv('WIKI_PASS')
return original(question, password, default, force)
pywikibot.input = new_input
if os.getenv('WIKI_PASS'):
patch_wiki()
Which works for me.
There are better options, like using BotPasswords or OAuth, see more in https://lists.wikimedia.org/pipermail/pywikibot/2019-December/009968.html
see https://phabricator.wikimedia.org/T248471 for a proposed patch to make password use from API much easier
I am trying to write a custom ansible module.
I need to have some debug info during module execution.
However the following lines fo not print anything even with the highest verbosity level enabled (-vvvv) and with export ANSIBLE_DEBUG=true
from ansible.module_utils.basic import AnsibleModule
module.log(msg="some_message"))
The only time I am ever able to see some msg printed is via the following method:
module.exit_json(changed=True, msg=_msg)
It should be emphasized that AnsibleModule.log() will not send the output to neither stout or stderr. It will send it to the default system logging facility.
In my case this was /var/log/syslog.
I am using AWS ec2 module and I would like to log what userdata is sent to AWS with every command, but I don't want verbose output from all other useless tasks.
Is there any way I can enable verbosity of the ec2 module?
I agree with #techraf that there is no out of the box way to do this.
But Ansible is easily tuneable with plugins!
Drop this code as <playbook_dir>/callback_plugins/verbose_tasks.py:
from ansible.plugins.callback import CallbackBase
import json
try:
from __main__ import display
except ImportError:
display = None
class CallbackModule(CallbackBase):
def v2_runner_on_any(self, result, ignore_errors=False):
if (result._task.action in ['file','stat']):
print '####### DEBUG ########'
print json.dumps(result._result,indent=4)
print '####### DEBUG ########'
v2_runner_on_ok = v2_runner_on_any
v2_runner_on_failed = v2_runner_on_any
You can tune what modules' results you want to print by changing ['file','stat'] list.
If you need only ec2, replace it with ['ec2'].
I don't think there is a straightforward way.
As a workaround you could run the whole play with no_log: true and add no_log: false explicitly to your task calling ec2 action. Then run the playbook with -v.
I'm trying to use parallel computing from ipython parallel library. But I have little knowledge about it and I find the doc difficult to read from someone who knows nothing about parallel computing.
Funnily, all tutorials I found just re-use the example in the doc, with the same explanation, which from my point of view, is useless.
Basically what I'd like to do is running few scripts in background so they are executed in the same time. In bash it would be something like :
for my_file in $(cat list_file); do
python pgm.py my_file &
done
But bash interpreter of Ipython notebook doesn't handle the background mode.
It seems that solution was to use parallel library from ipython.
I tried :
from IPython.parallel import Client
rc = Client()
rc.block = True
dview = rc[:2] # I take only 2 engines
But then I'm stuck. I don't know how to run twice (or more) the same script or pgm at the same time.
Thanks.
One year later, I eventually managed to get what I wanted.
1) Create a function with what you want to do on the different cpu. Here it is just calling a script from the bash with the ! magic ipython command. I guess it would work with the call() function.
def my_func(my_file):
!python pgm.py {my_file}
Don't forget the {} when using !
Note also that the path to my_file should be absolute, since the clusters are where you started the notebook (when doing jupyter notebook or ipython notebook) which is not necessarily where you are.
2) Start your ipython notebook Cluster with the number of CPU you want.
Wait 2s and execute the following cell:
from IPython import parallel
rc = parallel.Client()
view = rc.load_balanced_view()
3) Get a list of file you want to process:
files = list_of_files
4) Map asynchronously your function with all your files to the view of your engines you just created. (not sure of the wording).
r = view.map_async(my_func, files)
While it's running you can do something else on the notebook (It runs in "background"!). You can also call r.wait_interactive() that enumerates interactively the number of files processed and the number of time spent so far and the number of files left. This will prevent you to run other cells (but you can interrupt it).
And if you have more files than engines, no worries, they will be processed as soon as an engine finishes with 1 file.
Hope this will help others !
This tutorial might be of some help:
http://nbviewer.ipython.org/github/minrk/IPython-parallel-tutorial/blob/master/Index.ipynb
Note also that I still have IPython 2.3.1, I don't know if it changed since Jupyter.
Edit: Still works with Jupyter, see here for difference and potential issues you may encounter
Note that if you use external libraries in your function, you need to import them on the different engines with:
%px import numpy as np
or
%%px
import numpy as np
import pandas as pd
Same with variable and other functions, you need to push them to the engine name space:
rc[:].push(dict(
foo=foo,
bar=bar))
If you're trying to executing some external scripts in parallel, you don't need to use IPython's parallel functionality. Replicating bash's parallel execution can be achieved with the subprocess module as follows:
import subprocess
procs = []
for i in range(10):
procs.append(subprocess.Popen(['ls', '/Users/shad/tmp/'], stdout=subprocess.PIPE))
results = []
for proc in procs:
stdout, _ = proc.communicate()
results.append(stdout)
Be wary that if your subprocess generates a lot of output, the process will block. If you print the output (results) you get:
print results
['file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n', 'file1\nfile2\n']
I'm dealing with some GB-sized numpy arrays in IPython. When I delete them, I definitely want them gone, in order to recover the memory. IPythons output cache is quite annoying there, as it keeps the objects alive even after deleting the last actively intended reference to them. I already set
c.TerminalInteractiveShell.cache_size = 0
in the IPython configuration, but this only disables caching of entries to _oh, the other variables like _, __ and so on are still created. I'm also aware of %xdel, but anyways, I'd prefer to disable it completely, as I rarely use the output history anyways, so that a plain del would work again right away.
Looking at IPython/core/displayhook.py Line 209-214 I would say that it is not configurable. You could try making a PR to add an option to disable it totally.
Enter
echo "__builtin__._ = True" > ~/.config/ipython/profile_default/startup/00-disable-history.py
and your history should be gone.
Edit:
Seems like the path to the config directory is sometimes a bit different, either ~/.config/ipython or just ~/.ipython/. So just check which one you got and adjust the path accordingly. The solution still works with jupyter console.
Seems that we can suppress the output cache by putting a ";" at the end of the line now.
See http://ipython.org/ipython-doc/stable/interactive/tips.html#suppress-output
Create an ipython profile:
!ipython profile create
The output might be (for ipython v4.0):
[ProfileCreate] Generating default config file: '/root/.ipython/profile_default/ipython_config.py'
[ProfileCreate] Generating default config file: '/root/.ipython/profile_default/ipython_kernel_config.py'
Then add the line 'c.InteractiveShell.cache_size = 0' to the ipython_kernel_config.py file by
!echo 'c.InteractiveShell.cache_size = 0' >> /root/.ipython/profile_default/ipython_kernel_config.py
Load another ipython kernel and check if this work
In [1]: 123
Out[1]: 123
In [2]: _1
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-51-21553803e553> in <module>()
----> 1 _1
NameError: name '_1' is not defined
In [3]: len(Out)
Out[3]: 0