Python PermissionError on some file operations - windows

I created the following Python 3.5 script:
import sys
from pathlib import Path
def test_unlink():
log = Path('log.csv')
fails = 0
num_tries = int(sys.argv[1])
for i in range(num_tries):
try:
log.write_text('asdfasdfasdfasdfasdfasdfasdfasdf')
with log.open('r') as logfile:
lines = logfile.readlines()
# Check the second line to account for the log file header
assert len(lines) == 1
log.unlink()
not log.exists()
except PermissionError:
sys.stdout.write('! ')
sys.stdout.flush()
fails += 1
assert fails == 0, '{:%}'.format(fails / num_tries)
test_unlink()
and run it like this: python test.py 10000. On Windows 7 Pro 64 with Python 3.5.2, the failure rate is not 0: it is small, but non-zero. Sometimes, it is not even that small: 5%! If you printout the exception, it will be this:
PermissionError: [WinError 5] Access is denied: 'C:\\...\\log.csv'
but it will sometimes occur at the exists(), other times at the write_text(), and I wouldn't be surprised if it happens at the unlink() and the open() too.
Note that the same script, same Python (3.5.2), but on linux (through http://repl.it/), does not have this issue: failure rate is 0.
I realize that a possible workaround could be:
while True:
try: log.unlink()
except PermissionError: pass
else: break
but this is tedious and error prone (several methods on Path instance would need this, easy to forget), and should (IMHO) not be necessary, so I don't think it is practical solution.
So, does anyone have an explanation for this, and a practical workaround, maybe a mode flag somewhere that can be set when Python starts?

Related

Using Python to run .exe automatically

I have some .exe applications that were given to me by a supplier of sensors. They allow me to grab data at specific times and convert file types. But, I need to run them through the cmd manually, and I am trying to automate the process with Python. I am having trouble getting this to work.
So far, I have:
import sys
import ctypes
import subprocess
def is_admin():
try:
return ctypes.windll.shell32.IsUserAnAdmin()
except:
return False
if is_admin():
process = subprocess.Popen('arcfetch C:/reftek/arc_pas *,2,*,20:280:12:00:000,+180', shell=True, cwd="C:/reftek/bin",
stdout=subprocess.PIPE, stderr=subprocess.PIPE,)
out = process.stdout.read()
err = process.stderr.read()
else:
# Re-run the program with admin rights
ctypes.windll.shell32.ShellExecuteW(None, "runas", sys.executable, __file__, None, 1)
But, the "arcfetch" exe was not ran.
Also, this requires me to allow Python to make changes to the hard drive each time, which won't work automatically.
Any assistance would be greatly appreciated!
After some playing around and assistance from comments, I was able to get it to work!
The final code:
import sys
import ctypes
import subprocess
def is_admin():
try:
return ctypes.windll.shell32.IsUserAnAdmin()
except:
return False
if is_admin():
subprocess.run('arcfetch C:/reftek/arc_pas *,2,*,20:280:12:00:000,+180', shell=True, check=True, cwd="C:/reftek/bin")
else:
# Re-run the program with admin rights
ctypes.windll.shell32.ShellExecuteW(None, "runas", sys.executable, __file__, None, 1)
Edit: This still has the admin issue, but I can change the security settings on my computer for that.

Rshell on windows freezes randomly while sending files to ESP32 running micropython via rsync or cp

I'm having trouble copying files to ESP32 device running micropython via rshell (remote micropython shell). The problem seems to occur randomly. Sometimes I manage to successfuly send all files (like 20 of them) with rsync and sometimes I cannot even copy one file with cp. There is no error message and script doesn't crash, it just stops working and freezes the console. I've tried with and without -a parameter (I'm sending only .py files). The last thing printed with debug is code to be run on microcontroller below and it just stops there. I didn't find any pattern. I've tried using some other esp32 device and other windows PC. Same results. I even tried lowering default 32 buffer to 16, no improvement. The worst thing is that is sometimes works fine and I cannot get constant results (even bad). It stops on random files, not always the same one.
Rsync command (with --mirror parameter) is very helpful and I could not just copy all files by hand.
EDIT: just tested on mac and it works fine. I guess it's just problem on windows...
Adding /pyboard/protocol/parser.py
----- About to send 2269 bytes of code to the pyboard -----
def recv_file_from_host(src_file, dst_filename, filesize, dst_mode='wb'):
"""Function which runs on the pyboard. Matches up with send_file_to_remote."""
import sys
import ubinascii
import os
if False:
try:
import pyb
usb = pyb.USB_VCP()
except:
try:
import machine
usb = machine.USB_VCP()
except:
usb = None
if usb and usb.isconnected():
# We don't want 0x03 bytes in the data to be interpreted as a Control-C
# This gets reset each time the REPL runs a line, so we don't need to
# worry about resetting it ourselves
usb.setinterrupt(-1)
try:
with open(dst_filename, dst_mode) as dst_file:
bytes_remaining = filesize
if not False:
bytes_remaining *= 2 # hexlify makes each byte into 2
buf_size = 32
write_buf = bytearray(buf_size)
read_buf = bytearray(buf_size)
while bytes_remaining > 0:
# Send back an ack as a form of flow control
sys.stdout.write('\x06')
read_size = min(bytes_remaining, buf_size)
buf_remaining = read_size
buf_index = 0
while buf_remaining > 0:
if False:
bytes_read = sys.stdin.buffer.readinto(read_buf, read_size)
else:
bytes_read = sys.stdin.readinto(read_buf, read_size)
if bytes_read > 0:
write_buf[buf_index:bytes_read] = read_buf[0:bytes_read]
buf_index += bytes_read
buf_remaining -= bytes_read
if False:
dst_file.write(write_buf[0:read_size])
else:
dst_file.write(ubinascii.unhexlify(write_buf[0:read_size]))
if hasattr(os, 'sync'):
os.sync()
bytes_remaining -= read_size
return True
except:
return False
output = recv_file_from_host(None, '/protocol/parser.py', 1467)
if output is None:
print("None")
else:
print(output)
-----
I had the same problem ,
when trying
C:\> cp src\main.py /pyboard/
the cmd freezes.
When I copied using the following
C:\> cp src/main.py /pyboard/
There where no issues, so maybe rshell has some problems when there are "\" in the path

How to completely skip writing files to disk with libtorrent downloads?

I have the following code to download a torrent off of a magnet URI.
#python
#lt.storage_mode_t(0) ## tried this, didnt work
ses = lt.session()
params = { 'save_path': "/save/here"}
ses.listen_on(6881,6891)
ses.add_dht_router("router.utorrent.com", 6881)
#ses = lt.session()
link = "magnet:?xt=urn:btih:395603fa..hash..."
handle = lt.add_magnet_uri(ses, link, params)
while (not handle.has_metadata()):
time.sleep(1)
handle.pause () # got meta data paused, and set priority
handle.file_priority(0, 1)
handle.file_priority(1,0)
handle.file_priority(2,0)
print handle.file_priorities()
#output is [1,0,0]
#i checked no files written into disk yet.
handle.resume()
while (not handle.is_finished()):
time.sleep(1) #wait until download
It works, However in this specific torrent, there are 3 files, file 0 - 2 kb, file 1 - 300mb, file 3 - 2kb.
As can be seen from the code, file 0 has a priority of 1, while the rest has priority 0 (i.e. don't download).
The problem is that when the 0 file finishes downloading, i want to it to stop and not download anymore. but it will sometimes download 1 file -partially, sometimes 100mb, or 200mb, sometimes couple kb and sometimes the entire file.
So my question is: How can i make sure only file 0 is downloaded, and not 1 and 2.
EDIT: I added a check for whether i got metadata, then set priority and then resume it, however this still downloads the second file partially.
The reason this happens is because of the race between adding the torrent (which starts the download) and you setting the file priorities.
To avoid this you can set the file priorities along with adding the torrent, something like this:
p = parse_magnet_uri(link)
p['file_priorities'] = [1, 0, 0]
handle = ses.add_torrent(p)
UPDATE:
You don't need to know the number of files, it's OK to provide file priorities for more files than ends up being in the torrent file. The remaining ones will just be ignored. However, if you don't want to download anything (except for the metadata/.torrent) from the swarm, a better way is to set the flag_upload_mode flag. See documentation.
p = parse_magnet_uri(link)
p['flags'] |= add_torrent_params_flags_t.flag_upload_mode
handle = ses.add_torrent(p)

using ipdb to debug python code in one cell (jupyter or Ipython)

I'm using jupyter (or Ipython) notebook with firefox, and want to debug some python code in the cell. I am using 'import ipdb; ipdb.set_trace()' as kind of breakpoint, for example my cell has the following code:
a=4
import ipdb; ipdb.set_trace()
b=5
print a
print b
which after execution with Shift+Enter gives me this error:
--------------------------------------------------------------------------
MultipleInstanceError Traceback (most recent call last)
<ipython-input-1-f2b356251c56> in <module>()
1 a=4
----> 2 import ipdb; ipdb.set_trace()
3 b=5
4 print a
5 print b
/home/nnn/anaconda/lib/python2.7/site-packages/ipdb/__init__.py in <module>()
14 # You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.
15
---> 16 from ipdb.__main__ import set_trace, post_mortem, pm, run, runcall, runeval, launch_ipdb_on_exception
17
18 pm # please pyflakes
/home/nnn/anaconda/lib/python2.7/site-packages/ipdb/__main__.py in <module>()
71 # the instance method will create a new one without loading the config.
72 # i.e: if we are in an embed instance we do not want to load the config.
---> 73 ipapp = TerminalIPythonApp.instance()
74 shell = get_ipython()
75 def_colors = shell.colors
/home/nnn/anaconda/lib/python2.7/site-packages/traitlets/config/configurable.pyc in instance(cls, *args, **kwargs)
413 raise MultipleInstanceError(
414 'Multiple incompatible subclass instances of '
--> 415 '%s are being created.' % cls.__name__
416 )
417
MultipleInstanceError: Multiple incompatible subclass instances of TerminalIPythonApp are being created.
The same error appears if I use this code not in the jupyter notebook in the browser, but in jupyter qtconsole.
What does this error mean and what to do to avoid it?
Is it possible to debug code in the cell step-by-step, using next, continue, etc commands of pdb debugger?
Had this problem also and it seems to be related to versions of jupyter and ipdb.
Solution is to use this instead of the ipdb library set_trace call:
from IPython.core.debugger import Tracer
Tracer()() #this one triggers the debugger
Source: http://devmartin.com/blog/2014/10/trigger-ipdb-within-ipython-notebook/
Annotated screenshot:
Tracer() is deprecated.
Use:
from IPython.core.debugger import set_trace
and then place set_trace() where breakpoint is needed.
from IPython.core.debugger import set_trace
def add_to_life_universe_everything(x):
answer = 42
set_trace()
answer += x
return answer
add_to_life_universe_everything(12)
This works fine and brings us a little bit more comfort (e.g. syntax highlighting) than just using the built-in pdb.
source
If using Jupyter Notebook
begin your cell with magic command "%%debug".
Then a ipdb line will be shown at the bottom of the cell which will help you navigate through the debugging session. Following commands should get you started:
n- execute current line and go to next line.
c- continue execution until next break point.
Make sure you restart the kernel each time you decide on debugging, so that all variables are freshly assigned.You can check the value of each variable through the ipdb line and you will see that the variable is undefined until you execute the line that assigns a value to that variable.
%%debug
import pdb
from pdb import set_trace as bp
def function_xyz():
print('before breakpoint')
bp() # This is a breakpoint.
print('after breakpoint')
My version of Jupyter is 5.0.0 and my corresponding ipython version is 6.1.0. I am using
import IPython.core.debugger
dbg = IPython.core.debugger.Pdb()
dbg.set_trace()
Tracer is listed as deprecated.
Update:
I tried using the method from another answer https://stackoverflow.com/a/43086430/8019692 below but got an error:
MultipleInstanceError: Multiple incompatible subclass instances of TerminalIPythonApp are being created.
I prefer my method to the %%debug magic since I can set breakpoints in functions defined in other cells and run the function in another cell. Jupyter/IPython drops into the debugger in my function where the breakpoint is set, and I can use the usual pdb commands. To each his own...
#lugger1, the accepted answer is deprecated.

pygame issue loading images (sprites)

This is the code:
"""
Hello Bunny - Game1.py
By Finn Fallowfield
"""
# 1 - Import library
import pygame
from pygame.locals import *
# 2 - Initialize the game
pygame.init()
width, height = 640, 480
screen=pygame.display.set_mode((width, height))
# 3 - Load images
player = pygame.image.load("resources/images/dude.png")
# 4 - keep looping through
while 1:
# 5 - clear the screen before drawing it again
screen.fill(0)
# 6 - draw the screen elements
screen.blit(player, (100,100))
# 7 - update the screen
pygame.display.flip()
# 8 - loop through the events
for event in pygame.event.get():
# check if the event is the X button
if event.type==pygame.QUIT:
# if it is quit the game
pygame.quit()
exit(0)
When I try and open the file with python launcher I get this error message:
File "/Users/finnfallowfield/Desktop/Code/Game1.py", line 15, in <module>
player = pygame.image.load("resources/images/dude.png")
pygame.error: Couldn't open resources/images/dude.png
I am running a ported 64 bit version of pygame, by the way.
I use Komodo Edit 8 on OS X Mountain Lion with Python 2.7.5
This is not really a pygame issue, but a general problem with loading files. You could get the same problem just trying to open the file for reading:
f = open("resources/images/dude.png")
You are using a "relative" to the image file. This means your program will look under the current working directory for this file. You can know what that is by checking os.getcwd(). The other type of path is an "absolute" path on OS X. This just means a path that starts with a slash.
A common trick I use is to load the images relative to my game source code. For example, if the dude.png is in the same directory as the python code, you could always find it like this:
base_path = os.path.dirname(__file__)
dude_path = os.path.join(base_path, "dude.png")
player = pygame.image.load(dude_path)
Hopefully this helps. You can probably find more information under general questions about loading files and file paths.

Resources