subprocess sometimes sends returns empty - windows

Question

I have the following class that is used to run a third party command line tool which I have no control over.
I run this ina Qthread in a PyQt Gui.
I turn the gui into an EXE using Pyinstaller
Problems are more prevalent when it is an EXE
class CLI_Interface:
def process_f(self, command, bsize=4096):
self.kill_process(CLI_TOOL)
startupinfo = STARTUPINFO()
startupinfo.dwFlags |= STARTF_USESHOWWINDOW
startupinfo.wShowWindow = SW_HIDE
p = Popen(command, stdout=PIPE, stderr=PIPE,
startupinfo=startupinfo, bufsize=bsize, universal_newlines=True)
try:
out, err = p.communicate(timeout=120)
except TimeoutExpired:
p.kill()
out, err = p.communicate()
return out.split(), err.split()
def kill_process(self, proc):
# Check process is running, Kill it if it is,
# return False if not.
# uses its own popen for Stderr >> stdout
# If we use the self.process_f method, it will create an infinite loop
startupinfo = STARTUPINFO()
startupinfo.dwFlags |= STARTF_USESHOWWINDOW
startupinfo.wShowWindow = SW_HIDE
try:
kill_proc = Popen("TaskKill /IM {} /T /F".format(proc), stdout=PIPE, stderr=STDOUT,
startupinfo=startupinfo, universal_newlines=True).communicate()[0]
if 'ERROR' not in kill_proc.split():
return True # Process Killed
else:
self.kill_process(proc)
except Exception as e:
return False
def download_data(self, code):
""" download data from the device based on a 5 digit code """
command = '"{}" -l {},{} {}'.format(CLI_TOOL_PATH,
code[0], code[2], code[1])
try:
p = self.process_f(command)
proc, err = p[0], p[1]
try:
if err[-2] == '-p':
return False
return True
except IndexError:
if not proc:
return False # This means there is no data but the file is still saved!!
pass
return True
except Exception as e:
return False
def ....
def ....
def ....
Thread:
class GetDataThread(QThread):
taskFinished = pyqtSignal()
notConnected = pyqtSignal()
def __init__(self, f, parent=None):
super(GetDataThread, self).__init__(parent)
self.f = f
def run(self):
is_dongle_connected()
DD = cli.download_data(self.f)
if not DD:
self.notConnected.emit()
else:
self.taskFinished.emit()
I either get a done! or error - This is normal when running from the command line.
Sometimes I get an empty list returned and I put this back into a recursive loop after killing the program.
However, it does not seem to restart properly and the problem continues - it gets stuck in a loop of nothing!.
Meanwhile, the csv files the cli tool produces are created as normal yet I have no data from stdout / stderr
Looking at processes the conhost and the cli tool are destroyed no problem.
The gui will continue to fail (until I unplug and plug in the dongle and / or restart the program / computer.
When I open the CLI and run the same command, it works fine or throws an error (which I catch in the program no problem)
I have tried setting a buffer as some files generated can reach 2.4mb
I tried setting a higher timeout to allow for it to finish.
There does not seem to be a correlation with file size though and it can get stuck at any size.
The flow is like so:
Gui >> CLI >> Dongle >> Sensor
Running on Windows 10
How can I make the connection more solid or debug what processes might still be lingering around and stopping this?
Is it blocking?
Is it a pipe buffer overflow? - If so How do I determine the correct bufsize?
Is it something to do with PyQt and Python Subprocess or Pyinstaller?
Would it be better to use QProcess instead of Subprocess?
Thanks in advance!