QProgressBar causing bad performance in QT5? - performance

Question

I'm developping a program which parses a file (365000 lines) in which I try to match some keywords after reading each line. This computation along with the update of my QProgressBar are made in another thread using QThread. Everything works fine except for the performance especially when I update the QProgressBar. I use a timer for the parsing and the result is just STUNNING. When I emit a signal to update the QProgressBar the program takes around 45 seconds but when I do not emit the signal for the QProgressBar update then the program takes around 0.40 sec =/
from PyQt5 import QtCore, QtWidgets, QtGui
import sys
import time
liste = ["failed", "exception"]
class ParseFileAsync(QtCore.QThread):
match = QtCore.pyqtSignal(str)
PBupdate = QtCore.pyqtSignal(int)
PBMax = QtCore.pyqtSignal(int)
def run(self):
cpt = 0
with open("test.txt", "r") as fichier:
fileLines = fichier.readlines()
lineNumber = len(fileLines)
self.PBMax.emit(lineNumber)
t0 = time.time()
for line in fileLines:
cpt+=1
self.PBupdate.emit(cpt)
for element in liste:
if element in line:
self.match.emit(line)
finalTime = time.time() - t0
print("over :", finalTime)
class Ui_MainWindow(QtWidgets.QMainWindow):
def __init__(self):
super().__init__()
self.setupUi(self)
self.thread = ParseFileAsync()
self.thread.match.connect(self.printError)
self.thread.PBupdate.connect(self.updateProgressBar)
self.thread.PBMax.connect(self.setMaximumProgressBar)
self.pushButton_GO.clicked.connect(self.startThread)
def printError(self, line):
self.textEdit.append(line)
def updateProgressBar(self, value):
self.progressBar.setValue(value)
def setMaximumProgressBar(self, value):
self.progressBar.setMaximum(value)
def startThread(self):
self.thread.start()
Console output:
over : 44.49321101765038 //QProgressBar updated
over : 0.3695987798147516 //QProgressBar not updated
Am I missing something or is that expected ?
EDIT :
I followed jpo38 and Matteo very good advices. I update the QProgressBar less frequently. The progression is still smooth and the performance is very good (around one second with this implementation). PSB :
class ParseFileAsync(QtCore.QThread):
match = QtCore.pyqtSignal(str)
PBupdate = QtCore.pyqtSignal(int)
PBMax = QtCore.pyqtSignal(int)
def run(self):
with open("test_long.log", "r") as fichier:
fileLines = fichier.readlines()
self.lineNumber = len(fileLines)
self.PBMax.emit(self.lineNumber)
if (self.lineNumber < 30):
self.parseFile(fileLines, False)
else:
self.parseFile(fileLines, True)
def parseFile(self, fileLines, isBig):
cpt = 0
if(isBig):
for line in fileLines:
cpt+=1
if(cpt % (int(self.lineNumber/30)) == 0):
self.PBupdate.emit(cpt)
for element in liste:
if element in line:
self.match.emit(line)
self.PBupdate.emit(self.lineNumber) #To avoid QProgressBar stopping at 99%
else:
for line in fileLines:
cpt+=1
self.PBupdate.emit(cpt)
for element in liste:
if element in line:
self.match.emit(line)

Answer 1

Updating a QProgressBar too often will definitely lead to performance issues. You should update the progress bar less often. You don't want/need to do that for every iteration...365000 times. When you read one line out of 365000, you progressed by 0.0002%, no need to update GUI for that...
Showing progression to user always has a cost...and we accept that because user prefers to wait a bit more and have progress information. However, showing progression must not multiply processing time by 100 as you experienced.
You can either emit the signal to update the progress bar only when progression significantly changed (for instance every time percentage value casted to a int changed, you can store a progression as int value to check that...or test if (line%(fileLines/100)==0) for instance...this will significantly decrease the cost of progress bar update).
Or you could start a QTimer to update the progress bar every 100ms for instance. Then you don't emit any signal from the for loop and just save the progression value to be used when timer times out.
If the file size is always 365000 lines, you can also decide to emit the signal every 1000 lines for instance (if line%1000==0). But the two earlier solutions are preferable because they will fix your performance issues whatever the file size is.

Answer 2

That's a classic problem, every experienced developer I know have a story about a supposedly long process where most of the time was actually taken by the progressbar update (most of these stories end up with removing the progress bar completely).
The point is that, very often, the "unit of work" you process (in your case the parsing of a line) is way smaller than the cost of a progress bar update - GUIs are fast compared to the user reflexes, but are still quite heavyweight when compared to, say, parsing a single line (especially if cross-thread machinery is involved).
In my experience, there are three common solutions:
if you notice that your process in general is "fast" you just drop the progressbar (or replace it with those unhelpful "forward and backwards" progress bars just to show that your program isn't hung if the program is some times fed with files way bigger than usual);
you just update it less frequently; you can emit your signal every 1/100 of your total progress; the advancement will be still smooth and you should not have performance problems (100 updates aren't going to take much time, although I guess they'll still dominate the time taken by your process if it normally takes 0.40 seconds) ;
you can decouple completely the progressbar update from the code that actually does the stuff. Instead of emitting a signal, update an integer class member with the current progress (which should be quite cheap); in the GUI thread use a timer to update the progressbar according to this member every - say - 0.5 seconds. You can even be smarter and avoid showing the progressbar completely if the process finishes before the first timer tick.