I know that using win32gui is the fastest way to get the pixel data compared to PIL and other such libraries. Up until now I was executing my code in windows 7 laptop(old) where it was working very fast but recently I bought a new laptop(processor is superior to my older one) which came with Windows 10 inbuilt but in windows 10 the same program is a lot slower and after having a closer look I understood that the getpixel part of code is making the entire program slow. so to be sure I calculated the time taken for the same code to execute in both windows 10 & windows 7 and what I found is the same code took just 0.03 seconds in windows 7 whereas it took 17 seconds in windows 10(even after having higher processing power). Is there any solution for this? is there any separate library for windows 10 which is faster?
from win32 import win32gui
import time
a = []
def getList():
temp = []
i_desktop_window_id = win32gui.GetDesktopWindow()
i_desktop_window_dc = win32gui.GetWindowDC(i_desktop_window_id)
long_colour = win32gui.GetPixel(i_desktop_window_dc, 100,100)
i_colour = int(long_colour)
px_tuple=((i_colour & 0xff), ((i_colour >> 8) & 0xff), ((i_colour >> 16) & 0xff))
temp.append(px_tuple)
win32gui.ReleaseDC(i_desktop_window_id, i_desktop_window_dc)
return temp
start = time.perf_counter()
for i in range(1000):
a= getList()
stop = time.perf_counter()
print(stop - start)
Related
Im working on Mac Book Pro, OS : Catalina 10.15.7.
At first I was using VS Code to develop in Go but the ( fan?? ) i guess started sounding like a turbo Jet after a while up to the point that the entire OS would shutdown on its own. ( I do not exactly recall what the message said, it was a black screen with white text saying something like your cpu utilization was too high etc, etc and we had to restart the system ).
Today I am trying to run this Python3 script :
#!/usr/local/bin/python3
import csv
import json
import boto3
import time
from multiprocessing import Pool
dynamodb = boto3.resource('dynamodb', endpoint_url='http://localhost:4566', region_name='us-east-2')
table = dynamodb.Table('myTable')
collection = []
count = 0
with open('items.csv', newline='') as f:
reader = csv.DictReader(f)
for row in reader:
obj = {}
collection.append({
"PK" : int(row['id']),
"SK" : "product",
"Name" : row['name']
})
def InsertItem(i):
table.put_item(Item=i)
if __name__ == '__main__':
with Pool(processes=25) as pool:
result = pool.map(InsertItem, collection, 50)
print(result)
And the same behavior occurs ( it does not seem to be related to VS Code now since im directly running this script from the terminal ), the fans are extremely noisy, the performance drops to almost 0 and i get the lollypop mouse pointer of death. ( which seems to be an omen of the PC about to restart itself ) and the process I mentioned above happens again.
Some hints of what is going on :
Im not the only one having this problem. Another teammate does React and is seeing the same behaviour. ( Hs is using VsCode too but I think the problem is more generic ).
It seems to appear only with "intensive" tasks. ( And please take "intensive" with a grain of salt. I do the very same tasks in my Ubuntu Machine with half the RAM and it does not even flinch ).
I have been using Mac for years, and I do not recall having this issue.
So, my question is, is someone else noticing something similar? Is there some workaround for this?
Last note: The python script you see above I tested last week, it did not take even 2 minutes to run. today with these issues its just lingers forever. And I can see for the prints I am doing that it attempts to insert stuff but it freezes without moving forward.
I am new to Ubuntu and I am setting up a new machine for deep learning using Keras and Tensorflow. I am fine tuning VGG16 on a set of pretty complex medical images. My machine specifications are:-
i7-6900K CPU # 3.20GHz × 16
GeForce GTX 1080 Ti x 4
62.8 GiB of RAM
My previous machine was an iMac with no GPU but an i7 quad core processor and 32GB of RAM. The iMac ran the following model although it took 32 hours to complete it.
Here is the code:-
img_width, img_height = 512, 512
top_model_weights_path = '50435_train_uip_possible_inconsistent.h5'
train_dir = '../../MasterHRCT/50435/Three-Classes/train'
validation_dir = '../../MasterHRCT/50435/Three-Classes/validation'
nb_train_samples = 50435
nb_validation_samples = 12600
epochs = 200
batch_size = 16
datagen = ImageDataGenerator(rescale=1. / 255)
model = applications.VGG16(include_top=False, weights='imagenet')
Then:-
generator_train = datagen.flow_from_directory(
train_dir,
target_size=(img_width, img_height),
shuffle=False,
class_mode=None,
batch_size=batch_size
)
bottleneck_features_train = model.predict_generator(
generator=generator_train,
steps=nb_train_samples // batch_size,
verbose=1
)
np.save(file="50435_train_uip_possible_inconsistent.npy", arr=bottleneck_features_train)
print("Completed train data")
generator_validation = datagen.flow_from_directory(
validation_dir,
target_size=(img_width, img_height),
shuffle=False,
class_mode=None,
batch_size=batch_size
)
bottleneck_features_validation = model.predict_generator(
generator=generator_validation,
steps=nb_validation_samples // batch_size,
verbose=1
)
np.save(file="12600_validate_uip_possible_inconsistent.npy", arr=bottleneck_features_validation)
print("Completed validation data")
Yesterday, I ran this code and it was super fast (nvidia-smi suggested that only one GPU was being used which I believe is expected for TF). The CPU hit 56% of maximum. Then it crashed - with a CUDA_OUT_OF_MEMORY error. So I lowered the batch size to 4. Again, its started really fast but then the CPU jumped to 100% and my system froze. I had to hard reboot.
I have tried again today, and the first time I get this error when simply trying to load the ImageNet Weights...
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[3,3,512,512]
[[Node: block4_conv2_2/random_uniform/RandomUniform = RandomUniform[T=DT_INT32, dtype=DT_FLOAT, seed=87654321, seed2=5932420, _device="/job:localhost/replica:0/task:0/gpu:0"](block4_conv2_2/random_uniform/shape)]]
On the command line it says:-
2017-08-08 06:13:57.937723: I tensorflow/core/common_runtime /bfc_allocator.cc:700] Sum Total of in-use chunks: 71.99MiB
2017-08-08 06:13:57.937739: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
Limit: 80150528
InUse: 75491072
MaxInUse: 80069120
NumAllocs: 177
MaxAllocSize: 11985920
Now clearly this is a memory issue - but why would it fail to even load the weights. My Mac can run this entire code albeit intractably slowly. I should note that this morning, I did get this code running once, but this time, it was ridiculously slow - slower than my Mac. My ignorant view is that something is chewing memory but I can't debug this...I am uncertain where to begin being new to Ubuntu. Having the precedence of seeing the code run super fast (and then toward the end crash) yesterday I wonder has the system 'reset' something or disabled something.
Help!
EDIT:
I cleared all the variables in jupyter notebook, dropped the batch size to 1 and reloaded and I managed to load the weights, but on running the first generator I get:
ResourceExhaustedError: OOM when allocating tensor with shape[1,512,512,64]
[[Node: block1_conv1/convolution = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](_arg_input_1_0_0/_105, block1_conv1/kernel/read)]]
I am not clear why I can succesfully run this on my Mac but not a machine with greater RAM, CPU and 4 GPUs...
We have built a system where videos are stored in mongodb. The videos are each a couple of hundred megabytes in size. The system is built in python3 using mongoengine. The c extensions of pymongo and bson are installed.
The definition of the mongoengine documents is:
class VideoStore(Document, GeneralMixin):
video = EmbeddedDocumentListField(SingleFrame)
mutdat = DateTimeField()
_collection = 'VideoStore'
def gen_video(self):
for one_frame in self.video:
yield self._get_single_frame(one_frame)
def _get_single_frame(self, one_frame):
if one_frame.frame.tell() != 0:
one_frame.frame.seek(0)
return pickle.loads(one_frame.frame.read())
class SingleFrame(EmbeddedDocument):
frame = FileField()
Reading a video in Linux takes about 3 to 4 seconds. However running the same code in Windows takes 13 to 17 seconds.
Does anyone out there have any experience with this problem and any kind of solution?
I have thought of and tested (to no avail):
increasing the chunksize
reading the video as a single blob without using yield
Storing the file as a single blob (so no storing of separate frames)
Use Linux, Windows is poorly supported. The use of "infinite" virtual memory among other things causes issues with Windows variants. This thread elaborates further:
Why Mongodb performance better on Linux than on Windows?
Q: In AutoIt, how do I check if Windows desktop is locked
What I have
Let's take this example AutoIt script. It pings an URL in an indefinite loop and reports the response time in a tray tip.
While 1
$responsetime = Ping("www.google.com")
TrayTip("", $responsetime, 1)
sleep(5000)
Wend
Desired result
The new script should only ping if the desktop is not locked. But I haven't found a reliable way to check for this
While 1
$isLocked = secretFunctionIdontKnow
If $isLocked = false Then
Local $responsetime = Ping("www.google.com")
TrayTip("", $responsetime, 1)
EndIf
sleep(5000)
Wend
Hint: The function WinExist("A") seems not reliable enough because a desktop without any open windows will report back 0 (=false) even when unlocked
Based on my comments above, you can use the following secretFunctionIdontKnow ;-)
Func secretFunctionIdontKnow()
$classes = StringSplit(WinGetClassList("[ACTIVE]"), #LF, 2)
Return $classes[0] == "TaskSwitcherWnd"
EndFunc
Fortunately the program switcher hasn't got a return value of TaskSwitcherWnd, so this seems to work for me under Windows 7 Enterprise SP1 64 Bit...
Don't miss the brackets in your function call ;-)
I'm currently working on a project involving saving/loading quite big MAT files (around 150 MB), and I realized that it was much slower to access a loaded cell array than the equivalent version created inside a script or a function.
I created this example to simulate my code and show the difference :
clear; clc;
disp('Test for computing with loading');
if exist('data.mat', 'file')
delete('data.mat');
end
n_tests = 10000;
data = {};
for i=1:n_tests
data{end+1} = rand(1, 4096);
end
% disp('Saving data');
% save('data.mat', 'data');
% clear('data');
%
% disp('Loading data');
% load('data.mat', '-mat');
for i=1:n_tests
tic;
for j=1:n_tests
d = sum((data{i} - data{j}) .^ 2);
end
time = toc;
disp(['#' num2str(i) ' computed in ' num2str(time) ' s']);
end
In this code, no MAT file is saved nor loaded. The average time for one iteration over i is 0.75s. When I uncomment the lines to save/load the file, the computation for one iteration over i takes about 6.2s (the saving/loading time is not taking into consideration). The difference is 8x slower !
I'm using MATLAB 7.12.0 (R2011a) 64 bits with Windows 7 64 bits, and the MAT files are saved with the version v7.3.
Can it be related to the compression of the MAT file? Or caching variables ?
Is there any way to prevent/avoid this ?
I also know this problem. I think it's also related to the inefficient managing of memory in matlab - and as I remember it's not doing well with swapping.
A 150MB file can easily hold a lot of data - maybe more than can be quickly allocated.
I just made a quick calculation for your example using the information by mathworks
In your case total_size = n_tests*121 + n_tests*(1*4096* 8) is about 313MB.
First I would suggest to save them in format 7 (instead of 7.3) - I noticed very poor performance in reading this new format. That alone could be the reason of your slowdown.
Personally I solved this in two ways:
Split the data in smaller sets and then use functions that load the data when needed or create it on the fly (can be elegantly done with classes)
Move the data into a database. SQLite and MySQL are great. Both work efficiently with MUCH larger datasets (in the TBs instead of GBs). And the SQL language is quite efficient to quickly get subsets to manipulate.
I test this code with Windows 64bit, matlab 64bit 2014b.
Without saving and loading, the computation is around 0.22s,
Save the data file with '-v7' and then load, the computation is around 0.2s.
Save the data file with '-v7.3' and then load, the computation is around 4.1s.
So it is related to the compression of the MAT file.