I have configured the environment to run on the new Metal Performance Shaders (MPS) backend for GPU training acceleration for PyTorch and when running Yolov5 on my Macbook M2 Air it is always creating an error.
RES_DIR = set_res_dir()
if TRAIN:
!python /Users/krishpatel/yolov5/train.py --data /Users/krishpatel/yolov5/roboflow/data.yaml --weights yolov5s.pt \
--img 640 --epochs {EPOCHS} --batch-size 32 --device mps --name {RES_DIR}
this is the error
screenshot of the error
UserWarning: The operator 'aten::nonzero' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
t = t[j] # filter
0%| | 0/20 [00:16<?, ?it/s]
Traceback (most recent call last):
File "/Users/krishpatel/yolov5/train.py", line 630, in <module>
main(opt)
File "/Users/krishpatel/yolov5/train.py", line 524, in main
train(opt.hyp, opt, device, callbacks)
File "/Users/krishpatel/yolov5/train.py", line 307, in train
loss, loss_items = compute_loss(pred, targets.to(device)) # loss scaled by batch_size
File "/Users/krishpatel/yolov5/utils/loss.py", line 125, in __call__
tcls, tbox, indices, anchors = self.build_targets(p, targets) # targets
File "/Users/krishpatel/yolov5/utils/loss.py", line 213, in build_targets
j, k = ((gxy % 1 < g) & (gxy > 1)).T
NotImplementedError: The operator 'aten::remainder.Tensor_out' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.
what should i do? because when running in just cpu it is taking a very very very long time? any suggestions would be appreciated.
So I tried to search everywhere but couldn't find anything for macbook. I think if it doesn't help i would have to run it on google colab but then what would be the point of buying expensive macbook and not use gpu for running.
Related
I want to denoise images using below code with GAN.
https://gitee.com/cleryer-1/image-denoise-using-wasserstein-gan
This is my modified version of code: https://drive.google.com/drive/folders/1xdRZ5IOnAx_VyhvBB67x19EoDe6CjHUZ?usp=sharing (i only changed image pathways in config.py )
I'm trying to implement same code on my own test and training data which is as follows:
(http://data.vision.ee.ethz.ch/cvl/DIV2K/DIV2K_train_LR_bicubic_X2.zip)
(http://data.vision.ee.ethz.ch/cvl/DIV2K/DIV2K_valid_LR_bicubic_X2.zip)
I followed the first link above and reshaped and added noise to my own data(256*256 size) with the help of "python image_operation.py -h" in conda. I also changed the amount of virtual memory but problem still occurs. I only changed path in the config.py file with original work.
It gave 2 different errors at different times, these are:
'NoneType' object is not iterable.
numpy.core._exceptions.MemoryError: Unable to allocate 4.69 GiB for an array with shape (102400, 64, 64, 3) and data type float32.
I also tried it on Google Colab Pro+ but it gives an same error. At the last line, "train()"Should i change something about filepaths?
**runfile('C:/Users/eren.alici/Desktop/pythons/train.py', wdir='C:/Users/eren.alici/Desktop/pythons')
generating patches
done
Traceback (most recent call last):
File "C:\Users\eren.alici\Desktop\pythons\train.py", line 83, in
train()
File "C:\Users\eren.alici\Desktop\pythons\train.py", line 24, in train
truth, noise = get_patch(truth, noise)
File "C:\Users\eren.alici\Desktop\pythons\utils.py", line 198, in get_patch
return np.array(out_raw), np.array(out_noise)
MemoryError: Unable to allocate 4.69 GiB for an array with shape (102400, 64, 64, 3) and data type float32**
edit: i tried to train just 20 image and it gives this error this time:
C:\Users\asus\AppData\Local\Microsoft\WindowsApps\python3.8.exe C:\Users\asus\PycharmProjects\pythonProject1\train.py
2022-12-31 16:50:37.740732: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2022-12-31 16:50:37.743082: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
generating patches
done
2022-12-31 16:51:48.724730: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2022-12-31 16:51:48.727183: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2022-12-31 16:51:48.851474: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-OLN8V00
2022-12-31 16:51:48.851846: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-OLN8V00
2022-12-31 16:51:48.860257: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-31 16:51:48.952904: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1e5e84fe770 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-12-31 16:51:48.953235: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
Traceback (most recent call last):
File "C:\Users\asus\PycharmProjects\pythonProject1\train.py", line 84, in <module>
train()
File "C:\Users\asus\PycharmProjects\pythonProject1\train.py", line 45, in train
for times in epoch_bar(range(truth.shape[0] // BATCH_SIZE)):
File "C:\Users\asus\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\progressbar\progressbar.py", line 150, in __next__
value = next(self.__iterable)
TypeError: 'NoneType' object is not an iterator
I have been trying to read a scipy matrix file in my streamlit app.
The app runs on local machine without any errors in app or console.
While deployed on Heroku it raises ValueError: not enough values to unpack (expected 5, got 2)
on the line co_occurrence_matrix = scipy.io.mmread("./database/matrix.mtx").
I've crosschecked the following points, and not sure where to look for the problem.
Matrix is created with
smatrix = scipy.sparse.csr_matrix(matrix)
scipy.isspmatrix(smatrix) #-> returns True
scipy.io.mmwrite("./database/matrix.mtx\",smatrix)
All library versions including python itself are identical between the two apps. Both checked on consoles with pip list. And requirements file is created with pip freeze.
Files are synchronized on git, git status returns 'up to date. Heroku uses the file on git, the local app runs on the synchronized file.
If it makes any difference, the .mtx file is uploaded via git lfs.
Heroku deploys the app successfully, yet inside the streamlit app, it gives the error.
Full Error:
File "/app/.heroku/python/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
exec(code, module.__dict__)File "/app/main.py", line 4, in <module>
from gorsellestirme_util import set_bg, Plotter, matrix, word_list, main_dfFile "/app/gorsellestirme_util.py", line 30, in <module>
matrix, word_list, main_df = matrix_reader()File "/app/.heroku/python/lib/python3.9/site-packages/streamlit/runtime/legacy_caching/caching.py", line 625, in wrapped_func
return get_or_create_cached_value()File "/app/.heroku/python/lib/python3.9/site-packages/streamlit/runtime/legacy_caching/caching.py", line 609, in get_or_create_cached_value
return_value = non_optional_func(*args, **kwargs)File "/app/gorsellestirme_util.py", line 12, in matrix_reader
co_occurrence_matrix = mmread("./database/matrix.mtx")File "/app/.heroku/python/lib/python3.9/site-packages/scipy/io/_mmio.py", line 77, in mmread
return MMFile().read(source)File "/app/.heroku/python/lib/python3.9/site-packages/scipy/io/_mmio.py", line 438, in read
self._parse_header(stream)File "/app/.heroku/python/lib/python3.9/site-packages/scipy/io/_mmio.py", line 502, in _parse_header
self.__class__.info(stream)File "/app/.heroku/python/lib/python3.9/site-packages/scipy/io/_mmio.py", line 234, in info
mmid, matrix, format, field, symmetry = \
As for the source of the problem: It is clearly stated in the Heroku docs that Heroku does not support git-lfs files. I've missed that point.
As a workaround, there are multiple build packs in Heroku elements. FYI, those buildpacks are also limited with Heroku's 500Mb filesize cap. And security has to be considered as an issue as those buildpacks require third-party access to your git.
I compiled SFML library and my app on Raspbian with this tutorial https://github.com/oomek/sfml-pi. After this I moved shared objects and app to buildroot system for Raspberry Pi 4. I chosed DISPMANX version, my goal was to run app without X server.
When I try to run app, I have error failed to add service - already in use?. I know that there was many similar topics, I tried this solutions:
Comment dtoverlay=vc4-kms-v3d in config.txt -> this line didn't existed in my config.
Change gpu_mem to 128 -> any improvement
My config.txt:
# Please note that this is only a sample, we recommend you to change it to fit
# your needs.
# You should override this file using a post-build script.
# See http://buildroot.org/manual.html#rootfs-custom
# and http://elinux.org/RPiconfig for a description of config.txt syntax
# We always use the same names, the real used variant is selected by
# BR2_PACKAGE_RPI_FIRMWARE_{DEFAULT,X,CD} choice
start_file=start.elf
fixup_file=fixup.dat
kernel=zImage
# To use an external initramfs file
#initramfs rootfs.cpio.gz
# Disable overscan assuming the display supports displaying the full resolution
# If the text shown on the screen disappears off the edge, comment this out
disable_overscan=1
# How much memory in MB to assign to the GPU on Pi models having
# 256, 512 or 1024 MB total memory
gpu_mem_256=128
gpu_mem_512=128
gpu_mem_1024=128
gpu_mem_1024=192
gpu_mem=128
# fixes rpi (3B, 3B+, 3A+, 4B and Zero W) ttyAMA0 serial console
dtoverlay=miniuart-bt
On buildroot I set opengl from gst1-plugins-base with dispmanx, gles2, egl and wayland. I didn't set mesa-3d.
Any idea how can I make my app working? Should I add something to my config.txt?
Geeting an error while using mirror in hardops. I am using mac and it is working fine in windows. can it not be used in Mac?
Traceback (most recent call last):
File "///Library/Application Support/Blender/2.82/scripts/addons/HOps/operators/Gizmos/mirror.py", line 347, in invoke
current_tool = ToolSelectPanelHelper._tool_get_active(context, 'VIEW_3D', None)[0][0]
File "/Applications/Blender.app/Contents/Resources/2.82/scripts/startup/bl_ui/space_toolsystem_common.py", line 250, in _tool_get_active
for item in ToolSelectPanelHelper._tools_flatten(cls.tools_from_context(context, mode)):
AttributeError: type object 'ToolSelectPanelHelper' has no attribute 'tools_from_context'
location: :-1
Maybe it can help: I had this error on windows with previous currium Hard Ops, which is solved with latest Hard Ops update : Neodymium
Link to Hard Ops on BlenderMarket
Checking whether there is an H2O instance running at http://localhost:54321..... not found.
Attempting to start a local H2O server...
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)
Starting server from
C:\Users\Ramakanth\Anaconda2\lib\site-packages\h2o\backend\bin\h2o.jar
Ice root: c:\users\ramaka~1\appdata\local\temp\tmpeaff8n JVM stdout:
c:\users\ramaka~1\appdata\local\temp\tmpeaff8n\h2o_Ramakanth_started_from_python.out
JVM stderr:
c:\users\ramaka~1\appdata\local\temp\tmpeaff8n\h2o_Ramakanth_started_from_python.err
Traceback (most recent call last): File "", line 1, in
File
"C:\Users\Ramakanth\Anaconda2\lib\site-packages\h2o\h2o.py", line 262,
in init
min_mem_size=mmin, ice_root=ice_root, port=port, extra_classpath=extra_classpath) File
"C:\Users\Ramakanth\Anaconda2\lib\site-packages\h2o\backend\server.py",
line 121, in start
mmax=max_mem_size, mmin=min_mem_size) File "C:\Users\Ramakanth\Anaconda2\lib\site-packages\h2o\backend\server.py",
line 317, in _launch_server
raise H2OServerError("Server process terminated with error code %d" % proc.returncode) h2o.exceptions.H2OServerError: Server process
terminated with error code 1
Assuming "build 9.0.1+11" means Java 9, that is your problem: H2O currently only supports Java 7 or Java 8. This is the ticket to follow for adding Java 9 support. In the meantime uninstall your current Java, then install Java 8.
UPDATE: It seems Java 9 is now supported, so upgrade to h2o 3.20 or later.
BTW, normally you should be giving a lot more information: which language you are using, what code you used to try and start H2O (or the commandline if you started it that way), what OS, what versions of Java, R, Python, etc., number of cores, amount of memory, etc.