I've got a MacBook (Mac OS X 10.9) with 16 Gb of RAM.
Two Pythons installed via Anaconda: 2.7.8 and 3.4.1. Both equipped with the latest scikit-learn 0.15.1.
While trying to run that simple code (just testing the possibility to serialize large matrixes):
import numpy as np
test_data = np.random.rand(10000, 60000)
print(test_data.nbytes / 2**30)
from sklearn.externals import joblib
joblib.dump(test_data, '/Users/va/Desktop/test_data.joblib')
Python 2.7.8 is doing well, but Python 3.4.1 stuck with the following error:
Failed to save <class 'numpy.ndarray'> to .npy file:
Traceback (most recent call last):
File "/Users/va/anaconda/python.app/Contents/lib/python3.4/site-
packages/sklearn/externals/joblib/numpy_pickle.py", line 240, in save
obj, filename = self._write_array(obj, filename)
File "/Users/va/anaconda/python.app/Contents/lib/python3.4/site-
packages/sklearn/externals/joblib/numpy_pickle.py", line 203, in _write_array
self.np.save(filename, array)
File "/Users/va/anaconda/python.app/Contents/lib/python3.4/site-
packages/numpy/lib/npyio.py", line 453, in save
format.write_array(fid, arr)
File "/Users/va/anaconda/python.app/Contents/lib/python3.4/site-
packages/numpy/lib/format.py", line 410, in write_array
fp.write(array.tostring('C'))
OSError: [Errno 22] Invalid argument
Traceback (most recent call last):
File "<ipython-input-3-90ed09e5c6d4>", line 1, in <module>
joblib.dump(test_data, '/Users/va/Desktop/test_data.joblib')
File "/Users/va/anaconda/python.app/Contents/lib/python3.4/site-
packages/sklearn/externals/joblib/numpy_pickle.py", line 368, in dump
pickler.dump(value)
File "/Users/va/anaconda/python.app/Contents/lib/python3.4/pickle.py", line 412, in dump
self.framer.end_framing()
File "/Users/va/anaconda/python.app/Contents/lib/python3.4/pickle.py", line 196, in end_framing
self.commit_frame(force=True)
File "/Users/va/anaconda/python.app/Contents/lib/python3.4/pickle.py", line 208, in commit_frame
write(data)
OSError: [Errno 22] Invalid argument
It appears the problem is in the amount of data to be stored. E.g., Python 3 handles np.random.rand(10000, 20000), which is 1.5 Gb, perfectly well.
Just in case, pickle didn't work as well:
import pickle
with open('/Users/va/Desktop/test_data.pkl', 'wb') as f:
pickle.dump(test_data, f, protocol=pickle.HIGHEST_PROTOCOL)
goes to:
Traceback (most recent call last):
File "<ipython-input-6-3f73f3011539>", line 3, in <module>
pickle.dump(test_data, f, protocol=pickle.HIGHEST_PROTOCOL)
OSError: [Errno 22] Invalid argument
On Windows 7 Python 3.4 works fine with both joblib and pickle.
Any suggestions how to solve that problem with Python 3 on Mac?
This happens to me on OS X 10.10 with Python 3.4.3 using pickle too
Instead I started using https://github.com/zopefoundation/zodbpickle, which is around 2-3 times slower, but definitely works with sklearn classifiers
Related
C:\Users\user>jupyter notebook
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 197, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\Scripts\jupyter-notebook.EXE_main.py", line 4, in
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\notebook\notebookapp.py", line 47, in
from zmq.eventloop import ioloop
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\zmq_init.py", line 125, in
from zmq import backend
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\zmq\backend_init.py", line 32, in
raise original_error from None
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\zmq\backend_init_.py", line 27, in
_ns = select_backend(first)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\zmq\backend\select.py", line 36, in select_backend
raise ImportError(f"Importing {name} failed with {e}") from e
ImportError: Importing zmq.backend.cython failed with builtins.type size changed, may indicate binary incompatibility. Expected 888 from C header, got 880 from PyObject
I had the same problem and was able to rectify it by uninstalling the Jupyter notebook version that didn't work for me(5.0, 4.8) and installing a version that did work (4.0). I couldn't find this exact problem online but similar problems have occurred with NumPy module installations also fixed with a different version installation.
try:
pip uninstall notebook
pip install notebook==4.0
Been struggling on this on for 3 days. I already have tensorflow working on 3 other mahcines but need to install this on a high performance CPU going forward.
Get following DLL Import error while trying to install tensorflow using pip.
Processor: Intel(R) Xeon(R) CPU
System Type: 64-bit
Based on other solutions i have already tried
Have followed all instructions on the Tensorflow installation page
have installed the Microsoft Visual C++ 2015 Redistributable Update
Tried with Conda Package after installing multiple Anaconda versions
Tried some alternate [whl][1] as I could not find my CPU in the AVX2
CPU List
Tried multiple individual versions (Python 3.5, 3.6, tensorflow
1.12.0, 1.9.0, 1.6.0 etc) (i.e. without anaconda)
pywrap_tensorflow_internal import
ImportError: DLL load failed with error code -1073741795
============================
(venv)
c:\software\Python35\venv\Lib\site-packages>echo %Path%
c:\software\Python35\venv\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\
System32\Wbem;C:\software\Python35;C:\software\Python35\venv\Lib\site-packages;
C:\software\Python35\venv\Lib\site-packages\tensorflow\include\tensorflow;C:\software\curl
(venv) c:\software\Python35>pip show tensorflow
Name: tensorflow
Version: 1.12.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: opensource#google.com
License: Apache 2.0
Location: c:\software\python35\venv\lib\site-packages
Requires: termcolor, grpcio, keras-preprocessing, wheel, absl-py, six, numpy, ke
ras-applications, astor, tensorboard, protobuf, gast
Required-by:
(venv) c:\software\Python35>cd c:\software\python35\venv\lib\site-packag
es
(venv) c:\software\Python35\venv\Lib\site-packages>python
Python 3.5.0 (v3.5.0:374f501f4567, Sep 13 2015, 02:27:37) [MSC v.1900 64 bit (AM
D64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
Traceback (most recent call last):
File "c:\software\Python35\venv\Lib\site-packages\tensorflow\python\pywrap
_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "c:\software\Python35\venv\Lib\site-packages\tensorflow\python\pywrap
_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "c:\software\Python35\venv\Lib\site-packages\tensorflow\python\pywrap
_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, descript
ion)
File "c:\software\Python35\venv\lib\imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "c:\software\Python35\venv\lib\imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: DLL load failed with error code -1073741795
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\software\Python35\venv\Lib\site-packages\tensorflow\__init__.py",
line 24, in <module>
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-im
port
File "c:\software\Python35\venv\Lib\site-packages\tensorflow\python\__init
__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "c:\software\Python35\venv\Lib\site-packages\tensorflow\python\pywrap
_tensorflow.py", line 74, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "c:\software\Python35\venv\Lib\site-packages\tensorflow\python\pywrap
_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "c:\software\Python35\venv\Lib\site-packages\tensorflow\python\pywrap
_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "c:\software\Python35\venv\Lib\site-packages\tensorflow\python\pywrap
_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, descript
ion)
File "c:\software\Python35\venv\lib\imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "c:\software\Python35\venv\lib\imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: DLL load failed with error code -1073741795
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
Tried Python 3.6.0 install and Tensorflow 1.5.0. Got "google.protobuf.pyext import _message" error. So as per this I upgraded to 3.6.1. I am able to Import tensorflow now! So looks like i have to be with tensorflow 1.5.0 version
I have been trying to get Theano as backend for Keras on my 32-bit Windows 10. I have installed a bunch of different things, such as CUDA and Python 3.5 (I have 3.6.5 as my default) and even installed with Anaconda. The Anaconda download wasn't there when I used pip freeze in my terminal, and my code didn't recognize it after the conda install, so I simply pip installed Theano, which always gives me this error when I run it. I normally use Atom, not Anaconda, so I'm not sure if a conda install doesn't work for Atom. I am also unsure of how to switch my code in Atom to Python 3.5, as I know Theano only works up to 3.5, or if it's even necessary. I had previously installed my g++ gcc compiler through MinGW, which I have a suspicion might be the culprit for this error as well, as similar errors on StackOverflow questions had to do with this. The error is listed below. This has been a nightmare installation process, so I apologize for all the info about everything I've been doing. Help would be much appreciated.
Also, when I import Keras in my terminal with python 3.5, it says there is no such module.
Something is horribly wrong.
The error:
Using Theano backend.
You can find the C code in this temporary file: C:\Users\AARONM~1\AppData\Local\Temp\theano_compilation_error_9whx6ow7
Traceback (most recent call last):
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\theano\gof\lazylinker_c.py", line 75, in <module>
raise ImportError()
ImportError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\theano\gof\lazylinker_c.py", line 92, in <module>
raise ImportError()
ImportError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Aaron Mazie\Desktop\tradingbots\testntrain6.py", line 11, in <module>
from keras.layers.core import Dense, Activation, Dropout
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\keras\__init__.py", line 3, in <module>
from . import utils
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\keras\utils\__init__.py", line 6, in <module>
from . import conv_utils
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\keras\utils\conv_utils.py", line 9, in <module>
from .. import backend as K
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\keras\backend\__init__.py", line 81, in <module>
from .theano_backend import *
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\keras\backend\theano_backend.py", line 7, in <module>
import theano
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\theano\__init__.py", line 110, in <module>
from theano.compile import (
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\theano\compile\__init__.py", line 12, in <module>
from theano.compile.mode import *
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\theano\compile\mode.py", line 11, in <module>
import theano.gof.vm
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\theano\gof\vm.py", line 673, in <module>
from . import lazylinker_c
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\theano\gof\lazylinker_c.py", line 127, in <module>
preargs=args)
File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\theano\gof\cmodule.py", line 2359, in compile_str
(status, compile_stderr.replace('\n', '. ')))
Exception: Compilation failed (return status=1): In file included from c:\mingw\lib\gcc\mingw32\6.3.0\include\c++\math.h:36:0,
. from C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\include/pyport.h:194,
. from C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\include/Python.h:53,
. from C:\Users\Aaron Mazie\AppData\Local\Theano\compiledir_Windows-10-10.0.16299-SP0-Intel64_Family_6_Model_142_Stepping_9_GenuineIntel-3.6.5-32\lazylinker_ext\mod.cpp:1:
. c:\mingw\lib\gcc\mingw32\6.3.0\include\c++\cmath:1157:11: error: '::hypot' has not been declared
. using ::hypot;
. ^~~~~
.
For anyone wondering, I uninstalled everything, including the Theano cache files manually, and watched this vid: https://www.youtube.com/watch?v=J70j6q7aOi4&t=131s&index=1&list=LLwpbw2KREgJdXEC0gLRTy4g
I am attempting to install TensorFlow on my Mac ( OSX El Capitan 10.11.2) following the instructions listed here
I used pip to install tensorflow for Python3 and it appears that everything installed successfully.
But when I go to test it, I am greeted with this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/__init__.py", line 23, in <module>
from tensorflow.python import *
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/__init__.py", line 45, in <module>
from tensorflow.python import pywrap_tensorflow
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in <module>
_pywrap_tensorflow = swig_import_helper()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/_pywrap_tensorflow.so, 10): no suitable image found. Did find:
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/_pywrap_tensorflow.so: mach-o, but wrong architecture
This error is not listed on the common problems and I couldn't find anything conclusive on my own.
Update:
#mrry Alrighty, import sys; print(sys.maxsize) is returning 2147483647 so python is running in 32bit mode.
I'm trying to make an application for a mac. When I use pyinstaller to make an exe on a windows machine, it works fine, but when I do it on a mac, the file complies, but then when executed I get the following errors.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller/loader/pyi_importers.py", line 270, in load_module
exec(bytecode, module.__dict__)
File "/Library/Frameworks/Python.framework/Versions/2.7/bin/build/new_client/out00-PYZ.pyz/FileDialog", line 11, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller/loader/pyi_importers.py", line 270, in load_module
exec(bytecode, module.__dict__)
File "/Library/Frameworks/Python.framework/Versions/2.7/bin/build/new_client/out00-PYZ.pyz/Tkinter", line 39, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller/loader/pyi_importers.py", line 409, in load_module
module = imp.load_module(fullname, fp, filename, self._c_ext_tuple)
ImportError: dlopen(/var/folders/vb/vb9xryxdEz467ZADn0LL7U+++TI/-Tmp-/_MEIbvFCo2/_tkinter.so, 2): Library not loaded: #loader_path/Tcl
Referenced from: /var/folders/vb/vb9xryxdEz467ZADn0LL7U+++TI/-Tmp-/_MEIbvFCo2/_tkinter.so
Reason: image not found
It appears to be something wrong with Tkinter and TCL. Here are the modules that are loaded by the program.
import socket, thread, time, json, traceback, FileDialog
from functools import partial
from mtTkinter import *
I was using openpyxl and it required jdcal in the datetime.py module. None of the hidden imports or any of those pyinstaller work-arounds helped, running the exe would still say jdcal not found. The work-around that I used was to just copy the few functions from jdcal directly into the datetime.py in the openpyxl code. Then ran pyinstaller -F program.py
and it worked! Not the best answer, but when you are trying to get a project out the door for a client, it works.