When running linux code on a local mac environment, I am encountering the following error associated with decorator functions and multiprocessing. I am running Python 3.8.10.
_pickle.PicklingError: Can't pickle <function unzip at 0x7ff3e899c1f0>: it's not the same object as endpoints.v1.s3.unzip
I am also using functools.wraps, which does not appear to help with this issue.
Related
My MAC Pro has the latest MPS GPU. I'm trying to reproduce some results from a colleague who read a pickle file using a local NVIDIA GPU.
First, I ensured that my mps gpu is being used:
import torch
device = torch.device("mps") if torch.backends.mps.is_available() else "cpu"
Then I ran the following codes:
import pickle
with open('patient_notes_agg.pickle', 'rb') as df:
patient_notes_agg = pickle.load(df)
patient_notes_agg.head()
I received the following error:
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
So I tried to follow the prompt and switch to use cpu instead of mps gpu, and I used the code below:
model = torch.load('patient_notes_agg.pickle', map_location=torch.device('cpu'))
Unfortunately, I still received the same error.
To give more details:
MacOS Montery 12.3
chip Apple M1 Max
pytorch version '1.13.0'
Python 3.10.7
And I don't use conda environment.
I went through several posts with similar issues on stackoverflow, but unfortunately they didn't help. Any suggestion will be appreciated.
I found a way to work around the issue. Not utilizing my M1 GPU but using cpu instead by creating a reading function. Thanks to this link.
import io
import pickle
class CPU_Unpickler(pickle.Unpickler):
def find_class(self, module, name):
if module == 'torch.storage' and name == '_load_from_bytes':
return lambda b: torch.load(io.BytesIO(b), map_location='cpu')
else: return super().find_class(module, name)
with open('patient_notes_agg.pickle', 'rb') as df:
patient_notes_agg = CPU_Unpickler(df).load()
patient_notes_agg.head()
We've been trying to get spyder 5.1.5 to work inside a windows server environment (version 2012 and 2016). Before We've been using Spyder 3.3.6 successfully, but we want to upgrade to get "black/autopep8" autoformatting.
We have this distributed CIFS-like shared drive inside this server that has some strict permissions (but my username definitely has read/write/modify access at the level where the script is).
Unfortunately, when I try to get spyder to SAVE my code (and autoformat using black), I get the following (trimmed) error below. Basically spyder cannot save due to a permission error that doesn't make sense to me.
2021-12-08 18:23:52,005 [DEBUG] [spyder.plugins.completion.providers.languageserver.client] -> python Response error: {'code': -32602, 'message': "PermissionError: [WinError 5] Access is denied: '\\\\\\\\data-dist\\\\DataVol_014\\\\group\\\\users\\\\myusername\\\\myscript.py'", 'data': {'traceback': [' File "C:\\ProgramData\\Anaconda3\\envs\\py37-dev\\lib\\site-packages\\pylsp_jsonrpc\\endpoint.py", line 116, in consume\n self._handle_request(message[\'id\'], message[\'method\'], message.get(\'params\'))\n', '
Troubleshooting this further, I certainly have read/write/modify access to the following levels of this path:
\\data-dist\DataVol_014\group\users\myusername\myscript.py
\\data-dist\DataVol_014\group\users\myusername\
\\data-dist\DataVol_014\group\users\
However, I cannot even view this level:
\\data-dist\DataVol_014\group\
Still this does not make sense to me. I don't understand why spyder would need this level of access.
Any help/guidance is appreciated.
The error certainly goes away if I copy the script to a local drive on said server and the black autoformatting works as expected.
I am writing (hopefully) a simply AWS Lambda that will do an RDS Oracle SQL SELECT and email the results. So far I have been using the Lambda Management Console, but all the examples I've run across talk about making a Lambda Deployment Package. So my first question is can I do this from the Lambda Management Console?
Next question I have is what to import for the Oracle DB API? In all the examples I have seen, they download and build a package with pip, but that would then seem to imply using a Deployment Package (see above). Trying to import any of these modules listed in the examples simply give "No module named "...
After writing the above I dug into the boto3 API referrence and came up with:
import boto3
client = boto3.client('rds-data')
But it gives the error: Unknown service: 'rds-data'.
So I'm still lost.
As you can probably tell, I'm new to the Lambda environment. Any suggestions or examples would be greatly appreciated. Thanks.
This is an update of the solution using the 18c Oracle client libraries. If it wasn't for main solution it would have taken me a lot longer to get my code working. This will hopefully help anyone that follows.
(an aside - I tried getting it working with the instantclient_19_3 but went round in circles for a day, and then tried with instantclient_18_5 and it worked)
Files downloaded and used
instantclient-basic-linux.x64-18.5.0.0.0dbru.zip (all files)
cx_Oracle 7.2.2 (https://cx-oracle.readthedocs.io/en/latest/release_notes.html#releasenotes)
libaio.so.1.0.1 (as described in main answer, renamed to libaio.so.1)
This then gave these files in the zip (lambda_function.py is my python source code)
zip contents
Apparently, AWS Lambda is using an older version of boto3, which does not have rds-data yet.
So I'm afraid you will have to create a deployment package containing a more recent version of boto3.
One way to do this, would be to:
Create your lambda handler file (in this case named index.py).
def my_handler(event, context):
client = boto3.client('rds-data')
print(client)
# do stuff
return "hello world"
Add a requirements.txt file in the same folder, which will contain something like:
awscli >= 1.16.118
boto3 >= 1.9.108
Now run this (depending on the setup on your computer, you can use pip instead of pip3) in the directory/folder of your index and requirement file:
pip3 install -r requirements.txt -t .
zip -r somezipname .
Next, upload this zip and change your handler 'entry point' to index.my_handler. The code should now run without errors.
older version of boto3 does not support rds-data.
but you can deploy package with zip folder.
i recommend you to use import cx-oracle
for that install cx-oracle using pip
and upload zip packages. check this
[How can I access Oracle from Python?
After much groaning and gnashing of teeth I have come up with a successful solution.
rds_data (as confirmed by AWS Support) is only supporting Aurora Databases. Wish the AWS documents mentioned this. 8{(>
Thanks to the answers above as well as Jason Landrey for hints as to the solution.
In order to access RDS/Oracle, you need to use cx_Oracle. But wait, there's more.
cx_Oracle is not in the standard Lambda environment, so you need to bring your own. My development environment is on Windows, but the Lambda environment is Linux. So, you need to download and install in your packaging directory I got mine from https://pypi.org/project/cx-Oracle/#files. Install locally with:
pip install cx_Oracle-7.1.2-cp37-cp37m-manylinux1_x86_64.whl -t .
You will see several file appear in . Then you need to find a Linux system and download /lib64/libaio.so.1.0.1 and call it libaio.so.1 in your packaging directory.
And then you need to download both Oracle instant client basic and SDK packages from http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html.
Create a zip file with all these items (including your own Python source). In doing so, rename Oracle instant client files libclntsh.so.11.1 to libclntsh.so and libocci.so.11.1 to libocci.so.
Upload the zip to a S3 bucket as the direct deploy is limited to 66mb and this zip is a bit larger.
Create a Lambda with the appropriate IAM permissions and VPC access, install the package and it should be good to go.
I found that if you don't include all the instant client files you start getting Oracle errors about missing timezone and NLS information.
List of zip contents (for me, YMMV):
7996693 08/24/2013 12:30 libnnz11.so
0 03/11/2019 16:10 cx_Oracle-7.1.1.data/
0 03/11/2019 16:10 cx_Oracle-7.1.1.data/data/
0 03/11/2019 16:10 cx_Oracle-7.1.1.data/data/cx_Oracle-doc/
0 03/11/2019 16:10 cx_Oracle-7.1.1.dist-info/
1325 03/13/2019 12:35 Email.py
1805 02/19/2019 21:11 cx_Oracle-7.1.1.data/data/cx_Oracle-doc/LICENSE.txt
163 02/19/2019 21:11 cx_Oracle-7.1.1.data/data/cx_Oracle-doc/README.txt
851 02/19/2019 21:11 cx_Oracle-7.1.1.dist-info/METADATA
628 02/19/2019 21:12 cx_Oracle-7.1.1.dist-info/RECORD
109 02/19/2019 21:12 cx_Oracle-7.1.1.dist-info/WHEEL
10 02/19/2019 21:11 cx_Oracle-7.1.1.dist-info/top_level.txt
2270301 02/19/2019 21:11 cx_Oracle.cpython-37m-x86_64-linux-gnu.so
2140 03/13/2019 14:21 getSecrets.py
5560 03/12/2019 08:48 libaio.so.1
53865194 08/24/2013 12:30 libclntsh.so
118738042 08/24/2013 12:30 libociei.so
7633 03/13/2019 16:39 scheduleReports.py
I have been working on a web scraping program and it has recently just stopped working and is giving me the following error.
Traceback (most recent call last):
File "C:/Users/Bob/Desktop/test 3.5.py", line 7, in <module>
with open(saveDB,'rb') as f:
TypeError: invalid file: WindowsPath('Z:/project1/MasterWellDB.txt')
Where Z: is a network drive, but I have also moved the file locally and the error still exists.
I have tried multiple python versions, uninstalled and reinstalled visual studio multiple times and I am still clueless.
Here is my code:
from pathlib import Path
import pickle
# Opening well database
saveDB = Path(r"Z:\project1\MasterWellDB.txt")
#open picked DB if avaiable else remake database
if saveDB.exists():
with open(saveDB,'rb') as f:
wells = pickle.load(f)
print('success!')
Any help would be greatly appreciated. Thanks!
i have already read a lot of threads with similar topics and the tag jedi-vim here on stackoverflow and also other pages.
But this problem still keeps me from doing my actual work :-/.
Up to now I have already tried several steps back and forth, but basically I did the following:
1. Install latest Python3.4 (32bit)
2. Got this precompiled VIM: (https:// bitbucket.org/Haroogan/vim-for-windows/downloads/vim-7.4.417-python-2.7-python-3.4-ruby-2.0.0-lua-5.2-perl-5.18-windows-x86.zip) (32Bit)
3. Installed Pathogen
4. Installed Flake8 successfully
5. Installed Jedi (systemwide?) with pip
6. Got Jedi-vim from github.com/davidhalter/jedi-vim and unpacked the dir jedi-vim-master to:
6.1. $HOME/vimfiles/bundle/ only
6.2. $VIM/vimfiles/bundle/ only
6.3. BOTH
python and python3 both work (before I ONLY had python3).
So
":python3 import sys; print(sys.version)" and
":python2 import sys; print(sys.version)" give an output.
In my vimrc I force jedi-vim to use python3:
"let g:jedi#force_py_version = 3"
And now the actual error is the following:
When I type e.g. "self." in a *.py-file he searches for half a sec and says:
E117: unknown function: pythoncomplete#Complete
-- Omni completion (^O^N^P) Pattern not found
When I want to call a function and want type the parameters, something pops up / gets faded in, but it's some unclear signs and not the actual parameters.
see here
Other info:
:scriptnames :
D:\tools\v\plugin\getscriptPlugin.vim
D:\tools\v\plugin\gzip.vim
D:\tools\v\plugin\matchparen.vim
D:\tools\v\plugin\netrwPlugin.vim
D:\tools\v\plugin\rrhelper.vim
D:\tools\v\plugin\spellfile.vim
D:\tools\v\plugin\tarPlugin.vim
D:\tools\v\plugin\tohtml.vim
D:\tools\v\plugin\vimballPlugin.vim
D:\tools\v\plugin\zipPlugin.vim
D:\tools\v\menu.vim
D:\tools\v\lang\menu_de.latin1.vim
D:\tools\v\lang\menu_de_de.latin1.vim
D:\tools\v\autoload\paste.vim
D:\tools\v_gvimrc
D:\tools\v\syntax\syntax.vim
D:\tools\v\syntax\synload.vim
D:\tools\v\syntax\syncolor.vim
D:\tools\v\filetype.vim
H:\vimfiles\autoload\pathogen.vim
D:\tools\v\ftoff.vim
D:\tools\v\syntax\nosyntax.vim
D:\tools\v\ftplugin.vim
D:\tools\v\indent.vim
D:\tools\v\syntax\python.vim
D:\tools\v\ftplugin\python.vim
D:\tools\v\bundle\jedi-vim-master\ftplugin\python\jedi.vim
D:\tools\v\bundle\jedi-vim-master\autoload\jedi.vim
D:\tools\v\bundle\vim-flake8-master\ftplugin\python_flake8.vim
H:\vimfiles\bundle\vim-flake8-master\ftplugin\python_flake8.vim
D:\tools\v\indent\python.vim
My vimrc:
https://drive.google.com/file/d/0Bwb4EPY-atMdOWhnOGJiNmxYbGM/view?usp=sharing
Your problem is that the jedi-vim completion function is never called.
The error message is about pythoncomplete#Complete, which is VIM's default, but it should be set to :set omnifunc=jedi#completions.
I feel like the last line in scriptnames (D:\tools\v\indent\python.vim) overwrites omnifunc. Don't know why and how it happens, but that's my gut feeling.