I'm trying to install/import a cython dependent module [ZigZag] into my python function on Azure functions (Linux platfrom), I'm working via Windows 8.1 (VSCode), Running Python 3.6 on both local machine and azure functions.
I tried including Cython in the requirements.txt file, this errors out during deployment (seems installation of external modules happens somewhat in parallel)
I tried creating a '.toml' file, no luck.
I also tried the idea found here Python google cloud function deployment failure - Madmom pip package still didn't work.
Finally I decided to deploy the function first (with cython included in the 'requirements.txt' file) without the zigzag module, and then in my python script use subprocess.Popen() to do a pip install zigzag and import it before the rest of the function runs.
This is where I get stuck. In my python file I have these lines included to do the above:
#function to be called in azure __init__.py file
def f....:
# check for zigzag
try:
import zigzag
except ImportError:
subprocess.Popen(["pip", "install", "ZigZag"], shell=True)
finally:
import zigzag
.
.
.
rest of the function...
This deploys successfully to azure functions, however it fails during execution with the error:
Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException :
Result: Failure Exception: ModuleNotFoundError: No module named 'zigzag
Any idea's how to do this correctly?
NB: This issue occurs on google cloud functions as well.
I see this is related to https://learn.microsoft.com/en-us/answers/questions/152557/index.html
As per the thread, the latest version for ZigZag is 0.2.2 and the supported python version is 3.5: https://pypi.org/project/ZigZag/
The azure function supports 3.6 and above python version which will make it incompatible. You can try installing the latest cython using requirements.txt and it will work as expected. But when you try to install ZigZaG it tries to find the dependency of cython package (Cython==0.26.1) that is compatible with it therefor it throws the error that cython package not found.
I have created a deployment package for AWS Lambda with my python file and the dependencies including sqlalchemy and psycopg2. The code works perfectly in accessing the DB locally. But when I imported this zip file, I am getting the following error.
No module named 'psycopg2._psycopg': ModuleNotFoundError
The stack trace of the error is,
{
"errorMessage": "No module named 'psycopg2._psycopg'",
"errorType": "ModuleNotFoundError",
"stackTrace": [
[
"/var/task/DBAccessLamdaHandler.py",
50,
"lambda_handler",
"engine = create_engine(rds_host)"
],
[
"/var/task/sqlalchemy/engine/__init__.py",
387,
"create_engine",
"return strategy.create(*args, **kwargs)"
],
[
"/var/task/sqlalchemy/engine/strategies.py",
80,
"create",
"dbapi = dialect_cls.dbapi(**dbapi_args)"
],
[
"/var/task/sqlalchemy/dialects/postgresql/psycopg2.py",
554,
"dbapi",
"import psycopg2"
],
[
"/var/task/psycopg2/__init__.py",
50,
"<module>",
"from psycopg2._psycopg import ( # noqa"
]
]
}
Any help is appreciable
The AWS Lambda runtime environment doesn't include the PostgreSQL libraries so you need to include them within your AWS Lambda upload.
One way to do this is to get them from the jkehler/awslambda-psycopg2 repo at GitHub. Note that you don't need to build this project from scratch as the repo includes a pre-built package in the psycopg2 folder that you can simply include in your Lambda upload.
The psycopg2 build library from jkehler/awslambda-psycopg2 was built for python 3.6 and make sure that while uploading your code to AWS lambda, select Python Runtime environment as 3.6, and it should work. I banged my head on this for a full day and then when i changed to 3.6, the import error just vanished.
If you are going to attempt to build it yourself, remember that you must build on a machine or VM with the same architecture as your target at AWS.
Like other answers, psycopg2-binary worked fine for python3.9 (it looks like other package awslambda-psycopg2 was only available for python3.6).
But, if you run on MacOs before sending to aws lambda, you must specify the platform to your pip install like this:
pip3.9 install --platform=manylinux1_x86_64 --only-binary=:all: psycopg2-binary
Latest as of 26/MAR/2020
I was skeptical about depending on a third-party library for my production code. On research the following works,
The issue happens only when the packages are built from MAC OS.
I can confirm today that the issue is fixed when I build the package from Centos 7 ( AWS AMI )
The following is my approach
requirement.txt
psycopg2-binary==2.8.4
Build process
pip install -r requirements.txt --target .
Lambda code is in the root directory
+-- lambda_function.py
+-- psycopg2
+-- psycopg2 files
Zip the directory and test the code in lambda works.
The only additional step is to build the package in Linux env instead of macOS using the Docker container. An example can be found here: Deploy AWS Amplify Python Lambda from macOS with Docker
Thanks to AWS Lambda Layers we can include a ready compiled psycopg2 layer for our selected Python versions directly to our Lambda function. Please use the version you need from this Github repo. Be careful to use the correct Python version and Region in Lambda and layer creation.
For Windows (Looks like in any OS which is not build on Amazon AMI or Centos), the easiest fix is to use
psycopg2 Python Library for AWS Lambda
which you can find https://github.com/jkehler/awslambda-psycopg2
I've had issues getting SQLAlchemy to run on AWS Lambda despite the fact that I have tried multiple psycopg2 versions, but what eventually solved it for me was to use an older Python version. I went from Python 3.9 to 3.7 and it was finally able to run (Using psycopg2-binary 2.8.4, but I didn't try other versions or the none binary version with 3.7)
I have faced a similar issue. I tried to create the layers using AWS Cloud shell it worked with Python3.8. But failed in Python3.9 though.
Please change the region name as per your AWS Region
sudo amazon-linux-extras install python3.8
curl -O https://bootstrap.pypa.io/get-pip.py
python3.8 get-pip.py --user
mkdir python
python3.8 -m pip install --platform=manylinux1_x86_64 --only-binary=:all: pandas numpy pymysql psycopg2-binary SQLAlchemy -t python/
zip -r layer.zip python
aws lambda publish-layer-version \
--layer-name mics-layer \
--description "Pandas Numpy psycopg2-binary SQLAlchemy pymysql" \
--zip-file fileb://layer.zip \
--compatible-runtimes python3.8 python3.9 \
--region eu-west-1
instead of using psycopg2, try using pg8000
in command prompt go the directory that you are currently working(example F:\example)
pip install pg8000 -t. (-t. is for installing pg8000 in the directory that you are working)
// handler.py
import pg8000
database=''
host=''
port=''
user=''
password=''
conn = pg8000.connect(database=database, host=host, port=port, user=user,
password=password)
def lambda_function(event, context):
.
.
.
after writing your code zip the code and upload in your aws lambda.
It worked for me!!!
If you are using CDK you can use SQLAlchemy on AWS Lambda by bundling with Docker and providing the right pip install command:
Python 3.8.
sqlachemy_layer = lambda_.LayerVersion(
scope=self,
id=LAYERS_SQLALCHEMY_LAYER,
layer_version_name=LAYERS_SQLALCHEMY_LAYER,
code=lambda_.Code.from_asset(
str(pathlib.Path(__file__).parent.joinpath("dependencies").resolve()),
bundling=BundlingOptions(
image=lambda_.Runtime.PYTHON_3_8.bundling_image,
command=[
"bash", "-c",
"pip install -r requirements-sqlalchemy.txt -t /asset-output/python && cp -au . /asset-output/python",
]
)
),
compatible_runtimes=[
lambda_.Runtime.PYTHON_3_8,
],
)
Python 3.9
sqlachemy_layer = lambda_.LayerVersion(
scope=self,
id=LAYERS_SQLALCHEMY_LAYER,
layer_version_name=LAYERS_SQLALCHEMY_LAYER,
code=lambda_.Code.from_asset(
str(pathlib.Path(__file__).parent.joinpath("dependencies").resolve()),
bundling=BundlingOptions(
image=lambda_.Runtime.PYTHON_3_9.bundling_image,
command=[
"bash", "-c",
"pip3.9 install --platform=manylinux1_x86_64 --only-binary=:all: -r requirements-sqlalchemy.txt -t /asset-output/python && cp -au . /asset-output/python",
]
)
),
compatible_runtimes=[
lambda_.Runtime.PYTHON_3_9,
],
)
The requirements-sqlalchemy.txt:
psycopg2-binary
sqlalchemy
And the file structure:
dependencies/
requirements-sqlalchemy.txt
my_stack.py
I am trying to use python-ldap with AWS Lambda. I downloaded the tarball from : https://pypi.python.org/pypi/python-ldap
and code to use lambda (lambda_function.py)
from ldap_dir.ldap_query.Lib import ldap
and uploaded the zip to Lambda.
where my directory structure is
ldap_dir -> ldap_query -> Lib -> ldap folder
ldap_dir -> lambda_function.py
Am I missing out something?
python-ldap is built on top of native OpenLDAP libraries. This article - even though unrelated to the python ldap module - describes how to bundle Python packages that have native dependencies.
The outline of this is the following:
Create an Amazon EC2 instance with Amazon Linux
Install compiler packages as well as the OpenLDAP developer package. yum install -y gcc openldap-devel
Create a virtual environment: virtualenv env
Activate the virtual environment: env/bin/activate
Upgrade pip (I am not sure this is necessary, but I got a warning without this): pip install --upgrade pip
Install python-ldap: pip install python-ldap
Create a handler Python script, for example, lambda.py with the following code:
import os
import subprocess
libdir = os.path.join(os.getcwd(), 'local', 'lib')
def handler(event, context):
command = 'LD_LIBRARY_PATH={} python ldap.py'.format(libdir)
subprocess.call(command, shell=True)
Implement your LDAP function, in this example ldap.py:
import ldap
print ldap.PORT
Create a zip package, let's say ldap.zip:
zip -9 ~/ldap.zip ldap.py
zip -9 ~/ldap.zip lambda.py
cd env/lib/python2.7/site-packages
zip -r9 ~/ldap.zip *
cd ../../../lib64/python2.7/site-packages
zip -r9 ~/ldap.zip *
Download the zip to your system (or put it into an S3 bucket). Now you can create your Lambda function using lambda.handler as the function name and use the zip file as the code.
I hope this helps.
one more step/check to the solution above:
still you might get No module named '_ldap', then check if the python version that you install on local/EC2 are the same as the Runtime on lambda
I am using Python 2.7 through Anaconda 2.7.8 and need Kapteyn 2.2 to perform Non-linear Least Squares fitting easily (it is probably an alternative to Scipy.optimize.leastsq() for dummies like me!).
After copy-pasting this from a previous post here on Stack Overflow:
conda install -c https://conda.binstar.org/dhirschfeld pyodbc
and then running on my cmd (as I did not have pyodbc installed I think, because of which maybe the command prompt on my Windows 7 64-bit system was not responding well to python setup.py install inside the Anaconda directory where I unzipped the Kapteyn .zip file downloaded from University of Groningen website.
But, after the installing pyodbc properly and running python setup.py install, the cmd gave me an error saying error: command 'C:\Users\windows 7\Anaconda\Scripts\gcc.bat' failed with exit status 1. Later, when I tried to import kmpfit module (needed for Non-linear least square fitting with Kapteyn), here is the problem:
import kapteyn
help(kapteyn)
Help on package kapteyn:
NAME
kapteyn - Kapteyn package.
FILE
c:\users\windows 7\anaconda\kapteyn\__init__.py
PACKAGE CONTENTS
_ni_support
celestial
doccer
filters
interpolation
maputils
mplutil
positions
rulers
shapes
tabarray
wcsgrat
DATA
__all__ = ['celestial', 'wcs', 'wcsgrat', 'tabarray', 'maputils', 'mpl...
__version__ = '2.2'
VERSION
2.2
As you can see, there is no module named kmpfit (or even wcs) here. But according to http://www.astro.rug.nl/software/kapteyn/intro.html, these two should be there.
Kindly help. I have never imported any module before.
Thanks in advance...:-)
I just managed to get this working (on Mac OSX, so you may have to adjust this). My steps were:
$ conda install pyodbc (didn't need to go through binstar)
Download & unarchive the kapteyn package, then navigate to its directory
$ python setup.py install, which used my OS's C compiler and Anaconda's python, and installed kapteyn to my anaconda distro's site-packages, as it should.
Check that kmpfit.so is in the kapteyn folder in site-packages, showing that kmpfit installed correctly.
>> from kapteyn import kmpfit failed, ImportError: cannot import name kmpfit. I did some digging and discovered that it was still importing kapteyn from the folder that I downloaded, not from site-packages.
Delete the downloaded kapteyn folder, then try again. It worked!
from distutils.core import setup
import py2exe, sys, os
sys.argv.append('py2exe')
setup(
options = {'py2exe': {'bundle_files': 1}},
windows = [{'script': "single.py"}],
zipfile = None,
)
in this setup file for py2exe where it says single.py is that where I place the name of my program?
I don't know your py2exe tool, but we usually use this way to convert py to exe:
Download and install Standard Python Software:
http://www.python.org/download/
Download PyInstaller via link below:
http://pyinstaller.python-hosting.com/
Unpack the archive, that you have downloaded!
In this examople, the directory of the unpacked files:
In the <UNPACKED_FILES_DIR> directory, run Configure.py.
It must be run before trying to build anything.
Create a spec file for your project:
python Makespec.py -F -p <PYTHON_LIB_PATH> <PYTHON_SCRIPT>
-F: Produce a single file deployment.
-p <PYTHON_LIB_PATH>: Set base path for import (like using PYTHONPATH).
( e.g.: C:\Program Files\Python24\Lib\ )
<PYTHON_SCRIPT>: Path to python script.
6 Build your project!
python Build.py <SPECFILE>
<SPECFILE>: Path to the specfile, that have been created in step 4!
The full path to <SPECFILE>:
<UNPACKED_FILES_DIR>/<PYTHON_SCRIPT>/<PYTHON_SCRIPT>.spec
The binary file will be placed in the directory of <SPECFILE>.
If you can restrict your code, then Shed Skin, PyPy, or Cython make true, fast executables.
Py2exe, PyInstaller, or bbfreeze can package Python up to 2.7 into single executables.
Cx_Freeze packages Python up to 3.x into an executable plus many other files.
Yes. Are you making a windowing application or a console application? See the example setup.py files that came with py2exe.