MonetDB set specific embedded python version - monetdb

In a Linux environment, I'd like to be able to set the embedded python3 to a specific python version. My current monetDB install seems to use 3.7 even though all default python paths point to 3.8.6.
Is there a way to set my 3.8.6 install as the default embedded python3 version?

After some experimentation, the problem seems to be the following: because you are installing MonetDB using apt, the installation pulls the python version that the distribution has packaged in apt. I assume that you are building your container starting from python:3.8. This is based on Debian Buster that packages Python 3.7.
In a bit more detail:
I built a container using the following Dockerfile:
FROM python:3.8
RUN apt-get update
RUN apt-get upgrade -y
COPY ./monetdb.list /etc/apt/sources.list.d/
COPY ./MonetDB-GPG-KEY /
RUN apt-key add /MonetDB-GPG-KEY
RUN apt-get update
RUN apt-get install -y monetdb5-sql monetdb-client monetdb-python3
with the files monetdb.list and MonetDB-GPG-KEY as described in the downloads page.
I then created a database farm and a new database in the container, setting embedpy3 to true:
root#bd0420e945e8:/# monetdbd create /tmp/dbfarm
root#bd0420e945e8:/# monetdbd start /tmp/dbfarm
root#bd0420e945e8:/# monetdb create -p monetdb pytestdb
created database with password for monetdb user: pytestdb
root#bd0420e945e8:/# monetdb set embedpy3=yes pytestdb
Using mclient I created a Python UDF that returns as a string the version of the embedded python interpreter:
CREATE FUNCTION pyversion ()
RETURNS STRING
LANGUAGE python {
import sys
return sys.version
};
When I called it I got the following result:
sql>select pyversion();
+-----------------------------------------------------+
| %2 |
+=====================================================+
| 3.7.3 (default, Jul 25 2020, 13:03:44) |
: [GCC 8.3.0] :
+-----------------------------------------------------+
1 tuple
I repeated the above process starting from a different distribution (Ubuntu Focal) that packages a different version of Python (It seems that Ubuntu has updated to Python 3.8.5 since the release of Focal).
FROM ubuntu:20.04
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y gnupg ca-certificates
COPY ./monetdb.list /etc/apt/sources.list.d/
COPY ./MonetDB-GPG-KEY /
RUN apt-key add /MonetDB-GPG-KEY
RUN apt-get update
RUN apt-get install -y monetdb5-sql monetdb-client monetdb-python3
In this container the function pyversion returns the following:
sql>select pyversion();
+-----------------------------------------------------+
| %2 |
+=====================================================+
| 3.8.5 (default, Jul 28 2020, 12:59:40) |
: [GCC 9.3.0] :
+-----------------------------------------------------+
1 tuple
In conclusion, I would suggest to either build MonetDB from source in the container (see the README for instructions), or build your container starting from a distribution that packages Python 3.8 in its package manager.

Related

Automate installation of package maintainer's version during Elasticsearch upgrade

I am running a shell script that installs elasticsearch. I was previously running 7.x on the ec2 instance and I'm upgrading to version 8.
When I run the script, I get the message that a configuration file has changed with the below instructions.
Since I can't select anything, it defaults to keeping my current version and that's not what I want.
I want to install the package maintainer's version.How can I run my script to automate the process?
Instruction that pops up during installation
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
**deployment script that handles elasticsearch installation **
Yes | Sudo DEBIAN_FRONTEND=noninteractive apt-get install elasticsearch
Yes|Sudo apt-get
I tried adding a yes to my sudo apt-get hoping to automate every interaction. I also turned off Debian interaction.
Yes | Sudo DEBIAN_FRONTEND=noninteractive apt-get install elasticsearch

How to control which Python version plpython3 uses?

I have a postgres 11 database running on CentOS 7 and am trying to use PL/python, but need to point to a specific version of the python interpreter. I need to use python3.9 whereas it is using python3.6. I am not able to uninstall python3.6 at the moment, so was hoping to point plpython3 at my python3.9 interpreter somehow.
I have installed the package postgresql-plpython3:
sudo yum install -y postgresql-plpython3
I have created an extension in PG and can run a simple PL/python function that reports the version used:
CREATE OR REPLACE FUNCTION pyver ()
RETURNS TEXT
AS $$
import sys
pyversion = sys.version
return pyversion
$$ LANGUAGE 'plpython3u';
Executing it returns this:
# select pyver();
pyver
-----------------------------------------
3.6.8 (default, Nov 16 2020, 16:55:22) +
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
(1 row)
On my operating systems I have the following python interpreter files, and the versions they are using:
$ /usr/bin/python3 --version
Python 3.9.6
$ python3 --version
Python 3.9.6
$ python3.6 --version
Python 3.6.8
Something within the plpython3 module seems to be hard-coded to point to my python3.6 version.
postgresql uses an embedded python interpreter, as stated here. Changing it requires re-compiling the pl/python extension (afaik). Compilation instructions here show the relevant bits: the option --with-python and the env var PYTHON

Cannot run yum due to libssl.so.10: cannot open shared object file: No such file or directory

I inadvertently ran the following command in AWS EC2 Lightsail instance
rpm --nodeps -e openssl-1.0.2k-16.150.amzn1.x86_64
and ever since I am unable to run any yum commands
[root#ip-172-26-3-161 abc]# yum update
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:
libssl.so.10: cannot open shared object file: No such file or directory
Please install a package which provides this module, or
verify that the module is installed correctly.
It's possible that the above module doesn't match the
current version of Python, which is:
2.7.16 (default, Oct 14 2019, 21:26:56)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
If you cannot solve this problem yourself, please go to
the yum faq at:
http://yum.baseurl.org/wiki/Faq
Any pointers on how to recover from this
You should find the original version of openssl, download it from amazon repo and install it with rpm command. And next time use command yum localinstall ... to install local packages.

Issue using M2Crypto on lambda (works on EC2)

I am trying to install a python function using M2Crypto in AWS Lambda.
I spun up an EC2 instance with the Lambda AMI image, installed M2Crypto into a virtualenv, and was able to get my function working on EC2.
Then I zipped up the site-package and uploaded to Lambda. I got this error
Unable to import module 'epd_M2Crypto':
/var/task/M2Crypto/_m2crypto.cpython-36m-x86_64-linux-gnu.so: symbol
sk_deep_copy, version libcrypto.so.10 not defined in file
libcrypto.so.10 with link time reference
There are similar questions and hints here and here. I tried uploading the offending lib (libcrypto.so.10) in the zip file, but still get the same error. I am assuming the error means that the EC2 version of libcrypto.so.10 (used to install M2Crypto) is different than the version on Lambda (that I trying to run with), so M2Crypto complains.
If I look at the versions of openssl they are different:
OpenSSL 1.0.0-fips 29 Mar 2010 (lambda version)
OpenSSL 1.0.2k-fips 26 Jan 2017 (ec2 version)
I don't think the answer is to downgrade openssl on ec2 as the 1.0.0 version is obsolete (AWS applies security patches but the version still shows as 1.0.0). (Also the yum doesn't have versions this old)
Here's the steps i used on the EC2 instance to get it working on EC2:
$ sudo yum -y update
$ sudo yum -y install python36
$ sudo yum -y install python-virtualenv
$ sudo yum -y groupinstall "Development Tools"
$ sudo yum -y install python36-devel.x86_64
$ sudo yum -y install openssl-devel.x86_64
$ mkdir ~/forlambda
$ cd ~/forlambda
$ virtualenv -p python3 venv
$ source venv/bin/activate
$ cd ~
$ pip install M2Crypto -t ~/forlambda/venv/lib/python3.6/site-packages/
$ cd ~/forlambda/venv/lib/python3.6/site-packages/
$ (create python function that uses M2Crypto)
$ zip -r9 ~/forlambda/archive.zip .
Then added to the zip file
/usr/bin/openssl
/usr/lib64/libcrypto.so.10
/usr/lib64/libssl.so.10
And uploaded to Lambda, which is where I am now stuck.
Do I need to do something to get Lambda to use the version of libcrypto.so.10 that I have included in the uploaded zip?
My function:
"""
Wrapper for M2Crypto
https://github.com/mcepl/M2Crypto
https://pypi.org/project/M2Crypto/
"""
from __future__ import print_function
from M2Crypto import RSA
import base64
import json
def decrypt_string(string_b64):
rsa = RSA.load_key('private_key.pem')
string_encrypted = base64.b64decode(string_b64)
bytes = rsa.private_decrypt(string_encrypted, 1)
string_plaintext = bytes.decode("utf-8")
response = {
's': string_plaintext,
'status': "OK",
'statuscode': 200
};
return response
def lambda_handler(event, context):
response = ""
action = event['action']
if action == "decrypt":
string_b64 = event['s']
response = decrypt_string(string_b64)
return response
AWS support provided a resolution, upgrading to use Python 3.7 where the issue is resolved:
Our internal team has confirmed that the issue is with Lambda's Python
runtime. In a few rare cases, when the Lambda function is being
initialised, Lambda is unable to link against the correct OpenSSL
libraries - instead linking against Lambda's own in-built OpenSSL
binaries.
The team suggests trying this out in the Python3.7 environment where
this behaviour has been fixed. Also, python3.7 is compiled with the
newer openssl 1.0.2 and you should not have to include the binaries in
the Lambda package. ... still had to include the OpenSSL binaries in
the package and could not get it working with the default libraries.
First I ran this command on the EC2 instance to make sure I had included the correct .so file in my .zip:
$ ldd -v _m2crypto.cpython-36m-x86_64-linux-gnu.so
The output of the ldd command (edited for brevity):
libssl.so.10 => /lib64/libssl.so.10 (0x00007fd5f1892000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fd5f1433000)
Based on the output above, I included /lib64/libcrypto.so.10 in my .zip.
Also (at the suggestion of AWS Support), on the Lambda console, under 'Environment variables', I added a key 'LD_LIBRARY_PATH' with value '/var/task'.
I'm not sure if I needed both those changes to fix my problem, but it works right now and after three days of troubleshooting I am afraid to touch it to see if it was one or the other that made it work.
It is perhaps too brutal, but would it be possible to use LD_PRELOAD to force using your preferred version of OpenSSL library?
AWS lambda runs code on an old version of amazon linux (amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2) as mentioned in the official documentation
https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
So to run a code that depends on shared libraries, it needs to be compiled in the same environment so it can link correctly.
What I usually do in such cases is that I create virtualenv using docker container. The virtualenv can than be packaged with lambda code.
Please note that if you need install anything using yum (in the docker container), you must use same release server as the amazon linux version:
yum --releasever=2017.03 install ...
virtualenv can be built using an EC2 instance as well instead of docker container (though, I find docker method easier). Just make sure that the AMI used for EC2 is same as the one used by lambda.

build caffe2 with conda failed

I build the caffe2 with anaconda following the page.
In the server with a single titanx, has cudnn7 and cuda9 but do not have nccl, so I download the nccl2 from nvidia and extract it to path/to/local/nccl2, and then edit the ./pytorch/conda/integrated/build.sh in the line 42 to be:"export NCCL_ROOT_DIR=path/to/local/nccl2".
Then I need to use caffe2 with python2, so I added "conda_args+=(" --python 2.7") " in the ./pytorch/scripts/build_anaconda.sh to use python2.7.
The building was succeed, but when I run python2 test.py from caffe2.python import core
It tells me:
WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
WARNING:root:Debug message: No module named caffe2_pybind11_state_hip
Segmentation fault (core dumped)
My question is:
a. why the conda does not support gpu?
b. if I am using a single gpu, is nccl necessary for building?
c. how to fix No module named caffe2_pybind11_state_hip
PyTorch or Caffe2: caffe2
How you installed PyTorch (conda, pip, source): conda
Build command you used (if compiling from source):./scripts/build_anaconda.sh --install-locally --cuda 9.0 --cudnn 7
OS:ubuntu16
PyTorch version:
Python version:2.7
CUDA/cuDNN version:9.1/7
GPU models and configuration:??
GCC version (if compiling from source):5.4.0
CMake version:not install
Versions of any other relevant libraries:
Thank you very much!
First of all get CUDA and install it:
sudo apt-get update && sudo apt-get install wget -y --no-install-recommends
wget "http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb"
sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
sudo apt-get update
sudo apt-get install cuda
Now proceed with installation from source ( do it in an environment):
FULL_CAFFE2=1 python setup.py install
You can find more info here: https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=compile#install-with-gpu-support
Follow the below procedure it worked for me
ubuntu#test:~$ cd $HOME
ubuntu#test:~$ conda create -n caffe2
ubuntu#test:~$ source activate caffe2
(caffe2) ubuntu#test:~$ git clone --recursive https://github.com/pytorch/pytorch.git && cd pytorch
(caffe2) ubuntu#test:~/pytorch$ git submodule update --init
(caffe2) ubuntu#test:~/pytorch$ CONDA_INSTALL_LOCALLY=1 ./scripts/build_anaconda.sh --cuda 8.0 --cudnn 7 -DUSE_CUDA=ON -DUSE_NCCL=ON

Resources