So I am building a DOTNET application that runs on Debian, and makes use of ogr2ogr to copy data from an oracle database towards an Postgres database.
The problem is that I cannot get GDAL to recognize the OCI driver.
These are the installation commands that I have collected for now:
#Install dependencies used by GDAL and ora2pg
apt-get update && apt-get install -y -q --no-install-recommends \
libc-bin unzip curl ca-certificates rpm libaio1 \
#Package manager for installing Oracle
alien \
# Install postgresql
postgresql-client \
# Used for the POSTGRES_HOME variable
libpq-dev \
#Package manager used for installation of perl database drivers
cpanminus \
# Proj build
sqlite libsqlite3-dev pkg-config g++ make
#Install Oracle
curl -o oracle-instantclient-basic.x86_64.rpm https://download.oracle.com/otn_software/linux/instantclient/199000/oracle-instantclient19.9-basic-19.9.0.0.0-1.x86_64.rpm
curl -o oracle-instantclient-devel.x86_64.rpm https://download.oracle.com/otn_software/linux/instantclient/199000/oracle-instantclient19.9-devel-19.9.0.0.0-1.x86_64.rpm
curl -o oracle-instantclient-sqlplus.x86_64.rpm https://download.oracle.com/otn_software/linux/instantclient/199000/oracle-instantclient19.9-sqlplus-19.9.0.0.0-1.x86_64.rpm
alien -i oracle-instantclient-basic.x86_64.rpm && alien -i oracle-instantclient-devel.x86_64.rpm && alien -i oracle-instantclient-sqlplus.x86_64.rpm
EXPORT ORACLE_HOME=/usr/lib/oracle/19.9/client64
EXPORT TNS_ADMIN=/usr/lib/oracle/19.9/client64/network/admin
EXPORT LD_LIBRARY_PATH=/usr/lib/oracle/19.9/client64/lib
EXPORT PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/oracle/19.9/client64/bin
#Install Postgres en Oracle drivers for perl, ora2pg
cpanm DBD::Oracle
cpanm DBD::Pg
#Setup
wget https://download.osgeo.org/proj/proj-6.3.2.tar.gz
tar -zxf proj-6.3.2.tar.gz -C /opt/
/opt/proj-6.3.2/configure --prefix=/usr --disable-static --enable-lto
make -C /opt/proj-6.3.2/
make install -C /opt/proj-6.3.2/
RUN wget http://download.osgeo.org/gdal/3.2.2/gdal-3.2.2.tar.gz
/opt/gdal-3.2.2/configure
make -C /opt/proj-6.3.2/
make install -C /opt/proj-6.3.2/
Is there anyone who can tell me what I am missing, because I cannot find any answers on the internet...
So after a lot off testing, my colleague found the problem.
Apparently the scripts search for a folder in $ORACLE_HOME/sdk.
Now by installing it, like above, it doesn't install the sdk folder on the correct location.
So we solved it by adding an extra step extracting the SDK zip package on the correct location.
This is the result:
Dockerfile
Above solution did not work for me. I needed a GDAl container with OCI. Based on some other info I found I build an image that works.
https://github.com/botenvouwer/gdal-oci
Currently you have to clone and build the image yourself. You should (but don't have to) pass the version to your docker build as ARG GDAL_VERSION.
Related
I'm trying to setup a docker file that uses the x86_64 centos:centos7 docker image to cross compile to aarch64. The only problem is I'm missing libgcc_s.so for cross compiling, and I'm not sure why. I also noticed that there were no official glibc-aarch64 so I went with Computational's packages. I've provided my docker file, and any help would be appreciated.
FROM centos:centos7
RUN yum -y update \
&& yum -y install epel-release \
&& yum -y install clang gcc gcc-c++ make wget \
&& yum -y install gcc-aarch64-linux-gnu gcc-c++-aarch64-linux-gnu
RUN yum install -y http://springdale.princeton.edu/data/springdale/7/x86_64/os/Computational/glibc-aarch64-linux-gnu-2.24-2.sdl7.2.noarch.rpm
RUN yum install -y http://springdale.princeton.edu/data/springdale/7/x86_64/os/Computational/glibc-aarch64-linux-gnu-devel-2.24-2.sdl7.2.noarch.rpm
CMD ["/bin/bash"]
I'm not sure on the exact reason why, but gcc-aarch64-linux-gnu and gcc-c++-arch64-linux-gnu were compiled with --disable-shared. This prevents them from building libgcc_s.so. My best guess on why is because the shared library isn't always used and it's more common to use libgcc.a.
The two solutions that I had were to either build libgcc from source, or to copy the libgcc_s.so from another repo with the same version. I ended up extracting libgcc_s.so from libgcc-4.8.5-44.el7.aarch64.rpm.
Is there a way to use gifsicle in AWS lambda?
I know there is a package called pygifsicle, but it seems it requires the gifsicle version of AWS Linux 2?
I don't see a binary built for RedHat version of gifsicle
So my questions are,
Do I need to build one for AWS Linux 2 to use it along with pygifsicle?
Even if I build gifsicle for AWS Linux 2, how to use it along with pygifsicle?
As I read the documentation you can build one binary for Building Gifsicle on UNIX and can package that with your lambda zip file which can be called as a normal command in lambda function.
Like it is being called in the pygifsicle
subprocess.call(["gifsicle", *options, *sources, "--colors",
str(colors), "--output", destination])
My Dockerfile where I'm building it from the source.
FROM public.ecr.aws/lambda/python:3.8-arm64
RUN yum -y install install make gcc wget gzip
RUN wget https://www.lcdf.org/gifsicle/gifsicle-1.93.tar.gz
RUN tar -xzf gifsicle-1.93.tar.gz
RUN cd gifsicle-1.93 && \
./configure && \
make && \
make install
COPY requirements.txt ./
RUN yum update -y && \
pip install -r requirements.txt
COPY . .
CMD ["app.handler"]
I'm trying to create a new lambda layer to import the zip file with psycopg2, because the library made my deployment package get over 3MB, and I can not see the inline code in my lambda function any more.
I created lambda layer for the following 2 cases with Python 3.7:
psycopg2_lib.zip (contains psycopg2, psycopg2_binary.libs and psycopg2_binary-2.8.5.dist-info folders)
psycopg2_only.zip which contains only the psycopg2 folder.
I added they new created layer into my lambda function.
But, in both cases, my lambda_function throws an error as follows:
{
"errorMessage": "Unable to import module 'lambda_function': No module named 'psycopg2'",
"errorType": "Runtime.ImportModuleError"
}
The error seems as if something went wrong with my zip file that they are not recognized. But when it works well in my deployment package.
Any help or reason would be much appriciated. Thanks!
not sure if the OP found a solution to this but in case others land here. I resolved this using the following steps:
download the code/clone the git from:
https://github.com/jkehler/awslambda-psycopg2
create the following directory tree, if building for python3.7, otherwise replace 'python3.7' with the version choice:
mkdir -p python/lib/python3.7/site-packages/psycopg2
choose the python version of interest and copy the files from the folders downloaded in step 1. to the directory tree in step 2. e.g. if building a layer for python 3.7:
cp psycopg2-3.7/* python/lib/python3.7/site-packages/psycopg2
create the zip file for the layer. e.g.: zip -r9 psycopg2-py37.zip python
create a layer in the console or cli and upload the zip
If you end up on this page in >= 2022 year. Use official psycopg2-binary https://pypi.org/project/psycopg2-binary/
Works well for me. Just
pip install --target ./python psycopg2-binary
zip -r python.zip python
Maintainers of psycopg2 do not recommend using psycopg2-binary because it comes with linked libpq and libssl and others that may cause issues in production under certain circumstances.
I may imagine this being an issue when upgrading postgresql server while bundled libpq is incompatible. I also had issues w/ psycopg2-binary on AWS Lambda running in arm64 environment.
I've resorted to building postgresql and psycopg in Docker running on linux/arm64 platform using public.ecr.aws/lambda/python:3.9 as the base image.
FROM public.ecr.aws/lambda/python:3.9
RUN yum -y update && \
yum -y upgrade && \
yum -y install libffi-devel postgresql-devel postgresql-libs zip rsync wget openssl openssl-devel && \
yum -y groupinstall "development tools" && \
pip install pipenv
ENTRYPOINT ["/bin/bash"]
The build script is the following and valid for aarch64 platform. Just change path to x86_64 version on Prepare psycopg2 step.
#!/usr/bin/env bash
set -e
PG_VERSION="14.5"
cd "$TERRAFORM_ROOT"
if [ ! -f "postgresql-$PG_VERSION.tar.bz2" ]; then
wget "https://ftp.postgresql.org/pub/source/v$PG_VERSION/postgresql-$PG_VERSION.tar.bz2"
tar -xf "$(pwd)/postgresql-$PG_VERSION.tar.bz2"
fi
if [ ! -d "psycopg2" ]; then
git clone https://github.com/psycopg/psycopg2.git
fi
# Build postgres
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION"
./configure --without-readline --without-zlib
make
make install
# Build psycopg2
cd "$TERRAFORM_ROOT/psycopg2"
make clean
python setup.py build_ext \
--pg-config "$TERRAFORM_ROOT/postgresql-14.5/src/bin/pg_config/pg_config"
# Prepare psycopg2
cd build/lib.linux-aarch64-3.9
mkdir -p python/
cp -r psycopg2 python/
zip -9 -r "$BUNDLE" ./python
# Prepare libpq
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION/src/interfaces/libpq/"
mkdir -p lib/
cp libpq.so.5 lib/
zip -9 -r "$BUNDLE" ./lib
where $BUNDLE is the path to already existing .zip file.
I also tried to statically build psycopg2 binary and link libpq.a, however, I have had quite a lot of issues with missing symbols.
From AWS post How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?:
To create a Lambda deployment package or layer that's compatible with Lambda Python runtimes when using pip outside of Linux operating system, run the pip install command with manylinux2014 as the value for the --platform parameter.
pip install \
--platform manylinux2014_x86_64 \
--target=my-lambda-function \
--implementation cp \
--python 3.9 \
--only-binary=:all: --upgrade \
psycopg2-binary
You can then zip the content of directory my-lambda-function
First of all, I have to say that I am not very experienced in ubuntu.
I would like to install this library to use with FFMPEG.
I am following these steps, but I can manage to install it well...
https://github.com/Netflix/vmaf/blob/master/resource/doc/VMAF_Python_library.md
Could someone tell me what are the exact steps (commands) that I have to follow.
On the other hand, someone knows if there are other metrics that can ffmpeg calculates directly (apart from PSNR or SSIM)?
Many thanks
The link you provided is just to install the VMAF python library. To use VMAF with ffmpeg you need to install libvmaf first and then compile ffmpeg with libvmaf filter enabled. However, as one user suggests, you can use already compiled ffmpeg packages https://www.johnvansickle.com/ffmpeg/
Moving to your questions. These are the step by step I followed to install FFmpeg+VMAF over an Ubuntu 20.04:
Installing dependencies:
apt update -qq
apt install pkg-config -y
apt-get install --no-install-recommends\
ninja-build \
python3 \
python3-pip \
python3-setuptools \
python3-wheel \
ninja-build \
wget \
doxygen \
autoconf \
automake \
cmake \
g++ \
gcc \
pkg-config \
make \
nasm \
yasm -y
pip3 install --user meson
Installing libvmaf:
export PATH="$PATH:$HOME/.local/bin"
wget https://github.com/Netflix/vmaf/archive/v1.5.1.tar.gz
tar -xzf v1.5.1.tar.gz
cd vmaf-1.5.1/libvmaf/
meson build --buildtype release
ninja -vC build
ninja -vC build test
ninja -vC build install
Installing ffmpeg:
wget https://ffmpeg.org/releases/ffmpeg-4.2.2.tar.bz2 && tar xjf ffmpeg-4.2.2.tar.bz2
cd ffmpeg-4.2.2
./configure --enable-libvmaf --enable-version3
make
make install
Sometimes, depending on your OS, you also need to setup your $PATH, $LD_LIBRARY_PATH, $PKG_CONFIG_PATH.
Download an already compiled ffmpeg from johnvansickle.com. It supports the libvmaf filter. For instructions on how to install it see the FAQ.
Here is a detailed guide on OTTVerse for compiling and installing FFmpeg with VMAF support on Ubuntu 18.04. It shouldn't take you more than a few minutes.
Be aware, that VMAF's libvmaf library will soon be dropped and replaced with a new library called libvmaf_rc. Hopefully, its compilation with FFmpeg will remain smooth as well.
sudo apt-get install nasm doxygen ninja-build meson
Download VMAF source from here and extract it.
cd into the vmaf source code folder.
cd into the libvmaf folder.
meson build --buildtype release
ninja -vC build
ninja -vC build install
Download ffmpeg source code from here.
Extract the source code and cd into the ffmpeg source code folder.
./configure --enable-gpl --enable-libx264 --enable-libx265 --enable-nonfree --enable-libvmaf --enable-version3
sudo make
sudo make install
Add the following export line to ~/.bashrc or ~/.bash_profile: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/x86_64-linux-gnu/.
Run source ~/.bashrc or source ~/.bash_profile.
Attribution.
i need to connect adminer with oracle but its not supported in the official image. is there any adminer image for oracle connectivity? or if someone can guide me on how to install oci8 extension in current image.
You can use the following image in order to connect to oracle
https://hub.docker.com/r/soivangoi/nginx-php-adminer-oci8
for the server to use, please note that the syntax is the following :
host_ip:port/service_name (as described in .tnsnames.ora)
Dockerfile:
FROM adminer:4.8.1
ENV LD_LIBRARY_PATH /usr/local/instantclient_21_1
ENV ORACLE_HOME /usr/local/instantclient_21_1
USER root
RUN apk update && apk upgrade --available \
&& apk add --no-cache bash autoconf build-base composer libaio libnsl libc6-compat busybox-extras
ADD tmp/. /tmp/.
RUN unzip -d /usr/local/ /tmp/instantclient-basic-linux.x64-21.1.0.0.0.zip
RUN unzip -d /usr/local/ /tmp/instantclient-sdk-linux.x64-21.1.0.0.0.zip
RUN unzip -d /usr/local/ /tmp/instantclient-sqlplus-linux.x64-21.1.0.0.0.zip
RUN ln -s /usr/lib/libnsl.so.2 /usr/lib/libnsl.so.1
RUN ln -s /lib/libc.so.6 /usr/lib/libresolv.so.2
RUN ln -s /lib64/ld-linux-x86-64.so.2 /usr/lib/ld-linux-x86-64.so.2
RUN pear upgrade --force && pecl upgrade
ADD tmp/instantclient.ini /etc/php.d/instantclient.ini
RUN docker-php-ext-configure oci8 --with-oci8=instantclient,$ORACLE_HOME
RUN docker-php-ext-install oci8
the oracle instant clien download URL:
https://download.oracle.com/otn_software/linux/instantclient/211000/instantclient-basic-linux.x64-21.1.0.0.0.zip
https://download.oracle.com/otn_software/linux/instantclient/211000/instantclient-sqlplus-linux.x64-21.1.0.0.0.zip
https://download.oracle.com/otn_software/linux/instantclient/211000/instantclient-sdk-linux.x64-21.1.0.0.0.zip
the instantclient.ini file content is extension=oci8.so