So I am building a DOTNET application that runs on Debian, and makes use of ogr2ogr to copy data from an oracle database towards an Postgres database.
The problem is that I cannot get GDAL to recognize the OCI driver.
These are the installation commands that I have collected for now:
#Install dependencies used by GDAL and ora2pg
apt-get update && apt-get install -y -q --no-install-recommends \
libc-bin unzip curl ca-certificates rpm libaio1 \
#Package manager for installing Oracle
alien \
# Install postgresql
postgresql-client \
# Used for the POSTGRES_HOME variable
libpq-dev \
#Package manager used for installation of perl database drivers
cpanminus \
# Proj build
sqlite libsqlite3-dev pkg-config g++ make
#Install Oracle
curl -o oracle-instantclient-basic.x86_64.rpm https://download.oracle.com/otn_software/linux/instantclient/199000/oracle-instantclient19.9-basic-19.9.0.0.0-1.x86_64.rpm
curl -o oracle-instantclient-devel.x86_64.rpm https://download.oracle.com/otn_software/linux/instantclient/199000/oracle-instantclient19.9-devel-19.9.0.0.0-1.x86_64.rpm
curl -o oracle-instantclient-sqlplus.x86_64.rpm https://download.oracle.com/otn_software/linux/instantclient/199000/oracle-instantclient19.9-sqlplus-19.9.0.0.0-1.x86_64.rpm
alien -i oracle-instantclient-basic.x86_64.rpm && alien -i oracle-instantclient-devel.x86_64.rpm && alien -i oracle-instantclient-sqlplus.x86_64.rpm
EXPORT ORACLE_HOME=/usr/lib/oracle/19.9/client64
EXPORT TNS_ADMIN=/usr/lib/oracle/19.9/client64/network/admin
EXPORT LD_LIBRARY_PATH=/usr/lib/oracle/19.9/client64/lib
EXPORT PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/oracle/19.9/client64/bin
#Install Postgres en Oracle drivers for perl, ora2pg
cpanm DBD::Oracle
cpanm DBD::Pg
#Setup
wget https://download.osgeo.org/proj/proj-6.3.2.tar.gz
tar -zxf proj-6.3.2.tar.gz -C /opt/
/opt/proj-6.3.2/configure --prefix=/usr --disable-static --enable-lto
make -C /opt/proj-6.3.2/
make install -C /opt/proj-6.3.2/
RUN wget http://download.osgeo.org/gdal/3.2.2/gdal-3.2.2.tar.gz
/opt/gdal-3.2.2/configure
make -C /opt/proj-6.3.2/
make install -C /opt/proj-6.3.2/
Is there anyone who can tell me what I am missing, because I cannot find any answers on the internet...
So after a lot off testing, my colleague found the problem.
Apparently the scripts search for a folder in $ORACLE_HOME/sdk.
Now by installing it, like above, it doesn't install the sdk folder on the correct location.
So we solved it by adding an extra step extracting the SDK zip package on the correct location.
This is the result:
Dockerfile
Above solution did not work for me. I needed a GDAl container with OCI. Based on some other info I found I build an image that works.
https://github.com/botenvouwer/gdal-oci
Currently you have to clone and build the image yourself. You should (but don't have to) pass the version to your docker build as ARG GDAL_VERSION.
I'm trying to create a new lambda layer to import the zip file with psycopg2, because the library made my deployment package get over 3MB, and I can not see the inline code in my lambda function any more.
I created lambda layer for the following 2 cases with Python 3.7:
psycopg2_lib.zip (contains psycopg2, psycopg2_binary.libs and psycopg2_binary-2.8.5.dist-info folders)
psycopg2_only.zip which contains only the psycopg2 folder.
I added they new created layer into my lambda function.
But, in both cases, my lambda_function throws an error as follows:
{
"errorMessage": "Unable to import module 'lambda_function': No module named 'psycopg2'",
"errorType": "Runtime.ImportModuleError"
}
The error seems as if something went wrong with my zip file that they are not recognized. But when it works well in my deployment package.
Any help or reason would be much appriciated. Thanks!
not sure if the OP found a solution to this but in case others land here. I resolved this using the following steps:
download the code/clone the git from:
https://github.com/jkehler/awslambda-psycopg2
create the following directory tree, if building for python3.7, otherwise replace 'python3.7' with the version choice:
mkdir -p python/lib/python3.7/site-packages/psycopg2
choose the python version of interest and copy the files from the folders downloaded in step 1. to the directory tree in step 2. e.g. if building a layer for python 3.7:
cp psycopg2-3.7/* python/lib/python3.7/site-packages/psycopg2
create the zip file for the layer. e.g.: zip -r9 psycopg2-py37.zip python
create a layer in the console or cli and upload the zip
If you end up on this page in >= 2022 year. Use official psycopg2-binary https://pypi.org/project/psycopg2-binary/
Works well for me. Just
pip install --target ./python psycopg2-binary
zip -r python.zip python
Maintainers of psycopg2 do not recommend using psycopg2-binary because it comes with linked libpq and libssl and others that may cause issues in production under certain circumstances.
I may imagine this being an issue when upgrading postgresql server while bundled libpq is incompatible. I also had issues w/ psycopg2-binary on AWS Lambda running in arm64 environment.
I've resorted to building postgresql and psycopg in Docker running on linux/arm64 platform using public.ecr.aws/lambda/python:3.9 as the base image.
FROM public.ecr.aws/lambda/python:3.9
RUN yum -y update && \
yum -y upgrade && \
yum -y install libffi-devel postgresql-devel postgresql-libs zip rsync wget openssl openssl-devel && \
yum -y groupinstall "development tools" && \
pip install pipenv
ENTRYPOINT ["/bin/bash"]
The build script is the following and valid for aarch64 platform. Just change path to x86_64 version on Prepare psycopg2 step.
#!/usr/bin/env bash
set -e
PG_VERSION="14.5"
cd "$TERRAFORM_ROOT"
if [ ! -f "postgresql-$PG_VERSION.tar.bz2" ]; then
wget "https://ftp.postgresql.org/pub/source/v$PG_VERSION/postgresql-$PG_VERSION.tar.bz2"
tar -xf "$(pwd)/postgresql-$PG_VERSION.tar.bz2"
fi
if [ ! -d "psycopg2" ]; then
git clone https://github.com/psycopg/psycopg2.git
fi
# Build postgres
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION"
./configure --without-readline --without-zlib
make
make install
# Build psycopg2
cd "$TERRAFORM_ROOT/psycopg2"
make clean
python setup.py build_ext \
--pg-config "$TERRAFORM_ROOT/postgresql-14.5/src/bin/pg_config/pg_config"
# Prepare psycopg2
cd build/lib.linux-aarch64-3.9
mkdir -p python/
cp -r psycopg2 python/
zip -9 -r "$BUNDLE" ./python
# Prepare libpq
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION/src/interfaces/libpq/"
mkdir -p lib/
cp libpq.so.5 lib/
zip -9 -r "$BUNDLE" ./lib
where $BUNDLE is the path to already existing .zip file.
I also tried to statically build psycopg2 binary and link libpq.a, however, I have had quite a lot of issues with missing symbols.
From AWS post How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?:
To create a Lambda deployment package or layer that's compatible with Lambda Python runtimes when using pip outside of Linux operating system, run the pip install command with manylinux2014 as the value for the --platform parameter.
pip install \
--platform manylinux2014_x86_64 \
--target=my-lambda-function \
--implementation cp \
--python 3.9 \
--only-binary=:all: --upgrade \
psycopg2-binary
You can then zip the content of directory my-lambda-function
I am trying to use Protobuf on Linux box. I downloaded the pre-compiled from github.
When I try to compile my .proto file or just check the protobuf version, it says
protoc: command not found.
I tried the same steps on Windows machine using pre-compiled protobuf version and it works fine there.
Install protoc for Linux and Mac
Linux
PROTOC_ZIP=protoc-3.15.8-linux-x86_64.zip
curl -OL https://github.com/google/protobuf/releases/download/v3.15.8/$PROTOC_ZIP
sudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc
sudo unzip -o $PROTOC_ZIP -d /usr/local include/*
rm -f $PROTOC_ZIP
Mac OS X
brew install protobuf
Alternately, if you don't have Homebrew.
PROTOC_ZIP=protoc-3.15.8-osx-x86_64.zip
curl -OL https://github.com/google/protobuf/releases/download/v3.15.8/$PROTOC_ZIP
sudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc
sudo unzip -o $PROTOC_ZIP -d /usr/local 'include/*'
rm -f $PROTOC_ZIP
source: http://google.github.io/proto-lens/installing-protoc.html
For Linux Ubuntu 20, only install with snap
snap install protobuf --classic
or via apt, with:
sudo apt install protobuf-compiler
You can try it:
Install grpc and protobuf
brew install grpc protobuf
I know this question is specifically asked for Linux and I could n't find any question as it relates to solving this error on Windows.
This might help people who encounter the same error on windows.
Step 1: Download the windows distribution (protoc-3.5.0-win32.zip) from the link protobuf and unzip locally to a folder
Step 2 : Add the folder path to the path in the system environment variables.
Step 3: close the command prompt, restart the command prompt and try the command protoc
I might be late to the party but I also had "command not found" when trying to run protoc. It turned out it was just missing the execute permission. A quick chmod +x protoc fixed this for me.
Symfony provides a wrapper/install, or you can use Composer. Is there a way to get the installer to run in Cygwin?
The easiest way I found to complete this was to install a Cygwin package manager, either apt-cyg or Sage (I'm using Sage since apt-cyg is no longer being actively worked on).
$ lynx -source rawgit.com/svnpenn/sage/master/sage > sage
$ install sage /bin
Next we will install the packages required for both the Composer route and the Installer route.
$ sage update
$ sage install wget curl sed
$ sage install php php-json php-phar php-mbstring php-iconv php-zlib php-ctype php-tokenizer php-simplexml
We should now have enough for Composer.
$ mkdir -p ~/tmp
$ cd ~/tmp
$ wget https://getcomposer.org/installer
We can double check by asking the Installer.
$ php installer --check
If all is good, proceed
$ php installer
$ rm installer
$ chmod a+x composer.phar
Install Composer globally (put it in PATH)
$ mv composer.phar /usr/local/bin/composer
If you want to use the alternate Symfony install via Composer, you should now be able to do that.
If we want to use the Symfony Installer, we'll need to grab it.
$ curl -L https://symfony.com/installer -o /usr/local/bin/symfony
$ chmod a+x /usr/local/bin/symfony
If you don't have a timezone set in your php.ini, you should set it before trying to run the Symfony Installer or else it will complain.
$ sed -i -e 's/;date.timezone/date.timezone = "America\/Toronto"/g' /etc/php5/php.ini
An explanation for the sed command is available in this answer. Set your own timezone as appropriate based on your location.
You should now be able to run the Symfony Installer to create your project
$ symfony new my_project_name lts
Where the lts part is for your version. Currently lts is 2.8.4. If you leave that part off you'll get the most current stable.
I hope this helps you (and perhaps me, next time I have to do this) save a bit of time.
I have been using wkthmltopdf to convert html to pdf documents on-the-fly on my linux web server. The program originally needed X11 or similar X server to run correctly, but through many requests by developers to have this run on servers without GUI, I am pretty sure it runs a virtual X server in the static version. I have been using the static (stand-alone) version of the program and it works great! I would put the executable file in a folder, and run:
./wkhtmltopdf file1.html file2.pdf
However I would like to install this program system-wide. I used the apt-get install wkhtmltopdf (just installed yesterday) and since I am running on a 64 bit system, I also needed apt-get install ia32-libs. After installation I can find the version like this:
wkhtmltopdf --version
output:
Name:
wkhtmltopdf 0.9.9
License:
Copyright (C) 2008,2009 Wkhtmltopdf Authors.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO
WARRANTY, to the extent permitted by law.
Authors:
Written by Jakob Truelsen. Patches by Mário Silva, Benoit Garret and Emmanuel
Bouthenot.
Now when I try to run the program installed via aptitude, I get the following error:
wkhtmltopdf: cannot connect to X server
Does anyone know how I can fix this? I guess this version is missing a virtual X server or something.
or try this (from http://drupal.org/node/870058)
Download wkhtmltopdf. Or better install it with a package manager:
sudo apt-get install wkhtmltopdf
Extract it and move it to /usr/local/bin/
Rename it to wkhtmltopdf so that now you have an executable at /usr/local/bin/wkhtmltopdf
Set permissions: sudo chmod a+x /usr/local/bin/wkhtmltopdf
Install required support packages.
sudo apt-get install openssl build-essential xorg libssl-dev
Check to see if it works: run
/usr/local/bin/wkhtmltopdf http://www.google.com test.pdf
If it works, then you are done. If you get the error "Cannot connect to X server" then continue to number 7.
We need to run it headless on a 'virtual' x server. We will do this with a package called xvfb.
sudo apt-get install xvfb
We need to write a little shell script to wrap wkhtmltopdf in xvfb. Make a file called wkhtmltopdf.sh and add the following:
xvfb-run -a -s "-screen 0 640x480x16" wkhtmltopdf "$#"
Move this shell script to /usr/local/bin, and set permissions:
sudo chmod a+x /usr/local/bin/wkhtmltopdf.sh
Check to see if it works once again: run
/usr/local/bin/wkhtmltopdf.sh http://www.google.com test.pdf
Note that http://www.google.com may throw an error like "A finished ResourceObject received a loading finished signal. This might be an indication of an iframe taking to long to load." You may want to test with a simpler page like http://www.example.com.
This solved the issue for me:
sudo apt-get install xvfb
xvfb-run --server-args="-screen 0, 1024x768x24" wkhtmltopdf file1.html file2.pdf
I tried to do sudo apt-get install wkhtmltopdf but without any success.
Instead I recommend you try:
Download the latest executable (.11 rc1) :
wget https://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.11.0_rc1-static-i386.tar.bz2
uncompress it :
tar -vxf wkhtmltopdf-0.11.0_rc1-static-i386.tar.bz2
rename it :
mv wkhtmltopdf-i386 wkhtmltopdf
chmod it to executable :
chmod a+x wkhtmltopdf
place it into /usr/bin :
sudo mv wkhtmltopdf /usr/bin
Just made it:
1- To download wkhtmltopdf dependencies
# apt-get install wkhtmltopdf
2- Download from source
# wget http://downloads.sourceforge.net/project/wkhtmltopdf/xxx.deb
# dpkg -i xxx.deb
3- Try
# wkhtmltopdf http://google.com google.pdf
Its working fine
It works!
I found method to resolve this problem without fake X server.
In newest version of wkhtmltopdf dont need X server for work, but it no into official linux repositories.
Solution for Ubuntu 14.04.4 LTS (trusty) i386
$ sudo apt-get install xfonts-75dpi
$ wget http://download.gna.org/wkhtmltopdf/0.12/0.12.2/wkhtmltox-0.12.2_linux-trusty-i386.deb
$ sudo dpkg -i wkhtmltox-0.12.2_linux-trusty-i386.deb
$ wkhtmltopdf http://www.google.com test.pdf
Solution for Ubuntu 14.04.4 LTS (trusty) amd64
$ sudo apt-get install xfonts-75dpi
$ wget http://download.gna.org/wkhtmltopdf/0.12/0.12.2/wkhtmltox-0.12.2_linux-trusty-amd64.deb
$ sudo dpkg -i wkhtmltox-0.12.2_linux-trusty-amd64.deb
$ wkhtmltopdf http://www.google.com test.pdf
User felixhummel got very good solution, but repository with utilite has changed.
Expanding on Timothy's answer...
If you're a web developer looking to use wkhtmltopdf as part of your web app, you can simply install it into your /usr/bin/ folder like so:
cd /usr/bin/
curl -C - -O http://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.11.0_rc1-static-i386.tar.bz2
tar -xvjf wkhtmltopdf-0.11.0_rc1-static-i386.tar.bz2
mv wkhtmltopdf-i386 wkhtmltopdf
You can now run it anywhere using wkhtmltopdf.
I personally use the Snappy library in PHP. Here is an example of how easy it is to create a PDF:
<?php
// Create new PDF
$pdf = new \Knp\Snappy\Pdf('wkhtmltopdf');
// Set output header
header('Content-Type: application/pdf');
// Generate PDF from HTML
echo $pdf->getOutputFromHtml('<h1>Title</h1><p>Your content goes here.</p>');
Update to latest wkhtmltopdf version from SourceForge (0.12 as of this writing). It does not need an X Server to run.
Example for Ubuntu 14.04:
$ cd /tmp/
$ wget -q http://downloads.sourceforge.net/project/wkhtmltopdf/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
$ dpkg -x wkhtmltox-0.12.2.1_linux-trusty-amd64.deb foo
$ echo '<p>hi</p>' | ./foo/usr/local/bin/wkhtmltopdf - /tmp/hi.pdf
Loading pages (1/6)
Counting pages (2/6)
Resolving links (4/6)
Loading headers and footers (5/6)
Printing pages (6/6)
Done
$ head -n3 /tmp/hi.pdf
%PDF-1.4
1 0 obj
<<
for 14.04.1-Ubuntu https://wkhtmltopdf.org/downloads.html
wget https://downloads.wkhtmltopdf.org/0.12/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz -O mktemp.tar.xz
tar xf mktemp.tar.xz
sudo cp wkhtmltox/bin/wkhtmltopdf /usr/bin/wkhtmltopdf
sudo chmod +x /usr/bin/wkhtmltopdf
rm mktemp.tar.xz
rm wkhtmltox -rf
apt-get update
apt-get install -y libxrender1 libxtst6 libxi6
wkhtmltopdf http://www.google.com test.pdf
sudo -i
apt-get install wkhtmltopdf xvfb libicu48
mv /usr/bin/wkhtmltopdf /usr/bin/wkhtmltopdf-origin
touch /usr/bin/wkhtmltopdf && chmod +x /usr/bin/wkhtmltopdf && cat > /usr/bin/wkhtmltopdf << END
#!/bin/bash
/usr/bin/xvfb-run -a -s "-screen 0 1024x768x24" /usr/bin/wkhtmltopdf-origin "\$#"
END
Problem is probably in old version of wkhtmltopdf - version 0.9 from distribution repository require running X server, but current version - 0.12.2.1 doesnt require it - can run headless.
Download package for your distribution from http://wkhtmltopdf.org/downloads.html and install it - for Ubuntu:
sudo apt-get install xfonts-75dpi
sudo dpkg -i wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
wkhtmltopdf > 0.11 doesn't have this X-server issue.
So installing 0.12.2.1 on a linux server.
At first install xvfb server:
sudo apt-get install xvfb
Get needed version of wkhtmltopdf from http://wkhtmltopdf.org/downloads.html
Install wkhtmltopdf:
sudo dpkg -i wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
or install with wget
URL='http://download.gna.org/wkhtmltopdf/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb'; FILE=`mktemp`; wget "$URL" -qO $FILE && sudo dpkg -i $FILE; rm $FILE
Install dependency (if needed):
sudo apt-get -f install
Create symblic link in /usr/local/bin/:
echo 'exec xvfb-run -a -s "-screen 0 640x480x16" wkhtmltopdf "$#"' | sudo tee /usr/local/bin/wkhtmltopdf.sh >/dev/null
sudo chmod a+x /usr/local/bin/wkhtmltopdf.sh
Now try below and it should work,
/usr/local/bin/wkhtmltopdf http://www.google.com test.pdf
I just figured out that I can simply move the static executable to the /usr/bin/ directory and execute it from anywhere.
solution for Centos7:
yum -y install xorg-x11-fonts-75dpi \
xorg-x11-fonts-Type1 \
&& rpm -Uvh http://download.gna.org/wkhtmltopdf/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-centos7-amd64.rpm
We run into this problem inside docker containers and the above install has wkhtmltopdf with patched QT
It is recommended to use at least 0.12.2.1.
Starting from wkhtmltopdf >= 0.12.2 it doesn't require X server or emulation anymore. You can download new version from http://wkhtmltopdf.org/downloads.html
I did follow the instructions here and made wkhtmltopdf work for me but I would like to offer a bit of perspective which I discovered while doing my own little dance with wkhtmltopdf - xvfb.
This is important because the same reason that causes it to throw the infamous cannot connect to X server error is also causing it to run with sever limitations even if you do provide it a X server. These limitations include not being able to take multiple input sources, set header and footers, etc (check the Reduced Functionality section of the manual).
wkhtmltox by itself doesn't require a X11, however it's making use of QT libraries which do. In newever versions of wkthmltox developers made a patch for QT which allows it to run with a X11.
Currently some versions are built against patched QT and some are not. You can check your version by running wkhtmltopds --version. There should be a line at the end saying Compiled against wkhtmltopdf patched qt.
So, to conclude, if you install and use a version that uses the patched libraries it should work on a linux server without the xvfb server, as I can confirm.
Pay attention: your file could be wkhtmltopdf.sh or wkhtmltopdf, check it on second step
You must copy it into directory : /usr/local/bin, make sur it's executable and add symlink of wkhtmltopdf.sh like :
1- the command :
sudo apt-get install wkhtmltopdf
2 - insert the binary in directory /usr/bin so the browser can't have permission to execute in this directory.
You must copy the wkhtmltopdf.sh to directory /usr/local/bin cause the browser have permission in this directory like:
sudo cp /usr/bin/wkhtmltopdf.sh /usr/local/bin/wkhtmltopdf.sh
3 - After make sur the binary have permission of execution like :
sudo chmod a+x /usr/local/bin/wkhtmltopdf.sh
4 - so now you can test, it's work like:
/usr/local/bin/wkhtmltopdf.sh http://www.google.com google.pdf
it make download the pdf in the current directory in your terminal
5 - Optional
now you can add symlink in your directory /usr/local/bin like
ln -s /usr/local/bin/wkhtmltopdf.sh /usr/local/bin/wkhtmltopdf
Just tell the Qt backend to not use X:
QT_QPA_PLATFORM=offscreen wkhtmltopdf <input> <outfile.pdf>
Download file from this link
Extract it and move executable file(/wkhtmltox/bin/wkhtmltopdf) to /usr/bin/
Rename it to wkhtmltopdf if current name is not wkhtmltopdf. So that now you have an executable at /usr/bin/wkhtmltopdf
Set permissions: sudo chmod a+x /usr/bin/wkhtmltopdf
Install required support packages. sudo apt-get install openssl build-essential xorg libssl-dev
Now, check with wkhtmltopdf http://www.google.com test.pdf
hint: detail information from this link
Just install a version 0.12.4 or higher. This seems to solve the problem.
See How can I install the latest wkhtmltopdf on Ubuntu 16.04?.
If you config wkhtmltopdf for Rails or Somethings in Centos, you can follow these step bellow:
Go to https://wkhtmltopdf.org/downloads.html and copied the link of rpm file.
In centos server bash.
wget link_of_wkhtmltopdf_rpm.rpm
rpm -ivh link_of_wkhtmltopdf_rpm.rpm
which wkhtmltopdf
=> You will get path of wkhtmltopdf.
Setup for wicked_pdf or pdfkit with path in step 4.
This is sample config with wickedpdf. config/initializers/wicked_pdf.rb
if Rails.env != "production"
path = %x[which wkhtmltopdf].gsub(/\n/, "")
else
path = "path_of_wkhtmltopdf_in_step_4"
end
WickedPdf.config = { exe_path: path }
Restart server.
DONE.
For 64-bit Use:
wget http://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.9.9-static-amd64.tar.bz2
tar xvjf wkhtmltopdf-0.9.9-static-amd64.tar.bz2
sudo mv wkhtmltopdf-amd64 /usr/bin/wkhtmltopdf
sudo chmod +x /usr/bin/wkhtmltopdf