Is there a way to use gifsicle in AWS lambda?
I know there is a package called pygifsicle, but it seems it requires the gifsicle version of AWS Linux 2?
I don't see a binary built for RedHat version of gifsicle
So my questions are,
Do I need to build one for AWS Linux 2 to use it along with pygifsicle?
Even if I build gifsicle for AWS Linux 2, how to use it along with pygifsicle?
As I read the documentation you can build one binary for Building Gifsicle on UNIX and can package that with your lambda zip file which can be called as a normal command in lambda function.
Like it is being called in the pygifsicle
subprocess.call(["gifsicle", *options, *sources, "--colors",
str(colors), "--output", destination])
My Dockerfile where I'm building it from the source.
FROM public.ecr.aws/lambda/python:3.8-arm64
RUN yum -y install install make gcc wget gzip
RUN wget https://www.lcdf.org/gifsicle/gifsicle-1.93.tar.gz
RUN tar -xzf gifsicle-1.93.tar.gz
RUN cd gifsicle-1.93 && \
./configure && \
make && \
make install
COPY requirements.txt ./
RUN yum update -y && \
pip install -r requirements.txt
COPY . .
CMD ["app.handler"]
Related
So I am building a DOTNET application that runs on Debian, and makes use of ogr2ogr to copy data from an oracle database towards an Postgres database.
The problem is that I cannot get GDAL to recognize the OCI driver.
These are the installation commands that I have collected for now:
#Install dependencies used by GDAL and ora2pg
apt-get update && apt-get install -y -q --no-install-recommends \
libc-bin unzip curl ca-certificates rpm libaio1 \
#Package manager for installing Oracle
alien \
# Install postgresql
postgresql-client \
# Used for the POSTGRES_HOME variable
libpq-dev \
#Package manager used for installation of perl database drivers
cpanminus \
# Proj build
sqlite libsqlite3-dev pkg-config g++ make
#Install Oracle
curl -o oracle-instantclient-basic.x86_64.rpm https://download.oracle.com/otn_software/linux/instantclient/199000/oracle-instantclient19.9-basic-19.9.0.0.0-1.x86_64.rpm
curl -o oracle-instantclient-devel.x86_64.rpm https://download.oracle.com/otn_software/linux/instantclient/199000/oracle-instantclient19.9-devel-19.9.0.0.0-1.x86_64.rpm
curl -o oracle-instantclient-sqlplus.x86_64.rpm https://download.oracle.com/otn_software/linux/instantclient/199000/oracle-instantclient19.9-sqlplus-19.9.0.0.0-1.x86_64.rpm
alien -i oracle-instantclient-basic.x86_64.rpm && alien -i oracle-instantclient-devel.x86_64.rpm && alien -i oracle-instantclient-sqlplus.x86_64.rpm
EXPORT ORACLE_HOME=/usr/lib/oracle/19.9/client64
EXPORT TNS_ADMIN=/usr/lib/oracle/19.9/client64/network/admin
EXPORT LD_LIBRARY_PATH=/usr/lib/oracle/19.9/client64/lib
EXPORT PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/oracle/19.9/client64/bin
#Install Postgres en Oracle drivers for perl, ora2pg
cpanm DBD::Oracle
cpanm DBD::Pg
#Setup
wget https://download.osgeo.org/proj/proj-6.3.2.tar.gz
tar -zxf proj-6.3.2.tar.gz -C /opt/
/opt/proj-6.3.2/configure --prefix=/usr --disable-static --enable-lto
make -C /opt/proj-6.3.2/
make install -C /opt/proj-6.3.2/
RUN wget http://download.osgeo.org/gdal/3.2.2/gdal-3.2.2.tar.gz
/opt/gdal-3.2.2/configure
make -C /opt/proj-6.3.2/
make install -C /opt/proj-6.3.2/
Is there anyone who can tell me what I am missing, because I cannot find any answers on the internet...
So after a lot off testing, my colleague found the problem.
Apparently the scripts search for a folder in $ORACLE_HOME/sdk.
Now by installing it, like above, it doesn't install the sdk folder on the correct location.
So we solved it by adding an extra step extracting the SDK zip package on the correct location.
This is the result:
Dockerfile
Above solution did not work for me. I needed a GDAl container with OCI. Based on some other info I found I build an image that works.
https://github.com/botenvouwer/gdal-oci
Currently you have to clone and build the image yourself. You should (but don't have to) pass the version to your docker build as ARG GDAL_VERSION.
I'm trying to create a new lambda layer to import the zip file with psycopg2, because the library made my deployment package get over 3MB, and I can not see the inline code in my lambda function any more.
I created lambda layer for the following 2 cases with Python 3.7:
psycopg2_lib.zip (contains psycopg2, psycopg2_binary.libs and psycopg2_binary-2.8.5.dist-info folders)
psycopg2_only.zip which contains only the psycopg2 folder.
I added they new created layer into my lambda function.
But, in both cases, my lambda_function throws an error as follows:
{
"errorMessage": "Unable to import module 'lambda_function': No module named 'psycopg2'",
"errorType": "Runtime.ImportModuleError"
}
The error seems as if something went wrong with my zip file that they are not recognized. But when it works well in my deployment package.
Any help or reason would be much appriciated. Thanks!
not sure if the OP found a solution to this but in case others land here. I resolved this using the following steps:
download the code/clone the git from:
https://github.com/jkehler/awslambda-psycopg2
create the following directory tree, if building for python3.7, otherwise replace 'python3.7' with the version choice:
mkdir -p python/lib/python3.7/site-packages/psycopg2
choose the python version of interest and copy the files from the folders downloaded in step 1. to the directory tree in step 2. e.g. if building a layer for python 3.7:
cp psycopg2-3.7/* python/lib/python3.7/site-packages/psycopg2
create the zip file for the layer. e.g.: zip -r9 psycopg2-py37.zip python
create a layer in the console or cli and upload the zip
If you end up on this page in >= 2022 year. Use official psycopg2-binary https://pypi.org/project/psycopg2-binary/
Works well for me. Just
pip install --target ./python psycopg2-binary
zip -r python.zip python
Maintainers of psycopg2 do not recommend using psycopg2-binary because it comes with linked libpq and libssl and others that may cause issues in production under certain circumstances.
I may imagine this being an issue when upgrading postgresql server while bundled libpq is incompatible. I also had issues w/ psycopg2-binary on AWS Lambda running in arm64 environment.
I've resorted to building postgresql and psycopg in Docker running on linux/arm64 platform using public.ecr.aws/lambda/python:3.9 as the base image.
FROM public.ecr.aws/lambda/python:3.9
RUN yum -y update && \
yum -y upgrade && \
yum -y install libffi-devel postgresql-devel postgresql-libs zip rsync wget openssl openssl-devel && \
yum -y groupinstall "development tools" && \
pip install pipenv
ENTRYPOINT ["/bin/bash"]
The build script is the following and valid for aarch64 platform. Just change path to x86_64 version on Prepare psycopg2 step.
#!/usr/bin/env bash
set -e
PG_VERSION="14.5"
cd "$TERRAFORM_ROOT"
if [ ! -f "postgresql-$PG_VERSION.tar.bz2" ]; then
wget "https://ftp.postgresql.org/pub/source/v$PG_VERSION/postgresql-$PG_VERSION.tar.bz2"
tar -xf "$(pwd)/postgresql-$PG_VERSION.tar.bz2"
fi
if [ ! -d "psycopg2" ]; then
git clone https://github.com/psycopg/psycopg2.git
fi
# Build postgres
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION"
./configure --without-readline --without-zlib
make
make install
# Build psycopg2
cd "$TERRAFORM_ROOT/psycopg2"
make clean
python setup.py build_ext \
--pg-config "$TERRAFORM_ROOT/postgresql-14.5/src/bin/pg_config/pg_config"
# Prepare psycopg2
cd build/lib.linux-aarch64-3.9
mkdir -p python/
cp -r psycopg2 python/
zip -9 -r "$BUNDLE" ./python
# Prepare libpq
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION/src/interfaces/libpq/"
mkdir -p lib/
cp libpq.so.5 lib/
zip -9 -r "$BUNDLE" ./lib
where $BUNDLE is the path to already existing .zip file.
I also tried to statically build psycopg2 binary and link libpq.a, however, I have had quite a lot of issues with missing symbols.
From AWS post How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?:
To create a Lambda deployment package or layer that's compatible with Lambda Python runtimes when using pip outside of Linux operating system, run the pip install command with manylinux2014 as the value for the --platform parameter.
pip install \
--platform manylinux2014_x86_64 \
--target=my-lambda-function \
--implementation cp \
--python 3.9 \
--only-binary=:all: --upgrade \
psycopg2-binary
You can then zip the content of directory my-lambda-function
I am entirely new to the concept of dockers. I am creating the following Dockerfile as an exercise.
FROM ubuntu:latest
MAINTAINER kesarling
RUN apt update && apt upgrade -y
RUN apt install nginx curl zip unzip -y
RUN apt install openjdk-14-jdk python3 python3-doc clang golang-go gcc g++ -y
RUN curl -s "https://get.sdkman.io" | bash
RUN bash /root/.sdkman/bin/sdkman-init.sh
RUN sdk version
RUN yes | bash -c 'sdk install kotlin'
CMD [ "echo","The development environment has now been fully setup with C, C++, JAVA, Python3, Go and Kotlin" ]
I am using SDKMAN! to install Kotlin. The problem initially was that instead of using RUN bash /root/.sdkman/bin/sdkman-init.sh, I was using RUN source /root/.sdkman/bin/sdkman-init.sh. However, it gave the error saying source not found. So, I tried using RUN . /root/.sdkman/bin/sdkman-init.sh, and it did not work. However, RUN bash /root/.sdkman/bin/sdkman-init.sh seems to work, as in does not give any error and tries to run the next command. However, the docker then gives error saying sdk: not found
Where am I going wrong?
It should be noted that these steps worked like charm for my host distribution (The one on which I'm running docker) which is Pop!_OS 20.04
Actually the script /root/.sdkman/bin/sdkman-init.sh sources the sdk
source is a built-in to bash rather than a binary somewhere on the filesystem.
source command executes the file in the current shell.
Each RUN instruction will execute any commands in a new layer on top of the current image and commit the results.
The resulting committed image will be used for the next step in the Dockerfile.
Try this:
FROM ubuntu:latest
MAINTAINER kesarling
RUN apt update && apt upgrade -y
RUN apt install nginx curl zip unzip -y
RUN apt install openjdk-14-jdk python3 python3-doc clang golang-go gcc g++ -y
RUN curl -s "https://get.sdkman.io" | bash
RUN /bin/bash -c "source /root/.sdkman/bin/sdkman-init.sh; sdk version; sdk install kotlin"
CMD [ "echo","The development environment has now been fully setup with C, C++, JAVA, Python3, Go and Kotlin" ]
SDKMAN in Ubuntu Dockerfile
tl;dr
the sdk command is not a binary but a bash script loaded into memory
Shell sessions are a "process", which means environment variables and declared shell function only exist for the duration that shell session exists; which lasts only as long as the RUN command.
Manually tweak your PATH
RUN apt-get update && apt-get install curl bash unzip zip -y
RUN curl -s "https://get.sdkman.io" | bash
RUN source "$HOME/.sdkman/bin/sdkman-init.sh" \
&& sdk install java 8.0.275-amzn \
&& sdk install sbt 1.4.2 \
&& sdk install scala 2.12.12
ENV PATH=/root/.sdkman/candidates/java/current/bin:$PATH
ENV PATH=/root/.sdkman/candidates/scala/current/bin:$PATH
ENV PATH=/root/.sdkman/candidates/sbt/current/bin:$PATH
Full Version
Oh wow this was a journey to figure out. Below each line is commented as to why certain commands are run.
I learnt a lot about how unix works and how sdkman works and how docker works and why the intersection of the three give very unusual behaviour.
# I am using a multi-stage build so I am just copying the built artifacts
# from this stage to keep final image small.
FROM ubuntu:latest as ScalaBuild
# Switch from `sh -c` to `bash -c` as the shell behind a `RUN` command.
SHELL ["/bin/bash", "-c"]
# Usual updates
RUN apt-get update && apt-get upgrade -y
# Dependencies for sdkman installation
RUN apt-get install curl bash unzip zip -y
#Install sdkman
RUN curl -s "https://get.sdkman.io" | bash
# FUN FACTS:
# 1) the `sdk` command is not a binary but a bash script loaded into memory
# 2) Shell sessions are a "process", which means environment variables
# and declared shell function only exist for
# the duration that shell session exists
RUN source "$HOME/.sdkman/bin/sdkman-init.sh" \
&& sdk install java 8.0.275-amzn \
&& sdk install sbt 1.4.2 \
&& sdk install scala 2.12.12
# Once the real binaries exist these are
# the symlinked paths that need to exist on PATH
ENV PATH=/root/.sdkman/candidates/java/current/bin:$PATH
ENV PATH=/root/.sdkman/candidates/scala/current/bin:$PATH
ENV PATH=/root/.sdkman/candidates/sbt/current/bin:$PATH
# This is specific to running a minimal empty Scala project and packaging it
RUN touch build.sbt
RUN sbt compile
RUN sbt package
FROM alpine AS production
# setup production environment image here
COPY --from=ScalaBuild /root/target/scala-2.12/ $INSTALL_PATH
ENTRYPOINT ["java", "-cp", "$INSTALL_PATH", "your.main.classfile"]
Generally you want to avoid using "version manager" type tools in Docker; it's better to install a specific version of the compiler or runtime you need.
In the case of Kotlin, it's a JVM application distributed as a zip file so it should be fairly easy to install:
FROM openjdk:15-slim
ARG KOTLIN_VERSION=1.3.72
# Get OS-level updates:
RUN apt-get update \
&& apt-get install --no-install-recommends --assume-yes \
curl \
unzip
# and if you need C/Python dependencies, those too
# Download and unpack Kotlin
RUN cd /opt \
&& curl -LO https://github.com/JetBrains/kotlin/releases/download/v${KOTLIN_VERSION}/kotlin-compiler-${KOTLIN_VERSION}.zip \
&& unzip kotlin-compiler-${KOTLIN_VERSION}.zip \
&& rm kotlin-compiler-${KOTLIN_VERSION}.zip
# Add its directory to $PATH
ENV PATH=/opt/kotlinc/bin:$PATH
The real problem with version managers is that they heavily depend on the tool setting environment variables. As #JeevanRao notes in their answer, each Dockerfile RUN command runs in a separate shell in a separate container, and any environment variable settings within that command get lost for the next command.
# Does absolutely nothing: environment variables do not stay set
RUN . /root/.sdkman/bin/sdkman-init.sh
Since an image generally contains only one application and its runtime, you don't need the ability to change which version of the runtime or compiler you're using. My Dockerfile example passes it as an ARG, so you can change it in the Dockerfile or pass a docker build --build-arg KOTLIN_VERSION=... option to use a different version.
There are several different versions of ffmpeg and ffprobe flying around, and each version has a different API.
If I apt-get install ffmpeg on Ubuntu 16.04, I get ffmpeg version 2.8.15-0ubuntu0.16.04.1. If I install apt-get install ffmpeg on Ubuntu 18.04, I get version 3.4.4-0ubuntu0.18.04.1.
When I visit the ffmpeg documentation, it says "The following documentation is regenerated nightly, and corresponds to the newest FFmpeg revision. Consult your locally installed documentation for older versions." That is, the hosted documentation is neither of those two versions.
So I have two questions:
What does it mean "your locally installed documentation"? Is it only talking about man ffmpeg? Or is there some way to host the documentation as a webpage?
Are there any places that simply host the older versions of the ffmpeg documentation?
For anyone who uses docker and wants to just host the docs without thinking too much, this is the Dockerfile I came up with.
FROM ubuntu:18.04
# Install requirements for ffmpeg doc generation
RUN apt-get update && apt-get install -y git build-essential texinfo yasm
# Install requirements for minimal webserver
RUN apt-get install -y webfs mime-support && update-mime
RUN git clone https://git.ffmpeg.org/ffmpeg.git
# Checkout the version that you want
RUN cd ffmpeg \
&& git checkout tags/n2.8.15 \
&& ./configure \
&& make doc
WORKDIR /ffmpeg/doc
CMD webfsd -F -p 80
Then you can
docker build -t ffmpeg-doc .
docker run --rm -it -p 80:80 ffmpeg-doc
And visit http://localhost for the list of generated files. The common ones will be http://localhost/ffmpeg.html or http://localhost/ffprobe.html.
What does it mean "your locally installed documentation"? Is it only talking about man ffmpeg?
It is referring to the various man pages and ffmpeg -h.
Are there any places that simply host the older versions of the ffmpeg documentation?
You can make it yourself. Install the build-essential and texinfo packages, download the source code for your FFmpeg version, then make the HTML documentation:
./configure
make doc
The HTML files will be located in the doc directory.
Alternatively, and more recommended, download or compile a recent version from the git master branch and use the online documentation.
I'm using Bitbucket's Pipelines to automate my deployments. I am working on an Angular Project but use Maven to deploy my artifacts.
I was looking around to find a Docker container that contained both npm and maven and came across jhipster (v3.7.1). After installing, I cannot run "mvn" or "./mvnw" in any way. It says the command cannot be found.
Suggestions appreciated. In the meantime, I'm using pbarnoux/maven-angular and this container works fine.
if you have a closer look at the jhipster dockerfile you can see that maven is not installed i.e. the command is not available and your information that mvn cannot be run is correct. As a solution you cold add the maven at a package to all the other utilities to be installed:
# install utilities apt-get install -y \
maven \
wget \
curl \
vim \
git \
zip \
bzip2 \
fontconfig \
python \
g++ \
build-essential && \