With Alpine, Alpine fully supports recent versions of librdkafka, I can just do apk add in my Dockerfile, and the following works:
FROM golang:1.13-alpine3.10 as builder
WORKDIR /app
COPY go.mod go.sum ./
COPY src ./src/
RUN set -eux; \
apk add --no-cache gcc git libc-dev librdkafka-dev; \
go build -o ./ ./...
Now, for a particular project, I need to make Debian friendly binaries, that will run on Debian/Ubuntu servers.
The problem is that:
The official Debian repositories only support really old 0.11.x versions of librdkafka. Even for stretch and buster including backports repos. They don't have more recent versions.
The official Confluent repositories only support librdkafka on Debian 8 (jessie). They don't support librdkafka at all on Debian 9 (stretch) or 10 (buster) due to a libssl version incompatibility.
The official golang images only support Debian 9 (stretch) and 10 (buster). They don't support Debian 8 (Jessie).
My options:
Use a dev branch of the Golang Kafka client that doesn't need librdkafka installed at the system level. This would be amazing if it was stable and recommended.
Manually install/build librdkafka on Debian 9/10.
Get a Debian 8 golang image?
Can I do Debian target builds from Alpine? I suspect no, but it's worth asking.
What is the recommended solution?
Here is the solution which worked for me. I had to download it from source and it gives the latest version
Sample Dockerfile looks like this:
FROM golang:1.12.9-alpine AS build-stage
LABEL app="application_name"
ENV PATH=$PATH:$GOROOT/bin:$GOPATH/bin
# Because of https://github.com/docker/docker/issues/14914
# required by go get
RUN apk add --update --no-cache alpine-sdk bash python ca-certificates \
libressl \
tar \
git openssh openssl yajl-dev zlib-dev cyrus-sasl-dev openssl-dev coreutils
WORKDIR /src/application_name
RUN git clone https://github.com/edenhill/librdkafka.git
WORKDIR /src/application_name/librdkafka
RUN /src/application_name/librdkafka/configure --prefix /usr
RUN make
RUN make install
WORKDIR /src/application_name
COPY . .
# build the application
RUN GOOS=linux go build -a -o image-name .
Related
I have a Confluence Kafka consumer written in Golang. I am trying to deploy it in a PKS cluster.
The Kafka config looks like this,
kafka.bootstrap.servers=server.myserver.com
kafka.security.protocol=SASL_SSL
kafka.sasl.mechanisms=GSSAPI
kafka.group.id=kafka-go-getting-started
kafka.auto.offset.reset=latest
kafka.topic=topic.consumer-topic
acks=all
I need to configure my Dockerfile for GSSAPI mechanism with SASL_SSL protocol. I have managed to resolve the GSSAPI thing, however, currently it shows,
**Failed to create consumer: Unsupported value "SASL_SSL" for configuration property "security.protocol": OpenSSL not available at build time**
Here is how my Dockerfile looks like:
FROM golang:1.19-alpine3.16 as c-bindings
RUN apk update && apk upgrade && apk add pkgconf git bash build-base sudo
RUN git clone https://github.com/edenhill/librdkafka.git
RUN cd librdkafka && ./configure && make && sudo make install
FROM c-bindings as app-builder
WORKDIR /go/app
COPY . .
RUN go mod download
RUN go mod verify
RUN go build -race -tags musl --ldflags "-extldflags -static -s -w" -o main ./main.go
FROM scratch AS app-runner
WORKDIR /go/app/
COPY --from=app-builder /go/app/main ./main
CMD ["/go/app/main"]`
Tried some ways in Dockerfile to make OpenSSL available, however things are stuck at same. Not sure if both GSSAPI mechanism as well as SASL_SSL protocol can be resolved over a common solution.
[Dec 05, 2022] Latest try:
Dockerfile,
FROM golang:1.19-alpine as c-bindings
RUN apk update && apk upgrade && apk add pkgconf git bash build-base sudo
FROM c-bindings as app-builder
WORKDIR /go/app
COPY . .
RUN go mod download
RUN go mod verify
RUN apk add zstd-dev
RUN apk add krb5
RUN apk add cyrus-sasl-gssapiv2
RUN apk add cyrus-sasl-dev
RUN apk add openssl-dev
RUN git clone https://github.com/edenhill/librdkafka.git
RUN cd librdkafka && ./configure --install-deps && make && sudo make install
COPY krb5.conf /etc/krb5.conf
COPY jaas.conf /etc/jaas.conf
RUN go build -race -tags dynamic -o main ./main.go
CMD ["/go/app/main"]
Kafka config -
kafka.bootstrap.servers=server.myserver.com
kafka.security.protocol=SASL_SSL
kafka.sasl.mechanism=GSSAPI
kafka.group.id=kafka-go-getting-started
kafka.auto.offset.reset=latest
kafka.topic=topic.consumer-topic
kafka.ssl.ca.location=/etc/ssl/certs/my-cert.pem
kafka.sasl.kerberos.service.name=kafka
kafka.sasl.kerberos.keytab=/etc/security/keytab/consumer.keytab
kafka.sasl.kerberos.principal=principal#myprincipal.COM
acks=all
Now the container is technically running. However, it is not able to run the Kafka consumer application with below errors -
GSSAPI Error: A token had an invalid MIC (unknown mech-code 0 for mech unknown)
that is because you are missing the SSL or SASL dependancies you would need to make sure that libssl-dev, hoewever it could also needs those libsasl2-dev, libsasl2-modules, but libssl-dev should be enough though
adding the following to the DOCKERFILE should help to resolve it
RUN apk add libressl-dev
here is the official libssl and the alpine pkg
I'm trying to create a new lambda layer to import the zip file with psycopg2, because the library made my deployment package get over 3MB, and I can not see the inline code in my lambda function any more.
I created lambda layer for the following 2 cases with Python 3.7:
psycopg2_lib.zip (contains psycopg2, psycopg2_binary.libs and psycopg2_binary-2.8.5.dist-info folders)
psycopg2_only.zip which contains only the psycopg2 folder.
I added they new created layer into my lambda function.
But, in both cases, my lambda_function throws an error as follows:
{
"errorMessage": "Unable to import module 'lambda_function': No module named 'psycopg2'",
"errorType": "Runtime.ImportModuleError"
}
The error seems as if something went wrong with my zip file that they are not recognized. But when it works well in my deployment package.
Any help or reason would be much appriciated. Thanks!
not sure if the OP found a solution to this but in case others land here. I resolved this using the following steps:
download the code/clone the git from:
https://github.com/jkehler/awslambda-psycopg2
create the following directory tree, if building for python3.7, otherwise replace 'python3.7' with the version choice:
mkdir -p python/lib/python3.7/site-packages/psycopg2
choose the python version of interest and copy the files from the folders downloaded in step 1. to the directory tree in step 2. e.g. if building a layer for python 3.7:
cp psycopg2-3.7/* python/lib/python3.7/site-packages/psycopg2
create the zip file for the layer. e.g.: zip -r9 psycopg2-py37.zip python
create a layer in the console or cli and upload the zip
If you end up on this page in >= 2022 year. Use official psycopg2-binary https://pypi.org/project/psycopg2-binary/
Works well for me. Just
pip install --target ./python psycopg2-binary
zip -r python.zip python
Maintainers of psycopg2 do not recommend using psycopg2-binary because it comes with linked libpq and libssl and others that may cause issues in production under certain circumstances.
I may imagine this being an issue when upgrading postgresql server while bundled libpq is incompatible. I also had issues w/ psycopg2-binary on AWS Lambda running in arm64 environment.
I've resorted to building postgresql and psycopg in Docker running on linux/arm64 platform using public.ecr.aws/lambda/python:3.9 as the base image.
FROM public.ecr.aws/lambda/python:3.9
RUN yum -y update && \
yum -y upgrade && \
yum -y install libffi-devel postgresql-devel postgresql-libs zip rsync wget openssl openssl-devel && \
yum -y groupinstall "development tools" && \
pip install pipenv
ENTRYPOINT ["/bin/bash"]
The build script is the following and valid for aarch64 platform. Just change path to x86_64 version on Prepare psycopg2 step.
#!/usr/bin/env bash
set -e
PG_VERSION="14.5"
cd "$TERRAFORM_ROOT"
if [ ! -f "postgresql-$PG_VERSION.tar.bz2" ]; then
wget "https://ftp.postgresql.org/pub/source/v$PG_VERSION/postgresql-$PG_VERSION.tar.bz2"
tar -xf "$(pwd)/postgresql-$PG_VERSION.tar.bz2"
fi
if [ ! -d "psycopg2" ]; then
git clone https://github.com/psycopg/psycopg2.git
fi
# Build postgres
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION"
./configure --without-readline --without-zlib
make
make install
# Build psycopg2
cd "$TERRAFORM_ROOT/psycopg2"
make clean
python setup.py build_ext \
--pg-config "$TERRAFORM_ROOT/postgresql-14.5/src/bin/pg_config/pg_config"
# Prepare psycopg2
cd build/lib.linux-aarch64-3.9
mkdir -p python/
cp -r psycopg2 python/
zip -9 -r "$BUNDLE" ./python
# Prepare libpq
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION/src/interfaces/libpq/"
mkdir -p lib/
cp libpq.so.5 lib/
zip -9 -r "$BUNDLE" ./lib
where $BUNDLE is the path to already existing .zip file.
I also tried to statically build psycopg2 binary and link libpq.a, however, I have had quite a lot of issues with missing symbols.
From AWS post How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?:
To create a Lambda deployment package or layer that's compatible with Lambda Python runtimes when using pip outside of Linux operating system, run the pip install command with manylinux2014 as the value for the --platform parameter.
pip install \
--platform manylinux2014_x86_64 \
--target=my-lambda-function \
--implementation cp \
--python 3.9 \
--only-binary=:all: --upgrade \
psycopg2-binary
You can then zip the content of directory my-lambda-function
There are several different versions of ffmpeg and ffprobe flying around, and each version has a different API.
If I apt-get install ffmpeg on Ubuntu 16.04, I get ffmpeg version 2.8.15-0ubuntu0.16.04.1. If I install apt-get install ffmpeg on Ubuntu 18.04, I get version 3.4.4-0ubuntu0.18.04.1.
When I visit the ffmpeg documentation, it says "The following documentation is regenerated nightly, and corresponds to the newest FFmpeg revision. Consult your locally installed documentation for older versions." That is, the hosted documentation is neither of those two versions.
So I have two questions:
What does it mean "your locally installed documentation"? Is it only talking about man ffmpeg? Or is there some way to host the documentation as a webpage?
Are there any places that simply host the older versions of the ffmpeg documentation?
For anyone who uses docker and wants to just host the docs without thinking too much, this is the Dockerfile I came up with.
FROM ubuntu:18.04
# Install requirements for ffmpeg doc generation
RUN apt-get update && apt-get install -y git build-essential texinfo yasm
# Install requirements for minimal webserver
RUN apt-get install -y webfs mime-support && update-mime
RUN git clone https://git.ffmpeg.org/ffmpeg.git
# Checkout the version that you want
RUN cd ffmpeg \
&& git checkout tags/n2.8.15 \
&& ./configure \
&& make doc
WORKDIR /ffmpeg/doc
CMD webfsd -F -p 80
Then you can
docker build -t ffmpeg-doc .
docker run --rm -it -p 80:80 ffmpeg-doc
And visit http://localhost for the list of generated files. The common ones will be http://localhost/ffmpeg.html or http://localhost/ffprobe.html.
What does it mean "your locally installed documentation"? Is it only talking about man ffmpeg?
It is referring to the various man pages and ffmpeg -h.
Are there any places that simply host the older versions of the ffmpeg documentation?
You can make it yourself. Install the build-essential and texinfo packages, download the source code for your FFmpeg version, then make the HTML documentation:
./configure
make doc
The HTML files will be located in the doc directory.
Alternatively, and more recommended, download or compile a recent version from the git master branch and use the online documentation.
I've posted this question already as an issue on the imagick git repository, but it has a very small user-base, so I'm hoping to get some help from here. I've been trying for a few days now to import https://github.com/gographics/imagick to Docker using the official goLang dockerfile for a project I'm working on, but have been unsuccessful. Since this package isn't popular, running apt-get won't work. I've (hesitantly) tried to just add the files to the container, but that didn't work. Here's the DockerFile I've built and the error it produces:
===DOCKERFILE===
# 1) Use the official go docker image built on debian.
FROM golang:latest
# 2) ENV VARS
ENV GOPATH $HOME/<PROJECT>
ENV PATH $HOME/<PROJECT>/bin:$PATH
# 3) Grab the source code and add it to the workspace.
ADD . /<GO>/src/<PROJECT>
ADD . /<GO>/gopkg.in
# Trying to add the files manually... Doesn't help.
ADD . /opt/local/share/doc/ImageMagick-6
# 4) Install revel and the revel CLI.
#(The commented out code is from previous attempts)
#RUN pkg-config --cflags --libs MagickWand
#RUN go get gopkg.in/gographics/imagick.v2/imagick
RUN go get github.com/revel/revel
RUN go get github.com/revel/cmd/revel
# 5) Does not work... Can't find the package.
#RUN apt-get install libmagickwand-dev
# 6) Get godeps from main repo
RUN go get github.com/tools/godep
# 7) Restore godep dependencies
WORKDIR /<GO>/src/<PROJECT>
RUN godep restore
# 8) Install Imagick
#RUN go build -tags no_pkgconfig gopkg.in/gographics/imagick.v2/imagick
# 9) Use the revel CLI to start up our application.
ENTRYPOINT revel run <PROJECT> dev 9000
# 10) Open up the port where the app is running.
EXPOSE 9000
===END DOCKERFILE===
This allows me to build the docker container, but when I try to run it, I get the following error in the logs of kinematic:
===DOCKER ERROR===
ERROR 2016/08/20 21:15:10 build.go:108: # pkg-config --cflags MagickWand MagickCore MagickWand MagickCore
pkg-config: exec: "pkg-config": executable file not found in $PATH
2016-08-20T21:15:10.081426584Z
ERROR 2016/08/20 21:15:10 build.go:308: Failed to parse build errors:
#pkg-config --cflags MagickWand MagickCore MagickWand MagickCore
pkg-config: exec: "pkg-config": executable file not found in $PATH
2016-08-20T21:15:10.082140143Z
===END DOCKER ERROR===
Most base images have package lists removed to avoid to reduce image size. Thus, in order to install something with apt-get, you first need to update the package lists and then install whatever package you wish. Then, after installing the package, remove all side-effects of running apt to avoid polluting the image with unneeded files (all that necessarily as a single RUN command).
The following Dockerfile should do the trick:
FROM golang:latest
RUN apt-get update \ # update package lists
&& apt-get install -y libmagickwand-dev \ # install the package
&& apt-get clean \ # clean package cache
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* # remove everything else
RUN go get gopkg.in/gographics/imagick.v2/imagick
Remember to add -y to apt-get install, because docker build is non-interactive.
I'm trying to compile linphone on mountain lion. I have downloaded the sources and followed all the instructions in the README.macos file. The first time, when I got to '$ port install ige-mac-integration' an error occurred. It said gtk2 had to be installed with x11, however, the instructions said to install it with quartz and no_x11. I tried installing gtk2 again, this time with x11, but when I get to the compiling process it tells me quartz is needed. I can't install both, because the gtk2 installation will complain about it, and it seems I need both to complete the entire process of installing and compiling linphone.
I have tried these steps with the downloadable sources, as well as the git sources, both gave me the same problems. Linphone does not have a forum, so I couldn't ask it on their site.
I just need to get the sources and start using them, so any recommendations on how to open the linphone source, edit it, and compile/run it would be very welcome. The ideal situation would be where I can just open an xcode project file.
This is the readme file:
**********************************
* Compiling linphone on macos X *
**********************************
You need:
- Xcode (download from apple or using appstore application)
- Macports: http://www.macports.org/
Download and install macports using its user friendly installer.
- Install build time dependencies
$ port install automake autoconf libtool intltool
- Install some linphone dependencies with macports
$ port install speex
$ port install libosip2 # WARNING: currently outdated in macport
$ port install libeXosip2 #WARNING: currently outdated in macport
$ port install ffmpeg-devel
$ port install libvpx
- Install srtp (optional) for call encryption
$ port install srtp
If that fails, get from source:
$ git clone git://git.linphone.org/srtp.git
$ cd srtp && autoconf && ./configure --prefix=/opt/local && make libsrtp.a
$ sudo make install
- Install zrtpcpp (optional), for unbreakable call encryption
$ port install cmake
$ git clone git://git.linphone.org/zrtpcpp.git
$ cd zrtpcpp && cmake -Denable_ccrtp=false . && make
$ sudo make install
- Install gtk. It is recommended to use the quartz backend for better integration.
$ port install gtk2 +quartz +no_x11
$ port install hicolor-icon-theme
- Compile and install the tunnelsu
If you got the source code from git, run ./autogen.sh first
Then or otherwise, do:
$ ./configure --prefix=/opt/local && make && sudo make install
- Compile linphone
If you got the source code from git, run ./autogen.sh first.
Then or otherwise, do:
$ ./configure --prefix=/opt/local --with-readline=/opt/local --disable-strict --disable-x11 --with-srtp=/opt/local --with-gsm=/opt/local --enable-zrtp && make
Install to /opt/local
$ sudo make install
Done.
If you want to generate a portable bundle, then install gtk-mac-bundler.
Use git:
$ git clone https://github.com/jralls/gtk-mac-bundler.git
$ cd gtk-mac-bundler && make install
$ export PATH=$PATH:~/.local/bin
#make this dummy charset.alias file for the bundler to be happy:
$ sudo touch touch /opt/local/lib/charset.alias
Then run, inside linphone source tree:
1. Run configure as told before but with "--enable-relativeprefix" appended.
$ make
$ make bundle
The resulting bundle is located in linphone build directory, together with a zipped version.
For a better appearance, you can install the gtk-quartz-engine (a gtk theme) that make gtk application more similar to other mac applications (but not perfect).
$ git clone https://github.com/jralls/gtk-quartz-engine.git
$ cd gtk-quartz-engine
$ autoreconf -i
$ ./configure --prefix=/opt/local && make
$ sudo make install
Generate a new bundle to have it included.
Below is the details to configure the dependencies and build linphone.
==============================================================
Step: 1:- Dependency configuration:
I just followed the instruction which is available in "README.macos" upto sqlite3 configuration. I had some problem with dependancies configuration.
Because of macport version, srtp configuration is failed. So I have downloaded "MacPorts-2.1.3-10.8-MountainLion.pkg" and installed manually.
Because of GTK version, libsoup is failed. So I have downloaded "GTK_2.18.5-X11.pkg" and installed manually.
After the manual installation, I have followed "README.macos" again for reconfiguration.
NOTE: Sometime terminal won't recognize "wget".
--> Solution: just execute below command.
echo 'alias wget="curl -O"' >> ~/.bash_profile
Step: 2:- Compile and Install:
I got some problem related to intltool while compiling.
Solution: just I set the path by executing below commands
export PATH=$PATH:/opt/local/bin
export MANPATH=$MANPATH:/opt/local/share/man
export INFOPATH=$INFOPATH:/opt/local/share/info
Now it is successfully compiled and installed.
Step: 3:- Creating bundle to run the app:
Just I followed below commands to make the build.
If you want to generate a portable bundle, then install gtk-mac-bundler.
Use git:
$ git clone https://github.com/jralls/gtk-mac-bundler.git
$ cd gtk-mac-bundler && make install
$ export PATH=$PATH:~/.local/bin
#make this dummy charset.alias file for the bundler to be happy:
$ sudo touch touch /opt/local/lib/charset.alias
Then run, inside linphone source tree:
1. Run configure as told before but with "--enable-relativeprefix" appended.
$ make
$ make bundle
It will create "linphone.app" file in current linphone directory. It will support only for "Mountain Lion".
Step: 4:- Support for lower version: (e.g.: Lion, Snow Leopard...)
We have to configure "libiconv hack" to supporting for lower version
I think it will help you.
I wrote an answer here. It's a bit long but I sincerely hope it is clear enough and will help you.
It contains the build settings and other libs you need to include.
Cool stuff I didn't need to run the make :)
Best,