I have a Confluence Kafka consumer written in Golang. I am trying to deploy it in a PKS cluster.
The Kafka config looks like this,
kafka.bootstrap.servers=server.myserver.com
kafka.security.protocol=SASL_SSL
kafka.sasl.mechanisms=GSSAPI
kafka.group.id=kafka-go-getting-started
kafka.auto.offset.reset=latest
kafka.topic=topic.consumer-topic
acks=all
I need to configure my Dockerfile for GSSAPI mechanism with SASL_SSL protocol. I have managed to resolve the GSSAPI thing, however, currently it shows,
**Failed to create consumer: Unsupported value "SASL_SSL" for configuration property "security.protocol": OpenSSL not available at build time**
Here is how my Dockerfile looks like:
FROM golang:1.19-alpine3.16 as c-bindings
RUN apk update && apk upgrade && apk add pkgconf git bash build-base sudo
RUN git clone https://github.com/edenhill/librdkafka.git
RUN cd librdkafka && ./configure && make && sudo make install
FROM c-bindings as app-builder
WORKDIR /go/app
COPY . .
RUN go mod download
RUN go mod verify
RUN go build -race -tags musl --ldflags "-extldflags -static -s -w" -o main ./main.go
FROM scratch AS app-runner
WORKDIR /go/app/
COPY --from=app-builder /go/app/main ./main
CMD ["/go/app/main"]`
Tried some ways in Dockerfile to make OpenSSL available, however things are stuck at same. Not sure if both GSSAPI mechanism as well as SASL_SSL protocol can be resolved over a common solution.
[Dec 05, 2022] Latest try:
Dockerfile,
FROM golang:1.19-alpine as c-bindings
RUN apk update && apk upgrade && apk add pkgconf git bash build-base sudo
FROM c-bindings as app-builder
WORKDIR /go/app
COPY . .
RUN go mod download
RUN go mod verify
RUN apk add zstd-dev
RUN apk add krb5
RUN apk add cyrus-sasl-gssapiv2
RUN apk add cyrus-sasl-dev
RUN apk add openssl-dev
RUN git clone https://github.com/edenhill/librdkafka.git
RUN cd librdkafka && ./configure --install-deps && make && sudo make install
COPY krb5.conf /etc/krb5.conf
COPY jaas.conf /etc/jaas.conf
RUN go build -race -tags dynamic -o main ./main.go
CMD ["/go/app/main"]
Kafka config -
kafka.bootstrap.servers=server.myserver.com
kafka.security.protocol=SASL_SSL
kafka.sasl.mechanism=GSSAPI
kafka.group.id=kafka-go-getting-started
kafka.auto.offset.reset=latest
kafka.topic=topic.consumer-topic
kafka.ssl.ca.location=/etc/ssl/certs/my-cert.pem
kafka.sasl.kerberos.service.name=kafka
kafka.sasl.kerberos.keytab=/etc/security/keytab/consumer.keytab
kafka.sasl.kerberos.principal=principal#myprincipal.COM
acks=all
Now the container is technically running. However, it is not able to run the Kafka consumer application with below errors -
GSSAPI Error: A token had an invalid MIC (unknown mech-code 0 for mech unknown)
that is because you are missing the SSL or SASL dependancies you would need to make sure that libssl-dev, hoewever it could also needs those libsasl2-dev, libsasl2-modules, but libssl-dev should be enough though
adding the following to the DOCKERFILE should help to resolve it
RUN apk add libressl-dev
here is the official libssl and the alpine pkg
Related
After initialising a project with anchor init, when I run anchor-test, it gives me this message at the last.
Unable to get recent blockhash. Test validator does not look started. Check .anchor/test-ledger/test-ledger-log.txt for errors. Consider increasing [test.startup_wait] in Anchor.toml.
The expected output at last should be the transaction hash but I don't get it.
When I use solana-test-validator it returns Aborted(Core Dumped). It seems like my CPU is not AVX2 compatible for which I am not able to run solana-test-validator.
The project is instructing me to build from source.
What to do now?
Sounds like you'll need to build solana-test-validator from source! This isn't very hard to do thankfully, and the steps go:
Install Rust:
$ curl https://sh.rustup.rs -sSf | sh
$ source $HOME/.cargo/env
$ rustup component add rustfmt
Install build dependencies, for Ubuntu:
$ sudo apt-get update
$ sudo apt-get install libssl-dev libudev-dev pkg-config zlib1g-dev llvm clang make
Download the source code:
$ git clone https://github.com/solana-labs/solana.git
$ cd solana
Build the test validator:
$ cargo build --bin solana-test-validator --release
Run it:
$ ./target/release/solana-test-validator
More information directly in the repo: https://github.com/solana-labs/solana#building
With Alpine, Alpine fully supports recent versions of librdkafka, I can just do apk add in my Dockerfile, and the following works:
FROM golang:1.13-alpine3.10 as builder
WORKDIR /app
COPY go.mod go.sum ./
COPY src ./src/
RUN set -eux; \
apk add --no-cache gcc git libc-dev librdkafka-dev; \
go build -o ./ ./...
Now, for a particular project, I need to make Debian friendly binaries, that will run on Debian/Ubuntu servers.
The problem is that:
The official Debian repositories only support really old 0.11.x versions of librdkafka. Even for stretch and buster including backports repos. They don't have more recent versions.
The official Confluent repositories only support librdkafka on Debian 8 (jessie). They don't support librdkafka at all on Debian 9 (stretch) or 10 (buster) due to a libssl version incompatibility.
The official golang images only support Debian 9 (stretch) and 10 (buster). They don't support Debian 8 (Jessie).
My options:
Use a dev branch of the Golang Kafka client that doesn't need librdkafka installed at the system level. This would be amazing if it was stable and recommended.
Manually install/build librdkafka on Debian 9/10.
Get a Debian 8 golang image?
Can I do Debian target builds from Alpine? I suspect no, but it's worth asking.
What is the recommended solution?
Here is the solution which worked for me. I had to download it from source and it gives the latest version
Sample Dockerfile looks like this:
FROM golang:1.12.9-alpine AS build-stage
LABEL app="application_name"
ENV PATH=$PATH:$GOROOT/bin:$GOPATH/bin
# Because of https://github.com/docker/docker/issues/14914
# required by go get
RUN apk add --update --no-cache alpine-sdk bash python ca-certificates \
libressl \
tar \
git openssh openssl yajl-dev zlib-dev cyrus-sasl-dev openssl-dev coreutils
WORKDIR /src/application_name
RUN git clone https://github.com/edenhill/librdkafka.git
WORKDIR /src/application_name/librdkafka
RUN /src/application_name/librdkafka/configure --prefix /usr
RUN make
RUN make install
WORKDIR /src/application_name
COPY . .
# build the application
RUN GOOS=linux go build -a -o image-name .
I've posted this question already as an issue on the imagick git repository, but it has a very small user-base, so I'm hoping to get some help from here. I've been trying for a few days now to import https://github.com/gographics/imagick to Docker using the official goLang dockerfile for a project I'm working on, but have been unsuccessful. Since this package isn't popular, running apt-get won't work. I've (hesitantly) tried to just add the files to the container, but that didn't work. Here's the DockerFile I've built and the error it produces:
===DOCKERFILE===
# 1) Use the official go docker image built on debian.
FROM golang:latest
# 2) ENV VARS
ENV GOPATH $HOME/<PROJECT>
ENV PATH $HOME/<PROJECT>/bin:$PATH
# 3) Grab the source code and add it to the workspace.
ADD . /<GO>/src/<PROJECT>
ADD . /<GO>/gopkg.in
# Trying to add the files manually... Doesn't help.
ADD . /opt/local/share/doc/ImageMagick-6
# 4) Install revel and the revel CLI.
#(The commented out code is from previous attempts)
#RUN pkg-config --cflags --libs MagickWand
#RUN go get gopkg.in/gographics/imagick.v2/imagick
RUN go get github.com/revel/revel
RUN go get github.com/revel/cmd/revel
# 5) Does not work... Can't find the package.
#RUN apt-get install libmagickwand-dev
# 6) Get godeps from main repo
RUN go get github.com/tools/godep
# 7) Restore godep dependencies
WORKDIR /<GO>/src/<PROJECT>
RUN godep restore
# 8) Install Imagick
#RUN go build -tags no_pkgconfig gopkg.in/gographics/imagick.v2/imagick
# 9) Use the revel CLI to start up our application.
ENTRYPOINT revel run <PROJECT> dev 9000
# 10) Open up the port where the app is running.
EXPOSE 9000
===END DOCKERFILE===
This allows me to build the docker container, but when I try to run it, I get the following error in the logs of kinematic:
===DOCKER ERROR===
ERROR 2016/08/20 21:15:10 build.go:108: # pkg-config --cflags MagickWand MagickCore MagickWand MagickCore
pkg-config: exec: "pkg-config": executable file not found in $PATH
2016-08-20T21:15:10.081426584Z
ERROR 2016/08/20 21:15:10 build.go:308: Failed to parse build errors:
#pkg-config --cflags MagickWand MagickCore MagickWand MagickCore
pkg-config: exec: "pkg-config": executable file not found in $PATH
2016-08-20T21:15:10.082140143Z
===END DOCKER ERROR===
Most base images have package lists removed to avoid to reduce image size. Thus, in order to install something with apt-get, you first need to update the package lists and then install whatever package you wish. Then, after installing the package, remove all side-effects of running apt to avoid polluting the image with unneeded files (all that necessarily as a single RUN command).
The following Dockerfile should do the trick:
FROM golang:latest
RUN apt-get update \ # update package lists
&& apt-get install -y libmagickwand-dev \ # install the package
&& apt-get clean \ # clean package cache
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* # remove everything else
RUN go get gopkg.in/gographics/imagick.v2/imagick
Remember to add -y to apt-get install, because docker build is non-interactive.
Given a binary, compiled with Go using GOOS=linux and GOARCH=amd64, deployed to a docker container based on alpine:3.3, the binary will not run if the docker engine host is Ubuntu (15.10):
sh: /bin/artisan: not found
This same binary (compiled for the same OS and arch) will run just fine if the docker engine host is busybox (which is the base for alpine) deployed within a VirtualBox VM on Mac OS X.
This same binary will also run perfectly fine if the container is based on one of Ubuntu images.
Any idea what this binary is missing?
This is what I've done to reproduce (successful run in VirtualBox/busybox on OS X not shown):
Build (building explicitly with flags even though the arch matches):
➜ artisan git:(master) ✗ GOOS=linux GOARCH=amd64 go build
Check it can run on the host:
➜ artisan git:(master) ✗ ./artisan
10:14:04.925 [ERROR] artisan: need a command, one of server, provision or build
Copy to docker dir, build, run:
➜ artisan git:(master) ✗ cp artisan docker/build/bin/
➜ artisan git:(master) ✗ cd docker
➜ docker git:(master) ✗ cat Dockerfile
FROM docker:1.10
COPY build/ /
➜ docker git:(master) ✗ docker build -t artisan .
Sending build context to Docker daemon 10.15 MB
Step 1 : FROM docker:1.10
...
➜ docker git:(master) ✗ docker run -it artisan sh
/ # /bin/artisan
sh: /bin/artisan: not found
Now changing the image base to phusion/baseimage:
➜ docker git:(master) ✗ cat Dockerfile
#FROM docker:1.10
FROM phusion/baseimage
COPY build/ /
➜ docker git:(master) ✗ docker build -t artisan .
Sending build context to Docker daemon 10.15 MB
Step 1 : FROM phusion/baseimage
...
➜ docker git:(master) ✗ docker run -it artisan sh
# /bin/artisan
08:16:39.424 [ERROR] artisan: need a command, one of server, provision or build
By default, if using the net package a build will likely produce a binary with some dynamic linking, e.g. to libc. You can inspect dynamically vs. statically link by viewing the result of ldd output.bin
There are two solutions I've come across:
Disable CGO, via CGO_ENABLED=0
Force the use of the Go implementation of net dependencies, netgo via go build -tags netgo -a -v, this is implemented for a certain platforms
From https://golang.org/doc/go1.2:
The net package requires cgo by default because the host operating system must in general mediate network call setup. On some systems, though, it is possible to use the network without cgo, and useful to do so, for instance to avoid dynamic linking. The new build tag netgo (off by default) allows the construction of a net package in pure Go on those systems where it is possible.
The above assumes that the only CGO dependency is the standard library's net package.
I had the same issue with a go binary, and I got it to work after adding this to my docker file:
RUN apk add --no-cache libc6-compat
Go compiler from your build machine probably links your binary with libraries on different location than in Alpine. In my case it was compiled with dependencies under /lib64 but Alpine does not use that folder.
FROM alpine:edge AS build
RUN apk update
RUN apk upgrade
RUN apk add --update go=1.8.3-r0 gcc=6.3.0-r4 g++=6.3.0-r4
WORKDIR /app
ENV GOPATH /app
ADD src /app/src
RUN go get server # server is name of our application
RUN CGO_ENABLED=1 GOOS=linux go install -a server
FROM alpine:edge
WORKDIR /app
RUN cd /app
COPY --from=build /app/bin/server /app/bin/server
CMD ["bin/server"]
I'm working on article about this issue. You can find draft with this solution here http://kefblog.com/2017-07-04/Golang-ang-docker .
What did the trick for me was enabling static linking in the linker options:
$ go build -ldflags '-linkmode external -w -extldflags "-static"'
The -linkmode option tells Go to use the external linker, the -extldflags option sets options to pass to the linker and the -w flag disables DWARF debug info to improve binary size.
See go tool link and Statically compiled Go programs, always, even with cgo, using musl
for more details
I had an app that required CGO_ENABLED=1.
The fix for me to run the compiled go binary in a debian-slim container was to build the binary using RUN GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go build -o goapp
And run the following commands in the debian slim
RUN apt-get update && apt-get install -y musl-dev
RUN ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1
Made me able to run the goapp afterwards
TIP: ldd goapp showed that libc.musl-x86_64 was missing in the container.
While executing a go binary inside a debian docker container, faced this issue:
/bin/bash: line 10: /my/go/binary: No such file or directory
The binary was built by using docker-in-docker (dind) from an alpine container using command:
GOOS=linux GOARCH=amd64 go build
Fixed it by using following env while building the binary:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build
I'm trying to compile linphone on mountain lion. I have downloaded the sources and followed all the instructions in the README.macos file. The first time, when I got to '$ port install ige-mac-integration' an error occurred. It said gtk2 had to be installed with x11, however, the instructions said to install it with quartz and no_x11. I tried installing gtk2 again, this time with x11, but when I get to the compiling process it tells me quartz is needed. I can't install both, because the gtk2 installation will complain about it, and it seems I need both to complete the entire process of installing and compiling linphone.
I have tried these steps with the downloadable sources, as well as the git sources, both gave me the same problems. Linphone does not have a forum, so I couldn't ask it on their site.
I just need to get the sources and start using them, so any recommendations on how to open the linphone source, edit it, and compile/run it would be very welcome. The ideal situation would be where I can just open an xcode project file.
This is the readme file:
**********************************
* Compiling linphone on macos X *
**********************************
You need:
- Xcode (download from apple or using appstore application)
- Macports: http://www.macports.org/
Download and install macports using its user friendly installer.
- Install build time dependencies
$ port install automake autoconf libtool intltool
- Install some linphone dependencies with macports
$ port install speex
$ port install libosip2 # WARNING: currently outdated in macport
$ port install libeXosip2 #WARNING: currently outdated in macport
$ port install ffmpeg-devel
$ port install libvpx
- Install srtp (optional) for call encryption
$ port install srtp
If that fails, get from source:
$ git clone git://git.linphone.org/srtp.git
$ cd srtp && autoconf && ./configure --prefix=/opt/local && make libsrtp.a
$ sudo make install
- Install zrtpcpp (optional), for unbreakable call encryption
$ port install cmake
$ git clone git://git.linphone.org/zrtpcpp.git
$ cd zrtpcpp && cmake -Denable_ccrtp=false . && make
$ sudo make install
- Install gtk. It is recommended to use the quartz backend for better integration.
$ port install gtk2 +quartz +no_x11
$ port install hicolor-icon-theme
- Compile and install the tunnelsu
If you got the source code from git, run ./autogen.sh first
Then or otherwise, do:
$ ./configure --prefix=/opt/local && make && sudo make install
- Compile linphone
If you got the source code from git, run ./autogen.sh first.
Then or otherwise, do:
$ ./configure --prefix=/opt/local --with-readline=/opt/local --disable-strict --disable-x11 --with-srtp=/opt/local --with-gsm=/opt/local --enable-zrtp && make
Install to /opt/local
$ sudo make install
Done.
If you want to generate a portable bundle, then install gtk-mac-bundler.
Use git:
$ git clone https://github.com/jralls/gtk-mac-bundler.git
$ cd gtk-mac-bundler && make install
$ export PATH=$PATH:~/.local/bin
#make this dummy charset.alias file for the bundler to be happy:
$ sudo touch touch /opt/local/lib/charset.alias
Then run, inside linphone source tree:
1. Run configure as told before but with "--enable-relativeprefix" appended.
$ make
$ make bundle
The resulting bundle is located in linphone build directory, together with a zipped version.
For a better appearance, you can install the gtk-quartz-engine (a gtk theme) that make gtk application more similar to other mac applications (but not perfect).
$ git clone https://github.com/jralls/gtk-quartz-engine.git
$ cd gtk-quartz-engine
$ autoreconf -i
$ ./configure --prefix=/opt/local && make
$ sudo make install
Generate a new bundle to have it included.
Below is the details to configure the dependencies and build linphone.
==============================================================
Step: 1:- Dependency configuration:
I just followed the instruction which is available in "README.macos" upto sqlite3 configuration. I had some problem with dependancies configuration.
Because of macport version, srtp configuration is failed. So I have downloaded "MacPorts-2.1.3-10.8-MountainLion.pkg" and installed manually.
Because of GTK version, libsoup is failed. So I have downloaded "GTK_2.18.5-X11.pkg" and installed manually.
After the manual installation, I have followed "README.macos" again for reconfiguration.
NOTE: Sometime terminal won't recognize "wget".
--> Solution: just execute below command.
echo 'alias wget="curl -O"' >> ~/.bash_profile
Step: 2:- Compile and Install:
I got some problem related to intltool while compiling.
Solution: just I set the path by executing below commands
export PATH=$PATH:/opt/local/bin
export MANPATH=$MANPATH:/opt/local/share/man
export INFOPATH=$INFOPATH:/opt/local/share/info
Now it is successfully compiled and installed.
Step: 3:- Creating bundle to run the app:
Just I followed below commands to make the build.
If you want to generate a portable bundle, then install gtk-mac-bundler.
Use git:
$ git clone https://github.com/jralls/gtk-mac-bundler.git
$ cd gtk-mac-bundler && make install
$ export PATH=$PATH:~/.local/bin
#make this dummy charset.alias file for the bundler to be happy:
$ sudo touch touch /opt/local/lib/charset.alias
Then run, inside linphone source tree:
1. Run configure as told before but with "--enable-relativeprefix" appended.
$ make
$ make bundle
It will create "linphone.app" file in current linphone directory. It will support only for "Mountain Lion".
Step: 4:- Support for lower version: (e.g.: Lion, Snow Leopard...)
We have to configure "libiconv hack" to supporting for lower version
I think it will help you.
I wrote an answer here. It's a bit long but I sincerely hope it is clear enough and will help you.
It contains the build settings and other libs you need to include.
Cool stuff I didn't need to run the make :)
Best,