High Latency is being observed in AWS ARM Graviton Processor in Comparison to AMD Processor for ASGI based Django Application - amazon-ec2

I am running an Asgi-based Django Application(Rest Framework) using AWS Kubernetes in the production environment. Everything is running fine at AMD Processor(c5.2xlarge, c5.4xlarge). To decrease the cost we are trying to migrate the application to AWS Graviton Processor(c6g.2xlarge, c6g.4xlarge). But we are observing an increase in the 90% latency to 10X.
The command used for running the application -
DD_DJANGO_INSTRUMENT_MIDDLEWARE=false ddtrace-run gunicorn --workers 1 --worker-tmp-dir /dev/shm --log-file=- --thread 2 --bind :8080 --log-level INFO --timeout 5000 asgi:application -k uvicorn.workers.UvicornWorker
I have one more application that is WSGI based and it's working fine at the graviton processor.
Attaching the docker code -
FROM python:3.9-slim
RUN apt update -y
RUN mv /var/lib/dpkg/info/libc-bin.* /tmp/ && apt install libc-bin && mv /tmp/libc-bin.* /var/lib/dpkg/info/
#
### Create a group and user to run our app
## ARG APP_USER=user
## RUN groupadd -r ${APP_USER} && useradd --no-log-init -r -g ${APP_USER} ${APP_USER}
#
## Install packages needed to run your application (not build deps):
## mime-support -- for mime types when serving static files
## postgresql-client -- for running database commands
## We need to recreate the /usr/share/man/man{1..8} directories first because
## they were clobbered by a parent image.
RUN set -ex \
&& RUN_DEPS=" \
libpcre3 \
git \
mime-support \
postgresql-client \
libmagic1\
fail2ban libjpeg-dev libtiff5-dev zlib1g-dev libfreetype6-dev liblcms2-dev libxslt-dev libxml2-dev \
gdal-bin sysstat libpq-dev binutils libproj-dev procps" \
&& seq 1 8 | xargs -I{} mkdir -p /usr/share/man/man{} \
&& apt-get update && apt-get install -y --no-install-recommends $RUN_DEPS \
&& rm -rf /var/lib/apt/lists/*
ADD requirements /requirements
#ADD package.json package.json
#
## Install build deps, then run `pip install`, then remove unneeded build deps all in a single step.
## Correct the path to your production requirements file, if needed.
RUN set -ex \
&& BUILD_DEPS=" \
build-essential \
libpcre3-dev \
libpq-dev \
" \
&& apt-get update && apt-get install -y --no-install-recommends $BUILD_DEPS \
# && npm install --production --no-save \
&& pip install --no-cache-dir -r /requirements/requirements.txt \
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $BUILD_DEPS \
&& rm -rf /var/lib/apt/lists/*
#
RUN rm -rf /requirements
RUN mkdir /code/
WORKDIR /code/
ADD . /code/
COPY ./scripts /scripts
RUN chmod +x /scripts/*
RUN mkdir -p /vol/web/media
RUN mkdir -p /vol/web/static
RUN groupadd -r user
RUN useradd --no-log-init -r -g user user
RUN chown -R user:user /vol
RUN chmod -R 755 /vol/web
USER user
Python Modules -
aiohttp==3.8.1
aiosignal==1.2.0
amqp==5.1.1
anyio==3.6.1
asgiref==3.5.2
asttokens==2.0.5
async-timeout==4.0.2
attrs==21.4.0
aws-requests-auth==0.4.3
Babel==2.9.1
backcall==0.2.0
billiard==3.6.4.0
black==22.6.0
boto3==1.9.62
botocore==1.12.253
bytecode==0.13.0
celery==5.2.7
certifi==2022.6.15
charset-normalizer==2.0.12
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
ddsketch==2.0.4
ddtrace==1.3.0
decorator==5.1.1
Django==4.0.1
django-appconf==1.0.5
django-cache-memoize==0.1.10
django-compressor==3.1
django-compressor-autoprefixer==0.1.0
django-cors-headers==3.11.0
django-datadog-logger==0.5.0
django-elasticsearch-dsl==7.2.2
django-elasticsearch-dsl-drf==0.22.4
django-environ==0.8.1
django-extensions==3.1.5
django-libsass==0.9
django-log-request-id==2.0.0
django-nine==0.2.5
django-prometheus==2.2.0
django-sites==0.11
django-storages==1.12.3
django-uuid-upload-path==1.0.0
django-versatileimagefield==2.2
djangorestframework==3.13.1
djangorestframework-gis==0.18
docutils==0.15.2
elasticsearch==7.17.1
elasticsearch-dsl==7.4.0
executing==0.8.3
frozenlist==1.3.0
geographiclib==1.52
geopy==2.2.0
gunicorn==20.1.0
h11==0.12.0
httpcore==0.14.7
httpx==0.21.3
idna==3.3
ipython==8.0.1
jedi==0.18.1
jmespath==0.10.0
JSON-log-formatter==0.5.1
kombu==5.2.4
libsass==0.21.0
matplotlib-inline==0.1.3
multidict==6.0.2
mypy-extensions==0.4.3
packaging==21.3
parso==0.8.3
pathspec==0.9.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.2.0
platformdirs==2.5.2
prometheus-client==0.14.1
prompt-toolkit==3.0.30
protobuf==4.21.2
psycopg2-binary==2.9.3
ptyprocess==0.7.0
pure-eval==0.2.2
Pygments==2.12.0
pyparsing==3.0.9
python-dateutil==2.8.2
python-dotenv==0.19.2
python-magic==0.4.27
pytz==2021.3
rcssmin==1.1.0
regex==2022.1.18
requests==2.27.1
rfc3986==1.5.0
rjsmin==1.2.0
s3transfer==0.1.13
six==1.16.0
sniffio==1.2.0
sqlparse==0.4.2
stack-data==0.3.0
tenacity==8.0.1
tomli==2.0.1
traitlets==5.3.0
typing_extensions==4.3.0
urllib3==1.25.11
uvicorn==0.17.0
vine==5.0.0
wcwidth==0.2.5
whitenoise==5.3.0
yarl==1.7.2
Docker build command -
docker buildx build --push --platform linux/amd64,linux/arm64 -t ${ECR_LATEST_TAGX} -t ${ECR_VERSION_TAGX} --output=type=image --file ../incr.Dockerfile ../

Related

Laravel sail up fail on Linux Mint 20.2

I am trying to build a dockerized laravel project following the documentation at https://laravel.com/docs/8.x/installation#getting-started-on-linux
I ran the following:
curl -s https://laravel.build/laravel-app | bash ,
cd laravel-app,
./vendor/bin/sail up
After pulling tons of files, I believe it comes to bulid the Dockerfile, not sure which as there are 3 Dockerfiles in 3 different folders inside laravel-app/vendor/laravel/sail/runtimes, there are 3 folders named 7.4, 8.0, and 8.1, in which each contains a Dockerfile but pretty much the differences are so subtle. These Dockerfiles have 18 commands, so after the pulls are finished it says Step 1/18, 2/18, etc... However it always messes up in step 9, it starts saying "Get: ## http://..ubnutu.com...." where ## is from 1 till 127 and then at the line
"Get:127 http://archive.ubuntu.com/ubuntu hirsute/universe amd64 gosu amd64 1.12-1build1 [745 kB]"
it always quits and says:
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/f/fontconfig/libfontconfig1_2.13.1-4.2ubuntu3_amd64.deb Connection timed out [IP: 91.189.88.142 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
My internet connection is stable but it always stops after the 127th Get line.
Also, I ran the command several times and same result happens, so I do not believe it's a connection issue.
Here's the components of the Dockerfile in folder 8.1, it is the 9th step where it is messed up (the second RUN command):
FROM ubuntu:21.04
LABEL maintainer="Taylor Otwell"
ARG WWWGROUP
ARG NODE_VERSION=16
WORKDIR /var/www/html
ENV DEBIAN_FRONTEND noninteractive
ENV TZ=UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get update \
&& apt-get install -y gnupg gosu curl ca-certificates zip unzip git supervisor sqlite3 libcap2-bin libpng-dev python2 \
&& mkdir -p ~/.gnupg \
&& chmod 600 ~/.gnupg \
&& echo "disable-ipv6" >> ~/.gnupg/dirmngr.conf \
&& apt-key adv --homedir ~/.gnupg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys E5267A6C \
&& apt-key adv --homedir ~/.gnupg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C300EE8C \
&& echo "deb http://ppa.launchpad.net/ondrej/php/ubuntu hirsute main" > /etc/apt/sources.list.d/ppa_ondrej_php.list \
&& apt-get update \
&& apt-get install -y php8.1-cli php8.1-dev \
php8.1-pgsql php8.1-sqlite3 php8.1-gd \
php8.1-curl \
php8.1-imap php8.1-mysql php8.1-mbstring \
php8.1-xml php8.1-zip php8.1-bcmath php8.1-soap \
php8.1-intl php8.1-readline \
php8.1-ldap \
php8.1-msgpack php8.1-igbinary php8.1-redis php8.1-swoole \
php8.1-memcached php8.1-pcov php8.1-xdebug \
&& php -r "readfile('https://getcomposer.org/installer');" | php -- --install-dir=/usr/bin/ --filename=composer \
&& curl -sL https://deb.nodesource.com/setup_$NODE_VERSION.x | bash - \
&& apt-get install -y nodejs \
&& npm install -g npm \
&& curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
&& echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list \
&& apt-get update \
&& apt-get install -y yarn \
&& apt-get install -y mysql-client \
&& apt-get install -y postgresql-client \
&& apt-get -y autoremove \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN setcap "cap_net_bind_service=+ep" /usr/bin/php8.1
RUN groupadd --force -g $WWWGROUP sail
RUN useradd -ms /bin/bash --no-user-group -g $WWWGROUP -u 1337 sail
COPY start-container /usr/local/bin/start-container
COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
COPY php.ini /etc/php/8.1/cli/conf.d/99-sail.ini
RUN chmod +x /usr/local/bin/start-container
EXPOSE 8000
ENTRYPOINT ["start-container"]
Also, I tried to execute the 9th step manually, turns out that probably the problem is in the part of step 9 which says
"echo "disable-ipv6" >> ~/.gnupg/dirmngr.conf", it results in "bash: /home/${user}/.gnupg/dirmngr.conf: Permission denied", I ran it with sudo and chmod u+x but same result.

neo4j 4.0 testing with embedded database DatabaseManagementServiceBuilder is found nowhere

I'm a beginner in neo4j. I'm trying to build tests using embedded neo4j database inside a springboot application. I haven't succeeded since the class DatabaseManagementServiceBuilder is found nowhere Please note I'm using version 4.0.2 Any help please ?
The full classname is org.neo4j.dbms.api.DatabaseManagementServiceBuilder.
Here is a sample class that uses the builder.
I've struggled with using an embedded neo4j db for my tests a few months back as well.
In case you don't find a suitable solution for the embedded version, I ended up starting a real instance of the db...
I adjusted a bit neo4j's official Dockerfile to use jdk instead of jre and was able to run my tests against it.
Here's the Dockerfile, starting from the official 3.4.5-enterprise Dockerfile:
FROM openjdk:8-jdk-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
make && \
rm -rf /var/lib/apt/lists/*
ADD maven-settings.xml /root/.m2/settings.xml
# from official neo4j:3.4.5-enterprise image content (changed base image to use jdk instead of jre)
RUN addgroup --system neo4j && adduser --system --no-create-home --home /var/lib/neo4j --ingroup neo4j neo4j
ENV NEO4J_SHA256=0629f17a99ba90d6900c98f332c775a732cc2ad6298b8df41a2872277b19e6e3 \
NEO4J_TARBALL=neo4j-enterprise-3.4.5-unix.tar.gz \
NEO4J_EDITION=enterprise \
NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
TINI_VERSION="v0.18.0" \
TINI_SHA256="12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855"
ARG NEO4J_URI=http://dist.neo4j.org/neo4j-enterprise-3.4.5-unix.tar.gz
RUN apt update \
&& apt install -y \
bash \
curl \
&& curl -L --fail --silent --show-error "https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini" > /sbin/tini \
&& echo "${TINI_SHA256} /sbin/tini" | sha256sum -c --strict --quiet \
&& chmod +x /sbin/tini \
&& curl --fail --silent --show-error --location --remote-name ${NEO4J_URI} \
&& echo "${NEO4J_SHA256} ${NEO4J_TARBALL}" | sha256sum -c --strict --quiet \
&& tar --extract --file ${NEO4J_TARBALL} --directory /var/lib \
&& mv /var/lib/neo4j-* /var/lib/neo4j \
&& rm ${NEO4J_TARBALL} \
&& mv /var/lib/neo4j/data /data \
&& chown -R neo4j:neo4j /data \
&& chmod -R 777 /data \
&& chown -R neo4j:neo4j /var/lib/neo4j \
&& chmod -R 777 /var/lib/neo4j \
&& ln -s /data /var/lib/neo4j/data
# Install latest su-exec
RUN set -ex; \
\
curl -o /usr/local/bin/su-exec.c https://raw.githubusercontent.com/ncopa/su-exec/master/su-exec.c; \
\
fetch_deps='gcc libc-dev'; \
apt-get update; \
apt-get install -y --no-install-recommends $fetch_deps; \
rm -rf /var/lib/apt/lists/*; \
gcc -Wall \
/usr/local/bin/su-exec.c -o/usr/local/bin/su-exec; \
chown root:root /usr/local/bin/su-exec; \
chmod 0755 /usr/local/bin/su-exec; \
rm /usr/local/bin/su-exec.c; \
\
apt-get purge -y --auto-remove $fetch_deps
ENV PATH /var/lib/neo4j/bin:$PATH
ARG NEO4J_AUTH=neo4j/neo4jtest
ENV NEO4J_AUTH=${NEO4J_AUTH}
WORKDIR /var/lib/neo4j
VOLUME /data
COPY docker-entrypoint.sh /docker-entrypoint.sh
EXPOSE 7474 7473 7687
ENTRYPOINT ["/sbin/tini", "-g", "--", "/docker-entrypoint.sh"]
CMD ["neo4j"]
I used the original docker-entrypoint.sh script.

Excecute .sh file on start of Docker container and let container run

I have a docker container that serves a webserver. On every startup of the container, I want to excecute a little shell script. The script that has to be executed has only one statement.
/var/www/html/app/Console/cake schema update -y
To achieve this, I created a .sh file called schemaupdate.sh which I copy into the docker container using the dockerfile into the /etc/init.d folder. Furthermore I make it executable and register it to the startup.
COPY schemaupdate.sh /etc/init.d/schemaupdate.sh
chmod 755 /etc/init.d/schemaupdate.sh
update-rc.d schemaupdate.sh defaults
The file is successfully copied into the container. However, the script is not executed when the docker container starts. When I manually call the sh file, everything is running fine.
How can I achieve, that the file / statement is executed on each startup of a container? It is important, that the script is executed at the startup and the container (so the webserver) still continues to run! The script only makes a little update check and after the check the webserver keeps on going.
The container is a debian based container. Here is inital dockerfile.
#start with base Image from php
FROM php:7.3-apache
#install system dependencies and enable PHP modules
RUN apt-get update && apt-get install -y \
libicu-dev \
libpq-dev \
libmcrypt-dev \
mysql-client \
git \
zip \
unzip \
&& rm -r /var/lib/apt/lists/* \
&& docker-php-ext-configure pdo_mysql --with-pdo-mysql=mysqlnd \
&& docker-php-ext-install \
intl \
mbstring \
pcntl \
pdo_mysql \
pdo_pgsql \
pgsql \
opcache
# zip \
# mcrypt \
#configure imap for mails
RUN apt-get update && \
apt-get install -y \
libc-client-dev libkrb5-dev && \
rm -r /var/lib/apt/lists/*
RUN docker-php-ext-configure imap --with-kerberos --with-imap-ssl && \
docker-php-ext-install -j$(nproc) imap
#install mcrypt
RUN apt-get update \
&& apt-get install -y libmcrypt-dev \
&& rm -rf /var/lib/apt/lists/* \
&& pecl install mcrypt-1.0.2 \
&& docker-php-ext-enable mcrypt
#install composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin/ --filename=composer
#set our application folder as an environment variable
ENV APP_HOME /var/www/html
#change uid and gid of apache to docker user uid/gid
RUN usermod -u 1000 www-data && groupmod -g 1000 www-data
#change the web_root to cakephp /var/www/html/webroot folder
#RUN sed -i -e "s/html/html\/webroot/g" /etc/apache2/sites-enabled/000-default.conf
# enable apache module rewrite
RUN a2enmod rewrite
#copy source files and run composer
#COPY src/ /var/www/html
#COPY src/ $APP_HOME
# install all PHP dependencies
#RUN composer install --no-interaction
#SET Volume
VOLUME /var/www/html/
#change ownership of our applications
RUN chown -R www-data:www-data $APP_HOME
#SET ENV VARIABLES
COPY schemaupdate.sh /etc/init.d/schemaupdate.sh
chmod 755 /etc/init.d/schemaupdate.sh
update-rc.d schemaupdate.sh defaults
EXPOSE 80
/etc/init.d/ isn't relevant. Containers aren't full blown operating systems with a heavyweight SysV init-style startup sequence. They run a single command, that's it.
You should either add the command as a RUN statement in the Dockerfile so its results are baked into the image, or you should have it called directly by the container's CMD or ENTRYPOINT directive.
I Finally used the Entrypoint. I deleted the COPY, chmod and update-rc. The Entrypoint looks like the following.
ENTRYPOINT [ "sh", "-c", "/var/www/html/app/Console/cake schema update -y && /var/www/html/app/Console/cake schema update -y && /usr/sbin/apachectl -D FOREGROUND"]
It first starts the update statment. After this is finished (so terminated), the apachectl is called to keep the webserver running.

docker container nginx start but keeps hanging

my dockerfile:
FROM AWS_ECR_IMAGE
RUN apt-get update && apt-get install -y \
cron \
python-dev \
git \
zlib1g-dev \
libffi-dev \
libssl-dev \
autotools-dev \
automake \
libbz2-dev \
libaio-dev \
libsasl2-dev \
python-pip
RUN pip install boto boto3 awscli
# Install Nginx.
RUN \
add-apt-repository -y ppa:nginx/stable && \
apt-get update && \
apt-get install -y nginx && \
rm -rf /var/lib/apt/lists/* && \
echo "\ndaemon off;" >> /etc/nginx/nginx.conf && \
chown -R www-data:www-data /var/lib/nginx
# Define mountable directories.
VOLUME ["/etc/nginx/sites-enabled", "/etc/nginx/certs", "/etc/nginx/conf.d", "/var/log/nginx", "/var/www/html"]
# Define working directory.
WORKDIR /etc/nginx
# Define default command.
CMD ["nginx"]
COPY nginx_conf /etc/nginx/sites-available/default
# Start service, replace server name, update web ui
COPY main.sh /opt/annotation-pipeline-docs/main.sh
RUN chmod 0755 /opt/annotation-pipeline-docs/main.sh
ENTRYPOINT [ "sh", "-c", "/opt/annotation-pipeline-docs/main.sh" ]
# Expose ports.
EXPOSE 80
And my entrypoint bash file (I need to update the server name first when the container runs) is:
#!/bin/bash -e
/usr/local/bin/aws s3 sync s3://${S3_Bucket}/docs/${ENVIRONMENT}/HEAD/ /var/www/html/
if [ "$ENVIRONMENT" == "prod" ]
then
sed -i.bak "s/REPLACE_ME/example.com/g" /etc/nginx/sites-available/default
else
sed -i.bak "s/REPLACE_ME/example-$ENVIRONMENT.com/g" /etc/nginx/sites-available/default
fi
nginx
while true; do
sleep 60
echo "s3 sync again:"
/usr/local/bin/aws s3 sync s3://${S3_Bucket}/docs/${ENVIRONMENT}/HEAD/ /var/www/html/
done
The issue is when
nginx
runs, it will hanging forever in the terminal:
and the while loop will never get called. Anyone know why is hanging and how to resolve it? Please help, tks in advanced.
The reason for my issue is waiting for the traffic, the while loop will never get called until Nginx start free the bash. However, Nginx will be running in the foreground and not release the focus.
The solution I tried is instead of letting Nginx running as a foreground service, I changed it run in the background. Since this is the only service in my container, should have no problem to do it.
the code changed is simply removed below line in my dockerfile:
echo "\ndaemon off;" >> /etc/nginx/nginx.conf
which will make Nginx as a foreground service

Failed to Call Access Method Exception when Creating a MedicationOrder in FHIR

I am using this http://fhirtest.uhn.ca/baseDstu2 test FHIR server and it worked okay so far.
Now I am getting an HTTP-500 - Failed to Call Access Method exception.
Anyone has any idea on what has gone wrong?
This happens frequently. Probably because someone tested weird queries or similar that put the server in an unstable status.
I suggest posting a comment in https://chat.fhir.org/#narrow/stream/hapi to get the server restarted,
or install http://hapifhir.io/doc_cli.html which does basically the same but you have full control.
I built a Dockerfile:
FROM debian:sid
MAINTAINER Günter Zöchbauer <guenter#yyy.com>
ENV DEBIAN_FRONTEND noninteractive
RUN \
apt-get -q update && \
DEBIAN_FRONTEND=noninteractive && \
apt-get install --no-install-recommends -y -q \
apt-transport-https \
apt-utils \
wget \
bzip2 \
default-jdk
# net-tools sudo procps telnet
RUN \
apt-get update && \
rm -rf /var/lib/apt/lists/*
https://github.com/jamesagnew/hapi-fhir/releases/download/v2.0/hapi-fhir-2.0-cli.tar.bz2 && \
ADD hapi-* /hapi_fhir_cli/
RUN ls -la
RUN ls -la /hapi_fhir_cli
ADD prepare_server.sh /hapi_fhir_cli/
RUN \
cd /hapi_fhir_cli && \
bash -c /hapi_fhir_cli/prepare_server.sh
ADD start.sh /hapi_fhir_cli/
WORKDIR /hapi_fhir_cli
EXPOSE 5555
ENTRYPOINT ["/hapi_fhir_cli/start.sh"]
Which requires in the same directory as the Dockerfile
prepare_server.sh
#!/usr/bin/env bash
ls -la
./hapi-fhir-cli run-server --allow-external-refs &
while ! timeout 1 bash -c "echo > /dev/tcp/localhost/8080"; do sleep 10; done
./hapi-fhir-cli upload-definitions -t http://localhost:8080/baseDstu2
./hapi-fhir-cli upload-examples -c -t http://localhost:8080/baseDstu2
start.sh
#!/usr/bin/env bash
cd /hapi_fhir_cli
./hapi-fhir-cli run-server --allow-external-refs -p 5555
Build
docker build myname/hapi_fhir_cli_dstu2 -t . #--no-cache
Run
docker run -d -p 5555:5555 [image id from docker build]
Hope this helps.

Resources