I am migrating to circleci2.0 and after successful build when tests are executed they are randomly failing with following error messages
/usr/local/go/pkg/tool/linux_amd64/link: signal: killed
/usr/local/go/pkg/tool/linux_amd64/link: flushing $WORK/b462/payment_step_svc.test: write $WORK/b462/svc.test: cannot allocate memory
I used the following config
jobs:
build:
docker:
- image: circleci/golang:latest
- image: rabbitmq:3.5.4
- image: redis
working_directory: /go/src/github.com/soniabhishek/taskrunner
environment:
GOOS: linux
GOARCH: amd64
GOPATH: /go
steps:
- checkout
- run:
name: Get dependencies
command: go get -t -d -v ./...
- run:
name: Build all
command: go build ./...
- run:
name: Test all
command: go test -v ./...
I have tried for many golang versions other than latest like (1.10.3).
Although I found the hack for this but I am not sure why is this happening,
all my tests run when I use CGO_ENABLED=0
Would like to know why is this issue occurring and permanent solution for this
There are two clues here that suggest you are consuming too much memory, and the OS is forcibly killing your "rogue" process. They are:
killed
cannot allocate memory
You can confirm this by getting an SSH session at the end of a failed build, and examining the kill history using dmesg. If it gives you a "sacrifice child" message, then you hit a memory limit so severe that the OS was forced to remove a process from memory.
You have several options. I put them here in the order I would suggest you prefer them:
Reduce your memory consumption by making changes to your Go program
Ask CircleCI to set you up with a larger Docker container (this will be a chargeable option). The default is 4G, I believe 8G and 16G are available
Swap to a Machine executor (presently free of charge, subject to change). This gives you 8G of RAM, but you lose some of the flexibility of the Docker approach (e.g. you have to use one of a few preset base images).
What worked for me in this situation was go test -p 1 to tell Go to run just one compilation at a time.
Without that Go will run the number of CPUs you have, which by default on CircleCI is two.
Related
For my work purposes, I have to build a plugin for zabbix Agent2 that has to on Windows systems. For simplicity purposes I chose to go with a loadable plugin but more on that later.
I followed the guide at # https://www.zabbix.com/documentation/guidelines/en/plugins/loadable_plugins to build the example myip plugin on my Windows laptop and after linking it through an additionnal config file in the C:\Program Files\Zabbix Agent 2\zabbix_agent2.d\plugins.d directory, it works.
It works meaning that I can query the new item both with:
> zabbix_get -s localhost -k myip
> zabbix_agent -c zabbi_agent2.conf -t myip.
The only issue is the zabbix_agent2 process CPU usage sky rocks when I linked my loadable plugin, and restarted the agent.
I am talking about a usage of about 17-20% on my 8cores/1.99Ghz i7 laptop. That is just insane.
If I unlink the agent2 and the loadable plugin, the agent cpu usage comes back to normal (less than 1%).
After looking at the agent log file (zabbi_agent2.log), it is full of lines like
2022/11/10 09:22:16.817154 failed to read response for plugin Myip, failed to read type header, The handle is invalid.
When I run zabbix_agent -v -c zabbi_agent2.conf -t myip with the -v (verbose) option, I see the same error in the output multiple times before the actual value of the metric.
I suspect the high CPU usage is caused by this error happening and being logged too frequently.
After having read the source code of the plugin-support at https://git.zabbix.com/projects/AP/repos/plugin-support/browse/plugin/comms/connection.go (line 42), I recognized the line of code creating the error.
As there are builtin plugins for the agent2 too, it tried to recompile the agent2 on windows but was getting an even worse error while i have not even added my plugin yet.
I acutually wrote a post on that as well here (# https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/454191-error-during-zabbix-agent2-compilation-from-sources and Error during Windows Zabbix Agent2 compilation from sources)
Resources:
Go: dl.google.com/go/go1.19.3.windows-amd64.msi
Zabbix Agent2: https://cdn.zabbix.com/zabbix/binaries/stable/6.2/6.2.4/zabbix_agent2-6.2.4-windows-amd64-openssl.msi
I build the same plugin for the agent2 on linux both as a loadable and builtin plugin and it is working flawlessly so i think it really is a windows issue.
I have been testing Quarkus, to check if it's worth the replacing cost, when running an empty Quarkus project with only resteasy and jsonb , the RAM usage is of just 5MB. But after adding only some entities from a big project, the project has grown to 100MB, which I find as an abusive usage, as in the Spring one, the full project is using 180MB.
Reproduction repository: https://github.com/KevinGuancheDarias/quarkus-owge-poo
running natively steps (requires docker)
$ mvn package -Pnative -Dquarkus.native.container-build=true -DskipTests
$ docker build --no-cache -f src/main/docker/Dockerfile.native -t quarkus/test .
$ docker run -i --rm -p 8080:8080 quarkus/test
Command used to test the sizes
docker stats 2985d8366e51 32db5040f6e7
Sample output where owge_all_dockerized.... is the full project running in Docker and suspicious_borg is the quarkus test
Quarkus extensions: agroal, cdi, hibernate-orm, hibernate-orm-panache, jdbc-mysql, mutiny, narayana-jta, resteasy, resteasy-jsonb, smallrye-context-propagation
Is this the expected RAM usage?
Update: Removing only hibernate ORM, but preserving agroal and jdbc-mysql, makes the memory go down to 6MB
Update: Please note, that both the quarkus test and the full project running in Spring boot are immediately started, and stopped, no time for GC to fill up the memory
Thanks in advance!.
Native graalvm execution has a different kind of memory management than when running traditional Java/jar.
Details are to be found in the official docs but basically it will grow in memory more aggressively but you can limit it by adding -mx 10m for example. Then it will not grow beyond 10 mb - which if is enough for your app will be fine.
Of course if your app actually needs more then increase it.
I'm trying to setup a Docker container to build my java project with maven
I have created my Docker file FROM maven:3.2-jdk-7 and built the image.
when I execute with:
docker run -it --rm --name my-maven-project -v "$PWD":/usr/src/app -v
"$HOME"/.m2:/root/.m2 -w /usr/src/app -v "$HOME"/.ssh:/root/.ssh test
mvn clean package -Dmaven.test.skip=true
It takes about 20 minute to complete.. but if I run the the same mvn command on my host it takes 2minutes
I have tried giving more memory to the container by using
-m 4gb
But it didn't change anything, looking at docker stats the container barely used more than 2G
I'm running all this from OSX
Is there anything I need to do to have a maven finish in a decent time? I very very surprised it takes THAT much when on the host it takes 2minutes..
This what docker stats says after maven has been building for 10 min
CPU: 201.13%
Mem usage / limit : 2.508GiB
MEM % : 62.69%
NET I/O: 3.01kB / 861B
BLOCK I/O: 57.7MB / 2.23MB
PIDS: 38
- EDIT -
It turns out Docker for mac does not play well when using mounted volume.
In order to avoid having to git clone the project inside the container I preferred using using -v "$PWD":/usr/src/app
To test I have directly git cloned the app form within the container and now the build takes a normal amount of time (4minutes)
Note that the git clone took... 6 min!!! instead (1min on host) so in total from git clone to final build it still takes 10min which is ridiculous.
So yea OSX and Docker is a big no no when using mounted volume...
I ran into this same issue using the same docker run syntax as you (docker run -v src:dest). A Maven build that took ~30 seconds on my OSX host was taking ~4 minutes in my container. I didn't solve it entirely, but switching to explicitly use a bind mount took my builds from around 4 minutes down to about 1.5 minutes. This still isn't an acceptable increase in build time for my use case, but it may help someone else. Try switching your docker run command to this:
docker run --name=my-maven-project -it \
--mount type=bind,source="$(pwd)",destination=/usr/src/app,consistency=delegated <docker image name>
NOTE: The consistency option at the very end is only valid on OSX, and has two other values, either of which may be more appropriate for your situation. I tried all three out of curiosity and build times were comparable between the delegated and cached options, meanwhile the consistent option nearly as slow as the way I was doing it before (unsurprisingly). Here's the documentation:
https://docs.docker.com/storage/bind-mounts/
So, unfortunately, despite bind mounts being "very performant," they're still apparently at least twice as slow as a native filesystem when it comes to maven builds, at least on OSX. With luck that will improve as time goes on.
In our Jenkins agents we are running about several (around 20) tests whose setup involves running docker-compose up for a "big" number of services/containers (around 14).
From time to time, I'll get the following error:
ERROR: for testdb-data UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
Haven't been able to reproduce this consistently. And I'm still trying to figure out whether or not there is a correlation with our agent's resources being at full use.
docker -v is 1.10.1 and docker-compose -v is 1.13.1.
Any ideas about what this may be related to?
Restarting docker service:
sudo systemctl restart docker
and setting DOCKER_CLIENT_TIMEOUT and COMPOSE_HTTP_TIMEOUT environment variables:
export DOCKER_CLIENT_TIMEOUT=120
export COMPOSE_HTTP_TIMEOUT=120
are two workarounds for now. But the issues are still open in docker compose github:
https://github.com/docker/compose/issues/3927
https://github.com/docker/compose/issues/4486
https://github.com/docker/compose/issues/3834
I had the same problem. It was solved after change the max-file size value from a number to a string.
Wrong config
logging:
options:
max-file: 10
max-size: 10m
Correct config
logging:
options:
max-file: "10"
max-size: 10m
docker-compose down
Running docker-compose down and then running docker-compose up --build may work. I am working on vscode and when I encountered a similar problem while building docker, this solution worked for me.
Before performing above mentioned command its better you refer to what is the purpose of docker-compose down
docker-compose restart
Fixed my issue. This will restart all stopped and running services.
docker-compose down
export DOCKER_CLIENT_TIMEOUT=120
export COMPOSE_HTTP_TIMEOUT=120
docker-compose up -d
following steps worked
sometimes it could because of docker internal cache, for me following steps worked:
1: docker-compose down -v --rmi all --> to remove all the image and cache
2: docker-compose up --build
we are in the middle of process migrating from Jenkins to Concourse CI and everything was pretty smooth so far. But now I have the issue, that I don't know how to solve. I would like to get any advices from the community.
What I am trying to do is a job that can run integrational or functional (web) tests using Selenium. There are few issues for us:
To run web tests I need to set up the database (and optionally, the searching engine, proxy and etc...) proxy to imitate the production environment as close at possible.
Ideally, it should be set up by docker-compose.
This database service should run in parallel of my tests
This database service should not return anything, neither error or success, because it only starts the database and nothing else
My web-tests should not be started until the database is ready
This database service should be stopped when all the web-tests were finished
As you can see, it's pretty non-trivial task. Of course, I can create an big uber-container that contains everything I need, but this is bad solution. Another option is to create a shell-script for that, but this is not flexible enough.
Is there any example how I could implement that or good practices for this issue?
Thanks!
Since version 1.3.0 it appears you can run Docker-compose in a task: https://github.com/concourse/concourse/issues/324
This appears to work:
jobs:
- name: docker-compose
public: true
serial: true
plan:
- do:
- task: docker-compose
timeout: 20m
privileged: true
config:
platform: linux
image_resource:
type: docker-image
source: {repository: "mumoshu/dcind", tag: "latest"}
run:
path: sh
args:
- -exc
- |
source /docker-lib.sh
start_docker
docker ps
docker-compose version
This is comment from author of Concourse:
There is no Docker binary or socket on the host - they're just running a Garden backend (probably Guardian). Concourse runs at an abstraction layer above Docker, so providing any sort of magic there doesn't really make sense.
The one thing missing post-1.3 is that Docker requires you to set up cgroups yourself. I forgot how annoying that is. I wish they did what Guardian does and auto-configure it, but what can ya do.
So, the full set of instructions is:
Use or a build an image with docker in it, e.g. docker:dind.
Run the following at the start of your task: https://github.com/concourse/docker-image-resource/blob/master/assets/common.sh#L1-L40
Spin up Docker with docker daemon &.
Then you can run docker-compose and friends as normal.
The downside of this is that you'll be fetching the images every time. #230 will address that.
In the long run, #324 (comment) is the direction I want to go.
See here https://github.com/concourse/concourse/issues/324
as in the accepted answer, the Slack archive data is deleted (due to Slack limit)
The docker image specialized for the usecase: https://github.com/meAmidos/dcind
It does not sound that complicated to me. I wrote a post on how to get something similar up and running here. I use some different containers for the stack and the test runner and fire up everything from an official docker:dind image with docker-compose installed on it...
Beyond the usual concourse CI stuff of fetching resources etc.
Performing a test-run would consist of :
Starting the web,rest, and other services with docker-compose up.
Starting the Testrunner service and fire the test-suites on the
web-page which communicates with the rest layer, which in turn is
dependent on the other services for responses.
Performing docker-compose down when the test-runner completes and
deciding the return-code of the task (0=fail, 1=success) based upon
the return code of the test-suite.
To cleanly setup and tear down the stack and test runner you could do something like the below, ( maybe you could use depends if your service is not started when the test begins, for me it works without)
# Setup the SUT stack:
docker-compose up -d
# Run the test-runner container outside of the SUT to be able to teardown the SUT when testing is completed:
docker-compose run --rm test-runner --entrypoint '/entrypoint.sh /protractor/project/conf-dev.js --baseUrl=http://web:9000/dist/ --suite=my_suite'
# Store the return-code from the tests and teardown:
rc=$?
docker-compose down
echo "exit code = $rc "
kill %1
exit $rc