When using TeamCity with the AWS Cloud Profile, it spins up the agent every 10 minutes for no reason.
I am using the agent to build a Yocto distribution, which takes a lot of time and resources. For that reason the agent is running on a large (expensive) instance which I only want to start when needed.
What I am trying to accomplish:
Have a cheap AWS instance running TeamCity server, always on.
Have a big, expensive AWS instance running a TeamCity Agent, but only turned on when needed.
Only spin up the agent machine when there is a build to run.
Place a tag in git when the build passes.
Preferably: Check for new git commits on the server, and start the agent when there is a commit.
Please note that I want to do tagging on the Agent, so server-side checkout will not work (according to the TeamCity documentation).
I created a setup in AWS where I have a small instance running TeamCity server, and a large (expensive) instance running the agent. In TeamCity, I configured a cloud profile in the Root project to automatically start the machine running the agent, and stop it when the build is finished or the machine has been idle for 5 minutes. Starting and stopping the AWS machine works, and triggering a build als works.
I created a project with a build configuration without any Triggers added to it. I also tried to increase the minimum checking interval for the VCS root for our git repository to 90000000 seconds (so that it will practically never spin up the agent machine).
For some reason, TeamCity spins up the AWS instance running the Agent every 10 minutes and it is not clear to me why it is doing this. The TeamCity log shows:
[2019-03-19 10:51:15,894] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'f1nqO09...' was created, agent: "yocto-build-agent1" {id=3}
[2019-03-19 10:56:33,250] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'f1nqO09...' was closed, agent: "yocto-build-agent1" {id=3}
[2019-03-19 10:56:33,257] INFO - jetbrains.buildServer.CLEANUP - Removing agent Unregistered agent "yocto-build-agent1" {id=3}
[2019-03-19 10:56:33,262] INFO - jetbrains.buildServer.AGENT - Agent "yocto-build-agent1" {id=3} has been unregistered. Reason: Agent has unregistered
[2019-03-19 11:02:30,612] INFO - jetbrains.buildServer.AGENT - Agent has been registered: "yocto-build-agent1" {id=4, protocol=unidirectional, host=10.55.20.127:9090, agentTypeId=3, pool=Default, registered since 2019-03-19 11:02:30.590}, not running a build
[2019-03-19 11:02:30,615] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'FcY5MVV...' was created, agent: "yocto-build-agent1" {id=4}
[2019-03-19 11:02:30,666] WARN - jetbrains.buildServer.AGENT - Agent "yocto-build-agent1" {id=4} is unauthorized on registration
[2019-03-19 11:07:09,821] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'FcY5MVV...' was closed, agent: "yocto-build-agent1" {id=4}
[2019-03-19 11:07:09,823] INFO - jetbrains.buildServer.AGENT - Agent "yocto-build-agent1" {id=4} has been unregistered. Reason: Agent has unregistered
[2019-03-19 11:07:22,455] INFO - jetbrains.buildServer.AGENT - Agent has been registered: "yocto-build-agent1" {id=4, protocol=unidirectional, host=10.55.20.127:9090, agentTypeId=3, pool=Default
The teamcity-clouds.log shows:
[2019-03-19 10:50:38,910] INFO [uled executor 1] - .instances.StoppedInstanceTask - Instance has changed status from stopped to Starting: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Starting}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 10:50:43,910] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'starting' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Starting}
[2019-03-19 10:51:15,883] INFO [nio-8111-exec-2] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 10:51:15,883] INFO [nio-8111-exec-2] - .server.impl.CloudEventsLogger - Detected cloud agent "yocto-build-agent1" {id=3}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Starting}
[2019-03-19 10:51:43,912] INFO [uled executor 2] - .server.impl.CloudEventsLogger - Cloud instance entered 'running' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}
[2019-03-19 10:56:32,807] INFO [uled executor 2] - te.IdleTimeoutTerminateFactory - Will stop instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} with idle time 5 minutes (idle time is 5 minutes)
[2019-03-19 10:56:32,808] INFO [uled executor 2] - l.instances.StopInstanceAction - Terminating instance: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, reason: "After 5 minutes idle"
[2019-03-19 10:56:32,808] INFO [uled executor 2] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 10:56:33,004] INFO [2 Stop Instance] - r.impl.DBCloudStateManagerImpl - Image: i-071eff7574b93d12f, Instance: i-071eff7574b93d12f, profile=amazon-2 is marked with state: stopped.
[2019-03-19 10:56:33,253] INFO [nio-8111-exec-7] - ces.terminate.AgentRemoverImpl - Removing cloud agent "yocto-build-agent1" {id=3} because: Instance has been terminated. Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 10:56:33,260] INFO [nio-8111-exec-7] - .server.impl.CloudEventsLogger - Cloud agent Unregistered agent "yocto-build-agent1" {id=3} is no longer recognized as cloud agent because instance has gone: profileId=amazon-2, imageId=i-071eff7574b93d12f, instanceId=i-071eff7574b93d12f
[2019-03-19 10:56:33,260] INFO [nio-8111-exec-7] - ces.terminate.AgentRemoverImpl - Removing cloud agent Unregistered agent "yocto-build-agent1" {id=3} because: Cloud instance (id=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, profileId=amazon-2) has gone (is not reported by cloud profile)
[2019-03-19 10:56:33,922] INFO [uled executor 2] - .server.impl.CloudEventsLogger - Cloud instance entered 'scheduled to stop' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Scheduled to stop}
[2019-03-19 10:56:43,923] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopping' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopping}
[2019-03-19 10:57:43,925] INFO [uled executor 1] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopped' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}
[2019-03-19 11:02:30,613] WARN [io-8111-exec-11] - .instances.StoppedInstanceTask - Agent "yocto-build-agent1" {id=4} was not matched to stopped instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:02:32,705] WARN [nio-8111-exec-4] - .instances.StoppedInstanceTask - Agent "yocto-build-agent1" {id=4} was not matched to stopped instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:02:43,939] INFO [uled executor 4] - .instances.StoppedInstanceTask - Instance has changed status from stopped to Running: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:02:48,939] INFO [uled executor 3] - .server.impl.CloudEventsLogger - Cloud instance entered 'running' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}
[2019-03-19 11:07:22,457] INFO [nio-8111-exec-1] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 11:07:22,458] INFO [nio-8111-exec-1] - .server.impl.CloudEventsLogger - Detected cloud agent "yocto-build-agent1" {id=4}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}
[2019-03-19 11:12:32,818] INFO [uled executor 5] - te.IdleTimeoutTerminateFactory - Will stop instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} with idle time 5 minutes (idle time is 5 minutes)
[2019-03-19 11:12:32,818] INFO [uled executor 5] - l.instances.StopInstanceAction - Terminating instance: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, reason: "After 5 minutes idle"
[2019-03-19 11:12:32,818] INFO [uled executor 5] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 11:12:33,050] INFO [2 Stop Instance] - r.impl.DBCloudStateManagerImpl - Image: i-071eff7574b93d12f, Instance: i-071eff7574b93d12f, profile=amazon-2 is marked with state: stopped.
[2019-03-19 11:12:33,403] INFO [nio-8111-exec-6] - ces.terminate.AgentRemoverImpl - Removing cloud agent "yocto-build-agent1" {id=4} because: Instance has been terminated. Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:12:33,415] INFO [nio-8111-exec-6] - .server.impl.CloudEventsLogger - Cloud agent Unregistered agent "yocto-build-agent1" {id=4} is no longer recognized as cloud agent because instance has gone: profileId=amazon-2, imageId=i-071eff7574b93d12f, instanceId=i-071eff7574b93d12f
[2019-03-19 11:12:33,415] INFO [nio-8111-exec-6] - ces.terminate.AgentRemoverImpl - Removing cloud agent Unregistered agent "yocto-build-agent1" {id=4} because: Cloud instance (id=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, profileId=amazon-2) has gone (is not reported by cloud profile)
[2019-03-19 11:12:33,960] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'scheduled to stop' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Scheduled to stop}
[2019-03-19 11:12:43,960] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopping' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopping}
[2019-03-19 11:15:43,967] INFO [uled executor 3] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopped' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}
(instance id and agent name have been changed to protect the innocent)
How can I prevent TeamCity Server from spinning up the agent machine, even if there are no triggers configured?
What is the best way to start the agent machine when there is a git commit? I was thinking of a separate build on a smaller agent which only checks commits, and make the larger build depend on that smaller build.
This is not normal behavior. Logic is: Teamcity will start a cloud build agent if there are no on-premise agents available. Then it will terminate if when idle, depending on the options you provide.
I created several cloud profiles, in the Root project. I use below options:
Additional terminate conditions / If idle, stop 3 minutes before the full hour
Maximum instances count: x
Not sure if that's helpful but I don't have any "was not matched to stopped instance" entries in the teamcity-clouds.log, which I can see in your log.
Also - did you try enabling debug log level for cloud profiles? You can do it in Administration / Diagnostics (/admin/admin.html?item=diagnostics). Then check log files again.
The logging in TeamCity says "State changed from Stopped to Starting", which made me think it was TeamCity that started this agent. As it turned out, the AWS Instance which is running the build agent was started by OpsWorks.
Amazon OpsWorks was configured to monitor the state of the machine and keep it "up". So when TeamCity correctly shuts the server down, Amazon OpsWorks detects the machine going down, and tries to start it up again. After a while TeamCity detects it being "up" and "idle" and stops it again. This goes on indefinetely.
It would be nice to have an OpsWorks feature in TeamCity Cloud configuration in addition to the EC2 and VMWare features. This way TeamCity can tell Opsworks that the machine is supposed to go down.
For now, we have fixed the problem by disabeling OpsWorks health monitoring for the agents.
I'm trying to setup Teamcity on an EC2 Ubuntu instance.
I've:
Downloaded the package through wget http://download.jetbrains.com/teamcity/TeamCity-9.0.4.tar.gz. Unpackaged it.
Unpackaged it: tar -xvzf TeamCity-9.0.4.tar.gz.
sh Teamcity/bin/teamcity-server.sh start.
I can't access the web-app, and when I check the logs, says:
INFO - jetbrains.buildServer.STARTUP - Server shutdown has been requested
WARN - jetbrains.buildServer.STARTUP - Could not determine the local IP address: ip-172-30-1-83: ip-172-30-1-83: unknown error
INFO - jetbrains.buildServer.STARTUP - Server IP address: -unknown-
INFO - jetbrains.buildServer.STARTUP - Server shutdown complete
Complete log:
INFO - jetbrains.buildServer.STARTUP - Starting TeamCity server
INFO - jetbrains.buildServer.STARTUP - Server shutdown has been requested
INFO - jetbrains.buildServer.STARTUP - TeamCity version: 9.0.4 (build 32407), data format version 709
INFO - jetbrains.buildServer.STARTUP - OS: Linux, User: ubuntu, JRE: 1.8.0_45-b14 Java HotSpot(TM) 64-Bit Server VM
INFO - jetbrains.buildServer.STARTUP - JVM parameters: -Djava.util.logging.config.file=/home/ubuntu/TeamCity/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx512m -XX:MaxPermSize=270m -Dlog4j.configuration=file:/home/ubuntu/TeamCity/bin/../conf/teamcity-server-log4j.xml -Dteamcity_logs=../logs/ -Djsse.enableSNIExtension=false -Djava.awt.headless=true -Djava.endorsed.dirs=/home/ubuntu/TeamCity/endorsed -Dcatalina.base=/home/ubuntu/TeamCity -Dcatalina.home=/home/ubuntu/TeamCity -Djava.io.tmpdir=/home/ubuntu/TeamCity/temp
INFO - jetbrains.buildServer.STARTUP - Garbage collection: Copy, MarkSweepCompact
WARN - jetbrains.buildServer.STARTUP - Could not determine the local IP address: ip-172-30-1-83: ip-172-30-1-83: unknown error
INFO - jetbrains.buildServer.STARTUP - Web server version: Apache Tomcat/7.0.59
INFO - jetbrains.buildServer.STARTUP - Server IP address: -unknown-
INFO - jetbrains.buildServer.STARTUP - Servlet real path: /home/ubuntu/TeamCity/webapps/ROOT/
INFO - jetbrains.buildServer.STARTUP - Current stage: TeamCity server is shutting down
INFO - jetbrains.buildServer.STARTUP - Stopping the main application
INFO - jetbrains.buildServer.STARTUP - The main servlet is not initialized, nothing to shut down.
INFO - jetbrains.buildServer.STARTUP - Server shutdown complete
Any hints on how to solve it?
I have Teamcity's web-app working on my local machine (also Ubuntu, and the same Teamcity package).
Read the shell script TeamCity/bin/teamcity-server.sh, it is written with limitation, the script can't run out of the application folder.
export CATALINA_OPTS
CATALINA_HOME=./..
CATALINA_BASE=./..
So you should cd to the bin folder and run :
cd Teamcity/bin
sh teamcity-server.sh start
Let me know if it fixs your issue or not.
Is it possible to run more than one Bamboo Agent per EC2 Elastic Instance?
We use Docker in our build system and the process seems mostly network IO bound. It would be nice if we could run multiple Agents on one machine.
By running multiple agents on one machine instead of starting multiple machines, we also don't need to worry about shipping Docker images between machines as artifacts.
Update 30 Oct 2014:
I tried copying the current startup script and adding a new home:
$ cat /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent2
#!/bin/sh
bambooAgentBin=$(cd -P -- $(dirname $0) && pwd)
. $bambooAgentBin/bambooShellFunctions.sh
echo Starting Elastic Bamboo Agent...
java -Dbamboo.home=/home/bamboo/bamboo-agent-home-2/ -Dimagefiles.version=3.3-SNAPSHOT -jar $bambooAgentBin/*installer*.jar 2>&1 | tee -a $(getHomeDir)/bamboo-elastic-agent.out
It seems like the elastic version sets up some kind of a tunnel and crashes because a tunnel is already running:
# su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent2 - bamboo &
[1] 14143
Starting Elastic Bamboo Agent...
2014-10-30 08:39:31,804 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-e1/5.6-OD-01-0070/cce16404c14f06456c6adc44181746abf8dd1206/ to /opt/bamboo-elastic-agent
2014-10-30 08:39:31,979 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-e1/5.6-OD-01-0070/cce16404c14f06456c6adc44181746abf8dd1206/ to /opt/bamboo-elastic-agent
2014-10-30 08:39:31,979 INFO [main] [S3Utils] Fetching the list of remote objects...
2014-10-30 08:39:33,006 INFO [main] [S3Utils] Found 579 files in s3://bamboo-agent-release-us-e1/5.6-OD-01-0070/cce16404c14f06456c6adc44181746abf8dd1206/
2014-10-30 08:39:33,059 INFO [main] [S3Utils] Found 463 files in /opt/bamboo-elastic-agent
2014-10-30 08:39:33,060 INFO [main] [S3Utils] Generating the list of files to fetch from S3...
2014-10-30 08:39:33,076 INFO [main] [S3Utils] Generating the list of files to remove...
2014-10-30 08:39:33,078 INFO [main] [S3Utils] Removing 0 files from /opt/bamboo-elastic-agent
2014-10-30 08:39:33,079 INFO [main] [S3Utils] Fetching 155 files to /opt/bamboo-elastic-agent
2014-10-30 08:39:39,969 INFO [main] [S3Utils] Fetched 113 MB from S3
2014-10-30 08:39:39,973 INFO [main] [ElasticAgentInstaller] Starting [java, -server, -Xms32m, -Xmx256m, -XX:MaxPermSize=128m, -XX:+HeapDumpOnOutOfMemoryError, -Dimagefiles.version=3.3, -Dbamboo.agent.installDir=/opt/bamboo-elastic-agent, -cp, /opt/bamboo-elastic-agent/boot/annotations-13.0.jar:/opt/bamboo-elastic-agent/boot/gson-2.2.2-atlassian-1.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-api-agent-bootstrap-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/commons-io-2.4.jar:/opt/bamboo-elastic-agent/boot/jackson-core-2.1.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-shared-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/atlassian-tunnel-0.21.jar:/opt/bamboo-elastic-agent/boot/stax-api-1.0-2.jar:/opt/bamboo-elastic-agent/boot/guava-bridge-11.0.2-atlassian-01.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/commons-codec-1.8.jar:/opt/bamboo-elastic-agent/boot/atlassian-util-concurrent-2.4.1.jar:/opt/bamboo-elastic-agent/boot/joda-time-2.3.jar:/opt/bamboo-elastic-agent/boot/log4j-1.2.15.jar:/opt/bamboo-elastic-agent/boot/guava-11.0.2-atlassian-01.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-bootstrap-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/commons-lang-2.6.jar:/opt/bamboo-elastic-agent/boot/atlassian-aws-1.0.71.jar:/opt/bamboo-elastic-agent/boot/jackson-databind-2.1.1.jar:/opt/bamboo-elastic-agent/boot/fugue-1.1.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-1.7.1.jar:/opt/bamboo-elastic-agent/boot/httpclient-4.2.5.jar:/opt/bamboo-elastic-agent/boot/commons-logging-1.0.4.jar:/opt/bamboo-elastic-agent/boot/jackson-annotations-2.1.1.jar:/opt/bamboo-elastic-agent/boot/bcprov-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-core-agent-bootstrap-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/bcpkix-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/atlassian-annotations-0.4.jar:/opt/bamboo-elastic-agent/boot/jsr305-1.3.9.jar:/opt/bamboo-elastic-agent/boot/httpcore-4.2.5.jar:, com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap]
2014-10-30 08:39:40,119 INFO [main] [ElasticAgentBootstrap] Starting Agent Bootstrap using Java 1.6.0_45 from Sun Microsystems Inc.
2014-10-30 08:39:40,410 INFO [main] [ElasticAgentBootstrap] Using tunnelling. Setting virtual host name to https://xxxxxxx.atlassian.net/builds/agentServer/
2014-10-30 08:39:40,410 INFO [main] [ElasticAgentBootstrap] Using tunnelling for HTTP(S). Registering 'httpt' and 'httpst' protocols.
2014-10-30 08:39:40,416 INFO [main] [ElasticAgentBootstrap] HTTP(S) tunnel: enabled
2014-10-30 08:39:40,416 INFO [main] [ElasticAgentBootstrap] JMS tunnel: enabled
2014-10-30 08:39:40,424 INFO [main] [ElasticAgentBootstrap] Starting tunnel server, waiting for 2 connections.
2014-10-30 08:39:40,425 FATAL [tunnellogger-thread] [TunnelServer] [com.atlassian.tunnel.tunnel.server.TunnelServer] Fatal error in TunnelServer.
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
at java.net.ServerSocket.bind(ServerSocket.java:328)
at java.net.ServerSocket.<init>(ServerSocket.java:194)
at java.net.ServerSocket.<init>(ServerSocket.java:150)
at javax.net.ssl.SSLServerSocket.<init>(SSLServerSocket.java:84)
at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.<init>(SSLServerSocketImpl.java:81)
at com.sun.net.ssl.internal.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:58)
at com.atlassian.tunnel.tunnel.server.TunnelServer.run(TunnelServer.java:54)
at java.lang.Thread.run(Thread.java:662)
Any idea for a workaround?
We (Atlassian Build Engineering) have created a set of plugins to run a number Docker based agents in a cluster (ECS) that come online, build a single job and then exit. It should be able to do what you outlined. We've recently open sourced the solution.
See https://bitbucket.org/atlassian/per-build-container for more details.
Amazon instances are historically bad providing I/O capacity (I am assuming you mean network I/O) per dollar as they tend to be optimized more for CPU and Memory intensive workloads. You may find that additional processes on the same node may not help.
If you install your docker image on a node and take a snapshot to create an AMI after the image is installed, you can launch as many ec2 instances with that AMI as you want. They will have the image preinstalled.
You can also use CloudFormation and/or Cloud Init to recompile or download your image on each Ec2 instance without having to worry about manually moving around docker image.
However, if you do want to run more than one bamboo agent on a node you should be able too as longs as you set the bamboo.home parameter differently for each agent instance.
java -Dbamboo.home=/agent1Home -jar atlassian-bamboo-agent-installer-X.X-SNAPSHOT.jar \
http://bamboo-host-server:8085/agentServer/
java -Dbamboo.home=/agent2Home -jar atlassian-bamboo-agent-installer-X.X-SNAPSHOT.jar \
http://bamboo-host-server:8085/agentServer/
I get an error when using jmeter with one master and one remote node. I did this steps:
Java & JMeter versions are the same (whith JMeter plugins)
I started the JMeter Server in the remote node
I added the remote node IP to the properties file of the master
I startet JMeter on the master and started the test script via remote start (the command line of the master contains "Using remote object …" with the correct IP
After that JMeter freezes and after 7 minutes I get the error Message "Engine is busy - please try later".
The Log of the master is:
2014/05/06 14:17:33 INFO - jmeter.gui.action.RemoteStart: Initialising remote engine: 172.19.147.41
2014/05/06 14:17:33 INFO - jmeter.engine.ClientJMeterEngine: running clientengine run method
2014/05/06 14:17:33 INFO - jmeter.samplers.SampleEvent: List of sample_variables: []
2014/05/06 14:17:33 INFO - jmeter.samplers.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
2014/05/06 14:17:33 INFO - jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2014/05/06 14:17:33 INFO - jmeter.samplers.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
2014/05/06 14:17:33 INFO - jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2014/05/06 14:25:50 ERROR - jmeter.engine.ClientJMeterEngine: Error in rconfigure() method java.lang.IllegalStateException: Engine is busy - please try later
2014/05/06 14:25:50 INFO - jmeter.engine.ClientJMeterEngine: Interrupting RMI Reaper
The log of the remote node:
2014/05/06 14:17:33 INFO - jmeter.samplers.BatchSampleSender: Using batching for this run. Thresholds: num=100, time=60000
2014/05/06 14:17:33 INFO - jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2014/05/06 14:17:33 INFO - jmeter.samplers.BatchSampleSender: Using batching for this run. Thresholds: num=100, time=60000
2014/05/06 14:17:33 INFO - jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2014/05/06 14:25:50 INFO - jmeter.engine.RemoteJMeterEngineImpl: Creating JMeter engine on host 172.19.147.41 base '.'
2014/05/06 14:25:50 INFO - jmeter.engine.RemoteJMeterEngineImpl: Remote client host: 172.19.147.38
2014/05/06 14:25:50 WARN - jmeter.engine.RemoteJMeterEngineImpl: Engine is busy - cannot create JMeter engine
Did I forget to add some configuration or where is the problem?
Thanke in regards...
This error is because of the slaves are still executing previous test plan.
i would suggest.
while stopping the jmeter tests , use 'Remote Stop all' option from master, which kills the running threads on remote/slave machine process.
2.Restart[relaunch the "jmeter-server.bat"] on the slave/remote jmeter process and start the test.
Your problem is that the runners(slaves) are still running the test from your first start command even though you tried to end the test. Previous answers on Stack did not work for me in solving this problem so here's my solution:
On every runner(slave) do the following:
1. kill -9 `pidof java`
This will kill all Java processes so be careful if running other applications.
2.cd into apache-jmeter/bin directory and type ./jmeter-server &
This will restart your runners(slaves) as jmeter-servers so they are ready to receive communication from the master.
Then you should be able to successfully restart your test from master.
IN your case , it seems you triggered the test from master twice.
So the message is saying to you, a test is currently running on slave , cannot start a new one.
How do you start test from controller ? and how do you start jmeter-server from slave node ?
Error in NonGUIDriver java.lang.IllegalStateException: Engine is busy - please try later
This error occurs while running JMeter in non-GUI mode. JmeterApacheJMeter.jar process aborts on master without sending STOP signal to slaves to specify the test status.
Perform steps below to fix the issue on Slave machines:
Step#1 : Kill any running JMeter-server agent process on JMeter Slave/s
ps ax | grep "ApacheJMeter.jar" | awk '{print $1}' | xargs kill
Step#2 : Start JMeter-server process on Slave/s
cd apache-jmeter-folder/bin/; ./jmeter-server &
Now resume your performance test on JMeter master.