Teamcity error - Image does not exist in Amazon EC2 - amazon-ec2

I am using teamcity server 8.0.3 (build 27540) and have set up several git projects. I am using Amazon EC2 images which are set up as cloud agents.
My setup starts up the agents and runs the tests on them when a new job is put to the queue.
I got it working fine for weeks and have not touched the server, and suddenly it stopped working.
In the cloud tabs I now get "Image error" with the message: "Image does not exist in Amazon EC2". Even though the image exists there!
If I open and save or re-create my cloud agent settings then one of the agents will be awoken by teamcity and do builds just fine, but the warning is still there and when the agent shuts down after my time delay I have to re-save the cloud agent settings to force a start on the agent.
I have tried restarting Teamcity with no avail.
This is the errors from teamcity-cloud.log
[2013-12-12 06:11:00,850] WARN [rome'{id=cp3} 1] - .clouds.amazon.image.ImageBase - Failed to fetch data of image AmazonImageInstance{id=i-d123cd29, amazonId=ami-463edb31} from EC2. Image does not exist in Amazon EC2
[2013-12-12 06:11:00,958] INFO [ue; Flush Queue] - .instances.StartInstanceAction - Starting cloud instance: profile 'EC2 TC Win Chrome'{id=cp3}, AmazonImageInstance{id=i-d123cd29, amazonId=ami-463edb31}, hash=EZObYYzWwxDOZ4o9svYSvGbdLqf5e7KQ, reason=Automatic start: Compatible with queued build Booking Log Client :: Win Chrome(promotionId=12008)
[2013-12-12 06:11:01,565] INFO [ue; Flush Queue] - .server.impl.CloudEventsLogger - Cloud instance start succeeded: profile 'EC2 TC Win Chrome'{id=cp3}, Amazon Instance{instanceId=i-d123cd29, imageId=i-d123cd29, amazonImageId=ami-463edb31, status: Scheduled to start}
[2013-12-12 06:11:03,135] INFO [rome'{id=cp3} 1] - r.impl.DBCloudStateManagerImpl - Image: i-d123cd29, Instance: i-d123cd29 is marked with state: running.
[2013-12-12 06:12:18,441] WARN [uled executor 4] - .instances.StoppedInstanceTask - Instance has changed status from stopped to Running: Amazon Instance{instanceId=i-d123cd29, imageId=i-d123cd29, amazonImageId=ami-463edb31, status: Running}, profile 'EC2 TC Win Chrome'{id=cp3}
[2013-12-12 06:12:20,759] WARN [rome'{id=cp3} 1] - .clouds.amazon.image.ImageBase - Failed to fetch data of image AmazonImageInstance{id=i-d123cd29, amazonId=ami-463edb31} from EC2. Image does not exist in Amazon EC2
[2013-12-12 06:13:04,668] INFO [nio-8111-exec-9] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-d123cd29, amazonId=ami-463edb31}, profile: profile 'EC2 TC Win Chrome'{id=cp3} was marked to CONTAIN agent
[2013-12-12 06:13:04,672] INFO [nio-8111-exec-9] - .server.impl.CloudEventsLogger - Detected cloud agent EC2-i-d123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}, profile 'EC2 TC Win Chrome'{id=cp3}, Amazon Instance{instanceId=i-d123cd29, imageId=i-d123cd29, amazonImageId=ami-463edb31, status: Running}
and from teamcity-server.log
[2013-12-12 06:13:04,648] WARN - jetbrains.buildServer.AGENT - Agent EC2-i-dd123cd29 is unauthorized with comment: Cloud instance (id=i-dd123cd29, imageId=i-dd123cd29, profileId=cp3) has gone (is not reported by cloud profile)
[2013-12-12 06:13:04,661] INFO - jetbrains.buildServer.SERVER - Agent has been registered: EC2-i-dd123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}, not running a build
[2013-12-12 06:13:04,690] INFO - tbrains.buildServer.ACTIVITIES - Agent "EC2-i-dd123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}" was authorised with comment "Virtual agent is authorized automatically."
[2013-12-12 06:13:05,022] INFO - tbrains.buildServer.ACTIVITIES - Build started; Booking Log Client :: Win Chrome {id=booking_log_client, internal id=bt18} #34 {build id=6213, promotion id=12008, branch=<default>, history=false, agent=EC2-i-dd123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}, triggered by "Nisse Nilsson" (##userId='10'). Started Thu Dec 12 06:13:04 UTC 2013}
[2013-12-12 06:13:57,586] INFO - jetbrains.buildServer.SERVER - Agent description updated for agent: EC2-i-dd123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}
[2013-12-12 06:23:54,323] INFO - jetbrains.buildServer.SERVER - Agent description updated for agent: EC2-i-dd123cd29 {id=24, host=10.254.1.23:9090, agentTypeId=122, pool=Win Chrome}

The error appears due to:
The AMI 'ami-463edb31' is no longer available in EC2's 'EU (Ireland)' region.
source:https://aws.amazon.com/amis/amazon-public-images-basic-microsoft-windows-server-2008-64-bit

Related

How to prevent TeamCity from spinning up an AWS cloud agent instance

When using TeamCity with the AWS Cloud Profile, it spins up the agent every 10 minutes for no reason.
I am using the agent to build a Yocto distribution, which takes a lot of time and resources. For that reason the agent is running on a large (expensive) instance which I only want to start when needed.
What I am trying to accomplish:
Have a cheap AWS instance running TeamCity server, always on.
Have a big, expensive AWS instance running a TeamCity Agent, but only turned on when needed.
Only spin up the agent machine when there is a build to run.
Place a tag in git when the build passes.
Preferably: Check for new git commits on the server, and start the agent when there is a commit.
Please note that I want to do tagging on the Agent, so server-side checkout will not work (according to the TeamCity documentation).
I created a setup in AWS where I have a small instance running TeamCity server, and a large (expensive) instance running the agent. In TeamCity, I configured a cloud profile in the Root project to automatically start the machine running the agent, and stop it when the build is finished or the machine has been idle for 5 minutes. Starting and stopping the AWS machine works, and triggering a build als works.
I created a project with a build configuration without any Triggers added to it. I also tried to increase the minimum checking interval for the VCS root for our git repository to 90000000 seconds (so that it will practically never spin up the agent machine).
For some reason, TeamCity spins up the AWS instance running the Agent every 10 minutes and it is not clear to me why it is doing this. The TeamCity log shows:
[2019-03-19 10:51:15,894] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'f1nqO09...' was created, agent: "yocto-build-agent1" {id=3}
[2019-03-19 10:56:33,250] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'f1nqO09...' was closed, agent: "yocto-build-agent1" {id=3}
[2019-03-19 10:56:33,257] INFO - jetbrains.buildServer.CLEANUP - Removing agent Unregistered agent "yocto-build-agent1" {id=3}
[2019-03-19 10:56:33,262] INFO - jetbrains.buildServer.AGENT - Agent "yocto-build-agent1" {id=3} has been unregistered. Reason: Agent has unregistered
[2019-03-19 11:02:30,612] INFO - jetbrains.buildServer.AGENT - Agent has been registered: "yocto-build-agent1" {id=4, protocol=unidirectional, host=10.55.20.127:9090, agentTypeId=3, pool=Default, registered since 2019-03-19 11:02:30.590}, not running a build
[2019-03-19 11:02:30,615] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'FcY5MVV...' was created, agent: "yocto-build-agent1" {id=4}
[2019-03-19 11:02:30,666] WARN - jetbrains.buildServer.AGENT - Agent "yocto-build-agent1" {id=4} is unauthorized on registration
[2019-03-19 11:07:09,821] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'FcY5MVV...' was closed, agent: "yocto-build-agent1" {id=4}
[2019-03-19 11:07:09,823] INFO - jetbrains.buildServer.AGENT - Agent "yocto-build-agent1" {id=4} has been unregistered. Reason: Agent has unregistered
[2019-03-19 11:07:22,455] INFO - jetbrains.buildServer.AGENT - Agent has been registered: "yocto-build-agent1" {id=4, protocol=unidirectional, host=10.55.20.127:9090, agentTypeId=3, pool=Default
The teamcity-clouds.log shows:
[2019-03-19 10:50:38,910] INFO [uled executor 1] - .instances.StoppedInstanceTask - Instance has changed status from stopped to Starting: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Starting}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 10:50:43,910] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'starting' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Starting}
[2019-03-19 10:51:15,883] INFO [nio-8111-exec-2] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 10:51:15,883] INFO [nio-8111-exec-2] - .server.impl.CloudEventsLogger - Detected cloud agent "yocto-build-agent1" {id=3}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Starting}
[2019-03-19 10:51:43,912] INFO [uled executor 2] - .server.impl.CloudEventsLogger - Cloud instance entered 'running' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}
[2019-03-19 10:56:32,807] INFO [uled executor 2] - te.IdleTimeoutTerminateFactory - Will stop instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} with idle time 5 minutes (idle time is 5 minutes)
[2019-03-19 10:56:32,808] INFO [uled executor 2] - l.instances.StopInstanceAction - Terminating instance: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, reason: "After 5 minutes idle"
[2019-03-19 10:56:32,808] INFO [uled executor 2] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 10:56:33,004] INFO [2 Stop Instance] - r.impl.DBCloudStateManagerImpl - Image: i-071eff7574b93d12f, Instance: i-071eff7574b93d12f, profile=amazon-2 is marked with state: stopped.
[2019-03-19 10:56:33,253] INFO [nio-8111-exec-7] - ces.terminate.AgentRemoverImpl - Removing cloud agent "yocto-build-agent1" {id=3} because: Instance has been terminated. Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 10:56:33,260] INFO [nio-8111-exec-7] - .server.impl.CloudEventsLogger - Cloud agent Unregistered agent "yocto-build-agent1" {id=3} is no longer recognized as cloud agent because instance has gone: profileId=amazon-2, imageId=i-071eff7574b93d12f, instanceId=i-071eff7574b93d12f
[2019-03-19 10:56:33,260] INFO [nio-8111-exec-7] - ces.terminate.AgentRemoverImpl - Removing cloud agent Unregistered agent "yocto-build-agent1" {id=3} because: Cloud instance (id=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, profileId=amazon-2) has gone (is not reported by cloud profile)
[2019-03-19 10:56:33,922] INFO [uled executor 2] - .server.impl.CloudEventsLogger - Cloud instance entered 'scheduled to stop' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Scheduled to stop}
[2019-03-19 10:56:43,923] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopping' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopping}
[2019-03-19 10:57:43,925] INFO [uled executor 1] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopped' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}
[2019-03-19 11:02:30,613] WARN [io-8111-exec-11] - .instances.StoppedInstanceTask - Agent "yocto-build-agent1" {id=4} was not matched to stopped instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:02:32,705] WARN [nio-8111-exec-4] - .instances.StoppedInstanceTask - Agent "yocto-build-agent1" {id=4} was not matched to stopped instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:02:43,939] INFO [uled executor 4] - .instances.StoppedInstanceTask - Instance has changed status from stopped to Running: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:02:48,939] INFO [uled executor 3] - .server.impl.CloudEventsLogger - Cloud instance entered 'running' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}
[2019-03-19 11:07:22,457] INFO [nio-8111-exec-1] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 11:07:22,458] INFO [nio-8111-exec-1] - .server.impl.CloudEventsLogger - Detected cloud agent "yocto-build-agent1" {id=4}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}
[2019-03-19 11:12:32,818] INFO [uled executor 5] - te.IdleTimeoutTerminateFactory - Will stop instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} with idle time 5 minutes (idle time is 5 minutes)
[2019-03-19 11:12:32,818] INFO [uled executor 5] - l.instances.StopInstanceAction - Terminating instance: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, reason: "After 5 minutes idle"
[2019-03-19 11:12:32,818] INFO [uled executor 5] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 11:12:33,050] INFO [2 Stop Instance] - r.impl.DBCloudStateManagerImpl - Image: i-071eff7574b93d12f, Instance: i-071eff7574b93d12f, profile=amazon-2 is marked with state: stopped.
[2019-03-19 11:12:33,403] INFO [nio-8111-exec-6] - ces.terminate.AgentRemoverImpl - Removing cloud agent "yocto-build-agent1" {id=4} because: Instance has been terminated. Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:12:33,415] INFO [nio-8111-exec-6] - .server.impl.CloudEventsLogger - Cloud agent Unregistered agent "yocto-build-agent1" {id=4} is no longer recognized as cloud agent because instance has gone: profileId=amazon-2, imageId=i-071eff7574b93d12f, instanceId=i-071eff7574b93d12f
[2019-03-19 11:12:33,415] INFO [nio-8111-exec-6] - ces.terminate.AgentRemoverImpl - Removing cloud agent Unregistered agent "yocto-build-agent1" {id=4} because: Cloud instance (id=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, profileId=amazon-2) has gone (is not reported by cloud profile)
[2019-03-19 11:12:33,960] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'scheduled to stop' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Scheduled to stop}
[2019-03-19 11:12:43,960] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopping' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopping}
[2019-03-19 11:15:43,967] INFO [uled executor 3] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopped' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}
(instance id and agent name have been changed to protect the innocent)
How can I prevent TeamCity Server from spinning up the agent machine, even if there are no triggers configured?
What is the best way to start the agent machine when there is a git commit? I was thinking of a separate build on a smaller agent which only checks commits, and make the larger build depend on that smaller build.
This is not normal behavior. Logic is: Teamcity will start a cloud build agent if there are no on-premise agents available. Then it will terminate if when idle, depending on the options you provide.
I created several cloud profiles, in the Root project. I use below options:
Additional terminate conditions / If idle, stop 3 minutes before the full hour
Maximum instances count: x
Not sure if that's helpful but I don't have any "was not matched to stopped instance" entries in the teamcity-clouds.log, which I can see in your log.
Also - did you try enabling debug log level for cloud profiles? You can do it in Administration / Diagnostics (/admin/admin.html?item=diagnostics). Then check log files again.
The logging in TeamCity says "State changed from Stopped to Starting", which made me think it was TeamCity that started this agent. As it turned out, the AWS Instance which is running the build agent was started by OpsWorks.
Amazon OpsWorks was configured to monitor the state of the machine and keep it "up". So when TeamCity correctly shuts the server down, Amazon OpsWorks detects the machine going down, and tries to start it up again. After a while TeamCity detects it being "up" and "idle" and stops it again. This goes on indefinetely.
It would be nice to have an OpsWorks feature in TeamCity Cloud configuration in addition to the EC2 and VMWare features. This way TeamCity can tell Opsworks that the machine is supposed to go down.
For now, we have fixed the problem by disabeling OpsWorks health monitoring for the agents.

How can I compile my C++ code with Mesos?

I got a cluster of 1 master node and 2 slaves and I'm trying to compile my application with mesos.
Basically, here is the command that I use:
mesos-execute --name=alc1 --command="ccmake -j myapp" --master=10.11.12.13:5050
Offers are made from the slave but this compilation task keeps failing.
[root#master-node ~]# mesos-execute --name=alc1 --command="ccmake -j myapp" --master=10.11.12.13:5050
I0511 22:26:11.623016 11560 sched.cpp:222] Version: 0.28.0
I0511 22:26:11.625602 11564 sched.cpp:326] New master detected at master#10.11.12.13:5050
I0511 22:26:11.625952 11564 sched.cpp:336] No credentials provided. Attempting to register without authentication
I0511 22:26:11.627279 11564 sched.cpp:703] Framework registered with 70582e35-5d6e-4915-a919-cae61c904fd9-0139
Framework registered with 70582e35-5d6e-4915-a919-cae61c904fd9-0139
task alc1 submitted to slave 70582e35-5d6e-4915-a919-cae61c904fd9-S2
Received status update TASK_RUNNING for task alc1
Received status update TASK_FAILED for task alc1
I0511 22:26:11.759610 11567 sched.cpp:1903] Asked to stop the driver
I0511 22:26:11.759639 11567 sched.cpp:1143] Stopping framework '70582e35-5d6e-4915-a919-cae61c904fd9-0139'
On the sandbox slave node, here is the stderr logs:
I0511 22:26:13.781070 5037 exec.cpp:143] Version: 0.28.0
I0511 22:26:13.785001 5040 exec.cpp:217] Executor registered on slave 70582e35-5d6e-4915-a919-cae61c904fd9-S2
sh: ccmake: command not found
I0511 22:26:13.892653 5042 exec.cpp:390] Executor asked to shutdown
Just to mentionned that commands like this work fine and get me the expected results:
[root#master-node ~]# mesos-execute --name=alc1 --command="find / -name a" --master=10.11.12.13:5050
I0511 22:26:03.733172 11550 sched.cpp:222] Version: 0.28.0
I0511 22:26:03.736112 11554 sched.cpp:326] New master detected at master#10.11.12.13:5050
I0511 22:26:03.736383 11554 sched.cpp:336] No credentials provided. Attempting to register without authentication
I0511 22:26:03.737730 11554 sched.cpp:703] Framework registered with 70582e35-5d6e-4915-a919-cae61c904fd9-0138
Framework registered with 70582e35-5d6e-4915-a919-cae61c904fd9-0138
task alc1 submitted to slave 70582e35-5d6e-4915-a919-cae61c904fd9-S2
Received status update TASK_RUNNING for task alc1
Received status update TASK_FINISHED for task alc1
I0511 22:26:04.184813 11553 sched.cpp:1903] Asked to stop the driver
I0511 22:26:04.184844 11553 sched.cpp:1143] Stopping framework '70582e35-5d6e-4915-a919-cae61c904fd9-0138'
I don't really get what is needed for even troubleshot this issue.

Teamcity on EC2 "Could not determine the local IP address (...): unknown error"

I'm trying to setup Teamcity on an EC2 Ubuntu instance.
I've:
Downloaded the package through wget http://download.jetbrains.com/teamcity/TeamCity-9.0.4.tar.gz. Unpackaged it.
Unpackaged it: tar -xvzf TeamCity-9.0.4.tar.gz.
sh Teamcity/bin/teamcity-server.sh start.
I can't access the web-app, and when I check the logs, says:
INFO - jetbrains.buildServer.STARTUP - Server shutdown has been requested
WARN - jetbrains.buildServer.STARTUP - Could not determine the local IP address: ip-172-30-1-83: ip-172-30-1-83: unknown error
INFO - jetbrains.buildServer.STARTUP - Server IP address: -unknown-
INFO - jetbrains.buildServer.STARTUP - Server shutdown complete
Complete log:
INFO - jetbrains.buildServer.STARTUP - Starting TeamCity server
INFO - jetbrains.buildServer.STARTUP - Server shutdown has been requested
INFO - jetbrains.buildServer.STARTUP - TeamCity version: 9.0.4 (build 32407), data format version 709
INFO - jetbrains.buildServer.STARTUP - OS: Linux, User: ubuntu, JRE: 1.8.0_45-b14 Java HotSpot(TM) 64-Bit Server VM
INFO - jetbrains.buildServer.STARTUP - JVM parameters: -Djava.util.logging.config.file=/home/ubuntu/TeamCity/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx512m -XX:MaxPermSize=270m -Dlog4j.configuration=file:/home/ubuntu/TeamCity/bin/../conf/teamcity-server-log4j.xml -Dteamcity_logs=../logs/ -Djsse.enableSNIExtension=false -Djava.awt.headless=true -Djava.endorsed.dirs=/home/ubuntu/TeamCity/endorsed -Dcatalina.base=/home/ubuntu/TeamCity -Dcatalina.home=/home/ubuntu/TeamCity -Djava.io.tmpdir=/home/ubuntu/TeamCity/temp
INFO - jetbrains.buildServer.STARTUP - Garbage collection: Copy, MarkSweepCompact
WARN - jetbrains.buildServer.STARTUP - Could not determine the local IP address: ip-172-30-1-83: ip-172-30-1-83: unknown error
INFO - jetbrains.buildServer.STARTUP - Web server version: Apache Tomcat/7.0.59
INFO - jetbrains.buildServer.STARTUP - Server IP address: -unknown-
INFO - jetbrains.buildServer.STARTUP - Servlet real path: /home/ubuntu/TeamCity/webapps/ROOT/
INFO - jetbrains.buildServer.STARTUP - Current stage: TeamCity server is shutting down
INFO - jetbrains.buildServer.STARTUP - Stopping the main application
INFO - jetbrains.buildServer.STARTUP - The main servlet is not initialized, nothing to shut down.
INFO - jetbrains.buildServer.STARTUP - Server shutdown complete
Any hints on how to solve it?
I have Teamcity's web-app working on my local machine (also Ubuntu, and the same Teamcity package).
Read the shell script TeamCity/bin/teamcity-server.sh, it is written with limitation, the script can't run out of the application folder.
export CATALINA_OPTS
CATALINA_HOME=./..
CATALINA_BASE=./..
So you should cd to the bin folder and run :
cd Teamcity/bin
sh teamcity-server.sh start
Let me know if it fixs your issue or not.

Bamboo: More than one agent per EC2 elastic instance?

Is it possible to run more than one Bamboo Agent per EC2 Elastic Instance?
We use Docker in our build system and the process seems mostly network IO bound. It would be nice if we could run multiple Agents on one machine.
By running multiple agents on one machine instead of starting multiple machines, we also don't need to worry about shipping Docker images between machines as artifacts.
Update 30 Oct 2014:
I tried copying the current startup script and adding a new home:
$ cat /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent2
#!/bin/sh
bambooAgentBin=$(cd -P -- $(dirname $0) && pwd)
. $bambooAgentBin/bambooShellFunctions.sh
echo Starting Elastic Bamboo Agent...
java -Dbamboo.home=/home/bamboo/bamboo-agent-home-2/ -Dimagefiles.version=3.3-SNAPSHOT -jar $bambooAgentBin/*installer*.jar 2>&1 | tee -a $(getHomeDir)/bamboo-elastic-agent.out
It seems like the elastic version sets up some kind of a tunnel and crashes because a tunnel is already running:
# su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent2 - bamboo &
[1] 14143
Starting Elastic Bamboo Agent...
2014-10-30 08:39:31,804 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-e1/5.6-OD-01-0070/cce16404c14f06456c6adc44181746abf8dd1206/ to /opt/bamboo-elastic-agent
2014-10-30 08:39:31,979 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-e1/5.6-OD-01-0070/cce16404c14f06456c6adc44181746abf8dd1206/ to /opt/bamboo-elastic-agent
2014-10-30 08:39:31,979 INFO [main] [S3Utils] Fetching the list of remote objects...
2014-10-30 08:39:33,006 INFO [main] [S3Utils] Found 579 files in s3://bamboo-agent-release-us-e1/5.6-OD-01-0070/cce16404c14f06456c6adc44181746abf8dd1206/
2014-10-30 08:39:33,059 INFO [main] [S3Utils] Found 463 files in /opt/bamboo-elastic-agent
2014-10-30 08:39:33,060 INFO [main] [S3Utils] Generating the list of files to fetch from S3...
2014-10-30 08:39:33,076 INFO [main] [S3Utils] Generating the list of files to remove...
2014-10-30 08:39:33,078 INFO [main] [S3Utils] Removing 0 files from /opt/bamboo-elastic-agent
2014-10-30 08:39:33,079 INFO [main] [S3Utils] Fetching 155 files to /opt/bamboo-elastic-agent
2014-10-30 08:39:39,969 INFO [main] [S3Utils] Fetched 113 MB from S3
2014-10-30 08:39:39,973 INFO [main] [ElasticAgentInstaller] Starting [java, -server, -Xms32m, -Xmx256m, -XX:MaxPermSize=128m, -XX:+HeapDumpOnOutOfMemoryError, -Dimagefiles.version=3.3, -Dbamboo.agent.installDir=/opt/bamboo-elastic-agent, -cp, /opt/bamboo-elastic-agent/boot/annotations-13.0.jar:/opt/bamboo-elastic-agent/boot/gson-2.2.2-atlassian-1.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-api-agent-bootstrap-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/commons-io-2.4.jar:/opt/bamboo-elastic-agent/boot/jackson-core-2.1.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-shared-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/atlassian-tunnel-0.21.jar:/opt/bamboo-elastic-agent/boot/stax-api-1.0-2.jar:/opt/bamboo-elastic-agent/boot/guava-bridge-11.0.2-atlassian-01.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/commons-codec-1.8.jar:/opt/bamboo-elastic-agent/boot/atlassian-util-concurrent-2.4.1.jar:/opt/bamboo-elastic-agent/boot/joda-time-2.3.jar:/opt/bamboo-elastic-agent/boot/log4j-1.2.15.jar:/opt/bamboo-elastic-agent/boot/guava-11.0.2-atlassian-01.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-bootstrap-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/commons-lang-2.6.jar:/opt/bamboo-elastic-agent/boot/atlassian-aws-1.0.71.jar:/opt/bamboo-elastic-agent/boot/jackson-databind-2.1.1.jar:/opt/bamboo-elastic-agent/boot/fugue-1.1.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-1.7.1.jar:/opt/bamboo-elastic-agent/boot/httpclient-4.2.5.jar:/opt/bamboo-elastic-agent/boot/commons-logging-1.0.4.jar:/opt/bamboo-elastic-agent/boot/jackson-annotations-2.1.1.jar:/opt/bamboo-elastic-agent/boot/bcprov-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-core-agent-bootstrap-5.6-OD-01-0070.jar:/opt/bamboo-elastic-agent/boot/bcpkix-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/atlassian-annotations-0.4.jar:/opt/bamboo-elastic-agent/boot/jsr305-1.3.9.jar:/opt/bamboo-elastic-agent/boot/httpcore-4.2.5.jar:, com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap]
2014-10-30 08:39:40,119 INFO [main] [ElasticAgentBootstrap] Starting Agent Bootstrap using Java 1.6.0_45 from Sun Microsystems Inc.
2014-10-30 08:39:40,410 INFO [main] [ElasticAgentBootstrap] Using tunnelling. Setting virtual host name to https://xxxxxxx.atlassian.net/builds/agentServer/
2014-10-30 08:39:40,410 INFO [main] [ElasticAgentBootstrap] Using tunnelling for HTTP(S). Registering 'httpt' and 'httpst' protocols.
2014-10-30 08:39:40,416 INFO [main] [ElasticAgentBootstrap] HTTP(S) tunnel: enabled
2014-10-30 08:39:40,416 INFO [main] [ElasticAgentBootstrap] JMS tunnel: enabled
2014-10-30 08:39:40,424 INFO [main] [ElasticAgentBootstrap] Starting tunnel server, waiting for 2 connections.
2014-10-30 08:39:40,425 FATAL [tunnellogger-thread] [TunnelServer] [com.atlassian.tunnel.tunnel.server.TunnelServer] Fatal error in TunnelServer.
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
at java.net.ServerSocket.bind(ServerSocket.java:328)
at java.net.ServerSocket.<init>(ServerSocket.java:194)
at java.net.ServerSocket.<init>(ServerSocket.java:150)
at javax.net.ssl.SSLServerSocket.<init>(SSLServerSocket.java:84)
at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.<init>(SSLServerSocketImpl.java:81)
at com.sun.net.ssl.internal.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:58)
at com.atlassian.tunnel.tunnel.server.TunnelServer.run(TunnelServer.java:54)
at java.lang.Thread.run(Thread.java:662)
Any idea for a workaround?
We (Atlassian Build Engineering) have created a set of plugins to run a number Docker based agents in a cluster (ECS) that come online, build a single job and then exit. It should be able to do what you outlined. We've recently open sourced the solution.
See https://bitbucket.org/atlassian/per-build-container for more details.
Amazon instances are historically bad providing I/O capacity (I am assuming you mean network I/O) per dollar as they tend to be optimized more for CPU and Memory intensive workloads. You may find that additional processes on the same node may not help.
If you install your docker image on a node and take a snapshot to create an AMI after the image is installed, you can launch as many ec2 instances with that AMI as you want. They will have the image preinstalled.
You can also use CloudFormation and/or Cloud Init to recompile or download your image on each Ec2 instance without having to worry about manually moving around docker image.
However, if you do want to run more than one bamboo agent on a node you should be able too as longs as you set the bamboo.home parameter differently for each agent instance.
java -Dbamboo.home=/agent1Home -jar atlassian-bamboo-agent-installer-X.X-SNAPSHOT.jar \
http://bamboo-host-server:8085/agentServer/
java -Dbamboo.home=/agent2Home -jar atlassian-bamboo-agent-installer-X.X-SNAPSHOT.jar \
http://bamboo-host-server:8085/agentServer/

JMeter remote test fails with error message "Engine is busy - please try later"

I get an error when using jmeter with one master and one remote node. I did this steps:
Java & JMeter versions are the same (whith JMeter plugins)
I started the JMeter Server in the remote node
I added the remote node IP to the properties file of the master
I startet JMeter on the master and started the test script via remote start (the command line of the master contains "Using remote object …" with the correct IP
After that JMeter freezes and after 7 minutes I get the error Message "Engine is busy - please try later".
The Log of the master is:
2014/05/06 14:17:33 INFO - jmeter.gui.action.RemoteStart: Initialising remote engine: 172.19.147.41
2014/05/06 14:17:33 INFO - jmeter.engine.ClientJMeterEngine: running clientengine run method
2014/05/06 14:17:33 INFO - jmeter.samplers.SampleEvent: List of sample_variables: []
2014/05/06 14:17:33 INFO - jmeter.samplers.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
2014/05/06 14:17:33 INFO - jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2014/05/06 14:17:33 INFO - jmeter.samplers.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
2014/05/06 14:17:33 INFO - jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2014/05/06 14:25:50 ERROR - jmeter.engine.ClientJMeterEngine: Error in rconfigure() method java.lang.IllegalStateException: Engine is busy - please try later
2014/05/06 14:25:50 INFO - jmeter.engine.ClientJMeterEngine: Interrupting RMI Reaper
The log of the remote node:
2014/05/06 14:17:33 INFO - jmeter.samplers.BatchSampleSender: Using batching for this run. Thresholds: num=100, time=60000
2014/05/06 14:17:33 INFO - jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2014/05/06 14:17:33 INFO - jmeter.samplers.BatchSampleSender: Using batching for this run. Thresholds: num=100, time=60000
2014/05/06 14:17:33 INFO - jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2014/05/06 14:25:50 INFO - jmeter.engine.RemoteJMeterEngineImpl: Creating JMeter engine on host 172.19.147.41 base '.'
2014/05/06 14:25:50 INFO - jmeter.engine.RemoteJMeterEngineImpl: Remote client host: 172.19.147.38
2014/05/06 14:25:50 WARN - jmeter.engine.RemoteJMeterEngineImpl: Engine is busy - cannot create JMeter engine
Did I forget to add some configuration or where is the problem?
Thanke in regards...
This error is because of the slaves are still executing previous test plan.
i would suggest.
while stopping the jmeter tests , use 'Remote Stop all' option from master, which kills the running threads on remote/slave machine process.
2.Restart[relaunch the "jmeter-server.bat"] on the slave/remote jmeter process and start the test.
Your problem is that the runners(slaves) are still running the test from your first start command even though you tried to end the test. Previous answers on Stack did not work for me in solving this problem so here's my solution:
On every runner(slave) do the following:
1. kill -9 `pidof java`
This will kill all Java processes so be careful if running other applications.
2.cd into apache-jmeter/bin directory and type ./jmeter-server &
This will restart your runners(slaves) as jmeter-servers so they are ready to receive communication from the master.
Then you should be able to successfully restart your test from master.
IN your case , it seems you triggered the test from master twice.
So the message is saying to you, a test is currently running on slave , cannot start a new one.
How do you start test from controller ? and how do you start jmeter-server from slave node ?
Error in NonGUIDriver java.lang.IllegalStateException: Engine is busy - please try later
This error occurs while running JMeter in non-GUI mode. JmeterApacheJMeter.jar process aborts on master without sending STOP signal to slaves to specify the test status.
Perform steps below to fix the issue on Slave machines:
Step#1 : Kill any running JMeter-server agent process on JMeter Slave/s
ps ax | grep "ApacheJMeter.jar" | awk '{print $1}' | xargs kill
Step#2 : Start JMeter-server process on Slave/s
cd apache-jmeter-folder/bin/; ./jmeter-server &
Now resume your performance test on JMeter master.

Resources