How to prevent TeamCity from spinning up an AWS cloud agent instance - amazon-ec2

Question

When using TeamCity with the AWS Cloud Profile, it spins up the agent every 10 minutes for no reason.
I am using the agent to build a Yocto distribution, which takes a lot of time and resources. For that reason the agent is running on a large (expensive) instance which I only want to start when needed.
What I am trying to accomplish:
Have a cheap AWS instance running TeamCity server, always on.
Have a big, expensive AWS instance running a TeamCity Agent, but only turned on when needed.
Only spin up the agent machine when there is a build to run.
Place a tag in git when the build passes.
Preferably: Check for new git commits on the server, and start the agent when there is a commit.
Please note that I want to do tagging on the Agent, so server-side checkout will not work (according to the TeamCity documentation).
I created a setup in AWS where I have a small instance running TeamCity server, and a large (expensive) instance running the agent. In TeamCity, I configured a cloud profile in the Root project to automatically start the machine running the agent, and stop it when the build is finished or the machine has been idle for 5 minutes. Starting and stopping the AWS machine works, and triggering a build als works.
I created a project with a build configuration without any Triggers added to it. I also tried to increase the minimum checking interval for the VCS root for our git repository to 90000000 seconds (so that it will practically never spin up the agent machine).
For some reason, TeamCity spins up the AWS instance running the Agent every 10 minutes and it is not clear to me why it is doing this. The TeamCity log shows:
[2019-03-19 10:51:15,894] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'f1nqO09...' was created, agent: "yocto-build-agent1" {id=3}
[2019-03-19 10:56:33,250] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'f1nqO09...' was closed, agent: "yocto-build-agent1" {id=3}
[2019-03-19 10:56:33,257] INFO - jetbrains.buildServer.CLEANUP - Removing agent Unregistered agent "yocto-build-agent1" {id=3}
[2019-03-19 10:56:33,262] INFO - jetbrains.buildServer.AGENT - Agent "yocto-build-agent1" {id=3} has been unregistered. Reason: Agent has unregistered
[2019-03-19 11:02:30,612] INFO - jetbrains.buildServer.AGENT - Agent has been registered: "yocto-build-agent1" {id=4, protocol=unidirectional, host=10.55.20.127:9090, agentTypeId=3, pool=Default, registered since 2019-03-19 11:02:30.590}, not running a build
[2019-03-19 11:02:30,615] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'FcY5MVV...' was created, agent: "yocto-build-agent1" {id=4}
[2019-03-19 11:02:30,666] WARN - jetbrains.buildServer.AGENT - Agent "yocto-build-agent1" {id=4} is unauthorized on registration
[2019-03-19 11:07:09,821] INFO - ldServer.AGENT.PollingProtocol - Polling Agent session 'FcY5MVV...' was closed, agent: "yocto-build-agent1" {id=4}
[2019-03-19 11:07:09,823] INFO - jetbrains.buildServer.AGENT - Agent "yocto-build-agent1" {id=4} has been unregistered. Reason: Agent has unregistered
[2019-03-19 11:07:22,455] INFO - jetbrains.buildServer.AGENT - Agent has been registered: "yocto-build-agent1" {id=4, protocol=unidirectional, host=10.55.20.127:9090, agentTypeId=3, pool=Default
The teamcity-clouds.log shows:
[2019-03-19 10:50:38,910] INFO [uled executor 1] - .instances.StoppedInstanceTask - Instance has changed status from stopped to Starting: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Starting}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 10:50:43,910] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'starting' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Starting}
[2019-03-19 10:51:15,883] INFO [nio-8111-exec-2] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 10:51:15,883] INFO [nio-8111-exec-2] - .server.impl.CloudEventsLogger - Detected cloud agent "yocto-build-agent1" {id=3}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Starting}
[2019-03-19 10:51:43,912] INFO [uled executor 2] - .server.impl.CloudEventsLogger - Cloud instance entered 'running' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}
[2019-03-19 10:56:32,807] INFO [uled executor 2] - te.IdleTimeoutTerminateFactory - Will stop instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} with idle time 5 minutes (idle time is 5 minutes)
[2019-03-19 10:56:32,808] INFO [uled executor 2] - l.instances.StopInstanceAction - Terminating instance: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, reason: "After 5 minutes idle"
[2019-03-19 10:56:32,808] INFO [uled executor 2] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 10:56:33,004] INFO [2 Stop Instance] - r.impl.DBCloudStateManagerImpl - Image: i-071eff7574b93d12f, Instance: i-071eff7574b93d12f, profile=amazon-2 is marked with state: stopped.
[2019-03-19 10:56:33,253] INFO [nio-8111-exec-7] - ces.terminate.AgentRemoverImpl - Removing cloud agent "yocto-build-agent1" {id=3} because: Instance has been terminated. Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 10:56:33,260] INFO [nio-8111-exec-7] - .server.impl.CloudEventsLogger - Cloud agent Unregistered agent "yocto-build-agent1" {id=3} is no longer recognized as cloud agent because instance has gone: profileId=amazon-2, imageId=i-071eff7574b93d12f, instanceId=i-071eff7574b93d12f
[2019-03-19 10:56:33,260] INFO [nio-8111-exec-7] - ces.terminate.AgentRemoverImpl - Removing cloud agent Unregistered agent "yocto-build-agent1" {id=3} because: Cloud instance (id=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, profileId=amazon-2) has gone (is not reported by cloud profile)
[2019-03-19 10:56:33,922] INFO [uled executor 2] - .server.impl.CloudEventsLogger - Cloud instance entered 'scheduled to stop' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Scheduled to stop}
[2019-03-19 10:56:43,923] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopping' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopping}
[2019-03-19 10:57:43,925] INFO [uled executor 1] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopped' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}
[2019-03-19 11:02:30,613] WARN [io-8111-exec-11] - .instances.StoppedInstanceTask - Agent "yocto-build-agent1" {id=4} was not matched to stopped instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:02:32,705] WARN [nio-8111-exec-4] - .instances.StoppedInstanceTask - Agent "yocto-build-agent1" {id=4} was not matched to stopped instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:02:43,939] INFO [uled executor 4] - .instances.StoppedInstanceTask - Instance has changed status from stopped to Running: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:02:48,939] INFO [uled executor 3] - .server.impl.CloudEventsLogger - Cloud instance entered 'running' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}
[2019-03-19 11:07:22,457] INFO [nio-8111-exec-1] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 11:07:22,458] INFO [nio-8111-exec-1] - .server.impl.CloudEventsLogger - Detected cloud agent "yocto-build-agent1" {id=4}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}
[2019-03-19 11:12:32,818] INFO [uled executor 5] - te.IdleTimeoutTerminateFactory - Will stop instance Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} with idle time 5 minutes (idle time is 5 minutes)
[2019-03-19 11:12:32,818] INFO [uled executor 5] - l.instances.StopInstanceAction - Terminating instance: Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, reason: "After 5 minutes idle"
[2019-03-19 11:12:32,818] INFO [uled executor 5] - r.impl.DBCloudStateManagerImpl - Image: AmazonImageInstance{id=i-071eff7574b93d12f, amazonId=i-071eff7574b93d12f}, profile: profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root} was marked to CONTAIN agent
[2019-03-19 11:12:33,050] INFO [2 Stop Instance] - r.impl.DBCloudStateManagerImpl - Image: i-071eff7574b93d12f, Instance: i-071eff7574b93d12f, profile=amazon-2 is marked with state: stopped.
[2019-03-19 11:12:33,403] INFO [nio-8111-exec-6] - ces.terminate.AgentRemoverImpl - Removing cloud agent "yocto-build-agent1" {id=4} because: Instance has been terminated. Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Running}, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}
[2019-03-19 11:12:33,415] INFO [nio-8111-exec-6] - .server.impl.CloudEventsLogger - Cloud agent Unregistered agent "yocto-build-agent1" {id=4} is no longer recognized as cloud agent because instance has gone: profileId=amazon-2, imageId=i-071eff7574b93d12f, instanceId=i-071eff7574b93d12f
[2019-03-19 11:12:33,415] INFO [nio-8111-exec-6] - ces.terminate.AgentRemoverImpl - Removing cloud agent Unregistered agent "yocto-build-agent1" {id=4} because: Cloud instance (id=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, profileId=amazon-2) has gone (is not reported by cloud profile)
[2019-03-19 11:12:33,960] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'scheduled to stop' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Scheduled to stop}
[2019-03-19 11:12:43,960] INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopping' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopping}
[2019-03-19 11:15:43,967] INFO [uled executor 3] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopped' state, profile 'AWS Cloud Profile'{id=amazon-2, projectId=_Root}, Amazon Instance{instanceId=i-071eff7574b93d12f, imageId=i-071eff7574b93d12f, status: Stopped}
(instance id and agent name have been changed to protect the innocent)
How can I prevent TeamCity Server from spinning up the agent machine, even if there are no triggers configured?
What is the best way to start the agent machine when there is a git commit? I was thinking of a separate build on a smaller agent which only checks commits, and make the larger build depend on that smaller build.

Answer 1

This is not normal behavior. Logic is: Teamcity will start a cloud build agent if there are no on-premise agents available. Then it will terminate if when idle, depending on the options you provide.
I created several cloud profiles, in the Root project. I use below options:
Additional terminate conditions / If idle, stop 3 minutes before the full hour
Maximum instances count: x
Not sure if that's helpful but I don't have any "was not matched to stopped instance" entries in the teamcity-clouds.log, which I can see in your log.
Also - did you try enabling debug log level for cloud profiles? You can do it in Administration / Diagnostics (/admin/admin.html?item=diagnostics). Then check log files again.

Answer 2

The logging in TeamCity says "State changed from Stopped to Starting", which made me think it was TeamCity that started this agent. As it turned out, the AWS Instance which is running the build agent was started by OpsWorks.
Amazon OpsWorks was configured to monitor the state of the machine and keep it "up". So when TeamCity correctly shuts the server down, Amazon OpsWorks detects the machine going down, and tries to start it up again. After a while TeamCity detects it being "up" and "idle" and stops it again. This goes on indefinetely.
It would be nice to have an OpsWorks feature in TeamCity Cloud configuration in addition to the EC2 and VMWare features. This way TeamCity can tell Opsworks that the machine is supposed to go down.
For now, we have fixed the problem by disabeling OpsWorks health monitoring for the agents.