In our setup maven builds are running on a TeamCity agent and are copied to the TeamCity server repository. Then I use teamcity-s3-plugin to copy them to S3 bucket. Overall size of the artifacts in the build is 3,3GB and it takes about 20 minutes to publish them to the server. We tested the connection between the TeamCity agent and server and it takes about 5 minutes to copy a file of this size, the speed for sptping data is 11,2 MB/s. It takes 3 minutes to copy all artifacts to S3. What might be the reason for the delay? Here is what I see in the agent log file:
[2015-07-15 09:01:25,235] DEBUG - r.artifacts.impl.HttpDiskCache - Http cache put: new entry [https://ci.adsquare.com/httpAuth/repository/download/adsBackendLiveMvn/1353.tcbuildid/jars/weather-service/
target/weather-service-0.0.1-SNAPSHOT.jar]
[2015-07-15 09:01:26,485] DEBUG - r.artifacts.impl.HttpDiskCache - Http disk cache - cached file from [/home/ads/agent/build01/work/864fff464676bc95/weather-service/target/weather-service-0.0.1-SNAPSHO
T.jar] under key [https://ci.adsquare.com/httpAuth/repository/download/adsBackendLiveMvn/1353.tcbuildid/jars/weather-service/target/weather-service-0.0.1-SNAPSHOT.jar]
... messages of this kind for other artifacts
[2015-07-15 09:01:28,390] DEBUG - jetbrains.buildServer.AGENT - Agent ping from server
[2015-07-15 09:01:48,895] DEBUG - jetbrains.buildServer.AGENT - Agent ping from server
...
[2015-07-15 09:22:19,356] DEBUG - jetbrains.buildServer.AGENT - Agent ping from server
[2015-07-15 09:22:39,862] DEBUG - jetbrains.buildServer.AGENT - Agent ping from server
[2015-07-15 09:22:42,083] INFO - jetbrains.buildServer.AGENT - Done publishing artifacts to 'jars', using 'WebPublisher', total files published: 144
[2015-07-15 09:22:42,083] INFO - jetbrains.buildServer.AGENT - Done publishing artifacts to 'jars', using 'ArtifactsCachePublisher', total files published: 144
[2015-07-15 09:22:42,083] INFO - jetbrains.buildServer.AGENT - Done: Publishing files
So what happens in the 20 minutes when the server pings the agent? Is it the time when the artifacts are copied? Why does it take 4 times longer than copying one file? Are there any configuration options to optimize the process?
And on a separate matter - is there a better way to copy files to S3? Is it possible to do it from agent host without publishing them to TeamCity server first?
Related
i have freshly installed on ec2 instance on ubuntu, the installation was successfull but after that when i try to run "airflow standalone" to run airflow, it won't start.
Below is the error i get, do i need to install anything else also before running "airflow standlone" command.
My ultimate goal is to deploy python code for data transformation in my data-pipeline.
please help ?
webserver | [2022-11-01 10:36:51,919] {webserver_command.py:217} ERROR - No response from gunicorn master within 120 seconds
webserver | [2022-11-01 10:36:51,920] {webserver_command.py:218} ERROR - Shutting down webserver
scheduler | [2022-11-01 10:39:38,254] {scheduler_job.py:1381} INFO - Resetting orphaned tasks for active dag runs
Is there any reason why my opendj takes more than 2 minutes to complete init setup ?
the setup command I'm using is like:
setup --cli -p 1389 --baseDN $BASE_DN -h localhost --rootUserDN "${OPENDJ_DIR_MANAGER}" --rootUserPassword "${OPENDJ_DIR_MANAGER_PW}" --addBaseEntry --acceptLicense --no-prompt --verbose
find below log:
[INFO] [2017-01-03T11:29:15+0000] Running setup.sh
[INFO] [2017-01-03T11:29:15+0000] Setting up default OpenDJ instance
See /tmp/opendj-setup-7089778767611065994.log for a detailed log of this
operation.
Configuring Directory Server ..... Done.
Creating Base Entry dc=example,dc=com ..... Done.
Starting Directory Server:
[03/Jan/2017:11:31:14 +0000] category=CORE severity=NOTICE
msgID=org.opends.messages.core.134 msg=OpenDJ 3.0.1-SNAPSHOT (build
20161229142204, revision number 1050009bd82b5c5f51f5d38b146243700ca01044)
starting up
OpenDJ is a Java based product. You might want to check the JVM and its default memory settings. Another option is DNS (OpenDJ does a reverse lookup at startup).
I installed a build agent on Windows 7 and always have the agent show up as "disconnected". The log teamcity-agent.txt shows below information
If this is the first time this agent registered on the server make sure it is authorized by administrator in the server web UI.
[2012-09-05 02:56:14,416] INFO - jetbrains.buildServer.AGENT - Agent registration finished.
[2012-09-05 02:56:14,418] INFO - jetbrains.buildServer.AGENT - Stop command was not performed. No build to stop with reason: BuildInterruptReason.AGENT_SHUTDOWN
[2012-09-05 02:56:14,418] INFO - jetbrains.buildServer.AGENT - Starting agent shutdown sequence, reason: Restart agent, failed to download upgrade from server
[2012-09-05 02:56:14,424] INFO - jetbrains.buildServer.AGENT - Host configuration for downloading updates: HostConfiguration[host=http://myteamcity:8080]
[2012-09-05 02:56:14,424] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/teamcity-agent.xml ==> C:\BuildAgent\temp\iQ0fjie6zR125neulJeqiYauT8zQ5zOI
[2012-09-05 02:56:15,070] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/agentSystemInfo.zip ==> C:\BuildAgent\update\plugins\agentSystemInfo.zip
[2012-09-05 02:56:15,747] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/amazonEC2.zip ==> C:\BuildAgent\update\plugins\amazonEC2.zip
[2012-09-05 02:56:16,617] INFO - agent.impl.AgentPortFileWriter - Delete agent runtime file from C:\BuildAgent\logs\buildAgent.port
[2012-09-05 02:56:16,619] INFO - jetbrains.buildServer.AGENT - Unregistering from build server: 18
[2012-09-05 02:56:16,800] INFO - jetbrains.buildServer.AGENT - Shutdown agent WebServer start
[2012-09-05 02:56:16,801] INFO - jetbrains.buildServer.AGENT - Shutdown agent WebServer finish
[2012-09-05 02:56:16,801] INFO - jetbrains.buildServer.AGENT - Shutdown agent finish
[2012-09-05 02:56:16,968] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/ant.zip ==> C:\BuildAgent\update\plugins\ant.zip
[2012-09-05 02:56:50,086] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/antPlugin.zip ==> C:\BuildAgent\update\plugins\antPlugin.zip
[2012-09-05 02:56:52,070] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/assembly-info-patcher.zip ==> C:\BuildAgent\update\plugins\assembly-info-patcher.zip
[2012-09-05 02:56:53,464] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/clearcase-agent.zip ==> C:\BuildAgent\update\plugins\clearcase-agent.zip
[2012-09-05 02:56:55,375] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/cmake-runner.zip ==> C:\BuildAgent\update\plugins\cmake-runner.zip
[2012-09-05 02:56:57,190] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/commandLineRunner.jar ==> C:\BuildAgent\update\plugins\commandLineRunner.jar
[2012-09-05 02:56:58,218] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/coveragePlugin.zip ==> C:\BuildAgent\update\plugins\coveragePlugin.zip
[2012-09-05 02:57:11,629] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/crashDetector.zip ==> C:\BuildAgent\update\plugins\crashDetector.zip
[2012-09-05 02:57:12,436] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/cvsAgent.zip ==> C:\BuildAgent\update\plugins\cvsAgent.zip
[2012-09-05 02:57:20,720] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/dotCover.zip ==> C:\BuildAgent\update\plugins\dotCover.zip
[2012-09-05 02:58:00,011] INFO - jetbrains.buildServer.AGENT - Downloading http://myteamcity:8080/update/plugins/dotNetPlugin.zip ==> C:\BuildAgent\update\plugins\dotNetPlugin.zip
How can I get the agent connected?
Here is a very comprehensive blog on re-connecting disconnected agents
http://michaelbarnesjr.wordpress.com/2012/10/18/teamcity-build-agent-disconnected/
Here are the steps (copied from the above link)
If an agent becomes disconnected for whatever reason, you can follow these steps to re-connect it:
ssh to the server that the disconnected agent is on.
go to the build agent’s bin directory (i.e. /opt/teamcity/buildagent/buildagent01/bin/)
su to the user that the agent runs as (i.e. appBuilder)
run agent.sh start
That will start the agent, and after a few seconds or so, the next time you go to TeamCity, it will be connected.
If Team City is running on a windows server, you can connect to that server, go to Services and restart the build agent that has become disconnected.
In my case the agent was started but listed as disconnected, after waiting several minutes it was still disconnect. So I went to the bin directory on the build agent like Peter suggested and ran the following two commands
./agent.sh stop
./agent.sh start
essentially just restarting the build agent. That did the trick for me.
First, make sure in the TeamCity website dashboard to authorize the agent if you haven't done so already.
Next, make sure the agents "ownPort" is not blocked by a firewall on the agent. This is the port that TeamCity will use to send messages to the agent. The value can be defined in "${agent.home}/conf/buildAgent.properties".
As of TeamCity 2017.2.1, the "ownPort" is not present, nor is there any embedded documentation (i.e. commented out examples). The default value is 8080, but you can just add the property and value you want like this:
## Defines which port the agent will use to receive messages from TeamCity.
## Defaults to 8080 if not present.
ownPort=9091
In my case under Windows 8 Professional:
TC build agent upgrade failed
I checked out BuildAgent\logs\teamcity-agent.log, there was trouble with the "upgrade process".
I checked out BuildAgent\logs\upgrade.log, there were more detailed "DEBUG" messages. The problem was "access denied" on shutting down service.
I added the user account of the TC server service to "Administrators" to let it stop the agent service.
I removed the user account of the TC server service from "Administrators" user group.
Restarting both service (TC server + Agent) led finally to have:
BuildAgent\logs\buildAgent.port
file: the agent was finally connected after successfull.
I had issues with the server and change the port for the server.
So, the agent was unable to connect to the server since I only changed the server port and not the agent properties.
Once I updated the agent properties, the agent started working again.
The properties are found under:
TeamCity\buildAgent\conf\buildAgent.properties
The property is found like:
serverUrl=http://localhost:8080
Starting the TeamCity Agent manually in the Services panel resolved the issue for me (on a Windows machine).
Problem was indicated in the upgrade.log (in BuildAgent\logs directory):
ERROR - r.agent.upgrade.UpgradeRunBase - Failed to start agent after upgrade: java.io.IOException: Failed to start TeamCity build agent service. Please check TeamCity build agent service user have enough permissions to stop and start the service.
I ran into this problem when working remotely over VPN and my IP address changed. In the conf/buildAgent.properties file, I changed my assigned IP for the serverURL property to use localhost instead, and the agent connected right up.
## The address of the TeamCity server. The same as is used to open
TeamCity web interface in the browser.
#serverUrl=http://111.222.333.444:8111
serverUrl=http://localhost:8111
I faced this issue on Windows 10, when I changed the port of TeamCity from "8111" to "80".
To fix this issue, I opened the file C:\TeamCity\buildAgent\conf\buildAgent.properties and chenged the first line from serverUrl=http\://localhost\:8111 to serverUrl=http\://localhost\:80.
We are trying to setup a build agent and every time we start it the log shows the following messages:
[2012-09-18 12:52:01,805] INFO - jetbrains.buildServer.AGENT - Starting agent shutdown sequence, reason: Restart agent, failed to download upgrade from server
[2012-09-18 12:52:01,821] INFO - jetbrains.buildServer.AGENT - Host configuration for downloading updates: HostConfiguration[host=http://localhost:8000]
[2012-09-18 12:52:01,821] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/teamcity-agent.xml ==> E:\buildAgent\temp\m8a1mAwTuLIngev3yRUMPUuaYWZFmMSh
[2012-09-18 12:52:01,849] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/agentSystemInfo.zip ==> E:\buildAgent\update\plugins\agentSystemInfo.zip
[2012-09-18 12:52:01,880] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/amazonEC2.zip ==> E:\buildAgent\update\plugins\amazonEC2.zip
[2012-09-18 12:52:01,921] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/ant.zip ==> E:\buildAgent\update\plugins\ant.zip
[2012-09-18 12:52:02,056] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/antPlugin.zip ==> E:\buildAgent\update\plugins\antPlugin.zip
[2012-09-18 12:52:02,078] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/assembly-info-patcher.zip ==> E:\buildAgent\update\plugins\assembly-info-patcher.zip
[2012-09-18 12:52:02,098] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/clearcase-agent.zip ==> E:\buildAgent\update\plugins\clearcase-agent.zip
[2012-09-18 12:52:02,106] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/commandLineRunner.jar ==> E:\buildAgent\update\plugins\commandLineRunner.jar
[2012-09-18 12:52:02,118] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/coveragePlugin.zip ==> E:\buildAgent\update\plugins\coveragePlugin.zip
[2012-09-18 12:52:02,151] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/crashDetector.zip ==> E:\buildAgent\update\plugins\crashDetector.zip
[2012-09-18 12:52:02,163] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/cvsAgent.zip ==> E:\buildAgent\update\plugins\cvsAgent.zip
[2012-09-18 12:52:02,183] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/dotCover.zip ==> E:\buildAgent\update\plugins\dotCover.zip
[2012-09-18 12:52:02,308] INFO - jetbrains.buildServer.AGENT - Downloading http://localhost:8000/update/plugins/dotNetPlugin.zip ==> E:\buildAgent\update\plugins\dotNetPlugin.zip
[2012-09-18 12:52:03,830] INFO - agent.impl.AgentPortFileWriter - Delete agent runtime file from E:\buildAgent\logs\buildAgent.port
[2012-09-18 12:52:03,831] INFO - jetbrains.buildServer.AGENT - Unregistering from build server: 5
Has anyone seen anything like this before? We've looked at the server logs and aren't seeing anything on them to indicate what could be wrong.
I ran into the same issue. Both the build agent and the server are installed on Windows Server 2012.
I stopped the build agent service and deleted the logs from BuildAgent\logs and restarted the build agent service so I could see a fresh log.
upgrade.log showed me that the build agent received a call from the teamcity server to upgrade. The log also showed the following:
"Please check TeamCity build agent service user have enough permissions to stop and start the service."
Using the Local Security Policy, I granted the build agent service user "logon as a service" rights but this is not sufficient to start and stop a service. By default, only members of the Administrators group can start, stop, pause, resume or restart a service. After I added the build agent service user to the Administrator group and restarted the build agent service, the upgrade finished successfully and the agent connected again.
Alternatively, as described by BatteryBackupUnit and https://web.archive.org/web/20171019005501/http://windowsitpro.com/security/letting-user-start-and-stop-services-without-granting-user-administrator-privileges, it's possible to give the TeamCity Windows user account the start, stop and pause permissions on the TeamCity Build Agent service. The article says that something like
subinacl /service spooler /grant=contoso\cortana=top
will suffice.
Or, if you don't want to use a tool that Microsoft no longer makes available, then you can use Process Explorer, as described at https://superuser.com/a/315709/12337.
I had this problem when running the agent with systemd on Ubuntu 18.04. The agent exits with code 143 when upgrading and this is interpreted as an error. It needs to be added to the list of acceptable error codes with SuccessExitStatus=143 0
Here's the full configuration:
[Unit]
Description=TeamCity Build Agent
After=network.target
[Service]
Type=forking
RemainAfterExit=yes
PIDFile=/build-agent/logs/buildAgent.pid
ExecStart=/build-agent/bin/agent.sh start
ExecStop=/build-agent/bin/agent.sh stop
User=build
Group=build
Restart=on-failure
RestartSec=5s
# agent will exit with 143 during upgrade process
SuccessExitStatus=143 0
[Install]
WantedBy=multi-user.target
The TeamCity build agent can sometimes take a long time to upgrade. If you believe that the upgrading takes too long with no positive results do as follows:
Uninstall the build agent
Download the build agent pack from your server
Reinstall the agent
Upgrading process should go faster after this. If it does not help it is better to contact JetBrains tech support.
A really simple way of doing this if your enterprise security policy doesn't allow you to add users to the 'Administrators' group, but the user your logged on as does have elevated rights. Stop the service running, via the Services.msc and then open a command prompt (with elevated rights) in the agent bin directory and run
agent.bat start
Let the agent update with this user and then once it has finished and you can see it successfully registered in the Teamcity UI. Kill the processes and restart the Service.
If you're using a "jailed" build user like we are, you'll see "Please check TeamCity build agent service user have enough permissions to stop and start the service."
There are workarounds but no graceful solution to this. Here's why:
Workaround 1: "TCBuildAgent" service needs "jailed" needs start/stop service credential access assigned to it. SubinACL is your best tool for assigning this (lengthy but full discussion).
TeamCity removes "TCBuildAgent" and installs a new one, effectively eliminating the service credentials assigned to it. So, you must issue SubinACL grant access commands every time you upgrade. Clumsy and annoying
Workaround 2: Aforementioned tip of adding "jailed" to Administrators group is viable and permits a proper upgrade, though violates the concept of a "jailed" user.
After much fiddling with SubinACL, I've thrown in the towel and just temporarily add "jailed" to Administrators group for the upgrade process then remove it afterward.
I trust that Jetbrains isn't gonna do something awful and malicious with TCBuildAgent... not during upgrade, anyway ;)
I had a similar problem. helped me to reinstall the antivirus.
http://devnet.jetbrains.com/thread/440728
Stop the agent:
./agent.bat stop
Reinstall agent.
Start the agent:
./agent.bat start
My TeamCity agent is contniually trying to download upgrades from the TeamCity server. It always downloads the files, then fails to start the installation, with the error:
WARN - jetbrains.buildServer.AGENT - Unable to start upgrade: Connection reset
My server is running at http://localhost:9912
How do I stop this endless cycle?
Latest version of TeamCity downloaded from JetBrains
I'm guessing you're running the agent as an AD account rather than under the machine account.
Run the agent service under an account with administrative permissions for the upgrade. I don't remember the specifics of what it's trying to write but it won't if it's a standard user.