Using Team City 2017.1
I am unable to use Agent Side checkout with my Ubuntu 14.04 build agent due to the following error:
[2017-06-22 13:41:12,779] INFO - jetbrains.buildServer.VCS.P4 - Running p4 login for user myUserId in [P4Port: redacted-server-address:1666; P4User: myUserId; perforce client mapping with 1 rules, VCSRoot: "ETG" {internal id=170}]
[2017-06-22 14:07:38,029] INFO - jetbrains.buildServer.VCS.P4 - Creating P4 workspace TC_p4_LinuxBuildAgent1_964e0a7b4154cd8c_85d77afe3e61a99a
[2017-06-22 14:07:38,225] INFO - jetbrains.buildServer.VCS.P4 - Creating/updating Perforce client specification:
Client: TC_p4_LinuxBuildAgent1_964e0a7b4154cd8c_85d77afe3e61a99a
Owner: myUserId
Description:
Created by TeamCity for user myUserId.
Root: /home/someuser/BuildAgent/work/964e0a7b4154cd8c
Options: noallwrite clobber nocompress unlocked nomodtime rmdir
Host: ubuntu
SubmitOptions: revertunchanged
LineEnd: local
View:
//ETS/GE_DEV/Build/... //TC_p4_LinuxBuildAgent1_964e0a7b4154cd8c_85d77afe3e61a99a/...
[2017-06-22 14:07:38,436] INFO - jetbrains.buildServer.VCS.P4 - Running p4 login for user myUserId in [P4Port: redacted-server-address:1666; P4User: myUserId; perforce client mapping with 1 rules, VCSRoot: "ETG" {internal id=170}]
[2017-06-22 14:07:39,016] WARN - l.patch.AbstractSourcesUpdater - Error while checkout on agent: Perforce password (P4PASSWD) invalid or unset. - while running 'p4 -c TC_p4_LinuxBuildAgent1_964e0a7b4154cd8c_85d77afe3e61a99a -u myUserId -p
redacted-server-address:1666 -H ubuntu client -i'
jetbrains.buildServer.vcs.VcsException: Perforce password (P4PASSWD) invalid or unset. - while running 'p4 -c TC_p4_LinuxBuildAgent1_964e0a7b4154cd8c_85d77afe3e61a99a -u myUserId -p redacted-server-address:1666 -H ubuntu client -i'
at jetbrains.buildServer.vcs.perforce.PerforceConnection.runCommand(PerforceConnection.java:271)
at jetbrains.buildServer.vcs.perforce.PerforceConnection.runCommand(PerforceConnection.java:257)
at jetbrains.buildServer.vcs.perforce.PerforceWorkspacesImpl.createOrUpdateWorkspace(PerforceWorkspacesImpl.java:80)
at jetbrains.buildServer.vcs.perforce.PerforceAgentSourceUpdater.createOrUpdateLocalWorkspace(PerforceAgentSourceUpdater.java:99)
at jetbrains.buildServer.vcs.perforce.PerforceAgentSourceUpdater.updateSources(PerforceAgentSourceUpdater.java:68)
at jetbrains.buildServer.vcs.perforce.PerforceAgentSourceUpdater.updatePerforceSources(PerforceAgentSourceUpdater.java:55)
at jetbrains.buildServer.vcs.perforce.PerforceSourceUpdatePolicy.updateSources(PerforceSourceUpdatePolicy.java:66)
at jetbrains.buildServer.agent.impl.vcs.AgentVcsManagerExImpl$CheckoutSupportImpl.updateSources(AgentVcsManagerExImpl.java:108)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesOnAgent$1.run(ProjectSourcesOnAgent.java:186)
at java.lang.Thread.run(Thread.java:745)
I am fairly certain our Perforce server uses ticket based auth. At the build machine, I can run p4 login (which prompts for password). This is successful and allows me to run p4 client which returns a User specification that includes an "AuthMethod: perforce" (the user specification does not include a "Password:" line).
I have tried a couple of different workarounds including:
Creating a .p4enviro file which includes the P4PASSWD
Setting an environment variable for P4PASSWD (in /etc/environment)
However, these have no effect...
The logs seems strange to me because the login appears to succeed (at least, no errors are logged). But, the checkout fails with the P4PASSWD error.
Also, the VCS root is using a Client mapping (but I have tried it with Client as well - same error is present).
Any help would be very appreciated!
This is resolved thanks to those who added comments above.
This issue was self-inflicted and caused by an entry in rc.local that was issuing the start command to agent.sh (thus causing the build agent to run as root).
If I would have just RTFM, I wouldn't have encountered this issue :)
Related
I've got a macOS 10.13 server running, on which I have recently had to change the hostname (upstream IT requirements) - and I suspect this has broken Kerberos.
Changing the hostname appears to have been successful: I exported the Open Directory setup, modified it, and reimported it into the updated setup - user accounts exist, and manual authentication works as expected. changeip is happy:
mac-mini:~ server_admin$ sudo changeip -checkhostname
dirserv:success = "success"
However SSO from client machines does not appear to be successful.
Attempting to run kinit with a valid user account shows this:
mac-mini:~ server_admin$ kinit test#MAC-MINI.EXAMPLE.COM
test#MAC-MINI.EXAMPLE.COM's password:
kinit: krb5_get_init_creds: Server (krbtgt/MAC-MINI.EXAMPLE.COM#MAC-MINI.EXAMPLE.COM) unknown
Looking at /etc/krb5.conf, I only see this:
[libdefaults]
kdc_timeout=5
...which is the same as it was on my previously-working configuration.
And now I'm a bit stumped. All the documentation for destroying and rebuilding Kerberos setups seem to be out of date. Any ideas?!
Thanks.
So I just upgraded my Mac mini (Late 2012) to macOS 10.12 (Sierra) and everything seems fine, but I’m running into one odd problem deploying code with Capistrano. I get the following error:
Permission denied (publickey).
Never had this problem before in Mac OS X 10.11 (El Capitan) or any version prior to it. Why is this suddenly happening now? Full output of the failed Capistrano deployment below:
jakes_mac:SomeCode jake$ cap staging deploy
INFO [hkdgad21] Running /usr/bin/env mkdir -p /tmp/somecode/ as jake#example.com
DEBUG [hkdgad21] Command: /usr/bin/env mkdir -p /tmp/somecode/
jake#example.com's password:
INFO [hkdgad21] Finished in 5.166 seconds with exit status 0 (successful).
DEBUG Uploading /tmp/somecode/git-ssh.sh 0.0%
INFO Uploading /tmp/somecode/git-ssh.sh 100.0%
INFO [xyz20312] Running /usr/bin/env chmod +x /tmp/somecode/git-ssh.sh as jake#example.com
DEBUG [xyz20312] Command: /usr/bin/env chmod +x /tmp/somecode/git-ssh.sh
INFO [xyz20312] Finished in 0.240 seconds with exit status 0 (successful).
INFO [abcdef01] Running /usr/bin/env git ls-remote --heads git#github.com:SomeUser/SomeCode.git as jake#example.com
DEBUG [abcdef01] Command: ( GIT_ASKPASS=/bin/echo GIT_SSH=/tmp/somecode/git-ssh.sh /usr/bin/env git ls-remote --heads git#github.com:SomeUser/SomeCode.git )
DEBUG [abcdef01] Permission denied (publickey).
DEBUG [abcdef01] fatal: Could not read from remote repository.
DEBUG [abcdef01]
DEBUG [abcdef01] Please make sure you have the correct access rights
DEBUG [abcdef01] and the repository exists.
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as jake#example.com: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
SSHKit::Command::Failed: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Tasks: TOP => git:check
(See full trace by running task with --trace)
The deploy has failed with an error: Exception while executing as jake#example.com: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Seems like it’s an issue with SSH keys not being automatically added as it used to be in Mac OS X 10.11 (El Capitan). Is this expected behavior from macOS Sierra or something connected to OpenSSH?
Method 1: Add all known keys to the SSH agent.
So one solution I found is to run ssh-add with the -A option—which adds all known identities to the SSH agent using any passphrases stored in your keychain—like this:
ssh-add -A
Now this works but it won’t persist across reboots. So if you want to never worry about this again, just open up your user’s ~/.bash_profile file like this:
nano ~/.bash_profile
And add this line to the bottom:
ssh-add -A 2>/dev/null;
Now when you open a new Terminal window, all should be good!
Method 2: Add only SSH keys that are in the keychain to the agent.
So while the ssh-add -A option should work for most basic cases, I ran into an issue recently where I had 6-7 Vagrant boxes (which uses SSH keys/identities for access) setup on a machine on top of the more common id_rsa.pub in place.
Long story short, I ended up being locked out of a remote server due to too many failed tries based on SSH keys/identities since the server access was based on a password and SSH keys/identities are SSH keys/identities. So the SSH agent tried all of my SSH keys, failed and I couldn’t even get to the password prompt.
The problem is that ssh-add -A will just arbitrarily add every single SSH key/identity you have to the agent even if it’s not necessary to do so; such as in the case of Vagrant boxes.
My solution after much testing was as follows.
First, if you have more SSH keys/identities added to your agent than you need—as shown with ssh-add -l then purge them all from the agent like so:
ssh-add -D
With that done, then start the SSH agent as a background process like so:
eval "$(ssh-agent -s)"
Now, it gets weird and I am not too sure why. In some cases you can specifically add the ~/.ssh/id_rsa.pub key/identity to the agent like so:
ssh-add ~/.ssh/id_rsa.pub
Type in your passphrase, hit Return and you should be good to go.
But in other cases simply running this is enough to get the key/identity added:
ssh-add -K
If that’s all worked, type in ssh-add -l and you should see one lone SSH key/identity listed.
All good? Now open up your .bash_profile:
nano ~/.bash_profile
And add this line to the bottom; comment or remove the -A version if you have that in place:
ssh-add -K
That will allow the SSH key/identity to be reloaded to the SSH agent on each startup/reboot.
UPDATE 1: Based on davidalger’s answer I discovered a nicer, global solution that can work for all user’s on a system. Just open up the global SSH config located here via sudo:
sudo nano /etc/ssh/ssh_config
And add this line to the bottom of the file:
AddKeysToAgent yes
Did that—after removing the .bash_profile fix and all is good as well.
UPDATE 2: Apple has now added a UseKeychain option to the open SSH config options and considers ssh-add -A a solution as well.
As of macOS Sierra 10.12.2, Apple (I assume) has added a UseKeychain config option for SSH configs. Checking the man page (via man ssh_config) shows the following info:
UseKeychain
On macOS, specifies whether the system should search for
passphrases in the user's keychain when attempting to use a par-
ticular key. When the passphrase is provided by the user, this
option also specifies whether the passphrase should be stored
into the keychain once it has been verified to be correct. The
argument must be ``yes'' or ``no''. The default is ``no''.
Which boils down to Apple seeing the solution as either adding ssh-add -A to your .bash_profile as explained in this Open Radar ticket or adding UseKeychain as one of the options in a per user ~/.ssh/config.
Is this expected behavior from macOS Sierra or something connected to OpenSSH?
This is due to a new feature in OpenSSH 7.2 causing a change in behavior on the part of the SSH client. From the release notes:
ssh(1): Add an AddKeysToAgent client option which can be set to
'yes', 'no', 'ask', or 'confirm', and defaults to 'no'. When
enabled, a private key that is used during authentication will be
added to ssh-agent if it is running (with confirmation enabled if
set to 'confirm').
There were other interesting (security related) features introduced as well, although the release was considered mainly a bug fix release. This particular feature resulted in a default behavior change on OS X since it's default value is 'no' and OS X (I don't know about other clients) previously added keys to the agent as they were used.
So if you add the following to your ~/.ssh/config file (or the global ssh_config that should be located in /etc/ssh/ssh_config), keys will again be added to your agent as they are used.
AddKeysToAgent yes
This one-liner makes it pretty easy:
echo "AddKeysToAgent yes" >> ~/.ssh/config
After doing this, I was able to achieve the expected behavior:
$ ssh-add -l
The agent has no identities.
$ ssh -T git#bitbucket.org
logged in as davidalger.
You can use git or hg to connect to Bitbucket. Shell access is disabled.
$ ssh-add -l
2048 SHA256:<snip> (RSA)
This help me to resolve the issue on MacOS Serra:
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa_file
This has got me the whole day digging for answers. The bottom line is that when I run svn as system user it seems to use the system-name to authenticate against the SVN server regardless of what credentials are passed. Following is the long explanation that made arrive at that conclusion.
When running from a Windows 7 Professional, if I run svn from the console under any normal user, the application works as expected: if credentials have been cached in %AppData%/Roaming/Subversion it will use them, if not it will prompt for username and password unless I use the options --username and --password. If I enter credentials using the options then the commit works with no problem. All good so far.
But when I try to run svn as the system user (nt authority\system) in the same computer, it behaves differently. To begin with, %AppData%/Roaming/Subversion points to C:\Windows\System32\config\systemprofile\AppData\Roaming\Subversion, and I make sure there is no auth folder there, so no credentials cached. Then I run svn without any parameters and it doesn't prompt for username/password, instead it executes the action and receives an error from subversion:
svn: E175013: Commit failed (details follow):
svn: E175013: MKACTIVITY of '/svn/Development/!svn/act/f20db493-48f1-9c43-a957-541584be555e': 403 Forbidden (http://<ip-address>)
If I run it indicating --username and --password, it gets the same error. But then I check the error logs from subversion and find this:
[Fri Aug 08 17:32:18 2014] [error] [client <client IP>] Access denied: '<clienthostname>$' MKACTIVITY Development:
[Fri Aug 08 17:32:18 2014] [error] [client <client IP>] Access denied: '<clienthostname>$' DELETE Development:
Where <clienthostname> is the hostname of the computer where I'm trying to commit from (note the '$' at the end of the subversion log, that's not part of the hostname but it does appear in the log as part of the username).
So that's the question: why is svn behaving differently when running as system user? Why does it use the hostname as username when authenticating against the SVN server? And why do other users work correctly?
Note: I believe my problem is different from the following questions in stackoverflow:
Subversion ignoring “--password” and “--username” options: I don't get any prompts to enter username and password, regardless of whether I indicate the options --username and --password or I don't
SVN Error when commiting Access denied: 'foobar' MKACTIVITY MYREPO: I saw this question and I tried double checking the case of all the items in the URL, no luck.
svn: MKACTIVITY 403 Forbidden: I have checked that no credentials are cached in %AppData%/Roaming/Subversion
For those who are wondering why I'm trying to run svn as system user, the answer is that I am trying to make a commit from TeamCity, which means it is the Build Agent the one executing the svn command. The Build Agent is a Windows Service and runs as system user, and the svn command fails in the way explained above.
Use the "--no-auth-cache" in your svn commands and you won't see this issues. You might however run into other set of issues.
If you do not use the no-auth-cache it tries to figure out default username and password anyways
A better way would be to create a .subversion folder and store the authentication in that folder. So for the system account, you can specify a differrent userid and password for the login.
I have a EC2 instance running CentOS and PBX on a flash installed.
I can connect to server using SSH. I want to reset PBX passwords, so i run:
passwd-master
after following steps, i get an error at the end :
Your manual password was accepted!
passwd-master - 2.0.4 released on 082612
CentOS release 6.3 (Final) - 64 Bit
**********************************************************************
* Now applying new password to maint/meetme/wwwadmin *
**********************************************************************
Applying wwwadmin password
Applying maint password
Applying meetme password
httpd service restarted to apply new passwords!
passwd-master - 2.0.4 released on 082612
CentOS release 6.3 (Final) - 64 Bit - ARI PATCH
**********************************************************************
* Now patching main.conf.php *
**********************************************************************
Now verifying /var/www/html/recordings/includes/main.conf.php exists
ERROR the file /var/www/html/recordings/includes/main.conf.php DOES NOT EXIST
or is zero length. Unable to continue now exiting
I noticed i dont have permission to access this folder : "/var/www/html/recordings/includes/"
Then i tried to set password of "maint" only, and i got an error at the end again :
passwd-maint
-------------------------------------------
Set password for AMP web GUI and maint GUI
User: maint
-------------------------------------------
New password:
Re-type new password:
Updating password for user maint
htpasswd: unable to update file /usr/local/apache/passwd/wwwpasswd
And its because i dont have access to "wwwpasswd" (not even read permission)
I login using default EC2-user obviously.
My question is, how can i fix this problem and change passwords?
I tried to use : sudo su command
and then run passwd-master , but it doesnt work.
after i use sudo su command, it doesnt know passwd-master and passwd-maint anymore and i get
bash: passwd-master: command not found
error.
Any advise please?
If its not in the path, then you have to run it like this ./passwd-master while in the directory where the script exists. – datasage
Thank you datasage. using " ./ " solved my problem . i didn't know about it.
Ok, so I have Hudson (v1.393) running in an Ubuntu VM and everything's working fine.
However I'm trying to add a Mac slave to the Ubuntu master and I've run in to a few problems.
I have set up SSH keys so that from the command line, the Ubuntu VM can ssh using the key into a user called hudson on the Mac.
In the Hudson slave configuration, I have "Launch slave agents on Unix machines via SSH" selected and have entered the host IP, username of the user on the slave and the location of my private key file on the master (which has been added to the authorised keys file on the slave).
However, the master fails to connect to the slave.
Looking at the log (below), it's trying to authenticate using a password.
Is this a fall back for a failed key based SSH attempt?
Is Hudson only trying to authenticate using a password, and I need to change something else to get it to use the key file which is defined in the configuration?
Is it just not possible to launch slave agents via ssh on a mac? (I know the name of this type of slave launch method explicity states Unix, but I was thinking (read: hoping) that it would work with OS X too)
Log
[01/14/11 10:38:07] [SSH] Opening SSH connection to 10.0.1.188:22.
[01/14/11 10:38:07] [SSH] Authenticating as hudson/******.
java.io.IOException: Password authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:319)
at com.trilead.ssh2.Connection.authenticateWithPassword(Connection.java:314)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:565)
at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:179)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:184)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: Authentication method password not supported by the server at this stage.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:289)
... 9 more
[01/14/11 10:38:07] [SSH] Connection closed.
If anyone has managed to conquer this type of set up before, or has any tips or ideas, I'd be very grateful!
Thanks
I've recently run into the same problem, trying to launch an agent on a Mac OS X 10.6 machine using SSH.
To get password authentication to work you'll need to edit /etc/sshd_config on the client node, setting PasswordAuthentication yes
In the Hudson dashboard take the node offline, make sure the configuration has a valid username and password, and launch the agent. Also make sure that the Remote FS root directory is owned by the build user you're connecting as.
For password-less ssh authentication, first check which user the Hudson master is running as. Lets assume that this is tomcat55. Generate a public/private SSH key pair (with an empty passphrase), then verify that the Hudson user can connect.
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tomcat55/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/tomcat55/.ssh/id_rsa.
Your public key has been saved in /home/tomcat55/.ssh/id_rsa.pub.
$ # authorize the hudson master on the hudson node
$ scp /home/tomcat55/.ssh/id_rsa.pub hudson#macnode:~/.ssh/authorized_keys
$ # test the connection
$ ssh -i /home/tomcat55/.ssh/id_rsa hudson#macnode
On the Hudson mac node, the /etc/sshd_config needs to allow for password-less access.
Protocol 2
PubkeyAuthentication yes
In the node configuration clear the password field, and set the private key field (in this example it is /home/tomcat55/.ssh/id_rsa). You should now be able to launch the agent:
[01/19/11 22:38:44] [SSH] Opening SSH connection to macnode:22.
[01/19/11 22:38:44] [SSH] Authenticating as hudson with /home/tomcat55/.ssh/id_rsa.
[01/19/11 22:38:45] [SSH] Authentication successful.
Check the /var/log/auth.log file on the Ubuntu machine. I'm betting you need to chmod 700 the .ssh directory of the hudson user.
I think the first answer (the selected one) is an awesome answer, but I did find a case where it is not the only solution.
In my case I have a Mac OS slave that was working and then I took that Mac down and brought up a new one. I thought I could just tweak the settings for the existing node's configuration to point it at the new Mac. It didn't work and I had all the same errors and problems described throughout this message thread.
Then I went in and deleted the node and recreated it with exactly the same settings and it worked. I suspect that SSH key fingerprint changed and by deleting the node and recreating it I was able to get it working. Whatever it is, the key component that caused it to fail is not a configuration option.