As a long-time lurker on SO, I'm popping my question cherry with a tricky one regarding CruiseControl.NET, Subversion, and a remote repository. Here's the problem:
Take a remote Subversion 1.6.6 repository running on Windows Server 2008 SP2, using Apache 2.2.14 as the gateway to enable access on Port 80 and 443 - we redirect unencrypted traffic to the secure port, and we have a properly-configured SSL layer running with a self-signed certificate. This all works - I can point a browser from my local machine (XP SP2 or Server 2008 SP2) to this repository, pass through the certificate verification painlessly, authenticate against the repository ACL, and see stuff in there. Equally, SVN commands executed on my local machine (either at the commandline or via TortoiseSVN) work flawlessly.
Now add CruiseControl.NET 1.4.4.83 - running for the moment on my local machine with a simple script to pull some sourcecode from the remote repository, and build a very basic application. This same script works perfectly against a local repository, so the only difference is the Subversion URL, which is now pointing to the remote server.
If I run CC.NET in a commandline shell (ccnet.exe) under my own account, it works.
If I run CC.NET as a service (ccservice.exe) however, it fails; by default, it runs as LOCALSERVICE, but I have already altered this to run with my credentials. In this mode, the very first Subversion command issued (SVN LOG) fails, complaining that it cannot connect to the server.
I have spent a good four or five days investigating this; I know it's not a firewall-type problem, because the exact same command that CC.NET issues works in a commandline shell, and of course I can connect to the remote server via TortoiseSVN and a browser. It's not the SSL and/or certificate getting in the way, as I have already imported the certificate - and again, this works fine in 'manual' mode using TortoiseSVN, a browser, or SVN commands at the commandline. It's not a DNS resolution issue, because I can specify the actual IP address of the remote server, and it still fails to connect - but of course connects OK if I run in commandline mode...
I've even downloaded and prowled through the CC.NET sourcecode to see if there's anything odd going on. As far as I can see, the only difference regarding issuing commands when running in service-mode is that an AllocConsole call has been made to prepare a console for the Subversion command processes to latch on to when they're spawned.
The best guess I can make, at this stage, is that the AllocConsole session is somehow fundamentally different from a standard commandline session - in that, even though the service is running under my user credentials, somehow the AllocConsole does not have 'proper' network access. However, I don't know enough about AllocConsole or the CC.NET sourcecode to be able to prove this, and I'm therefore at an impasse.
For the moment, I'm running CC.NET in commandline mode; however, this just feels unsatisfactory, because we prefer to run it in service mode (which works fine against repositories in our local domain) to avoid the necessity creating a scheduled task to start it up when the machine starts.
Anyone got any suggestions?
Cracked it. It wasn't a firewall problem, it was a proxy problem.
We use Bluecoat here (yes, I know, not my call) and so the Subversion 'servers' file, which lives at 'C:\Documents and Settings\All Users\Application Data\Subversion\' on all our developer PCs has an entry to let us bypass the proxy to get out to specific external repositories. However, we tend to use TortoiseSVN mostly, and access this file through the property sheet - until now, not realising that it's a Subversion file, not a TortoiseSVN file.
Naturally, since we don't use TortoiseSVN on the build servers, we've never messed with the 'servers' file. However, now that CruiseControl.NET needs to get to an external repository (previously it has only talked to our internal repositories) it was unable to get past the Bluecoat roadblock.
Once the puzzle clicked into place, I created a 'servers' file on the build box, added our bypass configuration to it, and suddenly CruiseControl.NET and Subversion in Service Mode can see the remote repository.
I'm still not entirely sure why it worked in commandline mode but not service mode, but I've fixed it so I'm happy. :)
Related
Our team has ~80 Windows development machines, and activities of each developer are logged as text files on the local storage of those machines.
To analyze the logged activities, I want to gather all log files from those machines. Additionally, the log files are updated constantly, so It is desirable to gather files with the command-line from my machine.
I’ve searched and found some solutions, but all of those are not suitable for our situation:
We cannot use PsExec, because tcp/135 and tcp/445 are both closed (countermeasure for WannaCry).
Administrative share is disabled.
telnet service is not up and is banned by security reasons.
WinRM is disabled on those machines by default.
It is difficult to install new software like OpenSSH on those machines (because of the rule of this project)
RDP is the only way to connect those machines. (I have an account on all machines)
How can I copy files from remote Windows machines with command-line through RDP?
Or, at least, is there any way to execute a command on remote Windows machines with command-line through RDP?
I think you can do this, though it is very hacky :)
For a basic setup, which just copies files once, what you would need to do is
Run a script in the remote session when it logs in. I can think of three ways to do this:
Use the "Alternate Shell" RDP file property. This runs a specified program in place of explorer.exe on login; you can use it to run "cmd.exe /c [your script]" for instance.
If that doesn't work (e.g. the remote machine doesn't respect it), you might be able to use a scheduled task that runs the script on login, but perhaps only for a specified user, or maybe the script could check the WinStation type to make sure this is actually an RDP connection before doing anything.
It's also possible to do this by connecting in RemoteApp mode and using the script as your "application", but that only works for Server and Enterprise editions of Windows.
Enable either drive redirection or clipboard redirection on the RDP connection, to give you a way to get data out.
Drive redirection is much simpler to script; you just have the remote script copy files to e.g. "\\tsclient\C\logs".
Clipboard redirection is theoretically possible - you have the remote script copy, then a local script paste - but would probably be a pain to get working in practice. I'm only mentioning it in case drive redirection isn't available for some reason.
You would probably want to script to then log the session off afterward.
You could then launch that from command-line by running "mstsc.exe [your RDP file]". The RDP files could be programmatically generated if needed (given you're working with 80 machines).
If you want a persistent connection you can execute commands over, that's more complicated, but still technically possible. Two ways I can think of:
Use the previous method to run a program on logon, but this time create a custom application that receives commands using a transport that isn't blocked and executes them in the session. I've done this with WCF over HTTP, for instance; it's not secure, of course.
Develop and install a service on the remote machine that opens an RDP virtual channel, and a corresponding RDP client plugin that communicates with it. You can then do whatever you want across the connection. While this solution would be the most likely to work, it's also the most heavyweight and time-consuming to implement so it's probably a last resort.
I'm wondering if anyone's managed to connect to a self-hosted Git repository on a Mediatemple GridServer (gs) via XCode? I tried setting this up last night, and this is as far as I've got:
Installed Git on my Mac(s)
Created subdomain for git (git.blahblah.com) on my gs service via Terminal SSH
Created empty repository within subdomain (git.blahblah.com/repo/repo.git) via Terminal SSH
Clicked on 'Connect to repo' from XCode's launch menu
Entered repo address (ssh://git.blahblah.com:22/repo/repo.git) in XCode's address bar
Clicked 'clone' once XCode has found the repo and is telling me that "Authentication [is] required"
Chosen the appropriate local folder on my machine to clone the repo to
Tried to enter username and password specified in MT admin panel - the same ones that I used via Terminal SSH to set up the repo in the first place
Get booted with an error message: either 'incorrect username and password' (they're not) or 'specified address does not appear to be a repository, connection unexpectedly terminated by host' (or similar)
Cry. Repeatedly.
Getting to step 6 causes my IP to be blocked by MT, even if I'm logged into the web control panel from the same IP at the same time. Unblocking the IP via the MT control panel has no effect; the IP is blocked as soon as I try and connect via XCode again.
I suspect it's actually step 4 that's causing problems as XCode is trying to 'ping' the repo without credentials as you type in the address (to resolve connectivity).
Has anyone got this working, or do I need to look at hosting my Git repos elsewhere? I'd rather not spend more money on hosting when I have a service that should suit this purpose sitting unused.
After several online sessions with MediaTemple support I've come to the conclusion that it's not going to work.
While I can host a repository on my own gs service just fine, the way that XCode tries to connect to it causes my IP to be blocked every single time. As the blocking is automatic (and can't be turned off) then it seems XCode and MediaTemple have reached an impasse.
It's not a huge issue as I'm the only developer on this project, and I can always use DropBox instead (I was just trying to find a better way of keeping versioned backups across my laptop and desktop). If I really need Git I can use another tool or Terminal.
If I had to point fingers, I'd blame XCode: it seems its support for Git is rather sketchy beyond connecting to an open, gitHub-hosted repository.
Quick update: the solution I've managed to get working is to set up an account on BitBucket, create a project and add it to local Git, commit that to BitBucket and then connect to BitBucket using XCode. I downloaded SourceTree for extra Git activity as per Stefan's suggestion, as I'm still not 100% comfortable using Terminal (I just like GUIs, is that wrong?).
Everything seems to be running smoothly so far; it's just a little annoying that I had to use a third party for Git when I thought my server (which I'm already paying for) could do the job.
I am starting with Bazaar (switching from Subversion, sorry if terminology is a bit off sometimes). Using Bazaar locally no problems.
Got bzr+ssh:// working on my Windows server (finally! and even ssh agent is working wow!)
Now I want a shared repository on the server from which the developers can branch to their machines. I want the repository in a specific folder, for example in C:\bzr\MyProject.
When I do:
bzr init-repo --no-trees bzr+ssh://myserver.com/MyProject
it creates the repository in C:\cygwin\home\user\MyProject in the home directory of my user account - it is understandable, but worrying.
Then as an experiment I also tried and succeeded:
bzr init-repo --no-trees bzr+ssh://myserver.com/C:\bzr\MyProject
This created the repository where I wanted. But how do I "map" or "alias" the URL (or bzr) so my developers can logon under their accounts and use URL
bzr+ssh://myserver.com/MyProject
to access the shared repository in C:\bzr\MyProject?
Obviously I don't want developers to use bzr+ssh://myserver.com/C:\bzr\MyProject because of the physical path in the URL.
Ok my own answer is if you want to run Bazaar on a Windows server for a development team who will access it over the internet,
and you are not familiar with Linux, Cygwin, SSHD and related stuff then it might turn out more complicated than you can bear.
I actually abandoned the Bazaar idea and gone with Mercurial. I must say Mercurial install on the server is also steep, but at least it is just Windows, IIS and only a bit of Python. Got it running in half a day.
Some of the problems that I had with bzr+ssh:// on a Windows server are:
Needs SSHD installed on the server. SSHD (from stripped down Cygwin) supplied with Bazaar refused to work. Had to install Cygwin and learn a bit of Linux stuff, how to run as a service, how to configure, how to generate keys.
Hard to add a new user in a way that does not request passwords typing for each command. Will need to generate a keypair, mock with copying the keys to th server in two locations (Cygwin's home user folder and Window's too). Probably need to log on the user to Windows to create a profile. Don't want developers logging on to the server actually.
Hard to set up a shared repository in a specific location on the server. Does not seem possible with bzr+ssh. Possible with sftp. Might need to use symlinks as bialix suggested above.
As a newbie to linux stuff I don't understand all implications of running sshd on the server and giving shell access to the developer accounts. Have to use bzr shell limited... documentation is scarce.
Basically, bzr+ssh:// on a Windows server seems to be what installing Subversion on a Windows server was like several years ago - hard. Hopefully it will get better with Bazaar too because I chose it over Mercurial initially.
my project is a PHP web application. This applies to my test server (local), not production server! I am also the solo developer on this project (however, that may change in the very far future). Also, all my source code is committed to a repository and the production server gets the source code from the repository.
I do my development in Windows while my test server runs on Ubuntu (perhaps you can also recommend me another distro that is easy to use and can serve as a good web server). I need an elegant way to interface between the two environments. Currently, I do my coding in Windows and then FTP the changed files to the test server. However, this is quite cumbersome and tedious since I have to manually go to my FTP client each time. Suggest me something elegant please! Perhaps FTP sync? or OpenVPN (where the root www directory on test server is acts like a folder in Windows)? Thanks for your awesome time!
Easiest would be in Ubuntu, right click a folder then click "Sharing Options", then share the folder. In Windows, connect to the share, and work on that copy.
If you're using version control, using continuous integration like Hudson ( http://hudson-ci.org/ ) would help if you create a task that builds/exports the website for the testing server. This approach would be better in the long term, but you'll waste a day setting it up initially.
I prefer SFTP to FTP.
That said, ExpanDrive lets you map SFTP servers to local drive letters, which then means you can use any text editor to access your files directly on the test server, or use other mechanisms to keep the files in sync. Since they show up as two local drives, you can use just about any product out there.
If you want to use FTP, you can just map the drive in Windows Explorer. If you open up My Computer, then go to Tools > Map Network Drive, you can map a FTP server folder to any local drive. Just type in the address as the folder, ie. ftp://mscharley#192.168.0.10/htdocs
This will atleast save you a trip to the FTP client...
Is there any reason you couldn't just test on your local computer? At my job, we all develop and do developer testing locally, most of us using Windows. Our production and test servers are all linux based. Working locally is really nice, because you don't need to worry about making changes on the server with every small change.
Another option would be to create a checkout or working copy of your code on the server, and then run svn up or svn export (or equivalent using your version control software) each time you change the code on the server (assuming you are sshd into the server). This is kind of slow, but it's easy. The other option would be to write a script that goes through the svn logs for the recent commit and only exports or updates the ones that changed. This is much faster, and for all I know, there is already something out there that this.
Finally, some IDEs allow you to edit files live over ftp\sftp. Basically the IDE downloads a copy of the code and then reuploads it when you save.
Currently I develop on windows (PHP) as well and deploy on a Linux box for testing and production. This is how I do it.
Set up a local development server with e.g. WAMP.
Set up your code base in version control, e.g. Subversion.
Checkout your code base onto the testing/staging server, not just only on your local dev. environment.
In the early stages of development you want to deploy to the testing environment A LOT to sort out any discrepancies between your windows and linux environments. When your programming efforts turn more into program flow type programming this constant testing will probably slow down. But still take the effort to test on a regular basis.
To test your code base on staging do an svn update. I just log in with an SSH session to do this. A key thing here to note is that you do not have to make any config changes to your code base. If you do need to make config changes to your environment on staging it worth while spending the time to SCRIPT this process rather than this being a manual process.
Do the same for production. I use an Subversion check out on production as well. Make sure you set you .htaccess file to deny access to your hidden .svn folders and script the deployment especially if there a config changes necessary.
Some ideas:
Use a server environment under windows (e.g. EasyPHP).
Use a development tool that can save over FTP (e.g. ultra edit).
Use a network drive connected to the remote machine via FTP.
Use a network drive connected to the remote machine via Samba.
Run a linux distro inside a virtualization tool (e.g. virtual box) and write from the windows host to a share directory of the guest host.
Use dropbox to sync files between machines (there is more a hack than an "enterprise" solution).
I've been trying to get psexec to run executables on remote machines from custom build tasks in visual studio. All of these commands work from the command line but running it from an application seems to be a problem. Some commands work, on other psexec hangs and consequently so do msbuild and visual studio 2005. I'm calling SharePoint's stsadm.exe in this case, but this problem occurs with a lot of programs, when running psexec from an application. There are lots of people having this problem, but there seems to be no solution, so my question is: Does anyone know a working alternative to psexec?
I've experienced 'hangs' when executing PSEXEC against a remote system, but I always attributed it to the security context in which the remote process was running under.
From PSEXEC help:
If you omit a username the remote
process runs in the same account from
which you execute PsExec, but because
the remote process is impersonating it
will not have access to network
resources on the remote system. When
you specify a username the remote
process executes in the account
specified, and will have access to any
network resources the account has
access to. Note that the password is
transmitted in clear text to the
remote system.
If your executing a process remotely, that then needs to access the database (stsadm.exe), then it could be failing trying to access the network resource, depending on how PSEXEC was executed. If thats the case, I'd imagine it would eventually time out and give some sort of resource unavailable message.
There are two things that generally done when executing deployment steps against a remote machine to prevent the behavior your describing:
Like rifferte mentioned, make sure all assets needed to
deploy are local to the remote
machine (copy files, etc) before
using PSEXEC to execute the script (*.bat, *.vbs, *.ps, etc) -
so that everything runs 'local' to
the remote machine.
Run PSEXEC using
a domain username/password when
executing it - note that this
information is passed in clear text
to the remote server.
There's always RCE.
RemCom is a small (10KB upx packed)
remoteshell / telnet replacement that
lets you execute processes on remote
windows systems, copy files on remote
systems, process there output and
stream it back. It allows execution of
remote shell commands directly with
full interactive console without
having to install any client software.
On local machines it is also able to
impersonate so can be used as a silent
replacement for Runas command.
You can try to have psexec call a bat program that executes what you need on the remote machine. I ran into this issue with installutil.exe. A simple bat file on the remote machine resolved it.
You should also share your experience on the sysinternals board. There may be something they can do in a future revision.