I'm working on some academic research projects involving scraping large data sets from the web using Python. It's been inconvenient to work on my academic institution's Linux server because (1) I don't have superuser access, meaning I'm dependent on the IT staff to install my packages, and (2) my disk quota is somewhat limited (I would ideally want ~10 GB). What is the simplest way for me to get access to a machine that solves these problems? I don't need huge processing power; I just need access to a reasonably fast machine that runs 24/7, so that my programs can run continuously, and above all, something very simple to get running, use, and maintain, since I have a few non-CS people working on this project with me. Linux would be preferable, but I'd consider Windows too.
I'm aware of Amazon Web Services, but am wondering if there's something more appropriate to my specific needs.
By the way, it would be a huge bonus if I could get some sort of remote desktop access to this machine so I wasn't limited to using SSH and SFTP.
Suggestions?
EDIT: I can't use VirtualBox or Virtual PC because I need the program to be running around the clock, and I need to turn off my laptop often, etc.
If you do want to stick with running on your CS department's machines, use virtualenv to solve your package installation woes. And if disk space is an issue, you could use S3 (and perhaps FUSE) to store huge amounts of data extremely cheaply.
However, if that's not really what you're after, I can recommend Slicehost very highly. They give you a virtual private server - so you have complete control over what gets installed, users, admin, etc.
In principle, it's very much like EC2 (which I prefer to use for "real" servers), but has a friendly interface, great customer service and is aimed at smaller projects like yours.
Use x11vnc with ssh.
'sudo apt-get install x11vnc' on your remote server.
Once you have that, you can access your remote server via vnc, but the great thing is that you can tunnel vnc over ssh like so:
ssh -X -C -L 5900:localhost:5900 remotehost x11vnc -localhost -display :0
For more details see the x11vnc manpage.
Or, just setup remote desktop -- (which is actually vnc) on your linux distribution. Most distributions come with a GUI to configure remote desktop access.
If you have a linux machine you can use, then SSH -X will allow you to start GUI programs. It's not remote desktop, but it's close.
ssh -X whoever#whatever.com
firefox
Then bam. A firefox window pops on your desktop.
I have been pretty happy with TekTonic Virtual Private Servers. It's a virtualized environment, but you have full root access to install any packages you need. I'm not sure what your CPU and memory constraints are, but if they aren't too extensive then this should fit the bill nicely for you. I don't know if you would be able to enable a remote desktop as I've never tried but it may be possible to install the requisite packages.
The plans range from $15/mo to $100/mo, the $15/mo plan comes with 294MB RAM, 13GB disk space, and 2.6GHz max CPU speed. I ran on that plan for quite a while and eventually moved up to the next level up with double the disk/cpu/mem, and I've been quite happy with it. I've been with them since 2003 and have yet to find anyone who offers equivalent plans at these prices.
Related
I'm trying to learn ML on GCP. Some of the Qwiklabs and Tutorials start with Cloud Shell to setup things like env variables and install Python packages, while others start by opening an SSH terminal into a VM to do those preliminary steps.
I can't really tell the difference between the two approaches, other than the fact that in the second case a VM needs to be provisioned first. Presumably, when you use Cloud Shell some sort of VM instance is being provisioned for you behind the scenes anyway.
So how are the two approaches different?
Cloud Shell is a product that is designed to give a large number of preconfigured tools that are kept updated, as well as being quick to start, accessable from the UI, and free. Basically, its a quick way to get an interactive shell. You can learn more about this environment from its documentation.
There are also limits to Cloud Shell -- you can only use it for 60 hours a week, if you go idle your session is terminated, and there is only 5GB of storage. It is also only an f1-micro instance, IIRC. So while it is provisioned for you (and free!), it isn't really useful for anything other than an interactive shell.
On the other hand, SSHing into a VM places you directly in a terminal on that VM, much like you would on any specific host -- you only have whatever tools that the image installed onto that VM provides (and many VMs come pretty bare bones, it depends on the image). But, you're now in a terminal on the host that is likely executing the code you want to work with, and it has as much CPU and RAM as you provisioned in that instance.
As far as guides pointing you to one or the other -- thats really up to them, but I suspect they'd point client / tool type work to the cloud shell (since its easy and a reasonably standard environment, which can even be scripted with tutorials), while they'd probably point how to install necessary software for use in production to a "real" VM.
First, I guess I'd have to figure out if I'm running remotely and second I'd have to figure out whether my remote connection is a standalone remote app or an app running on a terminal server (that may be tricky).
But, once I've figured out all those awful things, is there a way to run a windows function like ShellExecute locally instead of remotely?
The reason I'd want to do this is because I launch a web browser to view rather high bandwidth things that require javascript and flash and certain sysadmins who administer our product aren't too keen on having to make unnecessary and insecure modifications to their terminal server farm.
Yes, if the clients are running Windows and you can install software on them.
See Remote Desktop Services Virtual Channels in MSDN.
There is a free tool that does exactly what you want. I got reference from TechNet forums, it's named Remote Executer from http://www.mqtechnologies.com
Good luck
I am trying to set up a clean vm I will use in many of my devs. Hopefully I will use it many times and for a long time, so I want to get it right and set it up so performance is as good as possible. I have searched for a list of things to do, but strangely found only older posts, and none here.
My requirements are:
My host is Vista 32b, and guest is Windows2008 64b, using Vmware Workstation.
The VM should also be able to run on a Vmware ESX
I cannot move to other products (VirtualBox etc), but info about performance of each one is welcomed for reference. Anyway I guess most advices would apply to other OSs and other VM products.
I need network connectivity to my LAN
When developing/testing, guest will run several java processes, a DB and perform some file I/O
What I have found so far is:
HOWTO: Squeeze Every Last Drop of Performance Out of Your Virtual PCs: it's and old post, and about Virtual PC, but I guess most things still apply (and also apply to vmware).
I guess it makes a difference to disable all unnecessary services, but the ones mentioned in 1 seem like too few, I specifically always disable Windows Search. Any other service I should disable?
You can try to run the CD/DVD through vLite to remove unwanted crap. I'm not 100% sure if Windows 2008 server is supported but you could give it a try. I've successfully stripped down XP with nLite to about 200MB with only the bare minimum I need for testing software. You might be able to do something similar to Windows 2008 with vLite.
My host is Vista 32b, and guest is
Windows2008 64b,
First mistake. Seriously, why not running 64 bit even on Vista? This would give your VM a good memory space to work with, while now even if it is possible with VmWare it goes through really nasty API's in the Windows layer.
That said, why use Vista as host? Why not directly load a 2008 R2 host, configure it into workstation mode (heck, you even get our friendly AERO if you install all the things the server leaves out per default) and be happy with it?
I guess it makes a difference to
disable all unnecessary services,
Hm, seriously? I run a couple of Hyper-V hosting servers on top of physical domain controllers without any reconfiguration and with good enough (i.e. great) perforamnce. Helps I dont ahve the typical workstation bottleneck (i.e. one overloaded hard disc). I never found a reason to disable any service for squeezing the last performance out.
Guest will run many java processes, a
DB and perform lots of file I/O
Well, get proper hardware for that. I.e. a hardware RAID controller, and a LOT of drives - in accordance with your needs. DB is IO sensitive. VERY sensitive.
I'm working on a windows platform and want to be able to auto sync my files one way 'on change' to my virtual windows or linux web server - also need to be able to filter file types. i can connect to the remote machine via network drives.
i'm ideally looking for a free, easy to set up solution - a commercial product that does what I need is called ViceVersa but its a little overkill and costs :)
Thanks
Josh
I'd use rsync - simple, easy to setup, and provides the filters you need. Also very low on bandwidth after the first pass.
Here is a link explaining how to get it working in Windows
Whilst rsync doesn't allow 'on-change' auto-syncing, it is very fast when it scans a sync'ed directory (even very large ones), so you could schedule a frequent sync to overcome this.
Edit: You could combine it with a program like this, to trigger an rsync on folder contents change. Cheaper than viceversa
For other users, its worth mentioning lsyncd, it will auto sync on changes between two machines (by default deferring to rsync). Will only work on Linux though, but if thats not a problem it works great.
It also seems that Sparkleshare has finally released some working code (Dropbox clone). Havent tried it myself but does cross-platform synching and you can setup your own server.
I have decided to go with Subversion for a source control repository for my personal and side projects and I'm now trying to decide what OS to use. Currently my file server for my home network is Windows 7 beta. I'm wondering if I should wipe it and install Windows Server 2008 instead? Basically I'd like to know if there are things I could take advantage with a server OS that I can't with Windows 7. First thing that comes to mind is accessing subversion remotely with a VPN connection.
I'm a .net developer, but have dabbled in Linux a bit so I'm not completely turned off to the idea of an ubuntu or debian server...
I imagine the installation and configuration process might go off with fewer hitches if installed on Linux, just because of the package management, but that's assuming some experience with the package system of $whatever_distro. If you're comfortable with Windows, Subversion works perfectly well on there. I've set it up on both, but prefer the Linux installation process (easier Apache integration, in my view), but I had pre-existing Linux experience.
If you're familiar with Windows, I bet you'll find the installation and configuration process easier there. As others have said, many of the tools are cross-platform.
You can run a Subversion server on Windows or Linux (or whatever) so it really doesn't matter. Pick whichever one you already have and feel most comfortable with. Since you are a Windows developer I see no real reason to toss Linux into the mix though.
If your goal is to minimize the amount of work you put into the maintenance of subversion, go with the OS you are most comfortable with. Many maintenance scripts, and subversion hooks are written and available in perl and python which are available for both windows and linux.
One advantage to the Windows server OSes over their client counterparts is that the client OSes are limited as to the number of inbound connections. If you are going to be the only person working on the repo, this may not make a difference. However, if there are multiple people, then this would be an issue. XP Pro/Vista Ultimate are limited by Microsoft to 10 inbound connections. I cannot speak for Windows 7.
To make life easy, try VisualSVN Server. For personal projects there's no reason to setup a separate server just for SVN.
Windows 7 will be able to host Subversion with no problems whatsoever..
If your file-server is already setup and working under Windows 7, I'd say stick with that.. Adding SVN is no reason to install a new OS
You don't need a server at all to use subversion.
If you've already got a file server on your home network, and you're doing this only for you and your personal projects, just use a subversion client such as TortoiseSVN and create your repository (or repositories) on your file server via network share (or mapped network drive, etc).
I wouldn't recommend this for multi-user setups (unless each has their own repository), but for a single user this is the simplest option. And using this approach, to answer your question, you wouldn't gain anything by switching to a server OS such as Windows Server 2008.
I'd actually recommend going with a hosted Subversion provider instead of setting up Subversion on Windows or getting a second server for that purpose. I work for ProjectLocker, but if you Google "subversion hosting", you'll see there are a number of providers that offer free or reasonably priced solutions. The advantages:
It's a hosting provider's primary job to keep your code safe, secure, and accessible, so they focus on uptime, backups, and security monitoring so you don't have to
You don't have to learn how to be a system administrator or Subversion administrator; several providers have user interfaces that make it easy to manage users and permissions.
Hosting instead of DIY lets you focus on what you actually care about: writing great software
I suggest you take a look at ProjectLocker and some of the other providers and decide which one is right for you. You may decide that doing it yourself is the best option for you, but for many people in your situation, a hosted solution has met their needs.