First with the background...
We have a Linux server that supports multiple projects.
The Clearcase server and repository are installed on this Linux server.
Different projects require different cross-compilers and libraries, and all of them are installed on the server.
User can choose different tool sets by running different scripts, which exports different environment variable values such as include paths and compilers.
User needs to run cleartool to mount the repository.
Developers develop in Eclipse and have two options:
SSH into the server and run Eclipse through with X11 tunneling.
Install Eclipse locally on their Windows machine and invoke builds from the SSH terminal.
Now:
Problem with #1 is that Eclipse operations (typing, content assist, etc) are extremely laggy.
Problem with #2 is that the developers need to go through extra hoops to build their code.
This is what I have tried:
Set up Remote System Explorer, which allows remote editing of files and remote running of the compiler:
How to build a c++ project on a remote computer in Eclipse?
This approach works perfectly for files that do not need special environment variable values and mounting of Clearcase repository, but I could not figure out how to get all of these things to integrate.
It would be great if someone can let me know how I can direct RSE to run a script (may be different per project) to set the environment variables and to run the cleartool commands to mount the repository so that it can locate the files.
The cleartool command arguments would be different per user for setting up a particular view.
Some extra info that may help:
I have root access to the development server
The Clearcase filesystem is mapped to a drive on the Windows machine
Thanks in advance for saving me hours of frustration dealing with a slow network!
==================
Additional detail per comments:
- The VOB storage is located locally on the Linux server. We would SSH to the server and start Eclipse there, therefore the delay should not be due to dynamic vs snapshot view and GUI performance seems to be the real problem.
- We also mount the same view on Windows by using Region Synchronizer. When running the local copy of Eclipse installed on Windows, there is no performance problems.
So this question can probably be solved by answering either question:
1. How to improve X11 performance such that development on Linux will suffice?
2. How to set up Windows Eclipse to perform all the steps mentioned above when building projects?
I came here a similar question to your part two, but alas, no one has answered it. However, I have an answer to your part one: https://www.nomachine.com/. It speeds up X11 forwarding considerably.
Related
I am trying to build and deploy a gaia build from git repo in Windows. I am trying to deploy it in অ Flame.
I am trying to do it in a windows 7 with cygwin installed. After installing everything this is the error I am getting
This works just fine in a Linux machine, but I need to do this in Windows since right now I have access to it only.
Any pointers to what I am doing wrong here?
I'm afraid it's not going to work without significant effort for several reasons. Much better to use a VM with Linux on as even if it did work it will be really slow. Windows is slow at handling lots of file access and Cygwin slows it down even more.
For example in making a simple change to config.sh (full stack build) so it works on Cygwin I found it took hours to run (on a decent PC). And then I had a couple of corrupt git repos I had to hand fix.
I also looked at getting gaia's make to work, but stopped after the problem just got bigger.
Here's what I found for future reference
The build is not really portable, it expects a linux like environment
While cygwin gives good linux emulation most of the tools run are win32 native and handling path conversion for them requires not trivial changes due to assumptions. For example you can switch to the Win32 XPCshell and hack the command line paths to use cygpath, but environment variable are an extra source of dependency in the JS scripts and are all unix paths. ( I did manage this part).
these path and environment dependencies get magnified with the C build chain and other tools.
You need to change the mount to use noacl or else cygwin attaches ACLs to simulate file properties, thus breaking things. It's might even be a little faster without ACLS
I also tried MinGW which provides native versions without the emulation so should be faster. However it falls short of the requirements and its automatic path conversion heuristics get in the way.
you need to turn of any antivirus prog as they slow it down. in fact the very first time I used the old FIrefox WIndows build it would crash after a long time. Turned out to be a mem leak in the AV :(
So all-in-all it's too much hassle in terms of dev time to convert and probably maintain. A true Windows build would be better but then it's so easy these days to run a VM. You can even share directories between the guest and host so could flash from Windows.
I also tried with cygwin, but was unable to build the gaia source code on windows.
It's not straightforward to build the gaia source code on windows. Please follow these steps:-
Download Mozilla Build from MozillaBuild - Mozilla Wiki and install the tools in c:/mozilla-build (preferred). It includes everything (make, wget, python etc) you need to build gaia source code.
Run start-shell.bat. If build process failed with this batch file then run start-shell-msvc2013.bat if you have Visual Studio 2013 or start-shell-msvc2015.bat if you have Visual Studio 2015. (You need Visual Studio for the second step).
Browse to the gaia source code directory using the command cd Mozilla/gaia.
Run DEVICE_DEBUG=1 make command. Don't run DEVICE=1 make or make command (because you won't be able to debug the apps, I was able to connect to the Firefox OS 2.2 but was not able to debug the apps when I ran these commands).
If you are running this command for the first time, it will download the b2g_sdk otherwise it will create a folder profile with your custom profile.
Open the WEBIDE using Firefox (Nightly preferred) and point to the profile folder you just created.
Links for your reference:-
https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Developing_Gaia
https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Different_ways_to_run_Gaia
https://developer.mozilla.org/en-US/docs/Tools/WebIDE/Troubleshooting
https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Making_Gaia_code_changes
https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Windows_Prerequisites
I am starting with Bazaar (switching from Subversion, sorry if terminology is a bit off sometimes). Using Bazaar locally no problems.
Got bzr+ssh:// working on my Windows server (finally! and even ssh agent is working wow!)
Now I want a shared repository on the server from which the developers can branch to their machines. I want the repository in a specific folder, for example in C:\bzr\MyProject.
When I do:
bzr init-repo --no-trees bzr+ssh://myserver.com/MyProject
it creates the repository in C:\cygwin\home\user\MyProject in the home directory of my user account - it is understandable, but worrying.
Then as an experiment I also tried and succeeded:
bzr init-repo --no-trees bzr+ssh://myserver.com/C:\bzr\MyProject
This created the repository where I wanted. But how do I "map" or "alias" the URL (or bzr) so my developers can logon under their accounts and use URL
bzr+ssh://myserver.com/MyProject
to access the shared repository in C:\bzr\MyProject?
Obviously I don't want developers to use bzr+ssh://myserver.com/C:\bzr\MyProject because of the physical path in the URL.
Ok my own answer is if you want to run Bazaar on a Windows server for a development team who will access it over the internet,
and you are not familiar with Linux, Cygwin, SSHD and related stuff then it might turn out more complicated than you can bear.
I actually abandoned the Bazaar idea and gone with Mercurial. I must say Mercurial install on the server is also steep, but at least it is just Windows, IIS and only a bit of Python. Got it running in half a day.
Some of the problems that I had with bzr+ssh:// on a Windows server are:
Needs SSHD installed on the server. SSHD (from stripped down Cygwin) supplied with Bazaar refused to work. Had to install Cygwin and learn a bit of Linux stuff, how to run as a service, how to configure, how to generate keys.
Hard to add a new user in a way that does not request passwords typing for each command. Will need to generate a keypair, mock with copying the keys to th server in two locations (Cygwin's home user folder and Window's too). Probably need to log on the user to Windows to create a profile. Don't want developers logging on to the server actually.
Hard to set up a shared repository in a specific location on the server. Does not seem possible with bzr+ssh. Possible with sftp. Might need to use symlinks as bialix suggested above.
As a newbie to linux stuff I don't understand all implications of running sshd on the server and giving shell access to the developer accounts. Have to use bzr shell limited... documentation is scarce.
Basically, bzr+ssh:// on a Windows server seems to be what installing Subversion on a Windows server was like several years ago - hard. Hopefully it will get better with Bazaar too because I chose it over Mercurial initially.
I got a question of using Eclipse to develop for embedded Linux on a Windows host
Here are now I have and where I am.
1. a Windows host that have the latest Eclipse + CDT (c/c++ development tools) installed
2. a Ubuntu host (ssh + samba installed) that contains sources and toolschain to build the project. (the windows and ubuntu hosts are sitting within one network segment (In LAN).)
3. I can use the following commands to build this project under Ubuntu.
# chroot dummyroot
# cd /home/project/Build
# sh Build date +%Y%m%d%H%M%S
4. I am now trying to create an eclipse C++ project to achieve the goad of the step 3, but I have been stuck here for a while. any ideas of how it can be done?
Speaking from experience, attempting to develop embedded Linux on a Windows host is a world of pain and frustration. Emphasis on attempting -- I'd like to meet someone who completed a serious project in this way, or who can explain how working in Windows made job easier.
Windows can be a great development environment for many tasks, but it's a lousy, lousy environment for embedded Linux. There are tools out there to help you do this, but everything is much harder compared to working on a Linux host. The toolchains are older and buggier. You will constantly fight with your crosscompilers and GNU autotools trying to get packages to compile. (Wait till you try to compile one of the many packages that needs to build an intermediate binary and then executes that as part of its build process.)
You probably have reasons for keeping your desktop in Windows -- just run a Linux virtual machine.
Eclipse CDT can execute arbitrary commands, such as
ssh username#target build_script
using the external builder. If you are using gcc on the target eclipse cdt can parse the output of the gcc and make from the ssh session and send you to the correct source path (if that path matches the path generated by gcc). Although this might not work windows->linux
With Embedded Linux, I've attempted to run Virtual Machines using Ubunutu, Xubutunu, Debian. I have been developing for a long time. I design hardware and develop low level firmware, to test said hardware. I cannot get anything to work on instructions given, as in an unified IDE and development system such as I am used to (i.e. Codewarrior, MPLAB, Code Composer) If Embedded Linux is so useful and easy how is it I cannot get a single one of instructions from Yocto, Freescale, Timesys, anywhere to work? Every-time there is a directory change, or a directory that no longer exists, or even a file that is not there. Surely there something that I can use..
my project is a PHP web application. This applies to my test server (local), not production server! I am also the solo developer on this project (however, that may change in the very far future). Also, all my source code is committed to a repository and the production server gets the source code from the repository.
I do my development in Windows while my test server runs on Ubuntu (perhaps you can also recommend me another distro that is easy to use and can serve as a good web server). I need an elegant way to interface between the two environments. Currently, I do my coding in Windows and then FTP the changed files to the test server. However, this is quite cumbersome and tedious since I have to manually go to my FTP client each time. Suggest me something elegant please! Perhaps FTP sync? or OpenVPN (where the root www directory on test server is acts like a folder in Windows)? Thanks for your awesome time!
Easiest would be in Ubuntu, right click a folder then click "Sharing Options", then share the folder. In Windows, connect to the share, and work on that copy.
If you're using version control, using continuous integration like Hudson ( http://hudson-ci.org/ ) would help if you create a task that builds/exports the website for the testing server. This approach would be better in the long term, but you'll waste a day setting it up initially.
I prefer SFTP to FTP.
That said, ExpanDrive lets you map SFTP servers to local drive letters, which then means you can use any text editor to access your files directly on the test server, or use other mechanisms to keep the files in sync. Since they show up as two local drives, you can use just about any product out there.
If you want to use FTP, you can just map the drive in Windows Explorer. If you open up My Computer, then go to Tools > Map Network Drive, you can map a FTP server folder to any local drive. Just type in the address as the folder, ie. ftp://mscharley#192.168.0.10/htdocs
This will atleast save you a trip to the FTP client...
Is there any reason you couldn't just test on your local computer? At my job, we all develop and do developer testing locally, most of us using Windows. Our production and test servers are all linux based. Working locally is really nice, because you don't need to worry about making changes on the server with every small change.
Another option would be to create a checkout or working copy of your code on the server, and then run svn up or svn export (or equivalent using your version control software) each time you change the code on the server (assuming you are sshd into the server). This is kind of slow, but it's easy. The other option would be to write a script that goes through the svn logs for the recent commit and only exports or updates the ones that changed. This is much faster, and for all I know, there is already something out there that this.
Finally, some IDEs allow you to edit files live over ftp\sftp. Basically the IDE downloads a copy of the code and then reuploads it when you save.
Currently I develop on windows (PHP) as well and deploy on a Linux box for testing and production. This is how I do it.
Set up a local development server with e.g. WAMP.
Set up your code base in version control, e.g. Subversion.
Checkout your code base onto the testing/staging server, not just only on your local dev. environment.
In the early stages of development you want to deploy to the testing environment A LOT to sort out any discrepancies between your windows and linux environments. When your programming efforts turn more into program flow type programming this constant testing will probably slow down. But still take the effort to test on a regular basis.
To test your code base on staging do an svn update. I just log in with an SSH session to do this. A key thing here to note is that you do not have to make any config changes to your code base. If you do need to make config changes to your environment on staging it worth while spending the time to SCRIPT this process rather than this being a manual process.
Do the same for production. I use an Subversion check out on production as well. Make sure you set you .htaccess file to deny access to your hidden .svn folders and script the deployment especially if there a config changes necessary.
Some ideas:
Use a server environment under windows (e.g. EasyPHP).
Use a development tool that can save over FTP (e.g. ultra edit).
Use a network drive connected to the remote machine via FTP.
Use a network drive connected to the remote machine via Samba.
Run a linux distro inside a virtualization tool (e.g. virtual box) and write from the windows host to a share directory of the guest host.
Use dropbox to sync files between machines (there is more a hack than an "enterprise" solution).
I'm currently working on a server-side product which is a bit complex to deploy on a new server, which makes it an ideal candidate for testing out in a VM. We are already using Hudson as our CI system, and I would really like to be able to deploy a virtual machine image with the latest and greatest software as a build artifact.
So, how does one go about doing this exactly? What VM software is recommended for this purpose? How much scripting needs to be done to accomplish this? Are there any issues in particular when using Windows 2003 Server as the OS here?
Sorry to deny anyone an accepted answer here, but based on further research (thanks to your answers!), I've found a better solution and wanted to summarize what I've found.
First, both VirtualBox and VMWare Server are great products, and since both are free, each is worth evaluating. We've decided to go with VMWare Server, since it is a more established product and we can get support for it should we need. This is especially important since we are also considering distributing our software to clients as a VM instead of a special server installation, assuming that the overhead from the VMWare Player is not too high. Also, there is a VMWare scripting interface called VIX which one can use to directly install files to the VM without needing to install SSH or SFTP, which is a big advantage.
So our solution is basically as follows... first we create a "vanilla" VM image with OS, nothing else, and check it into the repository. Then, we write a script which acts as our installer, putting the artifacts created by Hudson on the VM. This script should have interfaces to copy files directly, over SFTP, and through VIX. This will allow us to continue distributing software directly on the target machine, or through a VM of our choice. This resulting image is then compressed and distributed as an artifact of the CI server.
Regardless of the VM software (I can recommend VirtualBox, too) I think you are looking at the following scenario:
Build is done
CI launches virtual machine (or it is always running)
CI uses scp/sftp to upload build into VM over the network
CI uses the ssh (if available on target OS running in VM) or other remote command execution facility to trigger installation in the VM environment
VMWare Server is free and a very stable product. It also gives you the ability to create snapshots of the VM slice and rollback to previous version of your virtual machine when needed. It will run fine on Win 2003.
In terms of provisioning new VM slices for your builds, you can simply copy and past the folder that contains the VMWare files, change the SID and IP of the new VM and you have a new machine. Takes 15 minutes depending on the size of your VM slice. No scripting required.
If you use VirtualBox, you'll want to look into running it headless, since it'll be on your server. Normally, VirtualBox runs as a desktop app, but it's possible to start VMs from the commandline and access the virtual machine over RDP.
VBoxManage startvm "Windows 2003 Server" -type vrdp
We are using Jenkins + Vagrant + Chef for this scenario.
So you can do the following process:
Version control your VM environment using vagrant provisioning scripts (Chef or Puppet)
Build your system using Jenkins/Hudson
Run your Vagrant script to fetch the last stable release from CI output
Save the VM state to reuse in future.
Reference:
vagrantup.com
I'd recommend VirtualBox. It is free and has a well-defined programming interface, although I haven't personally used it in automated build situations.
Choosing VMWare is currently NOT a bad choice.
However,
Just like VMWare gives support for VMWare server, SUN gives support for VirtualBOX.
You can also accomplish this task using VMWare Studio, which is also free.
The basic workflow is this:
1. Create an XML file that describes your virtual machine
2. Use studio to create the shell.
3. Use VMWare server to provision the virtual machine.