I have been running drush scripts (for Drupal) with Cygwin on my relatively fast windows machine, but I still have to wait about a minute for any drush command (specifically drush cache clear to execute).
I'm quite sure it has something to do with the speed of Cygwin since my fellow developers (who are running Linux) can run these scripts in about 5 seconds.
Is there a way to make Cygwin use more memory and/or CPU per terminal?
The problem you're running into is not some arbitrary limit in Cygwin that you can make go away with a settings change. It's an inherent aspect of the way Cygwin has to work to get the POSIX semantics programs built under it expect.
The POSIX fork() system call has no native equivalent on Windows, so Cygwin is forced to emulate it in a very inefficient way. Shell scripts cause a call to fork() every time they execute an external process, which happens quite a lot since the shell script languages are so impoverished relative to what we'd normally call a programming language. External programs are how shell scripts get anything of consequence done.
There are other inefficiencies in Cygwin, though if you profiled it, you'd probably find that that's the number one speed hit. In most places, the Cygwin layer between a program built using it and the underlying OS is pretty thin. The developers of Cygwin take a lot of pains to keep the layer as thin as possible while still providing correct POSIX semantics. The current uncommon thickness in the fork() call emulation is unavoidable short of Microsoft adding a native fork() type facility to their OS. Their incentives to do that aren't very good.
The solutions posted above as comments aren't bad.
Another possibility is to go through the drush script and see if there are calls to external programs you can replace with shell intrinsics or more efficient constructs. I wouldn't expect a huge speed improvement by doing that, but it has the nice property that you'll speed things up on the Linux side as well. (fork() is efficient on Linux, but starting external programs is still a big speed hit that you may not have to pay as often as you currently do.) For instance:
numlines=`grep somepattern $somefile | wc -l`
if [ $numlines -gt 0 ] ; then ...
would run faster as:
if grep -q somepattern $somefile ; then ...
The first version is arguably clearer, but it requires at least three external program invocations, and with primitive shells, four. (Do you see all of them?) The replacement requires only one external program invocation.
Also look at things that slow down Cygwin startup:
Trim down your Windows PATH (to the bare bones like %SystemRoot%\system32;%SystemRoot%)
Remove things you don't need from bashrc and bash_profile
Move things you only need in your terminal window from bashrc to bash_profile
One surprisingly large time suck in Cygwin is Bash completion. If you are using it (and you should because it's great), only source completion for the commands you need (rather than all of them which used to be the default). And, as mentioned above, source them from bash_profile, not bashrc.
You can give Cygwin a higher priority.
Write a new batch file, for example, "cygstart.bat" with the following content:
start "Cygwin" /high C:\cygwin\Cygwin.bat
The /high switch gives the shell a higher process priority.
Related
We are running a Windows .EXE file via a Cygwin script and are encountering performance issues. I have seen various posts about Cygwin performance issues, including this one, one of whose answers delves enough into Cygwin internals to make me think there could be a problem. However, these posts do seem to be more about startup time, tab completion, etc. Before we launch on a benchmarking witch hunt, I was hoping to ask: is there any intrinsic reason why a Windows .EXE could run slower if kicked off from Cygwin vs. BAT?
Not the actual program, no.
Housekeeping and stuff before running the program may vary. Cmd probably calls CreateProcess directly. Cygwin's bash may first do argument parsing, wildcard expansion, fork via Cygwin's slow implementation and call exec with the parsed arguments, which Cygwin has to piece together into a string again to pass to CreateProcess. But in the end, a new process is created which has no ties to its parent anymore. So how fast your program runs entirely depends on that program, not on who launched it.
I have searched the web for long time without success. How can I add a new command to the kernel command line? e.g. a command like ver without any arguments, that simply prints Hello World message when executed.
The Linux kernel does not handle any commands (but the GRUB bootloader can pass some arguments to the booted kernel). It is just handling system calls. You could add some new one, but learn about advanced linux programming & syscalls(2) before hacking the kernel.
Perhaps you want to add a new command available to your shell. This is quite simple, put a new executable file (either an ELF executable, e.g. compiled from C code, or some script) in a directory known to your PATH. You might also make a new shell function.
The system call to create a new process is fork(2) (sometimes emulated thru some appropriate call to clone(2)...). fork is difficult to understand, so take several hours to read about it. Almost every process (with a few exceptions, notably /sbin/init) is created by fork. The system call to execute a new program is execve(2). Your shell will often use fork & execve to run commands (except builtin ones, like cd). Use strace(1) to understand what syscalls are done (by some program).
PS. Don't dare hacking your kernel before understanding quite well user-land Linux system programming....
Lets say I had a series of operations I wanted to apply to some data. The programs implementing the operations are not necessarily written in the same language, but they all work by reading from STDIN and writing to STDOUT.
In a unix environment it can be set it up as a pipeline like:
cat data.txt | prog1.sh | prog2.pl | prog3.py | prog4 > out.txt
and it will execute the 4 operations concurrently on the stream of data.
Does the same happen in windows?
I remember testing this out a few years ago with cygwin on windows xp, but I only saw a single prog running in the task manager.
Has anything changed with cygwin, the new xp service packs, or windows 7/8 that would allow for concurrent pipelining? Or has it always worked and I just made a silly mistake in my tests?
I don't have access to a windows machine right now or I'd test it out myself. If someone knows what's going on, I appreciate any help.
While the Unix-like layer implemented by Cygwin has many flaws compared to a native POSIX system or to native Windows programming (especially where performance is concerned), the pipes it implements are quite "real." The programs in the pipeline will run concurrently and will process the data they receive in parallel.
However, as with any pipeline, the speed of the entire operation will be determined by the speed of the slowest component. So if one of the programs in the pipeline is markedly less efficient than the others, it will dominate the CPU usage in the process list.
my goal: to create a suite of scripts that do some common system tasks, which include these
copy/move/list/search/grep files
watch/start/stop processes
run queries against Oracle via sqlplus
i grew accustomed to using Cygwin/bash to ease my life at work, and frankly speaking, i don't want to move away from bash language and start learning PowerShell, for example - so i started searching for a way to run bash scripts on Windows, ... preferably something alternative to Cygwin.
the truth is that i am still not pleased with Cygwin installation, and the fact that there is no simple way around it, that it is targeting more or less expert users, and there are a number of things that might pop up during the installation. i mean. what i am trying to do now is to write a suite of scripts that will target someone less expert than me (and i am in no way a real expert) - in most cases some kind of an administrator who doesn't want to know the script details.
i am thinking that this user will also want to be able to run these scripts on another machine, and i want to be able to explain him/her how to do it, without just saying, call the support, and, me, eventually (so that we can install cygwin on another machine etc etc.)
i tried MinGW(msys) but it also needs manual steps to set things up - i mean, these manual steps have become something of a de facto standard in these Windows ports (sorry, maybe i have a mood for bragging). win-bash looked like it could be a solution, but i ended up trashing it too, because of the old bash version, and its inability to do things i was able to do in cygwin - specifically
here documents
things like "cmd /C dir *" (don't know why) - and yes, i do cmd /C dir in cases i am in some kind of shared network folder with thousands of files, and ls is significantly slower than dir
my questions at last:
am i doomed to use PowerShell? i guess i will, reluctantly, if i have to
is there a simple pre-packaged "slim" cygwin installation.. or, portable cygwin, even better? there is a cygwin-portable project on sourceforge, but it's not that doesn't need those manual steps, again, apparently - is there a way to automate those steps, perhaps? and if there is, i wonder why somebody hasn't done it already? - and then, would it be possible to call bash scripts from Windows command prompt using a simple command like "bash somescript.sh"?
thanks for your attention.
As mentioned here, the Cygwin installation can totally be scripted and parametrized to ran in a silently and automatic mode.
If you define the minimal list of cygwin packages you need, just use a little .bat script that call the cygmin setup executable like this
setup.exe --packages=list_of_packages_you_need --quiet-mode
If you wrap the cygwin install process, it should be tolerable for a less technical user.
The cygwin install can be streamlined using command-line args;
http://sources.redhat.com/ml/cygwin-apps/2003-03/msg00526.html
You can also automate the install of most cygwin packages through cyg-apt.
I haven't verified this but I suspect that msys implements a *nix look alike by creating windows executable versions of system commands. All of the common commands have an executable on my install of msys. If that is true then it should be possible to use them separate from a complete install.
Try copying "bash.exe", "cp.exe", etc. from the msys bin directory to a machine/vm that does not have an msys install and see if it works. You may need to copy some dll's or shared libraries as well. A windows dependency checker program would show which dll's an executable is using.
You could package up the stuff you use and make a simple installer or just copy the files with your scripts.
Take a look at MKS Toolkit. Unlike Cygwin, it can live within the Windows world. Files end in CR/LF like Windows files, and you don't have that /cygdrive/c stuff. Naked drive letters work fine in MKS Toolkit.
A few caveats:
I haven't used MKS Toolkit in a long time. See following reason.
MKS Toolkit is (sit down for this) $600 per license. Ouch! That's why I use Cygwin even though I don't think it's as good or works as well.
It's Kornshell based and not Bash (although this may be a bit different). Kornshell and BASH are 95% alike. However, that last 5% gets you. I actually like Kornshell better than BASH in many respects. Kornshell has the print statement which is way superior than the echo statement. Variable names don't disappear in blocks. You can easily do double loops because almost all the commands can take unit numbers of input and output. However, Kornshell doesn't have those neat escape characters in the prompt, and it's hard to find the exit status of a command in the middle of the pipeline.
If while an application is running one of the shared libraries it uses is written to or truncated, then the application will crash. Moving the file or removing it wholesale with 'rm' will not cause a crash, because the OS (Solaris in this case but I assume this is true on Linux and other *nix as well) is smart enough to not delete the inode associated with the file while any process has it open.
I have a shell script that performs installation of shared libraries. Sometimes, it may be used to reinstall versions of shared libraries that were already installed, without an uninstall first. Because applications may be using the already installed shared libraries, it's important the the script is smart enough to rm the files or move them out of the way (e.g. to a 'deleted' folder that cron could empty at a time when we know no applications will be running) before installing the new ones so that they're not overwritten or truncated.
Unfortunately, recently an application crashed just after an install. Coincidence? It's difficult to tell. The real solution here is to switch over to a more robust installation method than an old gigantic shell script, but it'd be nice to have some extra protection until the switch is made. Is there any way to wrap a shell script to protect it from overwriting or truncating files (and ideally failing loudly), but still allowing them to be moved or rm'd?
Standard UNIX file permissions won't do the trick because you can't distinguish moving/removing from overwriting/truncating. Aliases could work but I'm not sure what entirety of commands need to be aliased. I imagine something like truss/strace except before each action it checks against a filter whether to actually do it. I don't need a perfect solution that would work even against an intentionally malicious script.
You can prevent a script from overwriting through I/O redirection by
set noclobber
Preventing overwriting by cp and the like is harder. My inclination would be to reset the PATH for the script to run with PATH containing just a single entry, a "blessed" directory where you place commands that you know are safe. This might mean, for example, that your version of cp is arranged always to use the --remove-destination option (probably a GNU-ism). In any case, you arrange for the script to execute only commands from the blessed directory. You can then vet each such command individually.
It would be good if you could prevent a script from executing a command by absolute pathname, but I don't know how to do that. If you are doing installations in your regular directories, a chroot jail probably does not help unless you do a lot of loopback mounting to make those directories visible. And if the directories into which you're installing contain dangerous commands, I don't see how you can protect yourself against them completely.
Incidentally, I tried and failed to learn if install(1) removes the desitination before installing. It would be interseting to learn.
Bash script I presume? Is the script very long? If not, you can do this manually:
if [ ! -f /tmp/foo.txt ] #If file does not exist
then
...code
fi
But I think you're asking for a way to wrap this around the script. You can certainly monitor the file writes with strace but AFAIK it doesn't have the functionality to interrupt them, unless you set up some sort of Intrusion Detection System with rules etc.
But to be honest, unless it's a huge script, that's probably more trouble than it's worth
Write your own safe_install() functions and make sure they're the only methods used. If you really need to be sure, run two processes. One would have permissions to make changes and the other would drop all privileges early and tell the other script to do the actual disk work.