I read quite a lot on Google that ruby does not release back memory to the OS and I understand the as well because allocating memory from OS is a costly fair.
Which has led me to ask this
If a developer want the Ruby to release the memory back to OS from ruby how do they do it.
- I guess the answer is manually triggering a GC in Ruby.
But is it advisable to do (run manual GC) on a production application.
I'm thinking of creating a separate thread(or Actor) in Celluloid to start a GC at every 2 hours time.
Can anyone suggest. Is the above solution advisable for production.
Note: I think I also know GC.start (means stop the whole world in Ruby)
Note: Ruby Version 2.2.2 MRI Centos 4G RAM with 4 core.
And here is the reason for the question http://pastebin.com/rYdYwxLJ
I read quite a lot on Google that ruby does not release back memory to the OS and I understand the as well because allocating memory from OS is a costly fair.
That's simply not true. The Ruby Language doesn't say anything at all about releasing memory back to the OS. In fact, the Ruby Language doesn't say anything about memory, period.
There are a lot of different implementations of Ruby with a lot of different memory allocators and a lot of different garbage collectors. Some release memory back to the OS, some don't, some run on top of a managed runtime and don't even manage memory at all.
For example, JRuby running on J9 will release memory back to the OS. JRuby running on HotSpot with the G1 GC, ParallelScavenge GC, or ParallelOld GC will release memory back to the OS.
If a developer want the Ruby to release the memory back to OS from ruby how do they do it.
You don't. Either the implementation releases memory back to the OS, in which case you don't need to do anything, or it doesn't release memory back to the OS, in which you cannot do anything.
- I guess the answer is manually triggering a GC in Ruby.
If the implementation doesn't release memory back to the OS, then it doesn't release memory back to the OS. Period.
If the implementation does release memory back to the OS, then it will do that whether you manually trigger a GC or not.
By the way: you cannot manually trigger a GC in Ruby. You can suggest a GC (using GC::start), but not trigger it.
But is it advisable to do (run manual GC) on a production application.
Pretty much never. The GC knows better than you do whether it makes sense to run now or at a later time.
Note: I think I also know GC.start (means stop the whole world in Ruby)
No, GC::start simply suggests to do a GC. It does not force a GC. Also, GC does not mean stop-the-world. E.g. when running JRuby on the J9 JVM with the Metronome GC or running JRuby on the Azul Zing JVM with the Pauseless GC will never stop-the-world.
Note: Ruby Version 2.2.2
That's not the interesting bit. The interesting question is which implementation you are using, which garbage collector, and which OS. (And of course the versions and configuration settings for all of those.)
You likely misunderstood what you have read via Google links. The memory, that Ruby (here and after I mean MRI as default Ruby implementation) does not release back to OS (so-called Ruby Heaps,) is not released to OS. Period.
You have no ability to force Ruby to release Ruby Heaps back to OS (besides cold restart of the whole application.)
What might be released back to OS, is released by Ruby internally and it is a bad practice by all means to call GC manually, in 99% of cases that will slow down the application, in the rest 1% this will have no effect at all.
Related
Is there a way to completely free memory in TCL, meaning to reallocate it back to the operating system without closing the tclsh.85 process??
If you're doing this because you're about to unload the tcl DLL from your process, there's the Tcl_Finalize() API call. Otherwise, no, there's no way to do it by default. If you build Tcl with the PURIFY symbol defined (named for a commercial tool that does memory analysis) then Tcl will use the system memory allocator instead of its own high-performance thread-aware allocator; that might be able to release memory back to the OS. Or it might not; it's out of Tcl's hands entirely.
There's no script-level API for this at all. Most of the time, if a process uses a lot of memory it's either going to exit soon, going to reuse that much memory, or can rely on the OS just paging the unused memory out.
I've got a C++ program that is notably slower on OSX 10.8.2 than on Linux. Profiling shows that the reason is that calls to free (that result from STL operations, FWIW), are much slower on OSX, because they go and call madvise, and real time gets consumed in there.
Is there any way to modulate this behavior of OS/X?
Well, yes!
I had horrible performance issues with malloc/free in Linux and started looking for a replacement.
Two options came to mind tbbmalloc (part of Intel TBB which is free BTW) and Google malloc.
After extensive testing it wasn't clear which was faster (of the two) but both were significantly faster than LIBC's implementation.
I went with tbbmalloc since it was working smoother, google malloc had a bug that caused virtual memory to be very large (reserved but not committed) which was very bad for my app (IT daemons would kill it).
The good:
Much better performance than libc's malloc. Was 3x-300x in STL heavy app.
Simple integration. No code change. Add/change 1 line the executable's makefile. No change to SOs.
The bad:
Mem checkers will not with replacements. for memchk/valgrind/etc. revert to the original malloc.
App would take 10-30% more memory.
The app I developed was a CAD application that used 10s of GBs, building and destroying 10s of millions various structures (lots of STL maps, vectors, hash_maps).
How to do this:
In the linker command, add -ltbbmalloc and make sure the library is in the lib search path (-L flag).
I was wondering this and couldn't find anything except this
"Thread scheduler bug fixes and performance improvements. Threading on Ruby Enterprise Edition can be more than 10 times faster than official Ruby 1.8"
REE is derived from MRI 1.8.7. As such, it only used green threads. REE changes some parts of 1.8.7 (esp. in the areas memory management and garbage collection). But it still widely follows the design of the upstream MRI (the original Matz's Ruby Interpreter)
While YARV (1.9) switched to OS native threads, they still have a global interpreter lock making sure that only exactly one of these threads runs at a time.
There are a couple of Ruby implementations with OS native threads and without a GIL. The most prominent are JRuby (based on the JVM) and Rubinius (with its own VM). These implementations offer "real" concurrent threads.
Besides JRuby and Rubinius, who have got rid of an interpreter lock entirely, the state of affairs in CRuby/MRI has also made some progress with regard to concurrency.
One notable feature is that with the Bitmap Marking GC by Narihiro Nakamura, as of Ruby 2.0, another advantage of REE over CRuby will be gone: REE has a copy on write-friendly GC algorithm which made it attractive for achieving concurrency through processes (forking) rather than through threading. The new Bitmap Marking GC will have the same advantage of saving unnecessary copying of memory around when forking a new process.
The GIL (or GVL as it is officially called) is also not quite as bad as it sounds at first. For example, Ruby releases the interpreter lock when doing IO. Another feature that we see much more often lately is that C extension developers have the ability to manually release the lock by calling rb_thread_blocking_region, which will execute a C-level function with the GIL released. This can have huge effects if some operation in C is to be performed where we can rest assured that it will have no side effects. A nice example is RSA key generation - this runs completely in C with memory allocated by OpenSSL, so we can run it safely with the GIL released.
Fibers introduced in 1.9 or recent projects like Celluloid also cast a much more friendly light on the state of Ruby concurrency today as when compared to a few years ago.
Last not least, Koichi Sasada, the author of CRuby's VM, is actively working on the MVM technology, which will allow to run multiple VMs in a single Ruby process, and therefore achieving concurrency in yet another way.
Taking all the other performance improvements into account, there are less and less arguments for using REE, it's safe to switch to 1.9.3 or 2.0.0 once it's out, especially since the 1.8 series will no longer be actively developed and many popular projects have announced to quit their support for 1.8 sometime soon.
Edit:
As Holger pointed out, REE has also been EOLed, and there will be no port to 1.9 or further. So it's not only safe to switch, but also the right thing to do :)
Can the Ruby language be used to create an entire new mobile operating system or desktop operating system i.e. can it be used in system programming?
Well there are a few operating systems out there right now which use higher-level languages than C. Basically the ruby interpreter itself would need to be written in something low-level, and there would need to be some boot-loading code that loaded a fully-functional ruby interpreter into memory as a standalone kernel. Once the ruby interpreter is bootstrapped and running in kernel-mode (or one of the inner rings), there would be nothing stopping you from builing a whole OS on top of it.
Unfortunately, it would likely be very slow. Garbage collection for every OS function would probably be rather noticeable. The ruby interpreter would be responsible for basic things like task scheduling and the network stack, which using a garbage-collecting framework would slow things down considerably. To work around this, odds are good that the "performance critical" pieces would still be written in C.
So yes, technically speaking this is possible. But no one in their right mind would try it (queue crazy person in 3... 2...)
For all practical purposes: No.
While the language itself is not suited for such a task, it is imaginable (in some other universe ;-) that there a Ruby run-time developed with such a goal in mind.
The only "high level" -- yes, the quotes are there for a reason, I don't consider C very "high level" these days -- language I know of designed for Systems Programming is BitC. (Which is quite unlike Ruby.)
Happy coding.
Edit: Here is a list of "Lisp-based OSes". While not Ruby, the dynamically-typed/garbage-collected nature of (many) Lisp implementations makes for a favorable comparison: if those crazy Lispers can do/attempt it, then so can some Ruby fanatic ... or at least they can wish for it ;-) There is even a link to an OCaml OS on the list...
No, not directly
In the same way that Rails is built on top of Ruby, Ruby is built on top of the services that lower layers .. the real OS .. provide.
I suppose one could subset Ruby until it functionally resembled C and then build an OS out of that, but it wouldn't be worth it. Sure, it would have a nice if .. end but C syntax is perfectly usable and we already have C language systems. Also, operating systems don't handle character data very much, so all of the Ruby features to manipulate it wouldn't be as valuable in a kernel.
If we were starting from scratch today we might actually try (as various experimental projects have) to use garbage collected memory allocation in a kernel but we already have OS kernels.
People are making investments at the higher layers rather than redoing work already done. After all, with all the upper level software to run these days, a new kernel would need to present a compatible interface and the question would then be asked "why not just run the nice kernels we already have?".
Now, the application API for a mobile OS could indeed be done for Ruby. So, just as Android apps are written in Java, RubyPhone apps could be written in Ruby. But Ruby might not be the best possible starting point for a rich application platform. Its development so far has been oriented to server-side problems. There exist various graphical interface gems but I don't think they are widely used.
basically yes, but with a big disclamer .. which is basically Chris' answer but a different spin on it. Since for kernel performance it would kinda suck to use ruby, you'd probably want to build around a linux-ish kernel and just not load any of the rest of the operating system. This is basically what Android does: the kernel is a fork from Linux (and is maintained close to linux), the console is a webkit screen, and the interpreter is Java with some Android specific libraries. IE, Android is Java masquerading as an OS, .. you could do about the same thing with Ruby instead of Java and only a smallish hit to performance from java
While building a whole OS from scratch in Ruby seems like
a multi-billion project (think of all the drivers), a
linux kernel module that runs simple ruby scripts does
make sense for me - even it was only for prototyping
new linux drivers.
There is a very noticeable difference in application initiation time between running my specs from the command line with ruby 1.9.x vs. 1.8.7. My application initiates much faster with ruby 1.8.7 than with ruby 1.9.1 or 1.9.2. The application initiation difference is approximately 18 seconds. It takes about 5 seconds for my app to initialize with 1.8.7 and 23 seconds with 1.9.1 and 1.9.2.
Application initialization time is not a big deal for production, but it is a very big deal for BDD development. Every time I change my code and run my specs, I have to wait an additional 18 seconds per iteration.
I assume this application initialization time is attributed to YARV compiling bytecode as my application initializes.
Am I right about my YARV slowing down my application initialization, and is there a way to disable YARV on the command line. It would be very nice to be able to disable YARV only when I am running my specs.
YARV is a pure Ruby compiler. If you disable it, there's nothing left.
More precisely: YARV is a multi-phase implementation, where each of the phases is single-mode. It consists of a Ruby-to-YARV compiler and a YARV interpreter. If you remove the compiler, the only thing you are left with is the YARV bytecode interpreter. Unless you want to start writing your app in YARV bytecode, that interpreter is not going to be of much use to you.
This is in contrast to mixed-mode implementations such as JRuby and IronRuby which implement multiple execution modes (in particular, both a compiler and an interpreter) within a single phase. If you turn off the compiler in JRuby or IronRuby, you are still left with a usable execution engine, because they both also contain an interpreter. In fact, JRuby actually started out as a pure interpreter and added the compiler later and IronRuby started out as pure compiler and they added an interpreter exactly because of the same problem that you are seeing: compiling unit tests is simply a waste of time.
The only interpreted implementation of Ruby 1.9 right now is JRuby. Of course, there you have the whole JVM overhead to deal with. The best thing you can do is try how fast you can get JRuby to start up (use the nightly 1.6.0.dev builds from http://CI.JRuby.Org/snapshots/ since both 1.9 support and startup time are heavily worked on right this very moment) using either some very fast starting desktop-oriented JVM like IBM J9 or try JRuby's Nailgun support, which keeps a JVM running in the background.
You could also try to get rid of RubyGems, which generally eats up quite a lot of startup time, especially on YARV. (Use the --disable-gem commandline option to truly get rid of it.)
There's currently no way to disable YARV, simply because MRI 1.9 only includes the virtual machine, and not an interpreter. Maintaining both would be way too much job for the core team.
In the future there will probably be ways to cache the bytecode YARV generates (like Rubinius does). At the moment there is no way to load such bytecode through Ruby (see #971), but you could easily write a C extension which accomplishes it.
However, I would say that 18 seconds is way too much and it's probably a bug. I know there are some threads at ruby-core which discusses the slowness of require; maybe you find something interesting there!
the next RC of 1.9.2 out might be faster as it doesn't preload $: with all your gem lib dirs.