I'm working on a Ruby on Rails application which has a memory leak, so eventually it crashes when there's no more memory.
However, in the final stage it basically only running the GC and processing very few requests, so basically DoS-ing itself. This DoS time was between 1 hour and 6 hours for my application!
I tried to locate the memory leak but no luck so far, so now I want to find a workaround for the production server.
Is there a way to configure the MRI Ruby GC so that when it reaches the memory limit then it just crashes? I mean to crash at the first time when Ruby tries to allocate more memory and the operating system denies it.
As far as I know, you cannot do that.
But your have another options:
Setup something in your system, which will prevent ruby from using too much memory (oom maybe?)
Setup your webserver to kill itself - like in this gem
Related
I have a console application that runs for less than 500ms, but that according to BenchmarkDotNet allocates more than 100 MB.
I am trying to figure out what are those 100Mb because it does not add up. However I cannot find a tool to do so in Linux or Mac. Once the method the app calls is over, the GC can clean all that memory without problems, so it is not a leak I can see in a dump, unless I take the dump in the very exact moment before exiting the method. I am not clear which is the moment in which the algorithm peaks in memory usage.
I can take CPU traces using dotnet-trace and show it in the browser with Speedscope, but I cannot show in Speedscope a trace when using gc-verbose or gc-collect as provider.
Is there a way with dotnet-trace to print in the console the stats of the created objects or anything like that?
try out dotnet dump and checkout this article by Tess Ferrandez.
and maybe you can share a little bit more information please.
It must be obvious, but I cant get a usecase of Delayed Job, cause due to ruby`s Gargabe Collector specific, it doesnt free memory back to OS. And once delayed job process will take all memory anyway. And the only way is to restart delayed job process.
But if I restart delayed job process and there is currenlty running task - it will never be completed. Probably, there is some workaround to restart that task later, but this approach seems ugly to me.
I tried real jobs and some simple computatuin without any variables, symbols or links so I dont think that "my code leaks". Still, every new job increases memory of delayed_job process.
May be I use Delayed job for something that its not designed? Or it could be environment problem (besides, tried on local machine and on VPS) ?
Tested on: Ubuntu 14.04 and Debian 6 (both x86), Rails 3.2, delayed_job 4.0.2, delayed_job_active_record 4.0.1, ruby 2.1.2
I could give some code examples, but, as I mentioned, I tried both: real job and simple computation. So I won`t if it is not significant and my mistakes are fundamental.
Due to my conditions - my tasks can be executed for couple of minutes, read and write about 100K records to database and require a lot of computation, tasks cant be interrupted, and number of tasks limited by 10-20 dayli, may be - I only guess to use Resque, because it forks process everytime, so there should be no problems with accumulating memory with time.
So do I realy do something wrong or this is a nature of DJ - to occupie all memory or require a restart - and if I cant restart it, I shouldnt use its approach ?
Everything I read on the internet (not so much, by the way) tells that its rubys GC trouble that it doesnt free memory back to OS, and some advises to profile code for unlinked objects (it sounds the most realistic to my case, but, I tried a lot with code that doesnt create any objects, and I explicitly set everything to nil and call GC.start)
Last week we upgraded a small project which was running on Rails 3.0.1 to Rails 3.2.2.
Shortly after the upgrade we recognized that occasionally, 2-3 times a day, we're seening a huge Phusion Passenger process (1-5 GB).
We're running Phusion Passenger 3.0.11 and Ruby 1.9.3-p0. We already tried different Ruby version (1.9.2-p290 and 1.9.3-p125) without a success.
Afterwards, we tried to track our memory usage with Oink. Unfortunately Oink doesn't show the reason for the memory bloat - The large processes seem to not write to the logfiles anymore.
When we downgraded back to Rails 3.0.1 the problem is gone. Does anyone have similar problems? We really checked our code for possible memory leaks, such as too many ActiveRecord instances, but didn't find any.
Do you think it's worth it to try Unicorn instead of Passenger? Or is it likely that we'll run into the same problem?
Any suggestions about how to trace the memory leak are welcome. We already set up newrelic, but it doesn't show detailed information about the memory leak.
If this is Red Hat compatible Linux, you can use SystemTap. I am not sure if SystemTap is available on Debian/Ubuntu systems, if not there is alternative called DTrace. Here are some articles - I have been pretty successful with tracking down several regressions, although none of these were memory issues (maybe you can find a STP script that could do the trick for you). Read here:
http://lukas.zapletalovi.com/2012/02/peek-into-your-ruby-app-with-systemtap.html
http://lukas.zapletalovi.com/2012/01/probing-ruby-apps-with-systemtap-in.html
http://sourceware.org/systemtap/wiki/RubyMarker
The last link shows probes you can hoop into in SystemTaps in Ruby. There are thinks like gc runs or memory allocations, could help you. Good luck!
I would be interested to know what passenger-memory-stats shows and what type of Memory you have passenger set for PassengerMaxPoolSize PassengerPoolIdleTime and any other passenger settings.
How did you upgrade passenger?
What is your apache setup like? prefork or worker?
I suspect that you are seeing poor gc performance, attempt to tune them by putting this in a wrapper around the ruby that passenger uses:
#!/bin/sh
export RUBY_HEAP_MIN_SLOTS=600000
export RUBY_GC_MALLOC_LIMIT=59000000
export RUBY_FREE_MIN=200000
exec "/usr/bin/ruby" "$#"
It should fix some performance issues you are seeing in general due to poor ruby gc defaults.
I have a Ruby application that uses eventmachine and starts 16 processes that each manage 1000 connections.
Initially each process only uses around 150MB, however after some runtime they consume more and more towards 500MB and I am running out of memory and swap.
The amount of open connections (indicated by EM.connection_count) is normal (around 1000 all the time), so there shouldn't really be references to old connections anymore.
Unfortunately, memprof only runs under Ruby 1.8, so this is not an option in my case.
I don't want to build the ITAPPMONROBOT for my application just so I can keep it running 24/7/365. How can I find the memory leak here or how can I help the GC?
There is a known memory leak in 1.9.2's Kernel#method that affects most EM applications. See http://groups.google.com/group/eventmachine/browse_thread/thread/fa56ff02440a624d and http://redmine.ruby-lang.org/issues/show/3466#note-3
Windows Server 2008. How can I quickly use up RAM so to induce GC in my app. If there is a way to do it without needing Visual Studio or installing a language runtime it would be good.
EDIT: I don't want to have to write an app and then copy it over to the server. I'm looking for a way to do it quickly without writing an app that requires an IDE or installation of a runtime/compiler.
Perhaps a powershell or batch script?...
I don't think using up RAM outside your process is going to necessarily trigger GC.
If I understand your question correctly, you have a program Foo.exe that is written in some unknown language, running on some unknown runtime (are you not allowed to post the details for some reason, or do you just not know?), and you want to try to get that program's runtime to trigger a garbage collection. However, you want to do this by using up RAM outside of foo.exe.
You could do this by creating a simple batch file that just started up a hundred copies of IE or Word or whatever program you want. However, I don't think that will do what you want it to do. If your process has already allocated a certain amount of memory, it won't necessarily give that memory up or trigger GC just because other processes are being started. It may page to disk, or may force other programs to page to disk. But not all Garbage Collectors are alike, so we can't really help without more details. I'm pretty sure some VM's never give back memory once they've allocated it, even after GC.
You could run your program inside a virtual machine such as Virtual Box, where you specify the memory ceiling of the guest operating system.
I'm having trouble imagining a scenario where this would be necessary though. Could you provide more information about the problem?
If you are using java you can specify the max amount of memory using Xmx. Search for JVM memory setting