When ruby runs out of memory for it's internal allocations, it raises NoMemoryError.
We can catch this, and even do simple operations, but it seems that if we try to continue on, even after garbage collection, we will eventually hit a fatal memory allocation problem, which on my Linux box looks like:
[FATAL] failed to allocate memory
I suspect at this point this is my OS complaining about ruby running out of memory, instead of ruby catching it.
What I don't understand is that there are many cases where we should be able to continue after we rescue a NoMemoryError (and hence give up on the allocation that caused the failure) but it seems that ruby never cleans up the mess. A simple example is to cause (and rescue) a NoMemoryError by shifting too big of a number. For example:
begin
a = 1<<10000000000000
puts "Calculated answer somehow"
rescue NoMemoryError => e
puts "RESCUE: out of memory"
end
puts "Finished first begin/rescue/end"
GC.start # This is in vain
begin
a = 1<<10000000000000
puts "Calculated answer somehow"
rescue NoMemoryError => e
puts "RESCUE: out of memory"
end
puts "Finished second begin/rescue/end"
We catch the first one, which it seems should regain the memory from what I presume to be a BigNum overflow, but that memory never comes back, not even courtesy the garbage collection. So the next time we try to start allocating a bunch of memory (in this case with the same problem) we actually hit a fatal error. The full output:
RESCUE: out of memory
Finished begin/rescue/end
[FATAL] failed to allocate memory
Is there some way to fixup the state after Ruby has hit NoMemoryError if we rescue (and hence abort) the operation that took too much memory?
It feels like this might be a problem with the Garbage Collector, possibly not properly marking memory as no longer in use when we do a rescue, which makes me think there might be a GC bug that this is revealing.
Related
I wrote an algorithm inspired by the merge part of the merge sort.
def self.merge(arr)
if arr.length == 1
return arr
end
groups = []
(0...-(-arr.length/2)).each do |i|
groups << []
if !arr[2*i+1].nil?
arr[2*i].each do |cal1|
arr[2*i+1].each do |cal2|
mergecal = func(cal1,cal2)
if mergecal
groups[i] << mergecal
else
mergecal = nil
end
end
end
else
groups[i] = arr[2*i]
end
end
arr = nil
return merge(groups)
end
After the page using this algorithm is rendered, Task Manager reported around 500MB of RAM usage. Then by refreshing the same page again, memory usage have now reached 1GB. I tried adding GC.start(full_mark: true) to the controller just after the function call, but nothing seems to have changed. I'm not sure whether the memory leak has to code with my code or Ruby itself.
Ruby garbage collection doesn't immediately reduce the amount of memory your ruby program has allocated. Memory allocation is expensive so even if the objects you create are immediately collected by GC the memory is slowly released back to the OS. If you think this function has a memory leak you should try running it in a non-rails process where you have more control over object lifecycles. You can use GC.stat to get information about the number of live and free objects before and after you run the GC. It's also worth reading up on how ruby GC works I like this article.
I must be missing something but every single application I write in Ruby seems like leaking some memory. I use Ruby MRI 2.3 but I see the same behaviour with other versions.
Whenever I write a test application that does something inside a loop it is slowly leaking memory.
while true
#do something
sleep 0.1
end
For instance, I can write to array and then clean it in a loop, or just send http post request.
Here is just one example, but I have many examples like this:
require 'net/http'
require 'json'
require 'openssl'
class Tester
def send_http some_json
begin
#uri = URI('SERVER_URL')
#http = Net::HTTP.new(#uri.host, #uri.port)
#http.use_ssl = true
#http.keep_alive_timeout = 10
#http.verify_mode = OpenSSL::SSL::VERIFY_NONE
#http.read_timeout = 30
#req = Net::HTTP::Post.new(#uri.path, 'Content-Type' => 'application/json')
#req.body = some_json.to_json
res = #http.request(#req)
rescue Exception => e
puts e.message
puts e.backtrace.inspect
end
end
def run
while true
some_json = {"name": "My name"}
send_http(some_json)
sleep 0.1
end
end
end
Tester.new.run
The leak I see is very small, it can be 0.5 mb every hour.
I ran the code with MemoryProfiler and with GC::Profiler.enable and it shows that I have no leaks. So it must be 2 options:
There is a memory leak in C code. This might be possible but I don't use any external gems so I find it hard to believe that Ruby is leaking.
There is no memory leak and this is some sort of Ruby memory management mechanism. The thing is that I can defiantly see the memory growing. Until when will it grow? How much do I need to wait to know if it is a leak or now?
The same code runs perfectly fine with JRuby without any leaks.
I was amazed reading a post:
stack overlflow
from Joe Edgar:
Ruby’s history is mostly as a command line tool for text processing
and therefore it values quick startup and a small memory footprint. It
was not designed for long-running daemon/server processes
If what is written there is true and Ruby doesn't release memory back to OS then... We will always have a leak, right?
For instance:
Ruby asks for memory from OS.
OS provides the memory to Ruby.
Ruby frees the memory but GC still didn't run.
Ruby asks for more memory from OS.
OS provide more memory to Ruby.
Ruby runs GC but it is too late as Ruby already asked twice.
And so on and on.
What am I missing here?
Look Into GC Compaction and (Un)frozen String Literals
"Identical" Strings Aren't Necessarily Identical
Prior to Ruby 2.7.0, mainline Ruby didn't have compacting garbage collection. While I don't fully understand all the internals, the gist is that certain objects couldn't be garbage collected. Since you're using Ruby 2.3, that's something to keep in mind as you work on your memory allocation issues. Other non-YARV VMs may handle their internals differently, which is why you might see variation when using alternative engines like JRuby.
Even with Ruby 3.0.0-preview2, String literals aren't frozen by default, so your current implementation is creating a new String object with a unique object ID every tenth of a second. Consider the following:
3.times.map { 'foo'.__id__ }
#=> [240, 260, 280]
Even though the String objects seem identical, Ruby is actually allocating each one as a unique object in memory. Because a loop iteration is not a scope gate, those String objects can't be collected or compacted by YARV.
Enable Frozen String Literals by Default
You may have other issues as well, but it seems likely that your largest issue is keeping all of those String literals in scope indefinitely within your endless while-loop. You may be able to resolve your garbage collection problem (it's not a memory leak) by using frozen String literals instead. Consider the following:
# run irb with universally-frozen string literals
RUBYOPT="--enable-frozen-string-literal" irb
3.times.map { 'foo'.__id__ }
#=> [240, 240, 240]
You can solve this within your code in other ways as well, but reducing the number of String literals that remain in scope seems like a very sensible place to start.
I have code that is running in a manner similar to the sample code below. There are two threads that loop at certain time intervals. The first thread sets a flag and depending on the value of this flag the second thread prints out a result. My question is in a situation like this where only one thread is changing the value of the resource (#flag), and the second thread is only accessing its value but not changing it, is a mutex lock required? Any explanations?
Class Sample
def initialize
#flag=""
#wait_interval1 = 20
#wait_interval2 = 5
end
def thread1(x)
Thread.start do
loop do
if x.is_a?(String)
#flag = 0
else
#flag = 1
sleep #wait_interval
end
end
end
def thread2(y)
Thread.start do
loop do
if #flag == 0
if y.start_with?("a")
puts "yes"
else
puts "no"
end
end
end
end
end
end
As a general rule, the mutex lock is required (or better yet, a read/write lock so multiple reads can run in parallel and the exclusive lock's only needed when changing the value).
It's possible to avoid needing the lock if you can guarantee that the underlying accesses (both the read and the write) are atomic (they happen as one uninterruptible action so it's not possible for two to overlap). On modern multi-core and multi-processor hardware it's difficult to guarantee that, and when you add in virtualization and semi-interpreted languages like Ruby it's all but impossible. Don't be fooled into thinking that being 99.999% certain there won't be an overlap is enough, that just means you can expect an error due to lack of locking once every 100,000 iterations which translates to several times a second for your code and probably at least once every couple of seconds for the kind of code you'd see in a real application. That's why it's advisable to follow the general rule and not worry about when it's safe to break it until you've exhausted every other option for getting acceptable performance and shown through profiling that it's acquiring/releasing that lock that's the bottleneck.
I have code:
Process.spawn(RbConfig.ruby, "a ruby file", "arg")
and I wait and check its status by:
Process.wait
$?.success?
Most of the time, it works well. But sometimes, $?.success? is false and $?.to_i is 0x0100. It seems the failed process didn't get a chance to run any code before 0x0100 was returned (I didn't send any signal to the process). I wonder the meaning of 0x0100. I further want to know if Ruby's spawn may fail when the command is all right. Could anyone help?
Here is a quote from the Process::Status class documentation:
Posix systems record information on processes using a 16-bit integer. The lower bits record the process status (stopped, exited, signaled) and the upper bits possibly contain additional information (for example the program's return code in the case of exited processes). Pre Ruby 1.8, these bits were exposed directly to the Ruby program. Ruby now encapsulates these in a Process::Status object. To maximize compatibility, however, these objects retain a bit-oriented interface. In the descriptions that follow, when we talk about the integer value of stat, we're referring to this 16 bit value.
The method Process::Status#to_i returns this stat as a Fixnum.
OK, finally I got the answer: when a ruby process throws an uncaught exception, the process' exit code will be 0x0100. This is from my observation on Ubuntu 14.04 and Ruby 2.2. For example: there's ruby file a.rb, and in another file, say src.rb, there's a code snippet:
Process.spawn(RbConfig.ruby, "a.rb", "arg")
Process.wait
If a.rb throws an uncaught exception, then $?.to_i will be 0x0100. What's more, I also observed that a.rb sometimes didn't get executed before its process failed with 0x0100. So I guess it may have something to do with the Ruby interpreter since I'm sure a.rb is OK.
Anyway, there' no official document mentioning the exact behavior. So my experience is for your reference.
I have been using Ruby for a while now and I find, for bigger projects, it can take up a fair amount of memory. What are some best practices for reducing memory usage in Ruby?
Please, let each answer have one "best practice" and let the community vote it up.
When working with huge arrays of ActiveRecord objects be very careful... When processing those objects in a loop if on each iteration you are loading their related objects using ActiveRecord's has_many, belongs_to, etc. - the memory usage grows a lot because each object that belongs to an array grows...
The following technique helped us a lot (simplified example):
students.each do |student|
cloned_student = student.clone
...
cloned_student.books.detect {...}
ca_teachers = cloned_student.teachers.detect {|teacher| teacher.address.state == 'CA'}
ca_teachers.blah_blah
...
# Not sure if the following is necessary, but we have it just in case...
cloned_student = nil
end
In the code above "cloned_student" is the object that grows, but since it is "nullified" at the end of each iteration this is not a problem for huge array of students. If we didn't do "clone", the loop variable "student" would have grown, but since it belongs to an array - the memory used by it is never released as long as array object exists.
Different approach works too:
students.each do |student|
loop_student = Student.find(student.id) # just re-find the record into local variable.
...
loop_student.books.detect {...}
ca_teachers = loop_student.teachers.detect {|teacher| teacher.address.state == 'CA'}
ca_teachers.blah_blah
...
end
In our production environment we had a background process that failed to finish once because 8Gb of RAM wasn't enough for it. After this small change it uses less than 1Gb to process the same amount of data...
Don't abuse symbols.
Each time you create a symbol, ruby puts an entry in it's symbol table. The symbol table is a global hash which never gets emptied.
This is not technically a memory leak, but it behaves like one. Symbols don't take up much memory so you don't need to be too paranoid, but it pays to be aware of this.
A general guideline: If you've actually typed the symbol in code, it's fine (you only have a finite amount of code after all), but don't call to_sym on dynamically generated or user-input strings, as this opens the door to a potentially ever-increasing number
Don't do this:
def method(x)
x.split( doesn't matter what the args are )
end
or this:
def method(x)
x.gsub( doesn't matter what the args are )
end
Both will permanently leak memory in ruby 1.8.5 and 1.8.6. (not sure about 1.8.7 as I haven't tried it, but I really hope it's fixed.) The workaround is stupid and involves creating a local variable. You don't have to use the local, just create one...
Things like this are why I have lots of love for the ruby language, but no respect for MRI
Beware of C extensions which allocate large chunks of memory themselves.
As an example, when you load an image using RMagick, the entire bitmap gets loaded into memory inside the ruby process. This may be 30 meg or so depending on the size of the image.
However, most of this memory has been allocated by RMagick itself. All ruby knows about is a wrapper object, which is tiny(1).
Ruby only thinks it's holding onto a tiny amount of memory, so it won't bother running the GC. In actual fact it's holding onto 30 meg.
If you loop over a say 10 images, you can run yourself out of memory really fast.
The preferred solution is to manually tell the C library to clean up the memory itself - RMagick has a destroy! method which does this. If your library doesn't however, you may need to forcibly run the GC yourself, even though this is generally discouraged.
(1): Ruby C extensions have callbacks which will get run when the ruby runtime decides to free them, so the memory will eventually be successfully freed at some point, just perhaps not soon enough.
Measure and detect which parts of your code are creating objects that cause memory usage to go up. Improve and modify your code then measure again. Sometimes, you're using gems or libraries that use up a lot of memory and creating a lot of objects as well.
There are many tools out there such as busy-administrator that allow you to check the memory size of objects (including those inside hashes and arrays).
$ gem install busy-administrator
Example # 1: MemorySize.of
require 'busy-administrator'
data = BusyAdministrator::ExampleGenerator.generate_string_with_specified_memory_size(10.mebibytes)
puts BusyAdministrator::MemorySize.of(data)
# => 10 MiB
Example # 2: MemoryUtils.profile
Code
require 'busy-administrator'
results = BusyAdministrator::MemoryUtils.profile(gc_enabled: false) do |analyzer|
BusyAdministrator::ExampleGenerator.generate_string_with_specified_memory_size(10.mebibytes)
end
BusyAdministrator::Display.debug(results)
Output:
{
memory_usage:
{
before: 12 MiB
after: 22 MiB
diff: 10 MiB
}
total_time: 0.406452
gc:
{
count: 0
enabled: false
}
specific:
{
}
object_count: 151
general:
{
String: 10 MiB
Hash: 8 KiB
BusyAdministrator::MemorySize: 0 Bytes
Process::Status: 0 Bytes
IO: 432 Bytes
Array: 326 KiB
Proc: 72 Bytes
RubyVM::Env: 96 Bytes
Time: 176 Bytes
Enumerator: 80 Bytes
}
}
You can also try ruby-prof and memory_profiler. It is better if you test and experiment different versions of your code so you can measure the memory usage and performance of each version. This will allow you to check if your optimization really worked or not. You usually use these tools in development / testing mode and turn them off in production.