I have a webserver running on a VM and often we have high server load whose root cause has not yet found.
I have sofar tried to dig out some Disk activity, memory usage, swap usage and CPU usage during the incident
So far i have found the following.
High pgscand/s, pgsteal/s %vmeff noticed during the load.
%vmeff is 100 %
03:00:01 PM 1.06 65.63 662.44 0.00 1790.95 0.00 0.00 0.00 0.00
03:10:01 PM 4.39 56.46 1744.16 0.00 2370.19 0.00 221.60 221.60 100.00
The server runs on Redhat 7.5. Webserver is Apache and there are no outside hits to the webserver to cause any load. There are also no cronjobs running.
I would like to know the root cause of the issue
From reading the doc https://github.com/ruby-prof/ruby-prof:
It seems that some patches were originally required for these features,
for allocations:
http://rubyforge.org/tracker/index.php?func=detail&aid=11497&group_id=426&atid=1700
for memory use:
http://rubyforge.org/tracker/index.php?func=detail&aid=17676&group_id=1814&atid=7062
The latter of which claims to have been applied.
This github issue seems to back that up:
https://github.com/ruby-prof/ruby-prof/issues/86
But I have had absolutely no luck getting either of these modes to work in a 2.1.1 install, eg:
require 'ruby-prof'
RubyProf.measure_mode = RubyProf::MEMORY
RubyProf.start
1000.times do
s = "string"
end
RubyProf::FlatPrinter.new(RubyProf.stop).print
Produces:
Thread ID: 4183000
Fiber ID: 8413080
Total: 0.000000
Sort by: self_time
%self total self wait child calls name
NaN 0.000 0.000 0.000 0.000 1 Integer#times
NaN 0.000 0.000 0.000 0.000 1 Global#[No method]
I even went so far as to try install ruby 1.9.3 to do this profiling, but rvm seems unable to find any of the old patches. Is there some way to get this working? or have these features been abandoned?
According to a ruby-prof issue we need a patched version of ruby to use the memory profiler.
https://github.com/ruby-prof/ruby-prof/issues/166
https://github.com/skaes/rvm-patchsets
I am trying to use Ruby's standard lib profiler, and I am not using ruby-prof because this is a jruby project.
Though, I am allways getting something like:
% cumulative self self total
time seconds seconds calls ms/call ms/call name
0.00 0.93 0.00 1 0.00 930.00 #toplevel
The code that generates this is:
require 'profiler'
Profiler__.start_profile
# profiling_result = RubyProf.profile do
result= handle_request
# end
#Profiler__.stop_profile
# printer = RubyProf::FlatPrinter.new(result)
File.open("#{File.dirname(__FILE__)}/profiler/#{Time.now.to_i}.flat","w") do |f|
Profiler__.print_profile f
end
Profiler__.stop_profile
I would strongly recommend using the JRuby built-in profiler, detailed here: http://danlucraft.com/blog/2011/03/built-in-profiler-in-jruby-1.6/
The stdlib profiler uses feature of Ruby that incur a tremendous perf hit (set_trace_func, among others). You'll get better results using the built-in profiler, which avoids such overhead.
Mahoro is a libmagic wrapper. Right now my process for reading in a file is:
filetype = Mahoro.new.file(full_path)
File.open(full_path, get_access_string(filetype)) do |f|
The problem is that Mahoro seems to read the entire file, and not just the header strings. So I get a profiling result like:
%self total self wait child calls name
6.02 0.26 0.26 0.00 0.00 1 Mahoro#file
5.81 4.36 0.25 0.00 4.11 1 Parser#read_from_file
Each are taking .25 seconds, which implies that they are duplicating each other's work. Is there a way to get the file as a string from libmagic? That seems to be the only way to make this process more efficient.
Someone posted something similar here, however I'm curious to which would run faster and save me more memory? I know that Matz as made many enhancements in 1.9 but I also stand behind the decisions the Phusion guys have made as well.
My question is which is the best to run on my VM which only has 265mb of ram?
The 1.8.7 EE would be a safer bet right now. The main problem with Ruby apps is the distinct inability to share memory (the copy-on-write issue) and fixing that is the main aim of EE.
I manage 8 different sites, all running versions of our product running on a mix of Rails, Merb, Rack and Thin on all running on plain old Ruby 1.8.7. For a small Rails application, 256Mb would be ok.
You can see from below that our application is comprised of 6 processes; Rails (2) and Merb (4). The Rails processes (mongrel_rails) are using 104Mb of actual memory each. Our app is reasonably complex with responses of the order of 0.5s so we are looking at being able to handle around 4/5 concurrent users from the 2 Rails processes. Take a look at the shockingly small amount of shared memory to see why EE makes so much sense. I'd expect a much higher shared section with EE.
As they say 'your experience may differ' but there's nothing stopping you from even trying plain old Ruby/Rails and only moving to EE if you need to.
top - 08:57:48 up 128 days, 11:57, 1 user, load average: 0.07, 0.09, 0.09
Tasks: 76 total, 1 running, 75 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.4%us, 0.1%sy, 0.0%ni, 96.2%id, 0.0%wa, 0.0%hi, 0.0%si, 1.3%st
Mem: 1048796k total, 745840k used, 302956k free, 5192k buffers
Swap: 2097144k total, 634636k used, 1462508k free, 124816k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25875 root 20 0 271m 104m 4616 S 0 10.2 141:07.07 mongrel_rails
25872 root 20 0 263m 102m 4648 S 0 10.0 142:11.86 mongrel_rails
21089 root 20 0 192m 84m 2436 S 0 8.3 2:52.03 merb
21088 root 20 0 173m 80m 2436 S 0 7.9 2:51.73 merb
21090 root 20 0 179m 74m 2436 S 0 7.3 2:42.83 merb
21086 root 20 0 113m 34m 1660 S 11 3.4 3752:37 merb
4874 clavis 20 0 122m 31m 3804 S 0 3.1 127:52.87 profile_report
3662 mysql 20 0 368m 22m 3280 S 0 2.2 464:01.81 mysqld
Since Rails 3 will be highly optimized for Ruby 1.9, and Ruby 1.9 has MANY similar optimizations that REE has, Ruby 1.9 will likely be very close in memory usage, and faster than REE.
Yehuda talk about Rails 3:
http://railsonedge.blogspot.com/2009/03/yehuda-katz-talks-about-rails-30.html
http://www.oreillynet.com/pub/e/1338
Tiny Google Group discussion about REE for Ruby 1.9
http://groups.google.com/group/emm-ruby/browse_thread/thread/b5ab0f02c3faac7e#