I have installed Munin to provide some insight into server performance for a VPS server with some small rails and Sinatra applications. Is there a good resource for reading up on what to look for on the graphs Munin provides. Or a good resource on getting more details on specific measures (Fork rate, Swap in/out) - what they are telling me, what are signals that need to be looked into...
Mainly I am trying to learn about what measures I should pay attention to on the server side as I try to work with some small ruby application for fun.
When you install munin you get a bunch of default plugins which graph system metrics. To begin with, it would be a good idea to keep an eye on load-avg, cpu%, memory, swap in/out.
If you're not sure exactly how munin is calculating a specific metric, you can try reading the source code for the plugin script. Usually system metrics are obtained from the /proc filesystem. On a debian/ubuntu box, munin plugin scripts are installed (via a symlink) under /etc/munin/plugins. You can install your custom plugins by simply dropping them somewhere and symlinking to them from /etc/munin/plugins.
Related
I'm currently developing a web-app, and I alternate between Windows and Mac dev machines for this.
My problem is that pages render extremely slowly on Windows, but it's not my Ruby code running slowly, but rather that static files are getting served slowly.
A typical page takes about 200ms to render and get served in dev (both Mac and Windows are similar here), but it includes about 50 static files (in production it's just 5 to 10, once they get minified and combined, but in dev they're still separate).
Those 50 files take about 1.5 seconds to serve on the Mac, but about 10 seconds on Windows. Which makes it quite tortuous to test things...
I've tried both Webrick and Thin, they are about the same.
Has anybody found this problem and know how to improve this?
I've tried changing the Webrick conf to ":DoNotReverseLookup => true", as suggested in this answer, but it's not helping.
Any help will be greatly appreciated
Thanks!
Daniel
You are running into two existential problems that have plagued Ruby developers for a long time:
Webrick is slow. Always. Just don't even bother.
Ruby is always slower on Windows. Sometimes by orders of magnitude as you've found.
So if you insist on doing development on Windows proper (as opposed to developing only on Linux or developing on a Linux VM running on Windows), then we need to figure out some ways of putting lipstick on a pig.
Some ideas:
Make sure you run the latest version of Ruby.
Try deploying nginx with Thin as shown in this helpful albeit dated tutorial. This will help you get the most out of Thin's multithreading and asynchronicity.
Use Capistrano to deploy to Windows via this also dated GitHub project.
If you do decide you've had enough of developing Rails on an environment it wasn't designed for, you can set up a VM in the manner described here. The author reports significant speedup.
Use an Ubuntu VM inside VirtualBox, it's probably much closer to your deployment environment then mac and windows which means less "but it worked in development" troubles in production.
Also, you will save yourself a lot of time dealing with quirks of different ruby/gems implementations and various levels of headache due to native extensions.
You can:
set up internal network so you can use browser under windows to browse the app running inside the VM
use something like putty to open console sessions to VM
share a Dropbox/Sparkleshare folder with your Ubuntu VM so you always have same code between Windows and Mac box and Ubuntu VM
and this enables you to use your favorite editor under windows/macos to edit files inside the VM
you can use that same VM under Mac too
Ubuntu installation under VirtualBox is fast, easy and well documented, it's pretty much just a wizard. Alternatively, you can try finding a good vagrant recipee (see http://www.vagrantup.com/) or ask around to see if a colleague of yours would be willing to share his/her vbox.
I experienced a performance decrease on development (due to real-time compilation) working with projects with a large number of assets, but I was not on Windows.
I assume the large performance difference may be caused by some inefficient asset compilation under Windows.
I don't any Windows development experience, I haven't been using a Windows machine for a long time, however I registered a noticeable performance increase in parallel asset processing (in development) when I switched to multi-thread servers, specifically Puma. Keep in mind that, in any case, the default Rails webserver (Webrick) is very inefficient.
As Konstantine explained in this answer, there are currently several choices available. You can group them by the processing mode.
Skipping all the background history about Ruby threads, multi-process etc, I'd invite you to try Puma in your machine and see if it improves the load.
Puma works better with Ruby implementations that offer real multi threading, but quoting the official readme
On MRI, there is a Global Interpreter Lock (GIL) that ensures only one thread can be run at a time. But if you're doing a lot of blocking IO (such as HTTP calls to external APIs like Twitter), Puma still improves MRI's throughput by allowing blocking IO to be run concurrently (EventMachine-based servers such as Thin turn off this ability, requiring you to use special libraries).
Puma is designed to provide a simple and high performance request/response pipeline to Rack apps.
Is anybody know a good testing tool that can produce a graph containing the CPU cycle and RAM usage?
What I will do for ex. is I will run an application and while the application is running the testing tool will record CPU cycle and RAM Usage and it will make a graph as an output.
Basically what I'm trying to test is how much heavy load an application put on RAM and CPU.
Thanks in advance.
In case this is Windows the easiest way is probably Performance Monitor (perfmon.exe).
You can configure the counters you are interested in (Such as Processor Time/Commited Bytes/et) and create a Data Collector Set that measures these counters at the desired interval. There are even templates for basic System Performance Report or you can add counters for the particular process you are interested in.
You can schedule the time where you want to execute the sampling and you will be able to see the result using PerfMon or export to a file for further processing.
Video tutorial for the basics: http://www.youtube.com/watch?v=591kfPROYbs
Good Sample where it shows how to monitor SQL:
http://www.brentozar.com/archive/2006/12/dba-101-using-perfmon-for-sql-performance-tuning/
Loadrunner is the best I can think of ; but its very expensive too ! Depending on what you are trying to do, there might be cheaper alternatives.
Any tool which can either hook to the standard Windows or 'NIX system utilities can do this. This has been a defacto feature set on just about every commercial tool for the past 15 years (HP, IBM, Microfocus, etc). Some of the web only commercial tools (but not all) and the hosted services offer this as wekll. For the hosted services you will generally need to punch a hole through your firewall for them to get access to the hosts for monitoring purposes.
On the open source fron this is a totally mixed bag. Some have it, some don't. Some support one platform, but not others (i.e. support Windows, but not 'NIX or vice-versa).
What tools are you using? It is unfortunately common for people to have performance tools in use and not be aware of their existing toolset's monitoring capabilities.
All of the major commercial performance testing tools have this capability, as well as a fair number of the open source ones. The ability to integrate monitor data with response time data is key to the identification of bottlenecks in the system.
If you have a commercial tool and your staff is telling you that it cannot be done then what they are really telling you is that they don't know how to do this with the tool that you have.
It can be done using jmeter, once you install the agent in the target machine you just need to add the perfmon monitor to your test plan.
It will produce 2 result files, the pefmon file and the requests log.
You could also build a plot that compares the resource compsumtion to the load, and througput. The throughput stops increasing when some resource capacity is exceeded. As you can see in the image CPU time increases as the load increases.
JMeter perfmon plugin: http://jmeter-plugins.org/wiki/PerfMon/
I know this is an old thread but I was looking for the same thing today and as I did not found something that was simple to use and produced graphs I made this helper program for apachebench:
https://github.com/juanluisbaptiste/apachebench-graphs
It will run apachebench and plot the results and percentile files using gnuplot.
I hope it helps someone.
I'm probably dreaming here, but am wondering if there's any possibility of completely embedding a minimal CouchDB engine within a Windows application, such that the app can be run without requiring installation (of CouchDB/Erlang) on the user's computer.
I already provide this slimmed down / bundled ability - check here https://github.com/dch/couchdb/downloads and specifically the lean bundle at 16MiB erlang + all couch love here https://github.com/downloads/dch/couchdb/couchdb-1.1.0+COUCHDB-1152_otp_R14B03_lean.7z
Some brief notes on bundling and embedding couchdb on windows at wiki.apache.org/couchdb/Quirks_on_Windows including how to hide the erlang window (erl.exe -detached) at startup.
Ask on CouchDB #user mailing list if you want more info or help while you have a crack at this.
While not a code solution, you could use one of the bundling applications that can embed files and other files into one executable. One example would be BoxedApp.
Why bother. It is so easy to install Erlang on Windows. Just bundle up the whole thing including the erl.exe binary and have your installer unzip it into a folder. The only thing that you would need to change would be the batch files, or better yet, discard them and write your own batch file to start up CouchDb. Also, it is a good idea to use a different port that either the normal Erlang port (or the usual CouchDB port) and maybe even get Erlang to use localhost as its "shortname".
The CouchDB wiki does provide at least a few tips for Integrating CouchDB into your Windows Applications. YMMV, from what I can tell it's more or less just tips on creating a relocatable build. You'll want to likely generate a solid random admin user/password into the local.ini file during the install process and set up proper permissions on all created databases (to protect against any potential cross-site scripting vulnerabilities) in addition to ensuring the socket binding only happens on the default localhost interface.
I'm the administrator for a company intranet and I'd like to start producing videos. However, we have a very small bandwidth tunnel between our locations, and I'd like to avoid hogging it by streaming videos by multiple users.
I'd like to synchronize the files to servers at each of the locations. Then I'd like the browser (or the intranet) to detect which site I'm at. From there, I'd like it to request the video from the closest location.
I've never done this, and was wondering if there is already a solution out there for this. It looks like Hadoop may do this, but I guess I'd like to hear that from someone using it before I commit to learning it.
I don't know how to achieve exactly what you want, but hadoop does not do what you desire.
Hadoop provides an infrastructure to process large amounts of data (like e.g. log file analysis) in a distributed environment (cluster) but the machines in the cluster are usually connected with high speed communication links (may be even on the same server rack).
So I can answer the last part of your question and tell you that hadoop not a good fit for your type of problem. You might still want to learn what hadoop gives you, so that you may be able to use it in another scenario, though.
You might also want to check out serverfault to find an answer to your problem, as it seems you are looking to a more system administrator like answer than a programming one.
Here are good materials to learn about hadoop
I've been looking at ways people test their apps in order decide where to do caching or apply some extra engineering effort, and so far httperf and a simple sesslog have been quite helpful.
What tools and tricks did you apply on your projects?
I use httperf for a high level view of performance.
Rails has a performance script built in, that uses the ruby-prof gem to analyse calls deep within the Rails stack. There is an awesome Railscast on Request Profiling using this technique.
NewRelic have some seriously cool analysis tools that give near real-time data.
They just made it a "Lite" version available for free.
I use jmeter for session-based testing - it allows very fine-grained control over pages you want to hit, parameters to inject, loops to go through, etc. It's great for simulating how many real users your site can handle, rather than just performance testing a set of static urls. You can distribute tests over multiple machines quite easily by loading up the jmeter-server on computers with publicly accessible IP's. I have found some limitations in the number of users/threads any one machine can throw at a server at once (it depends on the test), but jmeter has helped my team improve our apps capacity for users to 6x.
It doesn't have any fancy graphing -- I actually use my own in-house graphing with gruff that can do performance analysis on request time for certain pages and actions.
I'm evaluating a new opensource web page instrumentation and measurement suite called Jiffy. It's not particularly for ruby, it works for all kind of webapps
There's also a Jiffy Firebug Extension for rendering the metrics inside the browser.
I also suggest you look at Browser Mob for load testing.
A colleague of mine has also posted some interesting thoughts on this.