Is it possible to profile a Ruby application to see how much it interacts with the file system?
Background: In the past I've written code that reads files within a loop when it only needs to do so once. I'd like to make sure that I eliminate all such code.
There are already perfectly capable programs for this purpose out there, that you don't need to duplicate. I don't think you should complicate your program with special logic checking for a relatively obscure programming error (at least, I've never accidentally committed the error you describe). In such cases, the solution is to check a performance characteristic from outside the program. Assuming you are on Linux, I would turn to a test/spec that exercises your code and watches iostat (or similar) in a seperate thread.
Sure, you can simply require 'profile at the top of a script:
# myscript.rb
require 'profile'
Dir.entries(".").each { |e| puts e}
$ ruby myscript.rb
(list of filenames...)
% cumulative self self total
time seconds seconds calls ms/call ms/call name
0.00 0.00 0.00 1 0.00 0.00 Dir#open
0.00 0.00 0.00 1 0.00 0.00 Dir#each
0.00 0.00 0.00 1 0.00 0.00 Enumerable.to_a
0.00 0.00 0.00 1 0.00 0.00 Dir#entries
0.00 0.00 0.00 56 0.00 0.00 IO#write
0.00 0.00 0.00 28 0.00 0.00 IO#puts
0.00 0.00 0.00 28 0.00 0.00 Kernel.puts
0.00 0.00 0.00 1 0.00 0.00 Array#each
0.00 0.01 0.00 1 0.00 10.00 #toplevel
Or you can just pass in an option on the command line:
$ ruby -r profile myscript.rb
If you want finer control over what to profile, take a look at the ruby-prof library.
I'm a sucker for brute force.
File.open(path, 'r') do |file|
...
end
File.mv(path, path + '.hidden') # Temporary
If the code tries to open the file twice, it won't find it the second time. After the test, you can reverse the rename with a shell one-liner. In Bash (and probably in other *nix shells as well):
for i in `ls *.hidden` ; do mv $i ${i%.hidden} ; done
Related
I'm running a KornShell script which originally has 61 input arguments:
./runOS.ksh 2.8409 24 40 0.350 0.62917 8 1 2 1.00000 4.00000 0.50000 0.00 1 1 4900.00 1.500 -0.00800 1.500 -0.00800 1 100.00000 20.00000 4 1.0 0.0 0.0 0.0 1 90 2 0.10000 0.10000 0.10000 1.500 -0.008 3.00000 0.34744 1.500 -0.008 1.500 -0.008 0.15000 0.21715 1.500 -0.008 0.00000 1 1.334 0 0.243 0.073 0.642 0.0229 38.0 0.03071 2 0 15 -1 20 1
I only vary 6 of them. Would it make a difference in performance if I fixed the remaining 55 arguments inside the script and just call the variable ones, say:
./runOS.ksh 2.8409 24 40 0.350 0.62917 8
If anyone has a quick/general answer to this, it will be highly appreciated, since it might take me a long time to fix the 55 extra arguments inside the script and I'm afraid it won't change anything.
There's no performance impact, as you're asking, but I see other threads:
What is the commandline limitation for your system? You mention 61 input parameters, some of them having a length of 8 characters. If the number of input parameters increases, you might have problems with the maximum command length.
Are you performing 440 million scripts? That's too much, far too much. You need to consider why you're doing this: you mention needing to wait ±153 days for their execution to finish, which is far too much (and unpredictable).
I'm currently trying to benchmark a bash script in 4 different versions. Each one does a giant rsync job and it usually takes a very long time to finish. There are many steps in the bash script which involves setting up and tearing down the environment to rsync to.
However, when I ran strace on the bash scripts, I get surprisingly short results, which leads me to believe that strace is not actually tracing the time waiting for a command like rsync(which might be spawned in a subshell and is completely not recorded by rsync), or, it's waking up intermittently and sleep for another amount of time of which strace is not counting. Here's a snippet:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.98 12.972555 120116 108 52 wait4
0.01 0.000751 13 56 clone
0.00 0.000380 1 553 rt_sigprocmask
0.00 0.000303 2 197 85 stat
0.00 0.000274 2 134 read
0.00 0.000223 19 12 open
0.00 0.000190 48 4 getdents
0.00 0.000110 1 82 8 close
0.00 0.000110 1 153 rt_sigaction
0.00 0.000084 1 61 getegid
0.00 0.000074 4 19 write
So what tools can I use that are similar to strace, OR, maybe I'm missing some type of recursive flag in strace to find out correctly where my bash script is waiting on?
I would like something along the lines of:
% time command
------ --------
... rsync
... ls
Any suggestions would be appreciated. Thank you!
I implemented some code, which runs in a loop:
loop do
..
end
In that loop, I handle keypresses with Curses library. If I press N and entered something, I start a new Thread, which counts time( loop do .. end again)
The question is, why loop or while true causes 100% cpu load on one of the cpu cores? Is the problem actaully in loop?
Is there a way to do infinite loop with lower cpu consumption in ruby?
The full sources available here
UPD - Strace
$ strace -c -p 5480
Process 5480 attached - interrupt to quit
^CProcess 5480 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
51.52 0.002188 0 142842 ioctl
24.21 0.001028 0 71421 select
14.22 0.000604 0 47614 gettimeofday
10.05 0.000427 0 47614 rt_sigaction
0.00 0.000000 0 25 write
0.00 0.000000 0 16 futex
------ ----------- ----------- --------- --------- ----------------
100.00 0.004247 309532 total
After some thinking and suggestions from user2246674 I managed to resolve the issue. It was not inside the threads, it was the main loop.
I had such code inside the main loop:
c = Curses.getch
unless c.nil?
# input handling
After adding sleep 1 to else problem was resolved. It does nothing when there's no input from Curses, then checks again in one second, and this stops it from actively polling STDIN and generating high CPU load
I just started learning scala. Is there any way to find CPU time and real time and the cores used by the program when using actor model??
Thanks in advance.
You may use a profiler such as VisualVM or more adhoc and pleasant solution: Typesafe console.
how about using the unix system time function ?
time scalac HelloWorld.scala
If you are running in Linux (specifically here I'm describing Ubuntu) mpstat is a useful program.
You can install it using the following command:
sudo apt-get install sysstat
Once installed you can run a bash script to record CPU info. In this case I'm logging 10 times a second. This will block until you press Control-C to kill the logging. (Also, consider removing the sleep 0.1; for better data.)
while x=0; do mpstat -P ALL >> cpu_log.txt; sleep 0.1; done
In another terminal you can fire off your program (or ANY program) and see track the performance data. The output looks like this:
03:21:08 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
03:21:08 PM all 0.37 0.00 0.33 0.50 0.00 0.02 0.00 0.00 98.78
03:21:08 PM 0 0.51 0.00 0.45 0.57 0.00 0.03 0.00 0.00 98.43
03:21:08 PM 1 0.29 0.00 0.26 0.45 0.00 0.01 0.00 0.00 99.00
I'm suffering performance issues with my application, and one of my suspects is excessive IO.
iostat shows rate of 10K blocks written per second. How can I tell if this is a lot or not? How can I know the limit of the specific machine and disk?
Edit:Following Elliot's request:
iostat output:
avg-cpu: %user %nice %system %iowait %steal %idle
16.39 0.00 0.52 11.43 0.00 71.66
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
cciss/c0d0 315.20 0.00 10341.80 0 51709
uptime:
2:08am up 17 days 17:26, 5 users, load average: 9.13, 9.32, 8.73
top:
top - 02:10:02 up 17 days, 17:27, 5 users, load average: 8.89, 9.18, 8.72
Tasks: 202 total, 2 running, 200 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.9%us, 0.7%sy, 0.0%ni, 90.5%id, 2.9%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 96556M total, 15930M used, 80626M free, 221M buffers
Swap: 196615M total, 93M used, 196522M free, 2061M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20027 root 16 0 10.5g 9.8g 12m S 74 10.4 2407:55 /usr/intel/pkgs/java/1.6.0.31-64/jre//bin/java -server -X
Thanks
I can tell you from experience that's a very high block write rate for most systems. However, your system could be perfectly capable of handling that--depends on what kind of hardware you have. What's important is your server load figure and the iowait percentage. If your server load is high (i.e., higher than the number of cores on your system) and your load largely consists of iowait, then you have a problem.
Could you share with us the full output of iostat, uptime, and a snapshot of top -c output while your application is running?
Perspective:
If it's spinning disk, that's a high value.
If it's an SSD or a SAN with a write cache, that's reasonable.
Use iostat -x for wide and extended metrics:
[xxx#xxxxxxxxxx]$ iostat -x
Linux 2.6.32-358.6.2.el6.x86_64 (boxname.goes.here) 12/12/2013 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.57 0.00 0.21 0.00 0.00 99.21
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.06 28.38 0.04 3.99 0.99 259.15 64.58 0.01 2.11 0.55 0.22
The %util is your friend. If you look at iostat.c (see it at: http://code.google.com/p/tester-higkoo/source/browse/trunk/Tools/iostat/iostat.c) you can see it calculates this percentage by looking at the amount of time (in processor ticks) spent doing IO versus the total number of ticks that have passed. In other words, the PERCENTAGE-UTIL is the percent of time the IO was in a busy state.