I might have misunderstood something.. but I expected the running of a code in the editor and in the command window to be the same..
I have the following code in the editor
display(' ');
display('script');
fac = #(n) prod(1:n);
n=20;
display(['- use of anonymous function: fac = #(n) prod(1:n); with n = ' num2str(n)]);
tic; fac(n); toc;
display(['- use of build in MatLab function: factorial(); with n = ' num2str(n)]);
tic; factorial(n); toc;
before I run (is that called compiled or execute?) I quickly type in the same command in the command window. Which gives me this:
So all of a sudden the factorial function in the editor got a boost when it comes to performance.. What just happened?
First, the calculation you are trying to measure is too quick to get anything like an accurate reading. It is way below the measurement noise you get when using tic/toc because of other processes and activity on the system. To get more accurate measures repeat the calculation many times... (I would repeat the calculation so it takes 10-20 seconds).
Second, there is indeed a difference between code typed interactively and in a script. I think the JIT only runs for code run from a script or function m-file, where and not for stuff running in the interpreter (but I am struggling to find a reference for this now - I will add if I find it).
you mean the differences in running time between running in command window and running from script? I think they're caused by everything else also running on your computer, matlab's own memory management and a few other things...
it is explained in greater detail here: http://www.mathworks.com/matlabcentral/newsreader/view_thread/296850
Another method of measuring the time spent on cpu is the function cputime. It has however lower precision. So expand your test case in that it takes a longer time (n=2e7 eg factorial is capped at 171) to see results with cputime.
Conclusion: one swallow does not a summer make
Extend your testing with more cases.
n=200;
tic;
for ii=1:1e4
factorial(n);
end;
toc
Related
I have large loop that contains abs([imaginary]).
It took me a lot of time to complete the program.
I tried multiple other ways to compute abs() such as when C is imaginary, (real(C)^2+imag(C)^2).^.5.
This result same as abs(C).
When I used tic,toc, (real(C)^2+imag(C)^2).^.5 was slightly faster. So I substituted and ran it again.
However profile shows that when I had abs() was much faster.
How can this happen and how can I make abs(C) process faster?
i take it form your comment that you are using large loops, matlab is not that efficent with those, example:
test = randn(10000000,2);
testC = complex(test(:,1),test(:,2));
%%vector
tic
foo = abs(testC);
toc
%%loop
bar = zeros(size(foo));
tic
for i=1:10000000
bar(i) = abs(testC(i));
end
toc
gives you something like
Elapsed time is 0.106635 seconds.
Elapsed time is 0.928885 seconds.
Thats why i would recommend to calculate abs() outside the loop. If replacing the loop in total is no option you can do so only in Parts. For exsample you could use your loop until you got all your complex numbers, end the loop, calc abs() then start a new loop with those results. Also if each iteration of your loop is independat of other iteration results you might want to look in parfor as an replacement for for-loops
The algorithm repeats the same thing again-and-again. I expected to get the same time in each trial but I got very unexpected times for the four identical trials
in which I expected the curves to be identical but they act totally differently. The reason is probably in the tic/toc precision.
What kind of profiling/timing tools should I use in Matlab?
What am I doing wrong in the below code? How reliable is the tic/toc profiling?
Anyway to guarantee consistent results?
Algorithm
A=[];
for ii=1:25
tic;
timerval=tic;
AlgoCalculatesTheSameThing();
tElapsed=toc(timerval);
A=[A,tElapsed];
end
You should try timeit.
Have a look at this related question:
How to benchmark Matlab processes?
A snippet from Sam Roberts answer to the other question:
It handles many subtle issues related to benchmarking MATLAB code for you, such as:
ensuring that JIT compilation is used by wrapping the benchmarked code in a function
warming up the code
running the code several times and averaging
Have a look at this question for discussion regarding warm up:
Why does Matlab run faster after a script is "warmed up"?
Update:
Since timeit was first submitted at the fileexchange, the source code is available here and can be studied and analyzed (as opposed to most other MATLAB functions).
From the header of timeit.m:
% TIMEIT handles automatically the usual benchmarking procedures of "warming
% up" F, figuring out how many times to repeat F in a timing loop, etc.
% TIMEIT also compensates for the estimated time-measurement overhead
% associated with tic/toc and with calling function handles. TIMEIT returns
% the median of several repeated measurements.
You can go through the function step-by-step. The comments are very good and descriptive in my opinion. It is of course possible that Mathworks has changed parts of the code, but the overall functionality is there.
For instance, to account for the time it takes to run tic/toc:
function t = tictocTimeExperiment
% Call tic/toc 100 times and return the average time required.
It is later substracted from the total time.
The following is said regarding number of computations:
function t = roughEstimate(f, num_f_outputs)
% Return rough estimate of time required for one execution of
% f(). Basic warmups are done, but no fancy looping, medians,
% etc.
This rough estimate is used to determine how many times the computations should run.
If you want to change the number of computation times, you can modify the timeit function yourself, as it is available. I would recommend you to save it as my_timeit, or something else, so that you avoid overwriting the built-in version.
Qualitatively there are large differences between the same runs. I did the same four trials as in the question and tested them with the methods suggested so far and I created my own version of the timeit timeitH because the timeit has too large standard deviation between different trials. The timeitH returns far more robust results to other methods because it warm ups the code similarly to the timeit and then it has increased the amount of outer loops in the original timeit from 11 to 50.
The below has the four trials done with the three different methods. The closer the curves are to each other, the better.
TimeitH: results pretty good!
Some observations.
timeit: result smoothed but bumps
tic/toc: easy to adjust for larger cases to get the standard deviation smaller in computation times but no warming up
timeitH: download the code and change 60th line to num_outer_iterations = 50; to get smoother results
In summarum
I think the timeitH is the best candidate here, yet only tested in evaluating sparse polynomials. The timeit and tic/toc like 500 times do not result into robust results.
Timeit
500 trials and average with tic/toc
Algorithm for the 500 trials with tic/toc
for ii=1:25
numTrials = 500;
tic;
for ii=1:numTrials
AlgoCalculatesTheSameThing();
end
tTotal = toc;
tElapsed = tTotal/numTrials;
A=[A,tElapsed];
end
Is the time for AlgoCalculatesTheSameThing() relatively short (fractions of sec or a few sec) or long (multi-minutes or hours)? If the former I would suggest doing it more like this: move your timing functions outside your loop, then compute averages:
A=[];
numTrials = 25;
tic;
for ii=1:numTrials
AlgoCalculatesTheSameThing();
end
tTotal = toc;
tAvg = tTotal/numTrials;
If the event is short enough (fraction of sec) then you should also increase the value of numTrials to 100s or even 1000s.
You have to consider that with any timing function there will be error bars (like in any other measurement). If the event your timing is short enough, the uncertainties in your measurement can be relatively big, keeping in mind that the resolution of tic and toc also has some finite value.
More discussion on the accuracy of tic and toc can be found here.
You need to work out these uncertainties for your specific application, so do experiments: perform averages over a number of trials and then compute the standard deviation to get a sense of the "scatter" or uncertainly in your results.
Searching for an idea how to avoid using loop in my Matlab code, I found following comments under one question on SE:
The statement "for loops are slow in Matlab" is no longer generally true since Matlab...euhm, R2008a?
and
Have you tried to benchmark a for loop vs what you already have? sometimes it is faster than vectorized code...
So I would like to ask, is there commonly used way to test the speed of a process in Matlab? Can user see somewhere how much time the process takes or the only way is to extend the processes for several minutes in order to compare the times between each other?
The best tool for testing the performance of MATLAB code is Steve Eddins' timeit function, available here from the MATLAB Central File Exchange.
It handles many subtle issues related to benchmarking MATLAB code for you, such as:
ensuring that JIT compilation is used by wrapping the benchmarked code in a function
warming up the code
running the code several times and averaging
Update: As of release R2013b, timeit is part of core MATLAB.
Update: As of release R2016a, MATLAB also includes a performance testing framework that handles the above issues for you in a similar way to timeit.
You can use the profiler to assess how much time your functions, and the blocks of code within them, are taking.
>> profile on; % Starts the profiler
>> myfunctiontorun( ); % This can be a function, script or block of code
>> profile viewer; % Opens the viewer showing you how much time everything took
Viewer also clears the current profile data for next time.
Bear in mind, profile does tend to slow execution a bit, but I believe it does so in a uniform way across everything.
Obviously if your function is very quick, you might find you don't get reliable results so if you can run it many times or extend the computation that would improve matters.
If it's really simple stuff you're testing, you can also just time it using tic and toc:
>> tic; % Start the timer
>> myfunctionname( );
>> toc; % End the timer and display elapsed time
Also if you want multiple timers, you can assign them to variables:
>> mytimer = tic;
>> myfunctionname( );
>> toc(mytimer);
Finally, if you want to store the elapsed time instead of display it:
>> myresult = toc;
I think that I am right to state that many of us time Matlab by wrapping the block of code we're interested in between tic and toc. Furthermore, we take care to ensure that the total time is of the order of 10s of seconds (rather than 1s of seconds or 100s of seconds) and repeat it 3 - 5 times and take some measure of central tendency (such as the mean) and draw our conclusions from that.
If the piece of code takes less than, say 10s, then repeat it as many times as necessary to bring it into the range, being careful to avoid any impact of one iteration on the next. And if the code naturally takes 100s of seconds or longer, either spend longer on the testing or try it with artificially small input data to run more quickly.
In my experience it's not necessary to run programs for minutes to get data on average run time with acceptably low variance. If I run a program 5 times and one (or two) of the results is wildly different from the mean I'll re-run it.
Of course, if the code has any features which make its run time non-deterministic then it's a different matter.
I am still somewhat new to parallel computing in Matlab. I have used OpenMP in C successfully, but could not get better performance in Matlab.
First, since I'm machine at a university that I am new to, I verified that the machine I am on has the Parallel Computing Toolbox by typing ver in the command prompt and it displayed: Parallel Computing Toolbox Version 5.2 (R2011b). Note that the machine has 4 cores
I tried simple examples of using parfor vs. for, but for always won, though this might be because of the overhead cost. I was doing simple things like the example here: MATLAB parfor is slower than for -- what is wrong?
Before trying to apply parfor to my bigger more complicated program (I need to compute 500 evaluations of a function and each evaluation takes about a minute, so parallelizing will help here), I would very much like to see a concrete example where parfor beats for. . Examples are abundant for OpenMP, but did not find a simple example that I can copy and paste that shows parfor is better than for
I use the following code (once per Matlab session) in order to use parfor:
pools = matlabpool('size');
cpus = feature('numCores');
if pools ~= (cpus - 1)
if pools > 0
matlabpool('close');
end
matlabpool('open', cpus - 1);
end
This leaves 1 core for other processes.
Note, the feature() command is undocumented.
There is an example of improved performance from parfor on Loren Shure's MATLAB blog.
Her example is simply computing the rank of a magic square matrix:
function ranks = parMagic(n)
ranks = zeros(1,n);
parfor (ind = 1:n)
ranks(ind) = rank(magic(ind)); % last index could be ind,not n-ind+1
end
Serg describes how to "enable" parallel functionality.
Here is a very simple cut and paste example to test it with as requested. Simply copy and paste the follwing into an mfile and run it.
function parfortest()
enable_parallel;
pause on
tic;
N=500;
for i=1:N
sequential_answer=slow_fun(i);
end
sequential_time=toc
tic;
parfor i=1:N
sequential_answer=slow_fun(i);
end
parallel_time=toc
end
function result=slow_fun(x)
pause(0.001);
result=x;
end
If you have run the code to enable parallel as shown in the answer by Serg you should get a pretty obvious improvement in performance.
So, I've recently started using Matlab's built-in profiler on a regular basis, and I've noticed that while its usually great at showing which lines are taking up the most time, sometimes it'll tell me a large chunk of time is being used on the end statement of a for loop.
Now, seeing as such a line is just used for denoting the end of the loop, I can't imagine how it could use anything other than a trivial amount of processing.
I've seen a specific version of this question asked on matlab central, but a consensus didn't seem to be reached.
EDIT: Here's a minimal example of this problem:
for i =1:1000
x = 1;
x = [x 1];
% clear x;
end
Even if you uncomment the clear, the end line still takes up a lot of computation (about 20%), and the clear actually increases the absolute amount of computation performed by the end line.
When I've seen this in my code, it's been the deallocation of large temporaries created in the loop. Each new variable created in the loop is deallocated at the end.