Matlab: Speed up loop applied to each of 820,000 elements - performance

Question

I have a set of rainfall data, with a value every 15 minutes over many years, giving 820,000 rows.
The aim (eventually) of my code is to create columns which categorise the data which can then be used to extract relevant chunks of data for further analysis.
I am a Matlab novice and would appreciate some help!
The first steps I have got working sufficiently fast. However, some steps are very slow.
I have tried pre-allocating arrays, and using the lowest intX (8 or 16 depending on situation) possible, but other steps are so slow they don't complete.
The slow ones are for loops, but I don't know if they can be vectorised/split into chunks/anything else to speed them up.
I have a variable "rain" which contains a value for every time step/row.
I have created a variable called "state" of 0 if no rain, and 1 if there is rain.
Also a variable called "begin" which has 1 if it is the first row of a storm, and 0 if not.
The first slow loop is to create a "spell" variable - to give each rain storm a number.
% Generate blank column for spell of size (rain) - preallocate
spell = zeros(size(st),1,'int16');
% Start row for analysis
x=1;
% Populate "spell" variable with a storm number in each row of rain, for the storm number it belongs to (storm number calculated by adding up the number of "begin" values up to that point
for i=1:size(state)
if(state(x)==1)
spell(x) = sum(begin(1:x));
end
x=x+1;
end
The next stage is about length of each storm. The first steps are fast enough.
% List of storm numbers
spellnum = unique(spell);
% Length of each spell
spelllength = histc(spell,spellnum);
The last step below (the for loop) is too slow and just crashes.
% Generate blank column for length
length = zeros(size(state),1,'int16');
% Starting row
x = 1;
% For loop to output the total length of the storm for each row of rain within that storm
for i=1:size(state)
for j=1:size(state)
position = find(spell==x);
for k=1:size(state)
length(position) = spelllength(x+1);
end
end
x=x+1;
end
Is it possible to make this more efficient?
Apologies if examples already exist - I'm not sure what the process would be called!
Many thanks in advance.

Answer 1

Mem. allocation/reallocation tips:
try to create the results directly from expression (eventually trimming another, more general result);
if 1. is not possible, try to pre-allocate whenever possible (when you have an upper limit for the result);
if 2. is not possible try to grow cell-arrays rather than massive matrices (because a matrix requires a contiguous memory area)
Type-choice tips:
try to use always double in intermediate results, because is the basic numeric data type in MATLAB; avoiding conversions back and forth;
use other types for intermediate results only if there's a memory constraint that can be alleviated by using a smaller-size type.
Linearisation tips:
fastest linearisation uses matrix-wise or element-wise basic algebraic operations combined with logical indexing.
loops are not that bad starting with MATLAB R2008;
the worst-performing element-wise processing functions are arrayfun, cellfun and structfun with anonymous functions, because anon functions evaluate the slowest;
try not to calculate the same things twice, even if this gives you better linearisation.
First block:
% Just calculate the entire cumulative sum over begin, then
% trim the result. Check if the cumsum doesn't overflow.
spell = cumsum(begin);
spell(state==0) = 0;
Second block:
% The same, not sure how could you speed this up; changed
% the name of variables to my taste, though.
spell_num = unique(spell);
spell_length = histc(spell,spell_num);
Third block:
% Fix the following issues:
% - the most-inner "for" does not make sense because it rewrites
% several times the same thing;
% - the same looping variable "i" is re-used in three nested loops,
% - thename of the standard function "length" is obscured by declaring
% a variable named "length".
for x = 1:numel(spell_num)
storm_selector = (spell==spell_num(x));
storm_length(storm_selector) = spell_length(x+1);
end;

Answer 2

The combination of code I have ended up using is a mixture from #CST_Link and #Sifu. Thank you very much for your help! I don't think Stackoverflow lets me accept two answers, so for clarity by putting it all together, here is the code which everyone's helped me create!
The only slow part is the for loop in block three, but this still runs in a few minutes, which is good enough for me, and infinitely better than my attempt.
First block:
%% Spell
%spell is cumulative sum of begin
spell = cumsum(begin);
%% start row
x=1;
%% Replace all rows of spell with no rain with 0
spell(state==0)=0
Second block (unchanged except better variable names):
%% Spell number = all values of spell
spell_num = unique(spell);
%% Spell length = how many of each value of spell
spell_length = histc(spell,spell_num);
Third block:
%% Generate blank column for spell of size (state)
spell_length2 = zeros(length(state),1);
%%
for x=1:length(state)
position = find(spell==x);
spell_length2(position) = spell_length(x+1);
end

Answer 3

for the first part if i am following what you are doing
i created some data matching your description for testing.
please tell me if i missed something
state=[ 1 0 0 0 0 1 1 1 1 1 0 1 0 0 1 0 1 1 1 1 0];
begin=[ 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0];
spell = zeros(length(state),1,'int16');
%Start row for analysis
x=1;
% Populate "spell" variable with a storm number in each row of rain, for the storm number it belongs to (storm number calculated by adding up the number of "begin" values up to that point
for i=1:length(state)
if(state(x)==1)
spell(x) = sum(begin(1:x));
end
x=x+1;
end
% can be accomplished by simply using cumsum ( no need for extra variables if you are short in memory)
spell2=cumsum(begin);
spell3=spell2.*(state==1);
and the output for both spell and spell3 as shown
[spell.'; spell3]
0 0 0 0 0 1 1 1 1 1 0 2 0 0 2 0 3 3 3 3 0
0 0 0 0 0 1 1 1 1 1 0 2 0 0 2 0 3 3 3 3 0

Answer 4

Why don't you do that instead?
% For loop to output the total length of the storm for each row of rain within that storm
for x=1:size(state)
position = find(spell==x);
length(position) = spelllength(x+1);
end
I replaced the i iterator for x, that removes 2 lines and some computation.
I then proceeded to removed the two nested loops as they were litteraly useless (each loop would output the same thing)
That's already a good start..