i am looking to speed up my code. I have tried looking into vectorization and apply etc. But i am not sure on how to implement that on my piece of code. Hopefully someone can help me.
for j in df.col1.unique(): #this is the big problem
temp = df.col2[df.col1 == j]
for k in range(1,len(temp)+1):
colName = 'col2' + str(k)
value = temp.iloc[k-1]
df.loc[df.col1 == j, [colName]] = value
If there are any questions about the code please let me know.
The current code has a quadratic complexity due to the expression like df.col1 == j and df.col1 == j iterating over the whole dataframe executed likely a lot of time. The solution is to use a groupby. Here is an untested example:
groupedDf = df.groupby('col1')
for j,locDf in groupedDf:
temp = locDf.col2
for k in range(len(temp)):
colName = f'col2{k+1}'
value = temp.iloc[k]
df.loc[groupedDf.indices[j], [colName]] = value
It should be even faster to assign all the columns using. Here is another untested example:
groupedDf = df.groupby('col1')
for j,locDf in groupedDf:
temp = locDf.col2
colNames = [f'col2{k+1}' for k in range(len(temp))]
df.loc[groupedDf.indices[j], colNames ] = temp
Related
So if every variable has a value assigned to it ( I mean value is an attribute of variable) . How do i most efficiently pick out the variable that has the max or min value ?
Can you give an example in Python please ?
So this would be my approach in Python :
domains = []
for var in variables :
domains.append(var.value)
min= min(domains)
for var in variables :
if var.value == min :
return var
domains.index(min(domains))
You may say that is not efficient but asymptotically you can't do any better when your list is not sorted.
A general sketch would be something like this (improvements would be providing a way of extracting the wanted value, once you have a list, the actual element can easily be retrieved using the index):
def MinAndIndex(l):
minval = l[0]
min_ix = 0
cnt = 0
for e in l:
if e < minval:
minval = e
min_ix = cnt
cnt += 1
return minval, min_ix
Here is my little script for simulating Levy motion:
clear all;
clc; close all;
t = 0; T = 1000; I = T-t;
dT = T/I; t = 0:dT:T; tau = T/I;
alpha = 1.5;
sigma = dT^(1/alpha);
mu = 0; beta = 0;
N = 1000;
X = zeros(N, length(I));
for k=1:N
L = zeros(1,I);
for i = 1:I-1
L( (i + 1) * tau ) = L(i*tau) + stable2( alpha, beta, sigma, mu, 1);
end
X(k,1:length(L)) = L;
end
q = 0.1:0.1:0.9;
quant = qlines2(X, q, t(1:length(X)), tau);
hold all
for i = 1:length(quant)
plot( t, quant(i) * t.^(1/alpha), ':k' );
end
Where stable2 returns a stable random variable with given parameters (you may replace it with normrnd(mu, sigma) for this case, it's not crucial); qlines2 returns quantiles needed for plotting.
But I don't want to talk about math here. My problem is that this implementation is pretty slow, and I would like to speed it up. Unfortunately, computer science is not my main field - I heard something about methods like memoization, vectorization and that there is a lot of other techniques, but I don't know how to use them.
For example, I'm pretty sure I should replace this filthy double for-loop somehow, but I'm not sure what to do instead.
EDIT: Maybe I should use (and learn...) another language (Python, C, any functional one)? I always though that Matlab/OCTAVE is designed for numerical computation, but if change, then for which one?
The crucial bit is, as you said, the for loops, Matlab does not like those, so vectorization is indeed the keyword. (Together with preallocating the space.
I just altered you for loop section somewhat so that you do not have to reset L over and over again, instead we save all Ls in a bigger matrix (also I elimiated the length(L) command).
L = zeros(N,I);
for k=1:N
for i = 1:I-1
L(k,(i + 1) * tau ) = L(k,i*tau) + normrnd(mu, sigma);
end
X(k,1:I) = L(k,1:I);
end
Now you can already see that X(k,1:I) = L(k,1:I); in the loop is obsolete and that also means that we can switch the order of the loops. This is crucial, because the i-steps are recursive (depend on the previous step) that means we cannot vectorize this loop, we can only vectorize the k-loop.
Now your original code needed 9.3 seconds on my machine, the new code still needs about the same time)
L = zeros(N,I);
for i = 1:I-1
for k=1:N
L(k,(i + 1) * tau ) = L(k,i*tau) + normrnd(mu, sigma);
end
end
X = L;
But now we can apply the vectorization, instead of looping throu all rows (the loop over k) we can instead eliminate this loop, and doing all rows at "once".
L = zeros(N,I);
for i = 1:I-1
L(:,(i + 1) * tau ) = L(:,i*tau) + normrnd(mu, sigma); %<- this is not yet what you want, see comment below
end
X = L;
This code need only 0.045 seconds on my machine. I hope you still get the same output, because I have no idea what you are calculating, but I also hope you could see how you go about vectorizing code.
PS: I just noticed that we now use the same random number in the last example for the whole column, this is obviously not what you want. Instad you should generate a whole vector of random numbers, e.g:
L = zeros(N,I);
for i = 1:I-1
L(:,(i + 1) * tau ) = L(:,i*tau) + normrnd(mu, sigma,N,1);
end
X = L;
PPS: Great question!
I write in variable 'O' some values using
for i = 1:size(I,1)
for j = 1:size(1,I)
h = i * j;
O{h} = I(i, j) * theta(h);
end
end
I - double, theta - double.
I need to sum()all 'O' values, but when I do it its give me error: sum: wrong type argument 'cell'.
How can I sum() it?
P.s. when I want to see O(), its give me
O =
{
[1,1] = 0.0079764
[1,2] = 0.0035291
[1,3] = 0.0027539
[1,4] = 0.0034392
[1,5] = 0.017066
[1,6] = 0.0082958
[1,7] = 1.4764e-04
[1,8] = 0.0024597
[1,9] = 1.1155e-04
[1,10] = 0.0010342
[1,11] = 0.0039654
[1,12] = 0.0047713
[1,13] = 0.0054305
[1,14] = 3.3794e-04
[1,15] = 0.014323
[1,16] = 0.0026826
[1,17] = 0.013864
[1,18] = 0.0097778
[1,19] = 0.0058029
[1,20] = 0.0020726
[1,21] = 0.0016430
etc...
The exact answer to your question is to use cell2mat
sum (cell2mat (your_cell_o))
However, this is the very wrong way to solve your problem. The thing is that you should not have created a cell array in first place. You should have created a numeric array:
O = zeros (size (I), class (I));
for i = 1:rows (I)
for j = 1:columns (I)
h = i * j;
O(h) = I(i, j) * theta(h);
endfor
endfor
but even this is just really bad and slow. Octave is a language to vectorize operations. Instead, you should have:
h = (1:rows (I))' .* (1:columns (I)); # automatic broadcasting
O = I .* theta (h);
which assumes your function theta behaves properly and if givena matrix will compute the value for each of the element of h and return something of the same size.
If you get an error about wrong sizes, I will guess you have an old version of Octave that does not perform automatic broadcasting. If so, update Octave. If you really can't, then:
h = bsxfun (#times, (1:rows (I))', 1:columns (I));
I have a cell array of size m x 1 and each cell is again s x t cell array (size varies). I would like to concatenate vertically. The code is as follows:
function(cell_out) = vert_cat(cell_in)
[row,col] = cellfun(#size,cell_in,'Uni',0);
fcn_vert = #(x)([x,repmat({''},size(x,1),max(cell2mat(col))-size(x,2))]);
cell_out = cellfun(fcn_vert,cell_in,'Uni',0); % Taking up lot of time
cell_out = vertcat(cell_out{:});
end
Step 3 takes a lot of time. Is it the right way to do or is there any another faster way to achieve this?
cellfun has been found to be slower than loops (kind of old, but agrees with what I have seen).
In addition, repmat has also been a performance hit in the past (though that may be different now).
Try this two-loop code that aims to accomplish your task:
function cellOut = vert_cat(c)
nElem = length(c);
colPad = zeros(nElem,1);
nRow = zeros(nElem,1);
for k = 1:nElem
[nRow(k),colPad(k)] = size(c{k});
end
colMax = max(colPad);
colPad = colMax - colPad;
cellOut = cell(sum(nRow),colMax);
bottom = cumsum(nRow) - nRow + 1;
top = bottom + nRow - 1;
for k = 1:nElem
cellOut(bottom(k):top(k),:) = [c{k},cell(nRow(k),colPad(k))];
end
end
My test for this code was
A = rand(20,20);
A = mat2cell(A,ones(20,1),ones(20,1));
C = arrayfun(#(c) A(1:c,1:c),randi([1,15],1,5),'UniformOutput',false);
ccat = vert_cat(c);
I used this pice of code to generate data:
%generating some dummy data
m=1000;
s=100;
t=100;
cell_in=cell(m,1);
for idx=1:m
cell_in{idx}=cell(randi(s),randi(t));
end
Applying some minor modifications, I was able to speed up the code by a factor of 5
%Minor modifications of the original code
%use arrays instead of cells for row and col
[row,col] = cellfun(#size,cell_in);
%claculate max(col) once
tcol=max(col);
%use cell instead of repmat to generate an empty cell
fcn_vert = #(x)([x,cell(size(x,1),tcol-size(x,2))]);
cell_out = cellfun(fcn_vert,cell_in,'Uni',0); % Taking up lot of time
cell_out = vertcat(cell_out{:});
Using simply a for loop is even faster, because the data is only moved once
%new approac. Basic idea: move every data only once
[row,col] = cellfun(#size,cell_in);
trow=sum(row);
tcol=max(col);
r=1;
cell_out2 = cell(trow,tcol);
for idx=1:numel(cell_in)
cell_out2(r:r+row(idx)-1,1:col(idx))=cell_in{idx};
r=r+row(idx);
end
I have a piece of code here I need to streamline as it is greatly increasing the runtime of my script:
size=300;
resultLength = (size+1)^3;
freqResult=zeros(1, resultLength);
inc=1;
for i=0:size,
for j=0:size,
for k=0:size,
freqResult(inc)=(c/2)*sqrt((i/L)^2+(j/W)^2+(k/H)^2);
inc=inc+1;
end
end
end
c, L, W, and H are all constants. As the size input gets over about 400, the runtime is too long to wait for, and I can watch my disk space draining by the gigabyte. Any advice?
Thanks!
What about this:
[kT, jT, iT] = ind2sub([size+1, size+1, size+1], [1:(size+1)^3]);
for indx = 1:numel(iT)
i = iT(indx) - 1;
j = jT(indx) - 1;
k = kT(indx) - 1;
freqResult1(indx) = (c/2)*sqrt((i/L)^2+(j/W)^2+(k/H)^2);
end
On my PC, for size = 400, version with 3 loops takes 136s and this one takes 19s.
For more "matlaby" way u could also even do as follows:
[kT, jT, iT] = ind2sub([size+1, size+1, size+1], [1:(size+1)^3]);
func = #(i, j, k) (c/2)*sqrt((i/L)^2+(j/W)^2+(k/H)^2);
freqResult2 = arrayfun(func, iT-1, jT-1, kT-1);
But for some reason, this is slower then the above version.
A faster solution can be (based on Marcin's answer):
[k, j, i] = ind2sub([size+1, size+1, size+1], [1:(size+1)^3]);
freqResult = (c/2)*sqrt(((i-1)/L).^2+((j-1)/W).^2+((k-1)/H).^2);
It takes about 5 seconds to run on my PC for size = 300
The following is even faster (but it doesn't look very good):
k = repmat(0:size,[1 (size+1)^2]);
j = repmat(kron(0:size, ones(1,size+1)),[1 (size+1)]);
i = kron(0:size, ones(1,(size+1)^2));
freqResult = (c/2)*sqrt((i/L).^2+(j/W).^2+(k/H).^2);
which takes ~3.5s for size = 300