Related
i have written a fairly large class for the calculation of measurement uncertainties, but it is painfully slow. Profiling the code shows that the slowest operation, by far, is to insert the computation results into a large sparse matrix. About 97% of all time is spent on that operation. The matrix keeps the uncertainties of all measurement data, and I cannot change the data structures without breaking a lot of other code. So my only option is to optimize the data insertion step. This is done about 5700 times in my benchmark, and every time the amout of data increases.
First solution, extremely slow:
% this automatically sums up duplicate yInd entries
[zInd_grid, yInd_grid] = ndgrid(1:numel(z), yInd(:));
Uzw = sparse(zInd_grid(:), yInd_grid(:), Uzy(:), numel(z), numel(obj.w));
% this automatically sums up duplicate yInd entries
dz_dw = sparse(zInd_grid(:), yInd_grid(:), dz_dy(:), numel(z), numel(obj.w));
obj.w = [obj.w; z(:)]; % insert new measurement results into the column vector obj.w
obj.Uww = [obj.Uww, transpose(Uzw); Uzw, Uzz]; % insert new uncertainties of the measurement results
obj.dw_dw = [obj.dw_dw, transpose(dz_dw); dz_dw, dz_dz]; % insert "dependencies" of new measurements on old results
The line obj.Uww = [obj.Uww, transpose(Uzw); Uzw, Uzz]; is the slowest, by far. Perhaps it is slow because Matlab needs to allocate a new, larger buffer for obj.Uww and copy everything over. Thus I changed the code to the following:
% Preallocation in the class constructor
obj.w = spalloc(nnz_w, 1, nnz_w);
obj.Uww = spalloc(nnz_w, nnz_w, nnz_Uww);
obj.dw_dw = spalloc(nnz_w, nnz_w, nnz_dw_dw);
obj.num_w = 0; % to manually keep track of the "real" size of obj.w, obj.Uww and obj.dw_dw
The class constructor is called with the sizes the three properties w, Uww and dw_dw will have at the end of the computation (nzz_w is approximately 0.1 million, nzz_Uww is at 9 million and nzz_dw_dw is about 1.6 million). Thus, no new allocation of memory should be needed. This is the inserting step now:
% this automatically sums up duplicate yInd entries
[zInd_grid, yInd_grid] = ndgrid(1:numel(z), yInd(:));
Uzw = sparse(zInd_grid(:), yInd_grid(:), Uzy(:), numel(z), obj.num_w);
% this automatically sums up duplicate yInd entries
dz_dw = sparse(zInd_grid(:), yInd_grid(:), dz_dy(:), numel(z), obj.num_w);
wInd = 1:obj.num_w;
obj.w(zInd, 1) = z(:); % insert new measurement results
obj.num_w = zInd(end); % new "real" size of w, Uww and dw_dw
obj.Uww(zInd, wInd) = Uzw; % about 51% of all computation time
obj.Uww(wInd, zInd) = transpose(Uzw); % about 15% of all computation time
obj.Uww(zInd, zInd) = Uzz; % about 14.4% of all computation time
obj.dw_dw(zInd, wInd) = dz_dw; % about 13% of all computation time
obj.dw_dw(wInd, zInd) = transpose(dz_dw); % about 3.5% of all computation time
obj.dw_dw(zInd, zInd) = dz_dz; % less than 3.5% of all computation time
Still, these lines account for 97% of all computation time, and no speed improvement. Thus I tried version three:
obj.w = [obj.w; z(:)];
[zInd_zy, yInd_zy] = ndgrid(zInd, yInd(:));
[zzInd_i, zzInd_j] = ndgrid(zInd, zInd);
[Uww_i, Uww_j, Uww_v] = find(obj.Uww); % 14% of all computation time
Uww_new = sparse( ... % this statement takes 66% of all computation time
[Uww_i; zInd_zy(:); yInd_zy(:); zzInd_i(:)], ...
[Uww_j; yInd_zy(:); zInd_zy(:); zzInd_j(:)], ...
[Uww_v; Uzy(:); Uzy(:); Uzz(:)], ...
numel(obj.w), numel(obj.w));
[dw_dw_i, dw_dw_j, dw_dw_v] = find(obj.dw_dw);
dw_dw_new = sparse( ... % 14% of all computation time
[dw_dw_i; zInd_zy(:); yInd_zy(:); zzInd_i(:)], ...
[dw_dw_j; yInd_zy(:); zInd_zy(:); zzInd_j(:)], ...
[dw_dw_v; dz_dy(:); dz_dy(:); dz_dz(:)], ...
numel(obj.w), numel(obj.w));
obj.Uww = Uww_new;
obj.dw_dw = dw_dw_new;
which is even slower thant the two other versions. Why is inserting into an already preallocated array so slow? And how can I speed it up?
(All the matrices are symmetric, but I did not try to exploit that yet.)
I don't understand the details of your update pattern, but keep in mind that Matlab stores sparse matrices internally in compressed-sparse column format. So adding entries in sequence column-by-column is significantly faster than other orders. E.g., on my old version of Matlab (R2006a), this:
n=10000;
nz=400000;
v=floor(n*rand(nz,3))+1;
fprintf('Random\n');
A=sparse(n, n);
tic
for k=1:nz
A(v(k,1), v(k,2))=v(k,3);
end
toc
fprintf('Row-wise\n');
v=sortrows(v);
A=sparse(n, n);
tic
for k=1:nz
A(v(k,1), v(k,2))=v(k,3);
end
toc
fprintf('Column-wise\n');
v=sortrows(v, [2 1]);
A=sparse(n, n);
tic
for k=1:nz
A(v(k,1), v(k,2))=v(k,3);
end
toc
gives this:
>> sparsetest
Random
Elapsed time is 19.276089 seconds.
Row-wise
Elapsed time is 20.714324 seconds.
Column-wise
Elapsed time is 1.498150 seconds.
Likely best of all would be if you can somehow just collect the nonzeros in a form suitable for spconvert or sparse and then make the whole sparse matrix at the end, but I gather that you might not be able to do that.
#bg2b Pointed out how adding data column-wise is much faster.
It turns out adding rows to a sparse matrix or sparse vector is painfully slow (more precisely, to the lower triangular part). Thus, I now store only the upper triangular part of the sparse matrix, because that is fast. When I need data from the matrix, I recreate it from the upper triangular sparse matrix. See the end of the benchmarking script for this.
This is my benchmarking script. It nicely shows the exponential increase in computation time for adding data.
% Benchmark the extension of sparse matrices of the form
% Uww_new = [ Uww, Uwz; ...
% Uzw, Uzz];
% where Uzw = transpose(Uwz). Uww and Uzz are always square and symmetric.
close all;
clearvars;
rng(70101557, 'twister'); % the seed is the number of the stack overflow question
density = 0.25;
nZ = 10;
iterations = 5e2;
n = nZ * iterations;
nonzeros = n*n*density;
all_Uzz = cell(iterations, 1);
all_Uwz = cell(iterations, 1);
for k = 1:iterations
% Uzz must be symmetric!
Uzz_nonsymmetric = sprand(nZ, nZ, density);
all_Uzz{k} = (Uzz_nonsymmetric + transpose(Uzz_nonsymmetric))./2;
all_Uwz{k} = sprand((k-1)*nZ, nZ, density);
end
f = figure();
ax = axes(f);
hold(ax, 'on');
grid(ax, 'on');
xlabel(ax, 'Iteration');
ylabel(ax, 'Elapsed time in seconds.');
h = gobjects(1, 0);
name = 'Optimised.';
fprintf('%s\n', name);
Uww_optimised = spalloc(0, 0, nonzeros);
t1 = tic();
elapsedTimes = NaN(iterations, 1);
for k = 1:iterations
Uzz = all_Uzz{k};
Uwz = all_Uwz{k};
zInd = size(Uww_optimised, 1) + (1:size(Uzz, 1));
wInd = 1:size(Uww_optimised, 1);
Uzz_triu = triu(Uzz);
Uww_optimised(wInd, zInd) = Uwz; % add columns
Uww_optimised(zInd, zInd) = Uzz_triu; % add rows and columns
elapsedTimes(k, 1) = toc(t1);
end
toc(t1)
h = [h, plot(ax, 1:iterations, seconds(elapsedTimes), 'DisplayName', name)];
name = 'Only Uwz and Uzz.';
fprintf('%s\n', name);
Uww_wz_zz = spalloc(0, 0, nonzeros);
t1 = tic();
elapsedTimes = NaN(iterations, 1);
for k = 1:iterations
Uzz = all_Uzz{k};
Uwz = all_Uwz{k};
zInd = size(Uww_wz_zz, 1) + (1:size(Uzz, 1));
wInd = 1:size(Uww_wz_zz, 1);
Uzw = transpose(Uwz);
Uww_wz_zz(wInd, zInd) = Uwz; % add columns
% Uww_wz_zz(zInd, wInd) = Uzw; % add rows
Uww_wz_zz(zInd, zInd) = Uzz; % add rows and columns
elapsedTimes(k, 1) = toc(t1);
end
toc(t1)
h = [h, plot(ax, 1:iterations, seconds(elapsedTimes), 'DisplayName', name)];
name = 'Only Uzw and Uzz.';
fprintf('%s\n', name);
Uww_zw_zz = spalloc(0, 0, nonzeros);
t1 = tic();
elapsedTimes = NaN(iterations, 1);
for k = 1:iterations
Uzz = all_Uzz{k};
Uwz = all_Uwz{k};
zInd = size(Uww_zw_zz, 1) + (1:size(Uzz, 1));
wInd = 1:size(Uww_zw_zz, 1);
Uzw = transpose(Uwz);
% Uww_zw_zz(wInd, zInd) = Uwz;
Uww_zw_zz(zInd, wInd) = Uzw;
Uww_zw_zz(zInd, zInd) = Uzz;
elapsedTimes(k, 1) = toc(t1);
end
toc(t1)
h = [h, plot(ax, 1:iterations, seconds(elapsedTimes), 'DisplayName', name)];
name = 'Uzw, Uwz and Uzz.';
fprintf('%s\n', name);
Uww = spalloc(0, 0, nonzeros);
t1 = tic();
elapsedTimes = NaN(iterations, 1);
for k = 1:iterations
Uzz = all_Uzz{k};
Uwz = all_Uwz{k};
zInd = size(Uww, 1) + (1:size(Uzz, 1));
wInd = 1:size(Uww, 1);
Uzw = transpose(Uwz);
Uww(wInd, zInd) = Uwz;
Uww(zInd, wInd) = Uzw;
Uww(zInd, zInd) = Uzz;
elapsedTimes(k, 1) = toc(t1);
end
toc(t1)
h = [h, plot(ax, 1:iterations, seconds(elapsedTimes), 'DisplayName', name)];
leg = legend(ax, h, 'Location', 'northwest');
assert(issymmetric(Uww));
assert(istriu(Uww_optimised));
assert(isequal(Uww, Uww_optimised + transpose(triu(Uww_optimised, 1))));
%% Get Uyy from Uww_optimised. Uyy is a symmetric subset of Uww
yInd = randi(size(Uww_optimised, 1), 1, nZ); % indices to extract
[yIndRowInds, yIndColInds] = ndgrid(yInd, yInd);
indsToFlip = yIndRowInds > yIndColInds;
temp = yIndColInds(indsToFlip);
yIndColInds(indsToFlip) = yIndRowInds(indsToFlip);
yIndRowInds(indsToFlip) = temp;
linInd = sub2ind(size(Uww_optimised), yIndRowInds, yIndColInds);
assert(issymmetric(linInd));
Uyy = Uww_optimised(linInd);
assert(issymmetric(Uyy));
I wrote an algorithm that does something and prints the output. The input for my algorithm is a list with some integers.
Here is the sample input as a list.
`mylist = [5,6,14,15,16,17,18,19,20,28,40,41,42,43,44,55]`
and here is my algorithm
```
tduration = 0
duration = 0
avg = 0
bottleneck = 0
y = 0
x = 0
while x<len(mylist)-4 and y<len(mylist)-1 :
if mylist[x+4] == mylist[x]+4:
y = x + 4
print("MY LIST X = ",mylist[x])
print("X = ", x)
print ("Y = ", y)
while True:
if y==len(mylist)-1 or mylist[y+1] > mylist[y]+10:
bottleneck = bottleneck + 1
duration = mylist[y] - mylist[x] + 1
tduration = tduration + duration
avg = tduration/bottleneck
x = y + 1
print("MY LIST Y = " , mylist[y])
print("Duration = " , duration)
break
else:
y = y + 1
else:
x = x + 1
print("BottleneckCount = ", bottleneck, "\nAverageDuration = ", avg)
```
Now I want to transform this "Algorithm" to a User Defined Function (UDF) in PySpark and then apply this UDF to a DataFrame with one column. There is one list in each row of this DataFrame. Sample DataFrame has 1 column and 2 rows. row1 is a list of [10,11,19,20,21,22,23,24,25,33,45] and row2 is a list of [55,56,57,58,59,60,80,81,82,83,84,85,92,115] so the UDF should be applied to each row of DataFrame separately and give the results for each row in another column.
Thank you in advance for your time and help. I will upvote your answers
Here's a way you can do:
import pyspark.sql.functions as F
from pyspark.sql.types import IntegerType, ArrayType
def calculate(mylist):
tduration = 0
duration = 0
avg = 0
bottleneck = 0
y = 0
x = 0
while x<len(mylist)-4 and y<len(mylist)-1 :
if mylist[x+4] == mylist[x]+4:
y = x + 4
print("MY LIST X = ",mylist[x])
print("X = ", x)
print ("Y = ", y)
while True:
if y==len(mylist)-1 or mylist[y+1] > mylist[y]+10:
bottleneck = bottleneck + 1
duration = mylist[y] - mylist[x] + 1
tduration = tduration + duration
avg = tduration/bottleneck
x = y + 1
print("MY LIST Y = " , mylist[y])
print("Duration = " , duration)
break
else:
y = y + 1
else:
x = x + 1
return bottleneck, avg
# sample data frame to use
df = spark.createDataFrame(
[
[[10,11,19,20,21,22,23,24,25,33,45]],
[[55,56,57,58,59,60,80,81,82,83,84,85,92,115]],
],
['col1',]
)
df.show()
+--------------------+
| col1|
+--------------------+
|[10, 11, 19, 20, ...|
|[55, 56, 57, 58, ...|
+--------------------+
# convert values to int --- edit
f_to_int = F.udf(lambda x: list(map(int, x)))
df = df.withColumn('col1', f_to_int('col1'))
# create udf
func = F.udf(lambda x: calculate(x), ArrayType(IntegerType()))
# apply udf
df = df.withColumn('vals', func('col1'))
# create new cols
df = df.select("col1", df.vals[0].alias('bottleneck'), df.vals[1].alias('avg'))
df.show()
+--------------------+----------+----+
| col1|bottleneck| avg|
+--------------------+----------+----+
|[10, 11, 19, 20, ...| 1|null|
|[55, 56, 57, 58, ...| 2|null|
+--------------------+----------+----+
YOLO answered this question and it is a complete answer. The only problem is that in the last column for "avg", we are getting NULL values.
I realized that I can solve this problem by using this "func" instead of that "func" in YOLO's answer.
import pyspark.sql.types as T
func = F.udf(lambda x: calculate(x), T.StructType(
[T.StructField("val1", T.IntegerType(), True),
T.StructField("val2", T.FloatType(), True)]))
I'm using the next code to plot in a pie chart the percentage of values in a matrix that are greater/smaller than 1. The thing is that when I want to put the title above the graph, it overlaps with the label of one of the groups.
I tried replacing it with text() but it didn't worked, and Documentation on pie say nothing to this. How can I avoid this overlap?
eigen = []; % Modes array
c2 = 170; % Sound speed divided by 2
%% Room dimensions
lx = 5.74;
ly = 8.1;
lz = 4.66;
i = 1; % Index for modes array
for nz = 0:50
for ny = 0:50
for nx = 0:50
aux = c2 * sqrt((nx/lx)^2+(ny/ly)^2+(nz/lz)^2);
if aux < 400 %% If value is into our range of interest
eigen(i) = aux;
i=i+1;
end
end
end
end
eigen = round(sort(eigen'),1);
eigen
% dif = eigen(2:end)-eigen(1:end-1); % Distance between modes
x = 0; %% dif >= 1
y = 0; %% dif <= 1
dif = [];
for i=2:length(eigen)
if eigen(i)-eigen(i-1) >= 1
x = x+1;
else
y = y+1;
end
end
figure
dif = [x,y];
explode = [1 1];
graf = pie(dif,explode);
hText = findobj(graf,'Type','text');
percentValues = get(hText,'String');
txt = {'Smaller than 1 Hz: ';'Greater than 1 Hz: '};
combinedtxt = strcat(txt,percentValues);
oldExtents_cell = get(hText,'Extent');
oldExtents = cell2mat(oldExtents_cell);
hText(1).String = combinedtxt(1);
hText(2).String = combinedtxt(2);
title('Distance between modes')
You can rotate the pie chart so that the figure look better. Further, you can use position to allocate your text as follows,
figure
dif = [x,y];
explode = [1 1];
graf = pie(dif,explode);
hText = findobj(graf,'Type','text');
percentValues = get(hText,'String');
txt = {'Smaller than 1 Hz: ';'Greater than 1 Hz: '};
combinedtxt = strcat(txt,percentValues);
oldExtents_cell = get(hText,'Extent');
oldExtents = cell2mat(oldExtents_cell);
hText(1).String = combinedtxt(1);
hText(2).String = combinedtxt(2);
view([90 90]) % this is to rotate the chart
textPositions_cell = get(hText,{'Position'});
textPositions = cell2mat(textPositions_cell);
textPositions(:,1) = textPositions(:,1) + 0.2; % replace 0.2 with any offset value you want
hText(1).Position = textPositions(1,:);
hText(2).Position = textPositions(2,:);
title('Distance between modes')
You can change only the text position (without rotation) by deleting view command.
I am using Excel 2007 which supports Columns upto 16,384 Columns. I would like to obtain the Column name corresponding Column Number.
Currently, I am using the following code. However this code supports upto 256 Columns. Any idea how to obtain Column Name if the column number is greater than 256.
function loc = xlcolumn(column)
if isnumeric(column)
if column>256
error('Excel is limited to 256 columns! Enter an integer number <256');
end
letters = {'A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z'};
count = 0;
if column-26<=0
loc = char(letters(column));
else
while column-26>0
count = count + 1;
column = column - 26;
end
loc = [char(letters(count)) char(letters(column))];
end
else
letters = ['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z'];
if size(column,2)==1
loc =findstr(column,letters);
elseif size(column,2)==2
loc1 =findstr(column(1),letters);
loc2 =findstr(column(2),letters);
loc = (26 + 26*loc1)-(26-loc2);
end
end
Thanks
As a diversion, here is an all function handle example, with (almost) no file-based functions required. This is based on the dec2base function, since Excel column names are (almost) base 26 numbers, with the frustrating difference that there are no "0" characters.
Note: this is probably a terrible idea overall, but it works. Better solutions are probably found elsewhere in the file exchange.
First, the one file based function that I couldn't get around, to perform arbitrary depth function composition.
function result = compose( fnHandles )
%COMPOSE Compose a set of functions
% COMPOSE({fnHandles}) returns a function handle consisting of the
% composition of the cell array of input function handles.
%
% For example, if F, G, and H are function handles with one input and
% one output, then:
% FNCOMPOSED = COMPOSE({F,G,H});
% y = FNCOMPOSED(x);
% is equivalent to
% y = F(G(H(x)));
if isempty(fnHandles)
result = #(x)x;
elseif length(fnHandles)==1
result = fnHandles{1};
else
fnOuter = fnHandles{1};
fnRemainder = compose(fnHandles(2:end));
result = #(x)fnOuter(fnRemainder(x));
end
Then, the bizarre, contrived path to convert base26 values into the correct string
%Functions leading to "getNumeric", which creates a numeric, base26 array
remapUpper = #(rawBase)(rawBase + (rawBase>='A')*(-55)); %Map the letters 'A-P' to [10:26]
reMapLower = #(rawBase)(rawBase + (rawBase<'A')*(-48)); %Map characters '0123456789' to [0:9]
getRawBase = #(x)dec2base(x, 26);
getNumeric = #(x)remapUpper(reMapLower(getRawBase(x)));
%Functions leading to "correctNumeric"
% This replaces zeros with 26, and reduces the high values entry by 1.
% Similar to "borrowing" as we learned in longhand subtraction
borrowDownFrom = #(x, fromIndex) [x(1:(fromIndex-1)) (x(fromIndex)-1) (x(fromIndex+1)+26) (x((fromIndex+2):end))];
borrowToIfNeeded = #(x, toIndex) (x(toIndex)<=0)*borrowDownFrom(x,toIndex-1) + (x(toIndex)>0)*(x); %Ugly numeric switch
getAllConditionalBorrowFunctions = #(numeric)arrayfun(#(index)#(numeric)borrowToIfNeeded(numeric, index),(2:length(numeric)),'uniformoutput',false);
getComposedBorrowFunction = #(x)compose(getAllConditionalBorrowFunctions(x));
correctNumeric = #(x)feval(getComposedBorrowFunction(x),x);
%Function to replace numerics with letters, and remove leading '#' (leading
%zeros)
numeric2alpha = #(x)regexprep(char(x+'A'-1),'^#','');
%Compose complete function
num2ExcelName = #(x)arrayfun(#(x)numeric2alpha(correctNumeric(getNumeric(x))), x, 'uniformoutput',false)';
Now test using some stressing transitions:
>> num2ExcelName([1:5 23:28 700:704 727:729 1024:1026 1351:1355 16382:16384])
ans =
'A'
'B'
'C'
'D'
'E'
'W'
'X'
'Y'
'Z'
'AA'
'AB'
'ZX'
'ZY'
'ZZ'
'AAA'
'AAB'
'AAY'
'AAZ'
'ABA'
'AMJ'
'AMK'
'AML'
'AYY'
'AYZ'
'AZA'
'AZB'
'AZC'
'XFB'
'XFC'
'XFD'
This function I wrote works for any number of columns (until Excel runs out of columns). It just requires a column number input (e.g. 16368 will return a string 'XEN').
If the application of this concept is different than my function, it's important to note that a column of x number of A's begins every 26^(x-1) + 26^(x-2) + ... + 26^2 + 26 + 1. (e.g. 'AAA' begins on 26^2 + 26 + 1 = 703)
function [col_str] = let_loc(num_loc)
test = 2;
old = 0;
x = 0;
while test >= 1
old = 26^x + old;
test = num_loc/old;
x = x + 1;
end
num_letters = x - 1;
str_array = zeros(1,num_letters);
for i = 1:num_letters
loc = floor(num_loc/(26^(num_letters-i)));
num_loc = num_loc - (loc*26^(num_letters-i));
str_array(i) = char(65 + (loc - 1));
end
col_str = strcat(str_array(1:length(str_array)));
end
Hope this saves someone some time!
I have a R x C matrix filled to the k-th row and empty below this row. What i need to do is to fill the remaining rows. In order to do this, i have a function that takes 2 entire rows as arguments, process these rows and output 2 fresh rows (these outputs will fill the empty rows of the matrix, in batches of 2). I have a fixed matrix containing all 'pairs' of rows to be processed, but my for loop is not helping performance:
# the processRows function:
processRows = function(r1, r2)
{
# just change a little bit the two rows and return it in a compact way
nr1 = r1 * 0.1
nr2 = -r2 * 0.1
matrix (c(nr1, nr2), ncol = 2)
}
# M is the matrix
# nrow(M) and k are even, so nLeft is even
M = matrix(1:48, ncol = 3)
# half to fill (can be more or less, but k is always even)
k = nrow(M)/2
# simulate empty rows to be filled
M[-(1:k), ] = 0
cat('before fill')
print(M)
# number of empty rows to fill
nLeft = nrow(M) - k
nextRow = k + 1
# each row in idxList represents a 'pair' of rows to be processed
# any pairwise combination of non-empty rows could happen
# make it reproducible
set.seed(1)
idxList = matrix (sample(1:k, k), ncol = 2, byrow = TRUE)
for ( i in 1 : (nLeft / 2))
{
row1 = M[idxList[i, 1],]
row2 = M[idxList[i, 2],]
# the two columns in 'results' will become 2 rows in M
results = processRows(row1, row2)
# fill the matrix
M[nextRow, ] = results[, 1]
nextRow = nextRow + 1
M[nextRow, ] = results[, 2]
nextRow = nextRow + 1
}
cat('after fill')
print(M)
Okay, here is your code first. We run this so that we have a copy of the "true" matrix, the one we hope to reproduce, faster.
#### Original Code (aka Gold Standard) ####
M = matrix(1:48, ncol = 3)
k = nrow(M)/2
M[-(1:k), ] = 0
nLeft = nrow(M) - k
nextRow = k + 1
idxList = matrix(1:k, ncol = 2)
for ( i in 1 : (nLeft / 2))
{
row1 = M[idxList[i, 1],]
row2 = M[idxList[i, 2],]
results = matrix(c(2*row1, 3*row2), ncol = 2)
M[nextRow, ] = results[, 1]
nextRow = nextRow + 1
M[nextRow, ] = results[, 2]
nextRow = nextRow + 1
}
Now here is the vectorized code. The basic idea is if you have 4 rows you are processing. Rather than passing them as vectors one at a time, do it at once. That is:
(1:3) * 2
(1:3) * 2
(1:3) * 2
(1:3) * 2
is the same (but slower) as:
c(1:3, 1:3, 1:3, 1:3) * 2
So first, we will use your same setup code, then create the rows to be processed as two long vectors (where all 4 original rows are just strung together as in my simple example above). Then, we take those results, and transform them into matrices with the appropriate dimensions. The last trick is to assign the results back in in just two steps. You can assign to multiple rows of a matrix at once, so we use seq() to get odd and even numbers so assign the first and second column of the results to, respectively.
#### Vectorized Code (testing) ####
M2 = matrix(1:48, ncol = 3)
k2 = nrow(M2)/2
M2[-(1:k2), ] = 0
nLeft2 = nrow(M2) - k2
nextRow2 = k2 + 1
idxList2 = matrix(1:k2, ncol = 2)
## create two long vectors of all rows to be processed
row12 <- as.vector(t(M2[idxList2[, 1],]))
row22 <- as.vector(t(M2[idxList2[, 2],]))
## get all results
results2 = matrix(c(2*row12, 3*row22), ncol = 2)
## add results back
M2[seq(nextRow2, nextRow2 + nLeft2-1, by = 2), ] <- matrix(results2[,1], nLeft2/2, byrow=TRUE)
M2[seq(nextRow2+1, nextRow2 + nLeft2, by = 2), ] <- matrix(results2[,2], nLeft2/2, byrow=TRUE)
## check that vectorized code matches your examples
all.equal(M, M2)
Which on my machine gives:
> all.equal(M, M2)
[1] TRUE