Related
I am working on linear model predictive control and I need to calculate some matrices for the controller, only.. it takes a lot of time to calculate one of them and I would like to ask if there is a better way to code this calculation. I am using MATLAB, but I do understand FORTRAN also.
Well, I want to calculate a matrix (Φ) but the way I am doing it takes to much time to calculate it. Φ matrix is of the form (the right one):
MPC_matrices.
Here is the book where I found this image in case you need to refer to (especially page 8).
Now the code that I wrote in MATLAB goes like this:(moved it after EDIT)
Considering that I will have quite big NS, Np and Nc variables it will take a whole lot of time to do this calculation. Is there an optimal way (or at least better than mine) to speed up this calculation?
EDIT
After considering #Daniel's & user2682877's comments I tested this
clear;clc
Np = 80;
Nc = Np / 2;
m = 3;
q = 1;
Niter = 30;
MAT = zeros(Niter,5);
for I=1:Niter
NS = 10 * I;
A = rand(NS,NS);
B = rand(NS,m);
C = rand(1,NS);
tic
Phi1 = zeros(Np*q,Nc*m);
CB = C * B;
for i=1:Np
for j=1:Nc
if j<i
Phi1( (q*i-(q-1)):(q*i) , (m*j-(m-1)):(m*j) ) = C * A^(i-1-(j-1)) * B;
elseif j==i
Phi1( (q*i-(q-1)):(q*i) , (m*j-(m-1)):(m*j) ) = CB;
end
end
end
t1 = toc;
% Daniel's suggestion
tic
Phi2=zeros(Np*q,Nc*m);
CB = C * B;
for diffij=0:Np-1
if diffij>0
F=C * A^diffij * B;
else
F=CB;
end
for i=max(1,diffij+1):min(Np,Nc+diffij)
j=i-diffij;
Phi2( (q*i-(q-1)):(q*i) , (m*j-(m-1)):(m*j) ) = F;
end
end
t2 = toc;
% user2682877 suggestion
tic
Phi3=zeros(Np*q,Nc*m);
temp = B;
% 1st column
Phi3( (q*1-(q-1)):(q*1) , (m*1-(m-1)):(m*1) ) = C * B;
for i=2:Np
% reusing temp
temp = A * temp;
Phi3( (q*i-(q-1)):(q*i) , (m*1-(m-1)):(m*1) ) = C * temp;
end
% remaining columns
for j=2:Nc
for i=j:Nc
Phi3( (q*i-(q-1)):(q*i) , (m*j-(m-1)):(m*j) ) =...
Phi3( (q*(i-j+1)-(q-1)):(q*(i-j+1)) , (m*1-(m-1)):(m*1) );
end
end
t3 = toc;
MAT(I,:) = [I, NS, t1, t2 ,t3];
fprintf('I iteration = %g \n', I);
end
figure(1)
clf(1)
hold on
plot(MAT(:,2),MAT(:,3),'b')
plot(MAT(:,2),MAT(:,4),'r')
plot(MAT(:,2),MAT(:,5),'g')
hold off
legend('My <Unfortunate> Idea','Daniel`s suggestion','user2682877 suggestion')
xlabel('NS variable')
ylabel('Time, s')
And here is the resulting figure:
Keep in mind that now NS = 300 ,BUT as I upgrate my model (I intent include more and more equations and variables in the state space model) this variables (mostly NS and Np) will be bigger and bigger.
#Daniel's 2nd comment, I know that I perform more calculations than those I should, but my lack of experience limits my ideas on upgrating this one.
#durasm's comment, I am not quite familiar with parfor, but I will test it.
Referring to the answers: I will test your suggestions as soon as I understand them (...) and get back to you.
Results
It is obvious that my initial thought is only a bit worse than what suggested here.. Thank you guys! You were extremely helpful!
There is only a limited set of results from your calculation C * A^(i-1-(j-1)) * B which only depends on the difference between i and j. Not to repeatedly calculate it, my solution iterates this difference and i, then calculates j depending on these two variables.
Phi=zeros(Np*q,Nc*m);
CB = C * B;
for diffij=0:Np-1
if diffij>0
F=C * A^diffij * B;
else
F=CB;
end
for i=max(1,diffij+1):min(Np,Nc+diffij)
j=i-diffij;
Phi( (q*i-(q-1)):(q*i) , (m*j-(m-1)):(m*j) ) = F;
end
end
Performance comparison:
Maybe you can try this:
Phi=zeros(Np*q,Nc*m);
temp = B;
% 1st column
Phi( (q*1-(q-1)):(q*1) , (m*1-(m-1)):(m*1) ) = C * B;
for i=2:Np
% reusing temp
temp = A * temp;
Phi( (q*i-(q-1)):(q*i) , (m*1-(m-1)):(m*1) ) = C * temp;
end
% remaining columns
for j=2:Nc
for i=j:Np
Phi( (q*i-(q-1)):(q*i) , (m*j-(m-1)):(m*j) ) = Phi( (q*(i-j+1)-(q-1)):(q*(i-j+1)) , (m*1-(m-1)):(m*1) );
end
end
Sometimes one requires random numbers with the condition that they are unique.
The classic algorithm is to keep looping until you hit the different numbers by chance, some pseudo code:
minval = 0 ;
maxval = 4 ; // random max will be one less than maxval
val1 = random( minval , maxval ) ;
val2 = random( minval , maxval ) ;
while( val1 == val2 ) {
val2 = random( minval , maxval ) ;
}
I have a time critical and memory limited program and was wondering if there are any algorithms that avoid the continuous loop brute force method without using extra memory like a look up table.
Probably a simple solution but it's a late and tired evening here.
Any tips?
Yes, you can exclude the already found number like this:
minval = 0 ;
maxval = 4 ; // random max will be one less than maxval
val1 = random( minval , maxval ) ;
val2 = random( minval , maxval - 1 ) ;
if(val2 >= val1) val2++;
The second time, specify an interval of one less than the first one, and shift the part above the already found number up by one.
A1, B1, C1, A2, B2 and C2 are 6 matrix with the same dimensions 4435X2000.
I have to find the values i, j and k for which A1(k,2000) == A2(i,j) and B1(k,2000) == B2(i,j) and C1(k,2000) == C2(i,j) , with the condition X(k)==1 and Y(i,j)==1
The objective is to find: counter, L, T and D
Is there a way to make this code faster? Can I avoid loops?
counter=0;
L(1)=0;
T(1)=0;
D(1)=0;
for k=1:4435
if X(k)==1 % X is a vector (4435x1)
F(k,:) = [A1(k,2000) B1(k,2000) C1(k,2000)]
for i=1:4435
for j=100:1999
if Y(i,j)==1 % Y is a matrix (4435x1999)
if F(k,:) == [A2(i,j) B2(i,j) C2(i,j)]
counter = counter+1;
L(counter)=k;
T(counter)=i;
D(counter)=j;
end
end
end
end
end
end
I want a solution that will save me at least 80% of the computation time!
and not have the error message: Out of memory
See how this works out for you -
%// Store X-Y data by calling X() and Y() functions
X_data = X(1:4435);
Y_data = Y(1:4435,100:1999);
range1 = 100:1999 %// define range for columns
A2 = A2(:,range1); %// Crop out A2, B2, C2 based on column-range
B2 = B2(:,range1);
C2 = C2(:,range1);
Y_data = Y_data(:,range1)==1;
%// Indices for dim-3
idx_X = find(X_data==1)
%// Map X==1 onto A1, B1, C1
A1Lr = A1(X_data==1,end)
B1Lr = B1(X_data==1,end)
C1Lr = C1(X_data==1,end)
%// Setup output array to store L, T, D as single Nx3 output array
out = zeros(sum(Y_data(:))*numel(A1Lr),3);
%// Try out(sum(Y_data(:)==1)*numel(A1Lr),3)=0; instead for speed!
%// Start collecting output indices
count = 1;
for iter1 = 1:numel(A1Lr)
[R,C] = find(Y_data & A2==A1Lr(iter1) & B2==B1Lr(iter1) & C2==C1Lr(iter1));
nR = numel(R);
out(count:count+nR-1,:) = [R C repmat(iter1,nR,1)];
count = count + nR;
end
out(find(out(:,1)==0,1):end,:)=[];
%// Packup the outputs
T = out(:,1)
D = out(:,2) + range1(1)-1
L = idx_X(out(:,3))
It is very difficult to determine what your code is actually supposed to accomplish, without really working to interpret your code. However, I'll give it a crack:
% Determine where X is true.
XTrue = X == 1;
% Extract values from A1,B1,C1 where X is true.
F ( XTrue , 1 : 3 ) = [ A1(XTrue,2000) B1(XTrue,2000) C1(XTrue,2000) ];
% Determine where Y is true.
YTrueIndex = find ( Y == 1 );
% Determine where the extracted values match
counter = [];
L = [];
T = [];
D = [];
for ( ii = 1 : length(YTrueIndex) )
indexCurrent = YTrueIndex(ii)
FRowsThatMatch = F(:,1)==A2(indexCurrent) & F(:,2)==B2(indexCurrent) & F(:,3)==C2(indexCurrent);
matchCount = length ( find ( FRowsThatMatch ) );
if ( matchCount > 0 )
counter = counter + matchCount;
[ i , j ] = ind2sub ( size ( Y ) , indexCurrent );
L = [ L , find ( FRowsThatMatch ) ];
T = [ T , ones(matchCount,1)*i ];
D = [ D , ones(matchCount,2)*j ];
end
end
Given a grid (or table) with x*y cells. Each cell contains a value. Most of these cells have a value of 0, but there may be a "hot spot" somewhere on this grid with a cell that has a high value. The neighbours of this cell then also have a value > 0. As farer away from the hot spot as lower the value in the respective grid cell.
So this hot spot can be seen as the top of a hill, with decreasing values the farer we are away from this hill. At a certain distance the values drop to 0 again.
Now I need to determine the cell within the grid that represents the grid's center of gravity. In the simple example above this centroid would simply be the one cell with the highest value. However it's not always that simple:
the decreasing values of neighbour cells around the hot spot cell may not be equally distributed, or a "side of the hill" may fall down to 0 sooner than another side.
there is another hot spot/hill with values > 0 elsewehere within the grid.
I could think that this is kind of a typical problem. Unfortunately I am no math expert so I don't know what to search for (at least I have not found an answer in Google).
Any ideas how can I solve this problem?
Thanks in advance.
You are looking for the "weighted mean" of the cell values. Assuming each cell has a value z(x,y), then you can do the following
zx = sum( z(x, y) ) over all values of y
zy = sum( z(x, y) ) over all values of x
meanX = sum( x * zx(x)) / sum ( zx(x) )
meanY = sum( y * zy(y)) / sum ( zy(y) )
I trust you can convert this into a language of your choice...
Example: if you know Matlab, then the above would be written as follows
zx = sum( Z, 1 ); % sum all the rows
zy = sum( Z, 2 ); % sum all the columns
[ny nx] = size(Z); % find out the dimensions of Z
meanX = sum((1:nx).*zx) / sum(zx);
meanY = sum((1:ny).*zy) / sum(zy);
This would give you the meanX in the range 1 .. nx : if it's right in the middle, the value would be (nx+1)/2. You can obviously scale this to your needs.
EDIT: one more time, in "almost real" code:
// array Z(N, M) contains values on an evenly spaced grid
// assume base 1 arrays
zx = zeros(N);
zy = zeros(M);
// create X profile:
for jj = 1 to M
for ii = 1 to N
zx(jj) = zx(jj) + Z(ii, jj);
next ii
next jj
// create Y profile:
for ii = 1 to N
for jj = 1 to M
zy(ii) = zy(ii) + Z(ii, jj);
next jj
next ii
xsum = 0;
zxsum = 0;
for ii = 1 to N
zxsum += zx(ii);
xsum += ii * zx(ii);
next ii
xmean = xsum / zxsum;
ysum = 0;
zysum = 0;
for jj = 1 to M
zysum += zy(jj);
ysum += jj * zy(ii);
next jj
ymean = ysum / zysum;
This Wikipedia entry may help; the section entitled "A system of particles" is all you need. Just understand that you need to do the calculation once for each dimension, of which you apparently have two.
And here is a complete Scala 2.10 program to generate a grid full of random integers (using dimensions specified on the command line) and find the center of gravity (where rows and columns are numbered starting at 1):
object Ctr extends App {
val Array( nRows, nCols ) = args map (_.toInt)
val grid = Array.fill( nRows, nCols )( util.Random.nextInt(10) )
grid foreach ( row => println( row mkString "," ) )
val sum = grid.map(_.sum).sum
val xCtr = ( ( for ( i <- 0 until nRows; j <- 0 until nCols )
yield (j+1) * grid(i)(j) ).sum :Float ) / sum
val yCtr = ( ( for ( i <- 0 until nRows; j <- 0 until nCols )
yield (i+1) * grid(i)(j) ).sum :Float ) / sum
println( s"Center is ( $xCtr, $yCtr )" )
}
You could def a function to keep the calculations DRYer, but I wanted to keep it as obvious as possible. Anyway, here we run it a couple of times:
$ scala Ctr 3 3
4,1,9
3,5,1
9,5,0
Center is ( 1.8378378, 2.0 )
$ scala Ctr 6 9
5,1,1,0,0,4,5,4,6
9,1,0,7,2,7,5,6,7
1,2,6,6,1,8,2,4,6
1,3,9,8,2,9,3,6,7
0,7,1,7,6,6,2,6,1
3,9,6,4,3,2,5,7,1
Center is ( 5.2956524, 3.626087 )
I am failing over and over trying to get this indicator to run quantised with 2 buffers in mql4. After a long time reading I have put 2 extra buffers in to squish it :/ because:
the indicator is sitting between 0.1430-0.1427 at present but doesn't have a fixed top and bottom.
I can't seem to suss it; cool indicator but won't play fair!
#property indicator_separate_window
#property indicator_buffers 4
#property indicator_color1 Lime
#property indicator_color2 Red
#property indicator_color3 CLR_NONE
#property indicator_color4 CLR_NONE
//#property indicator_minimum 0
//#property indicator_maximum 100
extern int P = 13;
extern int T = 3000;
extern double P2 = 0.001;
//int MIN = 0;
//int MAX = 100;
double G[];
double R[];
double B3[];
double B4[];
int init(){
IndicatorBuffers(4);
SetIndexBuffer( 0, G );SetIndexStyle( 0, DRAW_LINE, STYLE_SOLID, 1, Lime );
SetIndexBuffer( 1, R );SetIndexStyle( 1, DRAW_LINE, STYLE_SOLID, 1, Red );
SetIndexBuffer( 2, B3 );SetIndexStyle( 2, DRAW_NONE );
SetIndexBuffer( 3, B4 );SetIndexStyle( 3, DRAW_NONE );
return(0);
}
int start(){
if ( T >= Bars ) T = Bars;
SetIndexDrawBegin( 0, Bars - T + P + 1 );
SetIndexDrawBegin( 1, Bars - T + P + 1 );
SetIndexDrawBegin( 2, Bars - T + P + 1 );
SetIndexDrawBegin( 3, Bars - T + P + 1 );
int Z, C, Opt = IndicatorCounted();
if ( Bars <= 38 ) return(0);
if ( Opt < P ){
for ( Z = 1; Z <= 0; Z++ ) G[T-Z] = 0.0;
for ( Z = 1; Z <= 0; Z++ ) R[T-Z] = 0.0;
}
Z = T - P - 1;
while( Z >= 0 ){
double A, S1, S2;
S1 = 0.0; for ( C = 0; C <= P - 1; C++ ){ S1 = S1 + ( High[Z+C] + Low[Z+C] ) / 2;}
S2 = 0.0; for ( C = 0; C <= P - 1; C++ ){ S2 = S2 + ( ( High[Z+C] + Low[Z+C] ) * ( C+1 ) / 2 );}
A = S1 / S2;
// if ( A < MIN ){ MIN = A;}
// if ( A > MAX ){ MAX = A;}
// A = ( MIN / MAX ) * 100;
G[Z] = A;
if ( Z > 0 ){ R[Z-1] = A;}
Z--;
}
for ( int N = T-P-2; N >= 0; N-- ){
if ( N > 0 ){
if ( G[N-1] > G[N] ){ R[N] = EMPTY_VALUE; continue;}
if ( G[N-1] < G[N] ){ G[N] = R[N]; continue;}
}
B3[0] = G[0] + P2;
B4[0] = G[0] - P2; //forced quantise using 2 extra buffers
}
return(0);
}
Let´s split the task first
0) indicator logic
1) indicator quantisation step
2) indicator performance
MQL4 Custom Indicator programming relies on deeper understanding of underlying MetaTrader4 Terminal platform. Each external Market Event, changing a traded instrument price, is signalled to a lcoalhost by a network delivery of a QUOTE ... message from MetaTrader4 Server. This is aka Tick and it triggers a call to a function originally called start(), in newer New-MQL4.56789 renamed to OnTick().
The below modified MQL4 listing contains remarks for core-logic disambiguation, which must precede all the below listed steps.
1) indicator quantisation step
While the code is still very inefficient ( as per [2] below ) the logic does not include any straight hurdle form having the output quantised to any form thereof { binary | ternary | arbitrary-number-of-states }-quantised system. Whence the indicator core-logic is cleared, the quantisation step is just a trivial conversion from R(1) to I(1).
2) indicator performance
Any Tick arrival may, but need not modify either High[0] or Low[0], which are the only variable parts of the proposed Custom Indicator calculus.
This is the core idea on how to reduce the scope of re-calculations, that the MQL4 code has to realise per tick. In recent versions of MT4, all Custom Indicators share a single thread, the more stress has been put on efficient algorithmisation of Custom Indicators, at these may block the platform's trading decisions on poor, inefficient code-loops and convolution/recursion re-executions.
#property indicator_separate_window
#property indicator_buffers 4
#property indicator_color1 Lime
#property indicator_color2 Red
#property indicator_color3 CLR_NONE
#property indicator_color4 CLR_NONE
extern int P = 13;
extern int T = 3000;
extern double P2 = 0.001;
double G[]; // 0: LINE
double R[]; // 1: LINE
double B3[]; // 2: BUT NEVER PAINTED, NEVER CONSUMED _?_
double B4[]; // 3: BUT NEVER PAINTED, NEVER CONSUMED _?_
int init(){
IndicatorBuffers(4);
SetIndexBuffer( 0, G );SetIndexStyle( 0, DRAW_LINE, STYLE_SOLID, 1, Lime );
SetIndexBuffer( 1, R );SetIndexStyle( 1, DRAW_LINE, STYLE_SOLID, 1, Red );
SetIndexBuffer( 2, B3 );SetIndexStyle( 2, DRAW_NONE );
SetIndexBuffer( 3, B4 );SetIndexStyle( 3, DRAW_NONE );
return(0);
}
int start(){
if ( Bars <= 38 ) return(0); // JIT/RET in case Bars < 39 --^ --^ --^ --^
if ( T >= Bars ) T = Bars; // (TRIM´d) T < Bars .OR. = Bars
int aDrawBegins = Bars - T + P + 1; // ( extern P = 13 ) + 1 + ( Bars - ( extern T = 3000 if T < Bars else Bars ) )
//tIndexDrawBegin( 0, Bars - T + P + 1 ); // PREF: ( reused 4x )
SetIndexDrawBegin( 0, aDrawBegins ); // Draw 14+ last candles -- PREF: why a per tick hard-coded SHIFTING / enforced re-draw?
SetIndexDrawBegin( 1, aDrawBegins ); // Draw 14+ last candles -- PREF: why a per tick hard-coded SHIFTING / enforced re-draw?
SetIndexDrawBegin( 2, aDrawBegins ); // Draw 14+ last candles -- PREF: why a per tick hard-coded SHIFTING / enforced re-draw?
SetIndexDrawBegin( 3, aDrawBegins ); // Draw 14+ last candles -- PREF: why a per tick hard-coded SHIFTING / enforced re-draw?
double A, S1, S2; // auxiliary var for bar-mid-price calculi
int Z; // auxiliary stepper
int Opt = IndicatorCounted(); // Opt ( NEVER RE-USED )
if ( Opt < P ){ // if ( ( extern P = 13 ) > IndicatorCounted() )
// ----------------------- ??? ----------------------------------------------------- NEVER EXEC´d: for( Z = 1++ v/s Z <= 0 )
for ( Z = 1; Z <= 0; Z++ ) G[T-Z] = 0.0; // .STO G[T-Z], 0., BUT NEVER EXEC´d: for( Z = 1++ v/s Z <= 0 )
for ( Z = 1; Z <= 0; Z++ ) R[T-Z] = 0.0; // .STO R[T-Z], 0., BUT NEVER EXEC´d: for( Z = 1++ v/s Z <= 0 )
// ----------------------- ??? ----------------------------------------------------- NEVER EXEC´d: for( Z = 1++ v/s Z <= 0 )
}
Z = T - P - 1; // .STO Z, ( T = Bars (TRIM´d) ) - ( extern P = 13 ) - 1
while( Z >= 0 ){ // .DEC Z
// !!! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ // PERF: very inefficient to RE-calc STATIC ( ( extern P = 13 ) - 1 )-DEEP CONVOLUTIONS per tick !!
S1 = 0.0; for ( int C = 0; C <= P - 1; C++ ){ S1 = S1 + ( High[Z+C] + Low[Z+C] ) / 2; }
S2 = 0.0; for ( int C = 0; C <= P - 1; C++ ){ S2 = S2 + ( ( High[Z+C] + Low[Z+C] ) * ( C+1 ) / 2 );}
// !!! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ // PERF: very inefficient to RE-calc STATIC ( ( extern P = 13 ) - 1 )-DEEP CONVOLUTIONS per tick !!
A = S1 / S2;
G[Z] = A; // .STO G[Z], A if Z >= 0
if ( Z > 0 ){ R[Z-1] = A;} // .STO R[Z-1], A if Z > 0
Z--;
}
for ( int N = T - P - 2; N >= 0; N-- ){ // .STO N, ( T = Bars (TRIM´d) ) - ( extern P = 13 ) - 2
if ( N > 0 ){ // N > 0:
if ( G[N-1] > G[N] ){ R[N] = EMPTY_VALUE; continue;} // .BLNK R[N], EMPTY if G[N-1] > G[N]
if ( G[N-1] < G[N] ){ G[N] = R[N]; continue;} // .SET G[N], R[N] if G[N-1] < G[N]
}
// ?? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ // WHY MANY-TIMES RE-ASSIGNED A CONST. VALUE HERE, INSIDE A FOR(){...}-loop body? -------------- ??
B3[0] = G[0] + P2; // .STO B3[0], G[0] + ( extern P2 = 0.001 )
B4[0] = G[0] - P2; // .STO B4[0], G[0] - ( extern P2 = 0.001 )
// forced quantise using 2 extra buffers
// ?? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ // WHY MANY-TIMES RE-ASSIGNED A CONST. VALUE HERE, INSIDE A FOR(){...}-loop body? -------------- ??
}
return(0);
}
New-MQL4.56789 syntax
The OnCalculate() function is called only in custom indicators when it's necessary to calculate the indicator values by the Calculate event. This usually happens when a new tick is received for the symbol, for which the indicator is calculated. This indicator is not required to be attached to any price chart of this symbol.
The first rates_total parameter contains the number of bars, available to the indicator for calculation, and corresponds to the number of bars available in the chart.
We should note the connection between the return value of OnCalculate() and the second input parameter prev_calculated. During the OnCalculate() function call, the prev_calculated parameter contains a value returned by OnCalculate() during previous call. This allows for economical algorithms for calculating the custom indicator in order to avoid repeated calculations for those bars that haven't changed since the previous run of this function.
For this, it is usually enough to return the value of the rates_total parameter, which contains the number of bars in the current function call. If since the last call of OnCalculate() the price data has changed (a deeper history downloaded or history blanks filled), the value of the input parameter prev_calculated will be set to zero by the terminal.
//+------------------------------------------------------------------+
//| Custom indicator iteration function |
//+------------------------------------------------------------------+
int OnCalculate( const int rates_total,
const int prev_calculated,
const datetime &time[],
const double &open[],
const double &high[],
const double &low[],
const double &close[],
const long &tick_volume[],
const long &volume[],
const int &spread[]
)
{
// Get the number of bars available now for the current Symbol and chart period
int barsNow = Bars( _Symbol, PERIOD_CURRENT );
// .RET value of prev_calculated for a next call
return( rates_total );
}