Related
I got an assignment in a video processing course - to implement the Lucas-Kanade algorithm. Since we have to do it in the pyramidal model, I first build a pyramid for each of the 2 input images, and then for each level I perform a number of LK iterations. in each step (iteration), the following code runs (note: the images are zero-padded so I can handle the image edges easily):
function [du,dv]= LucasKanadeStep(I1,I2,WindowSize)
It = I2-I1;
[Ix, Iy] = imgradientxy(I2);
Ixx = imfilter(Ix.*Ix, ones(5));
Iyy = imfilter(Iy.*Iy, ones(5));
Ixy = imfilter(Ix.*Iy, ones(5));
Ixt = imfilter(Ix.*It, ones(5));
Iyt = imfilter(Iy.*It, ones(5));
half_win = floor(WindowSize/2);
du = zeros(size(It));
dv = zeros(size(It));
A = zeros(2);
b = zeros(2,1);
%iterate only on the relevant parts of the images
for i = 1+half_win : size(It,1)-half_win
for j = 1+half_win : size(It,2)-half_win
A(1,1) = Ixx(i,j);
A(2,2) = Iyy(i,j);
A(1,2) = Ixy(i,j);
A(2,1) = Ixy(i,j);
b(1,1) = -Ixt(i,j);
b(2,1) = -Iyt(i,j);
U = pinv(A)*b;
du(i,j) = U(1);
dv(i,j) = U(2);
end
end
end
mathematically what I'm doing is calculating for every pixel (i,j) the following optical flow:
as you can see, in the code I am calculating this for each pixel, which takes quite a long time (the whole processing for 2 images - including building 3 levels pyramids and 3 LK steps like the one above on each level - takes about 25 seconds (!) on a remote connection to my university servers).
My question: Is there a way to calculate this single LK step without the nested for loops? it must be more efficient because the next step of the assignment is to stabilize a short video using this algorithm.. thanks.
I ran your code on my system and did profiling. Here is what I got.
As you can see inverting the matrix(pinv) is taking most of the time. You can try and vectorise your code I guess, but I am not sure how to do it. But I do know a trick to improve the compute time. You have to exploit the minimum variance of the matrix A. That is, compute the inverse only if the minimum variance of A is greater than some threshold. This will improve the speed as you won't be inverting the matrix for all the pixel.
You do this by modifying your code to the one shown below.
function [du,dv]= LucasKanadeStep(I1,I2,WindowSize)
It = double(I2-I1);
[Ix, Iy] = imgradientxy(I2);
Ixx = imfilter(Ix.*Ix, ones(5));
Iyy = imfilter(Iy.*Iy, ones(5));
Ixy = imfilter(Ix.*Iy, ones(5));
Ixt = imfilter(Ix.*It, ones(5));
Iyt = imfilter(Iy.*It, ones(5));
half_win = floor(WindowSize/2);
du = zeros(size(It));
dv = zeros(size(It));
A = zeros(2);
B = zeros(2,1);
%iterate only on the relevant parts of the images
for i = 1+half_win : size(It,1)-half_win
for j = 1+half_win : size(It,2)-half_win
A(1,1) = Ixx(i,j);
A(2,2) = Iyy(i,j);
A(1,2) = Ixy(i,j);
A(2,1) = Ixy(i,j);
B(1,1) = -Ixt(i,j);
B(2,1) = -Iyt(i,j);
% +++++++++++++++++++++++++++++++++++++++++++++++++++
% Code I added , threshold better be outside the loop.
lambda = eig(A);
threshold = 0.2
if (min(lambda)> threshold)
U = A\B;
du(i,j) = U(1);
dv(i,j) = U(2);
end
% end of addendum
% +++++++++++++++++++++++++++++++++++++++++++++++++++
% U = pinv(A)*B;
% du(i,j) = U(1);
% dv(i,j) = U(2);
end
end
end
I have set the threshold to 0.2. You can experiment with it. By using eigen value trick I was able to get the compute time from 37 seconds to 10 seconds(shown below). Using eigen, pinv hardly takes up the time like before.
Hope this helped. Good luck :)
Eventually I was able to find a much more efficient solution to this problem.
It is based on the formula shown in the question. The last 3 lines are what makes the difference - we get a loop-free code that works way faster. There were negligible differences from the looped version (~10^-18 or less in terms of absolute difference between the result matrices, ignoring the padding zone).
Here is the code:
function [du,dv]= LucasKanadeStep(I1,I2,WindowSize)
half_win = floor(WindowSize/2);
% pad frames with mirror reflections of itself
I1 = padarray(I1, [half_win half_win], 'symmetric');
I2 = padarray(I2, [half_win half_win], 'symmetric');
% create derivatives (time and space)
It = I2-I1;
[Ix, Iy] = imgradientxy(I2, 'prewitt');
% calculate dP = (du, dv) according to the formula
Ixx = imfilter(Ix.*Ix, ones(WindowSize));
Iyy = imfilter(Iy.*Iy, ones(WindowSize));
Ixy = imfilter(Ix.*Iy, ones(WindowSize));
Ixt = imfilter(Ix.*It, ones(WindowSize));
Iyt = imfilter(Iy.*It, ones(WindowSize));
% calculate the whole du,dv matrices AT ONCE!
invdet = (Ixx.*Iyy - Ixy.*Ixy).^-1;
du = invdet.*(-Iyy.*Ixt + Ixy.*Iyt);
dv = invdet.*(Ixy.*Ixt - Ixx.*Iyt);
end
I will like to implement "Adaptive Watershed Segmentation" in Matlab.
There are six steps in this algorithm. Input is figure(a) and result is figure(d).
Would you please to help me check is there any mistake in my code, and I don't know how to implement the sixth step.
Thank you so much!
Load image:
input_image = imread('test.gif');
Step 1 : Calculate D(x,y) at each (x,y), obtain the Euclidian distance map of the binary image and assign each value of M(x,y) as 0.
DT = bwdist(input_image,'euclidean'); % Trandform distance:Euclidian distance
[h,w]=size(DT);
M = zeros(h,w);
Step 2 : Smooth the distance map using Gaussian filter to merge the adjacent maxima, set M(x,y) as 1 if D(x,y) is a local maximum, and then obtain the marker map of the distance map.
H = fspecial('gaussian');
gfDT = imfilter(DT,H);
M = imregionalmax(gfDT); % maker map, M = local maximum of gfDT
Step3 : Scan the marker map pixel by pixel. If M(x0,y0) is 1, seek the spurious maxima in its neighbourhood with a radius of D(x ,y ).When M(x,y) equals 1 and sqr((x − x0)^2 + (y − y0)^2 ) ≤ D(x0, y0) , set M(x,y) as 0 if D(x,y) < D(x0,y0).
for x0 = 1:h
for y0 = 1:w
if M(x0,y0) == 1
r = ceil(gfDT(x0,y0));
% range begin:(x0-r,y0-r) end:(x0+r,y0+r)
xb = x0-r;
if xb <= 0
xb =1;
end
yb = y0-r;
if yb <= 0
yb =1;
end
xe = x0+r;
if xe > w
xe = w;
end
ye = y0+r;
if ye > h
ye = h;
end
for x = yb:ye
for y = xb:xe
if M(x,y)==1
Pos = [x0,y0 ;x,y];
Dis = pdist(Pos,'euclidean');
IFA = Dis<= (gfDT(x0,y0));
IFB = gfDT(x,y)<gfDT(x0,y0);
if ( IFA && IFB)
M(x,y) = 0;
end
end
end
end
end
end
end
Step 4:
Calculate the inverse of the distance map,and the local maxima turn out to be the local minima.
igfDT = -(gfDT);
STep5:
Segment the distance map according to the markers by the conventional watershed algorithm and obtain the segmentation of binary image.
I2 = imimposemin(igfDT,M);
L = watershed(I2);
igfDT (L==0)=0;
Step 6 : Straighten the watershed lines by linking the ends of the watershed lines with a straight line and reclassifying the pixels along the straight line.
I don't know how to implement this step
Try distance transform and then watershed transform.
im=imread('n6BRI.gif');
imb=bwdist(im);
sigma=3;
kernel = fspecial('gaussian',4*sigma+1,sigma);
im2=imfilter(imb,kernel,'symmetric');
L = watershed(max(im2(:))-im2);
[x,y]=find(L==0);
lblImg = bwlabel(L&~im);
figure,imshow(label2rgb(lblImg,'jet','k','shuffle'));
Here's essentially my problem. Also maybe I am not familiar enough with Euler angles and what I'm attempting to do is not possible.
I have 2 points in 3d space.
p1 (1,2,3)
p2 (4,5,6)
In order to get the unit vectors for these two points I'm doing this basically.
var productX = (position.X2 - position.X1);
var productY = (position.Y2 - position.Y1);
var productZ = (position.Z2 - position.Z1);
var normalizedTotal = Math.sqrt(productX * productX + productY * productY + productZ * productZ);
var unitVectorX, unitVectorY, unitVectorZ;
if(normalizedTotal == 0)
{
unitVectorX = productX;
unitVectorY = productY;
unitVectorZ = productZ;
}
else
{
unitVectorX = productX / normalizedTotal;
unitVectorY = productY / normalizedTotal;
unitVectorZ = productZ / normalizedTotal;
}
So now I have a unit vector x y z for these 2 3d points.
I'm attempting now to convert from directional vector to euler angle. Is this possible. What am I missing here as I can't find any good resource on how to do this.
Thanks for the help.
Sometimes a picture helps.
maybe this will give a better example of what i'm trying to solve for.
Given 2 points, I have determined a midpoint, length, and now i'm trying to figure out hte angles to set so that the cylinder is correctly oriented around the x,y,z axis. I think I need to figure out all 3 angles not just 1 and 2 is that correct? I think the euler angles from a directional vector bit through you off.
What you want is a transformation from Cartesian coordinates of the vector
v = (v_x, v_y, v_z)
to the spherical coordinates r, ψ and θ where
v = ( r*COS(ψ)*COS(θ), r*SIN(θ), r*SIN(ψ)*COS(θ) )
This is done with the following equations
r = SQRT(v_x^2+v_y^2+v_z^2)
TAN(ψ) = (v_z)/(v_x)
TAN(θ) = (v_y)/(v_x^2+v_z^2)
To get the angles ψ and θ, use the ATAN2(dy,dx) function as in
ψ = ATAN2(v_z, v_x)
θ = ATAN2(v_y, SQRT(v_x^2+v_z^2))
Now that you have the along direction vector
j = ( COS(ψ)*COS(θ), SIN(θ), SIN(ψ)*COS(θ) )
you can get the two perpendicular vectors from
i = ( SIN(ψ), 0, -COS(ψ) )
k = ( COS(ψ)*SIN(θ), -COS(θ), SIN(ψ)*SIN(θ) )
These three vectors make up the columns of the 3×3 rotation matrix
| SIN(ψ) COS(ψ)*COS(θ) COS(ψ)*SIN(θ) |
E =[i j k] = | 0 SIN(θ) -COS(θ) |
| -COS(ψ) SIN(ψ)*COS(θ) SIN(ψ)*SIN(θ) |
In terms of Euler angles the above is equivalent to
E = RY(π/2-ψ)*RX(π/2-θ)
Example
Two points p_1=(3,2,3) and p_2=(5,6,4) define the vector
v = (5,6,4) - (3,2,3) = (2,4,1)
NOTE: I am using the notation of v[i] for the i-th element of the vector, as in v[1]=2 above. This is neither like C, Python which is zero based, nor like VB, FORTRAN or MATLAB which uses parens () for the index.
Using the expressions above you get
r = √(2^2+4^2+1^2) = √21
TAN(ψ) = 1/2
TAN(θ) = 4/√(2^2+1^2) = 4/√5
ψ = ATAN2(1,2) = 0.463647
θ = ATAN2(4,√5) = 1.061057
Now to find the direction vectors
j = ( COS(ψ)*COS(θ), SIN(θ), SIN(ψ)*COS(θ) ) = (0.4364, 0.87287, 0.21822 )
i = ( SIN(ψ), 0, -COS(ψ) ) = (0.44721, 0, -0.89443 )
k = ( COS(ψ)*SIN(θ), -COS(θ), SIN(ψ)*SIN(θ) ) = (0.78072, -0.48795, 0.39036)
Put the direction vectors as columns of the local to world coordinate transformation (rotation)
E[1,1] = i[1] E[1,2] = j[1] E[1,3] = k[1]
E[2,1] = i[2] E[2,2] = j[2] E[2,3] = k[2]
E[3,1] = i[3] E[3,2] = j[3] E[3,3] = k[3]
| 0.447213595499957 0.436435780471984 0.780720058358826 |
| |
E = | 0 0.872871560943969 -0.487950036474266 |
| |
| -0.894427190999915 0.218217890235992 0.390360029179413 |
I am implementing a fast optimization algorithm using fixed point method in matlab. The goal of that method is that find optimal value of u. Denote u={u_i,i=1..2}. The optimal value of u can be obtained as following steps:
Sorry about my image because I cannot type mathematics equation in here.
To do that task, I tried to find u follows above steps. However, I don't know how to implement the term \sum_{j!=i} (u_j-1) in equation 25. This is my code. Please see it and could you give me some comment or suggestion about my implementation to correct them. Currently, I tried to run that code but it give an incorrect answer.
function u = compute_u_TV(Im0, N_class)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Initialization
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
theta=0.001;
gamma=0.01;
tau=0.1;
sigma=0.1;
N_class=2; % only have u1 and u2
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Iterative segmentation process
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
for i=1:N_class
v(:,:,i) = Im0/max(Im0(:)); % u between 0 and 1.
qxv(:,:,i) = zeros(size(Im0));
qyv(:,:,i) = zeros(size(Im0));
u(:,:,i) = v(:,:,i);
for iteration=1:10000
u_temp=u;
% Update v
Divqi = ( BackwardX(qxv(:,:,i)) + BackwardY(qyv(:,:,i)) );
Term = Divqi - u(:,:,i)/ (theta*gamma);
TermX = ForwardX(Term);
TermY = ForwardY(Term);
Norm = sqrt(TermX.^2 + TermY.^2);
Denom = 1 + tau*Norm;
%Equation 24
qxv(:,:,i) = (qxv(:,:,i) + tau*TermX)./Denom;
qyv(:,:,i) = (qyv(:,:,i) + tau*TermY)./Denom;
v(:,:,i) = u(:,:,i) - theta*gamma* Divqi; %Equation 23
% Update u
u(:,:,i) = (v(:,:,i) - theta* gamma* Divqi -theta*gamma*sigma*(sum(u(:))-u(:,:,i)-1))./(1+theta* gamma*sigma);
u(:,:,i) = max(u(:,:,i),0);
u(:,:,i) = min(u(:,:,i),1);
check=u_temp(:,:,i)-u(:,:,i);
if(abs(sum(check(:)))<=0.1)
break;
end
end
end
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Sub-functions- X.Berson
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function [dx]=BackwardX(u);
[Ny,Nx] = size(u);
dx = u;
dx(2:Ny-1,2:Nx-1)=( u(2:Ny-1,2:Nx-1) - u(2:Ny-1,1:Nx-2) );
dx(:,Nx) = -u(:,Nx-1);
function [dy]=BackwardY(u);
[Ny,Nx] = size(u);
dy = u;
dy(2:Ny-1,2:Nx-1)=( u(2:Ny-1,2:Nx-1) - u(1:Ny-2,2:Nx-1) );
dy(Ny,:) = -u(Ny-1,:);
function [dx]=ForwardX(u);
[Ny,Nx] = size(u);
dx = zeros(Ny,Nx);
dx(1:Ny-1,1:Nx-1)=( u(1:Ny-1,2:Nx) - u(1:Ny-1,1:Nx-1) );
function [dy]=ForwardY(u);
[Ny,Nx] = size(u);
dy = zeros(Ny,Nx);
dy(1:Ny-1,1:Nx-1)=( u(2:Ny,1:Nx-1) - u(1:Ny-1,1:Nx-1) );
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% End of sub-function
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
You should do
u(:,:,i) = (v(:,:,i) - theta* gamma* Divqi -theta*gamma*sigma* ...
(sum(u(:,:,1:size(u,3) ~= i),3) -1))./(1+theta* gamma*sigma);
The part you were searching for is
sum(u(:,:,1:size(u,3) ~= i),3)
Let's decompose this :
1:size(u,3) ~= i
is a vector containing all values from 1 to the max size of u on the third dimension except i.
Then
u(:,:,1:size(u,3) ~= i)
is all the matrix of the third dimension of u except for j = i
Finally,
sum(...,3)
is the sum of all the matrix by the thrid dimension.
Let me know if it does help!
For a fixed and given tform, the imwarp command in the Image Processing Toolbox
B = imwarp(A,tform)
is linear with respect to A, meaning there exists some sparse matrix W, depending on tform but independent of A, such that the above can be equivalently implemented
B(:)=W*A(:)
for all A of fixed known dimensions [n,n]. My question is whether there are fast/efficient options for computing W. The matrix form is necessary when I need the transpose operation W.'*B(:), or if I need to do W\B(:) or similar linear algebraic things which I can't do directly through imwarp alone.
I know that it is possible to compute W column-by-column by doing
E=zeros(n);
W=spalloc(n^2,n^2,4*n^2);
for i=1:n^2
E(i)=1;
tmp=imwarp(E,tform);
E(i)=0;
W(:,i)=tmp(:);
end
but this is brute force and slow.
The routine FUNC2MAT is somewhat more optimal in that it uses the loop to compute/gather the sparse entry data I,J,S of each column W(:,i). Then, after the loop, it uses this to construct the overall sparse matrix. It also offers the option of using a PARFOR loop. However, this is still slower than I would like.
Can anyone suggest more speed-optimal alternatives?
EDIT:
For those uncomfortable with my claim that imwarp(A,tform) is linear w.r.t. A, I include the demo script below, which tests that the superposition property is satisfied for random input images and tform data. It can be run repeatedly to see that the nonlinearityError is always small, and easily attributable to floating point noise.
tform=affine2d(rand(3,2));
%tform=projective2d(rand(3));
fun=#(A) imwarp(A,tform,'cubic');
I1=rand(100); I2=rand(100);
c1=rand; c2=rand;
LHS=fun(c1*I1+c2*I2); %left hand side
RHS=c1*fun(I1)+c2*fun(I2); %right hand side
linearityError = norm(LHS(:)-RHS(:),'inf')
That's actually pretty simple:
W = sparse(B(:)/A(:));
Note that W is not unique, but this operation probably produces the most sparse result. Another way to calculate it would be
W = sparse( B(:) * pinv(A(:)) );
but that results in a much less sparse (yet still valid) result.
I constructed the warping matrix using the optical flow fields [u,v] and it is working well for my application
% this function computes the warping matrix
% M x N is the size of the image
function [ Fw ] = generateFwi( u,v,M,N )
Fw = zeros(M*N, M*N);
k =1;
for i=1:M
for j= 1:N
newcoord(1) = i+u(i,j);
newcoord(2) = j+v(i,j);
newi = newcoord(1);
newj = newcoord(2);
if newi >0 && newj >0
newi1x = floor(newi);
newi1y = floor(newj);
newi2x = floor(newi);
newi2y = ceil(newj);
newi3x = ceil(newi); % four nearest points to the given point
newi3y = floor(newj);
newi4x = ceil(newi);
newi4y = ceil(newj);
x1 = [newi,newj;newi1x,newi1y];
x2 = [newi,newj;newi2x,newi2y];
x3 = [newi,newj;newi3x,newi3y];
x4 = [newi,newj;newi4x,newi4y];
w1 = pdist(x1,'euclidean');
w2 = pdist(x2,'euclidean');
w3 = pdist(x3,'euclidean');
w4 = pdist(x4,'euclidean');
if ceil(newi) == floor(newi) && ceil(newj)==floor(newj) % both the new coordinates are integers
Fw(k,(newi1x-1)*N+newi1y) = 1;
else if ceil(newi) == floor(newi) % one of the new coordinates is an integer
w = w1+w2;
w1new = w1/w;
w2new = w2/w;
W = w1new*w2new;
y1coord = (newi1x-1)*N+newi1y;
y2coord = (newi2x-1)*N+newi2y;
if y1coord <= M*N && y2coord <=M*N
Fw(k,y1coord) = W/w2new;
Fw(k,y2coord) = W/w1new;
end
else if ceil(newj) == floor(newj) % one of the new coordinates is an integer
w = w1+w3;
w1 = w1/w;
w3 = w3/w;
W = w1*w3;
y1coord = (newi1x-1)*N+newi1y;
y2coord = (newi3x-1)*N+newi3y;
if y1coord <= M*N && y2coord <=M*N
Fw(k,y1coord) = W/w3;
Fw(k,y2coord) = W/w1;
end
else % both the new coordinates are not integers
w = w1+w2+w3+w4;
w1 = w1/w;
w2 = w2/w;
w3 = w3/w;
w4 = w4/w;
W = w1*w2*w3 + w2*w3*w4 + w3*w4*w1 + w4*w1*w2;
y1coord = (newi1x-1)*N+newi1y;
y2coord = (newi2x-1)*N+newi2y;
y3coord = (newi3x-1)*N+newi3y;
y4coord = (newi4x-1)*N+newi4y;
if y1coord <= M*N && y2coord <= M*N && y3coord <= M*N && y4coord <= M*N
Fw(k,y1coord) = w2*w3*w4/W;
Fw(k,y2coord) = w3*w4*w1/W;
Fw(k,y3coord) = w4*w1*w2/W;
Fw(k,y4coord) = w1*w2*w3/W;
end
end
end
end
else
Fw(k,k) = 1;
end
k=k+1;
end
end
end