I am implementing the partial derivative equations from the Horn & Schunck paper on optical flow. However, even for relative small images (320x568), it takes a frustratingly long time (~30-40 seconds) to complete. I assume this is due to the 320 x 568 = 181760 loop iterations, but I can't figure out a more efficient way to do this (short of a MEX file).
Is there some way to turn this into a more efficient MATLAB operation (a convolution perhaps)? I can figure out how to do this as a convolution for It but not Ix and Iy. I've also considered matrix shifting, but that only works for It as well, as far as I can figure out.
Has anyone else run into this issue and found a solution?
My code is below:
function [Ix, Iy, It] = getFlowParams(img1, img2)
% Make sure image dimensions match up
assert(size(img1, 1) == size(img2, 1) && size(img1, 2) == size(img2, 2), ...
'Images must be the same size');
assert(size(img1, 3) == 1, 'Images must be grayscale');
% Dimensions of original image
[rows, cols] = size(img1);
Ix = zeros(numel(img1), 1);
Iy = zeros(numel(img1), 1);
It = zeros(numel(img1), 1);
% Pad images to handle edge cases
img1 = padarray(img1, [1,1], 'post');
img2 = padarray(img2, [1,1], 'post');
% Concatenate i-th image with i-th + 1 image
imgs = cat(3, img1, img2);
% Calculate energy for each pixel
for i = 1 : rows
for j = 1 : cols
cube = imgs(i:i+1, j:j+1, :);
Ix(sub2ind([rows, cols], i, j)) = mean(mean(cube(:, 2, :) - cube(:, 1, :)));
Iy(sub2ind([rows, cols], i, j)) = mean(mean(cube(2, :, :) - cube(1, :, :)));
It(sub2ind([rows, cols], i, j)) = mean(mean(cube(:, :, 2) - cube(:, :, 1)));
end
end
2D convolution is the way to go here as also predicted in the question to replace those heavy mean/average calculations. Also, those iterative differentiations could be replaced by MATLAB's diff. Thus, incorporating all that, a vectorized implementation would be -
%// Pad images to handle edge cases
img1 = padarray(img1, [1,1], 'post');
img2 = padarray(img2, [1,1], 'post');
%// Store size parameters for later usage
[m,n] = size(img1);
%// Differentiation along dim-2 on input imgs for Ix calculations
df1 = diff(img1,[],2)
df2 = diff(img2,[],2)
%// 2D Convolution to simulate average calculations & reshape to col vector
Ixvals = (conv2(df1,ones(2,1),'same') + conv2(df2,ones(2,1),'same'))./4;
Ixout = reshape(Ixvals(1:m-1,:),[],1);
%// Differentiation along dim-1 on input imgs for Iy calculations
df1 = diff(img1,[],1)
df2 = diff(img2,[],1)
%// 2D Convolution to simulate average calculations & reshape to col vector
Iyvals = (conv2(df1,ones(1,2),'same') + conv2(df2,ones(1,2),'same'))./4
Iyout = reshape(Iyvals(:,1:n-1),[],1);
%// It just needs elementwise diffentiation between input imgs.
%// 2D convolution to simulate mean calculations & reshape to col vector
Itvals = conv2(img2-img1,ones(2,2),'same')./4
Itout = reshape(Itvals(1:m-1,1:n-1),[],1)
Benefits with such a vectorized implementation would be :
Memory efficiency : No more concatenation along the third dimension that would incur memory overhead. Again, performance wise it would be a benefit as we won't need to index into such heavy arrays.
The iterative differentiations inside the loopy codes are replaced by differentiation with diff, so this should be another improvement.
Those expensive average calculations are replaced by very fast convolution calculations and this should be the major improvement section.
A faster method, with improved results (in the cases I have observed) is the following:
function [Ix, Iy, It] = getFlowParams(imNew,imPrev)
gg = [0.2163, 0.5674, 0.2163];
f = imNew + imPrev;
Ix = f(:,[2:end end]) - f(:,[1 1:(end-1)]);
Ix = conv2(Ix,gg','same');
Iy = f([2:end end],:) - f([1 1:(end-1)],:);
Iy = conv2(Iy,gg ,'same');
It = 2*conv2(gg,gg,imNew - imPrev,'same');
This handles the boundary cases elegantly.
I made this as part of my optical flow toolbox, where you can easily view H&S, Lucas Kanade and more in realtime. In the toolbox, the function is called grad3D.m. You may also want to check out grad3Drec.m, in the same toolbox, which adds simple temporal blurring.
Related
I have to use an inverse filter to remove the blurring from this image
.
Unfortunately, I have to figure out the transfer function H of the imaging
system used to get these sharper images, It should be Gaussian. So, I should determine the approximate width of the Gaussian by trying different Gaussian widths in an inverse filter and judging which resulting images look the “best”.
The best result will be optimally sharp – i.e., edges will look sharp but will not have visible ringing.
I tried by using 3 approaches:
I created a transfer function with N dimensions (odd number, for simplicity), by creating a grid of N dimensions, and then applying the Gaussian function to this grid. After that, we add zeroes to this transfer function in order to get the same size as the original image. However, after applying the filter to the original image, I just see noise (too many artifacts).
I created the transfer function with size as high as the original image, by creating a grid of the same size as the original image. If sigma is too small, then the PSF FFT magnitude is wide. Otherwise it gets thinner. If sigma is small, then the image is even more blurred, but if we set a very high sigma value then we get the same image (not better at all).
I used the fspecial function, playing with sizes of sigma and h. But still I do not get anything sharper than the original blurred image.
Any ideas?
Here is the code used for creating the transfer function in Approach 1:
%Create Gaussian Filter
function h = transfer_function(N, sigma, I) %N is the dimension of the kernel
%create a 2D-grid that is the same size as the Gaussian filter matrix
grid = -floor(N/2) : floor(N/2);
[x, y] = meshgrid(grid, grid);
arg = -(x.*x + y.*y)/(2*sigma*sigma);
h = exp(arg); %gaussian 2D-function
kernel = h/sum(h(:)); %Normalize so that total weight equals 1
[rows,cols] = size(I);
add_zeros_w = (rows - N)/2;
add_zeros_h = (cols - N)/2;
h = padarray(kernel,[add_zeros_w add_zeros_h],0,'both'); % h = kernel_final_matrix
end
And this is the code for every approach:
I = imread('lena_blur.jpg');
I1 = rgb2gray(I);
figure(1),
I1 = double(I1);
%---------------Approach 1
% N = 5; %Dimension Assume is an odd number
% sigma = 20; %The bigger number, the thinner the PSF in FREQ
% H = transfer_function(N, sigma, I1);
%I1=I1(2:end,2:end); %To simplify operations
imagesc(I1); colormap('gray'); title('Original Blurred Image')
I_fft = fftshift(fft2(I1)); %Shift the image in Fourier domain to let its DC part in the center of the image
% %FILTER-----------Approach 2---------------
% N = 5; %Dimension Assume is an odd number
% sigma = 20; %The bigger number, the thinner the PSF in FREQ
%
%
% [x,y] = meshgrid(-size(I,2)/2:size(I,2)/2-1, -size(I,1)/2:size(I,1)/2-1);
% H = exp(-(x.^2+y.^2)*sigma/2);
% %// Normalize so that total area (sum of all weights) is 1
% H = H /sum(H(:));
%
% %Avoid zero freqs
% for i = 1:size(I,2) %Cols
% for j = 1:size(I,1) %Rows
% if (H(i,j) == 0)
% H(i,j) = 1e-8;
% end
% end
% end
%
% [rows columns z] = size(I);
% G_filter_fft = fft2(H,rows,columns);
%FILTER---------------------------------
%Filter--------- Aproach 3------------
N = 21; %Dimension Assume is an odd number
sigma = 1.25; %The bigger number, the thinner the PSF in FREQ
H = fspecial('gaussian',N,sigma)
[rows columns z] = size(I);
G_filter_fft = fft2(H,rows,columns);
%Filter--------- Aproach 3------------
%DISPLAY FFT PSF MAGNITUDE
figure(2),
imshow(fftshift(abs(G_filter_fft)),[]); title('FFT PSF magnitude 2D');
% Yest = Y_blurred/Gaussian_Filter
I_restoration_fft = I_fft./G_filter_fft;
I_restoration = (ifft2(I_restoration_fft));
I_restoration = abs(I_restoration);
I_fft = abs(I_fft);
% Display of Frequency domain (To compare with the slides)
figure(3),
subplot(1,3,1);
imagesc(I_fft);colormap('gray');title('|DFT Blurred Image|')
subplot(1,3,2)
imshow(log(fftshift(abs(G_filter_fft))+1),[]) ;title('| Log DFT Point Spread Function + 1|');
subplot(1,3,3)
imagesc(abs(I_restoration_fft));colormap('gray'); title('|DFT Deblurred|')
% imshow(log(I_restoration+1),[])
%Display PSF FFT in 3D
figure(4)
hf_abs = abs(G_filter_fft);
%270x270
surf([-134:135]/135,[-134:135]/135,fftshift(hf_abs));
% surf([-134:134]/134,[-134:134]/134,fftshift(hf_abs));
shading interp, camlight, colormap jet
xlabel('PSF FFT magnitude')
%Display Result (it should be the de-blurred image)
figure(5),
%imshow(fftshift(I_restoration));
imagesc(I_restoration);colormap('gray'); title('Deblurred Image')
%Pseudo Inverse restoration
% cam_pinv = real(ifft2((abs(G_filter_fft) > 0.1).*I_fft./G_filter_fft));
% imshow(fftshift(cam_pinv));
% xlabel('pseudo-inverse restoration')
A possible solution is deconvwr. I will first show its performance starting from an undistorted lena image. So, I know exactly the gaussian blurring function. Note that setting estimated_nsr to zero will destroy the performance completely due to quantisation noise.
I_ori = imread('lenaTest3.jpg'); % Download an original undistorted lena file
N = 19;
sigma = 5;
H = fspecial('gaussian',N,sigma)
estimated_nsr = 0.05;
I = imfilter(I_ori, H)
wnr3 = deconvwnr(I, H, estimated_nsr);
figure
subplot(1, 4, 1);
imshow(I_ori)
subplot(1, 4, 2);
imshow(I)
subplot(1, 4, 3);
imshow(wnr3)
title('Restoration of Blurred, Noisy Image Using Estimated NSR');
subplot(1, 4, 4);
imshow(H, []);
The best parameters I found for your problem by trial and error.
N = 19;
sigma = 2;
H = fspecial('gaussian',N,sigma)
estimated_nsr = 0.05;
EDIT: calculating exactly the used blurring filter
If you download an undistorted lena (I_original_fft), you can calculate the used blurring filter as follows:
G_filter_fft = I_fft./I_original_fft
How to vectorize this code in MATLAB?
If possible, I wish matrix B to be a sparse matrix.
%% Y is a matrix l*n
%% X is a matrix k*n
B = [];
for i=1:l
for j=1:n
temp1 = zeros(1,n*l);
temp1((i-1)*n+j) = -1;
temp2 = zeros(1,l*k);
temp2((i-1)*k+1:i*k) = (-Y(i,j)).*(X(:,j)');
B = [B;[temp1,temp2]];
end
end
I don't know how to vectorize this code, please help! Thanks!
Making use of bsxfun for masking and vectorized calculations of the elementwise multiplications, here's a vectorized approach -
%// Create left part of the output that is basically an identity matrix
parte1 = -eye(n*l);
%// Setup right part of output
parte2 = zeros(n*l,l*k);
%// Mask to set elements from the calculations of (-Y(i,j)).*(X(:,j)')
M = bsxfun(#eq,reshape(repmat(1:l,n,1),[],1),reshape(repmat(1:l,k,1),1,[]));
%// OR concisely : M = kron(eye(l),ones(n,k))==1
%// Perform vectorized calculations of (-Y(i,j)).*(X(:,j)') and set those
%// into second part at masked places
parte2(M) = -bsxfun(#times,permute(X,[2,1,3]),permute(Y,[2,3,1]));
%// Finally concatenate those parts for final output
out = [parte1,parte2];
I have an image of size as RGB uint8(576,720,3) where I want to classify each pixel to a set of colors. I have transformed using rgb2lab from RGB to LAB space, and then removed the L layer so it is now a double(576,720,2) consisting of AB.
Now, I want to classify this to some colors that I have trained on another image, and calculated their respective AB-representations as:
Cluster 1: -17.7903 -13.1170
Cluster 2: -30.1957 40.3520
Cluster 3: -4.4608 47.2543
Cluster 4: 46.3738 36.5225
Cluster 5: 43.3134 -17.6443
Cluster 6: -0.9003 1.4042
Cluster 7: 7.3884 11.5584
Now, in order to classify/label each pixel to a cluster 1-7, I currently do the following (pseudo-code):
clusters;
for each x
for each y
ab = im(x,y,2:3);
dist = norm(ab - clusters); // norm of dist between ab and each cluster
[~, idx] = min(dist);
end
end
However, this is terribly slow (52 seconds) because of the image resolution and that I manually loop through each x and y.
Are there some built-in functions I can use that performs the same job? There must be.
To summarize: I need a classification method that classifies pixel images to an already defined set of clusters.
Approach #1
For a N x 2 sized points/pixels array, you can avoid permute as suggested in the other solution by Luis, which could slow down things a bit, to have a kind of "permute-unrolled" version of it and also let's bsxfun work towards a 2D array instead of a 3D array, which must be better with performance.
Thus, assuming clusters to be ordered as a N x 2 sized array, you may try this other bsxfun based approach -
%// Get a's and b's
im_a = im(:,:,2);
im_b = im(:,:,3);
%// Get the minimum indices that correspond to the cluster IDs
[~,idx] = min(bsxfun(#minus,im_a(:),clusters(:,1).').^2 + ...
bsxfun(#minus,im_b(:),clusters(:,2).').^2,[],2);
idx = reshape(idx,size(im,1),[]);
Approach #2
You can try out another approach that leverages fast matrix multiplication in MATLAB and is based on this smart solution -
d = 2; %// dimension of the problem size
im23 = reshape(im(:,:,2:3),[],2);
numA = size(im23,1);
numB = size(clusters,1);
A_ext = zeros(numA,3*d);
B_ext = zeros(numB,3*d);
for id = 1:d
A_ext(:,3*id-2:3*id) = [ones(numA,1), -2*im23(:,id), im23(:,id).^2 ];
B_ext(:,3*id-2:3*id) = [clusters(:,id).^2 , clusters(:,id), ones(numB,1)];
end
[~, idx] = min(A_ext * B_ext',[],2); %//'
idx = reshape(idx, size(im,1),[]); %// Desired IDs
What’s going on with the matrix multiplication based distance matrix calculation?
Let us consider two matrices A and B between whom we want to calculate the distance matrix. For the sake of an easier explanation that follows next, let us consider A as 3 x 2 and B as 4 x 2 sized arrays, thus indicating that we are working with X-Y points. If we had A as N x 3 and B as M x 3 sized arrays, then those would be X-Y-Z points.
Now, if we have to manually calculate the first element of the square of distance matrix, it would look like this –
first_element = ( A(1,1) – B(1,1) )^2 + ( A(1,2) – B(1,2) )^2
which would be –
first_element = A(1,1)^2 + B(1,1)^2 -2*A(1,1)* B(1,1) + ...
A(1,2)^2 + B(1,2)^2 -2*A(1,2)* B(1,2) … Equation (1)
Now, according to our proposed matrix multiplication, if you check the output of A_ext and B_ext after the loop in the earlier code ends, they would look like the following –
So, if you perform matrix multiplication between A_ext and transpose of B_ext, the first element of the product would be the sum of elementwise multiplication between the first rows of A_ext and B_ext, i.e. sum of these –
The result would be identical to the result obtained from Equation (1) earlier. This would continue for all the elements of A against all the elements of B that are in the same column as in A. Thus, we would end up with the complete squared distance matrix. That’s all there is!!
Vectorized Variations
Vectorized variations of the matrix multiplication based distance matrix calculations are possible, though there weren't any big performance improvements seen with them. Two such variations are listed next.
Variation #1
[nA,dim] = size(A);
nB = size(B,1);
A_ext = ones(nA,dim*3);
A_ext(:,2:3:end) = -2*A;
A_ext(:,3:3:end) = A.^2;
B_ext = ones(nB,dim*3);
B_ext(:,1:3:end) = B.^2;
B_ext(:,2:3:end) = B;
distmat = A_ext * B_ext.';
Variation #2
[nA,dim] = size(A);
nB = size(B,1);
A_ext = [ones(nA*dim,1) -2*A(:) A(:).^2];
B_ext = [B(:).^2 B(:) ones(nB*dim,1)];
A_ext = reshape(permute(reshape(A_ext,nA,dim,[]),[1 3 2]),nA,[]);
B_ext = reshape(permute(reshape(B_ext,nB,dim,[]),[1 3 2]),nB,[]);
distmat = A_ext * B_ext.';
So, these could be considered as experimental versions too.
Use pdist2 (Statistics Toolbox) to compute the distances in a vectorized manner:
ab = im(:,:,2:3); % // get A, B components
ab = reshape(ab, [size(im,1)*size(im,2) 2]); % // reshape into 2-column
dist = pdist2(clusters, ab); % // compute distances
[~, idx] = min(dist); % // find minimizer for each pixel
idx = reshape(idx, size(im,1), size(im,2)); % // reshape result
If you don't have the Statistics Toolbox, you can replace the third line by
dist = squeeze(sum(bsxfun(#minus, clusters, permute(ab, [3 2 1])).^2, 2));
This gives squared distance instead of distance, but for the purposes of minimizing it doesn't matter.
I'm working on an iris recognition algorithm that processes these kind of images into unique codes for identification and authentication purposes.
After filtering, intelligently thresholding, then finding edges in the image, the next step is obviously to fit circles to the pupil and iris. I've looked around the the technique to use is the circular Hough Transform. Here is the code for my implementation. Sorry about the cryptic variable names.
print "Populating Accumulator..."
# Loop over image rows
for x in range(w):
# Loop over image columns
for y in range(h):
# Only process black pixels
if inp[x,y] == 0:
# px,py = 0 means pupil, otherwise pupil center
if px == 0:
ra = r_min
rb = r_max
else:
rr = sqrt((px-x)*(px-x)+(py-y)*(py-y))
ra = int(rr-3)
rb = int(rr+3)
# a is the width of the image, b is the height
for _a in range(a):
for _b in range(b):
for _r in range(rb-ra):
s1 = x - (_a + a_min)
s2 = y - (_b + b_min)
r1 = _r + ra
if (s1 * s1 + s2 * s2 == r1 * r1):
new = acc[_a][_b][_r]
if new >= maxVotes:
maxVotes = new
print "Done"
# Average all circles with the most votes
for _a in range(a):
for _b in range(b):
for _r in range(r):
if acc[_a][_b][_r] >= maxVotes-1:
total_a += _a + a_min
total_b += _b + b_min
total_r += _r + r_min
amount += 1
top_a = total_a / amount
top_b = total_b / amount
top_r = total_r / amount
print top_a,top_b,top_r
This is written in python and uses the Python Imaging Library to do image processing. As you can see, this is a very naive brute force method of finding circles. It works, but takes several minutes. The basic idea is to draw circles from rmin to rmax wherever there is a black pixel (from thresholding and edge-detection), the build an accumulator array of the number of times a location on the image is "voted" on. Whichever x, y, and r has the most votes is the circle of interest. I tried to use the fact that the iris and pupil have about the same center (variables ra and rb) to reduce some of the complexity of the r loop, but the pupil detection takes so long that it doesn't matter.
Now, obviously my implementation is very naive. It uses a three dimensional parameter space (x, y, and r), which unfortunately makes it run slower than is acceptable. What kind of improvements can I make? Is there any way to reduce this to a two-dimensional parameter space? Is there a more efficient way of accessing and setting pixels that I'm not aware of?
On a side note, are there any other techniques for improving the overall runtime of this algorithm that I'm not aware of? Such as methods to approximate the maximum radius of the pupil or iris?
Note: I've tried to use OpenCV for this as well, but I could not tune the parameters enough to be consistently accurate.
Let me know if there's any other information that you need.
NOTE: Once again I misinterpreted my own code. It is technically 5-dimensional, but the 3-dimensional x,y,r loop only operates on black pixels.
Assuming you want the position of the circle rather than a measure of R.
If you have a decent estimate of the possible range of R then a common technique is to run the algorithm for a first guess of fixed R, adjust it and try again.
I am plotting a 7x7 pixel 'image' in MATLAB, using the imagesc command:
imagesc(conf_matrix, [0 1]);
This represents a confusion matrix, between seven different objects. I have a thumbnail picture of each of the seven objects that I would like to use as the axes tick labels. Is there an easy way to do this?
I don't know an easy way. The axes properties XtickLabel which determines the labels, can only be strings.
If you want a not-so-easy way, you could do something in the spirit of the following non-complete (in the sense of a non-complete solution) code, creating one label:
h = imagesc(rand(7,7));
axh = gca;
figh = gcf;
xticks = get(gca,'xtick');
yticks = get(gca,'ytick');
set(gca,'XTickLabel','');
set(gca,'YTickLabel','');
pos = get(axh,'position'); % position of current axes in parent figure
pic = imread('coins.png');
x = pos(1);
y = pos(2);
dlta = (pos(3)-pos(1)) / length(xticks); % square size in units of parant figure
% create image label
lblAx = axes('parent',figh,'position',[x+dlta/4,y-dlta/2,dlta/2,dlta/2]);
imagesc(pic,'parent',lblAx)
axis(lblAx,'off')
One problem is that the label will have the same colormap of the original image.
#Itmar Katz gives a solution very close to what I want to do, which I've marked as 'accepted'. In the meantime, I made this dirty solution using subplots, which I've given here for completeness. It only works up to a certain size input matrix though, and only displays well when the figure is square.
conf_mat = randn(5);
A = imread('peppers.png');
tick_images = {A, A, A, A, A};
n = length(conf_mat) + 1;
% plotting axis labels at left and top
for i = 1:(n-1)
subplot(n, n, i + 1);
imshow(tick_images{i});
subplot(n, n, i * n + 1);
imshow(tick_images{i});
end
% generating logical array for where the confusion matrix should be
idx = 1:(n*n);
idx(1:n) = 0;
idx(mod(idx, n)==1) = 0;
% plotting the confusion matrix
subplot(n, n, find(idx~=0));
imshow(conf_mat);
axis image
colormap(gray)