matlab: efficient computation of local histograms within circular neighboorhoods - performance

I've an image over which I would like to compute a local histogram within a circular neighborhood. The size of the neighborhood is given by a radius. Although the code below does the job, it's computationally expensive. I run the profiler and the way I'm accessing to the pixels within the circular neighborhoods is already expensive.
Is there any sort of improvement/optimization based maybe on vectorization? Or for instance, storing the neighborhoods as columns?
I found a similar question in this post and the proposed solution is quite in the spirit of the code below, however the solution is still not appropriate to my case. Any ideas are really welcomed :-) Imagine for the moment, the image is binary, but the method should also ideally work with gray-level images :-)
[rows,cols] = size(img);
hist_img = zeros(rows, cols, 2);
[XX, YY] = meshgrid(1:cols, 1:rows);
for rr=1:rows
for cc=1:cols
distance = sqrt( (YY-rr).^2 + (XX-cc).^2 );
mask_radii = (distance <= radius);
bwresponses = img(mask_radii);
[nelems, ~] = histc(double(bwresponses),0:255);
% do some processing over the histogram
...
end
end
EDIT 1 Given the received feedback, I tried to update the solution. However, it's not yet correct
radius = sqrt(2.0);
disk = diskfilter(radius);
fun = #(x) histc( x(disk>0), min(x(:)):max(x(:)) );
output = im2col(im, size(disk), fun);
function disk = diskfilter(radius)
height = 2*ceil(radius)+1;
width = 2*ceil(radius)+1;
[XX,YY] = meshgrid(1:width,1:height);
dist = sqrt((XX-ceil(width/2)).^2+(YY-ceil(height/2)).^2);
circfilter = (dist <= radius);
end

Following on the technique I described in my answer to a similar question you could try to do the following:
compute the index offsets from a particular voxel that get you to all the neighbors within a radius
Determine which voxels have all neighbors at least radius away from the edge
Compute the neighbors for all these voxels
Generate your histograms for each neighborhood
It is not hard to vectorize this, but note that
It will be slow when the neighborhood is large
It involves generating an intermediate matrix that is NxM (N = voxels in image, M = voxels in neighborhood) which could get very large
Here is the code:
% generate histograms for neighborhood within radius r
A = rand(200,200,200);
radius = 2.5;
tic
sz=size(A);
[xx yy zz] = meshgrid(1:sz(2), 1:sz(1), 1:sz(3));
center = round(sz/2);
centerPoints = find((xx - center(1)).^2 + (yy - center(2)).^2 + (zz - center(3)).^2 < radius.^2);
centerIndex = sub2ind(sz, center(1), center(2), center(3));
% limit to just the points that are "far enough on the inside":
inside = find(xx > radius+1 & xx < sz(2) - radius & ...
yy > radius + 1 & yy < sz(1) - radius & ...
zz > radius + 1 & zz < sz(3) - radius);
offsets = centerPoints - centerIndex;
allPoints = 1:prod(sz);
insidePoints = allPoints(inside);
indices = bsxfun(#plus, offsets, insidePoints);
hh = histc(A(indices), 0:0.1:1); % <<<< modify to give you the histogram you want
toc
A 2D version of the same code (which might be all you need, and is considerably faster):
% generate histograms for neighborhood within radius r
A = rand(200,200);
radius = 2.5;
tic
sz=size(A);
[xx yy] = meshgrid(1:sz(2), 1:sz(1));
center = round(sz/2);
centerPoints = find((xx - center(1)).^2 + (yy - center(2)).^2 < radius.^2);
centerIndex = sub2ind(sz, center(1), center(2));
% limit to just the points that are "far enough on the inside":
inside = find(xx > radius+1 & xx < sz(2) - radius & ...
yy > radius + 1 & yy < sz(1) - radius);
offsets = centerPoints - centerIndex;
allPoints = 1:prod(sz);
insidePoints = allPoints(inside);
indices = bsxfun(#plus, offsets, insidePoints);
hh = histc(A(indices), 0:0.1:1); % <<<< modify to give you the histogram you want
toc

You're right, I don't think that colfilt can be used as you're not applying a filter. You'll have to check the correctness, but here's my attempt using im2col and your diskfilter function (I did remove the conversion to double so it now output logicals):
function circhist
% Example data
im = randi(256,20)-1;
% Ranges - I do this globally for the whole image rather than for each neighborhood
mini = min(im(:));
maxi = max(im(:));
edges = linspace(mini,maxi,20);
% Disk filter
radius = sqrt(2.0);
disk = diskfilter(radius); % Returns logical matrix
% Pad array with -1
im_pad = padarray(im, (size(disk)-1)/2, -1);
% Convert sliding neighborhoods to columns
B = im2col(im_pad, size(disk), 'sliding');
% Get elements from each column that correspond to disk (logical indexing)
C = B(disk(:), :);
% Apply histogram across columns to count number of elements
out = histc(C, edges)
% Display output
figure
imagesc(out)
h = colorbar;
ylabel(h,'Counts');
xlabel('Neighborhood #')
ylabel('Bins')
axis xy
function disk = diskfilter(radius)
height = 2*ceil(radius)+1;
width = 2*ceil(radius)+1;
[XX,YY] = meshgrid(1:width,1:height);
dist = sqrt((XX-ceil(width/2)).^2+(YY-ceil(height/2)).^2);
disk = (dist <= radius);
If you want to set your ranges (edges) based on each neighborhood then you'll need to make sure that the vector is always the same length if you want to build a big matrix (and then the rows of that matrix won't correspond to each other).
You should note that the shape of the disk returned by fspecial is not as circular as what you were using. It's meant to be used a smoothing/averaging filter so the edges are fuzzy (anti-aliased). Thus when you use ~=0 it will grab more pixels. It'd stick with your own function, which is faster anyways.

You could try processing with an opposite logic (as briefly explained in the comment)
hist = zeros(W+2*R, H+2*R, Q);
for i = 1:R+1;
for j = 1:R+1;
if ((i-R-1)^2+(j-R-1)^2 < R*R)
for q = 0:1:Q-1;
hist(i:i+W-1,j:j+H-1,q+1) += (image == q);
end
end
end
end

Related

Interp2 of image with transformed coordinates

I have 2 greyscale images that i am trying to align using scalar scaling 1 , rotation matrix [2,2] and translation vector [2,1]. I can calculate image1's transformed coordinates as
y = s*R*x + t;
Below the resulting images are shown.
The first image is image1 before transformation,
the second image is image1 (red) with attempted interpolation using interp2 shown on top of image2 (green)
The third image is when i manually insert the pixel values from image1 into an empty array (that has the same size as image2) using the transformed coordinates.
From this we can see that the coordinate transformation must have been successful, as the images are aligned although not perfectly (which is to be expected since only 2 coordinates were used in calculating s, R and t) .
How come interp2 is not producing a result more similar to when i manually insert pixel values?
Below the code for doing this is included:
Interpolation code
function [transformed_image] = interpolate_image(im_r,im_t,s,R,t)
[m,n] = size(im_t);
% doesn't help if i use get_grid that the other function is using here
[~, grid_xr, grid_yr] = get_ipgrid(im_r);
[x_t, grid_xt, grid_yt] = get_ipgrid(im_t);
y = s*R*x_t + t;
yx = reshape(y(1,:), m,n);
yy = reshape(y(2,:), m,n);
transformed_image = interp2(grid_xr, grid_yr, im_r, yx, yy, 'nearest');
end
function [x, grid_x, grid_y] = get_ipgrid(image)
[m,n] = size(image);
[grid_x,grid_y] = meshgrid(1:n,1:m);
x = [reshape(grid_x, 1, []); reshape(grid_y, 1, [])]; % X is [2xM*N] coordinate pairs
end
The manual code
function [transformed_image] = transform_image(im_r,im_t,s,R,t)
[m,n] = size(im_t);
[x_t, grid_xt, grid_yt] = get_grid(im_t);
y = s*R*x_t + t;
ymat = reshape(y',m,n,2);
yx = ymat(:,:,1);
yy = ymat(:,:,2);
transformed_image = zeros(m,n);
for i = 1:m
for j = 1:n
% make sure coordinates are inside
if (yx(i,j) < m & yy(i,j) < n & yx(i,j) > 0.5 & yy(i,j) > 0.5)
transformed_image(round(yx(i,j)),round(yy(i,j))) = im_r(i,j);
end
end
end
end
function [x, grid_x, grid_y] = get_grid(image)
[m,n] = size(image);
[grid_y,grid_x] = meshgrid(1:n,1:m);
x = [grid_x(:) grid_y(:)]'; % X is [2xM*N] coordinate pairs
end
Can anyone see what i'm doing wrong with interp2? I feel like i have tried everything
Turns out i got interpolation all wrong.
In my question i calculate the coordinates of im1 in im2.
However the way interpolation works is that i need to calculate the coordinates of im2 in im1 such that i can map the image as shown below.
This means that i also calculated the wrong s,R and t since they were used to transform im1 -> im2, where as i needed im2 -> im1. (this is also called the inverse transform). Below is the manual code, that is basically the same as interp2 with nearest neighbour interpolation
function [transformed_image] = transform_image(im_r,im_t,s,R,t)
[m,n] = size(im_t);
[x_t, grid_xt, grid_yt] = get_grid(im_t);
y = s*R*x_t + t;
ymat = reshape(y',m,n,2);
yx = ymat(:,:,1);
yy = ymat(:,:,2);
transformed_image = zeros(m,n);
for i = 1:m
for j = 1:n
% make sure coordinates are inside
if (yx(i,j) < m & yy(i,j) < n & yx(i,j) > 0.5 & yy(i,j) > 0.5)
transformed_image(i,j) = im_r(round(yx(i,j)),round(yy(i,j)));
end
end
end
end

How to rotate a 2D theoretical path to fit or be overlaid on a path from videoprocessing

I track the motion of an object from a video file in MATLAB and save the locations from each frame in a numberOfFrames x 2 array.
I know the theoretical path or intended path. When recording the movie the camera is at some unknown angle in space. Therefore, the path is skewed. The only information I have is the scaling between the pixels and millimeters by using the object diameter.
Now I would like to rotate the intended path, and move it around until it is overlaid on the tracked motion path.
I start with my theoretical path (Pth) then rotate it in 3-dimensions to "Pthr". After that I loop over each point in "M". And for each point in "M", I look for the closest point from "Pthr". Then, I repeat for the next point in "M". This probably has a problem of choosing the same point in "Pthr" for multiple points in "M".
I noticed this is sensitive to my initial guess and it gives terrible results.
Also, M is not a perfect path, since it is experimental measurements it is no where near perfect. measured vs. theoretical unrotated path
% M = [Mx,My], is location in x and y, from motion tracking.
% scale = 20; % pixels/mm, using the size of object
% I build the theoretical path (Pth) goes from to (0,0,0) to (0,3,0) to
% (3,3,0) to (3,0,0) to be approximately the same length as M
Pthup = linspace(0,3,num)';
Pthdwn = linspace(3,0,num)';
Pth0 = zeros(size(Pthup));
Pth3 = 3*ones(size(Pthup));
% Pth is approximately same length as M
Pth = scale*[Pth0 Pthup Pth0;Pthup Pth3 Pth0;Pth3 Pthdwn Pth0];
% using fmincon in matlab to minimize the sum of the square
lb = [145 0 -45 min(min(M)) min(min(M))]; %upper bound
ub = [180 90 45 max(max(M)) max(max(M))]; %lower bound
coro = [180 0 0 mean(Mx) mean(My)]; %initial guess
% initial guess (theta(x),theta(y),theta(z), shift in x, shift in y)
cnt = 0; er = 1;
while (abs(er)>0.1)
[const,fval] = fmincon(#(cor)findOrientation(cor,Pth,M),coro,[],[],[],[],lb,ub);
er = sum(const-coro);
coro = const;
cnt = 1+cnt;
if (cnt>50)
cnt = cnt;
break
end
end
%% function findOrientation keeps rotating Pth until it is closest to M
function [Eo] = findOrientation(cor,Pth,M)
% cor = [angle of rotations, center coordinate];(degrees, non-dimensiolaized in pixels)
% M = measured coordinates from movie in pixel
% coor: is output of the form [x-coordiante,y-coordinate, absolute distance from Center(i,:)]
% F = sum of least square, sum(coor(:,3))
%% Rotation of theoretical path about z,y,x and shifting in it in xy
thx = cor(1);
thy = cor(2);
thz = cor(3);
xy = cor([4:5]);
% T = [cosd(thn) -sind(thn);
% sind(thn) cosd(thn)]; %rotation matrix in 3D
Tz = [cosd(thz) -sind(thz) 0;
sind(thz) cosd(thz) 0;0 0 1]; %rotation matrix
Ty = [cosd(thy) 0 -sind(thy);0 1 0;
sind(thy) 0 cosd(thy)]; %rotation matrix
Tx = [1 0 0;0 cosd(thx) -sind(thx);
0 sind(thx) cosd(thx)]; %rotation matrix
Pthr = zeros(size(Pth));
for i = 1:size(Pth,1)
xp = Tz*Pth(i,:)';
xp = Ty*xp;
xp = Tx*xp;
Pthr(i,:) = xp.';
end
Pthr = Pthr(:,[1,2]); % omit third value because it is 2D
Pthr = Pthr + [cor([4:5])];
rin = sqrt(Pthr(:,1).^2+Pthr(:,2).^2); %theoretical radius
Centern = sqrt(M(:,1).^2 + M(:,2).^2);%measured radius
for i = 1:size(M,1) %loop over each point in tracked motion
sub = Pthr-M(i,:); %subtracting M(i,:) from all Pthr
for j = 1:length(sub)
dist(j,1) = norm(sub(j,:));% distance from M(i,:) to all ri
end
%index is based on the min absolute distance between Pthr and M(i,:). It chooses the closest Pthr to a specific M(i,:)
[mn, index] = min(dist);
erri = abs(rin(index)-Centern(i))./rin(index);
coor(i,:) = erri;
end
Eo = sum(coor);

Check for pixel values in a neighborhood

I'm trying to write a MATLAB script that does the following:
Given: pixel coordinates(x,y) for a .jpg image
Goal: Check, within a 5 pixel radius of given coordinates, if there is a pixel of a certain value.
For example, let's say I'm given the coordinates (100,100), then I want to check the neighborhood of (100,100) within my image for any pixels that are black (0,0,0). So perhaps, pixel (103, 100) and (104,100) might have the value (0,0,0).
Current code:
x_coord = uint32(coord(:,1));
y_coord = uint32(coord(:,2));
count = 0;
for i = 1:length(x_coord)
%(img(x,y) returns pixel value at that (x,y)
%Note 0 = black. Indicating that, at that position, the image is just
% black
if img(x_coord(i),y_coord(i)) == 0
count = count + 1;
end
end
It currently only checks at an exact location. Not in a local neighborhood. How to could I extend this?
EDIT: Also note, as long as there as at least one pixel in the neighborhood with the value, I increment count. I'm not trying to enumerate how many pixels in the neighborhood have that value, just trying to find evidence of at least one pixel that has that value.
EDIT:
Even though I am unable to identify an error with the code, I am not able to get the exact results I want. Here is the code I am using.
val = 0; %pixel value to check
N = 50; % neighbourhood radius
%2D grid of coordinates surrounding center coordinate
[R, C] = ndgrid(1 : size(img, 1), 1 : size(img, 2));
for kk = 1 : size(coord, 1)
r = coord(kk, 1); c = coord(kk, 2); % Get pixel locations
% mask of valid locations within the neighbourhood (avoid boundary problems)
mask = (R - r).^2 + (C - c).^2 <= N*N;
pix = img(mask); % Get the valid pixels
valid = any(pix(:) ~= val);
% Add either 0 or 1 depending if we have found any matching pixels
if(valid == 1)
img = insertMarker(img, [r c], 'x', 'color', 'red', 'size', 10);
imwrite(img, images(i).name,'tiff');
end
count = count + valid;
end
An easier way to do this would be to use indexing to grab a neighbourhood, then to check to see if any of the pixels in the neighbourhood have the value that you're looking for, use any on a flattened version of this neighbourhood. The trick with grabbing the right neighbourhood is to first generate a 2D grid of coordinates that span the entire dimensions of your image, then simply use the equation of a circle with the centre of it being each coordinate you are looking at and determine those locations that satisfy the following equation:
(x - a)^2 + (y - b)^2 <= N^2
N is the radius of the observation window, (a, b) is a coordinate of interest while (x, y) is a coordinate in the image. Use meshgrid to generate the coordinates.
You would use the above equation to create a logical mask, index into your image to pull the locations that are valid within the mask and check how many pixels match the one you want. Another added benefit with the above approach is that you are not subject to any out of bounds errors. Because you are pre-generating the list of all valid coordinates in your image, generating the mask will confine you within the boundaries of the image so you never have to check for out of boundaries conditions.... even when you specify coordinates to search that are out of bounds.
Specifically, assuming your image is stored in img, you would do:
count = 0; % Remembers total count of pixels matching a value
val = 0; % Value to match
N = 50; % Radius of neighbourhood
% Generate 2D grid of coordinates
[x, y] = meshgrid(1 : size(img, 2), 1 : size(img, 1));
% For each coordinate to check...
for kk = 1 : size(coord, 1)
a = coord(kk, 1); b = coord(kk, 2); % Get the pixel locations
mask = (x - a).^2 + (y - b).^2 <= N*N; % Get a mask of valid locations
% within the neighbourhood
pix = img(mask); % Get the valid pixels
count = count + any(pix(:) == val); % Add either 0 or 1 depending if
% we have found any matching pixels
end
The proposed solution:
fc = repmat(-5:5,11,1);
I = (fc.^2+fc'.^2)<=25;
fc_x = fc(I);
fc_y = fc'; fc_y = fc_y(I);
for i = 1:length(x_coord)
x_toCheck = fc_x + x_coord(i);
y_toCheck = fc_y + y_coord(i);
I = x_toCheck>0 & x_toCheck<=yourImageWidth;
I = I.*(y_toCheck>0 & y_toCheck<=yourImageHeight);
x_toCheck = x_toCheck(logical(I));
y_toCheck = y_toCheck(logical(I));
count = sum(img(x_toCheck(:),y_toCheck(:)) == 0);
end
If your img function can only check one pixel at a time, just add a for loop:
for i = 1:length(x_coord)
x_toCheck = fc_x + x_coord(i);
y_toCheck = fc_y + y_coord(i);
I = x_toCheck>0 & x_toCheck<=yourImageWidth;
I = I.*(y_toCheck>0 & y_toCheck<=yourImageHeight);
x_toCheck = x_toCheck(logical(I));
y_toCheck = y_toCheck(logical(I));
for j = 1:length(x_toCheck)
count = count + (img(x_toCheck(j),y_toCheck(j)) == 0);
end
end
Step-by-step:
You first need to get all the coordinates within 5 pixels range of the given coordinate.
We start by building a square of 11 pixels in length/width.
fc = repmat(-5:5,11,1);
fc_x = fc;
fc_y = fc';
plot(fc_x,fc_y,'.');
We now need to build a filter to get rid of those points outside the 5-pixel radius.
I = (fc.^2+fc'.^2)<=25;
Apply the filter, so we can get a circle of 5-pixel radius.
fc_x = fc_x(I);
fc_y = fc_y(I);
Next translate the centre of the circle to the given coordinate:
x_toCheck = fc_x + x_coord(i);
y_toCheck = fc_y + y_coord(i);
You need to check whether part of the circle is outside the range of your image:
I = x_toCheck>0 & x_toCheck<=yourImageWidth;
I = I.*(y_toCheck>0 & y_toCheck<=yourImageHeight);
x_toCheck = x_toCheck(logical(I));
y_toCheck = y_toCheck(logical(I));
Finally count the pixels:
count = sum(img(x_toCheck,y_toCheck) == 0);

Remove barrel distortion from an image in MATLAB [duplicate]

BOUNTY STATUS UPDATE:
I discovered how to map a linear lens, from destination coordinates to source coordinates.
How do you calculate the radial distance from the centre to go from fisheye to rectilinear?
1). I actually struggle to reverse it, and to map source coordinates to destination coordinates. What is the inverse, in code in the style of the converting functions I posted?
2). I also see that my undistortion is imperfect on some lenses - presumably those that are not strictly linear. What is the equivalent to-and-from source-and-destination coordinates for those lenses? Again, more code than just mathematical formulae please...
Question as originally stated:
I have some points that describe positions in a picture taken with a fisheye lens.
I want to convert these points to rectilinear coordinates. I want to undistort the image.
I've found this description of how to generate a fisheye effect, but not how to reverse it.
There's also a blog post that describes how to use tools to do it; these pictures are from that:
(1) : SOURCE Original photo link
Input : Original image with fish-eye distortion to fix.
(2) : DESTINATION Original photo link
Output : Corrected image (technically also with perspective correction, but that's a separate step).
How do you calculate the radial distance from the centre to go from fisheye to rectilinear?
My function stub looks like this:
Point correct_fisheye(const Point& p,const Size& img) {
// to polar
const Point centre = {img.width/2,img.height/2};
const Point rel = {p.x-centre.x,p.y-centre.y};
const double theta = atan2(rel.y,rel.x);
double R = sqrt((rel.x*rel.x)+(rel.y*rel.y));
// fisheye undistortion in here please
//... change R ...
// back to rectangular
const Point ret = Point(centre.x+R*cos(theta),centre.y+R*sin(theta));
fprintf(stderr,"(%d,%d) in (%d,%d) = %f,%f = (%d,%d)\n",p.x,p.y,img.width,img.height,theta,R,ret.x,ret.y);
return ret;
}
Alternatively, I could somehow convert the image from fisheye to rectilinear before finding the points, but I'm completely befuddled by the OpenCV documentation. Is there a straightforward way to do it in OpenCV, and does it perform well enough to do it to a live video feed?
The description you mention states that the projection by a pin-hole camera (one that does not introduce lens distortion) is modeled by
R_u = f*tan(theta)
and the projection by common fisheye lens cameras (that is, distorted) is modeled by
R_d = 2*f*sin(theta/2)
You already know R_d and theta and if you knew the camera's focal length (represented by f) then correcting the image would amount to computing R_u in terms of R_d and theta. In other words,
R_u = f*tan(2*asin(R_d/(2*f)))
is the formula you're looking for. Estimating the focal length f can be solved by calibrating the camera or other means such as letting the user provide feedback on how well the image is corrected or using knowledge from the original scene.
In order to solve the same problem using OpenCV, you would have to obtain the camera's intrinsic parameters and lens distortion coefficients. See, for example, Chapter 11 of Learning OpenCV (don't forget to check the correction). Then you can use a program such as this one (written with the Python bindings for OpenCV) in order to reverse lens distortion:
#!/usr/bin/python
# ./undistort 0_0000.jpg 1367.451167 1367.451167 0 0 -0.246065 0.193617 -0.002004 -0.002056
import sys
import cv
def main(argv):
if len(argv) < 10:
print 'Usage: %s input-file fx fy cx cy k1 k2 p1 p2 output-file' % argv[0]
sys.exit(-1)
src = argv[1]
fx, fy, cx, cy, k1, k2, p1, p2, output = argv[2:]
intrinsics = cv.CreateMat(3, 3, cv.CV_64FC1)
cv.Zero(intrinsics)
intrinsics[0, 0] = float(fx)
intrinsics[1, 1] = float(fy)
intrinsics[2, 2] = 1.0
intrinsics[0, 2] = float(cx)
intrinsics[1, 2] = float(cy)
dist_coeffs = cv.CreateMat(1, 4, cv.CV_64FC1)
cv.Zero(dist_coeffs)
dist_coeffs[0, 0] = float(k1)
dist_coeffs[0, 1] = float(k2)
dist_coeffs[0, 2] = float(p1)
dist_coeffs[0, 3] = float(p2)
src = cv.LoadImage(src)
dst = cv.CreateImage(cv.GetSize(src), src.depth, src.nChannels)
mapx = cv.CreateImage(cv.GetSize(src), cv.IPL_DEPTH_32F, 1)
mapy = cv.CreateImage(cv.GetSize(src), cv.IPL_DEPTH_32F, 1)
cv.InitUndistortMap(intrinsics, dist_coeffs, mapx, mapy)
cv.Remap(src, dst, mapx, mapy, cv.CV_INTER_LINEAR + cv.CV_WARP_FILL_OUTLIERS, cv.ScalarAll(0))
# cv.Undistort2(src, dst, intrinsics, dist_coeffs)
cv.SaveImage(output, dst)
if __name__ == '__main__':
main(sys.argv)
Also note that OpenCV uses a very different lens distortion model to the one in the web page you linked to.
(Original poster, providing an alternative)
The following function maps destination (rectilinear) coordinates to source (fisheye-distorted) coordinates. (I'd appreciate help in reversing it)
I got to this point through trial-and-error: I don't fundamentally grasp why this code is working, explanations and improved accuracy appreciated!
def dist(x,y):
return sqrt(x*x+y*y)
def correct_fisheye(src_size,dest_size,dx,dy,factor):
""" returns a tuple of source coordinates (sx,sy)
(note: values can be out of range)"""
# convert dx,dy to relative coordinates
rx, ry = dx-(dest_size[0]/2), dy-(dest_size[1]/2)
# calc theta
r = dist(rx,ry)/(dist(src_size[0],src_size[1])/factor)
if 0==r:
theta = 1.0
else:
theta = atan(r)/r
# back to absolute coordinates
sx, sy = (src_size[0]/2)+theta*rx, (src_size[1]/2)+theta*ry
# done
return (int(round(sx)),int(round(sy)))
When used with a factor of 3.0, it successfully undistorts the images used as examples (I made no attempt at quality interpolation):
Dead link
(And this is from the blog post, for comparison:)
If you think your formulas are exact, you can comput an exact formula with trig, like so:
Rin = 2 f sin(w/2) -> sin(w/2)= Rin/2f
Rout= f tan(w) -> tan(w)= Rout/f
(Rin/2f)^2 = [sin(w/2)]^2 = (1 - cos(w))/2 -> cos(w) = 1 - 2(Rin/2f)^2
(Rout/f)^2 = [tan(w)]^2 = 1/[cos(w)]^2 - 1
-> (Rout/f)^2 = 1/(1-2[Rin/2f]^2)^2 - 1
However, as #jmbr says, the actual camera distortion will depend on the lens and the zoom. Rather than rely on a fixed formula, you might want to try a polynomial expansion:
Rout = Rin*(1 + A*Rin^2 + B*Rin^4 + ...)
By tweaking first A, then higher-order coefficients, you can compute any reasonable local function (the form of the expansion takes advantage of the symmetry of the problem). In particular, it should be possible to compute initial coefficients to approximate the theoretical function above.
Also, for good results, you will need to use an interpolation filter to generate your corrected image. As long as the distortion is not too great, you can use the kind of filter you would use to rescale the image linearly without much problem.
Edit: as per your request, the equivalent scaling factor for the above formula:
(Rout/f)^2 = 1/(1-2[Rin/2f]^2)^2 - 1
-> Rout/f = [Rin/f] * sqrt(1-[Rin/f]^2/4)/(1-[Rin/f]^2/2)
If you plot the above formula alongside tan(Rin/f), you can see that they are very similar in shape. Basically, distortion from the tangent becomes severe before sin(w) becomes much different from w.
The inverse formula should be something like:
Rin/f = [Rout/f] / sqrt( sqrt(([Rout/f]^2+1) * (sqrt([Rout/f]^2+1) + 1) / 2 )
I blindly implemented the formulas from here, so I cannot guarantee it would do what you need.
Use auto_zoom to get the value for the zoom parameter.
def dist(x,y):
return sqrt(x*x+y*y)
def fisheye_to_rectilinear(src_size,dest_size,sx,sy,crop_factor,zoom):
""" returns a tuple of dest coordinates (dx,dy)
(note: values can be out of range)
crop_factor is ratio of sphere diameter to diagonal of the source image"""
# convert sx,sy to relative coordinates
rx, ry = sx-(src_size[0]/2), sy-(src_size[1]/2)
r = dist(rx,ry)
# focal distance = radius of the sphere
pi = 3.1415926535
f = dist(src_size[0],src_size[1])*factor/pi
# calc theta 1) linear mapping (older Nikon)
theta = r / f
# calc theta 2) nonlinear mapping
# theta = asin ( r / ( 2 * f ) ) * 2
# calc new radius
nr = tan(theta) * zoom
# back to absolute coordinates
dx, dy = (dest_size[0]/2)+rx/r*nr, (dest_size[1]/2)+ry/r*nr
# done
return (int(round(dx)),int(round(dy)))
def fisheye_auto_zoom(src_size,dest_size,crop_factor):
""" calculate zoom such that left edge of source image matches left edge of dest image """
# Try to see what happens with zoom=1
dx, dy = fisheye_to_rectilinear(src_size, dest_size, 0, src_size[1]/2, crop_factor, 1)
# Calculate zoom so the result is what we wanted
obtained_r = dest_size[0]/2 - dx
required_r = dest_size[0]/2
zoom = required_r / obtained_r
return zoom
I took what JMBR did and basically reversed it. He took the radius of the distorted image (Rd, that is, the distance in pixels from the center of the image) and found a formula for Ru, the radius of the undistorted image.
You want to go the other way. For each pixel in the undistorted (processed image), you want to know what the corresponding pixel is in the distorted image.
In other words, given (xu, yu) --> (xd, yd). You then replace each pixel in the undistorted image with its corresponding pixel from the distorted image.
Starting where JMBR did, I do the reverse, finding Rd as a function of Ru. I get:
Rd = f * sqrt(2) * sqrt( 1 - 1/sqrt(r^2 +1))
where f is the focal length in pixels (I'll explain later), and r = Ru/f.
The focal length for my camera was 2.5 mm. The size of each pixel on my CCD was 6 um square. f was therefore 2500/6 = 417 pixels. This can be found by trial and error.
Finding Rd allows you to find the corresponding pixel in the distorted image using polar coordinates.
The angle of each pixel from the center point is the same:
theta = arctan( (yu-yc)/(xu-xc) ) where xc, yc are the center points.
Then,
xd = Rd * cos(theta) + xc
yd = Rd * sin(theta) + yc
Make sure you know which quadrant you are in.
Here is the C# code I used
public class Analyzer
{
private ArrayList mFisheyeCorrect;
private int mFELimit = 1500;
private double mScaleFESize = 0.9;
public Analyzer()
{
//A lookup table so we don't have to calculate Rdistorted over and over
//The values will be multiplied by focal length in pixels to
//get the Rdistorted
mFisheyeCorrect = new ArrayList(mFELimit);
//i corresponds to Rundist/focalLengthInPixels * 1000 (to get integers)
for (int i = 0; i < mFELimit; i++)
{
double result = Math.Sqrt(1 - 1 / Math.Sqrt(1.0 + (double)i * i / 1000000.0)) * 1.4142136;
mFisheyeCorrect.Add(result);
}
}
public Bitmap RemoveFisheye(ref Bitmap aImage, double aFocalLinPixels)
{
Bitmap correctedImage = new Bitmap(aImage.Width, aImage.Height);
//The center points of the image
double xc = aImage.Width / 2.0;
double yc = aImage.Height / 2.0;
Boolean xpos, ypos;
//Move through the pixels in the corrected image;
//set to corresponding pixels in distorted image
for (int i = 0; i < correctedImage.Width; i++)
{
for (int j = 0; j < correctedImage.Height; j++)
{
//which quadrant are we in?
xpos = i > xc;
ypos = j > yc;
//Find the distance from the center
double xdif = i-xc;
double ydif = j-yc;
//The distance squared
double Rusquare = xdif * xdif + ydif * ydif;
//the angle from the center
double theta = Math.Atan2(ydif, xdif);
//find index for lookup table
int index = (int)(Math.Sqrt(Rusquare) / aFocalLinPixels * 1000);
if (index >= mFELimit) index = mFELimit - 1;
//calculated Rdistorted
double Rd = aFocalLinPixels * (double)mFisheyeCorrect[index]
/mScaleFESize;
//calculate x and y distances
double xdelta = Math.Abs(Rd*Math.Cos(theta));
double ydelta = Math.Abs(Rd * Math.Sin(theta));
//convert to pixel coordinates
int xd = (int)(xc + (xpos ? xdelta : -xdelta));
int yd = (int)(yc + (ypos ? ydelta : -ydelta));
xd = Math.Max(0, Math.Min(xd, aImage.Width-1));
yd = Math.Max(0, Math.Min(yd, aImage.Height-1));
//set the corrected pixel value from the distorted image
correctedImage.SetPixel(i, j, aImage.GetPixel(xd, yd));
}
}
return correctedImage;
}
}
I found this pdf file and I have proved that the maths are correct (except for the line vd = *xd**fv+v0 which should say vd = **yd**+fv+v0).
http://perception.inrialpes.fr/CAVA_Dataset/Site/files/Calibration_OpenCV.pdf
It does not use all of the latest co-efficients that OpenCV has available but I am sure that it could be adapted fairly easily.
double k1 = cameraIntrinsic.distortion[0];
double k2 = cameraIntrinsic.distortion[1];
double p1 = cameraIntrinsic.distortion[2];
double p2 = cameraIntrinsic.distortion[3];
double k3 = cameraIntrinsic.distortion[4];
double fu = cameraIntrinsic.focalLength[0];
double fv = cameraIntrinsic.focalLength[1];
double u0 = cameraIntrinsic.principalPoint[0];
double v0 = cameraIntrinsic.principalPoint[1];
double u, v;
u = thisPoint->x; // the undistorted point
v = thisPoint->y;
double x = ( u - u0 )/fu;
double y = ( v - v0 )/fv;
double r2 = (x*x) + (y*y);
double r4 = r2*r2;
double cDist = 1 + (k1*r2) + (k2*r4);
double xr = x*cDist;
double yr = y*cDist;
double a1 = 2*x*y;
double a2 = r2 + (2*(x*x));
double a3 = r2 + (2*(y*y));
double dx = (a1*p1) + (a2*p2);
double dy = (a3*p1) + (a1*p2);
double xd = xr + dx;
double yd = yr + dy;
double ud = (xd*fu) + u0;
double vd = (yd*fv) + v0;
thisPoint->x = ud; // the distorted point
thisPoint->y = vd;
This can be solved as an optimization problem. Simply draw on curves in images that are supposed to be straight lines. Store the contour points for each of those curves. Now we can solve the fish eye matrix as a minimization problem. Minimize the curve in points and that will give us a fisheye matrix. It works.
It can be done manually by adjusting the fish eye matrix using trackbars! Here is a fish eye GUI code using OpenCV for manual calibration.

Image with accordion effect

I have read in an image file to MATLAB and I am trying to stretch it in one direction, but a variable amount (sinusoidal). This would create an accordion affect on the image. I have toyed around with imresize, however that only resizes the image linearly. I would like the amount of "stretch" to vary for each image line. I tried to convey this with the following code:
periods = 10; % Number of "stretch" cycles
sz = size(original_image,2)/periods;
s = 0;
x = 0;
for index = 1:periods
B = original_image(:,round(s+1:s+sz));
if mod(index,2) == 0
amp = 1.5;
else
amp = 0.75;
end
xi = size(B,2)*amp;
new_image(:,x+1:x+xi) = imresize(B, [size(B,1) size(B,2)*amp]);
s = s + sz;
x = x+xi;
end
You can see that segments of the image are stretched, then compressed, then stretched, etc, like an accordion. However, each segment has a uniform amount of stretch, whereas I'd like it to be increasing then decreasing as you move along the image.
I have also looked at MATLAB's example of Applying a Sinusoidal Transformation to a Checkerboard which seems very applicable to my problem, however I have been trying and I cannot get this to produce the desired result for my image.
Any help is much appreciated.
UPDATE:
Thank you for Answer #1. I was unable to get it to work for me, but also realized it would resulted in loss of data, as the code only called for certian lines in the original image, and other lines would have been ignored.
After experimenting further, I developed the code below. I used a checkerboard as an example. While combersome, it does get the job done. However, upon trying the script with an actual high-resolution image, it was extremely slow and ended up failing due to running out of memory. I believe this is because of the excessive number of "imresize" commands that are used in loop.
I = checkerboard(10,50);
I = imrotate(I,90);
[X Y] = size(I);
k = 4; % Number of "cycles"
k = k*2;
x = 1;
y = 2;
z = 2;
F = [];
i = 1;
t = 0;
s = 0;
for j = 1:k/2
t = t + 1;
for inc = round(s+1):round(Y/k*t)
Yi = i + 1;
F(:,(x:y)) = imresize(I(:,(inc:inc)),[X Yi]);
x = y + 1;
y = x + z;
z = z + 1;
i = i + 1;
end
y = y - 2;
z = z - 4;
for inc = round(Y/k*t+1):round(Y/k*(t+1))
Yi = i - 1;
F(:,(x:y)) = imresize(I(:,(inc:inc)),[X Yi]);
x = y + 1;
y = x + z;
z = z - 1;
i = i - 1;
end
y = y + 2;
z = z + 4;
s = Y/k*(t+1);
t = t + 1;
end
Fn = imresize(F, [X Y]);
imshow(Fn);
Does anyone know of a simpler way to achieve this? If you run the code above, you can see the effect I am trying to achieve. Unfortunately, my method above does not allow me to adjust the amplitude of the "stretch" either, only the number of "cycles," or frequency. Help on this would also be appreciated. Much thanks!
Here is how I would approach it:
Determine how the coordinate of each point in your Final image F maps into your Initial image I of size (M,N)
Since you want to stretch horizontally only, given a point (xF,yF) in your final image, that point would be (xI,yI) in your initial image where xI and yI can be obtained as follows:
yI = yF;
xI = xF + Lsin(xFK);
Notes:
these equations do not guarantee that xI remains within the range [1:N] so cropping needs to be added
K controls the how many wrinkles you want to have in your accordion effect. For example, if you only want one wrinkle, K would be 2*pi/N
L controls how much stretching you want to apply
Then simply express your image F from image I with the transforms you have in 1.
Putting it all together, the code below creates a sample image I and generates the image F as follows:
% Generate a sample input image
N=500;
xF=1:N;
I=(1:4)'*xF/N*50;
% Set the parameters for your accordion transform
K=2*pi/N;
L=100;
% Apply the transform
F=I(:, round(min(N*ones(1,N), max(ones(1,N), (xF + L*sin(xF*K))))) );
% Display the input and output images side by side
image(I);
figure;
image(F);
If you run this exact code you get:
As you can see, the final image on the right stretches the center part of the image on the left, giving you an accordion effect with one wrinkle.
You can fiddle with K and L and adjust the formula to get the exact effect you want, but note how by expressing the transform in a matrix form MATLAB executes the code in a fraction of second. If there is one take away for you is that you should stay away from for loops and complex processing whenever you can.
Have fun!

Resources