Object Detection in an Image - algorithm

I want to detect some elements in an Image.
For this goal, i get the image and the specified element (like a nose) and from Pixel(0,0) start to search for my element.
But the software performance is awful because i traverse the pixels one by one.
I think i need some smart algorithm for this problem.
And maybe the machine learning algorithm useful for this.
What's your idea?

I would start with viola jones object detection framework.
This is a supervised learning technique, that allows you to detect any kind of object with high provavility.
(even though the article mainly refers to faces, but it is designed for general objects..).
If you chose this approach - your main chore is going to be to obtain a classified training set. You can later evaluate how good your algorithm is using cross-validation.
AFAIK, it is implemented in OpenCV library (I am not familiar with the library to offer help)

You can do a very fast cross correlation using the Fourier transformation of your image and search pattern
A good implementation is for example OpenCV's matchTemplate function
This will work best if your pattern always has the same rotation and scale accross your image.
If it does not, you can repeat the search with several scaled/rotated versions of your pattern.
One advantage of this approach is that no training phase is required.
Another, simpler approach that would work in particular with your pattern is this:
Use connected component labeling to identify blobs with the right number of white pixels to be the center rectangle of your element. This will eliminate all but a few false positives. Concentrate your search on the remaining few spots.
Again OpenCV has a nice Blob library for that sort of stuff.

If you're looking for simple geometric shapes in computer-generated images like the example you provided, then you don't need to bother with machine learning.
For example, here's one of the components you're trying to find in the original image:
(Image removed by request)
Assuming this component is always drawn at the same dimensions, the top and bottom lines are always going to be 21 pixels apart. You can narrow down your search space considerably by combining this image with a copy of itself shifted vertically by 21 pixels, and taking the lighter of the two images as the pixel value at each position.
(Image removed by request)
Similarly, the vertical lines at the left and right of this component are 47 pixels apart, so we can repeat this process with a 47px horizontal shift. This results in a vertical bar about 24px tall at the position of the component.
(Image removed by request)
You can detect these bars quite easily by looking for runs of black pixels between 22 and 26 pixels long in the vertical columns of the processed image. This will provide you with a short list of candidate positions where you can check for the presence of this component more thoroughly, e.g. by calculating a local 2D cross correlation.
Here are the results after processing the whole image. Reaching this stage should only take a few milliseconds.
(Image removed by request)

Related

Detecting hexagonal shapes in greyscale or binary image

For my bachelor thesis I need to analyse images taken in the ocean to count and measure the size of water particles.
my problem:
besides the wanted water particles, the images show hexagonal patches all over the image in:
- different sizes
- not regular shape
- different greyscale values
(Example image below!)
It is clear that these patches will falsify my image analysis concerning the size and number of particles.
For this reason this patches need to be detected and deleted somehow.
Since it will be just a little part of the work in my thesis, I don't want to spend much time in it and already tried classic ways like: (imageJ)
playing with the threshold (resulting in also deleting wanted water particles)
analyse image including the hexagonal patches and later sort out the biggest areas (the hexagonal patches have quite the biggest areas, but you will still have a lot of haxagons)
playing with filters: using gaussian filter on a duplicated image and subtract the copy from the original deletes many patches (in reducing the greyscale value) but also deletes little wanted water particles and so again falsifies the result
a more complicated and time consuming solution would be to use a implemented library in for example matlab or opencv to detect points, that describe the shapes.
but so far I could not find any code that fits my task.
Does anyone of you have created such a code I could use for my task or any other idea?
You can see a lot of hexagonal patches in different depths also.
the little spots with an greater pixel value are the wanted particles!
Image processing is quite an involved area so there are no hard and fast rules.
But if it was me I would 'Mask' the image. This involves either defining what you want to keep or remove as a pixel 'Mask'. You then scan the mask over the image recursively and compare the mask to the image portion selected. You then select or remove the section (depending on your method) if it meets your criterion.
One such example of a criteria would be the spatial and grey-scale error weighted against a likelihood function (eg Chi-squared, square mean error etc.) or a Normal distribution that you define the uncertainty..
Some food for thought
Maybe you can try with the Hough transform:
https://en.wikipedia.org/wiki/Hough_transform
Matlab have an built-in function, hough, wich implements this, but only works for lines. Maybe you can start from that and change it to recognize hexagons.

algorithm - warping image to another image and calculate similarity measure

I have a query on calculation of best matching point of one image to another image through intensity based registration. I'd like to have some comments on my algorithm:
Compute the warp matrix at this iteration
For every point of the image A,
2a. We warp the particular image A pixel coordinates with the warp matrix to image B
2b. Perform interpolation to get the corresponding intensity form image B if warped point coordinate is in image B.
2c. Calculate the similarity measure value between warped pixel A intensity and warped image B intensity
Cycle through every pixel in image A
Cycle through every possible rotation and translation
Would this be okay? Is there any relevant opencv code we can reference?
Comments on algorithm
Your algorithm appears good although you will have to be careful about:
Edge effects: You need to make sure that the algorithm does not favour matches where most of image A does not overlap image B. e.g. you may wish to compute the average similarity measure and constrain the transformation to make sure that at least 50% of pixels overlap.
Computational complexity. There may be a lot of possible translations and rotations to consider and this algorithm may be too slow in practice.
Type of warp. Depending on your application you may also need to consider perspective/lighting changes as well as translation and rotation.
Acceleration
A similar algorithm is commonly used in video encoders, although most will ignore rotations/perspective changes and just search for translations.
One approach that is quite commonly used is to do a gradient search for the best match. In other words, try tweaking the translation/rotation in a few different ways (e.g. left/right/up/down by 16 pixels) and pick the best match as your new starting point. Then repeat this process several times.
Once you are unable to improve the match, reduce the size of your tweaks and try again.
Alternative algorithms
Depending on your application you may want to consider some alternative methods:
Stereo matching. If your 2 images come from stereo camera then you only really need to search in one direction (and OpenCV provides useful methods to do this)
Known patterns. If you are able to place a known pattern (e.g. a chessboard) in both your images then it becomes a lot easier to register them (and OpenCV provides methods to find and register certain types of pattern)
Feature point matching. A common approach to image registration is to search for distinctive points (e.g. types of corner or more general places of interest) and then try to find matching distinctive points in the two images. For example, OpenCV contains functions to detect SURF features. Google has published a great paper on using this kind of approach in order to remove rolling shutter noise that I recommend reading.

simple case of optical flow

General: I'm hoping that the use-case I'm about to describe is a simple case of an optical flow problem and since I don't have much knowledge on the subject, I was wondering if anyone has any suggestions on how I can approach solving my problem.
Research I've already done: I have began reading the High Accuracy Optical Flow Estimation Based on a Theory for Warping paper and am planning on looking over the Particle Video paper. I have found a MATLAB High Accuracy Optical Flow implementation of optical flow. However, the papers (and the code) seem to describe concepts that are very involved and may require a lot of time for me to dig in and understand. I am hoping that the solution to my problem may be more simple.
Problem: I have a sequence of images. The images depict a material breakage process, where the material and background are black and the cracks are white. I am interested in traversing the sequence of images in reverse in an attempt to map all of the cracks that have formed in the breakage process to the first black image. You can think of the material as a large puzzle and I am trying to put the pieces back together in the reverse order that they broke.
In each image, there can be some cracks that are just emerging and/or some cracks that have been fully formed (and thus created a fragment). Throughout the breakage process, some fragments may separate and break further. The fragments can also move farther away from one another (the change is slight between subsequent frames).
Desired Output: All of the cracks/lines in the sequence mapped to the first image in the sequence.
Additional Notes: Images are available in grayscale format (i.e. original) as well as in binary format, where the cracks have been outlined in white and the background is completely black. See below for some image examples.
The top row shows the original images and the bottom row shows the binary images. As you can see, the crack that goes down the middle grows wider and wider as the image sequence progresses. Thus, the bottom crack moves together with the lower fragment. When traversing the sequence in reverse, I hope to algorithmically realize that the middle crack comes together as one (and map it correctly to the first image), and also map the bottom crack correctly, keeping its correct correspondence (size and position) with the bottom fragment.
A sequence typically contains about 30~40 images, so I've just shown the beginning subset. Also, although these images don't show it, it is possible that a particular image only contains the beginning of the crack (i.e. its initial appearance) and in subsequent images it gets longer and longer and may join with other cracks.
Language: Although not necessary, I would like to implement the solution using MATLAB (just because most of the other code that relates to the project has been done in MATLAB). However, if OpenCV may be easier, I am flexible in my language/library usage.
Any ideas are greatly appreciated.
Traverse forward rather than reverse, and don't use optical flow. Use the fracture lines to segment the black parts, track the centroid of each black segment over time. Whenever a new fracture line appears that cuts across a black segment, split the segment into two and continue tracking each segment separately.
From this you should be able to construct a tree structure representing the segmentation of the black parts over time. The fracture lines can be added as metadata to this tree, perhaps assigning fracture lines to the segment node in which they first appeared.
I would advise you to follow your initial idea of backtracking the cracks. Yo kind of know how the cracks look like so you can track all the points that belong to the crack. You just track all the white points with an optical flow tracker, start with Lukas-Kanade tracker and see where you get. The high-accuracy optical flow method is a global one and more general, I'll track all the pixels in the image trying to keep some smoothness everywhere. The LK is a local method that will just use a small window around each point to do the tracking. The problem is that appart from the cracks all the pixels are plain black so nothing to track there, you'll just waist time trying to track something that you can't track and you don't need to track.
If lines are very straight you might end up with what's called the aperture problem and you'll get inaccurate results. You can also try some shape fitting/deformation based on snakes.
I agree to damian. Most optical flow methods like the HAOF rely on the first-order taylor approximation of the intensity constancy constrian equation I(x,t)=I(x+v,t+dt). That mean the solution depends on image derivatives where the gradient determine the motion vector magnitude and angle i.e. you need a certain amount of texture. However the very low texture of your non-binarised images could be enough. You could try histogram equalization to increase the contrast of your input data but it is important to apply the same transformation for both input images. e.g. as follows:
cv::Mat equalizeMat(grayInp1.rows, grayInp1.cols * 2 , CV_8UC1);
grayInp1.copyTo(equalizeMat(cv::Rect(0,0,grayInp1.cols,grayInp1.rows)));
grayInp2.copyTo(equalizeMat(cv::Rect(grayInp1.cols,0,grayInp2.cols,grayInp2.rows)));
cv::equalizeHist(equalizeMat,equalizeMat);
equalizeMat(cv::Rect(0,0,grayInp1.cols,grayInp1.rows)).copyTo(grayInp1);
equalizeMat(cv::Rect(grayInp1.cols,0,grayInp2.cols,grayInp2.rows)).copyTo(grayInp2);
// estimate optical flow

Recognizing distortions in a regular grid

To give you some background as to what I'm doing: I'm trying to quantitatively record variations in flow of a compressible fluid via image analysis. One way to do this is to exploit the fact that the index of refraction of the fluid is directly related to its density. If you set up some kind of image behind the flow, the distortion in the image due to refractive index changes throughout the fluid field leads you to a density gradient, which helps to characterize the flow pattern.
I have a set of routines that do this successfully with a regular 2D pattern of dots. The dot pattern is slightly distorted, and by comparing the position of the dots in the distorted image with that in the non-distorted image, I get a displacement field, which is exactly what I need. The problem with this method is resolution. The resolution is limited to the number of dots in the field, and I'm exploring methods that give me more data.
One idea I've had is to use a regular grid of horizontal and vertical lines. This image will distort the same way, but instead of getting only the displacement of a dot, I'll have the continuous distortion of a grid. It seems like there must be some standard algorithm or procedure to compare one geometric grid to another and infer some kind of displacement field. Nonetheless, I haven't found anything like this in my research.
Does anyone have some ideas that might point me in the right direction? FYI, I am not a computer scientist -- I'm an engineer. I say that only because there may be some obvious approach I'm neglecting due to coming from a different field. But I can program. I'm using MATLAB, but I can read Python, C/C++, etc.
Here are examples of the type of images I'm working with:
Regular: Distorted:
--------
I think you are looking for the Digital Image Correlation algorithm.
Here you can see a demo.
Here is a Matlab Implementation.
From Wikipedia:
Digital Image Correlation and Tracking (DIC/DDIT) is an optical method that employs tracking & image registration techniques for accurate 2D and 3D measurements of changes in images. This is often used to measure deformation (engineering), displacement, and strain, but it is widely applied in many areas of science and engineering.
Edit
Here I applied the DIC algorithm to your distorted image using Mathematica, showing the relative displacements.
Edit
You may also easily identify the maximum displacement zone:
Edit
After some work (quite a bit, frankly) you can come up to something like this, representing the "displacement field", showing clearly that you are dealing with a vortex:
(Darker and bigger arrows means more displacement (velocity))
Post me a comment if you are interested in the Mathematica code for this one. I think my code is not going to help anybody else, so I omit posting it.
I would also suggest a line tracking algorithm would work well.
Simply start at the first pixel line of the image and start following each of the vertical lines downwards (You just need to start this at the first line to get the starting points. This can be done by a simple pattern that moves orthogonally to the gradient of that line, ergo follows a line. When you reach a crossing of a horizontal line you can measure that point (in x,y coordinates) and compare it to the corresponding crossing point in your distorted image.
Since your grid is regular you know that the n'th measured crossing point on the m'th vertical black line are corresponding in both images. Then you simply compare both points by computing their distance. Do this for each line on your grid and you will get, by how far each crossing point of the grid is distorted.
This following a line algorithm is also used in basic Edge linking algorithms or the Canny Edge detector.
(All this are just theoretic ideas and I cannot provide you with an algorithm to it. But I guess it should work easily on distorted images like you have there... but maybe it is helpful for you)

How do I locate black rectangles in a grid and extract the binary code from that

i'm working in a project to recognize a bit code from an image like this, where black rectangle represents 0 bit, and white (white space, not visible) 1 bit.
Somebody have any idea to process the image in order to extract this informations? My project is written in java, but any solution is accepted.
thanks all for support.
I'm not an expert in image processing, I try to apply Edge Detection using Canny Edge Detector Implementation, free java implementation find here. I used this complete image [http://img257.imageshack.us/img257/5323/colorimg.png], reduce it (scale factor = 0.4) to have fast processing and this is the result [http://img222.imageshack.us/img222/8255/colorimgout.png]. Now, how i can decode white rectangle with 0 bit value, and no rectangle with 1?
The image have 10 line X 16 columns. I don't use python, but i can try to convert it to Java.
Many thanks to support.
This is recognising good old OMR (optical mark recognition).
The solution varies depending on the quality and consistency of the data you get, so noise is important.
Using an image processing library will clearly help.
Simple case: No skew in the image and no stretch or shrinkage
Create a horizontal and vertical profile of the image. i.e. sum up values in all columns and all rows and store in arrays. for an image of MxN (width x height) you will have M cells in horizontal profile and N cells in vertical profile.
Use a thresholding to find out which cells are white (empty) and which are black. This assumes you will get at least a couple of entries in each row or column. So black cells will define a location of interest (where you will expect the marks).
Based on this, you can define in lozenges in the form and you get coordinates of lozenges (rectangles where you have marks) and then you just add up pixel values in each lozenge and based on the number, you can define if it has mark or not.
Case 2: Skew (slant in the image)
Use fourier (FFT) to find the slant value and then transform it.
Case 3: Stretch or shrink
Pretty much the same as 1 but noise is higher and reliability less.
Aliostad has made some good comments.
This is OMR and you will find it much easier to get good consistent results with a good image processing library. www.leptonica.com is a free open source 'C' library that would be a very good place to start. It could process the skew and thresholding tasks for you. Thresholding to B/W would be a good start.
Another option would be IEvolution - http://www.hi-components.com/nievolution.asp for .NET.
To be successful you will need some type of reference / registration marks to allow for skew and stretch especially if you are using document scanning or capturing from a camera image.
I am not familiar with Java, but in Python, you can use the imaging library to open the image. Then load the height and the widths, and segment the image into a grid accordingly, by Height/Rows and Width/Cols. Then, just look for black pixels in those regions, or whatever color PIL registers that black to be. This obviously relies on the grid like nature of the data.
Edit:
Doing Edge Detection may also be Fruitful. First apply an edge detection method like something from wikipedia. I have used the one found at archive.alwaysmovefast.com/basic-edge-detection-in-python.html. Then convert any grayscale value less than 180 (if you want the boxes darker just increase this value) into black and otherwise make it completely white. Then create bounding boxes, lines where the pixels are all white. If data isn't terribly skewed, then this should work pretty well, otherwise you may need to do more work. See here for the results: http://imm.io/2BLd
Edit2:
Denis, how large is your dataset and how large are the images? If you have thousands of these images, then it is not feasible to manually remove the borders (the red background and yellow bars). I think this is important to know before proceeding. Also, I think the prewitt edge detection may prove more useful in this case, since there appears to be less noise:
The previous method of segmenting may be applied, if you do preprocess to bin in the following manner, in which case you need only count the number of black or white pixels and threshold after some training samples.

Resources