I need to process a raster file by scanning every point and radiating around it to find other points of interest. As you may know, a GIS raster file contains millions, if not tens or hundreds of millions of pixel points. I need a platform that will allow me to process this data efficiently. I am slightly familiar with vba but from what I see such image processing capabilities are beyond its scope.
You probably do not want to be starting by coding this from scratch; raster processing in GIS is almost as old as the hills. I suggest you look at something like QGIS as a starting point. If it or the many raster plugins don't do what you want it's time to start learning how to write your own; likely not in VBA.
You probably also want to discover the many helpful answers on gis.stackexchange.com
Related
I am currently working on a project that is utilizing traffic camera video streams and analyzing them. I have written an algorithm in Octave, a .m file, that is able to return the outlines of multiple cars as individual blobs, and its outline and center coordinates. Specifically, for the blob part, I am using BWLabel and BWBoundaries and applying it to a binary image that I have created where 1 is car and 0 is background, according to the rest of my algorithm. I have an image below that shows this.
http://imgur.com/25hgrUP
All of the blobs are cars, including the one with about 5 blobs next to each other. All of these blobs are one van, but the different colors and features have thrown off the detection system. Does anyone know of a way to easily combine all of these blobs that are in close proximity into one blob. I am talking about an existing algorithm or function that is already in Octave packages or Matlab toolboxes. If not, I will write the code from scratch and make it happen. This question was just a call to ask if there are pre-existing solutions, not a call to write code for me, unless you want to :).
Thanks for your help,
AeroVTP
You can solve this (to an extent) with morphological closing, in Matlab it's imclose. You'll need to be careful, though as noise that's too close may be included, and true blobs too far away may be excluded.
Although imclose is a good idea, running it just one time has a much "stronger" effect on the image than just running the 'erode' and 'dilate' functions multiple times. I personally ran the erode and dilate functions 5 times, in succession, to get the best results for the earlier image. Running the Dilate and Erode commands separately give more control for good image processing.
Imclose is just a function that applies the erode and dilate functions repeatedly. In Octave, the function to dilate and erode is
editedImage = bwmorph(initialImage, 'dilate', 5 %number of times to apply% );
editedImage = bwmorph(initialImage, 'erode', 5 %number of times to apply% );
I also invented my own commenting structure :).
Thanks to wbest for initial imClose idea.
I am new to d3 geo. My task is to make a map of Boston and add some interactive features to it.
So far I've been able to get an outline of Boston. But the base map should be comparable to something you'd see in Google Maps - it should have buildings, roads, street names and city names, rivers, etc. A basic geography that makes the region more familiar.
For now, I don't need to pan, and may have just two or three zoom states.
All the visualizations I've seen that overlay interactive features onto maps like this seem to use images for the underlying maps: windhistory, polymaps, google maps and more. So I guess my questions are:
Why do some maps use images for the "backdrop"? Is it just the easiest way to build on top of existing maps? Is it more performant?
If I go with the images approach, are there any limitations to the features I can add? I'm hoping to do things like windmaps, animations, heatmaps, etc.
What are the copyright implications for using images? I imagine the answer to this is, "depends on which images I use," but are there some standard libraries that have no strings attached? For example I know if I use Google Maps, I have to display their logo, there's an API limit, etc. Are there any standard sources that are completely open?
Are there any examples where geography is added purely through TopoJSON?
Sorry if some of these seem obvious, but I am completely new to maps and just don't know the standard practices. Thanks for any help!
A quick take on your questions. Hopefully someone with more mapping experience can give you more detail:
Why do some maps use images for the "backdrop"?
File size and computation time, mostly. Drawing complete maps with buildings, roads, and topography requires a lot of data and a lot of time for the browser to render it. If your browser DOM gets too complicated, it can slow down all interactions even after the original drawing.
If I go with the images approach, are there any limitations to the features I can add?
There's a reason most interactive maps use multiple layers. The background images are best for the underlying "lay of the land" type imagery, anything you want to be interactive should be on top with SVG.
What are the copyright implications for using images?
If you're using someone's images, you have to follow their licence. You might want to look at the OpenStreetMap project.
Are there any examples where geography is added purely through TopoJSON?
I suppose that depends on what you mean by "geography"; Mike Bostock has generated topoJSON for a variety of features based on US Atlas data.
As for whether it makes sense: TopoJSON encodes paths/boundaries directly, and encodes regions as the area enclosed by a set of boundaries. You could use it to encode streets and rivers and even building outlines, but you're not saving any file size relative regular GeoJSON because those paths generally aren't duplicated the way that region boundaries are. Relative to using image tiles, any improvement in file size would be countered with increased processing time.
I am wondering if there is a pre-existing algorithm/library/framework to compare two images to see if one is a re-sized version of the other? The programming language doesn't matter at this stage.
If there is nothing out there, I'd need to write something up. What I have thought of so far:
(Expensive) Resize the larger to the smaller and compare pixel by pixel.
Better yet, just resize a few random "areas" on the picture and compare. If they match, convert more, etc...
Break the image into a number of rows and columns and do some sort of parity math on the color values.
The problem I see with the first two ideas especially, is that there are different ways to re-size a picture in the first place, so the math will likely not work out the same at all. Some re-sizing adds blur, etc....
If anyone could point me to some good literature on this subject, that would be great. My googling turns up mostly shareware applications which is not what I want.
The goal is to have this running in the back of a webserver.
The best approach depends on the characteristics of the images you are comparing, what percentage of probability it is that the images are the same, and when they are different, are they typically off by a lot or could it be as minute as a single pixel difference?
If the answers to the above is that the images you need to compare will be completely random then going with the expensive solution, or some available package might be the best bet.
If it is that you know that the images are different more often than not, and that the images typically differ quite a lot, and you really want to hand-roll a solution you could implement some initial 'quick compare' steps that would be less expensive and that would quickly identify a lot of the cases where the images are different.
For example you could resize the larger image, then either compare pixel-by-pixel (or calculate a hash of the pixel values) only a 'diagonal line' of the image (top left pixel to bottom right pixel) and by doing so exclude differing images and only do the more expensive comparison for those that pass this test.
Or take a pre-set number of points at whatever is a 'good distribution' depending on the type of image and only do the more expensive comparison for those that pass this test.
If you know a lot about the images you will be comparing, they have known characteristics and they are different more often than they are the same, implementing a cheap 'quick elimination compare' along the lines of the above could be worthwhile.
You need to look into dHash algorithm for this.
I wrote a pure java library just for this few days back. You can feed it with directory path(includes sub-directory), and it will list the duplicate images in list with absolute path which you want to delete. Alternatively, you can use it to find all unique images in a directory too.
It used awt api internally, so can't be used for Android though. Since, imageIO has problem reading alot of new types of images, i am using twelve monkeys jar which is internally used.
https://github.com/srch07/Duplicate-Image-Finder-API
Jar with dependencies bundled internally can be downloaded from, https://github.com/srch07/Duplicate-Image-Finder-API/blob/master/archives/duplicate_image_finder_1.0.jar
The api can find duplicates among images of different sizes too.
I 'm trying to find an efficient way of acceptable complexity to
detect an object in an image so I can isolate it from its surroundings
segment that object to its sub-parts and label them so I can then fetch them at will
It's been 3 weeks since I entered the image processing world and I've read about so many algorithms (sift, snakes, more snakes, fourier-related, etc.), and heuristics that I don't know where to start and which one is "best" for what I'm trying to achieve. Having in mind that the image dataset in interest is a pretty large one, I don't even know if I should use some algorithm implemented in OpenCV or if I should implement one my own.
Summarize:
Which methodology should I focus on? Why?
Should I use OpenCV for that kind of stuff or is there some other 'better' alternative?
Thank you in advance.
EDIT -- More info regarding the datasets
Each dataset consists of 80K images of products sharing the same
concept e.g. t-shirts, watches, shoes
size
orientation (90% of them)
background (95% of them)
All pictures in each datasets look almost identical apart from the product itself, apparently. To make things a little more clear, let's consider only the 'watch dataset':
All the pictures in the set look almost exactly like this:
(again, apart form the watch itself). I want to extract the strap and the dial. The thing is that there are lots of different watch styles and therefore shapes. From what I've read so far, I think I need a template algorithm that allows bending and stretching so as to be able to match straps and dials of different styles.
Instead of creating three distinct templates (upper part of strap, lower part of strap, dial), it would be reasonable to create only one and segment it into 3 parts. That way, I would be confident enough that each part was detected with respect to each other as intended to e.g. the dial would not be detected below the lower part of the strap.
From all the algorithms/methodologies I've encountered, active shape|appearance model seem to be the most promising ones. Unfortunately, I haven't managed to find a descent implementation and I'm not confident enough that that's the best approach so as to go ahead and write one myself.
If anyone could point out what I should be really looking for (algorithm/heuristic/library/etc.), I would be more than grateful. If again you think my description was a bit vague, feel free to ask for a more detailed one.
From what you've said, here are a few things that pop up at first glance:
Simplest thing to do it binarize the image and do Connected Components using OpenCV or CvBlob library. For simple images with non-complex background this usually yeilds objects
HOwever, looking at your sample image, texture-based segmentation techniques may work better - the watch dial, the straps and the background are wisely variant in texture/roughness, and this could be an ideal way to separate them.
The roughness of a portion can be easily found by the Eigen transform (explained a bit on SO, check the link to the research paper provided there), then the Mean Shift filter can be applied on the output of the Eigen transform. This will give regions clearly separated according to texture. Both the pyramidal Mean Shift and finding eigenvalues by SVD are implemented in OpenCV, so unless you can optimize your own code its better (and easier) to use inbuilt functions (if present) as far as speed and efficiency is concerned.
I think I would turn the problem around. Instead of hunting for the dial, I would use a set of robust features from the watch to 'stitch' the target image onto a template. The first watch has a set of squares in the dial that are white, the second watch has a number of white circles. I would per type of watch:
Segment out the squares or circles in the dial. Segmentation steps can be tricky as they are usually both scale and light dependent
Estimate the centers or corners of the above found feature areas. These are the new feature points.
Use the Hungarian algorithm to match features between the template watch and the target watch. Alternatively, one can take the surroundings of each feature point in the original image and match these using cross correlation
Use matching features between the template and the target to estimate scaling, rotation and translation
Stitch the image
As the image is now in a known form, one can extract the regions simply via pre set coordinates
Does anyone has recommendation of data structures for relative large maps with high resolution, something like 400mile x 400mile with 10-15ft resolution. Using 2D array, that would be roughly 2Mx2M cells.
The map only needs to store the elevation and terrain (earth, water, rock, etc.), and I don't think storing tiles is a good strategy.
Thank you!
It depends on what you need to do with it: view it, store it, analyze it, etc...
One thing I can say, however, is that that file will be HUGE at your stated resolution, and you should consider splitting it up into at least a few tiles, even better at 1x1 mile tiles.
The list of raster formats supported by GDAL could serve as a good starting point for exploring various formats, keeping in mind that many software packages (GRASS, ArcGIS, etc. use GDAL to read and write most raster formats). Note also that some file formats have maximum sizes which may prevent you from using them with your very large file.
For analysis and non-viewable storage, HDF5 format might be of interest.
If you want people to see the data as a map over the web, then creating small image tile overlays will be the fastest approach to sharing such a large dataset.