Related
I am solving a optimization problem using a european electric grid with 9241 nodes. To do that I created a connectivity matrix and inserted it in an Excel file. Using code from smaller problems (less nodes), which takes the information from the Excel file to write a CPLEX matrix, I tried to apply it to this bigger problem and I get the following error:
Exception from IBM ILOG Concert: Can not read data from excel. PEGASE-9241data.dat /Teste 8:36-37 E:\Program Files\IBM\ILOG\CPLEX_Studio129\opl\oplide\workspace\Teste\PEGASE-9241data.dat OPL Problem Marker
Some important things to know:
- The Excel file is .xlsx
- I wrote the connectivity matrix into the Excel file using Matlab (worked for smaller problems)
When I try the same code with a smaller problem (3120 nodes) It works fine.
Model:
range n = 1..9241;
dvar boolean x[n];
dvar boolean y[n][n];
//Conectivity matrix
int A[n][n] =...;
//Vector ZIB
int z[n] =...;
//Objective Function
dexpr int total = sum (p in n) x[p];
minimize (total);
subject to {
forall (i in n)
sum (j in n)
(A[i][j]*x[j] +A[i][j]*z[j]*y[i][j]) >= 1;
forall (j in n)
sum(i in n)
A[i][j]*y[i][j] == z[j];
}
Data:
SheetConnection sheet("E:\\Program Files\\IBM\\ILOG\\CPLEX_Studio129\\opl\\oplide\\workspace\\Teste\\PEGASEmatriz9241.xlsx");
A from SheetRead(sheet,"matriz9241");
z from SheetRead(sheet,"zib9241");
I think you have a memory issue. I managed to run the scalable example
https://www.ibm.com/developerworks/community/forums/html/topic?id=c2469c56-db27-4816-9cf2-f596513ce555&ps=25
til n=7000
So in your case I would divide your square into 4 squares. Let me share the full model to do that.
First to write into the Excel file in order to let you test.
.mod
execute
{
// http://cwestblog.com/2013/09/05/javascript-snippet-convert-number-to-column-name/
function toColumnName(num) {
for (var ret = '', a = 1, b = 26; (num -= a) >= 0; a = b, b *= 26) {
ret = String.fromCharCode(parseInt((num % b) / a) + 65) + ret;
}
return ret;
}
// 1,1 => A1 1,4 => D1 2,27 => AA2
function convertR1C1toA1(r,c)
{
return(toColumnName(c)+r);
}
}
int n=4;
int cell[i in 1..n][j in 1..n]=i*j;
int cell1[i in 1..n div 2][j in 1..n div 2]=cell[i][j];
int cell2[i in 1..n div 2][j in n div 2+1..n]=cell[i][j];
int cell3[i in n div 2+1..n][j in 1..n div 2]=cell[i][j];
int cell4[i in n div 2+1..n][j in n div 2+1 ..n]=cell[i][j];
string sheetWriteString1;
string sheetWriteString2;
string sheetWriteString3;
string sheetWriteString4;
execute
{
sheetWriteString1=convertR1C1toA1(1,1)+":"+convertR1C1toA1(n/2,n/2);
writeln("sheetWriteString1=",sheetWriteString1);
sheetWriteString2=convertR1C1toA1(1,n/2+1)+":"+convertR1C1toA1(n/2,n);
writeln("sheetWriteString2=",sheetWriteString2);
sheetWriteString3=convertR1C1toA1(n/2+1,1)+":"+convertR1C1toA1(n,n/2);
writeln("sheetWriteString3=",sheetWriteString3);
sheetWriteString4=convertR1C1toA1(n/2+1,n/2+1)+":"+convertR1C1toA1(n,n);
writeln("sheetWriteString4=",sheetWriteString4);
}
.dat
SheetConnection s("f2.xlsx");
cell1 to SheetWrite(s,sheetWriteString1);
cell2 to SheetWrite(s,sheetWriteString2);
cell3 to SheetWrite(s,sheetWriteString3);
cell4 to SheetWrite(s,sheetWriteString4);
and then in order to read f2.xls
.mod
int n=...;
string sheetWriteString1=...;
string sheetWriteString2=...;
string sheetWriteString3=...;
string sheetWriteString4=...;
int cell1[i in 1..n div 2][j in 1..n div 2]=...;
int cell2[i in 1..n div 2][j in n div 2+1..n]=...;
int cell3[i in n div 2+1..n][j in 1..n div 2]=...;
int cell4[i in n div 2+1..n][j in n div 2+1..n]=...;
int cell[i in 1..n][j in 1..n]=
(i<=n div 2)?((j<=n div 2)?cell1[i][j]:cell2[i][j]):((j<=n div 2)?cell3[i][j]:cell4[i][j]);
assert forall(i,j in 1..n) cell[i][j]==i*j;
.dat
SheetConnection s("f2.xlsx");
n=4;
sheetWriteString1="A1:B2";
sheetWriteString2="C1:D2";
sheetWriteString3="A3:B4";
sheetWriteString4="C3:D4";
cell1 from SheetRead(s,sheetWriteString1);
cell2 from SheetRead(s,sheetWriteString2);
cell3 from SheetRead(s,sheetWriteString3);
cell4 from SheetRead(s,sheetWriteString4);
I am studying the GMG background subtraction algorithm as described in this paper. As OpenCV 3.0 also has an implementation of the GMG algorithm (as the additional package opencv_contrib), I try to study the two together. However, I am not quite sure about the meaning of the two parameters maxFeatures and quantizationLevels, as I want to map them to the descriptions in the paper.
Quoting the source code in OpenCV 3.0 (file modules\bgsegm\src\bgfg_gmg.cpp):
//! Total number of distinct colors to maintain in histogram.
int maxFeatures;
and
//! Number of discrete levels in each channel to be used in histograms.
int quantizationLevels;
And quoting the paper (Section II B) (with some mathematical symbol and variable names modified since LaTex is not supported here):
"... Of the T observed features, select the F_tot <= F_max most recently observed unique features; let I is a subset of {1, 2, ... T}, where |I| = F_tot, be the corresponding time index set. (If T > F_max, it is possible that F_tot, the number of distinct features observed, exceeds the limit F_max. In that case, we throw away the oldest observations so that F_tot <= F_max). Then, we calculate an average to generate the initial histogram: H(T) = (1/F_tot)∑f(r). This puts equal weight, 1/F_tot, in F_tot unique bins of the histogram."
From the above description, I was originally convinced that maxFeatures in OpenCV 3.0 refers to F_max in the paper, and quantizationLevels refers to F_tot. However, this does not sound right for two reasons: (1) The paper mentions "F_tot is the number of distinct features observed", and (2) the OpenCV source code does not pose any relationship between maxFeatures and quantizationLevels, while the paper clearly suggests that the former should be larger than or equal to the latter.
So, what are the meaning of maxFeatures and quantizationLevels? And is quantizationLevels a parameter introduced by OpenCV for the calculation of histogram?
After further studying the source code in OpenCV, I believe that maxFeatures refer to F_max in the paper, while quantizationLevels is actually the number of bins in the histogram. The reason is as follows:
In the function insertFeature(), which contains the following code:
static bool insertFeature(unsigned int color, float weight, unsigned int* colors, float* weights, int& nfeatures, int maxFeatures)
{
int idx = -1;
for (int i = 0; i < nfeatures; ++i) {
if (color == colors[i]) {
// feature in histogram
weight += weights[i];
idx = i;
break;
}
}
if (idx >= 0) { // case 1
// move feature to beginning of list
::memmove(colors + 1, colors, idx * sizeof(unsigned int));
::memmove(weights + 1, weights, idx * sizeof(float));
colors[0] = color;
weights[0] = weight;
}
else if (nfeatures == maxFeatures) { // case 2
// discard oldest feature
::memmove(colors + 1, colors, (nfeatures - 1) * sizeof(unsigned int));
::memmove(weights + 1, weights, (nfeatures - 1) * sizeof(float));
colors[0] = color;
weights[0] = weight;
}
else { // case 3
colors[nfeatures] = color;
weights[nfeatures] = weight;
++nfeatures;
return true;
}
return false;
}
Case 1: When the color matches one item in the array colors[], the corresponding array element is obtained.
Case 2: When the color does not match any item in the array colors[] and maxFeatures is reached (nFeatures stores the number of items stored in the array), then the oldest feature is removed for the new color.
Case 3: When the color does not match any item in the array colors[] and maxFeatures is not reached yet, add the color to a new item of the array, and nFeatures is incremented by 1.
Hence maxFeatures should correspond to F_max (the maximum number of features) in the paper.
In addition, in the function apply():
static unsigned int apply(const void* src_, int x, int cn, double minVal, double maxVal, int quantizationLevels)
{
const T* src = static_cast<const T*>(src_);
src += x * cn;
unsigned int res = 0;
for (int i = 0, shift = 0; i < cn; ++i, ++src, shift += 8)
res |= static_cast<int>((*src - minVal) * quantizationLevels / (maxVal - minVal)) << shift;
return res;
}
This function maps the pixel's color intensity pointed to by the pointer src_ to a bin according to the values of maxVal, minVal and quantizationLevels, such that if quantizationLevels = q, the result of the code:
static_cast<int>((*src - minVal) * quantizationLevels / (maxVal - minVal))
must be an integer in the range [0, q-1]. However, there can be cn channels (for example, in RGB, cn = 3, hence the shift operation and the possible number of bins (denote it as b) is therefore quantizationLevels to the power of cn. So if b > F_max, we have to discard the (b - F_max) oldest features.
Hence, in OpenCV, if we set maxFeatures to be >= quantizationLevels ^ cn, we never have to discard the oldest features, because we allow more than enough bins, or more than enough different unique features.
I need a data structure for storing float values at an uniformly sampled 3D mesh:
x = x0 + ix*dx where 0 <= ix < nx
y = y0 + iy*dy where 0 <= iy < ny
z = z0 + iz*dz where 0 <= iz < nz
Up to now I have used my Array class:
Array3D<float> A(nx, ny,nz);
A(0,0,0) = 0.0f; // ix = iy = iz = 0
Internally it stores the float values as an 1D array with nx * ny * nz elements.
However now I need to represent an mesh with more values than I have RAM,
e.g. nx = ny = nz = 2000.
I think many neighbour nodes in such an mesh may have similar values so I was thinking if there was some simple way that I could "coarsen" the mesh adaptively.
For instance if the 8 (ix,iy,iz) nodes of an cell in this mesh have values that are less than 5% apart; they are "removed" and replaced by just one value; the mean of the 8 values.
How could I implement such a data structure in a simple and efficient way?
EDIT:
thanks Ante for suggesting lossy compression. I think this could work the following way:
#define BLOCK_SIZE 64
struct CompressedArray3D {
CompressedArray3D(int ni, int nj, int nk) {
NI = ni/BLOCK_SIZE + 1;
NJ = nj/BLOCK_SIZE + 1;
NK = nk/BLOCK_SIZE + 1;
blocks = new float*[NI*NJ*NK];
compressedSize = new unsigned int[NI*NJ*NK];
}
void setBlock(int I, int J, int K, float values[BLOCK_SIZE][BLOCK_SIZE][BLOCK_SIZE]) {
unsigned int csize;
blocks[I*NJ*NK + J*NK + K] = compress(values, csize);
compressedSize[I*NJ*NK + J*NK + K] = csize;
}
float getValue(int i, int j, int k) {
int I = i/BLOCK_SIZE;
int J = j/BLOCK_SIZE;
int K = k/BLOCK_SIZE;
int ii = i - I*BLOCK_SIZE;
int jj = j - J*BLOCK_SIZE;
int kk = k - K*BLOCK_SIZE;
float *compressedBlock = blocks[I*NJ*NK + J*NK + K];
unsigned int csize = compressedSize[I*NJ*NK + J*NK + K];
float values[BLOCK_SIZE][BLOCK_SIZE][BLOCK_SIZE];
decompress(compressedBlock, csize, values);
return values[ii][jj][kk];
}
// number of blocks:
int NI, NJ, NK;
// number of samples:
int ni, nj, nk;
float** blocks;
unsigned int* compressedSize;
};
For this to be useful I need a lossy compression that is:
extremely fast, also on small datasets (e.g. 64x64x64)
compress quite hard > 3x, never mind if it looses quite a bit of info.
Any good candidates?
It sounds like you're looking for a LOD (level of detail) adaptive mesh. It's a recurring theme in video games and terrain simulation.
For terrain, see here: http://vterrain.org/LOD/Papers/ -- look for the ROAM video which is IIRC not only adaptive by distance, but also by view direction.
For non-terrain entities, there is a huge body of work (here's one example: Generic Adaptive Mesh Refinement).
I would suggest to use OctoMap to handle large 3D data.
And to extend it as shown here to handle geometrical properties.
Does anybody know how to find the local maxima in a grayscale IPL_DEPTH_8U image using OpenCV? HarrisCorner mentions something like that but I'm actually not interested in corners ...
Thanks!
A pixel is considered a local maximum if it is equal to the maximum value in a 'local' neighborhood. The function below captures this property in two lines of code.
To deal with pixels on 'plateaus' (value equal to their neighborhood) one can use the local minimum property, since plateaus pixels are equal to their local minimum. The rest of the code filters out those pixels.
void non_maxima_suppression(const cv::Mat& image, cv::Mat& mask, bool remove_plateaus) {
// find pixels that are equal to the local neighborhood not maximum (including 'plateaus')
cv::dilate(image, mask, cv::Mat());
cv::compare(image, mask, mask, cv::CMP_GE);
// optionally filter out pixels that are equal to the local minimum ('plateaus')
if (remove_plateaus) {
cv::Mat non_plateau_mask;
cv::erode(image, non_plateau_mask, cv::Mat());
cv::compare(image, non_plateau_mask, non_plateau_mask, cv::CMP_GT);
cv::bitwise_and(mask, non_plateau_mask, mask);
}
}
Here's a simple trick. The idea is to dilate with a kernel that contains a hole in the center. After the dilate operation, each pixel is replaced with the maximum of it's neighbors (using a 5 by 5 neighborhood in this example), excluding the original pixel.
Mat1b kernelLM(Size(5, 5), 1u);
kernelLM.at<uchar>(2, 2) = 0u;
Mat imageLM;
dilate(image, imageLM, kernelLM);
Mat1b localMaxima = (image > imageLM);
Actually after I posted the code above I wrote a better and very very faster one ..
The code above suffers even for a 640x480 picture..
I optimized it and now it is very very fast even for 1600x1200 pic.
Here is the code :
void localMaxima(cv::Mat src,cv::Mat &dst,int squareSize)
{
if (squareSize==0)
{
dst = src.clone();
return;
}
Mat m0;
dst = src.clone();
Point maxLoc(0,0);
//1.Be sure to have at least 3x3 for at least looking at 1 pixel close neighbours
// Also the window must be <odd>x<odd>
SANITYCHECK(squareSize,3,1);
int sqrCenter = (squareSize-1)/2;
//2.Create the localWindow mask to get things done faster
// When we find a local maxima we will multiply the subwindow with this MASK
// So that we will not search for those 0 values again and again
Mat localWindowMask = Mat::zeros(Size(squareSize,squareSize),CV_8U);//boolean
localWindowMask.at<unsigned char>(sqrCenter,sqrCenter)=1;
//3.Find the threshold value to threshold the image
//this function here returns the peak of histogram of picture
//the picture is a thresholded picture it will have a lot of zero values in it
//so that the second boolean variable says :
// (boolean) ? "return peak even if it is at 0" : "return peak discarding 0"
int thrshld = maxUsedValInHistogramData(dst,false);
threshold(dst,m0,thrshld,1,THRESH_BINARY);
//4.Now delete all thresholded values from picture
dst = dst.mul(m0);
//put the src in the middle of the big array
for (int row=sqrCenter;row<dst.size().height-sqrCenter;row++)
for (int col=sqrCenter;col<dst.size().width-sqrCenter;col++)
{
//1.if the value is zero it can not be a local maxima
if (dst.at<unsigned char>(row,col)==0)
continue;
//2.the value at (row,col) is not 0 so it can be a local maxima point
m0 = dst.colRange(col-sqrCenter,col+sqrCenter+1).rowRange(row-sqrCenter,row+sqrCenter+1);
minMaxLoc(m0,NULL,NULL,NULL,&maxLoc);
//if the maximum location of this subWindow is at center
//it means we found the local maxima
//so we should delete the surrounding values which lies in the subWindow area
//hence we will not try to find if a point is at localMaxima when already found a neighbour was
if ((maxLoc.x==sqrCenter)&&(maxLoc.y==sqrCenter))
{
m0 = m0.mul(localWindowMask);
//we can skip the values that we already made 0 by the above function
col+=sqrCenter;
}
}
}
The following listing is a function similar to Matlab's "imregionalmax". It looks for at most nLocMax local maxima above threshold, where the found local maxima are at least minDistBtwLocMax pixels apart. It returns the actual number of local maxima found. Notice that it uses OpenCV's minMaxLoc to find global maxima. It is "opencv-self-contained" except for the (easy to implement) function vdist, which computes the (euclidian) distance between points (r,c) and (row,col).
input is one-channel CV_32F matrix, and locations is nLocMax (rows) by 2 (columns) CV_32S matrix.
int imregionalmax(Mat input, int nLocMax, float threshold, float minDistBtwLocMax, Mat locations)
{
Mat scratch = input.clone();
int nFoundLocMax = 0;
for (int i = 0; i < nLocMax; i++) {
Point location;
double maxVal;
minMaxLoc(scratch, NULL, &maxVal, NULL, &location);
if (maxVal > threshold) {
nFoundLocMax += 1;
int row = location.y;
int col = location.x;
locations.at<int>(i,0) = row;
locations.at<int>(i,1) = col;
int r0 = (row-minDistBtwLocMax > -1 ? row-minDistBtwLocMax : 0);
int r1 = (row+minDistBtwLocMax < scratch.rows ? row+minDistBtwLocMax : scratch.rows-1);
int c0 = (col-minDistBtwLocMax > -1 ? col-minDistBtwLocMax : 0);
int c1 = (col+minDistBtwLocMax < scratch.cols ? col+minDistBtwLocMax : scratch.cols-1);
for (int r = r0; r <= r1; r++) {
for (int c = c0; c <= c1; c++) {
if (vdist(Point2DMake(r, c),Point2DMake(row, col)) <= minDistBtwLocMax) {
scratch.at<float>(r,c) = 0.0;
}
}
}
} else {
break;
}
}
return nFoundLocMax;
}
The first question to answer would be what is "local" in your opinion. The answer may well be a square window (say 3x3 or 5x5) or circular window of a certain radius. You can then scan over the entire image with the window centered at each pixel and pick the highest value in the window.
See this for how to access pixel values in OpenCV.
This is very fast method. It stored founded maxima in a vector of
Points.
vector <Point> GetLocalMaxima(const cv::Mat Src,int MatchingSize, int Threshold, int GaussKernel )
{
vector <Point> vMaxLoc(0);
if ((MatchingSize % 2 == 0) || (GaussKernel % 2 == 0)) // MatchingSize and GaussKernel have to be "odd" and > 0
{
return vMaxLoc;
}
vMaxLoc.reserve(100); // Reserve place for fast access
Mat ProcessImg = Src.clone();
int W = Src.cols;
int H = Src.rows;
int SearchWidth = W - MatchingSize;
int SearchHeight = H - MatchingSize;
int MatchingSquareCenter = MatchingSize/2;
if(GaussKernel > 1) // If You need a smoothing
{
GaussianBlur(ProcessImg,ProcessImg,Size(GaussKernel,GaussKernel),0,0,4);
}
uchar* pProcess = (uchar *) ProcessImg.data; // The pointer to image Data
int Shift = MatchingSquareCenter * ( W + 1);
int k = 0;
for(int y=0; y < SearchHeight; ++y)
{
int m = k + Shift;
for(int x=0;x < SearchWidth ; ++x)
{
if (pProcess[m++] >= Threshold)
{
Point LocMax;
Mat mROI(ProcessImg, Rect(x,y,MatchingSize,MatchingSize));
minMaxLoc(mROI,NULL,NULL,NULL,&LocMax);
if (LocMax.x == MatchingSquareCenter && LocMax.y == MatchingSquareCenter)
{
vMaxLoc.push_back(Point( x+LocMax.x,y + LocMax.y ));
// imshow("W1",mROI);cvWaitKey(0); //For gebug
}
}
}
k += W;
}
return vMaxLoc;
}
Found a simple solution.
In this example, if you are trying to find 2 results of a matchTemplate function with a minimum distance from each other.
cv::Mat result;
matchTemplate(search, target, result, CV_TM_SQDIFF_NORMED);
float score1;
cv::Point displacement1 = MinMax(result, score1);
cv::circle(result, cv::Point(displacement1.x+result.cols/2 , displacement1.y+result.rows/2), 10, cv::Scalar(0), CV_FILLED, 8, 0);
float score2;
cv::Point displacement2 = MinMax(result, score2);
where
cv::Point MinMax(cv::Mat &result, float &score)
{
double minVal, maxVal;
cv::Point minLoc, maxLoc, matchLoc;
minMaxLoc(result, &minVal, &maxVal, &minLoc, &maxLoc, cv::Mat());
matchLoc.x = minLoc.x - result.cols/2;
matchLoc.y = minLoc.y - result.rows/2;
return minVal;
}
The process is:
Find global Minimum using minMaxLoc
Draw a filled white circle around global minimum using min distance between minima as radius
Find another minimum
The the scores can be compared to each other to determine, for example, the certainty of the match,
To find more than just the global minimum and maximum try using this function from skimage:
http://scikit-image.org/docs/dev/api/skimage.feature.html#skimage.feature.peak_local_max
You can parameterize the minimum distance between peaks, too. And more. To find minima, use negated values (take care of the array type though, 255-image could do the trick).
You can go over each pixel and test if it is a local maxima. Here is how I would do it.
The input is assumed to be type CV_32FC1
#include <vector>//std::vector
#include <algorithm>//std::sort
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/core/core.hpp"
//structure for maximal values including position
struct SRegionalMaxPoint
{
SRegionalMaxPoint():
values(-FLT_MAX),
row(-1),
col(-1)
{}
float values;
int row;
int col;
//ascending order
bool operator()(const SRegionalMaxPoint& a, const SRegionalMaxPoint& b)
{
return a.values < b.values;
}
};
//checks if pixel is local max
bool isRegionalMax(const float* im_ptr, const int& cols )
{
float center = *im_ptr;
bool is_regional_max = true;
im_ptr -= (cols + 1);
for (int ii = 0; ii < 3; ++ii, im_ptr+= (cols-3))
{
for (int jj = 0; jj < 3; ++jj, im_ptr++)
{
if (ii != 1 || jj != 1)
{
is_regional_max &= (center > *im_ptr);
}
}
}
return is_regional_max;
}
void imregionalmax(
const cv::Mat& input,
std::vector<SRegionalMaxPoint>& buffer)
{
//find local max - top maxima
static const int margin = 1;
const int rows = input.rows;
const int cols = input.cols;
for (int i = margin; i < rows - margin; ++i)
{
const float* im_ptr = input.ptr<float>(i, margin);
for (int j = margin; j < cols - margin; ++j, im_ptr++)
{
//Check if pixel is local maximum
if ( isRegionalMax(im_ptr, cols ) )
{
cv::Rect roi = cv::Rect(j - margin, i - margin, 3, 3);
cv::Mat subMat = input(roi);
float val = *im_ptr;
//replace smallest value in buffer
if ( val > buffer[0].values )
{
buffer[0].values = val;
buffer[0].row = i;
buffer[0].col = j;
std::sort(buffer.begin(), buffer.end(), SRegionalMaxPoint());
}
}
}
}
}
For testing the code you can try this:
cv::Mat temp = cv::Mat::zeros(15, 15, CV_32FC1);
temp.at<float>(7, 7) = 1;
temp.at<float>(3, 5) = 6;
temp.at<float>(8, 10) = 4;
temp.at<float>(11, 13) = 7;
temp.at<float>(10, 3) = 8;
temp.at<float>(7, 13) = 3;
vector<SRegionalMaxPoint> buffer_(5);
imregionalmax(temp, buffer_);
cv::Mat debug;
cv::cvtColor(temp, debug, cv::COLOR_GRAY2BGR);
for (auto it = buffer_.begin(); it != buffer_.end(); ++it)
{
circle(debug, cv::Point(it->col, it->row), 1, cv::Scalar(0, 255, 0));
}
This solution does not take plateaus into account so it is not exactly the same as matlab's imregionalmax()
I think you want to use the
MinMaxLoc(arr, mask=NULL)-> (minVal, maxVal, minLoc, maxLoc)
Finds global minimum and maximum in array or subarray
function on you image
I have an application where a Hilbert R-Tree (wikipedia) (citeseer) would seem to be an appropriate data structure. Specifically, it requires reasonably fast spatial queries over a data set that will experience a lot of updates.
However, as far as I can see, none of the descriptions of the algorithms for this data structure even mention how to actually calculate the requisite Hilbert Value; which is the distance along a Hilbert Curve to the point.
So any suggestions for how to go about calculating this?
Fun question!
I did a bit of googling, and the good news is, I've found an implementation of Hilbert Value.
The potentially bad news is, it's in Haskell...
http://www.serpentine.com/blog/2007/01/11/two-dimensional-spatial-hashing-with-space-filling-curves/
It also proposes a Lebesgue distance metric you might be able to compute more easily.
Below is my java code adapted from C code in the paper "Encoding and decoding the Hilbert order" by Xian Lu and Gunther Schrack, published in Software: Practice and Experience Vol. 26 pp 1335-46 (1996).
Hope this helps. Improvements welcome !
Michael
/**
* Find the Hilbert order (=vertex index) for the given grid cell
* coordinates.
* #param x cell column (from 0)
* #param y cell row (from 0)
* #param r resolution of Hilbert curve (grid will have Math.pow(2,r)
* rows and cols)
* #return Hilbert order
*/
public static int encode(int x, int y, int r) {
int mask = (1 << r) - 1;
int hodd = 0;
int heven = x ^ y;
int notx = ~x & mask;
int noty = ~y & mask;
int temp = notx ^ y;
int v0 = 0, v1 = 0;
for (int k = 1; k < r; k++) {
v1 = ((v1 & heven) | ((v0 ^ noty) & temp)) >> 1;
v0 = ((v0 & (v1 ^ notx)) | (~v0 & (v1 ^ noty))) >> 1;
}
hodd = (~v0 & (v1 ^ x)) | (v0 & (v1 ^ noty));
return interleaveBits(hodd, heven);
}
/**
* Interleave the bits from two input integer values
* #param odd integer holding bit values for odd bit positions
* #param even integer holding bit values for even bit positions
* #return the integer that results from interleaving the input bits
*
* #todo: I'm sure there's a more elegant way of doing this !
*/
private static int interleaveBits(int odd, int even) {
int val = 0;
// Replaced this line with the improved code provided by Tuska
// int n = Math.max(Integer.highestOneBit(odd), Integer.highestOneBit(even));
int max = Math.max(odd, even);
int n = 0;
while (max > 0) {
n++;
max >>= 1;
}
for (int i = 0; i < n; i++) {
int bitMask = 1 << i;
int a = (even & bitMask) > 0 ? (1 << (2*i)) : 0;
int b = (odd & bitMask) > 0 ? (1 << (2*i+1)) : 0;
val += a + b;
}
return val;
}
See uzaygezen.
The code and java code above are fine for 2D data points. But for higher dimensions you may need to look at Jonathan Lawder's paper: J.K.Lawder. Calculation of Mappings Between One and n-dimensional Values Using the Hilbert Space-filling Curve.
I figured out a slightly more efficient way to interleave bits. It can be found at the Stanford Graphics Website. I included a version that I created that can interleave two 32 bit integers into one 64 bit long.
public static long spreadBits32(int y) {
long[] B = new long[] {
0x5555555555555555L,
0x3333333333333333L,
0x0f0f0f0f0f0f0f0fL,
0x00ff00ff00ff00ffL,
0x0000ffff0000ffffL,
0x00000000ffffffffL
};
int[] S = new int[] { 1, 2, 4, 8, 16, 32 };
long x = y;
x = (x | (x << S[5])) & B[5];
x = (x | (x << S[4])) & B[4];
x = (x | (x << S[3])) & B[3];
x = (x | (x << S[2])) & B[2];
x = (x | (x << S[1])) & B[1];
x = (x | (x << S[0])) & B[0];
return x;
}
public static long interleave64(int x, int y) {
return spreadBits32(x) | (spreadBits32(y) << 1);
}
Obviously, the B and S local variables should be class constants but it was left this way for simplicity.
Michael,
thanks for your Java code! I tested it and it seems to work fine, but I noticed that the bit-interleaving function overflows at recursion level 7 (at least in my tests, but I used long values), because the "n"-value is calculated using highestOneBit()-function, which returns the value and not the position of the highest one bit; so the loop does unnecessarily many interleavings.
I just changed it to the following snippet, and after that it worked fine.
int max = Math.max(odd, even);
int n = 0;
while (max > 0) {
n++;
max >>= 1;
}
If you need a spatial index with fast delete/insert capabilities, have a look at the PH-tree. It partly based on quadtrees but faster and more space efficient. Internally it uses a Z-curve which has slightly worse spatial properties than an H-curve but is much easier to calculate.
Paper: http://www.globis.ethz.ch/script/publication/download?docid=699
Java implementation: http://globis.ethz.ch/files/2014/11/ph-tree-2014-11-10.zip
Another option is the X-tree, which is also available here:
https://code.google.com/p/xxl/
Suggestion: A good simple efficient data structure for spatial queries is a multidimensional binary tree.
In a traditional binary tree, there is one "discriminant"; the value that's used to determine whether you take the left branch or the right branch. This can be considered to be the one-dimensional case.
In a multidimensional binary tree, you have multiple discriminants; consecutive levels use different discriminants. For example, for two dimensional spacial data, you could use the X and Y coordinates as discriminants. Consecutive levels would use X, Y, X, Y...
For spatial queries (for example finding all nodes within a rectangle) you do a depth-first search of the tree starting at the root, and you use the discriminant at each level to avoid searching down branches that contain no nodes in the given rectangle.
This allows you to potentially cut the search space in half at each level, making it very efficient for finding small regions in a massive data set. (BTW, this data structure is also useful for partial-match queries, i.e. queries that omit one or more discriminants. You just search down both branches at levels with an omitted discriminant.)
A good paper on this data structure: http://portal.acm.org/citation.cfm?id=361007
This article has good diagrams and algorithm descriptions: http://en.wikipedia.org/wiki/Kd-tree