Fast algorithm for image distortion - algorithm

I am working on a tool which distorts images, the purpose of the distortion is to project images to a sphere screen. The desired output is as the following image.
The code I use is as follow - for every Point(x, y) in the destination area, I calculate the corresponding pixel (sourceX, sourceY) in the original image to retrieve from.
But this approach is awkwardly slow, in my test, processing the sunset.jpg (800*600) requires more than 1500ms, if I remove the Mathematical/Trigonometrical calculations, calling cvGet2D and cvSet2D alone require more than 1200ms.
Is there a better way to do this? I am using Emgu CV (a .NET wrapper library for OpenCV) but examples in other language is also OK.
private static void DistortSingleImage()
{
System.Diagnostics.Stopwatch stopWatch = System.Diagnostics.Stopwatch.StartNew();
using (Image<Bgr, Byte> origImage = new Image<Bgr, Byte>("sunset.jpg"))
{
int frameH = origImage.Height;
using (Image<Bgr, Byte> distortImage = new Image<Bgr, Byte>(2 * frameH, 2 * frameH))
{
MCvScalar pixel;
for (int x = 0; x < 2 * frameH; x++)
{
for (int y = 0; y < 2 * frameH; y++)
{
if (x == frameH && y == frameH) continue;
int x1 = x - frameH;
int y1 = y - frameH;
if (x1 * x1 + y1 * y1 < frameH * frameH)
{
double radius = Math.Sqrt(x1 * x1 + y1 * y1);
double theta = Math.Acos(x1 / radius);
int sourceX = (int)(theta * (origImage.Width - 1) / Math.PI);
int sourceY = (int)radius;
pixel = CvInvoke.cvGet2D(origImage.Ptr, sourceY, sourceX);
CvInvoke.cvSet2D(distortImage, y, x, pixel);
}
}
}
distortImage.Save("Distort.jpg");
}
Console.WriteLine(stopWatch.ElapsedMilliseconds);
}
}

From my personal experience, I was doing some stereoscopic vision stuff, the best way to talk to openCV is through own wrapper, you could put your method in c++ and call it from c#, that would give you 1 call to native, faster code, and because under the hood Emgu's keeping OpenCV data, it's also possible to create an image with emgu, process it natively and enjoy processed image in c# again.
The get/set methods looks like Gdi's GetPixel / SetPixel ones, and, according to documentation they are "slow but safe way".
For staying with Emgu only, documentation tells that if you want to iterate over pixels, you should access .Data property:
The safe (slow) way
Suppose you are working on an Image. You can obtain the pixel on the y-th row and x-th column by calling
Bgr color = img[y, x];
Setting the pixel on the y-th row and x-th column is also simple
img[y,x] = color;
The fast way
The Image pixels values are stored in the Data property, a 3D array. Use this property if you need to iterate through the pixel values of the image.

Related

Processing: Efficiently create uniform grid

I'm trying to create a grid of an image (in the way one would tile a background with). Here's what I've been using:
PImage bgtile;
PGraphics bg;
int tilesize = 50;
void setup() {
int t = millis();
fullScreen(P2D);
background(0);
bgtile = loadImage("bgtile.png");
int bgw = ceil( ((float) width) / tilesize) + 1;
int bgh = ceil( ((float) height) / tilesize) + 1;
bg = createGraphics(bgw*tilesize,bgh*tilesize);
bg.beginDraw();
for(int i = 0; i < bgw; i++){
for(int j = 0; j < bgh; j++){
bg.image(bgtile, i*tilesize, j*tilesize, tilesize, tilesize);
}
}
bg.endDraw();
print(millis() - t);
}
The timing code says that this takes about a quarter of a second, but by my count there's a full second once the window opens before anything shows up on screen (which should happen as soon as draw is first run). Is there a faster way to get this same effect? (I want to avoid rendering bgtile hundreds of times in the draw loop for obvious reasons)
One way could be to make use of the GPU and let OpenGL repeat a texture for you.
Processing makes it fairly easy to repeat a texture via textureWrap(REPEAT)
Instead of drawing an image you'd make your own quad shape and instead of calling vertex(x, y) for example, you'd call vertex(x, y, u, v); passing texture coordinates (more low level info on the OpenGL link above). The simple idea is x,y would control the geometry on screen and u,v would control how the texture is applied to the geometry.
Another thing you can control is textureMode() which allows you control how you specify the texture coordinates (U, V):
IMAGE mode is the default: you use pixel coordinates (based on the dimensions of the texture)
NORMAL mode uses values between 0.0 and 1.0 (also known as normalised values) where 1.0 means the maximum the texture can go (e.g. image width for U or image height for V) and you don't need to worry about knowing the texture image dimensions
Here's a basic example based on the textureMode() example above:
PImage img;
void setup() {
fullScreen(P2D);
noStroke();
img = loadImage("https://processing.org/examples/moonwalk.jpg");
// texture mode can be IMAGE (pixel dimensions) or NORMAL (0.0 to 1.0)
// normal means 1.0 is full width (for U) or height (for V) without having to know the image resolution
textureMode(NORMAL);
// this is what will make handle tiling for you
textureWrap(REPEAT);
}
void draw() {
// drag mouse on X axis to change tiling
int tileRepeats = (int)map(constrain(mouseX,0,width), 0, width, 1, 100);
// draw a textured quad
beginShape(QUAD);
// set the texture
texture(img);
// x , y , U , V
vertex(0 , 0 , 0 , 0);
vertex(width, 0 , tileRepeats, 0);
vertex(width, height, tileRepeats, tileRepeats);
vertex(0 , height, 0 , tileRepeats);
endShape();
text((int)frameRate+"fps",15,15);
}
Drag the mouse on the Y axis to control the number of repetitions.
In this simple example both vertex coordinates and texture coordinates are going clockwise (top left, top right, bottom right, bottom left order).
There are probably other ways to achieve the same result: using a PShader comes to mind.
Your approach caching the tiles in setup is ok.
Even flattening your nested loop into a single loop at best may only shave a few milliseconds off, but nothing substantial.
If you tried to cache my snippet above it would make a minimal difference.
In this particular case, because of the back and forth between Java/OpenGL (via JOGL), as far as I can tell using VisualVM, it looks like there's not a lot of room for improvement since simply swapping buffers takes so long (e.g. bg.image()):
An easy way to do this would be to use processing's built in get(); which saves a PImage of the coordinates you pass, for example: PImage pic = get(0, 0, width, height); will capture a "screenshot" of your entire window. So, you can create the image like you already are, and then take a screenshot and display that screenshot.
PImage bgtile;
PGraphics bg;
PImage screenGrab;
int tilesize = 50;
void setup() {
fullScreen(P2D);
background(0);
bgtile = loadImage("bgtile.png");
int bgw = ceil(((float) width) / tilesize) + 1;
int bgh = ceil(((float) height) / tilesize) + 1;
bg = createGraphics(bgw * tilesize, bgh * tilesize);
bg.beginDraw();
for (int i = 0; i < bgw; i++) {
for (int j = 0; j < bgh; j++) {
bg.image(bgtile, i * tilesize, j * tilesize, tilesize, tilesize);
}
}
bg.endDraw();
screenGrab = get(0, 0, width, height);
}
void draw() {
image(screenGrab, 0, 0);
}
This will still take a little bit to generate the image, but once it does, there is no need to use the for loops again unless you change the tilesize.
#George Profenza's answer looks more efficient than my solution, but mine may take a little less modification to the code you already have.

How is this ray casting algorithm flawed?

Matrix operations performed on the GPU can be pretty hard to debug because GPU operations don't really allow for console logs.
I've written one designed for a real time 2D rendering engine based on a very simple form of I guess what could be called ray casting and am having trouble figuring out what's wrong with it (it's outputting [0,0,0,255,0,0,0,255,...] instead of populating colors).
this.thread.x is the index of the current unit (color channel) in the matrix being operated on.
scene is a buffer made up of 6-unit clumps, each value containing, in order:
The type of entity, always 1 for "sprite" in this case.
The sprite ID, corresponding the the index in this.constants.textures containing the buffer for the entity's sprite.
X offset, the left edge of the sprite
Y offset, the top edge of the sprite
width of the sprite
height of the sprite
bufferWidth is the width of the render area multiplied by 4 channels.
this.constants.textures is an array containing buffers of each sprite which the sprite IDs from the scene refer to.
Note: For those curious, this is being done with GPU.js, a JavaScript lib that converts a JS func into GLSL code to be run via WebGL.
function(scene, sceneLength, bufferWidth) {
var channel = this.thread.x % 4;
if (channel === 3) {
return 255;
}
var x = this.thread.x % bufferWidth;
var y = Math.floor(this.thread.x / bufferWidth);
for (let i1 = 0; i1 < sceneLength; i1 += 6) {
var id = scene[i1 + 1];
var x1 = scene[i1 + 2];
var y1 = scene[i1 + 3];
var w1 = scene[i1 + 4];
var h1 = scene[i1 + 5];
var r1 = scene[i1 + 6];
var offsetX1 = x1 - x;
if (offsetX1 > 0 && offsetX1 < w1) {
var offsetY1 = y1 - y;
if (offsetY1 > 0 && offsetY1 < h1) {
var c1 = offsetY1 * w1 * 4 + offsetX1 * 4;
var c1R = c1 - (c1 % 4);
var c1A = c1R + 3;
if (this.constants.textures[id][c1A] != 0) {
return this.constants.textures[id][c1];
}
}
}
}
return 0;
}
Explanation for the concept I'm trying to implement:
With a matrix operation, when you want to draw a sprite if you were to perform a pass on the entire render area, you'd be doing far more work than necessary. If you break the rendering area down into chunks and only update the sections involved in the sprite being drawn, that would be a fairly decent way to do it. It would certainly be good enough for real time game rendering. This would be a multi-pass approach, where sprites are rendered one at a time.
Alternatively, for what seems to me to be the most optimal approach possible, instead of that, we can use a single-pass approach that performs a single matrix operation for the entire rendering area, evaluating for each color channel what should be there based on doing a very basic form of collision detection with each sprite in the scene and the relevant pixel in that sprite.
You're calculating your sprite offsets backwards, the calculations should be:
var offsetX1 = x - x1;
and
var offsetY1 = y - y1;
The offsets should increase as x and y increase (assuming the sprite co-ordinates have the same co-ordinate system as the screen co-ordinates), so you shouldn't be subtracting x and y.

How can i get device tilt in xamarin forms?

I´d like to get the tilt of the device, so i can use this to mesure the tilt os some surface, laying down the device over the surface.
Right now i am using Device Motion Plugin for xamarin forms from here https://github.com/rdelrosario/xamarin-plugins
and the code below:
CrossDeviceMotion.Current.Start(MotionSensorType.Accelerometer);
CrossDeviceMotion.Current.SensorValueChanged += (s, a) =>
{
switch (a.SensorType)
{
case MotionSensorType.Accelerometer:
{
Debug.WriteLine("A: {0},{1},{2}", ((MotionVector)a.Value).X, ((MotionVector)a.Value).Y,
((MotionVector)a.Value).Z);
Exposicao.Inclinacao = ((MotionVector)a.Value).Z;
break;
}
case MotionSensorType.Compass:
{
// Debug.WriteLine("H: {0}", a.Value);
Exposicao.Bussola = (double)a.Value.Value;
break;
}
}
};
The compass part is ok, the accelerometer part is working but there are some but´s.
If i am not wrong, i get the tilt in Z axis, so z.Value.Value.
This value is diferent for android and ios, lets focus in android.
z values are from 10 when device is laying down on flat surface, to 0 if device is stand up, lets focus only in just one quadrant.
I am doing something wrong to achieve what i explained?
How can i convert those values to a Angle between 0 and 90? It seems not linear, so the 5 does not seem 45 degrees.
Thanks
I'd probably roll out my own platform implementation for the feature you're looking for. The DeviceMotion library looks a bit simple for your purposes, as can be seen from the answer below. I'm pretty sure you can use it as a good starting point but it needs to be extended a little.
Android
On Android, you should use the Rotation Vector Sensor which uses a Kalman filter (with accelerometer, magnetometer and gyroscope) to get accurate measurements of the device's rotation:
The rotation vector represents the orientation of the device as a combination of an angle and an axis, in which the device has rotated through an angle θ around an axis (x, y, or z).
Image from the official Android documentation
iOS:
For iOS, you have to do a bit more work yourself. The key is to make use of CMAttitude, which describes the attitude of the device relative to an initial attitude. I found a snippet I've saved to my collection from an unknown source (can't credit the original author) here:
public void CalculateLeanAngle ()
{
motionManager = new CMMotionManager ();
motionManager.DeviceMotionUpdateInterval = 0.02;
if (motionManager.DeviceMotionAvailable) {
motionManager.StartDeviceMotionUpdates(CMAttitudeReferenceFrame.XArbitraryZVertical, NSOperationQueue.CurrentQueue, (data, error) => {
CMQuaternion quat = motionManager.DeviceMotion.Attitude.Quaternion;
double x = quat.x;
double y = quat.y;
double w = quat.w;
double z = quat.z;
double degrees = 0.0;
//Roll
double roll = Math.Atan2 (2 * y * w - 2 * x * z, 1 - 2 * y * y - 2 * z * z);
degrees = Math.Round (-applyKalmanFiltering (roll) * 180.0 / Constants.M_PI);
});
}
public double applyKalmanFiltering (double yaw)
{
if (motionLastYaw == 0)
motionLastYaw = yaw;
float q = 0.1f; // process noise
float r = 0.1f; // sensor noise
float p = 0.1f; // estimated error
float k = 0.5f; // kalman filter gain
double x = motionLastYaw;
p = p + q;
k = p / (p + r);
x = x + k * (yaw - x);
p = (1 - k) * p;
motionLastYaw = x;
return motionLastYaw;
}
Image from the official Xamarin documentation
I'll try to look for the original source when I have more time but I'm pretty sure this will work out of the box for your purposes.

Processing - creating circles from current pixels

I'm using processing, and I'm trying to create a circle from the pixels i have on my display.
I managed to pull the pixels on screen and create a growing circle from them.
However i'm looking for something much more sophisticated, I want to make it seem as if the pixels on the display are moving from their current location and forming a turning circle or something like this.
This is what i have for now:
int c = 0;
int radius = 30;
allPixels = removeBlackP();
void draw {
loadPixels();
for (int alpha = 0; alpha < 360; alpha++)
{
float xf = 350 + radius*cos(alpha);
float yf = 350 + radius*sin(alpha);
int x = (int) xf;
int y = (int) yf;
if (radius > 200) {radius =30;break;}
if (c> allPixels.length) {c= 0;}
pixels[y*700 +x] = allPixels[c];
updatePixels();
}
radius++;
c++;
}
the function removeBlackP return an array with all the pixels except for the black ones.
This code works for me. There is an issue that the circle only has the numbers as int so it seems like some pixels inside the circle won't fill, i can live with that. I'm looking for something a bit more complex like I explained.
Thanks!
Fill all pixels of scanlines belonging to the circle. Using this approach, you will paint all places inside the circle. For every line calculate start coordinate (end one is symmetric). Pseudocode:
for y = center_y - radius; y <= center_y + radius; y++
dx = Sqrt(radius * radius - y * y)
for x = center_x - dx; x <= center_x + dx; x++
fill a[y, x]
When you find places for all pixels, you can make correlation between initial pixels places and calculated ones and move them step-by-step.
For example, if initial coordinates relative to center point for k-th pixel are (x0, y0) and final coordinates are (x1,y1), and you want to make M steps, moving pixel by spiral, calculate intermediate coordinates:
calc values once:
r0 = Sqrt(x0*x0 + y0*y0) //Math.Hypot if available
r1 = Sqrt(x1*x1 + y1*y1)
fi0 = Math.Atan2(y0, x0)
fi1 = Math.Atan2(y1, x1)
if fi1 < fi0 then
fi1 = fi1 + 2 * Pi;
for i = 1; i <=M ; i++
x = (r0 + i / M * (r1 - r0)) * Cos(fi0 + i / M * (fi1 - fi0))
y = (r0 + i / M * (r1 - r0)) * Sin(fi0 + i / M * (fi1 - fi0))
shift by center coordinates
The way you go about drawing circles in Processing looks a little convoluted.
The simplest way is to use the ellipse() function, no pixels involved though:
If you do need to draw an ellipse and use pixels, you can make use of PGraphics which is similar to using a separate buffer/"layer" to draw into using Processing drawing commands but it also has pixels[] you can access.
Let's say you want to draw a low-res pixel circle circle, you can create a small PGraphics, disable smoothing, draw the circle, then render the circle at a higher resolution. The only catch is these drawing commands must be placed within beginDraw()/endDraw() calls:
PGraphics buffer;
void setup(){
//disable sketch's aliasing
noSmooth();
buffer = createGraphics(25,25);
buffer.beginDraw();
//disable buffer's aliasing
buffer.noSmooth();
buffer.noFill();
buffer.stroke(255);
buffer.endDraw();
}
void draw(){
background(255);
//draw small circle
float circleSize = map(sin(frameCount * .01),-1.0,1.0,0.0,20.0);
buffer.beginDraw();
buffer.background(0);
buffer.ellipse(buffer.width / 2,buffer.height / 2, circleSize,circleSize);
buffer.endDraw();
//render small circle at higher resolution (blocky - no aliasing)
image(buffer,0,0,width,height);
}
If you want to manually draw a circle using pixels[] you are on the right using the polar to cartesian conversion formula (x = cos(angle) * radius, y = sin(angle) * radius).Even though it's focusing on drawing a radial gradient, you can find an example of drawing a circle(a lot actually) using pixels in this answer

How do I visualize audio data?

I would like to have something that looks something like this. Two different colors are not nessesary.
(source: sourceforge.net)
I already have the audio data (one sample/millisecond) from a stereo wav in two int arrays, one each for left and right channel. I have made a few attempts but they don't look anywhere near as clear as this, my attempts get to spikey or a compact lump.
Any good suggestions? I'm working in c# but psuedocode is ok.
Assume we have
a function DrawLine(color, x1, y1, x2, y2)
two int arrays with data right[] and left[] of lenght L
data values between 32767 and -32768
If you make any other assumptions just write them down in your answer.
for(i = 0; i < L - 1; i++) {
// What magic goes here?
}
This is how it turned out when I applied the solution Han provided. (only one channel)
alt text http://www.imagechicken.com/uploads/1245877759099921200.jpg
You'll likely have more than 1 sample for each pixel. For each group of samples mapped to a single pixel, you could draw a (vertical) line segment from the minimum value in the sample group to the maximum value. If you zoom in to 1 sample per pixel or less, this doesn't work anymore, and the 'nice' solution would be to display the sinc interpolated values.
Because DrawLine cannot paint a single pixel, there is a small problem when the minimum and maximum are the same. In that case you could copy a single pixel image in the desired position, as in the code below:
double samplesPerPixel = (double)L / _width;
double firstSample = 0;
int endSample = firstSample + L - 1;
for (short pixel = 0; pixel < _width; pixel++)
{
int lastSample = __min(endSample, (int)(firstSample + samplesPerPixel));
double Y = _data[channel][(int)firstSample];
double minY = Y;
double maxY = Y;
for (int sample = (int)firstSample + 1; sample <= lastSample; sample++)
{
Y = _data[channel][sample];
minY = __min(Y, minY);
maxY = __max(Y, maxY);
}
x = pixel + _offsetx;
y1 = Value2Pixel(minY);
y2 = Value2Pixel(maxY);
if (y1 == y2)
{
g->DrawImageUnscaled(bm, x, y1);
}
else
{
g->DrawLine(pen, x, y1, x, y2);
}
firstSample += samplesPerPixel;
}
Note that Value2Pixel scales a sample value to a pixel value (in the y-direction).
You might want to look into the R language for this. I don't have very much experience with it, but it's used largely in statistical analysis/visualization scenarios. I would be surprised if they didn't have some smoothing function to get rid of the extremes like you mentioned.
And you should have no trouble importing your data into it. Not only can you read flat text files, but it's also designed to be easily extensible with C, so there is probably some kind of C# interface as well.

Resources