Getting image of the player with Kinect in XNA performance issues - performance

Question

I am developing an XNA game that is using Kinect. The player seen on the screen is the real image of the person who is playing in front of Kinect sensor. For eliminating the background and getting only the player's image I am doing these operations in kinect.AllFramesReady:
using (ColorImageFrame colorVideoFrame = imageFrames.OpenColorImageFrame())
{
if (colorVideoFrame != null)
{
//Getting the image of the colorVideoFrame to a Texture2D named colorVideo
}
//And setting its information on a Color array named colors with GetData
colorVideo.GetData(colors);
}
using (DepthImageFrame depthVideoFrame = imageFrames.OpenDepthImageFrame())
{
if (depthVideoFrame != null){
//Copying the the image to a DepthImagePixel array
//Using only the pixels with PlayerIndex > 0 to create a Color array
//And then setting the colors of this array from the 'colors' array by using MapDepthPointToColorPoint method, provided by Kinect SDK
//Finally I use SetData function in order to set the colors to a Texture2D I created before
}
}
But the performance is very low unsurprisingly. Because I have to use GetData for a color array with 640*480 = 307200 length (because of the ColorImageFormat) and SetData for another color array with 320*480 = 76800 length (because of the DepthImageFormat) in every frame!
I wonder if there is any other solutions for this problem, any alternatives for SetData and GetData maybe. Because I know that these functions moving data between the GPU and CPU and that is an expensive operation for big data. Thanks for any help.

Answer 1

The Kinect for Windows Toolbox comes with a "GreenScreen-WPF" example, which should provide some insight into processing the information. Because you are working in XNA there may be some differences, but the overall concepts should work between the two examples.
The example works by extracting multiple players. Here is the business end of the processing function:
private void SensorAllFramesReady(object sender, AllFramesReadyEventArgs e)
{
// in the middle of shutting down, so nothing to do
if (null == this.sensor)
{
return;
}
bool depthReceived = false;
bool colorReceived = false;
using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
{
if (null != depthFrame)
{
// Copy the pixel data from the image to a temporary array
depthFrame.CopyDepthImagePixelDataTo(this.depthPixels);
depthReceived = true;
}
}
using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
{
if (null != colorFrame)
{
// Copy the pixel data from the image to a temporary array
colorFrame.CopyPixelDataTo(this.colorPixels);
colorReceived = true;
}
}
// do our processing outside of the using block
// so that we return resources to the kinect as soon as possible
if (true == depthReceived)
{
this.sensor.CoordinateMapper.MapDepthFrameToColorFrame(
DepthFormat,
this.depthPixels,
ColorFormat,
this.colorCoordinates);
Array.Clear(this.greenScreenPixelData, 0, this.greenScreenPixelData.Length);
// loop over each row and column of the depth
for (int y = 0; y < this.depthHeight; ++y)
{
for (int x = 0; x < this.depthWidth; ++x)
{
// calculate index into depth array
int depthIndex = x + (y * this.depthWidth);
DepthImagePixel depthPixel = this.depthPixels[depthIndex];
int player = depthPixel.PlayerIndex;
// if we're tracking a player for the current pixel, do green screen
if (player > 0)
{
// retrieve the depth to color mapping for the current depth pixel
ColorImagePoint colorImagePoint = this.colorCoordinates[depthIndex];
// scale color coordinates to depth resolution
int colorInDepthX = colorImagePoint.X / this.colorToDepthDivisor;
int colorInDepthY = colorImagePoint.Y / this.colorToDepthDivisor;
// make sure the depth pixel maps to a valid point in color space
// check y > 0 and y < depthHeight to make sure we don't write outside of the array
// check x > 0 instead of >= 0 since to fill gaps we set opaque current pixel plus the one to the left
// because of how the sensor works it is more correct to do it this way than to set to the right
if (colorInDepthX > 0 && colorInDepthX < this.depthWidth && colorInDepthY >= 0 && colorInDepthY < this.depthHeight)
{
// calculate index into the green screen pixel array
int greenScreenIndex = colorInDepthX + (colorInDepthY * this.depthWidth);
// set opaque
this.greenScreenPixelData[greenScreenIndex] = opaquePixelValue;
// compensate for depth/color not corresponding exactly by setting the pixel
// to the left to opaque as well
this.greenScreenPixelData[greenScreenIndex - 1] = opaquePixelValue;
}
}
}
}
}
// do our processing outside of the using block
// so that we return resources to the kinect as soon as possible
if (true == colorReceived)
{
// Write the pixel data into our bitmap
this.colorBitmap.WritePixels(
new Int32Rect(0, 0, this.colorBitmap.PixelWidth, this.colorBitmap.PixelHeight),
this.colorPixels,
this.colorBitmap.PixelWidth * sizeof(int),
0);
if (this.playerOpacityMaskImage == null)
{
this.playerOpacityMaskImage = new WriteableBitmap(
this.depthWidth,
this.depthHeight,
96,
96,
PixelFormats.Bgra32,
null);
MaskedColor.OpacityMask = new ImageBrush { ImageSource = this.playerOpacityMaskImage };
}
this.playerOpacityMaskImage.WritePixels(
new Int32Rect(0, 0, this.depthWidth, this.depthHeight),
this.greenScreenPixelData,
this.depthWidth * ((this.playerOpacityMaskImage.Format.BitsPerPixel + 7) / 8),
0);
}
}
If you are interested in only a single player, you could look into using the player mask to more quickly extract the appropriate pixel set. You'd fi
using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
{
if (skeletonFrame != null && skeletonFrame.SkeletonArrayLength > 0)
{
if (_skeletons == null || _skeletons.Length != skeletonFrame.SkeletonArrayLength)
{
_skeletons = new Skeleton[skeletonFrame.SkeletonArrayLength];
}
skeletonFrame.CopySkeletonDataTo(_skeletons);
// grab the tracked skeleton and set the playerIndex for use pulling
// the depth data out for the silhouette.
this.playerIndex = -1;
for (int i = 0; i < _skeletons.Length; i++)
{
if (_skeletons[i].TrackingState != SkeletonTrackingState.NotTracked)
{
this.playerIndex = i+1;
}
}
}
}
You can then step through the depth data to extract the appropriate bits:
depthFrame.CopyPixelDataTo(this.pixelData);
for (int i16 = 0, i32 = 0; i16 < pixelData.Length && i32 < depthFrame32.Length; i16++, i32 += 4)
{
int player = pixelData[i16] & DepthImageFrame.PlayerIndexBitmask;
if (player == this.playerIndex)
{
// the player we are tracking
}
else if (player > 0)
{
// a player, but not the one we want.
}
else
{
// background or something else we don't care about
}
}
I'm pulling this code from a control I use to produce a silhouette, so it does not deal with the color stream. However, making a call to MapDepthFrameToColorFrame at the appropriate time should allow you to deal with the color stream data and extract the corresponding pixels to the player's mask.