Kinect Color and Depth Stream Performance - performance

Question

I am developing an XNA game. In my game I am using Kinect's Color Stream and Depth Stream to get the image of the player only. For this cause I check the depth pixels and find the pixels with PlayerIndex > 0. Then I map these depth points to color points with MapDepthPointToColorPoint method and get the colors of these pixels from color video.
This works normally but the performance is very bad (especially when this is for a game). When I closed color stream and set the pixels with black color (I mean pixels with player index), it's all smooth. But when I enable color stream it is not very effective.
I tried 2 things in AllFramesReady function for this :
1-
using (ColorImageFrame colorVideoFrame = imageFrames.OpenColorImageFrame())
{
//color logic
}
using (DepthImageFrame depthVideoFrame = imageFrames.OpenDepthImageFrame())
{
//depth logic
}
and
2-
using (ColorImageFrame colorVideoFrame = imageFrames.OpenColorImageFrame())
{
colorReceived=true;
}
if (colorReceived){
//color logic
}
using (DepthImageFrame depthVideoFrame = imageFrames.OpenDepthImageFrame())
{
depthReceived=true;
}
if (depthReceived){
//depth logic
}
The second one seems to have a better performance because it applies color and depth logic outside of using blocks and returns resources to Kinect as soon as possible. But when the second player comes in performance decreases drastically. But sometimes the image of the player just disappears for 1 or 2 frames when I use the second choice.
What more can I do to increase performance of color and depth streams? Thanks for any help.