I am capturing data from the web camera by using DirectShow api. To change white balance value I call IAMVideoProcAmp::Set method.
I have noticed that for some cameras white balance value is being changed immediately (after 1-2 frames new values is already applied). But for other cameras it is being applied incrementally during 50-60 frames. It is too long for me.
May be someone has faced with the same problem. Can I configure how fast new value will be applied or does it depend on the camera's driver?
IAMVideoProcAmp::Set is all you have. There is no generic way to change white balance or affect the way changes take effect. If you are interested in specific models of cameras, you might check with tech support if there is SDK available and model-specific ways to setup the device.
Related
Is it possible to modify the Vuforia video stream for better tracking performance?
Step 1: Get the raw pixel data from the VuforiaBehaviour.Instance.CameraDevice.GetCameraImage();
Step 2: Modify the pixels with post processing via custom shaders in Unity. For example apply a threshold or edge detection.
Step 3: Vuforia Engine uses the modified video input to track images.
That´s the idea but I´m not sure if Vuforia is gonna pass the modified video into the Vuforia Engine then or still uses the unmodified video input for tracking?
If anybody has experience with that I would be thankful for your help! :)
Vuforia Engine assumes that the input images look like "natural" images. Passing an image belonging to a different domain (e.g., the result of an edge detector) is unlikely to improve tracking performance.
That said, tracking performance is affected by image quality. For example, if images are blurry, tracking robustness is going to suffer. If this is the case you might want to look at trying to adjust system camera parameters via the platform API (iOS, Android, etc.). However, please note that this might or might not be possible depending on the platform. Also, on some platforms when a device tracker like ARKit or ARCore is used, the platform tracker itself adjusts the camera parameters for good tracking performance. For example it might keep the exposure time low to reduce blur.
I'm looking for more information on how to use drift-correction correctly (using Unity SDK).
On the Tango website it says "Drift-corrected frames come through the Area Description reference frame", that the frame pair Start Of Service -> Device "does not include drift correction" and for Area Description -> Start Of Service that it "provides updates only when a localization event or a drift correction occurs".
The way I'd like to use a drift-corrected pose is like in the TangoPointCloud prefab, where depth points are multiplied by a matrix startServiceTDevice which results from the frame pair SoS -> Device. Assuming that the drift-corrected frame is in the AD frame, I'd need SoS -> AD. Since only AD -> SoS is available, I tried with this one and its inverse. The resulting pose is too small though to make any sense (even if using it the wrong direction, the translation shouldn't be close to zero if I had been walking around). Then I considered that the AD frame might actually be something like a drift-corrected Start of Service, but then again I can't find any significant/visible difference between AD -> Device and SoS -> Device, definitely no loop closures in it. I'm requesting and applying poses after finishing my scan, so drifts should have been detected by then.
On the Tango website it's further said that "There will be a period after Startup during which drift-corrected frames are not available.", yet the AD -> SoS pose is available (and valid) from the beginning and I couldn't yet produce a situation where it wasn't (e.g. no motion, rapid motion...).
Is drift correction working at all? Or am I using it all wrong?
PS: On the latest stackoverflow post it sounds as if drift correction would be for relocalization after tracking loss only. However, I find this hard to believe since the Tango website describes drift correction as "When the device sees a place it knows it has seen earlier in your session, it realizes it has traveled in a loop and adjusts its path to be more consistent with its previous observations.".
Drift correction is working as experimental features at this moment, there's corner cases that it will break. I will go into more details later.
In order to use drift correction pose, you will need to use ADF_T_Device frame pair (ADF is base frame, Device is target frame). In the example of using drift-correction pose to project points into world space, you don't need to do Adf_T_ss * ss_T_device transform, instead, all you only need to use ADF_T_device frame directly. If this is in Unity, you can just check the use area description pose on PointCloud prefab.
Corner cases that breaks drift-correction:
User shakes the device right after starting the experience.
Under the hood, drift correction is constructing a more dense but more accurate version of ADF. If user covers camera or shake device at the very beginning, that will cause that no ADF (or features) being saved in the buffer. Thus the API could get into a state that never gives any valid pose from ADF_T_Device frame pair.
Device lost tracking, and user moved to a new space without relocalizing.
This is similar to the first case. If user moved to a new space without relocalizing after lost tracking, device will never relocalized, thus no valid pose will be available through ADF_T_device frame.
Drift correction API is still experimental, we are trying to address above issues from API level as well.
I am getting my first Tango in the next day or so; worked a little bit with Occipital's Structure Sensor - which is where my background in depth perceiving camera's come from.
Has anyone used multiple Tango at once (lets say 6-10), looking at the same part of a room, using depth for identification and placement of 3d character/content? I have been told that multiple devices looking at the same part of a room will confuse each Tango as they will see the other Tango's IR dots.
Thanks for your input.
Grisly
I have not tried to use several Tangos, but I have however tried to use my Tango in a room where I had a Kinect 2 sensor, which caused the Tango to go bananas. It seems however like the Tango has lower intensity on its IR projector in comparison, but I would still say that it is a reasonable assumption that it will not work.
It might work under certain angles but I doubt that you will be able to find a configuration of that many cameras without any of them interfering with each other. If you would make it work however, I would be very interested to know how.
You could lower the depth camera rate (defaults to 5/second I believe) to avoid conflicts, but that might not be desirable given what you're using the system for.
Alternatively, only enable the depth camera when placing your 3D models on surfaces, then disable said depth camera when it is not needed. This can also help conserve CPU and battery power.
It did not work. Occipital Structure Sensor on the other hand, did work (multiple devices in one place)!
I've been developing a virtual camera app for depth cameras and I'm extremely interested in the Tango project. I have several questions regarding the cameras on board. I can't seem to find these specs anywhere in the developer section or forums, so I understand completely if these cant be answered publicly. I thought I would ask regardless and see if the current device is suitable for my app.
Are the depth and color images from the rgb/ir camera captured simultaneously?
What frame rates is the rgb/ir capable of? e.g. 30, 25, 24? And at what resolutions?
Does the motion tracking camera run in sync with the rgb/ir camera? If not what frame rate (or refresh rate) does the motion tracking camera run at? Also if they do not run on the same clock does the API expose a relative or an absolute time stamp for both cameras?
What manual controls (if any) are exposed for the color camera? Frame rate, gain, exposure time, white balance?
If the color camera is fully automatic, does it automatically drop its frame rate in low light situations?
Thank you so much for your time!
Edit: Im specifically referring to the new tablet.
Some guessing
No, the actual image used to generate the point cloud is not the droid you want - I put up a picture on Google+ that shows what you get when you get one of the images that has the IR pattern used to calculate depth (an aside - it looks suspiciously like a Serpinski curve to me
Image frame rate is considerably higher than point cloud frame rate, but seems variable - probably a function of the load that Tango imposes
Motion tracking, i.e. pose, is captured at a rate roughly 3x the pose cloud rate
Timestamps are done with the most fascinating double precision number - in prior releases there was definitely artifacts/data in the lsb's of the double - I do a getposeattime (callbacks used for ADF localization) when I pick up a cloud, so supposedly I've got a pose aligned with the cloud - images have very low timestamp correspondance with pose and cloud data - it's very important to note that the 3 tango streams (pose,image,cloud) all return timestamps
Don't know about camera controls yet - still wedging OpenCV into the cloud services :-) Low light will be interesting - anecdotal data indicates that Tango has a wider visual spectrum than we do, which makes me wonder if fiddling with the camera at the point of capture to change image quality, e.g. dropping the frame rate, might not cause Tango problems
I am creating an augmented reality application that uses the depth information to change how objects render within the color image. I'm not sure how frequently I should expect new frames or how to make sure I'm matching the correct depth samples with the right color image frames.
The Tango Phone Development Kit and Tango Tablet Development Kit both update the RGB image at an average of 25hz. The depth for the Phone is sampled at 5hz, while the Tablet currently runs at 2-3hz, but may increase in later software releases.
The color and depth data are not synchronized, but in both platforms the API provides timestamps for all data as well as an interface to request data at a given timestamp, so the application can decide how best to manage the data.