I have a Google Tango application where position repeat accuracy of a learned area description is critical.
https://www.youtube.com/watch?v=jMbNykdPDRo
When area description tracking is activated, the repeat accuracy normally is high. However, if finding learned feature points in a learned area description fails after a certain time (e.g. after lighting conditions changed or the device entered a not yet learned area), repeat accuracy decreases (due to starting missing area description).
How could we get some feedback from the area tracking success to detect such as case?
Examples: how many feature points/frames were successfully tracked against some ADF, or how confident/accurate the pose position actually is for a certain area description?
I cannot find anything useful in the API.
I know such a that field is existing in the Tango Core as you can clearly see in the Tango Debug Overlay App ('Tracking Success').
http://grauonline.de/alexwww/tmp/tango_debug_overlay_app.jpg
The tracking data is not available through API, the tool you saw from the picture is developed by Tango team, and it can be downloaded here.
On top of API, the best way to identify pose quality is to check pose status returned from callbacks, and check TangoEvent data.
Related
I would need to make my dron Mavic2 Pro to visit approx 10 locations in relatively low altitude 1.7 m. In each location the camera should look at right direction and the mission paused to let the mobile application process some CV tasks. I am not sure what is the best approach to make a mission that is partially processed on mobile platform? What to use in DJI mobile sdk api to pause mission when the location is reached?
I am going to use a time line mission composed from sequence of GoToAction. I wonder if this is a good way to do it. Is there a better solution?
Is MissionControl.Listener right place to interrupt a mission when a TimeLineElement finish or should I use WaypointReachedTrigger?
I wasn't able to find any suitable example.
Please add specific programming question. Otherwise, all answer is primarily opinion-based see https://stackoverflow.com/help/dont-ask for detail
DJI method allows you to control the drones gimbal and gps navigation through MissionAction. The gotoaction is a subclass of the Missionaction class. the gotoaction only goes to some GPS location . So you need other things in the mission action such as gimbalattiudeaction and camera capture action to perform camera pointing and capturing. See the Fig below.
For CV tasks, it is easy to link DJI app to OpenCV. But I highly not recommend you to do so as the task such as dection using CNN system takes too much resources. The popular approach is to upload the image taken in local buffer to a local server with GPU for processing in near real-time manner. See the Fig below, I`m using WSDK with windows online OCR for detection. Video at https://youtu.be/CcndnHkriyA . I tried with local phone based approch, but result is limited by the model accuracy. And I could not apply high accuracy model because of the processing demand for such model is high. You can see my demo in the Fig below
What you want is pretty ez to implement but hard to perfect. Flying in low altitude(1.7m) requires you to have some degree of obstacle avoidance and GPSless path planning. The one implemented in Mavic hardware is only simple avoidance or slip through. For a bit more complex like go around a wall or maze-like environment, it better to add your own global path planer and local path planner. For the feedback you can use the SVO method to get odometry and map the local sparse obstacle map for inflated radius calculation. See Fig below.
Fig taken from video https://www.youtube.com/watch?v=2YnIMfw6bJY.
The feedback code is available at https://github.com/uzh-rpg/rpg_svo.
The path planning code you can try with ETH`s https://github.com/ethz-asl/mav_voxblox_planning as well.
Good luck with your work.
I'm looking for more information on how to use drift-correction correctly (using Unity SDK).
On the Tango website it says "Drift-corrected frames come through the Area Description reference frame", that the frame pair Start Of Service -> Device "does not include drift correction" and for Area Description -> Start Of Service that it "provides updates only when a localization event or a drift correction occurs".
The way I'd like to use a drift-corrected pose is like in the TangoPointCloud prefab, where depth points are multiplied by a matrix startServiceTDevice which results from the frame pair SoS -> Device. Assuming that the drift-corrected frame is in the AD frame, I'd need SoS -> AD. Since only AD -> SoS is available, I tried with this one and its inverse. The resulting pose is too small though to make any sense (even if using it the wrong direction, the translation shouldn't be close to zero if I had been walking around). Then I considered that the AD frame might actually be something like a drift-corrected Start of Service, but then again I can't find any significant/visible difference between AD -> Device and SoS -> Device, definitely no loop closures in it. I'm requesting and applying poses after finishing my scan, so drifts should have been detected by then.
On the Tango website it's further said that "There will be a period after Startup during which drift-corrected frames are not available.", yet the AD -> SoS pose is available (and valid) from the beginning and I couldn't yet produce a situation where it wasn't (e.g. no motion, rapid motion...).
Is drift correction working at all? Or am I using it all wrong?
PS: On the latest stackoverflow post it sounds as if drift correction would be for relocalization after tracking loss only. However, I find this hard to believe since the Tango website describes drift correction as "When the device sees a place it knows it has seen earlier in your session, it realizes it has traveled in a loop and adjusts its path to be more consistent with its previous observations.".
Drift correction is working as experimental features at this moment, there's corner cases that it will break. I will go into more details later.
In order to use drift correction pose, you will need to use ADF_T_Device frame pair (ADF is base frame, Device is target frame). In the example of using drift-correction pose to project points into world space, you don't need to do Adf_T_ss * ss_T_device transform, instead, all you only need to use ADF_T_device frame directly. If this is in Unity, you can just check the use area description pose on PointCloud prefab.
Corner cases that breaks drift-correction:
User shakes the device right after starting the experience.
Under the hood, drift correction is constructing a more dense but more accurate version of ADF. If user covers camera or shake device at the very beginning, that will cause that no ADF (or features) being saved in the buffer. Thus the API could get into a state that never gives any valid pose from ADF_T_Device frame pair.
Device lost tracking, and user moved to a new space without relocalizing.
This is similar to the first case. If user moved to a new space without relocalizing after lost tracking, device will never relocalized, thus no valid pose will be available through ADF_T_device frame.
Drift correction API is still experimental, we are trying to address above issues from API level as well.
As far as I know, the main features of Project Tango SDK are:
motion tracking
area learning
depth perception
But what about object recognition & tracking?
I didn't see anything about that in the SDK, though I assume that Tango hardware would be really good at it (compared with traditional smartphones). Why?
Update 2017/06/05
A marker detection API has been introduced in the Hopak release
There are already good libraries for object tracking in 2D images and the additional features of project tango would likely add only marginal improvement in performance(of existing functions) for major overhauls of the library to support a small set of evolving hardware.
How do you think project tango could improve on existing object recognition & tracking?
With a 3d model of the object to be tracked, and knowledge of the motion and pose of the camera, one could predict what the next image of the tracked object 'should' look like. If the next image is different than predicted, it could be assumed that the tracked object has moved from its prior position. And the actual new 3D image could indicate the tracked object's vectors. That certainly has uses in navigating a live environment.
But that sounds like the sort of solution a self driving car might use. And that would be a valuable piece of tech worth keeping away from competitors despite its value to the community.
This is all just speculation. I have no first hand knowledge.
I'm not really sure what you're expecting for an "open question", but I can tell you one common way that people exploit Tango's capabilities to aid object recognition & tracking. Tango's point cloud, image callbacks, and pose data can be used as input for a library like PCL (http://pointclouds.org/).
Simply browsing the documentation & tutorials will give you a good idea of what's possible and how it can be achieved.
http://pointclouds.org/documentation/
Beyond that, you might browse the pcl-users mail archives:
http://www.pcl-users.org/
The Tango pose is drifting around while holding the device still when the camera is facing a region without too many distinct features, i.e. facing a white wall. Typically the drift direction is away from the target it is facing. I understand it is hard for the device to localize itself under such a condition due to lack of landmarks. However, is there a mechanism to let the device know that it has difficulties in getting reliable pose, then I am able to tell the device stop doing something until the device is relocalized by going back to the area with rich landmarks or features.
Note: the pose status is still showing valid in this case.
Please check tango_client_api_header
you are looking for TANGO_POSE_INVALID
another way you want is integrating UX library.
check this below:
https://developers.google.com/project-tango/ux/ux-framework#exception_handling
or you can write up your own handle for different module.
Is there anyway to disable the "inertial motion sensors" for a program, so that my program does not track the devices acceleration? I've noticed that if a user suddenly moves and then suddenly stops with the device the motion tracking becomes inaccurate.
Even if you could, "disabling the sensors" is not a good idea. And when you say "disabling" I assume you mean setting them to an initialization state rather than just ignoring the data stream. The 3-axis accelerometer and gyroscope data are fused with position data to provide your relative location during motion tracking. You have no way of knowing which of these data streams is the source of the inaccuracy, and just turning off acceleration would likely require a re-calibration of all sensors so that tracking (data fusion) is accurate.
Replicate the error with as much data as possible (speed, stopping time, orientation of the tablet, distance to the nearest object, nearest object characteristics, etc.) and report it to the project Tango team.