Problem Description:
we have a camera that is sending video of a live sports game in 30 frames per second.
on the other side we have a screen that is representing immediately every fram that is coming.
Assumptions
*frames will arrive in order
1.what will be the experience for a person that is wathcing the screen?
2.what can we do in order to improve it?
Your playback will have very variable framerate which would cause visible artifacts during any smooth movement ...
To remedy this You need to implement image FIFO that will cover bigger time that is your worst delay difference (Idealy at least 2x times more). So if you got 300ms-100ms delay difference and 30 fps then minimal FIFO size is:
n = 2 * (300-100) * 0.001 * 30 = 12 images
Now the reproduction should be like this:
init playback
simply start obtaining images into FIFO until the FIFO is half FULL (contains images for biggest delay difference)
playback
so any incoming image is inserted into FIFO at time of receival (unless FIFO is full in which case you wait until you have room to place new image or skip frame). Meanwhile in some thread or timer (that runs in parallel) you fetch image from FIFO every 1/30 seconds and render it (if the FIFO is empty you use last image and you can even go to bullet #1 again).
playback stop
once FIFO is empty for longer duration then some threshold (no new frames are incoming) you stop the playback.
The FIFO size reserve and point when to start playback depends on the image source timing properties (so it does not overrun nor underrun the FIFO)...
In case you need to implement your own FIFO class then cyclic buffer of constant size is your friend (so you do not need to copy all the stored images on FIFO in/out operations).
Related
I'm trying to check 360 Video performance with version 11.62465, the 360 video output can't reach to 60P when we play FHD#60P 360 video. The video decode will update video frame in every 14ms~17 ms, but the application can't call SbPlayerGetCurrentFrame() 16ms, then the drop frame happens. The drop frame rate is nearly 20%. I tried to use chrome://tracing/ check the performance and found that sometimes the rasterizer lasted 40ms, the CPU duration is only 8ms, does this mean the GPU ability is not enough? Will the cobalt render thread be blocked by other modules?
If the CPU duration per-frame is 8ms, then it does sound like the GPU is not powerful enough to process each video frame fast enough. The Cobalt render thread should never be blocked by other modules, though it may be worth double checking that your implementation of SbPlayerGetCurrentFrame() is not taking a long time to render (perhaps it acquires a lock?).
You can use chrome://tracing/ to check the performance of the renderer when a non-360 FHD#60P video is playing, and compare that performance to when a 360 video is playing. This will tell you whether the renderer's performance is affected by the decode-to-texture process or not.
I have started looking into the client side performance of my app using Chrome's Timeline tools. However, whilst I have found many articles on how to use them, information on how to interpret the results is more sparse and often vague.
Currently I am looking at scroll performance and attempting to hit 60FPS.
This screenshot show's the results of my most recent timeline recording.
As can be seen most frames are over 60 FPS and several are over 30 FPS.
If I zoom in on one particular frame - the one with duration 67.076ms I can see a few things:
the duration of the frame is 67ms, but aggregated time is 204ms
201ms of this time is spent painting BUT the two paint events in
this frame are of duration 1.327 ms and 0.106 ms
The total duration for the JS event, update layer tree and paint events is
only 2.4 ms
There is a long green hollow bar (rasterize Paint)
which lasts the duration of the frame and in fact starts before and
continues after it.
I have a few questions on this:
the aggregated time is far longer than the frame time - is it
correct to assume that these are parrallel processes?
the paint time for the frame (204ms) far exceeds the time for the two paint events (1.433ms) - is this because it includes the
rasterize paint events
why does the rasterize paint event span multiple frames?
where would one start optimizing this?
Finally can someone point me to some good resources on understanding this?
This is somewhat unfortunate result of the way the 'classic' waterfall Timeline view coalesces multiple events. If you expand that long "Rasterize Paint" event, you'll see a bunch of individual events, which are going to be somewhat shorter. You may really want to switch to Flame Chart mode for troubleshooting rendering performance, where the rasterize events are shown on appropriate threads.
We get timestamps as a double value for pose, picture, and point data - they aren't always aligned - how do I calculate the temporal distance between two time stamps ? Yes, I know how to subtract two doubles, but I'm not at all sure of how the delta corresponds to time.
I have some interesting timestamp data that sheds light on your question, without exactly answering it. I have been trying to match up depth frames with image frames - just as a lot of people posting under this Tango tag. My data did not match exactly and I thought there were problems with my projection matrices and point reprojection. Then I checked the timestamps on my depth frames and image frames and found that they were off by as much as 130 milliseconds. A lot! Even though I was getting the most recent image whenever a depth frame was available. So I went back to test just the timestamp data.
I am working in Native with code based on the point-cloud-jni-example. For each of onXYZijAvailable(), onFrameAvailable(), and onPoseAvailable() I am dumping out time information. In the XYZ and Frame cases I am copying the returned data to a static buffer for later use. For this test I am ignoring the buffered image frame, and the XYZ depth data is displayed in the normal OpenGL display loop of the example code. The data captured looks like this:
callback type : systime : timestamp : last pose
I/tango_jni_example( 3247): TM CLK Img 5.420798 110.914437 110.845522
I/tango_jni_example( 3247): TM CLK XYZ 5.448181 110.792470 110.845522
I/tango_jni_example( 3247): TM CLK Pose 5.454577 110.878850
I/tango_jni_example( 3247): TM CLK Img 5.458924 110.947708 110.878850
I/tango_jni_example( 3247): TM CLK Pose 5.468766 110.912178
The system time is from std::chrono::system_clock::now() run inside of each callback. (Offset by a start time at app start.) The timestamp is the actual timestamp data from the XYZij, image, or pose struct. For depth and image I also list the most recent pose timestamp (from start-of-service to device, with given time of 0.0). A quick analysis of about 2 minutes of sample data leads to the following initial conclusions:
Pose data is captured at VERY regular intervals of 0.033328 seconds.
Depth data is captured at pretty regular intervals of 0.2 seconds.
Image data is captured at odd intervals
with 3 or 4 frames at 0.033 seconds
then 1 frame at about 0.100 seconds
often followed by a second frame with the same timestamp
(even though it is not reported until the next onFrameAvailable()?)
That is the actual timestamp data in the returned structs. The "real?" elapsed time between callbacks is much more variable. The pose callback fires anywhere from 0.010 to 0.079 seconds, even though the pose timestamps are rock solid at 0.033. The image (frame) callback fires 4 times at between 0.025 and 0.040 and then gives one long pause of around 0.065. That is where two images with the same timestamp are returned in successive calls. It appears that the camera is skipping a frame?
So, to match depth, image, and pose you really need to buffer multiple returns with their corresponding timestamps (ring buffer?) and then match them up by whichever value you want as master. Pose times are the most stable.
Note: I have not tried to get a pose for a particular "in between" time to see if the returned pose is interpolated between the values given by onPoseAvailable().
I have the logcat file and various awk extracts available. I am not sure how to post those (1000's of lines).
I think the fundamental question would be how to sync the pose, depth and color image data together into a single frame. So to answer that, there are actually two step
Sync pose to either color image or depth: to do that, the simplest way is to use the TangoService_getPoseAtTime function, that basically gives you the ability to query a pose with certain timestamp. i.e, you have a depth point cloud available, and it gives you a timestamp of that depth frame, then you could use the depth point cloud timestamp to query the corresponding pose.
Sync color image and depth image: currently, you would have to buffer either the depth point cloud or the color image at the application level, and base on one of their's timestamp, query the other's data in the buffer. There is a field name color_image in the TangoXYZij data structure, and the comment says it's reserved for future use, so the built-in sync up feature might be coming in future releases.
I have an animated gif much like this one, where the cyclic nature of the animation causes some frames to be repeated twice within one full loop of the animation.
(From here)
I am currently storing each frame separately in the gif. Is it possible to only store each repeated frame once, to effectively halve the storage space required?
I am creating my gif in MATLAB using the movie2gif converter, but would be happy with an alternative method for gif creation or a post-processing tool.
EDIT
What I mean by the frame repetition is best explained in the context of this example image. There is a frame shown just as the left-hand ball leaves the row of balls. That exact frame is repeated a few frames later, when the left-hand ball is now on its way back to hit the row of balls again. Because of the ordering of frames, it is currently needed to store this
frame twice.
To clarify what I am looking for: I want a way of saving the gif (or post-processing the gif) such that I can keep the full animation sequence (e.g. of 30 frames), but frames which repeat are soft-linked back to the first showing of them, thus removing the need to store them twice.
Judging from the description of movie2gif and its input arguements, it does not appear to be possible. Furthermore, when reading how gifs work (and LZW) compression I can imagine that it is not even possible to reduce the size of a gif like this.
If you want to save only the images that are minimally required and don't mind building the image before you can see it, then you can just store each image and an indexing vector.
In your case it may be possible to find a way to just save half of the image, and then play it in a cycle: forward-backward-forward ... but I don't know whether this is possible.
I am trying to create a small project wherein I need to capture/read the video frame buffer and calculate the average RGB value of the screen.
I don't need to write anything on the screen. I'm doing this in Windows.
Can anyone help me with any Windows API which will read the video frame buffer and calculate the average RGB value?
What I came to know is that I need to write a kernel driver which will have access to read the frame buffer.
Is this the only solution?
Is there any other way of reading frame buffer?
Is there an algorithm to calculate the RGB value from frame buffer data?
If you want really good performance, you might have to use directx and capture the backbuffer to a texture. Using mipmaps, it will automatically create downsamples all the way to 1X1. Justgrab the color of that 1 pixel and you're good to go.
Good luck, though. I'm working on implimenting this as we speak. I'm creating an ambient light control for my room. I was getting about 15FPS using device contexts and StretchBLT. Only got decent performance if I grabbed 1 pixel with GetPixel(). That's an i5 3570K # 4.5GHz
But with the directx method, you could technically get hundreds if not thousands of frames per second. (when I make a spinning triangle, my 660 gets about 24,000 FPS. It couldn't be TOO much slower, minus the CPU calls.)