Rendering to CAMetalLayer from dedicated render thread / loop - macos

In Windows World, a dedicated render thread would loop something similar to this:
void RenderThread()
{
while (!quit)
{
UpdateStates();
RenderToDirect3D();
// Can either present with no synchronisation,
// or synchronise after 1-4 vertical blanks.
// See docs for IDXGISwapChain::Present
PresentToSwapChain();
}
}
What is the equivalent in Cocoa with CAMetalLayer? All the examples deal with updates being done in the main thread, either using MTKView (with it's internal timer) or using CADisplayLink in the iOS examples.
I want to be in control of the whole render loop, rather than just receiving a callback at some non-specified interval (and ideally blocking for V-Sync if it's enabled).

At some level, you're going to be throttled by the availability of drawables. A CAMetalLayer has a fixed pool of drawables available, and calling nextDrawable will block the current thread until a drawable becomes available. This doesn't imply you have to call nextDrawable at the top of your render loop, though.
If you want to draw on your own schedule without getting blocked waiting on a drawable, render to an off-screen renderbuffer (i.e., a MTLTexture with dimensions matching your drawable size), and then blit from the most-recently-drawn texture to a drawable's texture and present on whatever cadence you prefer. This can be useful for getting frame timings, but every frame you draw and then don't display is wasted work. It also increases the risk of judder.
Your options are limited when it comes to getting callbacks that match the v-sync cadence. Your best is almost certainly a CVDisplayLink scheduled in the default and tracking run loop modes, though this has caveats.
You could use something like a counting semaphore in concert with a display link if you want to free-run without getting too far ahead.
If your application is able to maintain a real-time framerate, you'll normally be rendering a frame or two ahead of what's going on the glass, so you don't want to literally block on v-sync; you just want to inform the window server that you'd like presentation to match v-sync. On macOS, you do this by setting the layer's displaySyncEnabled to true (the default). Turning this off may cause tearing on certain displays.

At the point where you want to render to screen, you obtain the drawable from the layer by calling nextDrawable. You obtain the drawable's texture from its texture property. You use that texture to set up the render target (color attachment) of a MTLRenderPassDescriptor. For example:
id<CAMetalDrawable> drawable = layer.nextDrawable;
id<MTLTexture> texture = drawable.texture;
MTLRenderPassDescriptor *desc = [MTLRenderPassDescriptor renderPassDescriptor];
desc.colorAttachments[0].texture = texture;
From here, it's pretty similar to what you do in an MTKView's drawRect: method. You create a command buffer (if you don't already have one), create a render command encoder using the descriptor, encode drawing commands, end encoding, tell the command buffer to present the drawable (using a -presentDrawable:... method), and commit the command buffer. Whatever was drawn to the drawable's texture is what will end up on-screen when it's presented.
I agree with Warren that you probably don't really want to sync your loop with the display refresh. You want parallelism. You want the CPU to be working on the next frame while the GPU is rendering the most current frame (and the display is showing the last frame).
The fact that there's a limit on how many drawables may be in flight at once and that nextDrawable will block waiting for one will prevent your render loop from getting too far ahead. (You'll probably use some other synchronization before that, like for managing a small pool of buffers.) If you want only double-buffering and not triple-buffering, you can set the layer's maximumDrawableCount to 2 instead of its default value of 3.

Related

Calling functions only after drawing is completed

I am making a drawing on NSView using a timer that is set to update every .02 seconds. On update some physical simulation makes a step, and then Canvas!.needsDisplay = true. It works when app is in foreground (usually), but when some lags happen, simulation progresses forward despite the fact that view hasn't reflected it yet. How do I pause the timer during these times to make simulation happen only when NSView can show it? I do not want to call step_over from inside drawRect, cause it seems like a bad idea, because then it would be harder to stop the simulation.
Generally this kind of update should be done the other way around, by letting the display ask you for frames as it can display them. This is done with a CADisplayLink CVDisplayLink (this is Mac; CADisplayLink is iOS). Configure it with a method you want to be called when a frame can be drawn.
Generally you do want your simulation to keep moving forward, even if it means dropping frames occasionally. For that, you check the timestamp and use that to work out what time to use for your new frame. But if you only want to move forward when the display can show it, then just update once per call.
Note that generating at 50fps is often going to mismatch the system that's trying to draw at 60fps, so you're going to wind up missing frames occasionally. That's one of several reasons not to try to push drawing with a timer.
See also Alternative of CADisplayLink for Mac OS X. Note that trying to draw at 50fps with Core Graphics usually isn't going to give good results in any case. The right tool here in OS X is Core Animation (or SpriteKit for games on 10.10, or OpenGL for more advanced high-speed rendering). You can do very basic animations with an NSTimer (and we did for years before Core Animation came along), but it's not really a tool for complex drawing.

Rendering OpenGL just once rather than every frame

Nearly every example I see of OpenGL ES involves it updating every frame, even if the image itself is not moving in any way.
I did some tests and I see it works quite fine to just render (using drawArrays etc) and then present the render buffer (these two actions, together) just once and then not do either again until you have something change onscreen.
Is this "normal" ? I just don't see this really done much. Once drawn, the graphics stay on the screen without additional constant rendering.
Is this acceptable?
Yes, it is acceptable and completely valid. You also need to take account to render again when the context is lost. To give you an example, using Android standard OpenGL helper classes there is an option to only draw when needed, not in loop (RENDERMODE_WHEN_DIRTY).

NSWindow Flip Animation - Like iWork

I'm attempting to implement window-flipping identical to that in iWork -
https://dl.dropbox.com/u/2338382/Window%20Flipping.mov
However, I can't quite seem to find a straightforward way of doing this. Some tutorials suggest sticking snapshot-images of both sides of the window in a bigger, transparent window and animate those. This might work, but seems a bit hacky, and the sample code is always bloated. Some tutorials suggest using private APIs, and since this app may be MAS-bound, I'd like to avoid that.
How should I go about implementing this? Does anyone have any hints?
NSWindow+Flipping
I've rewritten the ancient code linked below into NSWindow+Flipping. You can grab these source files from my misc. Cocoa collection on GitHub, PCSnippets.
You can achieve this using CoreGraphics framework. Take a look at this:
- (void) flipWithDuration: (float) duration forwards: (BOOL) forwards
{
CGSTransitionSpec spec;
CGSTransitionHandle transitionHandle;
CGSConnection cid = CGSDefaultConnection;
spec.type = CGSFlip;
spec.option = 0x80 | (forwards ? 2 : 1);
spec.wid = [self windowNumber];
spec.backColor = nil;
transitionHandle = -1;
CGSNewTransition (cid, &spec, &transitionHandle);
CGSInvokeTransition (cid, transitionHandle, duration);
[[NSRunLoop currentRunLoop] runUntilDate:
[NSDate dateWithTimeIntervalSinceNow: duration]];
CGSReleaseTransition (cid, transitionHandle);
}
You can download sample project: here. More info here.
UPDATE:
Take a look at this project. It's actually what You need.
About this project:
This category on NSWindow allows you to switch one window for
another, using the "flip" animation popularized by Dashboard widgets.
This was a nice excuse to learn something about CoreImage and how to
use it in Cocoa. The demo app shows how to use it. Scroll to the end
to see what's new in this version!
Basically, all you need to do is something like:
[someWindow flipToShowWindow:someOtherWindow forward:YES];
However, this code makes some assumptions: — someWindow (the initial
window) is already visible on-screen. — someOtherWindow (the final
window) is not already visible on-screen. — Both windows can be
resized to the same size, and aren't too large or complicated — the
latter conditions being less important the faster your CPU/video card
is. — The windows won't go away while the animation is running. — The
user won't try to click on the animated window or do something while
the animation is running.
The implementation is quite straightforward. I move the final to the
same position and size as the initial window. I then position a larger
transparent window so it covers that frame. I render both window
contents into CIImages, hide both windows, and start the animation.
Each frame of the animation renders a perspective-distorted image into
the transparent window. When the animation is done, I show the final
window. Some tricks are used to make this faster; the flipping window
is setup only once; the final window is hidden by setting its alpha to
0.0, not by ordering it out and later ordering it back in again, for instance.
The main bottleneck is the CoreImage filter, and the first frame
always takes much longer to render — 4 or 6 times what it takes for
the remaining frames. I suppose this time is spent with setup and
downloading to the video card. So I calculate the time this takes and
draw a second frame at a stage where the rotation begins to show. The
animation begins at this point, but, if those first two frames took
too long, I stretch the duration to make sure that at least 5 more
frames will get rendered. This will happen with slow hardware or large
windows. At the end, I don't render the last frame at all and swap the
final window in instead.

Should I use NSOperation or NSRunLoop?

I am trying to monitor a stream of video output from a FireWire camera. I have created an Interface Builder interface with buttons and an NSImageView. While image monitoring is occurring within an endless loop, I want to:
change some camera parameters on the fly (gain, gamma, etc.)
tell the monitoring to stop so I can save an image to a file (set a flag that stops the while loop)
Using the button features, I have been unable to loop the video frame monitor, while still looking for a button press (much like using the keypressed feature from C.) Two options present themselves:
Initiate a new run loop (for which I cannot get an autoreleasepool to function ...)
Initiate an NSOperation - how do I do this in a way which allows me to connect with an Xcode button push?
The documentation is very obtuse about the creation of such objects. If I create an NSOperation as per the examples I've found, there seems to be no way to communicate with it with an object from Interface Builder. When I create an NSRunLoop, I get an object leak error, and I can find no example of how to create an autoreleasepool that actually responds to the RunLoop I've created. Nevermind that I haven't even attempted to choose which objects get sampled by the secondary run loop ...
Because Objective C is (obviously!) not my native tongue, I am looking for solutions with baby steps, sorry to say ...
Thanks in advance
I've needed to do almost exactly the same as you, only with a continuous video display from the FireWire camera. In my case, I used the libdc1394 library to perform the frame capture and camera property adjustment for our FireWire cameras. I know you can also do this using some of the Carbon Quicktime functions, but I found libdc1394 to be a little easier to understand.
For the video capture loop, I tried a number of different approaches, from a separate thread that polls the camera and has locks around shared resources, to using one NSOperationQueue for interaction with the camera, and finally settled on using a CVDisplayLink to poll the camera in a way that matches the refresh rate of the screen.
The CVDisplayLink is configured using the following code:
CGDirectDisplayID displayID = CGMainDisplayID();
CVReturn error = kCVReturnSuccess;
error = CVDisplayLinkCreateWithCGDisplay(displayID, &displayLink);
if (error)
{
NSLog(#"DisplayLink created with error:%d", error);
displayLink = NULL;
}
CVDisplayLinkSetOutputCallback(displayLink, renderCallback, self);
and it calls the following function to trigger the retrieval of a new camera frame:
static CVReturn renderCallback(CVDisplayLinkRef displayLink,
const CVTimeStamp *inNow,
const CVTimeStamp *inOutputTime,
CVOptionFlags flagsIn,
CVOptionFlags *flagsOut,
void *displayLinkContext)
{
return [(SPVideoView *)displayLinkContext renderTime:inOutputTime];
}
The CVDisplayLink is started and stopped using the following:
- (void)startRequestingFrames;
{
CVDisplayLinkStart(displayLink);
}
- (void)stopRequestingFrames;
{
CVDisplayLinkStop(displayLink);
}
Rather than using a lock on the FireWire camera communications, whenever I need to adjust the exposure, gain, etc. I change corresponding instance variables and set the appropriate bits within a flag variable to indicate which settings to change. On the next retrieval of a frame, the callback method from the CVDisplayLink changes the appropriate settings on the camera to match the locally stored instance variables and clears that flag.
Display to the screen is handled through an NSOpenGLView (CAOpenGLLayer introduced too many visual artifacts when updating at this rate, and its update callbacks ran on the main thread). Apple has some extensions you can use to provide these frames as textures using DMA for better performance.
Unfortunately, nothing that I've described here is introductory-level stuff. I have about 2,000 lines of code for these camera-handling functions in our software and this took a long time to puzzle out. If Apple could add the manual camera settings adjustments to the QTKit Capture APIs, I could remove almost all of this.
If all you're trying to do is see/grab the output of a connected camera, the answer is probably neither.
Use QTKit's QTCaptureView. Problem solved. Want to grab a frame? Also no problem. Don't try to roll your own - QTKit's stuff is optimized and part of the OS. I'm pretty sure you can affect camera properties as you wanted but if not, plan B should work.
Plan b: Use a scheduled, recurring NSTimer to ask QTKit to grab a frame every so often ("how" linked above) and apply your image manipulations to the frame (maybe with Core Image) before displaying in your NSImageView.

SVG Background animation and frame rate

I've made an animation with SVG. It's like a slowly changing wallpaper. The idea is that you should barely notice it is changing.
It's purely decorative, and I don't want it to drain any resources. Is there a way to set the frame rate in SVG? I thought setting it to a very low number might do the trick? I'm using Raphael, by the way.
Deep in Raphael's guts, you will find the logic that controls the frame rate for non-keyframe animations:
animationElements[length] && setTimeout(animation);
By omitting an actual timeout value, Raphael is basically telling the browser to run the method as fast as it can (within scheduling constraints provided by the DOM specification and the browser implementation). You could either tweak that function to use a user-supplied parameter (or put a number there, though that will affect all animations), or use Peter's suggestion.

Resources