I have a CVPixelBuffer that I'm trying to efficiently draw on screen.
The not-efficient way of turning into an NSImage works but is very slow, dropping about 40% of my frames.
Therefore, I've tried rendering it on-screen using CIContext's drawImage:inRect:fromRect. The CIContext was initialized with a NSOpenGLContext who's view was set to my VC's view. When I have a new image, I call the drawImage method which doesn't spit out any errors... but doesn't display anything on screen either (it did log errors when my contexts were not correctly setup).
I've tried to find an example of how this is done on MacOS, but everything seems to be for iOS nowadays.
EDIT:
Here's some of the code I am using. I've left out irrelevant sections
On viewDidLoad I init the GL and CI contexts
NSOpenGLPixelFormatAttribute pixelFormatAttr[] = {
kCGLPFAAllRenderers, 0
};
NSOpenGLPixelFormat *glPixelFormat = [[NSOpenGLPixelFormat alloc] initWithAttributes: pixelFormatAttr];
NSOpenGLContext *glContext = [[NSOpenGLContext alloc] initWithFormat:glPixelFormat shareContext:nil];
glContext.view = self.view;
self.ciContext = [CIContext contextWithCGLContext:glContext.CGLContextObj pixelFormat:glPixelFormat.CGLPixelFormatObj colorSpace:nil options:nil];
Then, when a new frame is ready, I do:
dispatch_async(dispatch_get_main_queue(), ^{
[vc.ciContext drawImage:ciImage inRect:vc.view.bounds fromRect:ciImage.extent];
vc.isRendering = NO;
});
I am not sure I'm calling draw in the right place, but I can't seem to find out where is this supposed to go.
If the CVPixelBuffer has the kCVPixelBufferIOSurfaceCoreAnimationCompatibilityKey attribute, the backing IOSurface (retrieved via CVPixelBufferGetIOSurface) can be passed directly to the contents property of a CALayer.
This is probably the most efficient way to display a CVPixelBuffer.
Related
I'm using a MTKView to draw Metal content. It's configured as follows:
mtkView = MTKView(frame: self.view.frame, device: device)
mtkView.colorPixelFormat = .bgra8Unorm
mtkView.delegate=self
mtkView.sampleCount=4
mtkView.isPaused=true
mtkView.enableSetNeedsDisplay=true
setFrameSize is overriden to trigger a redisplay.
Whenever the view resizes it scales its old content before it redraws everything. This gives a jittering feeling.
I tried setting the contentGravity property of the MTKView's layer to a non-resizing value, but that totally messes up the scale and position of the content. It seems MTKView doesn't want me to fiddle with that parameter.
How can I make sure that during a resize the content is always properly redrawn?
In my usage of Metal and MTKView, I tried various combinations of presentsWithTransaction and waitUntilScheduled without success. I still experienced occasional frames of stretched content in between frames of properly rendered content during live resize.
Finally, I dropped MTKView altogether and made my own NSView subclass that uses CAMetalLayer and resize looks good now (without any use of presentsWithTransaction or waitUntilScheduled). One key bit is that I needed to set the layer's autoresizingMask to get the displayLayer method to be called every frame during window resize.
Here's the header file:
#import <Cocoa/Cocoa.h>
#interface MyMTLView : NSView<CALayerDelegate>
#end
Here's the implementation:
#import <QuartzCore/CAMetalLayer.h>
#import <Metal/Metal.h>
#implementation MyMTLView
- (id)initWithFrame:(NSRect)frame
{
if (!(self = [super initWithFrame:frame])) {
return self;
}
// We want to be backed by a CAMetalLayer.
self.wantsLayer = YES;
// We want to redraw the layer during live window resize.
self.layerContentsRedrawPolicy = NSViewLayerContentsRedrawDuringViewResize;
// Not strictly necessary, but in case something goes wrong with live window
// resize, this layer placement makes it more obvious what's going wrong.
self.layerContentsPlacement = NSViewLayerContentsPlacementTopLeft;
return self;
}
- (CALayer*)makeBackingLayer
{
CAMetalLayer* metalLayer = [CAMetalLayer layer];
metalLayer.device = MTLCreateSystemDefaultDevice();
metalLayer.delegate = self;
// *Both* of these properties are crucial to getting displayLayer to be
// called during live window resize.
metalLayer.autoresizingMask = kCALayerHeightSizable | kCALayerWidthSizable;
metalLayer.needsDisplayOnBoundsChange = YES;
return metalLayer;
}
- (CAMetalLayer*)metalLayer
{
return (CAMetalLayer*)self.layer;
}
- (void)setFrameSize:(NSSize)newSize
{
[super setFrameSize:newSize];
self.metalLayer.drawableSize = newSize;
}
- (void)displayLayer:(CALayer*)layer
{
// Do drawing with Metal.
}
#end
For reference, I do all my Metal drawing in MTKView's drawRect method.
I have the same problem with glitches on view resizing. You can even reproduce it in the HelloTriangle example from the Apple's developer site. However the effect is minimized because the triangle is drawn near the middle of the screen, and it's the content closest to the edge of the window, opposite the corner that drags, that is effected worst. The developer notes regarding use of presentsWithTransaction and waitUntilScheduled do not work for me either.
My solution was to add a Metal layer beneath the window.contentView.layer, and to make that layer large enough that it rarely needs to be resized. The reason this works is that, unlike the window.contentView.layer, which sizes itself automatically to the view (in turn maintaining the window size), you have explicit control of the sublayer size. This eliminates the flickering.
This helped me - https://github.com/trishume/MetalTest
He uses MetalLayer and careful setting of various properties. Everything is pretty smooth even with two side by side in synchronised scroll views with 45megapixel images.
A link to my original problem How do I position an image correctly in MTKView?
i have some problem. So i have code which update song name and picture from php. Song name work and also updated but picture not work, in php file all work but in my project - no. How make update picture from url after 10 sec for example. Thanks.
-(void)viewWillDraw {
NSURL *artistImageURL = [NSURL URLWithString:#"http://site.ru/ParseDataField/kiss.php?image"];
NSImage *artistImage = [[NSImage alloc] initWithContentsOfURL:artistImageURL];
[dj setImage:artistImage];
dispatch_queue_t queue = dispatch_get_global_queue(0,0);
dispatch_async(queue, ^{
NSError* error = nil;
NSString* text = [NSString stringWithContentsOfURL:[NSURL URLWithString:#"http://site.ru/ParseDataField/kiss.php?artist"]
encoding:NSASCIIStringEncoding
error:&error];
dispatch_async(dispatch_get_main_queue(), ^{
[labelName setStringValue:text];
});
});
}
You should really consider placing this code someplace other than -viewWillDraw. This routine can be called multiple times for the same NSView under some circumstances and, more importantly, you need to call [super viewWillDraw] to make sure that things will actually draw correctly (if anything is drawn in the view itself).
For periodic updates (such as every 10 seconds), you should consider using NSTimer to trigger the retrieval of the next object.
As for the general question of why your image isn't being drawn correctly, you should probably consider putting the image retrieval and drawing code into the same structure as your label retrieval and drawing code. This will get the [dj setImage: artistImage] method call outside of the viewWillDraw chain which is likely causing some difficulty here.
Struggling with this one. I have a custom NSStatusItemView that I'm trying to animate. I've added the following code to my status item view to kick off the animation:
- (void)setAnimated
{
CABasicAnimation *anim = [CABasicAnimation animationWithKeyPath:#"opacity"];
anim.duration = 1.0;
anim.repeatCount = HUGE_VALF;
anim.autoreverses = YES;
anim.fromValue=[NSNumber numberWithFloat:1.0];
anim.toValue=[NSNumber numberWithFloat:0.0];
[self.layer addAnimation: anim forKey: #"animateOpacity"];
[self setWantsLayer:YES];
[self setNeedsDisplay:YES];
}
When I call this method, nothing happens. Yet if I move this code to my drawRect method, then the view properly animates at launch. Not entirely sure what I need to do to be able to tell it to start animated after the fact but the above method is not doing it and I have no idea why! Any ideas?
Ok answer myself so the googles have record of the answer!
The problem was, kinda of, a lack of understanding of drawRect. When my setAnimated method calls setNeedsDisplay, it calls drawRect again, effectively undoing what is done in the setAnimated method.
There were two things I did to properly fix this. First, I modified the setAnimated method to accept a BOOL argument and set a isAnimated property on the view to that value. I then, in drawRect, check this BOOL value and do the animation if it is set to YES.
Secondly, it seems, you need to call [self setWantsLayer: YES] the first time the view is drawn. So I call this in drawRect the very first time it is run, so that later animation will work.
Re-asking the question:
When you add an animation for the contents key, a CATransitionAnimation is apparently being triggered that fades the original contents property to the first value in the animation's values array, resulting in a .25 second fade. And it looks bad! I have suppressed every animatable property using all the methods discussed here (returning null animations through a delegate, into the actions dictionary, using CATransaction), but none of these seem to be targeting this particular transition animation.
I have been looking into what property could possibly be responsible for this, but cannot figure it out.
I need to suppress the transition animation that is occurring when you add an animation to the contents key.
As I'm at such a loss, I will put the keyframe animation that is being added for you to see. I figure maybe I am doing something wrong here? Just a note, that array is just an array of 6 CGImageRefs (the frames of the animation).
+ (CAKeyframeAnimation *)moveLeftAnimation {
CAKeyframeAnimation *animation = [CAKeyframeAnimation animationWithKeyPath:#"contents"];
animation.values = [NSArray arrayWithArray:[Setzer walkingLeftSprite]];
animation.duration = 0.5f;
animation.keyTimes = [NSArray arrayWithObjects:
[NSNumber numberWithFloat:0.0],
[NSNumber numberWithFloat:0.2],
[NSNumber numberWithFloat:0.4],
[NSNumber numberWithFloat:0.6],
[NSNumber numberWithFloat:0.8],
[NSNumber numberWithFloat:1.0],
nil];
animation.timingFunction = [CAMediaTimingFunction functionWithName:kCAMediaTimingFunctionLinear];
return animation;
}
Also, I have an animation that handles the position key, in the sprite's action dictionary:
+ (CABasicAnimation *)moveAnimation {
CABasicAnimation *moveAnimation = [CABasicAnimation animation];
moveAnimation.timingFunction = [CAMediaTimingFunction functionWithName:kCAMediaTimingFunctionLinear];
moveAnimation.duration = 0.5f;
return moveAnimation;
}
I am thinking maybe this transition is occurring when you change the layer's position? I don't know...
Please help! This is driving me NUTS!
You can do something along the lines of what I describe in this answer, where I disable the implicit animations for various layer properties by setting the appropriate values in the actions dictionary on that layer.
In your case, I believe something like
NSMutableDictionary *newActions = [[NSMutableDictionary alloc] initWithObjectsAndKeys:
[NSNull null], #"contents",
nil];
layer.actions = newActions;
[newActions release];
should prevent the implicit animation of a layer's contents property until you explicitly animate it.
To prevent any animation, you could set an object as the delegate of your CALayer and then implement the ‑actionForLayer:forKey: delegate method and return a null object:
- (id<CAAction>)actionForLayer:(CALayer*)layer forKey:(NSString*)key
{
if(layer == yourLayer)
{
if([key isEqualToString:#"contents"])
{
return (id<CAAction>)[NSNull null];
}
}
return nil;
}
Here are a few notes on how this puzzle was solved, and how everyone's answers provided a piece of the puzzle.
To restate the problem: when I added a keyframe animation to the #contents key of a CALayer, there appeared to be a .25 second fade transition between the original contents property and the first frame of the keyframe animation. This looked bad, and I wanted to get rid of it.
At first, I thought surely that by using a CATransaction I could suppress this implicit transition animation, as that is what Apple's docs lead you to believe. Using a transaction, I suppressed in every possible way you could imagine, and yet it was still happening. Then I tried returning NULL animations for every animatable property via a dictionary. No luck. Then I did the same thing, but with a delegate. Still no luck.
What I didn't mention is that at the same time the animation was being added and the layer was being moved, two sublayers beneath it were being removed from the their superlayers. I tried adding custom animations for the onOrderOut key, but to no avail. Then I stumbled upon another question here on StackOverflow, about adding a custom animation for the onOrderOut key. It turns out, quite simply, that you can't, and that if you wan to implement some other animation for when a sublayer is removed, you have to use the delagate method animationDidStop. How can I use custom animations for onOrderOut in Core Animation?
So at this point I was convinced that this ghost image had nothing to do with the actual layer in question, the sprite itself. To test this, I just didn't add the sublayers that went beneath it. Sure enough, there was no lingering ghost when I moved the sprite around. It looked perfect. Then I added the layers beneath there was the ghost. It was almost like the sprite's contents were drawn into the layers beneath it, so that when they were removed, there was a sort of imprint.
Instead of removing the sublayers, I just tried hiding them. Bingo. It was perfect. The same fade transition occurred, but there was no imprint of the sprite left. I still don't understand why this is so.
Then, because I still needed to remove those layers, I implemented the animationDidStop delegate method for the sprite's various movement animations to remove them.
This is the original:
This is the new version:
So while I don't understand, technically, why there appears to be an imprint, I am all but certain that it concerns what goes on behind the scenes when you remove a sublayer. Also, for interest sake, I still wanted that sublayer to be hidden on animation start, so I just set it to hidden and provided my own transition animation.
So thanks to everyone for their help. This is a strange use case, I know, but if you are ever thinking of making a 2d sprites-based Final Fantasy Tactics ripoff, then hopefully my pain will be to your benefit!
class AnimationLayer: CALayer {
override class func defaultAction(forKey event: String) -> CAAction? {
let transition = CATransition()
transition.duration = 0
return transition
}
}
I am creating an NSView, which in its drawRect method creates and adds an NSImageView as a subview.
I would like to rotate this NSImageView (circleView), or [self]. So in another method, I am trying to do that:
-(void)startAnimation {
CABasicAnimation* spinAnimation = [CABasicAnimation animationWithKeyPath:#"transform.rotation.z"];
spinAnimation.toValue = [NSNumber numberWithFloat:5*2*M_PI];
spinAnimation.duration = 2;
[circleView.layer addAnimation:spinAnimation forKey:#"spinAnimation"];
}
However, this code doesn't do anything when the method is called. Am I doing something wrong? I tried self.layer and circleView.layer but neither seem to work...
The answer to this issue is that the NSView/NSImageView didn't have a backing layer to do the animation with.
You set this with:
[NSView setWantsLayer:YES];
Your're just defining the animation, you're not actually calling the animation block that performs the animation.
You need to call a block defined between:
+ beginAnimations:context:
+ commitAnimations
To actually run the animation.
Edit01: My bad. I didn't pay attention to the tags and added the answer for the iPhone API.