When I load an NSImage with imageNamed, the OS owns the memory, but when I call CGImageForProposedRect, Apple says "This may be an existing CGImage if one is available. If not, a new CGImage is created."
If a new image is created, then I would think I have to release it.
I am not getting a crash either way so it is hard to tell what is correct.
CGImageRef imageRef = [[NSImage imageNamed:resource] CGImageForProposedRect:&imageRect context:NULL hints:nil];
if (imageRef)
{
// do stuff
// is this needed?
CGImageRelease(imageRef);
}
Related
I am trying to capture video frames from my macBook camera and process them on the fly (for later face detection). To reduce memory usage, I want to reduce the capture resolution from the preset value 1200x720 to 640x480.
Here is my code to setup the Capture Session:
_session = [[AVCaptureSession alloc] init];
if ([_session canSetSessionPreset:AVCaptureSessionPreset640x480]){
[_session setSessionPreset:AVCaptureSessionPreset640x480];
NSLog(#"resolution preset changed");
}
// configure input
_camera = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
_deviceInput = [AVCaptureDeviceInput deviceInputWithDevice:_camera error:nil];
// configure output
_videoOutput = [[AVCaptureVideoDataOutput alloc] init];
NSDictionary* newSettings = #{ (NSString*) kCVPixelBufferPixelFormatTypeKey: #(kCVPixelFormatType_32BGRA)};
_videoOutput.videoSettings = newSettings;
//discard if the data output queue is blocked
[_videoOutput setAlwaysDiscardsLateVideoFrames:YES];
// process frames on another queue
dispatch_queue_t videoDataOutputQueue;
videoDataOutputQueue = dispatch_queue_create("VideoDataOutputQueue", DISPATCH_QUEUE_SERIAL);
[_videoOutput setSampleBufferDelegate:videoBufferDelegate queue:videoDataOutputQueue];
[_session addInput:_deviceInput];
[_session addOutput:_videoOutput];
[_session startRunning];
After this, the session sets up appropriately, it logs "resolution preset changed" correctly and forwards the video data to the delegate on another queue to process it. When I inspect session.sessionPreset, it says that the preset is AVCaptureSessionPreset640x480.
Now in the delegate:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
//get the image from buffer, transform it to CIImage
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
self.image = [[CIImage alloc] initWithCVPixelBuffer:pixelBuffer];
When inspecting self.image.extent.size, it shows and incorrect size of 1200x720, as if I had not changed the preset before. Even when inspecting the method's arguement sampleBuffer, it shows the dimension as 1200x720.
I browsed the internet and the apple reference for a couple of hours now, but could not manage to find a solution. I hope you can save me!
I seem to have found a solution by myself (or at least a workaround). Commenting out the following lines led to the intended changes of the buffer's resolution:
NSDictionary* settings = #{ (NSString*) kCVPixelBufferPixelFormatTypeKey: #(kCVPixelFormatType_32BGRA)};
_videoOutput.videoSettings = settings;
My assumption is, that setting these compression settings results in an overwriting of the AVCaptureSessionPreset. However, I find it not completely clear why this should be the case (compression settings shouldn't have an influence on resolution settings, right?).
Currently I'm making an isometric map editor (in Swift) where each map tile is added to an SKNode called mapLayer. I want to know whether it is possible to produce a png image from this mapLayer, once I have finished designing a map?
Not in Swift but it should give you a good idea on how to do it.
You can capture your screen to a UIImage by doing this:
CGRect bounds = self.scene.view.bounds;
UIGraphicsBeginImageContextWithOptions(bounds.size, NO, [UIScreen mainScreen].scale);
[self drawViewHierarchyInRect:bounds afterScreenUpdates:YES];
UIImage* screenshotImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
Then create a PNG image from the UIImage (and write it to disk) like this:
// Create paths to output images
NSString *pngPath = [NSHomeDirectory() stringByAppendingPathComponent:#"Documents/Test.png"];
NSString *jpgPath = [NSHomeDirectory() stringByAppendingPathComponent:#"Documents/Test.jpg"];
Source here.
// Write a UIImage to JPEG with minimum compression (best quality)
// The value 'image' must be a UIImage object
// The value '1.0' represents image compression quality as value from 0.0 to 1.0
[UIImageJPEGRepresentation(image, 1.0) writeToFile:jpgPath atomically:YES];
// Write image to PNG
[UIImagePNGRepresentation(image) writeToFile:pngPath atomically:YES];
// Let's check to see if files were successfully written...
// Create file manager
NSError *error;
NSFileManager *fileMgr = [NSFileManager defaultManager];
// Point to Document directory
NSString *documentsDirectory = [NSHomeDirectory() stringByAppendingPathComponent:#"Documents"];
// Write out the contents of home directory to console
NSLog(#"Documents directory: %#", [fileMgr contentsOfDirectoryAtPath:documentsDirectory error:&error]);
Source here.
Here's an example of how to write the contents of the view to a PNG file. First, define a UIView extension that captures the contents of the view and converts it to a UIImage.
extension UIView {
func screenGrab() -> UIImage {
// Uncomment this (and comment out the next statement) for retina screen capture
// UIGraphicsBeginImageContextWithOptions(bounds.size, false, UIScreen.mainScreen().scale)
UIGraphicsBeginImageContextWithOptions(bounds.size, false, 1.0)
drawViewHierarchyInRect(self.bounds, afterScreenUpdates: true)
let image = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
return image
}
}
Then, define a UIImage extension that writes a UIImage to a file.
extension UIImage {
func writeToFile(fileName:String) {
let path = NSHomeDirectory().stringByAppendingPathComponent("Documents/"+fileName)
UIImagePNGRepresentation(self).writeToFile(path, atomically: true)
}
}
Lastly, capture and write the contents of the view to a PNG file with
self.view?.screenGrab().writeToFile("capture.png")
and here's how to determine where the PNGs will be stored:
println(NSHomeDirectory().stringByAppendingPathComponent("Documents"))
This is one of the instances where macOS and iOS behave very differently. Spoiler: the usual methods of capturing the contents of an NSView to NSImage don't work.
NSImage(data:inside:) and NSView.bitmapImageRepForCachingDisplay(in:to) work fine for NSView, but if you embed an SKView or apply them to the SKView directly, you'll only get a blank canvas.
The solution lies in thinking SpriteKit:
An SKScene is an SKNode at heart (actually, an SKEmitterNode), and SKView has a method called texture(from: SKNode) which gives you an SKTexture. SKTexture has a cgImage() property, and there you are.
This assumes three properties in the NSViewController that presents the scene: skView: SKView, scene: SKScene?, and an NSImageView for display named snapView.
if let rendered = skView.texture(from: scene!){
snapView.image = NSImage(cgImage: rendered.cgImage(), size: self.skView.bounds.size)
(Don't force unwrap, it's only for demo purposes)
Hi all you awesome coders! I've put together this thing from various helpful sources over the last couple of weeks (including a lot of posts from stackoverflow), trying to create something that will take a webcam feed and detect smiles when they occur (might as well draw boxes around the faces and the smiles as well, that doesn't seem like it would be hard once they are detected). Please give me some lee-way if it's messy code because I'm still very much learning.
Currently I'm stuck at trying to pass the image to a CIImage so it can be analysed for faces (I plan to deal with smiles after the face hurdle is overcome). As it is the compiler succeeds if I comment out the block after (5) - it brings up a simple AVCaptureVideoPreviewLayer in a window. I think this is what I've called "rootLayer", so it's like the first layer of the displayed output, and after I detect faces in the video frames I'll show a rectangle following the "bounds" of any detected face in a new layer overlaid on top of this one, and I've called that layer "previewLayer"... correct?
But with the block after (5) there, the compiler throws out three errors -
Undefined symbols for architecture x86_64:
"_CMCopyDictionaryOfAttachments", referenced from:
-[AVRecorderDocument captureOutput:didOutputSampleBuffer:fromConnection:] in AVRecorderDocument.o
"_CMSampleBufferGetImageBuffer", referenced from:
-[AVRecorderDocument captureOutput:didOutputSampleBuffer:fromConnection:] in AVRecorderDocument.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Can anyone tell me where I'm going wrong and what my next steps are?
Thanks for any help, I've been stuck at this point for a couple of days and I can't figure it out, all the examples I can find are for IOS and don't work in OSX.
- (id)init
{
self = [super init];
if (self) {
// Move the output part to another function
[self addVideoDataOutput];
// Create a capture session
session = [[AVCaptureSession alloc] init];
// Set a session preset (resolution)
self.session.sessionPreset = AVCaptureSessionPreset640x480;
// Select devices if any exist
AVCaptureDevice *videoDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
if (videoDevice) {
[self setSelectedVideoDevice:videoDevice];
} else {
[self setSelectedVideoDevice:[AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeMuxed]];
}
NSError *error = nil;
// Add an input
videoDeviceInput = [AVCaptureDeviceInput deviceInputWithDevice:videoDevice error:&error];
[self.session addInput:self.videoDeviceInput];
// Start the session (app opens slower if it is here but I think it is needed in order to send the frames for processing)
[[self session] startRunning];
// Initial refresh of device list
[self refreshDevices];
}
return self;
}
-(void) addVideoDataOutput {
// (1) Instantiate a new video data output object
AVCaptureVideoDataOutput * captureOutput = [[AVCaptureVideoDataOutput alloc] init];
captureOutput.videoSettings = #{ (NSString *) kCVPixelBufferPixelFormatTypeKey : #(kCVPixelFormatType_32BGRA) };
// discard if the data output queue is blocked (while CI processes the still image)
captureOutput.alwaysDiscardsLateVideoFrames = YES;
// (2) The sample buffer delegate requires a serial dispatch queue
dispatch_queue_t captureOutputQueue;
captureOutputQueue = dispatch_queue_create("CaptureOutputQueue", DISPATCH_QUEUE_SERIAL);
[captureOutput setSampleBufferDelegate:self queue:captureOutputQueue];
dispatch_release(captureOutputQueue); //what does this do and should it be here or after we receive the processed image back?
// (3) Define the pixel format for the video data output
NSString * key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
NSNumber * value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];
NSDictionary * settings = #{key:value};
[captureOutput setVideoSettings:settings];
// (4) Configure the output port on the captureSession property
if ( [self.session canAddOutput:captureOutput] )
[session addOutput:captureOutput];
}
// Implement the Sample Buffer Delegate Method
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
// I *think* I have a video frame now in some sort of image format... so have to convert it into a CIImage before I can process it:
// (5) Convert CMSampleBufferRef to CVImageBufferRef, then to a CI Image (per weichsel's answer in July '13)
CVImageBufferRef cvFrameImage = CMSampleBufferGetImageBuffer(sampleBuffer); // Having trouble here, prog. stops and won't recognise CMSampleBufferGetImageBuffer.
CFDictionaryRef attachments = CMCopyDictionaryOfAttachments(kCFAllocatorDefault, sampleBuffer, kCMAttachmentMode_ShouldPropagate);
self.ciFrameImage = [[CIImage alloc] initWithCVImageBuffer:cvFrameImage options:(__bridge NSDictionary *)attachments];
//self.ciFrameImage = [[CIImage alloc] initWithCVImageBuffer:cvFrameImage];
//OK so it is a CIImage. Find some way to send it to a separate CIImage function to find the faces, then smiles. Then send it somewhere else to be displayed on top of AVCaptureVideoPreviewLayer
//TBW
}
- (NSString *)windowNibName
{
return #"AVRecorderDocument";
}
- (void)windowControllerDidLoadNib:(NSWindowController *) aController
{
[super windowControllerDidLoadNib:aController];
// Attach preview to session
CALayer *rootLayer = self.previewView.layer;
[rootLayer setMasksToBounds:YES]; //aaron added
self.previewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:self.session];
[self.previewLayer setBackgroundColor:CGColorGetConstantColor(kCGColorBlack)];
[self.previewLayer setVideoGravity:AVLayerVideoGravityResizeAspect];
[self.previewLayer setFrame:[rootLayer bounds]];
//[newPreviewLayer setAutoresizingMask:kCALayerWidthSizable | kCALayerHeightSizable]; //don't think I need this for OSX?
[self.previewLayer setVideoGravity:AVLayerVideoGravityResizeAspect];
[rootLayer addSublayer:previewLayer];
// [newPreviewLayer release]; //what's this for?
}
(moved from the comments section)
Wow. I guess two days and one StackOverflow post is what it takes to figure out that I haven't added CoreMedia.framework to my project.
Setting the scene
I am working on a video processing app that runs from the command line to read in, process and then export video. I'm working with 4 tracks.
Lots of clips that I append into a single track to make one video. Let's call this the ugcVideoComposition.
Clips with Alpha which get positioned on a second track and using layer instructions, is set composited on export to play back over the top of the ugcVideoComposition.
A music audio track.
An audio track for the ugcVideoComposition containing the audio from the clips appended into the single track.
I have this all working, can composite it and export it correctly using AVExportSession.
The problem
What I now want to do is apply filters and gradients to the ugcVideoComposition.
My research so far suggests that this is done by using AVReader and AVWriter, extracting a CIImage, manipulating it with filters and then writing that out.
I haven't yet got all the functionality I had above working, but I have managed to get the ugcVideoComposition read in and written back out to disk using the AssetReader and AssetWriter.
BOOL done = NO;
while (!done)
{
while ([assetWriterVideoInput isReadyForMoreMediaData] && !done)
{
CMSampleBufferRef sampleBuffer = [videoCompositionOutput copyNextSampleBuffer];
if (sampleBuffer)
{
// Let's try create an image....
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CIImage *inputImage = [CIImage imageWithCVImageBuffer:imageBuffer];
// < Apply filters and transformations to the CIImage here
// < HOW TO GET THE TRANSFORMED IMAGE BACK INTO SAMPLE BUFFER??? >
// Write things back out.
[assetWriterVideoInput appendSampleBuffer:sampleBuffer];
CFRelease(sampleBuffer);
sampleBuffer = NULL;
}
else
{
// Find out why we couldn't get another sample buffer....
if (assetReader.status == AVAssetReaderStatusFailed)
{
NSError *failureError = assetReader.error;
// Do something with this error.
}
else
{
// Some kind of success....
done = YES;
[assetWriter finishWriting];
}
}
}
}
As you can see, I can even get the CIImage from the CMSampleBuffer, and I'm confident I can work out how to manipulate the image and apply any effects etc. I need. What I don't know how to do is put the resulting manipulated image BACK into the SampleBuffer so I can write it out again.
The question
Given a CIImage, how can I put that into a sampleBuffer to append it with the assetWriter?
Any help appreciated - the AVFoundation documentation is terrible and either misses crucial points (like how to put an image back after you've extracted it, or is focussed on rendering images to the iPhone screen which is not what I want to do.
Much appreciated and thanks!
I eventually found a solution by digging through a lot of half complete samples and poor AVFoundation documentation from Apple.
The biggest confusion is that while at a high level, AVFoundation is "reasonably" consistent between iOS and OSX, the lower level items behave differently, have different methods and different techniques. This solution is for OSX.
Setting up your AssetWriter
The first thing is to make sure that when you set up the asset writer, you add an adaptor to read in from a CVPixelBuffer. This buffer will contain the modified frames.
// Create the asset writer input and add it to the asset writer.
AVAssetWriterInput *assetWriterVideoInput = [AVAssetWriterInput assetWriterInputWithMediaType:[[videoTracks objectAtIndex:0] mediaType] outputSettings:videoSettings];
// Now create an adaptor that writes pixels too!
AVAssetWriterInputPixelBufferAdaptor *adaptor = [AVAssetWriterInputPixelBufferAdaptor
assetWriterInputPixelBufferAdaptorWithAssetWriterInput:assetWriterVideoInput
sourcePixelBufferAttributes:nil];
assetWriterVideoInput.expectsMediaDataInRealTime = NO;
[assetWriter addInput:assetWriterVideoInput];
Reading and Writing
The challenge here is that I couldn't find directly comparable methods between iOS and OSX - iOS has the ability to render a context directly to a PixelBuffer, where OSX does NOT support that option. The context is also configured differently between iOS and OSX.
Note that you should include the QuartzCore.Framework into your XCode Project as well.
Creating the context on OSX.
CIContext *context = [CIContext contextWithCGContext:
[[NSGraphicsContext currentContext] graphicsPort]
options: nil]; // We don't want to always create a context so we put it outside the loop
Now you want want to loop through, reading off the AssetReader and writing to the AssetWriter... but note that you are writing via the adaptor created previously, not with the SampleBuffer.
while ([adaptor.assetWriterInput isReadyForMoreMediaData] && !done)
{
CMSampleBufferRef sampleBuffer = [videoCompositionOutput copyNextSampleBuffer];
if (sampleBuffer)
{
CMTime currentTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
// GRAB AN IMAGE FROM THE SAMPLE BUFFER
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithInt:kCVPixelFormatType_32BGRA], kCVPixelBufferPixelFormatTypeKey,
[NSNumber numberWithInt:640.0], kCVPixelBufferWidthKey,
[NSNumber numberWithInt:360.0], kCVPixelBufferHeightKey,
nil];
CIImage *inputImage = [CIImage imageWithCVImageBuffer:imageBuffer options:options];
//-----------------
// FILTER IMAGE - APPLY ANY FILTERS IN HERE
CIFilter *filter = [CIFilter filterWithName:#"CISepiaTone"];
[filter setDefaults];
[filter setValue: inputImage forKey: kCIInputImageKey];
[filter setValue: #1.0f forKey: kCIInputIntensityKey];
CIImage *outputImage = [filter valueForKey: kCIOutputImageKey];
//-----------------
// RENDER OUTPUT IMAGE BACK TO PIXEL BUFFER
// 1. Firstly render the image
CGImageRef finalImage = [context createCGImage:outputImage fromRect:[outputImage extent]];
// 2. Grab the size
CGSize size = CGSizeMake(CGImageGetWidth(finalImage), CGImageGetHeight(finalImage));
// 3. Convert the CGImage to a PixelBuffer
CVPixelBufferRef pxBuffer = NULL;
// pixelBufferFromCGImage is documented below.
pxBuffer = [self pixelBufferFromCGImage: finalImage andSize: size];
// 4. Write things back out.
// Calculate the frame time
CMTime frameTime = CMTimeMake(1, 30); // Represents 1 frame at 30 FPS
CMTime presentTime=CMTimeAdd(currentTime, frameTime); // Note that if you actually had a sequence of images (an animation or transition perhaps), your frameTime would represent the number of images / frames, not just 1 as I've done here.
// Finally write out using the adaptor.
[adaptor appendPixelBuffer:pxBuffer withPresentationTime:presentTime];
CFRelease(sampleBuffer);
sampleBuffer = NULL;
}
else
{
// Find out why we couldn't get another sample buffer....
if (assetReader.status == AVAssetReaderStatusFailed)
{
NSError *failureError = assetReader.error;
// Do something with this error.
}
else
{
// Some kind of success....
done = YES;
[assetWriter finishWriting];
}
}
}
}
Creating the PixelBuffer
There MUST be an easier way, however for now, this works and is the only way I found to get directly from a CIImage to a PixelBuffer (via a CGImage) on OSX. The following code is cut and paste from AVFoundation + AssetWriter: Generate Movie With Images and Audio
- (CVPixelBufferRef) pixelBufferFromCGImage: (CGImageRef) image andSize:(CGSize) size
{
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
[NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
nil];
CVPixelBufferRef pxbuffer = NULL;
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, size.width,
size.height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options,
&pxbuffer);
NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
CVPixelBufferLockBaseAddress(pxbuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
NSParameterAssert(pxdata != NULL);
CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pxdata, size.width,
size.height, 8, 4*size.width, rgbColorSpace,
kCGImageAlphaNoneSkipFirst);
NSParameterAssert(context);
CGContextConcatCTM(context, CGAffineTransformMakeRotation(0));
CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image),
CGImageGetHeight(image)), image);
CGColorSpaceRelease(rgbColorSpace);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
return pxbuffer;
}
Try using: SDAVAssetExportSession
SDAVAssetExportSession on GITHub
and then implementing a delegate to process the pixels
- (void)exportSession:(SDAVAssetExportSession *)exportSession renderFrame:(CVPixelBufferRef)pixelBuffer withPresentationTime:(CMTime)presentationTime toBuffer:(CVPixelBufferRef)renderBuffer
{ Do CIImage and CIFilter inside here }
Following is the code snippet that I have:
// Make Auto release pool
NSAutoreleasePool * autoReleasePool = [[NSAutoreleasePool alloc] init];
try
{
if (mCapture)
{
// Get the image reference
NSImage* image = NULL;
image = [mCapture getCurrentFrameImage];
// Get the TIFF data
NSData *pDataTifData = [[NSData alloc] initWithData:[image TIFFRepresentation]];
NSBitmapImageRep *pBitmapImageRep = [[NSBitmapImageRep alloc] initWithData:pDataTifData];
// Convert to BMP data
NSData *pDataBMPData;
pDataBMPData = [pBitmapImageRep representationUsingType: NSPNGFileType
properties: nil];
// Save to specified path
ASL::String strPath = ASL::MakeString(capInfo->thefile.name);
NSString* pPath = (NSString*)ASL::MakeCFString(strPath);
[pDataBMPData writeToFile:pPath
atomically: YES];
::CFRelease(pPath);
pDataBMPData = nil;
[pBitmapImageRep release];
pBitmapImageRep = nil;
[pDataTifData release];
pDataTifData = nil;
image = nil;
}
}
catch(...)
{
}
[autoReleasePool drain];
Note that image = [mCapture getCurrentFrameImage]; is returning an autoreleased NSImage. I am releasing objects and also have NSAutoreleasePool in place. But still it is leaking about 3-4 MB of memory everytime this code snippet is executed. I am not sure where the mistake is.
You could simplify this code a lot by making captureCurrentFrameImage return an NSBitmapImageRep instead of an NSImage, since you never actually use an NSImage for anything here. You can wrap the image rep in an image when necessary, and for this code, simply use the image rep by itself to produce the PNG data. Among other things, this saves you a trip through the TIFF representation.
If it still leaks after you make those changes, run your app under Instruments's Leaks template; the two instruments in that template, Leaks and ObjectAlloc, will help you hunt down whatever leaks you have.