TL;DR: In macOS 10.13, an MTLTexture has a maximum width and height of 16,384. What strategies can you use to be able to process and display images larger than 16,384 pixels using Metal?
In a photo viewer that I'm currently working on, I've moved most of the viewing of images into a Metal backed view that uses Core Image for doing any image adjustments. This is working really well but I've recently started testing against some really large images (panoramas) and I'm now hitting some limits that I'm not entirely sure how to workaround while remaining relatively performant.
My current environment looks like this:
Load and decode an image from from an NSURL into an IOSurface. This is done using either Image IO directly or a Core Image pipeline that renders into the IOSurface. The IOSurface is then passed from an XPC service back into the main app.
In the main app, a new MTLTexture is created that is backed by the IOSurface. Then, a CIImage is created from the MTLTexture and that CIImage is used throughout an image pipeline as the root "source image".
However, if I attempt to open an image larger that 16,384 pixels in one dimension, then I'm unable to create the original IOSurface 16,384 on my laptop. (13" MBP-TB 2016)
But even if I could create a larger IOSurface, then I'm still stuck with the same limit on the MTLTexture.
See: Apple's Metal Feature Table Set
I'm curious what strategies others would recommend to allow one to open large image files while still taking advantage of Core Image and Metal.
One attempt I've made is to just have the root source image be a CIImage that was created with a CGImageRef. However, there's a significant drop in performance between that arrangement and a CIImage backed by a texture for even smaller sized images.
Another idea I've had, but haven't yet explored, was to use CIImageProvider in some capacity but I'm not entirely sure how I'd go about "tiling" potentially several IOSurfaces or MTLTextures, and if that even makes sense or if it would be better to just allocate a single large buffer to read from. (Or perhaps use dispatch_data in some capacity?)
(macOS 10.13 or even 10.14 would be fine.)
Related
I am creating a simple photo catalogue application for macOS to see whether the latest APIs can significantly improve performance of loading directories with large numbers of images.
So far it looks pretty promising and loading around 600 45MB RAW image thumbnails using QLThumbnailGenerator and CGImageSourceCreateWithURL is super fast allowing thumbnail images and image metadata to be displayed almost instantly.
Displaying these images in a NSCollectionView using a CALayer in the NSCollectionViewItem's view also appears to be extremely fast and scrolling is very smooth.
I did find that QLThumbnailGeneratorseems to start failing after a few hundred images and starts returning error code 108 if I call the api in a continuous loop - I fixed that by calling CGImageSourceCopyPropertiesAtIndex immediately after the thumbnail generator api call - so maybe there is a timing issue or not enough file handles or something if the api is called to quickly and for too long.
However I am still having trouble rendering a full sized image to the display - here I am using a NSScrollView with a layer backed NSView documentView. Everything is super fast until the following call:
view.layer.contents = cgImage
And at this point the entire main thread hangs until the image has loaded - and this may take a few seconds.
Once it has loaded it's fine and zooming in and out by changing the documentView frame size is very fast - scrolling around the full size image is also super smooth without any of the typical hiccups.
Is there a way of loading these images without causing the UI to freeze ?
I've seen the recent WWDC2020 session where they demonstrate similar scrolling of large numbers of images but I haven't been able to find anything useful on loading large images other than CATiledLayer - but it's not really clear if that is the right answer for this problem.
The old Apple sample RawExpose seemed to be an option but most of that code is deprecated and it seems one has to use MetalKit not instead of GLKit - unfortunately there is no example of using MetaKit with Core Image that I can find.
FYI - I tried using some the new SwiftUI CollectionView and List but they seem to be significantly slower than AppKit and I found some of the collection view items never render - of course these could just be bugs in the macOS 11 beta.
OK - well I finally figured it out and it's complicated but simple. It's complicated because there are so many options to choose from and so many outdated sample apps to look at. In any event I think I have solved most if not all the issues related to using metal backed CALayers and rendering realtime updates of the images as CIFilter adjustments are applied. There are many pieces to the puzzle and happy to share if anyone is looking for help.
Some key pointers:
I am using CAMetalLayer and NSView
I override the CAMetalLayer.display(layer:) method and call the layer.setNeedsDisplay() when the user slides an adjustment slider.
I chain together all the CIFilters, including the RAW filter created with CIFilter(imageUrl:)
Most importantly I use the RAW filters scaleFactor parameter to size the image - encountered major performance issues using any other method to resize the image for the views size
Don't expect high performance if the image is zoomed right in - 50% is seems to be the limit for 45megapixel RAW imaged from Nikon D850.
A short video of the result is here https://youtu.be/5wp0CIWAoIM
What APIs do I need to use, and what precautions do I need to take, when writing to an IOSurface in an XPC process that is also being used as the backing store for an MTLTexture in the main application?
In my XPC service I have the following:
IOSurface *surface = ...;
CIRenderDestination *renderDestination = [... initWithIOSurface:surface];
// Send the IOSurface to the client using an NSXPCConnection.
// In the service, periodically write to the IOSurface.
In my application I have the following:
IOSurface *surface = // ... fetch IOSurface from NSXPConnection.
id<MTLTexture> texture = [device newTextureWithDescriptor:... iosurface:surface];
// The texture is used in a fragment shader (Read-only)
I have an MTKView that is running it's normal update loop. I want my XPC service to be able to periodically write to the IOSurface using Core Image and then have the new contents rendered by Metal on the app side.
What synchronization is needed to ensure this is done properly? A double or triple buffering strategy is one, but that doesn't really work for me because I might not have enough memory to allocate 2x or 3x the number of surfaces. (The example above uses one surface for clarity, but in reality I might have dozens of surfaces I'm drawing to. Each surface represents a tile of an image. An image can be as large as JPG/TIFF/etc allows.)
WWDC 2010-442 talks about IOSurface and briefly mentions that it all "just works", but that's in the context of OpenGL and doesn't mention Core Image or Metal.
I originally assumed that Core Image and/or Metal would be calling IOSurfaceLock() and IOSurfaceUnlock() to protect read/write access, but that doesn't appear to be the case at all. (And the comments in the header file for IOSurfaceRef.h suggest that the locking is only for CPU access.)
Can I really just let Core Image's CIRenderDestination write at-will to the IOSurface while I read from the corresponding MTLTexture in my application's update loop? If so, then how is that possible if, as the WWDC video states, all textures bound to an IOSurface share the same video memory? Surely I'd get some tearing of the surface's content if reading and writing occurred during the same pass.
The thing you need to do is ensure that the CoreImage drawing has completed in the XPC before the IOSurface is used to draw in the application. If you were using either OpenGL or Metal on both sides, you would either call glFlush() or [-MTLRenderCommandEncoder waitUntilScheduled]. I would assume that something in CoreImage is making one of those calls.
I can say that it will likely be obvious if that's not happening because you will get tearing or images that are half new rendering and half old rendering if things aren't properly synchronized. I've seen that happen when using IOSurfaces across XPCs.
One thing you can do is put some symbolic breakpoints on -waitUntilScheduled and -waitUntilCompleted and see if CI is calling them in your XPC (assuming the documentation doesn't explicitly tell you). There are other synchronization primitives in Metal, but I'm not very familiar with them. They may be useful as well. (It's my understanding that CI is all Metal under the hood now.)
Also, the IOSurface object has methods -incrementUseCount, -decrementUseCount, and -localUseCount. It might be worth checking those to see if CI sets them appropriately. (See <IOSurface/IOSurfaceObjC.h> for details.)
I am having some memory issues with our android app when handling bitmaps (duh!).
We are having multiple activities loading images from a server, this could be a background image for the activity.
This background image could be the same for multiple activities, and right now each activity is loading its own background image.
This means if the flow is ac1->ac2->ac3->ac4 the same image will be loaded 4 times and using 4x memory.
How do I optimize imagehandling for this scenario? Do I create an image cache where the image is stored and then each activity ask the cache first for images. If this is the case, how do I know when to garbage collect the image from the cache?
Any suggestions, link to good tutorials or similar is highly appreciated.
Regards
EDIT:
When downloading images for the device the exact sizes is used, meaning that if the ui element needs an 100x100 pixel image it gets that size and therefore no need for scaling. So i am not sure about downscaling the image when loading it into the memory. Maybe it is needed to unload images in the activity when moving on the the next and then reload when going back.
One thing you might want to try is scaling down your bitmaps (make a thumbnail) to a size that is more appropriate to your device. It's pretty easy to quickly use up all the RAM on an Android device with a bitmap if you don't scale it down. This recipe shows how to do so on the fly. You could adapt this to save the images to disk.
You could also create your own implementation of the LRUCache to cache the images for your app.
After ready your update I will give you an other tip then.
I can still post the patch too if people want that..
What you need to do with those bitmaps is call them with a using block. That way Android will unload the bitmap as soon as that block is executed.
Example:
using(Bitmap usedBitmap = new Bitmap()){
//Do stuff with the Bitmap here
//You can call usedBitmap.Dispose() but it's not really needed
}
With this code your app shouldn't keep all the used bitmaps in memory.
I'm developing an app for WP7 (VS2010 Express for Windows Phone RTM and WP Emulator), but now im facing a big problem related to memory usage.
The app itself has like 12 views, and some of them are reused with different data. It's a newsreader, so the views are mostly listboxes (image thumbnail, stackpanels and textblocks).
The first view has a listbox with 30 items. This takes about 20MB of RAM, but as i navigate between views the current and peak RAM usage start to rise. Well the peak usage it's around 55MB which i think is OK, but the app has a gallery section in which after selecting a thumbnail it navigates to another view which displays a downloaded image (JPEG, 1131px × 1647px, ~486KB) but initially fit to the screen. Until here all good, but for scaling purposes i'm using Laurent Bugnion's Multitouch Behavior and the problem is that when i zoom in the image, the memory usage gets near 90MB (like 87MB last time i tested with the maximum scale size at 2.5).
Also after i navigate the views the current ram usage may permantly reach and stay at 35MB, which i beleive is due to the device caching some things.
So, as the title says, how can i avoid such huge ram usage?
Edit----
Also i'd like to ask if the fact that in my app one can navigate from any view to almost every other and that in between there's always a page transition animation (like the one for the phone list application template in the beta tools), may be contributing to the excessive memory usage.
Generally speaking you want to keep images as small as feasible (in dimensions and color depth, not necessarily file size). In order to display an image, the device must decompress it to an actual Bitmap, so in this case you're looking at a 1131x1647pixel image, let's say at 16bpp, means that you have a 3.7MB memory footprint for the image, not the 486k file size.
Depending on your zoom/rotate, there may be a second copy buffer, so you can effectively double that. It doesn't take long at that rate to get to 90MB. I'd certainly try either download smaller image files or try resizing them locally and then using the resized image.
Well maybe i should look into deepzoom (but when i first read about it, i thought it was for using it with the same image at different sizes, like google maps in satellite view), but yesterday i solved it by using a webbrowser inside my view, so if before i had:
<Image x:Name="imgPlaceHolder" delay:LowProfileImageLoader.UriSource="{Binding Path=ActualImageSource}" MaxHeight="800"
MaxWidth="480" >
<interac:Interaction.Behaviors >
<tbeh:MultiTouchBehavior x:Name="ImageMTB" IsScaleEnabled="True"
MinimumScale="0.4"
MaximumScale="2.5"
IsRotateEnabled="False"
IsDebugModeActive="False"
IsTranslateXEnabled="True"
IsTranslateYEnabled="True"
>
</tbeh:MultiTouchBehavior>
</interac:Interaction.Behaviors>
</Image>
I changed that to:
<phone:WebBrowser Source="{Binding Path=ActualImageSource}" x:Name="wbbigimage" />
It uses less memory and takes care of the zoom with no problems.
I'll wait a bit for someone with a better solution before checking this one.
A couple of things to note. First, make sure that you are clearing out any lists or images that you are using on pages durring the onNavigatedFrom method. The way WP7 Deals with images is "interesting". One of the things I have found to help is to null out the background of any panorama controls when the page is not being displayed.
As for your specific issue, have you considered using a MultiScaleImage (Deep Zoom) to cut down on memory?
http://msdn.microsoft.com/en-us/library/system.windows.controls.multiscaleimage(VS.95).aspx
We would like to display very large (50mb plus) images in Internet Explorer. We would like to avoid compression as compression algorithms are not what CSI would have us believe that they are and the resulting files are too lossy.
As a result, we have come up with two options: Silverlight Deep Zoom or a Flash based solution (such as Zoomify). The issue is that both of these require conversion to a tiled output and/or conversion to a specific file type (Zoomify supports a single proprietary file type, PFF).
What we are wondering is if a solution exists which will allow us to view the image without a conversion before hand.
PS: I know that you can write an application to tile the images (as needed or after the load process) and output them; however, we would like to do this without chopping up the file.
The tiled approach really is the right way to do it.
Your users don't want to download a 50mb file before they can start viewing the image. You don't want to spend the bandwidth to serve 50 megs to every user who might only view a fraction of your image.
If you serve the whole file, users will eventually be able to load and view it, but it won't run smoothly for most of them.
There is no simple non-tiled way to serve just a portion of an image unless you want to use a server-side library like imagemagik or PIL to extract a specific subset of the image for each user. You probably don't want to do that because it will place a significant load on your server.
Alternatively, you might use something like google's map tool to provide zooming and scaling. Some comments on doing that are available here:
http://webtide.wordpress.com/2008/08/27/custom-google-maps/
Take a look at OpenSeadragon. To make a image can work with OpenSeadragon, you should generate a zoomable image format which mentioned here. Then follow starting guide here
The browser isn't going to smoothly load a 50 meg file; if you don't chop it up, there's no reasonable way to make it not lag.
If you dont want to tile, you could have the server open the file and render a screen sized view of the image for display in the browser at the particular zoom resolution requested. This way you arent sending 50 meg files across the line when someone only wants to get an overview of the image. That is, the browser requests a set of coordinates and an output size in pixels, the server opens the larger image and creates a smaller image that fits the desired view, and sends that back to the web browser.
As far as compression, you say its too lossy, but if thats what you are seeing you are probably using the wrong compression algorithm or setting for the type of image you have. The jpg format has quality settings to control lossiness, and PNG compression is lossless (the pixels you get after decompressing are the exact values you had prior to compression). So consider changing what you are using as compression, and dont just rely on the default settings in an image editor.