I am shuffling hundreds of bitmaps into and out of RAM as my users browse through their asset library. There is a maximum storage space for bitmaps. Right now, doing the conservative thing, I am dumping all loaded bitmaps when the asset view changes (think of a page flip etc.).
This results in rather jarring delays in the fetch/render cycles, even if the asset was already loaded in the last page.
I am now wondering how I could write a cache that would retain some bitmaps in memory, releasing them on timeout or when they are explicitly dropped by the app.
I am using assets from multiple sources, including local disk and network URIs..., so simply using the XAML mechanisms won't work (as that only is applicable for network URIs, not local assets).
To do that, I would need to know, how much memory the app actually has to work with for bitmaps. Is there any way to find out at runtime???
Related
I'm trying to store a lot of images for a project in xcassets. Around 2,000. However, they are all very small/highly optimized at only around 3-5kB each, so the overall total will be maybe 6-10MB. Is this going to cause a memory problem? Is this not recommended?
I know I could have the user download them from a server on a need basis, but it would be easier if I could include them all right from the app download.
How many images can you store in XCAssets?
Well... As per Apple on Maximum build file sizes:
Your app’s total uncompressed size must be less than 4GB
So technically you can store as many assets that can fit in this limit.
It's not going to cause memory problem because these images are stored in the app's bundle which is eventually stuff stored on disk.
It's just going to increase the app download size... for now. It will affect the app's memory once they are used/displayed.
Best example for this are games, they generally have many assets and almost always in the app's bundle itself. In some cases you would even download expansion packs that contain more assets but seldom will you see a game art download, atleast not while playing because smoothness is a key factor whether it be a game or a simple app.
Around 2,000. However, they are all very small/highly optimized at only around 3-5kB each, so the overall total will be maybe 6-10MB.
Firstly, 10MB is hardly anything to worry about but anyways the point is that if you load all 2000 images at the same time, i.e each image is loaded in a UIImageView, only then will they be loaded in app's memory.
This wouldn't matter whether you had to load the images from the app's bundle or you had to download it from a server.
How many images should you store in XCAssets?
Now that depends on you.
Consider the following before deciding:
Are these images likely to change?
Do my users need all 2000 images?
Do we want to maintain a list of urls for 2000 images?
Do we need to ensure that the images download properly?
Would users mind waiting a few milliseconds for an image to load?
If an image doesn't download, for whatever reason, then what should be an appropriate failure mechanism?
Depending on how you answer the above, you will know what path you need to take.
The Image Qt Quick component has a cache property. The documentation on that property, however, is very scant. In particular, it does not say what the cache is used for, and when images are loaded from the cache.
Can you tell me the implications of setting Image.cache to either true or false?
I'm asking because I'm tempted to save a lot of memory by setting cache: false on all my Images.
The docs do say this:
Images are cached and shared internally, so if several Image items have the same source, only one copy of the image will be loaded.
But does this mean that cache: true only has a point when several Image items have the same source? And does it mean that cache: false results in no gain whatsoever?
Note: In my tests with ~1000 images mem usage dropped from 247MB to 237MB when I disabled cache. And this app has a lot of other things in it too. So I'm happy with this win. But my questions stand.
The difference you have is in the margin of error. QML is very careless with memory management. Make sure the behavior is consistent before you write it off as a gain.
What the doc actually says about Image.cache is:
Specifies whether the image should be cached. The default value is
true. Setting cache to false is useful when dealing with large images,
to make sure that they aren't cached at the expense of small 'ui
element' images.
Which I assume is that the image will be kept in (presumably) GPU RAM whereas non-cached images are transferred there on demand. As far as I know, all Images, visible or not, will take up system RAM, which has always been odd to me, since having an image that is solely for the purpose of visualizing on a GPU backend can actually omit the copy in RAM and only keep the image in VRAM.
The doc entry you quote looks like it concerns the scenegraph, that caching is implicit, which only makes sense, because there is no reason storing the same data twice, it will refcount each image url so that if it is used multiple times it will resolve to the same single packet of data. Image.cache has nothing to do with that cache.
At any rate, I'd say the only viable solution to make sizable gains is to do lazy loading. Using vectors won't save you much, since the images are internally cached rasterized. Using more compressed images will reduce the binary size, but they are still cached decoded in memory.
If your images are stuff like grayscale icon stencils, they render tremendously well to signed distance field representation. In my project I managed to reduce 83 MB worth of such icons to 1.25 MB of SDFs with no loss of quality, in fact with the SDFs I was able to enlarge the GUI even more without getting jagged edges. And the shader to render an SDF into a raster of the appropriate size is only a tad more complex than a flat color shader, so performance differences are negligible, but since this happens on the fly I also get the same amount of memory saved from both RAM and VRAM.
I'm building a desktop application for macOS that allows a user to browse a collection of videos. Generating thumbnails for each video is a relatively expensive operation so I'd like to cache any thumbnails that have been generated. This cache does not need to persist across application launches. Speed is not that critical because just about anything is faster than decoding frames of a video.
I'm considering using RocksDB to store the thumbnails and was wondering if this is an appropriate use of RocksDB. Most of the RocksDB documentation involves setting and getting strings as values, but in recent versions the addition of PinnableSlice appears to make it easier for storing binary data. In my application, I'd be storing heavily compressed JPEG data for thumbnails up to 512 x 512px.
Is there anything I should take into consideration when using RocksDB in this fashion? Are there any configuration options that would be beneficial to tune or adjust for such a use-case? I would prefer if RocksDB kept a low in-memory overhead because I'm willing to take the performance hit of reading back from disk when a thumbnail is needed.
(And if using RocksDB as an image cache is a really bad idea to begin with, then I'd like to better understand why.)
I was wondering whether it is a good idea to create a "system" wide rendering server that is responsible for the rendering of all application elements. Currently, applications usually have their own context, meaning whatever data might be identical across different applications, it will be duplicated in GPU memory and the more frequent resource management calls only decrease the count of usable render calls. From what I understand, the OpenGL execution engine/server itself is sequential/single threaded in design. So technically, everything that might be reused across applications, and especially heavy stuff like bitmap or geometry caches for text and UI, is just clogging the server with unnecessary transfers and memory usage.
Are there any downsides to having a scenegraph shared across multiple applications? Naturally, assuming the correct handling of clients which accidentally freeze.
I was wondering whether it is a good idea to create a "system" wide rendering server that is responsible for the rendering of all application elements.
That depends on the task at hand. A small detour: Take a webbrowser for example, where JavaScript performs manipulations on the DOM; CSS transform and SVG elements define graphical elements. Each JavaScript called in response to an event may run as a separate thread/lighweight process. In a matter of sense the webbrowser is a rendering engine (heck they're internally even called rendering engines) for a whole bunch of applications.
And for that it's a good idea.
And in general display servers are a very good thing. Just have a look at X11, which has an incredible track record. These days Wayland is all the hype, and a lot of people drank the Kool-Aid, but you actually want the abstraction of a display server. However not for the reasons you thought. The main reason to have a display server is to avoid redundant code (not redundant data) and to have only a single entity to deal with the dirty details (color spaces, device physical properties) and provide optimized higher order drawing primitives.
But in regard with the direct use of OpenGL none of those considerations matter:
Currently, applications usually have their own context, meaning whatever data might be identical across different applications,
So? Memory is cheap. And you don't gain performance by coalescing duplicate data, because the only thing that matters for performance is the memory bandwidth required to process this data. But that bandwidth doesn't change because it only depends on the internal structure of the data, which however is unchanged by coalescing.
In fact deduplication creates significant overhead, since when one application made changes, that are not to affect the other application a copy-on-write operation has to be invoked which is not for free, usually means a full copy, which however means that while making the whole copy the memory bandwidth is consumed.
However for a small, selected change in the data of one application, with each application having its own copy the memory bus is blocked for much shorter time.
it will be duplicated in GPU memory and the more frequent resource management calls only decrease the count of usable render calls.
Resource management and rendering normally do not interfere with each other. While the GPU is busy turning scalar values into points, lines and triangles, the driver on the CPU can do the housekeeping. In fact a lot of performance is gained by keeping making the CPU do non-rendering related work while the GPU is busy rendering.
From what I understand, the OpenGL execution engine/server itself is sequential/single threaded in design
Where did you read that? There's no such constraint/requirement on this in the OpenGL specifications and real OpenGL implementations (=drivers) are free to parallelize as much as they want.
just clogging the server with unnecessary transfers and memory usage.
Transfer happens only once, when the data gets loaded. Memory bandwidth consumption is unchanged by deduplication. And memory is so cheap these days, that data deduplication simply isn't worth the effort.
Are there any downsides to having a scenegraph shared across multiple applications? Naturally, assuming the correct handling of clients which accidentally freeze.
I think you completely misunderstand the nature of OpenGL. OpenGL is not a scene graph. There's no scene, there are mo models in OpenGL. Each applications has its own layout of data and eventually this data gets passed into OpenGL to draw pixels onto the screen.
To OpenGL however there are just drawing commands to turn arrays of scalar values into points, lines and triangles on the screen. There's nothing more to it.
I'm writing an application leveraging JavaFX that scrolls a large amount of image content on and off of the screen every 20-30 seconds. It's meant to be able to run for multiple hours, pulling in completely new content and discarding old content every couple minutes. I have 512Mb of graphics memory on my system and after several minutes, all of that memory has been consumed by JavaFX and no matter what I do with my JavaFX scene, none of it is released. I've been very careful to discard nodes when they drop off of the scene, and at most I have 50-60 image nodes in memory at one time. I really need to be able to do a hard release of the graphics memory that was backing these images, but haven't been able to figure out how to accomplish that, as the Image interface in JavaFX seems to be very high level. JavaFX will continue to run fine, but other graphics heavy applications will fail to load due to limited resources.
I'm looking for something like the flush() method on java.awt.image.Image:
http://docs.oracle.com/javase/7/docs/api/java/awt/Image.html#flush()
I'm running java 7u13 on Linux.
EDIT:
I managed to work out a potential workaround ( see below ), but have also entered a JavaFX JIRA ticket to request the functionality described above:
RT-28661
Add explicit access to a native resource cleanup function on nodes.
The best workaround that I could come up with was to set my JVM's max heap to half of the available limit of my graphics card. ( I have 512mb of graphics memory, so I set this to -Xmx256m ) This forces the GC to be more proactive in cleaning up my discarded javafx.image.Image objects, which in turn seems to trigger graphics memory cleanup on the part of JavaFX.
Previously my heap space was set to 512mb, ( I have 4gb of system memory, so this is a very manageable limit ). The problem with that seems to be that the JVM was being very lazy about cleaning up my images until it started approaching this 512mb limit. Since all of my image data was copied into graphics memory, this meant I had most likely exhausted my graphics memory before the JVM really started really caring about cleanup.
I did try some of the suggestions by jewelsea:
I am calling setCache(false), so this may be having a positive affect, but I didn't notice an improvement until I dropped my max heap size.
I tried running with Java8 with some positive results. It did seem to behave better in graphics memory management, but it still ate up all of my memory and didn't seem to start caring about graphics memory until I was almost out. If reducing your the application's heap limit is not feasible, then evaluating the Java8 pre-release may be worthwhile.
I will be posting some feature requests to the JavaFX project and will provide links to the JIRA tickets.
Perhaps you are encountering behaviour related to the root cause of the following issue:
RT-16011 Need mechanism for PG nodes to know when they are no longer part of a scene graph
From the issue description:
Some PG nodes contain handles to non-heap resources, such as GPU textures, which we would want to aggressively reclaim when the node is no longer part of a scene graph. Unfortunately, there is no mechanism to report this state change to them so that they can release their resources so we must rely on a combination of GC, Ref queues, and sometimes finalization to reclaim the resources. Lazy reclamation of some of these resources can result in exceptions when garbage collection gets behind and we run out of these limited resources.
There are numerous other related issues you can see when you look at the issue page I linked (signup is required to view the issue, but anybody can signup).
A sample related issue is:
RT-15516 image data associated with cached nodes that are removed from a scene are not aggressively released
On which a user commented:
I found a workaround for my app just settihg up an using of cashe to false for all frequently using nodes. 2 days working without any crashes.
So try calling setCache(false) on your nodes.
Also try using a Java 8 preview release where some of these issues have been fixed and see if it increases the stability of your application. Though currently, even in the Java 8 branch, there are still open issues such as the following:
RT-25323 Need a unified Texture resource management system for Prism
Currently texture resources are managed separately in at least 2 places depending on how it is used; one is a texture cache for images and the other is the ImagePool for RTTs. This approach is flawed in its design, i.e. the 2 caches are unaware of each other and it assumes system has unlimited native resources.
Using a video card with more memory may either reduce or eliminate the issue.
You may also wish to put together a minimal executable example which demonstrates your issue and raise a bug request against the JavaFX Runtime project so that a JavaFX developer can investigate your scenario and see if it is new or a duplicate of a known issue.