I'm attempting to perform a fairly large CoreData import (around 25,000 rows) while still maintaining a fairly low memory footprint. I've read the documentation surrounding efficient importing of data and have endeavoured to implement everything suggested there (including setting things like my MOC's undoManager to nil).
Unfortunately, my applications memory usage still climbs to around 180MB when running the below code. Upon completion the application will sit at around the 180MB mark, regardless of the final NSAutoreleasePool drain call.
Running the application through Allocations shows that 95% of the memory usage is attributable to my [self.moc save:&error] call. What am I doing wrong here?
- (void)generateCache
{
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
NSUInteger count = 0, batchSize = 1000;
// SNIP SNIP
// Iterate over our directory structure
for(NSString *item in directoryStructure)
{
NSDictionary *info = [fm attributesOfItemAtPath:item error:nil];
FileRecord *record = (FileRecord *)[NSEntityDescription insertNewObjectForEntityForName:#"FileRecord" inManagedObjectContext:self.moc];
record.size = [NSNumber numberWithUnsignedLongLong:[info fileSize]];
record.path = item;
count ++;
if(count == batchSize)
{
NSError *error = nil;
if([self.moc save:&error])
{
NSLog(#"MOC saved down and reset");
[self.moc reset];
[pool drain];
pool = [[NSAutoreleasePool alloc] init];
count = 0;
}
}
}
// Perform any necessary last minute MOC saves
if (count != 0) {
[self.moc save:nil];
[self.moc reset];
}
// Drain our NSAutoreleasePool
[pool drain];
// Tell our main thread that we're done
if ([self respondsToSelector:#selector(completedCache)])
{
[self performSelectorOnMainThread:#selector(completedCache) withObject:nil waitUntilDone:NO];
}
}
Instead of dealing with auto-release pools, why not explicitly manage the life-cycle of your managed objects by creating them with NSManagedObject's initWithEntity:insertIntoManagedObjectContext:? You can safely release them after modifying the object's properties since a managed object context retains a newly inserted object -- till its saved to the persistent store.
Also, I should mention a couple of problems that I see with your code:
As someone mentioned above, you are not logging errors from the save: operation. You really should -- that may highlight some (possibly unrelated) problem.
If save: is successful, you really should not need to call reset. See this section in the core data guide.
Related
I have an NSMutableArray I need to add objects to from multiple blocks I have dispatched. Is this an acceptable way to make sure that the array is safely being changed? These are already being dispatched from inside and NSOperation and running in the background. I was loading the data from within that thread serially but it was getting very slow to load a list of locations at once.
NSMutableArray *weatherObjects = [[NSMutableArray alloc] init];
ForecastDownloader *forecastDownloader = [[ForecastDownloader alloc] init];
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_group_t group = dispatch_group_create();
dispatch_queue_t serialQueue;
serialQueue = dispatch_queue_create("us.mattshepherd.ForecasterSerialQueue", NULL);
for (NSDictionary *theLocation in self.weatherLocations) {
// Add a task to the group
dispatch_group_async(group, queue, ^{
NSLog(#"dispatching...");
int i = 0;
WeatherObject *weatherObject = [forecastDownloader getForecast:[theLocation objectForKey:#"lat"] lng:[theLocation objectForKey:#"lng"] weatherID:[[theLocation objectForKey:#"id"] intValue]];
}
if(!weatherObject){
//need to implement delegate method to show problem updating weather
NSLog(#"problem updating weather data");
}else{
NSLog(#"got weather for location...");
dispatch_sync(serialQueue, ^{
[weatherObjects addObject:weatherObject];
});
}
});
}
// wait on the group to block the current thread.
dispatch_group_wait(group, DISPATCH_TIME_FOREVER);
NSLog(#"finished getting weather for all locations...");
//we will now do something with the weatherObjects
That's not going to work because you're making a new lock each time, rather than using a single lock for the variable (analogy: imagine a locked door to a room. If everyone gets their own door with a lock, it hardly matters that they lock it, since everyone else will come in their own door).
You can either use a single NSLock for all iterations, or (basically equivalently) a single serial dispatch queue.
We are trying to create multiple pdf files by using UIView.layer's renderInContext method
The below code run in a autoreleasepool.
---loop
UIGraphicsBeginPDFContextToFile(documentDirectoryFilename, CGRectZero, nil);
UIGraphicsBeginPDFPage();
CGContextRef pdfContext = UIGraphicsGetCurrentContext();
[view.layer renderInContext:pdfContext];
view =nil;
pdfContext =nil
UIGraphicsEndPDFContext()
--loop ends
After couple of iterations resident memory increases to 20 mb and subsequent iteration adds to the total memory used.
Some how ARC is not releasing the memory used in rendering the pdf and thus the application crashes with low memory.
Application is using ARC
Any help or pointers to resolve the issue would be much appreciated.
Thanks
UPDATED:
Thanks for the quick reply.
My apologies for having a typo in the pseudocode.
Here is sample code from a test project.
-(BOOL)addUIViewToPDFFile:(UIView*)view newBounds:(CGRect)newBounds pdfFile:(NSString*)pdfFile{
BOOL retFlag =YES;
NSArray* documentDirectories = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask,YES);
NSString* documentDirectory = [documentDirectories objectAtIndex:0];
NSString* documentDirectoryFilename = [documentDirectory stringByAppendingPathComponent:pdfFile];
UIGraphicsBeginPDFContextToFile(documentDirectoryFilename, CGRectZero, nil);
CGContextRef pdfContext = UIGraphicsGetCurrentContext();
if([NSThread isMainThread]){
NSLog(#"Main Thread");
}else{
NSLog(#"Not in Main Thread");
}
UIGraphicsBeginPDFPage();
logMemUsage1();
[view.layer renderInContext:pdfContext];
UIGraphicsEndPDFContext();
pdfContext = nil;
logMemUsage1();
return retFlag;
}
- (IBAction)createPdf:(id)sender {
for(int i=0;i<10;i++){
#autoreleasepool {
PDFViewController *vc = [[PDFViewController alloc] init];
[vc loadView];
NSString *PdfFileName =[ NSString stringWithFormat:#"TestPdf%d.pdf",i ];
[self addUIViewToPDFFile:vc.view newBounds:self.view.frame pdfFile:PdfFileName];
vc = nil;
}
}
}
One more to point to add is that when application goes into the background on press of the hardware home button and comes back again to foreground the memory seems to get cleared and released.
I am expecting that after each iteration the memory which is used for creating the pdf should be released.
I am writing a dictionary application, and i am trying to import raw data from strings, one string per word. Amongst other things, the raw input strings contain the names of the parts of speech the corresponding word belongs to. In my datamodel I have a separate entity for Words and PartOfSpeech, and i want to create one entity of the type PartOfSpeech for each unique part of speech there may be in the input strings, and establish the relationships from the Words to the relevant pars of speech. The PartOfSpeech entity has just one Atribute, name, and one-to-many relationship to the Word:
My first implementation of getting unique PartOfSpeech entities involved caching them in a mutable array and filtering it each time with a predicate. It worked, but it was slow. I decided to speed it up a bit by caching the PartsOfSpeech in an NSDictionary, and now when i try and save the datastore after the import, i get the error "Cannot save objects with references outside of their own stores.". It looks like the problem is in the dictionary, but how can i solve it?
Here is the code that worked:
(in both sniplets managedObjectContext is an ivar, and processStringsInBackground: method runs on a background thread using performSelectorInBackground:withObject: method)
- (void) processStringsInBackground:(NSFetchRequest *)wordStringsReq {
NSError *err = NULL;
NSFetchRequest *req = [[NSFetchRequest alloc] init];
[req setEntity:[NSEntityDescription entityForName:#"PartOfSpeech" inManagedObjectContext:managedObjectContext]];
err = NULL;
NSMutableArray *selectedPartsOfSpeech = [[managedObjectContext executeFetchRequest:req error:&err] mutableCopy];
NSPredicate *p = [NSPredicate predicateWithFormat:#"name like[c] $name"];
// NSPredicate *formNamePredicate = [NSPredicate predicateWithFormat:<#(NSString *)predicateFormat#>]
...
for (int i = 0; i < count; i++){
...
currentPos = [self uniqueEntityWithName:#"PartOfSpeech" usingMutableArray:selectedPartsOfSpeech predicate:p andDictionary:[NSDictionary dictionaryWithObject:partOfSpeech forKey:#"name"]];
...
}
}
- (NSManagedObject *) uniqueEntityWithName:(NSString *) entityName usingMutableArray:(NSMutableArray *)objects predicate:(NSPredicate *)aPredicate andDictionary:(NSDictionary *) params {
NSPredicate *p = [aPredicate predicateWithSubstitutionVariables:params];
NSArray *filteredArray = [objects filteredArrayUsingPredicate:p];
if ([filteredArray count] > 0) {
return [filteredArray objectAtIndex:0];
}
NSManagedObject *newObject = [NSEntityDescription insertNewObjectForEntityForName:entityName inManagedObjectContext:managedObjectContext];
NSArray *dicKeys = [params allKeys];
for (NSString *key in dicKeys) {
[newObject willChangeValueForKey:key];
[newObject setPrimitiveValue:[params valueForKey:key] forKey:key];
[newObject didChangeValueForKey:key];
}
[objects addObject:newObject];
return newObject;
}
And here is the same, but with caching using NSMutableDictionary, which fails to save afterwards:
- (void) processStringsInBackground:(NSFetchRequest *)wordStringsReq {
NSError *err = NULL;
[req setEntity:[NSEntityDescription entityForName:#"PartOfSpeech" inManagedObjectContext:managedObjectContext]];
NSArray *selectedPartsOfSpeech = [managedObjectContext executeFetchRequest:req error:&err];
NSMutableDictionary *partsOfSpeechChache = [[NSMutableDictionary alloc] init];
for (PartOfSpeech *pos in selectedPartsOfSpeech) {
[partsOfSpeechChache setObject:pos forKey:pos.name];
}
...
for (int i = 0; i < count; i++){
...
currentPos = [self uniqueEntity:#"PartOfSpeech" withName:partOfSpeech usingDictionary:partsOfSpeechChache];
...
}
}
- (NSManagedObject *)uniqueEntity:(NSString *) entityName withName:(NSString *) name usingDictionary:(NSMutableDictionary *) dic {
NSManagedObject *pos = [dic objectForKey:name];
if (pos != nil) {
return pos;
}
NSManagedObject *newPos = [NSEntityDescription insertNewObjectForEntityForName:entityName inManagedObjectContext:managedObjectContext];
[newPos willChangeValueForKey:#"name"];
[newPos setPrimitiveValue:name forKey:#"name"];
[newPos didChangeValueForKey:#"name"];
[dic setObject:newPos forKey:name];
return newPos;
}
Could you help me to find the problem?
Best regards,
Timofey.
The error is caused by forming a relationship between managedObjects that don't share the same persistent store. You can do that by:
Creating a managed object with initialization without inserting it into a context.
Deleting a managed object from a context while retaining it in another object e.g. array, and then forming a relationship with it.
Accidentally creating two Core Data stacks so that you have two context and two stores.
Confusing configurations in a multi-store context.
I don't see any part of the code you provided that would trigger the problem.
It turns out, that it is wrong to pass NSManagedContext to a thread different from the one it was created in. Instead, one should pass the NSPersistenceStroreCoordinator to another thread, and create a new managed object context there. In order to merge the changes into the "main" context, one should save the other thread's context, receive the notification on the completion of the save on the main thread and merge the changes (see apple docs regarding Core Data and concurrency, can't give you the link, because i read it in Xcode). So here are the changes i made to my code to make it work (only posting the changed lines):
— (void) processStringsInBackground:(NSDictionary *) params {
NSFetchRequest *wordStringsReq = [params objectForKey:#"wordStringsReq"];
NSPersistentStoreCoordinator *coordinator = [params objectForKey:#"coordinator"];
NSManagedObjectContext *localContext = [[NSManagedObjectContext alloc] init];
[localContext setPersistentStoreCoordinator:coordinator];
(all the references to the managedObjectContext were replaced by localContext
And on the main thread, i call this method thusly:
.......
NSDictionary *dict = [NSDictionary dictionaryWithObjectsAndKeys:req, #"wordStringsReq", persistentStoreCoordinator, #"coordinator", nil]; //the params i pass to the background method
[[NSNotificationCenter defaultCenter] addObserver:self selector:#selector(handleNotification:) name:#"NSManagingContextDidSaveChangesNotification" object:nil]; //register to receive the notification from the save
[self performSelectorInBackground:#selector(processStringsInBackground:) withObject:dict];
}
- (void) handleNotification:(NSNotification *) notific {
NSLog(#"got notification, %#", [notific name]);
[managedObjectContext mergeChangesFromContextDidSaveNotification:notific];
}
Good luck!
Good answers, though a bit dated. The fine documentation notes that the main NSManagedObjectContext should never be used in worker threads. Instead, create a separate NSManagedObjectContext private to the worker using the "main" MOC as a parent, and then that instead. Here's the relevant "Concurrency" page from the Core Data Programming Guide:
https://developer.apple.com/library/prerelease/ios/documentation/Cocoa/Conceptual/CoreData/Concurrency.html
Snippet (Swift)
let jsonArray = … //JSON data to be imported into Core Data
let moc = … //Our primary context on the main queue
let privateMOC = NSManagedObjectContext(concurrencyType: .PrivateQueueConcurrencyType)
privateMOC.parentContext = moc
privateMOC.performBlock {
for jsonObject in jsonArray {
let mo = … //Managed object that matches the incoming JSON structure
//update MO with data from the dictionary
}
do {
try privateMOC.save()
} catch {
fatalError("Failure to save context: \(error)")
}
}
Following is the code snippet that I have:
// Make Auto release pool
NSAutoreleasePool * autoReleasePool = [[NSAutoreleasePool alloc] init];
try
{
if (mCapture)
{
// Get the image reference
NSImage* image = NULL;
image = [mCapture getCurrentFrameImage];
// Get the TIFF data
NSData *pDataTifData = [[NSData alloc] initWithData:[image TIFFRepresentation]];
NSBitmapImageRep *pBitmapImageRep = [[NSBitmapImageRep alloc] initWithData:pDataTifData];
// Convert to BMP data
NSData *pDataBMPData;
pDataBMPData = [pBitmapImageRep representationUsingType: NSPNGFileType
properties: nil];
// Save to specified path
ASL::String strPath = ASL::MakeString(capInfo->thefile.name);
NSString* pPath = (NSString*)ASL::MakeCFString(strPath);
[pDataBMPData writeToFile:pPath
atomically: YES];
::CFRelease(pPath);
pDataBMPData = nil;
[pBitmapImageRep release];
pBitmapImageRep = nil;
[pDataTifData release];
pDataTifData = nil;
image = nil;
}
}
catch(...)
{
}
[autoReleasePool drain];
Note that image = [mCapture getCurrentFrameImage]; is returning an autoreleased NSImage. I am releasing objects and also have NSAutoreleasePool in place. But still it is leaking about 3-4 MB of memory everytime this code snippet is executed. I am not sure where the mistake is.
You could simplify this code a lot by making captureCurrentFrameImage return an NSBitmapImageRep instead of an NSImage, since you never actually use an NSImage for anything here. You can wrap the image rep in an image when necessary, and for this code, simply use the image rep by itself to produce the PNG data. Among other things, this saves you a trip through the TIFF representation.
If it still leaks after you make those changes, run your app under Instruments's Leaks template; the two instruments in that template, Leaks and ObjectAlloc, will help you hunt down whatever leaks you have.
I have an method which save files to the internet, it works but just slow. Then I'd like to make the user interface more smooth, so I create an NSThread to handle the slow task.
I am seeing a list of errors like:
_NSAutoreleaseNoPool(): Object 0x18a140 of class NSCFString autoreleased with no pool in place - just leaking
Without NSThread, I call the method like:
[self save:self.savedImg];
And I used the following to use NSThread to call the method:
NSThread* thread1 = [[NSThread alloc] initWithTarget:self
selector:#selector(save:)
object:self.savedImg];
[thread1 start];
Thanks.
Well first of all, you are both creating a new thread for your saving code and then using NSUrlConnection asynchronously. NSUrlConnection in its own implementation would also spin-off another thread and call you back on your newly created thread, which mostly is not something you are trying to do. I assume you are just trying to make sure that your UI does not block while you are saving...
NSUrlConnection also has synchronous version which will block on your thread and it would be better to use that if you want to launch your own thread for doing things. The signature is
+ sendSynchronousRequest:returningResponse:error:
Then when you get the response back, you can call back into your UI thread. Something like below should work:
- (void) beginSaving {
// This is your UI thread. Call this API from your UI.
// Below spins of another thread for the selector "save"
[NSThread detachNewThreadSelector:#selector(save:) toTarget:self withObject:nil];
}
- (void) save {
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
// ... calculate your post request...
// Initialize your NSUrlResponse and NSError
NSUrlConnection *conn = [NSUrlConnection sendSyncronousRequest:postRequest:&response error:&error];
// Above statement blocks until you get the response, but you are in another thread so you
// are not blocking UI.
// I am assuming you have a delegate with selector saveCommitted to be called back on the
// UI thread.
if ( [delegate_ respondsToSelector:#selector(saveCommitted)] ) {
// Make sure you are calling back your UI on the UI thread as below:
[delegate_ performSelectorOnMainThread:#selector(saveCommitted) withObject:nil waitUntilDone:NO];
}
[pool release];
}
You need to mainly create an autorelease pool for the thread. Try changing your save method to be like this:
- (void) save:(id)arg {
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
//Existing code
[pool drain];
}
You will not you that the above does not call release on the NSAutoreleasePool. This is a special case. For NSAutoreleasePool drain is equivalent to release when running without GC, and converts to a hint to collector that it might be good point to run a collection.
You may need to create a run loop. I will add to Louis's solution:
BOOL done = NO;
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
[NSRunLoop currentRunLoop];
// Start the HTTP connection here. When it's completed,
// you could stop the run loop and then the thread will end.
do {
SInt32 result = CFRunLoopRunInMode(kCFRunLoopDefaultMode, 1, YES);
if ((result == kCFRunLoopRunStopped) || (result == kCFRunLoopRunFinished)) {
done = YES;
}
} while (!done);
[pool release];
Within the thread, you need to create a new autorelease pool before you do anything else, otherwise the network operations will have issues as you saw.
I don't see any reason for you to use threads for this. Simply doing it asynchronously on the run loop should work without blocking the UI.
Trust in the run loop. It's always easier than threading, and is designed to provide the same result (a never-blocked UI).