Calloc is called from some signal handler in my production code. I know that it is not safe to call calloc from a signal handler.
I need to find the point where is this done in my production code, but backtrace is of no help. Please guide
print $_siginfo.si_signo
I know above tells the signal number of the current signal. But that does not help me to find the point.
Following is the call stack, the stacktrace just below calloc is of no use.
#26 0x00002aaaaaccfec9 in calloc () from
#27 0x00000000047c522f in Ox4074f61149c39b9f ()
#28 0x00000000047aa93a in Ox4074ff7b773cfdf6 ()
#29 0x000000000468c981 in Ox4074ea3d1f5df78f ()
#30 0x000000000468e667 in ?? ()
#31 0x000000000468f2dd in ?? ()
#32 0x0000000004686f2c in Ox4074ea18721a0b5d ()
#33 0x00000000047b3727 in Ox4075027a04de15e3 ()
#34 0x00000000047efdd4 in ?? ()
#35 <signal handler called>
#36 0x00002aaabc5d0332 in __lll_lock_wait () from /lib64/libpthread.so.0
#37 0x00002aaabc5cb5d8 in _L_lock_854 () from /lib64/libpthread.so.0
#38 0x00002aaabc5cb4a7 in pthread_mutex_lock () from
/lib64/libpthread.so.0
Please guide
The most likely explanation for the stack trace you observed: you are analyzing a core dump from production machine on a different machine, and using the wrong system libraries to do so.
See this answer on how to do this properly.
Related
I've been tracking down an occasional crash in my app, which comes from an EXC_BAD_ACCESS KERN_PROTECTION_FAILURE, and I finally found something that it could be by using the Address Sanitizer. Unfortunately, I have no idea what the ASan output is telling me.
I've set up my Core Data stack based on this tutorial's stack. The ASan warning warning shows up here:
func mergeChanges(from transactions: [NSPersistentHistoryTransaction]) {
let context = viewContext
context.perform {
for transaction in transactions {
let note = transaction.objectIDNotification()
context.mergeChanges(fromContextDidSave: note) // heap buffer overflow!
try? self.storeHistoryToken(transaction.token)
}
}
}
I don't know what I could do to fix this code, since it seems pretty simple. It does happen when there's a lot of data being imported to Core Data through Batch Insertion Requests.
The output of the sanitizer into the debug terminal looks like this:
==59326==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6280004dbcff at pc 0x00010c71c8bd bp 0x7ff7b724b9d0 sp 0x7ff7b724b190
READ of size 15360 at 0x6280004dbcff thread T0
#0 0x10c71c8bc in wrap_memcpy+0x16c (libclang_rt.asan_iossim_dynamic.dylib:x86_64+0x1b8bc)
#1 0x7ff804aafd81 in fetchResultSetReallocCurrentRow+0x206 (CoreData:x86_64+0x1f2d81)
#2 0x7ff804a80dce in -[NSSQLiteConnection fetchResultSet:usingFetchPlan:]+0x317 (CoreData:x86_64+0x1c3dce)
#3 0x7ff804ab0ed1 in newFetchedRowsForFetchPlan_MT+0x471 (CoreData:x86_64+0x1f3ed1)
#4 0x7ff804bb0807 in _executeObjectFaultRequest+0x307 (CoreData:x86_64+0x2f3807)
#5 0x7ff804bb26e7 in _executeNewRowValuesForObjectFaultRequest+0xb6 (CoreData:x86_64+0x2f56e7)
#6 0x7ff804abe237 in -[NSSQLObjectFaultRequestContext executeRequestCore:]+0x14 (CoreData:x86_64+0x201237)
#7 0x7ff80491f370 in -[NSSQLStoreRequestContext executeRequestUsingConnection:]+0x1b0 (CoreData:x86_64+0x62370)
#8 0x7ff8049debfa in __52-[NSSQLDefaultConnectionManager handleStoreRequest:]_block_invoke+0x37 (CoreData:x86_64+0x121bfa)
#9 0x7ff804a729ca in __37-[NSSQLiteConnection performAndWait:]_block_invoke+0x1b (CoreData:x86_64+0x1b59ca)
#10 0x1123d2f5a in _dispatch_client_callout+0x7 (libdispatch.dylib:x86_64+0x4f5a)
#11 0x1123e4d71 in _dispatch_lane_barrier_sync_invoke_and_complete+0x83 (libdispatch.dylib:x86_64+0x16d71)
#12 0x7ff804a723d2 in -[NSSQLiteConnection performAndWait:]+0x8e (CoreData:x86_64+0x1b53d2)
#13 0x7ff8049deb13 in -[NSSQLDefaultConnectionManager handleStoreRequest:]+0xe3 (CoreData:x86_64+0x121b13)
#14 0x7ff804b68065 in -[NSSQLCoreDispatchManager routeStoreRequest:]+0x10e (CoreData:x86_64+0x2ab065)
#15 0x7ff804a32fd4 in -[NSSQLCore dispatchRequest:withRetries:]+0x84 (CoreData:x86_64+0x175fd4)
#16 0x7ff804a33ec3 in -[NSSQLCore newValuesForObjectWithID:withContext:error:]+0x137 (CoreData:x86_64+0x176ec3)
#17 0x7ff8049f8858 in __95-[NSPersistentStoreCoordinator(_NSInternalMethods) newValuesForObjectWithID:withContext:error:]_block_invoke+0x5b (CoreData:x86_64+0x13b858)
#18 0x7ff8049e64e6 in -[NSPersistentStoreCoordinator _routeLightweightBlock:toStore:]+0xeb (CoreData:x86_64+0x1294e6)
#19 0x7ff8049f85bd in -[NSPersistentStoreCoordinator(_NSInternalMethods) newValuesForObjectWithID:withContext:error:]+0x15c (CoreData:x86_64+0x13b5bd)
#20 0x7ff80496466f in _PFFaultHandlerLookupRow+0x204 (CoreData:x86_64+0xa766f)
#21 0x7ff804966c69 in _PF_FulfillDeferredFault+0xd6 (CoreData:x86_64+0xa9c69)
#22 0x7ff8049864a6 in _sharedIMPL_pvfk_core+0x8a (CoreData:x86_64+0xc94a6)
#23 0x7ff804976326 in _PF_Handler_Public_GetProperty+0xed (CoreData:x86_64+0xb9326)
#24 0x7ff800b9e985 in -[NSFunctionExpression expressionValueWithObject:context:]+0x6c2 (Foundation:x86_64+0x4a0985)
#25 0x7ff800b58fda in -[NSComparisonPredicate evaluateWithObject:substitutionVariables:]+0x146 (Foundation:x86_64+0x45afda)
#26 0x7ff800b5c0c0 in -[NSCompoundPredicateOperator evaluatePredicates:withObject:substitutionVariables:]+0x116 (Foundation:x86_64+0x45e0c0)
#27 0x7ff800b5ba14 in -[NSCompoundPredicate evaluateWithObject:substitutionVariables:]+0x109 (Foundation:x86_64+0x45da14)
#28 0x7ff804a18812 in -[NSFetchedResultsController _preprocessUpdatedObjects:insertsInfo:deletesInfo:updatesInfo:sectionsWithDeletes:newSectionNames:treatAsRefreshes:]+0x1f0 (CoreData:x86_64+0x15b812)
#29 0x7ff804a19c48 in __82-[NSFetchedResultsController(PrivateMethods) _core_managedObjectContextDidChange:]_block_invoke+0x7d7 (CoreData:x86_64+0x15cc48)
#30 0x7ff8049a0112 in developerSubmittedBlockToNSManagedObjectContextPerform+0x96 (CoreData:x86_64+0xe3112)
#31 0x7ff80499fffc in -[NSManagedObjectContext performBlockAndWait:]+0xc4 (CoreData:x86_64+0xe2ffc)
#32 0x7ff804a1945e in -[NSFetchedResultsController _core_managedObjectContextDidChange:]+0x6d (CoreData:x86_64+0x15c45e)
#33 0x7ff80035964a in __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__+0x88 (CoreFoundation:x86_64+0x5664a)
#34 0x7ff800359580 in ___CFXRegistrationPost_block_invoke+0x55 (CoreFoundation:x86_64+0x56580)
#35 0x7ff80AddressSanitizer report breakpoint hit. Use 'thread info -s' to get extended information about the report.
(lldb) thread info -s
thread #1: tid = 0x1361ef, 0x000000010c751210 libclang_rt.asan_iossim_dynamic.dylib`__asan::AsanDie(), queue = 'SQLQueue 0x613000128340 for MyAppModel.sqlite', stop reason = Heap buffer overflow
{
"access_size": 15360,
"access_type": 0,
"address": 108301900430591,
"description": "heap-buffer-overflow",
"instrumentation_class": "AddressSanitizer",
"pc": 4503750845,
"stop_type": "fatal_error"
}
Can anyone help me make sense of this?
Why are none of the addresses from the gdb backtrace output present in the otx (otool) assembly output of this program? I was under the impression that they corresponded to the offsets in my otool asm output... I apologize if this is a silly question.
(gdb) bt
#0 0x9a4e1aa2 in __semwait_signal ()
#1 0x9a4e175e in _pthread_cond_wait ()
#2 0x9a4e12b1 in pthread_cond_timedwait$UNIX2003 ()
#3 0x0d8f3f87 in unregister_ShockwaveFlash ()
#4 0x0d45cf19 in dyld_stub_write$UNIX2003 ()
#5 0x0d6f9d7e in dyld_stub_write$UNIX2003 ()
#6 0x0d72db5a in dyld_stub_write$UNIX2003 ()
#7 0x0d72e24c in dyld_stub_write$UNIX2003 ()
#8 0x0d8eb5a2 in unregister_ShockwaveFlash ()
#9 0x0d95b9c9 in unregister_ShockwaveFlash ()
#10 0x9277df60 in CAOpenGLLayerDraw ()
#11 0x9277e897 in -[CAOpenGLLayer _display] ()
#12 0x92503ef1 in CALayerDisplayIfNeeded ()
#13 0x925032bc in CA::Context::commit_transaction ()
#14 0x92502f04 in CA::Transaction::commit ()
#15 0x958a1dd2 in __CFRunLoopDoObservers ()
#16 0x9585d3ea in CFRunLoopRunSpecific ()
#17 0x9585d1f1 in CFRunLoopRunInMode ()
#18 0x95530e04 in RunCurrentEventLoopInMode ()
#19 0x95530bb9 in ReceiveNextEventCommon ()
#20 0x956b9084 in _AcquireNextEvent ()
#21 0x956aed40 in RunApplicationEventLoop ()
#22 0x100083e2 in fkDecode ()
#23 0x10026167 in fkCall ()
#24 0x100081f6 in fkDecode ()
#25 0x10026167 in fkCall ()
#26 0x100081f6 in fkDecode ()
#27 0x10003be5 in fkRunMain ()
#28 0x00001e84 in main ()</code>
The followings may help you :
Variables defined in Global and Local Scope may have different address due to Address Space Layout Randomization (ASLR)
Heap allocated variables may have different address due to ASLR, if your program is multi-threaded
On Linux GDB is by default disables ASLR. Try it on Mac disabling ASLR.
I have a recurring intermittent EXC_BAD_ACCESS crash documented here.
This question: What steps can I take to make sure that this isn't a framework/lib error, and is actually a fault with my code? (Aside from the obvious, yes, duh, it's my code.)
I'm struggling with Instruments and getting a stack trace; are there resources I should be using to learn about this aspect of programming?
EDIT: I think this is a stack trace:
#0 0x0000cad8 in std::string ofToString<float>(float const&) at /Developer/of_007_iphone/libs/openFrameworks/utils/ofUtils.h:79
#1 0x000064ac in testApp::draw() ()
#2 0x0036d78c in ofAppiPhoneWindow::timerLoop() ()
#3 0x0037e698 in -[ofxiPhoneAppDelegate timerLoop] ()
#4 0x3095cbde in __NSFireTimer ()
#5 0x3579eca0 in __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ ()
#6 0x3579e6ac in __CFRunLoopDoTimer ()
#7 0x3576e300 in __CFRunLoopRun ()
#8 0x3576dd7a in CFRunLoopRunSpecific ()
#9 0x3576dc88 in CFRunLoopRunInMode ()
#10 0x336ace8c in GSEventRunModal ()
#11 0x318f0f94 in -[UIApplication _run] ()
#12 0x318ee4d4 in UIApplicationMain ()
#13 0x0036e9c4 in ofAppiPhoneWindow::runAppViaInfiniteLoop(ofBaseApp*) ()
#14 0x003a6804 in ofRunApp(ofBaseApp*) ()
#15 0x00002b34 in main ()
Ok, and another one. Wasn't even sure this was a separate error before:
#0 0x00019244 in std::vector<std::complex<float>, std::allocator<std::complex<float> > >::capacity() const at /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS4.3.sdk/usr/include/c++/4.2.1/bits/stl_vector.h:434
#1 0x00026608 in std::vector<std::complex<float>, std::allocator<std::complex<float> > >::operator=(std::vector<std::complex<float>, std::allocator<std::complex<float> > > const&) at /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS4.3.sdk/usr/include/c++/4.2.1/bits/vector.tcc:137
#2 0x00018708 in Analyzer::calcFFT() at /Developer/of_007_iphone/apps/cwi007/iTicTacToe/src/gameplay/pitch.cc:86
#3 0x0001881c in Analyzer::process() at /Developer/of_007_iphone/apps/cwi007/iTicTacToe/src/gameplay/pitch.cc:197
#4 0x00004378 in testApp::audioIn(float*, int, int) at /Developer/of_007_iphone/apps/cwi007/iTicTacToe/src/testApp.mm:362
#5 0x004a3fa0 in recordingCallback(void*, unsigned long*, AudioTimeStamp const*, unsigned long, unsigned long, AudioBufferList*) at /Developer/of_007_iphone/libs/openFrameworks/sound/ofxiPhoneSoundStream.mm:143
#6 0x361ccae0 in AUIOHelper::NotifyInputAvailable(AudioTimeStamp const&, unsigned long, AudioBufferList const&) ()
#7 0x361b9b90 in AURemoteIO::PerformIO(unsigned int, unsigned int, XAudioTimeStamp const&, XAudioTimeStamp const&, int&) ()
#8 0x361b9cfc in AURIOCallbackReceiver_PerformIO ()
#9 0x361b0fcc in _XPerformIO ()
#10 0x360dccbc in mshMIGPerform ()
#11 0x36173850 in MSHMIGDispatchMessage ()
#12 0x361c0b5c in AURemoteIO::IOThread::Entry(void*) ()
#13 0x3609ebb4 in CAPThread::Entry(CAPThread*) ()
#14 0x33c14684 in _pthread_start ()
The first step is to look at the stack trace in the Debugger Navigator or the crash log. Find the crashed thread and look at its stack. If any of your own code is in there, chances are it was your fault.
It won't stand up in court. For one thing, you might have done everything right and tripped a bug in framework or library code, and the moment where you tripped their bug is the moment caught in the stack trace (since that's where the crash happened). It's rare, but it does happen.
The other thing is that if you don't see your own code in the stack trace of the crashed thread, that doesn't prove you innocent. Your code might be guilty on another thread (usually indicating a thread-safety violation: you did something thread-unsafe, either knowingly on another thread without knowing it was unsafe or unknowingly on the wrong thread). Or you might be guilty in the past, having set up a crash to occur (e.g., by over-releasing an object) that occurred later (by messaging the dead object).
No matter what, the only way to determine whose bug it is is to investigate. Find where the crash happened, find what happened, and find why it happened. Once you know those three things, you'll know who did it, and whether you have to fix it (your bug) or report it and work around it (someone else's).
I have a release version server process running under linux 64-bit systems. It got crashed and left a coredump file. I use gdb to debug it like this:
gdb svr coredump file
And got the following backtraces:
(gdb) where
#0 0x0000000000432691 in ?? ()
#1 0x00002b07655a50cc in ?? ()
#2 0x00002b07655a50c4 in ?? ()
#3 0x00007fff9fade920 in ?? ()
#4 0x0000000000439db3 in ?? ()
#5 0x00007fff9fade910 in ?? ()
#6 0x00007fff9fade910 in ?? ()
#7 0x00007fff9fade8e0 in ?? ()
#8 0x00000000004663e2 in ?? ()
#9 0x00007fff9fade910 in ?? ()
#10 0x00007fff9fade910 in ?? ()
#11 0x00007fff9fade930 in ?? ()
#12 0x0000000000605108 in ?? ()
#13 0x00002b07655a274c in ?? ()
#14 0x0000000000ebc678 in ?? ()
#15 0x169f49f100000001 in ?? ()
#16 0x00000000021dc6c0 in ?? ()
#17 0x00002b07655a284c in ?? ()
#18 0x00002b07655a27dc in ?? ()
#19 0x00007fff9fade960 in ?? ()
#20 0x000000000043a03b in ?? ()
#21 0x00007fff9fade960 in ?? ()
#22 0x0000000000994d02 in ?? ()
#23 0x00000000000ecd57 in ?? ()
#24 0x00002b07655a274c in ?? ()
#25 0x00002b07655a274c in ?? ()
#26 0x00002b07655a27dc in ?? ()
#27 0x00007fff9fade980 in ?? ()
#28 0x000000000060a5eb in ?? ()
#29 0x000000009fadeb50 in ?? ()
#30 0x00002b07655a274c in ?? ()
#31 0x00007fff9fade9d0 in ?? ()
#32 0x000000000060a8f0 in ?? ()
#33 0x00007fff9fadeb50 in ?? ()
#34 0x00007fff9fadea10 in ?? ()
#35 0x00002b07655a274c in ?? ()
#36 0x00007fff9fadea10 in ?? ()
#37 0x000000009fade9d0 in ?? ()
#38 0x00007fff9fadeb58 in ?? ()
#39 0x0000000000000000 in ?? ()
The address info can not be analyzed by addr2line, what is the problem and how can I find what is the root cause of coredump?
Thanks!
Are you running GDB on the machine on which the core dump was produced?
For GDB to correctly reconstruct the crash stack trace, it must have access to exact binaries which were used at the time of the crash (or you get garbage).
The easiest way to achieve this is to analyze the core on the machine on which it was produced. If that's not feasible, you must copy all shared libraries to e.g. /tmp/solib-on-server/ (preserve the path, so e.g. /lib/libc.so.6 goes into /tmp/solib-on-server/lib/libc.so.6), then use GDB set solib-absolute-prefix /tmp/solib-on-server command before loading the core. E.g.
gdb -ex 'set solib-absolute-prefix /tmp/solib-on-server' \
-ex 'core /path/to/core' /path/to/executable
Its very hard to debug programs without debug symbols. As you are using release version of the application the core dump will not contain any debug information.
I am not sure, but if you can correlate the stack trace with the ".map" file of your application you could be getting somewhere. If I was in your position I would have recompiled the app with debug symbols(-g compiler flag) and then proceed on debugging.
You can use below commands in gdb to set shared library path.
set solib-search-path [Directories]
[Directories] : paths separated by colon(Ex /usr/lib:/usr/lib64)
show solib-search-path
These two commands helped me to get some info.
I'm debugging a Cocoa app, which calls the system libraries. I get debug symbols for my own code, but not the system libraries, which stops me debugging any further. How can I get debug symbols for the system library.
I'm using gdb, compiling against the 10.5 SDK on 10.6.4 (gcc 4.2).
Here is the stack-trace I'd like to get symbols for:
(gdb) bt
#0 0x93713e43 in CFQSortArray ()
#1 0x936f4c49 in CFArraySortValues ()
#2 0x958f0bc0 in ColorSyncProfileCopyTagSignatures ()
#3 0x9591d218 in CMMProfile::Usable ()
#4 0x9591cb5d in DoValidateProfile ()
#5 0x9591cc75 in AppleCMMValidateProfile ()
#6 0x958f1e52 in ColorSyncProfileVerify ()
#7 0x91b13b88 in validateProfile ()
#8 0x91b13aed in CMSValidateProfile ()
#9 0x93b27f6c in CGCMSInterfaceValidateProfile ()
#10 0x93b27f4c in CGCMSUtilsValidateProfile ()
#11 0x93b27e4b in CGColorSpaceCreateICCBased ()
#12 0x93b27203 in create_generic_color_space ()
#13 0x93b26f92 in CGColorSpaceCreateWithIndex ()
#14 0x90228ad5 in +[NSColorSpace specialColorSpaceWithID:] ()
#15 0x90228a0c in +[NSColorSpace genericRGBColorSpace] ()
#16 0x9022881e in -[NSBitmapImageRep _bitmapImageRep_setColorSpaceName:] ()
#17 0x904f6e39 in -[NSBitmapImageRep initWithBitmapDataPlanes:pixelsWide:pixelsHigh:bitsPerSample:samplesPerPixel:hasAlpha:isPlanar:colorSpaceName:bitmapFormat:bytesPerRow:bitsPerPixel:] ()
#18 0x902282ae in -[NSBitmapImageRep _initWithSharedBitmap:rect:] ()
#19 0x90227efe in -[NSImage _addRepresentationWithSharedKitWindow:rect:] ()
#20 0x90226d06 in +[NSImage _findSystemImageNamed:] ()
#21 0x90226a48 in +[NSImage imageNamed:] ()
#22 0x902268ea in -[NSCustomResource _loadImageWithName:] ()
#23 0x9022681e in -[NSCustomResource loadImageWithName:] ()
#24 0x90225d5f in -[NSCustomResource awakeAfterUsingCoder:] ()
#25 0x915a2208 in _decodeObjectBinary ()
#26 0x915a14e4 in _decodeObject ()
#27 0x9022556f in -[NSMenuItem initWithCoder:] ()
#28 0x915a21ec in _decodeObjectBinary ()
#29 0x915a2ff8 in -[NSKeyedUnarchiver _decodeArrayOfObjectsForKey:] ()
#30 0x915a3665 in -[NSArray(NSArray) initWithCoder:] ()
#31 0x915a21ec in _decodeObjectBinary ()
#32 0x915a14e4 in _decodeObject ()
#33 0x90229ff0 in -[NSMenu initWithCoder:] ()
#34 0x915a21ec in _decodeObjectBinary ()
#35 0x915a2ff8 in -[NSKeyedUnarchiver _decodeArrayOfObjectsForKey:] ()
#36 0x915a3665 in -[NSArray(NSArray) initWithCoder:] ()
#37 0x915a21ec in _decodeObjectBinary ()
#38 0x915a14e4 in _decodeObject ()
#39 0x9021ff29 in -[NSIBObjectData initWithCoder:] ()
#40 0x915a21ec in _decodeObjectBinary ()
#41 0x915a14e4 in _decodeObject ()
#42 0x9021f4a8 in loadNib ()
#43 0x9021eb5b in +[NSBundle(NSNibLoading) _loadNibFile:nameTable:withZone:ownerBundle:] ()
#44 0x9021e811 in +[NSBundle(NSNibLoading) loadNibFile:externalNameTable:withZone:] ()
As described in this apple TechNote, you want to set DYLD_IMAGE_SUFFIX to _debug.
http://developer.apple.com/library/mac/#technotes/tn2004/tn2124.html
there are further problems then (not having the right version on the right system) which are hard to overcome, so I haven't tested this.
I doubt you'd find anything but link symbol names in system libraries. Certainly no line number info, and no source code (it's a closed-source commercial product). The best you can do is step through the assembler instructions in these functions.
You can use gdb finish command to run until selected stack frame returns. Do that several times to return to your application code. Or just next your way over library function calls.