I've been fighting with this for a few days now. Whenever I run a watchKit App from the simulator my mac will get the black screen of death and reboot my machine. I am running XCode 8.2.1 running simulators for iPhone 6 or 7 and watches with watchOS 3 for both the original watch and version 2 in both sizes. The watch simulator will start, but whenever it attempts to launch a watchkit app that was deployed with XCode my machine will crash. This can be any watchkit app, even a simple Hello World app.
I have also tried completely uninstalling XCode 8 and 7 as well as all of the simulators and derived info, but the watch apps will still crash my machine. I've filed a bug report with Apple, but they won't reply back to me. I am also running on a Mid 2014 MacBook Pro with Sierra version 10.12.3.
Has anyone else experienced this? I'm running out of ideas to fix this and I cannot do anything watch related until I get past this. I've listed some of the logs from the activity monitor below as well as the Panic report:
ACTIVITY LOG:
Mar 24 09:04:17 C02N618RG3QD logd[74]: _purge_uuidtext unlinking /var/db/uuidtext/FD/2940F9BA8137988BB56CC86EFEDC84
Mar 24 09:04:17 C02N618RG3QD logd[74]: _purge_uuidtext unlinking /var/db/uuidtext/FE/872728C5713EA69D9D3FBC6C653C19
Mar 24 09:04:17 C02N618RG3QD logd[74]: _purge_uuidtext unlinking /var/db/uuidtext/FE/B838E71BA739F4AA6F121E88E788B5
Mar 24 09:04:17 C02N618RG3QD logd[74]: _purge_uuidtext unlinking /var/db/uuidtext/FF/0B49D70BCF30419AD992D76F372F19
Mar 24 09:04:17 C02N618RG3QD logd[74]: _purge_uuidtext total: 3188, in_use:2294, marked:0, recent:254, deleted 640
Mar 24 09:04:17 C02N618RG3QD logd[74]: _purge_uuidtext slib tree cleaned up (0)
Mar 24 09:04:17 C02N618RG3QD logd[74]: _purge_uuidtext uuid tree cleaned up (3)
Mar 24 09:04:17 C02N618RG3QD logd[74]: _purge_uuidtext cleaned up (0)
Mar 24 09:04:17 C02N618RG3QD logd[74]: Purged 114692767 bytes from uuidtext.
Mar 24 09:04:18 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe03260b130>: 9223372036854847488.000000]
Mar 24 09:04:18 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe034800ba0>: 9223372036854855680.000000]
Mar 24 09:04:18 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe03260ac80>: 9223372036854855680.000000]
Mar 24 09:04:18 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe0349017f0>: 9223372036854855680.000000]
Mar 24 09:04:18 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe034a07340>: 9223372036854855680.000000]
Mar 24 09:04:18 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe0324107b0>: 9223372036854855680.000000]
Mar 24 09:04:18 C02N618RG3QD useractivityd[1022]: BUG in libdispatch client: kevent[EVFILT_MACHPORT] monitored resource vanished before the source cancel handler was invoked
Mar 24 09:04:19 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe034b0e0c0>: 9223372036854855680.000000]
Mar 24 09:04:19 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe0326088d0>: 9223372036854855680.000000]
Mar 24 09:04:19 C02N618RG3QD UserEventAgent[1178]: assertion failed: 16D32 14S471a: com.apple.cts + 20609 [D0046D83-09B7-3C10-A783-396F29B9B380]: 0x0
Mar 24 09:04:21 C02N618RG3QD com.apple.xpc.launchd[1] (com.mcafee.ssm.ScanManager): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.
Mar 24 09:04:22 C02N618RG3QD com.apple.xpc.launchd[1] (com.apple.nowplayingtouchui): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.
Mar 24 09:04:22 C02N618RG3QD com.apple.xpc.launchd[1] (com.apple.quicklook[30456]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.quicklook
Mar 24 09:04:32 C02N618RG3QD com.apple.xpc.launchd[1] (com.mcafee.ssm.ScanManager): Service only ran for 1 seconds. Pushing respawn out by 9 seconds.
Mar 24 09:04:33 C02N618RG3QD com.apple.xpc.launchd[1] (com.apple.nowplayingtouchui): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.
Mar 24 09:04:39 C02N618RG3QD IBCocoaTouchImageCatalogTool[30517]: assertion failed: 16D32: libxpc.dylib + 34617 [D3538D2B-EEA9-3C74-890E-6F4AE4F51F0E]: 0x8d
Mar 24 09:04:40 C02N618RG3QD IBCocoaTouchImageCatalogTool[30517]: BUG in libdispatch client: kevent[EVFILT_VNODE] add: "Bad file descriptor" - 0x9
Mar 24 09:04:42 C02N618RG3QD Interface Builder WatchKit Tool[30530]: assertion failed: 16D32: libxpc.dylib + 34617 [D3538D2B-EEA9-3C74-890E-6F4AE4F51F0E]: 0x8d
Mar 24 09:04:42 --- last message repeated 2 times ---
Mar 24 09:04:42 C02N618RG3QD Interface Builder WatchKit Tool[30530]: BUG in libdispatch client: kevent[EVFILT_VNODE] add: "Bad file descriptor" - 0x9
Mar 24 09:04:42 C02N618RG3QD com.apple.xpc.launchd[1] (com.mcafee.ssm.ScanManager): Service only ran for 1 seconds. Pushing respawn out by 9 seconds.
Mar 24 09:04:43 C02N618RG3QD com.apple.xpc.launchd[1] (com.apple.nowplayingtouchui): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.
Mar 24 09:04:47 C02N618RG3QD Interface Builder Cocoa Touch Tool[30670]: assertion failed: 16D32 14C89: libxpc.dylib + 31485 [0339A402-6531-30E9-8BC2-589C463CD6D6]: 0x8d
Mar 24 09:04:47 C02N618RG3QD Interface Builder Cocoa Touch Tool[30671]: assertion failed: 16D32 14C89: libxpc.dylib + 31485 [0339A402-6531-30E9-8BC2-589C463CD6D6]: 0x8d
Mar 24 09:04:47 --- last message repeated 1 time ---
Mar 24 09:04:47 C02N618RG3QD Interface Builder Cocoa Touch Tool[30670]: assertion failed: 16D32 14C89: libxpc.dylib + 31485 [0339A402-6531-30E9-8BC2-589C463CD6D6]: 0x8d
Mar 24 09:04:47 C02N618RG3QD Interface Builder Cocoa Touch Tool[30671]: BUG in libdispatch: 16D32 14C89 - 1296 - 0x2
Mar 24 09:04:47 C02N618RG3QD Interface Builder Cocoa Touch Tool[30670]: BUG in libdispatch: 16D32 14C89 - 1296 - 0x2
Mar 24 09:04:47 C02N618RG3QD Interface Builder Cocoa Touch Tool[30671]: BUG in libdispatch client: kevent[EVFILT_VNODE] add: "Bad file descriptor" - 0x9
Mar 24 09:04:47 C02N618RG3QD Interface Builder Cocoa Touch Tool[30670]: BUG in libdispatch client: kevent[EVFILT_VNODE] add: "Bad file descriptor" - 0x9
Mar 24 09:04:47 C02N618RG3QD Interface Builder Cocoa Touch Tool[30671]: assertion failed: 16D32 14C89: libxpc.dylib + 31485 [0339A402-6531-30E9-8BC2-589C463CD6D6]: 0x8d
Mar 24 09:04:47 C02N618RG3QD Interface Builder Cocoa Touch Tool[30670]: assertion failed: 16D32 14C89: libxpc.dylib + 31485 [0339A402-6531-30E9-8BC2-589C463CD6D6]: 0x8d
Mar 24 09:04:48 C02N618RG3QD IBCocoaTouchImageCatalogTool[30687]: BUG in libdispatch: 16D32 14C89 - 1296 - 0x2
Mar 24 09:04:48 C02N618RG3QD IBCocoaTouchImageCatalogTool[30687]: BUG in libdispatch client: kevent[EVFILT_VNODE] add: "Bad file descriptor" - 0x9
Mar 24 09:04:52 C02N618RG3QD com.apple.xpc.launchd[1] (com.mcafee.ssm.ScanManager): Service only ran for 1 seconds. Pushing respawn out by 9 seconds.
Mar 24 09:04:53 C02N618RG3QD com.apple.xpc.launchd[1] (com.apple.nowplayingtouchui): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.
Mar 24 09:04:58 C02N618RG3QD Simulator (Watch)[30853]: BUG in libdispatch client: kevent[EVFILT_MACHPORT] monitored resource vanished before the source cancel handler was invoked
Mar 24 09:05:00 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe032706fc0>: 9223372036854857728.000000]
Mar 24 09:05:00 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe0327081f0>: 9223372036854857728.000000]
Mar 24 09:05:00 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe0326114b0>: 9223372036854857728.000000]
Mar 24 09:05:00 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe03250fbf0>: 9223372036854857728.000000]
Mar 24 09:05:00 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe03250d9d0>: 9223372036854857728.000000]
AND
Mar 24 09:05:53 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe034801f40>: 9223372036854861824.000000]
Mar 24 09:05:53 C02N618RG3QD diagnosticd[380]: JSON->NS failed [type" : "u",
"timestamp" : 58798649826108
},
"timeGMTsec" : 1490360753,
"procid" : {
"procid" : 30972,
"_type" : "u"
},
"persisted" : {
"_type" : "b",
"persisted" : 0
},
"thread" : {
"_type" : "u",
"thread" : 529028
},
"pid" : {
"pid" : 30972,
"_type" : "u"
},
"procpath" : "\/Applications\/Xcode.app\/Contents\/Developer\/Platforms\/iPhoneSimulator.platform\/Developer\/SDKs\/iPhoneSimulator.sdk\/System\/Library\/CoreServices\/SpringBoard.app\/SpringBoard",
"SIMULATOR_UDID" : "4855B6A5-7A82-49AE-94E4-DBA2BED65585"
}]
Mar 24 09:05:53 C02N618RG3QD diagnosticd[380]: Unexpected xpc type [<double: 0x7fe032403a30>: 9223372036854861824.000000]
PANIC REPORT
Anonymous UUID: 56E742E7-D1F7-AB9B-90CC-58BFFAFB991E
Fri Mar 24 09:07:31 2017
*** Panic Report ***
panic(cpu 4 caller 0xffffff8018e05ecd): Kernel trap at 0xffffff8018dfc620, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0xffffff806783403c, CR3: 0x00000002cb9b6256, CR4: 0x00000000001627e0
RAX: 0x00000000ffffff01, RBX: 0x000000002d1e202c, RCX: 0x00000000ffffffa1, RDX: 0x000000000000000a
RSP: 0xffffff921b9fb7c0, RBP: 0xffffff921b9fb7c0, RSI: 0xffffff7f9c364723, RDI: 0xffffff806783403c
R8: 0x0000000000000002, R9: 0x0000000000000ff8, R10: 0xe326417ed3850092, R11: 0xffffff804d3b5bb8
R12: 0x000000000000000a, R13: 0x0000000000000054, R14: 0x0000000000906065, R15: 0xffffff803a652010
RFL: 0x0000000000010206, RIP: 0xffffff8018dfc620, CS: 0x0000000000000008, SS: 0x0000000000000010
Fault CR2: 0xffffff806783403c, Error code: 0x0000000000000000, Fault CPU: 0x4, PL: 0, VF: 1
Backtrace (CPU 4), Frame : Return Address
0xffffff921b9fb450 : 0xffffff8018cf210c
0xffffff921b9fb4d0 : 0xffffff8018e05ecd
0xffffff921b9fb6b0 : 0xffffff8018ca3743
0xffffff921b9fb6d0 : 0xffffff8018dfc620
0xffffff921b9fb7c0 : 0xffffff7f9c33e471
0xffffff921b9fb800 : 0xffffff7f9c33e4fa
0xffffff921b9fb830 : 0xffffff7f9c33e719
0xffffff921b9fb910 : 0xffffff7f9c33e839
0xffffff921b9fb950 : 0xffffff7f9c3604fa
0xffffff921b9fb9a0 : 0xffffff7f9c3603b4
0xffffff921b9fb9c0 : 0xffffff7f9c3474b7
0xffffff921b9fbe60 : 0xffffff7f9c3479e1
0xffffff921b9fbf40 : 0xffffff8202aea6a0
0xffffff921b9fbf50 : 0xffffff8019227066
0xffffff921b9fbfb0 : 0xffffff8018ca3c38
Kernel Extensions in backtrace:
com.verdasys.dgagent(23.20.7)[90F4996A-8100-3509-BBCA-9778AAA30E30]#0xffffff7f9c338000->0xffffff7f9c374fff
dependency: com.apple.security.sandbox(300.0)[8997F7FD-7C04-38F6-86DB-BACDE9137374]#0xffffff7f99937000
BSD process name corresponding to current thread: xpcproxy_sim
Mac OS version:
16D32
Kernel version:
Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64
Kernel UUID: C67A8D03-DEAC-35B8-8F68-06FF7B687215
Kernel slide: 0x0000000018a00000
Kernel text base: 0xffffff8018c00000
__HIB text base: 0xffffff8018b00000
System model name: MacBookPro11,3 (Mac-2BD1B31983FE1663)
System uptime in nanoseconds: 2958365287234
last loaded kext at 359084323236: com.apple.filesystems.smbfs 3.1.1 (addr 0xffffff7f998b3000, size 425984)
loaded kexts:
com.McAfee.FMPSysCore 1
com.verdasys.dgagent 0023.20.7
com.intel.kext.intelhaxm 1.1.1
com.squirrels.airparrot.framebuffer 5
com.squirrels.driver.AirParrotSpeakers 1.8
com.apple.filesystems.smbfs 3.1.1
com.apple.driver.AppleHWSensor 1.9.5d0
com.apple.filesystems.autofs 3.0
com.apple.driver.AGPM 110.23.14
com.apple.driver.ApplePlatformEnabler 2.7.0d0
com.apple.driver.X86PlatformShim 1.0.0
com.apple.driver.pmtelemetry 1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.iokit.IOBluetoothSerialManager 5.0.3f1
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AppleHV 1
com.apple.driver.AppleOSXWatchdog 1
com.apple.driver.AppleGraphicsDevicePolicy 3.13.79
com.apple.driver.AppleHDAHardwareConfigDriver 278.56
com.apple.driver.AppleHDA 278.56
com.apple.driver.AppleUpstreamUserClient 3.6.4
com.apple.GeForce 10.1.4
com.apple.driver.AppleIntelHD5000Graphics 10.2.2
com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0
com.apple.driver.AppleFIVRDriver 4.1.0
com.apple.driver.AppleBacklight 170.9.10
com.apple.driver.AppleLPC 3.1
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 5.0.3f1
com.apple.nvidia.NVDAStartup 10.1.4
com.apple.driver.AudioAUUC 1.70
com.apple.driver.AppleSMCLMU 208
com.apple.driver.AppleCameraInterface 5.57.0
com.apple.driver.AppleMuxControl 3.13.79
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.driver.AppleIntelFramebufferAzul 10.2.2
com.apple.driver.AppleThunderboltIP 3.0.8
com.apple.driver.AppleMCCSControl 1.2.15
com.apple.driver.AppleUSBDisplays 380
com.apple.iokit.SCSITaskUserClient 394.30.2
com.apple.driver.AppleUSBStorageCoexistentDriver 404.30.1
com.apple.driver.AppleUSBCardReader 404.30.1
com.apple.iokit.IOBluetoothUSBDFU 5.0.3f1
com.apple.driver.AppleUSBTCKeyEventDriver 252
com.apple.driver.AppleUSBTCKeyboard 252
com.apple.driver.AppleUSBTCButtons 252
com.apple.driver.CoreStorageFsck 540.30.1
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.BootCache 40
com.apple.filesystems.hfs.kext 366.30.3
com.apple.iokit.IOAHCIBlockStorage 295.20.1
com.apple.driver.AppleAHCIPort 326
com.apple.iokit.AppleBCM5701Ethernet 10.2.7
com.apple.driver.AppleFWOHCI 5.5.5
com.apple.driver.AirPort.Brcm4360 1130.3.1a5
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.driver.AppleACPIButtons 5.0
com.apple.driver.AppleRTC 2.0
com.apple.driver.AppleHPET 1.8
com.apple.driver.AppleSMBIOS 2.1
com.apple.driver.AppleACPIEC 5.0
com.apple.driver.AppleAPIC 1.7
com.apple.nke.applicationfirewall 172
com.apple.security.quarantine 3
com.apple.security.TMSafetyNet 8
com.apple.kext.triggers 1.0
com.apple.iokit.IOSerialFamily 11
com.apple.driver.AppleSSE 1.0
com.apple.driver.DspFuncLib 278.56
com.apple.kext.OSvKernDSPLib 525
com.apple.nvidia.driver.NVDAGK100Hal 10.1.4
com.apple.nvidia.driver.NVDAResman 10.1.4
com.apple.iokit.IOSurface 153.3
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.iokit.IOFireWireIP 2.2.7
com.apple.driver.X86PlatformPlugin 1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d8
com.apple.iokit.IOBluetoothHostControllerUSBTransport 5.0.3f1
com.apple.iokit.IOBluetoothHostControllerTransport 5.0.3f1
com.apple.iokit.IOBluetoothFamily 5.0.3f1
com.apple.driver.AppleHDAController 278.56
com.apple.iokit.IOHDAFamily 278.56
com.apple.driver.AppleSMBusPCI 1.0.14d1
com.apple.driver.AppleBacklightExpert 1.1.0
com.apple.iokit.IONDRVSupport 513.1
com.apple.driver.AppleGraphicsControl 3.13.79
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.AppleGraphicsDeviceControl 3.13.79
com.apple.iokit.IOAcceleratorFamily2 289.32
com.apple.driver.AppleThunderboltEDMSink 4.1.1
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.AppleSMBusController 1.0.14d1
com.apple.iokit.IOGraphicsFamily 513.1
com.apple.driver.AppleUSBComposite 900.4.1
com.apple.driver.AppleUSBAudio 307.7
com.apple.iokit.IOAudioFamily 205.12
com.apple.vecLib.kext 1.2.0
com.apple.iokit.IOUSBMassStorageClass 4.0.4
com.apple.iokit.IOSCSIBlockCommandsDevice 394.30.2
com.apple.iokit.IOUSBMassStorageDriver 131.1.1
com.apple.iokit.IOSCSIArchitectureModelFamily 394.30.2
com.apple.driver.AppleThunderboltDPInAdapter 4.6.1
com.apple.driver.AppleThunderboltDPOutAdapter 4.6.1
com.apple.driver.AppleThunderboltDPAdapterFamily 4.6.1
com.apple.driver.AppleThunderboltPCIUpAdapter 2.1.1
com.apple.driver.AppleThunderboltPCIDownAdapter 2.1.1
com.apple.driver.AppleUSBMultitouch 258
com.apple.driver.usb.IOUSBHostHIDDevice 1.1
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.usb.networking 5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.1
com.apple.driver.CoreStorage 540.30.1
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.AppleUSBMergeNub 900.4.1
com.apple.driver.usb.AppleUSBHub 1.1
com.apple.driver.AppleXsanScheme 3
com.apple.iokit.IOAHCIFamily 288
com.apple.driver.usb.AppleUSBEHCIPCI 1.1
com.apple.driver.usb.AppleUSBEHCI 1.1
com.apple.iokit.IOEthernetAVBController 1.0.3b4
com.apple.iokit.IOFireWireFamily 4.6.5
com.apple.driver.AppleThunderboltNHI 4.1.8
com.apple.iokit.IOThunderboltFamily 6.3.3
com.apple.iokit.IO80211Family 1200.12.2
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.2
com.apple.driver.corecapture 1.0.4
com.apple.driver.AppleIntelLpssGspi 3.0.60
com.apple.driver.usb.AppleUSBXHCIPCI 1.1
com.apple.driver.usb.AppleUSBXHCI 1.1
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.iokit.IOUSBFamily 900.4.1
com.apple.driver.AppleUSBHostMergeProperties 1.1
com.apple.driver.AppleEFINVRAM 2.1
com.apple.driver.AppleEFIRuntime 2.0
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.IOSMBusFamily 1.1
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleKeyStore 2
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.1
com.apple.driver.AppleBusPowerController 1.0
com.apple.driver.DiskImages 444.41.1
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOReportFamily 31
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleACPIPlatform 5.0
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.Libm 1
com.apple.kec.pthread 1
com.apple.kec.corecrypto 1.0
You have Verdasys Digital Guardian Agent and McAfee installed, both of which have injected kernel extensions that may be causing the panic.
It appears to be happening during an XPC call's security checks. The Simulator is a separate user space sharing the host Mac kernel. My guess is the developers of one or both kernel extensions are not properly handling certain data or are mangling private kernel data structures. $rip is the address of the next instruction. Page faulting on that address means something has gone seriously wrong.
I suggest uninstalling all 3rd-party kernel extensions then attempting to reproduce the problem. If that fixes it then file a bug with Verdasys or McAfee to have them fix their kernel extensions.
Related
I am using a (bash) alias on an Ubuntu 22.04 system to quickly tail the system log without all the garbage that usually is of no importance. (I do want all this logged, but not displayed all the time.) The current alias looks something like this:
$ tail -f --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: (REFUSED unexpected RCODE|timed out) resolving)|\[INFO\] (New connection from ::1|Logout.)"
This works fine and give me some output:
Dec 7 13:32:15 root named[1044]: clients-per-query decreased to 20
Dec 7 13:32:29 root mariadbd[1188]: 2022-12-07 13:32:29 83187 [Warning] Aborted connection 83187 to db: 'XXX' user: 'XXX' host: 'localhost' (Got an error reading communication packets)
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.168.192.bl.blocklist.de/A/IN': 46.252.24.212#53
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.215.85.bl.blocklist.de/A/IN': 46.252.24.212#53
Dec 7 13:39:00 root systemd[1]: Starting Clean php session files...
Dec 7 13:39:03 root systemd[1]: phpsessionclean.service: Deactivated successfully.
Dec 7 13:39:03 root systemd[1]: Finished Clean php session files.
Dec 7 13:39:03 root systemd[1]: phpsessionclean.service: Consumed 3.081s CPU time.
Dec 7 13:52:15 root named[1044]: clients-per-query decreased to 19
Dec 7 13:53:06 root named[1044]: connection refused resolving '_.54.142.bl.blocklist.de/A/IN': 46.252.24.212#53
Dec 7 13:55:58 root mariadbd[1188]: 2022-12-07 13:55:58 83536 [Warning] Access denied for user 'xxx'#'localhost' (using password: YES)
Dec 7 13:55:58 root mariadbd[1188]: 2022-12-07 13:55:58 83537 [Warning] Access denied for user ''#'localhost' (using password: NO)
Dec 7 13:59:01 root kernel: [241023.546785] sh (1569030): drop_caches: 3
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> Received signal: wake up
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> ClamAV update process started at Wed Dec 7 14:04:19 2022
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> ^Your ClamAV installation is OUTDATED!
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> ^Local version: 0.103.6 Recommended version: 0.103.7
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> DON'T PANIC! Read https://docs.clamav.net/manual/Installing.html
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> daily.cld database is up-to-date (version: 26743, sigs: 2013590, f-level: 90, builder: raynman)
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
Dec 7 14:04:52 root named[1044]: validating _spf.eu.mailgun.org/TXT: no valid signature found
Dec 7 14:09:00 root systemd[1]: Starting Clean php session files...
Dec 7 14:09:03 root systemd[1]: phpsessionclean.service: Deactivated successfully.
Dec 7 14:09:03 root systemd[1]: Finished Clean php session files.
Dec 7 14:09:03 root systemd[1]: phpsessionclean.service: Consumed 3.038s CPU time.
Dec 7 14:09:38 root clamd[489]: Wed Dec 7 14:09:38 2022 -> SelfCheck: Database status OK.
However, when I pipe this output through ccze, I get no output at all:
$ tail -f --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: (REFUSED unexpected RCODE|timed out) resolving)|\[INFO\] (New connection from ::1|Logout.)" | \
ccze -A
It gets weirder: after removing the RCODE line from my exclusion list it works:
$ tail -f --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: timed out resolving)|\[INFO\] (New connection from ::1|Logout.)" | \
ccze -A
Dec 7 13:32:15 root named[1044]: clients-per-query decreased to 20
Dec 7 13:32:29 root mariadbd[1188]: 2022-12-07 13:32:29 83187 [Warning] Aborted connection 83187 to db: 'xxx' user: 'xxx' host: 'localhost' (Got an error reading communication packets)
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.168.192.bl.blocklist.de/A/IN': 46.252.24.212#53
(...)
And if I first cat the syslog to a file and then pipe it through ccze, it also works:
$ tail --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: (REFUSED unexpected RCODE|timed out) resolving)|\[INFO\] (New connection from ::1|Logout.)" \
> syslog-ccze.bug
$ cat syslog-ccze.bug | ccze -A
Dec 7 13:32:15 root named[1044]: clients-per-query decreased to 20
Dec 7 13:32:29 root mariadbd[1188]: 2022-12-07 13:32:29 83187 [Warning] Aborted connection 83187 to db: 'xxx' user: 'xxx' host: 'localhost' (Got an error reading communication packets)
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.168.192.bl.blocklist.de/A/IN': 46.252.24.212#53
(...)
For some reason, ccze doesn't like me filtering out the "unexpected RCODE" named messages. Am I missing something obvious, or am I hitting some weird race condition in ccze, or what is happening?
Bah. Once you take the time to actually write down your question, you'll find answers by yourself. :-(
ccze apparently has an internal buffer. It needs a minimum amount of lines before outputting ANYTHING. Increasing the --lines= parameter for tail in my alias helped.
This seems to be undocumented behaviour, so I'll leave the question here to be searchable. Hope it helps and saves you some time. :-)
I am using yocto to build my image, am trying to integrate RAUC(system update solution). RAUC now is installed in my image but it's not working correctly.
rauc status gives :
(rauc:350): rauc-WARNING **: 07:50:14.624: Failed getting primary slot: Failed getting primary slot: fw_printenv failed with exit code: 234
=== System Info ===
Compatible: tx6q-1036
Variant:
Booted from: (null) (/dev/mmcblk3p2)
=== Bootloader ===
Activated: (null) ((null))
=== Slot States ===
o [rootfs.1] (/dev/mmcblk0p3, ext4, inactive)
bootname: B
boot status: bad
o [rootfs.0] (/dev/mmcblk0p2, ext4, inactive)
bootname: A
boot status: bad
systemctl status rauc gives :
rauc.service - Rauc Update Service
Loaded: loaded (/lib/systemd/system/rauc.service; static)
Active: active (running) since Tue 2021-03-23 07:49:41 UTC; 8min ago
Docs: https://rauc.readthedocs.io
Main PID: 292 (rauc)
Memory: 1.4M
CGroup: /system.slice/rauc.service
`-292 /usr/bin/rauc --mount=/run/rauc service
Mar 23 07:49:41 AMR83FA51 rauc[292]: mounting slot /dev/mmcblk0p2
Mar 23 07:49:41 AMR83FA51 rauc[292]: Failed to mount slot /dev/mmcblk0p2: Slot >
Mar 23 07:49:41 AMR83FA51 rauc[292]: Failed to resolve realpath for '/dev/mmcbl>
Mar 23 07:49:41 AMR83FA51 rauc[292]: Failed to resolve realpath for '/dev/mmcbl>
Mar 23 07:49:41 AMR83FA51 rauc[292]: rauc mark: Failed to determine slot states>
Mar 23 07:50:14 AMR83FA51 rauc[353]: Configuration file wrong or corrupted
Mar 23 07:50:14 AMR83FA51 rauc[292]: Failed getting primary slot: Failed gettin>
Mar 23 07:50:14 AMR83FA51 rauc[292]: Failed to resolve realpath for '/dev/mmcbl>
Mar 23 07:50:14 AMR83FA51 rauc[292]: Failed to resolve realpath for '/dev/mmcbl>
Mar 23 07:50:14 AMR83FA51 rauc[355]: Configuration file wrong or corrupted
The /etc/fw_env.config content is : /boot/uboot.env 0x0000 0x20000
I'm a beginner with u_boot and I don't know much about u-boot environment, could someone help to better understand this step and how to fix it.
A mac user was having some clock errors, and thought they had seen someone using remote/VNC action on their screen. I went through the system.log and most of this activity is showing at times when the laptop was off and unplugged (no battery) and the user was asleep.
System.log file here- https://ghostbin.com/paste/mcukf
These were the lines that interested me.
Java connection causing clock to be off.
23:54:32 Ushas-Air Java Updater[531]: Original euid:501
Apr 24 23:54:32 Ushas-Air com.apple.xpc.launchd[1] (com.apple.preference.datetime.remoteservice[366]): Service exited due to signal: Killed: 9 sent by com.apple.preference.datetime.re[366]
Apr 24 23:54:32 Ushas-Air Java Updater[531]: Host name is javadl-esd-secure.oracle.com
Apr 24 23:54:32 Ushas-Air Java Updater[531]: Feed URL: https
Apr 24 23:54:32 Ushas-Air Java Updater[531]: Hostname check passed. Valid Oracle hostname
Apr 24 23:54:33 Ushas-Air com.apple.xpc.launchd[1] (com.apple.bsd.dirhelper[523]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.bsd.dirhelper
Apr 24 23:54:36 Ushas-Air java[541]: objc[541]: Class JavaLaunchHelper is implemented in both /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java (0x1023604c0) and /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/jli/./libjli.dylib (0x119327480). One of the two will be used. Which one is undefined.
Instances of IMRemoteURLConnection Agent happening
Apr 25 00:14:11 Ushas-MacBook-Air com.apple.xpc.launchd[1] (com.apple.imfoundation.IMRemoteURLConnectionAgent): Unknown key for integer: _DirtyJetsamMemoryLimit
Apr 25 00:01:22 Ushas-MacBook-Air com.apple.xpc.launchd[1] (com.apple.imfoundation.IMRemoteURLConnectionAgent): Unknown key for integer: _DirtyJetsamMemoryLimit
Apr 25 00:05:57 Ushas-MacBook-Air com.apple.xpc.launchd[1] (com.apple.preferences.users.remoteservice[762]): Service exited due to signal: Killed: 9 sent by com.apple.preferences.users.remo[762]
Multiple cache deletes requested after.
Apr 25 00:01:27 Ushas-MacBook-Air logd[57]: _handle_cache_delete_with_urgency(0x7fdf19412a60, 3, 0)
Apr 25 00:01:27 Ushas-MacBook-Air logd[57]: _handle_cache_delete_with_urgency(0x7fdf19412a60, 3, 0)
Apr 25 00:01:31 Ushas-MacBook-Air com.apple.preferences.icloud.remoteservice[700]: BUG in libdispatch client: kevent[EVFILT_MACHPORT] monitored resource vanished before the source cancel handler was invoked
Apr 25 00:01:33 Ushas-MacBook-Air logd[57]: _handle_cache_delete_with_urgency(0x7fdf19658620, 3, 0)
Apr 25 00:01:33 Ushas-MacBook-Air logd[57]: _volume_contains_cached_data(is /private/var/db/diagnostics/ in /) - YES
Apr 25 00:01:34 Ushas-MacBook-Air logd[57]: 239517600 bytes of purgeable space from log files
Apr 25 00:01:34 Ushas-MacBook-Air logd[57]: _purge_uuidtext only runs at urgency 0 (3)
Apr 25 00:01:34 Ushas-MacBook-Air logd[57]: 0 bytes of purgeable space from uuidtext files
And appears to be launching the FamilyCircleFramework
Apr 24 23:56:11 Ushas-Air com.apple.xpc.launchd[1] (com.apple.imfoundation.IMRemoteURLConnectionAgent): Unknown key for integer: _DirtyJetsamMemoryLimit
Apr 24 23:56:16 --- last message repeated 1 time ---
Apr 24 23:56:16 Ushas-Air familycircled[615]: objc[615]: Class FAFamilyCloudKitProperties is implemented in both /System/Library/PrivateFrameworks/FamilyCircle.framework/Versions/A/FamilyCircle (0x7fffbe466a60) and /System/Library/PrivateFrameworks/FamilyCircle.framework/Versions/A/Resources/familycircled (0x10aa01178). One of the two will be used. Which one is undefined.
Apr 24 23:56:16 Ushas-Air familycircled[615]: objc[615]: Class FAFamilyMember is implemented in both /System/Library/PrivateFrameworks/FamilyCircle.framework/Versions/A/FamilyCircle (0x7fffbe466880) and /System/Library/PrivateFrameworks/FamilyCircle.framework/Versions/A/Resources/familycircled (0x10aa01268). One of the two will be used. Which one is undefined.
Apr 24 23:56:16 Ushas-Air familycircled[615]: objc[615]: Class FAFamilyCircle is implemented in both /System/Library/PrivateFrameworks/FamilyCircle.framework/Versions/A/FamilyCircle (0x7fffbe466a10) and /System/Library/PrivateFrameworks/FamilyCircle.framework/Versions/A/Resources/familycircled (0x10aa01358). One of the two will be used. Which one is undefined.
Activity related to Findmyfriends happening. The mac owner doesn't use FindMyFriends, or have a mac phone.
Apr 25 00:30:00 Ushas-MacBook-Air syslogd[40]: Configuration Notice:
ASL Module "com.apple.mobileme.fmf1.internal" sharing output destination "/var/log/FindMyFriendsApp/FindMyFriendsApp.asl" with ASL Module "com.apple.mobileme.fmf1".
Output parameters from ASL Module "com.apple.mobileme.fmf1" override any specified in ASL Module "com.apple.mobileme.fmf1.internal".
Apr 25 00:30:00 Ushas-MacBook-Air syslogd[40]: Configuration Notice:
ASL Module "com.apple.mobileme.fmf1.internal" sharing output destination "/var/log/FindMyFriendsApp" with ASL Module "com.apple.mobileme.fmf1".
Output parameters from ASL Module "com.apple.mobileme.fmf1" override any specified in ASL Module "com.apple.mobileme.fmf1.internal".
Apr 25 00:30:00 Ushas-MacBook-Air syslogd[40]: Configuration Notice:
The keybaglogd being shared with com.apple.mkb
Apr 25 00:30:00 Ushas-MacBook-Air syslogd[40]: Configuration Notice:
ASL Module "com.apple.mkb.internal" sharing output destination "/private/var/log/keybagd.log" with ASL Module "com.apple.mkb".
iOS apps are not starting in iPhone simulator(iPhone 5 - iOS 10.0 used ).
There are errors from logs
CoreSimulator.log:
mlaunch[1482] <Error>: Error Domain=FBSOpenApplicationServiceErrorDomain Code=1 "The request to open "com.xamarin.acquaint-native" failed." UserInfo={NSLocalizedDescription=The request to open "com.xamarin.acquaint-native" failed., BSErrorCodeDescription=RequestDenied, NSUnderlyingError=0x7fe15ceb7540 {Error Domain=FBSOpenApplicationErrorDomain Code=1 "(null)" UserInfo={BSErrorCodeDescription=Unspecified}}, NSLocalizedFailureReason=The request was denied by service delegate (SBMainWorkspace) for reason: Unspecified.}
System.log(Debug mode from Xamarin Studio or VS 2015):
Nov 10 00:00:01 macmini CoreSimulatorBridge[5136]: Requesting launch of com.companyname.FasterNotesMobile.iOS with options: {
arguments = (
"-monodevelop-port",
51746
);
environment = {
"DYLD_INSERT_LIBRARIES" = "/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/lib/monotouch-fixes.dylib";
};
}
Nov 10 00:00:01 macmini CoreSimulatorBridge[5136]: [Common] [FBSSystemService][0xe350] Sending request to open "com.companyname.FasterNotesMobile.iOS"
Nov 10 00:00:01 macmini com.apple.CoreSimulator.SimDevice.3D2EDEEE-4261-4B99-9B31-971D08B0124E.launchd_sim[5110] (UIKitApplication:com.companyname.FasterNotesMobile.iOS[0xe451][5510]): Program specified by service does not contain one of the requested architectures:
Nov 10 00:00:01 macmini SpringBoard[5127]: [Common] Unable to get pid for 'UIKitApplication:com.companyname.FasterNotesMobile.iOS[0xe451]': No such process (err 3)
Nov 10 00:00:01 macmini SpringBoard[5127]: [Common] Bootstrapping failed for <FBApplicationProcess: 0x7a79b830; com.companyname.FasterNotesMobile.iOS; pid: -1>
Nov 10 00:00:01 macmini CoreSimulatorBridge[5136]: [Common] [FBSSystemService][0xe350] Error handling open request for com.companyname.FasterNotesMobile.iOS: <NSError: 0x7b871a30; domain: FBSOpenApplicationServiceErrorDomain; code: 1 (RequestDenied); reason: "The request was denied by service delegate (SBMainWorkspace) for reason: Unspecified."> {
description = "The request to open "com.companyname.FasterNotesMobile.iOS" failed.";
failureReason = "The request was denied by service delegate (SBMainWorkspace) for reason: Unspecified.";
underlyingError = <NSError: 0x7b871c10; domain: FBSOpenApplicationErrorDomain; code: 1 (Unspecified)> {
description = "The operation couldn’t be completed. (FBSOpenApplicationErrorDomain error 1.)";
};
}
Nov 10 00:00:01 macmini CoreSimulatorBridge[5136]: Error Launching: Error Domain=FBSOpenApplicationServiceErrorDomain Code=1 "The request to open "com.companyname.FasterNotesMobile.iOS" failed." UserInfo={NSLocalizedFailureReason=The request was denied by service delegate (SBMainWorkspace) for reason: Unspecified., NSLocalizedDescription=The request to open "com.companyname.FasterNotesMobile.iOS" failed., BSErrorCodeDescription=RequestDenied, NSUnderlyingError=0x7b871c10 {Error Domain=FBSOpenApplicationErrorDomain Code=1 "(null)" UserInfo={BSErrorCodeDescription=Unspecified}}}
Nov 10 00:00:01 macmini com.apple.CoreSimulator.SimDevice.3D2EDEEE-4261-4B99-9B31-971D08B0124E.launchd_sim[5110] (UIKitApplication:com.companyname.FasterNotesMobile.iOS[0xea1e][5511]): Program specified by service does not contain one of the requested architectures:
Nov 10 00:00:01 macmini SpringBoard[5127]: [Common] Unable to get pid for 'UIKitApplication:com.companyname.FasterNotesMobile.iOS[0xea1e]': No such process (err 3)
Nov 10 00:00:01 macmini SpringBoard[5127]: [Common] Bootstrapping failed for <FBApplicationProcess: 0x80a24f00; com.companyname.FasterNotesMobile.iOS; pid: -1>
Nov 10 00:00:05 macmini calaccessd[5164]: 25534103: Beginning CalScheduledTaskCacheTrimAndExtendAndUpdateOverdue
Nov 10 00:00:05 macmini calaccessd[5164]: 25534103: ScheduledTaskCache last saved 499237200
Nov 10 00:00:05 macmini calaccessd[5164]: 25534103: today is EKCalendarDate <0x7c17e740> { date = Thursday, November 10, 2016 at 12:00:00 AM CST }, comparisonResult is -1
Nov 10 00:00:05 macmini calaccessd[5164]: 25534103: Updating overdue with today 500450400.000000
The log above are for acquaint pre build application. I tried several other application, the same story.
It used to work fine.
I am using latest Xamarin iOS SDK update(that maybe the issue)
Got same problem with a new project and the latest Xamarin.
To fix it I had to change "Supported architectures" to "i386" in the project options. The default was "x86_64".
Do you have the latest version of Xcode?
You won't be able to simulate on IOs 10 unless you have Xcode 8.2 (or 8.1, not sure) installed on your Mac station.
Apparently there is a known problem of XFS locking up the kernel/processes and corrupting volumes under heavy traffic.
Some web pages talk about it, but I was not able to figure out which pages are new and may have a solution.
My company's deployments have Debian with kernel 3.4.107, xfsprogs 3.1.4, and large storage arrays.
We have large data (PB) and high throughput (GB/sec) using async IO to several large volumes.
We constantly experience these unpredictable lockups on several systems.
Kernel logs/dmesg show something like the following:
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986515] INFO: task Sr2dReceiver-5:46829 blocked for more than 120 seconds.
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986518] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986520] Sr2dReceiver-5 D ffffffff8105b39e 0 46829 7284 0x00000000
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986524] ffff881e71f57b38 0000000000000082 000000000000000b ffff884066763180
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986529] 0000000000000000 ffff884066763180 0000000000011180 0000000000011180
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986532] ffff881e71f57fd8 ffff881e71f56000 0000000000011180 ffff881e71f56000
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986536] Call Trace:
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986545] [<ffffffff814ffe9f>] schedule+0x64/0x66
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986548] [<ffffffff815005f3>] rwsem_down_failed_common+0xdb/0x10d
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986551] [<ffffffff81500638>] rwsem_down_write_failed+0x13/0x15
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986555] [<ffffffff8126b583>] call_rwsem_down_write_failed+0x13/0x20
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986558] [<ffffffff814ff320>] ? down_write+0x25/0x27
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986572] [<ffffffffa01f29e0>] xfs_ilock+0xbc/0x12e [xfs]
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986580] [<ffffffffa01eec71>] xfs_rw_ilock+0x2c/0x33 [xfs]
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986586] [<ffffffffa01eec71>] ? xfs_rw_ilock+0x2c/0x33 [xfs]
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986593] [<ffffffffa01ef234>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986600] [<ffffffffa01ef358>] xfs_file_buffered_aio_write+0x67/0x179 [xfs]
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986603] [<ffffffff8150099a>] ? _raw_spin_unlock_irqrestore+0x30/0x3d
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986611] [<ffffffffa01ef81d>] xfs_file_aio_write+0x163/0x1b5 [xfs]
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986614] [<ffffffff8106f1af>] ? futex_wait+0x22c/0x244
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986619] [<ffffffff8110038e>] do_sync_write+0xd9/0x116
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986622] [<ffffffff8150095f>] ? _raw_spin_unlock+0x26/0x31
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986634] [<ffffffff8106f2f1>] ? futex_wake+0xe8/0xfa
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986637] [<ffffffff81100d1d>] vfs_write+0xae/0x10a
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986639] [<ffffffff811015b3>] ? fget_light+0xb0/0xbf
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986642] [<ffffffff81100dd3>] sys_pwrite64+0x5a/0x79
2016 Mar 24 04:42:34 hmtmzhbgb01-ssu-1 kernel: [2358750.986645] [<ffffffff81506912>] system_call_fastpath+0x16/0x1b
Lockups leave the system in a bad state. The processes in D state that hang cannot even be killed with signal 9.
The only way to resume operations is to reboot, repair XFS and then the system works for another while.
But occasionally after the lockup we cannot even repair some volumes, as they get totally corrupted and we need to rebuild them with mkfs.
As a last resort, we now run xfs-repair periodically and this reduced the frequency of lockups and data loss to a certain extent.
But the incidents still occur often enough, so we need some solution.
I was wondering if there is a solution for this with kernel 3.4.107, e.g. some patch that we may apply.
Due to the large number of deployments and other software issues, we cannot upgrade the kernel in the near future.
However, we are working towards updating our applications so that we can run on kernel 3.16 in our next releases.
Does anyone know if this XFS lockup problem was fixed in 3.16?
Some people have experienced this but it was not a problem with XFS it was because the kernel was unable to flush dirty pages within the 120s time period. Have a look here but please check the numbers they're using as default on your own system.
http://blog.ronnyegner-consulting.de/2011/10/13/info-task-blocked-for-more-than-120-seconds/
and here
http://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/
You can see what you're dirty cache ratio is by running this
sysctl -a | grep dirty
or
cat /proc/sys/vm/dirty_ratio
The best write up on this I could find is here...
https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/
Essentially you need to tune your application to make sure that it can write the dirty buffers to disk within the time period or change the timer period etc.
You can also see some interesting paramaters as follows
sysctl -a | grep hung
You could increase the timeout permanently using /etc/sysctl.conf as follows...
kernel.hung_task_timeout_secs = 300
Does anyone know if this XFS lockup problem was fixed in 3.16?
It is said so in A Short Guide to Kernel Debugging:
Searching for “xfs splice deadlock” turns up an email thread from 2011 that describes this
problem. However, bisecting the kernel source repository shows that
the bug wasn’t really addressed until April, 2014 (8d02076) for release in Linux 3.16.