NoHostAvailableException while bulk loading data into Cassandra - cassandra-2.0
I'm trying to load 1 Million LUBM dataset into Cassandra. But, I keep getting this Exception after about 200,000 inserts.
Exception in thread "main" com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1 (com.datastax.driver.core.exceptions.DriverException: Timeout during read))
The server is still alive. I can still execute CQL queries using Datastax Dev Center. Prior to this, I was getting WriteTimeoutException and hence I tweaked some options in .yaml
Here are the specs of my system:
CPU- Intel I5 - 2 cores|
Memory- 4GB |
Hard Disk- 500 GB 4 Drives.
I have an OWL file from which the data is read and is inserted into Cassandra. I have mixed workloads. Sometimes, based upon a condition, I need to retrieve the data from the row and append the new data to it and write it to the same row. This mustn't be the cause for this problem because this condition must have occurred plenty of times during the 200k inserts.
I'm using an external hard disk for storing commitlogs.
concurrent_reads is set to 16.
concurrent_writes is set to 32.
read_timeout has been set to 10000 and so as write_timeout.
MAX_HEAP_SIZE="2G"
HEAP_NEWSIZE="200M"
As soon as I get hat exception, I see this log with this pattern
INFO [MemoryMeter:1] 2014-03-25 16:28:56,498 Memtable.java (line 449) CFS(Keyspace='rdf', ColumnFamily='lubm1m.lubm1m_teacherof_idx') liveRatio is 15.733535462912831 (just-counted was 15.733535462912831). calculation took 91ms for 665 cells
INFO [MemoryMeter:1] 2014-03-25 16:30:01,560 Memtable.java (line 449) CFS(Keyspace='system', ColumnFamily='sstable_activity') liveRatio is 15.896188158961882 (just-counted was 15.896188158961882). calculation took 37ms for 411 cells
INFO [ScheduledTasks:1] 2014-03-25 16:33:22,252 GCInspector.java (line 116) GC for ParNew: 15581 ms for 1 collections, 395151608 used; max is 1038876672
INFO [ScheduledTasks:1] 2014-03-25 16:33:24,894 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2014-03-25 16:33:26,956 StatusLogger.java (line 70) ReadStage 0 0 343094 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:26,957 StatusLogger.java (line 70) RequestResponseStage 0 0 1 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:26,996 StatusLogger.java (line 70) ReadRepairStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,004 StatusLogger.java (line 70) MutationStage 0 0 601747 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,024 StatusLogger.java (line 70) ReplicateOnWriteStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,031 StatusLogger.java (line 70) GossipStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,066 StatusLogger.java (line 70) AntiEntropyStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,067 StatusLogger.java (line 70) MigrationStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,098 StatusLogger.java (line 70) MemtablePostFlusher 0 0 52 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,099 StatusLogger.java (line 70) MemoryMeter 0 0 139 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,100 StatusLogger.java (line 70) FlushWriter 0 0 31 0 16
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,101 StatusLogger.java (line 70) MiscStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,130 StatusLogger.java (line 70) PendingRangeCalculator 0 0 1 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,133 StatusLogger.java (line 70) commitlog_archiver 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,134 StatusLogger.java (line 70) InternalResponseStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,142 StatusLogger.java (line 70) HintedHandoff 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,223 StatusLogger.java (line 79) CompactionManager 0 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,232 StatusLogger.java (line 81) Commitlog n/a 0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,262 StatusLogger.java (line 93) MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,262 StatusLogger.java (line 103) Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,287 StatusLogger.java (line 105) KeyCache 17212 51380224 all
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,335 StatusLogger.java (line 111) RowCache 0 0 all
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,336 StatusLogger.java (line 118) ColumnFamily Memtable ops,data
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,336 StatusLogger.java (line 121) system.schema_triggers 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,337 StatusLogger.java (line 121) system.local 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,337 StatusLogger.java (line 121) system.peers 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,338 StatusLogger.java (line 121) system.batchlog 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,338 StatusLogger.java (line 121) system.NodeIdInfo 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,339 StatusLogger.java (line 121) system.compactions_in_progress 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,339 StatusLogger.java (line 121) system.schema_keyspaces 4,1130
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,340 StatusLogger.java (line 121) system.compaction_history 38,11220
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,340 StatusLogger.java (line 121) system.paxos 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,340 StatusLogger.java (line 121) system.schema_columns 3361,891818
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,341 StatusLogger.java (line 121) system.schema_columnfamilies 1426,394018
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,341 StatusLogger.java (line 121) system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,342 StatusLogger.java (line 121) system.range_xfers 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,344 StatusLogger.java (line 121) system.peer_events 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,344 StatusLogger.java (line 121) system.hints 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,345 StatusLogger.java (line 121) system.sstable_activity 1542,196000
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,345 StatusLogger.java (line 121) KeyspaceCumulus.POS_RN_NUM 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,346 StatusLogger.java (line 121) KeyspaceCumulus.DICT_SO_REVERSE 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,346 StatusLogger.java (line 121) KeyspaceCumulus.SPO_RN_DT 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,347 StatusLogger.java (line 121) KeyspaceCumulus.DICT_P_REVERSE 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,347 StatusLogger.java (line 121) KeyspaceCumulus.SPOC 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,348 StatusLogger.java (line 121) KeyspaceCumulus.POSC 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,348 StatusLogger.java (line 121) KeyspaceCumulus.counter 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,349 StatusLogger.java (line 121) KeyspaceCumulus.SCHEMA_O_PROPS 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,350 StatusLogger.java (line 121) KeyspaceCumulus.SCHEMA_D_PROPS 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,350 StatusLogger.java (line 121) KeyspaceCumulus.PREFIX_TO_NS 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,352 StatusLogger.java (line 121) KeyspaceCumulus.POS_RN_DT 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,352 StatusLogger.java (line 121) KeyspaceCumulus.SPO_RN_NUM 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,353 StatusLogger.java (line 121) KeyspaceCumulus.SCHEMA_CLASSES 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,353 StatusLogger.java (line 121) KeyspaceCumulus.OSPC 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,354 StatusLogger.java (line 121) KeyspaceCumulus.DICT_SO 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,354 StatusLogger.java (line 121) KeyspaceCumulus.DICT_P 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,355 StatusLogger.java (line 121) rdf.animal 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,355 StatusLogger.java (line 121) rdf.lubm20 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,356 StatusLogger.java (line 121) rdf.bsbm50 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,356 StatusLogger.java (line 121) rdf.tim 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,357 StatusLogger.java (line 121) rdf.bsbm250 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,357 StatusLogger.java (line 121) rdf.lubm1m 152700,22140112
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,358 StatusLogger.java (line 121) rdf.lubm3 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,358 StatusLogger.java (line 121) system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2014-03-25 16:33:27,359 StatusLogger.java (line 121) system_traces.events 0,0
What could be the problem?
increase socket timeout with SocketOptions. You may try to configure it on datastax client and server side as well. Also, see if reducing consistency could be any help?
-Vivek
Related
QtCreator doesn't open on Mac
Problem:
I need to install Qt Creator for my university course on my mac and it doesn't open. I've tried reinstalling different versions a lot of times but it gives me a Segmentation Fault everytime.
Everything else is updated to the latest Version and i don't know what to try anymore.
Any tips and solutions would be appreciated.
This is one of the error messages:
DebuggerItem \"/Applications/Xcode.app/Contents/Developer/usr/bin/lldb\" ({8999bbd1-bcb9-4fd1-843a-47fec36eb8b6}) read from \"/Users/Samy/.config/QtProject/qtcreator/debuggers.xml\" dropped since the command is not executable."
zsh: segmentation fault /Applications/Qt\ Creator.app/Contents/MacOS/Qt\ Creator
Here is the Report if I try opening the App:
Process: Qt Creator [54924]
Path: /Applications/Qt Creator.app/Contents/MacOS/Qt Creator
Identifier: org.qt-project.qtcreator
Version: 4.11.0 (4.11.0)
Code Type: X86-64 (Native)
Parent Process: ??? [1]
Responsible: Qt Creator [54924]
User ID: 501
Date/Time: 2020-01-08 22:12:42.699 +0100
OS Version: Mac OS X 10.15 (19A602)
Report Version: 12
Anonymous UUID: 103F9158-D5E0-F1B8-BA89-222AB7C6F587
Sleep/Wake UUID: 3FB53B09-90D6-4FB8-AB3A-2D663E75701E
Time Awake Since Boot: 170000 seconds
Time Since Wake: 15000 seconds
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Signal: Segmentation fault: 11
Termination Reason: Namespace SIGNAL, Code 0xb
Terminating Process: exc handler [54924]
VM Regions Near 0:
-->
__TEXT 000000010d082000-000000010d096000 [ 80K] r-x/rwx SM=COW /Applications/Qt Creator.app/Contents/MacOS/Qt Creator
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libUtils.4.11.0.dylib 0x000000010d1ab7b0 QtPrivate::QFunctorSlotObject<Utils::FancyMainWindowPrivate::FancyMainWindowPrivate(Utils::FancyMainWindow*)::$_2, 1, QtPrivate::List<bool>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) + 48
1 org.qt-project.QtCore 0x000000010e3369b5 0x10e10f000 + 2259381
2 org.qt-project.QtWidgets 0x000000010d3f040e QAction::setChecked(bool) + 174
3 libDebugger.dylib 0x0000000114a1a2ef Utils::DebuggerMainWindow::restorePersistentSettings() + 1151
4 libDebugger.dylib 0x0000000114a19c5d Utils::DebuggerMainWindow::DebuggerMainWindow() + 925
5 libDebugger.dylib 0x0000000114a218fb Utils::Perspective::Perspective(QString const&, QString const&, QString const&, QString const&) + 267
6 libDebugger.dylib 0x0000000114a24d25 Debugger::Internal::DebuggerPluginPrivate::DebuggerPluginPrivate(QStringList const&) + 1781
7 libDebugger.dylib 0x0000000114a36747 Debugger::Internal::DebuggerPlugin::initialize(QStringList const&, QString*) + 39
8 libExtensionSystem.4.11.0.dylib 0x000000010d0cbecb ExtensionSystem::Internal::PluginSpecPrivate::initializePlugin() + 107
9 libExtensionSystem.4.11.0.dylib 0x000000010d0be6a0 ExtensionSystem::Internal::PluginManagerPrivate::loadPlugin(ExtensionSystem::PluginSpec*, ExtensionSystem::PluginSpec::State) + 656
10 libExtensionSystem.4.11.0.dylib 0x000000010d0b6f90 ExtensionSystem::Internal::PluginManagerPrivate::loadPlugins() + 592
11 org.qt-project.qtcreator 0x000000010d08de28 main + 14520
12 libdyld.dylib 0x00007fff6e91b405 start + 1
Thread 1:
0 libsystem_pthread.dylib 0x00007fff6eb245b4 start_wqthread + 0
Thread 2:
0 libsystem_pthread.dylib 0x00007fff6eb245b4 start_wqthread + 0
Thread 3:
0 libsystem_pthread.dylib 0x00007fff6eb245b4 start_wqthread + 0
Thread 4:
0 libsystem_pthread.dylib 0x00007fff6eb245b4 start_wqthread + 0
Thread 5:
0 libsystem_pthread.dylib 0x00007fff6eb245b4 start_wqthread + 0
Thread 6:: com.apple.CFSocket.private
0 libsystem_kernel.dylib 0x00007fff6ea6b7c6 __select + 10
1 com.apple.CoreFoundation 0x00007fff3764a92a __CFSocketManager + 632
2 libsystem_pthread.dylib 0x00007fff6eb27d76 _pthread_start + 125
3 libsystem_pthread.dylib 0x00007fff6eb245d7 thread_start + 15
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x0000000000000000 rcx: 0x00007ffee2b7cf50 rdx: 0x0000600000d5a3c0
rdi: 0x0000000000000000 rsi: 0x000060000184bba0 rbp: 0x00007ffee2b7ce70 rsp: 0x00007ffee2b7ce60
r8: 0x0000000000000000 r9: 0x0000000000000000 r10: 0x000000011c7d7be0 r11: 0x000000010d5461d0
r12: 0x0000600003c9bea0 r13: 0x000000011c805d40 r14: 0x0000000000000001 r15: 0x0000600000d5a3c0
rip: 0x000000010d1ab7b0 rfl: 0x0000000000010246 cr2: 0x0000000000000000
Logical CPU: 2
Error Code: 0x00000004 (no mapping for user data write)
Trap Number: 14
[Number jibberish]`enter code here`
External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 114998
thread_create: 0
thread_set_state: 0
VM Region Summary:
ReadOnly portion of Libraries: Total=780.0M resident=0K(0%) swapped_out_or_unallocated=780.0M(100%)
Writable regions: Total=573.9M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=573.9M(100%)
VIRTUAL REGION
REGION TYPE SIZE COUNT (non-coalesced)
=========== ======= =======
Accelerate framework 128K 1
Activity Tracing 256K 1
CG backing stores 248K 2
CoreImage 8K 2
CoreUI image data 72K 1
JS VM register file 4360K 4
JS VM register file (reserved) 2040K 1 reserved VM address space (unallocated)
Kernel Alloc Once 8K 1
MALLOC 167.2M 43
MALLOC guard page 16K 4
MALLOC_NANO (reserved) 384.0M 1 reserved VM address space (unallocated)
STACK GUARD 56.0M 7
Stack 11.0M 7
VM_ALLOCATE 132K 9
WebAssembly memory 4096K 1
__DATA 39.4M 428
__DATA_CONST 41K 3
__FONT_DATA 4K 1
__GLSLBUILTINS 5176K 1
__LINKEDIT 382.9M 106
__OBJC_RO 31.8M 1
__OBJC_RW 1764K 2
__TEXT 397.2M 410
__UNICODE 564K 1
mapped file 332.3M 43
shared memory 640K 15
=========== ======= =======
TOTAL 1.8G 1096
TOTAL, minus reserved VM space 1.4G 1096
Model: MacBookAir7,2, BootROM 190.0.0.0.0, 2 processors, Dual-Core Intel Core i5, 1,6 GHz, 8 GB, SMC 2.27f2
Graphics: kHW_IntelHDGraphics6000Item, Intel HD Graphics 6000, spdisplays_builtin
Memory Module: BANK 0/DIMM0, 4 GB, DDR3, 1600 MHz, 0x02FE, -
Memory Module: BANK 1/DIMM0, 4 GB, DDR3, 1600 MHz, 0x02FE, -
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x117), Broadcom BCM43xx 1.0 (7.77.105.1 AirPortDriverBrcmNIC-1429)
Bluetooth: Version 7.0.0f8, 3 services, 18 devices, 1 incoming serial ports
Network Service: Wi-Fi, AirPort, en0
Serial ATA Device: APPLE SSD SM0128G, 121,33 GB
USB Device: USB 3.0 Bus
USB Device: BRCM20702 Hub
USB Device: Bluetooth USB Host Controller
Thunderbolt Bus: MacBook Air, Apple Inc., 27.2
Mac Os 10.13 logs me out randomly | Help reading logs
Apparently out of the blue, I have started to experience random crashes on my Mac Pro 4.1, flashed to 5.1. It's running High Sierra 10.13.6 (17G9016), with 48Gb of Ram and two cards (one GT120 and a flashed MacVidCard Titan X Pascal). It is a soft crash in the sense that it just logs me out of my session and then I need to log in again, so it doesn't completely reboot. However it's very annoying since I can't finish any task properly and need to reopen all the files I might be working with, which generally deal with heavy graphics (CAD applications or video editing). I am not fluent in reading crash logs, so any help reading this one is much appreciated. Here's the last crash log: Process: WindowServer [144] Path: /System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/Resources/WindowServer Identifier: WindowServer Version: 600.00 (312.103.11) Code Type: X86-64 (Native) Parent Process: launchd [1] Responsible: WindowServer [144] User ID: 88 Date/Time: 2019-11-17 18:15:12.313 +0000 OS Version: Mac OS X 10.13.6 (17G9016) Report Version: 12 Anonymous UUID: B4B1B7C1-08E2-6A09-6684-E949A6C2D95F Time Awake Since Boot: 230 seconds System Integrity Protection: enabled Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000016 Exception Note: EXC_CORPSE_NOTIFY Termination Signal: Segmentation fault: 11 Termination Reason: Namespace SIGNAL, Code 0xb Terminating Process: exc handler [0] VM Regions Near 0x16: --> __TEXT 0000000108ebe000-0000000108ebf000 [ 4K] r-x/rwx SM=COW [/System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/Resources/WindowServer] Application Specific Information: StartTime:2019-11-17 18:11:36 GPU:NV MetalDevice for accelerator(0x4507): 0x7fb86ae03ef8 (MTLDevice: 0x7fb86c004800) MetalDevice for accelerator(0x4e07): 0x7fb86d00d858 (MTLDevice: 0x7fb86c004800) IOService:/AppleACPIPlatformExpert/PCI0#0/AppleACPIPCI/IOU0#3/IOPP/PXS1#0/NVDA,Display-B#1/NVDA IOService:/AppleACPIPlatformExpert/PCI0#0/AppleACPIPCI/IOU1#7/IOPP/PXS2#0/NVDA,Display-A#0/NVDATesla Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 libGPUSupport.dylib 0x00007fff6606b10c gpumRestoreTextureData + 60 1 libGFXShared.dylib 0x00007fff56f366d9 gfxUploadPluginTextureLevel + 111 2 GLEngine 0x00007fff57bf55e7 gleSynchronizePluginTextureLevels + 279 3 GLEngine 0x00007fff57bbfaaf gleBindFramebuffer + 321 4 GLEngine 0x00007fff57b0855c glBindFramebuffer_Exec + 142 5 com.apple.SkyLight 0x00007fff6f32ec24 CaptureSurfaceGL::CreateAndAttachTextureToFBO() + 162 6 com.apple.SkyLight 0x00007fff6f32e3d8 CaptureSurfaceGL::PrepareForPopulation(CGRect, WSPixelFormat, float, bool, CGColorSpace*, CGXDisplayDevice*, unsigned int) + 74 7 com.apple.SkyLight 0x00007fff6f444601 WSCALayerBackingUpdateFlatteningIfNeeded + 1284 8 com.apple.SkyLight 0x00007fff6f3cd2b3 CGXNextSurface + 3246 9 com.apple.SkyLight 0x00007fff6f472211 generate_layers_for_window_surfaces(CGXRedrawState*, CGXWindow*, CGSOrderOp, unsigned int, int*, WSCompositeSourceLayer**, CGSRegionObject*) + 5581 10 com.apple.SkyLight 0x00007fff6f46cca6 generate_layers_for_window(CGXRedrawState*, CGXWindow*) + 3193 11 com.apple.SkyLight 0x00007fff6f46afc1 CGXUpdateDisplay + 13505 12 com.apple.SkyLight 0x00007fff6f467882 update_display_callback(void*, double) + 257 13 com.apple.SkyLight 0x00007fff6f4ae552 run_timer_pass + 495 14 com.apple.SkyLight 0x00007fff6f4dc1c9 CGXRunOneServicesPass + 247 15 com.apple.SkyLight 0x00007fff6f4dcd84 SLXServer + 832 16 WindowServer 0x0000000108ebedde 0x108ebe000 + 3550 17 libdyld.dylib 0x00007fff75559015 start + 1 Thread 1: 0 libsystem_kernel.dylib 0x00007fff756aa28a __workq_kernreturn + 10 1 libsystem_pthread.dylib 0x00007fff75871009 _pthread_wqthread + 1035 2 libsystem_pthread.dylib 0x00007fff75870be9 start_wqthread + 13 Thread 2: 0 libsystem_kernel.dylib 0x00007fff756a020a mach_msg_trap + 10 1 libsystem_kernel.dylib 0x00007fff7569f724 mach_msg + 60 2 com.apple.CoreDisplay 0x00007fff4d4cb685 0x7fff4d41c000 + 718469 3 com.apple.CoreDisplay 0x00007fff4d4cb813 0x7fff4d41c000 + 718867 4 libsystem_pthread.dylib 0x00007fff75871661 _pthread_body + 340 5 libsystem_pthread.dylib 0x00007fff7587150d _pthread_start + 377 6 libsystem_pthread.dylib 0x00007fff75870bf9 thread_start + 13 Thread 3:: com.apple.coreanimation.render-server 0 libsystem_kernel.dylib 0x00007fff756a020a mach_msg_trap + 10 1 libsystem_kernel.dylib 0x00007fff7569f724 mach_msg + 60 2 com.apple.QuartzCore 0x00007fff5890ae62 CA::Render::Server::server_thread(void*) + 870 3 com.apple.QuartzCore 0x00007fff5890aae6 thread_fun + 25 4 libsystem_pthread.dylib 0x00007fff75871661 _pthread_body + 340 5 libsystem_pthread.dylib 0x00007fff7587150d _pthread_start + 377 6 libsystem_pthread.dylib 0x00007fff75870bf9 thread_start + 13 Thread 4: 0 libsystem_kernel.dylib 0x00007fff756aa28a __workq_kernreturn + 10 1 libsystem_pthread.dylib 0x00007fff75871009 _pthread_wqthread + 1035 2 libsystem_pthread.dylib 0x00007fff75870be9 start_wqthread + 13 Thread 0 crashed with X86 Thread State (64-bit): rax: 0x00007fb871800000 rbx: 0x00007fb86c0e8800 rcx: 0x0000000000000000 rdx: 0x0000000000000000 rdi: 0x00007fb86f832a00 rsi: 0x00007fb86c0e8800 rbp: 0x00007ffee6d40380 rsp: 0x00007ffee6d40320 r8: 0x0000000000000000 r9: 0x000000016f236ee7 r10: 0x0000000000000003 r11: 0x0000000000000206 r12: 0x0000000000000000 r13: 0x0000000000000000 r14: 0x0000000000000000 r15: 0x00007fb86f832a00 rip: 0x00007fff6606b10c rfl: 0x0000000000010246 cr2: 0x0000000000000016 Logical CPU: 4 Error Code: 0x00000004 Trap Number: 14 Binary Images: […] External Modification Summary: Calls made by other processes targeting this process: task_for_pid: 3 thread_create: 0 thread_set_state: 0 Calls made by this process: task_for_pid: 0 thread_create: 0 thread_set_state: 0 Calls made by all processes on this machine: task_for_pid: 318 thread_create: 0 thread_set_state: 0 VM Region Summary: ReadOnly portion of Libraries: Total=380.3M resident=0K(0%) swapped_out_or_unallocated=380.3M(100%) Writable regions: Total=1.6G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=1.6G(100%) VIRTUAL REGION REGION TYPE SIZE COUNT (non-coalesced) =========== ======= ======= Activity Tracing 256K 2 CG backing stores 173.6M 139 CG framebuffers 1.0G 9 CG framebuffers (reserved) 56.1M 9 reserved VM address space (unallocated) CG image 668K 9 CoreAnimation 64.1M 53 CoreGraphics 4K 2 Dispatch continuations 32.0M 2 Kernel Alloc Once 8K 2 MALLOC 321.3M 146 MALLOC guard page 48K 13 MALLOC_LARGE (reserved) 160K 2 reserved VM address space (unallocated) OpenGL GLSL 256K 4 STACK GUARD 56.0M 6 Stack 10.0M 6 VM_ALLOCATE 1352K 50 __CGSERVER 4K 2 __DATA 51.1M 249 __FONT_DATA 4K 2 __GLSLBUILTINS 2588K 2 __LINKEDIT 194.5M 10 __SLSERVER 4K 2 __TEXT 185.8M 250 __UNICODE 560K 2 mapped file 25.8M 3 shared memory 180K 15 =========== ======= ======= TOTAL 2.1G 965 TOTAL, minus reserved VM space 2.1G 965
Why are no dts pts written to my mp4 container
Based on my (self-answered) question here Muxing AVPackets into mp4 file - revisited, I have to ask, what could be the reason why there are no values written for pts/dts in the resulting mp4 container. I examined the container file with the tool MediaInfo. I observe that only the very first Frame contains a value for pts in the container. After that, pts is not even shown in the mp4 file anymore, but dts is, with a value of all zeros. This is the output from MediaInfo for the first 3 frames: 0000A2 slice_layer_without_partitioning (IDR) - 0 (0x0) - Frame 0 - slice_type I - frame_num 0 - DTS 00:00:00.000 - PTS 00:00:00.017 (141867 bytes) 0000A2 Header (5 bytes) 0000A2 zero_byte: 0 (0x00) 0000A3 start_code_prefix_one_3bytes: 1 (0x000001) 0000A6 nal_ref_idc: 3 (0x3) - (2 bits) 0000A6 nal_unit_type: 5 (0x05) - (5 bits) 0000A7 slice_header (3 bytes) 0000A7 first_mb_in_slice: 0 (0x0) 0000A7 slice_type: 7 (0x07) - I 0000A8 pic_parameter_set_id: 0 (0x0) 0000A8 frame_num: 0 (0x0) 0000A8 idr_pic_id: 0 (0x0) 0000A8 no_output_of_prior_pics_flag: No 0000A8 long_term_reference_flag: No 0000A9 slice_qp_delta: -5 (0xFFFFFFFB) 0000AA disable_deblocking_filter_idc: 0 (0x0) 0000AA slice_alpha_c0_offset_div2: 0 (0x0) 0000AA slice_beta_offset_div2: 0 (0x0) 0000AA slice_data (141856 bytes) 0000AA (ToDo): (Data) 022ACD slice_layer_without_partitioning (IDR) - 0 (0x0) - Frame 0 - slice_type I - frame_num 0 - DTS 00:00:00.000 - PTS 00:00:00.017 - first_mb_in_slice 8040 (2248 bytes) 022ACD Header (5 bytes) 022ACD zero_byte: 0 (0x00) 022ACE start_code_prefix_one_3bytes: 1 (0x000001) 022AD1 nal_ref_idc: 3 (0x3) - (2 bits) 022AD1 nal_unit_type: 5 (0x05) - (5 bits) 022AD2 slice_header (6 bytes) 022AD2 first_mb_in_slice: 8040 (0x001F68) 022AD5 slice_type: 7 (0x07) - I 022AD6 pic_parameter_set_id: 0 (0x0) 022AD6 frame_num: 0 (0x0) 022AD6 idr_pic_id: 0 (0x0) 022AD6 no_output_of_prior_pics_flag: No 022AD6 long_term_reference_flag: No 022AD7 slice_qp_delta: -5 (0xFFFFFFFB) 022AD8 disable_deblocking_filter_idc: 0 (0x0) 022AD8 slice_alpha_c0_offset_div2: 0 (0x0) 022AD8 slice_beta_offset_div2: 0 (0x0) 022AD8 slice_data (2237 bytes) 022AD8 (ToDo): (Data) 023395 1 (36212 bytes) 023395 slice_layer_without_partitioning (non-IDR) - 2 (0x2) - Frame 1 - slice_type P - frame_num 1 - DTS 00:00:00.000 (36017 bytes) 023395 Header (5 bytes) 023395 zero_byte: 0 (0x00) 023396 start_code_prefix_one_3bytes: 1 (0x000001) 023399 nal_ref_idc: 3 (0x3) - (2 bits) 023399 nal_unit_type: 1 (0x01) - (5 bits) 02339A slice_header (3 bytes) 02339A first_mb_in_slice: 0 (0x0) 02339A slice_type: 5 (0x5) - P 02339A pic_parameter_set_id: 0 (0x0) 02339A frame_num: 1 (0x1) 02339B num_ref_idx_active_override_flag (0 bytes) 02339B num_ref_idx_active_override_flag: Yes 02339B num_ref_idx_l0_active_minus1: 0 (0x0) 02339B ref_pic_list_modification_flag_l0: No 02339B adaptive_ref_pic_marking_mode_flag: No 02339C cabac_init_idc: 0 (0x0) 02339C slice_qp_delta: -3 (0xFFFFFFFD) 02339C disable_deblocking_filter_idc: 0 (0x0) 02339C slice_alpha_c0_offset_div2: 0 (0x0) 02339D slice_beta_offset_div2: 0 (0x0) 02339D slice_data (36012 bytes) 02339D (ToDo): (Data) 02C046 slice_layer_without_partitioning (non-IDR) - 2 (0x2) - Frame 1 - slice_type P - frame_num 1 - DTS 00:00:00.000 - first_mb_in_slice 8040 (195 bytes) 02C046 Header (5 bytes) 02C046 zero_byte: 0 (0x00) 02C047 start_code_prefix_one_3bytes: 1 (0x000001) 02C04A nal_ref_idc: 3 (0x3) - (2 bits) 02C04A nal_unit_type: 1 (0x01) - (5 bits) 02C04B slice_header (6 bytes) 02C04B first_mb_in_slice: 8040 (0x001F68) 02C04E slice_type: 5 (0x5) - P 02C04E pic_parameter_set_id: 0 (0x0) 02C04E frame_num: 1 (0x1) 02C04F num_ref_idx_active_override_flag (0 bytes) 02C04F num_ref_idx_active_override_flag: Yes 02C04F num_ref_idx_l0_active_minus1: 0 (0x0) 02C04F ref_pic_list_modification_flag_l0: No 02C04F adaptive_ref_pic_marking_mode_flag: No 02C050 cabac_init_idc: 0 (0x0) 02C050 slice_qp_delta: -3 (0xFFFFFFFD) 02C050 disable_deblocking_filter_idc: 0 (0x0) 02C050 slice_alpha_c0_offset_div2: 0 (0x0) 02C051 slice_beta_offset_div2: 0 (0x0) 02C051 slice_data (190 bytes) 02C051 (ToDo): (Data) 02C109 1 (26280 bytes) 02C109 slice_layer_without_partitioning (non-IDR) - 4 (0x4) - Frame 2 - slice_type P - frame_num 2 - DTS 00:00:00.000 (26157 bytes) 02C109 Header (5 bytes) 02C109 zero_byte: 0 (0x00) 02C10A start_code_prefix_one_3bytes: 1 (0x000001) 02C10D nal_ref_idc: 3 (0x3) - (2 bits) 02C10D nal_unit_type: 1 (0x01) - (5 bits) 02C10E slice_header (3 bytes) 02C10E first_mb_in_slice: 0 (0x0) 02C10E slice_type: 5 (0x5) - P 02C10E pic_parameter_set_id: 0 (0x0) 02C10E frame_num: 2 (0x2) 02C10F num_ref_idx_active_override_flag (0 bytes) 02C10F num_ref_idx_active_override_flag: Yes 02C10F num_ref_idx_l0_active_minus1: 0 (0x0) 02C10F ref_pic_list_modification_flag_l0: No 02C10F adaptive_ref_pic_marking_mode_flag: No 02C110 cabac_init_idc: 0 (0x0) 02C110 slice_qp_delta: -2 (0xFFFFFFFE) 02C110 disable_deblocking_filter_idc: 0 (0x0) 02C110 slice_alpha_c0_offset_div2: 0 (0x0) 02C111 slice_beta_offset_div2: 0 (0x0) 02C111 slice_data (26152 bytes) 02C111 (ToDo): (Data) 032736 slice_layer_without_partitioning (non-IDR) - 4 (0x4) - Frame 2 - slice_type P - frame_num 2 - DTS 00:00:00.000 - first_mb_in_slice 8040 (123 bytes) 032736 Header (5 bytes) 032736 zero_byte: 0 (0x00) 032737 start_code_prefix_one_3bytes: 1 (0x000001) 03273A nal_ref_idc: 3 (0x3) - (2 bits) 03273A nal_unit_type: 1 (0x01) - (5 bits) 03273B slice_header (6 bytes) 03273B first_mb_in_slice: 8040 (0x001F68) 03273E slice_type: 5 (0x5) - P 03273E pic_parameter_set_id: 0 (0x0) 03273E frame_num: 2 (0x2) 03273F num_ref_idx_active_override_flag (0 bytes) 03273F num_ref_idx_active_override_flag: Yes 03273F num_ref_idx_l0_active_minus1: 0 (0x0) 03273F ref_pic_list_modification_flag_l0: No 03273F adaptive_ref_pic_marking_mode_flag: No 032740 cabac_init_idc: 0 (0x0) 032740 slice_qp_delta: -2 (0xFFFFFFFE) 032740 disable_deblocking_filter_idc: 0 (0x0) 032740 slice_alpha_c0_offset_div2: 0 (0x0) 032741 slice_beta_offset_div2: 0 (0x0) 032741 slice_data (118 bytes) 032741 (ToDo): (Data) 0327B1 1 (21125 bytes) It goes on like that, even though I set pts and dts. The settings may not be correct already (I do some calculations like (1 / framerate) * FrameNumber), but I would expect at least some numbers in pts and dts, when I set the according fields in the avPacket structure and write that via av_interleaved_write_frame(outFmtCtx, &avPacket); to the file. What could be wrong here? Edit: (please see below in the comments the download to my testdata and source file) One thing that bugs me is the fact, if I compare the output of MediaInfo from my file and that of the muxing.c generated is, that in the header, the muxing.c generated already mentions the duration of the file as 9960 ms, whereas mine is only 40 ms. muxing.c also does call avformat_write_header before even one frame is drawn. Yes, I suppose that the header will be updated, when the either av_interleaved_write_frame or av_write_trailer is called, but I totally not understand the mechanics behind it. Maybe somebody can enlighten me with some background information of any kind. Additionally, I think it could be necessarry to extract some SPS and PPS from my raw data (preceding the I-slice), and give that as extra data to the avformat_write_header call. But I just cannot figure out myself if I have to do that at all and if so, how to do it.
Titan/Cassandra read scalability, random servers going down or dropping messages
We are using Titan-1.0.0 supported by backed as cassandra 2.1.7. The cluster consists of 36 VMs each of 16GB RAM (6GB heapsize) and 16 CPU cores. We are using SSD disk for cassandra data and a normal HDD for commitlogs. The RF is 3 and reads/writes are being done at CF=2 each. The Java version is 1.8.0_45 There is heavy reads being served through this cluster. We are facing random outages with some servers at an average interval of around twice-thrice in day one of the server goes down. Let me share a particular instance of around 10-May-2017 04:00 AM, there were too many READ/MUTATION drops and we had to restart the server. Following are some of parameters that we have collected when the servers go down:
Can someone please please help us with some pointers, as to what is going wrong here and what can be done to stabilize the servers.
gc.log
2017-05-11T21:09:44.018+0530: 1900952.543: Total time for which application threads were stopped: 0.0156063 seconds, Stopping threads took: 0.0125466 seconds 2017-05-11T21:09:45.021+0530: 1900953.546: Total time for which application threads were stopped: 0.0031342 seconds, Stopping threads took: 0.0006697 seconds 2017-05-11T21:09:46.025+0530: 1900954.550: Total time for which application threads were stopped: 0.0037100 seconds, Stopping threads took: 0.0011525 seconds 2017-05-11T21:09:47.031+0530: 1900955.556: Total time for which application threads were stopped: 0.0057972 seconds, Stopping threads took: 0.0030861 seconds 2017-05-11T21:09:48.034+0530: 1900956.559: Total time for which application threads were stopped: 0.0029592 seconds, Stopping threads took: 0.0007129 seconds 2017-05-11T21:09:49.061+0530: 1900957.586: Total time for which application threads were stopped: 0.0268731 seconds, Stopping threads took: 0.0242117 seconds 2017-05-11T21:10:03.095+0530: 1900971.620: Total time for which application threads were stopped: 0.0045643 seconds, Stopping threads took: 0.0009709 seconds 2017-05-11T21:10:04.099+0530: 1900972.624: Total time for which application threads were stopped: 0.0033764 seconds, Stopping threads took: 0.0007673 seconds 2017-05-11T21:10:07.104+0530: 1900975.629: Total time for which application threads were stopped: 0.0056445 seconds, Stopping threads took: 0.0019495 seconds 2017-05-11T21:10:07.447+0530: 1900975.972: Total time for which application threads were stopped: 0.0045721 seconds, Stopping threads took: 0.0009695 seconds {Heap before GC invocations=85258 (full 0): garbage-first heap total 6291456K, used 5059586K [0x0000000640000000, 0x0000000640206000, 0x00000007c0000000) region size 2048K, 1844 young (3776512K), 105 survivors (215040K) Metaspace used 36253K, capacity 36654K, committed 36784K, reserved 1081344K class space used 3844K, capacity 3965K, committed 4016K, reserved 1048576K 2017-05-11T21:10:07.453+0530:
1900975.978: [GC pause (GCLocker Initiated GC) (young) Desired survivor size 242221056 bytes, new threshold 15 (max 15)
- age 1: 182659896 bytes, 182659896 total
- age 2: 21418160 bytes, 204078056 total
- age 3: 871016 bytes, 204949072 total
- age 4: 3600512 bytes, 208549584 total
- age 5: 1313096 bytes, 209862680 total
- age 6: 21152 bytes, 209883832 total , 0.0972867 secs] [Parallel Time: 83.6 ms, GC Workers: 13]
[GC Worker Start (ms): Min: 1900975980.9, Avg: 1900975986.0, Max: 1900975989.4, Diff: 8.5]
[Ext Root Scanning (ms): Min: 8.5, Avg: 11.9, Max: 17.4, Diff: 8.9, Sum: 154.3]
[Update RS (ms): Min: 26.8, Avg: 30.8, Max: 34.2, Diff: 7.4, Sum: 399.8]
[Processed Buffers: Min: 31, Avg: 87.3, Max: 144, Diff: 113, Sum: 1135]
[Scan RS (ms): Min: 0.1, Avg: 2.8, Max: 6.2, Diff: 6.1, Sum: 36.6]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 0.9]
[Object Copy (ms): Min: 31.5, Avg: 32.3, Max: 33.2, Diff: 1.7, Sum: 419.9]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.3, Sum: 3.1]
[GC Worker Total (ms): Min: 74.7, Avg: 78.1, Max: 83.3, Diff: 8.7, Sum: 1014.7]
[GC Worker End (ms): Min: 1900976063.9, Avg: 1900976064.1, Max: 1900976064.3, Diff: 0.3] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 1.4 ms] [Other: 12.1 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 1.0 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 1.6 ms]
[Humongous Reclaim: 0.4 ms]
[Free CSet: 4.5 ms] [Eden: 3478.0M(3476.0M)->0.0B(252.0M) Survivors: 210.0M->54.0M Heap: 4941.5M(6144.0M)->1175.0M(6144.0M)] Heap after GC invocations=85259 (full 0): garbage-first heap total 6291456K, used 1203201K [0x0000000640000000, 0x0000000640206000, 0x00000007c0000000) region size 2048K, 27 young (55296K), 27 survivors (55296K) Metaspace used 36253K, capacity 36654K, committed 36784K, reserved 1081344K class space used 3844K, capacity 3965K, committed 4016K, reserved 1048576K } [Times: user=0.58 sys=0.21, real=0.09 secs] 2017-05-11T21:10:07.551+0530:
1900976.076: Total time for which application threads were stopped: 0.1015506 seconds, Stopping threads took: 0.0004470 seconds 2017-05-11T21:10:08.557+0530: 1900977.082: Total time for which application threads were stopped: 0.0053236 seconds, Stopping threads took: 0.0010397 seconds {Heap before GC invocations=85259 (full 0): garbage-first heap total 6291456K, used 1461249K [0x0000000640000000, 0x0000000640206000, 0x00000007c0000000) region size 2048K, 153 young (313344K), 27 survivors (55296K) Metaspace used 36253K, capacity 36654K, committed 36784K, reserved 1081344K class space used 3844K, capacity 3965K, committed 4016K, reserved 1048576K 2017-05-11T21:10:08.995+0530: 1900977.520: [GC pause (G1 Evacuation Pause) (mixed) Desired survivor size 20971520 bytes, new threshold 3 (max 15)
- age 1: 19769440 bytes, 19769440 total
- age 2: 801568 bytes, 20571008 total
- age 3: 20679216 bytes, 41250224 total
- age 4: 738736 bytes, 41988960 total
- age 5: 3536096 bytes, 45525056 total
- age 6: 1242648 bytes, 46767704 total
- age 7: 19208 bytes, 46786912 total , 0.1879873 secs] [Parallel Time: 175.3 ms, GC Workers: 13]
[GC Worker Start (ms): Min: 1900977524.7, Avg: 1900977525.0, Max: 1900977525.3, Diff: 0.6]
[Ext Root Scanning (ms): Min: 14.5, Avg: 14.8, Max: 15.3, Diff: 0.8, Sum: 192.9]
[Update RS (ms): Min: 7.1, Avg: 7.9, Max: 13.3, Diff: 6.1, Sum: 103.2]
[Processed Buffers: Min: 4, Avg: 37.0, Max: 93, Diff: 89, Sum: 481]
[Scan RS (ms): Min: 55.7, Avg: 60.9, Max: 61.8, Diff: 6.1, Sum: 791.9]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.7, Diff: 0.7, Sum: 0.9]
[Object Copy (ms): Min: 90.3, Avg: 90.7, Max: 91.2, Diff: 0.9, Sum: 1179.5]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 0.8]
[GC Worker Total (ms): Min: 174.3, Avg: 174.6, Max: 174.8, Diff: 0.6, Sum: 2269.2]
[GC Worker End (ms): Min: 1900977699.6, Avg: 1900977699.6, Max: 1900977699.7, Diff: 0.1] [Code Root Fixup: 0.2 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.9 ms] [Other: 11.5 ms]
[Choose CSet: 3.3 ms]
[Ref Proc: 0.4 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 1.1 ms]
[Humongous Reclaim: 0.1 ms]
[Free CSet: 3.6 ms] [Eden: 252.0M(252.0M)->0.0B(3656.0M) Survivors: 54.0M->30.0M Heap: 1427.0M(6144.0M)->890.0M(6144.0M)] Heap after GC invocations=85260 (full 0): garbage-first heap total 6291456K, used 911310K [0x0000000640000000, 0x0000000640206000, 0x00000007c0000000) region size 2048K, 15 young (30720K), 15 survivors (30720K) Metaspace used 36253K, capacity 36654K, committed 36784K, reserved 1081344K class space used 3844K, capacity 3965K, committed 4016K, reserved 1048576K } [Times: user=1.59 sys=0.60, real=0.19 secs]
jstat
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 100.00 2.29 37.85 98.56 95.74 79645 6351.963 0 0.000 6351.963
0.00 100.00 4.63 37.85 98.56 95.74 79645 6351.963 0 0.000 6351.963
0.00 100.00 7.81 37.85 98.56 95.74 79645 6351.963 0 0.000 6351.963
0.00 100.00 10.46 37.85 98.56 95.74 79645 6351.963 0 0.000 6351.963
0.00 100.00 13.17 37.81 98.56 95.74 79645 6351.963 0 0.000 6351.963
0.00 100.00 15.30 38.11 98.56 95.74 79645 6351.963 0 0.000 6351.963
/var/log/cassandra/system.log
WARN [SharedPool-Worker-1] 2017-05-11 21:08:49,621 SliceQueryFilter.java:319 - Read 286 live and 12978 tombstone cells in icmsgraph_10042017.edgestore for key: f800000000104600 (see tombstone_warn_threshold). 2147483647 columns were requested, slices=[b02ea000800000350080000002-b02ea000800000350080000003]
WARN [SharedPool-Worker-1] 2017-05-11 21:08:52,647 SliceQueryFilter.java:319 - Read 286 live and 12978 tombstone cells in icmsgraph_10042017.edgestore for key: f800000000104600 (see tombstone_warn_threshold). 2147483647 columns were requested, slices=[b02ea000800000350080000002-b02ea000800000350080000003]
WARN [GossipTasks:1] 2017-05-11 21:09:05,030 Gossiper.java:714 - Gossip stage has 1 pending tasks; skipping status check (no nodes will be marked down)
WARN [GossipTasks:1] 2017-05-11 21:09:16,588 FailureDetector.java:249 - Not marking nodes down due to local pause of 16886451559 > 5000000000
INFO [ScheduledTasks:1] 2017-05-11 21:09:16,954 MessagingService.java:888 - 1924 READ messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,153 StatusLogger.java:51 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,154 StatusLogger.java:66 - MutationStage 0 0 32568360 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,155 StatusLogger.java:66 - RequestResponseStage 1 0 12112153129 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,155 StatusLogger.java:66 - ReadRepairStage 0 0 593141839 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,155 StatusLogger.java:66 - CounterMutationStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,155 StatusLogger.java:66 - ReadStage 15 0 12168491480 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,155 StatusLogger.java:66 - MiscStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,156 StatusLogger.java:66 - HintedHandoff 0 1 3531 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,156 StatusLogger.java:66 - GossipStage 0 0 5798815 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,156 StatusLogger.java:66 - CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,156 StatusLogger.java:66 - InternalResponseStage 0 0 537 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,156 StatusLogger.java:66 - CommitLogArchiver 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,157 StatusLogger.java:66 - CompactionExecutor 0 0 29915673 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,157 StatusLogger.java:66 - ValidationExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,157 StatusLogger.java:66 - MigrationStage 0 0 3990 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,157 StatusLogger.java:66 - AntiEntropyStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,158 StatusLogger.java:66 - PendingRangeCalculator 0 0 255 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,159 StatusLogger.java:66 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,159 StatusLogger.java:66 - MemtableFlushWriter 0 0 7705 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,159 StatusLogger.java:66 - MemtablePostFlush 0 0 44884 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,159 StatusLogger.java:66 - MemtableReclaimMemory 0 0 7705 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,160 StatusLogger.java:75 - CompactionManager 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,160 StatusLogger.java:87 - MessagingService n/a 57/3
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,160 StatusLogger.java:97 - Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,160 StatusLogger.java:99 - KeyCache 104857584 104857600 all
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,160 StatusLogger.java:105 - RowCache 0 0 all
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,160 StatusLogger.java:112 - ColumnFamily Memtable ops,data
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,162 StatusLogger.java:115 - system.compaction_history 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,162 StatusLogger.java:115 - system.hints 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,162 StatusLogger.java:115 - system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,162 StatusLogger.java:115 - system.schema_columnfamilies 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,162 StatusLogger.java:115 - system.schema_triggers 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,162 StatusLogger.java:115 - system.size_estimates 762300,93404798
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.paxos 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.peer_events 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.range_xfers 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.compactions_in_progress 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.peers 283,103428
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.schema_keyspaces 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.schema_usertypes 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.local 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.sstable_activity 315,63532
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.schema_columns 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,163 StatusLogger.java:115 - system.batchlog 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_10042017.titan_ids 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_10042017.graphindex 1843,277464
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_10042017.system_properties_lock_ 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_10042017.system_properties 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_10042017.edgestore_lock_ 6,1191
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_10042017.systemlog 741,242777
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_10042017.txlog 741,242777
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_10042017.graphindex_lock_ 1332,277240
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_10042017.edgestore 75171,16624309
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_05052017.system_properties_lock_ 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,169 StatusLogger.java:115 - icmsgraph_05052017.system_properties 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,170 StatusLogger.java:115 - icmsgraph_05052017.edgestore_lock_ 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,170 StatusLogger.java:115 - icmsgraph_05052017.txlog 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,170 StatusLogger.java:115 - icmsgraph_05052017.edgestore 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,170 StatusLogger.java:115 - icmsgraph_05052017.graphindex 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,170 StatusLogger.java:115 - icmsgraph_05052017.titan_ids 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,170 StatusLogger.java:115 - icmsgraph_05052017.systemlog 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:17,170 StatusLogger.java:115 - icmsgraph_05052017.graphindex_lock_ 0,0
WARN [SharedPool-Worker-24] 2017-05-11 21:09:17,200 SliceQueryFilter.java:319 - Read 286 live and 12978 tombstone cells in icmsgraph_10042017.edgestore for key: f800000000104600 (see tombstone_warn_threshold). 2147483647 columns were requested, slices=[b02ea000800000350080000002-b02ea000800000350080000003]
WARN [SharedPool-Worker-18] 2017-05-11 21:09:17,201 SliceQueryFilter.java:319 - Read 286 live and 12978 tombstone cells in icmsgraph_10042017.edgestore for key: f800000000104600 (see tombstone_warn_threshold). 2147483647 columns were requested, slices=[b02ea000800000350080000002-b02ea000800000350080000003]
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,170 MessagingService.java:888 - 1 READ messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,170 StatusLogger.java:51 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,171 StatusLogger.java:66 - MutationStage 0 0 32568376 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,171 StatusLogger.java:66 - RequestResponseStage 0 0 12112193109 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,171 StatusLogger.java:66 - ReadRepairStage 0 0 593143878 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,171 StatusLogger.java:66 - CounterMutationStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,171 StatusLogger.java:66 - ReadStage 0 0 12168508581 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,171 StatusLogger.java:66 - MiscStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - HintedHandoff 0 1 3531 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - GossipStage 0 0 5798844 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - InternalResponseStage 0 0 537 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - CommitLogArchiver 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - CompactionExecutor 0 0 29915673 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - ValidationExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - MigrationStage 0 0 3990 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - AntiEntropyStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - PendingRangeCalculator 0 0 255 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,172 StatusLogger.java:66 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,173 StatusLogger.java:66 - MemtableFlushWriter 0 0 7705 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,173 StatusLogger.java:66 - MemtablePostFlush 0 0 44884 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,173 StatusLogger.java:66 - MemtableReclaimMemory 0 0 7705 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,173 StatusLogger.java:75 - CompactionManager 0 0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,173 StatusLogger.java:87 - MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,173 StatusLogger.java:97 - Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,173 StatusLogger.java:99 - KeyCache 104857584 104857600 all
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,174 StatusLogger.java:105 - RowCache 0 0 all
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,173 StatusLogger.java:99 - KeyCache 104857584 104857600 all
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,174 StatusLogger.java:105 - RowCache 0 0 all
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,174 StatusLogger.java:112 - ColumnFamily Memtable ops,data
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,179 StatusLogger.java:115 - icmsgraph_10042017.titan_ids 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,179 StatusLogger.java:115 - icmsgraph_10042017.graphindex 1844,277804
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,179 StatusLogger.java:115 - icmsgraph_10042017.system_properties_lock_ 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,179 StatusLogger.java:115 - icmsgraph_10042017.system_properties 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,179 StatusLogger.java:115 - icmsgraph_10042017.edgestore_lock_ 6,1191
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,179 StatusLogger.java:115 - icmsgraph_10042017.systemlog 741,242777
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,179 StatusLogger.java:115 - icmsgraph_10042017.txlog 741,242777
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,179 StatusLogger.java:115 - icmsgraph_10042017.graphindex_lock_ 1332,277240
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,179 StatusLogger.java:115 - icmsgraph_10042017.edgestore 75176,16625645
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,180 StatusLogger.java:115 - icmsgraph_05052017.system_properties_lock_ 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,180 StatusLogger.java:115 - icmsgraph_05052017.system_properties 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,180 StatusLogger.java:115 - icmsgraph_05052017.edgestore_lock_ 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,180 StatusLogger.java:115 - icmsgraph_05052017.txlog 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,180 StatusLogger.java:115 - icmsgraph_05052017.edgestore 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,180 StatusLogger.java:115 - icmsgraph_05052017.graphindex 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,180 StatusLogger.java:115 - icmsgraph_05052017.titan_ids 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,180 StatusLogger.java:115 - icmsgraph_05052017.systemlog 0,0
INFO [ScheduledTasks:1] 2017-05-11 21:09:22,180 StatusLogger.java:115 - icmsgraph_05052017.graphindex_lock_ 0,0
WARN [SharedPool-Worker-34] 2017-05-11 21:09:24,177 SliceQueryFilter.java:319 - Read 286 live and 12978 tombstone cells in icmsgraph_10042017.edgestore for key: f800000000104600 (see tombstone_warn_threshold). 2147483647 columns were requested, slices=[b02ea000800000350080000002-b02ea000800000350080000003]
ThreadDump
Around 600 Thrift threads:
"Thrift:20118" #124110 daemon prio=5 os_prio=0 tid=0x00007feab9a71460 nid=0xb1c6 runnable [0x00007fea6a578000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked (a java.io.BufferedInputStream)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:205)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
72 MessagingService-Incoming threads ...
"MessagingService-Incoming-/192.168.33.67" #124106 prio=5 os_prio=0 tid=0x00007feab98436b0 nid=0xb0ca runnable [0x00007fea7503a000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
- locked (a java.lang.Object) at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:192)
- locked (a java.lang.Object) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
- locked (a sun.nio.ch.SocketAdaptor$SocketInputStream) at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:215) at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:149) at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:101) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:169) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88)
sar -q
09:10:01 IST 5 1396 16.02 9.24 6.31 0
I think the problem is cassandra's tombstone In Cassandra whenever a delete, insert explicitly null or TTL expire Tombstone created. Tombstones are a mechanism which allows Cassandra to write fast but it has an operational price to pay. First of all, tombstones are themselves records. They take up space and can substantially increase the amount of storage you require. Secondly, Large Number of Tombstones Causes Latency and Heap Pressure From your cassandra's system log it seems that about 13K tombstone is created only for a single vertex (Vertex Key ID : 0xf800000000104600). It's huge. You can see that when this warning appear on cassandra's system.log, Heap Pressure increases and so gc.log printed at the same time. What should i do ? Cassandra will fully drop those tombstones when a compaction triggers, only after local_delete_time + gc_grace_seconds as defined on the table the data belongs to. Run nodetool repair regularly, Once a week. You can force a major compaction on one or more table with nodetool compact Change the table compaction class. Which Compaction class to use Read More : http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html https://opencredo.com/cassandra-tombstones-common-issues/
Slow booting process after adding mem=16M in boot parameters
My linux-3.0 kernel was panicking saying ERROR: Failed to allocate 0x1000 bytes below 0x0. while booting. So I changed the bootargs and added a boot parameter mem = 16M. Now it boots fine but it takes a lot of time to boot. I have tried with higher mem value also but it does not work. Below are the logs: `Machine: KZM9D arm_add_memory: 0 0x40000000 0x1000000 Memory policy: ECC disabled, Data cache writealloc bootmem_init: max_low=0x266240, max_high=0x266240 <6>Section 8256 and 8250 (node 0)<c> have a circular dependency on usemap and pgdat allocations <7>On node 0 totalpages: 0 <7>On node 1 totalpages: 0 <7>On node 2 totalpages: 0 <7>On node 3 totalpages: 0 <7>On node 4 totalpages: 0 <7>On node 5 totalpages: 0 <7>On node 6 totalpages: 0 <7>On node 7 totalpages: 0 high_memory: e0000000 Zone PFN ranges: Normal 0x00040000 -> 0x00041000 Movable zone start PFN for each node early_node_map[1] active PFN ranges 0: 0x00040000 -> 0x00041000 <7>On node 0 totalpages: 4096 <7> Normal zone: 36 pages used for memmap <7> Normal zone: 0 pages reserved <7> Normal zone: 4060 pages, LIFO batch:0 <6>boottime: reserved memory at 0x40002000 size 0x2000 mm_init_owner <6>PERCPU: Embedded 8 pages/cpu #c087f000 s9824 r8192 d14752 u32768 <7>pcpu-alloc: s9824 r8192 d14752 u32768 alloc=8*4096 <7>pcpu-alloc: [0] 0 [0] 1 build_all_zonelists Built 1 zonelists in Node order, mobility grouping on. Total pages: 4060 Policy zone: Normal page_alloc_init <5>Kernel command line: console=ttyS1,115200n8 root=/dev/nfs ip=9.8.7.6 nfsroot=1.2.3.7:/tftpboot/arm/ rootwait rw mem=16M parse_early_param <6>PID hash table entries: 64 (order: -4, 256 bytes) <6>Dentry cache hash table entries: 2048 (order: 2, 24576 bytes) <6>Inode-cache hash table entries: 1024 (order: 0, 4096 bytes) <6>Memory: 16MB = 16MB total <5>Memory: 7824k/7824k available, 8560k reserved, 0K highmem <5>Virtual kernel memory layout: vector : 0xffff0000 - 0xffff1000 ( 4 kB) fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB) DMA : 0xffc00000 - 0xffe00000 ( 2 MB) vmalloc : 0xe0800000 - 0xf0000000 ( 248 MB) lowmem : 0xc0000000 - 0xe0000000 ( 512 MB) modules : 0xbf000000 - 0xc0000000 ( 16 MB) .text : 0xc0008000 - 0xc0704024 (7153 kB) .init : 0xc0705000 - 0xc0740660 ( 238 kB) .data : 0xc0742000 - 0xc078dc18 ( 304 kB) .bss : 0xc078dc18 - 0xc07f2950 ( 404 kB) <6>Preemptible hierarchical RCU implementation. <6>NR_IRQS:374`