Embedded Buildroot and Yocto Linux crash 50% of the time - linux-kernel

I am getting the below error 50% of the time during startup on an STM32 based processor (OSD32MP15x).
I've tried over 40 versions configs and setups none work and I am under high pressure to deliver this piece of hardware to a client. To me this looks like some kind of timing issue because it doesn't always happen.
I've tried to disable USB related configs, change DTS files and strip out everything that is not needed and last but not least change power config in the linux config and DTS.
Questions:
How can I debug the below?
What does this error look like to you?
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.4.56 (yme#ubuntu18) (gcc version 9.3.0 (Buildroot 2020.02-00011-g7ea8a52)) #1 SMP PREEMPT Wed Nov 4 17:25:46 CET 2020
[ 0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: IM45
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] Reserved memory: created DMA memory pool at 0x10000000, size 0 MiB
[ 0.000000] OF: reserved mem: initialized node mcuram2#10000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x10040000, size 0 MiB
[ 0.000000] OF: reserved mem: initialized node vdev0vring0#10040000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x10041000, size 0 MiB
[ 0.000000] OF: reserved mem: initialized node vdev0vring1#10041000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x10042000, size 0 MiB
[ 0.000000] OF: reserved mem: initialized node vdev0buffer#10042000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x30000000, size 0 MiB
[ 0.000000] OF: reserved mem: initialized node mcuram#30000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x38000000, size 0 MiB
[ 0.000000] OF: reserved mem: initialized node retram#38000000, compatible id shared-dma-pool
[ 0.000000] cma: Reserved 128 MiB at 0xd2000000
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.1 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
[ 0.000000] psci: SMC Calling Convention v1.0
[ 0.000000] percpu: Embedded 20 pages/cpu s50124 r8192 d23604 u81920
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 113536
[ 0.000000] Kernel command line: root=/dev/mmcblk1p2 rootwait rw console=ttySTM0,115200
[ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 305152K/458752K available (11264K kernel code, 844K rwdata, 3364K rodata, 1024K init, 216K bss, 22528K reserved, 131072K cma-reserved, 0K highmem)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] random: get_random_bytes called from start_kernel+0x31c/0x4f4 with crng_init=0
[ 0.000000] arch_timer: cp15 timer(s) running at 24.00MHz (virt).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x588fe9dc0, max_idle_ns: 440795202592 ns
[ 0.000009] sched_clock: 56 bits at 24MHz, resolution 41ns, wraps every 4398046511097ns
[ 0.000027] Switching to timer-based delay loop, resolution 41ns
[ 0.001228] Console: colour dummy device 80x30
[ 0.001276] Calibrating delay loop (skipped), value calculated using timer frequency.. 48.00 BogoMIPS (lpj=240000)
[ 0.001296] pid_max: default: 32768 minimum: 301
[ 0.001560] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.001581] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.002900] CPU: Testing write buffer coherency: ok
[ 0.003447] /cpus/cpu#0 missing clock-frequency property
[ 0.003484] /cpus/cpu#1 missing clock-frequency property
[ 0.003504] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.060001] Setting up static identity map for 0xc0100000 - 0xc0100060
[ 0.079971] rcu: Hierarchical SRCU implementation.
[ 0.104229] smp: Bringing up secondary CPUs ...
[ 0.180623] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.180859] smp: Brought up 1 node, 2 CPUs
[ 0.180884] SMP: Total of 2 processors activated (96.00 BogoMIPS).
[ 0.180895] CPU: All CPU(s) started in SVC mode.
[ 0.181728] devtmpfs: initialized
[ 0.201048] VFP support v0.3: implementor 41 architecture 2 part 30 variant 7 rev 5
[ 0.201702] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.201736] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[ 0.208702] pinctrl core: initialized pinctrl subsystem
[ 0.211039] NET: Registered protocol family 16
[ 0.223325] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.225695] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[ 0.225715] hw-breakpoint: maximum watchpoint size is 8 bytes.
[ 0.226146] Serial: AMBA PL011 UART driver
[ 0.228991] stm32-pm-domain pm_domain: domain core-ret-power-domain registered
[ 0.229030] stm32-pm-domain pm_domain: subdomain core-power-domain registered
[ 0.229046] stm32-pm-domain pm_domain: domains probed
[ 0.264062] SCSI subsystem initialized
[ 0.264934] usbcore: registered new interface driver usbfs
[ 0.265017] usbcore: registered new interface driver hub
[ 0.265170] usbcore: registered new device driver usb
[ 0.265576] pps_core: LinuxPPS API ver. 1 registered
[ 0.265590] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti#linux.it>
[ 0.265624] PTP clock support registered
[ 0.265887] EDAC MC: Ver: 3.0.0
[ 0.267262] Advanced Linux Sound Architecture Driver Initialized.
[ 0.269092] clocksource: Switched to clocksource arch_sys_counter
[ 0.994946] thermal_sys: Registered thermal governor 'step_wise'
[ 0.995599] NET: Registered protocol family 2
[ 0.996743] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[ 0.996799] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[ 0.996866] TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 0.996962] TCP: Hash tables configured (established 4096 bind 4096)
[ 0.997119] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 0.997165] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 0.997480] NET: Registered protocol family 1
[ 0.998493] RPC: Registered named UNIX socket transport module.
[ 0.998514] RPC: Registered udp transport module.
[ 0.998525] RPC: Registered tcp transport module.
[ 0.998536] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.999750] hw perfevents: enabled with armv7_cortex_a7 PMU driver, 5 counters available
[ 1.001772] Initialise system trusted keyrings
[ 1.002136] workingset: timestamp_bits=14 max_order=17 bucket_order=3
[ 1.015364] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 1.016787] NFS: Registering the id_resolver key type
[ 1.016838] Key type id_resolver registered
[ 1.016850] Key type id_legacy registered
[ 1.017081] ntfs: driver 2.1.32 [Flags: R/O].
[ 1.017623] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[ 1.018434] fuse: init (API version 7.31)
[ 1.019683] NET: Registered protocol family 38
[ 1.019706] Key type asymmetric registered
[ 1.019719] Asymmetric key parser 'x509' registered
[ 1.019785] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[ 1.019800] io scheduler mq-deadline registered
[ 1.019811] io scheduler kyber registered
[ 1.029372] STM32 USART driver initialized
[ 1.050170] brd: module loaded
[ 1.065205] loop: module loaded
[ 1.070532] libphy: Fixed MDIO Bus: probed
[ 1.072541] CAN device driver interface
[ 1.075529] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 1.075582] ehci-platform: EHCI generic platform driver
[ 1.076011] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 1.076048] ohci-platform: OHCI generic platform driver
[ 1.076725] usbcore: registered new interface driver cdc_acm
[ 1.076739] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[ 1.080942] i2c /dev entries driver
[ 1.084746] stm32-cpufreq stm32-cpufreq: Failed to get chip info: -517
[ 1.087467] sdhci: Secure Digital Host Controller Interface driver
[ 1.087483] sdhci: Copyright(c) Pierre Ossman
[ 1.087493] Synopsys Designware Multimedia Card Interface Driver
[ 1.087913] sdhci-pltfm: SDHCI platform and OF driver helper
[ 1.088990] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.090464] usbcore: registered new interface driver usbhid
[ 1.090481] usbhid: USB HID core driver
[ 1.090775] arm-smc-mbox mailbox-0: ARM SMC mailbox enabled.
[ 1.090909] arm-smc-mbox mailbox-1: ARM SMC mailbox enabled.
[ 1.092452] remoteproc remoteproc0: releasing m4
[ 1.094961] Unable to get STM32 DDR PMU clock
[ 1.099244] drop_monitor: Initializing network drop monitor service
[ 1.099591] NET: Registered protocol family 17
[ 1.099625] can: controller area network core (rev 20170425 abi 9)
[ 1.099760] NET: Registered protocol family 29
[ 1.099779] can: raw protocol (rev 20170425)
[ 1.099791] can: broadcast manager protocol (rev 20170425 t)
[ 1.099809] can: netlink gateway (rev 20190810) max_hops=1
[ 1.100648] Key type dns_resolver registered
[ 1.100825] ThumbEE CPU extension supported.
[ 1.100855] Registering SWP/SWPB emulation handler
[ 1.101320] registered taskstats version 1
[ 1.101336] Loading compiled-in X.509 certificates
[ 1.112592] Loaded X.509 cert 'Build time autogenerated kernel key: a1bb00e898260e4deb50c76fbad98451248b76f2'
[ 1.132155] arm-scmi firmware:scmi-0: SCMI Protocol v2.0 'ST:' Firmware version 0x0
[ 1.135402] remoteproc remoteproc0: releasing m4
[ 1.136357] Unable to get STM32 DDR PMU clock
[ 1.159535] remoteproc remoteproc0: releasing m4
[ 1.160502] Unable to get STM32 DDR PMU clock
[ 1.199623] stm32-mdma 58000000.dma: STM32 MDMA driver registered
[ 1.202939] stm32-dma 48000000.dma: STM32 DMA driver registered
[ 1.229428] random: fast init done
[ 1.229493] stm32-crc32 58009000.crc: Initialized
[ 1.233994] random: crng init done
[ 1.241968] remoteproc remoteproc0: releasing m4
[ 1.243240] stm32-ddr-pmu: probed (ID=0x00140061 VER=0x00000010), DDR#533MHz
[ 1.243899] /soc/interrupt-controller#5000d000: bank0
[ 1.243916] /soc/interrupt-controller#5000d000: bank1
[ 1.243929] /soc/interrupt-controller#5000d000: bank2
[ 1.245454] stm32mp157-pinctrl soc:pin-controller#50002000: GPIOA bank added
[ 1.245888] stm32mp157-pinctrl soc:pin-controller#50002000: GPIOB bank added
[ 1.246272] stm32mp157-pinctrl soc:pin-controller#50002000: GPIOC bank added
[ 1.246674] stm32mp157-pinctrl soc:pin-controller#50002000: GPIOD bank added
[ 1.247034] stm32mp157-pinctrl soc:pin-controller#50002000: GPIOE bank added
[ 1.247478] stm32mp157-pinctrl soc:pin-controller#50002000: GPIOF bank added
[ 1.247852] stm32mp157-pinctrl soc:pin-controller#50002000: GPIOG bank added
[ 1.248211] stm32mp157-pinctrl soc:pin-controller#50002000: GPIOH bank added
[ 1.248577] stm32mp157-pinctrl soc:pin-controller#50002000: GPIOI bank added
[ 1.248617] stm32mp157-pinctrl soc:pin-controller#50002000: Pinctrl STM32 initialized
[ 1.249706] stm32mp157-pinctrl soc:pin-controller-z#54004000: GPIOZ bank added
[ 1.249733] stm32mp157-pinctrl soc:pin-controller-z#54004000: Pinctrl STM32 initialized
[ 1.252232] 4000f000.serial: ttySTM1 at MMIO 0x4000f000 (irq = 36, base_baud = 6527435) is a stm32-usart
[ 1.253971] stm32-usart 40010000.serial: interrupt mode for rx (no dma)
[ 1.253994] stm32-usart 40010000.serial: interrupt mode for tx (no dma)
[ 1.254033] 40010000.serial: ttySTM0 at MMIO 0x40010000 (irq = 37, base_baud = 4000000) is a stm32-usart
[ 2.327555] printk: console [ttySTM0] enabled
[ 2.336702] stm32_rtc 5c004000.rtc: registered as rtc0
[ 2.341362] stm32_rtc 5c004000.rtc: Date/Time must be initialized
[ 2.346472] stm32_rtc 5c004000.rtc: registered rev:1.2
[ 2.373942] stm32f7-i2c 5c002000.i2c: doesn't use DMA
[ 2.379061] stpmic1 0-0033: PMIC Chip Version: 0x21
[ 2.383768] genirq: irq_chip stm32-exti-h did not update eff. affinity mask of irq 42
[ 2.391628] BUCK1: supplied by vin
[ 2.397045] BUCK2: supplied by vin
[ 2.402086] BUCK3: supplied by vin
[ 2.407216] BUCK4: supplied by vin
[ 2.412185] LDO1: supplied by v3v3
[ 2.417921] LDO2: supplied by vin
[ 2.423176] LDO3: supplied by vdd_ddr
[ 2.428498] LDO4: supplied by vin
[ 2.433721] LDO5: supplied by vin
[ 2.438846] LDO6: supplied by v3v3
[ 2.444180] VREF_DDR: supplied by vin
[ 2.449317] BOOST: supplied by vin
[ 2.454858] 8<--- cut here ---
[ 2.454968] VBUS_OTG: supplied by bst_out
[ 2.456448] Unable to handle kernel NULL pointer dereference at virtual address 00000050
[ 2.468540] pgd = 65ccf75c
[ 2.471247] [00000050] *pgd=00000000
[ 2.474801] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[ 2.480100] Modules linked in:
[ 2.483148] CPU: 0 PID: 73 Comm: irq/42-pmic_irq Not tainted 5.4.56 #1
[ 2.489664] Hardware name: STM32 (Device Tree Support)
[ 2.494809] PC is at mutex_trylock+0xc/0x84
[ 2.498977] LR is at regulator_lock+0x1c/0xbc
[ 2.503319] pc : [<c0b0dfec>] lr : [<c05d48ec>] psr: 60000013
[ 2.509580] sp : df7c7ecc ip : 010a1d00 fp : c0cbe554
[ 2.514798] r10: c0cbe984 r9 : df7c3800 r8 : 0000000e
[ 2.520016] r7 : df7eec68 r6 : 00000050 r5 : c11b5550 r4 : 00000000
[ 2.526539] r3 : df7c7ecc r2 : df7c600c r1 : 00000000 r0 : 00000050
[ 2.533064] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 2.540194] Control: 10c5387d Table: c000406a DAC: 00000051
[ 2.545935] Process irq/42-pmic_irq (pid: 73, stack limit = 0xa70e693e)
[ 2.552542] Stack: (0xdf7c7ecc to 0xdf7c8000)
[ 2.556897] 7ec0: c05d48ec df7ed0c0 c11b5550 df7eec00 c05e813c
[ 2.565073] 7ee0: df7f8bc0 00000000 df7eec00 c017fba0 00000020 00000024 df7ca800 00000001
[ 2.573247] 7f00: 0000000e c069b370 00000001 df1a6540 df7ca700 c1104c88 df7c6000 df7eb380
[ 2.581422] 7f20: df7ca700 df7c6000 00000001 df7eb380 c017c470 ffffe000 00000000 c017c48c
[ 2.589599] 7f40: df7ca700 df7eb3a4 df7c6000 c017c7a4 40000013 00000000 c017c59c c1104c88
[ 2.597774] 7f60: 00000000 df5a9d00 df7eb3c0 df1cbab4 df7c6000 df7eb380 c017c670 df5a9d1c
[ 2.605949] 7f80: 00000000 c0148a70 00000000 df7eb3c0 c0148954 00000000 00000000 00000000
[ 2.614124] 7fa0: 00000000 00000000 00000000 c01010e8 00000000 00000000 00000000 00000000
[ 2.622298] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 2.630473] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 2.638666] [<c0b0dfec>] (mutex_trylock) from [<c05d48ec>] (regulator_lock+0x1c/0xbc)
[ 2.646486] [<c05d48ec>] (regulator_lock) from [<c05e813c>] (stpmic1_boost_irq_handler+0x44/0x1cc)
[ 2.655438] [<c05e813c>] (stpmic1_boost_irq_handler) from [<c017fba0>] (handle_nested_irq+0xc4/0x130)
[ 2.664654] [<c017fba0>] (handle_nested_irq) from [<c069b370>] (regmap_irq_thread+0x328/0x508)
[ 2.673260] [<c069b370>] (regmap_irq_thread) from [<c017c48c>] (irq_thread_fn+0x1c/0x78)
[ 2.681344] [<c017c48c>] (irq_thread_fn) from [<c017c7a4>] (irq_thread+0x134/0x200)
[ 2.688999] [<c017c7a4>] (irq_thread) from [<c0148a70>] (kthread+0x11c/0x144)
[ 2.696132] [<c0148a70>] (kthread) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
[ 2.703338] Exception stack(0xdf7c7fb0 to 0xdf7c7ff8)
[ 2.708385] 7fa0: 00000000 00000000 00000000 00000000
[ 2.716562] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 2.724735] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 2.731348] Code: e8bd81f0 e52de004 e1a0300d e3c32d7f (e5903000)
[ 2.737474] ---[ end trace 63a5f7bb4baf46f1 ]---
[ 2.742083] genirq: exiting task "irq/42-pmic_irq" (73) is an active IRQ thread (irq 42)
Edit link linux codebase: v5.4-stm32mp
Edit add DTS config:
pmic:stpmic#33{
compatible = "st,stpmic1";
reg = <0x33>;
interrupts-extended = <&exti_pwr 55 IRQ_TYPE_EDGE_FALLING>;
interrupt-controller;
#interrupt-cells = <2>;
status = "okay";
regulators{
compatible = "st,stpmic1-regulators";
buck1-supply = <&vin>;
buck2-supply = <&vin>;
buck3-supply = <&vin>;
buck4-supply = <&vin>;
ldo1-supply = <&v3v3>;
ldo2-supply = <&vin>;
ldo3-supply = <&vdd_ddr>;
ldo4-supply = <&vin>;
ldo5-supply = <&vin>;
ldo6-supply = <&v3v3>;
vref_ddr-supply = <&vin>;
boost-supply = <&vin>;
pwr_sw1-supply = <&bst_out>;
pwr_sw2-supply = <&bst_out>;
vddcore:buck1{
regulator-name = "vddcore";
regulator-min-microvolt = <1200000>;
regulator-max-microvolt = <1350000>;
regulator-always-on;
regulator-initial-mode = <0>;
regulator-over-current-protection;
};
vdd_ddr:buck2{
regulator-name = "vdd_ddr";
regulator-min-microvolt = <1350000>;
regulator-max-microvolt = <1350000>;
regulator-always-on;
regulator-initial-mode = <0>;
regulator-over-current-protection;
};
vdd:buck3{
regulator-name = "vdd";
regulator-min-microvolt = <3300000>;
regulator-max-microvolt = <3300000>;
regulator-always-on;
st,mask-reset;
regulator-initial-mode = <0>;
regulator-over-current-protection;
};
v3v3:buck4{
regulator-name = "v3v3";
regulator-min-microvolt = <3300000>;
regulator-max-microvolt = <3300000>;
regulator-always-on;
regulator-over-current-protection;
regulator-initial-mode = <0>;
};
v1v8_audio:ldo1{
regulator-name = "v1v8_audio";
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <1800000>;
regulator-always-on;
interrupts = <IT_CURLIM_LDO1 0>;
};
v3v3_hdmi:ldo2{
regulator-name = "v3v3_hdmi";
regulator-min-microvolt = <3300000>;
regulator-max-microvolt = <3300000>;
regulator-always-on;
interrupts = <IT_CURLIM_LDO2 0>;
};
vtt_ddr:ldo3{
regulator-name = "vtt_ddr";
regulator-min-microvolt = <500000>;
regulator-max-microvolt = <750000>;
regulator-always-on;
regulator-over-current-protection;
};
vdd_usb:ldo4{
regulator-name = "vdd_usb";
regulator-min-microvolt = <3300000>;
regulator-max-microvolt = <3300000>;
interrupts = <IT_CURLIM_LDO4 0>;
regulator-always-on;
};
vdda:ldo5{
regulator-name = "vdda";
regulator-min-microvolt = <2900000>;
regulator-max-microvolt = <2900000>;
interrupts = <IT_CURLIM_LDO5 0>;
regulator-boot-on;
};
v1v2_hdmi:ldo6{
regulator-name = "v1v2_hdmi";
regulator-min-microvolt = <1200000>;
regulator-max-microvolt = <1200000>;
regulator-always-on;
interrupts = <IT_CURLIM_LDO6 0>;
};
vref_ddr:vref_ddr{
regulator-name = "vref_ddr";
regulator-always-on;
regulator-over-current-protection;
};
bst_out:boost{
regulator-name = "bst_out";
interrupts = <IT_OCP_BOOST 0>;
regulator-always-on;
};
vbus_otg:pwr_sw1{
regulator-name = "vbus_otg";
interrupts = <IT_OCP_OTG 0>;
};
vbus_sw:pwr_sw2{
regulator-name = "vbus_sw";
interrupts = <IT_OCP_SWOUT 0>;
regulator-active-discharge = <1>;
regulator-always-on;
};
};
onkey{
compatible = "st,stpmic1-onkey";
interrupts = <IT_PONKEY_F 0>, <IT_PONKEY_R 0>;
interrupt-names = "onkey-falling", "onkey-rising";
power-off-time-sec = <10>;
status = "okay";
};
watchdog {
compatible = "st,stpmic1-wdt";
status = "disabled";
};
};

It looks like the problem was introduced by the patch: regulator: stpmic1: Usb boost over-current protection workaround.
The "boost" interrupt handler stpmic1_boost_irq_handler calls regulator_lock(usb_data->boost_rdev);
The "boost" interrupt handler (stpmic1_boost_irq_handler) is set up by the stpmic1_boost_register function, but usb_data->boost_rdev is still null at this time.
usb_data->boost_rdev is set to the return value of the stpmic1_boost_register function by thestpmic1_regulator_probe function.
If the "boost" interrupt handler stpmic1_boost_irq_handler gets called before usb_data->boost_rdev has been set by the stpmic1_regulator_probe function, then the crash occurs.
A workaround would be to set usb_data->boost_rdev = rdev; in the stpmic1_boost_register function after it has set rdev but before it sets up the interrupt handler.
This should be reported to the author of the patch. See the "Signed-off-by:" line in the patch for the email address.

Related

embedded linux buidling/booting issue

I am trying to cross compile the Linux for a dual-core ARM Cortex A9 device. While booting I got the following prints on the UART which is showing some sync issue when Core-1 is starting. I remember I have enabled multi programming while compiling the kernel. What part of the building process must have gone wrong here?
zImage building or
Generating the root file system?
Uart Prints:
U-Boot SPL 2013.01.01 (Sep 13 2019 - 01:35:54)
BOARD : Altera SOCFPGA Cyclone V Board
CLOCK: EOSC1 clock 25000 KHz
CLOCK: EOSC2 clock 25000 KHz
CLOCK: F2S_SDR_REF clock 0 KHz
CLOCK: F2S_PER_REF clock 0 KHz
CLOCK: MPU clock 800 MHz
CLOCK: DDR clock 400 MHz
CLOCK: UART clock 100000 KHz
CLOCK: MMC clock 50000 KHz
CLOCK: QSPI clock 3125 KHz
RESET: COLD
INFO : Watchdog enabled
SDRAM: Initializing MMR registers
SDRAM: Calibrating PHY
SEQ.C: Preparing to start memory calibration
SEQ.C: CALIBRATION PASSED
SDRAM: 1024 MiB
ALTERA DWMMC: 0
reading u-boot.img
reading u-boot.img
U-Boot 2013.01.01-00140-g0c9a3bdf3c (Sep 13 2019 - 01:28:34)
CPU : Altera SOCFPGA Platform
BOARD : Altera SOCFPGA Cyclone V Board
I2C: ready
DRAM: 1 GiB
MMC: ALTERA DWMMC: 0
*** Warning - bad CRC, using default environment
In: serial
Out: serial
Err: serial
Skipped ethaddr assignment due to invalid EMAC address in EEPROM
Net: mii0
Warning: failed to set MAC address
Hit any key to stop autoboot: 0
reading u-boot.scr
589 bytes read in 5 ms (114.3 KiB/s)
## Executing script at 02000000
-- Programming FPGA --
reading soc_system.rbf
7007204 bytes read in 361 ms (18.5 MiB/s)
## Starting application at 0x3FF785A4 ...
## Application terminated, rc = 0x0
-- Setting Env Variables --
reading zImage
4389688 bytes read in 246 ms (17 MiB/s)
reading soc_system.dtb
24091 bytes read in 9 ms (2.6 MiB/s)
## Flattened Device Tree blob at 00000100
Booting using the fdt blob at 0x00000100
Loading Device Tree to 03ff7000, end 03fffe1a ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.9.0-ge4c2b71c147d (gurmeet#gurmeet-VirtualBox) (gcc version 7.1.1 20170510 (Linaro GCC 7.1-2017.05) ) #1 SMP Sat Sep 14 10:50:07 IST 2019
[ 0.000000] CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt:Machine model: Terasic DE10 NANO
[ 0.000000] cma: Reserved 16 MiB at 0x3f000000
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] percpu: Embedded 14 pages/cpu #ef7ca000 s26892 r8192 d22260 u57344
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260608
[ 0.000000] Kernel command line: console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1012076K/1048576K available (7168K kernel code, 464K rwdata, 1436K rodata, 1024K init, 173K bss, 20116K reserved, 16384K cma-reserved, 245760K highmem)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
[ 0.000000] vmalloc : 0xf0800000 - 0xff800000 ( 240 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xf0000000 ( 768 MB)
[ 0.000000] pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
[ 0.000000] modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
[ 0.000000] .text : 0xc0008000 - 0xc0800000 (8160 kB)
[ 0.000000] .init : 0xc0a00000 - 0xc0b00000 (1024 kB)
[ 0.000000] .data : 0xc0b00000 - 0xc0b742b4 ( 465 kB)
[ 0.000000] .bss : 0xc0b742b4 - 0xc0b9f80c ( 174 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] L2C-310 erratum 769419 enabled
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[ 0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[ 0.000000] L2C-310 cache controller enabled, 8 ways, 512 kB
[ 0.000000] L2C-310: CACHE_ID 0x410030c9, AUX_CTRL 0x46060001
[ 0.000000] clocksource: timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[ 0.000007] sched_clock: 32 bits at 100MHz, resolution 10ns, wraps every 21474836475ns
[ 0.000019] Switching to timer-based delay loop, resolution 10ns
[ 0.000248] GIC: PPI13 is secure or misconfigured
[ 0.000285] GIC: PPI13 is secure or misconfigured
[ 0.000481] Console: colour dummy device 80x30
[ 0.000506] Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=1000000)
[ 0.000521] pid_max: default: 32768 minimum: 301
[ 0.000625] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.000637] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.001356] CPU: Testing write buffer coherency: ok
[ 0.001386] ftrace: allocating 23073 entries in 68 pages
[ 0.042460] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.042709] Setting up static identity map for 0x100000 - 0x100058
[ 0.044448] GIC: PPI13 is secure or misconfigured
[ 0.044460] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.044549] Brought up 2 CPUs
[ 0.044568] SMP: Total of 2 processors activated (400.00 BogoMIPS).
[ 0.044575] CPU: All CPU(s) started in SVC mode.
[ 0.045480] devtmpfs: initialized
[ 0.051307] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.051529] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.053213] NET: Registered protocol family 16
[ 0.055578] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.056834] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.056846] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.115503] vgaarb: loaded
[ 0.115792] SCSI subsystem initialized
[ 0.116038] usbcore: registered new interface driver usbfs
[ 0.116099] usbcore: registered new interface driver hub
[ 0.116166] usbcore: registered new device driver usb
[ 0.116328] sopc#0:usbphy#0 supply vcc not found, using dummy regulator
[ 0.117339] pps_core: LinuxPPS API ver. 1 registered
[ 0.117351] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti#linux.it>
[ 0.117377] PTP clock support registered
[ 0.117557] FPGA manager framework
[ 0.118986] clocksource: Switched to clocksource timer
[ 0.151378] NET: Registered protocol family 2
[ 0.151919] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 0.151999] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.152126] TCP: Hash tables configured (established 8192 bind 8192)
[ 0.152208] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 0.152255] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 0.152425] NET: Registered protocol family 1
[ 0.152936] RPC: Registered named UNIX socket transport module.
[ 0.152947] RPC: Registered udp transport module.
[ 0.152954] RPC: Registered tcp transport module.
[ 0.152961] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.153628] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 0.154456] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 0.155247] workingset: timestamp_bits=30 max_order=18 bucket_order=0
[ 0.164377] NFS: Registering the id_resolver key type
[ 0.164411] Key type id_resolver registered
[ 0.164419] Key type id_legacy registered
[ 0.164435] Installing knfsd (copyright (C) 1996 okir#monad.swb.de).
[ 0.165053] ntfs: driver 2.1.32 [Flags: R/W].
[ 0.165373] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[ 0.166615] bounce: pool size: 64 pages
[ 0.166628] io scheduler noop registered (default)
[ 0.174406] dma-pl330 ffe01000.dma: Loaded driver for PL330 DMAC-341330
[ 0.174424] dma-pl330 ffe01000.dma: DBUFF-512x8bytes Num_Chans-8 Num_Peri-32 Num_Events-8
[ 0.178453] Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
[ 0.179707] console [ttyS0] disabled
[ 0.179744] ffc02000.serial: ttyS0 at MMIO 0xffc02000 (irq = 24, base_baud = 6250000) is a 16550A
[ 0.766263] console [ttyS0] enabled
[ 0.771581] brd: module loaded
[ 0.774834] altera_ilc ff230000.ilc: Failed to register interrupt handler
[ 0.781634] altera_ilc ff230000.ilc: Failed to register interrupt handler
[ 0.788418] altera_ilc ff230000.ilc: Driver successfully loaded
[ 0.796364] libphy: Fixed MDIO Bus: probed
[ 0.801310] CAN device driver interface
[ 0.805522] socfpga-dwmac ff702000.ethernet: snps,phy-addr property is deprecated
[ 0.813151] stmmac - user ID: 0x10, Synopsys ID: 0x37
[ 0.818182] Ring mode enabled
[ 0.821237] DMA HW capability register supported[ 0.825749] Enhanced/Alternate descriptors
[ 0.829922] Enabled extended descriptors
[ 0.833913] RX Checksum Offload Engine supported
[ 0.838596] COE Type 2
[ 0.841040] TX Checksum insertion supported
[ 0.845290] Enable RX Mitigation via HW Watchdog Timer
[ 0.856614] libphy: stmmac: probed
[ 0.860049] eth0: PHY ID 00221622 at 1 IRQ POLL (stmmac-0:01) active
[ 0.866923] ffb40000.usb supply vusb_d not found, using dummy regulator
[ 0.873577] ffb40000.usb supply vusb_a not found, using dummy regulator
[ 0.880327] dwc2 ffb40000.usb: EPs: 16, dedicated fifos, 8064 entries in SPRAM
[ 0.887867] dwc2 ffb40000.usb: DWC OTG Controller
[ 0.892600] dwc2 ffb40000.usb: new USB bus registered, assigned bus number 1
[ 0.899660] dwc2 ffb40000.usb: irq
[ 0.989000] mmc_host mmc0: Bus speed (slot 0) = 50000000Hz (slot req 400000Hz, actual 396825HZ div = 63)
[ 1.019146] dw_mmc ff704000.flash: 1 slots initialized
[ 1.024610] OF: /sopc#0/leds/fpga0: could not get #gpio-cells for /sopc#0/bridge#0xc0000000/leds#0x100000000
[ 1.034522] leds-gpio: probe of sopc#0:leds failed with error -22
[ 1.040771] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.046944] usbcore: registered new interface driver usbhid
[ 1.052514] usbhid: USB HID core driver
[ 1.056592] fpga_manager fpga0: Altera SOCFPGA FPGA Manager registered
[ 1.063805] altera_hps2fpga_bridge sopc#0:fpgabridge#0: fpga bridge [hps2fpga] registered
[ 1.072200] altera_hps2fpga_bridge sopc#0:fpgabridge#1: fpga bridge [lwhps2fpga] registered
[ 1.080687] altera_hps2fpga_bridge sopc#0:fpgabridge#2: fpga bridge [fpga2hps] registered
[ 1.089235] altera_fpga2sdram_bridge sopc#0:fpgabridge#3: fpga bridge [fpga2sdram] registered
[ 1.097728] altera_fpga2sdram_bridge sopc#0:fpgabridge#3: driver initialized with handoff 000001ff
[ 1.107005] oprofile: using arm/armv7-ca9
[ 1.111851] NET: Registered protocol family 10
[ 1.117061] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 1.123642] NET: Registered protocol family 17
[ 1.128087] NET: Registered protocol family 15
[ 1.132545] can: controller area network core (rev 20120528 abi 9)
[ 1.138755] NET: Registered protocol family 29
[ 1.143200] can: raw protocol (rev 20120528)
[ 1.147454] can: broadcast manager protocol (rev 20161123 t)
[ 1.153104] can: netlink gateway (rev 20130117) max_hops=1
[ 1.158791] 8021q: 802.1Q VLAN Support v1.8
[ 1.163021] Key type dns_resolver registered
[ 1.167350] ThumbEE CPU extension supported.
[ 1.171622] Registering SWP/SWPB emulation handler
[ 1.179767] hctosys: unable to open rtc device (rtc0)
[ 1.184823] of_cfs_init
[ 1.187332] of_cfs_init: OK
[ 1.190841] ttyS0 - failed to request DMA
[ 1.194993] Waiting for root device /dev/mmcblk0p2...
[ 1.238212] mmc0: Problem switching card into high-speed mode!
[ 1.244050] mmc_host mmc0: Bus speed (slot 0) = 50000000Hz (slot req 25000000Hz, actual 25000000HZ div = 1)
[ 1.253817] mmc0: new SDHC card at address 0001
[ 1.258775] mmcblk0: mmc0:0001 ASTC 7.45 GiB
[ 1.265186] mmcblk0: p1 p2 p3
[ 1.320560] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 1.350985] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 1.359101] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 1.366797] devtmpfs: error mounting -2
[ 1.373057] Freeing unused kernel memory: 1024K (c0a00000 - c0b00000)
[ 1.379834] Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
[ 1.392908] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.9.0-ge4c2b71c147d #1
[ 1.399924] Hardware name: Altera SOCFPGA
[ 1.403946] [<c0112004>] (unwind_backtrace) from [<c010cc88>] (show_stack+0x20/0x24)
[ 1.411671] [<c010cc88>] (show_stack) from [<c04255b8>] (dump_stack+0x90/0xa4)
[ 1.418874] [<c04255b8>] (dump_stack) from [<c01e9978>] (panic+0xec/0x258)
[ 1.425738] [<c01e9978>] (panic) from [<c0739004>] (kernel_init+0x10c/0x11c)
[ 1.432769] [<c0739004>] (kernel_init) from [<c0108478>] (ret_from_fork+0x14/0x3c)
[ 1.440309] CPU1: stopping
[ 1.443013] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.9.0-ge4c2b71c147d #1
[ 1.450029] Hardware name: Altera SOCFPGA
[ 1.454033] [<c0112004>] (unwind_backtrace) from [<c010cc88>] (show_stack+0x20/0x24)
[ 1.461749] [<c010cc88>] (show_stack) from [<c04255b8>] (dump_stack+0x90/0xa4)
[ 1.468947] [<c04255b8>] (dump_stack) from [<c0110604>] (handle_IPI+0x2e0/0x2fc)
[ 1.476316] [<c0110604>] (handle_IPI) from [<c0101544>] (gic_handle_irq+0x98/0x9c)
[ 1.483857] [<c0101544>] (gic_handle_irq) from [<c010d84c>] (__irq_svc+0x6c/0x90)
[ 1.491305] Exception stack(0xef11bf58 to 0xef11bfa0)
[ 1.496335] bf40: 00000000 000004c6
[ 1.504479] bf60: ef7da370 c011bca0 ffffe000 00000002 c0b0302c c0b0307c c0b13c83 00000001
[ 1.512622] bf80: c08c86ec ef11bfb4 ef11bfb8 ef11bfa8 c0108f44 c0108f48 60000013 ffffffff
[ 1.520771] [<c010d84c>] (__irq_svc) from [<c0108f48>] (arch_cpu_idle+0x48/0x4c)
[ 1.528142] [<c0108f48>] (arch_cpu_idle) from [<c073dd44>] (default_idle_call+0x30/0x3c)
[ 1.536205] [<c073dd44>] (default_idle_call) from [<c0164760>] (cpu_startup_entry+0xe8/0x148)
[ 1.544699] [<c0164760>] (cpu_startup_entry) from [<c01100a8>] (secondary_start_kernel+0x15c/0x164)
[ 1.553708] [<c01100a8>] (secondary_start_kernel) from [<0010194c>] (0x10194c)
[ 1.560906] ---[ end Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
There was an error in mounting the ext4 partition of SD card. Rebuilding the Kernel and build root with right configuration solved the issue. (Enabled Support for large (2TB+) block devices and files).

What's the significance of CONFIG_CGROUPS in Kernel 4.12?

I have an embedded system with power pc architecture 87xx. If I don't enable the "CONFIG_CGROUPS" in Linux kernel I can't mount the devtmpf i.e. ubifs file system.I am trying to understand what't the dependency of CGROUPS in mounting the nand filesystem running UBIFS.If I enable the CGROUPS then I can't execute the /sbin/init of the mounted root file system. What I could be doing wrong here?
This happens if I don't turned the CONFIG_CGROUPS=y in kernel.
Zone ranges:
DMA [mem 0x0000000000000000-0x000000001fffffff]
Normal empty
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000001fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000001fffffff]
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: ubi.mtd=1 rootfstype=ubifs root=ubi0:discofs rw console=tty S1,115200 serno=91101316 sku=700064090C hwrev==00:90:5E:12:32:05 mac1=00:90:5E:1 2:32:06 mac2=00:90:5E:12:32:07 mac3=00:90:5E:12:32:08 mac4=00:90:5E:12:32:09
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 511944K/524288K available (5972K kernel code, 272K rwdata, 1192K rodata, 192K init, 128K bss, 12344K reserved, 0K cma-reserved)
Kernel virtual memory layout:
* 0xfffdf000..0xfffff000 : fixmap
* 0xfdffe000..0xfe000000 : early ioremap
* 0xe1000000..0xfdffe000 : vmalloc & ioremap
NR_IRQS:512 nr_irqs:512 16
IPIC (128 IRQ sources) at e1000700
clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x12049cd416, max_id le_ns: 440795202745 ns
clocksource: timebase mult[ccccccd] shift[24] registered
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
devtmpfs: initialized
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645 041785100000 ns
futex hash table entries: 256 (order: -1, 3072 bytes)
NET: Registered protocol family 16
PCI: Probing PCI hardware
Freescale Elo series DMA driver
vgaarb: loaded
SCSI subsystem initialized
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti#l inux.it>
PTP clock support registered
clocksource: Switched to clocksource timebase
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 2, 16384 bytes)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
UDP hash table entries: 256 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
fsl-elo-dma e00082a8.dma: #0 (fsl,elo-dma-channel), irq 71
fsl-elo-dma e00082a8.dma: #1 (fsl,elo-dma-channel), irq 71
fsl-elo-dma e00082a8.dma: #2 (fsl,elo-dma-channel), irq 71
fsl-elo-dma e00082a8.dma: #3 (fsl,elo-dma-channel), irq 71
workingset: timestamp_bits=30 max_order=17 bucket_order=0
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered
jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
SGI XFS with security attributes, no debug enabled
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
Oops: Exception in kernel mode, sig: 4 [#1]
disco
Modules linked in:
CPU: 0 PID: 8 Comm: kdevtmpfs Not tainted 4.12.28-disco-standard #1
task: df41b580 task.stack: df438000
NIP: c0100000 LR: c00fb708 CTR: c00b462c
REGS: df439c50 TRAP: 0700 Not tainted (4.12.28-disco-standard)
MSR: 00089032 <EE,ME,IR,DR,RI>
CR: 880ad872 XER: 00000000
GPR00: c00fb708 df439d00 df41b580 df69c170 00000000 df69c170 00000002 00000000
GPR08: 00000000 00000007 00000001 0e4e1c2f 280ad872 00000000 c0040514 df41d120
GPR16: 00000000 014000c0 00000007 00000000 00000000 0000002f 00000000 c078c840
GPR24: fffff000 00000000 00000000 df69c1cc df69c170 df41fd18 00000000 df69c170
NIP [c0100000] iput+0x24/0x1c8
LR [c00fb708] d_delete+0xb8/0xfc
Call Trace:
[df439d00] [c00fb000] dentry_unlink_inode+0xf0/0x164 (unreliable)
[df439d20] [c00fb708] d_delete+0xb8/0xfc
[df439d40] [c00eecc4] vfs_unlink+0x1b8/0x210
[df439d70] [c03c4f2c] handle_remove+0x1b0/0x324
[df439e50] [c03c5200] devtmpfsd+0x160/0x34c
[df439f00] [c0040650] kthread+0x13c/0x140
[df439f40] [c00103f0] ret_from_kernel_thread+0x5c/0x64
Instruction dump:
39200001 7d234b78 4e800020 9421ffe0 93e1001c 7c7f1b79 41820094 7c0802a6
93a10014 90010024 93c10018 813f0058 <27051956> 9188a4bd 5d2e9c29 0039c48b
---[ end trace df78166105247941 ]---
My guess is you have a systemd-based system. In which case, init is provided by systemd, so is population of /dev. Systemd, in turn, makes extensive use of CGROUPS.
The ubifs/devtmpfs link is not a direct one, IMO. ubifs could be used without it too as long as /dev is correctly populated.

AWS ec2 putty issue

I was able to connect an EC2 instance through SSH
However, it keeps fail and timed out after i followed to step 3 of below article
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SSL-on-an-instance.html
Literally, I have just installed ssl_mod and reboot the instance, afterwards the instance cannot be accessed again by putty.
And There is no way to access the EC2 instance anymore.
Any way to solve this issue?
-----System log------
[ 0.000000] Linux version 4.14.101-75.76.amzn1.x86_64 (mockbuild#gobi-build-60003) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC)) #1 SMP Tue Feb 19 18:14:51 UTC 2019
[ 0.000000] Command line: root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295 LANG=en_US.UTF-8 KEYTABLE=us
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable
[ 0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.7 present.
[ 0.000000] DMI: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[ 0.000000] Hypervisor detected: Xen HVM
[ 0.000000] Xen version 4.2.
[ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
[ 0.000000] You might have to change the root device
[ 0.000000] from /dev/hd[a-d] to /dev/xvd[a-d]
[ 0.000000] in your root= kernel command line option
[ 0.000000] tsc: Fast TSC calibration using PIT
[ 0.000000] e820: last_pfn = 0x40000 max_arch_pfn = 0x400000000
[ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.000000] found SMP MP-table at [mem 0x000fbc50-0x000fbc5f] mapped at [ffffffffff240c50]
[ 0.000000] RAMDISK: [mem 0x37087000-0x37feffff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000EA020 000024 (v02 Xen )
[ 0.000000] ACPI: XSDT 0x00000000FC00E2A0 000054 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: FACP 0x00000000FC00DF60 0000F4 (v04 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: DSDT 0x00000000FC0021C0 00BD19 (v02 Xen HVM 00000000 INTL 20090123)
[ 0.000000] ACPI: FACS 0x00000000FC002180 000040
[ 0.000000] ACPI: FACS 0x00000000FC002180 000040
[ 0.000000] ACPI: APIC 0x00000000FC00E060 0000D8 (v02 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: HPET 0x00000000FC00E1B0 000038 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: WAET 0x00000000FC00E1F0 000028 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: SSDT 0x00000000FC00E220 000031 (v02 Xen HVM 00000000 INTL 20090123)
[ 0.000000] ACPI: SSDT 0x00000000FC00E260 000031 (v02 Xen HVM 00000000 INTL 20090123)
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000003fffffff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x3ffde000-0x3fffffff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.000000] DMA32 [mem 0x0000000001000000-0x000000003fffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009dfff]
[ 0.000000] node 0: [mem 0x0000000000100000-0x000000003fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000003fffffff]
[ 0.000000] ACPI: PM-Timer IO Port: 0xb008
[ 0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 low level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 low level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level)
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[ 0.000000] smpboot: Allowing 15 CPUs, 14 hotplug CPUs
[ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff]
[ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[ 0.000000] e820: [mem 0x40000000-0xfbffffff] available for PCI devices
[ 0.000000] Booting paravirtualized kernel on Xen HVM
[ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:15 nr_cpu_ids:15 nr_node_ids:1
[ 0.000000] percpu: Embedded 44 pages/cpu #ffff88803e200000 s140120 r8192 d31912 u262144
[ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 257928
[ 0.000000] Policy zone: DMA32
[ 0.000000] Kernel command line: root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295 LANG=en_US.UTF-8 KEYTABLE=us
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Memory: 988260K/1048180K available (8204K kernel code, 1920K rwdata, 2768K rodata, 2044K init, 3876K bss, 59920K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=15, Nodes=1
[ 0.000000] Kernel/User page tables isolation: enabled
[ 0.000000] ftrace: allocating 25766 entries in 101 pages
[ 0.004000] Hierarchical RCU implementation.
[ 0.004000] RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=15.
[ 0.004000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=15
[ 0.004000] NR_IRQS: 524544, nr_irqs: 952, preallocated irqs: 16
[ 0.004000] xen:events: Using 2-level ABI
[ 0.004000] xen:events: Xen HVM callback vector for event delivery is enabled
[ 0.004000] Console: colour VGA+ 80x25
[ 0.004000] console [tty1] enabled
[ 0.004000] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
[ 0.004000] console [ttyS0] enabled
[ 0.004000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 30580167144 ns
[ 0.008000] tsc: Fast TSC calibration using PIT
[ 0.020004] tsc: Detected 2399.921 MHz processor
[ 0.024009] Calibrating delay loop (skipped), value calculated using timer frequency.. 4800.12 BogoMIPS (lpj=9600256)
[ 0.032003] pid_max: default: 32768 minimum: 301
[ 0.036018] ACPI: Core revision 20170728
[ 0.046999] ACPI: 3 ACPI AML tables successfully acquired and loaded
[ 0.052025] Security Framework initialized
[ 0.056038] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.060085] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.064019] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.072006] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.076233] mce: CPU supports 2 MCE banks
[ 0.080026] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 1024
[ 0.084002] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 1024, 1GB 4
[ 0.092004] Spectre V2 : Mitigation: Full generic retpoline
[ 0.096002] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[ 0.100005] Speculative Store Bypass: Vulnerable
[ 0.113770] smpboot: Max logical packages: 15
[ 0.116757] x2apic: IRQ remapping doesn't support X2APIC mode
[ 0.120003] Switched APIC routing to physical flat.
[ 0.126535] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[ 0.174541] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.180019] installing Xen timer for CPU 0
[ 0.188076] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2676 v3 # 2.40GHz (family: 0x6, model: 0x3f, stepping: 0x2)
[ 0.192033] cpu 0 spinlock event irq 53
[ 0.195817] Performance Events: unsupported p6 CPU model 63 no PMU driver, software events only.
[ 0.196036] Hierarchical SRCU implementation.
[ 0.200295] NMI watchdog: Perf event create on CPU 0 failed with -2
[ 0.204006] NMI watchdog: Perf NMI watchdog permanently disabled
[ 0.208185] smp: Bringing up secondary CPUs ...
[ 0.211932] smp: Brought up 1 node, 1 CPU
[ 0.212008] smpboot: Total of 1 processors activated (4800.12 BogoMIPS)
[ 0.216442] devtmpfs: initialized
[ 0.219675] x86/mm: Memory block size: 128MB
[ 0.220224] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.224020] futex hash table entries: 4096 (order: 6, 262144 bytes)
[ 0.228273] NET: Registered protocol family 16
[ 0.232242] cpuidle: using governor ladder
[ 0.236007] cpuidle: using governor menu
[ 0.239387] ACPI: bus type PCI registered
[ 0.240005] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 0.244347] PCI: Using configuration type 1 for base access
[ 0.250764] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.252194] ACPI: Added _OSI(Module Device)
[ 0.256011] ACPI: Added _OSI(Processor Device)
[ 0.260005] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 0.264007] ACPI: Added _OSI(Processor Aggregator Device)
[ 0.271170] ACPI: Interpreter enabled
[ 0.272019] ACPI: (supports S0 S4 S5)
[ 0.275388] ACPI: Using IOAPIC for interrupt routing
[ 0.276049] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.280422] ACPI: Enabled 2 GPEs in block 00 to 0F
[ 0.334308] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 0.336012] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
[ 0.340012] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[ 0.344020] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[ 0.349096] acpiphp: Slot [0] registered
[ 0.353056] acpiphp: Slot [3] registered
[ 0.356406] acpiphp: Slot [4] registered
[ 0.359996] acpiphp: Slot [5] registered
[ 0.360414] acpiphp: Slot [6] registered
[ 0.363957] acpiphp: Slot [7] registered
[ 0.364418] acpiphp: Slot [8] registered
[ 0.368008] acpiphp: Slot [9] registered
[ 0.371580] acpiphp: Slot [10] registered
[ 0.372415] acpiphp: Slot [11] registered
[ 0.375900] acpiphp: Slot [12] registered
[ 0.376411] acpiphp: Slot [13] registered
[ 0.380000] acpiphp: Slot [14] registered
[ 0.380427] acpiphp: Slot [15] registered
[ 0.384079] acpiphp: Slot [16] registered
[ 0.387800] acpiphp: Slot [17] registered
[ 0.388411] acpiphp: Slot [18] registered
[ 0.392000] acpiphp: Slot [19] registered
[ 0.392412] acpiphp: Slot [20] registered
[ 0.395992] acpiphp: Slot [21] registered
[ 0.396419] acpiphp: Slot [22] registered
[ 0.400000] acpiphp: Slot [23] registered
[ 0.400406] acpiphp: Slot [24] registered
[ 0.403896] acpiphp: Slot [25] registered
[ 0.404408] acpiphp: Slot [26] registered
[ 0.408261] acpiphp: Slot [27] registered
[ 0.411817] acpiphp: Slot [28] registered
[ 0.412417] acpiphp: Slot [29] registered
[ 0.416044] acpiphp: Slot [30] registered
[ 0.419707] acpiphp: Slot [31] registered
[ 0.420356] PCI host bridge to bus 0000:00
[ 0.423606] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
[ 0.424009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]
[ 0.428008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[ 0.432006] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfbffffff window]
[ 0.436005] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.447197] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7]
[ 0.448006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6]
[ 0.452004] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177]
[ 0.456004] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376]
[ 0.460733] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
[ 0.460733] * this clock source is slow. Consider trying other clock sources
[ 0.465592] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI
[ 0.476152] ACPI: PCI Interrupt Link [LNKA] (IRQs *5 10 11)
[ 0.480222] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[ 0.484220] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[ 0.488237] ACPI: PCI Interrupt Link [LNKD] (IRQs *5 10 11)
[ 0.511891] xen:balloon: Initialising balloon driver
[ 0.520142] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[ 0.524000] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[ 0.524010] pci 0000:00:02.0: vgaarb: bridge control possible
[ 0.528004] vgaarb: loaded
[ 0.531106] EDAC MC: Ver: 3.0.0
[ 0.532587] PCI: Using ACPI for IRQ routing
[ 0.536465] NetLabel: Initializing
[ 0.539477] NetLabel: domain hash size = 128
[ 0.540005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
[ 0.544024] NetLabel: unlabeled traffic allowed by default
[ 0.548042] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[ 0.552017] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[ 0.555868] hpet0: 3 comparators, 64-bit 62.500000 MHz counter
[ 0.558213] clocksource: Switched to clocksource xen
[ 0.568083] VFS: Disk quotas dquot_6.6.0
[ 0.573690] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.581819] pnp: PnP ACPI init
[ 0.586116] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved
[ 0.594537] system 00:01: [io 0x08a0-0x08a3] has been reserved
[ 0.601500] system 00:01: [io 0x0cc0-0x0ccf] has been reserved
[ 0.608222] system 00:01: [io 0x04d0-0x04d1] has been reserved
[ 0.612991] system 00:07: [io 0x10c0-0x1141] has been reserved
[ 0.617641] system 00:07: [io 0xb044-0xb047] has been reserved
[ 0.641810] pnp: PnP ACPI: found 8 devices
[ 0.651918] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[ 0.659212] NET: Registered protocol family 2
[ 0.662987] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.668084] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[ 0.672829] TCP: Hash tables configured (established 8192 bind 8192)
[ 0.677593] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 0.684367] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 0.689599] NET: Registered protocol family 1
[ 0.693893] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 0.698659] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[ 0.703208] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[ 0.713135] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[ 0.723220] Unpacking initramfs...
[ 0.864063] random: fast init done
[ 0.968913] Freeing initrd memory: 15780K
[ 0.974092] audit: initializing netlink subsys (disabled)
[ 0.981075] Initialise system trusted keyrings
[ 0.984736] random: get_random_bytes called from key_alloc+0x209/0x430 with crng_init=1
[ 0.990943] Key type blacklist registered
[ 0.994506] audit: type=2000 audit(1551756108.573:1): state=initialized audit_enabled=0 res=1
[ 1.000910] workingset: timestamp_bits=36 max_order=18 bucket_order=0
[ 1.006273] zbud: loaded
[ 1.284352] Key type asymmetric registered
[ 1.287742] Asymmetric key parser 'x509' registered
[ 1.291498] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 1.297323] io scheduler noop registered (default)
[ 1.301463] xen:grant_table: Grant tables using version 1 layout
[ 1.305870] Grant table initialized
[ 1.309041] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
[ 1.313703] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 1.346384] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 1.352887] xen_netfront: Initialising Xen virtual ethernet driver
[ 1.358641] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[ 1.366855] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 1.370698] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 1.375506] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[ 1.382858] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
[ 1.387621] rtc_cmos 00:02: alarms up to one day, 114 bytes nvram, hpet irqs
[ 1.392869] hidraw: raw HID events driver (C) Jiri Kosina
[ 1.397109] NET: Registered protocol family 17
[ 1.400947] sched_clock: Marking stable (1400624129, 0)->(2253558423, -852934294)
[ 1.407324] registered taskstats version 1
[ 1.411259] Loading compiled-in X.509 certificates
[ 1.415069] zswap: loaded using pool lzo/zbud
[ 1.520120] xenbus_probe_frontend: Device with no driver: device/vbd/51712
[ 1.525300] rtc_cmos 00:02: setting system clock to 2019-03-05 03:21:49 UTC (1551756109)
[ 1.533509] Freeing unused kernel memory: 2044K
[ 1.544082] Write protecting the kernel read-only data: 14336k
[ 1.548953] Freeing unused kernel memory: 2016K
[ 1.554984] Freeing unused kernel memory: 1328K
[ 1.575051] device-mapper: uevent: version 1.0.3
[ 1.579046] device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel#redhat.com
[ 1.589941] udevd[853]: starting version 173
[ 1.655508] SCSI subsystem initialized
[ 1.690059] Invalid max_queues (4), will use default max: 1.
[ 1.720512] scsi host0: ata_piix
[ 1.738747] blkfront: xvda: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
[ 1.747115] scsi host1: ata_piix
[ 1.750384] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc100 irq 14
[ 1.755294] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc108 irq 15
[ 1.785880] xvda: xvda1
growroot: NOCHANGE: disk=/dev/xvda partition=1: size=16773086, it cannot be grown
[ 1.992025] tsc: Refined TSC clocksource calibration: 2400.000 MHz
[ 1.996677] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns
[ 2.007196] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
[ 2.017844] dracut: Mounted root filesystem /dev/xvda1
[ 2.054536] dracut: Switching root
[ 2.102376] random: init: uninitialized urandom read (12 bytes read)
Couple of questions:
1. What OS are you using? I'm assuming Amazon Linux but want to confirm...
2. Did you do anything to change security groups in the Management Console? What are your security group / VPC config?
3. Are you shutting down the instance then turning it back on or did you reboot? How did you do this (either command or management console action)?
4. In the management console can you get the system log and paste here?
It is likely:
1. You configured a firewall or your security groups are not right.
2. You shut down an instance with an auto-assigned public IP. In this case the IP will change.
3. You did something that nuked a system service which is preventing the machine from booting.

Kernel panic on linux-omap 4.3.0-rc5 on IGEPv2 board

I'm trying to run the latest linux-omap kernel Linux-4.3.0-rc5-12674-g80923cf8e that I got from the official linux-omap GIT.
I compiled it with the multi_v7_defconfig, disabling unneeded boards and adding some debug options like earlyprintk.
I am now stuck with a kernel panic and I can't find what the problem is.
The board is IGEPv2 and I am using the omap3-igep0020.dts device tree file.
Kernel is loaded using the latest u-boot, if it helps.
Here's the bootlog. Any idea ?
## Booting kernel from Legacy Image at 80008000 ...
Image Name: Linux-4.3.0-rc5-12674-g80923cf8e
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 4945528 Bytes = 4.7 MiB
Load Address: 80008000
Entry Point: 80008000
Verifying Checksum ... OK
## Flattened Device Tree blob at 81f00000
Booting using the fdt blob at 0x81f00000
Loading Kernel Image ... OK
Loading Device Tree to 8ffec000, end 8ffff0a2 ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.3.0-rc5-12674-g80923cf8eb3b (b#b43) (gcc version 7.2.0 (Ubuntu/Linaro 7.2.0-6ubuntu1) ) #2 SMP Wed Feb 28 00:49:54 EST 2018
[ 0.000000] CPU: ARMv7 Processor [411fc083] revision 3 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[ 0.000000] Machine model: IGEPv2 Rev. C (TI OMAP AM/DM37x)
[ 0.000000] bootconsole [earlycon0] enabled
[ 0.000000] cma: Reserved 64 MiB at 0x9b800000
[ 0.000000] Memory policy: Data cache writeback
[ 0.000000] On node 0 totalpages: 130816
[ 0.000000] free_area_init_node: node 0, pgdat c09aee00, node_mem_map dfad9000
[ 0.000000] Normal zone: 1024 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 130816 pages, LIFO batch:31
[ 0.000000] CPU: All CPU(s) started in SVC mode.
[ 0.000000] OMAP3430/3530 ES3.1 (l2cache iva sgx neon isp )
[ 0.000000] PERCPU: Embedded 12 pages/cpu #dfa86000 s18572 r8192 d22388 u49152
[ 0.000000] pcpu-alloc: s18572 r8192 d22388 u49152 alloc=12*4096
[ 0.000000] pcpu-alloc: [0] 0
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 129792
[ 0.000000] Kernel command line: mem=512M console=ttyO2,115200n8 debug earlyprintk omapfb.mode=dvi:1024x768MR-16#60 root=/dev/mmcblk0p2 rw rootwait
[ 0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 0.000000] Memory: 442476K/523264K available (6856K kernel code, 370K rwdata, 2264K rodata, 396K init, 295K bss, 15252K reserved, 65536K cma-reserved, 0K highmem)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
[ 0.000000] vmalloc : 0xe0800000 - 0xff000000 ( 488 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)
[ 0.000000] pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
[ 0.000000] modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
[ 0.000000] .text : 0xc0008000 - 0xc08f0590 (9122 kB)
[ 0.000000] .init : 0xc08f1000 - 0xc0954000 ( 396 kB)
[ 0.000000] .data : 0xc0954000 - 0xc09b0980 ( 371 kB)
[ 0.000000] .bss : 0xc09b3000 - 0xc09fcc0c ( 296 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] ⇥ Build-time adjustment of leaf fanout to 32.
[ 0.000000] ⇥ RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=1.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=1
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] IRQ: Found an INTC at 0xfa200000 (revision 4.0) with 96 interrupts
[ 0.000000] Clocking rate (Crystal/Core/MPU): 26.0/332/600 MHz
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[ 0.008697] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.014770] Console: colour dummy device 80x30
[ 0.019439] Calibrating delay loop... 597.60 BogoMIPS (lpj=2988032)
[ 0.084869] pid_max: default: 32768 minimum: 301
[ 0.089874] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.096801] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.105255] CPU: Testing write buffer coherency: ok
[ 0.110809] CPU0: thread -1, cpu 0, socket -1, mpidr 0
[ 0.116394] Setting up static identity map for 0x80008340 - 0x80008398
[ 0.124786] Brought up 1 CPUs
[ 0.127929] SMP: Total of 1 processors activated (597.60 BogoMIPS).
[ 0.134521] CPU: All CPU(s) started in SVC mode.
[ 0.141571] devtmpfs: initialized
[ 0.190948] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 1
[ 0.221343] omap_hwmod: mcbsp2_sidetone using broken dt data from mcbsp
[ 0.229003] omap_hwmod: mcbsp3_sidetone using broken dt data from mcbsp
[ 0.296447] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.311370] pinctrl core: initialized pinctrl subsystem
[ 0.319793] NET: Registered protocol family 16
[ 0.328643] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.338317] cpuidle: using governor ladder
[ 0.342712] cpuidle: using governor menu
[ 0.347473] Reprogramming SDRC clock to 332000000 Hz
[ 0.360107] OMAP GPIO hardware version 2.5
[ 0.385833] omap-gpmc 6e000000.gpmc: GPMC revision 5.0
[ 0.391235] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[ 0.400299] platform 49042000.serial: Cannot lookup hwmod 'uart4'
[ 0.412506] No ATAGs?
[ 0.414703] hw-breakpoint: debug architecture 0x4 unsupported.
[ 0.422790] OMAP DMA hardware revision 4.0
[ 0.427307] Serial: AMBA PL011 UART driver
[ 0.458862] omap-dma-engine 48056000.dma-controller: OMAP DMA engine driver
[ 0.474639] vgaarb: loaded
[ 0.478942] SCSI subsystem initialized
[ 0.483337] libata version 3.00 loaded.
[ 0.488037] usbcore: registered new interface driver usbfs
[ 0.493957] usbcore: registered new interface driver hub
[ 0.499603] usbcore: registered new device driver usb
[ 0.506896] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4#48000000/scm#2000/pinmux#30/pinmux_i2c1_pins, deferring probe
[ 0.520629] omap_i2c 48072000.i2c: bus 1 rev3.3 at 100 kHz
[ 0.526519] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4#48000000/scm#2000/pinmux#30/pinmux_i2c3_pins, deferring probe
[ 0.539611] pps_core: LinuxPPS API ver. 1 registered
[ 0.544860] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti#linux.it>
[ 0.554412] PTP clock support registered
[ 0.558990] EDAC MC: Ver: 3.0.0
[ 0.565734] clocksource: Switched to clocksource 32k_counter
[ 0.593200] NET: Registered protocol family 2
[ 0.598846] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.606323] TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.613128] TCP: Hash tables configured (established 4096 bind 4096)
[ 0.619903] UDP hash table entries: 256 (order: 1, 8192 bytes)
[ 0.626068] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[ 0.632873] NET: Registered protocol family 1
[ 0.638061] RPC: Registered named UNIX socket transport module.
[ 0.644226] RPC: Registered udp transport module.
[ 0.649230] RPC: Registered tcp transport module.
[ 0.654144] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.660919] PCI: CLS 0 bytes, default 64
[ 0.668243] hw perfevents: enabled with armv7_cortex_a8 PMU driver, 5 counters available
[ 0.680969] futex hash table entries: 256 (order: 2, 16384 bytes)
[ 0.707794] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.715759] NFS: Registering the id_resolver key type
[ 0.721191] Key type id_resolver registered
[ 0.725585] Key type id_legacy registered
[ 0.729949] ntfs: driver 2.1.32 [Flags: R/O].
[ 0.735290] jffs2: version 2.2. (NAND) <0xc2><0xa9> 2001-2006 Red Hat, Inc.
[ 0.743438] fuse init (API version 7.23)
[ 0.750671] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[ 0.758483] io scheduler noop registered
[ 0.762634] io scheduler deadline registered
[ 0.767242] io scheduler cfq registered (default)
[ 0.776153] pinctrl-single 48002030.pinmux: 284 pins at pa fa002030 size 568
[ 0.783874] pinctrl-single 48002a00.pinmux: 46 pins at pa fa002a00 size 92
[ 0.791748] Unhandled fault: external abort on non-linefetch (0x1028) at 0xfa0025a0
[ 0.799713] pgd = c0004000
[ 0.802581] [fa0025a0] *pgd=48011452(bad)
[ 0.806823] Internal error: : 1028 [#1] SMP ARM
[ 0.811553] Modules linked in:
[ 0.814819] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc5-12674-g80923cf8eb3b #2
[ 0.823028] Hardware name: Generic OMAP36xx (Flattened Device Tree)
[ 0.829589] task: db074000 ti: db078000 task.ti: db078000
[ 0.835235] PC is at pcs_readw+0x0/0x10
[ 0.839294] LR is at pcs_probe+0x388/0x804
[ 0.843597] pc : [<c02be958>] lr : [<c02c0700>] psr: 20000153
[ 0.843597] sp : db079de0 ip : 00000000 fp : 0000002e
[ 0.855560] r10: 00000000 r9 : 00000002 r8 : 00000000
[ 0.861053] r7 : 00000001 r6 : 00000000 r5 : 00000002 r4 : db2d4e10
[ 0.867828] r3 : c02be958 r2 : db21fc00 r1 : 00000002 r0 : fa0025a0
[ 0.874664] Flags: nzCv IRQs on FIQs off Mode SVC_32 ISA ARM Segment none
[ 0.882171] Control: 10c5387d Table: 80004019 DAC: 00000051
[ 0.888183] Process swapper/0 (pid: 1, stack limit = 0xdb078218)
[ 0.894439] Stack: (0xdb079de0 to 0xdb07a000)
[ 0.899017] 9de0: c0824d64 dfac90a0 c0974cd8 00000000 00000002 dfac90a0 0000000c c0824fcc
[ 0.907531] 9e00: db2d4ea8 00000000 db161c00 00000000 db2d4e40 db2d4e28 db2da9b0 db2da9b0
[ 0.916046] 9e20: c09b082f c0156d78 db2da9b0 00000000 00000000 c08302ec 00000001 c0156f34
[ 0.924560] 9e40: db163cd0 c08302ec 00000001 000000ec db163cd0 db2da7d0 db2da9b0 db163cd0
[ 0.933074] 9e60: c08302ec db161c10 ffffffed c0974cd8 fffffdfb 00000000 000000ec c094d048
[ 0.941589] 9e80: 00000000 c038b544 c09ec6ec db161c10 c0974cd8 00000000 00000000 c0389e2c
[ 0.950103] 9ea0: db161c10 c0974cd8 db161c44 00000000 00000000 c0389f9c 00000000 c0974cd8
[ 0.958587] 9ec0: c0389f18 c03882d0 db00cb5c db162434 c0974cd8 db2d9780 c0987578 c038940c
[ 0.967102] 9ee0: c0824d64 c09b3000 c0974cd8 c0974cd8 c0959bb0 db2db240 c0912288 c038a68c
[ 0.975616] 9f00: c0959bb0 c0959bb0 db2db240 c000985c 00000000 c014e320 c014ddd8 db11ab80
[ 0.984130] 9f20: 00000000 c096a858 00000000 c06aad30 dfeff132 c00504d0 cccccccd c08bebb8
[ 0.992645] 9f40: c085f958 00000006 00000006 c07f9410 c096a840 c09b3000 c09b3000 c0939830
[ 1.001159] 9f60: c09b3000 c09b3000 c0939834 c093983c 00000007 c08f1dc0 00000006 00000006
[ 1.009674] 9f80: 00000000 c08f159c 00000000 c0683b30 00000000 00000000 00000000 00000000
[ 1.018188] 9fa0: 00000000 c0683b38 00000000 c000f638 00000000 00000000 00000000 00000000
[ 1.026702] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1.035186] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 fdfeffff f7ff777f
[ 1.043731] [<c02be958>] (pcs_readw) from [<c02c0700>] (pcs_probe+0x388/0x804)
[ 1.051269] [<c02c0700>] (pcs_probe) from [<c038b544>] (platform_drv_probe+0x48/0xa0)
[ 1.059417] [<c038b544>] (platform_drv_probe) from [<c0389e2c>] (driver_probe_device+0x1ec/0x2d8)
[ 1.068664] [<c0389e2c>] (driver_probe_device) from [<c0389f9c>] (__driver_attach+0x84/0x88)
[ 1.077423] [<c0389f9c>] (__driver_attach) from [<c03882d0>] (bus_for_each_dev+0x4c/0x9c)
[ 1.085937] [<c03882d0>] (bus_for_each_dev) from [<c038940c>] (bus_add_driver+0x18c/0x210)
[ 1.094543] [<c038940c>] (bus_add_driver) from [<c038a68c>] (driver_register+0x78/0xf4)
[ 1.102874] [<c038a68c>] (driver_register) from [<c000985c>] (do_one_initcall+0x8c/0x1d8)
[ 1.111389] [<c000985c>] (do_one_initcall) from [<c08f1dc0>] (kernel_init_freeable+0x158/0x1e8)
[ 1.120452] [<c08f1dc0>] (kernel_init_freeable) from [<c0683b38>] (kernel_init+0x8/0xe8)
[ 1.128875] [<c0683b38>] (kernel_init) from [<c000f638>] (ret_from_fork+0x14/0x3c)
[ 1.136779] Code: e5d00000 e6ef0070 f57ff04f e12fff1e (e1d000b0)
[ 1.143157] ---[ end trace 51c6d6f40dd2a6d2 ]---
[ 1.148040] In-band Error seen by MPU at address 0
[ 1.153167] ------------[ cut here ]------------
[ 1.158020] WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_smx.c:166 omap3_l3_app_irq+0xc4/0x118()
[ 1.167236] Modules linked in:
[ 1.170471] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G D 4.3.0-rc5-12674-g80923cf8eb3b #2
[ 1.179962] Hardware name: Generic OMAP36xx (Flattened Device Tree)
[ 1.186523] [<c00174f0>] (unwind_backtrace) from [<c0012ff4>] (show_stack+0x10/0x14)
[ 1.194580] [<c0012ff4>] (show_stack) from [<c028eea8>] (dump_stack+0x80/0x90)
[ 1.202117] [<c028eea8>] (dump_stack) from [<c00365f0>] (warn_slowpath_common+0x80/0xb0)
[ 1.210540] [<c00365f0>] (warn_slowpath_common) from [<c00366bc>] (warn_slowpath_null+0x1c/0x24)
[ 1.219665] [<c00366bc>] (warn_slowpath_null) from [<c02b5ec0>] (omap3_l3_app_irq+0xc4/0x118)
[ 1.228546] [<c02b5ec0>] (omap3_l3_app_irq) from [<c00787e0>] (handle_irq_event_percpu+0x48/0x138)
[ 1.237884] [<c00787e0>] (handle_irq_event_percpu) from [<c0078918>] (handle_irq_event+0x48/0x6c)
[ 1.247100] [<c0078918>] (handle_irq_event) from [<c007b584>] (handle_level_irq+0xac/0x144)
[ 1.255798] [<c007b584>] (handle_level_irq) from [<c0077f24>] (generic_handle_irq+0x24/0x34)
[ 1.264587] [<c0077f24>] (generic_handle_irq) from [<c00781b0>] (__handle_domain_irq+0x5c/0xb4)
[ 1.273620] [<c00781b0>] (__handle_domain_irq) from [<c0013bd4>] (__irq_svc+0x54/0x70)
[ 1.281860] [<c0013bd4>] (__irq_svc) from [<c0038028>] (do_exit+0x3e0/0x988)
[ 1.289215] [<c0038028>] (do_exit) from [<c0013324>] (die+0x32c/0x330)
[ 1.296020] [<c0013324>] (die) from [<c0009380>] (do_DataAbort+0xb4/0xb8)
[ 1.303100] [<c0009380>] (do_DataAbort) from [<c0013b60>] (__dabt_svc+0x40/0x60)
[ 1.310821] Exception stack(0xdb079d90 to 0xdb079dd8)
[ 1.316101] 9d80: fa0025a0 00000002 db21fc00 c02be958
[ 1.324615] 9da0: db2d4e10 00000002 00000000 00000001 00000000 00000002 00000000 0000002e
[ 1.333129] 9dc0: 00000000 db079de0 c02c0700 c02be958 20000153 ffffffff
[ 1.340026] [<c0013b60>] (__dabt_svc) from [<c02be958>] (pcs_readw+0x0/0x10)
[ 1.347381] [<c02be958>] (pcs_readw) from [<c02c0700>] (pcs_probe+0x388/0x804)
[ 1.354888] [<c02c0700>] (pcs_probe) from [<c038b544>] (platform_drv_probe+0x48/0xa0)
[ 1.363067] [<c038b544>] (platform_drv_probe) from [<c0389e2c>] (driver_probe_device+0x1ec/0x2d8)
[ 1.372283] [<c0389e2c>] (driver_probe_device) from [<c0389f9c>] (__driver_attach+0x84/0x88)
[ 1.381072] [<c0389f9c>] (__driver_attach) from [<c03882d0>] (bus_for_each_dev+0x4c/0x9c)
[ 1.389587] [<c03882d0>] (bus_for_each_dev) from [<c038940c>] (bus_add_driver+0x18c/0x210)
[ 1.398193] [<c038940c>] (bus_add_driver) from [<c038a68c>] (driver_register+0x78/0xf4)
[ 1.406524] [<c038a68c>] (driver_register) from [<c000985c>] (do_one_initcall+0x8c/0x1d8)
[ 1.415039] [<c000985c>] (do_one_initcall) from [<c08f1dc0>] (kernel_init_freeable+0x158/0x1e8)
[ 1.424072] [<c08f1dc0>] (kernel_init_freeable) from [<c0683b38>] (kernel_init+0x8/0xe8)
[ 1.432495] [<c0683b38>] (kernel_init) from [<c000f638>] (ret_from_fork+0x14/0x3c)
[ 1.440368] ---[ end trace 51c6d6f40dd2a6d3 ]---
[ 1.445343] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 1.445343]
[ 1.454925] Rebooting in 300
I'm confused because there seem to be several stacktraces.
Also I wasn't able to get a more detailed stacktrace despite compiling the debug info in the kernel (but I suspect the debug info is not actually there as the file is surprisingly small)
The last supported kernel from the manufacturer ISEE for this board is 2.6.37 which is horribly old. There does not seem to be any problem with running more recent kernels but it's possible the device tree files have not been updated for long... I'm just looking for pointers of where this error might come from and I'll try to fix it.
Thanks !
I don't know what pcs is, but looks like it reads from wrong memory region.
[ 1.340026] [<c0013b60>] (__dabt_svc) from [<c02be958>] (pcs_readw+0x0/0x10)
[ 1.347381] [<c02be958>] (pcs_readw) from [<c02c0700>] (pcs_probe+0x388/0x804)
[ 1.354888] [<c02c0700>] (pcs_probe) from [<c038b544>] (platform_drv_probe+0x48/0xa0)
The reason why you getting page fault is an data abort in SVC (kernel) mode. That might happens because:
There is no any physical memory or IO space under physical addresses, which was specified in device tree for PCS driver. (possibly the used dtb is not compatible with the soc).
The memory which was accessed by pcs_readw is not mapped at all (or mapped with wrong attributes).
To figure out you need to check where virtual addresses, which are accessed by pcs_readw was mapped. What physical addresses are used. Check soc manual to be sure that this physical address fits in soc memory map.
Alex above nailed it !
I was writing an answer because I also found the problem in the meantime.
Basically, the board I am using, IGEPv2, was released in several revisions which use different CPUs. Apparently I have an early one (revision RB1) which uses an OMAP3530 CPU whereas the most recent ones are using a DM3730 CPU.
So, in the omap3-igep.dtsi file (used in the dts) I changed the include from omap36xx.dtsi to omap34xx.dtsi and boom, it boots successfully.
Wifi should not be working (because they changed the GPIO of the RESET pins of the Wifi and Bluetooth controller between the revisions) but now the system boots properly so it's just a matter of tracking the changes between the versions and editing the DTS files accordingly.
Hope this helps someone in the same boat - not sure if I'll be able to send these changes upstream so at least if someone Google the issue they'll find it !

unable to attach ubi to mtd partition

We are using MT29F8G08ABABA in our DM365 based custom board.
We are using NAND boot.
RootFS is in UBIFS format,
Below is basic achitecture of the NAND.
I am using following commands to create ubifs,
$(UBIFS_PATH)/mkfs.ubifs -r $(TARGET_FS) -m 4096 -e 520192 -c 75 -o ubifs.img
$(UBIFS_PATH)/ubinize -o $(SYSTEM_CFG)_$(HARDWARE_CFG)_ubifs -m 4096 -p 512KiB $(HARDWARE_CFG)_ubinize.cfg
dm368_ubinize.cfg content is as follows,
[ubifs]
mode=ubi
image=ubifs.img
vol_id=0
vol_size=37888KiB
vol_type=dynamic
vol_name=rootfs
vol_flags=autoresize
Then i put this ubifs to NAND using following commands,
loady 0x82000000 ---> ubifs is loaded to RAM
nand scrub 0x1800000 0x2800000
nand erase 0x1800000 0x2800000
nand write 0x82000000 0x1800000 0x2800000
and when i boot the kernel there is error while detecting UBIFS from kernel,
Below is kernel log for the same,
[ 0.000000] Linux version 2.6[ 0.000000] Linux version 2.6.37_IPNC_DM368_5.1.0 (still-image#stillimage-VirtualBox) (gcc version 4.3.3 (GCC) ) #1 PREEMPT Sat Jan 4 11:09:29 IST 2014
[ 0.000000] CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00053177
[ 0.000000] CPU: VIVT data cache, VIVT instruction cache
[ 0.000000] Machine: DaVinci DM36x IPNC
[ 0.000000] Memory policy: ECC disabled, Data cache writeback
[ 0.000000] DaVinci dm36x_rev1.2 variant 0x8
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 12192
[ 0.000000] Kernel command line: mem=48M console=ttyS0,115200n8 noinitrd ip=off rw ubi.mtd=3,4096 rootfstype=ubifs root=ubi0:rootfs cmemk.phys_start=0x83000000 cmemk.phys_end=0x88000000 cmemk.phys_start_1=0x00001000 cmemk.phys_end_1=0x00008000 cmemk.pools_1=1x28672 cmemk.allowOverlap=1 cmemk.useHeapIfPoolUnavailable=1 nohz=off highres=off clocksource=acpi_pm lpj=1077248
[ 0.000000] PID hash table entries: 256 (order: -2, 1024 bytes)
[ 0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
[ 0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Memory: 48MB = 48MB total
[ 0.000000] Memory: 44772k/44772k available, 4380k reserved, 0K highmem
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
[ 0.000000] DMA : 0xff000000 - 0xffe00000 ( 14 MB)
[ 0.000000] vmalloc : 0xc3800000 - 0xfea00000 ( 946 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xc3000000 ( 48 MB)
[ 0.000000] modules : 0xbf000000 - 0xc0000000 ( 16 MB)
[ 0.000000] .init : 0xc0008000 - 0xc002c000 ( 144 kB)
[ 0.000000] .text : 0xc002c000 - 0xc0390000 (3472 kB)
[ 0.000000] .data : 0xc0390000 - 0xc03b6600 ( 154 kB)
[ 0.000000] SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] Preemptable hierarchical RCU implementation.
[ 0.000000] RCU-based detection of stalled CPUs is disabled.
[ 0.000000] Verbose stalled-CPUs detection is disabled.
[ 0.000000] NR_IRQS:245
[ 0.000000] Console: colour dummy device 80x30
[ 0.000407] Calibrating delay loop (skipped) preset value.. 215.44 BogoMIPS (lpj=1077248)
[ 0.000494] pid_max: default: 32768 minimum: 301
[ 0.000794] Security Framework initialized
[ 0.001001] Mount-cache hash table entries: 512
[ 0.002188] CPU: Testing write buffer coherency: ok
[ 0.004944] devtmpfs: initialized
[ 0.010877] DaVinci: 8 gpio irqs
[ 0.011931] NET: Registered protocol family 16
[ 0.066045] dm365_rmd_cs_rdk_150_wl_init
[ 0.066124] No entry for the specified index
[ 0.069960] No entry for the specified index
[ 0.192448] bio: create slab <bio-0> at 0
[ 0.199765] SCSI subsystem initialized
[ 0.206445] usbcore: registered new interface driver usbfs
[ 0.208607] usbcore: registered new interface driver hub
[ 0.210484] usbcore: registered new device driver usb
[ 0.215823] vpss vpss: dm365_vpss vpss probed
[ 0.215919] vpss vpss: dm365_vpss vpss probe success
[ 0.222681] Advanced Linux Sound Architecture Driver Version 1.0.23.
[ 0.228358] Switching to clocksource timer0_1
[ 0.356682] NET: Registered protocol family 2
[ 0.357150] IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.358399] TCP established hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.358604] TCP bind hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.358736] TCP: Hash tables configured (established 2048 bind 2048)
[ 0.358784] TCP reno registered
[ 0.358845] UDP hash table entries: 256 (order: 0, 4096 bytes)
[ 0.358939] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[ 0.359791] NET: Registered protocol family 1
[ 0.466150] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.469156] JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[ 0.474898] msgmni has been set to 87
[ 0.481735] io scheduler noop registered (default)
[ 0.484414] CSL: Module install successful, device major num = 254
[ 0.484478] CSL: Module version 0.10.00, built on Dec 13 2013 22:33:02
[ 0.484552] I2C: Module install successful, device major num = 253
[ 0.484609] DMA: Module install successful, device major num = 252
[ 0.484654] DRV: Module install successful
[ 0.484692] DRV: Module built on Dec 13 2013 22:33:04
[ 0.484733] DRV: EDMACC.QUEPRI = 00002777
[ 0.484769] DRV: SYSTEM.MSTPRI0 = 00440011
[ 0.484804] DRV: SYSTEM.MSTPRI1 = 00000444
[ 0.484839] DRV: ISP.BCR = 00000002
[ 0.484874] DRV: SYSTEM.MISC = 00000399
[ 0.484911] EDMAK module: built on Dec 13 2013 at 22:32:35
[ 0.484957] Reference Linux version 2.6.37
[ 0.484996] File /opt/workdir/OV9715_Ver5_0/dvsdk_ipnctools/linuxutils_2_26_02_05/packages/ti/sdo/linuxutils/edma/src/module/edmak.c
[ 0.487539] IRQK module: built on Dec 13 2013 at 22:32:33
[ 0.487612] Reference Linux version 2.6.37
[ 0.487653] File /opt/workdir/OV9715_Ver5_0/dvsdk_ipnctools/linuxutils_2_26_02_05/packages/ti/sdo/linuxutils/irq/src/module/irqk.c
[ 0.490558] irqk initialized
[ 0.490663] CMEMK module: built on Dec 13 2013 at 22:32:30
[ 0.490723] Reference Linux version 2.6.37
[ 0.490762] File /opt/workdir/OV9715_Ver5_0/dvsdk_ipnctools/linuxutils_2_26_02_05/packages/ti/sdo/linuxutils/cmem/src/module/cmemk.c
[ 0.498082] allocated heap buffer 0xc4000000 of size 0x5000000
[ 0.498145] heap fallback enabled - will try heap if pool buffer is not available
[ 0.498205] CMEM Range Overlaps Kernel Physical - allowing overlap
[ 0.498257] CMEM phys_start (0x1000) overlaps kernel (0x80000000 -> 0x83000000)
[ 0.498473] cmemk initialized
[ 0.498551] Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
[ 0.611337] serial8250.0: ttyS0 at MMIO 0x1c20000 (irq = 40) is a 16550A
[ 1.137052] console [ttyS0] enabled
[ 1.211160] serial8250.0: ttyS1 at MMIO 0x1d06000 (irq = 41) is a 16550A
[ 1.254825] brd: module loaded
[ 1.272762] NAND device: Manufacturer ID: 0x2c, Chip ID: 0x38 (Micron )
[ 1.281908] Bad block table not found for chip 0
[ 1.287959] Bad block table not found for chip 0
[ 1.292787] Scanning device for bad blocks
[ 1.729485] Creating 7 MTD partitions on "davinci_nand.0":
[ 1.735177] 0x000000000000-0x000000c00000 : "bootloader"
[ 1.748031] 0x000000c00000-0x000001400000 : "params"
[ 1.760990] 0x000001400000-0x000001800000 : "kernel"
[ 1.773829] 0x000001800000-0x000004000000 : "filesystem1"
[ 1.787411] 0x000004000000-0x000004800000 : "data1"
[ 1.800510] 0x000004800000-0x000007000000 : "filesystem2"
[ 1.813667] 0x000007000000-0x000040000000 : "data2"
[ 1.828113] davinci_nand davinci_nand.0: controller rev. 2.3
[ 1.839036] UBI: attaching mtd3 to ubi0
[ 1.843400] UBI: physical eraseblock size: 262144 bytes (256 KiB)
[ 1.849714] UBI: logical eraseblock size: 256000 bytes
[ 1.855391] UBI: smallest flash I/O unit: 2048
[ 1.860292] UBI: sub-page size: 512
[ 1.864953] UBI: VID header offset: 4096 (aligned 4096)
[ 1.871094] UBI: data offset: 6144
[ 1.877230] UBI error: validate_ec_hdr: bad data offset 8192, expected 6144
[ 1.884383] UBI error: validate_ec_hdr: bad EC header
[ 1.889484] UBI error: ubi_io_read_ec_hdr: validation failed for PEB 0
[ 1.896907] UBI error: ubi_init: cannot attach mtd3
[ 1.905311] spi_davinci spi_davinci.0: Controller at 0xfec66000
[ 1.970163] davinci_mdio davinci_mdio.0: davinci mdio revision 1.4
[ 1.976404] davinci_mdio davinci_mdio.0: no live phy, scanning all
[ 1.983779] davinci_mdio: probe of davinci_mdio.0 failed with error -5
[ 1.993145] i2c /dev entries driver
[ 2.000461] Linux media interface: v0.10
[ 2.005482] Linux video capture interface: v2.00
[ 2.013867] vpbe-osd vpbe-osd: OSD sub device probe success
[ 2.021111] vpbe-venc vpbe-venc: VENC sub device probe success
[ 2.028379] vpbe-v4l2 vpbe-v4l2: vpbe v4l2 device registered
[ 2.034923] Setting default output to Composite
[ 2.039507] Setting default mode to ntsc
[ 2.043823] vpbe-v4l2 vpbe-v4l2: Trying to register VPBE display device.
[ 2.050741] vpbe-v4l2 vpbe-v4l2: layer=c2989a00,layer->video_dev=c2989ae8
[ 2.059118] vpbe-v4l2 vpbe-v4l2: Trying to register VPBE display device.
[ 2.066191] vpbe-v4l2 vpbe-v4l2: layer=c2989c00,layer->video_dev=c2989ce8
[ 2.076313] watchdog watchdog: heartbeat 60 sec
[ 2.090661] Error: Driver 'davinci-mcbsp' is already registered, aborting...
[ 2.102840] asoc: cq93vc-hifi <-> davinci-vcif mapping ok
[ 2.115028] ALSA device list:
[ 2.118092] #0: DaVinci DM365 EVM
[ 2.123853] nf_conntrack version 0.5.0 (699 buckets, 2796 max)
[ 2.133076] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 2.138907] TCP cubic registered
[ 2.142616] NET: Registered protocol family 17
[ 2.163517] UBIFS error (pid 1): ubifs_mount: cannot open "ubi0:rootfs", error -19
[ 2.171615] VFS: Cannot open root device "ubi0:rootfs" or unknown-block(0,0)
[ 2.178710] Please append a correct "root=" boot option; here are the available partitions:
[ 2.187357] 1f00 12288 mtdblock0 (driver?)
[ 2.192678] 1f01 8192 mtdblock1 (driver?)
[ 2.197807] 1f02 4096 mtdblock2 (driver?)
[ 2.203056] 1f03 40960 mtdblock3 (driver?)
[ 2.208184] 1f04 8192 mtdblock4 (driver?)
[ 2.213414] 1f05 40960 mtdblock5 (driver?)
[ 2.218542] 1f06 933888 mtdblock6 (driver?)
[ 2.223740] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 2.232105] Backtrace:
[ 2.234600] Function entered at [<c0030504>] from [<c030ec74>]
[ 2.240524] r7:00008000 r6:c2813000 r5:c00263ec r4:c03b6e00
[ 2.246275] Function entered at [<c030ec5c>] from [<c030ecd4>]
[ 2.252188] Function entered at [<c030ec78>] from [<c0008fdc>]
[ 2.258049] r3:00000000 r2:c2824000 r1:c2825f78 r0:c0331f90
[ 2.263847] Function entered at [<c0008d78>] from [<c0009134>]
[ 2.269707] Function entered at [<c00090a0>] from [<c00084b4>]
[ 2.275614] r5:c002597c r4:c03b6618
[ 2.279248] Function entered at [<c00083a4>] from [<c0046a58>]
[ 2.285156] r5:c00083a4 r4:00000000
Now my doubt is why kernel is reporting physical eraseblock size as 256KiB? from architecture of NAND it looks to be 512KiB.
Any suggestions/pointer for the errors ?
Thank you in advance
As i have mentioned in above answer,
I was not able to create a file or folder because filesystem size and partitions size was same.
So i increased the partition size to 128MB(nand_partition in kernel) and i recreated the UBIFS. I flashed the new kernel and UBIFS in NAND.
And it started working, now i am able to create folder and/or file anywhere in the root file system.
I used below commands to create ubifs,
$(UBIFS_PATH)/mkfs.ubifs -r $(TARGET_FS) -m 2048 -e 258048 -c 506 -o ubifs.img
$(UBIFS_PATH)/ubinize -o $(SYSTEM_CFG)_$(HARDWARE_CFG)_ubifs -m 2048 -p 256KiB -O 2048 $(HARDWARE_CFG)_ubinize.cfg
content of ubinize.cfg is as follows,
[ubifs]
peb=0x40000
min_io_size=0x800
compress_type=zlib
mode=ubi
image=ubifs.img
vol_id=0
vol_size=124MiB
vol_type=dynamic
vol_name=rootfs
vol_flags=autoresize
Some calculations which will help understand
For 256Kib and pagesize 2048
Symbol Meaning Value for 128MB fs
SP PEB Size 256KiB
SL LEB Size 256kib - 2* 2kiB = 252Kib
P Total number of PEBs on the MTD device 128MiB / 256KiB = 512
B Number of PEBs reserved for bad PEB handling 1% of P = 5
O The overhead related to storing EC and VID headers in bytes, i.e. O = SP - SL 4KiB
UBI Overhead = (B + 4) * SP + O * (P - B - 4)
= (5 + 4) * 256Kib + 4 KiB * (512 - 5 - 4)
= 2304Kib + 2012Kib
= 4316 KiB
= 16.859375 PEBs (round to 17)
512 - 17 = 498 peb = 498*256kiB PEBS = 127488KiB availble for user data
we have to use -c <X> where x = 127488Kib / 252kib = 505.9 = 506
Hope this helps someone facing the similar issue.
PEB size (physical erase block) I think needs configuring as part of the ubinize config file, try adding something like this to the file:
[flash]
peb=0x20000
min_io_size=0x800
compress_type=zlib
(I've got a PEB size of 128KiB, so you'll need to change accordingly).
Also you might find UBIFS MTD pages useful - there's a question about bad VID header offset.
Hope this helps.

Resources