PCI Express AER Driver issues on Linux - linux-kernel

I'm debugging a PCIe hardware issue on Linux and I want to enable PCIe AER driver on linux to catch any AER errors reported by my hardware device. I'm following this wiki:
https://www.kernel.org/doc/Documentation/PCI/pcieaer-howto.txt
My syslog shows AER is enabled
0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-108-generic root=UUID=a9f6d189-c13d-485c-a504-ba0aa0127e2e ro quiet splash aerdriver.forceload=y crashkernel=512M-:192M vt.handoff=1
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-108-generic root=UUID=a9f6d189-c13d-485c-a504-ba0aa0127e2e ro quiet splash aerdriver.forceload=y crashkernel=512M-:192M vt.handoff=1
[ 0.640130] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[ 0.661638] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[ 0.678143] acpi PNP0A08:02: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[ 0.694863] acpi PNP0A08:03: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[ 4.747041] acpi PNP0A08:04: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[ 4.751760] acpi PNP0A08:05: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[ 4.758480] acpi PNP0A08:06: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[ 4.763990] acpi PNP0A08:07: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[ 5.463432] pcieport 0000:00:01.1: AER enabled with IRQ 34
[ 5.463450] pcieport 0000:00:07.1: AER enabled with IRQ 35
[ 5.463472] pcieport 0000:00:08.1: AER enabled with IRQ 37
[ 5.463517] pcieport 0000:10:01.1: AER enabled with IRQ 38
[ 5.463547] pcieport 0000:10:07.1: AER enabled with IRQ 39
[ 5.463575] pcieport 0000:10:08.1: AER enabled with IRQ 41
[ 5.463604] pcieport 0000:20:03.1: AER enabled with IRQ 42
[ 5.463635] pcieport 0000:20:07.1: AER enabled with IRQ 44
[ 5.463663] pcieport 0000:20:08.1: AER enabled with IRQ 46
[ 5.463782] pcieport 0000:30:03.1: AER enabled with IRQ 47
[ 5.463811] pcieport 0000:30:07.1: AER enabled with IRQ 49
[ 5.463843] pcieport 0000:30:08.1: AER enabled with IRQ 51
[ 5.463872] pcieport 0000:40:07.1: AER enabled with IRQ 62
[ 5.463895] pcieport 0000:40:08.1: AER enabled with IRQ 64
[ 5.463930] pcieport 0000:50:07.1: AER enabled with IRQ 66
[ 5.463965] pcieport 0000:50:08.1: AER enabled with IRQ 68
[ 5.464000] pcieport 0000:60:07.1: AER enabled with IRQ 70
[ 5.464044] pcieport 0000:60:08.1: AER enabled with IRQ 72
[ 5.464071] pcieport 0000:70:07.1: AER enabled with IRQ 74
[ 5.464099] pcieport 0000:70:08.1: AER enabled with IRQ 76
The hardware device is a Samsung SSD connected to the Root Complex by a PCIe switch
PCIe Topology : Root Complex - <PLDA PCIe switch + FPGA> - Samsung EVO SSD
Unfortunately, I'm seeing a lot of NVMe related errors but no AER errors are outputted.
Jun 26 12:13:54 ndra-Diesel kernel: [ 1080.672606] nvme1n1: p1 p2
Jun 26 12:14:27 ndra-Diesel kernel: [ 1113.542592] nvme nvme1: I/O 832 QID 5 timeout, aborting
Jun 26 12:14:27 ndra-Diesel kernel: [ 1113.542617] nvme nvme1: I/O 833 QID 5 timeout, aborting
Jun 26 12:14:27 ndra-Diesel kernel: [ 1113.542627] nvme nvme1: I/O 834 QID 5 timeout, aborting
Jun 26 12:14:27 ndra-Diesel kernel: [ 1113.542636] nvme nvme1: I/O 835 QID 5 timeout, aborting
Jun 26 12:14:27 ndra-Diesel kernel: [ 1113.542645] nvme nvme1: I/O 872 QID 5 timeout, aborting
Jun 26 12:14:27 ndra-Diesel kernel: [ 1113.542654] nvme nvme1: I/O 873 QID 5 timeout, aborting
Jun 26 12:14:27 ndra-Diesel kernel: [ 1113.542662] nvme nvme1: I/O 874 QID 5 timeout, aborting
Jun 26 12:14:27 ndra-Diesel kernel: [ 1113.542670] nvme nvme1: I/O 875 QID 5 timeout, aborting
Jun 26 12:14:58 ndra-Diesel kernel: [ 1144.262425] nvme nvme1: I/O 832 QID 5 timeout, reset controller
Jun 26 12:15:29 ndra-Diesel kernel: [ 1174.982243] nvme nvme1: I/O 16 QID 0 timeout, reset controller
Jun 26 12:15:40 ndra-Diesel gnome-software[6474]: no app for changed ubuntu-dock#ubuntu.com
I have custom compiled my kernel with following options:
cat /boot/config-4.15.0-108-generic | grep -i PCIE
CONFIG_ACPI_APEI_PCIEAER=y
CONFIG_PCIEAER=y
CONFIG_PCIEAER_INJECT=y
CONFIG_PCIEPORTBUS=y
The lspci output for my Samsung NVMe shows that it has AER Capability:
37:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd Device a801
Flags: bus master, fast devsel, latency 0, IRQ 54, NUMA node 3
Memory at b6500000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
Capabilities: [100] Advanced Error Reporting <----------------------------- SEE THIS
Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158] Power Budgeting <?>
Capabilities: [168] #19
Capabilities: [188] Latency Tolerance Reporting
Capabilities: [190] L1 PM Substates
Kernel driver in use: nvme
Kernel modules: nvme
But the lscpi for PLDA switch doesn't show it has AER Capability
3:00.0 PCI bridge: PLDA XpressSwitch (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 53, NUMA node 3
Bus: primary=33, secondary=34, subordinate=3b, sec-latency=0
Memory behind bridge: b6400000-b65fffff
Capabilities: [80] Express Upstream Port, MSI 00
Capabilities: [e0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [f8] Power Management version 3
Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
Capabilities: [300] #19
Kernel driver in use: pcieport
Kernel modules: shpchp
I have two Questions :
The Samsung NVMe is behind the PLDA switch in the topology and the switch doesn't have AER capability. Can this be the reason I'm not seeing AER errors from the NVMe ?
Do I need to do anything else to enable AER on linux ?

Try adding this to the end of your config file:
pcie_ports=native

Related

UBI failed to attach mtd3

I'm trying to create an operating system that works with UBIFS on SPI Nor flash. I used buildroot. Buildroot created rootfs.ubi and rootfs.ubifs therefore I didn't create any ubifs.img(I thought don't need it). I didn't take any error when compile buildroot.
Buildroot output files:
rootfs.ubi
rootfs.ubifs
sun5i-a13.dtb
u-boot.bin
u-boot-sunxi-with-spl.bin
uImage
Write output file to SPI Nor Flash:
#!/bin/bash
DIR="/home/ubuntu/buildroot-spi-nor/output/images"
sudo sunxi-fel -v -p \
spiflash-write 0x0 ${DIR}/u-boot-sunxi-with-spl.bin \
spiflash-write 0xE0000 ${DIR}/sun5i-a13.dtb \
spiflash-write 0x100000 ${DIR}/uImage \
spiflash-write 0x600000 ${DIR}/rootfs.ubi \
uboot ${DIR}/u-boot-sunxi-with-spl.bin
U-boot configs:
BOOTARGS = ubi.mtd=3 rootfstype=ubifs rw rootwait rootfs=/dev/mtdblock3 loglevel=9
BOOTCOMMAND = run distro_bootcmd;sspi;sf probe;sf read 0x43000000 0xE0000 0x4000;sf read 0x42000000 0x100000 0x2F0000;bootm 0x42000000 - 0x43000000;
MTDIDS_DEFAULT = nor0=spi0.0
MTDPARTS_DEFAULT = mtdparts=spi0.0:0x000E0000(boot),0x00020000(dtb),0x00400000(kernel),0x00a00000(rootfs)
Part of SPI flash of .dts file:
&spi0 {
pinctrl-names = "default";
pinctrl-0 = <&spi0_pins_a>, <&spi0_cs0_pins_a>;
status = "okay";
flash#0 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "mxicy,mx25l12805d", "jedec,spi-nor";
spi-max-frequency = <1000000>;
reg = <0>;
mode = <0>;
partition#0 {
label = "bootloader";
reg = <0x0 0xE0000>;
// read-only;
};
partition#E0000 {
label = "dtb";
reg = <0xE0000 0x20000>;
// read-only;
};
partition#100000 {
label = "kernel";
reg = <0x100000 0x500000>;
};
partition#600000 {
label = "rootfs";
reg = <0x600000 0xA00000>;
};
};
};
I have 4 mdt partition and I strive mount ubifs to rootfs partition.
After writing rootfs.ubifs to 0x400000 adress of SPI Nor Flash Dmesg output
[ 0.045841] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
[ 0.046046] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.046096] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[ 0.046200] pinctrl core: initialized pinctrl subsystem
[ 0.047135] NET: Registered protocol family 16
[ 0.047738] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.048490] thermal_sys: Registered thermal governor 'step_wise'
[ 0.048852] hw-breakpoint: debug architecture 0x4 unsupported.
[ 0.063346] vgaarb: loaded
[ 0.063718] SCSI subsystem initialized
[ 0.063873] usbcore: registered new interface driver usbfs
[ 0.063938] usbcore: registered new interface driver hub
[ 0.064001] usbcore: registered new device driver usb
[ 0.065175] clocksource: Switched to clocksource timer
[ 0.065959] simple-framebuffer 5fe89000.framebuffer: framebuffer at 0x5fe89000, 0x177000 bytes, mapped to 0x(ptrval)
[ 0.066018] simple-framebuffer 5fe89000.framebuffer: format=x8r8g8b8, mode=800x480x32, linelength=3200
[ 0.071954] Console: switching to colour frame buffer device 100x30
[ 0.080800] simple-framebuffer 5fe89000.framebuffer: fb0: simplefb registered!
[ 0.090750] PCI: CLS 0 bytes, default 64
[ 0.095320] Initialise system trusted keyrings
[ 0.098848] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[ 0.106227] Key type asymmetric registered
[ 0.109588] Asymmetric key parser 'x509' registered
[ 0.112865] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[ 0.116197] io scheduler mq-deadline registered
[ 0.119426] io scheduler kyber registered
[ 0.126048] sun5i-pinctrl 1c20800.pinctrl: initialized sunXi PIO driver
[ 0.177235] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled
[ 0.182159] sun5i-pinctrl 1c20800.pinctrl: supply vcc-pg not found, using dummy regulator
[ 0.206521] 1c28400.serial: ttyS0 at MMIO 0x1c28400 (irq = 33, base_baud = 1500000) is a U6_16550A
[ 0.669462] printk: console [ttyS0] enabled
[ 0.684378] loop: module loaded
[ 0.691437] sun5i-pinctrl 1c20800.pinctrl: supply vcc-pc not found, using dummy regulator
[ 0.704102] spi-nor spi0.0: mx25l12805d (16384 Kbytes)
[ 0.712595] 4 fixed-partitions partitions found on MTD device spi0.0
[ 0.722225] Creating 4 MTD partitions on "spi0.0":
[ 0.730299] 0x000000000000-0x0000000e0000 : "bootloader"
[ 0.740017] 0x0000000e0000-0x000000100000 : "dtb"
[ 0.749058] 0x000000100000-0x000000600000 : "kernel"
[ 0.758323] 0x000000500000-0x000000f00000 : "rootfs"
[ 0.768061] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 0.777963] ehci-pci: EHCI PCI platform driver
[ 0.785658] ehci-platform: EHCI generic platform driver
[ 0.794364] ehci-platform 1c14000.usb: EHCI Host Controller
[ 0.803083] ehci-platform 1c14000.usb: new USB bus registered, assigned bus number 1
[ 0.814054] ehci-platform 1c14000.usb: irq 23, io mem 0x01c14000
[ 0.845204] ehci-platform 1c14000.usb: USB 2.0 started, EHCI 1.00
[ 0.854424] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.12
[ 0.865765] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 0.876007] usb usb1: Product: EHCI Host Controller
[ 0.883853] usb usb1: Manufacturer: Linux 5.12.0-rc3 ehci_hcd
[ 0.892601] usb usb1: SerialNumber: 1c14000.usb
[ 0.900637] hub 1-0:1.0: USB hub found
[ 0.907371] hub 1-0:1.0: 1 port detected
[ 0.914828] usbcore: registered new interface driver usb-storage
[ 0.924210] usb_phy_generic usb_phy_generic.0.auto: supply vcc not found, using dummy regulator
[ 0.936722] musb-hdrc musb-hdrc.1.auto: MUSB HDRC host driver
[ 0.945503] musb-hdrc musb-hdrc.1.auto: new USB bus registered, assigned bus number 2
[ 0.956512] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.12
[ 0.967679] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 0.977779] usb usb2: Product: MUSB HDRC host driver
[ 0.985617] usb usb2: Manufacturer: Linux 5.12.0-rc3 musb-hcd
[ 0.994187] usb usb2: SerialNumber: musb-hdrc.1.auto
[ 1.002466] hub 2-0:1.0: USB hub found
[ 1.009019] hub 2-0:1.0: 1 port detected
[ 1.017193] dummy_hcd dummy_hcd.0: USB Host+Gadget Emulator, driver 02 May 2005
[ 1.027371] dummy_hcd dummy_hcd.0: Dummy host controller
[ 1.035441] dummy_hcd dummy_hcd.0: new USB bus registered, assigned bus number 3
[ 1.045868] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.12
[ 1.057120] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.067275] usb usb3: Product: Dummy host controller
[ 1.075111] usb usb3: Manufacturer: Linux 5.12.0-rc3 dummy_hcd
[ 1.083871] usb usb3: SerialNumber: dummy_hcd.0
[ 1.091833] hub 3-0:1.0: USB hub found
[ 1.098560] hub 3-0:1.0: 1 port detected
[ 1.106573] sunxi-wdt 1c20c90.watchdog: Watchdog enabled (timeout=16 sec, nowayout=0)
[ 1.119058] sun4i-ss 1c15000.crypto-engine: Die ID 1
[ 1.127422] sun5i-pinctrl 1c20800.pinctrl: supply vcc-pf not found, using dummy regulator
[ 1.139675] sunxi-mmc 1c0f000.mmc: Got CD GPIO
[ 1.148837] usbcore: registered new interface driver usbhid
[ 1.157506] usbhid: USB HID core driver
[ 1.165056] Loading compiled-in X.509 certificates
[ 1.181214] ubi0: attaching mtd3
[ 1.188137] sunxi-mmc 1c0f000.mmc: initialized, max. request size: 16384 KB
[ 1.208030] random: fast init done
[ 1.354154] ubi0: scanning is finished
[ 1.360876] ubi0 error: ubi_read_volume_table: the layout volume was not found
[ 1.371165] ubi0 error: ubi_attach_mtd_dev: failed to attach mtd3, error -22
[ 1.381210] UBI error: cannot attach mtd3
[ 1.389077] Waiting for root device ...
[ 31.845194] vcc3v0: disabling
[ 31.851016] vcc3v3: disabling
[ 31.856753] vcc5v0: disabling
[ 196.405220] random: crng init done
After writing rootfs.ubi to 0x600000 adress of SPI Nor Flash Dmesg output
[ 0.714492] spi-nor spi0.0: mx25l12805d (16384 Kbytes)
[ 0.722995] 4 fixed-partitions partitions found on MTD device spi0.0
[ 0.732653] Creating 4 MTD partitions on "spi0.0":
[ 0.740754] 0x000000000000-0x0000000e0000 : "bootloader"
[ 0.750508] 0x0000000e0000-0x000000100000 : "dtb"
[ 0.759608] 0x000000100000-0x000000600000 : "kernel"
[ 0.768897] 0x000000500000-0x000000f00000 : "rootfs"
[ 0.778640] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 0.788514] ehci-pci: EHCI PCI platform driver
[ 0.796182] ehci-platform: EHCI generic platform driver
[ 0.804863] ehci-platform 1c14000.usb: EHCI Host Controller
[ 0.813581] ehci-platform 1c14000.usb: new USB bus registered, assigned bus number 1
[ 0.824528] ehci-platform 1c14000.usb: irq 23, io mem 0x01c14000
[ 0.855269] ehci-platform 1c14000.usb: USB 2.0 started, EHCI 1.00
[ 0.864542] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.12
[ 0.875891] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 0.886138] usb usb1: Product: EHCI Host Controller
[ 0.893979] usb usb1: Manufacturer: Linux 5.12.0-rc3 ehci_hcd
[ 0.902719] usb usb1: SerialNumber: 1c14000.usb
[ 0.910764] hub 1-0:1.0: USB hub found
[ 0.917459] hub 1-0:1.0: 1 port detected
[ 0.924897] usbcore: registered new interface driver usb-storage
[ 0.934270] usb_phy_generic usb_phy_generic.0.auto: supply vcc not found, using dummy regulator
[ 0.946772] musb-hdrc musb-hdrc.1.auto: MUSB HDRC host driver
[ 0.955538] musb-hdrc musb-hdrc.1.auto: new USB bus registered, assigned bus number 2
[ 0.966547] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.12
[ 0.977706] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 0.987801] usb usb2: Product: MUSB HDRC host driver
[ 0.995636] usb usb2: Manufacturer: Linux 5.12.0-rc3 musb-hcd
[ 1.004203] usb usb2: SerialNumber: musb-hdrc.1.auto
[ 1.012477] hub 2-0:1.0: USB hub found
[ 1.019014] hub 2-0:1.0: 1 port detected
[ 1.026691] musb_handle_intr_disconnect 983: unhandled DISCONNECT transition (a_idle)
[ 1.038345] dummy_hcd dummy_hcd.0: USB Host+Gadget Emulator, driver 02 May 2005
[ 1.048618] dummy_hcd dummy_hcd.0: Dummy host controller
[ 1.056812] dummy_hcd dummy_hcd.0: new USB bus registered, assigned bus number 3
[ 1.067365] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.12
[ 1.078654] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.088859] usb usb3: Product: Dummy host controller
[ 1.096802] usb usb3: Manufacturer: Linux 5.12.0-rc3 dummy_hcd
[ 1.105650] usb usb3: SerialNumber: dummy_hcd.0
[ 1.113672] hub 3-0:1.0: USB hub found
[ 1.120483] hub 3-0:1.0: 1 port detected
[ 1.128629] sunxi-wdt 1c20c90.watchdog: Watchdog enabled (timeout=16 sec, nowayout=0)
[ 1.141063] sun4i-ss 1c15000.crypto-engine: Die ID 1
[ 1.149452] sun5i-pinctrl 1c20800.pinctrl: supply vcc-pf not found, using dummy regulator
[ 1.161773] sunxi-mmc 1c0f000.mmc: Got CD GPIO
[ 1.170983] usbcore: registered new interface driver usbhid
[ 1.179725] usbhid: USB HID core driver
[ 1.187351] Loading compiled-in X.509 certificates
[ 1.203610] ubi0: attaching mtd3
[ 1.210562] sunxi-mmc 1c0f000.mmc: initialized, max. request size: 16384 KB
[ 1.221794] ubi0 error: validate_ec_hdr: bad VID header offset 512, expected 64
[ 1.232292] ubi0 error: validate_ec_hdr: bad EC header
[ 1.240488] Erase counter header dump:
[ 1.247177] magic 0x55424923
[ 1.253797] version 1
[ 1.259563] ec 0
[ 1.265285] vid_hdr_offset 512
[ 1.271135] data_offset 2048
[ 1.277052] image_seq 1557467774
[ 1.283518] hdr_crc 0xfc6f20a5
[ 1.289894] erase counter header hexdump:
[ 1.296464] 00000000: 55 42 49 23 01 00 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 08 00 5c d5 12 7e 00 00 00 00 UBI#....................\..~....
[ 1.315406] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fc 6f 20 a5 .............................o .
[ 1.334481] CPU: 0 PID: 1 Comm: swapper Not tainted 5.12.0-rc3 #2
[ 1.343368] Hardware name: Allwinner sun4i/sun5i Families
[ 1.351584] [<c010b088>] (unwind_backtrace) from [<c01088f4>] (show_stack+0x10/0x14)
[ 1.362271] [<c01088f4>] (show_stack) from [<c042daec>] (validate_ec_hdr+0x70/0xcc)
[ 1.372919] [<c042daec>] (validate_ec_hdr) from [<c042e53c>] (ubi_io_read_ec_hdr+0x188/0x1d0)
[ 1.384549] [<c042e53c>] (ubi_io_read_ec_hdr) from [<c0432a28>] (ubi_attach+0x2d8/0x1094)
[ 1.395818] [<c0432a28>] (ubi_attach) from [<c0429448>] (ubi_attach_mtd_dev+0x57c/0x9ac)
[ 1.406991] [<c0429448>] (ubi_attach_mtd_dev) from [<c081d018>] (ubi_init+0x1a8/0x214)
[ 1.417956] [<c081d018>] (ubi_init) from [<c01015f4>] (do_one_initcall+0x70/0x194)
[ 1.428550] [<c01015f4>] (do_one_initcall) from [<c0800f94>] (kernel_init_freeable+0x164/0x1b0)
[ 1.440343] [<c0800f94>] (kernel_init_freeable) from [<c052ccc0>] (kernel_init+0x8/0x114)
[ 1.451636] [<c052ccc0>] (kernel_init) from [<c0100150>] (ret_from_fork+0x14/0x24)
[ 1.462347] Exception stack(0xc0c4bfb0 to 0xc0c4bff8)
[ 1.470542] bfa0: 00000000 00000000 00000000 00000000
[ 1.481944] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1.493296] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 1.503033] ubi0 error: ubi_io_read_ec_hdr: validation failed for PEB 0
[ 1.512812] ubi0 error: ubi_attach_mtd_dev: failed to attach mtd3, error -22
[ 1.523275] UBI error: cannot attach mtd3
[ 1.531472] Waiting for root device ...
[ 2.435240] random: fast init done
[ 31.845256] vcc3v0: disabling
[ 31.851464] vcc3v3: disabling
[ 31.857643] vcc5v0: disabling
Buildroot version: 2020.02.11
U-boot version: U-Boot 2021.04-rc3
Kernel version: 5.12.0-rc3
It will be a very general question, but how can I mount and mount rootfs and ubifs and run the system?
Thanks.
Try modifying command sunxi-fel
sudo sunxi-fel -v -p \
spiflash-write 0x0 ${DIR}/u-boot-sunxi-with-spl.bin \
spiflash-write 0xE0000 ${DIR}/sun5i-a13.dtb \
spiflash-write 0x100000 ${DIR}/uImage \
spiflash-write 0x600000 ${DIR}/rootfs.ubi \
uboot ${DIR}/u-boot-sunxi-with-spl.bin
And dts file
partition#600000 {
label = "rootfs";
reg = <0x600000 0xA00000>;
};

AWS EC2 : Connection Timed Out - SSH

Unable to connect via SSH to my Amazon Linux AMI, However I am able to connect to other instance with same VPC, security group and key.
These are the system logs, appreciate any help on this.
> [ 0.000000] Linux version 4.14.101-75.76.amzn1.x86_64 (mockbuild#gobi-build-60003) (gcc version 7.2.1 20170915 (Red Hat
> 7.2.1-2) (GCC)) #1 SMP Tue Feb 19 18:14:51 UTC 2019
> [ 0.000000] Command line: root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295
> LANG=en_US.UTF-8 KEYTABLE=us
> [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
> [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
> [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
> [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
> [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
> [ 0.000000] e820: BIOS-provided physical RAM map:
> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
> [ 0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
> [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable
> [ 0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved
> [ 0.000000] NX (Execute Disable) protection: active
> [ 0.000000] SMBIOS 2.7 present.
> [ 0.000000] DMI: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
> [ 0.000000] Hypervisor detected: Xen HVM
> [ 0.000000] Xen version 4.2.
> [ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
> [ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
> [ 0.000000] You might have to change the root device
> [ 0.000000] from /dev/hd[a-d] to /dev/xvd[a-d]
> [ 0.000000] in your root= kernel command line option
> [ 0.000000] tsc: Fast TSC calibration using PIT
> [ 0.000000] e820: last_pfn = 0x40000 max_arch_pfn = 0x400000000
> [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
> [ 0.000000] found SMP MP-table at [mem 0x000fbc50-0x000fbc5f] mapped at [ffffffffff240c50]
> [ 0.000000] RAMDISK: [mem 0x37088000-0x37feffff]
> [ 0.000000] ACPI: Early table checksum verification disabled
> [ 0.000000] ACPI: RSDP 0x00000000000EA020 000024 (v02 Xen )
> [ 0.000000] ACPI: XSDT 0x00000000FC00E2A0 000054 (v01 Xen HVM 00000000 HVML 00000000)
> [ 0.000000] ACPI: FACP 0x00000000FC00DF60 0000F4 (v04 Xen HVM 00000000 HVML 00000000)
> [ 0.000000] ACPI: DSDT 0x00000000FC0021C0 00BD19 (v02 Xen HVM 00000000 INTL 20090123)
> [ 0.000000] ACPI: FACS 0x00000000FC002180 000040
> [ 0.000000] ACPI: FACS 0x00000000FC002180 000040
> [ 0.000000] ACPI: APIC 0x00000000FC00E060 0000D8 (v02 Xen HVM 00000000 HVML 00000000)
> [ 0.000000] ACPI: HPET 0x00000000FC00E1B0 000038 (v01 Xen HVM 00000000 HVML 00000000)
> [ 0.000000] ACPI: WAET 0x00000000FC00E1F0 000028 (v01 Xen HVM 00000000 HVML 00000000)
> [ 0.000000] ACPI: SSDT 0x00000000FC00E220 000031 (v02 Xen HVM 00000000 INTL 20090123)
> [ 0.000000] ACPI: SSDT 0x00000000FC00E260 000033 (v02 Xen HVM 00000000 INTL 20090123)
> [ 0.000000] No NUMA configuration found
> [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000003fffffff]
> [ 0.000000] NODE_DATA(0) allocated [mem 0x3ffde000-0x3fffffff]
> [ 0.000000] Zone ranges:
> [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
> [ 0.000000] DMA32 [mem 0x0000000001000000-0x000000003fffffff]
> [ 0.000000] Normal empty
> [ 0.000000] Movable zone start for each node
> [ 0.000000] Early memory node ranges
> [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009dfff]
> [ 0.000000] node 0: [mem 0x0000000000100000-0x000000003fffffff]
> [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000003fffffff]
> [ 0.000000] ACPI: PM-Timer IO Port: 0xb008
> [ 0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47
> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 low level)
> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 low level)
> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level)
> [ 0.000000] Using ACPI (MADT) for SMP configuration information
> [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
> [ 0.000000] smpboot: Allowing 15 CPUs, 14 hotplug CPUs
> [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
> [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff]
> [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
> [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
> [ 0.000000] e820: [mem 0x40000000-0xfbffffff] available for PCI devices
> [ 0.000000] Booting paravirtualized kernel on Xen HVM
> [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
> [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:15 nr_cpu_ids:15 nr_node_ids:1
> [ 0.000000] percpu: Embedded 44 pages/cpu #ffff88803e200000 s140120 r8192 d31912 u262144
> [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
> [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 257928
> [ 0.000000] Policy zone: DMA32
> [ 0.000000] Kernel command line: root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295
> LANG=en_US.UTF-8 KEYTABLE=us
> [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
> [ 0.000000] Memory: 988264K/1048180K available (8204K kernel code, 1920K rwdata, 2768K rodata, 2044K init, 3876K bss, 59916K
> reserved, 0K cma-reserved)
> [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=15, Nodes=1
> [ 0.000000] Kernel/User page tables isolation: enabled
> [ 0.000000] ftrace: allocating 25766 entries in 101 pages
> [ 0.004000] Hierarchical RCU implementation.
> [ 0.004000] RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=15.
> [ 0.004000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=15
> [ 0.004000] NR_IRQS: 524544, nr_irqs: 952, preallocated irqs: 16
> [ 0.004000] xen:events: Using 2-level ABI
> [ 0.004000] xen:events: Xen HVM callback vector for event delivery is enabled
> [ 0.004000] Console: colour VGA+ 80x25
> [ 0.004000] console [tty1] enabled
> [ 0.004000] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
> [ 0.004000] console [ttyS0] enabled
> [ 0.004000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 30580167144 ns
> [ 0.008000] tsc: Fast TSC calibration using PIT
> [ 0.024004] tsc: Detected 2400.059 MHz processor
> [ 0.028010] Calibrating delay loop (skipped), value calculated using timer frequency.. 4800.15 BogoMIPS (lpj=9600312)
> [ 0.040005] pid_max: default: 32768 minimum: 301
> [ 0.044020] ACPI: Core revision 20170728
> [ 0.059670] ACPI: 3 ACPI AML tables successfully acquired and loaded
> [ 0.068038] Security Framework initialized
> [ 0.076206] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
> [ 0.092100] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
> [ 0.112022] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
> [ 0.116007] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes)
> [ 0.120233] mce: CPU supports 2 MCE banks
> [ 0.124030] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 1024
> [ 0.132003] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 1024, 1GB 4
> [ 0.148005] Spectre V2 : Mitigation: Full generic retpoline
> [ 0.152004] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
> [ 0.160004] Speculative Store Bypass: Vulnerable
> [ 0.173851] smpboot: Max logical packages: 15
> [ 0.176833] x2apic: IRQ remapping doesn't support X2APIC mode
> [ 0.184004] Switched APIC routing to physical flat.
> [ 0.190772] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
> [ 0.238435] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
> [ 0.244018] installing Xen timer for CPU 0
> [ 0.248079] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2676 v3 # 2.40GHz (family: 0x6, model: 0x3f, stepping: 0x2)
> [ 0.252050] cpu 0 spinlock event irq 53
> [ 0.256035] Performance Events: unsupported p6 CPU model 63 no PMU driver, software events only.
> [ 0.260056] Hierarchical SRCU implementation.
> [ 0.264304] NMI watchdog: Perf event create on CPU 0 failed with -2
> [ 0.268008] NMI watchdog: Perf NMI watchdog permanently disabled
> [ 0.272202] smp: Bringing up secondary CPUs ...
> [ 0.275796] smp: Brought up 1 node, 1 CPU
> [ 0.276011] smpboot: Total of 1 processors activated (4800.15 BogoMIPS)
> [ 0.280482] devtmpfs: initialized
> [ 0.283941] x86/mm: Memory block size: 128MB
> [ 0.284230] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
> [ 0.288031] futex hash table entries: 4096 (order: 6, 262144 bytes)
> [ 0.292282] NET: Registered protocol family 16
> [ 0.296242] cpuidle: using governor ladder
> [ 0.300010] cpuidle: using governor menu
> [ 0.304044] ACPI: bus type PCI registered
> [ 0.308007] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> [ 0.312456] PCI: Using configuration type 1 for base access
> [ 0.318787] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
> [ 0.320217] ACPI: Added _OSI(Module Device)
> [ 0.324021] ACPI: Added _OSI(Processor Device)
> [ 0.328007] ACPI: Added _OSI(3.0 _SCP Extensions)
> [ 0.332006] ACPI: Added _OSI(Processor Aggregator Device)
> [ 0.339182] ACPI: Interpreter enabled
> [ 0.340021] ACPI: (supports S0 S4 S5)
> [ 0.343460] ACPI: Using IOAPIC for interrupt routing
> [ 0.344043] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
> [ 0.348445] ACPI: Enabled 2 GPEs in block 00 to 0F
> [ 0.412557] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
> [ 0.416018] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
> [ 0.420033] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
> [ 0.424030] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
> [ 0.429218] acpiphp: Slot [0] registered
> [ 0.433312] acpiphp: Slot [3] registered
> [ 0.436509] acpiphp: Slot [4] registered
> [ 0.440524] acpiphp: Slot [5] registered
> [ 0.444541] acpiphp: Slot [6] registered
> [ 0.448541] acpiphp: Slot [7] registered
> [ 0.452544] acpiphp: Slot [8] registered
> [ 0.458516] acpiphp: Slot [9] registered
> [ 0.460544] acpiphp: Slot [10] registered
> [ 0.464527] acpiphp: Slot [11] registered
> [ 0.468568] acpiphp: Slot [12] registered
> [ 0.472652] acpiphp: Slot [13] registered
> [ 0.476608] acpiphp: Slot [14] registered
> [ 0.480511] acpiphp: Slot [15] registered
> [ 0.484590] acpiphp: Slot [16] registered
> [ 0.489551] acpiphp: Slot [17] registered
> [ 0.492548] acpiphp: Slot [18] registered
> [ 0.496621] acpiphp: Slot [19] registered
> [ 0.500593] acpiphp: Slot [20] registered
> [ 0.504608] acpiphp: Slot [21] registered
> [ 0.508563] acpiphp: Slot [22] registered
> [ 0.512509] acpiphp: Slot [23] registered
> [ 0.516537] acpiphp: Slot [24] registered
> [ 0.520580] acpiphp: Slot [25] registered
> [ 0.524589] acpiphp: Slot [26] registered
> [ 0.528546] acpiphp: Slot [27] registered
> [ 0.532617] acpiphp: Slot [28] registered
> [ 0.536625] acpiphp: Slot [29] registered
> [ 0.540551] acpiphp: Slot [30] registered
> [ 0.544492] acpiphp: Slot [31] registered
> [ 0.548403] PCI host bridge to bus 0000:00
> [ 0.552006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
> [ 0.556005] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]
> [ 0.560005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
> [ 0.564009] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfbffffff window]
> [ 0.568008] pci_bus 0000:00: root bus resource [bus 00-ff]
> [ 0.580994] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7]
> [ 0.584013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6]
> [ 0.588009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177]
> [ 0.592013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376]
> [ 0.597469] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
> [ 0.597469] * this clock source is slow. Consider trying other clock sources
> [ 0.601963] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI
> [ 0.617879] ACPI: PCI Interrupt Link [LNKA] (IRQs *5 10 11)
> [ 0.620484] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
> [ 0.624308] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
> [ 0.628273] ACPI: PCI Interrupt Link [LNKD] (IRQs *5 10 11)
> [ 0.659090] xen:balloon: Initialising balloon driver
> [ 0.672470] pci 0000:00:02.0: vgaarb: setting as boot VGA device
> [ 0.676000] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
> [ 0.676028] pci 0000:00:02.0: vgaarb: bridge control possible
> [ 0.680005] vgaarb: loaded
> [ 0.684298] EDAC MC: Ver: 3.0.0
> [ 0.688228] PCI: Using ACPI for IRQ routing
> [ 0.692903] NetLabel: Initializing
> [ 0.696007] NetLabel: domain hash size = 128
> [ 0.700005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
> [ 0.704026] NetLabel: unlabeled traffic allowed by default
> [ 0.708043] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
> [ 0.712018] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
> [ 0.716006] hpet0: 3 comparators, 64-bit 62.500000 MHz counter
> [ 0.722208] clocksource: Switched to clocksource xen
> [ 0.732257] VFS: Disk quotas dquot_6.6.0
> [ 0.735847] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> [ 0.741239] pnp: PnP ACPI init
> [ 0.743942] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved
> [ 0.749560] system 00:01: [io 0x08a0-0x08a3] has been reserved
> [ 0.755099] system 00:01: [io 0x0cc0-0x0ccf] has been reserved
> [ 0.759542] system 00:01: [io 0x04d0-0x04d1] has been reserved
> [ 0.765628] system 00:07: [io 0x10c0-0x1141] has been reserved
> [ 0.769988] system 00:07: [io 0xb044-0xb047] has been reserved
> [ 0.774201] random: fast init done
> [ 0.802572] pnp: PnP ACPI: found 8 devices
> [ 0.815571] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
> [ 0.823851] NET: Registered protocol family 2
> [ 0.828295] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
> [ 0.836223] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
> [ 0.842471] TCP: Hash tables configured (established 8192 bind 8192)
> [ 0.848377] UDP hash table entries: 512 (order: 2, 16384 bytes)
> [ 0.854312] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
> [ 0.865357] NET: Registered protocol family 1
> [ 0.870057] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
> [ 0.874521] pci 0000:00:01.0: PIIX3: Enabling Passive Release
> [ 0.880434] pci 0000:00:01.0: Activating ISA DMA hang workarounds
> [ 0.886961] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
> [ 0.900418] Unpacking initramfs...
> [ 1.140068] Freeing initrd memory: 15776K
> [ 1.144270] audit: initializing netlink subsys (disabled)
> [ 1.149487] Initialise system trusted keyrings
> [ 1.153120] random: get_random_bytes called from key_alloc+0x209/0x430 with crng_init=1
> [ 1.159656] Key type blacklist registered
> [ 1.164319] audit: type=2000 audit(1621604183.240:1): state=initialized audit_enabled=0 res=1
> [ 1.173559] workingset: timestamp_bits=36 max_order=18 bucket_order=0
> [ 1.181126] zbud: loaded
> [ 1.462923] Key type asymmetric registered
> [ 1.466712] Asymmetric key parser 'x509' registered
> [ 1.471526] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
> [ 1.478219] io scheduler noop registered (default)
> [ 1.482503] xen:grant_table: Grant tables using version 1 layout
> [ 1.488560] Grant table initialized
> [ 1.492864] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
> [ 1.498136] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> [ 1.532534] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
> [ 1.539067] xen_netfront: Initialising Xen virtual ethernet driver
> [ 1.546587] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
> [ 1.557894] serio: i8042 KBD port at 0x60,0x64 irq 1
> [ 1.563616] serio: i8042 AUX port at 0x60,0x64 irq 12
> [ 1.569522] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
> [ 1.578461] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
> [ 1.584470] rtc_cmos 00:02: alarms up to one day, 114 bytes nvram, hpet irqs
> [ 1.591556] hidraw: raw HID events driver (C) Jiri Kosina
> [ 1.597392] NET: Registered protocol family 17
> [ 1.602551] sched_clock: Marking stable (1602191383, 0)->(2508005586, -905814203)
> [ 1.608985] registered taskstats version 1
> [ 1.613380] Loading compiled-in X.509 certificates
> [ 1.617465] zswap: loaded using pool lzo/zbud
> [ 1.724098] xenbus_probe_frontend: Device with no driver: device/vbd/768
> [ 1.733164] rtc_cmos 00:02: setting system clock to 2021-05-21 13:36:24 UTC (1621604184)
> [ 1.740878] Freeing unused kernel memory: 2044K
> [ 1.752090] Write protecting the kernel read-only data: 14336k
> [ 1.759007] Freeing unused kernel memory: 2016K
> [ 1.766675] Freeing unused kernel memory: 1328K
> [ 1.788444] device-mapper: uevent: version 1.0.3
> [ 1.794113] device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel#redhat.com
> [ 1.808846] udevd[853]: starting version 173
> [ 1.859269] SCSI subsystem initialized
> [ 1.951311] scsi host0: ata_piix
> [ 1.964023] Invalid max_queues (4), will use default max: 1.
> [ 1.970827] scsi host1: ata_piix
> [ 1.973941] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc100 irq 14
> [ 1.979650] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc108 irq 15
> [ 2.011829] blkfront: xvda: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
> [ 2.026784] xvda: xvda1
> [ 2.144087] tsc: Refined TSC clocksource calibration: 2399.999 MHz
> [ 2.148782] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2298364cab5, max_idle_ns: 440795214892 ns
> growroot: NOCHANGE: disk=/dev/xvda partition=1: size=16773086, it cannot be grown
> [ 2.214915] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
> [ 2.224557] dracut: Remounting /dev/disk/by-label/\x2f with -o noatime,nofail,ro
> [ 2.232584] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
> [ 2.240125] dracut: Mounted root filesystem /dev/xvda1
> [ 2.273312] dracut: Switching root
> [ 2.321659] random: init: uninitialized urandom read (12 bytes read)
> image_name="amzn-ami-hvm"
> image_version="2018.03"
> image_arch="x86_64"
> image_file="amzn-ami-hvm-2018.03.0.20180811-x86_64.ext4.gpt"
> image_stamp="cc81-f2f3"
> image_date="20180811012746"
> recipe_name="amzn ami"
> recipe_id="5b283820-dc60-a7ea-d436-39fa-439f-02ea-5c802dbd"
> Welcome to Amazon Linux AMI
> Starting udev: [ 2.454862] udevd[1570]: starting version 173
> [ 2.518886] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
> [ 2.543884] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input4
> [ 2.611788] ACPI: Power Button [PWRF]
> [ 2.615632] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input5
> [ 2.669689] ACPI: Sleep Button [SLPF]
> [ 2.678057] mousedev: PS/2 mouse device common for all mice
> [ OK ]
>
>
Setting hostname localhost.localdomain: [ OK ]
Setting up Logical Volume Management: [ 3.718097] random: lvm: uninitialized urandom read (4 bytes read)
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
[ OK ]
Checking filesystems
Checking all file systems.
[ OK ]
Remounting root filesystem in read-write mode: [ 3.759745] EXT4-fs (xvda1): re-mounted. Opts: (null)
[ OK ]
Mounting local filesystems: [ OK ]
Enabling local filesystem quotas: [ OK ]
[ 3.791456] random: dd: uninitialized urandom read (4096 bytes read)
Enabling /etc/fstab swaps: [ OK ]
Entering non-interactive startup
Starting LVM metadata daemon: [ OK ]
Starting LVM poll daemon: [ OK ]
[ 3.924597] random: vgs: uninitialized urandom read (4 bytes read)
[ 4.010580] NET: Registered protocol family 10
[ 4.018195] Segment Routing with IPv6
Bringing up loopback interface: [ OK ]
Bringing up interface eth0:
Determining IP information for eth0... done.
Determining IPv6 information for eth0... done.
[ OK ]
Starting auditd: [ OK ]
Starting system logger: [ OK ]
Starting irqbalance: [ OK ]
Starting rngd: [ 5.871299] urandom_read: 1 callbacks suppressed
[ 5.871300] random: rngd: uninitialized urandom read (16 bytes read)
Unable to open file: /dev/hwrng
[ OK ]
[ 5.910283] random: crng init done
Starting rpcbind: [ OK ]
Starting NFS statd: [ OK ]
Starting system message bus: [ OK ]
Mounting filesystems: [ OK ]
Starting acpi daemon: [ OK ]
Retrigger failed udev events--type=failed is deprecated and will be removed from a future udev release.
[ OK ]
Starting cloud-init: Cloud-init v. 0.7.6 running 'init-local' at Fri, 21 May 2021 13:36:29 +0000. Up 6.53 seconds.
Starting cloud-init: Cloud-init v. 0.7.6 running 'init' at Fri, 21 May 2021 13:36:29 +0000. Up 6.73 seconds.
ci-info: +++++++++++++++++++++++Net device info++++++++++++++++++++++++
ci-info: Device Up Address Mask Hw-Address
ci-info: lo True 127.0.0.1 255.0.0.0 .
ci-info: eth0 True 172.31.10.11 255.255.240.0 0a:3b:55:e4:8b:7e
ci-info: ++++++++++++++++++++++++++++++Route info+++++++++++++++++++++++++++++++
ci-info: Route Destination Gateway Genmask Interface Flags
ci-info: 0 0.0.0.0 172.31.0.1 0.0.0.0 eth0 UG
ci-info: 1 169.254.169.254 0.0.0.0 255.255.255.255 eth0 UH
ci-info: 2 172.31.0.0 0.0.0.0 255.255.240.0 eth0 U
May 21 13:36:29 cloud-init[2433]: __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'sudo yum install -y
http...'
Starting cloud-init: Cloud-init v. 0.7.6 running 'modules:config' at Fri, 21 May 2021 13:36:30 +0000. Up 7.43 seconds.
Starting sshd: [ OK ]
Starting ntpd: [ OK ]
Starting sendmail: [ OK ]
Starting sm-client: [ OK ]
Starting httpd: [ OK ]
Starting mongod: [ OK ]
My issue was resolved. Please follow below post to check procedure that was followed. Hope it helps someone else as well.
https://stackoverflow.com/a/67648386/3224357

Error in booting from uSD-card - No filesystem could mount root, tried:Ext4

I am trying to build an embedded Linux to be run on PocketBeagle using Buildroot; and following the instructions in (source1). The resulting images are copied then to the uSD-card.
When inserting the uSD-card in the Pocketbeagle and monitor the booting process via a serial connection; I got the following error:
No filesystem could mount root, tried:
ext4
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,1)
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,1).
The used resources are:
Buildroot: 2018.02
File system: ext4
Bootloader: U-Boot (custom version 2018.01)
Kernel version: 4.14.24
Applying all the patches stated in source above and can be found in (source2):
- 0001-Stripped-back-pocketbeagle-devicetree.patch
- 0001-am335x_evm-uEnv.txt-bootz-n-fixes.patch
- 0002-U-Boot-BeagleBone-Cape-Manager.patch
- 0003-pocketbeagle-tweaks.patch
Runing the build results in the following files and images:
MLO: First stage bootloader
U-boot.img: Second stage bootloader
uEnv.txt: U-boot environmnt
zImage: Linux kernel image
am335x-pocketbeagle.dtb: Linux kernel Device Tree Blob
rootfs.ext4: Root filesystem image
The uEnv.txt is as follows:
fdtfile=am335x-pocketbeagle.dtb
bootpart=0:1
bootdir=
bootargs=console=ttyO0,115200n8 root=/dev/mmcblk0p1 rw rootfstype=ext4 rootwait
uenvcmd=run loadimage;run loadfdt;printenv bootargs;bootz ${loadaddr} - ${fdtaddr};
Then by using genimage in buildroot (and rebuilding) to generate the uSd-image that contains two partitions:
1. FAT partition containing the bootloader images, the kernel image and Device Tree
2. Rootfs partition containing the root filesystem ext4
The resulting image is called sdcard.img is copies to the uSD-card by using program etcher (etcher-electron-1.4.3-x86_64.AppImage) or by (sudo dd if=output/images/sdcard.img of=/dev/mmcblk0 bs=1M)
I have used two different uSD-cards to test the build, but the result was the same:
1. Kingstone 32 GB (SDHC 10)
2. Sandisk 16GB (SDHC 10)
It's worth to say that I have used the same uSD-cards to load the original image of the PocketBeagle from the official site and boot from them, the cards boots without problem.
Any idea to help is highly appreciated.
ADDENDUM
Another error that is shown in the log file occurs in the beginning of the boot:
U-Boot SPL 2018.01 (Mar 24 2018 - 21:13:25)
Trying to boot from MMC1
*** Warning - bad CRC, using default environment
reading u-boot.img
reading u-boot.img
U-Boot 2018.01 (Mar 24 2018 - 21:13:25 +0100)
CPU : AM335X-GP rev 2.1 I2C: ready DRAM: 512 MiB
No match for driver 'omap_hsmmc'
No match for driver 'omap_hsmmc'
Some drivers were not found
Reset Source: Power-on reset has occurred.
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
*** Warning - bad CRC, using default environment
The full log is as follows:
U-Boot SPL 2018.01 (Mar 24 2018 - 21:13:25)
Trying to boot from MMC1
*** Warning - bad CRC, using default environment
reading u-boot.img
reading u-boot.img
U-Boot 2018.01 (Mar 24 2018 - 21:13:25 +0100)
CPU : AM335X-GP rev 2.1
I2C: ready
DRAM: 512 MiB
No match for driver 'omap_hsmmc'
No match for driver 'omap_hsmmc'
Some drivers were not found
Reset Source: Power-on reset has occurred.
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
*** Warning - bad CRC, using default environment
Model: BeagleBoard.org PocketBeagle
<ethaddr> not set. Validating first E-fuse MAC
Net: No ethernet found.
Press SPACE to abort autoboot in 2 seconds
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
gpio: pin 56 (gpio 56) value is 0
gpio: pin 55 (gpio 55) value is 0
gpio: pin 54 (gpio 54) value is 0
gpio: pin 53 (gpio 53) value is 1
switch to partitions #0, OK
mmc0 is current device
gpio: pin 54 (gpio 54) value is 1
Checking for: /uEnv.txt ...
reading uEnv.txt
218 bytes read in 4 ms (52.7 KiB/s)
gpio: pin 55 (gpio 55) value is 1
Loaded environment from /uEnv.txt
Importing environment from mmc ...
Checking if uenvcmd is set ...
gpio: pin 56 (gpio 56) value is 1
Running uenvcmd ...
reading /zImage
5540120 bytes read in 351 ms (15.1 MiB/s)
loading /am335x-pocketbeagle.dtb ...
reading /am335x-pocketbeagle.dtb
33516 bytes read in 9 ms (3.6 MiB/s)
bootargs=console=ttyO0,115200n8 root=/dev/mmcblk0p1 rw rootfstype=ext4 rootwait
## Flattened Device Tree blob at 88000000
Booting using the fdt blob at 0x88000000
Loading Device Tree to 8fff4000, end 8ffff2eb ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.14.24 (ammar#ammar-System-Product-Name) (gcc version 7.2.1 20171011 (Linaro GCC 7.2-2017.11)) #1 SMP Sat Mar 24 21:23:50 CET 2018
[ 0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: TI AM335x PocketBeagle
[ 0.000000] Memory policy: Data cache writeback
[ 0.000000] cma: Reserved 16 MiB at 0x9e800000
[ 0.000000] CPU: All CPU(s) started in SVC mode.
[ 0.000000] AM335X ES2.1 (sgx neon)
[ 0.000000] random: fast init done
[ 0.000000] percpu: Embedded 17 pages/cpu #df93f000 s40872 r8192 d20568 u69632
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 129412
[ 0.000000] Kernel command line: console=ttyO0,115200n8 root=/dev/mmcblk0p1 rw rootfstype=ext4 rootwait
[ 0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 0.000000] Memory: 479900K/522240K available (8192K kernel code, 810K rwdata, 2412K rodata, 1024K init, 7547K bss, 25956K reserved, 16384K cma-reserved, 0K highmem)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
[ 0.000000] vmalloc : 0xe0000000 - 0xff800000 ( 504 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xdfe00000 ( 510 MB)
[ 0.000000] pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
[ 0.000000] modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
[ 0.000000] .text : 0xc0008000 - 0xc0900000 (9184 kB)
[ 0.000000] .init : 0xc0c00000 - 0xc0d00000 (1024 kB)
[ 0.000000] .data : 0xc0d00000 - 0xc0dca9b8 ( 811 kB)
[ 0.000000] .bss : 0xc0dcc000 - 0xc152af94 (7548 kB)
[ 0.000000] Running RCU self tests
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU event tracing is enabled.
[ 0.000000] RCU lockdep checking is enabled.
[ 0.000000] RCU restricting CPUs from NR_CPUS=2 to nr_cpu_ids=1.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts
[ 0.000000] OMAP clockevent source: timer2 at 24000000 Hz
[ 0.000017] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
[ 0.000044] clocksource: timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[ 0.000097] OMAP clocksource: timer1 at 24000000 Hz
[ 0.000569] timer_probe: no matching timers found
[ 0.001470] Console: colour dummy device 80x30
[ 0.001516] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[ 0.001530] ... MAX_LOCKDEP_SUBCLASSES: 8
[ 0.001542] ... MAX_LOCK_DEPTH: 48
[ 0.001554] ... MAX_LOCKDEP_KEYS: 8191
[ 0.001566] ... CLASSHASH_SIZE: 4096
[ 0.001578] ... MAX_LOCKDEP_ENTRIES: 32768
[ 0.001589] ... MAX_LOCKDEP_CHAINS: 65536
[ 0.001601] ... CHAINHASH_SIZE: 32768
[ 0.001613] memory used by lock dependency info: 4655 kB
[ 0.001626] per task-struct memory footprint: 1536 bytes
[ 0.001663] Calibrating delay loop... 996.14 BogoMIPS (lpj=4980736)
[ 0.078740] pid_max: default: 32768 minimum: 301
[ 0.079147] Security Framework initialized
[ 0.079285] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.079306] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.082392] CPU: Testing write buffer coherency: ok
[ 0.083889] CPU0: thread -1, cpu 0, socket -1, mpidr 0
[ 0.085406] Setting up static identity map for 0x80100000 - 0x80100078
[ 0.085975] Hierarchical SRCU implementation.
[ 0.087625] smp: Bringing up secondary CPUs ...
[ 0.087648] smp: Brought up 1 node, 1 CPU
[ 0.087665] SMP: Total of 1 processors activated (996.14 BogoMIPS).
[ 0.087679] CPU: All CPU(s) started in SVC mode.
[ 0.091643] devtmpfs: initialized
[ 0.116033] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
[ 0.116853] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.116916] futex hash table entries: 256 (order: 2, 16384 bytes)
[ 0.118550] pinctrl core: initialized pinctrl subsystem
[ 0.123225] NET: Registered protocol family 16
[ 0.130326] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.171692] omap_hwmod: debugss: _wait_target_disable failed
[ 0.226390] cpuidle: using governor menu
[ 0.237629] OMAP GPIO hardware version 0.1
[ 0.262333] No ATAGs?
[ 0.262360] hw-breakpoint: debug architecture 0x4 unsupported.
[ 0.262655] omap4_sram_init:Unable to allocate sram needed to handle errata I688
[ 0.262675] omap4_sram_init:Unable to get sram pool needed to handle errata I688
[ 0.298256] edma 49000000.edma: TI EDMA DMA engine driver
[ 0.303666] SCSI subsystem initialized
[ 0.305374] omap_i2c 44e0b000.i2c: could not find pctldev for node /ocp/l4_wkup#44c00000/scm#210000/pinmux#800/pinmux_i2c0_pins, deferring probe
[ 0.307447] omap_i2c 4802a000.i2c: bus 1 rev0.11 at 400 kHz
[ 0.307764] omap_i2c 4819c000.i2c: could not find pctldev for node /ocp/l4_wkup#44c00000/scm#210000/pinmux#800/mma8452_pins_default, deferring probe
[ 0.308058] pps_core: LinuxPPS API ver. 1 registered
[ 0.308076] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti#linux.it>
[ 0.308128] PTP clock support registered
[ 0.312478] clocksource: Switched to clocksource timer1
[ 0.448001] VFS: Disk quotas dquot_6.6.0
[ 0.448193] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 0.473020] NET: Registered protocol family 2
[ 0.474843] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.474948] TCP bind hash table entries: 4096 (order: 5, 147456 bytes)
[ 0.476120] TCP: Hash tables configured (established 4096 bind 4096)
[ 0.476506] UDP hash table entries: 256 (order: 2, 20480 bytes)
[ 0.476689] UDP-Lite hash table entries: 256 (order: 2, 20480 bytes)
[ 0.477446] NET: Registered protocol family 1
[ 0.479566] RPC: Registered named UNIX socket transport module.
[ 0.479614] RPC: Registered udp transport module.
[ 0.479629] RPC: Registered tcp transport module.
[ 0.479643] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.482107] hw perfevents: no interrupt-affinity property for /pmu, guessing.
[ 0.483357] hw perfevents: enabled with armv7_cortex_a8 PMU driver, 5 counters available
[ 0.487455] audit: initializing netlink subsys (disabled)
[ 0.490304] audit: type=2000 audit(0.480:1): state=initialized audit_enabled=0 res=1
[ 0.490640] workingset: timestamp_bits=14 max_order=17 bucket_order=3
[ 0.493682] NFS: Registering the id_resolver key type
[ 0.494058] Key type id_resolver registered
[ 0.494099] Key type id_legacy registered
[ 0.494253] jffs2: version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
[ 0.500669] io scheduler noop registered
[ 0.500700] io scheduler deadline registered
[ 0.500822] io scheduler cfq registered (default)
[ 0.500841] io scheduler mq-deadline registered
[ 0.500856] io scheduler kyber registered
[ 0.503256] pinctrl-single 44e10800.pinmux: 142 pins at pa f9e10800 size 568
[ 0.510527] Serial: 8250/16550 driver, 6 ports, IRQ sharing enabled
[ 0.516492] omap_uart 44e09000.serial: no wakeirq for uart0
[ 0.516942] 44e09000.serial: ttyO0 at MMIO 0x44e09000 (irq = 30, base_baud = 3000000) is a OMAP UART0
[ 1.250345] console [ttyO0] enabled
[ 1.255586] omap_uart 48022000.serial: no wakeirq for uart1
[ 1.261829] 48022000.serial: ttyO1 at MMIO 0x48022000 (irq = 31, base_baud = 3000000) is a OMAP UART1
[ 1.272744] omap_uart 48024000.serial: no wakeirq for uart2
[ 1.278844] 48024000.serial: ttyO2 at MMIO 0x48024000 (irq = 32, base_baud = 3000000) is a OMAP UART2
[ 1.289570] omap_uart 481a8000.serial: no wakeirq for uart4
[ 1.295781] 481a8000.serial: ttyO4 at MMIO 0x481a8000 (irq = 33, base_baud = 3000000) is a OMAP UART4
[ 1.334046] brd: module loaded
[ 1.362138] loop: module loaded
[ 1.368162] mtdoops: mtd device (mtddev=name/number) must be supplied
[ 1.383297] libphy: Fixed MDIO Bus: probed
[ 1.390588] i2c /dev entries driver
[ 1.398755] omap_hsmmc 48060000.mmc: Got CD GPIO
[ 1.465093] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.472542] oprofile: using arm/armv7
[ 1.477095] Initializing XFRM netlink socket
[ 1.481985] NET: Registered protocol family 10
[ 1.493215] Segment Routing with IPv6
[ 1.497190] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 1.505262] NET: Registered protocol family 17
[ 1.509973] NET: Registered protocol family 15
[ 1.515090] Key type dns_resolver registered
[ 1.519724] omap_voltage_late_init: Voltage driver support not added
[ 1.526481] sr_dev_init: No voltage domain specified for smartreflex0. Cannot initialize
[ 1.535007] sr_dev_init: No voltage domain specified for smartreflex1. Cannot initialize
[ 1.543566] ThumbEE CPU extension supported.
[ 1.548049] Registering SWP/SWPB emulation handler
[ 1.553116] SmartReflex Class3 initialized
[ 1.623490] mmc0: host does not support reading read-only switch, assuming write-enable
[ 1.635560] mmc0: new high speed SDHC card at address 0007
[ 1.644119] mmcblk0: mmc0:0007 SD32G 28.8 GiB
[ 1.655081] mmcblk0: p1 p2
[ 1.671199] tps65217 0-0024: TPS65217 ID 0xe version 1.2
[ 1.677649] omap_i2c 44e0b000.i2c: bus 0 rev0.11 at 400 kHz
[ 1.686633] omap_i2c 4819c000.i2c: bus 2 rev0.11 at 400 kHz
[ 1.694194] hctosys: unable to open rtc device (rtc0)
[ 1.699490] sr_init: No PMIC hook to init smartreflex
[ 1.705144] sr_init: platform driver register failed for SR
[ 1.720670] List of all partitions:
[ 1.724586] 0100 16384 ram0
[ 1.724597] (driver?)
[ 1.730999] 0101 16384 ram1
[ 1.731008] (driver?)
[ 1.737842] 0102 16384 ram2
[ 1.737854] (driver?)
[ 1.744389] 0103 16384 ram3
[ 1.744400] (driver?)
[ 1.750791] 0104 16384 ram4
[ 1.750800] (driver?)
[ 1.757269] 0105 16384 ram5
[ 1.757278] (driver?)
[ 1.763737] 0106 16384 ram6
[ 1.763746] (driver?)
[ 1.770139] 0107 16384 ram7
[ 1.770148] (driver?)
[ 1.776597] 0108 16384 ram8
[ 1.776607] (driver?)
[ 1.783061] 0109 16384 ram9
[ 1.783071] (driver?)
[ 1.789456] 010a 16384 ram10
[ 1.789465] (driver?)
[ 1.795999] 010b 16384 ram11
[ 1.796008] (driver?)
[ 1.802540] 010c 16384 ram12
[ 1.802549] (driver?)
[ 1.809029] 010d 16384 ram13
[ 1.809038] (driver?)
[ 1.815571] 010e 16384 ram14
[ 1.815580] (driver?)
[ 1.822056] 010f 16384 ram15
[ 1.822065] (driver?)
[ 1.828679] b300 30228480 mmcblk0
[ 1.828690] driver: mmcblk
[ 1.835867] b301 16384 mmcblk0p1 00000000-01
[ 1.835877]
[ 1.843044] b302 524288 mmcblk0p2 00000000-02
[ 1.843053]
[ 1.850164] No filesystem could mount root, tried:
[ 1.850173] ext4
[ 1.855340]
[ 1.858913] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,1)
[ 1.867873] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,1)
The failure to mount the rootfs is occurring because the correct partition of the SDcard has not been specified.
In the kernel command line, the kernel is (incorrectly) instructed to find the rootfs in the partition corresponding to /dev/mmcblk0p1.
bootargs=console=ttyO0,115200n8 root=/dev/mmcblk0p1 rw rootfstype=ext4 rootwait
and
[ 0.000000] Kernel command line: console=ttyO0,115200n8 root=/dev/mmcblk0p1 rw rootfstype=ext4 rootwait
However the mmc subsystem reports that there are two partitions on your SDcard, and the boot log reports the device names of these partitions for you twice.
First:
[ 1.635560] mmc0: new high speed SDHC card at address 0007
[ 1.644119] mmcblk0: mmc0:0007 SD32G 28.8 GiB
[ 1.655081] mmcblk0: p1 p2
And then:
[ 1.720670] List of all partitions:
...
[ 1.828679] b300 30228480 mmcblk0
[ 1.828690] driver: mmcblk
[ 1.835867] b301 16384 mmcblk0p1 00000000-01
[ 1.835877]
[ 1.843044] b302 524288 mmcblk0p2 00000000-02
So the kernel command line (through bootargs) should specify the second partition, /dev/mmcblk0p2, rather than the first partition for the rootfs.
Since the storage area for your U-Boot environment has not been properly initialized (i.e. the "Warning - bad CRC, using default environment" messages) (or your uEnv.txt file is missing or corrupt), U-Boot is booting the Linux kernel with a hardcoded bootargs value that is defined in your board's configuration file.
You could either edit that file and re-build U-Boot, or abort the autoboot and use
setenv bootargs console=ttyO0,115200n8 root=/dev/mmcblk0p2 rw rootfstype=ext4 rootwait and saveenv commands to correct this mis-specification.

Where can I load a GPIO module at the earliest?

I wrote a kernel module which works as expected. But I want that to be loaded at the beginning of the boot process. So I moved this code to
OpenWRT/build_dir/target-i386_geode_eglibc-2.19/linux-x86_alix2/linux-3.10.49/arch/x86/platform
and my code is here:
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/gpio.h>
#include <linux/delay.h>
#include <linux/cs5535.h>
MODULE_AUTHOR("Ramana");
MODULE_DESCRIPTION("POWER LED DRIVER");
#ifdef MODULE_LICENSE
MODULE_LICENSE("Dual BSD/GPL");
#endif
#define HW_VERSION_GPIO 15
#define LATCH_GPIO 6
#define DATA_GPIO 25
#define CLOCK_GPIO 27
#define HIGH 1
#define LOW 0
static void set_power_led(void)
{
uint8_t i = 0;
/*
* Configure Pins Q8 Q7 Q6....Q0 in shift register
* Set 0 to glow LED
*
* shift_reg:
* indices 0, 1 and 2 are for LED3
* 0 1 1 -> Blue_ON, GREEN_OFF, RED_OFF
*
* indices 3, 4 and 5 are for LED1
*
* indices 6, 7 and 8 are for LED2
* 0 1 1 -> Blue_ON, GREEN_OFF, RED_OFF
*
* The pins Q9 Q10 and Q11 are don't care, so we are not using here
*/
uint8_t shift_reg[9] = {0, 1, 1, 1, 1, 1, 1, 1, 1};
/*
* Clear register before set
*/
for (i = 0; i < 9 ; i ++) {
gpio_set_value(CLOCK_GPIO, LOW);
if (shift_reg[i] == 0) {
gpio_set_value(DATA_GPIO, HIGH);
} else {
gpio_set_value(DATA_GPIO, LOW);
}
gpio_set_value(CLOCK_GPIO, HIGH);
}
gpio_set_value(LATCH_GPIO, HIGH);
msleep(1);
gpio_set_value(LATCH_GPIO, LOW);
}
static int __init power_led_init(void)
{
/*
* If GPIO 15 is high, it is old hardware
*/
printk(KERN_INFO "LED INIT\n");
if (!gpio_is_valid(LATCH_GPIO)) {
printk(KERN_INFO "LEDs: Latch gpio is not valid\n");
return -ENODEV;
}
if (!gpio_is_valid(DATA_GPIO)) {
printk(KERN_INFO "LEDs: Data gpio is not valid\n");
return -ENODEV;
}
if (!gpio_is_valid(CLOCK_GPIO)) {
printk(KERN_INFO "LEDs: Clock gpio is not valid\n");
return -ENODEV;
}
gpio_request(LATCH_GPIO, "sysfs");
gpio_request(DATA_GPIO, "sysfs");
gpio_request(CLOCK_GPIO, "sysfs");
gpio_direction_output(LATCH_GPIO, LOW);
gpio_direction_output(DATA_GPIO, LOW);
gpio_direction_output(CLOCK_GPIO, LOW);
set_power_led();
printk(KERN_INFO "Power LED: registered\n");
return 0;
}
static void __exit power_led_exit(void)
{
uint8_t i;
for (i = 0; i < 9 ; i++) {
gpio_set_value(CLOCK_GPIO, LOW);
gpio_set_value(DATA_GPIO, LOW);
gpio_set_value(CLOCK_GPIO, HIGH);
}
gpio_set_value(LATCH_GPIO, HIGH);
msleep(1);
gpio_set_value(LATCH_GPIO, LOW);
}
module_init(power_led_init);
module_exit(power_led_exit);
With this there is a kernel panic:
[ 0.104709] LED INIT
[ 0.105306] BUG: unable to handle kernel NULL pointer dereference at 0000004c
[ 0.106284] IP: [<c115acc2>] __gpio_set_value+0x12/0x80
[ 0.106284] *pde = 00000000
[ 0.106284] Oops: 0000 [#1]
[ 0.106284] Modules linked in:
[ 0.106284] CPU: 0 PID: 1 Comm: swapper Not tainted 3.10.49 #33
[ 0.106284] task: cf834000 ti: cf840000 task.ti: cf840000
[ 0.106284] EIP: 0060:[<c115acc2>] EFLAGS: 00010286 CPU: 0
[ 0.106284] EIP is at __gpio_set_value+0x12/0x80
[ 0.106284] EAX: c13a1624 EBX: c13a1624 ECX: ffffffea EDX: 00000000
[ 0.106284] ESI: 00000000 EDI: 00000000 EBP: cf841f80 ESP: cf841f24
[ 0.106284] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 0.106284] CR0: 8005003b CR2: 0000004c CR3: 01371000 CR4: 00000090
[ 0.106284] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 0.106284] DR6: ffff0ff0 DR7: 00000400
[ 0.106284] Stack:
[ 0.106284] cf841f3b cf841f44 0000003e c133b05e c12b0d03 006e6967 01010101 01010101
[ 0.106284] 00000000 c133afbb c1000172 cfdff401 00060006 c13019c0 c12ced19 cfdff460
[ 0.106284] 00000000 cfdff460 00000200 c114956a c136c480 00000006 0000003e cf840000
[ 0.106284] Call Trace:
[ 0.106284] [<c133b05e>] ? power_led_init+0xa3/0x112
[ 0.106284] [<c133afbb>] ? alix_init+0xf6/0xf6
[ 0.106284] [<c1000172>] ? do_one_initcall+0xb2/0x150
[ 0.106284] [<c114956a>] ? strcpy+0xa/0x20
[ 0.106284] [<c132da22>] ? kernel_init_freeable+0xd1/0x173
[ 0.106284] [<c132d4aa>] ? do_early_param+0x77/0x77
[ 0.106284] [<c124ad68>] ? kernel_init+0x8/0x170
[ 0.106284] [<c1251322>] ? ret_from_kernel_thread+0x6/0x28
[ 0.106284] [<c1251337>] ? ret_from_kernel_thread+0x1b/0x28
[ 0.106284] [<c124ad60>] ? rest_init+0x60/0x60
[ 0.106284] Code: d6 ab aa aa aa ff d1 5b 5e 5f c3 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 57 f
[ 0.106284] EIP: [<c115acc2>] __gpio_set_value+0x12/0x80 SS:ESP 0068:cf841f24
[ 0.106284] CR2: 000000000000004c
[ 0.106284] ---[ end trace 23021a4cac17faa2 ]---
[ 0.107751] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009
[ 0.107751]
Why this is crashing here and where can I add this to make this module loaded at the earliest.
Full minicom log
PC Engines ALIX.3 v0.99h
640 KB Base Memory
153603174448128645128089697280113664130048146432162816179200195584211968228352244736261120 KB Extended Memory
01F0 Master 045A InnoDisk Corp. - iCF4000 8GB
Phys C/H/S 16000/16/63 Log C/H/S 1003/255/63 LBA
GRUB loading....
Booting `OpenWrt'
[ 0.000000] Linux version 3.10.49 (savari#Ramana) (gcc version 4.8.3 (OpenWrt/Linaro GCC 4.8-2014.04 unknown) ) #40 Tue Nov 8 13:11:49 IST 2016
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000000fffffff] usable
[ 0.000000] BIOS-e820: [mem 0x00000000fff00000-0x00000000ffffffff] reserved
[ 0.000000] Notice: NX (Execute Disable) protection missing in CPU!
[ 0.000000] DMI not present or invalid.
[ 0.000000] e820: last_pfn = 0x10000 max_arch_pfn = 0x100000
[ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[ 0.000000] init_memory_mapping: [mem 0x0fc00000-0x0fffffff]
[ 0.000000] init_memory_mapping: [mem 0x08000000-0x0fbfffff]
[ 0.000000] init_memory_mapping: [mem 0x00100000-0x07ffffff]
[ 0.000000] 256MB LOWMEM available.
[ 0.000000] mapped low ram: 0 - 10000000
[ 0.000000] low ram: 0 - 10000000
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00001000-0x00ffffff]
[ 0.000000] Normal [mem 0x01000000-0x0fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00001000-0x0009ffff]
[ 0.000000] node 0: [mem 0x00100000-0x0fffffff]
[ 0.000000] Using APIC driver default
[ 0.000000] No local APIC present or hardware disabled
[ 0.000000] APIC: disable apic facility
[ 0.000000] APIC: switched to apic NOOP
[ 0.000000] e820: [mem 0x10000000-0xffefffff] available for PCI devices
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 64927
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz block2mtd.block2mtd=/dev/hda2,131072,rootfs,5 root=/dev/mtdblock0 rootfstype=jffs2 rootwait console=tty0 console=ttyS0,38400n8 noinitrd
[ 0.000000] PID hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
[ 0.000000] Initializing CPU#0
[ 0.000000] Memory: 255600k/262144k available (2377k kernel code, 6156k reserved, 873k data, 260k init, 0k highmem)
[ 0.000000] virtual kernel memory layout:
[ 0.000000] fixmap : 0xfffa3000 - 0xfffff000 ( 368 kB)
[ 0.000000] vmalloc : 0xd0800000 - 0xfffa1000 ( 759 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xd0000000 ( 256 MB)
[ 0.000000] .init : 0xc132d000 - 0xc136e000 ( 260 kB)
[ 0.000000] .data : 0xc1252630 - 0xc132cd00 ( 873 kB)
[ 0.000000] .text : 0xc1000000 - 0xc1252630 (2377 kB)
[ 0.000000] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] NR_IRQS:2304 nr_irqs:256 16
[ 0.000000] console [ttyS0] enabled
[ 0.000000] tsc: Fast TSC calibration using PIT
[ 0.000000] tsc: Detected 498.062 MHz processor
[ 0.003005] Calibrating delay loop (skipped), value calculated using timer frequency.. 996.12 BogoMIPS (lpj=498062)
[ 0.005013] pid_max: default: 32768 minimum: 301
[ 0.007653] Mount-cache hash table entries: 512
[ 0.011056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[ 0.011056] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0
[ 0.011056] tlb_flushall_shift: -1
[ 0.012009] CPU: Geode(TM) Integrated Processor by AMD PCS (fam: 05, model: 0a, stepping: 02)
[ 0.018327] Performance Events: no PMU driver, software events only.
[ 0.026472] NET: Registered protocol family 16
[ 0.030862] PCI: PCI BIOS revision 2.10 entry at 0xfced9, last bus=0
[ 0.031012] PCI: Using configuration type 1 for base access
[ 0.051103] bio: create slab <bio-0> at 0
[ 0.057403] SCSI subsystem initialized
[ 0.060534] PCI: Probing PCI hardware
[ 0.062281] PCI host bridge to bus 0000:00
[ 0.063030] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.064033] pci_bus 0000:00: root bus resource [mem 0x00000000-0xffffffff]
[ 0.065020] pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
[ 0.078410] Switching to clocksource pit
[ 0.086307] NET: Registered protocol family 2
[ 0.088599] TCP established hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.089847] TCP bind hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.090949] TCP: Hash tables configured (established 2048 bind 2048)
[ 0.092136] TCP: reno registered
[ 0.093867] UDP hash table entries: 256 (order: 0, 4096 bytes)
[ 0.095398] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[ 0.096873] NET: Registered protocol family 1
[ 0.100630] platform rtc_cmos: registered platform RTC device (no PNP device found)
[ 0.104665] alix: system is recognized as "PC Engines ALIX.3 v0.99h"
[ 0.106619] LED INIT
[ 0.107229] BUG: unable to handle kernel NULL pointer dereference at 0000004c
[ 0.108207] IP: [<c115acc2>] __gpio_set_value+0x12/0x80
[ 0.108207] *pde = 00000000
[ 0.108207] Oops: 0000 [#1]
[ 0.108207] Modules linked in:
[ 0.108207] CPU: 0 PID: 1 Comm: swapper Not tainted 3.10.49 #40
[ 0.108207] task: cf834000 ti: cf840000 task.ti: cf840000
[ 0.108207] EIP: 0060:[<c115acc2>] EFLAGS: 00010286 CPU: 0
[ 0.108207] EIP is at __gpio_set_value+0x12/0x80
[ 0.108207] EAX: c13a1624 EBX: c13a1624 ECX: ffffffea EDX: 00000000
[ 0.108207] ESI: 00000000 EDI: 00000000 EBP: cf841f80 ESP: cf841f24
[ 0.108207] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 0.108207] CR0: 8005003b CR2: 0000004c CR3: 01371000 CR4: 00000090
[ 0.108207] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 0.108207] DR6: ffff0ff0 DR7: 00000400
[ 0.108207] Stack:
[ 0.108207] cf841f3b cf841f44 0000003e c133b031 c12b0deb 006e6967 01010101 01010101
[ 0.108207] 00000000 c133afbb c1000172 00000001 00060006 c1301bf0 c12ceeb1 cfdff460
[ 0.108207] 00000000 cfdff460 00000200 00000000 c136c480 00000006 0000003e cf840000
[ 0.108207] Call Trace:
[ 0.108207] [<c133b031>] ? power_led_init+0x76/0xe5
[ 0.108207] [<c133afbb>] ? alix_init+0xf6/0xf6
[ 0.108207] [<c1000172>] ? do_one_initcall+0xb2/0x150
[ 0.108207] [<c132da22>] ? kernel_init_freeable+0xd1/0x173
[ 0.108207] [<c132d4aa>] ? do_early_param+0x77/0x77
[ 0.108207] [<c124b058>] ? kernel_init+0x8/0x170
[ 0.108207] [<c1251622>] ? ret_from_kernel_thread+0x6/0x28
[ 0.108207] [<c1251637>] ? ret_from_kernel_thread+0x1b/0x28
[ 0.108207] [<c124b050>] ? rest_init+0x60/0x60
[ 0.108207] Code: d6 ab aa aa aa ff d1 5b 5e 5f c3 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 57 89 d7 56 53 e8 46 f4 ff ff 85 c0 89 c3 74 5d 8b 30 <f6> 46 4c 01 74 10 ba 8a 07 00 00 b8 5c da 2c c1 e8 b9 bb ec ff
[ 0.108207] EIP: [<c115acc2>] __gpio_set_value+0x12/0x80 SS:ESP 0068:cf841f24
[ 0.108207] CR2: 000000000000004c
[ 0.108207] ---[ end trace 7b3836317c1bee78 ]---
[ 0.108879] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009
[ 0.108879]
I think you have put this in kernel source directory. You should place this in drivers/gpio/ in kernel source. Then add the entries in Kconfig and Makefile of drivers/gpio/ directory. In Kconfig, you can specify the dependency. Selecting it with make menuconfig, you will be able to compile this module as a part of kernel, and it will be loaded at boot-time when kernel is loaded. You can decrease the time further in this case using early_initcall() instead of module_init().
If you are not putting this at kernel source (i.e. not building the driver as part of kernel), then you should call this using "insmod my_module.ko" in a shell script, and put it in init.d and call it at your desired runlevel.

Kernel stalls when accessing serial device on FPGA

I have two UART devices on an FPGA exposed to Linux on an Altera Cyclone V SoC. I have modified the DTS to incorporate these devices, and Linux picks them up on boot:
[ 0.879942] (NULL device *): ttyAL0 at MMIO 0xff200400 (irq = 41, base_baud = 3125000) is a Altera UART
[ 0.890050] (NULL device *): ttyAL1 at MMIO 0xff200420 (irq = 44, base_baud = 3125000) is a Altera UART
Resulting in a ttyAL0 and ttyAL1 in /dev/. The devices also appear in the relevant device subdirectory in /sys/devices/soc/ with the driver symlink present, for example:
lrwxrwxrwx 1 root root 0 Jun 20 10:36 driver -> ../../../bus/platform/drivers/altera_uart
-rw-r--r-- 1 root root 4096 Jun 20 10:36 driver_override
-r--r--r-- 1 root root 4096 Jun 20 10:36 modalias
drwxr-xr-x 2 root root 0 Jun 20 10:36 power
lrwxrwxrwx 1 root root 0 Jun 20 10:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Jun 20 10:36 uevent
However if I try to open the port either programmatically, or with cat or setserial, there is a 20s stall before the RCU scheduler throws an exception:
[ 202.242133] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0, t=2102 jiffies, g=124, c=123, q=254)
[ 202.252516] INFO: Stall ended before state dump start
[ 223.252109] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=125 c=124 q=229)
[ 223.260843] Task dump for CPU 0:
[ 223.264066] klogd R running 0 954 1 0x00000002
[ 223.270566] [<c0017984>] (unwind_backtrace) from [<c00137e0>] (show_stack+0x20/0x24)
[ 223.278319] [<c00137e0>] (show_stack) from [<c004b6cc>] (sched_show_task+0xb0/0x104)
[ 223.286045] [<c004b6cc>] (sched_show_task) from [<c004e34c>] (dump_cpu_task+0x48/0x4c)
[ 223.293941] [<c004e34c>] (dump_cpu_task) from [<c006ae60>] (rcu_dump_cpu_stacks+0xa0/0xcc)
[ 223.302188] [<c006ae60>] (rcu_dump_cpu_stacks) from [<c006e520>] (rcu_check_callbacks+0x488/0x790)
[ 223.311137] [<c006e520>] (rcu_check_callbacks) from [<c0072db0>] (update_process_times+0x50/0x70)
[ 223.319982] [<c0072db0>] (update_process_times) from [<c0083258>] (tick_sched_timer+0x78/0x27c)
[ 223.328656] [<c0083258>] (tick_sched_timer) from [<c00735f4>] (__run_hrtimer+0x90/0x1bc)
[ 223.336719] [<c00735f4>] (__run_hrtimer) from [<c0073ef4>] (hrtimer_interrupt+0x140/0x31c)
[ 223.344955] [<c0073ef4>] (hrtimer_interrupt) from [<c0016b58>] (twd_handler+0x40/0x50)
[ 223.352867] [<c0016b58>] (twd_handler) from [<c00669bc>] (handle_percpu_devid_irq+0x90/0x124)
[ 223.361364] [<c00669bc>] (handle_percpu_devid_irq) from [<c0062684>] (generic_handle_irq+0x3c/0x4c)
[ 223.370377] [<c0062684>] (generic_handle_irq) from [<c0062948>] (__handle_domain_irq+0x6c/0xb4)
[ 223.379042] [<c0062948>] (__handle_domain_irq) from [<c00086b0>] (gic_handle_irq+0x34/0x6c)
[ 223.387362] [<c00086b0>] (gic_handle_irq) from [<c0014380>] (__irq_svc+0x40/0x54)
[ 223.394811] Exception stack(0xded29cf8 to 0xded29d40)
[ 223.399842] 9ce0: 00000001 c06cb200
[ 223.407986] 9d00: 00000000 00000000 c0687b34 00000000 00000082 00000001 df418800 c06c416c
[ 223.416128] 9d20: ded28000 ded29d9c 00000000 ded29d40 c06cb200 c0029330 200f0113 ffffffff
[ 223.424285] [<c0014380>] (__irq_svc) from [<c0029330>] (__do_softirq+0xc4/0x2f0)
[ 223.431656] [<c0029330>] (__do_softirq) from [<c00297f8>] (irq_exit+0x88/0xc0)
[ 223.438851] [<c00297f8>] (irq_exit) from [<c006294c>] (__handle_domain_irq+0x70/0xb4)
[ 223.446649] [<c006294c>] (__handle_domain_irq) from [<c00086b0>] (gic_handle_irq+0x34/0x6c)
[ 223.454965] [<c00086b0>] (gic_handle_irq) from [<c0014380>] (__irq_svc+0x40/0x54)
[ 223.462412] Exception stack(0xded29e08 to 0xded29e50)
[ 223.467443] 9e00: dfbd3540 df782ac0 00000000 0000996f df59d6c0 dfbd3540
[ 223.475584] 9e20: c0695e20 00000000 df59c1c0 df59c540 ded28030 ded29e6c ded29e70 ded29e50
[ 223.483725] 9e40: c047bad0 c004756c 600f0013 ffffffff
[ 223.488762] [<c0014380>] (__irq_svc) from [<c004756c>] (finish_task_switch+0x78/0x11c)
[ 223.496661] [<c004756c>] (finish_task_switch) from [<c047bad0>] (__schedule+0x230/0x5f4)
[ 223.504726] [<c047bad0>] (__schedule) from [<c047bed4>] (schedule+0x40/0x8c)
[ 223.511746] [<c047bed4>] (schedule) from [<c0061a58>] (do_syslog+0x51c/0x5a8)
[ 223.518855] [<c0061a58>] (do_syslog) from [<c0061b00>] (SyS_syslog+0x1c/0x20)
[ 223.525968] [<c0061b00>] (SyS_syslog) from [<c000f820>] (ret_fast_syscall+0x0/0x30)
I don't know why this is happening but I have noticed two interesting (i.e. wrong) things about how Linux sees my devices. The first is that their IRQs, even though correctly reported during boot and any bind/unbind operations, are not listed in /proc/interrupts (they would appear as ff200400.serial2 and ff200420.serial3):
CPU0 CPU1
29: 47565 47091 GIC 29 twd
74: 0 0 GIC 74 0009
75: 0 0 GIC 75 000A
76: 0 0 GIC 76 000A
77: 0 0 GIC 77 0004
78: 0 0 GIC 78 0003
79: 0 0 GIC 79 0006
80: 0 0 GIC 80 0011
81: 0 0 GIC 81 0011
82: 0 0 GIC 82 0010
171: 10554 0 GIC 171 dw-mci
186: 0 0 GIC 186 dw_spi65535
190: 0 0 GIC 190 ffc04000.i2c
191: 0 0 GIC 191 ffc05000.i2c
192: 0 0 GIC 192 ffc06000.i2c
193: 0 0 GIC 193 ffc07000.i2c
194: 465 0 GIC 194 serial
199: 0 0 GIC 199 timer0
207: 0 0 GIC 207 fpga-mgr
IPI0: 0 0 CPU wakeup interrupts
IPI1: 0 0 Timer broadcast interrupts
IPI2: 591 3015 Rescheduling interrupts
IPI3: 0 0 Function call interrupts
IPI4: 1 5 Single function call interrupts
IPI5: 0 0 CPU stop interrupts
IPI6: 0 0 IRQ work interrupts
IPI7: 0 0 completion interrupts
Err: 0
The other observation is that in /sys/class/tty, the ttyAL* entries are links to virtual devices instead of the physical ones:
...
lrwxrwxrwx 1 root root 0 Jun 20 10:49 tty8 -> ../../devices/virtual/tty/tty8
lrwxrwxrwx 1 root root 0 Jun 20 10:49 tty9 -> ../../devices/virtual/tty/tty9
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyAL0 -> ../../devices/virtual/tty/ttyAL0
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyAL1 -> ../../devices/virtual/tty/ttyAL1
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyS0 -> ../../devices/soc/ffc02000.serial0/tty/ttyS0
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyS1 -> ../../devices/soc/ffc03000.serial1/tty/ttyS1
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyp0 -> ../../devices/virtual/tty/ttyp0
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyp1 -> ../../devices/virtual/tty/ttyp1
...
You can see the other two physical devices ttyS0 and ttyS1 ('real' UARTs on the ARM part of the SoC), I expected my devices to be in the same format. If you refer to the /sys/devices/soc/ device subdirectory listing above, you'll notice that it does not have a corresponding tty subdirectory - presumably part of the reason why I have a virtual TTY associated with the device.
So my question is: Why is my physical serial device appearing as virtual, and is that the reason I'm suffering kernel stalls?
In case I am missing vital information in the DTS, here are my UART additions:
uart2: serial2#ff200400 {
compatible = "altr,uart-1.0";
reg = <0xff200400 0x20>;
interrupts = <0 9 4>;
clock-frequency = <50000000>;
current-speed = <115200>;
};
uart3: serial3#ff200420 {
compatible = "altr,uart-1.0";
reg = <0xff200420 0x20>;
interrupts = <0 12 4>;
clock-frequency = <50000000>;
current-speed = <115200>;
};
They are child nodes of a soc node where the interrupt controller is specified.
I finally discovered the issue, and it's unsurprising judging from the RCU scheduler stack trace: My IRQs are wrong.
I don't quite understand the exact mechanics of it as I'm not a firmware engineer, but the UART modules were on a IRQ offset of 40, so the their IRQs were not 9 and 12 as I thought, but 49 and 52. Updating the DTS to match caused everything to work as expected.

Resources