PXE boot fail with kernel panic: Unable to mount root fs - linux-kernel

I am having an issue on some servers since some time, and fail to find the issue.
These are x86_64 server, with Intel Xeon, configured to boot in UEFI over network, through an iPXE rom.
Kernel and initramfs are the ones from Centos 8 (tried 8.0 and 8.2).
But when booting, I always end up with (on every servers, so should not be related to an hardware failure):
[ 5.542304] hid-generic 0003:0557:2221.0002: input,hidraw1: USB HID v1.00 Keyboard [Winbond Electronics Corp Hermon USB hidmouse Device] on usb-0000:00:1a.0-1.3/input1
[ 5.599611] rtc_cmos 00:02: setting system clock to 2020-06-26 19:21:43 UTC (1593199303)
[ 5.620965] md: Waiting for all devices to be available before autodetect
[ 5.640580] md: If you don't use raid, use raid=noautodetect
[ 5.659949] md: Autodetecting RAID arrays.
[ 5.676869] md: autorun ...
[ 5.691838] md: ... autorun DONE.
[ 5.707883] List of all partitions:
[ 5.723667] No filesystem could mount root, tried:
[ 5.723667]
[ 5.754724] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 5.775237] CPU: 9 PID: 1 Comm: swapper/0 Not tainted 4.18.0-193.6.3.el8_2.x86_64 #1
[ 5.815778] Call Trace:
[ 5.830360] dump_stack+0x5c/0x80
[ 5.845920] panic+0xe7/0x2a9
[ 5.860995] mount_block_root+0x2c5/0x2e9
[ 5.877407] ? do_early_param+0x91/0x91
[ 5.892617] prepare_namespace+0x135/0x16b
[ 5.907676] kernel_init_freeable+0x22e/0x258
[ 5.922607] ? rest_init+0xaa/0xaa
[ 5.937398] kernel_init+0xa/0xff
[ 5.950991] ret_from_fork+0x35/0x40
[ 5.964572] Kernel Offset: 0x2ca00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 6.000387] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---
The iPXE boot script is:
#!ipxe
kernel http://10.10.0.1/vmlinuz-4.18.0-193.6.3.el8_2.x86_64 initrd=initramfs-4.18.0-193.6.3.el8_2.x86_64.img selinux=0 rd.shell rd.debug root=live:http://10.10.0.1/squashfs.img rw console=tty0 console=ttyS1,115200
initrd http://10.10.0.1/initramfs-4.18.0-193.6.3.el8_2.x86_64.img
boot
Which generate this in the console:
http://10.10.0.1/vmlinuz-4.18.0-193.6.3.el8_2.x86_64... ok
http://10.10.0.1/initramfs-4.18.0-193.6.3.el8_2.x86_64.img... ok
INTEL 0x6f080f70 MAC reset (081c0261/80280783 was 081c0261/80280783)
INTEL 0x6f080f70 MAC reset (081c0261/80280783 was 081c0261/80280783)
INTEL 0x6f081ab0 MAC reset (081c0261/80280787 was 081c0261/80280787)
[ 0.000000] Linux version 4.18.0-193.6.3.el8_2.x86_64 (mockbuild#kbuilder.bsys.centos.org) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP Wed Jun 10 11:09:32 UTC 2020
[ 0.000000] Command line: vmlinuz-4.18.0-193.6.3.el8_2.x86_64 initrd=initramfs-4.18.0-193.6.3.el8_2.x86_64.img selinux=0 rd.shell rd.debug root=live:http://10.10.0.1/squashfs.img rw console=tty0 console=ttyS1,115200
...
On the server side, on apache logs I have:
10.10.2.1 - - [26/Jun/2020:19:55:42 +0200] "GET /vmlinuz-4.18.0-193.6.3.el8_2.x86_64 HTTP/1.1" 200 8913656 "-" "iPXE/1.0.0+"
10.10.2.1 - - [26/Jun/2020:19:55:42 +0200] "GET /initramfs-4.18.0-193.6.3.el8_2.x86_64.img HTTP/1.1" 200 53703611 "-" "iPXE/1.0.0+"
So it seems to be working perfectly.
This is diskless boot here, but whatever I try (kickstart diskfull install, or even kernel+initrd alone) I always end up to this kernel panic...
I tried to reset BIOS settings, try to boot in legacy/pcbios instead of UEFI, tried to desactivate sata disks, etc. Always this same error. Also tried to use kernel+initrd from Centos ISO (checked checksum), tried to use the ones from my management node. Nothing. Do I miss something obvious?
Does any of you have an idea or already faced this kind of issue?
Many thanks in advance :-)
With my best regards
Beuk

Related

syslogd prints log messages while Linux kernel booting

I am working on an Embedded board. Using the USB-Uart converter, board serial logs are accessed on my Desktop.
On my Desktop I am using the minicom application, to view the board's logs.
1] Power on the embedded board.
2] Kernel booting logs get printed on minicom.
3] It is observed that some booting messages are get printed twice on minicom, e.g.
[ 7.553435] LCD210 platform: Hook NA51068 driver.
Dec 22 12:43:37 (none) user.warn kernel: [ 7.553435] LCD210 platform: Hook NA51068 driver.
[ 7.587133] LCD200:0: suspend_state:1
Dec 22 12:43:37 (none) user.warn kernel: [ 7.587133] LCD200:0: suspend_state:1
In the above logs, line no 1 is printed by the Linux kernel,
[ 7.553435] LCD210 platform: Hook NA51068 driver.
And line no 2. is printed by syslogd on the minicom, while the kernel is booting.
Dec 22 12:43:37 (none) user.warn kernel: [ 7.553435] LCD210 platform: Hook NA51068 driver.
The above message is also present in /var/log/messages file.
I just want to avoid printing the same message twice.
How do I stop the syslogd, to don't print the messages on serial terminal, while the Linux kernel is booting?

Beaglebone black uImage boot issues

I am trying to boot the beaglebone black using the uImage file that I created using the mkimage command. There is a emmc to sd image flasher package available online which has a uImage in it and also other required files (https://elinux.org/BeagleBone_Black_Extracting_eMMC_contents), I followed this website and created a flasher card of the uImage that I made using mkimage utility.
I am getting the following error during boot from SD card having that uImage:
U-Boot SPL 2016.01-00763-g3faf221 (Jan 29 2016 - 20:47:08)
Trying to boot from MMC
Card doesn't support part_switch
MMC partition switch failed
*** Warning - MMC partition switch failed, using default environment
reading u-boot.img
reading u-boot.img
U-Boot 2016.01-00763-g3faf221 (Jan 29 2016 - 20:47:08 +0000)
Watchdog enabled
I2C: ready
DRAM: 512 MiB
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
*** Warning - bad CRC, using default environment
Net: <ethaddr> not set. Validating first E-fuse MAC
cpsw, usb_ether
Press SPACE to abort autoboot in 2 seconds
=> set sdboot 1
=> boot
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
switch to partitions #0, OK
mmc0 is current device
SD/MMC found on device 0
reading boot.scr
** Unable to read file boot.scr **
reading uEnv.txt
235 bytes read in 5 ms (45.9 KiB/s)
Loaded environment from uEnv.txt
Importing environment from mmc ...
Running uenvcmd ...
reading uImage
5497536 bytes read in 302 ms (17.4 MiB/s)
reading /am335x-boneblack.dtb
25926 bytes read in 8 ms (3.1 MiB/s)
## Booting kernel from Legacy Image at 82000000 ...
Image Name: Linux kernel
Created: 2015-09-07 21:47:16 UTC
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 5497472 Bytes = 5.2 MiB
Load Address: 80008000
Entry Point: 80008000
Verifying Checksum ... OK
## Flattened Device Tree blob at 81ff0000
Booting using the fdt blob at 0x81ff0000
Loading Kernel Image ... OK
Loading Device Tree to 8fff6000, end 8ffff545 ... OK
Starting kernel ...
Uncompressing Linux... done, booting the kernel.
[ 0.177572] omap2_mbox_probe: platform not supported
[ 0.333923] tps65217-bl tps65217-bl: no platform data provided
[ 0.397544] bone-capemgr bone_capemgr.9: slot #0: No cape found
[ 0.434652] bone-capemgr bone_capemgr.9: slot #1: No cape found
[ 0.471760] bone-capemgr bone_capemgr.9: slot #2: No cape found
[ 0.508869] bone-capemgr bone_capemgr.9: slot #3: No cape found
[ 0.528815] omap_hsmmc mmc.5: of_parse_phandle_with_args of 'reset' failed
[ 0.591166] pinctrl-single 44e10800.pinmux: pin 44e10854 already requested by 44e10800.pinmux; cannot claim for gpio-leds.8
[ 0.602845] pinctrl-single 44e10800.pinmux: pin-21 (gpio-leds.8) status -22
[ 0.610128] pinctrl-single 44e10800.pinmux: could not request pin 21 on device pinctrl-single
[ 0.686675] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 0.695344] [<c0010f5d>] (unwind_backtrace+0x1/0x98) from [<c04994c3>] (panic+0x5b/0x158)
[ 0.703882] [<c04994c3>] (panic+0x5b/0x158) from [<c07bc94b>] (mount_block_root+0xf7/0x178)
[ 0.712595] [<c07bc94b>] (mount_block_root+0xf7/0x178) from [<c07bcb09>] (mount_root+0x95/0xb0)
[ 0.721676] [<c07bcb09>] (mount_root+0x95/0xb0) from [<c07bcbef>] (prepare_namespace+0xcb/0x11c)
[ 0.730836] [<c07bcbef>] (prepare_namespace+0xcb/0x11c) from [<c0495aff>] (kernel_init+0x7/0x98)
[ 0.740007] [<c0495aff>] (kernel_init+0x7/0x98) from [<c000c7fd>] (ret_from_fork+0x11/0x34)
Any help will be useful.
The kernel cannot find the root partition. The kernel command line is set in U-Boot via the environment variable bootargs.

Qemu Freescale i.MX6 DualLite SABRE : root filesystem does not mount

Goal: emulate the "sabrelite : Freescale i.MX6 Quad SABRE Lite Board (Cortex A9)" that Qemu specifically supports (doing 'qemu-system-arm -M ?' it shows up).
Qemu ver: 2.10.1 (host: fedora-27).
I have successfully cross-compiled and built a 4.1.46 Linux kernel (used the imx_v6_v7_defconfig config file) as well as a simple "skeleton" root filesystem (busybox-based). (FYI, I have a similar working setup for the ARM Cortex-A9 Versatile Express platform - I do this using my own home-spun embedded Linux system called SEALS).
Looking at the U-Boot config file used by similar boards, I figured to use 'root=/dev/mmcblk0p0' as the root= param for the kernel.
So, to try it out I then run qemu as follows (pl scroll horizontally as well to see):
qemu-system-arm -m 512 -M sabrelite -kernel zImage -drive file=rfs.img,format=raw -append "console=ttymxc0 rootfstype=ext4 root=/dev/mmcblk0p0 rw rootwait init=/sbin/init " -nographic -dtb imx6dl-sabresd.dtb
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.1.46 (kai#klaptop) (gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29) ) #2 SMP Mon Nov 27 17:16:22 IST 2017
[ 0.000000] CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[ 0.000000] Machine model: Freescale i.MX6 DualLite SABRE Smart Device Board
[ 0.000000] cma: Reserved 16 MiB at 0x2f000000
[...]
So it starts to boot up just fine. But then:
[...]
[ 2.210965] /soc/aips-bus#02100000/usdhc#02194000: voltage-ranges unspecified
[ 2.211796] sdhci-esdhc-imx 2194000.usdhc: Got CD GPIO
[ 2.212199] sdhci-esdhc-imx 2194000.usdhc: Got WP GPIO
[ 2.214392] sdhci-esdhc-imx 2194000.usdhc: could not get ultra high speed state, work on normal mode
[ 2.218084] sdhci-esdhc-imx 2194000.usdhc: No vmmc regulator found
[ 2.218367] sdhci-esdhc-imx 2194000.usdhc: No vqmmc regulator found
[ 2.265431] mmc0: SDHCI controller on 2194000.usdhc [2194000.usdhc] using ADMA
[ 2.267300] mmc0: mmc_rescan_try_freq: trying to init card at 400000 Hz
[ 2.281912] /soc/aips-bus#02100000/usdhc#02198000: voltage-ranges unspecified
[ 2.282956] sdhci-esdhc-imx 2198000.usdhc: Got CD GPIO
[ 2.283703] sdhci-esdhc-imx 2198000.usdhc: Got WP GPIO
[ 2.284044] sdhci-esdhc-imx 2198000.usdhc: could not get ultra high speed state, work on normal mode
[ 2.284892] sdhci-esdhc-imx 2198000.usdhc: No vmmc regulator found
[ 2.285167] sdhci-esdhc-imx 2198000.usdhc: No vqmmc regulator found
[ 2.298029] mmc0: mmc_rescan_try_freq: trying to init card at 300000 Hz
[ 2.337904] mmc1: SDHCI controller on 2198000.usdhc [2198000.usdhc] using ADMA
[ 2.357051] /soc/aips-bus#02100000/usdhc#0219c000: voltage-ranges unspecified
[ 2.358313] sdhci-esdhc-imx 219c000.usdhc: No vmmc regulator found
[ 2.358642] sdhci-esdhc-imx 219c000.usdhc: No vqmmc regulator found
[ 2.368204] mmc0: mmc_rescan_try_freq: trying to init card at 200000 Hz
[ 2.414722] mmc2: SDHCI controller on 219c000.usdhc [219c000.usdhc] using ADMA
[ 2.440456] mmc0: mmc_rescan_try_freq: trying to init card at 100000 Hz
[...]
[ 2.986441] No soundcards found.
[ 3.007698] Waiting for root device /dev/mmcblk0p0...
Keeps waiting forever here ...
I understand that, on an actual physical board, one would have to "format" or partition the MMC (or SD) card, and have u-boot load up the kernel and rootfs into RAM. But am currently interested in getting the IMX6 working on Qemu...
So, my actual question: how can I get the root filesystem mounted and operational on Qemu?
Any help appreciated! TIA,
There are two problems here. Firstly, your command line isn't actually creating an SD card: the -drive option creates a drive object but doesn't try to plug it in anywhere (because the sabrelite board doesn't define a "default kind of block drive"). To actually plug in the drive to an emulated sd card you need
-drive file=yourfile.img,format=raw,id=mycard -device sd-card,drive=mycard
Secondly, there are bugs in QEMU's current imx6 sd controller emulation, because if you do that then the guest kernel continuously prints
[ 28.971663] mmc1: Timeout waiting for hardware interrupt.
[ 28.973619] mmc1: error -110 whilst initialising SD card
...so it has found the emulated card but isn't getting an interrupt it expects.
These can be fixed by a patch currently on the qemu-devel mailing lists and going through code review: http://patchwork.ozlabs.org/patch/834805/ plus a simple change to hw/arm/fsl-imx6.c to make it create TYPE_IMX_USDHC devices rather than TYPE_SYSBUS_SDHCI. (Basically the imx6's SD controller isn't a completely standard compatible sdhci controller but what we were creating in the QEMU model was the plain variety.)
If you do all that then you can boot a kernel that can see the mmc card:
[ 8.878283] mmc1: new SD card at address 4567
[ 8.910566] mmcblk0: mmc1:4567 QEMU! 256 MiB
With a little luck we'll be able to have this fixed in the 2.12 release of QEMU, which will be out in some time in spring 2018.
Edit as of 9 Mar 2018 -- the relevant fixes are now in QEMU master (commits fd1e5c81796, df2a5cf4c8) and will be in 2.12.

AT91SAM9263ek booting Linux with Device Tree failed

I have problem booting Linux 3.16.1. I have compiled sources from http://www.kernel.org with at91sam9263_defconfig.
I have added Flattened Device Tree support in Boot options.
Followin tips suggested in this (https://www.slideshare.net/softpapa/device-tree-support-on-arm-linux-8930303) presentation to turn on Support device tree in /proc but i don't have that option in menuconfig.
I have U-Boot bootloader version 2014.10rc2 which supports device tree.
I have generated dtb from script shipped with kernel:
make at91sam9263ek.dtb
And now i'm getting this error:
Welcome to minicom 2.5
OPTIONS: I18n
Compiled on Feb 9 2011, 14:45:00.
Port /dev/ttyS0
Press CTRL-A Z for help on special keys
RomBOOT
>
U-Boot 2014.10-rc2-00200-g9170818-dirty (Sep 23 2014 - 15:16:39)
CPU: AT91SAM9263
Crystal frequency: 16.368 MHz
CPU clock : 199.919 MHz
Master clock : 99.960 MHz
DRAM: 64 MiB
WARNING: Caches not enabled
NAND: 256 MiB
MMC: mci: 0
In: serial
Out: serial
Err: serial
Net: macb0
Warning: Your board does not use generic board. Please read
doc/README.generic-board and take action. Boards not
upgraded by the late 2014 may break or be removed.
Hit any key to stop autoboot: 0
U-Boot> tftp uImage
macb0: Starting autonegotiation...
macb0: Autonegotiation complete
macb0: link up, 100Mbps full-duplex (lpa: 0xcde1)
Using macb0 device
TFTP from server 192.168.1.247; our IP address is 192.168.1.240
Filename 'uImage'.
Load address: 0x22000000
Loading: #################################################################
#################################################################
#################################################################
##############
1.2 MiB/s
done
Bytes transferred = 3068016 (2ed070 hex)
U-Boot> tftp 20000000 dt
macb0: link up, 100Mbps full-duplex (lpa: 0xcde1)
Using macb0 device
TFTP from server 192.168.1.247; our IP address is 192.168.1.240
Filename 'dt'.
Load address: 0x20000000
Loading: #
340.8 KiB/s
done
Bytes transferred = 13279 (33df hex)
U-Boot> bootm 22000000 - 20000000
## Booting kernel from Legacy Image at 22000000 ...
Image Name: Linux-3.16.1
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 3067952 Bytes = 2.9 MiB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
## Flattened Device Tree blob at 20000000
Booting using the fdt blob at 0x20000000
Loading Kernel Image ... OK
Loading Device Tree to 23ea3000, end 23ea93de ... OK
Starting kernel ...
Uncompressing Linux... done, booting the kernel.
Error: unrecognized/unsupported device tree compatible list:
[ 'atmel,at91sam9263ek' 'atmel,at91sam9263' 'atmel,at91sam9' ]
Available machine support:
ID (hex) NAME
000004b2 Atmel AT91SAM9263-EK
Please check your kernel config and/or bootloader.
Solution:
Add this line to .config:
CONFIG_MACH_AT91SAM9_DT=y
The correct configuration for this board when using device tree is at91_dt_defconfig.
However, I am quite surprised to see someone trying to use such an old kernel. This board is fully supported upstream. Why don't you use v5.3? If this doesn't work, please report any bug, we will be happy to help correct them.

Suspend/Wakeup in Beagleboard using Angstrom

I want to suspend and resume my Beagleboard C4.
I checked out the latest Angstrom kernel using,
git clone git://git.kernel.org/pub/scm/linux/kernel/git/tmlind/linux-omap-2.6.git linux-omap-2.6
But the kernel is not booting properly. It is stuck at
[ 4.755645] EXT3-fs (mmcblk0p2): recovery complete
[ 4.991027] EXT3-fs (mmcblk0p2): mounted filesystem with ordered data mode
[ 4.999023] VFS: Mounted root (ext3 filesystem) readonly on device 179:2.
[ 5.006652] Freeing init memory: 304K
[ 6.316253] EXT3-fs (mmcblk0p2): using internal journal
Has anyone resolved this issue?
I compiled the kernel directly on the beagleboard. Same result.
I compiled the kernel using toolchain. Same result.
Is it something related to the u-boot/File system?

Resources