Another useful technique to debug Linux kernel crashes at boot is to analyze the kernel log after the crash. This is only possible if the RAM memory is persistent across reboots and does not get initialized by the bootloader.
As U-Boot keeps the memory intact, we can use this method to peek at the kernel login memory in search of clues.
Looking at the kernel source, we can see how the log ring buffer is set up in kernel/printk/printk.c and also note that it is stored in __log_buf:
- To find the location of the kernel buffer, we will use the System.map file created by the Linux build process, which maps symbols with virtual addresses using the following command:
$ grep __log_buf System.map 80c37864 b __log_buf
- To convert the virtual address to a physical address, we look at how __virt_to_phys() is defined for ARM:
x - PAGE_OFFSET + PHYS_OFFSET
- The PAGE_OFFSET variable is defined in the kernel configuration as follows:
config PAGE_OFFSET
hex
default 0x40000000 if VMSPLIT_1G
default 0x80000000 if VMSPLIT_2G
default 0xC0000000
Some ARM platforms, such as the i.MX6, will dynamically patch the __virt_to_phys() translation at runtime, so PHYS_OFFSET will depend on where the kernel is loaded into memory. As this can vary, the calculation we just saw is platform-specific.
For the Wandboard, the physical address for 0x80c37864 is 0x10c37864.
- We can then force a reboot using a magic SysRq key, which needs to be enabled in the kernel configuration with CONFIG_MAGIC_SYSRQ, but is enabled in the Wandboard by default:
$ echo b > /proc/sysrq-trigger
- We then dump that memory address from U-Boot as follows:
=> md.l 0x10c37864 10c37864: 00000000 00000000 00210034 c6000000 ........4.!..... 10c37874: 746f6f42 20676e69 756e694c 6e6f2078 Booting Linux on 10c37884: 79687020 61636973 5043206c 78302055 physical CPU 0x 10c37894: 00000030 00000000 00000000 009300a4 0............... 10c378a4: a6000000 756e694c 65762078 6f697372 ....Linux versio 10c378b4: 2e34206e 35312e31 3433312d 2d393930 n 4.1.15-134099- 10c378c4: 35616667 30623765 35393866 69642d62 gfa5e7b0f895b-di 10c378d4: 20797472 656c6128 6f6c4078 696c2d67 rty (alex@log-li 10c378e4: 2d78756e 612d7068 7a6e6f67 20296c61 nux-hp-agonzal) 10c378f4: 63636728 72657620 6e6f6973 312e3720 (gcc version 7.1 10c37904: 2820302e 29434347 23202920 4d532031 .0 (GCC) ) #1 SM 10c37914: 52502050 504d4545 72462054 75412069 P PREEMPT Fri Au 10c37924: 35322067 3a393120 313a3535 45432030 g 25 19:55:10 CE 10c37934: 32205453 00373130 00000000 00000000 ST 2017......... 10c37944: 00400050 c6000000 3a555043 4d524120 [email protected]: ARM 10c37954: 50203776 65636f72 726f7373 31345b20 v7 Processor [41