Good Day, Friends!

Some of my SBCs are printing some error to the kernel log (dmesg) and most of the time show malfunctioning behaviour afterwards. On the same device some boots can be successful and there are no malfunctions at all. The tricky part is that the device is operational until some reboots. Other device do not show any malfunction after several times of reflashing and around 130+ reboots.
Some of the output is indicating that there is an eMMC error.

[Fri May 31 15:05:42 2024 <   26.288612>] blk_update_request: I/O error, dev mmcblk1, sector 13717576 op 0x1:(WRITE) flags 0x0 phys_seg 10 prio class 0
[Fri May 31 15:05:42 2024 <    0.002546>] EXT4-fs warning (device mmcblk1p3): ext4_end_bio:347: I/O error 10 writing to inode 16334 starting block 1714707)
[Fri May 31 15:05:42 2024 <    0.002575>] Buffer I/O error on device mmcblk1p3, logical block 5641
[Fri May 31 15:05:42 2024 <    0.002579>] Buffer I/O error on device mmcblk1p3, logical block 5642
[Fri May 31 15:05:42 2024 <    0.002445>] Buffer I/O error on device mmcblk1p3, logical block 5643
[Fri May 31 15:05:42 2024 <    0.002414>] Buffer I/O error on device mmcblk1p3, logical block 5644
[Fri May 31 15:05:42 2024 <    0.002368>] Buffer I/O error on device mmcblk1p3, logical block 5645
[Fri May 31 15:05:42 2024 <    0.002361>] Buffer I/O error on device mmcblk1p3, logical block 5646
[Fri May 31 15:05:42 2024 <    0.002351>] Buffer I/O error on device mmcblk1p3, logical block 5647
[Fri May 31 15:05:42 2024 <    0.002276>] Buffer I/O error on device mmcblk1p3, logical block 5648
[Fri May 31 15:05:42 2024 <    0.002259>] Buffer I/O error on device mmcblk1p3, logical block 5649
[Fri May 31 15:05:42 2024 <    0.002202>] Buffer I/O error on device mmcblk1p3, logical block 5650
[Fri May 31 15:05:42 2024 <    0.002216>] EXT4-fs warning (device mmcblk1p3): ext4_end_bio:347: I/O error 10 writing to inode 16335 starting block 1714709)

Or like

Fri May 31 15:10:38 2024 <    0.002265>] Buffer I/O error on device mmcblk1p3, logical block 7190
[Fri May 31 15:10:38 2024 <    0.002224>] EXT4-fs warning (device mmcblk1p3): ext4_end_bio:347: I/O error 10 writing to inode 16339 starting block 1716249)
[Fri May 31 17:05:15 2024 < 6877.280226>] brcmfmac: brcmf_sdio_rxglom: glom read of 7680 bytes failed: -84
[Fri May 31 17:05:15 2024 <    0.002549>] brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame
[Fri May 31 17:05:16 2024 <    1.203433>] blk_update_request: I/O error, dev mmcblk1, sector 13769810 op 0x1:(WRITE) flags 0x4000 phys_seg 1 prio class 0
[Fri May 31 17:05:16 2024 <    0.002745>] blk_update_request: I/O error, dev mmcblk1, sector 13770321 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Fri May 31 17:05:16 2024 <    0.002585>] EXT4-fs warning (device mmcblk1p3): ext4_end_bio:347: I/O error 10 writing to inode 32595 starting block 1721344)
[Fri May 31 17:05:16 2024 <    0.002665>] buffer_io_error: 2 callbacks suppressed

Or like

Fri May 31 17:05:26 2024 <    0.002139>] blk_update_request: I/O error, dev mmcblk1, sector 14334036 op 0x1:(WRITE) flags 0x4000 phys_seg 2 prio class 0
[Fri May 31 17:05:26 2024 <    0.001844>] EXT4-fs warning (device mmcblk1p3): ext4_end_bio:347: I/O error 10 writing to inode 32595 starting block 1792012)
[Fri May 31 17:05:26 2024 <    0.003852>] blk_update_request: I/O error, dev mmcblk1, sector 14340184 op 0x1:(WRITE) flags 0x4000 phys_seg 1 prio class 0
[Fri May 31 17:05:26 2024 <    0.002171>] blk_update_request: I/O error, dev mmcblk1, sector 14340695 op 0x1:(WRITE) flags 0x4000 phys_seg 1 prio class 0

Since eMMC and BT/Wifi chip are on the same SDIO bus, a malfunction of the connectivity can occour.

[Sat May 25 08:25:51 2024 <  468.446931>] brcmfmac: brcmf_sdio_readframes: read 84 bytes from channel 2 failed: -84
[Sat May 25 08:25:51 2024 <    0.002573>] brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame

Or like

[Sat May 25 08:12:40 2024 <   88.291871>] brcmfmac: brcmf_sdio_readframes: RXHEADER FAILED: -84
[Sat May 25 08:12:40 2024 <    0.002577>] brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame, send NAK

Or like

[Wed May 22 18:40:17 2024 <   55.495985>] brcmfmac: brcmf_sdio_readframes: RXHEADER FAILED: -84
[Wed May 22 18:40:17 2024 <    0.002653>] brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame, send NAK
[Wed May 22 18:40:19 2024 <    2.509178>] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
[Wed May 22 18:42:04 2024 <  104.247478>] Bluetooth: hci0: link tx timeout
[Wed May 22 18:42:05 2024 <    1.051471>] Bluetooth: hci0: link tx timeout

Full with malfunction

Full log expected output, default operation

Which tests are suitable for such kind of error?

Which tools are available on Linux to test or reproduce this error?

Thanks in advance everyone!