Ext4 fall down go boom

After a reboot today whilst installing KDE 4.3.1 I noticed the following messages in my kernel (2.6.31-rc8) logs (courtesy of the KDE file watcher that was following /var/log/kern.log):

Sep 6 13:53:10 quad kernel: [ 142.842723] EXT4-fs error (device dm-7): ext4_mb_generate_buddy: EXT4-fs: group 287: 5812 blocks in bitmap, 5418 in gd
Sep 6 13:53:11 quad kernel: [ 143.452041] JBD: Spotted dirty metadata buffer (dev = dm-7, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
Sep 6 13:53:11 quad kernel: [ 143.486915] JBD: Spotted dirty metadata buffer (dev = dm-7, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
Sep 6 13:53:11 quad kernel: [ 143.486942] JBD: Spotted dirty metadata buffer (dev = dm-7, blocknr = 0). There's a risk of filesystem corruption in case of system crash.

That didn’t look too good, so I immediately did a “git pull” and happily found 2.6.31-rc9 was out so built that and then did a dual backup, rsync’ing to my local ZFS-FUSE drive (which takes snapshots so I can go backwards in time) and also an rsnapshot to a USB external disk. Then with trepidation I rebooted and found myself looking at an fsck error on /home due to shared blocks between an image and part of my local clone of Linus’s kernel git tree (d’oh!). Whilst the fsck got the filesystem mountable again it did result in not being able to view the kernel git tree due to missing files so I decided it was far safer to just revert to my latest backup, which worked like a charm (phew!).

Moral of the story – keeping backups is good – keeping lots of backups is even better, especially when running with release candidate kernels! 😉