In the process of flushing yum and trying to get CentOS to update, the system froze and I had to do a hard reset. Upon rebooting, the OS failed to load with several error messages:
XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1757 of file fs/xfs/xfs_alloc.c. ... Corruption of in-memory data detected. Shutting down filesystem Please unmount the filesystem and rectify the problem(s) Failed to recover EFIs task mount:365 blocked for more than 120 seconds.
Looks like some file system corruption and the last error about the system trying to perform a mount just kept repeating. Running xfs_repair in the console did nothing in this environment.
Next I tried booting from a CentOS install ISO and I chose the option to Rescue an existing installation. Choosing any of the 3 options in rescue mode would just result it the system hanging indefinitely without ever reaching a console. I chose Option 3 to skip to console and alt tabbing revealed:
Running ... mount -t xfs -o defaults,ro /dev/mapper/centos-root /mnt/sysimage
Now it is clear that the system is hanging on trying to mount the root partition which we know has problems. However I could not find a way to enter the console without rescue mode automatically trying to mount the root partition.
Next I tried booting a CentOS live ISO. Turns out it has no xfs tools so that was a dead end.
I decided to try booting a SystemRescueCd which has some xfs tools on it.
I chose the default boot option and reached a console. Ideally one should create a dump of the damaged partition by running xfs_metadump, restore the dump to an image using xfs_mdrestore, and then perform the repair on that image. That way you can evaluate the repair done to the image to see if it will work or irreversibly damage your actual data.
I went ahead and just did the repair on the actual data:
xfs_repair -L /dev/mapper/centos-rootThe repair completed and after a reboot CentOS started successfully.