Kernel upgrade broke system - kernel panic and unable to boot

[itsjustarumour]

Fedora 16.x86_64
Kernel 3.3.2-6.fc16.x86_64
NVidia binary blob.

Well, I thought it would never happen to me, but I installed a kernel update and its borked my system - and as luck would have it, this is my pride-and-joy "production" machine... BAH.

On rebooting, I just got the message:

Kernel Panic - not synching: VFS: Unable to mount root fs on unknown-block (0,0)

I had two old kernels available, but discovered I couldn't now boot from either of those either - I got the blue lines to indicate the OS was loading, but then at the end they froze. Checking the verbose output showed it was sticking at:

Started LSB: Automatically install DKMS modules for new kernels (OK)

I burnt a copy of the DVD .ISO and booted off that into rescue mode... so far so good. I discovered there was an incomplete yum transaction pending. Running a quick

yum-complete-transaction

fixed that bit, but I was still unable to boot.

I was able to get an ethernet connection working, and ran yum-update and yum-upgrade much installed another new kernel, but still no joy. I tried running

yum reinstall kernel

for each of my three kernels in turn, but that told me that the kernel wasn't available.

I've tried a few things, and have ended up uninstalling both newer kernels, leaving me with just the one last known-good kernel. I checked that the initrd /boot/initramfs-kernel-version.img existed for the kernel version I was tying to boot in /boot/, but that all looked in order (I tried running dracut --force anyway).

At some point (I can't remember exactly where, and I'm not at the machine at the moment) I got this output that seems to show what the problem is:

E: FATAL: Could not load /lib/modules/3.1.0-7.fc16.x86_64/modules.dep:
No such file or directory
find: @/lib/modes/3/1/0-7.fc16.x86_64/@ : No such file or directory

So - my question is this - how do I reload (or rebuild?) that particular kernel module? Or is my system completely borked? I have to confess I'm rapidly reaching the outer limits of my Fedora knowledge on this one, and most fixes I've read assume the ability to choose a working kernel at the grub boot screen, which I don't have...

Any help or wisdom gratefully received!

Best,

itsjustarumour

[glennzo]

A shameless bump for the gentleman with the broken system.

Glennzo yells from the corner of the room "hey! can we get some help over here?"

Wish I could be of help. What kernels are actually installed?

PHP Code:

rpm -qa | grep kernel 


What does find show with regard to modules.dep?

PHP Code:

find / -name modules.dep 


Do you get the same error no matter what kernel you attempt to boot?

[smr54]

What I have found, from time to time, is that an update will break UUID. (I'm not saying this is the case this time, but, each time, the symptoms are similar to your description--can't mount, gives an error about some modules not loading).

So, I've always been able to fix it by booting into rescue mode and changing fstab from the UUID to /dev/sdwhatever, and the same with grub.)

Might be worth a try, but obviously, backup fstab and grub.cfg first.

[itsjustarumour]

Originally Posted by glennzo

A shameless bump for the gentleman with the broken system.

Glennzo yells from the corner of the room "hey! can we get some help over here?"

Wish I could be of help. What kernels are actually installed?

PHP Code:

rpm -qa | grep kernel 


What does find show with regard to modules.dep?

PHP Code:

find / -name modules.dep 


Do you get the same error no matter what kernel you attempt to boot?

Many thanks for the bump Glennzo, and glad for the help!

So, without further ado, heres what I've got (I've accessed my install using the DVD ISO and "Rescue Mode", then run "chroot /mnt/sysimage")...

Running

PHP Code:

rpm -qa | grep kernel 


gives:

sh-4.2# rpm -qa | grep kernel
libreport-plugin-kerneloops-2.0.8-4.fc16.x86_64
kernel-devel-3.3.4-3.fc16.x86_64
kernel-devel-3.3.4-1.fc16.x86_64
kernel-headers-3.3.4-3.fc16.x86_64
kernel-devel-3.3.2-6.fc16.x86_64
kernel-3.3.4-3.fc16.x86_64
kernel-3.3.2-6.fc16.x86_64
abrt-addon-kerneloops-2.0.7-3.fc16.x86_64
sh-4.2#

Running

PHP Code:

find / -name modules.dep 


gives:

sh-4.2# find / -name modules.dep
/lib/modules/3.3.4-3.fc16.x86_64/modules.dep
/lib/modules/3.3.2-6.fc16.x86_64/modules.dep

As far as I can tell I got the same error message off every kernel I tried to boot from. Right now, I only have one kernel left showing in the grub boot menu - 3.3.2-6.fc16.x86_64 which was the last kernel I know to be "good" before I ran into problems.

Does any of that help?

Cheers,

itsjustarumour

---------- Post added at 07:15 PM ---------- Previous post was at 07:12 PM ----------

Originally Posted by smr54

What I have found, from time to time, is that an update will break UUID. (I'm not saying this is the case this time, but, each time, the symptoms are similar to your description--can't mount, gives an error about some modules not loading).

So, I've always been able to fix it by booting into rescue mode and changing fstab from the UUID to /dev/sdwhatever, and the same with grub.)

Might be worth a try, but obviously, backup fstab and grub.cfg first.

Thanks for that smr54, has given me a couple of other ideas for things to look at. Cheers!

itsjustarumour

[itsjustarumour]

Not making much progress unfortunately.

I've booted into rescue mode from the DVD and had a look at /etc/fstab and /boot/grub2/grub.cfg..... and I'm not quite sure what I'm meant to do from there

[smr54]

Well, find the long UUID number from grub--somewhere in the menu entry part, and make sure it matches what you have in /etc/fstab. There's a slight chance they'll be different and if so, that's the problem. As I said, it's not happened often to me, but when it has, I'd see something similar to what you describe.

[goodfella]

Originally Posted by itsjustarumour

Not making much progress unfortunately.

I've booted into rescue mode from the DVD and had a look at /etc/fstab and /boot/grub2/grub.cfg..... and I'm not quite sure what I'm meant to do from there

Boot off you DVD and do:
$ ls -s /dev/disk/by-uuid

you will see the symbolic links to /dev/sdxx names.

Substitute the /dev/sdxx names for the UUID's in your /etc/fstab and reboot.

[diamond_ramsey]

Originally Posted by goodfella

Boot off you DVD and do:
$ ls -s /dev/disk/by-uuid

you will see the symbolic links to /dev/sdxx names.

Substitute the /dev/sdxx names for the UUID's in your /etc/fstab and reboot.

Very cool, I am checking my "by-uuid" data and noting the location.

[solo2101]

I had this before... If I remember correctly....... I remove the bad kernel, and then I "re-update" my system...

[itsjustarumour]

Originally Posted by smr54

Well, find the long UUID number from grub--somewhere in the menu entry part, and make sure it matches what you have in /etc/fstab. There's a slight chance they'll be different and if so, that's the problem. As I said, it's not happened often to me, but when it has, I'd see something similar to what you describe.

OK, thanks for that swr54

I've had a look at what you suggest. Heres the "menuentry" part of my /boot/grub2/grub.cfg

###BEGIN /etc/grub.d/10_linux ###
menuentry 'Fedora (3.3.2-6.fc16.x86_64)' --class fedora --class gnu-linux --class gnu --class os {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
set root='(hd0,gpt2)'
search --no-floppy --fs-uuid --set=root 8d914f4f-d517-4fc8-9cae-2bcc83f6f9e5
echo 'Loading Fedora (3.3.2-6.fc16.x86_64)'
linux /vmlinuz-3.3.2-6.fc16.x86_64 root=/dev/mapper/vg_coolermaster-lv_root ro rd.lvm.lv=vg_coolermaster/lv_root rd.dm=0 quiet SYSFONT=latarcyrheb-sun16 rhgb rd.md=0 KEYTABLE=uk rd.luks=0 rd.lvm.lv=vg_coolermaster/lv_swap LANG=e$LANG=en_GB.UTF-8 nouveau.modeset=0 rd.driver.blacklist=nouveau
echo 'Loading initial ramdisk...'
initrd /initramfs-3.3.2-6.fc16.x86_64.img

... and heres my /etc/fstab

#
#/etc/fstab
#Created by Anaconda on Fri Dec 16 17:2706 2011
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_coolermaster-lv_root /
UUID=8d914f4f-d517-4fc8-9cae-2bcc83f6f9e5 /boot
/dev/mapper/vg_coolermaster-lv_swap swap

So looking at both of those, the UUID is the same - 8d914f4f-d517-4fc8-9cae-2bcc83f6f9e5. Is there anything obvious I'm missing there? I admit I'm on a bit of a learning curve with this...

---------- Post added at 04:59 PM ---------- Previous post was at 04:55 PM ----------

Originally Posted by goodfella

Boot off you DVD and do:
$ ls -s /dev/disk/by-uuid

you will see the symbolic links to /dev/sdxx names.

Substitute the /dev/sdxx names for the UUID's in your /etc/fstab and reboot.

@goodfella - thanks for the reply! I've given it a try

Running:

PHP Code:

ls -s /dev/disk/by-uuid 


gives me:

sh-4.2# ls -s /dev/disk/by-uuid
total 0
0 8d914f4f-d517-4fc8-9cae-2bcc83f6f9e5 0 b48c5c4e-b2a5-49ed-a587-38396ffed6b9 0 c5117af6-2d8a-4162-bff1-8008d85a9103 0 e7372ef0-69e5-4197-9cc3-d3609a071550
sh-4.2#

Does that look right? The first UUID is the correct one - 8d914f4f-d517-4fc8-9cae-2bcc83f6f9e5 - which already appears in my /etc/fstab (see previous post)

[goodfella]

Originally Posted by goodfella

Boot off you DVD and do:
$ ls -s /dev/disk/by-uuid

you will see the symbolic links to /dev/sdxx names.

Substitute the /dev/sdxx names for the UUID's in your /etc/fstab and reboot.

Sorry, it should be "ls -l" not "ls -s"

Here is what it looks like on my machine for one directory entry:

lrwxrwxrwx. 1 root root 10 May 10 09:09 0110326c-5e3f-4387-a0c1-b6f94690f049 -> ../../sdc4

So if sdc4 is what I want then I put /dev/sdc4 in fstab.

[itsjustarumour]

Originally Posted by goodfella

Sorry, it should be "ls -l" not "ls -s"

Here is what it looks like on my machine for one directory entry:

lrwxrwxrwx. 1 root root 10 May 10 09:09 0110326c-5e3f-4387-a0c1-b6f94690f049 -> ../../sdc4

So if sdc4 is what I want then I put /dev/sdc4 in fstab.

Ah - thanks for the update! Worked just as you suggested, but on reboot, I still have the same problem - the boot process is halting at what verbose output shows to be "Started LSB: Automatically install DKMS modules for new kernels (OK)".

So back to square one...

[EMGoodyear]

Have you tried removing the Nvidia driver? I know I am having the GPU display problem at the moment. From what I have seen elsewhere it has been fixed, but no go for me and my 9600GT.

Reference:

http://forums.fedoraforum.org/showthread.php?t=277911

http://forums.fedoraforum.org/showth...da+kernel+boot

[itsjustarumour]

Originally Posted by EMGoodyear

Have you tried removing the Nvidia driver? I know I am having the GPU display problem at the moment. From what I have seen elsewhere it has been fixed, but no go for me and my 9600GT.

Reference:

Thanks for the post, but I did try that... and alas no joy.

I've given up for now, guess its just one of those things - perfect time I guess for a reinstall with F17 being released today