Kernel version update 2.6.23 for Fedora 7 breaks SATA ICH8R driver

[xylog]

I have been able to boot in all kernel versions up to and including 2.6.22 but with the latest kernel update the system can't boot. I currently have two kernels installed 2.6.22.4-65.fc7, which works fine and 2.6.23.1-10.fc7 which does not boot. Here is my hardware info:

OS: Fedora release 7 (Moonshine)
Default run level: 5
Language: en_US.UTF-8
Platform: i686
BogoMIPS: 4264.23
CPU Vendor: GenuineIntel
CPU Model: Intel(R) Core(TM)2 CPU 6420 @ 2.13GHz
Number of CPUs: 2
CPU Speed: 2128
System Memory: 2001
System Swap: 1983
Vendor: Unknown
System:
Form factor: unknown
Kernel: 2.6.22.4-65.fc7
SELinux Enabled: False
SELinux Enforce: targeted

Devices
=================================
(32902:10273:32902:20813) pci, ahci, OTHER, 82801HR/HO/HH (ICH8R/DO/DH) 6 port SATA AHCI Controller

[xylog]

Relevant DMESG output:

SCSI subsystem initialized
libata version 2.21 loaded.
ahci 0000:00:1f.2: version 2.2
ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 19 (level, low) -> IRQ 19
usb 7-3: new high speed USB device using ehci_hcd and address 2
usb 7-3: configuration #1 chosen from 1 choice
usb 4-2: new low speed USB device using uhci_hcd and address 3
ahci 0000:00:1f.2: AHCI 0001.0100 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
ahci 0000:00:1f.2: flags: 64bit ncq led clo pio slum part
PCI: Setting latency timer of device 0000:00:1f.2 to 64
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
ata1: SATA max UDMA/133 cmd 0xf8830100 ctl 0x00000000 bmdma 0x00000000 irq 218
ata2: SATA max UDMA/133 cmd 0xf8830180 ctl 0x00000000 bmdma 0x00000000 irq 218
ata3: SATA max UDMA/133 cmd 0xf8830200 ctl 0x00000000 bmdma 0x00000000 irq 218
ata4: SATA max UDMA/133 cmd 0xf8830280 ctl 0x00000000 bmdma 0x00000000 irq 218
ata5: SATA max UDMA/133 cmd 0xf8830300 ctl 0x00000000 bmdma 0x00000000 irq 218
ata6: SATA max UDMA/133 cmd 0xf8830380 ctl 0x00000000 bmdma 0x00000000 irq 218
usb 4-2: configuration #1 chosen from 1 choice
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1.00: ATA-7: WDC WD5000YS-01MPB1, 09.02E09, max UDMA/133
ata1.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32)
ata1.00: configured for UDMA/133
hiddev96: USB HID v1.10 Device [APC Back-UPS ES 725 FW:802.n5.D USB FW:n5] on usb-0000:00:1d.1-2
ata2: SATA link down (SStatus 0 SControl 300)
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: ATA-7: WDC WD2500KS-00MJB0, 02.01C03, max UDMA/133
ata3.00: 488397168 sectors, multi 0: LBA48
ata3.00: configured for UDMA/133
ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata4.00: ATA-7: WDC WD2500KS-00MJB0, 02.01C03, max UDMA/133
ata4.00: 488397168 sectors, multi 0: LBA48
ata4.00: configured for UDMA/133
ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata5.00: ATA-7: WDC WD5000AAKS-00TMA0, 12.01C01, max UDMA/133
ata5.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32)
ata5.00: configured for UDMA/133
ata6: SATA link down (SStatus 0 SControl 300)
scsi 0:0:0:0: Direct-Access ATA WDC WD5000YS-01M 09.0 PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sda: sda1 sda2
sd 0:0:0:0: [sda] Attached SCSI disk
scsi 2:0:0:0: Direct-Access ATA WDC WD2500KS-00M 02.0 PQ: 0 ANSI: 5
sd 2:0:0:0: [sdb] 488397168 512-byte hardware sectors (250059 MB)
sd 2:0:0:0: [sdb] Write Protect is off
sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 2:0:0:0: [sdb] 488397168 512-byte hardware sectors (250059 MB)
sd 2:0:0:0: [sdb] Write Protect is off
sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sdb: sdb1
sd 2:0:0:0: [sdb] Attached SCSI disk
scsi 3:0:0:0: Direct-Access ATA WDC WD2500KS-00M 02.0 PQ: 0 ANSI: 5
sd 3:0:0:0: [sdc] 488397168 512-byte hardware sectors (250059 MB)
sd 3:0:0:0: [sdc] Write Protect is off
sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 3:0:0:0: [sdc] 488397168 512-byte hardware sectors (250059 MB)
sd 3:0:0:0: [sdc] Write Protect is off
sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sdc: sdc1
sd 3:0:0:0: [sdc] Attached SCSI disk
scsi 4:0:0:0: Direct-Access ATA WDC WD5000AAKS-0 12.0 PQ: 0 ANSI: 5
sd 4:0:0:0: [sdd] 976773168 512-byte hardware sectors (500108 MB)
sd 4:0:0:0: [sdd] Write Protect is off
sd 4:0:0:0: [sdd] Mode Sense: 00 3a 00 00
sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 4:0:0:0: [sdd] 976773168 512-byte hardware sectors (500108 MB)
sd 4:0:0:0: [sdd] Write Protect is off
sd 4:0:0:0: [sdd] Mode Sense: 00 3a 00 00
sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sdd: sdd1
sd 4:0:0:0: [sdd] Attached SCSI disk
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 17 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:02:00.0 to 64
scsi6 : pata_marvell
ata7: PATA max UDMA/100 cmd 0x00011018 ctl 0x00011026 bmdma 0x00011000 irq 16
ata8: DUMMY
BAR5:00:00 01:7F 02:22 03A 04:00 05:00 06:00 07:00 08:00 09:00 0A:00 0B:00 0C:01 0D:00 0E:00 0F:00
ata7.00: ATAPI: LITE-ON DVDRW LH-18A1P, GL0G, max UDMA/66
ata7.00: limited to UDMA/33 due to 40-wire cable
ata7.00: configured for UDMA/33
scsi 6:0:0:0: CD-ROM LITE-ON DVDRW LH-18A1P GL0G PQ: 0 ANSI: 5
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel@redhat.com
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: dm-0: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 98271243
ext3_orphan_cleanup: deleting unreferenced inode 98274224
ext3_orphan_cleanup: deleting unreferenced inode 98274189
ext3_orphan_cleanup: deleting unreferenced inode 98274109
ext3_orphan_cleanup: deleting unreferenced inode 98271379
ext3_orphan_cleanup: deleting unreferenced inode 98271373
ext3_orphan_cleanup: deleting unreferenced inode 98271355
ext3_orphan_cleanup: deleting unreferenced inode 98271278
ext3_orphan_cleanup: deleting unreferenced inode 77049246
ext3_orphan_cleanup: deleting unreferenced inode 77065537
ext3_orphan_cleanup: deleting unreferenced inode 77060918
EXT3-fs: dm-0: 11 orphan inodes deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: Disabled at runtime.
SELinux: Unregistering netfilter hooks
audit(1194189763.121:2): selinux=0 auid=4294967295
sd 0:0:0:0: Attached scsi generic sg0 type 0
sd 2:0:0:0: Attached scsi generic sg1 type 0
sd 3:0:0:0: Attached scsi generic sg2 type 0
sd 4:0:0:0: Attached scsi generic sg3 type 0
scsi 6:0:0:0: Attached scsi generic sg4 type 5
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.01 (21-Jan-2007)
iTCO_wdt: Found a ICH8DH TCO device (Version=2, TCOBASE=0x0460)
iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI
Copyright (c) 1999-2006 Intel Corporation.
ACPI: PCI Interrupt 0000:00:19.0[A] -> GSI 20 (level, low) -> IRQ 22
PCI: Setting latency timer of device 0000:00:19.0 to 64
e1000: 0000:00:19.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1) 00:19:d1:80:25:78
sr0: scsi3-mmc drive: 48x/48x writer dvd-ram cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 6:0:0:0: Attached scsi CD-ROM sr0

[Brian1]

Is the output from the 2.6.31 kernel?
So it does boot to a point and lock?
How far does it get?

Brian

[xylog]

This is output from the 2.6.22 kernel. The 2.6.23 kernel does not boot the hard disk so I cannot give you output from it.

[Brian1]

What is in /boot/grub/grub.conf file?
Also the list of files in /boot

If not booting with no text at all then hard to say.
Did you wait say 5 minutes to see if anything happened?
Does the harddrive led show activity like older kernels do when booting?

Brian

[xylog]

It boots to a kernel panic when it cant mount the root filesystem. Here are my grub.conf and /boot:

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
# initrd /initrd-version.img
#boot=/dev/sda
default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Fedora (2.6.23.1-10.fc7)
root (hd0,0)
kernel /vmlinuz-2.6.23.1-10.fc7 ro root=/dev/VolGroup00/LogVol00 clock=pmtmr
initrd /initrd-2.6.23.1-10.fc7.img
title Fedora (2.6.22.4-65.fc7)
root (hd0,0)
kernel /vmlinuz-2.6.22.4-65.fc7 ro root=/dev/VolGroup00/LogVol00 clock=pmtmr
initrd /initrd-2.6.22.4-65.fc7.img


-rw-r--r-- 1 root root 76K 2007-08-21 22:46 config-2.6.22.4-65.fc7
-rw-r--r-- 1 root root 77K 2007-10-19 15:45 config-2.6.23.1-10.fc7
drwxr-xr-x 2 root root 1.0K 2007-11-04 10:16 grub
-rw------- 1 root root 3.5M 2007-06-15 15:26 initrd-2.6.21-coretemp-2.6.21-1.3228.fc7.img
-rw------- 1 root root 3.5M 2007-06-24 19:42 initrd-2.6.21-custom-1.3228.fc7.img
-rw------- 1 root root 3.5M 2007-06-15 11:52 initrd-2.6.21-prep.img
-rw------- 1 root root 3.5M 2007-08-24 19:20 initrd-2.6.22.4-65.fc7.img
-rw------- 1 root root 3.5M 2007-09-03 19:01 initrd-2.6.22.4-syncerndis.img
-rw------- 1 root root 3.4M 2007-11-04 10:16 initrd-2.6.23.1-10.fc7.img
drwx------ 2 root root 12K 2007-06-15 06:36 lost+found
lrwxrwxrwx 1 root root 26 2007-09-03 18:47 System.map -> System.map-2.6.22.4-65.fc7
-rw-r--r-- 1 root root 897K 2007-06-24 19:42 System.map-2.6.21-custom-1.3228.fc7
-rw-r--r-- 1 root root 897K 2007-06-24 19:38 System.map-2.6.21-custom-1.3228.fc7.old
-rw-r--r-- 1 root root 897K 2007-06-15 11:51 System.map-2.6.21-prep
-rw-r--r-- 1 root root 920K 2007-08-21 22:46 System.map-2.6.22.4-65.fc7
-rw-r--r-- 1 root root 825K 2007-10-19 15:45 System.map-2.6.23.1-10.fc7
lrwxrwxrwx 1 root root 23 2007-09-03 18:47 vmlinuz -> vmlinuz-2.6.22.4-65.fc7
-rw-r--r-- 1 root root 1.8M 2007-06-24 19:42 vmlinuz-2.6.21-custom-1.3228.fc7
-rw-r--r-- 1 root root 1.8M 2007-06-24 19:38 vmlinuz-2.6.21-custom-1.3228.fc7.old
-rw-r--r-- 1 root root 1.8M 2007-06-15 11:51 vmlinuz-2.6.21-prep
-rw-r--r-- 1 root root 1.9M 2007-08-21 22:46 vmlinuz-2.6.22.4-65.fc7
-rw-r--r-- 1 root root 1.9M 2007-10-19 15:45 vmlinuz-2.6.23.1-10.fc7

[Brian1]

Maybe remove the clock=pmtmr mght help.
Other than that if nothing happens at boot then I am at a lost without actually being there.
Contact the Fedora team for help. May be a bug that is hardware specific.
What other hardware is in the system. ' /sbin/lspci -v '

Brian

[xylog]

I tried removing clock=pmtmr also tried agp=off but it did not boot. I also tried vga=792 but I still couldn't see enough of message to get anything useful. Still from what I can see the sata drivers are failing to load. Interestingly I found a Fedora 7.92 rescue CD with kernel-2.6.23-0.214 and it boots fine.

[brr872002]

XT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: dm-0: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 98271243
ext3_orphan_cleanup: deleting unreferenced inode 98274224
ext3_orphan_cleanup: deleting unreferenced inode 98274189
ext3_orphan_cleanup: deleting unreferenced inode 98274109
ext3_orphan_cleanup: deleting unreferenced inode 98271379
ext3_orphan_cleanup: deleting unreferenced inode 98271373
ext3_orphan_cleanup: deleting unreferenced inode 98271355
ext3_orphan_cleanup: deleting unreferenced inode 98271278
ext3_orphan_cleanup: deleting unreferenced inode 77049246
ext3_orphan_cleanup: deleting unreferenced inode 77065537
ext3_orphan_cleanup: deleting unreferenced inode 77060918
EXT3-fs: dm-0: 11 orphan inodes deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
It seems some file system errors installed files may be corrupt.
code
yum remove kernel 2.6.23.1-10

#shutdown -F -t secs 1

yum clean all
yum update kernel

Try it.

[xylog]

Uninstalling and re-installing the kernel did not help.

If I get the time I am going to try connecting a serial cable and capturing the output on my laptop.

Just a shot in the dark here but did you happen to change the drives identity?

In other words what I mean is that previously fedora named EIDE hard drives as "hdx" (where x is a,b,c, etc...).

With F7 (might be with F8 too) if you have not wiped the partitions on the drive and written over the mbr (master boot record) of the disk then the disk will not mount. The reason is because the OS is looking for the name sdx (where x is a,b,c, etc) as the name for the drive rather than the old naming convention. If the OS doesn't see the new name it assumes there is something wrong with the system and won't mount the drive.

This may be the reason why you keep getting so many "ext3_orphan_cleanup: deleting unreferenced inode" in your DMESG output.

[xylog]

I didn't change the MBR. As I said it still boots fine in the older kernel (and newer test kernel ) I really dont have the time to properly test this but if I did, I would likely build a new vanilla version of the 2.6.23.1 kernel and see if I have the same problem. If not I could look at the Fedora specific patches and see if any of them break it. As it is I am happy enough to run on the older kernel until a newer one comes out. Also, I will likely be upgrading to F8 soon anyway.

[xylog]

Well stranger and stranger. I can boot successfully from the Fedora 8 Live CD so I did the upgrade to Fedora 8 and guess what - I couldnt boot. So it's something specific to my build and not a specific Fedora problem it seems. I have managed to get up and running on F8 with and older 2.6.21 kernel. Not sure where to look next for the solution to my problem. At this point my only option seems to be to do a fresh re-install of Fedora 8.

[xylog]

I installed a fresh copy of Fedora 8 and restored my home directories from backup and I am reasonably well back to a working setup. I am, however, left with a bitter taste in my mouth - I had previously thought a kernel update to be self contained, and not rely things outside of the kernel package to boot, however this appears to be what was happening in my case - the same kernel that boots off a fresh install would not boot when installed in my existing OS. I would still like to understand what could have caused this to happen, in order to avoid it in the future. Any thoughts on what might have caused it would be appreciated. I have backup images of the broken config, so I can try and verify any theories.