F13-14 Hangs and clock drifts

[Goulou]

Hi All,

I've been running a fedora server for over a year now, and has seen some very strange issue, that really make it uncomfortable to use, and that I cannot manage to solve easily.
The problem is as follow : sometimes (every 20-30 minutes or so : this time is quite random), the server completely hangs.
I'm using it mainly remotely, through ssh and nfs. What I see when it "hangs" is the following
-no ping response
-nfs stalled
-ssh sessions hangs (for example, if I run a "top" command", it just isn't updated anymore
-disk activity led stays completely off (in normal activity, it is almost always blinking, even due to "internal" server activities, so network disconnection doesn't explain that either)

The most strange thing is that, to resume it, I basically have 3 options : wait (sometimes few tens of seconds, usually between 2 and 4 minutes!), just hit a key on the keyboard!, or do something like un/plugging any king of usb peripheral (which makes me think of some interruption mechanism that is stalled).

When it "wakes up", I see my "top" session over ssh suddenly being quickly updated hundreds of times (for all it has not received during the "pause"), ping says that the packets have actually all been received (with long times, for example, packet 1 : 80xxxms, packet 2 : 79xxxms, packet 3 : 78xxxms..... packet 79 : 1xxxms, packet 80 : 0.xxxms), the disk is quite overloaded for a few seconds, and everything is back at normal!!
Furthermore, I think this i related, but my clock drifts for few hours per day (3 holding minutes every 20-30 minutes makes me think there is some relation!)
I tried to set up ntpd to compensate it, but sometimes the suspensions are just too long, and I ended up with

Nov 27 01:22:36 server ntpd[699]: 0.0.0.0 0617 07 panic_stop +1203 s; set clock manually within 1000 s

and ntpd dies...

You'll ask me to have a look at the log, which I did by

tail -f /var/log/* /var/log/*/*

, but when the suspension happens, there is absolutely nothing new in all those files!

Version information :

uname -a
Linux vmserver.grelot.net 2.6.35.6-48.fc14.x86_64 #1 SMP Fri Oct 22 15:36:08 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

I said in the title "F13-F14", because I already had the problem with F13, but after some time it disappeared (I may have changed some configuration, I have to admit...). I still have a backup of the "/etc" tree of that "working" F13.
The hardware for F13[working] and F14[not_working] is the same : Phenom II X4 on an ASUS M3N78 PRO, data on raid 10, system on separated SATA disk, 8Gb RAM, some qemu vms running (between 4 and 6).

It would be really great if someone had an idea about this problem : it annoyed me for several months now, and I cannot stay calm anymore when I suddenly cannot do anything but wait, since I'm not near enough to press the "Ctrl" key on the keyboard!!

Thanks a lot in advance,

Goulou.

[jpollard]

It sort of sounds like a hardware problem with your clock.

You might try booting the system with nohz=off for testing. In one system
I have (an old Sony laptop) when the system warms up the clock starts
acting strange (not the time clock, but an interrupt clock). In this system
it would hang the system with a burst of interrupts - would lock the system
up (or severely slow it down) for 10-20 seconds. nohz=off seemed to help
(but the hardware problem remains).

[Goulou]

Unfortunately I don't have any screen plugged onto the server, so I'm afraid I'll have to wait for few days before any attempts at modifying boot parameters.
However, you can be sure I'll tell you the result.
Meanwhile, I'll try to have a look at the bios, see if anything looks wrong.

By the way, are there any drawbacks with putting the "nohz=off" option? Does it disable the tickless capability of the kernel? And in that case, won't that increase cpu overall usage?

Thanks for your help!

Goulou

[jpollard]

Yes, it does increase CPU usage a bit - but if it works smoother, then it
will help usage, especially if it turns out to be a hardware problem.

[Goulou]

I found a display to plug, and did test the nohz=off option... Without success

For information :

 cat /proc/cmdline 
ro root=UUID=e7cc31ce-8afd-4157-a8f1-bc3ce70d8f24 rd_MD_UUID=bf9db417:dc08c1db:b7c728be:f4ed51e6 rd_LVM_LV=vg_raid10/lv_swap rd_NO_LUKS rd_NO_DM LANG=fr_FR.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=fr-latin9 rhgb nohz=off

I really don't know what else to try... Any idea?

[vallimar]

Did you try disabling HPET? Add the "hpet=disable" to the kernel line when booting to see if that helps.
Something else to try is "clocksource=pit". Your problem could be apic or acpi related as well so you could try altering some of those settings if problems persist.

[sidebrnz]

You didn't say (or I didn't see) which way your clock's drifting. If it's running slow, it may well be nothing more than your CMOS battery needs replacing. The BIOS is designed to make the clock run slow as a warning that you need to change the battery.

[marko]

Originally Posted by sidebrnz

You didn't say (or I didn't see) which way your clock's drifting. If it's running slow, it may well be nothing more than your CMOS battery needs replacing. The BIOS is designed to make the clock run slow as a warning that you need to change the battery.

Checkout the BIOS update listing for that board, I don't know if I've ever seen a board
with so many updates:

http://support.asus.com/download/dow...e=map&f_type=3


Especially look at
revision 0603 -> "2. Resolve system freezing problem when using certain memory. "

and
revision 1003 -> "Improve stability and performance of LAN function. "

You might want to update the bios ?

[Goulou]

The bios may have been the problem : my cpu model was not "officially" recognised by the hardware (it said "model unknown" at startup and in lshw), while it is now.

I updated the bios to the latest version (I have to admit that they have really improved the process!! : Put it on a usb stick, go to the bios, launch the built-in utility, browse the stick and update!). So far (2 hours) I didn't notice any problem.
Last week was horrible thought : after 1 week, the delay was of 32 hours!!

I also removed the nohz=off option, since it didn't seem to do anything.

I'll update the post if it drifts again, otherwise it means that bios update solved it! (it may be of interest for future users).

Thanks a lot for your help.

Goulou