Allocated filehandles always grow till max and then it crashes

[haydenm315]

I recently installed Fedora Core 3 for one of our advertising servers and am running into an interesting problem. The number of allocated file handles continues to grow until it reaches the configured max. Once it reaches the max, messages can be found in the system log.

kernel: VFS: file-max limit 21989 reached

Other messages can also be found
kernel: HighMem: empty
kernel: Out of Memory: Killed process 12736 (crond).
kernel: Memory: 223176k/229312k available (2068k kernel

The server is an advertising server and interestingly enough there is a perl script run from cron every minute that checks to see if there are problems with the server. Our advertising software has been in use since the RedHat 6.2 days and I haven't seen this problem before.

I've done some searching on the web and have taken note to /proc/sys/fs/file-max and /proc/sys/fs/file-nr

file-nr is currently below
6120 0 21989

anyone familiar with the proc structure probably knows file-max is the 3rd column of file-nr.

I've searched the net for a number of days and can't seem to find a solution. Interestingly enough, I've commented out the cron entries related to the health monitoring and the files haven't gone up in the last 20 minutes. Any idea why we'd start having this problem with Fedora?

[Jman]

That out of memory error is also I problem. Maybe the perl script isn't exiting so you have hundreds of perl processes opening files.

My suggestion is to monitor the monitor system, that is, run top and watch to see what's hogging all the memory.

What else runs on this server, httpd?

[haydenm315]

Originally Posted by Jman

That out of memory error is also I problem. Maybe the perl script isn't exiting so you have hundreds of perl processes opening files.

My suggestion is to monitor the monitor system, that is, run top and watch to see what's hogging all the memory.

What else runs on this server, httpd?

Interestingly enough, I just got back from vacation and the number of filehandles open looks normal. Uptime verifies it hasn't been rebooted in 8 days. I left the monitoring program commented in the crontab, so at least it's pinpointed. I'm just trying to figure out why it hasn't had problems on other systems. I've enabled it again and am going to poke around and see what I can find.

[AndyGreen]

Jman's idea sounds like a good one.... if the script that "runs every minute" does not check to see if another instance is already running (which should be a reportable error by the sound of things) then it can over an hour or two build up a monstrous cancerlike army of stalled perl scripts that grows until the rivets pop out of your machine.

I have an hourly script on a server that emails me my http logs since the last report every hour, with some formatting. It tries to do a host on the IP address, if that fails as it often does it falls back to a whois. I learnt by this that when a *registrars* whois server is down, there is NO TIMEOUT in jwhois, nor is one configurable. Therefore overnight my machine also "caught cancer" the same way until I saw in the morning no report emails had arrived. I could then ssh in and nuke the processes.