Fedora Linux Support Community & Resources Center
  #1  
Old 20th April 2005, 04:08 PM
haydenm315 Offline
Registered User
 
Join Date: Apr 2005
Posts: 3
Allocated filehandles always grow till max and then it crashes

I recently installed Fedora Core 3 for one of our advertising servers and am running into an interesting problem. The number of allocated file handles continues to grow until it reaches the configured max. Once it reaches the max, messages can be found in the system log.

kernel: VFS: file-max limit 21989 reached

Other messages can also be found
kernel: HighMem: empty
kernel: Out of Memory: Killed process 12736 (crond).
kernel: Memory: 223176k/229312k available (2068k kernel

The server is an advertising server and interestingly enough there is a perl script run from cron every minute that checks to see if there are problems with the server. Our advertising software has been in use since the RedHat 6.2 days and I haven't seen this problem before.

I've done some searching on the web and have taken note to /proc/sys/fs/file-max and /proc/sys/fs/file-nr

file-nr is currently below
6120 0 21989

anyone familiar with the proc structure probably knows file-max is the 3rd column of file-nr.

I've searched the net for a number of days and can't seem to find a solution. Interestingly enough, I've commented out the cron entries related to the health monitoring and the files haven't gone up in the last 20 minutes. Any idea why we'd start having this problem with Fedora?
Reply With Quote
  #2  
Old 21st April 2005, 05:10 AM
Jman Offline
Registered User
 
Join Date: Mar 2004
Location: Minnesota, USA
Age: 29
Posts: 7,909
That out of memory error is also I problem. Maybe the perl script isn't exiting so you have hundreds of perl processes opening files.

My suggestion is to monitor the monitor system, that is, run top and watch to see what's hogging all the memory.

What else runs on this server, httpd?
Reply With Quote
  #3  
Old 29th April 2005, 03:46 PM
haydenm315 Offline
Registered User
 
Join Date: Apr 2005
Posts: 3
Quote:
Originally Posted by Jman
That out of memory error is also I problem. Maybe the perl script isn't exiting so you have hundreds of perl processes opening files.

My suggestion is to monitor the monitor system, that is, run top and watch to see what's hogging all the memory.

What else runs on this server, httpd?
Interestingly enough, I just got back from vacation and the number of filehandles open looks normal. Uptime verifies it hasn't been rebooted in 8 days. I left the monitoring program commented in the crontab, so at least it's pinpointed. I'm just trying to figure out why it hasn't had problems on other systems. I've enabled it again and am going to poke around and see what I can find.
Reply With Quote
  #4  
Old 29th April 2005, 03:55 PM
AndyGreen Offline
Registered User
 
Join Date: Apr 2005
Location: Northants, UK
Posts: 2,026
Jman's idea sounds like a good one.... if the script that "runs every minute" does not check to see if another instance is already running (which should be a reportable error by the sound of things) then it can over an hour or two build up a monstrous cancerlike army of stalled perl scripts that grows until the rivets pop out of your machine.

I have an hourly script on a server that emails me my http logs since the last report every hour, with some formatting. It tries to do a host on the IP address, if that fails as it often does it falls back to a whois. I learnt by this that when a *registrars* whois server is down, there is NO TIMEOUT in jwhois, nor is one configurable. Therefore overnight my machine also "caught cancer" the same way until I saw in the morning no report emails had arrived. I could then ssh in and nuke the processes.
__________________
Freelance RedHat Certified Engineer - http://northantsIT.com
Reply With Quote
Reply

Tags
allocated, crashes, filehandles, grow, max, till

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
IO APIC resources couldnt be allocated polhen Installation, Upgrades and Live Media 8 30th October 2009 02:15 PM
Wants to change allocated memory SunilReddy Installation, Upgrades and Live Media 3 31st December 2007 07:36 PM
Can Linux Grow Up? Xavior Linux Chat 99 20th April 2007 06:07 PM
Char Device Driver - Buffer Length - Allocated/ Used sciconf Using Fedora 0 14th August 2006 11:15 PM


Current GMT-time: 07:19 (Friday, 19-12-2014)

TopSubscribe to XML RSS for all Threads in all ForumsFedoraForumDotOrg Archive
logo

All trademarks, and forum posts in this site are property of their respective owner(s).
FedoraForum.org is privately owned and is not directly sponsored by the Fedora Project or Red Hat, Inc.

Privacy Policy | Term of Use | Posting Guidelines | Archive | Contact Us | Founding Members

Powered by vBulletin® Copyright ©2000 - 2012, vBulletin Solutions, Inc.

FedoraForum is Powered by RedHat
Creek Side - Stockholm-Arlanda Airport (ARN) Travel Photos on Instagram - Marina of Koper Travel Photos - Te Whiti Park Instagram Photos - Texas State Fair Photos on Instagram