Fedora Linux Support Community & Resources Center
  #1  
Old 2nd April 2012, 07:58 PM
cotton213 Offline
Registered User
 
Join Date: May 2007
Posts: 4
linuxfirefox
nfsd spikes when remote mount

We are installing Fedora 16 on our office workstations (up from F12). We are not using NFSv4 (although I can't seem to make the nfsd4 process go away). We export filesystems in the usual way with /etc/exports, but when an older system (an F12, Slackware, RHEL) mounts this filesystem and tries to do anything with the files there, the number of nfsd process on the server (the F16 system) doubles or triples and the CPU usages spike to as high as 50% per process. This renders the mount unusable, and the server itself unusable.

The only workaround that I can find is to specify udp for the protocol when mounting. Any idea why this is? It's not a good workaround. If I must do something like this, I'd prefer to set something on the server rather that on 100 potential clients.

Any insight would be appreciated.
Reply With Quote
  #2  
Old 2nd April 2012, 09:00 PM
jpollard Online
Registered User
 
Join Date: Aug 2009
Location: Waldorf, Maryland
Posts: 6,815
linuxfirefox
Re: nfsd spikes when remote mount

The NFS processes are part of the kernel. They are not user level processes. The NFSv4 process falls back to v3 (or v2) when the protocol is v3/2 specific.

If you have 100 clients you will want about 64 server processes (past experience). The default of 8 causes a REAL performance problem for high loads. (/etc/sysconfig/nfs). The parameter there is RPCNFSDCOUNT.

The actual number you want depends on how many concurrent accesses you want to support. The default of 8 works for 8-16 simultaneous use... but on an 8 core server, 64 should work better. Part of the problem is that the buffers used by client communications saturate the 8 real quick... more actually reduces the CPU load by eliminating some process thrashing (causing even more load). For average access (not a simultaneous mount from all systems at once) 32 should do reasonably well.

Oh, and you can disable the tcp use in that file too (--no-tcp option to nfsd in the RPCNFSDARGS parameter value). Once set it does take a reboot to get put in the kernel and verified.

I don't know your hardware (cpu/memory/network/disk) structure so testing is needed to tune NFS appropriately.

My background on this was tuning NFS for 16TB backup device copying files to it from a NAS. The NFS server was a Suse system from SGI. Until I upped the RPCNFSDCOUNT parameter it would take many hours (over 10) to scan the filesystem (50 million files). After upping it to 64 things went down to 45 minutes.

BTW, I don't recommend using F16 for a production server. There are just too many things that don't work as expected.

Last edited by jpollard; 2nd April 2012 at 09:02 PM.
Reply With Quote
  #3  
Old 2nd April 2012, 10:05 PM
cotton213 Offline
Registered User
 
Join Date: May 2007
Posts: 4
linuxfirefox
Re: nfsd spikes when remote mount

Thanks so much for your quick reply. I will try manipulating the RPCNFSDCOUNT and see what that gets me. To clarify a bit more, the Fedora 16 system is just a workstation. The engineers export subversion workspaces so that they can be accessed by build computers. We never have a bunch of build systems accessing the export at once, but any one or two build computers out of a dozen or more will access the export at any given time.

We never had a problem with this until F16. The engineer's build gets about 10 minutes in and then suddenly the nfsd processes multiply and spike -- freezing both computers.

I can reproduce this by doing a "find" or something on these mounted files. F16 to F16 communication seems to work ok (doesn't hang either computer).

Thanks again -- your hints may help me with my insanely long backup problems, too.
Reply With Quote
  #4  
Old 3rd April 2012, 08:00 PM
jpollard Online
Registered User
 
Join Date: Aug 2009
Location: Waldorf, Maryland
Posts: 6,815
linuxfirefox
Re: nfsd spikes when remote mount

Just a follow on -

If you think about it, each file access takes an NFSD to process it. This is why finds saturate so fast. It also depends on what the find is doing - just identifying files by name pattern is not so bad, but a find based on date causes each file inode to be accessed, that that takes another for each inode... Then there are the cache manipulations involved on the server after returning the data...

I'm not sure why the nfsd processes multiply.. unless there is some overload feature I hadn't run into before. Maybe there is another limit (something like a maxnfsd count).

I do understand that a build could acess a lot of files relatively quickly (and possibly in parallel too) and that would lock the thing up.

BTW, the client locks should be interruptable (unless a non-interupptable option is used), allowing them to continue, though that would abort any builds going on.
Reply With Quote
Reply

Tags
mount, nfsd, remote, spikes

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
mount remote dvd-rw drive BdclrzktUv Using Fedora 0 18th April 2011 07:15 PM
CPU usage spikes Linkmax Using Fedora 7 25th September 2006 08:47 AM
mount remote device via nfs drs Using Fedora 0 7th May 2006 05:57 AM
mount remote share chrisman81 Using Fedora 1 21st February 2005 05:29 AM


Current GMT-time: 15:44 (Friday, 22-08-2014)

TopSubscribe to XML RSS for all Threads in all ForumsFedoraForumDotOrg Archive
logo

All trademarks, and forum posts in this site are property of their respective owner(s).
FedoraForum.org is privately owned and is not directly sponsored by the Fedora Project or Red Hat, Inc.

Privacy Policy | Term of Use | Posting Guidelines | Archive | Contact Us | Founding Members

Powered by vBulletin® Copyright ©2000 - 2012, vBulletin Solutions, Inc.

FedoraForum is Powered by RedHat