#StackBounty: #linux #debian #kernel #nfs Poor NFS performance under high load

Bounty: 100

I have two Debian servers connected to a shared NFS server.

Mounts:

nfs.internal.com:/volume1/SHARE on /share type nfs (rw,nosuid,relatime,sync,vers=3,rsize=131072,wsize=131072,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,soft,noac,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.0.100,mountvers=3,mountport=892,mountproto=tcp,fsc,lookupcache=none,local_lock=none,addr=172.16.0.100)

Sometimes, the load on the server will increase. There seems to be a breakpoint where the load causes NFS operations to slow down enough that a runaway load condition occurs and we eventually reach a limit on processes configured for the service run on this server (1500).

When the load increases on one server, the other is not affected, so I don’t believe networking or the NFS server to be the culprit.

Under normal conditions, load ~10

ops/s       rpc bklog
20637.000           0.000

getattr:           ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)
               10453.000        2334.176           0.223        0 (0.0%)           0.339           0.467
access:            ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)
                4893.000        1139.078           0.233        0 (0.0%)           0.338           0.469

Under "medium" load of ~100:

ops/s       rpc bklog
13374.200           0.000

read:              ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)
                  10.600        1202.169         113.412        0 (0.0%)           0.623          26.868
write:             ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)
                   7.600           5.462           0.719        0 (0.0%)           0.553           7.395

I don’t have it saved, but at the highest load levels of 1500, the avg exe time will reach 4-5 seconds.

From what I understand, an exe value that is much higher than RTT means that the requests are queuing on the local server rather than waiting on the NFS server.

It sounds like the number of simultaneous requests to the NFS server is specified by sunrpc.tcp_slot_table_entries. In recent kernel versions this value is dynamically increased up tosunrpc.tcp_max_slot_table_entries, however I have never seen it go above the default of two.

Could this be the cause of the performance issues, and what would prevent this value from scaling like it should be?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.