when rpc.mountd flushes auth.unix.gid

[Colin Hudler]

We have a few hundred computers mounting an NFS server in a typical
LDAP-based users (nss) setup. We frequently add and remove exports and
use exportfs -r to update etab. Every time we do so, the clients report
"NFS server not responding" and start backing off their requests. After
a painful 3-5 minutes, they recover and life is normal again.

We discovered that when the rpc.mountd cache flushing occurs, our NIS
system is overwhelmed with grouplist requests and this obviously blocks
things. We are working on that problem separately, and I admit this to
be a weakness in our setup. My question is simple.

Why does it flush auth.unix.gid when the etab changed? I think it makes
unnecessary work for rpc.mountd because the gids are unlikely to have
changed, and they already have a reasonable expiration policy.

--
Colin
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Tom Haynes]

We have a few hundred computers mounting an NFS server in a typical L=

DAP-based users (nss) setup. We frequently add and remove exports and u=
se exportfs -r to update stab.

I know this isn=92t your question, but you would be better served by us=
ing explicit exportfs -a and exportfs -r commands for the specific chan=
ges.

Every time we do so, the clients report "NFS server not responding" a=

nd start backing off their requests. After a painful 3-5 minutes, they =
recover and life is normal again.

=20
We discovered that when the rpc.mountd cache flushing occurs, our NIS=

system is overwhelmed with grouplist requests and this obviously block=
s things. We are working on that problem separately, and I admit this t=
o be a weakness in our setup. My question is simple.

=20
Why does it flush auth.unix.gid when the etab changed? I think it mak=

es unnecessary work for rpc.mountd because the gids are unlikely to hav=
e changed,

Another assumption is that exports rarely change. I expect your setup i=
s an exception to the rule.

and they already have a reasonable expiration policy.

One way to read what the man page states for exportfs -r:

-r Reexport all directories, synchronizing /var/lib/nfs/=
etab with /etc/exports and files under /etc/exports.d.
This option removes entries in /var/lib/nfs/etab which ha=
ve been deleted from /etc/exports or files under
/etc/exports.d, and removes any entries from the kernel e=
xport table which are no longer valid.

is that it only removes entries which have been deleted.

Instead, it removes all entries and reexports those that are still vali=
d. The remove of all is what blows away the auth.unix.gid caching.

Using exportfs -a <path> and exportfs -r <path> should solve this for y=
ou.

=20
--
Colin
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" =

in

More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Colin Hudler]

We have a few hundred computers mounting an NFS server in a typical =

LDAP-based users (nss) setup. We frequently add and remove exports and =
use exportfs -r to update stab.

I know this isn=92t your question, but you would be better served by =

using explicit exportfs -a and exportfs -r commands for the specific ch=
anges.
Thank you for the insight and suggestions. We are considering changing=20
our methods but it requires breaking some (long-standing, internal)=20
abstractions and weighing the risk associated with that. In short, our=20
automations suck and cannot be changed so easily. However, manual-mode=20
is always an option.

Every time we do so, the clients report "NFS server not responding" =

and start backing off their requests. After a painful 3-5 minutes, they=
recover and life is normal again.

We discovered that when the rpc.mountd cache flushing occurs, our NI=

S system is overwhelmed with grouplist requests and this obviously bloc=
ks things. We are working on that problem separately, and I admit this =
to be a weakness in our setup. My question is simple.

Why does it flush auth.unix.gid when the etab changed? I think it ma=

kes unnecessary work for rpc.mountd because the gids are unlikely to ha=
ve changed,

Another assumption is that exports rarely change. I expect your setup=

is an exception to the rule.

and they already have a reasonable expiration policy.

-r Reexport all directories, synchronizing /var/lib/n=

fs/etab with /etc/exports and files under /etc/exports.d.

This option removes entries in /var/lib/nfs/etab which=

have been deleted from /etc/exports or files under

/etc/exports.d, and removes any entries from the kerne=

l export table which are no longer valid.

is that it only removes entries which have been deleted.
Instead, it removes all entries and reexports those that are still va=

lid. The remove of all is what blows away the auth.unix.gid caching.

Using exportfs -a <path> and exportfs -r <path> should solve this for=

you.


Understood. I am now tempted to rework exportfs -r into a loop over=20
dump(). Thanks again.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Jeff Layton]

On Fri, 17 Oct 2014 09:42:14 -0500

Post by Colin Hudler
We have a few hundred computers mounting an NFS server in a typical
LDAP-based users (nss) setup. We frequently add and remove exports and
use exportfs -r to update etab. Every time we do so, the clients report
"NFS server not responding" and start backing off their requests. After
a painful 3-5 minutes, they recover and life is normal again.
We discovered that when the rpc.mountd cache flushing occurs, our NIS
system is overwhelmed with grouplist requests and this obviously blocks
things. We are working on that problem separately, and I admit this to
be a weakness in our setup. My question is simple.
Why does it flush auth.unix.gid when the etab changed? I think it makes
unnecessary work for rpc.mountd because the gids are unlikely to have
changed, and they already have a reasonable expiration policy.

Most likely because no one really cared until now.

When exports change, cache_flush() is called and that function flushes
out all of the kernel caches.

I expect that could be made to do something a bit more granular, but
you may need to do some archaeology in mountd/exportfs (and the kernel)
to ensure that you're not missing anything.

[Tom Haynes]

Post by Jeff Layton
On Fri, 17 Oct 2014 09:42:14 -0500

Post by Colin Hudler
We have a few hundred computers mounting an NFS server in a typical
LDAP-based users (nss) setup. We frequently add and remove exports and
use exportfs -r to update etab. Every time we do so, the clients report
"NFS server not responding" and start backing off their requests. After
a painful 3-5 minutes, they recover and life is normal again.
We discovered that when the rpc.mountd cache flushing occurs, our NIS
system is overwhelmed with grouplist requests and this obviously blocks
things. We are working on that problem separately, and I admit this to
be a weakness in our setup. My question is simple.
Why does it flush auth.unix.gid when the etab changed? I think it makes
unnecessary work for rpc.mountd because the gids are unlikely to have
changed, and they already have a reasonable expiration policy.

Most likely because no one really cared until now.
When exports change, cache_flush() is called and that function flushes
out all of the kernel caches.
I expect that could be made to do something a bit more granular, but
you may need to do some archaeology in mountd/exportfs (and the kernel)
to ensure that you're not missing anything.

One thing would be to not remove the exports which are going to be added back in.

The catch here is that you have to account for new entries which need to be added.

Post by Jeff Layton
--
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Jeff Layton]

On Fri, 17 Oct 2014 17:24:14 -0500

Post by Tom Haynes

Post by Jeff Layton
On Fri, 17 Oct 2014 09:42:14 -0500

Post by Colin Hudler
We have a few hundred computers mounting an NFS server in a typical
LDAP-based users (nss) setup. We frequently add and remove exports and
use exportfs -r to update etab. Every time we do so, the clients report
"NFS server not responding" and start backing off their requests. After
a painful 3-5 minutes, they recover and life is normal again.
We discovered that when the rpc.mountd cache flushing occurs, our NIS
system is overwhelmed with grouplist requests and this obviously blocks
things. We are working on that problem separately, and I admit this to
be a weakness in our setup. My question is simple.
Why does it flush auth.unix.gid when the etab changed? I think it makes
unnecessary work for rpc.mountd because the gids are unlikely to have
changed, and they already have a reasonable expiration policy.

Most likely because no one really cared until now.
When exports change, cache_flush() is called and that function flushes
out all of the kernel caches.
I expect that could be made to do something a bit more granular, but
you may need to do some archaeology in mountd/exportfs (and the kernel)
to ensure that you're not missing anything.

One thing would be to not remove the exports which are going to be added back in.
The catch here is that you have to account for new entries which need to be added.

I'm not sure that flushing the uid or gid caches is really necessary on
an exports change at all. I don't think we expect that info to change.

In practical terms, we might be able to change exportfs to just flush
the nfsd.fh and nfsd.export caches instead of a full cache_flush() ?

[Jeff Layton]

On Fri, 17 Oct 2014 21:21:18 -0500

Post by Jeff Layton
On Fri, 17 Oct 2014 17:24:14 -0500

Post by Tom Haynes

Post by Jeff Layton
On Fri, 17 Oct 2014 09:42:14 -0500

Post by Colin Hudler
We have a few hundred computers mounting an NFS server in a typical
LDAP-based users (nss) setup. We frequently add and remove exports and
use exportfs -r to update etab. Every time we do so, the clients report
"NFS server not responding" and start backing off their requests. After
a painful 3-5 minutes, they recover and life is normal again.
We discovered that when the rpc.mountd cache flushing occurs, our NIS
system is overwhelmed with grouplist requests and this obviously blocks
things. We are working on that problem separately, and I admit this to
be a weakness in our setup. My question is simple.
Why does it flush auth.unix.gid when the etab changed? I think it makes
unnecessary work for rpc.mountd because the gids are unlikely to have
changed, and they already have a reasonable expiration policy.

Most likely because no one really cared until now.
When exports change, cache_flush() is called and that function flushes
out all of the kernel caches.
I expect that could be made to do something a bit more granular, but
you may need to do some archaeology in mountd/exportfs (and the kernel)
to ensure that you're not missing anything.

One thing would be to not remove the exports which are going to be added back in.
The catch here is that you have to account for new entries which need to be added.

I'm not sure that flushing the uid or gid caches is really necessary on
an exports change at all. I don't think we expect that info to change.

Is there a manual way to flush these caches?
Bump down the default TTL?

The manual way is to write to /proc/net/rpc/*/flush (which is what
cache_flush() in nfs-utils does). The comments over it say:

/* flush the kNFSd caches.
* Set the flush time to the mtime of _PATH_ETAB or
* if force, to now.
* the caches to flush are:
* auth.unix.ip nfsd.export nfsd.fh
*/

...but it looks like auth.unix.gid was added in 2007 and the
comment wasn't updated.

Post by Jeff Layton
In practical terms, we might be able to change exportfs to just flush
the nfsd.fh and nfsd.export caches instead of a full cache_flush() ?
--