Discussion:
Performance: NFS v3 mounting with 'noac' option
AK
2007-03-07 14:54:09 UTC
Permalink
Hi All,

Trying to understand what performance impact could
result if we mount the file system with 'noac' option.

We have multiple file systems begin mounted from the
same NFS server. One of our user's application
requires to mount the file system with 'noac' option.

We have user directories under /scratch file system,
Now we would like to mount this file system with
'noac' option so that we can make one of our user
happy. But this would afftect every user. Isn't it
true?

So Now if we just decide to create a new FS
/newScratchUserA and mount it with 'noac' option;
WILL THIS AFFECT the other files systems mounted from
the same server, if so how bad could it be.

Are there any other alternatives to reduce the impact
of using this option on other mounted File systems.

really appreciate any advise ....

~ thanks!



____________________________________________________________________________________
Never miss an email again!
Yahoo! Toolbar alerts you the instant new Mail arrives.
http://tools.search.yahoo.com/toolbar/features/mail/

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
Peter Staubach
2007-03-07 17:55:48 UTC
Permalink
Post by AK
Hi All,
Trying to understand what performance impact could
result if we mount the file system with 'noac' option.
We have multiple file systems begin mounted from the
same NFS server. One of our user's application
requires to mount the file system with 'noac' option.
What about this application requires that the "noac" option be used?
Post by AK
We have user directories under /scratch file system,
Now we would like to mount this file system with
'noac' option so that we can make one of our user
happy. But this would afftect every user. Isn't it
true?
Well, you could try mounting each user's home directory individually.
The "noac" option should only apply to a specific mounted instance.
Post by AK
So Now if we just decide to create a new FS
/newScratchUserA and mount it with 'noac' option;
WILL THIS AFFECT the other files systems mounted from
the same server, if so how bad could it be.
Are there any other alternatives to reduce the impact
of using this option on other mounted File systems.
One of the impacts of using the "noac" option is much more network
traffic and it will be composed of small packets. If the server is
busy handling all of the additional GETATTR requests, then it won't
be handling other requests. The client will also be a bit busier
with generating the additional requests, instead of just doing what
might be considered more useful work.

So, there should be no direct impact to other mounted file systems.
However, there may be indirect impacts.

ps

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
AK
2007-03-07 21:32:57 UTC
Permalink
Post by AK
Post by AK
We have multiple file systems begin mounted from
the
Post by AK
same NFS server. One of our user's application
requires to mount the file system with 'noac'
option.
What about this application requires that the "noac"
option be used?
application is romio with mpich2 ...
Post by AK
One of the impacts of using the "noac" option is
much more network
traffic and it will be composed of small packets.
If the server is
busy handling all of the additional GETATTR
requests, then it won't
be handling other requests. The client will also be
a bit busier
with generating the additional requests, instead of
just doing what
might be considered more useful work.
So, there should be no direct impact to other
mounted file systems.
However, there may be indirect impacts.
indirect impacts: can you please elaborate on this.

~ thanks!



____________________________________________________________________________________
It's here! Your new message!
Get new email alerts with the free Yahoo! Toolbar.
http://tools.search.yahoo.com/toolbar/features/mail/

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
Peter Staubach
2007-03-07 22:39:28 UTC
Permalink
Post by AK
Post by AK
Post by AK
We have multiple file systems begin mounted from
the
Post by AK
same NFS server. One of our user's application
requires to mount the file system with 'noac'
option.
What about this application requires that the "noac"
option be used?
application is romio with mpich2 ...
Sorry, I'm not familiar with those. What about either one of those
require "noac" semantics?
Post by AK
Post by AK
One of the impacts of using the "noac" option is
much more network
traffic and it will be composed of small packets.
If the server is
busy handling all of the additional GETATTR
requests, then it won't
be handling other requests. The client will also be
a bit busier
with generating the additional requests, instead of
just doing what
might be considered more useful work.
So, there should be no direct impact to other
mounted file systems.
However, there may be indirect impacts.
indirect impacts: can you please elaborate on this.
I thought that I did. :-)

If the system is busy handling the "noac" aspects of one file system,
then it isn't doing something else on one of the other file systems.
System resources are limited and using up more of them for one thing
leaves less for others.

ps

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
AK
2007-03-08 14:55:50 UTC
Permalink
Post by Peter Staubach
Post by AK
Post by Peter Staubach
What about this application requires that the
"noac"
Post by AK
Post by Peter Staubach
option be used?
application is romio with mpich2 ...
Sorry, I'm not familiar with those. What about
either one of those
require "noac" semantics?
"To use ROMIO on NFS, file locking with fcntl must
work correctly on the NFS installation. On some
installations, fcntl locks don't work. To get them to
work, you need to use Version 3 of NFS, ensure that
the lockd daemon is running on all the machines, and
have the system administrator mount the NFS file
system with the "noac" option (no attribute caching).
Turning off attribute caching may reduce performance,
but it is necessary for correct behavior."
Post by Peter Staubach
Post by AK
Post by Peter Staubach
One of the impacts of using the "noac" option is
much more network
traffic and it will be composed of small packets.
If the server is
busy handling all of the additional GETATTR
requests, then it won't
be handling other requests. The client will also
be
Post by AK
Post by Peter Staubach
a bit busier
with generating the additional requests, instead
of
Post by AK
Post by Peter Staubach
just doing what
might be considered more useful work.
So, there should be no direct impact to other
mounted file systems.
However, there may be indirect impacts.
indirect impacts: can you please elaborate on
this.
I thought that I did. :-)
If the system is busy handling the "noac" aspects of
one file system,
then it isn't doing something else on one of the
other file systems.
System resources are limited and using up more of
them for one thing
leaves less for others.
Got it!!! when you mentioned "However, there may be
indirect impacts." in the last line, i was like there
could be more, and hence curious :-)
Post by Peter Staubach
ps
____________________________________________________________________________________
The fish are biting.
Get more visitors on your site using Yahoo! Search Marketing.
http://searchmarketing.yahoo.com/arp/sponsoredsearch_v2.php

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
Peter Staubach
2007-03-08 16:13:24 UTC
Permalink
Post by AK
Post by Peter Staubach
Post by AK
Post by Peter Staubach
What about this application requires that the
"noac"
Post by AK
Post by Peter Staubach
option be used?
application is romio with mpich2 ...
Sorry, I'm not familiar with those. What about
either one of those
require "noac" semantics?
"To use ROMIO on NFS, file locking with fcntl must
work correctly on the NFS installation. On some
installations, fcntl locks don't work. To get them to
work, you need to use Version 3 of NFS, ensure that
the lockd daemon is running on all the machines, and
have the system administrator mount the NFS file
system with the "noac" option (no attribute caching).
Turning off attribute caching may reduce performance,
but it is necessary for correct behavior."
I think that I would assume that fcntl style locking works on all current
implementations, that this is not NFS version dependent, and also that
"noac" is not required in combination with the locking in order to get
correct behavior.

ps

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
Talpey, Thomas
2007-03-08 16:19:46 UTC
Permalink
Post by Peter Staubach
Post by AK
"To use ROMIO on NFS, file locking with fcntl must
work correctly on the NFS installation. On some
installations, fcntl locks don't work. To get them to
work, you need to use Version 3 of NFS, ensure that
the lockd daemon is running on all the machines, and
have the system administrator mount the NFS file
system with the "noac" option (no attribute caching).
Turning off attribute caching may reduce performance,
but it is necessary for correct behavior."
I think that I would assume that fcntl style locking works on all current
implementations, that this is not NFS version dependent, and also that
"noac" is not required in combination with the locking in order to get
correct behavior.
Ouch - I agree with Peter, this advice is way, way out of date.
<http://www-unix.mcs.anl.gov/romio/>
<http://www-unix.mcs.anl.gov/romio/nfs-instr.txt>
Post by Peter Staubach
"The following are some instructions we received from Ian Wells of HP
for setting the noac option on NFS. We have not tried them
ourselves. We are including them here because you may find
them useful. Note that some of the steps may be specific to HP
systems, and ..."
Leave noac alone, the Linux client will do the right thing with
its cache when locking is available.

Tom.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
AK
2007-03-08 16:52:14 UTC
Permalink
Hmmm..
Tom & Peter, thank you for feedbacks ...

I agree, but when I read through the web and other
doucmentation ... I found that if NFS's local caching
is enable, doing a parallel IO on a file is a problem
some, because there is no global synchronization.
Please correct me if I am wrong. I believe GFS works
that way and we don't need to turn off caching there.

With the current setup when the user runs the MPI I/O
program, he runs into following errors.

File locking failed in ADIOI_Set_lock. If the file
system is NFS, you need to use NFS version 3, ensure
that the lockd daemon is running on all the machines,
and mount the directory with the 'noac' option (no
attribute caching).

Is there an alternative to this ?? One more thing
since we have nfs on our rocks cluster, the nfsd,
rpc.statd etc are not started on the compute nodes
...will starting this help the file locking ??

~thank you!
Post by AK
Post by Peter Staubach
Post by AK
"To use ROMIO on NFS, file locking with fcntl
must
Post by Peter Staubach
Post by AK
work correctly on the NFS installation. On some
installations, fcntl locks don't work. To get
them to
Post by Peter Staubach
Post by AK
work, you need to use Version 3 of NFS, ensure
that
Post by Peter Staubach
Post by AK
the lockd daemon is running on all the machines,
and
Post by Peter Staubach
Post by AK
have the system administrator mount the NFS file
system with the "noac" option (no attribute
caching).
Post by Peter Staubach
Post by AK
Turning off attribute caching may reduce
performance,
Post by Peter Staubach
Post by AK
but it is necessary for correct behavior."
I think that I would assume that fcntl style
locking works on all current
Post by Peter Staubach
implementations, that this is not NFS version
dependent, and also that
Post by Peter Staubach
"noac" is not required in combination with the
locking in order to get
Post by Peter Staubach
correct behavior.
Ouch - I agree with Peter, this advice is way, way
out of date.
<http://www-unix.mcs.anl.gov/romio/>
<http://www-unix.mcs.anl.gov/romio/nfs-instr.txt>
Post by Peter Staubach
"The following are some instructions we received
from Ian Wells of HP
Post by Peter Staubach
for setting the noac option on NFS. We have not
tried them
Post by Peter Staubach
ourselves. We are including them here because you
may find
Post by Peter Staubach
them useful. Note that some of the steps may be
specific to HP
Post by Peter Staubach
systems, and ..."
Leave noac alone, the Linux client will do the right
thing with
its cache when locking is available.
Tom.
____________________________________________________________________________________
Now that's room service! Choose from over 150,000 hotels
in 45,000 destinations on Yahoo! Travel to find your fit.
http://farechase.yahoo.com/promo-generic-14795097

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
Talpey, Thomas
2007-03-08 17:06:05 UTC
Permalink
Post by AK
Hmmm..
Tom & Peter, thank you for feedbacks ...
I agree, but when I read through the web and other
doucmentation ... I found that if NFS's local caching
is enable, doing a parallel IO on a file is a problem
some, because there is no global synchronization.
I'm sure Peter will jump in too - the point is you *do* have global
synchronization, from the locks. And in addition, modern clients
know that locking requires them to treat their caches differently,
and they handle these cases.

This did not used to be the case in all NFS clients, but any Linux
client has for quite some time handled this correctly. The only
reason to use noac today is to force clients to do no caching
when they are performing shared writes and are incapable of,
or refuse to do, locking. And even then, it's risky in the extreme.
In addition to being slow.
Post by AK
Please correct me if I am wrong. I believe GFS works
that way and we don't need to turn off caching there.
With the current setup when the user runs the MPI I/O
program, he runs into following errors.
File locking failed in ADIOI_Set_lock. If the file
system is NFS, you need to use NFS version 3, ensure
that the lockd daemon is running on all the machines,
and mount the directory with the 'noac' option (no
attribute caching).
If the app is failing to take a lock, yes fix it. But don't take
the second part of the advice.

Remind us again - what client are you using? Some of them do
not start the client's locking automatically. Perhaps that's the
case with yours.
Post by AK
Is there an alternative to this ?? One more thing
since we have nfs on our rocks cluster, the nfsd,
rpc.statd etc are not started on the compute nodes
...will starting this help the file locking ??
You do not want or need the nfsd on the compute nodes. But you
do need the rpc.statd. If it wasn't started automatically, then either
you are running a version that has it integrated in the kernel, or you
have to perform additional client setup. Let us know that client
version...

Tom.
Post by AK
~thank you!
Post by AK
Post by Peter Staubach
Post by AK
"To use ROMIO on NFS, file locking with fcntl
must
Post by Peter Staubach
Post by AK
work correctly on the NFS installation. On some
installations, fcntl locks don't work. To get
them to
Post by Peter Staubach
Post by AK
work, you need to use Version 3 of NFS, ensure
that
Post by Peter Staubach
Post by AK
the lockd daemon is running on all the machines,
and
Post by Peter Staubach
Post by AK
have the system administrator mount the NFS file
system with the "noac" option (no attribute
caching).
Post by Peter Staubach
Post by AK
Turning off attribute caching may reduce
performance,
Post by Peter Staubach
Post by AK
but it is necessary for correct behavior."
I think that I would assume that fcntl style
locking works on all current
Post by Peter Staubach
implementations, that this is not NFS version
dependent, and also that
Post by Peter Staubach
"noac" is not required in combination with the
locking in order to get
Post by Peter Staubach
correct behavior.
Ouch - I agree with Peter, this advice is way, way
out of date.
<http://www-unix.mcs.anl.gov/romio/>
<http://www-unix.mcs.anl.gov/romio/nfs-instr.txt>
Post by Peter Staubach
"The following are some instructions we received
from Ian Wells of HP
Post by Peter Staubach
for setting the noac option on NFS. We have not
tried them
Post by Peter Staubach
ourselves. We are including them here because you
may find
Post by Peter Staubach
them useful. Note that some of the steps may be
specific to HP
Post by Peter Staubach
systems, and ..."
Leave noac alone, the Linux client will do the right
thing with
its cache when locking is available.
Tom.
______________________________________________________________________
______________
Now that's room service! Choose from over 150,000 hotels
in 45,000 destinations on Yahoo! Travel to find your fit.
http://farechase.yahoo.com/promo-generic-14795097
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Loading...