fsid question

[Jason Keltz]

Hi.

I'm running a RedHat Enterprise 4 system.

I have a question about the fsid option in /etc/exports that I was
hoping someone might be able to help with.

I am working with LVM under Linux. If I export LVM volumes via NFS, the
fsid is based on major/minor number. If I then remove a logical volume
on the NFS server and reboot the server, the major and minor numbers for
the other volumes are adjusted, which of course causes problems for the
existing NFS exports (which either end up stale/pointing to a different
logical volume)! The fsid option would fix this by allowing me to
specify a number independent of major/minor number.

A couple of questions:

1) How do I find out the fsid of an existing export so that I can
"hard-code" it in the /etc/exports file?
2) For filesystems that are not mounted, how do I choose a number that
the system will not try to allocate to a different nfs export that isn't
using the fsid option? Will the system check existing fsids, or it just
allocated the fsid based on a formula?
3) What is the best way for me to generate my own fsid?

Any assistance would be appreciated...

Jason.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Wendy Cheng]

Post by Jason Keltz
Hi.
I'm running a RedHat Enterprise 4 system.
I have a question about the fsid option in /etc/exports that I was
hoping someone might be able to help with.
I am working with LVM under Linux. If I export LVM volumes via NFS, the
fsid is based on major/minor number.

This statement needs correction. An NFS fsid is not based on major/minor
numbers - it is is a 32 bit number that admin (root) can arbitrary
choose and use to uniquely identify an NFS export. More on this in the
following comment.

Post by Jason Keltz
If I then remove a logical volume
on the NFS server and reboot the server, the major and minor numbers for
the other volumes are adjusted, which of course causes problems for the
existing NFS exports (which either end up stale/pointing to a different
logical volume)! The fsid option would fix this by allowing me to
specify a number independent of major/minor number.

This is correct (fsid option would allow admin to identify an NFS
export, independent of major/minor number).

Post by Jason Keltz
1) How do I find out the fsid of an existing export so that I can
"hard-code" it in the /etc/exports file?

The admin (root, or whoever has access rights to /etc/exports file) can
pick one number between 0 and 0xffffffff and add it into /etc/exports.
It comes and goes with each "exportfs" command. This id is used to
construct NFS filehandle that will be sent to NFS client upon requests.
NFS client uses file handle to communicate with NFS server on which file
they want to have access. The fsid, if specified, is part of the file
handle that uniquely identifies an export (one entry in /etc/exports file).

So as rule of thumb, you would not want to change fsid as long as there
are NFS clients still out there trying to access the files on the
server. It is, however, not a permanent ID since there is no filesystem
"on-disk" record of it.

Post by Jason Keltz
2) For filesystems that are not mounted, how do I choose a number that
the system will not try to allocate to a different nfs export that isn't
using the fsid option? Will the system check existing fsids, or it just
allocated the fsid based on a formula?

No, NFS fsid is *not* allocated (nor generated) by system. There is no
co-relation between the physical filesystem and NFS fsid, if the
filesystem is not NFS exported. Note that a filesystem can be mounted
but not NFS exported - in this case, the filesystem will not recognize
the NFS fsid. I suspect that you confuse fsid with either filehandle or
some specific filesystem implementation fsid (few cluster filesystems I
know of have its own fsid - but it differs from the "NFS" fsid we
discuss here).

Post by Jason Keltz
3) What is the best way for me to generate my own fsid?

It is a number picked by admin and must be between 0 and 0xFFFFFFFF.

Hopefully this makes things clear (since our cluster suites use NFS fsid
extensively).

-- Wendy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Jason Keltz]

Hi Wendy,

Thanks for your response..

A few more comments...

Post by Wendy Cheng

Post by Jason Keltz
1) How do I find out the fsid of an existing export so that I can
"hard-code" it in the /etc/exports file?

The admin (root, or whoever has access rights to /etc/exports file) can
pick one number between 0 and 0xffffffff and add it into /etc/exports.
It comes and goes with each "exportfs" command. This id is used to
construct NFS filehandle that will be sent to NFS client upon requests.
NFS client uses file handle to communicate with NFS server on which file
they want to have access. The fsid, if specified, is part of the file
handle that uniquely identifies an export (one entry in /etc/exports file).
So as rule of thumb, you would not want to change fsid as long as there
are NFS clients still out there trying to access the files on the
server. It is, however, not a permanent ID since there is no filesystem
"on-disk" record of it.

The problem is that right now, I have several hundred clients who have
mounted the exported filesystems via NFS where I did not specify an
explicit fsid in the NFS sever /etc/exports file. In order to start
using the fsid option, I need to know the existing fsids so that I can
hard-code those into /etc/exports. This way, if the server is rebooted,
the clients will not be affected. I could then choose fsids for NEW nfs
exported filesystems and hard-code those fsids into /etc/exports. Is
there a Linux command, or an entry in /proc on the file server that will
let me know the existing fsids?

Post by Wendy Cheng

Post by Jason Keltz
3) What is the best way for me to generate my own fsid?

It is a number picked by admin and must be between 0 and 0xFFFFFFFF.

Is the fsid specified in hex?

Since the system will allocate its own fsid for NFS exported filesystems
that do not have a user-specified fsid, how do I choose fsids for my
user-specified fsid exports that will not conflict with those that the
system will use?

Thanks,

Jason.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[James Pearson]

Post by Jason Keltz
Hi Wendy,
Thanks for your response..
A few more comments...

Post by Wendy Cheng

Post by Jason Keltz
1) How do I find out the fsid of an existing export so that I can
"hard-code" it in the /etc/exports file?

The admin (root, or whoever has access rights to /etc/exports file) can
pick one number between 0 and 0xffffffff and add it into /etc/exports.
It comes and goes with each "exportfs" command. This id is used to
construct NFS filehandle that will be sent to NFS client upon requests.
NFS client uses file handle to communicate with NFS server on which file
they want to have access. The fsid, if specified, is part of the file
handle that uniquely identifies an export (one entry in /etc/exports file).
So as rule of thumb, you would not want to change fsid as long as there
are NFS clients still out there trying to access the files on the
server. It is, however, not a permanent ID since there is no filesystem
"on-disk" record of it.

The problem is that right now, I have several hundred clients who have
mounted the exported filesystems via NFS where I did not specify an
explicit fsid in the NFS sever /etc/exports file. In order to start
using the fsid option, I need to know the existing fsids so that I can
hard-code those into /etc/exports. This way, if the server is rebooted,
the clients will not be affected. I could then choose fsids for NEW nfs
exported filesystems and hard-code those fsids into /etc/exports. Is
there a Linux command, or an entry in /proc on the file server that will
let me know the existing fsids?

I believe you can add a fsid to /etc/exports and re-export the file
systems - existing mounts will not be affected. New mounts will pick up
the new fsid. The existing mounts will also pick up the new fsid the
next time they remount the file system.

I've certainly done this in the past and had no problems.

James Pearson

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Jason Keltz]

Post by James Pearson

Post by Jason Keltz
The problem is that right now, I have several hundred clients who have
mounted the exported filesystems via NFS where I did not specify an
explicit fsid in the NFS sever /etc/exports file. In order to start
using the fsid option, I need to know the existing fsids so that I can
hard-code those into /etc/exports. This way, if the server is
rebooted, the clients will not be affected. I could then choose fsids
for NEW nfs exported filesystems and hard-code those fsids into
/etc/exports. Is there a Linux command, or an entry in /proc on the
file server that will let me know the existing fsids?

I believe you can add a fsid to /etc/exports and re-export the file
systems - existing mounts will not be affected. New mounts will pick up
the new fsid. The existing mounts will also pick up the new fsid the
next time they remount the file system.
I've certainly done this in the past and had no problems.
James Pearson

Hi James.

Thanks for your message.

I'm not sure that I understand this though. If the system is generating
an fsid for NFS exports in the absence of a user-specified fsid, then
changing the fsid and re-exporting the filesystem *should* essentially
confuse the client? How does one choose an fsid that is not already in
use on the system? I imagine there must be some way to list the
currently used fsids for exported filesystems.

Jas.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[James Pearson]

Post by Jason Keltz
Hi James.
Thanks for your message.
I'm not sure that I understand this though. If the system is generating
an fsid for NFS exports in the absence of a user-specified fsid, then
changing the fsid and re-exporting the filesystem *should* essentially
confuse the client? How does one choose an fsid that is not already in
use on the system? I imagine there must be some way to list the
currently used fsids for exported filesystems.

See the thread starting at:

<http://marc.info/?l=linux-nfs&m=110295083427898&w=2>

James Pearson


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Jason Keltz]

Post by James Pearson

Post by Jason Keltz
Hi James.
Thanks for your message.
I'm not sure that I understand this though. If the system is
generating an fsid for NFS exports in the absence of a user-specified
fsid, then changing the fsid and re-exporting the filesystem *should*
essentially confuse the client? How does one choose an fsid that is
not already in use on the system? I imagine there must be some way to
list the currently used fsids for exported filesystems.

<http://marc.info/?l=linux-nfs&m=110295083427898&w=2>

That makes sense, but what is still not clear to me is why I can't have
the system show me the existing fsid that it is using in the absence of
a user-specified fsid.

Jason.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[J. Bruce Fields]

Post by Jason Keltz
That makes sense, but what is still not clear to me is why I can't have
the system show me the existing fsid that it is using in the absence of
a user-specified fsid.

There *is* no "existing fsid". The fsid that you provide in the "fsid="
export option is totally different from what's used by default to
generate filehandles for that filesystem.

--b.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Wendy Cheng]

.........................................If the system is generating
an fsid for NFS exports in the absence of a user-specified fsid, then
changing the fsid and re-exporting the filesystem *should* essentially
confuse the client? How does one choose an fsid that is not already in
use on the system? I imagine there must be some way to list the
currently used fsids for exported filesystems.

I think I see where the confusion comes from. To understand this, let's
start with a thing called NFS "file handle".

When an NFS client request file access, server packages few info into a
max 64 bytes identifier called "file handle" that is sent back to
client. Depending on the version of Linux OS you have, it may consist of
file inode number, directory inode number, export device major and minor
number, or the admin-specified "fsid", etc. If the /etc/exports doesn't
specify an "fsid", then the major and minor numbers of the exported
device are used. Each file handle has a field called "fsid-type" to tell
whether this file handle uses admin-specified "fsid" or device
major-minor number.

In your case, the current FileHandle sent to NFS clients uses device
major and minor numbers. You can safely pick *any* number you like as
the fsid and re-exports the entries (say by doing an "exportfs -a"
command). Any *new* NFS client requests will get a new FileHandle that
uses the specified "fsid". However, the *existing" FileHandle(s) any NFS
client still keeps and uses will have the "old" way - that is the device
major-minor pair. The server knows how to interpret them (based on the
type specified in the "fsid-type" field of the file handle).

This implies the following:

1. If server for whatever reason gets rebooted and somehow the device
major-minor number gets altered, the client that keeps the "old" file
handle will get a "stale file handle" error. Some of the NFS clients
will re-do lookup to obtain the correct FileHandle(s). Some of them will
simply fail. There is really not much you can do as a system administrator.
2. You want to keep the "fsid" same (and unique) for obvious reasons
after you export them.
3. Any 32 bit integer can be used (say, 1, 2, 3, etc - no need to use
hex number).

-- Wendy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[J. Bruce Fields]

Post by Wendy Cheng
1. If server for whatever reason gets rebooted and somehow the device
major-minor number gets altered, the client that keeps the "old" file
handle will get a "stale file handle" error.

Actually, have you checked this? From a quick look at

linux/fs/nfsd/nfsfh.c:fh_verify()
and

nfs-utils/utils/mountd/cache.c:nfsd_fh()

it looks to me like nfsd should still allow both the old and new
filehandles.

--b.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Trond Myklebust]

Post by J. Bruce Fields

Post by Wendy Cheng
1. If server for whatever reason gets rebooted and somehow the device
major-minor number gets altered, the client that keeps the "old" file
handle will get a "stale file handle" error.

Actually, have you checked this? From a quick look at
linux/fs/nfsd/nfsfh.c:fh_verify()
and
nfs-utils/utils/mountd/cache.c:nfsd_fh()
it looks to me like nfsd should still allow both the old and new
filehandles.

That is correct. A server kernel upgrade should not normally result in a
client error. See also Neil's recently posted patches that replace the
fileid with the disk UUID. Those again attempt to preserve backward
compatibility for filehandles.

Cheers,
Trond


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Roger Heflin]

Post by Wendy Cheng
1. If server for whatever reason gets rebooted and somehow the device
major-minor number gets altered, the client that keeps the "old" file
handle will get a "stale file handle" error. Some of the NFS clients
will re-do lookup to obtain the correct FileHandle(s). Some of them will
simply fail. There is really not much you can do as a system administrator.

I think this can also happen if you are using the major+minor numbers with
say lvm, and change the order of the underlying LVM's, this might even
be a risk with the fsid option if the underlying filesystem that it was
supposed to be exporting fails to get mounted before the export.

Server reboots, a different device is put at the same major and minor
numbers
than was there before,now the server now gets hammered with inode numbers
that don't exist (valid range for filesystem, but don't exist on the "new"
filesystem that is at the same major and minor numbers as the old one was).

I have seen this actually completely use up all of the CPU on a NFS server,
eventually causing it to be completely unusable, and of course the issue
survives
reboots of the server as the requests are coming from the clients, and the
clients don't appear to ever give up, and if you have enough clients you
can make the server unusable fairly quickly. I have seen the machine
actually hang bringing up NFS from this.

Roger

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Chris Osicki]

On Mon, 26 Mar 2007 17:00:36 -0400
Wendy Cheng <***@redhat.com> wrote:

Hi

Post by Wendy Cheng

Post by Jason Keltz
Hi.
I'm running a RedHat Enterprise 4 system.
I have a question about the fsid option in /etc/exports that I was
hoping someone might be able to help with.
I am working with LVM under Linux. If I export LVM volumes via NFS, the
fsid is based on major/minor number.

This statement needs correction. An NFS fsid is not based on major/minor
numbers - it is is a 32 bit number that admin (root) can arbitrary
choose and use to uniquely identify an NFS export. More on this in the
following comment.

And this statement needs correction too ;-)
fsid _is_ based on major/minor number. I learned it the hard way,
seeing two filesystem swapped on NFS clients because of minor
number change after switching a cluster package to another node.

Then I read carefully fsid section in man 5 exports.

Regards,
Chris

[snip]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Wendy Cheng]

Post by Chris Osicki
On Mon, 26 Mar 2007 17:00:36 -0400
Hi

Post by Wendy Cheng

Post by Jason Keltz
Hi.
I'm running a RedHat Enterprise 4 system.
I have a question about the fsid option in /etc/exports that I was
hoping someone might be able to help with.
I am working with LVM under Linux. If I export LVM volumes via NFS, the
fsid is based on major/minor number.

This statement needs correction. An NFS fsid is not based on major/minor
numbers - it is is a 32 bit number that admin (root) can arbitrary
choose and use to uniquely identify an NFS export. More on this in the
following comment.

And this statement needs correction too ;-)
fsid _is_ based on major/minor number. I learned it the hard way,
seeing two filesystem swapped on NFS clients because of minor
number change after switching a cluster package to another node.
Then I read carefully fsid section in man 5 exports.

Well, I still don't see anywhere there is a documented definition that
says "(NFS) fsid _is_ based on major/minor number". The man page
describes it as "the filesystem identification portion of the file
handle and file attributes used on the wire". My opinion is that an NFS
"fsid" is the number you specify in the export file - nothing more.

On the other hand, arguments like these are kind of silly. The important
thing is to advocate and make sure people know how to use "fsid" in a
clustered environment.

-- Wendy


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Klaus Steinberger]

Post by Wendy Cheng
Well, I still don't see anywhere there is a documented definition that
says "(NFS) fsid _is_ based on major/minor number". The man page
describes it as "the filesystem identification portion of the file

I could not tell you out of memory where it is defined, too bad it's not in
the man page: Should be enhanced!

Definitly the default for fsid is based on major/minor number, as long as you
don't have a fsid option in you're exports file.

Post by Wendy Cheng
On the other hand, arguments like these are kind of silly. The important
thing is to advocate and make sure people know how to use "fsid" in a
clustered environment.

Not only in clustered environment. It is necessary to use fsid in an LVM
environment. I learned this also the hard way, I always wondered about stale
handles on clients after server reboots and reexports, until I learned that
lvm minor numbers can change, and until I discovered fsid.

But be careful! There is nothing that prevents you from using same fsid for
different exports.

Sincerly,
Klaus