[bug #26641] Wrong directory hard link count message on automounted directory.

[Mario Castelán Castro-2]

URL:
  <http://savannah.gnu.org/bugs/?26641>

                 Summary: Wrong directory hard link count message on
automounted directory.
                 Project: findutils
            Submitted by: None
            Submitted on: Fri 22 May 2009 05:01:37 PM UTC
                Category: find
                Severity: 3 - Normal
              Item Group: Wrong result
                  Status: None
                 Privacy: Public
             Assigned to: None
         Originator Name: Phil Dumont
        Originator Email: phil@...
             Open/Closed: Open
         Discussion Lock: Any
                 Release: 4.2.27
           Fixed Release: None

    _______________________________________________________

Details:

On a linux distro (CentOS 5.2, 5.3 and 5.4) an invocation of this command:

$ find /net/fs/export/vtrak3b/keeper/

...resulted in this message to stderr:

find: WARNING: Hard link count is wrong for /net/fs/export/vtrak3b/keeper/:
this may be a bug in your filesystem driver.  Automatically turning on find's
-noleaf option.  Earlier results may have failed to include directories that
should have been searched.

The problem seems to be a result of interaction with automount.

As soon as you access /net/host, if host is exporting any paths other than /,
the automount daemon will create a mount point /net/host/path/to/export for
every path exported by host, but doesn't actually mount anything until you
read the directory at the root of the exported path.

At first, find sees a hard link count of 2 on /net/fs/export/vtrak3b/keeper,
because the mount point was there, but the mount hadn't occurred yet, and it
was the mount point being stat(2)ed.  Since the directory was newly created
and empty (no subdirs), the only links were keeper's '.' and vtrak3b's
'keeper'.  Once find tried to read keeper, that triggered the automount, and
thenceforth the mount point was hidden and it was the mounted directory (which
did have sub-directories and therefore a higher hard link count) being seen.

There doesn't seem to be a Real Problem, but the message seems a bit ominous
until you figure out what's going on (which took me a while).  Any chance it
could be avoided?  Maybe postpone the stat of the directory until after
opening the directory triggers the automount?




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?26641>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/

[Mario Castelán Castro-2]

Follow-up Comment #1, bug #26641 (project findutils):

I reckon it could be argued that the bug is with automount.  Perhaps a
stat(2) of an automounted mount point should trigger the automount?  Still, if
it's not too difficult for find to take evasive action, it would be nice.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?26641>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/

[Mario Castelán Castro-2]

Follow-up Comment #2, bug #26641 (project findutils):

I posted to autofs@..., asking whether it would make sense for
automount to do a mount triggered by a stat() of the mount point.  They
replied that this is avoided, by design, to prevent a "mount storm" whenever a
stat()ing 'ls' (common, with the widespread use of colorized ls) of a
directory with many mount points is done.  Apparently, large directories of
mount points are not uncommon (e.g., an indirect map that uses the "browse"
option).

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?26641>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/

[Mario Castelán Castro-2]

Follow-up Comment #3, bug #26641 (project findutils):

Could the folks at autofs@... instead be persuaded that for any
directory that contains auto mount points that the containing directory return
a st_link count of 1 (meaning unknown number of subdirectories)?  That alone
would be enough to force find to realize that for that particular directory,
the traditional semantics of 2 + subdirectories won't hold, and the leaf
optimization will be skipped for just that directory without adversely
affecting other directories where the optimization is still valid.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?26641>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/

[Mario Castelán Castro-2]

Follow-up Comment #4, bug #26641 (project findutils):

Could you ask this of the autofs folks yourself?  I am, of course, willing to
do my part to help resolve this issue, but I would think that me being a
middle-man between findutils developers and autofs developers would be more of
a hindrance than a help.

My personal opinion (for what it's worth) is that autofs setting st_link to 1
is not a good general solution.  It might serve as a useful clue to find, but
could confuse other code elsewhere (not to mention humans).

Would this be reasonable?:

Just do 2 stat(2)s, one before, one after opening the directory.  If the
st_dev field of the 2 stats yield differing st_dev values, I think that's a
fairly foolproof(?) indication that an automount has occurred, in which case
you can throw away (ignore) the first stat and use the second.  The
performance implication may not be negligible, but could probably be mitigated
if the 2nd stat is only done *after* you have reason to suspect the st_link
value.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?26641>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/

[Mario Castelán Castro-2]

Follow-up Comment #5, bug #26641 (project findutils):

I don't use autofs, nor do I know where to report such a bug.  As a user,
your bug report will probably go further (but do feel free to point them to
this page).

Any code that is not prepared to handle st_nlink count of 1 is already
broken.  There are already systems, like cygwin, that ALWAYS return st_nlink
of 1 for all directories, so any code ported to cygwin already has dealt with
that issue.

Doing 2 stat()s is a performance hog and breaks some atomicity guarantees -
besides, what hueristic would you use to decide that an st_link count is
suspect?  If there is a reliable hueristic, then we should already be using it
to exclude directories where st_nlink is greater than 1 but wrong.  Really, it
boils down to whether optimizations are disabled (the current decision for
that is when st_nlink is 1 or when you configured with leaf optimization
disabled); the behavior without optimization is always correct, although it is
provably slower on systems where st_nlink is reliable.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?26641>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/

[Mario Castelán Castro-2]

Follow-up Comment #6, bug #26641 (project findutils):

find seems to be already doing *something* to determine that the st_link
count is wrong -- that's what leads to the printing of the warning and the
automatic turning on of the -noleaf option, yes?.  All I'm saying is, wherever
it is that find does:

  printf("WARNING: Hard link count is wrong...");
  turn_on_noleaf();

change it to

  fstat(dirfd, &new_stat_buf);
  if (stat_buf.st_dev != new_stat_buf.st_dev)
    /* probably an automount happened; just update stat info */
    stat_buf = new_stat_buf;
  else {
    printf("WARNING: Hard link count is wrong...");
    turn_on_noleaf();
  }

The performance implications should be minimal, because the extra stat is
only ever done whenever the -noleaf evasive action would have been taken,
which is seldom.

Again, I haven't read the code (still), so I apologize if this is naive.

 


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?26641>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/

[Mario Castelán Castro-2]

Follow-up Comment #7, bug #26641 (project findutils):

Regarding my previous comment:

Oops.  Can't do fstat.  That will just give the stat of the (now hidden)
mount point.  Have to stat the path.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?26641>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/

[Mario Castelán Castro-2]

Follow-up Comment #8, bug #26641 (project findutils):

On further reflection, fstat *would* work.  The variable I was calling
"dirfd" will have come from an open(2) (or dirfd(opendir()) or something like
that), which will have triggered the automount, and the file descriptor will
be for the mounted file system's root, not the mount point.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?26641>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/