Willing to debug bug #3542 (23.0.94; File access via UNC path slow again under Windows)

[Mathias Dahl-3]

I understand that the release of 23.1 is not far away and I am a bit
worried that this bug won't be solved before that. Is there anything I
can do to help? I use a lot of UNC file names / paths in my work and
the current slowness is very annoying (both opening files and listing
them in Dired). As this was a problem in the past too it isn't
unreasonable to believe that the bug has reappeared by mistake when
fixing some other thing (could it be related to TRAMP?).

I am willing to go bug hunting as long as it's in elisp-land and would
like to get some hints on where to start digging.

For details see:
http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=3542

Thanks!

/Mathias

[Chong Yidong]

Mathias Dahl <mathias.dahl@...> writes:

> I understand that the release of 23.1 is not far away and I am a bit
> worried that this bug won't be solved before that. Is there anything I
> can do to help? I use a lot of UNC file names / paths in my work and
> the current slowness is very annoying (both opening files and listing
> them in Dired). As this was a problem in the past too it isn't
> unreasonable to believe that the bug has reappeared by mistake when
> fixing some other thing (could it be related to TRAMP?).

If you have the time, try reverting to previous versions (e.g. `cvs up
-Pd -D 2009-01-01') and find out which checkin introduced the problem.
A binary search should narrow it down quite quickly.

[Eli Zaretskii]

> From: Mathias Dahl <mathias.dahl@...>
> Date: Sun, 5 Jul 2009 23:01:53 +0200
>
> I understand that the release of 23.1 is not far away and I am a bit
> worried that this bug won't be solved before that. Is there anything I
> can do to help? I use a lot of UNC file names / paths in my work and
> the current slowness is very annoying (both opening files and listing
> them in Dired). As this was a problem in the past too it isn't
> unreasonable to believe that the bug has reappeared by mistake when
> fixing some other thing (could it be related to TRAMP?).
>
> I am willing to go bug hunting as long as it's in elisp-land and would
> like to get some hints on where to start digging.

Please post some measurements (the original report says "a few
seconds", which is quite vague) and the basic performance data of your
machine.  Also, please tell if this happens with every file, even one
visited in Fundamental mode, or only with some files.  I will then try
to reproduce this.  Best I remember, I didn't have such problems on my
machine, but maybe I misremember.

[Mathias Dahl-3]

> Please post some measurements (the original report says "a few
> seconds", which is quite vague) and the basic performance data of your
> machine.  Also, please tell if this happens with every file, even one
> visited in Fundamental mode, or only with some files.  I will then try
> to reproduce this.  Best I remember, I didn't have such problems on my
> machine, but maybe I misremember.

I did all tests below in emacs -Q.

//gbgfs1/archive75 is a share that contains 260 directories and three
files. The server is located in our office.

C-x d //gbgfs1/archive75 RET

 Emacs 22.1.1: 1 s
 Emacs 23.0.95.1: 95 s

So, that is about 95 times slower...

Opening a subfolder in dired in the directory listing I got in the
above operation. The subfolder opened contains only one folder.

 Emacs 22.1.1: immediate (under 1 s)
 Emacs 23.0.95.1: 3 s

Opening a file, 2470 bytes, using RET in dired and in a directory
further down in the structure above:

 Emacs 22.1.1: immediate (under 1 s)
 Emacs 23.0.95.1: 3 s

I opened a file that was about 20 times larger than the above, from
the same folder, and it took about the same time, so it seems that the
actual reading of the file contents is not affected.

Opening a folder with about 270 files even further down in the same structure:

 Emacs 22.1.1: 2-3 s
 Emacs 23.0.95.1: 86 s

This seems to happen for any type of file I open.

My PC has a Pentium 4 CPU, running at 3 GHz, with 1.5 GBG of RAM,
according to My Computer -> RMB -> Properties -> General and I am
using Windows XP Professional, Version 2002, Service Pack 2.

Any tests in particular you want me to run?

/Mathias

[Mathias Dahl-3]

> If you have the time, try reverting to previous versions (e.g. `cvs up
> -Pd -D 2009-01-01') and find out which checkin introduced the problem.
> A binary search should narrow it down quite quickly.

I don't compile it myself, instead I rely on the precompiled pretests
over at alpha.gnu.org. But maybe I will find the time to download them
all and test, to narrow it down at least somewhat.

Considering the results of my Dired experiments, does it suggest the
problem is in some low level file listing function someplace?

[Chong Yidong]

Mathias Dahl <mathias.dahl@...> writes:

> I don't compile it myself, instead I rely on the precompiled pretests
> over at alpha.gnu.org. But maybe I will find the time to download them
> all and test, to narrow it down at least somewhat.

Please use CVS.  That lets you track down the problem with better
granularity, and it's more convenient.  See
http://savannah.gnu.org/cvs/?group=emacs for a description of how to
check out the cvs repository.  After that, as I said, the command to
revert to earlier revisions is (e.g.) `cvs up -Pd -D 2009-01-01'.

> Considering the results of my Dired experiments, does it suggest the
> problem is in some low level file listing function someplace?

No idea, sorry.

[Eli Zaretskii]

> From: Mathias Dahl <mathias.dahl@...>
> Date: Mon, 6 Jul 2009 09:38:07 +0200
> Cc: emacs-devel@...
>
> //gbgfs1/archive75 is a share that contains 260 directories and three
> files. The server is located in our office.
>
> C-x d //gbgfs1/archive75 RET
>
>  Emacs 22.1.1: 1 s
>  Emacs 23.0.95.1: 95 s
>
> So, that is about 95 times slower...

Ouch!  I'm quite sure I would have noticed such a terrible slowdown.

Let me time similar commands tomorrow in my office.

> Any tests in particular you want me to run?

Not yet.  But perhaps install StraceNT
(http://stracent.en.softonic.com/) and see what are the differences
between Emacs 22 and Emacs 23.  Maybe you will even be able to see the
system call(s) that take most of this extra time.

TIA

[Mathias Dahl-3]

> Please use CVS.  That lets you track down the problem with better
> granularity, and it's more convenient.  See
> http://savannah.gnu.org/cvs/?group=emacs for a description of how to
> check out the cvs repository.  After that, as I said, the command to
> revert to earlier revisions is (e.g.) `cvs up -Pd -D 2009-01-01'.

Just so that I understand you correctly, are you suggesting I download
and compile CVS emacs and do the "cvs up" trick and repeat this
process until I get a compiled version that works? If so, it might
take a while since I have never compiled Emacs myself under w32. Maybe
I will try to follow the guide Lennart has provided with me, we'll
see. Debugging elisp had been much more fun :)

Is there some low level primitive built in C that is always involved
in file listings etc that could be the culprit?

[Miles Bader-2]

Mathias Dahl <mathias.dahl@...> writes:
> Just so that I understand you correctly, are you suggesting I download
> and compile CVS emacs and do the "cvs up" trick and repeat this
> process until I get a compiled version that works? If so, it might
> take a while since I have never compiled Emacs myself under w32. Maybe
> I will try to follow the guide Lennart has provided with me, we'll
> see. Debugging elisp had been much more fun :)

It should be much easier (and faster) if you use the git mirror and "git bisect".

[e.g., "git clone git://git.sv.gnu.org/emacs.git"]

-Miles

--
Monday, n. In Christian countries, the day after the baseball game.

[Eli Zaretskii]

> Date: Mon, 06 Jul 2009 23:01:19 +0300
> From: Eli Zaretskii <eliz@...>
> Cc: emacs-devel@...
>
> > From: Mathias Dahl <mathias.dahl@...>
> > Date: Mon, 6 Jul 2009 09:38:07 +0200
> > Cc: emacs-devel@...
> >
> > //gbgfs1/archive75 is a share that contains 260 directories and three
> > files. The server is located in our office.
> >
> > C-x d //gbgfs1/archive75 RET
> >
> >  Emacs 22.1.1: 1 s
> >  Emacs 23.0.95.1: 95 s
> >
> > So, that is about 95 times slower...
>
> Ouch!  I'm quite sure I would have noticed such a terrible slowdown.
>
> Let me time similar commands tomorrow in my office.

Sorry for a long delay.  I finally found a few minutes to look into
this.

First, I don't see a ~100-fold slowdown, I see only a 10-fold slowdown
(vs Emacs 22.3).  This is with a directory that has ~130
subdirectories.

I have a hypothesis for the reason of this slowdown, but I need a few
more facts from you to make sure the slowdown I see is caused by the
same factors as what you see:

 . Do you see significant difference in speed with cold and hot cache?
   That is, if you run Dired on the above directory, then kill the
   Dired buffer and immediately run Dired again on the same directory,
   what times do you see then in Emacs 23?

 . Does it help to set w32-get-true-file-attributes to nil?  If the
   previous test shows a significant speedup with hot cache, please
   try this with a cold cache, which may mean rebooting the machine
   where you are trying this, or maybe waiting long enough for the
   cache to "cool down" and in addition restarting Emacs (to get rid
   of Emacs's internal caching).

 . If w32-get-true-file-attributes does have a significant effect,
   please tell how many different user names and group names you see
   in the listing presented by "C-x d".

Thanks in advance.

> > Any tests in particular you want me to run?
>
> Not yet.  But perhaps install StraceNT
> (http://stracent.en.softonic.com/) and see what are the differences
> between Emacs 22 and Emacs 23.  Maybe you will even be able to see the
> system call(s) that take most of this extra time.

Did you have a chance to do this?  If so, can you post the results?

[Mathias Dahl-3]

> First, I don't see a ~100-fold slowdown, I see only a 10-fold slowdown
> (vs Emacs 22.3).  This is with a directory that has ~130
> subdirectories.

The more files there are, the bigger the slowdown seems to be, in my tests.

>  . Do you see significant difference in speed with cold and hot cache?
>   That is, if you run Dired on the above directory, then kill the
>   Dired buffer and immediately run Dired again on the same directory,
>   what times do you see then in Emacs 23?

Hehe, it got even worse the second time :)

>  . Does it help to set w32-get-true-file-attributes to nil?

Woooo! Um, yes... It helped all right. From 85 (first try) and 115
(second try as per reuest above), down to 1,4 s. Shall we call that as
success? :)

>   previous test shows a significant speedup with hot cache, please
>   try this with a cold cache, which may mean rebooting the machine
>   where you are trying this, or maybe waiting long enough for the
>   cache to "cool down" and in addition restarting Emacs (to get rid
>   of Emacs's internal caching).

I assume I don't need to test this now. If I do, ask me again.

>  . If w32-get-true-file-attributes does have a significant effect,
>   please tell how many different user names and group names you see
>   in the listing presented by "C-x d".

It looks like this:

  //gbgfs1/archive75:

  total used in directory 8728 available 8198544

  drwxrwxrwx  1 Administrators Domain Users       0 1970-01-01  .

  dr-xr-xr-x  1 mdahse         Domain Users       0 1970-01-01  ..

  drwxrwxrwx  1 demoruntime    Domain Users       0 2008-12-05  _ConfigBuild

  drwxrwxrwx  1 demoruntime    Domain Users       0 2008-06-09  acadc

...

and then all other lines after that has the same user and group names.
The one that differs above (mdahse) is my own username.

>> Not yet.  But perhaps install StraceNT
>> (http://stracent.en.softonic.com/) and see what are the differences
>> between Emacs 22 and Emacs 23.  Maybe you will even be able to see the
>> system call(s) that take most of this extra time.
>
> Did you have a chance to do this?  If so, can you post the results?

Not yet, no. Is it still interesting?

/Mathias

[Jason Rumney-4]

Mathias Dahl wrote:

>>  . Does it help to set w32-get-true-file-attributes to nil?
>>    
>
> Woooo! Um, yes... It helped all right. From 85 (first try) and 115
> (second try as per reuest above), down to 1,4 s. Shall we call that as
> success? :)
>  

Did you have it set to t in your .emacs, or does the default of 'local
not work for you?

[Mathias Dahl-3]

>>>  . Does it help to set w32-get-true-file-attributes to nil?
>>>
>>
>> Woooo! Um, yes... It helped all right. From 85 (first try) and 115
>> (second try as per reuest above), down to 1,4 s. Shall we call that as
>> success? :)
>>
>
> Did you have it set to t in your .emacs, or does the default of 'local not work for you?

I did not have it set so it had the value 'local before I changed it
to nil now. I googled for the variable now and saw this:

"If the variable w32-get-true-file-attributes is non-nil (the
default), Emacs tries to determine the accurate link counts for files.
This option is only useful on NTFS volumes, and it *considerably slows
down Dired and other features*, so use it only on fast machines."

Hmm... There is a suggestion to use it only on fast machines at the
same time it is turned on by default. Shouldn't it be the other way
around? Btw, I consider my machine to be quite fast even though it is
a couple of years old.

I suggest we change the default value of this variable to nil (or make
the code faster).

/Mathias

[Eli Zaretskii]

> From: Mathias Dahl <mathias.dahl@...>
> Date: Wed, 8 Jul 2009 22:47:37 +0200
> Cc: emacs-devel@...
>
> > First, I don't see a ~100-fold slowdown, I see only a 10-fold slowdown
> > (vs Emacs 22.3).  This is with a directory that has ~130
> > subdirectories.

To clarify, I see 1 sec with Emacs 22.3, and 10 to 11 seconds with
Emacs 23.0.95.

> The more files there are, the bigger the slowdown seems to be, in my tests.

Can you see if it's approximately linear in the number of files?

> >  . Do you see significant difference in speed with cold and hot cache?
> >   That is, if you run Dired on the above directory, then kill the
> >   Dired buffer and immediately run Dired again on the same directory,
> >   what times do you see then in Emacs 23?
>
> Hehe, it got even worse the second time :)

That's somewhat strange.  For me, the second time is very fast: 3-4
seconds instead of 11.

> >  . Does it help to set w32-get-true-file-attributes to nil?
>
> Woooo! Um, yes... It helped all right. From 85 (first try) and 115
> (second try as per reuest above), down to 1,4 s. Shall we call that as
> success? :)

Yes, probably.  I will send a patch today or tomorrow for you to try.

Thanks.

[Drew Adams]

> "If the variable w32-get-true-file-attributes is non-nil (the
> default), Emacs tries to determine the accurate link counts for files.
> This option is only useful on NTFS volumes, and it *considerably slows
> down Dired and other features*, so use it only on fast machines."
>
> Hmm... There is a suggestion to use it only on fast machines at the
> same time it is turned on by default. Shouldn't it be the other way
> around? Btw, I consider my machine to be quite fast even though it is
> a couple of years old.
>
> I suggest we change the default value of this variable to nil (or make
> the code faster).

Dumb question: Is there a way for Emacs to know whether the format is NTFS or
FAT(32)? If so, then Emacs could use nil for FAT volumes.

(If not, it seems like nil would be the best default value.)

[Jason Rumney-4]

Drew Adams wrote:

>> Hmm... There is a suggestion to use it only on fast machines at the
>> same time it is turned on by default. Shouldn't it be the other way
>> around? Btw, I consider my machine to be quite fast even though it is
>> a couple of years old.
>>
>> I suggest we change the default value of this variable to nil (or make
>> the code faster).
>>    
>
> Dumb question: Is there a way for Emacs to know whether the format is NTFS or
> FAT(32)? If so, then Emacs could use nil for FAT volumes.
>  

That suggestion dates back to the days when "fast machines" were running
at 200MHz or so. The problem here is not the speed of the machine, but
the network.  The default value of w32-get-true-file-attributes is
'local, which means nil when going across the network, but apparently
that is not being recognized in all places - which I think is what Eli
is investigating.

[Drew Adams]

> > Dumb question: Is there a way for Emacs to know whether the
> > format is NTFS or FAT(32)? If so, then Emacs could use nil
> > for FAT volumes.
>  
> That suggestion dates back to the days when "fast machines"
> were running at 200MHz or so. The problem here is not the
> speed of the machine, but the network.  The default value of
> w32-get-true-file-attributes is 'local, which means nil when
> going across the network, but apparently that is not being
> recognized in all places - which I think is what Eli
> is investigating.

I see (I think). I thought that the doc was saying that it should be non-nil
only for NTFS, which should be independent of whether the file system is local
or over a network.

[Eli Zaretskii]

> Date: Fri, 10 Jul 2009 00:25:25 +0800
> From: Jason Rumney <jasonr@...>
> CC: 'Mathias Dahl' <mathias.dahl@...>,
>  'Eli Zaretskii' <eliz@...>,
>  emacs-devel@...
>
> Drew Adams wrote:
> >> Hmm... There is a suggestion to use it only on fast machines at the
> >> same time it is turned on by default. Shouldn't it be the other way
> >> around? Btw, I consider my machine to be quite fast even though it is
> >> a couple of years old.
> >>
> >> I suggest we change the default value of this variable to nil (or make
> >> the code faster).
> >>    
> >
> > Dumb question: Is there a way for Emacs to know whether the format is NTFS or
> > FAT(32)? If so, then Emacs could use nil for FAT volumes.
> >  
> That suggestion dates back to the days when "fast machines" were running
> at 200MHz or so. The problem here is not the speed of the machine, but
> the network.  The default value of w32-get-true-file-attributes is
> 'local, which means nil when going across the network, but apparently
> that is not being recognized in all places - which I think is what Eli
> is investigating.

Right.  We didn't treat UNC file names as remote, it's as simple as
that.

I installed the patch below on the trunk.

Stefan and Yidong, is it okay to install on the release branch as
well?

2009-07-09  Eli Zaretskii  <eliz@...>

        * w32.c (stat): Treat UNC file names as residing on remote
        drives.  (Bug#3542)


--- src/w32.c.orig 2009-06-21 10:38:18.000000000 +0300
+++ src/w32.c 2009-07-09 16:31:51.250000000 +0300
@@ -3154,11 +3154,13 @@
  }
     }
 
-  /* GetDriveType needs the root directory of NAME's drive.  */
-  if (!(strlen (name) >= 2 && IS_DEVICE_SEP (name[1])))
-    devtype = GetDriveType (NULL); /* use root of current diectory */
+  if (IS_DIRECTORY_SEP (name[0]) && IS_DIRECTORY_SEP (name[1]))
+    devtype = DRIVE_REMOTE;   /* assume UNC name is remote */
+  else if (!(strlen (name) >= 2 && IS_DEVICE_SEP (name[1])))
+    devtype = GetDriveType (NULL); /* use root of current drive */
   else
     {
+      /* GetDriveType needs the root directory of NAME's drive.  */
       strncpy (drive_root, name, 3);
       drive_root[3] = '\0';
       devtype = GetDriveType (drive_root);

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@...>
> Cc: "'Mathias Dahl'" <mathias.dahl@...>,
>         "'Eli Zaretskii'" <eliz@...>, <emacs-devel@...>
> Date: Thu, 9 Jul 2009 10:03:42 -0700
>
> > > Dumb question: Is there a way for Emacs to know whether the
> > > format is NTFS or FAT(32)? If so, then Emacs could use nil
> > > for FAT volumes.
> >  
> > That suggestion dates back to the days when "fast machines"
> > were running at 200MHz or so. The problem here is not the
> > speed of the machine, but the network.  The default value of
> > w32-get-true-file-attributes is 'local, which means nil when
> > going across the network, but apparently that is not being
> > recognized in all places - which I think is what Eli
> > is investigating.
>
> I see (I think). I thought that the doc was saying that it should be non-nil
> only for NTFS, which should be independent of whether the file system is local
> or over a network.

Network drives can be NTFS volumes as well, and usually are if the
server is a Windows machine.  But accessing all of the features we now
support in file-attributes on Windows can be slow over the network,
especially if the network is busy.

I made the doc string less categorical, thanks for pointing that out.

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@...>
> Cc: "'Eli Zaretskii'" <eliz@...>, <emacs-devel@...>
> Date: Thu, 9 Jul 2009 09:11:10 -0700
>
> Dumb question: Is there a way for Emacs to know whether the format is NTFS or
> FAT(32)?

Yes (and it does).

> If so, then Emacs could use nil for FAT volumes.

Doing that by default is too radical, IMO.  Having this option non-nil
on FAT/FAT32/XFAT has its merits, although less so than on NTFS.  And
the speedup for local drives would be minimal.

> (If not, it seems like nil would be the best default value.)

I prefer to carry out a plan I have for speeding up the emulated
`stat' so that it works faster without sacrificing features.  Those
features are important for important use-cases, such as support for
files and directories that are private to the user (inaccessible by
other Windows users).

Note that the slow-down due to additional functionality is only
acutely visible when we call file-attributes on long lists of files,
like when it is called from directory-files-and-attributes.  A single
call to file-attributes for a single file is very fast even for
networked drives.

I think nil should be the last resort, for those pathological cases
where disk access is dog slow no matter what you do.  Setting the
option to nil by default would punish too many use-cases for now good
reason.