large pages (amd64)

[Wojciech Puchar-5]

i enabled
vm.pmap.pg_ps_enabled: 1


could you please explain what exactly this values means?
because i don't understand why promotions-demotions!=mappings

vm.pmap.pde.promotions: 2703
vm.pmap.pde.p_failures: 6290
vm.pmap.pde.mappings: 610
vm.pmap.pde.demotions: 289




other question - tried enabling it on my i386 laptop (256 megs ram),
always mappings==0, while promitions>demotions>0.

certainly there are apps that could be put on big pages, gimp editing 40MB
bitmap for example
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Alan Cox-9]

On Sun, Jun 28, 2009 at 12:36 PM, Wojciech Puchar <
wojtek@...> wrote:

> i enabled
> vm.pmap.pg_ps_enabled: 1
>
>
> could you please explain what exactly this values means?
> because i don't understand why promotions-demotions!=mappings
>

"mappings" is not what you seem to think it is.  vm.pmap.pde.mappings is the
number of 2/4MB page mappings that are created directly and not through the
incremental promotion process.  For example, it counts the 2/4MB page
mappings that are created when the text segment of a large executable, e.g.,
gcc, is pre-faulted at startup or when a graphics card's frame buffer is
mmap()ed.

Moreover, not every promoted mapping is demoted.  A promoted mapping may be
destroyed without demotion, for example, when a process exits.  This is, in
fact, the ideal case because it is cheaper to destroy a single 2/4MB page
mapping instead of 512 or 1024 4KB page mappings.

>
> vm.pmap.pde.promotions: 2703
> vm.pmap.pde.p_failures: 6290
> vm.pmap.pde.mappings: 610
> vm.pmap.pde.demotions: 289
>
>
>
>
> other question - tried enabling it on my i386 laptop (256 megs ram), always
> mappings==0, while promitions>demotions>0.
>

The default starting address for executables on i386 is not aligned to a
2/4MB page boundary.  Hence, "mappings" are much less likely to occur.


> certainly there are apps that could be put on big pages, gimp editing 40MB
> bitmap for example


Regards,
Alan

<http://lists.freebsd.org/mailman/listinfo/freebsd-hackers><freebsd-hackers-unsubscribe@...>
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Wojciech Puchar-5]

>
>       other question - tried enabling it on my i386 laptop (256 megs ram), always
>       mappings==0, while promitions>demotions>0.
>
>
> The default starting address for executables on i386 is not aligned to a 2/4MB page boundary.
> Hence, "mappings" are much less likely to occur.
>
>
>       certainly there are apps that could be put on big pages, gimp editing 40MB bitmap for
>       example
>
>
> Regards,
> Alan
>
>
>

how can i check how much (or maybe - what processes) 2MB pages are
actually allocated?
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Alan Cox-5]

Wojciech Puchar wrote:

>>
>>       other question - tried enabling it on my i386 laptop (256 megs
>> ram), always
>>       mappings==0, while promitions>demotions>0.
>>
>>
>> The default starting address for executables on i386 is not aligned
>> to a 2/4MB page boundary. Hence, "mappings" are much less likely to
>> occur.
>>
>>
>>       certainly there are apps that could be put on big pages, gimp
>> editing 40MB bitmap for
>>       example
>>
>>
>> Regards,
>> Alan
>>
>>
>>
> how can i check how much (or maybe - what processes) 2MB pages are
> actually allocated?

I'm afraid that you can't with great precision.  For a given program
execution, on an otherwise idle machine, you can only estimate the
number by looking at the change in the quantity "promotions + mappings -
demotions" before, during, and after the program execution.

A program can call mincore(2) in order to determine if a virtual address
is part of a 2 or 4MB virtual page.

Alan

_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Nathanael Hoyle-3]

Wojciech Puchar wrote:

> i enabled
> vm.pmap.pg_ps_enabled: 1
>
>
> could you please explain what exactly this values means?
> because i don't understand why promotions-demotions!=mappings
>
> vm.pmap.pde.promotions: 2703
> vm.pmap.pde.p_failures: 6290
> vm.pmap.pde.mappings: 610
> vm.pmap.pde.demotions: 289
>
>
>
>
> other question - tried enabling it on my i386 laptop (256 megs ram),
> always mappings==0, while promitions>demotions>0.
>
> certainly there are apps that could be put on big pages, gimp editing
> 40MB bitmap for example

Just to be clear, since you say i386 (I presume you mean architecture),
I believe the Physical Address Extensions which allowed 2MB Page Size
bit to be set was introduced with Pentium Pro. Processors prior to this
were limited to standard 4KB pages.

-Nathanael
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Joerg Sonnenberger]

On Sun, Jun 28, 2009 at 08:14:24PM -0400, Nathanael Hoyle wrote:
> Just to be clear, since you say i386 (I presume you mean architecture),  
> I believe the Physical Address Extensions which allowed 2MB Page Size  
> bit to be set was introduced with Pentium Pro. Processors prior to this  
> were limited to standard 4KB pages.

PSE != PAE != PSE36. Pentium Pro sounds about right though.

Joerg
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Alan Cox-9]

On Sun, Jun 28, 2009 at 7:14 PM, Nathanael Hoyle <nhoyle@...>wrote:

> Wojciech Puchar wrote:
>
>> i enabled
>> vm.pmap.pg_ps_enabled: 1
>>
>>
>> could you please explain what exactly this values means?
>> because i don't understand why promotions-demotions!=mappings
>>
>> vm.pmap.pde.promotions: 2703
>> vm.pmap.pde.p_failures: 6290
>> vm.pmap.pde.mappings: 610
>> vm.pmap.pde.demotions: 289
>>
>>
>>
>>
>> other question - tried enabling it on my i386 laptop (256 megs ram),
>> always mappings==0, while promitions>demotions>0.
>>
>> certainly there are apps that could be put on big pages, gimp editing 40MB
>> bitmap for example
>>
>
> Just to be clear, since you say i386 (I presume you mean architecture), I
> believe the Physical Address Extensions which allowed 2MB Page Size bit to
> be set was introduced with Pentium Pro. Processors prior to this were
> limited to standard 4KB pages.
>

No.  Many of those processors supported 4MB pages.

Regards,
Alan
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Nathanael Hoyle-3]

Alan Cox wrote:

> On Sun, Jun 28, 2009 at 7:14 PM, Nathanael Hoyle <nhoyle@...>wrote:
>
>  
>> Wojciech Puchar wrote:
>>
>>    
>>> i enabled
>>> vm.pmap.pg_ps_enabled: 1
>>>
>>>
>>> could you please explain what exactly this values means?
>>> because i don't understand why promotions-demotions!=mappings
>>>
>>> vm.pmap.pde.promotions: 2703
>>> vm.pmap.pde.p_failures: 6290
>>> vm.pmap.pde.mappings: 610
>>> vm.pmap.pde.demotions: 289
>>>
>>>
>>>
>>>
>>> other question - tried enabling it on my i386 laptop (256 megs ram),
>>> always mappings==0, while promitions>demotions>0.
>>>
>>> certainly there are apps that could be put on big pages, gimp editing 40MB
>>> bitmap for example
>>>
>>>      
>> Just to be clear, since you say i386 (I presume you mean architecture), I
>> believe the Physical Address Extensions which allowed 2MB Page Size bit to
>> be set was introduced with Pentium Pro. Processors prior to this were
>> limited to standard 4KB pages.
>>
>>    
>
> No.  Many of those processors supported 4MB pages.
>
> Regards,
> Alan
> _______________________________________________
>  

Having been corrected by both you and Joerg (thank you!), I went back to
re-verify my understanding. It appears that while I was slightly mixing
PAE in with PSE, PSE support for 4MB pages was introduced 'silently'
with the Pentium, and documented first with the Pentium Pro.  I haven't
found anything that points to earlier inclusion. Certainly the 80386
processor specifically, I am fairly confident would be limited to the
4KB pages.

Agreed? Or are you aware of earlier usage than the Pentium for 4MB pages?

Thanks,
-Nathanael
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Nathanael Hoyle-3]

Joerg Sonnenberger wrote:

> On Sun, Jun 28, 2009 at 08:14:24PM -0400, Nathanael Hoyle wrote:
>  
>> Just to be clear, since you say i386 (I presume you mean architecture),  
>> I believe the Physical Address Extensions which allowed 2MB Page Size  
>> bit to be set was introduced with Pentium Pro. Processors prior to this  
>> were limited to standard 4KB pages.
>>    
>
> PSE != PAE != PSE36. Pentium Pro sounds about right though.
>
> Joerg
>  

Thanks for the clarification/correction. PSE does not depend on PAE,
though both were introduced at roughly the same time (appears PSE was
introduced silently in Pentium, officially in Pentium Pro). I definitely
wasn't thinking of PSE36, which was the Pentium III.

-Nathanael
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Alan Cox-5]

Nathanael Hoyle wrote:
[snip]

> Having been corrected by both you and Joerg (thank you!), I went back
> to re-verify my understanding. It appears that while I was slightly
> mixing PAE in with PSE, PSE support for 4MB pages was introduced
> 'silently' with the Pentium, and documented first with the Pentium
> Pro.  I haven't found anything that points to earlier inclusion.
> Certainly the 80386 processor specifically, I am fairly confident
> would be limited to the 4KB pages.
>
> Agreed? Or are you aware of earlier usage than the Pentium for 4MB pages?
>

Yes, I agree.  I'm not aware of 4MB page support before the Pentium.

Regards,
Alan

_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Mel Flynn-6]

On Sunday 28 June 2009 15:41:49 Alan Cox wrote:
> Wojciech Puchar wrote:

> > how can i check how much (or maybe - what processes) 2MB pages are
> > actually allocated?
>
> I'm afraid that you can't with great precision.  For a given program
> execution, on an otherwise idle machine, you can only estimate the
> number by looking at the change in the quantity "promotions + mappings -
> demotions" before, during, and after the program execution.
>
> A program can call mincore(2) in order to determine if a virtual address
> is part of a 2 or 4MB virtual page.

Would it be possible to expose the super page count as kve_super in the
kinfo_vmentry struct so that procstat can show this information? If only to
determine if one is using the feature and possibly benefiting from it.

It looks like sys/kern/kern_proc.c could call mincore around the loop at line
1601 (rev 194498), but I know nothing about the vm subsystem to know the
implications or locking involved. There's still 16 bytes of spare to consume,
in the kve_vminfo struct though ;)
--
Mel
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Alan Cox-5]

Mel Flynn wrote:

> On Sunday 28 June 2009 15:41:49 Alan Cox wrote:
>  
>> Wojciech Puchar wrote:
>>    
>
>  
>>> how can i check how much (or maybe - what processes) 2MB pages are
>>> actually allocated?
>>>      
>> I'm afraid that you can't with great precision.  For a given program
>> execution, on an otherwise idle machine, you can only estimate the
>> number by looking at the change in the quantity "promotions + mappings -
>> demotions" before, during, and after the program execution.
>>
>> A program can call mincore(2) in order to determine if a virtual address
>> is part of a 2 or 4MB virtual page.
>>    
>
> Would it be possible to expose the super page count as kve_super in the
> kinfo_vmentry struct so that procstat can show this information? If only to
> determine if one is using the feature and possibly benefiting from it.
>
>  

Yes, I think so.

> It looks like sys/kern/kern_proc.c could call mincore around the loop at line
> 1601 (rev 194498), but I know nothing about the vm subsystem to know the
> implications or locking involved. There's still 16 bytes of spare to consume,
> in the kve_vminfo struct though ;)
>  

Yes, to start with, you could replace the call to pmap_extract() with a
call to pmap_mincore() and export a Boolean to user space that says,
"This region of the address space contains one or more superpage mappings."

Counting the number of superpages is a little trickier.  The problem
being that pmap_mincore() doesn't tell you how large the underlying page
is.  So, the loop at line 1601 can't easily tell where one superpage
ends and the next 4KB page or superpage begins, making counting the
number of superpages in a region a little tricky.  One possibility is to
change pmap_mincore() to return the page size (or the logarithm of the
page size) rather than a single bit.

If you want to give it a try, I'll be happy to help.  There aren't
really any "implications" or synchronization issues that you need to
worry about.

Alan



_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Mel Flynn-6]

On Tuesday 30 June 2009 00:24:00 Alan Cox wrote:

> Mel Flynn wrote:
> > On Sunday 28 June 2009 15:41:49 Alan Cox wrote:
> >> Wojciech Puchar wrote:
> >>> how can i check how much (or maybe - what processes) 2MB pages are
> >>> actually allocated?
> >>
> >> I'm afraid that you can't with great precision.  For a given program
> >> execution, on an otherwise idle machine, you can only estimate the
> >> number by looking at the change in the quantity "promotions + mappings -
> >> demotions" before, during, and after the program execution.
> >>
> >> A program can call mincore(2) in order to determine if a virtual address
> >> is part of a 2 or 4MB virtual page.
> >
> > Would it be possible to expose the super page count as kve_super in the
> > kinfo_vmentry struct so that procstat can show this information? If only
> > to determine if one is using the feature and possibly benefiting from it.
>
> Yes, I think so.
>
> > It looks like sys/kern/kern_proc.c could call mincore around the loop at
> > line 1601 (rev 194498), but I know nothing about the vm subsystem to know
> > the implications or locking involved. There's still 16 bytes of spare to
> > consume, in the kve_vminfo struct though ;)
>
> Yes, to start with, you could replace the call to pmap_extract() with a
> call to pmap_mincore() and export a Boolean to user space that says,
> "This region of the address space contains one or more superpage mappings."

How about attached?

% sudo procstat -av|grep 'S '
  PID      START        END PRT  RES PRES REF SHD  FL TP PATH
 1754 0x28900000 0x2ae00000 rw- 9385    0   3   0 --S df
 2141 0x2f900000 0x30800000 rw- 3719    0   1   0 --S df
 2146 0x3eec0000 0x4fac0000 rwx 1745    0   1   0 --S df

--
Mel


[procstat-superpages.patch]

---

Index: sys/sys/user.h
===================================================================
--- sys/sys/user.h (revision 195188)
+++ sys/sys/user.h (working copy)
@@ -348,6 +348,7 @@
 
 #define KVME_FLAG_COW 0x00000001
 #define KVME_FLAG_NEEDS_COPY 0x00000002
+#define KVME_FLAG_SUPER 0x00000004
 
 #if defined(__amd64__)
 #define KINFO_OVMENTRY_SIZE 1168
Index: sys/kern/kern_proc.c
===================================================================
--- sys/kern/kern_proc.c (revision 195188)
+++ sys/kern/kern_proc.c (working copy)
@@ -59,6 +59,7 @@
 #include <sys/signalvar.h>
 #include <sys/sdt.h>
 #include <sys/sx.h>
+#include <sys/mman.h>
 #include <sys/user.h>
 #include <sys/jail.h>
 #include <sys/vnode.h>
@@ -1599,8 +1600,13 @@
  kve->kve_resident = 0;
  addr = entry->start;
  while (addr < entry->end) {
- if (pmap_extract(map->pmap, addr))
+ int flags;
+
+ flags = pmap_mincore(map->pmap, addr);
+ if ( flags & MINCORE_INCORE )
  kve->kve_resident++;
+ if( flags & MINCORE_SUPER )
+ kve->kve_flags |= KVME_FLAG_SUPER;
  addr += PAGE_SIZE;
  }
 
Index: usr.bin/procstat/procstat_vm.c
===================================================================
--- usr.bin/procstat/procstat_vm.c (revision 195188)
+++ usr.bin/procstat/procstat_vm.c (working copy)
@@ -49,7 +49,7 @@
 
  ptrwidth = 2*sizeof(void *) + 2;
  if (!hflag)
- printf("%5s %*s %*s %3s %4s %4s %3s %3s %2s %-2s %-s\n",
+ printf("%5s %*s %*s %3s %4s %4s %3s %3s %3s %-2s %-s\n",
     "PID", ptrwidth, "START", ptrwidth, "END", "PRT", "RES",
     "PRES", "REF", "SHD", "FL", "TP", "PATH");
 
@@ -69,8 +69,9 @@
  printf("%3d ", kve->kve_ref_count);
  printf("%3d ", kve->kve_shadow_count);
  printf("%-1s", kve->kve_flags & KVME_FLAG_COW ? "C" : "-");
- printf("%-1s ", kve->kve_flags & KVME_FLAG_NEEDS_COPY ? "N" :
+ printf("%-1s", kve->kve_flags & KVME_FLAG_NEEDS_COPY ? "N" :
     "-");
+ printf("%-1s ", kve->kve_flags & KVME_FLAG_SUPER ? "S" : "-");
  switch (kve->kve_type) {
  case KVME_TYPE_NONE:
  str = "--";
Index: usr.bin/procstat/procstat.1
===================================================================
--- usr.bin/procstat/procstat.1 (revision 195188)
+++ usr.bin/procstat/procstat.1 (working copy)
@@ -332,6 +332,8 @@
 copy-on-write
 .It N
 needs copy
+.It S
+One or more superpage mappings are used
 .El
 .Sh EXIT STATUS
 .Ex -std

---

_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Robert N. M. Watson]

On Tue, 30 Jun 2009, Mel Flynn wrote:

>>> It looks like sys/kern/kern_proc.c could call mincore around the loop at
>>> line 1601 (rev 194498), but I know nothing about the vm subsystem to know
>>> the implications or locking involved. There's still 16 bytes of spare to
>>> consume, in the kve_vminfo struct though ;)
>>
>> Yes, to start with, you could replace the call to pmap_extract() with a
>> call to pmap_mincore() and export a Boolean to user space that says, "This
>> region of the address space contains one or more superpage mappings."
>
> How about attached?

I like the idea -- there are some style nits that need fixing though.
Assuming Alan is happy with the VM side of things, I can do the cleanup and
get it in the tree.

Robert N M Watson
Computer Laboratory
University of Cambridge

>
> % sudo procstat -av|grep 'S '
>  PID      START        END PRT  RES PRES REF SHD  FL TP PATH
> 1754 0x28900000 0x2ae00000 rw- 9385    0   3   0 --S df
> 2141 0x2f900000 0x30800000 rw- 3719    0   1   0 --S df
> 2146 0x3eec0000 0x4fac0000 rwx 1745    0   1   0 --S df
>
> --
> Mel
>
>

_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Alan Cox-5]

Robert Watson wrote:

>
> On Tue, 30 Jun 2009, Mel Flynn wrote:
>
>>>> It looks like sys/kern/kern_proc.c could call mincore around the
>>>> loop at line 1601 (rev 194498), but I know nothing about the vm
>>>> subsystem to know the implications or locking involved. There's
>>>> still 16 bytes of spare to consume, in the kve_vminfo struct though ;)
>>>
>>> Yes, to start with, you could replace the call to pmap_extract()
>>> with a call to pmap_mincore() and export a Boolean to user space
>>> that says, "This region of the address space contains one or more
>>> superpage mappings."
>>
>> How about attached?
>
> I like the idea -- there are some style nits that need fixing though.
> Assuming Alan is happy with the VM side of things, I can do the
> cleanup and get it in the tree.

Aside from the style nits, it looks good to me.

Alan

_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."