Nabble - MLton

Re: A Possible "Bug"?

2009-12-18T09:02:21Z

On Fri, Dec 18, 2009 at 9:15 AM, Henry Cejtin
<henry.cejtin@...> wrote:
> I would say that MLton is correct and SML/NJ is wrong.

Agreed, with the caveat noted below.

> The point is
> that the left hand side of the binding in
> val [[x]] = []
> is a non-exhaustive pattern and the right hand side does not match any of
> the cases. (The type is correct though.) Hence evaluating this should
> result in the Bind exception being raised.
>
> At least in v110.67 of SML/NJ, that is what happens.
>
> Perhaps you are being confused by the fact that the exception raising
> happens at evaluation time. In MLton, that is when you run the
> executable, but in SML/NJ, it is when you enter the expression to the
> read-eval-print loop.

There is a subtle difference in SML/NJ's behavior:

Standard ML of New Jersey v110.71 [built: Thu Oct 1 09:13:22 2009]
- val [[x]] = [];

unexpected exception (bug?) in SML/NJ: Bind [nonexhaustive binding failure]
raised at: stdIn:1.5-1.15
../compiler/MiscUtil/print/ppobj.sml:396.20
../compiler/TopLevel/interact/evalloop.sml:44.55

- val SOME x = NONE;

unexpected exception (bug?) in SML/NJ: Bind [nonexhaustive binding failure]
raised at: stdIn:1.5-1.18
../compiler/MiscUtil/print/ppobj.sml:396.20
../compiler/TopLevel/interact/evalloop.sml:44.55

- val NONE = SOME 1;

uncaught exception Bind [nonexhaustive binding failure]
raised at: stdIn:1.5-1.18

The last is what I would expect to see in all cases. The others seem
to be a case where the the REPL isn't expecting a Bind exception; it
seem to be related to patterns that should be binding variables (i.e.,
they are entered into the static environment), but then are not bound
due to the pattern match failure (i.e, they are not entered into the
dynamic environment).

> Note, if SML/NJ did not issue a warning about the binding not being
> exhaustive then that would be a bug in their code.

Except that the Defn states: "However, this warning should not be
given when the binding is a component of a top-level declaration...."
MLton gives pattern-match warnings in all cases.

> ----- Original Message ----
> From: Baojian Hua <huabj@...>
> To: mlton <mlton@...>
> Sent: Fri, December 18, 2009 4:02:57 AM
> Subject: [MLton] A Possible "Bug"?
>
> ...
>
> However, I happened to run this code:
> val [[x]] = [];
>
> MLton reports a warning message, but smlnj-110.71 reports
> an error (compiler bug?). I'd like to ask here that which
> behaves correctly?
>
> _______________________________________________
> MLton mailing list
> MLton@...
> http://mlton.org/mailman/listinfo/mlton

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: A Possible "Bug"?

2009-12-18T06:15:17Z

I would say that MLton is correct and SML/NJ is wrong. The point is
that the left hand side of the binding in
val [[x]] = []
is a non-exhastive pattern and the right hand side does not match any of
the cases. (The type is correct though.) Hence evaluating this should
result in the Bind exception being raised.

At least in v110.67 of SML/NJ, that is what happens.

Perhaps you are being confused by the fact that the exception raising
happens at evaluation time. In MLton, that is when you run the
executable, but in SML/NJ, it is when you enter the expression to the
read-eval-print loop.

Note, if SML/NJ did not issue a warning about the binding not being
exhaustive then that would be a bug in their code.

----- Original Message ----
From: Baojian Hua <huabj@...>
To: mlton <mlton@...>
Sent: Fri, December 18, 2009 4:02:57 AM
Subject: [MLton] A Possible "Bug"?

...

However, I happened to run this code:
val [[x]] = [];

MLton reports a warning message, but smlnj-110.71 reports
an error (compiler bug?). I'd like to ask here that which
behaves correctly?

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

A Possible "Bug"?

2009-12-18T02:02:57Z

Hi, all,

I've just read and modified some webpages that are
out-of-date due to some reasons.

For instance, I've deleted some sentences on
http://mlton.org/SMLNJDeviations

related to these code:
val [x] = [[]];
val _ = {1::x, "one"::x}

for that I've tried it in the latest smlnj-110.71, and
it seems that it runs smoothly.

However, I happened to run this code:
val [[x]] = [];

MLton reports a warning message, but smlnj-110.71 reports
an error (compiler bug?). I'd like to ask here that which
behaves correctly?

Thanks in advance.

-Paul

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

target-map considered harmful

2009-12-15T04:21:05Z

So, I've use a lot of MLton cross compilers and have been looking at
how to get MLton packages that achieve this. I was hoping in the
future to be able to type:
apt-get install mlton mlton-mingw32 mlton-mipsel
... and then get a mlton capable of compiling to my native system,
mingw32, or mipsel targets.

This seems quite easy to achieve, except the annoying target-map file
is one file. Adding the needed sml/basis/config/c/xxx/c-types.sml and
xxx/*.a files in the mlton-mingw32 package is no problem. However, I
can't just overwrite target-map.

Couldn't we instead have a personality file inside the target folders? eg:

cat self/personality
i386 linux
cat i586-mingw32msvc/personality
i386 mingw

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Call For Participation: DAMP 2010

2009-12-14T11:18:43Z

We have several good ML papers this year, including one looking at parallel constructs in MLton. Please consider coming to Madrid to hear what's going on and to share your thoughts.

Cheers,
Leaf Petersen (General Chair)

DAMP 2010: Workshop on
Declarative Aspects of Multicore Programming
Madrid, SPAIN
(colocated with POPL 2010)
January 19, 2010
damp10.cs.nmsu.edu

The advent of multicore architectures has profoundly increased the
importance of research in parallel computing. Modern platforms are
becoming more complex and heterogenous and novel solutions are needed
to account for their peculiarities.
Multicore architectures will differ in significant ways from their
multisocket predecessors. For example, the communication to compute
bandwidth ratio is likely to be higher, which will positively impact
performance. More generally, multicore architectures introduce several
new dimensions of variability in both performance guarantees and
architectural contracts, such as the memory model, that may not
stabilize for several generations of product.

Programs written in functional or (constraint-)logic programming
languages, or in other highly declarative languages with a controlled
use of side effects, can greatly simplify parallel programming. Such
declarative programming allows for a deterministic semantics even
when the underlying implementation might be highly non-deterministic.
In addition to simplifying programming this can simplify debugging and
analyzing correctness.

DAMP 2010 is the fifth in a series of one-day workshops seeking to
explore ideas in declarative programming language design that will
greatly simplify programming for multicore architectures, and more
generally for tightly coupled parallel architectures. DAMP seeks to
gather together researchers in declarative approaches to parallel
programming and to foster cross fertilization across different
approaches.

Preliminary Program:
====================
Tuesday January 19th, 2010:

Invited Talk [9:00-10:00]
* Parallelizing Constraint Programs.
Laurent Michel (University of Connecticut)

Coffee Break [10:00-10:30]

Session 1 [10:30-11:30]
* PASTHA - Parallelizing Stencil Calculations in Haskell
Michael Lesniak
* Ypnos: Declarative Parallel Structured Grid Programming
Dominic Orchard, Alan Mycroft, Max Bolingbroke

Coffee Break [11:30-12:00]

Session 2 [12:00-13:00]
* SequenceL: Transparency and Multi-core Parallelism
Brad Nemanich, Daniel Cooke, Nelson Rushton
* Efficient Parallel Programming in Poly/ML and Isabelle/ML
David Matthews, Makarius Wenzel

Lunch (PROVIDED by the Conference) [13:00-14:30]

Session 3 [14:30-15:30]
* S-Net for Multi-Memory Multicores
Clemens Grelck, Jukka Julku, Frank Penczek
* Compress-and-Conquer for Optimal Multicore Computing
Z. George Mou, Hai Liu, Paul Hudak

Coffee Break [15:30-16:00]

Sesssion 4 [16:00-17:00]
* Lightweight Ansynchrony using Parasitic Threads
KC Sivaramakrishnan, Lukasz Ziarek, Raghavendra Prasad,
Suresh Jagannathan
* A Parallel ASP Instantiator Based on DLV
Simona Perri, Francesco Ricca, Marco Sirianni

Invited Talk [17:00-18:00]
* Declarative Data-Parallel Programming with the Accellerator
System
Satnam Singh (Microsoft Research)

URL:
====

http://damp10.cs.nmsu.edu

_______________________________________________
MLton-user mailing list
MLton-user@...
http://mlton.org/mailman/listinfo/mlton-user

From forum: MLton - User

Re: max-heap setting for 64-bit applications

2009-12-14T10:50:44Z

On Mon, Dec 14, 2009 at 6:53 PM, Matthew Fluet <matthew.fluet@...> wrote:
> I was thinking about the 32-bit case, in which case fragmenting the 4G
> VM isn't terribly difficult.

Ok, then let me address your original comment again:
> But, the Windows specific mremap could still fail --- note that the
> growHeap function demands significant growth from mremap. If that
> fails, then it attempts the alloc/copy, but allowing for an alloc of a
> heap down to the minimum size.

This sounds like a bad idea, then. The problem applies equally to all
platforms (even linux).

So, to be clear, we're talking about this situation:
memory is available for the minimum size
that memory is NOT large enough to accommodate "significant growth"
the memory region contains the current mapping

Under this case, AFAICT, every system fails to grow the heap, though
it should succeed. Both windows and linux would've been fine if the GC
had attempted to use mremap backed off to a lower value.

I submit that this is a bug which should be fixed.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-14T10:15:53Z

* Matthew Fluet:

> Oh, I certainly understand that under-the-hood any VMM is likely to
> have the reserve/commit distinction; plus plenty more subtleties:
> sharing of zero-pages, copy-on-write for fork-exec, etc., etc. But,
> if it isn't exposed in a user-space ABI, then it isn't "there" for us.
> Windows, on the other hand, *requires* the user to manage the
> distinction via the published user-space ABI.

Linux exposes the distinction as well if you set vm.overcommit_memory
to 2. There's just no ABI guarantee how to reserve part of the
address space without having it to count towards the commit limit.
The PROT_NONE business seems to work, but most run-time libraries
allocating a single heap block get this wrong.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-14T09:57:51Z

On Mon, Dec 14, 2009 at 12:51 PM, Florian Weimer <fw@...> wrote:

> * Matthew Fluet:
>
>> (Or, on Windows, use the reserve/commit distinction.
>
> It's there on Linx, too.
>
>> On Posix, use the madvise to preemptively indicate page-out/page-in
>> behavior.
>
> Doesn't work. You have to map with PROT_NONE and then use mprotect to
> make available the parts you need.
>
> This is not a published kernel ABI, so it might break in the future.

Oh, I certainly understand that under-the-hood any VMM is likely to
have the reserve/commit distinction; plus plenty more subtleties:
sharing of zero-pages, copy-on-write for fork-exec, etc., etc. But,
if it isn't exposed in a user-space ABI, then it isn't "there" for us.
Windows, on the other hand, *requires* the user to manage the
distinction via the published user-space ABI.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-14T09:53:05Z

On Mon, Dec 14, 2009 at 12:45 PM, Wesley W. Terpstra <wesley@...> wrote:
> On Mon, Dec 14, 2009 at 5:50 PM, Matthew Fluet <matthew.fluet@...> wrote:
>> On Sat, Dec 12, 2009 at 8:24 AM, Wesley W. Terpstra <wesley@...> wrote:
>>> The mremap code does a memcpy, which is 100% sequential.
>> But, the Windows specific mremap could still fail --- note that the
>> growHeap function demands significant growth from mremap. If that
>> fails, then it attempts the alloc/copy, but allowing for an alloc of a
>> heap down to the minimum size.
>
> This could only happen if you fragmentted the entire 16TB of VM.

I was thinking about the 32-bit case, in which case fragmenting the 4G
VM isn't terribly difficult.

> I bet it's probably always faster to grow in-place. As there have been
> no issues with the mremap code since I wrote it, I'm confident enough
> in it to switch it to the default, committed.
>
> It might also be worth writing an improved copy/move code-path that
> allocates as it goes, thus avoiding swapping. ie:
> (commit/copy/decommit)s in blocks of 4MB at a time, releasing the
> source reservation as the last step. Not as amazing as a real mremap
> which doesn't need to copy at all, but at least it would have the same
> 'trash or not thrash' behaviour.

On Windows, yes. Reserve the whole target heap size, and during the
copy, commit/decommit as we go.

I don't think we can get the same kind of behavior on POSIX.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-14T09:51:13Z

* Matthew Fluet:

> (Or, on Windows, use the reserve/commit distinction.

It's there on Linx, too.

> On Posix, use the madvise to preemptively indicate page-out/page-in
> behavior.

Doesn't work. You have to map with PROT_NONE and then use mprotect to
make available the parts you need.

This is not a published kernel ABI, so it might break in the future.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-14T09:45:20Z

On Mon, Dec 14, 2009 at 5:50 PM, Matthew Fluet <matthew.fluet@...> wrote:
> On Sat, Dec 12, 2009 at 8:24 AM, Wesley W. Terpstra <wesley@...> wrote:
>> The mremap code does a memcpy, which is 100% sequential.
> But, the Windows specific mremap could still fail --- note that the
> growHeap function demands significant growth from mremap. If that
> fails, then it attempts the alloc/copy, but allowing for an alloc of a
> heap down to the minimum size.

This could only happen if you fragmentted the entire 16TB of VM.
My point was that the only swapping he (or anyone else) saw was
sequential, and should be as fast as a disk-backed copy.

>> As already mentioned, the design goal for windows mremap was not efficiency,
>> but to use more virtual address space. I think both agree that alloc/copy as
>> the last step makes more sense for 64-bit systems to reduce the likelihood
>> of thrashing.
>
> I guess the question is whether the large alloc/copy is really more
> efficient on even a 32-bit system.

I bet it's probably always faster to grow in-place. As there have been
no issues with the mremap code since I wrote it, I'm confident enough
in it to switch it to the default, committed.

It might also be worth writing an improved copy/move code-path that
allocates as it goes, thus avoiding swapping. ie:
(commit/copy/decommit)s in blocks of 4MB at a time, releasing the
source reservation as the last step. Not as amazing as a real mremap
which doesn't need to copy at all, but at least it would have the same
'trash or not thrash' behaviour.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: Re: [MLton-commit] r7386

2009-12-14T08:54:40Z

On Mon, Dec 14, 2009 at 11:47 AM, Wesley W. Terpstra <wesley@...> wrote:
> On Mon, Dec 14, 2009 at 5:11 PM, Matthew Fluet <matthew.fluet@...> wrote:
>> ... seems to be no good reason to ... for Snow Leopard.
>
> Apple?

The only thing I could think of, and even this is a stretch, is that
having dropped PowerPC with Snow Leopard, they don't need to maintain
bit-level compatibility with programs over the two architectures.
PowerPC presumably uses proper IEEE-754 arithmetic. But, it seems
like that would have been quite hard to establish, given that x86
needed to use x87 calling conventions. And, if one did establish it,
keeping IEEE-754 arithmetic on x86/amd64 would be ever so much nicer.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-14T08:50:48Z

On Sat, Dec 12, 2009 at 8:24 AM, Wesley W. Terpstra <wesley@...> wrote:
> On Sat, Dec 12, 2009 at 8:48 AM, Matthew Fluet <matthew.fluet@...>
> wrote:
>>
>> It isn't quite sequential access.
>
> Sorry. You're both wrong. ;) The mremap code does a memcpy, which is 100%
> sequential. That's the only transition between the two heaps on Windows ->
> inside mremap. It allocates the new heap, memcopies them, then frees the old
> heap. The only time the GC touches it is when it fixes up the addresses.

But, the Windows specific mremap could still fail --- note that the
growHeap function demands significant growth from mremap. If that
fails, then it attempts the alloc/copy, but allowing for an alloc of a
heap down to the minimum size. (Again, perhaps not extremely likely,
but in a program that is using FFI and doing C-side allocations that
somewhat fragment the virtual address space, I could imagine a 1.25G
heap being unable to be (significantly) grown to a heap of size 2.75G,
but the alloc/copy succeeds with a heap of 1.3G.)

>> >> the mremap function is described as using the Linux page table scheme
>> >> to
>> >> efficiently change the mapping between virtual addresses and (physical)
>> >> memory pages. It's purpose is to be more efficient than allocating a
>> >> new
>> >> map and copying.
>> >
>> > If I could ...
>>
>> I guess my point is that the way that you indicate that you aren't
>> more efficient than alloc/copy is by not providing mremap. Everything
>> else in the generic implementation with attempting the in place expand
>> is more efficient; it is just the starting off with the alloc/copy
>> that doesn't seem to make sense.
>
> As already mentioned, the design goal for windows mremap was not efficiency,
> but to use more virtual address space. I think both agree that alloc/copy as
> the last step makes more sense for 64-bit systems to reduce the likelihood
> of thrashing.

I guess the question is whether the large alloc/copy is really more
efficient on even a 32-bit system.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: Re: [MLton-commit] r7386

2009-12-14T08:47:26Z

On Mon, Dec 14, 2009 at 5:11 PM, Matthew Fluet <matthew.fluet@...> wrote:
> ... seems to be no good reason to ... for Snow Leopard.

Apple?

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-14T08:44:28Z

On Fri, Dec 11, 2009 at 10:56 PM, Henry Cejtin
<henry.cejtin@...> wrote:
> With regards to the max-heap argument, it would be really good to have
> an argument which is the maximum amount of memory MLton can use, or as
> close as we can come to that. If it tries to grow bigger, even if only
> temporarily to grow a heap, and can't grow it by any other allowed
> means, it should fail with out of memory.

I think the only way one could use such an argument would be to
malloc/mmap that whole space at the beginning of the program and never
release or resize it. We could still make use of a smaller heap
within that reserved space, and let the OS VMM page out the unused
pages. (Or, on Windows, use the reserve/commit distinction. On
Posix, use the madvise to preemptively indicate page-out/page-in
behavior. I think I recall seeing that some *BSDs allow you to get
the reserve/commit distinction that Windows provides.)

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: [MLton-commit] r7386

2009-12-14T08:11:09Z

On Mon, Dec 14, 2009 at 9:36 AM, Wesley Terpstra <wesley@...> wrote:

> Snow Leopard real regression no longer differs from normal x86.
>
>
> ----------------------------------------------------------------------
>
> D mlton/trunk/regression/real.x86-darwin.ok
>
> ----------------------------------------------------------------------
>
> Deleted: mlton/trunk/regression/real.x86-darwin.ok

Weird. I always assumed that the differences in the real regression
on darwin were due to differences in the implementation of libm.
Because MacOSX doesn't run on any pre-SSE2 Intel chips, the MacOSX
delivered libm could be compiled to use SSE2 (but using x87 for the
calling convention); seems to be no good reason to go to an x87-only
implementation for Snow Leopard.

Linux, on the other hand, does need to support x87 only chips, so
presumably that libm uses x87 instructions exclusively.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: [MLton-commit] r7385

2009-12-14T08:03:42Z

Good to hear. Presumably the warning arose from an older MLton
compiling (any) sources on Snow Leopard and the warning in
<ucontext.h> being emitted despite the "gcc -w" invocation.
-Matthew

On Mon, Dec 14, 2009 at 9:11 AM, Wesley Terpstra <wesley@...> wrote:

> Comfirmed that this is NOT necessary when building svn/HEAD from a similarly new MLton.
>
>
> ----------------------------------------------------------------------
>
> U mlton/trunk/runtime/platform/darwin.h
>
> ----------------------------------------------------------------------
>
> Modified: mlton/trunk/runtime/platform/darwin.h
> ===================================================================
> --- mlton/trunk/runtime/platform/darwin.h 2009-12-12 02:09:51 UTC (rev 7384)
> +++ mlton/trunk/runtime/platform/darwin.h 2009-12-14 14:11:34 UTC (rev 7385)
> @@ -1,5 +1,3 @@
> -#define _XOPEN_SOURCE
> -
> #include <fenv.h>
> #include <inttypes.h>
> #include <stdint.h>

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: Re: [MLton-commit] r7381

2009-12-13T07:13:21Z

On Sat, Dec 12, 2009 at 2:51 PM, Christof Leng
<cleng@...> wrote:
> I attached the output of make check for the real and world5 tests.

I notice the changes in the real regression you posted are the exact
reverse of the changes from real.ok to real.x86-darwin.ok. I guess
that means this is a regresion only in the sense that darwin/x86 has
regressed into linux/x86-like behaviour. :)

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: Re: [MLton-commit] r7381

2009-12-12T05:51:48Z

On 12.12.2009 14:10, Wesley W. Terpstra wrote:

My colleague reported this error message:/usr/include/ucontext.h:42:2: error: #error ucontext routines are deprecated, and require _XOPEN_SOURCE to be defined" ... then he added the #define and it worked.

I'm Wesley's colleague who did this change to compile. I don't have remote access to the machine right now, but I'll check on Monday what the comment in ucontext.h said. I don't know if it's better to define _XOPEN_SOURCE or switch to sys/ucontext.h. Both approaches seem to work with Snow Leopard.

I don't have a Snow Leopard system, but he got told me that he got this error on the newest svn/HEAD. I'll get him to double-check on Monday and send the real regression log.

I attached the output of make check for the real and world5 tests.

testing real
666c666
< 11.59195518
---
> > 11.59195423
669c669
< 0.9962721467
---
> > 0.9962720871
678c678
< 7.544136047
---
> > 7.544136524
713c713
< 0.001230000518
---
> > 0.001230000402
817c817
< 11.59195518
---
> > 11.59195423
821c821
< ~0.9962721467
---
> > ~0.9962720871
830c830
< ~7.544136047
---
> > ~7.544136524
866c866
< ~0.001230000518
---
> > ~0.001230000402
difference with -type-check true

testing world5
1,2c1,4
< success
< success
---
> > fread (0xbffff5ec, 1, 1, _) failed (only read 0; eof).
> > (Unknown error: 0)
> > unhandled exception: Fail: child failed
> > Nonzero exit status.
difference with -type-check true

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-12T05:24:29Z

On Sat, Dec 12, 2009 at 8:48 AM, Matthew Fluet <matthew.fluet@...> wrote:

It isn't quite sequential access.

Sorry. You're both wrong. ;) The mremap code does a memcpy, which is 100% sequential. That's the only transition between the two heaps on Windows -> inside mremap. It allocates the new heap, memcopies them, then frees the old heap. The only time the GC touches it is when it fixes up the addresses.

> I have a higher-level solution proposal: MLton.GC already has hooks that are
> executed after a GC to implement finalizers. Expose these to the user. If a
> user knows his application only consumes X memory on an error condition, he
> can test for this after a GC and terminate with an "Out of Memory" error as
> desired.

You still need to get out of the GC before the GC signal handler runs.

Of course, but we are talking about early detection. Under normal operation his program probably uses >much< less than the total RAM. Stopping at 1GB with an error would certainly be earlier than thrashing set in.

Right now we are focusing in on a micro-program which just consumes memory and we're talking about how to optimize it. It's a pathological program! We should instead be supporting David's underlying goal: detecting when his program enters run-away RAM consumption. This happens much much sooner than thrashing.

>> the mremap function is described as using the Linux page table scheme to
>> efficiently change the mapping between virtual addresses and (physical)
>> memory pages. It's purpose is to be more efficient than allocating a new
>> map and copying.
>
> If I could ...

I guess my point is that the way that you indicate that you aren't
more efficient than alloc/copy is by not providing mremap. Everything
else in the generic implementation with attempting the in place expand
is more efficient; it is just the starting off with the alloc/copy
that doesn't seem to make sense.

As already mentioned, the design goal for windows mremap was not efficiency, but to use more virtual address space. I think both agree that alloc/copy as the last step makes more sense for 64-bit systems to reduce the likelihood of thrashing.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: Re: [MLton-commit] r7381

2009-12-12T05:10:56Z

My colleague reported this error message:/usr/include/ucontext.h:42:2: error: #error ucontext routines are deprecated, and require _XOPEN_SOURCE to be defined" ... then he added the #define and it worked.

I don't have a Snow Leopard system, but he got told me that he got this error on the newest svn/HEAD. I'll get him to double-check on Monday and send the real regression log.

I notice that the error message explicitly mentions /usr/include/ucontext.h not sys/ucontext.h. As he was bootstrapping from your last experimental upload, he had to change its included darwin.h as the first step. It's possible that if he'd instead changed the installed darwin.h to use sys/ucontext.h it would also have worked.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-11T23:48:17Z

On Fri, Dec 11, 2009 at 7:17 PM, Wesley W. Terpstra <wesley@...> wrote:
> Stepping outside of the current discussion, a matter of practicality: is the
> paging really that bad? AFAIK, it is a memcpy running between the two heaps
> which is sequential access. It shouldn't take longer than it would take to
> copy a 3GB file on the disk.
>
> If it *is* taking longer, then we need to add a hint to the windows VM to
> tell it that we will be doing sequential access before the memcpy, then flip
> it back to random-access mode.

It isn't quite sequential access. See growHeap in
<src>/runtime/gc/heap.c. We actually perform the copy from back to
front, with a memcpy (presumably doing sequential access) for each 32M
chunk. The reason for starting the copy from the back is that we can
shrink the old heap as we go.

> On Fri, Dec 11, 2009 at 9:32 PM, Matthew Fluet <matthew.fluet@...>
> wrote:
>>
>> Right, and the annoying bit is that the previous heap was so close to
>> the max heap setting. Perhaps a reasonable heuristic is that if a
>> desired heap is "close-to" the max-heap size, just round up. Perhaps
>> 0.75 of max heap? In the max-heap 3G setting, this could still leave
>> you in the situation where you have a 2.25G allocation and a 3G
>> allocation at the same time to copy. Or 0.55 of max heap; that could
>> require 1.65G+3G at the time of the copy.
>
> I would be against yet another special case in the sizing rules. Any cutoff
> we pick is going to fail for someone else the same way, while artificially
> restricting the memory growth for others. His problem would be (mostly)
> fixed if we flipped windows mremap to only move if growth fails.

That assumes that the inplace growth can succeed. (Agreed that on
amd64, there is a good chance that it will succeed.) If the inplace
growth cannot succeed, then runtime will attempt a moving growth (via
an explicit create and copy), and we still overcommit memory (relative
to the max-heap setting).

> I have a higher-level solution proposal: MLton.GC already has hooks that are
> executed after a GC to implement finalizers. Expose these to the user. If a
> user knows his application only consumes X memory on an error condition, he
> can test for this after a GC and terminate with an "Out of Memory" error as
> desired.

You still need to get out of the GC before the GC signal handler runs.

>> the mremap function is described as using the Linux page table scheme to
>> efficiently change the mapping between virtual addresses and (physical)
>> memory pages. It's purpose is to be more efficient than allocating a new
>> map and copying.
>
> If I could ...

I guess my point is that the way that you indicate that you aren't
more efficient than alloc/copy is by not providing mremap. Everything
else in the generic implementation with attempting the in place expand
is more efficient; it is just the starting off with the alloc/copy
that doesn't seem to make sense.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: [MLton-commit] r7381

2009-12-11T23:28:21Z

Are you sure this is necessary? John Reppy has been compiling and
running MLton under Snow Leopard for a couple of months and only
required changing
#include <ucontext.h>
to
#include <sys/ucontext.h>
(r7231-7232).

On Fri, Dec 11, 2009 at 7:52 PM, Wesley Terpstra <wesley@...> wrote:

> Under Snow Leopard, the ucontext structure is only available if
> _XOPEN_SOURCE is defined.
>
>
> ----------------------------------------------------------------------
>
> U mlton/trunk/runtime/platform/darwin.h
>
> ----------------------------------------------------------------------
>
> Modified: mlton/trunk/runtime/platform/darwin.h
> ===================================================================
> --- mlton/trunk/runtime/platform/darwin.h 2009-12-11 19:51:40 UTC (rev 7380)
> +++ mlton/trunk/runtime/platform/darwin.h 2009-12-12 00:52:19 UTC (rev 7381)
> @@ -1,3 +1,5 @@
> +#define _XOPEN_SOURCE
> +
> #include <fenv.h>
> #include <inttypes.h>
> #include <stdint.h>
>
>
> _______________________________________________
> MLton-commit mailing list
> MLton-commit@...
> http://mlton.org/mailman/listinfo/mlton-commit
>

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-11T19:56:02Z

Is GC really sequential in terms of memory access? Certainly the writes
to new space are, but not the reads from old space, nor the writes to
old space (to set the forwarding pointer).

I have a vague recollection of Macsyma on the VAX (before even my time),
which no doubt used a mark/sweep collector, and they used vadvise to set
random behaviour (I thought) during GC.

With regards to the max-heap argument, it would be really good to have
an argument which is the maximum amount of memory MLton can use, or as
close as we can come to that. If it tries to grow bigger, even if only
temporarily to grow a heap, and can't grow it by any other allowed
means, it should fail with out of memory.

From: Wesley W. Terpstra <wesley@...>
To: Matthew Fluet <matthew.fluet@...>
Cc: mlton@...
Sent: Fri, December 11, 2009 6:17:34 PM
Subject: Re: [MLton] max-heap setting for 64-bit applications

Stepping outside of the current discussion, a matter of practicality: is the paging really that bad? AFAIK, it is a memcpy running between the two heaps which is sequential access. It shouldn't take longer than it would take to copy a 3GB file on the disk.

If it *is* taking longer, then we need to add a hint to the windows VM to tell it that we will be doing sequential access before the memcpy, then flip it back to random-access mode.

On Fri, Dec 11, 2009 at 9:32 PM, Matthew Fluet <matthew.fluet@...> wrote:

Right, and the annoying bit is that the previous heap was so close to
the max heap setting. Perhaps a reasonable heuristic is that if a
desired heap is "close-to" the max-heap size, just round up. Perhaps
0.75 of max heap? In the max-heap 3G setting, this could still leave
you in the situation where you have a 2.25G allocation and a 3G
allocation at the same time to copy. Or 0.55 of max heap; that could
require 1.65G+3G at the time of the copy.

I would be against yet another special case in the sizing rules. Any cutoff we pick is going to fail for someone else the same way, while artificially restricting the memory growth for others. His problem would be (mostly) fixed if we flipped windows mremap to only move if growth fails.

I have a higher-level solution proposal: MLton.GC already has hooks that are executed after a GC to implement finalizers. Expose these to the user. If a user knows his application only consumes X memory on an error condition, he can test for this after a GC and terminate with an "Out of Memory" error as desired.

> It seems like one strategy would be that keepomg the "max-heap"
> setting slightly under half the available physical memory should
> avoid the case where we're already using about 50% and then have
> to create another heap of the same size.

True. Although, you would really need to use about 50% of the
physical memory that you want the MLton process to have access to,
else you will page via competition with other processes.

I think this isn't quite right. You get paging when the working set of the other applications + heap of MLton program > RAM. If his heap stays small, there is no thrashing. Once it gets big, there wilt be thrashing iff the working set of apps + max-heap > RAM. Keeping the max-heap size warm wouldn't help, only cause more thrashing sooner.

Going to a single
contiguous heap, interpreting it as two semi-spaces when using a major
copying-collection would be nicer here, because fixed-heap would grab
the whole 3.25G up front.

Yes.

the mremap function is described as using the Linux page table scheme to efficiently change the mapping between virtual addresses and (physical) memory pages. It's purpose is to be more efficient than allocating a new map and copying.

If I could ...

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-11T16:17:34Z

Stepping outside of the current discussion, a matter of practicality: is the paging really that bad? AFAIK, it is a memcpy running between the two heaps which is sequential access. It shouldn't take longer than it would take to copy a 3GB file on the disk.

If it *is* taking longer, then we need to add a hint to the windows VM to tell it that we will be doing sequential access before the memcpy, then flip it back to random-access mode.

On Fri, Dec 11, 2009 at 9:32 PM, Matthew Fluet <matthew.fluet@...> wrote:

Right, and the annoying bit is that the previous heap was so close to
the max heap setting. Perhaps a reasonable heuristic is that if a
desired heap is "close-to" the max-heap size, just round up. Perhaps
0.75 of max heap? In the max-heap 3G setting, this could still leave
you in the situation where you have a 2.25G allocation and a 3G
allocation at the same time to copy. Or 0.55 of max heap; that could
require 1.65G+3G at the time of the copy.

> It seems like one strategy would be that keepomg the "max-heap"
> setting slightly under half the available physical memory should
> avoid the case where we're already using about 50% and then have
> to create another heap of the same size.

True. Although, you would really need to use about 50% of the
physical memory that you want the MLton process to have access to,
else you will page via competition with other processes.

Going to a single
contiguous heap, interpreting it as two semi-spaces when using a major
copying-collection would be nicer here, because fixed-heap would grab
the whole 3.25G up front.

Yes.

the mremap function is described as using the Linux page table scheme to efficiently change the mapping between virtual addresses and (physical) memory pages. It's purpose is to be more efficient than allocating a new map and copying.

If I could ...

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-11T12:32:45Z

On Fri, Dec 11, 2009 at 1:15 PM, David Hansel
<hansel@...> wrote:

> Thanks for the detailed description on MLton's memory usage. This
> does explain what is going on. I ran the example with "gc-messages
> max-heap 3000M" and I see the following before the trashing starts:
>
> [GC: Growing heap at 0x0000000080000000 of size 3,062,628,352 bytes
> (+ 23,986,176 bytes card/cross map),]
> [GC: to desired size of 3,121,217,536 bytes and minimum size
> of 3,070,031,512 bytes.]
>
> When running with the "max-heap 2950M" setting, all the other
> messages are the same but the final "growing" operation does not
> happen ("Out of memory" is reported).
>
> This seems to reflect exactly what Matthew suggested, MLton is
> trying to grow the heap to its maximum size which causes the
> thrashing.

Right, and the annoying bit is that the previous heap was so close to
the max heap setting. Perhaps a reasonable heuristic is that if a
desired heap is "close-to" the max-heap size, just round up. Perhaps
0.75 of max heap? In the max-heap 3G setting, this could still leave
you in the situation where you have a 2.25G allocation and a 3G
allocation at the same time to copy. Or 0.55 of max heap; that could
require 1.65G+3G at the time of the copy.

> It seems like one strategy would be that keepomg the "max-heap"
> setting slightly under half the available physical memory should
> avoid the case where we're already using about 50% and then have
> to create another heap of the same size.

True. Although, you would really need to use about 50% of the
physical memory that you want the MLton process to have access to,
else you will page via competition with other processes.

> Another observation I've made is that using "fixed-heap" instead
> of "max-heap" seems to avoid the thrashing (unless I set the
> size unreasonably high, like 5GB on my 6GB machine running a
> number of other applications). I had somewhat expected for MLton
> to immediately allocate the whole heap but (according to the
> task manager) that did not happen. Are there any other drawbacks
> about using fixed-heap?

It would make sense for fixed-heap to grab the whole heap at once.
However, because the semi-spaces of the copying garbage collector are
two separate heaps (not two halves of one heap), MLton will initially
only allocate one of the heaps and won't allocate the secondary heap
until the first (major) garbage collection. If you run with @MLton
fixed-heap 3G --, then MLton will allocate a 1.5G heap, wait for that
to fill, perform a major copy-collection into another 1.5G heap (now
we have two 1.5G heaps, matching our "fixed-heap" setting). In your
example allocating program, where there is almost no garbage, after
the major copy-collection, MLton will see a heap at near capacity and
attempt to resize it to 3G (the "fixed-heap" setting), in order to
have a larger heap for mark-compact collection. Going from 1.5G to
3G, even when both are allocated for the copy, probably isn't
noticeable as thrashing. If, after the copying collection, the live
data was a small fraction of the heap, then MLton stick with copying
collection (using a 1.5G heap), but won't deallocate the secondary
heap unless keeping both of them exceeds the available RAM.

I don't think that there are any drawbacks to using fixed-heap, as
long as you are willing to devote that amount of memory to the
process. Especially on a 64-bit system, if you set the fixed-heap to
a proportion of ram such that the jump from 1/2-fixed-heap to
fixed-heap isn't noticeable as thrashing, then you should have
predictable behavior. On a 32-bit system, it isn't as nice. Since we
now have lots of systems with >4G memory, but only 4G virtual address
space, it would be very nice to run processes with fixed-heap 3.25G.
Unfortunately, because of the above described semi-space behavior,
what end up happening is that a 1.125G heap is allocated and then it
can be difficult to allocate the other 1.125G heap in the address
space. And can be even harder (and, indeed, in the absence of a
proper mremap, impossible) to grow the 1.125G heap to 3.25G without
the runtime dumping the 1.125G heap to disk. On those systems, you
really want to grab the whole 3.25G up front. Going to a single
contiguous heap, interpreting it as two semi-spaces when using a major
copying-collection would be nicer here, because fixed-heap would grab
the whole 3.25G up front.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-11T11:44:59Z

On Thu, Dec 10, 2009 at 7:08 PM, Wesley W. Terpstra <wesley@...> wrote:

On Thu, Dec 10, 2009 at 9:37 PM, Matthew Fluet <matthew.fluet@...> wrote:

Note, that while Windows doesn't natively support mremap, Wesley implemented a few things that try to mimic mmap/mremap/munamp functionality under Windows. There seems to be some complication with the Windows memory system that requires committing in addition to reserving memory, and some manner of being able to extend an existing map with space before it and after it. I don't understand it.

When you reserve memory, you get virtual address space that does not overlap any previous reservations. When you commit (part of) that reservation, you get memory that can actually be used without segfaulting.

That part makes sense; I was simply noting that you go to some lengths to use this facility in an efficient manner and I wasn't familiar enough with that code to change its behavior.

Of course, even if mremap tries in-place heap growth first, this might not be possible, and a copy to a new memory location will still thrash.

Absolutely.

(This actually seems to be redundant with the behavior of the garbage collector proper.)

Perhaps, but I was trying to implement an mremap compatible to the specification of mremap, not how MLton uses mremap. If mremap fails to aquire the desired space, mmap will certainly also fail.

Fair enough. Although, and I freely admit that this is not necessarily part of the mremap specification, on Linux, the mremap function is described as using the Linux page table scheme to efficiently change the mapping between virtual addresses and (physical) memory pages. It's purpose is to be more efficient than allocating a new map and copying.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: Versioning and hosting

2009-12-11T11:33:11Z

On Fri, Dec 11, 2009 at 10:42 AM, Wesley W. Terpstra <wesley@...> wrote:

I received a FTBFS (fails to build from source) debian bug #560623 [1] today, which turns out to be a patch compatibility bug already fixed by Matthew (r7369). (BTW, what the hell?! patch is as old and stable a unix tool as they come; what genius changed its behaviour?)

Agreed that changing patch's behavior is quite bizarre.

In the process of fixing this I noticed that svn browsing is broken [2].

Just a couple days ago, Stephen disabled viewsvn.cgi because something was causing many copies to be spawned and overload the machine. We wanted to check the apache logs before turning it back on.

Before I upload a new version of the debian package and go through the hassle of getting all the ports to build cleanly again (they are currenly all green! [3]), I thought I'd also address bug #559014 [4], which is a complaint about the version number I picked for the MLton snapshot; I called it 20091107.

Since MLton doesn't use version numbers, I can't really call it 1.99+r7369 to make it clear it's a prerelease snapshot of version 2.0. However, my current version name is a bad choice because it implies that 20091107 was an "official" release. The best I can propose is 20091211+r7369. Suggestions / complaints?

Would something as simple as "20091211pre+r7369" be sufficiently indicative of a pre-release?

Of course, if those two regressions on osx were fixed, maybe we'd have an official release and the point would be moot. BTW, a colleague of mine reports everything works under Snow Leopard except the regressions world5 and real. A small fix was needed which I'll commit shortly.

I think I've got the two performance regressions I mentioned last time solved. I only know of the world5 regression failure on Leopard (x86-darwin); I wasn't sure that it was a failure on Snow Leopard (x86-darwin). What did you need to fix?

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-11T10:15:55Z

Hi Matthew and Wesley,

Thanks for the detailed description on MLton's memory usage. This
does explain what is going on. I ran the example with "gc-messages
max-heap 3000M" and I see the following before the trashing starts:

[GC: Growing heap at 0x0000000080000000 of size 3,062,628,352 bytes
(+ 23,986,176 bytes card/cross map),]
[GC: to desired size of 3,121,217,536 bytes and minimum size
of 3,070,031,512 bytes.]

When running with the "max-heap 2950M" setting, all the other
messages are the same but the final "growing" operation does not
happen ("Out of memory" is reported).

This seems to reflect exactly what Matthew suggested, MLton is
trying to grow the heap to its maximum size which causes the
thrashing.

My question now comes from a practical point of view: how can I
(if possible) avoid this from happening to customers using our
application? Of course we are trying to make our code as memory
efficient as possible but in some cases a scenario like this
(where a LOT of memory is being used) can occur. In that case
I would prefer it if the application died with an "Out of memory"
error as opposed to bringing the system to a standstill.

It seems like one strategy would be that keepomg the "max-heap"
setting slightly under half the available physical memory should
avoid the case where we're already using about 50% and then have
to create another heap of the same size.

Another observation I've made is that using "fixed-heap" instead
of "max-heap" seems to avoid the thrashing (unless I set the
size unreasonably high, like 5GB on my 6GB machine running a
number of other applications). I had somewhat expected for MLton
to immediately allocate the whole heap but (according to the
task manager) that did not happen. Are there any other drawbacks
about using fixed-heap?

Do you have any other suggestions?

Thanks and best regards,

David

Matthew Fluet wrote:

>
> The code below is an example that is supposed to allocate more
> and more memory until it is terminated. While it is running, it
> outputs roughly how much memory it has allocated so far.
>
>
> You could run with "@MLton gc-messages --" to see the actual heap sizes
> as the program runs.
>
>
> If I run the example with "@MLton max-heap 2950M --" run-time
> parameters then it keeps allocating memory, the numbers printed
> by the example being roughly in sync with what Windows' Task
> Manager shows. Around 3GB it is terminated with the message
> "Out of memory with max heap size 3,093,299,200",
> which seems like exactly what should happen.
>
>
> You can use "@MLton max-heap 2.95G --" and "@MLton max-heap 3G --" as
> shorter forms to express gigabyte sized heaps. That is, the
> {max,fixed}-heap options accept a floating-point number (to express
> fractional portions of a size) and the "G"/"g" size modifier.
>
>
> However, if I run the example with "@MLton max-heap 3000M --"
> then around the the same point where it was was stopped before
> (the program reports roughly 3.05GB of allocated memory), my
> system starts thrashing and I can see in the Task Manager
> that the program has allocated around 4.5GB of memory and is still
> trying to get more.
>
> Is this what you would expect? I thought that the max-heap setting
> would include the amount of additional memory required for garbage
> collection. Is that true? Or do I always have to just allow
> half of the system memory for max-heap to make sure to not allocate
> too much memory?
>
>
> You should run your program with "@MLton gc-messages --" to be sure, but
> I believe that the behavior you are seeing is explained by the heap
> resizing policy. There is very little space overhead in the garbage
> collector and runtime over that consumed by the ML heap. So, the
> max-heap option only bounds the size of the ML heap (which includes the
> card/cross map used for generational garbage collection). MLton will
> never attempt to create a heap that is larger than the max-heap option.
>
> However, MLton starts execution with a small heap and allows it to grow
> in response to the demands of the live data. After each garbage
> collection, MLton decides whether to grow or shrink the heap in order to
> have the heap size approximately 8x the live data at the end of the
> garbage collection, further modified by the constraints of the available
> ram, max-heap, etc. MLton will never try to *allocate* a heap that is
> larger than the max-heap option; however, in order to *obtain* a larger
> heap than the currently allocated heap, MLton may be required to
> allocate the new (bigger) heap and copy the data from the current heap
> over. The idea is that the user would like their program to continue
> running; suffering a little bit of paging during the time that the two
> heaps are allocated is preferable to aborting with an out-of-memory
> error. After the copy, the old heap is discarded and total memory usage
> is again bounded by max-heap. (Note: This is *not* the policy for the
> copying collection; that is, during a copying collection, we do not have
> two heaps of max-heap size. Instead, the semi-space size is fixed so as
> to be 1/2 of max-heap; or, if the live data is more than 1/2 the
> max-heap, the runtime will use the mark-compact collection.)
>
> Obtaining the larger heap from the old heap actually happens in two
> ways. For platforms that support it, we first attempt to mremap the
> existing heap to a larger size. If we are able to expand to at least
> half-way between the current size and the desired size, we take that as
> acceptable growth, even if it isn't quite the full desired heap size.
> If we can't expand, or the platform doesn't support mremap, then we
> attempt to create a heap of the desired size. This, clearly, results in
> a second heap that is allocated while the existing heap is still
> allocated. On a 32-bit system, allocating a second 3G+ heap while one
> heap of significant size is allocated will fail due to the virtual
> address space limitations; in that situation, and if the program is
> running with @MLton may-page-heap true --, MLton will attempt to write
> out the existing heap to a file, deallocate the existing heap, and now
> allocate the large heap. We hope that by deallocating the existing
> heap, there is now sufficient contiguous address space to create the
> heap of sufficient size. On a 64-bit system, there shouldn't ever be
> any trouble finding another 3G+ portion of the virtual address space
> while the existing heap is allocated. I suspect that this is what you
> are seeing.
>
> Note, that while Windows doesn't natively support mremap, Wesley
> implemented a few things that try to mimic mmap/mremap/munamp
> functionality under Windows. There seems to be some complication with
> the Windows memory system that requires committing in addition to
> reserving memory, and some manner of being able to extend an existing
> map with space before it and after it. I don't understand it. However,
> I do note that the generic version of mremap
> (<src>/runtime/platform/mremap.c), which is used by the Windows
> platforms, starts off trying to allocate an entirely new region of the
> desired size and copy the existing region into it. (This actually seems
> to be redundant with the behavior of the garbage collector proper.) The
> comment states: "Prefer a moving remap -> it results in less mmapped
> regions". Since such an allocation is likely to never fail on a 64-bit
> platform, you will (briefly) have multiple regions allocated at the same
> time, likely to exceed the max-heap setting. Of course, that
> "(briefly)" is the time it takes to allocate a 3G region and copy the
> existing (nearly) 3G region; if you have <6G physical memory, there will
> be paging.
>
> As for why things happen differently with the different max-heap
> settings, I think that you just experience a slightly different heap
> growth trajectory. This is a simple example, but suppose I compare
> running with fixed-heap 31M and fixed-heap 33M. And suppose a very
> simple heap-size doubling policy, starting with a heap of size 2M. So
> the first run will have heap sizes: 2M, 4M, 8M, 16M, 31M. In order to
> transition from 16M to 31M, you might required 46M memory (the old and
> new heaps both allocated at the same time), but if you have enough RAM,
> you probably won't notice it, because it isn't too much worse than your
> desired max-heap. The second run will have heap sizes: 2M, 4M, 8M, 16M,
> 32M, 33M. Now, to transition from 32M to 33M, you require 65M memory,
> and that probably becomes noticeable, since it significantly exceeds
> your desired max-heap.

--
----------------------------------------------------------
David Hansel
http://www.reactive-systems.com/
OpenPGP (GnuPG) public key file:
http://www.reactive-systems.com/~hansel/pgp_public_key.txt
----------------------------------------------------------

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Versioning and hosting

2009-12-11T07:42:43Z

I received a FTBFS (fails to build from source) debian bug #560623 [1] today, which turns out to be a patch compatibility bug already fixed by Matthew (r7369). (BTW, what the hell?! patch is as old and stable a unix tool as they come; what genius changed its behaviour?)

In the process of fixing this I noticed that svn browsing is broken [2]. Also the MLton server still runs woody and hasn't seen a security update in years. Not to mention downloads and svn from it are extremely slow. If there's interest, we could definitely offer some improved hosting in Germany.

Before I upload a new version of the debian package and go through the hassle of getting all the ports to build cleanly again (they are currenly all green! [3]), I thought I'd also address bug #559014 [4], which is a complaint about the version number I picked for the MLton snapshot; I called it 20091107.

Since MLton doesn't use version numbers, I can't really call it 1.99+r7369 to make it clear it's a prerelease snapshot of version 2.0. However, my current version name is a bad choice because it implies that 20091107 was an "official" release. The best I can propose is 20091211+r7369. Suggestions / complaints?

Of course, if those two regressions on osx were fixed, maybe we'd have an official release and the point would be moot. BTW, a colleague of mine reports everything works under Snow Leopard except the regressions world5 and real. A small fix was needed which I'll commit shortly.

[1] <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560623>
[2] <http://mlton.org/cgi-bin/viewsvn.cgi/mlton/trunk/?rev=HEAD>
[3] <https://buildd.debian.org/~luk/status/package.php?p=mlton>
[4] <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=559014>

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-10T16:08:15Z

On Thu, Dec 10, 2009 at 9:37 PM, Matthew Fluet <matthew.fluet@...> wrote:

Note, that while Windows doesn't natively support mremap, Wesley implemented a few things that try to mimic mmap/mremap/munamp functionality under Windows. There seems to be some complication with the Windows memory system that requires committing in addition to reserving memory, and some manner of being able to extend an existing map with space before it and after it. I don't understand it.

When you reserve memory, you get virtual address space that does not overlap any previous reservations. When you commit (part of) that reservation, you get memory that can actually be used without segfaulting.

You can freely commit/decommit memory within a reservation, changing the memory that windows must provide to the application. You >cannot< shrink or grow a reservation.

While it is possible to reserve+commit in one step, this doesn't ensure that the reservation is exclusive with any existing reservations. This is what caused the sporadic windows GC segfaults some years ago, heaps that overlapped.

With respect to mremap, my main concern was supporting heap growth in a restricted virtual memory scenario. eg: you have 900MB allocated and need to grow to 1.2G. This matters when self-compiling MLton on mingw32, for example. To support this, it was necessary to allow a heap to grow 'in-place'. That's why the mremap implementation starts probing for available memory before and after the existing heap.

Supposing there is available memory before/after the existing heap, I make new reservations to expand the heap. The windows.c file describes the invariant I used for the memory layout. The invariant I choose makes it safe to free/shrink/grow and be certain the memory was actually allocated to >this< heap.

However, I do note that the generic version of mremap (<src>/runtime/platform/mremap.c), which is used by the Windows platforms, starts off trying to allocate an entirely new region of the desired size and copy the existing region into it.

We could revisit this approach in the 64-bit setting. The msdn documentation makes vague warnings about the cost of making too many reservations, but I doubt this really matters. From the point of view of a system with a lot of virtual address space, but not a lot of physical RAM, it might make sense to try to always expand in-place first.

Of course, even if mremap tries in-place heap growth first, this might not be possible, and a copy to a new memory location will still thrash.

(This actually seems to be redundant with the behavior of the garbage collector proper.)

Perhaps, but I was trying to implement an mremap compatible to the specification of mremap, not how MLton uses mremap. If mremap fails to aquire the desired space, mmap will certainly also fail.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: max-heap setting for 64-bit applications

2009-12-10T12:37:38Z

The code below is an example that is supposed to allocate more
and more memory until it is terminated. While it is running, it
outputs roughly how much memory it has allocated so far.

You could run with "@MLton gc-messages --" to see the actual heap sizes as the program runs.

If I run the example with "@MLton max-heap 2950M --" run-time
parameters then it keeps allocating memory, the numbers printed
by the example being roughly in sync with what Windows' Task
Manager shows. Around 3GB it is terminated with the message
"Out of memory with max heap size 3,093,299,200",
which seems like exactly what should happen.

You can use "@MLton max-heap 2.95G --" and "@MLton max-heap 3G --" as shorter forms to express gigabyte sized heaps. That is, the {max,fixed}-heap options accept a floating-point number (to express fractional portions of a size) and the "G"/"g" size modifier.

However, if I run the example with "@MLton max-heap 3000M --"
then around the the same point where it was was stopped before
(the program reports roughly 3.05GB of allocated memory), my
system starts thrashing and I can see in the Task Manager
that the program has allocated around 4.5GB of memory and is still
trying to get more.

Is this what you would expect? I thought that the max-heap setting
would include the amount of additional memory required for garbage
collection. Is that true? Or do I always have to just allow
half of the system memory for max-heap to make sure to not allocate
too much memory?

You should run your program with "@MLton gc-messages --" to be sure, but I believe that the behavior you are seeing is explained by the heap resizing policy. There is very little space overhead in the garbage collector and runtime over that consumed by the ML heap. So, the max-heap option only bounds the size of the ML heap (which includes the card/cross map used for generational garbage collection). MLton will never attempt to create a heap that is larger than the max-heap option.

However, MLton starts execution with a small heap and allows it to grow in response to the demands of the live data. After each garbage collection, MLton decides whether to grow or shrink the heap in order to have the heap size approximately 8x the live data at the end of the garbage collection, further modified by the constraints of the available ram, max-heap, etc. MLton will never try to *allocate* a heap that is larger than the max-heap option; however, in order to *obtain* a larger heap than the currently allocated heap, MLton may be required to allocate the new (bigger) heap and copy the data from the current heap over. The idea is that the user would like their program to continue running; suffering a little bit of paging during the time that the two heaps are allocated is preferable to aborting with an out-of-memory error. After the copy, the old heap is discarded and total memory usage is again bounded by max-heap. (Note: This is *not* the policy for the copying collection; that is, during a copying collection, we do not have two heaps of max-heap size. Instead, the semi-space size is fixed so as to be 1/2 of max-heap; or, if the live data is more than 1/2 the max-heap, the runtime will use the mark-compact collection.)

Obtaining the larger heap from the old heap actually happens in two ways. For platforms that support it, we first attempt to mremap the existing heap to a larger size. If we are able to expand to at least half-way between the current size and the desired size, we take that as acceptable growth, even if it isn't quite the full desired heap size. If we can't expand, or the platform doesn't support mremap, then we attempt to create a heap of the desired size. This, clearly, results in a second heap that is allocated while the existing heap is still allocated. On a 32-bit system, allocating a second 3G+ heap while one heap of significant size is allocated will fail due to the virtual address space limitations; in that situation, and if the program is running with @MLton may-page-heap true --, MLton will attempt to write out the existing heap to a file, deallocate the existing heap, and now allocate the large heap. We hope that by deallocating the existing heap, there is now sufficient contiguous address space to create the heap of sufficient size. On a 64-bit system, there shouldn't ever be any trouble finding another 3G+ portion of the virtual address space while the existing heap is allocated. I suspect that this is what you are seeing.

Note, that while Windows doesn't natively support mremap, Wesley implemented a few things that try to mimic mmap/mremap/munamp functionality under Windows. There seems to be some complication with the Windows memory system that requires committing in addition to reserving memory, and some manner of being able to extend an existing map with space before it and after it. I don't understand it. However, I do note that the generic version of mremap (<src>/runtime/platform/mremap.c), which is used by the Windows platforms, starts off trying to allocate an entirely new region of the desired size and copy the existing region into it. (This actually seems to be redundant with the behavior of the garbage collector proper.) The comment states: "Prefer a moving remap -> it results in less mmapped regions". Since such an allocation is likely to never fail on a 64-bit platform, you will (briefly) have multiple regions allocated at the same time, likely to exceed the max-heap setting. Of course, that "(briefly)" is the time it takes to allocate a 3G region and copy the existing (nearly) 3G region; if you have <6G physical memory, there will be paging.

As for why things happen differently with the different max-heap settings, I think that you just experience a slightly different heap growth trajectory. This is a simple example, but suppose I compare running with fixed-heap 31M and fixed-heap 33M. And suppose a very simple heap-size doubling policy, starting with a heap of size 2M. So the first run will have heap sizes: 2M, 4M, 8M, 16M, 31M. In order to transition from 16M to 31M, you might required 46M memory (the old and new heaps both allocated at the same time), but if you have enough RAM, you probably won't notice it, because it isn't too much worse than your desired max-heap. The second run will have heap sizes: 2M, 4M, 8M, 16M, 32M, 33M. Now, to transition from 32M to 33M, you require 65M memory, and that probably becomes noticeable, since it significantly exceeds your desired max-heap.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

max-heap setting for 64-bit applications

2009-12-09T09:05:02Z

Hi all,

I have a few questions about the max-heap setting for 64-bit
applications built by MLton.

The code below is an example that is supposed to allocate more
and more memory until it is terminated. While it is running, it
outputs roughly how much memory it has allocated so far.

I compiled the example with MLton (r7353 or r7369, no difference)
on 64-bit Windows 7 as a 64-bit application. The computer has
6GB of RAM, of which 3.5-4.0GB are available to the application.

If I run the example with "@MLton max-heap 2950M --" run-time
parameters then it keeps allocating memory, the numbers printed
by the example being roughly in sync with what Windows' Task
Manager shows. Around 3GB it is terminated with the message
"Out of memory with max heap size 3,093,299,200",
which seems like exactly what should happen.

However, if I run the example with "@MLton max-heap 3000M --"
then around the the same point where it was was stopped before
(the program reports roughly 3.05GB of allocated memory), my
system starts thrashing and I can see in the Task Manager
that the program has allocated around 4.5GB of memory and is still
trying to get more.

Is this what you would expect? I thought that the max-heap setting
would include the amount of additional memory required for garbage
collection. Is that true? Or do I always have to just allow
half of the system memory for max-heap to make sure to not allocate
too much memory?

Thanks,

David

Example code:

fun mk10MBArray () =
let
val n = 2500000
val a = Array.array (n, 0)
fun loop i =
if i >= n then
()
else
(Array.update (a, i, i);
loop (i+1))
in
loop 0;
a
end

fun processArray (a, x : IntInf.int) =
let
val n = Array.length a
fun loop (i, x) =
if i >= n then
x
else
loop (i+10000, x + (IntInf.fromInt (Array.sub (a, i))))
in
loop (0, x)
end

fun processMem mls =
let
val n = length mls
val x = IntInf.fromInt 0
fun loop (i, x) =
if i >= n then
x
else
let
val a = List.nth (mls, i)
val x = processArray (a, x)
in
loop (i+1, x)
end
in
loop (0, IntInf.fromInt 0)
end

fun allocUntilDeath () =
let
val timer = Timer.startRealTimer ()
fun loop (mls, i) =
let
val t = Timer.checkRealTimer timer
val mls = (mk10MBArray ())::mls
in
processMem mls;
print (Time.toString t);
print ": ";
print (Real.toString i);
print " GB\n";
loop (mls, i+0.01)
end
in
loop ([], 0.0)
end

val () = (allocUntilDeath ())

--
----------------------------------------------------------
David Hansel
http://www.reactive-systems.com/
OpenPGP (GnuPG) public key file:
http://www.reactive-systems.com/~hansel/pgp_public_key.txt
----------------------------------------------------------

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

Re: feature request: hash function for Pointer.t

2009-12-06T14:29:07Z

On Sun, Dec 6, 2009 at 12:31 PM, John Reppy <jhr@...> wrote:

Because MLton doesn't provide a way to dynamically define callbacks, I need to manage
the a mapping from C pointers to ML functions. I'd like to use a hash table to do
this, but there isn't an easy way to compute hash keys from pointers. I'd like
to see a function

val hash : t -> word

added to the MLTON_POINTER signature.

If you believe that the C address is unique enough to be a hash key, doesn't
val hash = fn p => MLton.Pointer.diff(MLton.Pointer.null, p)
suffice as a hashing function? In any case, I think any hashing of a pointer would start with its address, so you can do a mixing function from there if you don't think the distribution of C addresses suffice as a hashing key. Admittedly, this is in terms of Word.word, which defaults to a 32-bit value; MLton.Pointer.diff performs the difference computation at the target pointer size, and then does a low-bits projection to a 32-bit word. Still, that ought to be decent for a hash function. Perhaps all of the MLton.Pointer functions should be in terms of LargeWord.word.

Also,
val MLton.hash : 'a -> Word32.word
is a polymorphic hash function that works with any ML value.

As a more long-term request, it would be nice to have something like the GHC notion
of stable pointers that would allow ML values to be passed to C code for later supply
as arguments to callbacks.

Yes, some notion of registering an exported ML value would be desirable.

-Matthew

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev

feature request: hash function for Pointer.t

2009-12-06T09:31:22Z

Because MLton doesn't provide a way to dynamically define callbacks, I need to manage
the a mapping from C pointers to ML functions. I'd like to use a hash table to do
this, but there isn't an easy way to compute hash keys from pointers. I'd like
to see a function

val hash : t -> word

added to the MLTON_POINTER signature.

As a more long-term request, it would be nice to have something like the GHC notion
of stable pointers that would allow ML values to be passed to C code for later supply
as arguments to callbacks.

- John

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

From forum: MLton - Dev