Possible bug in MLton-7230.msi for MinGW and 64 bit questions

[David Hansel]

Hi all,

We recently tried compiling our code with the MLton-7230 MinGW experimental
release from your web site.  Our code had been working fine when compiled
using the "MLton MLTONVERSION (built Tue Feb 03 16:33:30 2009 on orange)"
build.  It compiled fine with the 7230 release but the generated executable
always crashed immediately after starting.  We found that the crashes only
occurred when using the FFI interface and that they always happened during calls to "dlopen".
(line 142 in MLton\lib\mlton\sml\mlnlffi-lib\memory\linkage-dlopen.sml)
After some more digging we found a change between the two versions regarding the
RTLD_* constants used in file
MLton\lib\mlton\sml\mlnlffi-lib\memory\linkage-libdl.sml
For the older version,  the constants defined in that file were
   val RTLD_LAZY   = 0wx00001
   val RTLD_NOW    = 0wx00002
   val RTLD_GLOBAL = 0wx00100
   val RTLD_LOCAL  = 0wx00000
whereas with the 7230 version,  the constants from
MLton\lib\mlton\sml\mlnlffi-lib\memory\platform\rtld-flags.mingw.sml
are used which are defined as
   val RTLD_GLOBAL = 0wx2
   val RTLD_LAZY   = 0wx0
   val RTLD_LOCAL  = 0wx4
   val RTLD_NOW    = 0wx0

After changing the constants in rtld-flags.mingw.sml to the values from the previous
version,  the crashes went away.  Our code calls DynLinkage.open_lib with
parameters "global=true" and "lazy=true",  so the values of the constants
definitely come into play.  Should we be using different parameters or is this
a bug in MLton?

Also,  from the web site it sounds like the MLton-7230.msi version supports
generating Windows 64 bit executables,  is that true?  If so,  how do we tell
MLton to generate a 64 bit executable?  If not,  can you tell us what the
current state of Windows 64 bit support for MLton is?

Thank you very much and best regards,

David

--
  ----------------------------------------------------------
  David Hansel
  http://www.reactive-systems.com/
  OpenPGP (GnuPG) public key file:
  http://www.reactive-systems.com/~hansel/pgp_public_key.txt
  ----------------------------------------------------------

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Wesley W. Terpstra]

On Sun, Nov 1, 2009 at 3:43 AM, David Hansel
<hansel@...> wrote:
> After changing the constants in rtld-flags.mingw.sml to the values from the previous
> version,  the crashes went away.

This sounds to be a bug in the libdl we are getting from google. I was
planning to build a new msi this weekend anyway so I'll take a look
tomorrow.

> Also,  from the web site it sounds like the MLton-7230.msi version supports
> generating Windows 64 bit executables,  is that true?  If so,  how do we tell
> MLton to generate a 64 bit executable?

mlton -target x86_64-w64-mingw32 your-mlb-file-here.mlb

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Wesley W. Terpstra]

On Sun, Nov 1, 2009 at 3:43 AM, David Hansel
<hansel@...> wrote:
> whereas with the 7230 version,  the constants from
> MLton\lib\mlton\sml\mlnlffi-lib\memory\platform\rtld-flags.mingw.sml
> are used which are defined as
>   val RTLD_GLOBAL = 0wx2
>   val RTLD_LAZY   = 0wx0
>   val RTLD_LOCAL  = 0wx4
>   val RTLD_NOW    = 0wx0

These match the dlfcn we got from
http://code.google.com/p/dlfcn-win32/downloads/list
So the newer version of MLton is correct.

Inspecting the code for dlfcn reveals that only the RTLD_GLOBAL flag
has any effect at all. All other flags are ignored on windows.

With the old (incorrect flags) "global=true,lazy=true" -> 0x101, which
means: do nothing. That's because the actual flag for global is 0x2.
The new MLton gets it correct and uses 0x2 for the flags you provided.

The end result of this is that the new MLton is actually honouring the
global=true option whereas the old version was silently ignoring it.

> After changing the constants in rtld-flags.mingw.sml to the values from the previous
> version,  the crashes went away.  Our code calls DynLinkage.open_lib with
> parameters "global=true" and "lazy=true",  so the values of the constants
> definitely come into play.

Why are you setting "global=true"? Do you load subsequent libraries
that depend on earlier loaded libraries?

> Should we be using different parameters or is this a bug in MLton?

A mixture of both. The *older* MLton had a bug, which is resolved in
the newer MLton builds. To reproduce the behaviour you had before,
switch to "global=false".

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[David Hansel]

Hi Wesley,

> [...]
> Why are you setting "global=true"? Do you load subsequent libraries
> that depend on earlier loaded libraries?

This part of our code is very old and I do not know why the setting was
chosen.  Changing the calls to "global=false" does indeed fix the
crashes with the original constants in place.  Thank you very much for
your help.

> [...]
>> Also,  from the web site it sounds like the MLton-7230.msi version supports
>> generating Windows 64 bit executables,  is that true?  If so,  how do we tell
>> MLton to generate a 64 bit executable?
>
> mlton -target x86_64-w64-mingw32 your-mlb-file-here.mlb

Calling mlton -target x86_64-w64-mingw32 -verbose 2 test.sml
for a "Hello World" style test.sml gives me the following error:

openIn c:\program files (x86)\mlton\lib\mlton\x86_64-w64-mingw32\sizes:
The system cannot find the file specified.

Do I need some additional files or settings?

One other question:  Are FFI calls supported in the Win64 executables
produced by MLton?  Are there any special considerations regarding
64 bit FFI?

Thanks again,

David

--
  ----------------------------------------------------------
  David Hansel
  http://www.reactive-systems.com/
  OpenPGP (GnuPG) public key file:
  http://www.reactive-systems.com/~hansel/pgp_public_key.txt
  ----------------------------------------------------------



_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Wesley W. Terpstra]

On Mon, Nov 2, 2009 at 2:44 AM, David Hansel
<hansel@...> wrote:

>>> Also,  from the web site it sounds like the MLton-7230.msi version supports
>>> generating Windows 64 bit executables,  is that true?  If so,  how do we tell
>>> MLton to generate a 64 bit executable?
>>
>> mlton -target x86_64-w64-mingw32 your-mlb-file-here.mlb
>
> Calling mlton -target x86_64-w64-mingw32 -verbose 2 test.sml
> for a "Hello World" style test.sml gives me the following error:
>
> openIn c:\program files (x86)\mlton\lib\mlton\x86_64-w64-mingw32\sizes:
> The system cannot find the file specified.

You need to install the 64-bit pieces from the msi. When it asks you
complete/typical/custom, select complete. It sounds like you selected
typical which does not install the 64-bit toolchain.

> One other question:  Are FFI calls supported in the Win64 executables
> produced by MLton?

Yes

> Are there any special considerations regarding 64 bit FFI?

No.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[David Hansel]

Hi again,

Sorry for the trivial questions so far.  Is there any 64 bit specific
documentation that I am missing?

Regarding the 64 bit FFI support,  I am currently getting a linker error
whenever I attempt to use FFI functions.  The error is:

c:/program files (x86)/mlton/bin/../lib/gcc/x86_64-w64-mingw32/4.4.1/../../../..
/x86_64-w64-mingw32/bin/ld.exe: cannot find -ldl
(this is of course caused by the "-link-opt -ldl" parameter to MLton
but if I leave that out then I get missing symbols).

I did install the full 64 bit support.  Is there anything else
(more or less obvious) that I should have done?

Thanks,

David


On 11/1/2009 10:35 PM, Wesley W. Terpstra wrote:

> On Mon, Nov 2, 2009 at 2:44 AM, David Hansel
> <hansel@...> wrote:
>>>> Also,  from the web site it sounds like the MLton-7230.msi version supports
>>>> generating Windows 64 bit executables,  is that true?  If so,  how do we tell
>>>> MLton to generate a 64 bit executable?
>>> mlton -target x86_64-w64-mingw32 your-mlb-file-here.mlb
>> Calling mlton -target x86_64-w64-mingw32 -verbose 2 test.sml
>> for a "Hello World" style test.sml gives me the following error:
>>
>> openIn c:\program files (x86)\mlton\lib\mlton\x86_64-w64-mingw32\sizes:
>> The system cannot find the file specified.
>
> You need to install the 64-bit pieces from the msi. When it asks you
> complete/typical/custom, select complete. It sounds like you selected
> typical which does not install the 64-bit toolchain.
>
>> One other question:  Are FFI calls supported in the Win64 executables
>> produced by MLton?
>
> Yes
>
>>  Are there any special considerations regarding 64 bit FFI?
>
> No.

--
  ----------------------------------------------------------
  David Hansel
  Chief Technology Officer -- Reactive Systems, Inc.
  http://www.reactive-systems.com/
  [phone] ext. 102 -- hansel@...
  OpenPGP (GnuPG) public key file:
  http://www.reactive-systems.com/~hansel/pgp_public_key.txt
  ----------------------------------------------------------

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Wesley W. Terpstra]

On Mon, Nov 2, 2009 at 5:02 AM, David Hansel
<hansel@...> wrote:
> Sorry for the trivial questions so far.  Is there any 64 bit specific
> documentation that I am missing?

There is no 64-bit specific documentation. It's exactly the same as 32
bit, though.

> c:/program files (x86)/mlton/bin/../lib/gcc/x86_64-w64-mingw32/4.4.1/../../../..
> /x86_64-w64-mingw32/bin/ld.exe: cannot find -ldl
> (this is of course caused by the "-link-opt -ldl" parameter to MLton
> but if I leave that out then I get missing symbols).

The problem is that the dlfcn library from google is a 32-bit library.
You need a 64-bit library. This is a general problem; all libraries
you use must also be 64-bit.

Try the newest msi from <http://mlton.org/Experimental>. I included a
64-bit dlfcn in that build. It might not work, though. Please report
back if it does (or does not) work for you.

However, any other libraries you use you will have to find 64-bit
versions of yourself.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[David Hansel]

Hi again

On 11/1/2009 11:09 PM, Wesley W. Terpstra wrote:

> On Mon, Nov 2, 2009 at 5:02 AM, David Hansel
> <hansel@...> wrote:
>> Sorry for the trivial questions so far.  Is there any 64 bit specific
>> documentation that I am missing?
>
> There is no 64-bit specific documentation. It's exactly the same as 32
> bit, though.
>
>> c:/program files (x86)/mlton/bin/../lib/gcc/x86_64-w64-mingw32/4.4.1/../../../..
>> /x86_64-w64-mingw32/bin/ld.exe: cannot find -ldl
>> (this is of course caused by the "-link-opt -ldl" parameter to MLton
>> but if I leave that out then I get missing symbols).
>
> The problem is that the dlfcn library from google is a 32-bit library.
> You need a 64-bit library. This is a general problem; all libraries
> you use must also be 64-bit.
>
> Try the newest msi from <http://mlton.org/Experimental>. I included a
> 64-bit dlfcn in that build. It might not work, though. Please report
> back if it does (or does not) work for you.
>
> However, any other libraries you use you will have to find 64-bit
> versions of yourself.

Thanks for providing the 64-bit dlfcn.  We are aware of the 32-bit vs.
64-bit library incompatibilities (in fact the need to work together
with a 64-bit 3rd party library is partially driving our 64-bit
development).

We were able to compile our code as a 64-bit application and some
initial testing shows the FFI calls appear to work fine too (no
problems with the 64-bit dlfcn so far).

While most of our ML code appears to work fine,  we do have a
problem with one (rather large) module.  When compiled using the
native code-generator, the resulting executable will crash on startup.
However,  when compiled with the "C" code-generator the executable
works without problems.  The other parts of our code work fine for
both native and "C" code-generators.  We will try to break up the
module to see if we can narrow down the offending code.  Do you have
any other suggestions on how to debug a problem like this?  Are there
any (rough) estimates about the runtime performance penalty for
using the C code generator?

Thanks again,

David

--
  ----------------------------------------------------------
  David Hansel
  http://www.reactive-systems.com/
  OpenPGP (GnuPG) public key file:
  http://www.reactive-systems.com/~hansel/pgp_public_key.txt
  ----------------------------------------------------------

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Wesley W. Terpstra]

On Tue, Nov 3, 2009 at 3:00 AM, David Hansel
<hansel@...> wrote:
> While most of our ML code appears to work fine,  we do have a
> problem with one (rather large) module.  When compiled using the
> native code-generator, the resulting executable will crash on startup.

That could be caused by a lot of things.

> However,  when compiled with the "C" code-generator the executable
> works without problems.  The other parts of our code work fine for
> both native and "C" code-generators.  We will try to break up the
> module to see if we can narrow down the offending code.

If you narrow it down to something that looks like a MLton bug, please
post a test-case demonstrating the problem. Win64 does have a
"special" ABI, so it is possible there is an incorrect corner case in
MLton.

> Do you have any other suggestions on how to debug a problem like this?

Supposing it's not MLton, I'd look closely at your C types. The sizes
of of variables in C may have changed in a way that causes them to no
longer match up to the 32-bit assumptions made on the SML side of your
code.

> Are there any (rough) estimates about the runtime performance penalty for
> using the C code generator?

This depends heavily on the application. Sometimes the C-codegen can
even be faster. Sometimes it can be much slower.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton