Could FFI support pass-by-value of structs?

[Maurí­cio]

It's not usual, but it is allowed to have values of
structs passed between functions directly instead of
using pointers:

/*****/
struct ex {
     int x;
     int y;
     int z;
};

ex example_functions (ex p)
{
   (...)
}
/*****/

Would it be possible to allow that in Haskell FFI
by, say, allowing any instance of Storable to be
used in a 'foreign' declaration? Like:

--
data Ex = (...)

instance Storable Ex where
   sizeOf _ = ...
   alignment = sizeOf
   (...)

foreign import ccall "example_functions" exampleFunction
   :: Ex -> IO Ex
--

Thanks,
Maurício

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[John Meacham]

On Tue, Jun 16, 2009 at 10:48:19AM -0300, Maurí­cio wrote:

> It's not usual, but it is allowed to have values of
> structs passed between functions directly instead of
> using pointers:
>
> /*****/
> struct ex {
>     int x;
>     int y;
>     int z;
> };
>
> ex example_functions (ex p)
> {
>   (...)
> }
> /*****/
>
> Would it be possible to allow that in Haskell FFI
> by, say, allowing any instance of Storable to be
> used in a 'foreign' declaration? Like:

There are a couple problems with this. First, the storage layout for a
given C struct may be radically different depending on the back end,
even sometimes depending on the operating system. So you can't write a
portable Storable instance.

The other problem is that the storage layout isn't enough to use structs
as arguments as the calling conventions also come into play. Some
calling conventions want structs expanded and places on the stack,
others want a pointer to some stack allocated data, returning a struct
may or may not create a hidden first argument. Now, Haskell compilers
already have to tackle calling conventions for basic types, so it is
entirely plausable to solve this, the question is, is it worth the
effort? anything you write will end up not being portable anyway because
it depends on the structure layout.

The end result being, structure passing is complicated, so if you need
to interface to a library that has pre-defined structeres, use hsc2hs to
manually peek and poke into the correct offsets, it will make things
easier and be more portable.

        John

--
John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Felipe Lessa]

On Tue, Jun 16, 2009 at 10:48:19AM -0300, Maurí­cio wrote:

> /*****/
> struct ex {
>     int x;
>     int y;
>     int z;
> };
>
> ex example_functions (ex p)
> {
>   (...)
> }
> /*****/

You may adopt the approach I used with Hipmunk[1] where there is
a wrapper library written in C.  For your example you could do
something like

> void wr_example_functions(ex *p) {
>   *p = example_functions(*p);
> }

and in you Haskell code you write something like

> foreign import ccall unsafe "wrapper.h"
>     wr_example_functions :: Ptr Ex -> IO ()
>
> exampleFunctions :: Ex -> IO Ex
> exampleFunctions input =
>   with input $ \ptr -> do
>     wr_example_functions ptr
>     peek ptr

The structure is allocated on the stack, so the drawbacks are
only another function call and two structure copies, and probably
that shouldn't hurt a lot.  Note that depending on the C compiler
those costs may even get somewhat optimized (e.g. by inlining).

[1] http://hackage.haskell.org/package/Hipmunk

--
Felipe.
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Maurí­cio]

>> It's not usual, but it is allowed to have values of
>> structs passed between functions directly instead of
>> using pointers: (...)

>> Would it be possible to allow that in Haskell FFI (...)

> There are a couple problems with this. First, the storage layout for a
> given C struct may be radically different depending on the back end,

When you say struct layout, do you mean the offset of diferent
fields? I don't think this would be a problem, as your sugestion
of hsc2hs is one of many solutions to that. I'm not sugesting
that peek and poke methods of Storage instances should be created
automatically, I understand this would not worth the effort.

However, isn't just knowing the size and alignment enough to
write a generic struct handler that, by using the appropriate
calling convention, is going to work with any struct? If not,
I agree with you it's really not worth it (as we can use pointers
as Felipe sugested.

Maurício

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[John Meacham]

On Wed, Jun 24, 2009 at 10:23:29AM -0300, Maurí­cio wrote:
> However, isn't just knowing the size and alignment enough to
> write a generic struct handler that, by using the appropriate
> calling convention, is going to work with any struct? If not,
> I agree with you it's really not worth it (as we can use pointers
> as Felipe sugested.

No, unfortunately it is not. Depending on the actual data types inside
the struct, values may be passed in registers, the stack, or split
between the two. for instance, the floating point values in a given
struct may be passed in SSE registers on x86-64, while the integral
parts are passed in normal registers.

        John

--
John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Henning Thielemann]

Maurí­cio schrieb:

>>> It's not usual, but it is allowed to have values of
>>> structs passed between functions directly instead of
>>> using pointers: (...)
>
>>> Would it be possible to allow that in Haskell FFI (...)
>
>> There are a couple problems with this. First, the storage layout for a
>> given C struct may be radically different depending on the back end,
>
> When you say struct layout, do you mean the offset of diferent
> fields? I don't think this would be a problem, as your sugestion
> of hsc2hs is one of many solutions to that. I'm not sugesting
> that peek and poke methods of Storage instances should be created
> automatically, I understand this would not worth the effort.
>
> However, isn't just knowing the size and alignment enough to
> write a generic struct handler that, by using the appropriate
> calling convention, is going to work with any struct? If not,
> I agree with you it's really not worth it (as we can use pointers
> as Felipe sugested.

I tried that in
  http://hackage.haskell.org/package/storable-record/

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Duncan Coutts]

On Wed, 2009-06-24 at 18:48 -0700, John Meacham wrote:

> On Wed, Jun 24, 2009 at 10:23:29AM -0300, Maurí­cio wrote:
> > However, isn't just knowing the size and alignment enough to
> > write a generic struct handler that, by using the appropriate
> > calling convention, is going to work with any struct? If not,
> > I agree with you it's really not worth it (as we can use pointers
> > as Felipe sugested.
>
> No, unfortunately it is not. Depending on the actual data types inside
> the struct, values may be passed in registers, the stack, or split
> between the two. for instance, the floating point values in a given
> struct may be passed in SSE registers on x86-64, while the integral
> parts are passed in normal registers.

A better approach might be to not use a custom user-defined type with
its Storable instance and just use a tuple of FFI-types.

Using Maurí­cio's example:

/*****/
struct ex {
     int x;
     int y;
     int z;
};

ex example_functions (ex p)
{
   (...)
}
/*****/

foreign import ccall "example_functions" exampleFunction
   :: (CInt, CInt, CInt) -> IO (CInt, CInt, CInt)


So no generic user mechanism, but extend the FFI-marshalable types to
include tuples of FFI-types and to pass them using the C struct ABI.
I've no idea what happens when using C pack pragmas etc.

Duncan

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Bulat Ziganshin-2]

Hello Duncan,

Tuesday, June 30, 2009, 12:03:15 AM, you wrote:

> struct ex {
>      int x;
>      int y;
>      int z;
> };

> ex example_functions (ex p)

afaik, there is C ABI, that defines how to pass and return parameters
of simple types, it's common for all compilers supporting so-called
"cdecl" on any architecture (x86, x86-64, ppc, so on). C ABI doesn't
define how to pass structures, so we can't generate universal
assembler code for this example - it depends on what compiler we want
to interact to

moreover, even for pointers-to-structures, there is no definitive
layout due to packing problem

OTOH, we could try to use gcc agreements


--
Best regards,
 Bulat                            mailto:Bulat.Ziganshin@...

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Duncan Coutts]

On Tue, 2009-06-30 at 00:45 +0400, Bulat Ziganshin wrote:

> Hello Duncan,
>
> Tuesday, June 30, 2009, 12:03:15 AM, you wrote:
>
> > struct ex {
> >      int x;
> >      int y;
> >      int z;
> > };
>
> > ex example_functions (ex p)
>
> afaik, there is C ABI, that defines how to pass and return parameters
> of simple types, it's common for all compilers supporting so-called
> "cdecl" on any architecture (x86, x86-64, ppc, so on). C ABI doesn't
> define how to pass structures, so we can't generate universal
> assembler code for this example - it depends on what compiler we want
> to interact to

Actually passing structs and unions as arguments or function results is
specified by the C ABI. See for example the IA32 ABI:

http://www.caldera.com/developers/devspecs/abi386-4.pdf

linked from the LSB:
http://refspecs.linux-foundation.org/LSB_3.1.0/LSB-Core-IA32/LSB-Core-IA32/normativerefs.html

See the section Function Calling Sequence, 3-9. Specifically:
  3-14: Functions Returning Structures or Unions
  3-18: Structure and Union Arguments

On IA32 structs/unions passed as parameters go by value on the stack.
For structs/unions as function results, they are stored into a
caller-allocated area on the stack, pointed to by a "hidden" first arg.

It's different on each arch, but it's all completely specified.

> moreover, even for pointers-to-structures, there is no definitive
> layout due to packing problem

Actually the padding is also specified by the C ABI. See section 3-3.
Compiler pragmas to use packed layout are not ABI conformant (and so
typically are only used internally).

Duncan

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[John Meacham]

On Tue, Jun 30, 2009 at 01:18:32AM +0100, Duncan Coutts wrote:

> On IA32 structs/unions passed as parameters go by value on the stack.
> For structs/unions as function results, they are stored into a
> caller-allocated area on the stack, pointed to by a "hidden" first arg.
>
> It's different on each arch, but it's all completely specified.
>
> > moreover, even for pointers-to-structures, there is no definitive
> > layout due to packing problem
>
> Actually the padding is also specified by the C ABI. See section 3-3.
> Compiler pragmas to use packed layout are not ABI conformant (and so
> typically are only used internally).

Yup. Unfortunately this doesn't help us, since in order to pass
structures we need to know more than just the layout (offset and size)
of the type, but we need to know the underlying C types as well which a
Storable instance doesn't give us. depending on the ABI, struct passing
may not be as simple as 'pass on stack' vs 'pass in registers'. it may
be something like 'the floating point components get pulled out and put
in registers, which the other components get pushed onto the stack'. So,
without some compiler magic, like a 'deriving Storable' instance that
derived some hidden methods to contain this extra information, or
additions to storable, this won't work in general.

        John

--
John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Duncan Coutts]

On Tue, 2009-06-30 at 18:59 -0700, John Meacham wrote:

> On Tue, Jun 30, 2009 at 01:18:32AM +0100, Duncan Coutts wrote:
> > On IA32 structs/unions passed as parameters go by value on the stack.
> > For structs/unions as function results, they are stored into a
> > caller-allocated area on the stack, pointed to by a "hidden" first arg.
> >
> > It's different on each arch, but it's all completely specified.
> >
> > > moreover, even for pointers-to-structures, there is no definitive
> > > layout due to packing problem
> >
> > Actually the padding is also specified by the C ABI. See section 3-3.
> > Compiler pragmas to use packed layout are not ABI conformant (and so
> > typically are only used internally).
>
> Yup. Unfortunately this doesn't help us, since in order to pass
> structures we need to know more than just the layout (offset and size)
> of the type, but we need to know the underlying C types as well which a
> Storable instance doesn't give us. depending on the ABI, struct passing
> may not be as simple as 'pass on stack' vs 'pass in registers'. it may
> be something like 'the floating point components get pulled out and put
> in registers, which the other components get pushed onto the stack'. So,
> without some compiler magic, like a 'deriving Storable' instance that
> derived some hidden methods to contain this extra information, or
> additions to storable, this won't work in general.

Yeah, I'm not suggesting going via Storable (for all those reasons),
just extending the FFI to say tuples of FFI types get passed as the
corresponding C ABI structs. All the magic to match the current platform
C ABI then lives in the compiler.

I was only half-serious in suggesting this btw, though as far as I can
see it should actually work. It doesn't help with unions of course and
it adds complexity to the compiler.

Duncan

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Maurí­cio]

> Yeah, I'm not suggesting going via Storable (for all those reasons),
> just extending the FFI to say tuples of FFI types get passed as the
> corresponding C ABI structs. All the magic to match the current platform
> C ABI then lives in the compiler.

Agree. The tuples idea is far better than my first sugestion
on Storable instances.

> I was only half-serious in suggesting this btw, though as far as I can
> see it should actually work. It doesn't help with unions of course and
> it adds complexity to the compiler.

:(

The only known case I have of this beeing important is on GNU GSL,
where the complex functions use a complex type defined as a struct
(with a size 2 array as the only member). This actually just
emulates the 'complex' type of C99, I'm almost sure.

Do you imagine an objection on creating a ticket asking for
something like CComplex on Foreign.C.Types?


Maurício

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[John Meacham]

On Wed, Jul 01, 2009 at 02:22:36PM -0300, Maurí­cio wrote:

>> Yeah, I'm not suggesting going via Storable (for all those reasons),
>> just extending the FFI to say tuples of FFI types get passed as the
>> corresponding C ABI structs. All the magic to match the current platform
>> C ABI then lives in the compiler.
>
> Agree. The tuples idea is far better than my first sugestion
> on Storable instances.
>
>> I was only half-serious in suggesting this btw, though as far as I can
>> see it should actually work. It doesn't help with unions of course and
>> it adds complexity to the compiler.
>
> :(
>
> The only known case I have of this beeing important is on GNU GSL,
> where the complex functions use a complex type defined as a struct
> (with a size 2 array as the only member). This actually just
> emulates the 'complex' type of C99, I'm almost sure.

I don't know whether it is required that Complex be represented in such
a way though.


> Do you imagine an objection on creating a ticket asking for
> something like CComplex on Foreign.C.Types?

No, I would like it. Also add a 'CBool' that maps to the calling
convention for _Bool while you are at it.

You would need CComplexFloat and CComplexDouble... or perhaps we should
just define that  'Complex CFloat' and 'Complex CDouble' are FFI-able
types in the obvious way. that is probably better actually.

Another FFI related extension I have wanted at times is making user
defined types that derive an 'Enum' instance FFI-able, with an 'int'
calling convention, translated via toEnum and fromEnum.

--
John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Bulat Ziganshin-2]

Hello Duncan,

Tuesday, June 30, 2009, 4:18:32 AM, you wrote:

> Actually passing structs and unions as arguments or function results is
> specified by the C ABI. See for example the IA32 ABI:

> http://www.caldera.com/developers/devspecs/abi386-4.pdf

> linked from the LSB:
> http://refspecs.linux-foundation.org/LSB_3.1.0/LSB-Core-IA32/LSB-Core-IA32/normativerefs.html

sorry, it's "System V ABI, IA32 Supplement"

so situation is still the same - afaik, there is no common ABI for all
C compilers on x86 platform that describes structures passing and alignment


--
Best regards,
 Bulat                            mailto:Bulat.Ziganshin@...

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Duncan Coutts]

On Thu, 2009-07-02 at 01:26 +0400, Bulat Ziganshin wrote:

> Hello Duncan,
>
> Tuesday, June 30, 2009, 4:18:32 AM, you wrote:
>
> > Actually passing structs and unions as arguments or function results is
> > specified by the C ABI. See for example the IA32 ABI:
>
> > http://www.caldera.com/developers/devspecs/abi386-4.pdf
>
> > linked from the LSB:
> > http://refspecs.linux-foundation.org/LSB_3.1.0/LSB-Core-IA32/LSB-Core-IA32/normativerefs.html
>
> sorry, it's "System V ABI, IA32 Supplement"
>
> so situation is still the same - afaik, there is no common ABI for all
> C compilers on x86 platform that describes structures passing and alignment

You don't need it to be the same between Windows and Unix, it just has
to be standard on each platform, which it is. There are really only two
ABIs in common use on x86, the System V ABI and the MS one (which apart
from the stdcall calling convention only differs in the bitfield layout
iirc).

But within each platform it most definitely is the same between C
compilers[*], that's the whole point. These days there's even a standard
C++ ABI. That took ages to standardise, the C one has been around for
much much longer.

Duncan


[*] On Windows gcc can do both normal and MS layout of bitfields.

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Bulat Ziganshin-2]

Hello Duncan,

Thursday, July 2, 2009, 2:57:29 AM, you wrote:

> You don't need it to be the same between Windows and Unix, it just has
> to be standard on each platform, which it is. There are really only two
> ABIs in common use on x86, the System V ABI and the MS one (which apart
> from the stdcall calling convention only differs in the bitfield layout
> iirc).

you mean that on windows gcc, msvc and all other C compilers use the
same ABI for passing and packing structs?

(at least it does noting common to document you shown, it says about
SysV ABI, that may be common for all modern unicies - i don't know)


--
Best regards,
 Bulat                            mailto:Bulat.Ziganshin@...

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Duncan Coutts]

On Thu, 2009-07-02 at 03:01 +0400, Bulat Ziganshin wrote:

> Hello Duncan,
>
> Thursday, July 2, 2009, 2:57:29 AM, you wrote:
>
> > You don't need it to be the same between Windows and Unix, it just has
> > to be standard on each platform, which it is. There are really only two
> > ABIs in common use on x86, the System V ABI and the MS one (which apart
> > from the stdcall calling convention only differs in the bitfield layout
> > iirc).
>
> you mean that on windows gcc, msvc and all other C compilers use the
> same ABI for passing and packing structs?

When using the gcc option -mms-bitfields then yes, gcc and msvc use the
same ABI for packing structs. As the name suggests, the only difference
otherwise is that MSVC uses a non-standard layout for bitfields, but
since it's the de-facto native compiler it kind of defines the native C
ABI. So as it happens this isn't relevant for the FFI because it does
not support bitfields.

Duncan


_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Maurí­cio]

>> Do you imagine an objection on creating a ticket asking for
>> something like CComplex on Foreign.C.Types?
>
> No, I would like it. Also add a 'CBool' that maps to the calling
> convention for _Bool while you are at it.

Here is a draft. Please comment:



Proposal: complement Foreign.C.Types

A few more types could be usefull in Foreign.C.Types. These
are sugested to be included:

* CBool
     Related to 'bool' (or _Bool) from stdbool.h.

* CComplexFloat, CComplexDouble
     Related to 'float complex' and 'double complex'.


Pros:

Cons:

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[John Meacham]

On Wed, Jul 01, 2009 at 09:05:06PM -0300, Maurí­cio wrote:

>>> Do you imagine an objection on creating a ticket asking for
>>> something like CComplex on Foreign.C.Types?
>>
>> No, I would like it. Also add a 'CBool' that maps to the calling
>> convention for _Bool while you are at it.
>
> Here is a draft. Please comment:
>
>
>
> Proposal: complement Foreign.C.Types
>
> A few more types could be usefull in Foreign.C.Types. These
> are sugested to be included:
>
> * CBool
>     Related to 'bool' (or _Bool) from stdbool.h.

Good. It appears from the FFI spec that it is possible to define HsBool
as '_Bool' and let 'Bool' translate to a 'bool' calling convention.
However, it is not guarenteed so a CBool type that is guarenteed to map
to _Bool is still good.

It would be odd to define CBool as an intergral type though, perhaps we
should specify it be defined as

newtype CBool = CBool Bool

so we can convert between haskell Bool and CBool easily. Though,
for consistency with other C types, we may want to

> * CComplexFloat, CComplexDouble
>     Related to 'float complex' and 'double complex'.

We need a way to actually examine the values, since 'fromIntegral' won't
work. I think the easiest way is to actually reuse Complex which is
already in Haskell 98, defining

Complex Foo where Foo is a FFI marshable type translates to the complex
version of said type. so 'Complex CDouble' translates to the external C
type of 'double complex'.

We may also want to add 'long double' as a c marshable type too.


        John

--
John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Maurí­cio]

>> * CComplexFloat, CComplexDouble
>>     Related to 'float complex' and 'double complex'.
>
> We need a way to actually examine the values, since 'fromIntegral' won't
> work. I think the easiest way is to actually reuse Complex which is
> already in Haskell 98, defining

What do you think of

newtype CComplexFloat = CComplexFloat {crealf, cimagf :: CFloat}

'crealf' and 'cimagf' actually belong to the standard, and
are supposed to be used to read the real and imaginary part
of a float complex. There are also 'creal', 'cimag', 'creall',
'cimagl' that we would use in CComplexDouble and CComplexLDouble.

> We may also want to add 'long double' as a c marshable type too.

I think it's already there. Isn't it CLDouble?

Maurício

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe