Binding C libraries which use variable arguments (stdarg.h)

[Camarade_Tux]

Hi,

I am currently trying to bind a C function that takes variables
arguments, like foo(int a, ...). I can't find how to make a C stub for
that function.

I found camlidl didn't support idl's "vararg" keyword and pidl (p
stands for perl) didn't either. I was told to try libffi but it
doesn't support varargs unfortunately. I also thought about doing it
by hand but that's not the best idea for portability.

Any other idea? Hint^WPointer? (sorry for the bad joke ;-) )

Thanks.


PS: I was also told I was toast

 ---

Adrien Nader

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Basile Starynkevitch]

Adrien wrote:
> Hi,
>
> I am currently trying to bind a C function that takes variables
> arguments, like foo(int a, ...). I can't find how to make a C stub for
> that function.

  I am assuming that the a is the number of actual arguments, so you call
foo(3, x, y, z)
foo(5, t, t+1, t+3, 0, 4)
foo(0)

>
> Any other idea? Hint^WPointer? (sorry for the bad joke ;-) )

First, you could suppose that the a has a reasonable limit, say 100.

Then you could generate the glue code for each value of the argument a.
I mean generate ocaml code like

external f0: void -> uit = "f_0"
external f1: int -> unit = "f_1"
external f2: int -> int -> unit = "f_2"
external f3: int -> int -> int -> unit = "f_3"

let f a = match Array.length a with
  0 -> f0 ()
| 1 -> f1 a.[0]
| 2 -> f2 a.[0] a.[1]
| 3 -> f3 a.[0] a.[1] a.[2]
....
| _ -> failwith "too many components for f"

and generate C code for each of f_0 f_1 ...

and call f with an array ...

The specialized code generator is reasonably written in Ocaml

There are more crazy variants, including

try Ocaml varargs like Pierre Weis did in printf.ml. For plain mortals
like me this is white magic.

Assuming a Linux system, you could lazily generate the glue code and
invoke dynamic linker on it. So the general case would be to call the
code generator.

Time to go to bed. I am saying lot of non-sense.

Bye!


--
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: [phone]
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Goswin von Brederlow-2]

Basile STARYNKEVITCH <basile@...> writes:

> Adrien wrote:
>> Hi,
>>
>> I am currently trying to bind a C function that takes variables
>> arguments, like foo(int a, ...). I can't find how to make a C stub for
>> that function.
>
>  I am assuming that the a is the number of actual arguments, so you call
> foo(3, x, y, z)
> foo(5, t, t+1, t+3, 0, 4)
> foo(0)
>
>>
>> Any other idea? Hint^WPointer? (sorry for the bad joke ;-) )
>
> First, you could suppose that the a has a reasonable limit, say 100.
>
> Then you could generate the glue code for each value of the argument
> a. I mean generate ocaml code like
>
> external f0: void -> uit = "f_0"
> external f1: int -> unit = "f_1"
> external f2: int -> int -> unit = "f_2"
> external f3: int -> int -> int -> unit = "f_3"
>
> let f a = match Array.length a with
>  0 -> f0 ()
> | 1 -> f1 a.[0]
> | 2 -> f2 a.[0] a.[1]
> | 3 -> f3 a.[0] a.[1] a.[2]
> ....
> | _ -> failwith "too many components for f"
>
> and generate C code for each of f_0 f_1 ...
>
> and call f with an array ...
>
> The specialized code generator is reasonably written in Ocaml
>
> There are more crazy variants, including
>
> try Ocaml varargs like Pierre Weis did in printf.ml. For plain mortals
> like me this is white magic.
>
> Assuming a Linux system, you could lazily generate the glue code and
> invoke dynamic linker on it. So the general case would be to call the
> code generator.
>
> Time to go to bed. I am saying lot of non-sense.
>
> Bye!

Since ocaml functions with more than 5 args use an array you only do
that in ocaml for a few arguments and then you need to do this in
C. So do it in C for all. The stub takes an array and then switches on
the lentgh to call the real function.

Unfortunately you can not convert an array or list into a va_list. You
need to specifically catch each length and call foo(4, a[0], a[1],
a[2], a[3]) for each length.

MfG
        Goswin

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Camarade_Tux]

On 28/10/2009, Basile STARYNKEVITCH <basile@...> wrote:

> Adrien wrote:
>> Hi,
>>
>> I am currently trying to bind a C function that takes variables
>> arguments, like foo(int a, ...). I can't find how to make a C stub for
>> that function.
>
>   I am assuming that the a is the number of actual arguments, so you call
> foo(3, x, y, z)
> foo(5, t, t+1, t+3, 0, 4)
> foo(0)

Well, actually the only reason was that stdarg.h needs at least one
non-variable argument (as opposed to varargs.h). ;-)
Anyway, that doesn't change anything.

>> Any other idea? Hint^WPointer? (sorry for the bad joke ;-) )
>
> First, you could suppose that the a has a reasonable limit, say 100.
>
> Then you could generate the glue code for each value of the argument a.
> I mean generate ocaml code like
>
> external f0: void -> uit = "f_0"
> external f1: int -> unit = "f_1"
> external f2: int -> int -> unit = "f_2"
> external f3: int -> int -> int -> unit = "f_3"
>
> let f a = match Array.length a with
>   0 -> f0 ()
> | 1 -> f1 a.[0]
> | 2 -> f2 a.[0] a.[1]
> | 3 -> f3 a.[0] a.[1] a.[2]
> ....
> | _ -> failwith "too many components for f"
>
> and generate C code for each of f_0 f_1 ...
>
> and call f with an array ...
>
> The specialized code generator is reasonably written in Ocaml

I'll probably go for that. The API won't be as nice as it could but at
least, it's going to be safe and checked at compile-time.

> There are more crazy variants, including
>
> try Ocaml varargs like Pierre Weis did in printf.ml. For plain mortals
> like me this is white magic.

That could make a nicer API. Maybe for ocaml-gir 3 or 4. :-)

> Assuming a Linux system, you could lazily generate the glue code and
> invoke dynamic linker on it. So the general case would be to call the
> code generator.

On linux and x86, I could probably use an array where I would move by
sizeof(some_type). I really want portability however (I want to use
these bindings at least on windows too).

> Time to go to bed. I am saying lot of non-sense.
>
> Bye!

Thanks. :-)


 ---

Adrien Nader

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Camarade_Tux]

On 29/10/2009, Goswin von Brederlow <goswin-v-b@...> wrote:

> Basile STARYNKEVITCH <basile@...> writes:
>
>> Adrien wrote:
>>> Hi,
>>>
>>> I am currently trying to bind a C function that takes variables
>>> arguments, like foo(int a, ...). I can't find how to make a C stub for
>>> that function.
>>
>>  I am assuming that the a is the number of actual arguments, so you call
>> foo(3, x, y, z)
>> foo(5, t, t+1, t+3, 0, 4)
>> foo(0)
>>
>>>
>>> Any other idea? Hint^WPointer? (sorry for the bad joke ;-) )
>>
>> First, you could suppose that the a has a reasonable limit, say 100.
>>
>> Then you could generate the glue code for each value of the argument
>> a. I mean generate ocaml code like
>>
>> external f0: void -> uit = "f_0"
>> external f1: int -> unit = "f_1"
>> external f2: int -> int -> unit = "f_2"
>> external f3: int -> int -> int -> unit = "f_3"
>>
>> let f a = match Array.length a with
>>  0 -> f0 ()
>> | 1 -> f1 a.[0]
>> | 2 -> f2 a.[0] a.[1]
>> | 3 -> f3 a.[0] a.[1] a.[2]
>> ....
>> | _ -> failwith "too many components for f"
>>
>> and generate C code for each of f_0 f_1 ...
>>
>> and call f with an array ...
>>
>> The specialized code generator is reasonably written in Ocaml
>>
>> There are more crazy variants, including
>>
>> try Ocaml varargs like Pierre Weis did in printf.ml. For plain mortals
>> like me this is white magic.
>>
>> Assuming a Linux system, you could lazily generate the glue code and
>> invoke dynamic linker on it. So the general case would be to call the
>> code generator.
>>
>> Time to go to bed. I am saying lot of non-sense.
>>
>> Bye!
>
> Since ocaml functions with more than 5 args use an array you only do
> that in ocaml for a few arguments and then you need to do this in
> C. So do it in C for all. The stub takes an array and then switches on
> the lentgh to call the real function.
>
> Unfortunately you can not convert an array or list into a va_list. You
> need to specifically catch each length and call foo(4, a[0], a[1],
> a[2], a[3]) for each length.

Well, my problem is more when calling the C function since I can chose
the interface of the caml functions.

There's something I've not been sure and would like to ask however:
for the sake of consitency, I've been giving both a native and
bytecode function every time no matter the number of arguments. Could
that be a problem?

 ---

Adrien Nader

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Goswin von Brederlow-2]

Adrien <camaradetux@...> writes:

> On 29/10/2009, Goswin von Brederlow <goswin-v-b@...> wrote:
>> Basile STARYNKEVITCH <basile@...> writes:
>>
>>> Adrien wrote:
>>>> Hi,
>>>>
>>>> I am currently trying to bind a C function that takes variables
>>>> arguments, like foo(int a, ...). I can't find how to make a C stub for
>>>> that function.
>>>
>>>  I am assuming that the a is the number of actual arguments, so you call
>>> foo(3, x, y, z)
>>> foo(5, t, t+1, t+3, 0, 4)
>>> foo(0)
>>>
>>>>
>>>> Any other idea? Hint^WPointer? (sorry for the bad joke ;-) )
>>>
>>> First, you could suppose that the a has a reasonable limit, say 100.
>>>
>>> Then you could generate the glue code for each value of the argument
>>> a. I mean generate ocaml code like
>>>
>>> external f0: void -> uit = "f_0"
>>> external f1: int -> unit = "f_1"
>>> external f2: int -> int -> unit = "f_2"
>>> external f3: int -> int -> int -> unit = "f_3"
>>>
>>> let f a = match Array.length a with
>>>  0 -> f0 ()
>>> | 1 -> f1 a.[0]
>>> | 2 -> f2 a.[0] a.[1]
>>> | 3 -> f3 a.[0] a.[1] a.[2]
>>> ....
>>> | _ -> failwith "too many components for f"
>>>
>>> and generate C code for each of f_0 f_1 ...
>>>
>>> and call f with an array ...
>>>
>>> The specialized code generator is reasonably written in Ocaml
>>>
>>> There are more crazy variants, including
>>>
>>> try Ocaml varargs like Pierre Weis did in printf.ml. For plain mortals
>>> like me this is white magic.
>>>
>>> Assuming a Linux system, you could lazily generate the glue code and
>>> invoke dynamic linker on it. So the general case would be to call the
>>> code generator.
>>>
>>> Time to go to bed. I am saying lot of non-sense.
>>>
>>> Bye!
>>
>> Since ocaml functions with more than 5 args use an array you only do
>> that in ocaml for a few arguments and then you need to do this in
>> C. So do it in C for all. The stub takes an array and then switches on
>> the lentgh to call the real function.
>>
>> Unfortunately you can not convert an array or list into a va_list. You
>> need to specifically catch each length and call foo(4, a[0], a[1],
>> a[2], a[3]) for each length.
>
> Well, my problem is more when calling the C function since I can chose
> the interface of the caml functions.

If you can't change the C function to use something other than varargs
and you can't limit the number of args and implement a case for each
number then the only option left is to create the varargs from an
array. That will be really ugly and architecture and compiler
dependent. So lots of #ifdef cases in your C code. Something you
really do not want to do.

> There's something I've not been sure and would like to ask however:
> for the sake of consitency, I've been giving both a native and
> bytecode function every time no matter the number of arguments. Could
> that be a problem?

No. With >= 5 args you just need 2 functions, one for native and one
for bytecode. The docs have an example on how to do that. The problem
is calling the actual C function.

MfG
        Goswin

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Xavier Leroy]

[Invoking a variadic C argument from OCaml]

>> Assuming a Linux system, you could lazily generate the glue code and
>> invoke dynamic linker on it. So the general case would be to call the
>> code generator.
>
> On linux and x86, I could probably use an array where I would move by
> sizeof(some_type). I really want portability however (I want to use
> these bindings at least on windows too).

It might be worth investigating the "libffi" library.  If I guess
correctly, it lets you synthesize calls to native C functions when the
number and type of arguments are statically unknown.

- Xavier Leroy

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Camarade_Tux]

On 30/10/2009, Xavier Leroy <Xavier.Leroy@...> wrote:

> [Invoking a variadic C argument from OCaml]
>
>>> Assuming a Linux system, you could lazily generate the glue code and
>>> invoke dynamic linker on it. So the general case would be to call the
>>> code generator.
>>
>> On linux and x86, I could probably use an array where I would move by
>> sizeof(some_type). I really want portability however (I want to use
>> these bindings at least on windows too).
>
> It might be worth investigating the "libffi" library.  If I guess
> correctly, it lets you synthesize calls to native C functions when the
> number and type of arguments are statically unknown.

I had checked libffi but its readme (from gcc-4.3) stated variable
arguments weren't supported. After some time I had doubts about the
relevance of the documentation (the last change mentionned was from
98) and tried anyway.

It turned out it worked on linux x86_32, linux x86_64, netbsd sparc64
with 32-bit userland, debian's kfreebsd amd64 and ppc32. There's
windows which I haven't checked yet though.

I posted my test code on vpaste.net[1]. I think it's good enough to
actually check libffi works with variable arguments but it could
probably be made better. Any comment welcome.

I also asked on the libffi mailing-list and was told by someone it
should work almost everywhere except on really exotic architectures.
That means I'll probably go the libffi way. :-)

[1] http://vpaste.net/6vPst?

 ---

Adrien Nader

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Florian Weimer]

* Adrien:

> It turned out it worked on linux x86_32, linux x86_64, netbsd sparc64
> with 32-bit userland, debian's kfreebsd amd64 and ppc32. There's
> windows which I haven't checked yet though.

x86_64 has different calling conventions for varargs and non-varargs
functions.  If it works for you, it's only by accident (e.g. because
you aren't passing floating-point arguments).

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Camarade_Tux]

On 01/11/2009, Florian Weimer <fw@...> wrote:

> * Adrien:
>
>> It turned out it worked on linux x86_32, linux x86_64, netbsd sparc64
>> with 32-bit userland, debian's kfreebsd amd64 and ppc32. There's
>> windows which I haven't checked yet though.
>
> x86_64 has different calling conventions for varargs and non-varargs
> functions.  If it works for you, it's only by accident (e.g. because
> you aren't passing floating-point arguments).
>

I tried a few things with "double" at first since I'm on x86_64 and
got no warning and no problem whatsoever. The situation was a bit
different with "float" however :
> a.c:16: warning: ‘float’ is promoted to ‘double’ when passed through ‘...’
> a.c:16: note: (so you should pass ‘double’ not ‘float’ to ‘va_arg’)
> a.c:16: note: if this code is reached, the program will abort

Running the program yields a nice: "illegal hardware instruction  ./a.out". :)

The problem is the same on x86_32 however, and APIs are probably
correct with regard to that. I'll see if I can add checks in ocaml-gir
nonetheless. Thanks.

 ---

Adrien Nader

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Florian Weimer]

* Adrien:

> On 01/11/2009, Florian Weimer <fw@...> wrote:
>> * Adrien:
>>
>>> It turned out it worked on linux x86_32, linux x86_64, netbsd sparc64
>>> with 32-bit userland, debian's kfreebsd amd64 and ppc32. There's
>>> windows which I haven't checked yet though.
>>
>> x86_64 has different calling conventions for varargs and non-varargs
>> functions.  If it works for you, it's only by accident (e.g. because
>> you aren't passing floating-point arguments).
>>
>
> I tried a few things with "double" at first since I'm on x86_64 and
> got no warning and no problem whatsoever.

Could you check if libffi sets the %rax register to the number of
floating-point arguments?  (I've seen this working by mere chance in
many cases.)

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Camarade_Tux]

On 01/11/2009, Florian Weimer <fw@...> wrote:

> * Adrien:
>
>> On 01/11/2009, Florian Weimer <fw@...> wrote:
>>> * Adrien:
>>>
>>>> It turned out it worked on linux x86_32, linux x86_64, netbsd sparc64
>>>> with 32-bit userland, debian's kfreebsd amd64 and ppc32. There's
>>>> windows which I haven't checked yet though.
>>>
>>> x86_64 has different calling conventions for varargs and non-varargs
>>> functions.  If it works for you, it's only by accident (e.g. because
>>> you aren't passing floating-point arguments).
>>>
>>
>> I tried a few things with "double" at first since I'm on x86_64 and
>> got no warning and no problem whatsoever.
>
> Could you check if libffi sets the %rax register to the number of
> floating-point arguments?  (I've seen this working by mere chance in
> many cases.)
>

Unfortunately I don't know much about asm, I tried to check but failed
(couldn't find what "leaq" meant).
I found a few references indicating that libffi was properly doing that however.

 ---

Adrien Nader

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Richard Jones-4]

On Sun, Nov 01, 2009 at 03:58:47PM +0100, Adrien wrote:
> Unfortunately I don't know much about asm, I tried to check but failed
> (couldn't find what "leaq" meant).

The 'q' just means it's a Long Mode instruction (ie. x86-64's 64 bit
mode).  The instruction itself is "Load Effective Address" - it means
it does an address calculation as if you were going to load/store
something at that address, but instead puts the address itself (result
of that calculation) in the target register.

If I remember my rusty x86 assembler correctly, then:

  movl 4(%ebx),%eax ; load 4 bytes of data from address %ebx + 4, put in %eax
  lea 4(%ebx),%eax  ; add %ebx + 4 and put that in %eax

Rich.

--
Richard Jones
Red Hat

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Florian Weimer]

* Adrien:

> I found a few references indicating that libffi was properly doing
> that however.

I looked at the assembly, and it is doing the right thing, but for the
wrong reason.  The "only load sse registers if ssecount non-zero"
change in 2005 happens to put the SSE register count in the correct
register, but this wasn't the intent of the change based on the
ChangeLog entry. 8-)

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs