Linking with C Library

[Nyx]

I'm coding a JIT compiler for C source in OCaml, using LLVM. I'm pretty much done with the LLVM code generation. The problem is that I can't seem to call C library functions. I was told that all I needed to do to be able to link with libc functions was to declare them in my module and give them external linkage, but this does not seem to work. Please note that this is a JIT compiler. I am not generating a binary, but running the code generated in memory. It should be possible to dynamically link with the libc functions because the ocaml executable is linked with them.

The three libc functions I declared appear in my module dump as such:
declare i8* @malloc(i32)
declare void @free(i8*)
declare i32 @puts(i8*)

However, when trying to run my program, I get the following:
LLVM ERROR: Tried to execute an unknown external function: i32 (i8*)* puts

I'm not quite sure how to go about this. Any help would be appreciated.

- Maxime

[Jeffrey Yasskin]

In C, on Linux, you would have to link your JIT compiler with
-rdynamic or -Wl,-export-dynamic (they're synonyms). I'm not sure what
the equivalent linker flag is for OCaml.

You can see what symbols are available to the JIT with `nm -D`.

On Sun, Apr 4, 2010 at 8:41 AM, Nyx <mcheva@...> wrote:

>
> I'm coding a JIT compiler for C source in OCaml, using LLVM. I'm pretty much
> done with the LLVM code generation. The problem is that I can't seem to call
> C library functions. I was told that all I needed to do to be able to link
> with libc functions was to declare them in my module and give them external
> linkage, but this does not seem to work. Please note that this is a JIT
> compiler. I am not generating a binary, but running the code generated in
> memory. It should be possible to dynamically link with the libc functions
> because the ocaml executable is linked with them.
>
> The three libc functions I declared appear in my module dump as such:
> declare i8* @malloc(i32)
> declare void @free(i8*)
> declare i32 @puts(i8*)
>
> However, when trying to run my program, I get the following:
> LLVM ERROR: Tried to execute an unknown external function: i32 (i8*)* puts
>
> I'm not quite sure how to go about this. Any help would be appreciated.
>
> - Maxime

_______________________________________________
LLVM Developers mailing list
LLVMdev@...         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Nyx]

I tried running nm - D | grep "puts" on the binary compiled by the OCaml
compiler. It outputs the following:

08161b00 T camlRuntime__rt_fputs_208
08161a20 T camlRuntime__rt_puts_198
         U fputs

I'm assuming this means that fputs is linked dynamically, and puts is
not. I tried modifying my code to use fputs instead of puts instead, but
had no success, however, I still get:

LLVM ERROR: Tried to execute an unknown external function: i32 (i8*,
i32)* fputs

There has to be some way of telling LLVM to get some symbols
dynamically. What's rather frustrating is that the OCaml LLVM tutorial
seems to be doing exactly what I want:

http://llvm.org/docs/tutorial/OCamlLangImpl4.html

They define some "extern" functions by using "declare function", which
are then linked dynamically by the JIT. Unfortunately, this is already
what I'm doing, and I can't seem to spot what they are doing differently!

- Maxime

Jeffrey Yasskin wrote:

> In C, on Linux, you would have to link your JIT compiler with
> -rdynamic or -Wl,-export-dynamic (they're synonyms). I'm not sure what
> the equivalent linker flag is for OCaml.
>
> You can see what symbols are available to the JIT with `nm -D`.
>
> On Sun, Apr 4, 2010 at 8:41 AM, Nyx <mcheva@...> wrote:
>  
>> I'm coding a JIT compiler for C source in OCaml, using LLVM. I'm pretty much
>> done with the LLVM code generation. The problem is that I can't seem to call
>> C library functions. I was told that all I needed to do to be able to link
>> with libc functions was to declare them in my module and give them external
>> linkage, but this does not seem to work. Please note that this is a JIT
>> compiler. I am not generating a binary, but running the code generated in
>> memory. It should be possible to dynamically link with the libc functions
>> because the ocaml executable is linked with them.
>>
>> The three libc functions I declared appear in my module dump as such:
>> declare i8* @malloc(i32)
>> declare void @free(i8*)
>> declare i32 @puts(i8*)
>>
>> However, when trying to run my program, I get the following:
>> LLVM ERROR: Tried to execute an unknown external function: i32 (i8*)* puts
>>
>> I'm not quite sure how to go about this. Any help would be appreciated.
>>
>> - Maxime
>>    

_______________________________________________
LLVM Developers mailing list
LLVMdev@...         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Jeffrey Yasskin]

On Sun, Apr 4, 2010 at 5:24 PM, Maxime Chevalier-Boisvert
<mcheva@...> wrote:
> I tried running nm - D | grep "puts" on the binary compiled by the OCaml
> compiler. It outputs the following:
>
> 08161b00 T camlRuntime__rt_fputs_208
> 08161a20 T camlRuntime__rt_puts_198
>        U fputs
>
> I'm assuming this means that fputs is linked dynamically, and puts is not.

Don't assume; ask `man nm`.
http://www.linuxcommand.org/man_pages/nm1.html says that a 'U' in that
column means "The symbol is undefined." (It's there so the dynamic
loader knows to pull it out of a .so, I think.)

You need to figure out how to pass -rdynamic to the linker, like I
said before. http://llvm.org/docs/tutorial/OCamlLangImpl7.html
mentions it, but I don't know enough about the ocaml build process to
say whether that'll work.

> I
> tried modifying my code to use fputs instead of puts instead, but had no
> success, however, I still get:
>
> LLVM ERROR: Tried to execute an unknown external function: i32 (i8*, i32)*
> fputs
>
> There has to be some way of telling LLVM to get some symbols dynamically.
> What's rather frustrating is that the OCaml LLVM tutorial seems to be doing
> exactly what I want:
>
> http://llvm.org/docs/tutorial/OCamlLangImpl4.html
>
> They define some "extern" functions by using "declare function", which are
> then linked dynamically by the JIT. Unfortunately, this is already what I'm
> doing, and I can't seem to spot what they are doing differently!
>
> - Maxime
>
> Jeffrey Yasskin wrote:
>>
>> In C, on Linux, you would have to link your JIT compiler with
>> -rdynamic or -Wl,-export-dynamic (they're synonyms). I'm not sure what
>> the equivalent linker flag is for OCaml.
>>
>> You can see what symbols are available to the JIT with `nm -D`.
>>
>> On Sun, Apr 4, 2010 at 8:41 AM, Nyx <mcheva@...> wrote:
>>
>>>
>>> I'm coding a JIT compiler for C source in OCaml, using LLVM. I'm pretty
>>> much
>>> done with the LLVM code generation. The problem is that I can't seem to
>>> call
>>> C library functions. I was told that all I needed to do to be able to
>>> link
>>> with libc functions was to declare them in my module and give them
>>> external
>>> linkage, but this does not seem to work. Please note that this is a JIT
>>> compiler. I am not generating a binary, but running the code generated in
>>> memory. It should be possible to dynamically link with the libc functions
>>> because the ocaml executable is linked with them.
>>>
>>> The three libc functions I declared appear in my module dump as such:
>>> declare i8* @malloc(i32)
>>> declare void @free(i8*)
>>> declare i32 @puts(i8*)
>>>
>>> However, when trying to run my program, I get the following:
>>> LLVM ERROR: Tried to execute an unknown external function: i32 (i8*)*
>>> puts
>>>
>>> I'm not quite sure how to go about this. Any help would be appreciated.
>>>
>>> - Maxime
>>>
>
>

_______________________________________________
LLVM Developers mailing list
LLVMdev@...         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Nyx]

>> You need to figure out how to pass -rdynamic to the linker, like I
said before. http://llvm.org/docs/tutorial/OCamlLangImpl7.html
mentions it, but I don't know enough about the ocaml build process to
say whether that'll work.

I believe I'm already doing that, properly by passing -ccopt -rdynamic to ocamlopt:
ocamlopt -cc g++ -ccopt -rdynamic -linkall $(LIBFILES) -o alpha $(OBJFILES)

I've also tried writing a dummy "puts" function in a C file and linking that with my executable. nm -D then shows the following:

000000000054b690 T camlRuntime__rt_fputs_208
000000000054b590 T camlRuntime__rt_puts_198
                 U fputs
0000000000c43044 T puts

However, LLVM *still* gives me:
LLVM ERROR: Tried to execute an unknown external function: i32 (i8*)* puts

Something really wrong is happening. The following thread seems to indicate that this should all work easily:
http://old.nabble.com/jit-with-external-functions-td7769793.html

[Jeffrey Yasskin]

On Mon, Apr 5, 2010 at 3:27 PM, Nyx <mcheva@...> wrote:

>
>>> You need to figure out how to pass -rdynamic to the linker, like I
> said before. http://llvm.org/docs/tutorial/OCamlLangImpl7.html
> mentions it, but I don't know enough about the ocaml build process to
> say whether that'll work.
>
> I believe I'm already doing that, properly by passing -ccopt -rdynamic to
> ocamlopt:
> ocamlopt -cc g++ -ccopt -rdynamic -linkall $(LIBFILES) -o alpha $(OBJFILES)
>
> I've also tried writing a dummy "puts" function in a C file and linking that
> with my executable. nm -D then shows the following:
>
> 000000000054b690 T camlRuntime__rt_fputs_208
> 000000000054b590 T camlRuntime__rt_puts_198
>                 U fputs
> 0000000000c43044 T puts
>
> However, LLVM *still* gives me:
> LLVM ERROR: Tried to execute an unknown external function: i32 (i8*)* puts
>
> Something really wrong is happening. The following thread seems to indicate
> that this should all work easily:
> http://old.nabble.com/jit-with-external-functions-td7769793.html
>

Ok, you've passed my expertise. Maybe you have to link with -lc?
Hopefully one of the ocaml experts will chime in.

_______________________________________________
LLVM Developers mailing list
LLVMdev@...         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Graham Wakefield-2]

If the OCaml bindings include the functionality in  
llvm::sys::DynamicLibrary, perhaps you can use the functions there to  
manually request the puts symbol?

e.g. llvm::sys::DynamicLibrary::LoadLibraryPermanently() and  
llvm::sys::DynamicLibrary::SearchForAddressOfSymbol() ...?

Just a thought.

On Apr 5, 2010, at 6:56 PM, Jeffrey Yasskin wrote:

> On Mon, Apr 5, 2010 at 3:27 PM, Nyx <mcheva@...> wrote:
>>
>>>> You need to figure out how to pass -rdynamic to the linker, like I
>> said before. http://llvm.org/docs/tutorial/OCamlLangImpl7.html
>> mentions it, but I don't know enough about the ocaml build process to
>> say whether that'll work.
>>
>> I believe I'm already doing that, properly by passing -ccopt -
>> rdynamic to
>> ocamlopt:
>> ocamlopt -cc g++ -ccopt -rdynamic -linkall $(LIBFILES) -o alpha $
>> (OBJFILES)
>>
>> I've also tried writing a dummy "puts" function in a C file and  
>> linking that
>> with my executable. nm -D then shows the following:
>>
>> 000000000054b690 T camlRuntime__rt_fputs_208
>> 000000000054b590 T camlRuntime__rt_puts_198
>>                 U fputs
>> 0000000000c43044 T puts
>>
>> However, LLVM *still* gives me:
>> LLVM ERROR: Tried to execute an unknown external function: i32  
>> (i8*)* puts
>>
>> Something really wrong is happening. The following thread seems to  
>> indicate
>> that this should all work easily:
>> http://old.nabble.com/jit-with-external-functions-td7769793.html
>>
>
> Ok, you've passed my expertise. Maybe you have to link with -lc?
> Hopefully one of the ocaml experts will chime in.
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev@...         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
LLVMdev@...         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Nyx]

>> Ok, you've passed my expertise. Maybe you have to link with -lc?
Hopefully one of the ocaml experts will chime in.

I figured out what was wrong. LLVM needs to be compiled with --enable-libffi for the external symbol resolution to work properly. This is unfortunately not mentioned in the OCaml version of the LLVM Kaleidoscope tutorials!

[Nick Lewycky]

Nyx wrote:
>
>>> Ok, you've passed my expertise. Maybe you have to link with -lc?
> Hopefully one of the ocaml experts will chime in.
>
> I figured out what was wrong. LLVM needs to be compiled with --enable-libffi
> for the external symbol resolution to work properly. This is unfortunately
> not mentioned in the OCaml version of the LLVM Kaleidoscope tutorials!

If that fixed your problem then you aren't using the JIT.
--enable-libffi only affects the (very slow and unmaintained) interpreter.

You may want to focus on why you're failing to correctly initialize the
JIT. Are you calling InitializeNativeTarget()?

Nick

_______________________________________________
LLVM Developers mailing list
LLVMdev@...         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Nyx]

>> If that fixed your problem then you aren't using the JIT.
--enable-libffi only affects the (very slow and unmaintained) interpreter.
You may want to focus on why you're failing to correctly initialize the
JIT. Are you calling InitializeNativeTarget()?

Ah! I was using ExecutionEngine.create, instead of ExecutionEngine.create_jit

- Maxime