Exposing C++ objects to Io

[Tres Walsh]

Hello,

For the past few days I've been trying to embed the Io VM in my
application. Maybe it's because there's very little documentation on
it, or maybe I just don't know what the hell I'm doing. Looking at how
Io Addons are compiled, I apparently need to compile my program with
Io?

I also tried the wiki books c++ binding located
http://en.wikibooks.org/wiki/Io_Programming/Binding_Io_to_C%2B%2B

And it actually won't even compile, emitting errors that IoObject has
no data field (And looking in the source IoObjectData is the one that
contains it?)

All in all I'm extremely confused if someone could just help me
out/point me in the right direction/give a step by step instruction, I
would really appreciate it :)

--
http://treswalsh.com

[Steve Dekorte]

On 2009-10-03, at 10:45 PM, Tres Walsh wrote:
> For the past few days I've been trying to embed the Io VM in my
> application.

Hi Tres,

You'll need to link to the libiovm dynamic library.

In your headers:

        #include "IoState.h"
        void IoAddonsInit(IoObject *context);

To open an Io state:

        IoState *self = IoState_new();
        IoState_setBindingsInitCallback(self, (IoStateBindingsInitCallback *)
IoAddonsInit);
        IoState_init(self);

To eval some Io code:

        IoState_doCString_(self, "\"hello world\" println");

To close an Io state:

        IoState_free(self);

Note, only one thread should call into a given Io at a time but you  
can have multiple states open in a given process with different  
threads in each at the same time.

Hope this helps,
- Steve

[Steve Dekorte]

I forgot to mention this link:

http://www.iolanguage.com/scm/git/checkout/Io/docs/IoGuide.html#Embedding-IoState

[dennisf486]

Hi Tres; I too am working on embedding Io in a C++ program.  I'm working on an open source C++/Io binding library that I think could help you, but I'll leave the lengthy description of it to the bottom section of this email in order to first give you some tips about building addons.

(Tres, when we get our projects working we should put our heads together to come up with a definitive guide to making Io work with C++ programs.  The wiki is not very complete on the topic.  Also, I would like to show you how to make an Io prompt for your running VM available via telnet.  That way you will actually be able to type in Io commands and do debugging while the VM is concurrently running your scripts as well.  It takes only 12 lines of Io code!)

I too thought you had to have your project in the addons subdirectory of Io because there's no other examples.  However this causes problems - the Io build system doesn't detect updates to your header files, and you can't pass custom C++ compiler args without hacking build.io.

Instead, it is possible to build your addon in any IDE as an ordinary DLL or .so file and then load the out directly from Io.  You just have to have the right folder structure (which can be located anywhere) and then use some Io script commands to load it.

What I recommend is making a dummy version of your eventual addon in the addons folder (you can copy NullAddon).  Once you've verified it builds along with the other addons you can then use this as a template:  pay close attention to the cc commands the Io build system issues when you type make YourAddonName, especially the final one.  This will tell you what libraries you need to link (mainly -liovm).  If you just duplicate all these options in the C/C++ and linker options on your IDE, you will be building an identical binary!  Pay attention to the folder structure of the addon.  If you type sudo make install, your addon will end up in (I think) /usr/local/lib/io/YourAddonName.  It has some sub folders that Io looks in when it's loading addons.  Make a folder in your project where you want your addon to be compiled to, and copy /usr/local/lib/io/YourAddonName and all its subfolders into it.  Then delete the .so file that's in a subdirectory of it, and configure your IDE to write it output file to the directory where it used to be.

Io's addon loading code looks in a specific subdirectory for the .so to load.  It also looks for a depends.io file, and will refuse to load the addon if the file is empty.  Just open up depends.io and insert a single space, or if you have some addons you depend on, put their names there.

Another weird thing about the addon build process is that your .cpp file's exact names and where you put them initially matters (but only if you were using Io to build it).  This is because Io's build system automatically generates a file called >>>IoYourPluginInit.c<<<.  All it's doing is looking for files named IoSomeName.c or .cpp and generates some code that load protos for them based solely on the file name.  You could easily write the code it outputs yourself but it can cause frustration if you didn't realize the file was also being autogenerated.  At the end of the day, you just must export a function called "IoYourPluginInit" because that is what Io will try to run when it opens up your plugin's shared library.  The autogenerated file does that.  What I would do if I were you is use the autogenerated file as a starting point but make it my own and hand update it.  The one thing you have to have in there is a proto created after your plugin loads that has the same name as your addon.  If you want more protos just add more lines and call different proto functions and change their names - but don't remove the one that is named the same as your addon.

Read the .io scripts in the build folder of the io directory:  build.io, addon.io, etc.  They're not that hard to follow and will enlighten you as to whats going on during the addon building process.

OK so now you have duplicated all the conditions that the Io build system creates for addons, but it's in your own folder and you have control over it.  Io automatically looks in a certain folder when you type an addon name, and obviously it isn't going to know about our custom folder.  So we'll have to use Io code to make it look specifically in our custom folder.  That's ok - another frustrating thing I ran into while using the conventional addon building process was that often Io would get "stuck" using an old version of my addon and no matter what I didn' I couldn't get it to "see" the new version.  Loading it from our custom folder will be better because we can be sure we know we're getting the right one.

Put this into an Io script called "LoadYourPluginName.io" (can someone more experienced with Io assure me I'm "allowed" to reuse the addon loader like this??):

YourPluginLoader := Addon clone
YourPluginLoader setRootPath("path/to/your/plugins")
YourPluginLoader name = "YourPluginName"
YourPluginLoader load

Note that the path to your plugins is not the folder with your plugin name, but rather one level up.  The addon loader will append "name" to the path to get the full path.

You can also run these commands interactively at the Io prompt.  After you've set the root path and the name you type "YourPluginLoader dllPath" you can find out exactly where it's try to load the dll from.

Anyhow cd to your script directory and start up Io.  Type doFile("LoadYourPluginName.io") and it should load your plugin.  Then you can experiment with calling functions interactively.

Troubleshooting:

- problem: Io throws exception while loading the addon, says, "IoYourPluginName_proto not defined".
- solution:
-- You need that function (get it from the autogenerated *Init.c file)
-- You need to declare that function extern "C" (only if in cpp file)
-- You have the wrong name on your plugin (names must match)
-- Can be caused by random build errors.  Clean, fix errors, rebuild

- problem:  Io throws exception while loading addon, says some other symbol not defined
- solution:
-- You used an Io C function that isn't actually implemented
-- You used a function from some other library that isn't implemented
-- You forgot to link to a library in your linker build options

- problem:  Strange behavior, bugs you fixed reappearing
- solution:
-- I was constantly running into situations where Io ended up with an old version of the addon somehow.
-- You should put a debugging message into your proto function that prints the date/time the code was compiled, for a sanity check.  I think that's the C preprocessor macros __DATE__ and __TIME__.

- problem:  Io complains it can't read the depends.io file (even when it's there)
- solution:
-- Just copy the depends file from some other addon and overwrite it

OK now that's done I want to plug my binding library:

*********************************
* LikeMagic C++ Binding Library *
*********************************

It will be released under an open source license based on Io's own license.  It is still a work in progress but hopefully I'll have a usable alpha of the library soon.  I'll send you a copy as soon as I have enough working.  What the library does for you is if you give it the name of your C++ class and the names of  methods of that C++ class, it automatically detects how many arguments and what type of arguments the methods need, and generates an Io proto and methods that map to it.  I'm also doing something beyond that, which is that I treat sets of objects as one block for speed.  That is, say you have a game with an stl::vector of sprites, and you need to call an update function on all the sprites, instead of looping through Io and having the overhead of popping in and out of script for each sprite, you can use a LikeMagic set operation to make one call from script and the framework will call the same update method on all the sprites in the set.  You can also build up arbitrarily complex expressions ahead of time and then execute them all in parallel on a set of objects.  For very large sets, it will dispatch multiple threads to run the function(s) on multiple chunks of the set in parallel at the same time.  So even though the Io code runs single threaded, every time you do a set operation it will automatically take advantage of multiple CPU cores.



--- In iolanguage@..., Tres Walsh <tres.walsh@...> wrote:

>
> Hello,
>
> For the past few days I've been trying to embed the Io VM in my
> application. Maybe it's because there's very little documentation on
> it, or maybe I just don't know what the hell I'm doing. Looking at how
> Io Addons are compiled, I apparently need to compile my program with
> Io?
>
> I also tried the wiki books c++ binding located
> http://en.wikibooks.org/wiki/Io_Programming/Binding_Io_to_C%2B%2B
>
> And it actually won't even compile, emitting errors that IoObject has
> no data field (And looking in the source IoObjectData is the one that
> contains it?)
>
> All in all I'm extremely confused if someone could just help me
> out/point me in the right direction/give a step by step instruction, I
> would really appreciate it :)
>
> --
> http://treswalsh.com
>

[Tres Walsh]

Thanks for your responses. I perhaps I should have been a little clearer.
With my project, it is less of an addon, and is actually a Game Engine,
where Io is going to be used for the scripting elements. So I really am
unsure if I treat it as another addon? or something completely different?

I am already aware of the IoState.h, IoAddonInit etc. etc. (it's actually
how I tested to make sure that the static and dynamic library built with
mingw work :D )

So rather than try to work around it, how about we try this out right now
(woot walkthroughs!).

In my engine I have a class called Vector 3 (This is in c++, just a small
quick FYI). It is a templated class, since anyone can use a double or float
(or an int, if they're crazy :D), and for those who have never done c++
templates the class would be instantiated like so: Vector3*
some_variable_name = new Vector3<int>();

You can see the code here: http://sahchandler.pastey.net/126073

So to create the IoVector3 class what would be the FIRST thing we do?
(Besides the basic
class IoVector3 { }
)

I figure that rather than trying to stumble around in the dark, whilst
blindly asking questions, it is better that I just get a small walkthrough,
for a basic class. And hey maybe I can clean up that wikibooks tutorial if
it does. :)

Again I really appreciate all your help. I sort of feel bad for asking for
help, but my experience with embedding a scripting language is getting a
python script to load and run as if it were __main__ in a C application.

[dennisf486]

--- In iolanguage@..., Tres Walsh <tres.walsh@...> wrote:
>
> Thanks for your responses. I perhaps I should have been a little clearer.
> With my project, it is less of an addon, and is actually a Game Engine,
> where Io is going to be used for the scripting elements. So I really am
> unsure if I treat it as another addon? or something completely different?

I'm actually a little unsure of that myself.  I initially envisioned this:

My game exe -loads-> IoVM -runs-> script

But I didn't have any examples of how to actually do that.  I've shifted to this model:

Io.exe -loads-> my game addon

but only because I haven't got any examples of how to do the former, so I'm doing the latter because I have examples of addons to base it off of.

If I did the first model, where I have a game exe that creates an IoVM state, how is the vm supposed to call back to my C++ functions?  It would be something like this:

My game exe -loads-> IoVM -load addon-> A DLL with my C++ Io bindings -then-> Does what?  Connects back to the data in the EXE?  How does it make this connection?

It seems like it ought to be possible to have this all under one static exe without dynamic libraries but HOW?  I mean the addon loading processes specifically loads DLLs.  How would you get it to link to functions within the same exe without loading the DLL?

[Tres Walsh]

This is also something that I've been pondering. You're first example is how
the Unreal Engine does it I believe, with the second being a more Source
Engine approach.

It sounds like we may have to hack the Io source to get it to work the way
we want? I just don't know :-/

On Sun, Oct 4, 2009 at 12:40 PM, dennisf486 <dennisf486@...> wrote:

>     --- In iolanguage@... <iolanguage%40yahoogroups.com>, Tres
> Walsh <tres.walsh@...> wrote:
> >
> > Thanks for your responses. I perhaps I should have been a little clearer.
> > With my project, it is less of an addon, and is actually a Game Engine,
> > where Io is going to be used for the scripting elements. So I really am
> > unsure if I treat it as another addon? or something completely different?
>
> I'm actually a little unsure of that myself. I initially envisioned this:
>
> My game exe -loads-> IoVM -runs-> script
>
> But I didn't have any examples of how to actually do that. I've shifted to
> this model:
>
> Io.exe -loads-> my game addon
>
> but only because I haven't got any examples of how to do the former, so I'm
> doing the latter because I have examples of addons to base it off of.
>
> If I did the first model, where I have a game exe that creates an IoVM
> state, how is the vm supposed to call back to my C++ functions? It would be
> something like this:
>
> My game exe -loads-> IoVM -load addon-> A DLL with my C++ Io bindings
> -then-> Does what? Connects back to the data in the EXE? How does it make
> this connection?
>
> It seems like it ought to be possible to have this all under one static exe
> without dynamic libraries but HOW? I mean the addon loading processes
> specifically loads DLLs. How would you get it to link to functions within
> the same exe without loading the DLL?
>
>  
>

--
http://treswalsh.com

[dennisf486]

Well after I wrote that I had something of an epiphany.

I had been thinking in terms of the Io code calling functions in our exe and "how would it even know what functions to call, if it hadn't been compiled together with my code?"

But what I just realized is that Io calls our C/C++ functions ***using a pointer***.  The fact that it loads a DLL first when loading an addon is just how it gets some code loaded; but if the code were already statically present inside the exe you could skip the step of loading a DLL entirely.  All you really have to do, is call the right methods to perform the right operations on your IoVM state to shove an Io proto in there that has function pointers that point to C++ functions in your exe.  Just because you *can* use the addon loading process to do it doesn't mean that you *must* use that process.  Instead of loading an addon, you could build the proto "by hand" in your exe in C/C++ code, put the addresses of some of your C++ functions into the proto's method table, and put the address of your C++ game state object into the proto's data pointer.  Get that proto into your Io state (somehow) and voila!  Your script is bound to your C++ data and functions without addons being involved at all.

I'm not sure on the details of how to do it but I think this must be in general how it's done.  It has some advantages over the addon based architecture:  you can structure your project how you want instead of being forced to make it a DLL, and less chance of a deployment problem (everything in one exe so no need to worry about DLLs not being in right directory).  You could probably even statically link the Io VM and not need that as a DLL either.  Plus, when it comes to the main game engine being in a DLL I was a little worried someone might hack it by writing a shim DLL that sits between the game engine and the script and monitors everything that goes between them.  I know that putting everything in one exe doesn't magically make it more secure, but somehow the idea of having my whole game be a constellation of DLLs seems almost inviting trouble.

--- In iolanguage@..., Tres Walsh <tres.walsh@...> wrote:

>
> This is also something that I've been pondering. You're first example is how
> the Unreal Engine does it I believe, with the second being a more Source
> Engine approach.
>
> It sounds like we may have to hack the Io source to get it to work the way
> we want? I just don't know :-/
>
> On Sun, Oct 4, 2009 at 12:40 PM, dennisf486 <dennisf486@...> wrote:
>
> >     --- In iolanguage@... <iolanguage%40yahoogroups.com>, Tres
> > Walsh <tres.walsh@> wrote:
> > >
> > > Thanks for your responses. I perhaps I should have been a little clearer.
> > > With my project, it is less of an addon, and is actually a Game Engine,
> > > where Io is going to be used for the scripting elements. So I really am
> > > unsure if I treat it as another addon? or something completely different?
> >
> > I'm actually a little unsure of that myself. I initially envisioned this:
> >
> > My game exe -loads-> IoVM -runs-> script
> >
> > But I didn't have any examples of how to actually do that. I've shifted to
> > this model:
> >
> > Io.exe -loads-> my game addon
> >
> > but only because I haven't got any examples of how to do the former, so I'm
> > doing the latter because I have examples of addons to base it off of.
> >
> > If I did the first model, where I have a game exe that creates an IoVM
> > state, how is the vm supposed to call back to my C++ functions? It would be
> > something like this:
> >
> > My game exe -loads-> IoVM -load addon-> A DLL with my C++ Io bindings
> > -then-> Does what? Connects back to the data in the EXE? How does it make
> > this connection?
> >
> > It seems like it ought to be possible to have this all under one static exe
> > without dynamic libraries but HOW? I mean the addon loading processes
> > specifically loads DLLs. How would you get it to link to functions within
> > the same exe without loading the DLL?
> >
> >  
> >
>
>
>
> --
> http://treswalsh.com
>

[Steve Dekorte]

On 2009-10-04, at 3:41 PM, dennisf486 wrote:
>   Just because you *can* use the addon loading process to do it  
> doesn't mean that you *must* use that process.

Exactly.

>  Instead of loading an addon, you could build the proto "by hand" in  
> your exe in C/C++ code, put the addresses of some of your C++  
> functions into the proto's method table, and put the address of your  
> C++ game state object into the proto's data pointer.  Get that proto  
> into your Io state (somehow) and voila!  Your script is bound to  
> your C++ data and functions without addons being involved at all.

Right. And thanks to the terrible semantics of C/C++ dynamically  
loaded libs as they are implemented on most (perhaps all) current  
systems, the static route is probably the way to go for a game  
embedding project.

[Tres Walsh]

Well this is what the wikibooks tutorial does, with each method in the
IoGlue class being static

However when I perform this such as

static IoObject* proto(IoState* state)
{
     IoObject* self = IoObject_new(state);
     self->tag = tag(state, "Nameof_IoObject");
     self->data = 0; //This is where an error happen D:
}

We get an error that IoObject has no member named data. Which is partially
true, since it is IoObjectData that contains that member, and IoObject is
just a typedef for CollectorMarker (or something similar)

What I think might work is, creating your IoVM in a separate thread, but,
manually calling all of the protos for each class? If you're using a custom
build system, I'm sure you could generate a "small"(in the sense of what it
does on a grand scale) function that passes the IoVM State to all the
protos, so by the time the game is up and running, everything has been
properly initialized. (I think this is how the Unreal Engine does it, which
would explain for the 30-60 second startup times where you're just staring
at a splash screen.)

The problem still remains of running the "main" io script, although
theoretically, if you needed to you could just run
IoState_doCString_(self,"doFile(\"scriptName.io\")"); :)

On Sun, Oct 4, 2009 at 3:41 PM, dennisf486 <dennisf486@...> wrote:

>
>
> Well after I wrote that I had something of an epiphany.
>
> I had been thinking in terms of the Io code calling functions in our exe
> and "how would it even know what functions to call, if it hadn't been
> compiled together with my code?"
>
> But what I just realized is that Io calls our C/C++ functions ***using a
> pointer***. The fact that it loads a DLL first when loading an addon is just
> how it gets some code loaded; but if the code were already statically
> present inside the exe you could skip the step of loading a DLL entirely.
> All you really have to do, is call the right methods to perform the right
> operations on your IoVM state to shove an Io proto in there that has
> function pointers that point to C++ functions in your exe. Just because you
> *can* use the addon loading process to do it doesn't mean that you *must*
> use that process. Instead of loading an addon, you could build the proto "by
> hand" in your exe in C/C++ code, put the addresses of some of your C++
> functions into the proto's method table, and put the address of your C++
> game state object into the proto's data pointer. Get that proto into your Io
> state (somehow) and voila! Your script is bound to your C++ data and
> functions without addons being involved at all.
>
> I'm not sure on the details of how to do it but I think this must be in
> general how it's done. It has some advantages over the addon based
> architecture: you can structure your project how you want instead of being
> forced to make it a DLL, and less chance of a deployment problem (everything
> in one exe so no need to worry about DLLs not being in right directory). You
> could probably even statically link the Io VM and not need that as a DLL
> either. Plus, when it comes to the main game engine being in a DLL I was a
> little worried someone might hack it by writing a shim DLL that sits between
> the game engine and the script and monitors everything that goes between
> them. I know that putting everything in one exe doesn't magically make it
> more secure, but somehow the idea of having my whole game be a constellation
> of DLLs seems almost inviting trouble.
>
> --- In iolanguage@... <iolanguage%40yahoogroups.com>, Tres
> Walsh <tres.walsh@...> wrote:
> >
> > This is also something that I've been pondering. You're first example is
> how
> > the Unreal Engine does it I believe, with the second being a more Source
> > Engine approach.
> >
> > It sounds like we may have to hack the Io source to get it to work the
> way
> > we want? I just don't know :-/
> >
> > On Sun, Oct 4, 2009 at 12:40 PM, dennisf486 <dennisf486@...> wrote:
> >
> > > --- In iolanguage@... <iolanguage%40yahoogroups.com><iolanguage%
> 40yahoogroups.com>, Tres
> > > Walsh <tres.walsh@> wrote:
> > > >
> > > > Thanks for your responses. I perhaps I should have been a little
> clearer.
> > > > With my project, it is less of an addon, and is actually a Game
> Engine,
> > > > where Io is going to be used for the scripting elements. So I really
> am
> > > > unsure if I treat it as another addon? or something completely
> different?
> > >
> > > I'm actually a little unsure of that myself. I initially envisioned
> this:
> > >
> > > My game exe -loads-> IoVM -runs-> script
> > >
> > > But I didn't have any examples of how to actually do that. I've shifted
> to
> > > this model:
> > >
> > > Io.exe -loads-> my game addon
> > >
> > > but only because I haven't got any examples of how to do the former, so
> I'm
> > > doing the latter because I have examples of addons to base it off of.
> > >
> > > If I did the first model, where I have a game exe that creates an IoVM
> > > state, how is the vm supposed to call back to my C++ functions? It
> would be
> > > something like this:
> > >
> > > My game exe -loads-> IoVM -load addon-> A DLL with my C++ Io bindings
> > > -then-> Does what? Connects back to the data in the EXE? How does it
> make
> > > this connection?
> > >
> > > It seems like it ought to be possible to have this all under one static
> exe
> > > without dynamic libraries but HOW? I mean the addon loading processes
> > > specifically loads DLLs. How would you get it to link to functions
> within
> > > the same exe without loading the DLL?
> > >
> > >
> > >
> >
> >
> >
> > --
> > http://treswalsh.com
> >
>
>  
>

--
http://treswalsh.com

[Steve Dekorte]

On 2009-10-04, at 4:37 PM, Tres Walsh wrote:

> Well this is what the wikibooks tutorial does, with each method in  
> the IoGlue class being static
>
> However when I perform this such as
>
> static IoObject* proto(IoState* state)
> {
>      IoObject* self = IoObject_new(state);
>      self->tag = tag(state, "Nameof_IoObject");
>      self->data = 0; //This is where an error happen D:
> }
>
> We get an error that IoObject has no member named data. Which is  
> partially true, since it is IoObjectData that contains that member,  
> and IoObject is just a typedef for CollectorMarker (or something  
> similar)

That tutorial was written before all references in Io were made  
indirect. self->data has been replaced by the IoObject_dataPointer
(self) macro.

I recommend ignoring the tutorial and looking at Io's wrappers for  
it's own data structures (like IoMap.c, etc). Io is unusual in that  
addon object operate at exactly the same level that it's own data  
structures do. Io has no primitive byte code ops vs function calls.  
Instead, everything is a message send and some messages are  
implemented as C functions.

[Steve Dekorte]

On 2009-10-04, at 12:04 PM, Tres Walsh wrote:
> You can see the code here: http://sahchandler.pastey.net/126073
> So to create the IoVector3 class what would be the FIRST thing we do?

I threw this together by copying the MD5 addon and replacing a few  
bits. I didn't compile it - just meant to show the basics.

[Tres Walsh]

Well that definitely makes more sense. I'm going to try a few things,
and if they work I'll be sure to reply to this message.

Thanks for your help so far Steve (and dennis :D )

On Sun, Oct 4, 2009 at 4:50 PM, Steve Dekorte <steve@...> wrote:

>
> On 2009-10-04, at 4:37 PM, Tres Walsh wrote:
>> Well this is what the wikibooks tutorial does, with each method in
>> the IoGlue class being static
>>
>> However when I perform this such as
>>
>> static IoObject* proto(IoState* state)
>> {
>>      IoObject* self = IoObject_new(state);
>>      self->tag = tag(state, "Nameof_IoObject");
>>      self->data = 0; //This is where an error happen D:
>> }
>>
>> We get an error that IoObject has no member named data. Which is
>> partially true, since it is IoObjectData that contains that member,
>> and IoObject is just a typedef for CollectorMarker (or something
>> similar)
>
> That tutorial was written before all references in Io were made
> indirect. self->data has been replaced by the IoObject_dataPointer
> (self) macro.
>
> I recommend ignoring the tutorial and looking at Io's wrappers for
> it's own data structures (like IoMap.c, etc). Io is unusual in that
> addon object operate at exactly the same level that it's own data
> structures do. Io has no primitive byte code ops vs function calls.
> Instead, everything is a message send and some messages are
> implemented as C functions.
>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>

--
http://treswalsh.com

[Tres Walsh]

So my absolutely godawful idea of translating the example you [Steve]
gave me to c++. And I've hit a tad of a snag. Now I'm going to imagine
this problem has a simple solution, however I rarely typecast. SO. the
function in question is:



IoObject* IoVector3::New(IoState* state)
{

        IoObject* proto = IoState_protoWithInitFunction_(state, &IoVector3::Proto);
        return IOCLONE(proto);
}

Which elicits the following error:

error: invalid conversion from 'IoObject* (*)(IoState*)' to 'IoObject*
(*)(void*)'

Now if I attempt to perform a reinterpret_cast, or just a plain
(void*) type cast, it changes the invalid conversion to void* (or
IoObject*), instead of IoObject*(*)(IoState*)

So my question is what the heck is going on here? Why is the void* not
accepting the IoState*? Is this a c++ thing? I got this same thing to
work with the tag functions (for example: IoTag_cloneFunc_(tag,
reinterpret_cast<IoTagCloneFunc*>(&IoVector3::RawClone)); )

I know I've been asking alot, but I really do appreciate all the help
I'm getting :)

On Sun, Oct 4, 2009 at 6:54 PM, Tres Walsh <tres.walsh@...> wrote:

> Well that definitely makes more sense. I'm going to try a few things,
> and if they work I'll be sure to reply to this message.
>
> Thanks for your help so far Steve (and dennis :D )
>
> On Sun, Oct 4, 2009 at 4:50 PM, Steve Dekorte <steve@...> wrote:
>>
>> On 2009-10-04, at 4:37 PM, Tres Walsh wrote:
>>> Well this is what the wikibooks tutorial does, with each method in
>>> the IoGlue class being static
>>>
>>> However when I perform this such as
>>>
>>> static IoObject* proto(IoState* state)
>>> {
>>>      IoObject* self = IoObject_new(state);
>>>      self->tag = tag(state, "Nameof_IoObject");
>>>      self->data = 0; //This is where an error happen D:
>>> }
>>>
>>> We get an error that IoObject has no member named data. Which is
>>> partially true, since it is IoObjectData that contains that member,
>>> and IoObject is just a typedef for CollectorMarker (or something
>>> similar)
>>
>> That tutorial was written before all references in Io were made
>> indirect. self->data has been replaced by the IoObject_dataPointer
>> (self) macro.
>>
>> I recommend ignoring the tutorial and looking at Io's wrappers for
>> it's own data structures (like IoMap.c, etc). Io is unusual in that
>> addon object operate at exactly the same level that it's own data
>> structures do. Io has no primitive byte code ops vs function calls.
>> Instead, everything is a message send and some messages are
>> implemented as C functions.
>>
>>
>>
>> ------------------------------------
>>
>> Yahoo! Groups Links
>>
>>
>>
>>
>
>
>
> --
> http://treswalsh.com
>

--
http://treswalsh.com

[Tres Walsh]

Sorry for yet another email. I solved the problem so you cna disregard
the previous message 8D

I continue on my perilous journey~

On Sun, Oct 4, 2009 at 9:06 PM, Tres Walsh <tres.walsh@...> wrote:

> So my absolutely godawful idea of translating the example you [Steve]
> gave me to c++. And I've hit a tad of a snag. Now I'm going to imagine
> this problem has a simple solution, however I rarely typecast. SO. the
> function in question is:
>
>
>
> IoObject* IoVector3::New(IoState* state)
> {
>
>        IoObject* proto = IoState_protoWithInitFunction_(state, &IoVector3::Proto);
>        return IOCLONE(proto);
> }
>
> Which elicits the following error:
>
> error: invalid conversion from 'IoObject* (*)(IoState*)' to 'IoObject*
> (*)(void*)'
>
> Now if I attempt to perform a reinterpret_cast, or just a plain
> (void*) type cast, it changes the invalid conversion to void* (or
> IoObject*), instead of IoObject*(*)(IoState*)
>
> So my question is what the heck is going on here? Why is the void* not
> accepting the IoState*? Is this a c++ thing? I got this same thing to
> work with the tag functions (for example: IoTag_cloneFunc_(tag,
> reinterpret_cast<IoTagCloneFunc*>(&IoVector3::RawClone)); )
>
> I know I've been asking alot, but I really do appreciate all the help
> I'm getting :)
>
> On Sun, Oct 4, 2009 at 6:54 PM, Tres Walsh <tres.walsh@...> wrote:
>> Well that definitely makes more sense. I'm going to try a few things,
>> and if they work I'll be sure to reply to this message.
>>
>> Thanks for your help so far Steve (and dennis :D )
>>
>> On Sun, Oct 4, 2009 at 4:50 PM, Steve Dekorte <steve@...> wrote:
>>>
>>> On 2009-10-04, at 4:37 PM, Tres Walsh wrote:
>>>> Well this is what the wikibooks tutorial does, with each method in
>>>> the IoGlue class being static
>>>>
>>>> However when I perform this such as
>>>>
>>>> static IoObject* proto(IoState* state)
>>>> {
>>>>      IoObject* self = IoObject_new(state);
>>>>      self->tag = tag(state, "Nameof_IoObject");
>>>>      self->data = 0; //This is where an error happen D:
>>>> }
>>>>
>>>> We get an error that IoObject has no member named data. Which is
>>>> partially true, since it is IoObjectData that contains that member,
>>>> and IoObject is just a typedef for CollectorMarker (or something
>>>> similar)
>>>
>>> That tutorial was written before all references in Io were made
>>> indirect. self->data has been replaced by the IoObject_dataPointer
>>> (self) macro.
>>>
>>> I recommend ignoring the tutorial and looking at Io's wrappers for
>>> it's own data structures (like IoMap.c, etc). Io is unusual in that
>>> addon object operate at exactly the same level that it's own data
>>> structures do. Io has no primitive byte code ops vs function calls.
>>> Instead, everything is a message send and some messages are
>>> implemented as C functions.
>>>
>>>
>>>
>>> ------------------------------------
>>>
>>> Yahoo! Groups Links
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> http://treswalsh.com
>>
>
>
>
> --
> http://treswalsh.com
>

--
http://treswalsh.com

[dennisf486]

Sorry for the confusion:  the static keyword applied to functions in C++ actually has nothing to do with the term "static" in the context of "static compilation" vs. "dynamic loading".  When I was using the word static I just meant that having everything in one EXE is slightly better than a constellation of DLLs (because, as Steve said, there are some issues with most's OS's implementation of dynamic libraries with C++).

The keyword static as applied to functions in C++ just means that it's a class-level method intead of an object level method.  It just allows you to call the function directly without needing to create an instance object of your class first.  (Which is how it needs to be because Io isn't going to create your object for you, it's just going to call the function.)  It's the C++ equivalent of the Visual Basic "Shared" keyword.

Just to confuse matters, the keyword static as applied to C functions (or namespace level functions in C++) has yet a different meaning altogether!  I forget just what it does, maybe makes the function local to the compilation unit, or public to it, whatever.  It's actually likely to NOT do what you think it does.

Technically we should be declaring our C++ functions that will be callable from Io as 'extern "C"' (though that can only be applied to namespace, not class, level functions).  That does 2 things:  it ensures a calling convention compatible with C, and doesn't do name mangling on the function name.  It's actually unfortunate that it does 2 things at once, because extern "C" ends up disabling a lot of other C++ features in the process, like overloading, when sometimes all you might want is just for it to support C's parameter passing mechanism even if you don't need the name visible from C++.  In practice, 99% of C++ compilers are going to use the same calling convention for a static class level function as for a C function, so you're usually safe using the keyword static but the standard doesn't guarantee it (I wish it did).  You do however need to declare the entry point for the dll as extern "C" because it is looked up by name, but if you don't use the addon model then you don't even need to worry about that.


--- In iolanguage@..., Tres Walsh <tres.walsh@...> wrote:

>
> Well this is what the wikibooks tutorial does, with each method in the
> IoGlue class being static
>
> However when I perform this such as
>
> static IoObject* proto(IoState* state)
> {
>      IoObject* self = IoObject_new(state);
>      self->tag = tag(state, "Nameof_IoObject");
>      self->data = 0; //This is where an error happen D:
> }
>
> We get an error that IoObject has no member named data. Which is partially
> true, since it is IoObjectData that contains that member, and IoObject is
> just a typedef for CollectorMarker (or something similar)
>
> What I think might work is, creating your IoVM in a separate thread, but,
> manually calling all of the protos for each class? If you're using a custom
> build system, I'm sure you could generate a "small"(in the sense of what it
> does on a grand scale) function that passes the IoVM State to all the
> protos, so by the time the game is up and running, everything has been
> properly initialized. (I think this is how the Unreal Engine does it, which
> would explain for the 30-60 second startup times where you're just staring
> at a splash screen.)
>
> The problem still remains of running the "main" io script, although
> theoretically, if you needed to you could just run
> IoState_doCString_(self,"doFile(\"scriptName.io\")"); :)
>
> On Sun, Oct 4, 2009 at 3:41 PM, dennisf486 <dennisf486@...> wrote:
>
> >
> >
> > Well after I wrote that I had something of an epiphany.
> >
> > I had been thinking in terms of the Io code calling functions in our exe
> > and "how would it even know what functions to call, if it hadn't been
> > compiled together with my code?"
> >
> > But what I just realized is that Io calls our C/C++ functions ***using a
> > pointer***. The fact that it loads a DLL first when loading an addon is just
> > how it gets some code loaded; but if the code were already statically
> > present inside the exe you could skip the step of loading a DLL entirely.
> > All you really have to do, is call the right methods to perform the right
> > operations on your IoVM state to shove an Io proto in there that has
> > function pointers that point to C++ functions in your exe. Just because you
> > *can* use the addon loading process to do it doesn't mean that you *must*
> > use that process. Instead of loading an addon, you could build the proto "by
> > hand" in your exe in C/C++ code, put the addresses of some of your C++
> > functions into the proto's method table, and put the address of your C++
> > game state object into the proto's data pointer. Get that proto into your Io
> > state (somehow) and voila! Your script is bound to your C++ data and
> > functions without addons being involved at all.
> >
> > I'm not sure on the details of how to do it but I think this must be in
> > general how it's done. It has some advantages over the addon based
> > architecture: you can structure your project how you want instead of being
> > forced to make it a DLL, and less chance of a deployment problem (everything
> > in one exe so no need to worry about DLLs not being in right directory). You
> > could probably even statically link the Io VM and not need that as a DLL
> > either. Plus, when it comes to the main game engine being in a DLL I was a
> > little worried someone might hack it by writing a shim DLL that sits between
> > the game engine and the script and monitors everything that goes between
> > them. I know that putting everything in one exe doesn't magically make it
> > more secure, but somehow the idea of having my whole game be a constellation
> > of DLLs seems almost inviting trouble.
> >
> > --- In iolanguage@... <iolanguage%40yahoogroups.com>, Tres
> > Walsh <tres.walsh@> wrote:
> > >
> > > This is also something that I've been pondering. You're first example is
> > how
> > > the Unreal Engine does it I believe, with the second being a more Source
> > > Engine approach.
> > >
> > > It sounds like we may have to hack the Io source to get it to work the
> > way
> > > we want? I just don't know :-/
> > >
> > > On Sun, Oct 4, 2009 at 12:40 PM, dennisf486 <dennisf486@> wrote:
> > >
> > > > --- In iolanguage@... <iolanguage%40yahoogroups.com><iolanguage%
> > 40yahoogroups.com>, Tres
> > > > Walsh <tres.walsh@> wrote:
> > > > >
> > > > > Thanks for your responses. I perhaps I should have been a little
> > clearer.
> > > > > With my project, it is less of an addon, and is actually a Game
> > Engine,
> > > > > where Io is going to be used for the scripting elements. So I really
> > am
> > > > > unsure if I treat it as another addon? or something completely
> > different?
> > > >
> > > > I'm actually a little unsure of that myself. I initially envisioned
> > this:
> > > >
> > > > My game exe -loads-> IoVM -runs-> script
> > > >
> > > > But I didn't have any examples of how to actually do that. I've shifted
> > to
> > > > this model:
> > > >
> > > > Io.exe -loads-> my game addon
> > > >
> > > > but only because I haven't got any examples of how to do the former, so
> > I'm
> > > > doing the latter because I have examples of addons to base it off of.
> > > >
> > > > If I did the first model, where I have a game exe that creates an IoVM
> > > > state, how is the vm supposed to call back to my C++ functions? It
> > would be
> > > > something like this:
> > > >
> > > > My game exe -loads-> IoVM -load addon-> A DLL with my C++ Io bindings
> > > > -then-> Does what? Connects back to the data in the EXE? How does it
> > make
> > > > this connection?
> > > >
> > > > It seems like it ought to be possible to have this all under one static
> > exe
> > > > without dynamic libraries but HOW? I mean the addon loading processes
> > > > specifically loads DLLs. How would you get it to link to functions
> > within
> > > > the same exe without loading the DLL?
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > http://treswalsh.com
> > >
> >
> >  
> >
>
>
>
> --
> http://treswalsh.com
>

[Erik Max Francis]

dennisf486 wrote:
> Sorry for the confusion:  the static keyword applied to functions in C++ actually has nothing to do with the term "static" in the context of "static compilation" vs. "dynamic loading".  When I was using the word static I just meant that having everything in one EXE is slightly better than a constellation of DLLs (because, as Steve said, there are some issues with most's OS's implementation of dynamic libraries with C++).
>
> The keyword static as applied to functions in C++ just means that it's a class-level method intead of an object level method.  It just allows you to call the function directly without needing to create an instance object of your class first.  (Which is how it needs to be because Io isn't going to create your object for you, it's just going to call the function.)  It's the C++ equivalent of the Visual Basic "Shared" keyword.
>
> Just to confuse matters, the keyword static as applied to C functions (or namespace level
> functions in C++) has yet a different meaning altogether!  I forget just what it does, maybe
> makes the function local to the compilation unit, or public to it, whatever.  It's actually
> likely to NOT do what you think it does.

There's actually yet another definition of `static` in C and C++ which
you've missed.  It's used for variables inside function.  It's the
opposite of `auto`, something most people don't even realize is a keyword.

> Technically we should be declaring our C++ functions that will be callable from Io as
> 'extern "C"' (though that can only be applied to namespace, not class, level functions).  
> That does 2 things:  it ensures a calling convention compatible with C, and doesn't do name
> mangling on the function name.  It's actually unfortunate that it does 2 things at once, because
> extern "C" ends up disabling a lot of other C++ features in the process, like overloading, when
> sometimes all you might want is just for it to support C's parameter passing mechanism even if
> you don't need the name visible from C++.

You also want it to be _accessible_, which means you need to turn off
name mangling as well.  Thus either one without the other isn't much use
if C calling capability is your goal.  Overloading is a good inadvertent
example -- it distinguishes between the different varieties via name
mangling!

> In practice, 99% of C++ compilers are going to use
> the same calling convention for a static class level function as for a C function, so you're
> usually safe using the keyword static but the standard doesn't guarantee it (I wish it did).

Even if you were guaranteed the same calling convention (which, as you
say, you aren't), it still wouldn't work; the C++ static member function
would have a implementation-defined, mangled name, so you couldn't get
at it from the C side.

--
Erik Max Francis && max@... && http://www.alcyone.com/max/
  San Jose, CA, USA && 37 18 N 121 57 W && AIM/Y!M/Skype erikmaxfrancis
   Can I lay with you / As your wife
    -- India Arie

[dennisf486]

--- In iolanguage@..., Erik Max Francis <max@...> wrote:
> There's actually yet another definition of `static` in C and C++ which
> you've missed.  It's used for variables inside function.  It's the
> opposite of `auto`, something most people don't even realize is a keyword.

Right.  I left that out because I was only talking about variables, only functions, but to be 100% complete yes there's static variables too.

The old auto keyword is (has been? will be?) deprecated in C++; the new C++0x standard is going to reuse the keyword for the much more useful purpose of allowing you to declare a variable without having to say its type.  (It will take its type from the first thing you assign to it.)

> > That does 2 things:  it ensures a calling convention compatible with C, and doesn't do name
> > mangling on the function name.
>
> You also want it to be _accessible_, which means you need to turn off
> name mangling as well.

I thought I said that already?

> Even if you were guaranteed the same calling convention (which, as you
> say, you aren't), it still wouldn't work; the C++ static member function
> would have a implementation-defined, mangled name, so you couldn't get
> at it from the C side.

Uh no, wrong, actually it does work.  I have code where it works and the C++ binding example at http://en.wikibooks.org/wiki/Io_Programming/Binding_Io_to_C%2B%2B does too.  When C code calls a function *by pointer* the name doesn't matter.  Here's how it works: you are in your C++ file, you use the name of a C++ function to get its address, and you put that address into a function pointer field in an Io tag struct.  The Io C code reads that pointer out of the struct, and calls the function using the value of that pointer.   At no time does any C code need to "get at" the name; it is only called by pointer.

The only function you need to have an extern "C"/unmangled name for is just the entry point to the DLL if it's an addon, because Io is looking in the DLL for one specific name in that case.  But that only comes up if you are making an addon.

[Erik Max Francis]

dennisf486 wrote:
> The old auto keyword is (has been? will be?) deprecated in C++; the new C++0x standard
> is going to reuse the keyword for the much more useful purpose of allowing you to declare
> a variable without having to say its type.  (It will take its type from the first thing
> you assign to it.)

It's not deprecated in any C++ Standard or draft Standard I'm aware of.
  C++0x uses it for other things as well, but doesn't deprecate its
original use.  Plus there's also `register`, if you really want to list
them all.

> Uh no, wrong, actually it does work.  I have code where it works and the C++ binding example
> at http://en.wikibooks.org/wiki/Io_Programming/Binding_Io_to_C%2B%2B does too.  When C code
> calls a function *by pointer* the name doesn't matter.  Here's how it works: you are in your
> C++ file, you use the name of a C++ function to get its address, and you put that address
> into a function pointer field in an Io tag struct.  The Io C code reads that pointer out
> of the struct, and calls the function using the value of that pointer.   At no time does
> any C code need to "get at" the name; it is only called by pointer.

Well, that's fine, but it's still not guaranteed to work, because it's
not `extern "C"` and thus has no obligation to use the same calling
conventions.  That "it works" and doesn't crash on one particular
implementation doesn't mean that it's Standard confirming and that it
won't work on some perfectly good implementations.

> The only function you need to have an extern "C"/unmangled name for is just the entry point
> to the DLL if it's an addon, because Io is looking in the DLL for one specific name in that
> case.  But that only comes up if you are making an addon.

Not just unmangled, but `extern "C"`.  You need the entry point to have
the right calling conventions, too.  Which is why, as I said, it's not
much good to separate out the two features of what `extern "C"` does
which you were advocating before.

The proper Standard way is to make an `extern "C"` function reference
your real C++ static member functions, and then expose that.  Playing
with static member function pointers doesn't fix the calling convention
problem.

--
Erik Max Francis && max@... && http://www.alcyone.com/max/
  San Jose, CA, USA && 37 18 N 121 57 W && AIM/Y!M/Skype erikmaxfrancis
   Whoever named it necking was a poor judge of anatomy.
    -- Groucho Marx

[dennisf486]

--- In iolanguage@..., Erik Max Francis <max@...> wrote:
> It's not deprecated in any C++ Standard or draft Standard I'm aware of.

Then you need to read http://www.open-std.org/jtc1/sc22/wg21/docs papers/2008/n2546.htm "N2546 Removal of auto as a storage class specifier."

--- In iolanguage@..., Erik Max Francis <max@...> wrote:
> Not just unmangled, but `extern "C"`.  You need the entry point to have
> the right calling conventions, too.  Which is why, as I said, it's not
> much good to separate out the two features of what `extern "C"` does
> which you were advocating before.

Advocating separating the two effects of extern "C" was just an offhand remark and you're making me wish I hadn't mentioned it.  But if you must know, I in fact have a specific situation where I need the extern "C" calling convention but I still need to keep name mangling.

The binding library I'm working on uses C++ templates to generate code for functions callable from Io (by pointer).  You can't declare a function that is both a template and extern "C" because templates require name mangling.  But it seems reasonable to expect a template function to still work using the C calling convention, even if it doesn't have the C naming convention, because after all static class member template functions already happen to use the C calling convention in most implementations.

That's why I said it's a shame they conflated the two concepts in the standard.

> The proper Standard way is to make an `extern "C"` function reference
> your real C++ static member functions, and then expose that.  

Right but the problem I have is that the template code generates an unlimited number of instantiations of the static member function, one for each user-defined type that is registered with the library.  You can't cover that with a single extern "C" function because what is inside that extern "C" function would need to change depending on template parameters.

But even if I succeeded in coming up with a trick to call unlimited variations of a template function from a single extern "C" function (perhaps using pointers and callbacks), it still wouldn't work.  For some reason Io enforces that you have to register a unique proto() function for each Io proto you introduce from an addon.  So if I had only one extern "C" function and the user registered two different types to make Io protos out of, I couldn't register the same proto function for both of them.

With static member templates on the other hand, since each instantiation of a template with new types creates an actual new function to be emitted by the compiler, I can get a unique proto() function with a unique address for each type the user registers.

> Playing
> with static member function pointers doesn't fix the calling convention
> problem.

De jure says it isn't guaranteed, but de facto it works on the vast majority of compilers.  But I already said that.  So I'm not sure what you're getting at.  Are you still arguing that calling C++ static class member functions from Io doesn't "work" (in the practical sense) or are you arguing that it's morally incorrect to rely on it to work?

Given the choice between slightly-nonstandard code that works and standards compiliant code that doesn't do what I need it to, I see no reason give up on a very useful technique just because some compiler on some OS some time in the future MIGHT not support it.