Creating a DSL within Io for manipulating sets of C++ objects

[dennisf486]

I want to preface this by saying that I'm only asking about the Io-specific part of what I'm doing here, so don't be put off by the next two paragraphs.  It's just that if I don't give you some background on what I'm doing, you won't know what the heck I'm talking about.

My LikeMagic C++ binding library is going to be used in a game engine, and the better its performance, the more of the game I can script in Io instead of hardcoding in C++.  I discovered that due to the details of how the C++ functions are invoked by LikeMagic I can actually apply the same C++ function over a whole collection of objects just as easily as I can apply it to just one object.  I call this feature "SFMO" (pronounced "SwiFt-MO") which stands for "Single Function, Multiple Objects".  Just as SIMD (single instruction, multiple data) speeds up CPU hardware by applying the same CPU instruction over a vector of data elements, SFMO makes game object scripting more efficient because one invocation from script applies a single member function over an entire std::vector of C++ objects in one atomic operation (atomic from Io's point of view; there is a for-loop is on the C++ side).  The speed benefit is that you can direct *what* is to be performed in Io script, but the actual execution of that operation is done on the C++ side at nearly the full speed of C++ code.  It allows you to script the operations at a much more granular level without any speed penalty.

Originally I had envisioned an SQL or LINQ syntax for expressing this in script, such as this Io code:

ResultSet := From(C_plus_plus_Objects) Update(obj) Set(obj pos += vel)

What SFMO does is that "obj pos += vel" does NOT get evaluated by Io.  Instead, the expression results in a query plan that instructs the C++ side to perform the analogous operation on all obj in C_plus_plus_Objects.

A friend of mine suggested basing it off jquery syntax instead.  For those who don't know, jquery is just a really convenient Javascript library for interacting with web page objects; it exposes a function named "$" (the name is short and cryptic to make it quick to type).  Like LINQ, jquery allows you to operate on sets of objects.

The jquery-inspired form of the above would probably look like:

ResultSet := $(C_plus_plus_Objects) each(obj, obj pos += vel)

So here's my questions:

Which syntax do you in your personal opinions like better, LINQ/SQL inspired or jquery inspired?

I notice that Io does let you create a method with the name "$", so is there any reason not to use $ for my method?  (Possibly I don't even need the $() if C_plus_plus_Objects defines the each() method.)

I notice that Io does not have +=, but I found it's certainly possible to define a += method on my objects.  Is there a specific reason Io doesn't define += already?

Since C++ objects can overload operators, I'd like to be able to support all the overloaded C++ operators in my query syntax.  For instance, in the example above, the variable pos is of C++ class Point, and pos += vel should call the C++ function Point::operator +=(const Point& p).  I'd also like to be able to write (in Io but only used with SFMO): "foo ++" (send ++ message to foo) and "++ foo" (send foo message to ++) to invoke C++'s pre- and post-increment operators.  Again, note that these aren't going to directly do these operations in the Io side, they're just going to help build a query plan that tells LikeMagic to perform the action on the C++ side!  Will I run into any problems with this?  Are there any symbols that I *can't* use as method names in Io?

You may notice the "each" method in the jquery inspired form is a lot like the "method" method in Io, that is I want to pass the arguments and then the last part is the actual code.  Can I make "each()" have the same syntax as "method()" or is Io's "method()" somehow special?

Finally, is there any way to detect whether or not the ultimate result of a whole expression is going to be used or not?  In the example:

ResultSet := From(C_plus_plus_Objects) Update(obj) Set(obj pos += vel)

it will create a C++ vector object that will be stored in ResultSet.  Later you could also use that ResultSet in another SFMO statement, as in From(ResultSet) <do something else>.  However, a significant amount of overhead could be saved if the result set doesn't need to be generated at all.  For example, in:

From(C_plus_plus_Objects) Update(obj) Set(obj pos += vel)

we're just updating the positions and the result of the line is discarded.  If the SFMO expression could somehow detect whether or not its return value is to be assigned to anything or is ever used, then it could determine whether or not to generate a result set as it runs.

Possibly I could return some kind of proxy object, similar in concept to a block or a future, and then only do the operation if/when the block or future's result is used.  However, in the case when the result set really isn't need but the C++ objects are updated, then I would need to call some kind of activate() or invoke() method on the proxy to tell it "go ahead and run now".

In fact, I think I might actually want to tie in to Io's futures facility to perform SFMO evaluations.  That is, I'd like to monkey patch Futures in Io to allow the same syntax to work on SFMO set operations as for concurrent operations.  Is this feasible?  (I still want futures to work for concurrency too; I just mean that when a Future is applied to a SFMO object, I want that to mean obtain a proxy for a SFMO operation instead of a proxy for a coroutine operation.)

[Samuel A. Falvo II]

I cannot speak for your other questions, but these are answers I know
for a subset of them.

On Tue, Aug 4, 2009 at 9:23 AM, dennisf486<dennisf486@...> wrote:
> Which syntax do you in your personal opinions like better, LINQ/SQL inspired or jquery inspired?

I prefer JQuery instead, because it communicates more clearly what's
happening.  SQL is domain-specific towards tables, and one need look
no further than SQL as an example of a syntax gone out of control.

> I notice that Io does let you create a method with the name "$", so is there any reason not to use $ for my method?  (Possibly I don't even need the $() if C_plus_plus_Objects defines the each() method.)

I'd not use $ only because JQuery _does_ use $.  Thus, if anyone wants
to make an Io/Javascript bridge to use, or maybe even server-side
compile JQuery JS code, they'll have to find a substitute to $.

(Not likely to be a critical issue, but it's more a courtesy thing
than a technical thing.)

> Will I run into any problems with this?

Yes; Io is strictly left-to-right in parsing and evaluation, so you'll
find "++ obj" quite impossible, as you expect it to work.  The only
way around this is to write your own parser.  This is doable since Io
passes entire abstract syntax trees as easily as it passes objects and
values, but are you sure you want to invest that kind of pain and
suffering?

> You may notice the "each" method in the jquery inspired form is a lot like the "method" method in Io, that is I want to pass the arguments and then the last part is the actual code.  Can I make "each()" have the same syntax as "method()" or is Io's "method()" somehow special?

Nope.  You're perfectly free to define syntax just like method(),
if(), or for(), etc.  Io's syntax is specifically built for this.

> we're just updating the positions and the result of the line is discarded.  If the SFMO expression could somehow detect whether or not its return value is to be assigned to anything or is ever used, then it could determine whether or not to generate a result set as it runs.

Not possible, unfortunately.

--
Samuel A. Falvo II

[dennisf486]

Thanks for taking the time to grok my questions, Samuel.  As for not being able to implement the ++ operator:  I can at least create a ++() method which supports a "++(foo)" syntax.  As for "++ foo" syntax (without parens), I got the syntax to parse but it didn't have the result I intended.  Here's an excerpt from an Io session:

The post increment operator, pretty straightforward:
Io> obj := Object clone do( value := 1 )
Io> obj ++ := method(temp := value; value = value+1; temp)

Proving post increment works:
Io> obj ++
==> 1
Io> obj value
==> 2

Implementing the pre increment operator as an object that takes any message:
Io> ++ := Object clone
Io> ++ message := method(m, m value = m value + 1)

Testing the pre increment operator
Io> ++ obj

Here's my problem:
Expected "3" got this instead!
==>  Object_0x1073ea0:
  ++               = method(...)
  value            = 2

And it did not affect obj value:
Io> obj value
==> 2

So that doesn't work.  What about making ++ a method?
Io> ++ = method(obj, obj value = obj value + 1)
Io> ++(obj)
==> 3

That worked.

So I can implement ++ as long as the syntax is "++(obj)" instead of "++ obj".  I'm not sure why my implementation of the "++ message" method failed to work.  I tried implementing the slot "++ forward" as well, but that didn't work either.