Scala back-end

[Etienne M. Gagnon]

Hi Lucas,

Thanks for the contribution. I've added it to the repository.

So, what can you tell us about the retargeting experience? Also, do you
have any feedback about the ObjectMacro system (e.g. its syntax, the use
of its generated classes, etc.)?

Etienne

--
Etienne M. Gagnon, Ph.D.
SableCC:                                            http://sablecc.org




_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion

[Lucas Satabin]

Hi Etienne,

It was interesting to add this new target. My feedbacks are the following:

 [*] Adding a new .objectmacro file for an object oriented programming language as Scala was not that hard, by adapting the existing code for Java (It should be even easier for C#).
 [*] For the re-targeting itself, it is not really flexible and we really need to modify the existing code (the target check, the instantiation of the code generator). I think it could be interesting to create a more modular system to easily add new targets without modifying the main code. For example, by registering the new back-ends as plugins, it would avoid to recompile the entire project when adding a new back-end. However it is not hard to add the target, there three places where we need to add the target name: help display, target check, code generator instantiation
 [*] The ObjectMacro is really powerful and easy to understand and to use. The syntax is quite intuitive, and the generated code is also easy to use when we know the structure of the .objectmacro file (which should be the case for people using the classes...).

I hope this email will incite other people to implement new back-ends and help me to improve the Scala back-end.

Enjoy, and thanks to Etienne for the help and advices

Lucas

Etienne M. Gagnon schrieb:

Hi Lucas,

Thanks for the contribution. I've added it to the repository.

So, what can you tell us about the retargeting experience? Also, do you
have any feedback about the ObjectMacro system (e.g. its syntax, the use
of its generated classes, etc.)?

Etienne

  

_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion
  

_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion

[Etienne M. Gagnon]

Hi Lucas,

Thanks for the feedback. See my comment below.

>  [*] For the re-targeting itself, it is not really flexible and we
> really need to modify the existing code (the target check, the
> instantiation of the code generator). I think it could be interesting
> to create a more modular system to easily add new targets without
> modifying the main code. For example, by registering the new back-ends
> as plugins, it would avoid to recompile the entire project when adding
> a new back-end. However it is not hard to add the target, there three
> places where we need to add the target name: help display, target
> check, code generator instantiation

I initially wanted to go with a plugin architecture, as you are
suggesting (see: http://sablecc.org/wiki/CodeGeneration#EtiennesSolution
), but I finally decided to take the more flexible approach of having a
well-defined textual intermediate representation, instead. Let me
explain my motivation.

The main object of a plugin architecture is to allow third-party
developers to create independent targets and be able to use them without
modifying SableCC at all.

Now, let's look at the current architecture (intermediate
representation): It supports creating independent targets without
modifying SableCC by providing a parsable intermediate representation
and a grammar (with CST->AST simplifications) for parsing it. It even
supports creating these independent targets in a language of choice (as
long as it is targeted by SableCC and ObjectMacro).

A plugin architecture, on the other hand, opens a whole can of worms, as
far as configuration goes: Should it support a default system-wide
plugin directory? If yes, can it be implemented so that it works
seamlessly on any platform (Unix, Windows, other) without requiring
SableCC developers to know about these systems? Where do should the
configuration file be stored? Ot should the plugin directory location be
hardcoded within SableCC source code?

Really, I see no gain to the plugin architecture. It simply adds
configuration headaches. Even worse: it makes it quite hard to create an
independent target in any language other than Java. The current
architecture has no need to hardcode any directory (other than using the
"current" directory as default destination), and allows for using other
languages (or, it will once SableCC targets new languages).


Now, for integrating a target within SableCC, the current architecture
does require adding/modifying two lines of code within the existing
code. Of course, in this case, the target must also be written in Java.
But, other than these two lines of code, no other modification is
needed. Note that the exact same code can be used for implementing both
an independent target and an integrated target.

I really don't mind modifying these two lines of code, when a new target
is to be integrated. What's currently missing, I think, is a template
for people to use to develop an independent target, e.g. a main class
which demonstrates implementing an independent target using the existing
Java target, and some wiki documentation explaining how things work.

Anyway, that's the result of a long reflection I've put into it before
choosing and implementing the current architecture.

Have fun!

Etienne

--
Etienne M. Gagnon, Ph.D.
SableCC:                                            http://sablecc.org




_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion

[Lucas Satabin]

Hi Etienne,

I understand your point of view and I agree with your argumentation. I did not mean that it was hard to add the two lines of code, it is just that I was not sure if I added the lines everywhere where it is needed. It did not take me too long to find the lines to modify, just document these places could help people to be sure that they did all what is needed to add the target.
I think I did not understand all the advantages of this concept, but I will surely learn more by exploring the code. A user/developer documentation would be great, and I can help you to write it if you need (after I am sure I really understood everything...)

I am impatient to test this new SableCC version!

Lucas

Etienne M. Gagnon schrieb:

Hi Lucas,

Thanks for the feedback. See my comment below.

  

 [*] For the re-targeting itself, it is not really flexible and we
really need to modify the existing code (the target check, the
instantiation of the code generator). I think it could be interesting
to create a more modular system to easily add new targets without
modifying the main code. For example, by registering the new back-ends
as plugins, it would avoid to recompile the entire project when adding
a new back-end. However it is not hard to add the target, there three
places where we need to add the target name: help display, target
check, code generator instantiation
    

I initially wanted to go with a plugin architecture, as you are
suggesting (see: http://sablecc.org/wiki/CodeGeneration#EtiennesSolution
), but I finally decided to take the more flexible approach of having a
well-defined textual intermediate representation, instead. Let me
explain my motivation.

The main object of a plugin architecture is to allow third-party
developers to create independent targets and be able to use them without
modifying SableCC at all.

Now, let's look at the current architecture (intermediate
representation): It supports creating independent targets without
modifying SableCC by providing a parsable intermediate representation
and a grammar (with CST->AST simplifications) for parsing it. It even
supports creating these independent targets in a language of choice (as
long as it is targeted by SableCC and ObjectMacro).

A plugin architecture, on the other hand, opens a whole can of worms, as
far as configuration goes: Should it support a default system-wide
plugin directory? If yes, can it be implemented so that it works
seamlessly on any platform (Unix, Windows, other) without requiring
SableCC developers to know about these systems? Where do should the
configuration file be stored? Ot should the plugin directory location be
hardcoded within SableCC source code?

Really, I see no gain to the plugin architecture. It simply adds
configuration headaches. Even worse: it makes it quite hard to create an
independent target in any language other than Java. The current
architecture has no need to hardcode any directory (other than using the
"current" directory as default destination), and allows for using other
languages (or, it will once SableCC targets new languages).


Now, for integrating a target within SableCC, the current architecture
does require adding/modifying two lines of code within the existing
code. Of course, in this case, the target must also be written in Java.
But, other than these two lines of code, no other modification is
needed. Note that the exact same code can be used for implementing both
an independent target and an integrated target.

I really don't mind modifying these two lines of code, when a new target
is to be integrated. What's currently missing, I think, is a template
for people to use to develop an independent target, e.g. a main class
which demonstrates implementing an independent target using the existing
Java target, and some wiki documentation explaining how things work.

Anyway, that's the result of a long reflection I've put into it before
choosing and implementing the current architecture.

Have fun!

Etienne

  

_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion
  

_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion

[Niklas Matthies]

I was thinking (probably naively) that the idea with SableCC 4 was
that it would provide a programmatic API to its non back-end specific
part. That is, one could write a program that incorporates SableCC as
a library, feeding it grammar descriptions (as character streams or as
object structures) and receiving, as a result, an object structure
that is an abstract representation of the generated parser. The
program could then generate anything it likes from that.

(This is effectively the opposite of a plugin architecture for
SableCC. A plugin architecture would be backwards IMO, somehow it
never occurred to me as an option. ;))

Two examples of what I was envisaging I'd be able to do:

- Generate a parser in-memory and directly use it, without
intermediate files and compilation. We have a template processing
library where some aspects of the template syntax are configurable at
runtime, which is why we can't use SableCC to generate a template
parser. Being able to use SableCC without its back-end as a library
would make it possible to replace our hand-coded and difficult-to-
maintain "dynamic" parser with parsers generated at runtime with
SableCC from a templated grammar.

- Generate different syntax tree interfaces and classes for Java than
the built-in Java back-end, for example introducing more subtyping (as
in "Expr is-a ExprTerm is-a Term" rather than "ExprTerm is-a Term and
has-a Expr"), generating Visitor interfaces and various Decorator
implementations per syntactic category rather than only global ones,
stuff like that.

As far as I can gather from the current discussion, I probably set my
hopes to high?

-- Niklas Matthies

_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion

[Etienne M. Gagnon]

Hi Niklas,

On the contrary. Here's how SableCC (in progress) and ObjectMacro
(existing) work:

1- Parse input file
2- Semantic validation
3- Semantic Processing (e.g. built tables, data structures, etc.)
4- Generate AST of intermediate representation
5- Send intermediate AST to code generation back-end.

Currently, ObjectMacro has 3 back-ends:
- intermediate AST to Java set of classes back-end.
- intermediate AST to Scala set of classes back-end.
- intermediate AST to intermediate text representation (e.g. concrete
syntax) back-end.

By design, a back-ends does not make any access to internal data
structures (such as computed tables, original AST, etc.). A back-end is
limited to using the intermediate AST as input for generating code.
Furthermore, the back-end assumes that this AST is correct (it does make
the semantic analysis of the intermediate AST), as it is automatically
generated. This simplifies greatly the task of back-end developers.

So, it really is very flexible. You can hook your code in the system
after setp 4, by taking hold of the in-memory generated intermediate AST
and do your processing from that point. You can also use the
"intermediate" target to generate a text file and develop an independent
back-end that parses this file (using the provided intermediate grammar).

Makes sense?

Etienne

Niklas Matthies wrote:

> I was thinking (probably naively) that the idea with SableCC 4 was
> that it would provide a programmatic API to its non back-end specific
> part. [...]
> Two examples of what I was envisaging I'd be able to do:
> - Generate a parser in-memory and directly use it, [...]
> - Generate different syntax tree interfaces and classes for Java than
> the built-in Java back-end, [...]
>
> As far as I can gather from the current discussion, I probably set my
> hopes to high

--
Etienne M. Gagnon, Ph.D.
SableCC:                                            http://sablecc.org




_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion

[Niklas Matthies]

On Thu 2009-05-07 at 11:57h, Etienne M. Gagnon wrote on sablecc-discussion:
> Hi Niklas,
>
> On the contrary.

Great! :)

> Here's how SableCC (in progress) and ObjectMacro (existing) work:
>
> 1- Parse input file
> 2- Semantic validation
> 3- Semantic Processing (e.g. built tables, data structures, etc.)
> 4- Generate AST of intermediate representation
> 5- Send intermediate AST to code generation back-end.
:

> By design, a back-ends does not make any access to internal data
> structures (such as computed tables, original AST, etc.). A back-end is
> limited to using the intermediate AST as input for generating code.
> Furthermore, the back-end assumes that this AST is correct (it does make
> the semantic analysis of the intermediate AST), as it is automatically
> generated. This simplifies greatly the task of back-end developers.
>
> So, it really is very flexible. You can hook your code in the system
> after setp 4, by taking hold of the in-memory generated intermediate AST
> and do your processing from that point.

Sounds good. By "hook your code", you mean modifying the SableCC
sources? I imagined having something like the methods

    CompiledGrammar compileGrammar(Reader grammarStream);
    CompiledGrammar compileGrammar(Grammar spec);

that can be invoked in SableCC4.jar from the outside, where
CompiledGrammar would be the intermediate representation between steps
4 and 5, and (optionally) Grammar would be something corresponding to
the representation between steps 1 and 2. In other words, provide an
API function that does steps 1-4 or steps 2-4.

Furthermore, I hope the intermediate representation is not just an AST
of the (transformed) grammar specification, but also includes a
representation of the grammar's parsing tables, i.e. an equivalent of
lexer.dat and parser.dat?

-- Niklas Matthies

_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion

[Etienne M. Gagnon]

Hi Niklas,

I've added a ticket with your proposal.

Yes, the intermediate AST contains everything. (There's no separate .dat
files).

Etienne

Niklas Matthies wrote:

> Sounds good. By "hook your code", you mean modifying the SableCC
> sources? I imagined having something like the methods
>
>     CompiledGrammar compileGrammar(Reader grammarStream);
>     CompiledGrammar compileGrammar(Grammar spec);
> ...
> Furthermore, I hope the intermediate representation is not just an AST
> of the (transformed) grammar specification, but also includes a
> representation of the grammar's parsing tables, i.e. an equivalent of
> lexer.dat and parser.dat?
>  

--
Etienne M. Gagnon, Ph.D.
SableCC:                                            http://sablecc.org




_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion

[Niklas Matthies]

On Thu 2009-05-07 at 18:42h, Etienne M. Gagnon wrote on sablecc-discussion:
> Hi Niklas,
>
> I've added a ticket with your proposal.

Thanks!

> Yes, the intermediate AST contains everything. (There's no separate
> .dat files).

Good. :)

-- Niklas Matthies

_______________________________________________
SableCC-Discussion mailing list
SableCC-Discussion@...
http://lists.sablecc.org/listinfo/sablecc-discussion