[scala] A small change to compiler phases

[Kevin Wright-4]

It seems that there's an important class of compiler plugins that
struggle under the current setup.
Specifically, those that generate entirely synthetic classes, or
synthetic methods on existing classes.

This is possible to do if no type or symbol information is needed by
the plugin (it can run between parser and namer), but it's impossible
to synthesise methods after the typer phase unless those methods are
overrides, as any classes then trying to use these methods will
already have failed to compile.

A quick look at the existing architecture shows how this issue is
handled with other code already being synthesised by the compiler:

case classes: the companion object is generated in the namer.  In
addition the generated hashCode and equals methods are overrides from
the Any class
@BeanProperty: synthesis is mostly in namer, with some support for
setters in typer
@specialized: after typer, but (crucially) the code is valid and
type-checks before specialisation

It seems that the best place to handle a lot of synthesis is
immediately before typer, but this option is not available to a plugin
as typer is set runRightAfter namer.

I'm attempting to write a plugin that could very easily adopt a
similar approach to @BeanProperty, and I'm able to synthesize the
necessary entries in the AST with only symbol information (I'm
perfectly happy to allow the typer to then do its normal thing on
these generated constructs, there's nothing unusual in them) but the
tight coupling of the namer/typer phases prevents me from doing this.

Would it therefore be possible to have "typer runsAfter namer" instead
of "typer runsRightAfter namer" ?

[Chris Twiner]

On Mon, Nov 9, 2009 at 4:38 PM, Kevin Wright
<kev.lee.wright@...> wrote:
> It seems that there's an important class of compiler plugins that
> struggle under the current setup.
> Specifically, those that generate entirely synthetic classes, or
> synthetic methods on existing classes.

I have such a plugin and I am sure there are others.

> these generated constructs, there's nothing unusual in them) but the
> tight coupling of the namer/typer phases prevents me from doing this.
>
> Would it therefore be possible to have "typer runsAfter namer" instead
> of "typer runsRightAfter namer" ?
>


Of course I realise this would be tough to do, but whilst we are
asking for compiler changes...

I'd prefer an optional phase of "please typer don't complain I WILL
fix this tree for you, but may ask you to run again, oh and on that
second run could you please add the following synthetics in". This
would allow for the case where you need the type information from
typers (as per my plugin) but would like to add synthetics.

[Stephen Haberman]

> I'd prefer an optional phase of "please typer don't complain I WILL
> fix this tree for you, but may ask you to run again, oh and on that
> second run could you please add the following synthetics in". This
> would allow for the case where you need the type information from
> typers (as per my plugin) but would like to add synthetics.

Speaking with absolutely zero knowledge how of scala compiler plugins
work, this sounds vaguely like JDK6 annotation processors.

E.g. I can manually write some code that depends on a non-existent
FooClass, and so fails to compile on the first round. But then the
javac compiler runs the annotation processor, which creates the
FooClass.java source, so the compiler starts a 2nd round to compile
FooClass.java -> FooClass.class. Having done that, the compiler then
re-compiles/fixes the "FooClass not found" error from the first round,
and everything is now fine.

javac would even start up a 3rd round if the FooClass.java source that
the first annotation processor generated happened to have annotations
that triggered another annotation processor to generate yet more source
code. It keeps going until no more source files have been generated.

No idea whether multiple rounds makes sense for scala compiler plugins,
but in my experience it has worked well for annotation processors.

- Stephen

[Kevin Wright-4]

On Tue, Nov 10, 2009 at 6:07 AM, Stephen Haberman
<stephen@...> wrote:

>
>> I'd prefer an optional phase of "please typer don't complain I WILL
>> fix this tree for you, but may ask you to run again, oh and on that
>> second run could you please add the following synthetics in". This
>> would allow for the case where you need the type information from
>> typers (as per my plugin) but would like to add synthetics.
>
> Speaking with absolutely zero knowledge how of scala compiler plugins
> work, this sounds vaguely like JDK6 annotation processors.
>
> E.g. I can manually write some code that depends on a non-existent
> FooClass, and so fails to compile on the first round. But then the
> javac compiler runs the annotation processor, which creates the
> FooClass.java source, so the compiler starts a 2nd round to compile
> FooClass.java -> FooClass.class. Having done that, the compiler then
> re-compiles/fixes the "FooClass not found" error from the first round,
> and everything is now fine.
>
> javac would even start up a 3rd round if the FooClass.java source that
> the first annotation processor generated happened to have annotations
> that triggered another annotation processor to generate yet more source
> code. It keeps going until no more source files have been generated.
>
> No idea whether multiple rounds makes sense for scala compiler plugins,
> but in my experience it has worked well for annotation processors.
>
> - Stephen
>
>
>

I can imagine something like this working for a range of issues:

- During typer, postpone the display of any error messages
- Create an new phase typeSynthetics, immediately before superaccessors
- A unit can be flagged for re-compilation by any phase before this
  (so plugins would runAfter typer, but before typeSynthetics)
- In typeSynthetics, reprocess (namer-->typeSynthetics) any flagged unit
- Report any outstanding typer failures

[Kevin Wright-4]

On Tue, Nov 10, 2009 at 7:48 AM, Kevin Wright
<kev.lee.wright@...> wrote:

> On Tue, Nov 10, 2009 at 6:07 AM, Stephen Haberman
> <stephen@...> wrote:
>>
>>> I'd prefer an optional phase of "please typer don't complain I WILL
>>> fix this tree for you, but may ask you to run again, oh and on that
>>> second run could you please add the following synthetics in". This
>>> would allow for the case where you need the type information from
>>> typers (as per my plugin) but would like to add synthetics.
>>
>> Speaking with absolutely zero knowledge how of scala compiler plugins
>> work, this sounds vaguely like JDK6 annotation processors.
>>
>> E.g. I can manually write some code that depends on a non-existent
>> FooClass, and so fails to compile on the first round. But then the
>> javac compiler runs the annotation processor, which creates the
>> FooClass.java source, so the compiler starts a 2nd round to compile
>> FooClass.java -> FooClass.class. Having done that, the compiler then
>> re-compiles/fixes the "FooClass not found" error from the first round,
>> and everything is now fine.
>>
>> javac would even start up a 3rd round if the FooClass.java source that
>> the first annotation processor generated happened to have annotations
>> that triggered another annotation processor to generate yet more source
>> code. It keeps going until no more source files have been generated.
>>
>> No idea whether multiple rounds makes sense for scala compiler plugins,
>> but in my experience it has worked well for annotation processors.
>>
>> - Stephen
>>
>>
>>
>
> I can imagine something like this working for a range of issues:
>
> - During typer, postpone the display of any error messages
> - Create an new phase typeSynthetics, immediately before superaccessors
> - A unit can be flagged for re-compilation by any phase before this
>  (so plugins would runAfter typer, but before typeSynthetics)
> - In typeSynthetics, reprocess (namer-->typeSynthetics) any flagged unit
> - Report any outstanding typer failures
>

Thinking about this further, I'd also like to be able to REMOVE
methods via a compiler plugin.

The specific goal I have in mind is an annotation that indicates which
JVMs a particular method is valid on (i.e. a method implemented in
JDK7 could have a fallback available in a scala Rich wrapper, which is
only compiled for JDK <= 6, handy for the IO stuff).  This could also
be handled very cleanly by running a second pass of the namer/typer
phases.

For now, I'm looking to see if I can write a compiler plugin to run
after parser which will perform a local run of namer/typer, do its
stuff in the AST, then throw away the localised symbol/type info and
proceed naturally onto namer and subsequent phases in global.  I
suspect that minimising the amount of naming/typing performed in this
first pass (for reasons of performance) is going to be one of my
bigger challenges here, I'll have to look at what's possible in nsc by
way of building a dependency graph.