[scala-tools] Can we improve the Eclipse REPL?

[Martin Odersky]

I am teaching functional programming in Scala to a class of 2nd year
students. I am doing a lot of demonstrations directly, showing the
students the behavior of various language constructs, inviting
improvement proposals, etc. This works generally very well. One
annoying thing is that I have been flip-flopping constantly between
the eclipse plugin and the command line interpreter. Both have
strengths and weaknesses. The plugin is great for developing and
incrementally improving a set of classes. The REPL is great for
free-form experimentation and exploration. This made me really wish we
could combine the strengths of the two.

Ideally what I would wish for is the following:

 - being able to open a REPL with a given focus. Say I open a REPL for
element mypackage.myobject. Then I want automatic
   imports:

     import mypackage._
     import myobject._

   so that I can directly access definitions in my focus and its environment.

  - when I change code, the REPL should automatically re-load it. I
think this can be achieved using JavaRebel, but don't know enough
about it yet.
    If we don't want to rely on JavaRebel, one could also imagine to
re-load only if the focus file is changed. This could be done by
wrapping the focus
    file in a synthetic version object, which gets changed everytime
we edit he focus file. So we end up with many versions of the same
class(es),
    each wrapped in a different version object.

The current console support does neither (and, besides, it does not
work on Windows; the console window closes immediately after being
opened).

I believe if we can do such a thing, it would be really useful. It
would be a point where the Scala plugin would do something clearly
better than the
Java plugin. For the first time, we'd do more than catching up!


Cheers

 -- Martin

[Josh Suereth]

I feel obligated to respond since I contributed the initial part of the REPL code for the current eclipse plugin (based on the previous plugin's REPL).

First off, I agree with you.  The REPL would be a great and distinguishing feature.

Here's what (IMHO) needs to be done:

1. Migrate from my "Roll your own ProcessBuilder" to Eclipse's abstraction for executing Java processes.
1. Make sure we don't lose control of potential syntax highlighting/code completion
2. This should fix the windows/mac issues.  (Funny thing, I developed the initial REPL code on windows)
   
2. Integrate JavaRebel as either an optional plugin or a configuration item
1. The interpreter needs a Preference + Project Properties page
2. We should get some help from the Lift/Maven plugin team (or anyone else who uses JavaRebel), as I don't make much use in my day-to-day programming.
3. Ensure all "fun" keystrokes (tabs, arrows, etc.) get passed into the REPL process from eclipse
4. Expand the "auto-import when opened from package" feature to include "auto-import projects"
1. You'll need to expand the menu filters to include a filter for scala objects.  BTW this is where one of the "innocuous" error messages gets thrown form the scala plugin
5. Integrate syntax highlighting/code completion
1. Since the "console" is not a source editor, I'm not sure the regular eclipse APIs (DamageRepairer, etc.) work here. 
   
2. I'm not as familiar with the Presentation compiler, so I can't articulate this task further, but I see it being fairly large
   

Just an FYI, the only REPL I've ever designed before was for the Eclipse LOLCode plugin.   Any help accomplishing the above would be MUCH appreciated (I'm currently slightly tied down with working on Scala/Maven integration with my copious free time).

Thanks,
-Josh

On Thu, Nov 13, 2008 at 6:06 AM, martin odersky <martin.odersky@...> wrote:

I am teaching functional programming in Scala to a class of 2nd year
students. I am doing a lot of demonstrations directly, showing the
students the behavior of various language constructs, inviting
improvement proposals, etc. This works generally very well. One
annoying thing is that I have been flip-flopping constantly between
the eclipse plugin and the command line interpreter. Both have
strengths and weaknesses. The plugin is great for developing and
incrementally improving a set of classes. The REPL is great for
free-form experimentation and exploration. This made me really wish we
could combine the strengths of the two.

Ideally what I would wish for is the following:

 - being able to open a REPL with a given focus. Say I open a REPL for
element mypackage.myobject. Then I want automatic
  imports:

    import mypackage._
    import myobject._

  so that I can directly access definitions in my focus and its environment.

 - when I change code, the REPL should automatically re-load it. I
think this can be achieved using JavaRebel, but don't know enough
about it yet.
   If we don't want to rely on JavaRebel, one could also imagine to
re-load only if the focus file is changed. This could be done by
wrapping the focus
   file in a synthetic version object, which gets changed everytime
we edit he focus file. So we end up with many versions of the same
class(es),
   each wrapped in a different version object.

The current console support does neither (and, besides, it does not
work on Windows; the console window closes immediately after being
opened).

I believe if we can do such a thing, it would be really useful. It
would be a point where the Scala plugin would do something clearly
better than the
Java plugin. For the first time, we'd do more than catching up!


Cheers

 -- Martin

[Miles Sabin]

On Thu, Nov 13, 2008 at 11:06 AM, martin odersky <martin.odersky@...> wrote:

> Ideally what I would wish for is the following:
>
>  - being able to open a REPL with a given focus. Say I open a REPL for
> element mypackage.myobject. Then I want automatic
>   imports:
>
>     import mypackage._
>     import myobject._
>
> so that I can directly access definitions in my focus and its environment.

This should be relatively straightforward to do.

>  - when I change code, the REPL should automatically re-load it. I
> think this can be achieved using JavaRebel, but don't know enough
> about it yet.
>    If we don't want to rely on JavaRebel, one could also imagine to
> re-load only if the focus file is changed. This could be done by
> wrapping the focus file in a synthetic version object, which gets
> changed everytime we edit he focus file. So we end up with many
> versions of the same class(es), each wrapped in a different version
> object.

JavaRebel works spectacularly well for this, at least with the latest
Sun JVM. My only concerns about depending on it is that it's closed
source, and it's not at all clear to me what the "free" one year Scala
license actually permits. For instance, it's not at all clear to me
whether or not it's distributable by EPFL or anyone else as a
component of the Eclipse plugin.

Ultimately it's your call on the licensing issue. From my point of
view, I would either want there to be no issues with bundling
JavaRebel with the plugin at zero cost now and into the future (ie.
well beyond one year); or have the JavaRebel functionality shipped as
an independent component separately from the rest of the Scala plugin,
in which case it would probably make sense if ZeroTurnaround could
develop and distribute it themselves.

> The current console support does neither (and, besides, it does not
> work on Windows; the console window closes immediately after being
> opened).

Yes, that's unfortunate ... I'm not sure what the problem is, and I'd
be grateful for any help anyone can give me here.

> I believe if we can do such a thing, it would be really useful. It
> would be a point where the Scala plugin would do something clearly
> better than the Java plugin. For the first time, we'd do more than
> catching up!

Agreed!

Cheers,


Miles

--
Miles Sabin
tel:    [phone]
mobile: [phone]
skype:  milessabin

[Daniel Spiewak]

IIRC, we've heard from the JavaRebel guys on this before.  Assuming that I'm remembering correctly, the answer was that bundling the Scala version of JavaRebel with the plugin would be no problem at all.  JavaRebel provides a free version for use with Scala which does allow for redistribution.  At the very least, we could bundle the javarebel.jar and force the users to point to a license in the preferences.

One issue which would need ironing is the fact that JR checks for updates on startup.  Since the JAR itself would be bundled with the plugin, we really don't want that.  I honestly have no idea if this can be disabled.

Integrating JavaRebel is really a snap.  I'm attaching my `jrscala` script which I use to launch a Scala interactive shell pre-juiced with JavaRebel.  It's worth noting that there is some flakiness to worry about.  Occaisionally, really significant changes (e.g. a series of tasks adding members, removing others, adding new classes, etc) will cause JavaRebel to get confused and somehow get stuck halfway through the change set.  In other words, you would be using a version of your classes which is not exactly current.  This would need to be made clear to anyone who uses it (particularly students).

+1 for improving the SDT REPL overall.  Even basic syntax highlighting would be extremely nice.

Daniel

On Thu, Nov 13, 2008 at 8:07 AM, Miles Sabin <miles@...> wrote:

On Thu, Nov 13, 2008 at 11:06 AM, martin odersky <martin.odersky@...> wrote:
> Ideally what I would wish for is the following:
>
>  - being able to open a REPL with a given focus. Say I open a REPL for
> element mypackage.myobject. Then I want automatic
>   imports:
>
>     import mypackage._
>     import myobject._
>
> so that I can directly access definitions in my focus and its environment.

This should be relatively straightforward to do.

>  - when I change code, the REPL should automatically re-load it. I
> think this can be achieved using JavaRebel, but don't know enough
> about it yet.
>    If we don't want to rely on JavaRebel, one could also imagine to
> re-load only if the focus file is changed. This could be done by
> wrapping the focus file in a synthetic version object, which gets
> changed everytime we edit he focus file. So we end up with many
> versions of the same class(es), each wrapped in a different version
> object.

JavaRebel works spectacularly well for this, at least with the latest
Sun JVM. My only concerns about depending on it is that it's closed
source, and it's not at all clear to me what the "free" one year Scala
license actually permits. For instance, it's not at all clear to me
whether or not it's distributable by EPFL or anyone else as a
component of the Eclipse plugin.

Ultimately it's your call on the licensing issue. From my point of
view, I would either want there to be no issues with bundling
JavaRebel with the plugin at zero cost now and into the future (ie.
well beyond one year); or have the JavaRebel functionality shipped as
an independent component separately from the rest of the Scala plugin,
in which case it would probably make sense if ZeroTurnaround could
develop and distribute it themselves.

> The current console support does neither (and, besides, it does not
> work on Windows; the console window closes immediately after being
> opened).

Yes, that's unfortunate ... I'm not sure what the problem is, and I'd
be grateful for any help anyone can give me here.

> I believe if we can do such a thing, it would be really useful. It
> would be a point where the Scala plugin would do something clearly
> better than the Java plugin. For the first time, we'd do more than
> catching up!

Agreed!

Cheers,


Miles

--
Miles Sabin
tel:    [phone]
mobile: [phone]
skype:  milessabin

[Miles Sabin]

On Thu, Nov 13, 2008 at 5:09 PM, Daniel Spiewak <djspiewak@...> wrote:
> IIRC, we've heard from the JavaRebel guys on this before.  Assuming that I'm
> remembering correctly, the answer was that bundling the Scala version of
> JavaRebel with the plugin would be no problem at all.

I think we need something a little more than things that someone might
have said on a mailing list once if we remember correctly. I can't
find any kind of document on the ZeroTurnaround site which lays out
the details of the license ... in particular anything which says it's
freely redistributable.

> JavaRebel provides a free version for use with Scala which does allow for
> redistribution.  At the very least, we could bundle the javarebel.jar and
> force the users to point to a license in the preferences.

I think that JavaRebel is a fine product. But what you're suggesting
would basically be an open source project giving it free advertising.
It's not clear to me (yet) why the Scala community should be expected
to do that.

Cheers,


Miles

[Daniel Spiewak]

From a thread on the scala-tools list a few months ago:

> Another question:  Can scala redistribute JavaRebel?  Can it redistribute
> with a license?  I didn't notice any info on the website where they posted
> the free license, however in looking at their licensing agreement, I just
> want to make sure we honor ZeroTurnaround's interests (and their amazing
> donation).

I can assure you that you can redistribute JavaRebel; we are working
out the legal details at the moment, but the free Scala license is
meant for redistribution.

Jevgeni Kabanov
ZeroTurnaround

The thread has a few more choice tidbits.  Basically, we were discussing this exact issue (JavaRebel and the Scala REPL in SDT).

Daniel

On Thu, Nov 13, 2008 at 11:23 AM, Miles Sabin <miles@...> wrote:

On Thu, Nov 13, 2008 at 5:09 PM, Daniel Spiewak <djspiewak@...> wrote:

> IIRC, we've heard from the JavaRebel guys on this before.  Assuming that I'm
> remembering correctly, the answer was that bundling the Scala version of
> JavaRebel with the plugin would be no problem at all.

I think we need something a little more than things that someone might
have said on a mailing list once if we remember correctly. I can't
find any kind of document on the ZeroTurnaround site which lays out
the details of the license ... in particular anything which says it's
freely redistributable.

> JavaRebel provides a free version for use with Scala which does allow for
> redistribution.  At the very least, we could bundle the javarebel.jar and
> force the users to point to a license in the preferences.

I think that JavaRebel is a fine product. But what you're suggesting
would basically be an open source project giving it free advertising.
It's not clear to me (yet) why the Scala community should be expected
to do that.

Cheers,


Miles

[Miles Sabin]

On Thu, Nov 13, 2008 at 5:45 PM, Daniel Spiewak <djspiewak@...> wrote:
> From a thread on the scala-tools list a few months ago:
> > I can assure you that you can redistribute JavaRebel; we are working
> > out the legal details at the moment, but the free Scala license is
> > meant for redistribution.
> >
> > Jevgeni Kabanov
> > ZeroTurnaround

That's great ... but then this is moot until the legal details have in
fact been worked out.

Cheers,


Miles

[Daniel Spiewak]

The point is that if they were "working out" the details a few months ago, it's probably "worked out" by now.  Either way, an email to Jevgeni would clear up the whole cloud of uncertainty (assuming he's not already on the list).

As to the more fundamental issue on whether or not we should be giving JavaRebel "free advertising", I think that is a legitimate question.  The hang-up is that if we don't have JavaRebel integration baked into the plugin, then there is no way to get that functionality within SDT, even for developers who want it.  Would it be essentially saying that JavaRebel is an expected part of every Scala developer's toolset?  Yes, probably, but is that such a bad thing?  It comes down to the philosophical question of whether or not commercial tools should be encouraged from within OSS.  I'm sure Stallman would say "no", but to refuse to acknowledge commercial software severely limits the usefulness (and thus, the adoption) of open-source.  There are plenty of examples of real-world OSS which has integration with proprietary software.  For example, when you use Firefox to navigate to a .mp4 file, does that file open in the QuickTime plugin, or do you suffer through embedded VLC?  Isn't that just Firefox giving free advertising to Apple?

Coming back to JavaRebel, I think that this is a feature that enough developers have asked for -- including our Fearless Leader -- that it merits some serious attention.  If it were possible for JavaRebel users to just enable the functionality separately without impacting the rest of the user-base (in other words, without the obvious "JavaRebel License Here" type of hooks), I would agree that the obvious endorsement should be avoided.  However, that just isn't the case.  If SDT does not implement this functionality within the core plugin, it's not going to happen for anyone.

Daniel

On Thu, Nov 13, 2008 at 11:53 AM, Miles Sabin <miles@...> wrote:

On Thu, Nov 13, 2008 at 5:45 PM, Daniel Spiewak <djspiewak@...> wrote:
> From a thread on the scala-tools list a few months ago:

> > I can assure you that you can redistribute JavaRebel; we are working
> > out the legal details at the moment, but the free Scala license is
> > meant for redistribution.
> >
> > Jevgeni Kabanov
> > ZeroTurnaround

That's great ... but then this is moot until the legal details have in
fact been worked out.

Cheers,


Miles

[Miles Sabin]

On Thu, Nov 13, 2008 at 6:10 PM, Daniel Spiewak <djspiewak@...> wrote:
> Coming back to JavaRebel, I think that this is a feature that enough
> developers have asked for -- including our Fearless Leader -- that it merits
> some serious attention.

Whether it merits serious attention or not, neither I nor EPFL are
able redistribute JavaRebel without an explicit legal agreement on
ZeroTurnaround's part that allows us to do so. We don't have one of
those, so we can't do it. Period.

If there is anyone on this list from ZeroTurnaround, it'd be great if
they could chip and clarify things.

Cheers,


Miles

[ekabanov]

There is no special agreement, because we amended the license agreement with section 5. LICENSE TO DISTRIBUTE SOFTWARE (see http://www.zeroturnaround.com/javarebel-license/). This part was written by David Pollak and we decided that it is in our interest to allow redistributing JavaRebel freely.

Jevgeni Kabanov

Miles Sabin wrote:

On Thu, Nov 13, 2008 at 6:10 PM, Daniel Spiewak <[e-mail]> wrote:
> Coming back to JavaRebel, I think that this is a feature that enough
> developers have asked for -- including our Fearless Leader -- that it merits
> some serious attention.

Whether it merits serious attention or not, neither I nor EPFL are
able redistribute JavaRebel without an explicit legal agreement on
ZeroTurnaround's part that allows us to do so. We don't have one of
those, so we can't do it. Period.

If there is anyone on this list from ZeroTurnaround, it'd be great if
they could chip and clarify things.

Cheers,


Miles

[Miles Sabin]

On Thu, Nov 13, 2008 at 8:39 PM, ekabanov <ekabanov@...> wrote:
> There is no special agreement, because we amended the license agreement with
> section 5. LICENSE TO DISTRIBUTE SOFTWARE (see
> http://www.zeroturnaround.com/javarebel-license/). This part was written by
> David Pollak and we decided that it is in our interest to allow
> redistributing JavaRebel freely.

Thanks for the clarification ... I'm a little surprised that this
document isn't linked to more prominently from more obvious pages on
your site (tho' perhaps I've been a little unobservant?).

Unfortunately it doesn't encourage me. Although it grants a license to
redistribute the binary, it doesn't grant the same for a license key
(see esp. clause 4.iii). That means that any JavaRebel-based
functionality would be non-functional unless the end-user purchased
and installed an individual or corporate license (it's unclear from
this document what the status of the "free" Scala license is, or for
how long it's likely to be available free of charge). Pricing is par
for the course I guess, but not cheap,

  http://www.zeroturnaround.com/news/javarebel-price-and-licensing-changes/

As such I'm afraid I think it would be completely inappropriate to
include it directly in a piece of free and open source software. In
any case, I'm not prepared to do the work of integration (which, at an
absolute minimum includes no impact on people who choose not to obtain
a JavaRebel license, ie. no errors, no nag screens, no cognitive
dissonance or sales pitches in the documentation).

That said, I think JavaRebel is an excellent product. I've used it
myself and I've recommended it to people and organizations who'll very
likely purchase licenses. If ZeroTurnaround or an enthusiastic
JavaRebel user were to provide me with a suitable patch or, better
still, an independent plugin that adds support to the Eclipse plugin,
then I'd look on that very favourably.

Cheers,


Miles

--
Miles Sabin
tel:    [phone]
mobile: [phone]
skype:  milessabin