[scala-tools] Scala Refactoring Term Project

Hi all

Reading about people creating whole new IDEs in just six month, I almost feel
bad about this, but I'd still like to announce my Scala Refactoring term
project.

In short, the goal is to create an IDE independent refactoring tool/API for
Scala. The longer version and my proposal can be found on my blog [1].

Now, what I'd like to know from you:

- Which refactorings are you missing the most?
- How important is having refactoring support?
- If there was a simple API with which you could modify your sourcecode, would
you use it to create your own refactorings or code manipulation tools? (This
is at most a long-term idea, but imagine something like an intelligent search-
replace that understands your code).
- And any other comment you might have on the topic :-)

Regards

Mirko

[1] http://misto.ch/scala-refactoring-term-project/

[Ismael Juma]

Hi Mirko,

On Thu, 2009-09-10 at 22:04 +0200, Mirko Stocker wrote:
> Now, what I'd like to know from you:
>
> - Which refactorings are you missing the most?

Rename and extract variable/method are the ones I use all the time
without even thinking. Inline method/variable is also very useful and
others I use on occasion are change method signature, extract
interface/trait, pull up, push down, convert anonymous class to nested,
convert nested class to top-level and convert local variable to field.

> - How important is having refactoring support?

It's not essential, but basic support is very useful.

> - If there was a simple API with which you could modify your sourcecode, would
> you use it to create your own refactorings or code manipulation tools? (This
> is at most a long-term idea, but imagine something like an intelligent search-
> replace that understands your code).

Is this something like Project Jackpot?

Best,
Ismael

[Jorge Ortiz-3]

Mirko,

I think this is an excellent idea.

Another use case is code-generation. Currently, Scala does something fairly primitive and hackish to generate all the necessary TupleN and FunctionN classes. It'd be great to, eg, have a code tool that could take a Tuple2 source file and generate Tuple3...N source files.

--j

On Thu, Sep 10, 2009 at 1:04 PM, Mirko Stocker <me@...> wrote:

Hi all

Reading about people creating whole new IDEs in just six month, I almost feel
bad about this, but I'd still like to announce my Scala Refactoring term
project.

In short, the goal is to create an IDE independent refactoring tool/API for
Scala. The longer version and my proposal can be found on my blog [1].

Now, what I'd like to know from you:

- Which refactorings are you missing the most?
- How important is having refactoring support?
- If there was a simple API with which you could modify your sourcecode, would
you use it to create your own refactorings or code manipulation tools? (This
is at most a long-term idea, but imagine something like an intelligent search-
replace that understands your code).
- And any other comment you might have on the topic :-)

Regards

Mirko

[1] http://misto.ch/scala-refactoring-term-project/

[Colin Bullock]

- Which refactorings are you missing the most?

Here's a list of (rather novel) refactorings, gratuitously stolen from a previous thread[1]:

Curry Method (split a parameter list, and the arg lists of all callers).
Uncurry Method (merge split parameter list, including merging the arg lists of callers.  If method is called with partial args, either complain or automatically create a helper method which represents the partial application, and replace partial calls with it.)
Extract Trait (including searching for other classes which can have the same trait extracted.  Tricky with super calls, but not impossible)
Split Trait (splits trait into two traits (putting in self-types if needed), change all extending classes to extend both traits)
Extract Extractor (select a pattern, automatically create an extractor)
Extract Closure (similar to extract method, but creating a function object)
Introduce by-name parameter
Extract type definition (obvious)
Merge nested for-comprehensions into single for-comprehension (and converse)
Split guard from for-comprehension into nested if (and converse)
Convert for-comprehension into map/filter/flatmap chain (and converse)
Wrap parameter as Option (converting null checks, etc.)
Convert instanceOf/asInstance pair to match
Replace case clause with if body to guarded case clause(s) 

- Colin

[1] http://www.nabble.com/Scala%3A-Great-potential%2C-but-not-ready-for-production-td22064598i20.html#a22082676

[Clint Gilbert]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mirko Stocker wrote:

> Hi all
>
> Reading about people creating whole new IDEs in just six month, I almost feel
> bad about this, but I'd still like to announce my Scala Refactoring term
> project.
>
> In short, the goal is to create an IDE independent refactoring tool/API for
> Scala. The longer version and my proposal can be found on my blog [1].
>
> Now, what I'd like to know from you:
>
> - Which refactorings are you missing the most?

All of them. :)

Just kidding - the ones I use most in Java are

Rename field and method
Extract method
Move (classes between packages, mostly)
and occasionally Extract interface

There are undoubtedly some new Scala-specific ones that are possible and that I would use.
  Someone posted their ideas for some a little ways back.

> - How important is having refactoring support?

Extremely.  I use Scala for hobby projects now, and for the occasional quick tool at work.
 If the IDE support was better - and refactoring support is a very big part of that - I
would use Scala at work too.  Perhaps more importantly, my coworkers are holding off on
Scala until there's a refactoring IDE.

> - If there was a simple API with which you could modify your sourcecode, would
> you use it to create your own refactorings or code manipulation tools? (This
> is at most a long-term idea, but imagine something like an intelligent search-
> replace that understands your code).

I wouldn't, since I don't know enough about making Eclipse plugins, and I would strongly
prefer a tool that was as well-integrated with my IDE as the Eclipse JDT.  However, if
your API would help IDE plugin writers make better plugins more quickly, it would be a big
help.

> - And any other comment you might have on the topic :-)
>
> Regards
>
> Mirko
>
> [1] http://misto.ch/scala-refactoring-term-project/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFKqWHcrZoE3ArapxERCP/AAJ0dv+MCID06AKZiwqX/JlWpnp+JNACglzaJ
702yAnHoobPvRHAkcyfbEIw=
=ay1q
-----END PGP SIGNATURE-----

[Naftoli Gugenheim]

I'm sure Miles and the other IDE developers will be happy to take advantage of and integrate with such a library!
Are you envisioning also providing a command line interface, or just an API? (Just curious.)
P.S. You have 6 months to see if you should almost feel bad.


-------------------------------------
Clint Gilbert<clint.gilbert@...> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mirko Stocker wrote:

> Hi all
>
> Reading about people creating whole new IDEs in just six month, I almost feel
> bad about this, but I'd still like to announce my Scala Refactoring term
> project.
>
> In short, the goal is to create an IDE independent refactoring tool/API for
> Scala. The longer version and my proposal can be found on my blog [1].
>
> Now, what I'd like to know from you:
>
> - Which refactorings are you missing the most?

All of them. :)

Just kidding - the ones I use most in Java are

Rename field and method
Extract method
Move (classes between packages, mostly)
and occasionally Extract interface

There are undoubtedly some new Scala-specific ones that are possible and that I would use.
  Someone posted their ideas for some a little ways back.

> - How important is having refactoring support?

Extremely.  I use Scala for hobby projects now, and for the occasional quick tool at work.
 If the IDE support was better - and refactoring support is a very big part of that - I
would use Scala at work too.  Perhaps more importantly, my coworkers are holding off on
Scala until there's a refactoring IDE.

> - If there was a simple API with which you could modify your sourcecode, would
> you use it to create your own refactorings or code manipulation tools? (This
> is at most a long-term idea, but imagine something like an intelligent search-
> replace that understands your code).

I wouldn't, since I don't know enough about making Eclipse plugins, and I would strongly
prefer a tool that was as well-integrated with my IDE as the Eclipse JDT.  However, if
your API would help IDE plugin writers make better plugins more quickly, it would be a big
help.

> - And any other comment you might have on the topic :-)
>
> Regards
>
> Mirko
>
> [1] http://misto.ch/scala-refactoring-term-project/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFKqWHcrZoE3ArapxERCP/AAJ0dv+MCID06AKZiwqX/JlWpnp+JNACglzaJ
702yAnHoobPvRHAkcyfbEIw=
=ay1q
-----END PGP SIGNATURE-----

[Dmitry Grigoriev-2]

Mirko Stocker wrote:

> - Which refactorings are you missing the most?

Things I miss the most:

1. rename var/method/class/etc;
2. when I move classes between packages or from/to outer classes, it
requires adjusting names and imports in whole project;
3. change method signature (change class/method generic signature would
be useful too).

> - How important is having refactoring support?

Strict typization does its job... I mean manual refactorings are asspain
but they are incomparably easier than in dynamically typed languages.

So the point is productivity only; errors introduced by simple manual
refactorings are caught by compiler.

> - If there was a simple API with which you could modify your sourcecode, would
> you use it to create your own refactorings or code manipulation tools? (This
> is at most a long-term idea, but imagine something like an intelligent search-
> replace that understands your code).

The idea looks very interesting. But the question about practical use is
probably to those who deal with large projects, especially projects with
lots of computer-generated source code (best candidates for automated
project-specific project-wide refactorings). The only thing I can say is
that API and whole process of creating/installing custom refactoring
must be as simple as possible in all means.

> - And any other comment you might have on the topic :-)

I'm not a pro in this area, but just interesting: Scala IDE plugin
builds syntactic tree in realtime while I edit source code. Do you plan
to build your tool on top of Scala plugin? If no, then how and what
about effeciency?

--
Cheers,
dimgel

On Thursday 10 September 2009 22:13:15 Ismael Juma wrote:
> Is this something like Project Jackpot?

Haven't heard of Project Jackpot before, but yes, that sounds similar. But
this is nor part of the plan for this project, just an idea for other things
one might want to do with a refactoring engine.

Thanks for your list of refactorings, I'll collect these and consider them for
implementation.

Cheers

Mirko

On Thursday 10 September 2009 22:20:50 Jorge Ortiz wrote:
> Another use case is code-generation.

Code generation will naturally be part of the project, so I guess this might
be doable as well.

Regards

Mirko

On Thursday 10 September 2009 22:21:48 Colin Bullock wrote:
> Here's a list of (rather novel) refactorings, gratuitously stolen from a
>  previous thread[1]:

Thanks!

On Thursday 10 September 2009 22:48:56 Naftoli Gugenheim wrote:
> I'm sure Miles and the other IDE developers will be happy to take advantage
>  of and integrate with such a library! Are you envisioning also providing a
>  command line interface, or just an API? (Just curious.)

I'll provide the Eclipse integration as well (my advisor insists on it :) ).
I'm not sure about the CLI yet.. for refactorings that don't need much user
interaction (like rename), that shouldn't be too hard to implement.

>  P.S. You have 6
>  months to see if you should almost feel bad.

Thanks :)

On Friday 11 September 2009 05:42:22 Dmitry Grigoriev wrote:
> > - And any other comment you might have on the topic :-)
>
> I'm not a pro in this area, but just interesting: Scala IDE plugin
> builds syntactic tree in realtime while I edit source code. Do you plan
> to build your tool on top of Scala plugin? If no, then how and what
> about effeciency?

Honestly, I can't answer this question at this time.. the project hasn't
really started yet and I've just begun digging into the code. But of course,
if the automated refactoring takes longer than doing it by hand, then nobody
will use it, so I have to make it fast. After all, this shouldn't just be an
academic exercise but something useful.

[Mohamed Bana-5]

good luck.  i look forward to seeing what you can produce.

where do you plan on hosting the code?

Mirko Stocker wrote:

> Hi all
>
> Reading about people creating whole new IDEs in just six month, I almost feel
> bad about this, but I'd still like to announce my Scala Refactoring term
> project.
>
> In short, the goal is to create an IDE independent refactoring tool/API for
> Scala. The longer version and my proposal can be found on my blog [1].
>
> Now, what I'd like to know from you:
>
> - Which refactorings are you missing the most?
> - How important is having refactoring support?
> - If there was a simple API with which you could modify your sourcecode, would
> you use it to create your own refactorings or code manipulation tools? (This
> is at most a long-term idea, but imagine something like an intelligent search-
> replace that understands your code).
> - And any other comment you might have on the topic :-)
>
> Regards
>
> Mirko
>
> [1] http://misto.ch/scala-refactoring-term-project/
>  

[Miles Sabin]

On Fri, Sep 11, 2009 at 2:01 PM, Mirko Stocker <me@...> wrote:

> On Friday 11 September 2009 05:42:22 Dmitry Grigoriev wrote:
>> > - And any other comment you might have on the topic :-)
>>
>> I'm not a pro in this area, but just interesting: Scala IDE plugin
>> builds syntactic tree in realtime while I edit source code. Do you plan
>> to build your tool on top of Scala plugin? If no, then how and what
>> about effeciency?
>
> Honestly, I can't answer this question at this time.. the project hasn't
> really started yet and I've just begun digging into the code. But of course,
> if the automated refactoring takes longer than doing it by hand, then nobody
> will use it, so I have to make it fast. After all, this shouldn't just be an
> academic exercise but something useful.

This should work out just fine.

On trunk, the Scala IDE interacts with the presentation and build
compilers via scalac's filesystem abstraction. In the case of open
files being edited, their modified buffers are exposed to the
presentation compiler as virtual files. Something similar should fit
the refactoring scenario very nicely.

Things are different on <= 2.7.6 ... that had a rather arcane and
fragile interface to the compiler, and I don't see any practical way
that a clean IDE-independent refactoring tool could have been
integrated with it.

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Miles Sabin]

On Thu, Sep 10, 2009 at 9:04 PM, Mirko Stocker <me@...> wrote:
> In short, the goal is to create an IDE independent refactoring tool/API for
> Scala.

I encourage everyone to give as much input and other assistance to
Mirko as possible: this is a really valuable project which will be
enormously beneficial for the entire Scala community, not just for
users of the Scala IDE for Eclipse.

Being IDE-independent, it'll be relatively straightforward for a broad
slice of the community to contribute refactorings and formatters
without having to commit to any particular IDE (accurate refactoring
in Emacs anyone?) or deal with the complexities of developing within
and IDE framework.

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Seth Tisue]

>>>>> "Mirko" == Mirko Stocker <me@...> writes:

 Mirko> In short, the goal is to create an IDE independent refactoring
 Mirko> tool/API for Scala.

Awesome!  I suggest you consider integrating it with sbt.

--
Seth Tisue / http://tisue.net
lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/

[Randall Schulz]

On Friday September 11 2009, Seth Tisue wrote:
> >>>>> "Mirko" == Mirko Stocker <me@...> writes:
>
>  Mirko> In short, the goal is to create an IDE independent
> refactoring Mirko> tool/API for Scala.
>
> Awesome!  I suggest you consider integrating it with sbt.

That does sound like a good idea, and I'd like to see it accommodate
IDEA, too.

In my experience, an API generic enough to be retargetable the way you'd
like needs at least a few reference points of actual integration to be
likely to have the necessary affordances to be genuinely retargetable.
If you could look at Eclipse (and / or NetBeans), SBT and IDEA's
extension mechanisms and design the API to accommodate all of them, I
think you'd stand a pretty good chance of being reasonably universal. I
know that's a lot of work and may not fit within a student project, but
it's worth considering.

Separately, I wonder how many refactorings are useful in a batch
setting? I've only ever done automated refactorings using IDEA, so I
have only that reference point to draw on in my own experience. Does
anybody use or wish they could perform non-interactive refactorings?


Randall Schulz

[Miles Sabin]

On Fri, Sep 11, 2009 at 5:17 PM, Randall R Schulz <rschulz@...> wrote:
> Separately, I wonder how many refactorings are useful in a batch
> setting? I've only ever done automated refactorings using IDEA, so I
> have only that reference point to draw on in my own experience. Does
> anybody use or wish they could perform non-interactive refactorings?

It really depends what you mean by "batch". A rename refactoring of a
globally visible symbol has to be done atomically across an entire
codebase ... I can see perfect justification for calling that a batch
operation, even when it's being done inside, eg., Eclipse.

If what you really meant was "command-line", then I'm sure there are
plenty of Vim and Emacs users who'll tell you how much they want
something like this ;-)

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[David Bernard-3]

+1 for integration in command line (or my experimental scala-tools-server)

A suggestion for your tool to ease integration with other tools, split
it in 3 phases (at least) :
0. define the workspace/project + the request
1. generate the diff/patch  : the main work
    result available via API (+ a simple toString output)
2. apply the diff/patch

Why not doing the change directly in the source ?
* IDE could provide preview and list of changes and request confirmation
* IDE/Integrator could use its own command to apply the patch and
trigger some hooks (eg: renaming a file => renaming in the SCM and not
delete/add if supported)

Good luck.

/davidB


On Fri, Sep 11, 2009 at 18:23, Miles Sabin <miles@...> wrote:

> On Fri, Sep 11, 2009 at 5:17 PM, Randall R Schulz <rschulz@...> wrote:
>> Separately, I wonder how many refactorings are useful in a batch
>> setting? I've only ever done automated refactorings using IDEA, so I
>> have only that reference point to draw on in my own experience. Does
>> anybody use or wish they could perform non-interactive refactorings?
>
> It really depends what you mean by "batch". A rename refactoring of a
> globally visible symbol has to be done atomically across an entire
> codebase ... I can see perfect justification for calling that a batch
> operation, even when it's being done inside, eg., Eclipse.
>
> If what you really meant was "command-line", then I'm sure there are
> plenty of Vim and Emacs users who'll tell you how much they want
> something like this ;-)
>
> Cheers,
>
>
> Miles
>
> --
> Miles Sabin
> tel: [phone]
> skype:  milessabin
> http://www.chuusai.com/
> http://twitter.com/milessabin
>

[dcaoyuan]

+1

I'm working on refactoring feature on Scala for NetBeans, current progress:
1. Find usages across all opening projects.
2. Rename class/trait/object and its method/accessor across all
opening projects.

It will be really helpful for a independent refactoring project.

For NetBeans, the requirements are:
1. I'll give a (Scala Symbols#Symbol | file name | package name | etc)
 that is going to find usages or rename or move, and a set of source
files that are in projects.
2. the refactoring API should return a batch diff/patch result, in
String or a proper defined class.

Then, I can got it integrated into NetBeans refactoring framework.

Please take a look at the fields of
http://hg.netbeans.org/main/contrib/file/tip/scala.refactoring/src/org/netbeans/modules/scala/refactoring/DiffElement.scala
http://hg.netbeans.org/main/contrib/file/tip/scala.refactoring/src/org/netbeans/modules/scala/refactoring/WhereUsedElement.scala

That's what I need.

Whole Refactoring module code of Scala for NetBeans can be found at:
http://hg.netbeans.org/main/contrib/file/tip/scala.refactoring/src/org/netbeans/modules/scala/refactoring

-Caoyuan

On Sat, Sep 12, 2009 at 12:50 AM, David Bernard
<david.bernard.31@...> wrote:

> +1 for integration in command line (or my experimental scala-tools-server)
>
> A suggestion for your tool to ease integration with other tools, split
> it in 3 phases (at least) :
> 0. define the workspace/project + the request
> 1. generate the diff/patch  : the main work
>    result available via API (+ a simple toString output)
> 2. apply the diff/patch
>
> Why not doing the change directly in the source ?
> * IDE could provide preview and list of changes and request confirmation
> * IDE/Integrator could use its own command to apply the patch and
> trigger some hooks (eg: renaming a file => renaming in the SCM and not
> delete/add if supported)
>
> Good luck.
>
> /davidB
>
>
> On Fri, Sep 11, 2009 at 18:23, Miles Sabin <miles@...> wrote:
>> On Fri, Sep 11, 2009 at 5:17 PM, Randall R Schulz <rschulz@...> wrote:
>>> Separately, I wonder how many refactorings are useful in a batch
>>> setting? I've only ever done automated refactorings using IDEA, so I
>>> have only that reference point to draw on in my own experience. Does
>>> anybody use or wish they could perform non-interactive refactorings?
>>
>> It really depends what you mean by "batch". A rename refactoring of a
>> globally visible symbol has to be done atomically across an entire
>> codebase ... I can see perfect justification for calling that a batch
>> operation, even when it's being done inside, eg., Eclipse.
>>
>> If what you really meant was "command-line", then I'm sure there are
>> plenty of Vim and Emacs users who'll tell you how much they want
>> something like this ;-)
>>
>> Cheers,
>>
>>
>> Miles
>>
>> --
>> Miles Sabin
>> tel: [phone]
>> skype:  milessabin
>> http://www.chuusai.com/
>> http://twitter.com/milessabin
>>
>