How to make Scala compile faster?

[John Ky]

Hi Scala list,

I have a small project, which I build in Eclipse using the Scala plugin.  It's gotten to a size where compilation would take a long time - about a minute or so.  At this point it is becoming counterproductive because on many occasions, I need to do a clean and force a full build after some minor code changes because something in the compilation process became corrupted.

Is there any way, I can recode it to compile faster?  For instance - if I added return types declarations to my methods, would Scala compile my code faster?

Cheers,

-John

[Peter Robinett-3]

Are you using fsc (http://www.scala-lang.org/docu/files/tools/fsc.html)? I don't know if Eclipse's plugin uses it, but it's worth checking out.

Peter Robinett

On Tue, Oct 27, 2009 at 10:59 PM, John Ky <newhoggy@...> wrote:

Hi Scala list,

I have a small project, which I build in Eclipse using the Scala plugin.  It's gotten to a size where compilation would take a long time - about a minute or so.  At this point it is becoming counterproductive because on many occasions, I need to do a clean and force a full build after some minor code changes because something in the compilation process became corrupted.

Is there any way, I can recode it to compile faster?  For instance - if I added return types declarations to my methods, would Scala compile my code faster?

Cheers,

-John

[Miles Sabin]

On Wed, Oct 28, 2009 at 5:59 AM, John Ky <newhoggy@...> wrote:
> I have a small project, which I build in Eclipse using the Scala plugin.

Version?

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[John Ky]

Hi Miles,

Eclipse SDK    3.5.1.M20090917-0800    org.eclipse.sdk.ide
Scala Eclipse Plugin    2.7.6.final    ch.epfl.lamp.sdt.feature.group

Cheers,

-John

On Wed, Oct 28, 2009 at 9:41 PM, Miles Sabin <miles@...> wrote:

On Wed, Oct 28, 2009 at 5:59 AM, John Ky <newhoggy@...> wrote:
> I have a small project, which I build in Eclipse using the Scala plugin.

Version?

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Miles Sabin]

On Wed, Oct 28, 2009 at 10:45 AM, John Ky <newhoggy@...> wrote:
> Eclipse SDK    3.5.1.M20090917-0800    org.eclipse.sdk.ide
> Scala Eclipse Plugin    2.7.6.final    ch.epfl.lamp.sdt.feature.group

Thanks.

You're seeing bugs in Eclipse's dependency management which are common
to all 2.7.x and earlier releases. This has been completely replaced
for 2.8 and should be a great deal more accurate in the future.

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[John Ky]

Hi Miles,

2.8 works very well.

Thankyou.

-John

On Wed, Oct 28, 2009 at 9:55 PM, Miles Sabin <miles@...> wrote:

On Wed, Oct 28, 2009 at 10:45 AM, John Ky <newhoggy@...> wrote:
> Eclipse SDK    3.5.1.M20090917-0800    org.eclipse.sdk.ide
> Scala Eclipse Plugin    2.7.6.final    ch.epfl.lamp.sdt.feature.group

Thanks.

You're seeing bugs in Eclipse's dependency management which are common
to all 2.7.x and earlier releases. This has been completely replaced
for 2.8 and should be a great deal more accurate in the future.

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Miles Sabin]

On Wed, Oct 28, 2009 at 1:02 PM, John Ky <newhoggy@...> wrote:
> 2.8 works very well.
>
> Thankyou.

Umm ... well, I wasn't actually recommending you to switch to 2.8 just
yet ... it's unreleased, beta quality, all bets are off, etc., etc.

Nevertheless, if it works well for you, then that's great :-)

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Stefan Zeiger]

Miles Sabin wrote:
> You're seeing bugs in Eclipse's dependency management which are common
> to all 2.7.x and earlier releases. This has been completely replaced
> for 2.8 and should be a great deal more accurate in the future.

Is this new dependency management in 2.8 supposed to work already (or
rather two or three weeks ago when I last updated to the latest nightly
-- I haven't gotten around to working on ScalaQuery since then) or is it
still work in progress?

To me, it doesn't look any better than in 2.7 and I have to do a full
rebuild all the time. For example:
- Rename a class and save -> Sources which use the old name are not
recompiled and they don't show up as errors until I rebuild
- Change some fields and save -> Running the app throws an
IncompatibleClassChange error because dependent sources were not recompiled.

--
Stefan Zeiger | http://szeiger.de | http://novocode.com

[Mark Harrah]

I took a good look at the source for RefinedBuildManager recently.  I tried
some things out with BuildManagerTest and came up with several examples that
failed to be recompiled correctly.

I'd like to provide some tests that demonstrate the issues, but I couldn't find
any examples to go off of in the associated commits to the build manager
classes.  Is it possible to write such tests for partest?

As an example of the issues I found, adding a subclass to a sealed hierarchy
will not cause pattern matches on the hierarchy to be recompiled. This means
that warnings that a match is not exhaustive will not be printed, for example.

Thanks,
Mark

On Saturday 31 October 2009, Stefan Zeiger wrote:

> Miles Sabin wrote:
> > You're seeing bugs in Eclipse's dependency management which are common
> > to all 2.7.x and earlier releases. This has been completely replaced
> > for 2.8 and should be a great deal more accurate in the future.
>
> Is this new dependency management in 2.8 supposed to work already (or
> rather two or three weeks ago when I last updated to the latest nightly
> -- I haven't gotten around to working on ScalaQuery since then) or is it
> still work in progress?
>
> To me, it doesn't look any better than in 2.7 and I have to do a full
> rebuild all the time. For example:
> - Rename a class and save -> Sources which use the old name are not
> recompiled and they don't show up as errors until I rebuild
> - Change some fields and save -> Running the app throws an
> IncompatibleClassChange error because dependent sources were not
>  recompiled.

[Ismael Juma]

Hi Mark,

On Sun, 2009-11-01 at 22:30 -0500, Mark Harrah wrote:
> I took a good look at the source for RefinedBuildManager recently.  I tried
> some things out with BuildManagerTest and came up with several examples that
> failed to be recompiled correctly.

I'm happy to see this.

I was just saying on IRC the other day that it would be nice if you
would take a look at scalac's dependency manager given your experience
with sbt's one (and the fact that the latter works well according to
many people's experience). :)

Best,
Ismael

P.S. Someone else will have to answer your questions, as I don't know
the answer. ;)

[Iulian Dragos-2]

On Mon, Nov 2, 2009 at 4:30 AM, Mark Harrah <harrah@...> wrote:

I'd like to provide some tests that demonstrate the issues, but I couldn't find
any examples to go off of in the associated commits to the build manager
classes.  Is it possible to write such tests for partest?

Not at the moment, as I assume it involves several compilation rounds, modifying files in between. Partest was designed with a more static view of testing. How do you test sbt?
 

As an example of the issues I found, adding a subclass to a sealed hierarchy
will not cause pattern matches on the hierarchy to be recompiled. This means
that warnings that a match is not exhaustive will not be printed, for example.

It would be good to have tickets for these issues. Of course, a way to automatically test the build manager (or even the Eclipse plugin) is necessary, just that for the moment we don't have it.

iulian
 

Thanks,
Mark

On Saturday 31 October 2009, Stefan Zeiger wrote:
> Miles Sabin wrote:
> > You're seeing bugs in Eclipse's dependency management which are common
> > to all 2.7.x and earlier releases. This has been completely replaced
> > for 2.8 and should be a great deal more accurate in the future.
>
> Is this new dependency management in 2.8 supposed to work already (or
> rather two or three weeks ago when I last updated to the latest nightly
> -- I haven't gotten around to working on ScalaQuery since then) or is it
> still work in progress?
>
> To me, it doesn't look any better than in 2.7 and I have to do a full
> rebuild all the time. For example:
> - Rename a class and save -> Sources which use the old name are not
> recompiled and they don't show up as errors until I rebuild
> - Change some fields and save -> Running the app throws an
> IncompatibleClassChange error because dependent sources were not
>  recompiled.

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

[Mark Harrah]

On Monday 02 November 2009, Iulian Dragos wrote:
> On Mon, Nov 2, 2009 at 4:30 AM, Mark Harrah <harrah@...> wrote:
> > I'd like to provide some tests that demonstrate the issues, but I
> > couldn't find
> > any examples to go off of in the associated commits to the build manager
> > classes.  Is it possible to write such tests for partest?
>
> Not at the moment, as I assume it involves several compilation rounds,
> modifying files in between. Partest was designed with a more static view of
> testing. How do you test sbt?

Right, multiple compilation rounds and file changes.  I wrote a test framework
that reads the commands and actions to run from a script.  Commands are things
like copy, delete, or test for file existence.  Actions are any sbt action,
like compile, run, or any custom action defined for the project being tested.  
A test script might look like:

 $ copy-file changes/First.scala src/A.scala
 > compile [success]
 $ copy-file changes/Second.scala src/A.scala
 > compile [failure]

> > As an example of the issues I found, adding a subclass to a sealed
> > hierarchy
> > will not cause pattern matches on the hierarchy to be recompiled. This
> > means
> > that warnings that a match is not exhaustive will not be printed, for
> > example.
>
> It would be good to have tickets for these issues. Of course, a way to
> automatically test the build manager (or even the Eclipse plugin) is
> necessary, just that for the moment we don't have it.

Certainly, it was just that there were several issues and they are much easier
to explain with test cases.  Also, I saw that Miles made some changes that
looked like he might have fixed an issue, but I don't want to have to manually
retry these examples every time there is a change.

-Mark

[Mark Harrah]

On Monday 02 November 2009, Ismael Juma wrote:

> Hi Mark,
>
> On Sun, 2009-11-01 at 22:30 -0500, Mark Harrah wrote:
> > I took a good look at the source for RefinedBuildManager recently.  I
> > tried some things out with BuildManagerTest and came up with several
> > examples that failed to be recompiled correctly.
>
> I'm happy to see this.
>
> I was just saying on IRC the other day that it would be nice if you
> would take a look at scalac's dependency manager given your experience
> with sbt's one (and the fact that the latter works well according to
> many people's experience). :)

sbt's partial recompilation is the most conservative of all solutions I've
seen, so is the most correct short of recompiling everything, but doesn't try
anything too fancy to be faster.

The latest refined build manager is very aggressive in order to be faster but
still correct.  There are two things it does towards this:
 1) recompile dependencies only when public API has changed
 2) recompile only affected dependencies and not necessarily transitive
dependencies

I think the first one is mostly achieved by the refined build manager (I will
file the issue I found) and is something I think should go in sbt.

I get the impression that the second one needs to be spec'd (not necessarily
in The Spec) because it is tricky and needs review to ensure all cases are
caught, even if the build manager has to be documented as not being able to
handle certain cases.  Most issues come from #2.

-Mark

[Ismael Juma]

On Mon, 2009-11-02 at 08:18 -0500, Mark Harrah wrote:
> I think the first one is mostly achieved by the refined build manager (I will
> file the issue I found) and is something I think should go in sbt.

Thanks for the explanation. Do you think it makes sense for sbt to use
the refined build manager at some point? If not, why not?

Best,
Ismael

[Mark Harrah]

On Monday 02 November 2009, Ismael Juma wrote:
> On Mon, 2009-11-02 at 08:18 -0500, Mark Harrah wrote:
> > I think the first one is mostly achieved by the refined build manager (I
> > will file the issue I found) and is something I think should go in sbt.
>
> Thanks for the explanation. Do you think it makes sense for sbt to use
> the refined build manager at some point? If not, why not?

The refined build manager can only be used as a resident compiler, which means
no, it can't be used in sbt right now.

I like the first part of the refined build manager (detecting public API
changes), though.  I'd like to use something similar in sbt, but again, it has
to be modified to work in non-resident mode.

Ignoring the resident compiler issue, I'd want to see a spec or something
(SID?) on the second part before using it in sbt.  I don't care if I have to
write it, but it is something I'd want to see reviewed.

My current plan is to keep transitive recompilation but be smarter about when
it is triggered (i.e., use the first part of the refined build manager).

-Mark

[Miles Sabin]

On Mon, Nov 2, 2009 at 1:53 PM, Mark Harrah <harrah@...> wrote:
> On Monday 02 November 2009, Ismael Juma wrote:
>> Thanks for the explanation. Do you think it makes sense for sbt to use
>> the refined build manager at some point? If not, why not?
>
> The refined build manager can only be used as a resident compiler, which means
> no, it can't be used in sbt right now.

Sure, but I don't think it'd take much to get it to provide most of
what sbt needs ... do you see any serious roadblocks?

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Andrew Gaydenko]

On Monday 02 November 2009 17:15:58 Miles Sabin wrote:

> On Mon, Nov 2, 2009 at 1:53 PM, Mark Harrah <harrah@...> wrote:
> > On Monday 02 November 2009, Ismael Juma wrote:
> >> Thanks for the explanation. Do you think it makes sense for sbt to use
> >> the refined build manager at some point? If not, why not?
> >
> > The refined build manager can only be used as a resident compiler, which
> > means no, it can't be used in sbt right now.
>
> Sure, but I don't think it'd take much to get it to provide most of
> what sbt needs ... do you see any serious roadblocks?
>
> Cheers,
>
>
> Miles
>

Miles,

Can all these new refined build manager's goodnesses be used to replace
current ant fsc task (or be wrapped as a new one)?

[Miles Sabin]

On Mon, Nov 2, 2009 at 3:11 PM, Andrew Gaydenko <a@...> wrote:
> Can all these new refined build manager's goodnesses be used to replace
> current ant fsc task (or be wrapped as a new one)?

That would be a question for Iulian ... in principle I would have thought so.

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Iulian Dragos-2]

On Mon, Nov 2, 2009 at 4:11 PM, Andrew Gaydenko <a@...> wrote:

On Monday 02 November 2009 17:15:58 Miles Sabin wrote:
> On Mon, Nov 2, 2009 at 1:53 PM, Mark Harrah <harrah@...> wrote:
> > On Monday 02 November 2009, Ismael Juma wrote:
> >> Thanks for the explanation. Do you think it makes sense for sbt to use
> >> the refined build manager at some point? If not, why not?
> >
> > The refined build manager can only be used as a resident compiler, which
> > means no, it can't be used in sbt right now.
>
> Sure, but I don't think it'd take much to get it to provide most of
> what sbt needs ... do you see any serious roadblocks?
>
> Cheers,
>
>
> Miles
>

Miles,

Can all these new refined build manager's goodnesses be used to replace
current ant fsc task (or be wrapped as a new one)?

The current build manager cuts some corners in order to keep things simple. The biggest barrier to using it in fsc (or in the ant task) is that it requires running in resident mode. That means it keeps state around (external references of each managed file, dependency graph between compilation units, current types and members of classes). References and the dependency graph can be easily saved/restored, the problem is with 'members'. It uses compiler symbols and types, and therefore needs a compiler instance. They are used for building a change set between an old and a new version of the same file. In principle it could be rewritten to not rely on them, but a lot of functionality already in scalac would need to be duplicateed (for instance, subtype checks). There's a tradeoff between accuracy and time to code these features. Unfortunately I had not much time (getting very close to thesis writing) so I did what I thought would work fine for Eclipse. It can certainly be improved, but I had to (temporarily) hand over even maitnenance.

Mark, spec of the build manager is a very good idea, I'm sure it will reveal missing cases and be good documentation. Simply I have not enough time now...

iulian

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

[Iulian Dragos-2]

On Mon, Nov 2, 2009 at 2:07 PM, Mark Harrah <harrah@...> wrote:

On Monday 02 November 2009, Iulian Dragos wrote:
> On Mon, Nov 2, 2009 at 4:30 AM, Mark Harrah <harrah@...> wrote:
> > I'd like to provide some tests that demonstrate the issues, but I
> > couldn't find
> > any examples to go off of in the associated commits to the build manager
> > classes.  Is it possible to write such tests for partest?
>
> Not at the moment, as I assume it involves several compilation rounds,
> modifying files in between. Partest was designed with a more static view of
> testing. How do you test sbt?

Right, multiple compilation rounds and file changes.  I wrote a test framework
that reads the commands and actions to run from a script.  Commands are things
like copy, delete, or test for file existence.  Actions are any sbt action,
like compile, run, or any custom action defined for the project being tested.
A test script might look like:

 $ copy-file changes/First.scala src/A.scala
 > compile [success]
 $ copy-file changes/Second.scala src/A.scala
 > compile [failure]

Wonderful, is this a standalone project that we could use, or part of sbt?

thanks,
iulian

> > As an example of the issues I found, adding a subclass to a sealed
> > hierarchy
> > will not cause pattern matches on the hierarchy to be recompiled. This
> > means
> > that warnings that a match is not exhaustive will not be printed, for
> > example.
>
> It would be good to have tickets for these issues. Of course, a way to
> automatically test the build manager (or even the Eclipse plugin) is
> necessary, just that for the moment we don't have it.

Certainly, it was just that there were several issues and they are much easier
to explain with test cases.  Also, I saw that Miles made some changes that
looked like he might have fixed an issue, but I don't want to have to manually
retry these examples every time there is a change.

-Mark

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais