Language stability (was Re: Corporate scala levels)

[Justin du coeur]

I'm going to spin off Bernd's thread, with a related question that's been much on my mind the past couple of months.

I *adore* Scala -- IMO, it's the best language currently going for serious down-and-dirty programming.  It has everything I might want in a language, comes closest to the DWIM principle for me, and while it's pretty complex, that complexity scales nicely.  (That is, you can get a great deal done in it even if you only think of it as "Java++", and know little about the stranger and more interesting bits.)

That said, I haven't been able to argue for it seriously in a work environment.  Partly that's because it's still a little new and unproven, but even more it's because the language is still a little too much in flux.  As I understand it, 2.8 has a fair number of slight but non-trivial breaking changes in it, and I gather that's not all that unusual.

So here's a question, more for thought by the group than looking for a hard and fast answer: when is the language likely to settle down to the point where it is practical for a corporate environment?  One key aspect of this is accepting that, sooner or later, the language is probably going to need to stop having significant breaking changes, even of the "this is deprecated and will be removed in two releases" sort, if it's going to achieve widespread acceptance, because corporate engineering departments are, quite reasonably, leery of anything that isn't promising backward compatibility for a *long* time to come.  That doesn't mean eliminating language evolution, but it does mean that it becomes harder.

This isn't an easy question (which is why I'm posting it to -debate), and strikes to the heart of what the language is about.  I've often heard it said that Scala is trying to be a bit "realer" than some of the more academic languages, more focused on day-to-day needs of the engineer working on ordinary real-world problems.  This meta-issue of language stability is one of the *most* important day-to-day needs, but it implies that the language will eventually start getting a bit grungy: stabilizing a language usually means that eventually, that language will become obsolete.  I think that's a good thing, and part of the natural evolution of the field -- languages grow, stabilize, gradually age and then get replaced by their successors.  (Personally, I think of Scala as the much-needed successor to Java.)

Anyway, I toss it out for thought and discussion; forgive me if it's old hat, but I haven't seen it discussed yet in my limited time with the group.  I would very much *like* to be an advocate for Scala in my workplaces, but this is the key reason why I've been reluctant to do so heretofore: until I can promise a VP of Engineering that the language is stable enough for them, I don't think I can ever win that argument...

[Ricky Clarkson]

For reasons like this, I have only bothered to argue for Scala in
small projects and for automated tests.  The reasons that are like it
are:

* Poor IDE support.  This has changed since we last discussed it at work.
* Poor documentation.  The Scala libraries we use are fantastic (only
Specs, ScalaCheck and paulp's optional) but the core library is
sparsely documented; I normally read the source instead of the
scaladocs, and I can't really suggest that for, let's say, programmers
who only code in work hours.

However, the kind of companies that are scared of breaking changes
tend to be so scared that they run the same version for years anyway
(I bet you know someone stuck on Java 1.4 or 5).  Our company isn't
perfect, but instead of being scared of change, we'll happily upgrade
when we want to, but respond quickly when customers have problems (and
catch most problems before customers see them).  Luckily, most Scala
changes will manifest themselves as compile errors, not runtime
errors, making the risk that we release buggy code because of Scala
changes quite tiny.


2009/10/23 Justin du coeur <jducoeur@...>:

> I'm going to spin off Bernd's thread, with a related question that's been
> much on my mind the past couple of months.
>
> I *adore* Scala -- IMO, it's the best language currently going for serious
> down-and-dirty programming.  It has everything I might want in a language,
> comes closest to the DWIM principle for me, and while it's pretty complex,
> that complexity scales nicely.  (That is, you can get a great deal done in
> it even if you only think of it as "Java++", and know little about the
> stranger and more interesting bits.)
>
> That said, I haven't been able to argue for it seriously in a work
> environment.  Partly that's because it's still a little new and unproven,
> but even more it's because the language is still a little too much in flux.
> As I understand it, 2.8 has a fair number of slight but non-trivial breaking
> changes in it, and I gather that's not all that unusual.
>
> So here's a question, more for thought by the group than looking for a hard
> and fast answer: when is the language likely to settle down to the point
> where it is practical for a corporate environment?  One key aspect of this
> is accepting that, sooner or later, the language is probably going to need
> to stop having significant breaking changes, even of the "this is deprecated
> and will be removed in two releases" sort, if it's going to achieve
> widespread acceptance, because corporate engineering departments are, quite
> reasonably, leery of anything that isn't promising backward compatibility
> for a *long* time to come.  That doesn't mean eliminating language
> evolution, but it does mean that it becomes harder.
>
> This isn't an easy question (which is why I'm posting it to -debate), and
> strikes to the heart of what the language is about.  I've often heard it
> said that Scala is trying to be a bit "realer" than some of the more
> academic languages, more focused on day-to-day needs of the engineer working
> on ordinary real-world problems.  This meta-issue of language stability is
> one of the *most* important day-to-day needs, but it implies that the
> language will eventually start getting a bit grungy: stabilizing a language
> usually means that eventually, that language will become obsolete.  I think
> that's a good thing, and part of the natural evolution of the field --
> languages grow, stabilize, gradually age and then get replaced by their
> successors.  (Personally, I think of Scala as the much-needed successor to
> Java.)
>
> Anyway, I toss it out for thought and discussion; forgive me if it's old
> hat, but I haven't seen it discussed yet in my limited time with the group.
> I would very much *like* to be an advocate for Scala in my workplaces, but
> this is the key reason why I've been reluctant to do so heretofore: until I
> can promise a VP of Engineering that the language is stable enough for them,
> I don't think I can ever win that argument...
>

--
Ricky Clarkson
Java and Scala Programmer, AD Holdings
[phone]
Skype: ricky_clarkson
Google Talk: ricky.clarkson@...
Google Wave: ricky.clarkson@...

[Martin Odersky]

On Fri, Oct 23, 2009 at 9:45 PM, Justin du coeur <jducoeur@...> wrote:

> I'm going to spin off Bernd's thread, with a related question that's been
> much on my mind the past couple of months.
>
> I *adore* Scala -- IMO, it's the best language currently going for serious
> down-and-dirty programming.  It has everything I might want in a language,
> comes closest to the DWIM principle for me, and while it's pretty complex,
> that complexity scales nicely.  (That is, you can get a great deal done in
> it even if you only think of it as "Java++", and know little about the
> stranger and more interesting bits.)
>
> That said, I haven't been able to argue for it seriously in a work
> environment.  Partly that's because it's still a little new and unproven,
> but even more it's because the language is still a little too much in flux.
> As I understand it, 2.8 has a fair number of slight but non-trivial breaking
> changes in it, and I gather that's not all that unusual.
>

Depends what you call unusual. The way I see it over the last two
years any breaking changes can be regarded as bug fixes, and indeed I
can't remember having seen a single report where these changes upset
anybody's code.

We certainly don't like to break code, and we won't do it on a whim.
2.8 pays down a lot of technical debt. In the rush of previous
releases certain things crept into the distribution which in
retrospect should not be there. One example is the design of arrays,
which leaked at the edges. Another was some of the additions to
collections (projections, jcl) which were rather baroque and had
surprising behavior. Originally, 2.8 was meant as a quick code
breaking fix to collections before too much code relied on them (in
particular projections). It took much longer than expected to come up
with a good alternative collections design. In the meantime other
problems were discovered and we decided to roll fixes for them into
2.8. Better break code once than on every iteration.

So, yes, 2.8 will be a difficult release in that it will break some
code. We believe this was necessary to correct certain mistakes that
crept into the distribution. We do not want to make a habit of
breaking code. And we hope that there will be enough new useful things
in 2.8 to entice people to switch to it.

We also asked our users (including the larger corporate users) whether
we should do these breaking changes and the answer we got from
everyone was to go ahead and do it now rather than later (or regret
forever not having done it).

Cheers

 -- Martin

[bearfeeder]

On Fri, Oct 23, 2009 at 1:41 PM, martin odersky <martin.odersky@...> wrote:

On Fri, Oct 23, 2009 at 9:45 PM, Justin du coeur <jducoeur@...> wrote:
> I'm going to spin off Bernd's thread, with a related question that's been
> much on my mind the past couple of months.
>
> I *adore* Scala -- IMO, it's the best language currently going for serious
> down-and-dirty programming.  It has everything I might want in a language,
> comes closest to the DWIM principle for me, and while it's pretty complex,
> that complexity scales nicely.  (That is, you can get a great deal done in
> it even if you only think of it as "Java++", and know little about the
> stranger and more interesting bits.)
>
> That said, I haven't been able to argue for it seriously in a work
> environment.  Partly that's because it's still a little new and unproven,
> but even more it's because the language is still a little too much in flux.
> As I understand it, 2.8 has a fair number of slight but non-trivial breaking
> changes in it, and I gather that's not all that unusual.
>

Depends what you call unusual. The way I see it over the last two
years any breaking changes can be regarded as bug fixes, and indeed I
can't remember having seen a single report where these changes upset
anybody's code.

We certainly don't like to break code, and we won't do it on a whim.
2.8 pays down a lot of technical debt. In the rush of previous
releases certain things crept into the distribution which in
retrospect should not be there. One example is the design of arrays,
which leaked at the edges. Another was some of the additions to
collections (projections, jcl) which were rather baroque and had
surprising behavior. Originally, 2.8 was meant as a quick code
breaking fix to collections before too much code relied on them (in
particular projections). It took much longer than expected to come up
with a good alternative collections design. In the meantime other
problems were discovered and we decided to roll fixes for them into
2.8. Better break code once than on every iteration.

So, yes, 2.8 will be a difficult release in that it will break some
code. We believe this was necessary to correct certain mistakes that
crept into the distribution. We do not want to make a habit of
breaking code. And we hope that there will be enough new useful things
in 2.8 to entice people to switch to it.

We also asked our users (including the larger corporate users) whether
we should do these breaking changes and the answer we got from
everyone was to go ahead and do it now rather than later (or regret
forever not having done it).

I've been around Scala for 3 years (well, 2 weeks shy of 3 years).  The pace of Scala language/library change has radically slowed in that period of time.

While the 2.8 shift is the largest change in the last 2 years, I heartily endorse and support it.  What it is really doing is avoiding a lot of dead-ends in the language.  If Java had had a similar change in the 1.2-1.3 timeframe, Java would still be a language that could evolve.  Instead, Java is syntactically and semantically boxed in and cannot evolve.

I am working to make sure that Lift migrates to 2.8 smoothly.  I expect that between some of the changes to how packages work and the excellent support for building one's own collections, Scala 2.8+ will have a lot of room to evolve while preserving backward (at least source) compatibility.

Thanks,

David
 

Cheers

 -- Martin

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics

[Dave Griffith]

The usual way this problem is dealt with at this point in a language's evolution is for one or more for-profit consultancies to arise capable of doing maintenance on the core tools of some fixed versions the language.   That way crucial bug fixes to non-tip versions of the language implementations can be guaranteed, which is basically all it takes to make CTOs happy.   These consultancies will often have  close-but-not-too-close relations to the language implementers, and supplement their income with other consulting or training on projects in the language, easing the way for early adopters.   Even if the consultancies rarely have to make tool fixes, just knowing that such fixes are possible and that someone exists with the capability and inclination to do them opens a lot of doors.

I'm hoping that 2.8 is going to be stable enough for that sort of ecosystem to emerge (it's certainly featureful enough).  That seems to be the intent of this release.    You've correctly identified a key problem (indeed, I believe, the key remaining problem) preventing Scala from supplanting Java in the enterprise.   Hopefully, a path to a solution can be found.

(Yes, this is a naked invitation for someone on this list to start such a consultancy.  I'm sadly otherwise booked for a while.)

[Miles Sabin]

On Fri, Oct 23, 2009 at 10:16 PM, Dave  Griffith
<dave.l.griffith@...> wrote:
> (Yes, this is a naked invitation for someone on this list to start such a
> consultancy.  I'm sadly otherwise booked for a while.)

You've just given a very good description of what I do.

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Viktor Klang]

Also, as long as bugfixes are backported I fail to see what the problem is. If you want to stay on a particular version, please do so.

Noone's forcing anyone to download new binaries...

Doesn't matter if you break backwards or not, I still know shops stuck on Java 1.3

On Fri, Oct 23, 2009 at 11:32 PM, Miles Sabin <miles@...> wrote:

On Fri, Oct 23, 2009 at 10:16 PM, Dave  Griffith
<dave.l.griffith@...> wrote:
> (Yes, this is a naked invitation for someone on this list to start such a
> consultancy.  I'm sadly otherwise booked for a while.)

You've just given a very good description of what I do.

Cheers,


Miles

--
Miles Sabin
tel: [phone]
skype:  milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Blog: klangism.blogspot.com
Twttr: viktorklang
Code: github.com/viktorklang

[Ken Bloom-2]

On Fri, 23 Oct 2009 21:00:55 +0100, Ricky Clarkson wrote:
> However, the kind of companies that are scared of breaking changes tend
> to be so scared that they run the same version for years anyway (I bet
> you know someone stuck on Java 1.4 or 5).

Consider what happens in the Python community: they release a new version
of Python every few years, and it contains incompatible changes with the
previous version. People in the community seem to accept this, and Linux
distributions understand. In Debian there are two stable versions of
Python packaged (currently) and two more in the experimental distribution
for things that aren't yet considered release worthy. People target
particular versions of the language, and indicate that their code should
be run using that version of the language.

Heck even C and C++ compilers need to be treated this way, Ruby is
treated this way, and Perl is about to accumulate a lot of breaking
changes with the advent of Perl 6.

Targeting a particular version of a language is an even easier model to
use in the Java community where platform support for sharing jars between
two programs is so abysmal that everyone just packages up all the jars
their code depends on and ships them with their code. (And even so, any
principled way to support dependencies on the Java platform is likely to
take into account the fact that there will be incompatibilities between
versions of a library.) You can look at the overall stability and
consistency of Scala 2.7 or Scala 2.8, and its long term support
prognosis if strange bugs are found whose fixes don't break source
compatibility (like the finally-block fix in Scala 2.7.7) and decide
based on that which version you want to target. Then the worry about
breaking changes becomes much more academic.


--
Chanoch (Ken) Bloom. PhD candidate. Linguistic Cognition Laboratory.
Department of Computer Science. Illinois Institute of Technology.
http://www.iit.edu/~kbloom1/

[bjohanns]

Martin Odersky wrote:

On Fri, Oct 23, 2009 at 9:45 PM, Justin du coeur <[e-mail]> wrote:
> I'm going to spin off Bernd's thread, with a related question that's been
> much on my mind the past couple of months.
>
> I *adore* Scala -- IMO, it's the best language currently going for serious
> down-and-dirty programming.  It has everything I might want in a language,
> comes closest to the DWIM principle for me, and while it's pretty complex,
> that complexity scales nicely.  (That is, you can get a great deal done in
> it even if you only think of it as "Java++", and know little about the
> stranger and more interesting bits.)
>
[...]
>
[...]
We certainly don't like to break code, and we won't do it on a whim.
[...]
So, yes, 2.8 will be a difficult release in that it will break some
code. We believe this was necessary to correct certain mistakes that
crept into the distribution. We do not want to make a habit of
breaking code. And we hope that there will be enough new useful things
in 2.8 to entice people to switch to it.

[...]
Cheers

 -- Martin

note: some shortening in the quotes indicated by [...]

I don't think that breaking code with some release is a major problem. We've seen this with any product and OS. If people responsible for the technical stack are alerted to this fact they can take appropriate steps and handle that (either by keeping stuck with the old version or by scheduling a transition project).
Agreed, this should  not happen too often but it has to happen from time to time.

What "feels" appealing to me is the scope of scala: from scripting to glue code up to full fledged applications. But at the moment I would not dear to throw it at the developer staff of some of our clients.
And this is not solely a matter of the scala language but especially a matter of the tooling around it.

The eclipse IDE I tried so far didn't convince me (other IDEs were currently not in the scope of potential clients). It is mandatory for a smooth change (not technically but psychologically) that there is enough IDE support to overcome the very first obstacles of a new language. But the intrinsic complexity of the scala syntax turns this into a very hard task for the "helper" functions (autocomplete, code assist, context jump - just name it) to provide this. The eclipse plugin and the buildin build system (2.7.x) fails at large regarding these requirements (well - this is just a personal oppinion, but I am used to see corporate program/maintenance guys as "clients" and I came to know what they expect when confronted with a new environment).

So from my point of view the "language stability"  is not only defined in terms of "core language" and "core library" compatibility but takes also the environment (or "ecosystem") into account.
And it's the scala ecosystem which causes me to refrain from recommending it at the moment. (On the plus side - this shortcomming can be solved easier than fixing a broken languange).

I still fear that the syntactic power of scala can raise maintenance and acceptance nightmares when used in a "mixed experience" staff - non of them being a scala advocate firstplace. At least when there is no easy and integrated tool around to catch some dont's (i.e. enforce coding policies) and help inexperienced newbies efficiently (newbies here: seasoned IT personal in itself but they just don't know the language yet - a fact one has to take into account when talking to them - talk to them like talking to students and you're lost).

That's why I considered something like compiler policies to "tame" (i.e. deny) some of the more involved features of scala as long as there is no sufficient IDE support around.

Resume: I'm looking forward to the results of 2.8 (final) - it's been said that this release will set some things right. For me the most important part comes along the IDE support as it is vital to language acceptance. Once the IDE gets some of the "basic" (but still involved) things right I think the *perceptance* of "stability" will come along with it. At the moment (2.7.x) things still aren't looking too good (e.g. cut & paste of an example from the book into the eclipse editor yields a "no main in..." - currently a known error but simply a nogo when introducing a new language to senior stuff).

Greeting
Bernd

[sanity]

On Fri, Oct 23, 2009 at 2:45 PM, Justin du coeur <jducoeur@...> wrote:
> but even more it's because the language is still a little too much in flux.
> As I understand it, 2.8 has a fair number of slight but non-trivial breaking
> changes in it, and I gather that's not all that unusual.

Clearly there is a trade-off.  We can have a language with some
irritating an unfixable flaws, or we can tolerate the occasional
backwards-incompatible change.

I think Java is the former, and Scala should be the latter.  At some
point Scala must make the leap, but I think it would be a real wasted
opportunity to lock things in now.

Frankly, if your's is the kind of organization that requires an API
that won't change for the remainder of the lifetime of the language -
stick with Java for the time being and let the Scala team get it
right.

Ian.

--
Ian Clarke
CEO, Uprizer Labs
Email: ian@...
Ph: [phone]