Polymorphic extension methods

[Adam Warski-3]

Hello,

I've been wondering recently about if and how to implement, as I call  
them, "polymorphic" extension methods.
In Scala, extension methods are normally implemented using implicit  
conversions. However there's a major limitation - implicits don's  
support polymorphism.

My use case is following: I have some data-only objects, lets say they  
represent shapes. They may either use inheritance (e.g. Square extends  
Shape, Circle extends Shape), implement common interfaces (Square  
implements Shape, Circle implements Shape), or traits. They are  
manipulated and persisted (for example mapped with Hibernate to a DB)  
in a backend system.

Now I want to display the shapes on different frontend systems (using  
Swing, a web app, etc). So for each of the shapes I need an  
implementation of a draw() method (quite obviously, a square is drawn  
differently than a circle ;) ). I don't want to store frontend-
specific code in the backend (so that rules out simply putting the  
method in the class), and I want each frontend to have a different  
implementation (in Swing you draw differently than in a web app).

I think the best way to do this in Scala now is to use pattern  
matching, so in each of the frontends I would have a draw(Shape)  
method like this:
def draw(shape: Shape) = shape match {
   case s: Square => drawSquare(s)
   case c: Circle => drawCircle(c)
   case ...
}

def drawSquare(s: Square) = ...
def drawCircle(s: Square) = ...

in some (probably artificially created) wrapper class "Drawer" or sth  
like this.

Having to write the pattern matching, and to create a wrapper class to  
hold those draw methods isn't quite as nice as just having the methods  
in the classes. But why not do that? If we could extend a class with a  
method (also an abstract method!) than the example above would just be:
extend class Shape with { def draw }
extend class Square with { def draw = ... }
extend class Circle with { def draw = ... }

The extend class ... with { } syntax is completely from the top of my  
head, and probably a better one can be devised. But I hope that the  
idea is clear :)

That way, it would be possible to have simple data classes, group  
different functionalities in the extension methods and still have OO  
benefits. Also it could be a good compromise between the supporters of  
anemic models and DDD (rich models). And it would prevent bloating  
classes with lots of methods for all kinds of uses that an object may  
have.

How to implement this? To make it simple for the beginning, let's say  
that we require that only sealed case classes can use extension  
methods. The compiler (or rather - an appropriate compiler plug-in)  
would first collect information on which classes define extension  
methods. Having that information, we can check that if abstract  
extension methods are used, every subclass provides an implementation  
(because we require only sealed classes, we know all subclasses).  
Then, the methods and their usages may be re-written to the wrapper-
class+case-matching+methods-per-implementation form as above.

What do you think? Do you see any problems/holes/etc? :) Maybe it has  
been discussed before (although my search of Scala's mailing lists  
didn't reveal anything)?

Adam

[Jesper Nordenberg]

See:

http://www.scala-lang.org/docu/files/IC_TECH_REPORT_200433.pdf

for a solution to exactly this problem (it's called the expression problem).

/Jesper Nordenberg

Adam Warski wrote:

> Hello,
>
> I've been wondering recently about if and how to implement, as I call
> them, "polymorphic" extension methods.
> In Scala, extension methods are normally implemented using implicit
> conversions. However there's a major limitation - implicits don's
> support polymorphism.
>
> My use case is following: I have some data-only objects, lets say they
> represent shapes. They may either use inheritance (e.g. Square extends
> Shape, Circle extends Shape), implement common interfaces (Square
> implements Shape, Circle implements Shape), or traits. They are
> manipulated and persisted (for example mapped with Hibernate to a DB) in
> a backend system.
>
> Now I want to display the shapes on different frontend systems (using
> Swing, a web app, etc). So for each of the shapes I need an
> implementation of a draw() method (quite obviously, a square is drawn
> differently than a circle ;) ). I don't want to store frontend-specific
> code in the backend (so that rules out simply putting the method in the
> class), and I want each frontend to have a different implementation (in
> Swing you draw differently than in a web app).
>
> I think the best way to do this in Scala now is to use pattern matching,
> so in each of the frontends I would have a draw(Shape) method like this:
> def draw(shape: Shape) = shape match {
>   case s: Square => drawSquare(s)
>   case c: Circle => drawCircle(c)
>   case ...
> }
>
> def drawSquare(s: Square) = ...
> def drawCircle(s: Square) = ...
>
> in some (probably artificially created) wrapper class "Drawer" or sth
> like this.
>
> Having to write the pattern matching, and to create a wrapper class to
> hold those draw methods isn't quite as nice as just having the methods
> in the classes. But why not do that? If we could extend a class with a
> method (also an abstract method!) than the example above would just be:
> extend class Shape with { def draw }
> extend class Square with { def draw = ... }
> extend class Circle with { def draw = ... }
>
> The extend class ... with { } syntax is completely from the top of my
> head, and probably a better one can be devised. But I hope that the idea
> is clear :)
>
> That way, it would be possible to have simple data classes, group
> different functionalities in the extension methods and still have OO
> benefits. Also it could be a good compromise between the supporters of
> anemic models and DDD (rich models). And it would prevent bloating
> classes with lots of methods for all kinds of uses that an object may have.
>
> How to implement this? To make it simple for the beginning, let's say
> that we require that only sealed case classes can use extension methods.
> The compiler (or rather - an appropriate compiler plug-in) would first
> collect information on which classes define extension methods. Having
> that information, we can check that if abstract extension methods are
> used, every subclass provides an implementation (because we require only
> sealed classes, we know all subclasses). Then, the methods and their
> usages may be re-written to the
> wrapper-class+case-matching+methods-per-implementation form as above.
>
> What do you think? Do you see any problems/holes/etc? :) Maybe it has
> been discussed before (although my search of Scala's mailing lists
> didn't reveal anything)?
>
> Adam
>

[Paul Phillips-3]

On Sat, Oct 31, 2009 at 07:55:42PM +0100, Jesper Nordenberg wrote:
> http://www.scala-lang.org/docu/files/IC_TECH_REPORT_200433.pdf
>
> for a solution to exactly this problem (it's called the expression problem).

And if you like your expression problems with extra fancy, see "A
Practical Solution to the Expression Families Problem":

  http://www.comlab.ox.ac.uk/files/2140/ModularVisitor.pdf

where you can enjoy type signatures like

  trait BaseHandleDefault[V[-_, _], A]

--
Paul Phillips      | The important thing here is that the music is not in
In Theory          | the piano.  And knowledge and edification is not in the
Empiricist         | computer.  The computer is simply an instrument whose
pal, i pill push   | music is ideas.  -- Alan Kay

[Adam Warski-3]

Thanks, I'll read up (after work ... ;) )

However the fact that it's a problem which has it's own name doesn't  
bode well ;)

Adam

On Oct 31, 2009, at 7:55 PM, Jesper Nordenberg wrote:

> See:
>
> http://www.scala-lang.org/docu/files/IC_TECH_REPORT_200433.pdf
>
> for a solution to exactly this problem (it's called the expression  
> problem).
>
> /Jesper Nordenberg
>
> Adam Warski wrote:
>> Hello,
>> I've been wondering recently about if and how to implement, as I  
>> call them, "polymorphic" extension methods.
>> In Scala, extension methods are normally implemented using implicit  
>> conversions. However there's a major limitation - implicits don's  
>> support polymorphism.
>> My use case is following: I have some data-only objects, lets say  
>> they represent shapes. They may either use inheritance (e.g. Square  
>> extends Shape, Circle extends Shape), implement common interfaces  
>> (Square implements Shape, Circle implements Shape), or traits. They  
>> are manipulated and persisted (for example mapped with Hibernate to  
>> a DB) in a backend system.
>> Now I want to display the shapes on different frontend systems  
>> (using Swing, a web app, etc). So for each of the shapes I need an  
>> implementation of a draw() method (quite obviously, a square is  
>> drawn differently than a circle ;) ). I don't want to store  
>> frontend-specific code in the backend (so that rules out simply  
>> putting the method in the class), and I want each frontend to have  
>> a different implementation (in Swing you draw differently than in a  
>> web app).
>> I think the best way to do this in Scala now is to use pattern  
>> matching, so in each of the frontends I would have a draw(Shape)  
>> method like this:
>> def draw(shape: Shape) = shape match {
>>  case s: Square => drawSquare(s)
>>  case c: Circle => drawCircle(c)
>>  case ...
>> }
>> def drawSquare(s: Square) = ...
>> def drawCircle(s: Square) = ...
>> in some (probably artificially created) wrapper class "Drawer" or  
>> sth like this.
>> Having to write the pattern matching, and to create a wrapper class  
>> to hold those draw methods isn't quite as nice as just having the  
>> methods in the classes. But why not do that? If we could extend a  
>> class with a method (also an abstract method!) than the example  
>> above would just be:
>> extend class Shape with { def draw }
>> extend class Square with { def draw = ... }
>> extend class Circle with { def draw = ... }
>> The extend class ... with { } syntax is completely from the top of  
>> my head, and probably a better one can be devised. But I hope that  
>> the idea is clear :)
>> That way, it would be possible to have simple data classes, group  
>> different functionalities in the extension methods and still have  
>> OO benefits. Also it could be a good compromise between the  
>> supporters of anemic models and DDD (rich models). And it would  
>> prevent bloating classes with lots of methods for all kinds of uses  
>> that an object may have.
>> How to implement this? To make it simple for the beginning, let's  
>> say that we require that only sealed case classes can use extension  
>> methods. The compiler (or rather - an appropriate compiler plug-in)  
>> would first collect information on which classes define extension  
>> methods. Having that information, we can check that if abstract  
>> extension methods are used, every subclass provides an  
>> implementation (because we require only sealed classes, we know all  
>> subclasses). Then, the methods and their usages may be re-written  
>> to the wrapper-class+case-matching+methods-per-implementation form  
>> as above.
>> What do you think? Do you see any problems/holes/etc? :) Maybe it  
>> has been discussed before (although my search of Scala's mailing  
>> lists didn't reveal anything)?
>> Adam
>

[Adam Warski-3]

Hello,

> http://www.scala-lang.org/docu/files/IC_TECH_REPORT_200433.pdf
>
> for a solution to exactly this problem (it's called the expression  
> problem).

that's some really nice things that you can do with abstract type  
members, and quite elegant; I'll have to investigate and experiment :).
Although I think that some statements about traits and classes in  
Scala are a bit out-of-date, but the paper is already 5 years old, so  
things may have changed.

To define my problem using the "right" terminology, I'd like to have a  
nice way to extend existing data types with new operations. Extending  
data types with new variants is not so important as the data types (in  
my case, but I guess in many common cases) are persistent classes (map  
to DB tables), so any new data types go into the "core"/backend module  
anyway. I even ruled that case out completely by requiring only sealed  
case classes. To put the problem in yet another terminology, I'd like  
to have multimethods polymorphic on the first parameter.

In fact, my problem can be often solved with an ordinary visitor.  
What I don't like in such an approach is that it requires quite a lot  
of boilerplate code. You need to write a visitor and an accept method  
in each class. Plus, there may be different ranges of the inheritance/
implements hierarchy that the visitor may cover, so the number of  
visitors to write and maintain would increase constantly.

The solution with the pattern-matching requires less boilerplate and  
is more maintainable, but still isn't quite as nice as just adding a  
method to a class. Hence my proposition which is in reality simply  
syntax sugar for the pattern-matching solution.

There's also one more problem with the solutions described in the  
paper (it's more of a problem in the object-decomposition one, as in  
the functional-decomposition one you simply use a visitor to implement  
new operations). They require that if you want to use e.g. an  
operation from an extended data type, you need to instantiate the  
extended class. So then it's impossible to instantiate a class in the  
backend module, send it to a frontend module and use the new operation  
there. (The backend module doesn't know anything about the frontend  
module, so it also doesn't know anything about the extensions made  
there.)

--
Adam

[Randall Schulz]

On Saturday October 31 2009, Jesper Nordenberg wrote:
> See:
>
> http://www.scala-lang.org/docu/files/IC_TECH_REPORT_200433.pdf
>
> for a solution to exactly this problem (it's called the expression
> problem).
>
> /Jesper Nordenberg

If you're willing to sacrifice static type safety within for the
relevant portion of your program that, Kiama's [1] Rewriter module is a
very elegant solution to this class of problems. It completely
separates reusable traversal control combinators (which it supplies in
abundance) from the logic of your program specific to the types it
operates upon. The only real restriction is that it is limited to case
classes (more precisely, subclasses of Product).


[1] <http://code.google.com/p/kiama/>


Randall Schulz

[nuttycom]

On Sat, Oct 31, 2009 at 12:25 PM, Adam Warski <adam@...> wrote:

> Hello,
>
> I've been wondering recently about if and how to implement, as I call them,
> "polymorphic" extension methods.
> In Scala, extension methods are normally implemented using implicit
> conversions. However there's a major limitation - implicits don's support
> polymorphism.
>
> My use case is following: I have some data-only objects, lets say they
> represent shapes. They may either use inheritance (e.g. Square extends
> Shape, Circle extends Shape), implement common interfaces (Square implements
> Shape, Circle implements Shape), or traits. They are manipulated and
> persisted (for example mapped with Hibernate to a DB) in a backend system.
>
> Now I want to display the shapes on different frontend systems (using Swing,
> a web app, etc). So for each of the shapes I need an implementation of a
> draw() method (quite obviously, a square is drawn differently than a circle
> ;) ). I don't want to store frontend-specific code in the backend (so that
> rules out simply putting the method in the class), and I want each frontend
> to have a different implementation (in Swing you draw differently than in a
> web app).
>
> I think the best way to do this in Scala now is to use pattern matching, so
> in each of the frontends I would have a draw(Shape) method like this:
> def draw(shape: Shape) = shape match {
>  case s: Square => drawSquare(s)
>  case c: Circle => drawCircle(c)
>  case ...
> }
>
> def drawSquare(s: Square) = ...
> def drawCircle(s: Square) = ...
>
> in some (probably artificially created) wrapper class "Drawer" or sth like
> this.
>
> Having to write the pattern matching, and to create a wrapper class to hold
> those draw methods isn't quite as nice as just having the methods in the
> classes. But why not do that? If we could extend a class with a method (also
> an abstract method!) than the example above would just be:
> extend class Shape with { def draw }
> extend class Square with { def draw = ... }
> extend class Circle with { def draw = ... }
>
> The extend class ... with { } syntax is completely from the top of my head,
> and probably a better one can be devised. But I hope that the idea is clear
> :)
>
> That way, it would be possible to have simple data classes, group different
> functionalities in the extension methods and still have OO benefits. Also it
> could be a good compromise between the supporters of anemic models and DDD
> (rich models). And it would prevent bloating classes with lots of methods
> for all kinds of uses that an object may have.
>
> How to implement this? To make it simple for the beginning, let's say that
> we require that only sealed case classes can use extension methods. The
> compiler (or rather - an appropriate compiler plug-in) would first collect
> information on which classes define extension methods. Having that
> information, we can check that if abstract extension methods are used, every
> subclass provides an implementation (because we require only sealed classes,
> we know all subclasses). Then, the methods and their usages may be
> re-written to the wrapper-class+case-matching+methods-per-implementation
> form as above.
>
> What do you think? Do you see any problems/holes/etc? :) Maybe it has been
> discussed before (although my search of Scala's mailing lists didn't reveal
> anything)?
>
> Adam
>

I use a technique that uses an implicit method (to supply the
conversion) with an implicit parameter (which supplies the rendering
logic) to achieve what (I think) you're talking about here. I did it
in the context of Lift bindings, but I think that the technique is
generally applicable.

Blogged about it here:
http://logji.blogspot.com/2009/09/composable-bindings-in-lift.html

Kris

[Adam Warski-3]

Hello,

>
> I use a technique that uses an implicit method (to supply the
> conversion) with an implicit parameter (which supplies the rendering
> logic) to achieve what (I think) you're talking about here. I did it
> in the context of Lift bindings, but I think that the technique is
> generally applicable.

But if I understand correctly it doesn't support polymorphic  
definitions of bind().
If you wanted to have for example different implementations for bind()  
for AuthTransaction and CaptureTransaction, then even if you  
implemented DataBinding[AuthTransaction] and DataBinding
[CaptureTransaction], still the DataBinding[Transaction] would be used  
(when doing order.transactions.flatMap(_.bind(txnTemplate)).toSeq), as  
statically that's what the compiler knows about the type, and so  
applies the appropriate implicits it can find.

Adam

[nuttycom]

On Tue, Nov 3, 2009 at 8:45 AM, Adam Warski <adam@...> wrote:

> Hello,
>
>>
>> I use a technique that uses an implicit method (to supply the
>> conversion) with an implicit parameter (which supplies the rendering
>> logic) to achieve what (I think) you're talking about here. I did it
>> in the context of Lift bindings, but I think that the technique is
>> generally applicable.
>
> But if I understand correctly it doesn't support polymorphic definitions of
> bind().
> If you wanted to have for example different implementations for bind() for
> AuthTransaction and CaptureTransaction, then even if you implemented
> DataBinding[AuthTransaction] and DataBinding[CaptureTransaction], still the
> DataBinding[Transaction] would be used (when doing
> order.transactions.flatMap(_.bind(txnTemplate)).toSeq), as statically that's
> what the compiler knows about the type, and so applies the appropriate
> implicits it can find.
>
> Adam
>

That is true; it is up to DataBinding[Transaction] to define implicit
DataBinding[AuthTransaction] and DataBinding[CaptureTransaction] in
scope and then have some means that can be used to recover the runtime
type - either a match on a sealed class or (in the case where your
entities are persisted by Hibernate and may be proxied) using a
Visitor.

If you've discarded static type information along the way somewhere,
you've got to use a runtime technique to recover it. A polymorphic
method won't help you. If you have all the static type information you
need, then the correct implicit will be selected (well, so long as
DataBinding is invariant in its type parameter.)

Kris

[Dean Wampler-2]

When I was a "pure OO programmer", I would have considered the pattern matching option evil ;) Now, I would try very hard to ensure that I only draw shapes (using your example) in one place. I would now have one place to maintain changes and one place where all the drawing logic exists. Localizing this logic is better for modularity than spreading it around in each Shape definition. That's a major plus, IMHO.

My point is that you should first try to avoid needing the extension methods, as they add some complexity and increase the testing and maintenance burden. If you can't avoid extension methods, then one of the discussed solutions should be fine.

dean

On Sat, Oct 31, 2009 at 12:25 PM, Adam Warski <adam@...> wrote:

Hello,

I've been wondering recently about if and how to implement, as I call them, "polymorphic" extension methods.
In Scala, extension methods are normally implemented using implicit conversions. However there's a major limitation - implicits don's support polymorphism.

My use case is following: I have some data-only objects, lets say they represent shapes. They may either use inheritance (e.g. Square extends Shape, Circle extends Shape), implement common interfaces (Square implements Shape, Circle implements Shape), or traits. They are manipulated and persisted (for example mapped with Hibernate to a DB) in a backend system.

Now I want to display the shapes on different frontend systems (using Swing, a web app, etc). So for each of the shapes I need an implementation of a draw() method (quite obviously, a square is drawn differently than a circle ;) ). I don't want to store frontend-specific code in the backend (so that rules out simply putting the method in the class), and I want each frontend to have a different implementation (in Swing you draw differently than in a web app).

I think the best way to do this in Scala now is to use pattern matching, so in each of the frontends I would have a draw(Shape) method like this:
def draw(shape: Shape) = shape match {
 case s: Square => drawSquare(s)
 case c: Circle => drawCircle(c)
 case ...
}

def drawSquare(s: Square) = ...
def drawCircle(s: Square) = ...

in some (probably artificially created) wrapper class "Drawer" or sth like this.

Having to write the pattern matching, and to create a wrapper class to hold those draw methods isn't quite as nice as just having the methods in the classes. But why not do that? If we could extend a class with a method (also an abstract method!) than the example above would just be:
extend class Shape with { def draw }
extend class Square with { def draw = ... }
extend class Circle with { def draw = ... }

The extend class ... with { } syntax is completely from the top of my head, and probably a better one can be devised. But I hope that the idea is clear :)

That way, it would be possible to have simple data classes, group different functionalities in the extension methods and still have OO benefits. Also it could be a good compromise between the supporters of anemic models and DDD (rich models). And it would prevent bloating classes with lots of methods for all kinds of uses that an object may have.

How to implement this? To make it simple for the beginning, let's say that we require that only sealed case classes can use extension methods. The compiler (or rather - an appropriate compiler plug-in) would first collect information on which classes define extension methods. Having that information, we can check that if abstract extension methods are used, every subclass provides an implementation (because we require only sealed classes, we know all subclasses). Then, the methods and their usages may be re-written to the wrapper-class+case-matching+methods-per-implementation form as above.

What do you think? Do you see any problems/holes/etc? :) Maybe it has been discussed before (although my search of Scala's mailing lists didn't reveal anything)?

Adam

--
Dean Wampler
coauthor of "Programming Scala" (O'Reilly)
-  http://programmingscala.com

twitter: @deanwampler, @chicagoscala
Chicago-Area Scala Enthusiasts (CASE):
-  http://groups.google.com/group/chicagoscala
-  http://www.meetup.com/chicagoscala/ (Meetings)
http://www.linkedin.com/in/deanwampler
http://www.polyglotprogramming.com
http://aquarium.rubyforge.org
http://www.contract4j.org

[Adam Warski-3]

> That is true; it is up to DataBinding[Transaction] to define implicit
> DataBinding[AuthTransaction] and DataBinding[CaptureTransaction] in
> scope and then have some means that can be used to recover the runtime
> type - either a match on a sealed class or (in the case where your
> entities are persisted by Hibernate and may be proxied) using a
> Visitor.
These two (matching/visitor) are exactly the two solutions that I  
described in my post. They work, although if used repeatedly they  
produce quite a lot of "noise" (boilerplate code; especially the  
visitors).

> If you've discarded static type information along the way somewhere,
> you've got to use a runtime technique to recover it. A polymorphic
> method won't help you. If you have all the static type information you
> need, then the correct implicit will be selected (well, so long as
> DataBinding is invariant in its type parameter.)
Sure. But you could also imagine having some syntactic sugar which  
does that for you.

--
Adam

[Adam Warski-3]

Hello,

> When I was a "pure OO programmer", I would have considered the  
> pattern matching option evil ;) Now, I would try very hard to ensure  
> that I only draw shapes (using your example) in one place. I would  
> now have one place to maintain changes and one place where all the  
> drawing logic exists. Localizing this logic is better for modularity  
> than spreading it around in each Shape definition. That's a major  
> plus, IMHO.
That's why I think "polymorphic extension method" may convince even  
pure OO programmers that a more functional approach doesn't have to be  
evil ;). I agree that it's good to draw shapes only in one place, but  
nothing stops you from writing methods for drawing different shapes in  
one file. I was even writing that the extension methods would be a  
good way to group methods with a similar responsibility in one place  
(so group methods by what they do, not which class they belong to). If  
you write a pattern matching method to draw the shapes, then if the  
actual drawing takes more than 3 lines of code, you'll probably  
extract two private methods (drawCircle & drawSquare) anyway.  
Otherwise, the method will quickly get very long.

--
Adam