Software Transactional Memory in Scala: Convenience and Standardization

[Ben Hutchison-3]

Ive been experimenting with Software Transactional Memory (STM) in
Scala. There are clearly some issues around Convenience of APIs and
Standardization, that I'd like to discuss.

== Convenience ==

Ive looked at Nathan Bronson's CCSTM and Daniel Spiewaks blog Impl.
Both have a similar API, both with the same problems, which I think
arise from constraints of Scala itself.

CCSTM [http://github.com/nbronson/ccstm/blob/master/src/main/scala/edu/stanford/ppl/ccstm/STM.scala]
Daniel Spiewak [http://www.codecommit.com/blog/scala/improving-the-stm-multi-version-concurrency-control]

Example from CCSTM:

import edu.stanford.ppl.ccstm._
import edu.stanford.ppl.ccstm.STM._

 val a = Ref(10)
 val b = Ref(0)
 atomic(performTransfer(a, b, 5)(_))

 def performTransfer(from: Ref[Int], to: Ref[Int], amount:
Int)(implicit txn: Txn) {
 from := !from - amount
 to := !to + amount
 }

Problems:

1. You cannot pass a code block to STM.atomic(), because a block
cannot declare an implict Txn parameter. So atomic can only wrap a
single method invocation, or you must use a quite different syntax
(see CCSTM link above for alternative). Already this is resulting in
lots of local function defs in my code, to wrap up 2-3 lines as a
function.
2. You seem to need to append the "(_)" token to the end of method
invocation, to pass the Txn. OK, but it looks a little inelegant. Wish
I could make it go away.

Can anyone do better?

== Standardization ==

STM appears a classic case where API standardization yields greatest
benefit, because

1. It touches many places in the application code
2. It exposes a very narrow API, the same operations (eg atomic(),
Ref, retry()) are repeated over and over again.

I started working with CCSTM, and after just a day I was struck by
just how deeply coupled, almost inseparable, my code had become to
CCSTM. That situation worries me.

Could and STM API be practically standardized, such that an
application could swap between implementations?

-Ben

[Ben Hutchison-3]

On Mon, Nov 2, 2009 at 5:06 PM, Ben Hutchison <brhutchison@...> wrote:

> Problems:
>
> 1. You cannot pass a code block to STM.atomic(), because a block
> cannot declare an implict Txn parameter. So atomic can only wrap a
> single method invocation, or you must use a quite different syntax
> (see CCSTM link above for alternative). Already this is resulting in
> lots of local function defs in my code, to wrap up 2-3 lines as a
> function.
> 2. You seem to need to append the "(_)" token to the end of method
> invocation, to pass the Txn. OK, but it looks a little inelegant. Wish
> I could make it go away.
>
> Can anyone do better?

Both problems above relate to propagating the txn object along with
the thread. Implicit params are a statically-typed Scala-specific way
to do this, but in current form they make STM code quite clumsy.
Ideally, that could be somehow fixed directly.

Alternative 1:

Trade-away some static type checking for API convenience and use the
JavaWay: store the txn in a Thread Local variable?

This would allow for atomic {...} to wrap basically anything, with no
boilerplate needed to pass the txn around.

It has plenty of precedents in Java-land; Java enterprise building
blocks like Spring and Hibernate use this approach. But type checking
is lost, and we aspire to do better than that.

Alterative 2: STM Monad

Transactions must occur in a STM monad, which would hold the txn. My
monadic fu is not up to spelling out the details, but I suspect Jonas
Boner's TransactionMonad is a useful starting point. It too propagates
the Transaction in a thread local, and the application uses it via a
for-comprehension:

http://github.com/dpp/liftweb/blob/3783b9e2200cc57dd72baa1bd8cabdb1365ee923/lift-jta/src/main/scala/net/liftweb/transaction/TransactionContext.scala

-Ben

[Ben Hutchison-3]

On Mon, Nov 2, 2009 at 9:45 PM, Ben Hutchison <brhutchison@...> wrote:

> Problems:
>
> 1. You cannot pass a code block to STM.atomic(), because a block
> cannot declare an implict Txn parameter. [snip..]
> 2. You seem to need to append the "(_)" token to the end of method
> invocation, to pass the Txn.
>
> Alternative 1:
>
> Trade-away some static type checking for API convenience and use the
> JavaWay: store the txn in a Thread Local variable? [snip]
>
> Alterative 2: STM Monad
>

Ok, last post in this monologue, I promise.

Ive analyzed myself back to where I started: both my alternatives seem
inferior to supporting implicit params to anonymous functions. That
fixes both problems 1 & 2.

http://lampsvn.epfl.ch/trac/scala/ticket/1492

I cant see a practical & type-checked way to do STM in Scala unless
this is supported

... but Id love to be wrong, if anyone's found a way?

-Ben

[andrew cooke-3]

I am not sure if this is related, or clueless, or helpful, but I have
been wondering if there is any way to do something like this -
http://gilesbowkett.blogspot.com/2007/10/ruby-language-extension-with-keyword.html
- in Scala.

The idea is that there is some object that "owns" a scope, so any
unbound methods in that scope are taken from that object (not sure I
am explaining it well - see link above).  This is useful for DSLs
because the methods can form a mini-language while the object provides
shared state.  I think it's also related to Monads in some way (the
object can "accumulate" state in the same way as a Monad).  And, in
your case, the object would be the transaction.

However, it's more about reducing the amount of repeated typing than
reducing the coupling in the code.

Andrew


2009/11/2 Ben Hutchison <brhutchison@...>:

> On Mon, Nov 2, 2009 at 9:45 PM, Ben Hutchison <brhutchison@...> wrote:
>> Problems:
>>
>> 1. You cannot pass a code block to STM.atomic(), because a block
>> cannot declare an implict Txn parameter. [snip..]
>> 2. You seem to need to append the "(_)" token to the end of method
>> invocation, to pass the Txn.
>>
>> Alternative 1:
>>
>> Trade-away some static type checking for API convenience and use the
>> JavaWay: store the txn in a Thread Local variable? [snip]
>>
>> Alterative 2: STM Monad
>>
>
> Ok, last post in this monologue, I promise.
>
> Ive analyzed myself back to where I started: both my alternatives seem
> inferior to supporting implicit params to anonymous functions. That
> fixes both problems 1 & 2.
>
> http://lampsvn.epfl.ch/trac/scala/ticket/1492
>
> I cant see a practical & type-checked way to do STM in Scala unless
> this is supported
>
> ... but Id love to be wrong, if anyone's found a way?
>
> -Ben
>

[Jesper Nordenberg]

Ben Hutchison wrote:

> Ive analyzed myself back to where I started: both my alternatives seem
> inferior to supporting implicit params to anonymous functions. That
> fixes both problems 1 & 2.
>
> http://lampsvn.epfl.ch/trac/scala/ticket/1492
>
> I cant see a practical & type-checked way to do STM in Scala unless
> this is supported
>
> ... but Id love to be wrong, if anyone's found a way?

It's possible to work around this limitation, although the syntax is not
the nicest:

abstract class WithTransaction {
   implicit val transaction = createTransaction()
   abstract def p : Unit

   // Oversimplified transaction handling
   fn
   transaction.commit
}

new WithTransaction { def p = {...} }

/Jesper Nordenberg

[Jesper Nordenberg]

Jesper Nordenberg wrote:
>   // Oversimplified transaction handling
>   fn
     ^^

Should be p of course.

/Jesper Nordenberg

[Ben Hutchison-3]

On Mon, Nov 2, 2009 at 11:51 PM, Jesper Nordenberg <megagurka@...> wrote:

> It's possible to work around this limitation, although the syntax is not the
> nicest:
>
> abstract class WithTransaction {
>  implicit val transaction = createTransaction()
>  abstract def p : Unit
>
>  // Oversimplified transaction handling
>  fn
>  transaction.commit
> }
>
> new WithTransaction { def p = {...} }

Thanks for the input.

My complaint against that mechanism is that I cannot see how to
support transaction blocks that return a value. It only seems to work
for the Unit case?

-Ben

[Ben Hutchison-3]

On Tue, Nov 3, 2009 at 12:08 AM, Ben Hutchison <brhutchison@...> wrote:
> My complaint against that mechanism is that I cannot see how to
> support transaction blocks that return a value. It only seems to work
> for the Unit case?

//Actually, this should work.. (Cf CCSTM's Atomic class)
abstract class WithTransaction[T] {
implicit val transaction = createTransaction()
abstract def p : T

def run: T = {
  val result = p
  transaction.commit
  result
}
}

val result:ResultType = new WithTransaction[ResultType] { def p = {...} }.run

-Ben

[Jesper Nordenberg]

Ben Hutchison wrote:

> On Tue, Nov 3, 2009 at 12:08 AM, Ben Hutchison <brhutchison@...> wrote:
>> My complaint against that mechanism is that I cannot see how to
>> support transaction blocks that return a value. It only seems to work
>> for the Unit case?
>
> //Actually, this should work.. (Cf CCSTM's Atomic class)
> abstract class WithTransaction[T] {
> implicit val transaction = createTransaction()
> abstract def p : T
>
> def run: T = {
>   val result = p
>   transaction.commit
>   result
> }
> }
>
> val result:ResultType = new WithTransaction[ResultType] { def p = {...} }.run

Yes, but now it's even more clumsy to use :) Maybe there is a smarter
way to do this...

/Jesper Nordenberg

[Josh Suereth]

I was debating making a SID that allowed the following:


def withTransaction[T](f : Transaction => A) : A


withTransaction { implicit tx =>

   atomic(  someOp )
   atomic( someOtherOp )

   tx.commit()
}


Does this seem to be what you want?  I think it's a slightly minor change (allows you to declare function arguments as implicitly available on the scope instead of having do to this:

withTransaction { tx =>
    implicit val itx = tx
    ....
}


If there was a way to reduce this even further without adding unneeded complexity to the language (or not interacting well with  other features), I think we should look into it.

- Josh

On Mon, Nov 2, 2009 at 8:29 AM, Jesper Nordenberg <megagurka@...> wrote:

Ben Hutchison wrote:

On Tue, Nov 3, 2009 at 12:08 AM, Ben Hutchison <brhutchison@...> wrote:

My complaint against that mechanism is that I cannot see how to
support transaction blocks that return a value. It only seems to work
for the Unit case?

//Actually, this should work.. (Cf CCSTM's Atomic class)
abstract class WithTransaction[T] {
implicit val transaction = createTransaction()
abstract def p : T

def run: T = {
 val result = p
 transaction.commit
 result
}
}

val result:ResultType = new WithTransaction[ResultType] { def p = {...} }.run

Yes, but now it's even more clumsy to use :) Maybe there is a smarter way to do this...

/Jesper Nordenberg

[Jesper Nordenberg]

Yes, this is the same as ticket #1492 I think. The question is if there
also should be a concept of function objects that takes implicit
arguments, for example should this be allowed:

val fn = (implicit i : Int) => i * 2
implicit val i = 3
fn

If not, the semantics of implicit parameters differ between methods and
function objects.

And should anonymous functions with multiple parameter lists be allowed:

val fn = (i : Int)(implicit j : Int) => i * j

or do you have to write:

val fn = (i : Int) => (implicit j : Int) => i * j

/Jesper Nordenberg

Josh Suereth wrote:

> I was debating making a SID that allowed the following:
>
>
> def withTransaction[T](f : Transaction => A) : A
>
>
> withTransaction { implicit tx =>
>
>    atomic(  someOp )
>    atomic( someOtherOp )
>
>    tx.commit()
> }
>
>
> Does this seem to be what you want?  I think it's a slightly minor
> change (allows you to declare function arguments as implicitly available
> on the scope instead of having do to this:
>
> withTransaction { tx =>
>     implicit val itx = tx
>     ....
> }
>
>
> If there was a way to reduce this even further without adding unneeded
> complexity to the language (or not interacting well with  other
> features), I think we should look into it.

[Josh Suereth]

I think not here, as it doesn't really make sense.  The primary gain is having the implicit available inside the scope of the function.

To do otherwise would imply needing to have implicitness available in the type:  i.e.

Function[Foo, implicit Bar] -> And that leads to strangeness...  I didn't want to go that far in my SID, as I don't find that useful.


- Josh

On Mon, Nov 2, 2009 at 9:00 AM, Jesper Nordenberg <megagurka@...> wrote:

Yes, this is the same as ticket #1492 I think. The question is if there also should be a concept of function objects that takes implicit arguments, for example should this be allowed:

val fn = (implicit i : Int) => i * 2
implicit val i = 3
fn

If not, the semantics of implicit parameters differ between methods and function objects.

And should anonymous functions with multiple parameter lists be allowed:

val fn = (i : Int)(implicit j : Int) => i * j

or do you have to write:

val fn = (i : Int) => (implicit j : Int) => i * j

/Jesper Nordenberg

Josh Suereth wrote:

I was debating making a SID that allowed the following:


def withTransaction[T](f : Transaction => A) : A


withTransaction { implicit tx =>

  atomic(  someOp )
  atomic( someOtherOp )

  tx.commit()
}


Does this seem to be what you want?  I think it's a slightly minor change (allows you to declare function arguments as implicitly available on the scope instead of having do to this:

withTransaction { tx =>
   implicit val itx = tx
   ....
}


If there was a way to reduce this even further without adding unneeded complexity to the language (or not interacting well with  other features), I think we should look into it.

[Jonas Bonér-6]

I've implemented support for STM in the Akka project. It currently only supports STM in the context of Actors (Transactors):

http://wiki.github.com/jboner/akka/reference-software-transactional-memory-stm

But it's on the roadmap (very soon) to wrap up the TX management stuff in a Monad similar to what I did for Lift JTA: 

http://github.com/dpp/liftweb/blob/3783b9e2200cc57dd72baa1bd8cabdb1365ee923/lift-jta/src/main/scala/net/liftweb/transaction/TransactionContext.scala

It implements STM through managed references (and immutable data, including persistent datastructures), the Clojure way. But is based on Multiverse which is a more general purpose STM. I might take advantage of more of Multiverse features later, but I'm not sure since I think that the simplicity of the Clojure STM and it's view on state in general is the best way to approach the problem.

/Jonas

2009/11/2 Ben Hutchison <brhutchison@...>

Ive been experimenting with Software Transactional Memory (STM) in
Scala. There are clearly some issues around Convenience of APIs and
Standardization, that I'd like to discuss.

== Convenience ==

Ive looked at Nathan Bronson's CCSTM and Daniel Spiewaks blog Impl.
Both have a similar API, both with the same problems, which I think
arise from constraints of Scala itself.

CCSTM [http://github.com/nbronson/ccstm/blob/master/src/main/scala/edu/stanford/ppl/ccstm/STM.scala]
Daniel Spiewak [http://www.codecommit.com/blog/scala/improving-the-stm-multi-version-concurrency-control]

Example from CCSTM:

import edu.stanford.ppl.ccstm._
import edu.stanford.ppl.ccstm.STM._

 val a = Ref(10)
 val b = Ref(0)
 atomic(performTransfer(a, b, 5)(_))

 def performTransfer(from: Ref[Int], to: Ref[Int], amount:
Int)(implicit txn: Txn) {
 from := !from - amount
 to := !to + amount
 }

Problems:

1. You cannot pass a code block to STM.atomic(), because a block
cannot declare an implict Txn parameter. So atomic can only wrap a
single method invocation, or you must use a quite different syntax
(see CCSTM link above for alternative). Already this is resulting in
lots of local function defs in my code, to wrap up 2-3 lines as a
function.
2. You seem to need to append the "(_)" token to the end of method
invocation, to pass the Txn. OK, but it looks a little inelegant. Wish
I could make it go away.

Can anyone do better?

== Standardization ==

STM appears a classic case where API standardization yields greatest
benefit, because

1. It touches many places in the application code
2. It exposes a very narrow API, the same operations (eg atomic(),
Ref, retry()) are repeated over and over again.

I started working with CCSTM, and after just a day I was struck by
just how deeply coupled, almost inseparable, my code had become to
CCSTM. That situation worries me.

Could and STM API be practically standardized, such that an
application could swap between implementations?

-Ben

--
Jonas Bonér

twitter: @jboner
blog:    http://jonasboner.com
work:   http://scalablesolutions.se
code:   http://github.com/jboner
code:   http://akkasource.org
also:    http://letitcrash.com

[Jonas Bonér-6]

Actually you can use the STM outside actors already (I'm just a bit tired a bit so I forgot that I implemented that some time ago :-) ).

Use the STM directly using 'atomic' blocks and 'run-orElse' blocks:

import se.scalablesolutions.akka.stm.Transaction._

atomic {

  .. // do something within a transaction

}

run {

  .. // try to do something

} orElse {

  .. // if transaction clashes try do do something else to minimize contention

}

The transactions manage (enforce usage of) the TransactionalRef reference, which you can use directly with immutable data (it has a nice monadic API) or one of the Map and Vector transactional datastructures which wraps a persistent datastructure managed by a TransactionalRef.

object TransactionalState {

  def newMap[K, V] = TransactionalMap[K, V]()

  def newVector[T] = TransactionalVector[T]()

  def newRef[T] = TransactionalRef[T]()

}

Hope it gives some insight into what I'm doing.

/Jonas 

 

2009/11/2 Jonas Bonér <lists@...>

I've implemented support for STM in the Akka project. It currently only supports STM in the context of Actors (Transactors):

http://wiki.github.com/jboner/akka/reference-software-transactional-memory-stm

But it's on the roadmap (very soon) to wrap up the TX management stuff in a Monad similar to what I did for Lift JTA: 

http://github.com/dpp/liftweb/blob/3783b9e2200cc57dd72baa1bd8cabdb1365ee923/lift-jta/src/main/scala/net/liftweb/transaction/TransactionContext.scala

It implements STM through managed references (and immutable data, including persistent datastructures), the Clojure way. But is based on Multiverse which is a more general purpose STM. I might take advantage of more of Multiverse features later, but I'm not sure since I think that the simplicity of the Clojure STM and it's view on state in general is the best way to approach the problem.

/Jonas

2009/11/2 Ben Hutchison <brhutchison@...>

Ive been experimenting with Software Transactional Memory (STM) in
Scala. There are clearly some issues around Convenience of APIs and
Standardization, that I'd like to discuss.

== Convenience ==

Ive looked at Nathan Bronson's CCSTM and Daniel Spiewaks blog Impl.
Both have a similar API, both with the same problems, which I think
arise from constraints of Scala itself.

CCSTM [http://github.com/nbronson/ccstm/blob/master/src/main/scala/edu/stanford/ppl/ccstm/STM.scala]
Daniel Spiewak [http://www.codecommit.com/blog/scala/improving-the-stm-multi-version-concurrency-control]

Example from CCSTM:

import edu.stanford.ppl.ccstm._
import edu.stanford.ppl.ccstm.STM._

 val a = Ref(10)
 val b = Ref(0)
 atomic(performTransfer(a, b, 5)(_))

 def performTransfer(from: Ref[Int], to: Ref[Int], amount:
Int)(implicit txn: Txn) {
 from := !from - amount
 to := !to + amount
 }

Problems:

1. You cannot pass a code block to STM.atomic(), because a block
cannot declare an implict Txn parameter. So atomic can only wrap a
single method invocation, or you must use a quite different syntax
(see CCSTM link above for alternative). Already this is resulting in
lots of local function defs in my code, to wrap up 2-3 lines as a
function.
2. You seem to need to append the "(_)" token to the end of method
invocation, to pass the Txn. OK, but it looks a little inelegant. Wish
I could make it go away.

Can anyone do better?

== Standardization ==

STM appears a classic case where API standardization yields greatest
benefit, because

1. It touches many places in the application code
2. It exposes a very narrow API, the same operations (eg atomic(),
Ref, retry()) are repeated over and over again.

I started working with CCSTM, and after just a day I was struck by
just how deeply coupled, almost inseparable, my code had become to
CCSTM. That situation worries me.

Could and STM API be practically standardized, such that an
application could swap between implementations?

-Ben

--
Jonas Bonér

twitter: @jboner
blog:    http://jonasboner.com
work:   http://scalablesolutions.se
code:   http://github.com/jboner
code:   http://akkasource.org
also:    http://letitcrash.com

--
Jonas Bonér

twitter: @jboner
blog:    http://jonasboner.com
work:   http://scalablesolutions.se
code:   http://github.com/jboner
code:   http://akkasource.org
also:    http://letitcrash.com

[Jonas Bonér-6]

Update. I went ahead and added a monadic api to the transaction.

This API and the high-order fun API is for those who wants to do explicit programmatic tx management. 

And/or those that wants to use the STM outside the scope of Actors.  

It means that transactions can be used in a for-comprehension.

Since the TransactionalRef is already monadic, they can be used together in a for-comprehension.

Here are some examples from the ScalaDoc. 

 * Example of atomic transaction management using for comprehensions (monadic usage):

 *

 * <pre>

 * import se.scalablesolutions.akka.stm.Transaction._

 * for (tx <- Transaction) {

 *   ... // do transactional stuff

 * }

 *

 * val result = for (tx <- Transaction) yield {

 *   ... // do transactional stuff yielding a result

 * }

 * </pre>

 *

 * Example of using Transaction and TransactionalRef in for comprehensions (monadic usage):

 *

 * <pre>

 * // For example, if you have a List with TransactionalRef

 * val refs: List[TransactionalRef] = ...

 *

 * // You can use them together with Transaction in a for comprehension since TransactionalRef is also monadic

 * for {

 *   tx <- Transaction

 *   ref <- refs

 * } {

 *   ... // use the ref inside a transaction

 * }

 *

 * val result = for {

 *   tx <- Transaction

 *   ref <- refs

 * } yield {

 *   ... // use the ref inside a transaction, yield a result

 * }

 * </pre>

/Jonas

2009/11/2 Jonas Bonér <lists@...>

Actually you can use the STM outside actors already (I'm just a bit tired a bit so I forgot that I implemented that some time ago :-) ).

Use the STM directly using 'atomic' blocks and 'run-orElse' blocks:

import se.scalablesolutions.akka.stm.Transaction._

atomic {

  .. // do something within a transaction

}

run {

  .. // try to do something

} orElse {

  .. // if transaction clashes try do do something else to minimize contention

}

The transactions manage (enforce usage of) the TransactionalRef reference, which you can use directly with immutable data (it has a nice monadic API) or one of the Map and Vector transactional datastructures which wraps a persistent datastructure managed by a TransactionalRef.

object TransactionalState {

  def newMap[K, V] = TransactionalMap[K, V]()

  def newVector[T] = TransactionalVector[T]()

  def newRef[T] = TransactionalRef[T]()

}

Hope it gives some insight into what I'm doing.

/Jonas 

 

2009/11/2 Jonas Bonér <lists@...>

I've implemented support for STM in the Akka project. It currently only supports STM in the context of Actors (Transactors):

http://wiki.github.com/jboner/akka/reference-software-transactional-memory-stm

But it's on the roadmap (very soon) to wrap up the TX management stuff in a Monad similar to what I did for Lift JTA: 

http://github.com/dpp/liftweb/blob/3783b9e2200cc57dd72baa1bd8cabdb1365ee923/lift-jta/src/main/scala/net/liftweb/transaction/TransactionContext.scala

It implements STM through managed references (and immutable data, including persistent datastructures), the Clojure way. But is based on Multiverse which is a more general purpose STM. I might take advantage of more of Multiverse features later, but I'm not sure since I think that the simplicity of the Clojure STM and it's view on state in general is the best way to approach the problem.

/Jonas

2009/11/2 Ben Hutchison <brhutchison@...>

Ive been experimenting with Software Transactional Memory (STM) in
Scala. There are clearly some issues around Convenience of APIs and
Standardization, that I'd like to discuss.

== Convenience ==

Ive looked at Nathan Bronson's CCSTM and Daniel Spiewaks blog Impl.
Both have a similar API, both with the same problems, which I think
arise from constraints of Scala itself.

CCSTM [http://github.com/nbronson/ccstm/blob/master/src/main/scala/edu/stanford/ppl/ccstm/STM.scala]
Daniel Spiewak [http://www.codecommit.com/blog/scala/improving-the-stm-multi-version-concurrency-control]

Example from CCSTM:

import edu.stanford.ppl.ccstm._
import edu.stanford.ppl.ccstm.STM._

 val a = Ref(10)
 val b = Ref(0)
 atomic(performTransfer(a, b, 5)(_))

 def performTransfer(from: Ref[Int], to: Ref[Int], amount:
Int)(implicit txn: Txn) {
 from := !from - amount
 to := !to + amount
 }

Problems:

1. You cannot pass a code block to STM.atomic(), because a block
cannot declare an implict Txn parameter. So atomic can only wrap a
single method invocation, or you must use a quite different syntax
(see CCSTM link above for alternative). Already this is resulting in
lots of local function defs in my code, to wrap up 2-3 lines as a
function.
2. You seem to need to append the "(_)" token to the end of method
invocation, to pass the Txn. OK, but it looks a little inelegant. Wish
I could make it go away.

Can anyone do better?

== Standardization ==

STM appears a classic case where API standardization yields greatest
benefit, because

1. It touches many places in the application code
2. It exposes a very narrow API, the same operations (eg atomic(),
Ref, retry()) are repeated over and over again.

I started working with CCSTM, and after just a day I was struck by
just how deeply coupled, almost inseparable, my code had become to
CCSTM. That situation worries me.

Could and STM API be practically standardized, such that an
application could swap between implementations?

-Ben

--
Jonas Bonér

twitter: @jboner
blog:    http://jonasboner.com
work:   http://scalablesolutions.se
code:   http://github.com/jboner
code:   http://akkasource.org
also:    http://letitcrash.com

--
Jonas Bonér

twitter: @jboner
blog:    http://jonasboner.com
work:   http://scalablesolutions.se
code:   http://github.com/jboner
code:   http://akkasource.org
also:    http://letitcrash.com

--
Jonas Bonér

twitter: @jboner
blog:    http://jonasboner.com
work:   http://scalablesolutions.se
code:   http://github.com/jboner
code:   http://akkasource.org
also:    http://letitcrash.com

[Meredith Gregory]

Dear Jonas,

This is really cool. i have a question about the semantics of nested comprehensions. What is the intended semantics of this pseudo code?

import se.scalablesolutions.akka.stm.Transaction._

 for (txOuter <- OuterTransaction) {

   ... // do outer transactional stuff
   for (txInner <- InnerTransaction) {

   ... // do inner transactional stuff

   }

 }

Best wishes,

--greg

On Mon, Nov 2, 2009 at 1:39 PM, Jonas Bonér <lists@...> wrote:

Update. I went ahead and added a monadic api to the transaction.

This API and the high-order fun API is for those who wants to do explicit programmatic tx management. 

And/or those that wants to use the STM outside the scope of Actors.  

It means that transactions can be used in a for-comprehension.

Since the TransactionalRef is already monadic, they can be used together in a for-comprehension.

Here are some examples from the ScalaDoc. 

 * Example of atomic transaction management using for comprehensions (monadic usage):

 *

 * <pre>

 * import se.scalablesolutions.akka.stm.Transaction._

 * for (tx <- Transaction) {

 *   ... // do transactional stuff

 * }

 *

 * val result = for (tx <- Transaction) yield {

 *   ... // do transactional stuff yielding a result

 * }

 * </pre>

 *

 * Example of using Transaction and TransactionalRef in for comprehensions (monadic usage):

 *

 * <pre>

 * // For example, if you have a List with TransactionalRef

 * val refs: List[TransactionalRef] = ...

 *

 * // You can use them together with Transaction in a for comprehension since TransactionalRef is also monadic

 * for {

 *   tx <- Transaction

 *   ref <- refs

 * } {

 *   ... // use the ref inside a transaction

 * }

 *

 * val result = for {

 *   tx <- Transaction

 *   ref <- refs

 * } yield {

 *   ... // use the ref inside a transaction, yield a result

 * }

 * </pre>

/Jonas

2009/11/2 Jonas Bonér <lists@...>

Actually you can use the STM outside actors already (I'm just a bit tired a bit so I forgot that I implemented that some time ago :-) ).

Use the STM directly using 'atomic' blocks and 'run-orElse' blocks:

import se.scalablesolutions.akka.stm.Transaction._

atomic {

  .. // do something within a transaction

}

run {

  .. // try to do something

} orElse {

  .. // if transaction clashes try do do something else to minimize contention

}

The transactions manage (enforce usage of) the TransactionalRef reference, which you can use directly with immutable data (it has a nice monadic API) or one of the Map and Vector transactional datastructures which wraps a persistent datastructure managed by a TransactionalRef.

object TransactionalState {

  def newMap[K, V] = TransactionalMap[K, V]()

  def newVector[T] = TransactionalVector[T]()

  def newRef[T] = TransactionalRef[T]()

}

Hope it gives some insight into what I'm doing.

/Jonas 

 

2009/11/2 Jonas Bonér <lists@...>

I've implemented support for STM in the Akka project. It currently only supports STM in the context of Actors (Transactors):

http://wiki.github.com/jboner/akka/reference-software-transactional-memory-stm

But it's on the roadmap (very soon) to wrap up the TX management stuff in a Monad similar to what I did for Lift JTA: 

http://github.com/dpp/liftweb/blob/3783b9e2200cc57dd72baa1bd8cabdb1365ee923/lift-jta/src/main/scala/net/liftweb/transaction/TransactionContext.scala

It implements STM through managed references (and immutable data, including persistent datastructures), the Clojure way. But is based on Multiverse which is a more general purpose STM. I might take advantage of more of Multiverse features later, but I'm not sure since I think that the simplicity of the Clojure STM and it's view on state in general is the best way to approach the problem.

/Jonas

2009/11/2 Ben Hutchison <brhutchison@...>

Ive been experimenting with Software Transactional Memory (STM) in
Scala. There are clearly some issues around Convenience of APIs and
Standardization, that I'd like to discuss.

== Convenience ==

Ive looked at Nathan Bronson's CCSTM and Daniel Spiewaks blog Impl.
Both have a similar API, both with the same problems, which I think
arise from constraints of Scala itself.

CCSTM [http://github.com/nbronson/ccstm/blob/master/src/main/scala/edu/stanford/ppl/ccstm/STM.scala]
Daniel Spiewak [http://www.codecommit.com/blog/scala/improving-the-stm-multi-version-concurrency-control]

Example from CCSTM:

import edu.stanford.ppl.ccstm._
import edu.stanford.ppl.ccstm.STM._

 val a = Ref(10)
 val b = Ref(0)
 atomic(performTransfer(a, b, 5)(_))

 def performTransfer(from: Ref[Int], to: Ref[Int], amount:
Int)(implicit txn: Txn) {
 from := !from - amount
 to := !to + amount
 }

Problems:

1. You cannot pass a code block to STM.atomic(), because a block
cannot declare an implict Txn parameter. So atomic can only wrap a
single method invocation, or you must use a quite different syntax
(see CCSTM link above for alternative). Already this is resulting in
lots of local function defs in my code, to wrap up 2-3 lines as a
function.
2. You seem to need to append the "(_)" token to the end of method
invocation, to pass the Txn. OK, but it looks a little inelegant. Wish
I could make it go away.

Can anyone do better?

== Standardization ==

STM appears a classic case where API standardization yields greatest
benefit, because

1. It touches many places in the application code
2. It exposes a very narrow API, the same operations (eg atomic(),
Ref, retry()) are repeated over and over again.

I started working with CCSTM, and after just a day I was struck by
just how deeply coupled, almost inseparable, my code had become to
CCSTM. That situation worries me.

Could and STM API be practically standardized, such that an
application could swap between implementations?

-Ben

--
Jonas Bonér

twitter: @jboner
blog:    http://jonasboner.com
work:   http://scalablesolutions.se
code:   http://github.com/jboner
code:   http://akkasource.org
also:    http://letitcrash.com

--
Jonas Bonér

twitter: @jboner
blog:    http://jonasboner.com
work:   http://scalablesolutions.se
code:   http://github.com/jboner
code:   http://akkasource.org
also:    http://letitcrash.com

--
Jonas Bonér

twitter: @jboner
blog:    http://jonasboner.com
work:   http://scalablesolutions.se
code:   http://github.com/jboner
code:   http://akkasource.org
also:    http://letitcrash.com

--
L.G. Meredith
Managing Partner
Biosimilarity LLC
1219 NW 83rd St
Seattle, WA 98117

+1 206.650.3740

http://biosimilarity.blogspot.com

[Seth Tisue]

>>>>> "Josh" == Josh Suereth <joshua.suereth@...> writes:

 Josh> withTransaction { tx => implicit val itx = tx ....  }

If you don't want to have to make up a name, you can do:

withTransaction { tx => implicit val _ = tx ....  }

--
Seth Tisue @ Northwestern University / http://tisue.net
lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/

[Daniel Sobral]

Well, my 2cents on this whole thing is: if the STM solution isn't composable, then it isn't a solution. 

In particular, if inside transation f I call g on a library which, unknown to me, also do a transaction, the whole thing must work.

That is the great thing that STM enables, and non-composable STM solutions are dead ends.

On Mon, Nov 2, 2009 at 10:09 AM, Ben Hutchison <brhutchison@...> wrote:

On Mon, Nov 2, 2009 at 9:45 PM, Ben Hutchison <brhutchison@...> wrote:
> Problems:
>
> 1. You cannot pass a code block to STM.atomic(), because a block

> cannot declare an implict Txn parameter. [snip..]

> 2. You seem to need to append the "(_)" token to the end of method
> invocation, to pass the Txn.
>

> Alternative 1:
>
> Trade-away some static type checking for API convenience and use the

> JavaWay: store the txn in a Thread Local variable? [snip]
>
> Alterative 2: STM Monad
>

Ok, last post in this monologue, I promise.

Ive analyzed myself back to where I started: both my alternatives seem
inferior to supporting implicit params to anonymous functions. That
fixes both problems 1 & 2.

http://lampsvn.epfl.ch/trac/scala/ticket/1492

I cant see a practical & type-checked way to do STM in Scala unless
this is supported

... but Id love to be wrong, if anyone's found a way?

-Ben

--
Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.

[Josh Suereth]

no, the whole point was to reduce as much boilerplate as possible.

- josh

On Mon, Nov 2, 2009 at 6:30 PM, Seth Tisue <seth@...> wrote:

>>>>> "Josh" == Josh Suereth <joshua.suereth@...> writes:

 Josh> withTransaction { tx => implicit val itx = tx ....  }

If you don't want to have to make up a name, you can do:

withTransaction { tx => implicit val _ = tx ....  }

--
Seth Tisue @ Northwestern University / http://tisue.net
lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/

[Seth Tisue]

>>>>> "Josh" == Josh Suereth <joshua.suereth@...> writes:

 Josh> no, the whole point was to reduce as much boilerplate as
 Josh> possible.  - josh

Oh, I know.  But if your proposal is not accepted at least you have this
tiny consolation.

--
Seth Tisue @ Northwestern University / http://tisue.net
lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/