final transient fields serialization

[Pawel Veselov]

Hi,

it again caught my attention, and I though that may be there is something that can be done about this.
The issue is obvious -- having 'final transient' instance fields makes little sense if the object is ever serialized.
Logically, there may be perfect reasoning behind making an instance field final, as well as transient, in which case there is then no mechanism to reinitialize this field on object deserialization.

It seems that it would be nice if either the final fields were initialized in a separate block that would be executed on deserialization, or if readObject() could set them. After all you can have a code block that sets the final fields. Not sure how feasible that is, but IMHO, that is a short coming.

--
With best of best regards
Pawel S. Veselov

[David M. Lloyd-3]

On 11/09/2009 03:30 PM, Pawel Veselov wrote:
> Hi,
>
> it again caught my attention, and I though that may be there is
> something that can be done about this.
> The issue is obvious -- having 'final transient' instance fields makes
> little sense if the object is ever serialized.
> Logically, there may be perfect reasoning behind making an instance
> field final, as well as transient, in which case there is then no
> mechanism to reinitialize this field on object deserialization.

I've used final transient fields before to hold values which are not
relevant on the remote side (and thus can be null or 0), but point taken...

> It seems that it would be nice if either the final fields were
> initialized in a separate block that would be executed on
> deserialization, or if readObject() could set them. After all you can
> have a code block that sets the final fields. Not sure how feasible that
> is, but IMHO, that is a short coming.

One possible problem with this is that changing a final field might have
some JMM implications (case in point, CopyOnWriteArrayList uses
sun.misc.Unsafe#putObjectVolatile() to reinitialize the transient final
Lock field, though I don't see anywhere that ObjectInputStream itself takes
such precautions; one would think that java.lang.reflect.Field would take
care of this for you, but perhaps it does not).

A workaround for this shortcoming is as follows.  Instead of using a
transient final field with a custom readObject(), use a regular final field
and add a writeObject() method which uses PutFields to change its value to
a marker object.  The marker object should have a protected readResolve()
method which constructs the correct value for the field on the remote side.
  Using this trick might have odd results though.  For example, you can
cause two objects to have final references to each other, which is
ordinarily not possible without reflection.

Alternately, in some cases the actual new value might be computable on the
writing side.  In either case there may be a bandwidth penalty however.

You could also use writeReplace() to replace the entire object with a
minimal marker, which in turn does readResolve() on the remote side to
construct the proper instance (using a constructor, thus it would be able
to initialize all fields).  This approach fails in certain cyclic cases
though, because the readResolve() isn't executed until after the entire
object was deserialized, so any backreferences to the replaced object will
be filled in with the marker object until then.

- DML

[David Holmes - Sun Microsystems]

Pawel,

Pawel Veselov said the following on 11/10/09 07:30:
> it again caught my attention, and I though that may be there is
> something that can be done about this.
> The issue is obvious -- having 'final transient' instance fields makes
> little sense if the object is ever serialized.
> Logically, there may be perfect reasoning behind making an instance
> field final, as well as transient, in which case there is then no
> mechanism to reinitialize this field on object deserialization.

Not quite true. This problem - that final fields can only be set during
true construction and not during the pseudo-construction that occurs
during deserialization - has been realized for a long time. As part of
the Java 5 update we (I think it was done JSR-133) put in place the
mechanism whereby you can use reflection to set a final field provided
that setAccessible(true) has been invoked for that field. This is of
course a limited solution as you must have the security capability to
invoke setAccessible(true).

JSR-133 also addresses the Java Memory Model issues concerning
deserialization of objects with final fields - see Section 17.5.3 of the
Java Language Specification. (The notion of a "freeze action" on a final
field was in part motivated by the deserialization issue).

David Holmes

> It seems that it would be nice if either the final fields were
> initialized in a separate block that would be executed on
> deserialization, or if readObject() could set them. After all you can
> have a code block that sets the final fields. Not sure how feasible that
> is, but IMHO, that is a short coming.
>
> --
> With best of best regards
> Pawel S. Veselov

[Remi Forax]

Le 10/11/2009 00:38, David M. Lloyd a écrit :

> On 11/09/2009 03:30 PM, Pawel Veselov wrote:
>> Hi,
>>
>> it again caught my attention, and I though that may be there is
>> something that can be done about this.
>> The issue is obvious -- having 'final transient' instance fields makes
>> little sense if the object is ever serialized.
>> Logically, there may be perfect reasoning behind making an instance
>> field final, as well as transient, in which case there is then no
>> mechanism to reinitialize this field on object deserialization.
>
> I've used final transient fields before to hold values which are not
> relevant on the remote side (and thus can be null or 0), but point
> taken...
>
>> It seems that it would be nice if either the final fields were
>> initialized in a separate block that would be executed on
>> deserialization, or if readObject() could set them. After all you can
>> have a code block that sets the final fields. Not sure how feasible that
>> is, but IMHO, that is a short coming.
>
> One possible problem with this is that changing a final field might
> have some JMM implications (case in point, CopyOnWriteArrayList uses
> sun.misc.Unsafe#putObjectVolatile() to reinitialize the transient
> final Lock field, though I don't see anywhere that ObjectInputStream
> itself takes such precautions; one would think that
> java.lang.reflect.Field would take care of this for you, but perhaps
> it does not).

It does :)
Chnaging a volatile or a final field by reflection is done with a
put...Volatile().

[...]

>
> - DML

Rémi

[Peter Jones-21]

Pawel,

Sorry for the late followup, but you might also want to read these RFEs:

http://bugs.sun.com/view_bug.do?bug_id=6379948
http://bugs.sun.com/view_bug.do?bug_id=6252102

-- Peter


On Nov 9, 2009, at 6:54 PM, David Holmes - Sun Microsystems wrote:

> Pawel,
>
> Pawel Veselov said the following on 11/10/09 07:30:
>> it again caught my attention, and I though that may be there is  
>> something that can be done about this.
>> The issue is obvious -- having 'final transient' instance fields  
>> makes little sense if the object is ever serialized.
>> Logically, there may be perfect reasoning behind making an instance  
>> field final, as well as transient, in which case there is then no  
>> mechanism to reinitialize this field on object deserialization.
>
> Not quite true. This problem - that final fields can only be set  
> during true construction and not during the pseudo-construction that  
> occurs during deserialization - has been realized for a long time.  
> As part of the Java 5 update we (I think it was done JSR-133) put in  
> place the mechanism whereby you can use reflection to set a final  
> field provided that setAccessible(true) has been invoked for that  
> field. This is of course a limited solution as you must have the  
> security capability to invoke setAccessible(true).
>
> JSR-133 also addresses the Java Memory Model issues concerning  
> deserialization of objects with final fields - see Section 17.5.3 of  
> the Java Language Specification. (The notion of a "freeze action" on  
> a final field was in part motivated by the deserialization issue).
>
> David Holmes
>
>> It seems that it would be nice if either the final fields were  
>> initialized in a separate block that would be executed on  
>> deserialization, or if readObject() could set them. After all you  
>> can have a code block that sets the final fields. Not sure how  
>> feasible that is, but IMHO, that is a short coming.
>> --
>> With best of best regards
>> Pawel S. Veselov

[Peter Jones-21]

On Nov 10, 2009, at 2:30 AM, Rémi Forax wrote:
> Le 10/11/2009 00:38, David M. Lloyd a écrit :
>> On 11/09/2009 03:30 PM, Pawel Veselov wrote:
>>
[snip]

>>> It seems that it would be nice if either the final fields were
>>> initialized in a separate block that would be executed on
>>> deserialization, or if readObject() could set them. After all you  
>>> can
>>> have a code block that sets the final fields. Not sure how  
>>> feasible that
>>> is, but IMHO, that is a short coming.
>>
>> One possible problem with this is that changing a final field might  
>> have some JMM implications (case in point, CopyOnWriteArrayList  
>> uses sun.misc.Unsafe#putObjectVolatile() to reinitialize the  
>> transient final Lock field, though I don't see anywhere that  
>> ObjectInputStream itself takes such precautions; one would think  
>> that java.lang.reflect.Field would take care of this for you, but  
>> perhaps it does not).
>
> It does :)
> Chnaging a volatile or a final field by reflection is done with a  
> put...Volatile().

Yes for java.lang.reflect.Field, but David's comment reminds me of  
this bug:

http://bugs.sun.com/view_bug.do?bug_id=6647361

which I'm somewhat embarrassed to have neglected after filing...

-- Peter

[Pawel Veselov]

Hi Peter,

not at all, I was late on replying to all the answers I've got myself.

Anyway, my thoughts around this were that I wouldn't consider it reasonable to put that much, and that kind of code around re-instantiation of final transient fields, at least for the sheer sake of them being final. Reflection means referencing fields using string values, and also places significant overhead in the instructions that need to be executed (comparing to what it took to create an object), the code that needs to be written (makes for poor templating), and the kind of the operations that need to be performed (needing security access, reflection itself is a shady operation for a high level user code).

However, I don't really see any good way out of this situation, as final fields are guaranteed instantiation during object construction, but there is effectively no limit on how it can get assigned, and what other objects would it use during such.

I would think that it would be reasonable to then allow the readObject() to re-assign final transient fields under the same rules that currently apply to setting final fields by constructors. I understand that this is as bad of an idea, because readObject() can be called by other means, however, this can be made into a compiler warning, or something, and throw some VM errors if a final field is ever modified outside of the instantiation or deserialization process. I know this is somewhat lame, but I'm not the VM engineer :) But I still believe that the current state of things with final and transient modifiers is somewhat troublesome.

Thanks,

  Pawel.

On Sun, Nov 29, 2009 at 1:05 PM, Peter Jones <pcj@...> wrote:

Pawel,

Sorry for the late followup, but you might also want to read these RFEs:

http://bugs.sun.com/view_bug.do?bug_id=6379948
http://bugs.sun.com/view_bug.do?bug_id=6252102

-- Peter

On Nov 9, 2009, at 6:54 PM, David Holmes - Sun Microsystems wrote:

Pawel,

Pawel Veselov said the following on 11/10/09 07:30:

it again caught my attention, and I though that may be there is something that can be done about this.
The issue is obvious -- having 'final transient' instance fields makes little sense if the object is ever serialized.
Logically, there may be perfect reasoning behind making an instance field final, as well as transient, in which case there is then no mechanism to reinitialize this field on object deserialization.

Not quite true. This problem - that final fields can only be set during true construction and not during the pseudo-construction that occurs during deserialization - has been realized for a long time. As part of the Java 5 update we (I think it was done JSR-133) put in place the mechanism whereby you can use reflection to set a final field provided that setAccessible(true) has been invoked for that field. This is of course a limited solution as you must have the security capability to invoke setAccessible(true).

JSR-133 also addresses the Java Memory Model issues concerning deserialization of objects with final fields - see Section 17.5.3 of the Java Language Specification. (The notion of a "freeze action" on a final field was in part motivated by the deserialization issue).

David Holmes

It seems that it would be nice if either the final fields were initialized in a separate block that would be executed on deserialization, or if readObject() could set them. After all you can have a code block that sets the final fields. Not sure how feasible that is, but IMHO, that is a short coming.
--
With best of best regards
Pawel S. Veselov

--
With best of best regards
Pawel S. Veselov

[Peter Jones-21]

Pawel,

I agree that the current reflection approach is unwieldy and otherwise  
problematic.

Regarding a language solution, though, I don't see how to avoid making  
a mess of the definite assignment/unassignment semantics for final  
fields.  (Of course deserialization itself effectively subverts that  
DA at runtime, and reflection DU, but at least the language is  
unaware.)  It's hard to say much without a specific proposal.  But I  
think that a language solution that includes detailed knowledge of  
serialization semantics[*] would be highly undesirable, not only for  
language purity reasons but also so that any needed evolution of those  
library semantics is not locked with evolution of the language spec.  
I could imagine some general way to let a (private) method be declared  
to have constructor-like final field setting ability.  But readObject  
methods couldn't be required to have constructor-like DA semantics,  
because a OIS.defaultReadObject invocation reflectively sets non-
transient final instance fields.  So it would need to have constructor-
like DU semantics but not DA-- again, seems messy.  And practically  
speaking, I suspect that even a workable proposal would likely be  
deemed too esoteric to fit within the necessarily-constrained budgets  
for future language changes (that's only personal speculation).

So I think that the best hope is an improved reflective library  
solution.  Thinking out loud: say, a factory method that looks up a  
field in the immediate caller's class with no permission requirement,  
throws unchecked NoSuchFieldError instead of NoSuchFieldException, and  
produces an object that can be used to set the field even if it's  
final.  Might be nice if the object were type-parameterized, but that  
would rule out actually using java.lang.reflect.Field-- could be sort  
of like the java.util.concurrent.atomic.Atomic*FieldUpdater APIs  
(hmm...).  It would not address your string lookup issue (field  
literals would be helpful), but I doubt that the performance overhead  
of the reflective field set is much of a concern.  Note that, as far  
as I know, pursuing something like this is not in current JDK plans.

Cheers,
-- Peter

[*] As described in the RFEs referenced below, the problem is more  
nuanced than just with "transient" fields, such as if the serializable  
class declares serialPersistentFields and/or uses the GetField API.


On Nov 29, 2009, at 5:12 PM, Pawel Veselov wrote:

>
> Hi Peter,
>
> not at all, I was late on replying to all the answers I've got myself.
>
> Anyway, my thoughts around this were that I wouldn't consider it  
> reasonable to put that much, and that kind of code around re-
> instantiation of final transient fields, at least for the sheer sake  
> of them being final. Reflection means referencing fields using  
> string values, and also places significant overhead in the  
> instructions that need to be executed (comparing to what it took to  
> create an object), the code that needs to be written (makes for poor  
> templating), and the kind of the operations that need to be  
> performed (needing security access, reflection itself is a shady  
> operation for a high level user code).
>
> However, I don't really see any good way out of this situation, as  
> final fields are guaranteed instantiation during object  
> construction, but there is effectively no limit on how it can get  
> assigned, and what other objects would it use during such.
>
> I would think that it would be reasonable to then allow the  
> readObject() to re-assign final transient fields under the same  
> rules that currently apply to setting final fields by constructors.  
> I understand that this is as bad of an idea, because readObject()  
> can be called by other means, however, this can be made into a  
> compiler warning, or something, and throw some VM errors if a final  
> field is ever modified outside of the instantiation or  
> deserialization process. I know this is somewhat lame, but I'm not  
> the VM engineer :) But I still believe that the current state of  
> things with final and transient modifiers is somewhat troublesome.
>
> Thanks,
>   Pawel.
>
> On Sun, Nov 29, 2009 at 1:05 PM, Peter Jones <pcj@...>  
> wrote:
> Pawel,
>
> Sorry for the late followup, but you might also want to read these  
> RFEs:
>
> http://bugs.sun.com/view_bug.do?bug_id=6379948
> http://bugs.sun.com/view_bug.do?bug_id=6252102
>
> -- Peter
>
>
>
> On Nov 9, 2009, at 6:54 PM, David Holmes - Sun Microsystems wrote:
>
> Pawel,
>
> Pawel Veselov said the following on 11/10/09 07:30:
> it again caught my attention, and I though that may be there is  
> something that can be done about this.
> The issue is obvious -- having 'final transient' instance fields  
> makes little sense if the object is ever serialized.
> Logically, there may be perfect reasoning behind making an instance  
> field final, as well as transient, in which case there is then no  
> mechanism to reinitialize this field on object deserialization.
>
> Not quite true. This problem - that final fields can only be set  
> during true construction and not during the pseudo-construction that  
> occurs during deserialization - has been realized for a long time.  
> As part of the Java 5 update we (I think it was done JSR-133) put in  
> place the mechanism whereby you can use reflection to set a final  
> field provided that setAccessible(true) has been invoked for that  
> field. This is of course a limited solution as you must have the  
> security capability to invoke setAccessible(true).
>
> JSR-133 also addresses the Java Memory Model issues concerning  
> deserialization of objects with final fields - see Section 17.5.3 of  
> the Java Language Specification. (The notion of a "freeze action" on  
> a final field was in part motivated by the deserialization issue).
>
> David Holmes
>
> It seems that it would be nice if either the final fields were  
> initialized in a separate block that would be executed on  
> deserialization, or if readObject() could set them. After all you  
> can have a code block that sets the final fields. Not sure how  
> feasible that is, but IMHO, that is a short coming.
> --
> With best of best regards
> Pawel S. Veselov
>
>
>
>
> --
> With best of best regards
> Pawel S. Veselov

[Thomas Hawtin-2]

Peter Jones wrote:

> Regarding a language solution, though, I don't see how to avoid making a
> [...]

`readObject` is a `psuedo-constructor`. Ideally it would be a real
constructor (with choice of defaultReadObject/readFields similar to
super), but that would require significant changes to the JLS.

Similar problems appear for JAXB, ORMs, etc. Dependency Inject also has
problems with construction. Construction often involves way too much
boilerplate. There are lots of issues here, most I have little
experience with.

> So I think that the best hope is an improved reflective library
> solution.  Thinking out loud: say, a factory method that looks up a
> [...]

I guess something like

class MyClass implements Serializable {
     private final transient Thing thing;
     private static final FieldRead<Thing> THING_READER =
         FieldRead.newInstance("thing");
     ...
     private void readObject(...) ... {
         in.defaultReadObject();
         Thing thing = (Thing)in.readObject();
         THING_READER.set(thing);
     }
}

Gratuitous use of immediate caller. I just know people will attempt to
take client supplied arguments. Best we can do is put a few static
checks in javac and static analysis tools, and runtime checks into the
serialisation mechanism.

(Notes:
    FieldRead.newInstance
      - must be called by static initialiser
          (dynamic and perhaps static check).
      - static analysis (inc. javac) can check field name, type and
assignment).
      - errors by unchecked exception
    FieldRead.read
      - Dynamic check that called from readObject on this instance (can
cheat).
      - Dynamic check for no reassignment.
      - Dynamic check after readObject call for definite assignment.
      - Could also add static checks as well.
)

Tom Hawtin

[David M. Lloyd-3]

This is exactly the solution I present to users of JBoss Marshalling.  The
access check performed verifies that the field being updated is a
non-static instance field (final or otherwise, any access level) of the
caller's class; IllegalAccessException is then not thrown at "runtime" when
the field is changed.  Setting the field can be done via the same
assortment of set methods found on Field (set, setByte, setBoolean, ...).

- DML

On 12/03/2009 04:57 AM, Tom Hawtin wrote:

> Peter Jones wrote:
>
>> Regarding a language solution, though, I don't see how to avoid making
>> a [...]
>
> `readObject` is a `psuedo-constructor`. Ideally it would be a real
> constructor (with choice of defaultReadObject/readFields similar to
> super), but that would require significant changes to the JLS.
>
> Similar problems appear for JAXB, ORMs, etc. Dependency Inject also has
> problems with construction. Construction often involves way too much
> boilerplate. There are lots of issues here, most I have little
> experience with.
>
>> So I think that the best hope is an improved reflective library
>> solution. Thinking out loud: say, a factory method that looks up a [...]
>
> I guess something like
>
> class MyClass implements Serializable {
> private final transient Thing thing;
> private static final FieldRead<Thing> THING_READER =
> FieldRead.newInstance("thing");
> ...
> private void readObject(...) ... {
> in.defaultReadObject();
> Thing thing = (Thing)in.readObject();
> THING_READER.set(thing);
> }
> }
>
> Gratuitous use of immediate caller. I just know people will attempt to
> take client supplied arguments. Best we can do is put a few static
> checks in javac and static analysis tools, and runtime checks into the
> serialisation mechanism.
>
> (Notes:
> FieldRead.newInstance
> - must be called by static initialiser
> (dynamic and perhaps static check).
> - static analysis (inc. javac) can check field name, type and assignment).
> - errors by unchecked exception
> FieldRead.read
> - Dynamic check that called from readObject on this instance (can cheat).
> - Dynamic check for no reassignment.
> - Dynamic check after readObject call for definite assignment.
> - Could also add static checks as well.
> )
>
> Tom Hawtin