[PATCH] Make DateTime binary compatible with .NET

[Jonathan Hseu-2]

I previously discussed this patch on the mono IRC channel.

Motivation:

1. The current DateTime implementation is not binary compatible (with BinaryFormatter) with .NET's DateTime.  There are many bugs on the tracker related to this issue:

https://bugzilla.novell.com/show_bug.cgi?id=321869

https://bugzilla.novell.com/show_bug.cgi?id=325067

https://bugzilla.novell.com/show_bug.cgi?id=360429

My patch fixes this issue.  With BinaryFormatter, I serialized several DateTimes (with different DateTimeKinds) from Visual Studio and was able to successfully load them after the patch and tested that they were equivalent.

2. The current DateTime takes up 16 bytes.  The .NET one takes up 8 bytes.  This is the primary reason for the above incompatibility.  Mono's extra field was a "DateTimeKind kind".  I simply removed that and used flag bits on the ticks field to indicate the kind so as to be compatible with .NET.

The size difference is an issue for me, because I currently load 50 million DateTime records into memory, and 50 million * 8 bytes extra == 400 megabytes extra RAM used.

Caveats:

This new DateTime is binary incompatible (again with BinaryFormatter) with older versions of Mono.

Please look over the contents of the patch as I am very new to C#.  I welcome any comments.

Thanks,

Jonathan Hseu

_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[Robert Jordan]

Jonathan Hseu wrote:
> Caveats:
>
> This new DateTime is binary incompatible (again with BinaryFormatter) with
> older versions of Mono.

That's the reason why the fixes attached to

https://bugzilla.novell.com/show_bug.cgi?id=325067

were not accepted.

Robert

_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[Jonathan Hseu-2]

On Fri, Nov 6, 2009 at 9:18 AM, Robert Jordan <robertj@...> wrote:

Jonathan Hseu wrote:
> Caveats:
>
> This new DateTime is binary incompatible (again with BinaryFormatter) with
> older versions of Mono.

That's the reason why the fixes attached to

https://bugzilla.novell.com/show_bug.cgi?id=325067

were not accepted.

Robert

Going through the bug logs, it seems like the patch was first applied but then later removed because it was still incompatible with .NET's version, not because it was incompatible with older versions of mono.

My patch is completely different than the one in the bug report.  Mine _only_ affects the DateTime.cs file whereas the one in the bug report changed the serialization code.  My patch basically introduces the same field names and types that are also in .NET so that it is guaranteed to be compatible.

Also, given that mono's DateTime currently has an extra 8 byte overhead over .NET, shouldn't this issue be looked at again?

Jonathan Hseu

_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[Leszek Ciesielski]

On Fri, Nov 6, 2009 at 4:32 PM, Jonathan Hseu <vomjom@...> wrote:

> On Fri, Nov 6, 2009 at 9:18 AM, Robert Jordan <robertj@...> wrote:
>>
>> Jonathan Hseu wrote:
>> > Caveats:
>> >
>> > This new DateTime is binary incompatible (again with BinaryFormatter)
>> > with
>> > older versions of Mono.
>>
>> That's the reason why the fixes attached to
>>
>> https://bugzilla.novell.com/show_bug.cgi?id=325067
>>
>> were not accepted.
>>
>> Robert
>
> Going through the bug logs, it seems like the patch was first applied but
> then later removed because it was still incompatible with .NET's version,
> not because it was incompatible with older versions of mono.
> My patch is completely different than the one in the bug report.  Mine
> _only_ affects the DateTime.cs file whereas the one in the bug report
> changed the serialization code.  My patch basically introduces the same
> field names and types that are also in .NET so that it is guaranteed to be
> compatible.
> Also, given that mono's DateTime currently has an extra 8 byte overhead over
> .NET, shouldn't this issue be looked at again?
>

Would the patch get accepted if it still was able to deserialize old
(i.e. currently used) Mono DateTime format, while only outputting the
.Net compatible one? This is an incompatibility that has to get fixed
eventually.
_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[gertdriesen]

Robert,

We could only enable this only if the MONO_STRICT_MS_COMPLIANT is set to
"yes".
We already use this env var in System.Xml to enable strict MS compatibility.

Gert

-----Original Message-----
From: mono-devel-list-bounces@...
[mailto:mono-devel-list-bounces@...] On Behalf Of Robert Jordan
Sent: vrijdag 6 november 2009 16:19
To: mono-devel-list@...
Subject: Re: [Mono-dev] [PATCH] Make DateTime binary compatible with .NET

Jonathan Hseu wrote:
> Caveats:
>
> This new DateTime is binary incompatible (again with BinaryFormatter) with
> older versions of Mono.

That's the reason why the fixes attached to

https://bugzilla.novell.com/show_bug.cgi?id=325067

were not accepted.

Robert

_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list

_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[Robert Jordan]

Jonathan Hseu wrote:

> On Fri, Nov 6, 2009 at 9:18 AM, Robert Jordan <robertj@...> wrote:
>
>> Jonathan Hseu wrote:
>>> Caveats:
>>>
>>> This new DateTime is binary incompatible (again with BinaryFormatter)
>> with
>>> older versions of Mono.
>> That's the reason why the fixes attached to
>>
>> https://bugzilla.novell.com/show_bug.cgi?id=325067
>>
>> were not accepted.
>>
>> Robert
>>
>
> Going through the bug logs, it seems like the patch was first applied but
> then later removed because it was still incompatible with .NET's version,
> not because it was incompatible with older versions of mono.
>
> My patch is completely different than the one in the bug report.  Mine
> _only_ affects the DateTime.cs file whereas the one in the bug report
> changed the serialization code.  My patch basically introduces the same
> field names and types that are also in .NET so that it is guaranteed to be
> compatible.

Please read the bug carefully.

Here is the DateTime.cs patch:

https://bugzillafiles.novell.org/attachment.cgi?id=172471

Then search for "DateTimeISerializable.diff" at
https://bugzilla.novell.com/show_bug.cgi?id=325067
and read the comments.

Robert

_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[Jonathan Hseu-2]

On Fri, Nov 6, 2009 at 9:56 AM, Robert Jordan <robertj@...> wrote:

Jonathan Hseu wrote:
> My patch is completely different than the one in the bug report.  Mine
> _only_ affects the DateTime.cs file whereas the one in the bug report
> changed the serialization code.  My patch basically introduces the same
> field names and types that are also in .NET so that it is guaranteed to be
> compatible.

Please read the bug carefully.

Here is the DateTime.cs patch:

https://bugzillafiles.novell.org/attachment.cgi?id=172471

Then search for "DateTimeISerializable.diff" at

https://bugzilla.novell.com/show_bug.cgi?id=325067

and read the comments.

Well, my patch acts a little differently (it, by the way, passes all the regression tests for both 2.0 and 1.1).

I just tested it and it also makes 1.1 and 2.0 binary compatible with each other and .NET, assuming you are on a new version of mono for both profiles.

So, the only issue is that it'll stop being compatible with older versions of mono, but it's possible to make a workaround so that the new mono will be able to Deserialize older mono binary DateTimes.

And, in the long run, people will stop using older versions of Mono.  But people will continue to use .NET, and this issue will come up over and over again.

Jonathan Hseu

_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[Jonathan Pobst]

Jonathan Hseu wrote:
> And, in the long run, people will stop using older versions of Mono.
>  But people will continue to use .NET, and this issue will come up over
> and over again.

The long term fix would probably be add the code that allows the
deserializer to *read* either method now to the current branches (2.4
LTS, 2.6, and trunk).

At some point in the future (2.8? 3.0?), we could then change the writer
to *write* the .Net way, and it would be backwards compatible with
2.4.3+ as well as compatible with itself and .Net.

This way you would have to be using very different versions of Mono to
hit the incompatibility (2.2 <-> 3.0).

In the mean time, you can write the .Net way using a environment
variable like Gert suggested.  (MONO_STRICT_MS_COMPLIANT)

Jonathan
_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[knocte]

This case is a bit different than the Xml one, right? So why not using
the strict mode as default, and use MONO_BACKWARDS_COMPATIBLE for the
old behaviour.

Just my 2 cents.

        Andres


Gert Driesen escribió:

> Robert,
>
> We could only enable this only if the MONO_STRICT_MS_COMPLIANT is set to
> "yes".
> We already use this env var in System.Xml to enable strict MS compatibility.
>
> Gert
>
> -----Original Message-----
> From: mono-devel-list-bounces@...
> [mailto:mono-devel-list-bounces@...] On Behalf Of Robert Jordan
> Sent: vrijdag 6 november 2009 16:19
> To: mono-devel-list@...
> Subject: Re: [Mono-dev] [PATCH] Make DateTime binary compatible with .NET
>
> Jonathan Hseu wrote:
>> Caveats:
>>
>> This new DateTime is binary incompatible (again with BinaryFormatter) with
>> older versions of Mono.
>
> That's the reason why the fixes attached to
>
> https://bugzilla.novell.com/show_bug.cgi?id=325067
>
> were not accepted.
>
> Robert
>
> _______________________________________________
> Mono-devel-list mailing list
> Mono-devel-list@...
> http://lists.ximian.com/mailman/listinfo/mono-devel-list

_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[Ernesto-18]

Jonathan Pobst wrote:

> Jonathan Hseu wrote:
>  
>> And, in the long run, people will stop using older versions of Mono.
>>  But people will continue to use .NET, and this issue will come up over
>> and over again.
>>    
>
> The long term fix would probably be add the code that allows the
> deserializer to *read* either method now to the current branches (2.4
> LTS, 2.6, and trunk).
>
> At some point in the future (2.8? 3.0?), we could then change the writer
> to *write* the .Net way, and it would be backwards compatible with
> 2.4.3+ as well as compatible with itself and .Net.
>
> This way you would have to be using very different versions of Mono to
> hit the incompatibility (2.2 <-> 3.0).
>
> In the mean time, you can write the .Net way using a environment
> variable like Gert suggested.  (MONO_STRICT_MS_COMPLIANT)
>  

That sound like the best way to get a smooth transition. As Leszek said,
it has to be fixed eventually. So it's better to do it sooner than
later, and it's better to do it transparent than breaking something.

To add, a few bytes from http://www.mono-project.com/FAQ:_Technical

<quote>
Our policy is to do our best to make the framework classes compatible
between Mono and MS.NET, however sometimes this is not possible because
the internal implementation is too different. Notice also that when we
change a class to make it compatible with MS.NET, we lose compatibility
with older versions of Mono.
</quote>

That very same document mentions that serialization "compatibility is
not even guaranteed between different MS.NET versions or Mono versions".

My 2 cents.

Regards,
Ernesto

_______________________________________________
Mono-devel-list mailing list
Mono-devel-list@...
http://lists.ximian.com/mailman/listinfo/mono-devel-list