Re: <i18n dev> Using system tzdata

[masayoshi.okutsu]

I apologize for the delay in responding. There have been some
discussions in Sun about long term directions on tzdata maintenance. We
might need to migrate to another mechanisms to update tzdata from the
current tzupdater tool due to other platform dependent issues. But we
haven't made a decision yet, and the discussion involves Sun's
sustaining team, which is taking time.

There are some further comments on the proposed changes, including ones
from Mark Reinhold.

    * If a user used Sun's tzupdater tool to update tzdata in JDK/JRE,
      the runtime would still pick up the older tzdata in
      /usr/share/javazi, which would cause maintenance problems
      (confusions). Do you plan to tell users that the tzupdater tool
      should never be used to update tzdata?
    * "user.zoneinfo.dir" is evaluated every time readZoneInfoFile()
      gets called. This means that the runtime may pick up tzdata from
      different locations if an application changed it during its
      execution. Should the tzdata directory be determined only once
      before reading tzdata first time?
    * The property name should be named as sun.* so that it's clearly
      understood as internal.
    * The default value should not be set in HotSpot's os_linux.cpp
      file, but rather in java_props_md.c.
    * The proposal should include the default location for other platforms.

Thanks,
Masayoshi

On 9/20/2007 3:28 AM, Keith Seitz wrote:

> Ping?
> [I have not yet updated the RFE. I'm going to await further feedback.]
>
> Keith Seitz wrote:
>> Masayoshi Okutsu wrote:
>>> Sorry for taking time to respond.
>>
>> Likewise!
>>
>>> I've filed an RFE for this request to make sure Sun's JRE supports
>>> the same mechanism. Sun's Change Request ID (aka bug ID) is 6593486.
>>
>> I'll add my comments/patch there, too, but I wanted to briefly let
>> you know what I've been up to with all of this.
>>
>>>     * I think the "java." name space should be avoided for this
>>> purpose.
>>>       "user.timezone.dir" should be OK.
>>
>> Done.
>>
>>>     * I'd also suggest that "/usr/share/zoneinfo/java" be changed to
>>>       something like "/usr/share/javazi" because Java runtime may
>>> search
>>>       files under /usr/share/zoneinfo to detect the platform time zone
>>>       setting. It costs extra to skip all the files under
>>>       /usr/share/zoneinfo/java if the directory comes before other
>>>       directories in /usr/share/zoneinfo. Also I think
>>>       /usr/share/zoneinfo should contain the native Olson time zone
>>> data
>>>       files only.
>>
>> Done.
>>
>>>     * I'm not sure if the current implementation is robust enough to be
>>>       able to reject all broken data files. It does check magic
>>> numbers,
>>>       though. You may need extra tests with random directory paths. Or
>>>       you could add more checking to see if the directory is right,
>>> such
>>>       as existence of the ZoneInfoMappings file.
>>
>> I've added a little something so that the code will check for a
>> ZoneInfoMappings file before using the user.timezone.dir property.
>> Beyond this, there seems little reason to do any more extreme error
>> checking. The default is still to use the supplied, built-in tzdata.
>> If the user sets user.timezone.dir to something else, he should be
>> aware of what he is doing!
>>
>> Thank you for your feedback,
>> Keith
>>
>

[masayoshi.okutsu]

Sorry again for taking time. Finally, I talked to some sustaining team
members in Ireland to make a decision on the long-term tzdata
maintenance issue. We've determined that Sun would need to continue to
use the tzupdater tool as the main vehicle to deliver new tzdata files
for Java runtime. Therefore, the proposed property needs to be supported
for those who want to use another mechanism to update tzdata files.

I need to get approval on the proposed property support because it
exposes an interface (even it's internal). To do that, I need to make a
decision on the default value. I've been having some difficulty to think
of default locations for other platforms, especially Windows. So I
wondered if it's OK to make the property value undefined by default.
Would that be acceptable for you?

Thanks,
Masayoshi

On 12/21/2007 11:59 AM, Masayoshi Okutsu wrote:

> I apologize for the delay in responding. There have been some
> discussions in Sun about long term directions on tzdata maintenance.
> We might need to migrate to another mechanisms to update tzdata from
> the current tzupdater tool due to other platform dependent issues. But
> we haven't made a decision yet, and the discussion involves Sun's
> sustaining team, which is taking time.
>
> There are some further comments on the proposed changes, including
> ones from Mark Reinhold.
>
>    * If a user used Sun's tzupdater tool to update tzdata in JDK/JRE,
>      the runtime would still pick up the older tzdata in
>      /usr/share/javazi, which would cause maintenance problems
>      (confusions). Do you plan to tell users that the tzupdater tool
>      should never be used to update tzdata?
>    * "user.zoneinfo.dir" is evaluated every time readZoneInfoFile()
>      gets called. This means that the runtime may pick up tzdata from
>      different locations if an application changed it during its
>      execution. Should the tzdata directory be determined only once
>      before reading tzdata first time?
>    * The property name should be named as sun.* so that it's clearly
>      understood as internal.
>    * The default value should not be set in HotSpot's os_linux.cpp
>      file, but rather in java_props_md.c.
>    * The proposal should include the default location for other
> platforms.
>
> Thanks,
> Masayoshi
>
> On 9/20/2007 3:28 AM, Keith Seitz wrote:
>> Ping?
>> [I have not yet updated the RFE. I'm going to await further feedback.]
>>
>> Keith Seitz wrote:
>>> Masayoshi Okutsu wrote:
>>>> Sorry for taking time to respond.
>>>
>>> Likewise!
>>>
>>>> I've filed an RFE for this request to make sure Sun's JRE supports
>>>> the same mechanism. Sun's Change Request ID (aka bug ID) is 6593486.
>>>
>>> I'll add my comments/patch there, too, but I wanted to briefly let
>>> you know what I've been up to with all of this.
>>>
>>>>     * I think the "java." name space should be avoided for this
>>>> purpose.
>>>>       "user.timezone.dir" should be OK.
>>>
>>> Done.
>>>
>>>>     * I'd also suggest that "/usr/share/zoneinfo/java" be changed to
>>>>       something like "/usr/share/javazi" because Java runtime may
>>>> search
>>>>       files under /usr/share/zoneinfo to detect the platform time zone
>>>>       setting. It costs extra to skip all the files under
>>>>       /usr/share/zoneinfo/java if the directory comes before other
>>>>       directories in /usr/share/zoneinfo. Also I think
>>>>       /usr/share/zoneinfo should contain the native Olson time zone
>>>> data
>>>>       files only.
>>>
>>> Done.
>>>
>>>>     * I'm not sure if the current implementation is robust enough
>>>> to be
>>>>       able to reject all broken data files. It does check magic
>>>> numbers,
>>>>       though. You may need extra tests with random directory paths. Or
>>>>       you could add more checking to see if the directory is right,
>>>> such
>>>>       as existence of the ZoneInfoMappings file.
>>>
>>> I've added a little something so that the code will check for a
>>> ZoneInfoMappings file before using the user.timezone.dir property.
>>> Beyond this, there seems little reason to do any more extreme error
>>> checking. The default is still to use the supplied, built-in tzdata.
>>> If the user sets user.timezone.dir to something else, he should be
>>> aware of what he is doing!
>>>
>>> Thank you for your feedback,
>>> Keith
>>>
>>

[Keith Seitz]

Masayoshi Okutsu wrote:
> Therefore, the proposed property needs to be supported
> for those who want to use another mechanism to update tzdata files.

I don't believe that my proposed change in any way conflicts with the
status quo. If the property is defined, tzdata is read from the value of
the property. If the directory is not valid (or ZoneInfoMapping doesn't
exist) or if the property is otherwise undefined or invalid, it is
simply ignored, and the JDK-supplied tzdata is used instead. [Although I
admit, allowing this property to change makes me nervous. I would rather
it ALWAYS use the system data or NEVER use it -- rather like how OpenJDK
behaves today.]

> I need to get approval on the proposed property support because it
> exposes an interface (even it's internal). To do that, I need to make a
> decision on the default value.

Default, platform-specific values, no?

> I've been having some difficulty to think
> of default locations for other platforms, especially Windows.

I don't think Microsoft would ever release tzdata for use by Sun's JRE,
but I could just be pessimistic. O:-) The only hosts that this really
affects are Solaris and Linux.

> So I  wondered if it's OK to make the property value undefined by
> default. Would that be acceptable for you?

While our (Red Hat) hands are tied, of course, I would not agree with
the decision to turn this off by default, effectively ignoring the
problem that the patch is attempting to address.

Is there really no solution to this problem?

Keith

[Dalibor Topic]

Masayoshi Okutsu <Masayoshi.Okutsu@...> writes:

>
> Sorry again for taking time. Finally, I talked to some sustaining team
> members in Ireland to make a decision on the long-term tzdata
> maintenance issue. We've determined [...]

Hi,

I've tried to find that discussion in the mailing list archives, but I couldn't
find it. Could you point me to it, or, in case it was not on this list, post it
for your fellow OpenJDK developers to read?

thank you in advance,
dalibor topic

[Brian Beck-8]

Point taken.  Since the discussion was about maintenance of all the
existing (non-open source at least for the moment) JDKs, we did not have
it on this alias.  But there's nothing secret about the topic and it
does impact the openjdk.  We'll summarize for the gang here.

Brian.


Dalibor Topic wrote:

> Masayoshi Okutsu <Masayoshi.Okutsu@...> writes:
>
>  
>> Sorry again for taking time. Finally, I talked to some sustaining team
>> members in Ireland to make a decision on the long-term tzdata
>> maintenance issue. We've determined [...]
>>    
>
> Hi,
>
> I've tried to find that discussion in the mailing list archives, but I couldn't
> find it. Could you point me to it, or, in case it was not on this list, post it
> for your fellow OpenJDK developers to read?
>
> thank you in advance,
> dalibor topic
>  

[Dalibor Topic]

Brian Beck <brian.beck@...> writes:

>
> Point taken.  Since the discussion was about maintenance of all the
> existing (non-open source at least for the moment) JDKs, we did not have
> it on this alias.  But there's nothing secret about the topic and it
> does impact the openjdk.  We'll summarize for the gang here.
>

Thank you very much, Brian, I'm looking forward to it.

cheers,
dalibor topic

[masayoshi.okutsu]

Thank you for asking! :-) Here's the whole history on the time zone data
maintenance.

 From 1.1 to 1.3.1, time zone data was hard-coded in
src/share/classes/java/util/TimeZone.java which was compiled into
rt.jar. Also it was java.util.SimpleTimeZone-based, which made it
difficult to deal with real world time zone changes, such as GMT offset
changes. The date-time API came from Taligent OS, except for
java.util.Date, along with many other i18n classes (e.g., java.text). We
(java i18n team) took over the date-time API responsibility as of 1.4.0
development from Taligent, which later became IBM.

In 1.4.0, the whole time zone support was reimplemented so that real
world time zone changes, including past and future changes, could be
supported. We used the public zoneinfo data (a.k.a. Olson tzdata). One
of the reasons of the reimplementation was to make it easier to maintain
time zone data. Time zone data was no longer hard-coded, of course. A
Java version of zic (zoneinfo compiler), called 'javazic' internally,
was written to generate necessary information to support the
java.util.Calendar and java.util.TimeZone classes. (At that time we
decided to minimize date-time API changes because we recognized the API
had many problems and that a new API would be required to fix those
problems. Unfortunately, a new date-time API project proposal for 5.0
development wasn't approved. So we had to change our plans deal with the
date-time API problems.)

Time zone support was much improved in 1.4.0. However, the time zone
data maintenance was still an issue. We (java i18n) proposed a couple of
times to open up javazic so that users could update time zone data files
with a new Olson tzdata release. However, at that point, time zone data
maintenance wasn't considered a big problem, and our proposal wasn't
approved. Actually nobody else paid much attention to the issue.

Before long the world was changing. Many countries were changing their
daylight saving time rules and GMT offsets to deal with the energy
crisis. Olson tzdata was frequently updated. It wasn't quick enough to
just add the latest Olson tzdata to JRE/JDK update releases. Then, the
United States of America announced the DST rule change from 2007, which
completely changed the situation. We had to develop a way to update
tzdata for all existing JRE/JDK installations, including 1.3.1_xx. The
tzupdater tool (http://java.sun.com/javase/tzupdater_README.html) was
developed late 2006. (Actually there are two versions; one for 1.4+
which updates tzdata files in <JRE>/lib/zi, and the other for 1.3.1_xx
which updates rt.jar.)

Tzupdate (for 1.4+) worked. But we encountered a platform specific
problem with Solaris Zones. A workaround was added to the tool. But I
considered it as a temporary solution and proposed migration to platform
specific patch mechanisms to avoid software package maintenance
problems. We may encounter similar problems on other platforms. The
change proposal from Red Hat is related to this issue. That is, the
up2date utility on Red Hat systems should be able to be used to update
the tzdata files for the Java runtime. The meeting I referenced in my
previous message was to discuss my proposal for the migration.

Thank you for listening! So... what should be the best way to maintain
tzdata for Java SE installations?

Thanks,
Masayoshi


On 1/26/2008 12:46 AM, Dalibor Topic wrote:

> Masayoshi Okutsu <Masayoshi.Okutsu@...> writes:
>
>  
>> Sorry again for taking time. Finally, I talked to some sustaining team
>> members in Ireland to make a decision on the long-term tzdata
>> maintenance issue. We've determined [...]
>>    
>
> Hi,
>
> I've tried to find that discussion in the mailing list archives, but I couldn't
> find it. Could you point me to it, or, in case it was not on this list, post it
> for your fellow OpenJDK developers to read?
>
> thank you in advance,
> dalibor topic
>
>  

[Dalibor Topic]

Masayoshi Okutsu wrote:
> Thank you for asking! :-) Here's the whole history on the time zone data
> maintenance.

Thank you very much!

> Tzupdate (for 1.4+) worked. But we encountered a platform specific
> problem with Solaris Zones. A workaround was added to the tool. But I
> considered it as a temporary solution and proposed migration to platform
> specific patch mechanisms to avoid software package maintenance
> problems. We may encounter similar problems on other platforms. The
> change proposal from Red Hat is related to this issue. That is, the
> up2date utility on Red Hat systems should be able to be used to update
> the tzdata files for the Java runtime. The meeting I referenced in my
> previous message was to discuss my proposal for the migration.
>
> Thank you for listening! So... what should be the best way to maintain
> tzdata for Java SE installations?

 From the POV of operating system distributors that already need to
support tzdata for other applications, and therefore need to keep it
current anyway for their customers, I think it's preferable to rely on
system facilities where such facilities exist, than to have to push a
separate patch for the Java runtime.

cheers,
dalibor topic

[Martin Buchholz]

Dalibor Topic wrote:
> Masayoshi Okutsu wrote:
> From the POV of operating system distributors that already need to
> support tzdata for other applications, and therefore need to keep it
> current anyway for their customers, I think it's preferable to rely on
> system facilities where such facilities exist, than to have to push a
> separate patch for the Java runtime.

Perhaps the best behavior is to check whether the system database is
at least as up to date as the Java database.  Could Java know
about the "last" change to its own database, query the OS as to whether
it "knows about" this specific change, and if so, defer to it?

Martin

>
> cheers,
> dalibor topic

[masayoshi.okutsu]

Tzdata for Java does have the Olson version information (eg,
tzdata2007k), but I don't think the OS tzdata has it. But I don't know
what Java should do if it was possible to check OS tzdata version. Can
you elaborate your suggestion a bit more?

Thanks,
Masayoshi

On 1/30/2008 8:51 AM, Martin Buchholz wrote:

> Dalibor Topic wrote:
>  
>> Masayoshi Okutsu wrote:
>> From the POV of operating system distributors that already need to
>> support tzdata for other applications, and therefore need to keep it
>> current anyway for their customers, I think it's preferable to rely on
>> system facilities where such facilities exist, than to have to push a
>> separate patch for the Java runtime.
>>    
>
> Perhaps the best behavior is to check whether the system database is
> at least as up to date as the Java database.  Could Java know
> about the "last" change to its own database, query the OS as to whether
> it "knows about" this specific change, and if so, defer to it?
>
> Martin
>
>  
>> cheers,
>> dalibor topic
>>    

[Martin Buchholz]

I don't know anything about how the Olson database is implemented
or accessed on any particular OS..... I'm just brainstorming here.

Say someone creates their own timezone definition MYTZ and adds it
to the OS database.  If Java fails to lookup any such timezone,
it seems reasonable to consult the OS.
Similarly if the OS has a more up-to-date database with a newly
defined timezone.  (But that's rare)

If the JDK and the OS agree on the current timezone definition,
all is wonderful.  If they differ, perhaps it is worth figuring
out which is newer, even if that is difficult.  Perhaps it's worth
snooping around in /usr/share/lib/zoneinfo.

Martin

Masayoshi Okutsu wrote:

> Tzdata for Java does have the Olson version information (eg,
> tzdata2007k), but I don't think the OS tzdata has it. But I don't know
> what Java should do if it was possible to check OS tzdata version. Can
> you elaborate your suggestion a bit more?
>
> Thanks,
> Masayoshi
>
> On 1/30/2008 8:51 AM, Martin Buchholz wrote:
>> Dalibor Topic wrote:
>>  
>>> Masayoshi Okutsu wrote:
>>> From the POV of operating system distributors that already need to
>>> support tzdata for other applications, and therefore need to keep it
>>> current anyway for their customers, I think it's preferable to rely on
>>> system facilities where such facilities exist, than to have to push a
>>> separate patch for the Java runtime.
>>>    
>>
>> Perhaps the best behavior is to check whether the system database is
>> at least as up to date as the Java database.  Could Java know
>> about the "last" change to its own database, query the OS as to whether
>> it "knows about" this specific change, and if so, defer to it?
>>
>> Martin
>>
>>  
>>> cheers,
>>> dalibor topic
>>>    

[masayoshi.okutsu]

A problem is that the OS tzdata doesn't contain enough information for
supporting the java.util.Calendar API. Otherwise, it would be
unnecessary to provide JDK's own tzdata files.

Masayoshi

Martin Buchholz wrote:

> I don't know anything about how the Olson database is implemented
> or accessed on any particular OS..... I'm just brainstorming here.
>
> Say someone creates their own timezone definition MYTZ and adds it
> to the OS database.  If Java fails to lookup any such timezone,
> it seems reasonable to consult the OS.
> Similarly if the OS has a more up-to-date database with a newly
> defined timezone.  (But that's rare)
>
> If the JDK and the OS agree on the current timezone definition,
> all is wonderful.  If they differ, perhaps it is worth figuring
> out which is newer, even if that is difficult.  Perhaps it's worth
> snooping around in /usr/share/lib/zoneinfo.
>
> Martin
>
> Masayoshi Okutsu wrote:
>  
>> Tzdata for Java does have the Olson version information (eg,
>> tzdata2007k), but I don't think the OS tzdata has it. But I don't know
>> what Java should do if it was possible to check OS tzdata version. Can
>> you elaborate your suggestion a bit more?
>>
>> Thanks,
>> Masayoshi
>>
>> On 1/30/2008 8:51 AM, Martin Buchholz wrote:
>>    
>>> Dalibor Topic wrote:
>>>  
>>>      
>>>> Masayoshi Okutsu wrote:
>>>> From the POV of operating system distributors that already need to
>>>> support tzdata for other applications, and therefore need to keep it
>>>> current anyway for their customers, I think it's preferable to rely on
>>>> system facilities where such facilities exist, than to have to push a
>>>> separate patch for the Java runtime.
>>>>    
>>>>        
>>> Perhaps the best behavior is to check whether the system database is
>>> at least as up to date as the Java database.  Could Java know
>>> about the "last" change to its own database, query the OS as to whether
>>> it "knows about" this specific change, and if so, defer to it?
>>>
>>> Martin
>>>
>>>  
>>>      
>>>> cheers,
>>>> dalibor topic
>>>>    
>>>>        

[herceg_novi]

The
tzupdater tool (http://java.sun.com/javase/tzupdater_README.html) was
developed late 2006. (Actually there are two versions; one for 1.4+
which updates tzdata files in <JRE>/lib/zi, and the other for 1.3.1_xx
which updates rt.jar.)"

I actually need to locate the tzupdater tool for JDK 1.3.1. I called Sun and offered to pay for it, but they were unsure how to handle my request since I do not have an existing support contract with Sun. Is there an easy way of obtaining tzupdater for JDK 1.3.1?

[Sean Coffey]

AFAIK, you need a support contract of some description to obtain tzupdater131 tool.

Do you need the very latest tzdata version ? If not, you can obtain jdk1.3.1_24 from
http://java.sun.com/j2se/1.3/download.html which contains tzdata 2008h.

http://java.sun.com/javase/timezones/tzdata_versions.html

regards,
Sean.

herceg_novi wrote:

The 
  

tzupdater tool (http://java.sun.com/javase/tzupdater_README.html) was 
developed late 2006. (Actually there are two versions; one for 1.4+ 
which updates tzdata files in <JRE>/lib/zi, and the other for 1.3.1_xx 
which updates rt.jar.)"

    

I actually need to locate the tzupdater tool for JDK 1.3.1. I called Sun and
offered to pay for it, but they were unsure how to handle my request since I
do not have an existing support contract with Sun. Is there an easy way of
obtaining tzupdater for JDK 1.3.1? 

  

[herceg_novi]

Sean Coffey wrote:

Do you need the very latest tzdata version ? If not, you can obtain
jdk1.3.1_24 from
http://java.sun.com/j2se/1.3/download.html 
which contains tzdata 2008h.

Unfortunately this version is for Solaris only. The latest Windows JDK is jdk1.3.1_20, and timezone dates are incorrect in that version.

I contacted Sun to give me a quote for the contract, but it would be good to find an alternate quicker solution as well.