HPET/PIT timer accuracy

[Keir Fraser]

Does anyone on this list have an idea of expected accuracy of chipset
time sources (e.g., PIT/HPET)? I've been looking at relative frequency
stability of the CPU-local TSC and platform HPET on a few different
systems, and see weird instabilities that I cannot explain.

On a couple of systems based on Intel E7520 (Lindenhurst) chipset I see
the relative frequency is stable to within a couple of parts per
million (which is within measurement error). Either the two are clocked
from the same crystal, or two crystals with low frequency drift/wander
(but not unexpectedly low -- I don't think crystals should drift over
short time periods).

In contrast, two different systems based on AMD 8111 chipset perform
very badly. The relative frequency of TSC vs HPET is stable only to
within +/- 30ppm! This means that, during one second, timers based on
these two sources can diverge by around 30us, which is a *lot*. Clearly
the TSC and HPET are driven off different time sources, but I almost
cannot believe that both are driven off crystals -- one of the two must
have a 1Hz frequency drift/wander of 30ppm which is totally outside the
specs I'd expect of a crystal source.

Any ideas what could be going on?! This is something of a pain for the
new time code, which syncs the local TSCs to the chipset timer....

  -- Keir

(I am talking frequency stability here, by the way, not frequency
tolerance. I know a crystal can be 100s of ppm out of whack from what
is stamped on the can, but I think over short time periods it should
not drift noticeably from its natural frequency).


_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Ronald G Minnich]

I can only tell you we've seen weird numbers from the opteron tsc, to the
point that we don't entirely trust it. Not sure why.

ron


_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Ian Pratt]

> In contrast, two different systems based on AMD 8111 chipset
> perform very badly. The relative frequency of TSC vs HPET is
> stable only to within +/- 30ppm! This means that, during one
> second, timers based on these two sources can diverge by
> around 30us, which is a *lot*. Clearly the TSC and HPET are
> driven off different time sources, but I almost cannot
> believe that both are driven off crystals -- one of the two
> must have a 1Hz frequency drift/wander of 30ppm which is
> totally outside the specs I'd expect of a crystal source.
>
> Any ideas what could be going on?! This is something of a
> pain for the new time code, which syncs the local TSCs to the
> chipset timer....

These systems haven't got a 'spread spectrum'[*] PCI bus clock per
chance? It might be worth seeing if there's a BIOS option for it.

Ian

[*] This is a horrible hack used by some system vendors to cheat EM
emmisions regulations.

_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Michael Hohnbaum]

On Fri, 2005-07-29 at 18:26 +0100, Keir Fraser wrote:
> Does anyone on this list have an idea of expected accuracy of chipset
> time sources (e.g., PIT/HPET)? I've been looking at relative frequency
> stability of the CPU-local TSC and platform HPET on a few different
> systems, and see weird instabilities that I cannot explain.

The TSC is not a reliable time source.  There is no guarantee of
synchronization between multiple CPUs.  Also note that the TSC will
stop in ACPI C3 mode.  Probably not an issue for Xen today, but could
be in the future, especially as servers start doing more power
management.  The HPET is preferable for a time source on systems
that this is available on.

>
> On a couple of systems based on Intel E7520 (Lindenhurst) chipset I see
> the relative frequency is stable to within a couple of parts per
> million (which is within measurement error). Either the two are clocked
> from the same crystal, or two crystals with low frequency drift/wander
> (but not unexpectedly low -- I don't think crystals should drift over
> short time periods).
>
> In contrast, two different systems based on AMD 8111 chipset perform
> very badly. The relative frequency of TSC vs HPET is stable only to
> within +/- 30ppm! This means that, during one second, timers based on
> these two sources can diverge by around 30us, which is a *lot*. Clearly
> the TSC and HPET are driven off different time sources, but I almost
> cannot believe that both are driven off crystals -- one of the two must
> have a 1Hz frequency drift/wander of 30ppm which is totally outside the
> specs I'd expect of a crystal source.

These types of observations are not inconsistent with what others have
seen.  Very platform dependent.
>
> Any ideas what could be going on?! This is something of a pain for the
> new time code, which syncs the local TSCs to the chipset timer....

This is a reality of the various hardware time sources.  There are many
problems associated with this.  You might look at the OLS paper on
a proposed rewrite for the Linux time subsystem to get a feel for how
the kernel people are trying to deal with some of these issues.

                Michael
>
>   -- Keir
>
> (I am talking frequency stability here, by the way, not frequency
> tolerance. I know a crystal can be 100s of ppm out of whack from what
> is stamped on the can, but I think over short time periods it should
> not drift noticeably from its natural frequency).
>

--
Michael Hohnbaum                             503 578 5486
hohnbaum@...                          t/l 775 5486


_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Keir Fraser]

On 29 Jul 2005, at 19:17, Michael Hohnbaum wrote:

> The TSC is not a reliable time source.  There is no guarantee of
> synchronization between multiple CPUs.  Also note that the TSC will
> stop in ACPI C3 mode.  Probably not an issue for Xen today, but could
> be in the future, especially as servers start doing more power
> management.  The HPET is preferable for a time source on systems
> that this is available on.

We're not relying on TSC synchronisation between CPUs. They can run at
different frequencies, stop in deep sleep, and so on. But we *do* want
the frequency of each one to be as stable as possible. I'd expect to
get <1ppm stability from a crystal source at constant temperature, no
problem.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Keir Fraser]

On 29 Jul 2005, at 18:53, Ian Pratt wrote:

>> Any ideas what could be going on?! This is something of a
>> pain for the new time code, which syncs the local TSCs to the
>> chipset timer....
>
> These systems haven't got a 'spread spectrum'[*] PCI bus clock per
> chance? It might be worth seeing if there's a BIOS option for it.

I did check one of our machines, and spread spectrum was definitely
disabled. In any case, I'd expect that to jitter the clock at some high
frequency, but not add/remove ticks in a way that would be discernable
at 1Hz.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Ian Pratt]

 
> We're not relying on TSC synchronisation between CPUs. They
> can run at different frequencies, stop in deep sleep, and so
> on. But we *do* want the frequency of each one to be as
> stable as possible. I'd expect to get <1ppm stability from a
> crystal source at constant temperature, no problem.

I'd wager the TSC is stable and it's the HPET that's wandering...

We'll probably just have to add more damping on the re-calibration.

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Keir Fraser]

On 29 Jul 2005, at 19:29, Ian Pratt wrote:

>> We're not relying on TSC synchronisation between CPUs. They
>> can run at different frequencies, stop in deep sleep, and so
>> on. But we *do* want the frequency of each one to be as
>> stable as possible. I'd expect to get <1ppm stability from a
>> crystal source at constant temperature, no problem.
>
> I'd wager the TSC is stable and it's the HPET that's wandering...
>
> We'll probably just have to add more damping on the re-calibration.

I haven't done any further tests to see how low the frequency of
wandering goes. Perhaps we'll find the platform timer is more stable
over e.g., 60s periods.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Michael Hohnbaum]

On Fri, 2005-07-29 at 19:29 +0100, Keir Fraser wrote:

> On 29 Jul 2005, at 19:17, Michael Hohnbaum wrote:
>
> > The TSC is not a reliable time source.  There is no guarantee of
> > synchronization between multiple CPUs.  Also note that the TSC will
> > stop in ACPI C3 mode.  Probably not an issue for Xen today, but could
> > be in the future, especially as servers start doing more power
> > management.  The HPET is preferable for a time source on systems
> > that this is available on.
>
> We're not relying on TSC synchronisation between CPUs. They can run at
> different frequencies, stop in deep sleep, and so on. But we *do* want
> the frequency of each one to be as stable as possible. I'd expect to
> get <1ppm stability from a crystal source at constant temperature, no
> problem.

Not being a hardware expert, terminology and details might be a bit off,
but my understanding is that yes, this level of instability from the
hardware time source (not necessarily a crystal) is not uncommon.  HPET
time source tends to be more reliable (stable) than the TSC.  There
tend to be many factors that affect the accuracy of the TSC, most of
which I do not grok.

Much of the complexity in the Linux time subsystem is there to deal with
the inadequacy of hardware time sources.

                  Michael
>
>   -- Keir
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@...
> http://lists.xensource.com/xen-devel
>
--
Michael Hohnbaum                             503 578 5486
hohnbaum@...                          t/l 775 5486


_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Benjamin LaHaise]

On Fri, Jul 29, 2005 at 11:51:19AM -0600, Ronald G. Minnich wrote:
> I can only tell you we've seen weird numbers from the opteron tsc, to the
> point that we don't entirely trust it. Not sure why.

Opterons can execute RDTSC out of order.

                -ben
--
"Time is what keeps everything from happening all at once." -- John Wheeler

_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Petersson, Mats]

You have to consider that the HPET and TSC are both driven by a separate
PLL that is based of some sort of crystal/oscillator. These are often
not as precise as we'd want them to be. If you do a google with "crystal
oscillator ppm" (or some such), you will find that most of these are
within the 50-100 ppm ranges [these are the guaranteed values, which
probably means that the typical is around 25-50 ppm, much along the
lines of your findings]. If you want more precise crystals than that,
you'll be looking at expensive stuff, and when you want less than around
10 ppm, you're probably looking at temperature controlled ones (aka oven
oscillators). They are both expensive and bulky, and very unlikely to be
found on the motherboard. A 2 ppm, 2MHz clock generator that I know of
from a GSM base station is about 80 x 50 x 50 mm. Not sure how much it
cost, but I'm sure it's much more than the clocks that are on even the
most expensive motherboards.

It's possible (or even LIKELY) that the differences you see between the
AMD and Intel platforms stem from something along the lines of using
different or the same oscillator for different parts of the system. One
way to reduce the clock-drift between different parts (assuming this is
something desirable, and that is a debate of it's own) would be to use
the input to the CPU PLL as an input to the (PLL that drives) HPET. I
know for sure that the AMD chipset clock for HPET is NOT derived from
the same as the CPU clock source. I don't know how Intel has solved
their clocking...

And a note on RDTSC: There's a RDTSCP which is a "serializing"
instruction. It also clobbers, I think, ECX, with the current CPU
number. Standard RDTSC is not serializing, so it can be performed out of
order by several instructions. I doubt that this will cause much of the
error discussed, but it may contribute in a positive or negative way...

--
Mats

> -----Original Message-----
> From: xen-devel-bounces@...
> [mailto:xen-devel-bounces@...] On Behalf Of
> Keir Fraser
> Sent: 29 July 2005 19:54
> To: Ian Pratt
> Cc: xen-devel List; Michael Hohnbaum
> Subject: Re: [Xen-devel] HPET/PIT timer accuracy
>
>
> On 29 Jul 2005, at 19:29, Ian Pratt wrote:
>
> >> We're not relying on TSC synchronisation between CPUs.
> They can run
> >> at different frequencies, stop in deep sleep, and so on.
> But we *do*
> >> want the frequency of each one to be as stable as possible. I'd
> >> expect to get <1ppm stability from a crystal source at constant
> >> temperature, no problem.
> >
> > I'd wager the TSC is stable and it's the HPET that's wandering...
> >
> > We'll probably just have to add more damping on the re-calibration.
>
> I haven't done any further tests to see how low the frequency
> of wandering goes. Perhaps we'll find the platform timer is
> more stable over e.g., 60s periods.
>
>   -- Keir
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@...
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Keir Fraser]

On 1 Aug 2005, at 13:17, Petersson, Mats wrote:

> You have to consider that the HPET and TSC are both driven by a
> separate
> PLL that is based of some sort of crystal/oscillator. These are often
> not as precise as we'd want them to be. If you do a google with
> "crystal
> oscillator ppm" (or some such), you will find that most of these are
> within the 50-100 ppm ranges [these are the guaranteed values, which
> probably means that the typical is around 25-50 ppm, much along the
> lines of your findings]. If you want more precise crystals than that,
> you'll be looking at expensive stuff, and when you want less than
> around
> 10 ppm, you're probably looking at temperature controlled ones (aka
> oven
> oscillators). They are both expensive and bulky, and very unlikely to
> be
> found on the motherboard.

Those variations of up to 100ppm are primarily composed of two factors
which I can definitely discount from my own tests: drift due to
temperature change, and frequency tolerance. Where a separate frequency
tolerance figure is not given, that is often folded into the overall
stability figure: I know that can be 100ppm for a chipset crystal. And
that'll be why a 2ppm clock generator is so expensive and bulky:
tolerance is hard to achieve, and drift is hard to compensate for over
a wide temperature range (that circuitry is where the bulk comes from).

*However* I am talking about frequency jitter at 1Hz: any constant
deviation from the frequency stamped on the can, due to 'tolerances',
can obviously be discounted. Also, at 1Hz temperature changes will be
negligible, and the jitter continues at the same magnitude even when
the machine has warmed up and reached a steady temperature.

A crystal simply shouldn't jitter at that frequency, all external
factors held constant. It's resonating at its natural frequency (or
some harmonic) after all.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[Ian Pratt]

> You have to consider that the HPET and TSC are both driven by
> a separate PLL that is based of some sort of
> crystal/oscillator. These are often not as precise as we'd
> want them to be. If you do a google with "crystal oscillator
> ppm" (or some such), you will find that most of these are
> within the 50-100 ppm ranges [these are the guaranteed
> values, which probably means that the typical is around 25-50
> ppm, much along the lines of your findings].

The figures you are quoting are for frequency *accuracy*, not
*stability*. Crystals typically have good stability, varying mostly as a
result of temperature changes.

The only time I've ever seen such high instability on a crystal was when
the load capacitors were a couple of orders of magnitude out and it was
a miracle the circuit was oscillating at all...

Ian



_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

[King, Steven R]

The RDTSC instruction is out-of-order on Intel CPU's as well.  You can
place serializing instructions such as CPUID before and/or after RDTSC
to get better locality.  Unfortunately, CPUID expends many more cycles
than the RDTSC instruction on its own.

-----Original Message-----
From: xen-devel-bounces@...
[mailto:xen-devel-bounces@...] On Behalf Of Benjamin
LaHaise
Sent: Friday, July 29, 2005 6:33 PM
To: Ronald G. Minnich
Cc: Ian Pratt; xen-devel List
Subject: Re: [Xen-devel] HPET/PIT timer accuracy

On Fri, Jul 29, 2005 at 11:51:19AM -0600, Ronald G. Minnich wrote:
> I can only tell you we've seen weird numbers from the opteron tsc, to
> the point that we don't entirely trust it. Not sure why.

Opterons can execute RDTSC out of order.

                -ben
--
"Time is what keeps everything from happening all at once." -- John
Wheeler

_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@...
http://lists.xensource.com/xen-devel