Interrupt Descriptions

[John Baldwin]

A few folks have asked recently for the ability to add descriptive strings to
registered interrupt handlers.  This is especially true since the advent of
MSI with multiple interrupts per device.  I hacked up a prototype today that
adds a new 'bus_describe_intr()' that takes the IRQ resource, the void *
cookie returned by bus_setup_intr() and var args description and appends that
to the interrupt name in the thread and vmstat -i info.  The current patch
only has the MI bits and the MD bits for amd64 as well as a sample change to
the igb(4) driver.

The patch is at http://www.FreeBSD.org/~jhb/patches/intr_describe.patch.

An example from this patch is:

> vmstat -i
interrupt                          total       rate
irq1: atkbd0                           8          0
irq4: uart0                          751          5
irq6: fdc0                             6          0
irq14: ata0                           36          0
irq20: uhci0                          20          0
irq23: uhci3 ehci0                     2          0
irq28: mpt0                         1661         11
irq256: igb0:tx 0                    880          6
irq257: igb0:rx 0                   1098          7
irq258: igb0:link                      3          0
irq259: igb1:tx 0                      1          0
irq260: igb1:rx 0                    134          0
irq261: igb1:link                      3          0

--
John Baldwin
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Jack Vogel]

Good job John, will make my test department very happy :)
Once this is in CURRENT I will make changes to the ixgbe driver
to use it also.

Jack


On Wed, Sep 30, 2009 at 2:32 PM, John Baldwin <jhb@...> wrote:

> A few folks have asked recently for the ability to add descriptive strings
> to
> registered interrupt handlers.  This is especially true since the advent of
> MSI with multiple interrupts per device.  I hacked up a prototype today
> that
> adds a new 'bus_describe_intr()' that takes the IRQ resource, the void *
> cookie returned by bus_setup_intr() and var args description and appends
> that
> to the interrupt name in the thread and vmstat -i info.  The current patch
> only has the MI bits and the MD bits for amd64 as well as a sample change
> to
> the igb(4) driver.
>
> The patch is at http://www.FreeBSD.org/~jhb/patches/intr_describe.patch<http://www.FreeBSD.org/%7Ejhb/patches/intr_describe.patch>
> .
>
> An example from this patch is:
>
> > vmstat -i
> interrupt                          total       rate
> irq1: atkbd0                           8          0
> irq4: uart0                          751          5
> irq6: fdc0                             6          0
> irq14: ata0                           36          0
> irq20: uhci0                          20          0
> irq23: uhci3 ehci0                     2          0
> irq28: mpt0                         1661         11
> irq256: igb0:tx 0                    880          6
> irq257: igb0:rx 0                   1098          7
> irq258: igb0:link                      3          0
> irq259: igb1:tx 0                      1          0
> irq260: igb1:rx 0                    134          0
> irq261: igb1:link                      3          0
>
> --
> John Baldwin
> _______________________________________________
> freebsd-arch@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."
>

_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Bruce Evans-4]

On Wed, 30 Sep 2009, John Baldwin wrote:

> A few folks have asked recently for the ability to add descriptive strings to
> registered interrupt handlers.  This is especially true since the advent of
> MSI with multiple interrupts per device.  I hacked up a prototype today that

Interrupt names should be no longer than 4 (5 works sometimes) characters so
that they can be displayed by systat -v.

>...
>> vmstat -i
> interrupt                          total       rate
> irq1: atkbd0                           8          0

The 4-5 characters doesn't include "irqN".  (systat -v already does extra
work to avoid the bad naming scheme visible here.  It moves the "irqN" to
the end, and removes the ":", else truncating these names would give a
larger mess and there would only be space for 3-4 characters of the device
name after wasting 1 for the ":".)

"atkbd0" is already far too long, so systat -v would display "atkbd0 irq"
in the 10 columns available, except it omits "irq" in this case so
as to print the interrupt number, and actually displays "atkbd0 1".

> irq4: uart0                          751          5
> irq6: fdc0                             6          0
> irq14: ata0                           36          0
> irq20: uhci0                          20          0

These are short enough to be displayed by systat -v (after removal of the
":" in 2 cases and after removal of the ":" and the "irq" in 1 case).

> irq23: uhci3 ehci0                     2          0

systat -v has no chance of displaying multiple devices per interrupt.  I
think it finds no space to print even the irq number, and also loses
the ehci number after truncating at "uhci3 ehci".

> irq28: mpt0                         1661         11

Short enough.

> irq256: igb0:tx 0                    880          6

I think this gets truncated to "igb0:tx 0 ".  The device name doesn't
follows the rules for naming devices (should contain no whitespace), so
the final "0" looks like the interrupt number.

Interrupt numbers > 99 cause further problems for systat.  Now there
is only space for 3 characters in the device name ("xx0 irq256" takes
10 columns).

> irq257: igb0:rx 0                   1098          7
> irq258: igb0:link                      3          0
> irq259: igb1:tx 0                      1          0
> irq260: igb1:rx 0                    134          0
> irq261: igb1:link                      3          0

Similar messes.

systat has no support for varying the terminal width, and adding such
support would be difficult because the display is very complicated and
hard-coded, so the truncation occurs even if you have a wide terminal.
Adding support for the English parser needed to translate more cases would
be even more difficult.  The "irqN:" translation is MD (probably only
helps for amd64 and i386).

The above output shows related misformatting for vmstat.  Interrupt
name strings are supposed to be NUL-terminated and packed into a string
space, mainly to match the historical interface to vmstat, which was
designed mainly for space-efficiency.  I think they are still in a
string space, but at least on amd64 and i386 the implementation is
extremely space-inefficient, with many unused entries and every entry
padded with spaces to length MAXCOMLEN bytes (plus the NUL terminator).
MAXCOMLEN is 19, and most of the 19 are used for the verbose names in
the above.  Then vmstat leaves about 15 more spaces before the totals
column (it uses format "%-*s %20llu..."), giving the above ugly output.
The "*" in the format is supposed to be dynamic, but this is defeated
by the space-padding.  Leaving space for 20-digit numbers is bogus:
the 20 is a hard coding of the number of digits needed to display the
largest 64-bit long long, but long longs might be longer than that,
and the maximum is unreachable even with 64-bit long longs.

OTOH, MAXCOMLEN is a bit short for a description.

Bruce
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Poul-Henning Kamp]

In message <20091001090218.L21015@...>, Bruce Evans writes:

>Interrupt names should be no longer than 4 (5 works sometimes) characters so
>that they can be displayed by systat -v.

I disagree.  Bytes are cheaper now than they were on a PDP11.  We should
use as many as is necessary to convey sensible information.

--
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@...         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Andrew Brampton-6]

2009/9/30 John Baldwin <jhb@...>:

> A few folks have asked recently for the ability to add descriptive strings to
> registered interrupt handlers.  This is especially true since the advent of
> MSI with multiple interrupts per device.  I hacked up a prototype today that
> adds a new 'bus_describe_intr()' that takes the IRQ resource, the void *
> cookie returned by bus_setup_intr() and var args description and appends that
> to the interrupt name in the thread and vmstat -i info.  The current patch
> only has the MI bits and the MD bits for amd64 as well as a sample change to
> the igb(4) driver.
>
> The patch is at http://www.FreeBSD.org/~jhb/patches/intr_describe.patch.
>

Very cool feature, which I had been hoping for on FreeBSD.

Well done.
Andrew
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Rui Paulo-2]

On 1 Oct 2009, at 01:02, Bruce Evans wrote:

> On Wed, 30 Sep 2009, John Baldwin wrote:
>
>> A few folks have asked recently for the ability to add descriptive  
>> strings to
>> registered interrupt handlers.  This is especially true since the  
>> advent of
>> MSI with multiple interrupts per device.  I hacked up a prototype  
>> today that
>
> Interrupt names should be no longer than 4 (5 works sometimes)  
> characters so
> that they can be displayed by systat -v.

We should just change systat, IMHO, but I know it's not easy.

--
Rui Paulo



_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Andriy Gapon]

on 01/10/2009 00:32 John Baldwin said the following:

> A few folks have asked recently for the ability to add descriptive strings to
> registered interrupt handlers.  This is especially true since the advent of
> MSI with multiple interrupts per device.  I hacked up a prototype today that
> adds a new 'bus_describe_intr()' that takes the IRQ resource, the void *
> cookie returned by bus_setup_intr() and var args description and appends that
> to the interrupt name in the thread and vmstat -i info.  The current patch
> only has the MI bits and the MD bits for amd64 as well as a sample change to
> the igb(4) driver.
>
> The patch is at http://www.FreeBSD.org/~jhb/patches/intr_describe.patch.
>
> An example from this patch is:
>
>> vmstat -i
> interrupt                          total       rate
> irq1: atkbd0                           8          0
> irq4: uart0                          751          5
> irq6: fdc0                             6          0
> irq14: ata0                           36          0
> irq20: uhci0                          20          0
> irq23: uhci3 ehci0                     2          0
> irq28: mpt0                         1661         11
> irq256: igb0:tx 0                    880          6
> irq257: igb0:rx 0                   1098          7
> irq258: igb0:link                      3          0
> irq259: igb1:tx 0                      1          0
> irq260: igb1:rx 0                    134          0
> irq261: igb1:link                      3          0
>

Example above doesn't demonstrate what happens when there are shared interrupts.
Would it still look nice (with sufficiently long descriptions)?

--
Andriy Gapon
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Bruce Evans-4]

On Thu, 1 Oct 2009, Poul-Henning Kamp wrote:

> In message <20091001090218.L21015@...>, Bruce Evans writes:
>
>> Interrupt names should be no longer than 4 (5 works sometimes) characters so
>> that they can be displayed by systat -v.
>
> I disagree.  Bytes are cheaper now than they were on a PDP11.  We should
> use as many as is necessary to convey sensible information.

s/should/shall/ then.  It is the API that interrupt names must be no
longer than 4.  You can disagree with this being the best API but not
with what it is.

Using many bytes here results in no information being conveyed in some
cases, not sensible information, when the long description is truncated
to 3-4 characters to fit in the available space.

Bruce
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Robert Noland]

On Thu, 2009-10-01 at 15:49 +0300, Andriy Gapon wrote:

> on 01/10/2009 00:32 John Baldwin said the following:
> > A few folks have asked recently for the ability to add descriptive strings to
> > registered interrupt handlers.  This is especially true since the advent of
> > MSI with multiple interrupts per device.  I hacked up a prototype today that
> > adds a new 'bus_describe_intr()' that takes the IRQ resource, the void *
> > cookie returned by bus_setup_intr() and var args description and appends that
> > to the interrupt name in the thread and vmstat -i info.  The current patch
> > only has the MI bits and the MD bits for amd64 as well as a sample change to
> > the igb(4) driver.
> >
> > The patch is at http://www.FreeBSD.org/~jhb/patches/intr_describe.patch.
> >
> > An example from this patch is:
> >
> >> vmstat -i
> > interrupt                          total       rate
> > irq1: atkbd0                           8          0
> > irq4: uart0                          751          5
> > irq6: fdc0                             6          0
> > irq14: ata0                           36          0
> > irq20: uhci0                          20          0
> > irq23: uhci3 ehci0                     2          0
> > irq28: mpt0                         1661         11
> > irq256: igb0:tx 0                    880          6
> > irq257: igb0:rx 0                   1098          7
> > irq258: igb0:link                      3          0
> > irq259: igb1:tx 0                      1          0
> > irq260: igb1:rx 0                    134          0
> > irq261: igb1:link                      3          0
> >
>
> Example above doesn't demonstrate what happens when there are shared interrupts.
> Would it still look nice (with sufficiently long descriptions)?

vmstat -i already chops off lots of stuff if there are very many shared
irqs.

robert.

--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Bruce Evans-4]

On Thu, 1 Oct 2009, Rui Paulo wrote:

> On 1 Oct 2009, at 01:02, Bruce Evans wrote:
>
>> On Wed, 30 Sep 2009, John Baldwin wrote:
>>
>>> A few folks have asked recently for the ability to add descriptive strings
>>> to
>>> registered interrupt handlers.  This is especially true since the advent
>>> of
>>> MSI with multiple interrupts per device.  I hacked up a prototype today
>>> that
>>
>> Interrupt names should be no longer than 4 (5 works sometimes) characters
>> so
>> that they can be displayed by systat -v.

Make that: "Interrupt names must must contain only characters that
take 1 space to print, and must be no longer than 10 characters, or
unique and without garbage when blindly truncated to 10 characters,
or in a special form that can easily be understood and compressed to
<= 10 characters.  The special forms are..."

Similarly for device names (max length 4 or 5) and sleep message strings
(max length 6).

> We should just change systat, IMHO, but I know it's not easy.

It's impossible to make changes that increase display resource requirements
without breaking something.  Either small displays stop working or scrolling
is required.

Bruce
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Robert Noland]

On Thu, 2009-10-01 at 23:31 +1000, Bruce Evans wrote:

> On Thu, 1 Oct 2009, Rui Paulo wrote:
>
> > On 1 Oct 2009, at 01:02, Bruce Evans wrote:
> >
> >> On Wed, 30 Sep 2009, John Baldwin wrote:
> >>
> >>> A few folks have asked recently for the ability to add descriptive strings
> >>> to
> >>> registered interrupt handlers.  This is especially true since the advent
> >>> of
> >>> MSI with multiple interrupts per device.  I hacked up a prototype today
> >>> that
> >>
> >> Interrupt names should be no longer than 4 (5 works sometimes) characters
> >> so
> >> that they can be displayed by systat -v.
>
> Make that: "Interrupt names must must contain only characters that
> take 1 space to print, and must be no longer than 10 characters, or
> unique and without garbage when blindly truncated to 10 characters,
> or in a special form that can easily be understood and compressed to
> <= 10 characters.  The special forms are..."
>
> Similarly for device names (max length 4 or 5) and sleep message strings
> (max length 6).
>
> > We should just change systat, IMHO, but I know it's not easy.
>
> It's impossible to make changes that increase display resource requirements
> without breaking something.  Either small displays stop working or scrolling
> is required.

While I have always liked the fact that systat -v was capable of
displaying most anything that you would care about in one display, lack
of display real-estate is not a sufficient reason to limit the
descriptions.

Perhaps it is time for systat to get a bit of an overhaul...

robert.
 
> Bruce
> _______________________________________________
> freebsd-arch@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."
--
Robert Noland <rnoland@...>
FreeBSD

_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[John Baldwin]

On Wednesday 30 September 2009 8:02:57 pm Bruce Evans wrote:
> On Wed, 30 Sep 2009, John Baldwin wrote:
>
> > A few folks have asked recently for the ability to add descriptive strings to
> > registered interrupt handlers.  This is especially true since the advent of
> > MSI with multiple interrupts per device.  I hacked up a prototype today that
>
> Interrupt names should be no longer than 4 (5 works sometimes) characters so
> that they can be displayed by systat -v.

For the machines this is targeted at (i.e. ones with devices using MSI/MSI-X)
the number of interrupts is already well beyond what systat -v can currently
display anyway.  I believe most folks use vmstat -i or ithread stats in top
to measure real-time interrupts at this point.  Possibly systat could grow a
new interrupt-only mode that has sufficient room for full interrupt names.

FYI, it actually looks decent enough running a 7.x systat on a test 9.0 kernel:

                                     25000 frevn          pdpgs  2000 cpu0: time
                                                          intrn       igb0:tx 0
Disks   da0 pass0                                  473896 wire     25 igb0:rx 0
KB/t   0.00  0.00                                   13872 act         igb0:link
tps       0     0                                   10080 inact       igb1:tx 0
MB/s   0.00  0.00                                     128 cache     1 igb1:rx 0
%busy     0     0                                 7603460 free        igb1:link

For these devices with MSI, the description (tx 0, tx 1, link, etc.) is
actually more meaningful than the IRQ.

--
John Baldwin
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[John Baldwin]

On Thursday 01 October 2009 8:49:43 am Andriy Gapon wrote:

> on 01/10/2009 00:32 John Baldwin said the following:
> > A few folks have asked recently for the ability to add descriptive strings to
> > registered interrupt handlers.  This is especially true since the advent of
> > MSI with multiple interrupts per device.  I hacked up a prototype today that
> > adds a new 'bus_describe_intr()' that takes the IRQ resource, the void *
> > cookie returned by bus_setup_intr() and var args description and appends that
> > to the interrupt name in the thread and vmstat -i info.  The current patch
> > only has the MI bits and the MD bits for amd64 as well as a sample change to
> > the igb(4) driver.
> >
> > The patch is at http://www.FreeBSD.org/~jhb/patches/intr_describe.patch.
> >
> > An example from this patch is:
> >
> >> vmstat -i
> > interrupt                          total       rate
> > irq1: atkbd0                           8          0
> > irq4: uart0                          751          5
> > irq6: fdc0                             6          0
> > irq14: ata0                           36          0
> > irq20: uhci0                          20          0
> > irq23: uhci3 ehci0                     2          0
> > irq28: mpt0                         1661         11
> > irq256: igb0:tx 0                    880          6
> > irq257: igb0:rx 0                   1098          7
> > irq258: igb0:link                      3          0
> > irq259: igb1:tx 0                      1          0
> > irq260: igb1:rx 0                    134          0
> > irq261: igb1:link                      3          0
> >
>
> Example above doesn't demonstrate what happens when there are shared interrupts.
> Would it still look nice (with sufficiently long descriptions)?

Currently MSI interrupts generally aren't shared.  If they are shared what
happens is that the 'name:descr' bit is treated as the full name of the
device and the normal "+" or "*" truncation stuff kicks in so you would
maybe have:

irq280: igb0:tx 0 igb0:tx 1

except that doesn't fit in MAXCOMLEN so you would have:

irq280: igb0:tx 0+

For the ithread name we are stuck with MAXCOMLEN.  I would like to overhaul
the intrcnt stuff to possibly allow for longer names and be mostly MI (having
all the longs shared across cache lines that might be shared by multiple CPUs
isn't ideal either at this point).  I am considering having either an array or
list of intrcnt structures with a name and a count with a sysctl that exports
an array for vmstat to use (the kvm bits could always rebuild the list using
the same logic as the sysctl handler for crash dumps).  The current design is
very limiting and forces the weird space padding to make renaming intrcnt
descriptions manageable when adding/removing interrupt handlers.

--
John Baldwin
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Bruce Evans-4]

On Thu, 1 Oct 2009, Robert Noland wrote:

> On Thu, 2009-10-01 at 15:49 +0300, Andriy Gapon wrote:
>> on 01/10/2009 00:32 John Baldwin said the following:
>>> ...
>>> An example from this patch is:
>>>
>>>> vmstat -i
>>> interrupt                          total       rate
>>> irq1: atkbd0                           8          0
>>> irq4: uart0                          751          5
>>> irq6: fdc0                             6          0
>>> irq14: ata0                           36          0
>>> irq20: uhci0                          20          0
>>> irq23: uhci3 ehci0                     2          0
>>> irq28: mpt0                         1661         11
>>> irq256: igb0:tx 0                    880          6
>>> irq257: igb0:rx 0                   1098          7
>>> irq258: igb0:link                      3          0
>>> irq259: igb1:tx 0                      1          0
>>> irq260: igb1:rx 0                    134          0
>>> irq261: igb1:link                      3          0
>>
>> Example above doesn't demonstrate what happens when there are shared interrupts.

irq23 seems to be shared by uhci3 and ehci0.  This part of the description
is not new (I think device names are just concatenated until space runs out).

>> Would it still look nice (with sufficiently long descriptions)?
>
> vmstat -i already chops off lots of stuff if there are very many shared
> irqs.

Not vmstat -i, but the kernel.  vmstat -i attempts to display the
entire string, unlike systat -v, but the kernel only has space for
strings of length MAXCOMLEN - 1 (18) (ie_name[MAXCOMLEN] nd
ie_fullname[MAXCOMLEN] in <sys/interrupt.h>) (unless jhb increased
this).  This is the same limit as for command names of executables,
except for the off-by-1 error for interrupt names, and for various
bugs involving thread names (p_comm[MAXCOMLEN + 1], etc.).

Concatenating thread names with process names causes similar display
problems in ps and top, etc.  Space is short for displaying just
p_comm[] plus args, and showing td_name[] too needs more space.  Actually
concatenation doesn't work at all.  ps must select the "name" string
that works, but it gets this wrong somehow (ps -laxHw seems to display
only p_comm (ki_comm in ps), resulting in useless names for all idle
and interrupt threads, although ps has code to support td_name (ki_ocomm
is abused to hold the part of this that fits in ps).  -H works almost
right in top, and the truncated td_name string can be seen using the
undocumented tdnam keyword in ps).

Bruce
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Bruce Evans-4]

On Thu, 1 Oct 2009, John Baldwin wrote:

> On Wednesday 30 September 2009 8:02:57 pm Bruce Evans wrote:
>> On Wed, 30 Sep 2009, John Baldwin wrote:
>>
>>> A few folks have asked recently for the ability to add descriptive strings to
>>> registered interrupt handlers.  This is especially true since the advent of
>>> MSI with multiple interrupts per device.  I hacked up a prototype today that
>>
>> Interrupt names should be no longer than 4 (5 works sometimes) characters so
>> that they can be displayed by systat -v.
>
> For the machines this is targeted at (i.e. ones with devices using MSI/MSI-X)
> the number of interrupts is already well beyond what systat -v can currently
> display anyway.

This should be fixed before expanding the names into descriptions.

I wonder what uses all these interrupts.  Most FreeBSD cluster machines use
a whole 3 interrupts apart from acpi_timer, with 1 barely active (1 disk,
1 net, and 1 serial i/o probably only used for a serial console).  My
workstations use about 10 plus timers, but most are usually inactive
except for net, keyboard and sometimes disk.  Does anyone have systems
more than about 16 disk controllers and 16 network cards so as to need
more than 32 interrupts?

> I believe most folks use vmstat -i or ithread stats in top
> to measure real-time interrupts at this point.  Possibly systat could grow a
> new interrupt-only mode that has sufficient room for full interrupt names.

I don't find static info and thus vmstat -i very useful.  vmstat with a
repeat interval is useful, but for some reason vmstat -i never supported
the repeat interval (its multi-line display is certainly unsuitable for
repetitions being readable), and with systat -v repetition in vmstat -i
is rarely missed.  A repeating vmstat -i would have even more space
problems than systat -v, since it would want to repeat single line output.

systat and most things just want a unique tag, not a description or even
a full name.  The full name could be looked up from a tag by looking at
vmstat -i output which could show both.

>
> FYI, it actually looks decent enough running a 7.x systat on a test 9.0 kernel:
>
>                                     25000 frevn          pdpgs  2000 cpu0: time
>                                                          intrn       igb0:tx 0
> Disks   da0 pass0                                  473896 wire     25 igb0:rx 0
> KB/t   0.00  0.00                                   13872 act         igb0:link
> tps       0     0                                   10080 inact       igb1:tx 0
> MB/s   0.00  0.00                                     128 cache     1 igb1:rx 0
> %busy     0     0                                 7603460 free        igb1:link
>
> For these devices with MSI, the description (tx 0, tx 1, link, etc.) is
> actually more meaningful than the IRQ.

The irq is probably least useful for most devices.

The above is only reasonable because I changed systat to move "irqN:
" to " irqN" at the end.  "irq256: bce0" on FreeBSD cluster machines
is displayed reasonably as "bce0 256".  But systat shouldn't have to
do this.  At least the form of the names used by systat (== the one
given by the old intrnames API) should have names that are carefully
phrased to have the most useful info first, and to not waste space
with ": " or even "irq".

Bruce
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Jack Vogel]

What uses them, machines with multiqueue, in my ixgbe driver a rx/tx pair of
queues
is configured and bound to each cpu. This ability really matters when you do
virtualization
because the queues can be assigned per guest. So right now with 16 cpus a
single
Niantic or Oplin port (10G) has 33 MSIX vectors.

I brought this request to John because our validation group complained. On
Linux
they have a description for each interrupt and they want the same for
FreeBSD.

Welcome to the 21st century :)

Jack



On Thu, Oct 1, 2009 at 10:14 AM, Bruce Evans <brde@...> wrote:

> On Thu, 1 Oct 2009, John Baldwin wrote:
>
>  On Wednesday 30 September 2009 8:02:57 pm Bruce Evans wrote:
>>
>>> On Wed, 30 Sep 2009, John Baldwin wrote:
>>>
>>>  A few folks have asked recently for the ability to add descriptive
>>>> strings to
>>>> registered interrupt handlers.  This is especially true since the advent
>>>> of
>>>> MSI with multiple interrupts per device.  I hacked up a prototype today
>>>> that
>>>>
>>>
>>> Interrupt names should be no longer than 4 (5 works sometimes) characters
>>> so
>>> that they can be displayed by systat -v.
>>>
>>
>> For the machines this is targeted at (i.e. ones with devices using
>> MSI/MSI-X)
>> the number of interrupts is already well beyond what systat -v can
>> currently
>> display anyway.
>>
>
> This should be fixed before expanding the names into descriptions.
>
> I wonder what uses all these interrupts.  Most FreeBSD cluster machines use
> a whole 3 interrupts apart from acpi_timer, with 1 barely active (1 disk,
> 1 net, and 1 serial i/o probably only used for a serial console).  My
> workstations use about 10 plus timers, but most are usually inactive
> except for net, keyboard and sometimes disk.  Does anyone have systems
> more than about 16 disk controllers and 16 network cards so as to need
> more than 32 interrupts?
>
>  I believe most folks use vmstat -i or ithread stats in top
>> to measure real-time interrupts at this point.  Possibly systat could grow
>> a
>> new interrupt-only mode that has sufficient room for full interrupt names.
>>
>
> I don't find static info and thus vmstat -i very useful.  vmstat with a
> repeat interval is useful, but for some reason vmstat -i never supported
> the repeat interval (its multi-line display is certainly unsuitable for
> repetitions being readable), and with systat -v repetition in vmstat -i
> is rarely missed.  A repeating vmstat -i would have even more space
> problems than systat -v, since it would want to repeat single line output.
>
> systat and most things just want a unique tag, not a description or even
> a full name.  The full name could be looked up from a tag by looking at
> vmstat -i output which could show both.
>
>
>> FYI, it actually looks decent enough running a 7.x systat on a test 9.0
>> kernel:
>>
>>                                    25000 frevn          pdpgs  2000 cpu0:
>> time
>>                                                         intrn
>> igb0:tx 0
>> Disks   da0 pass0                                  473896 wire     25
>> igb0:rx 0
>> KB/t   0.00  0.00                                   13872 act
>> igb0:link
>> tps       0     0                                   10080 inact
>> igb1:tx 0
>> MB/s   0.00  0.00                                     128 cache     1
>> igb1:rx 0
>> %busy     0     0                                 7603460 free
>>  igb1:link
>>
>> For these devices with MSI, the description (tx 0, tx 1, link, etc.) is
>> actually more meaningful than the IRQ.
>>
>
> The irq is probably least useful for most devices.
>
> The above is only reasonable because I changed systat to move "irqN:
> " to " irqN" at the end.  "irq256: bce0" on FreeBSD cluster machines
> is displayed reasonably as "bce0 256".  But systat shouldn't have to
> do this.  At least the form of the names used by systat (== the one
> given by the old intrnames API) should have names that are carefully
> phrased to have the most useful info first, and to not waste space
> with ": " or even "irq".
>
> Bruce
>
> _______________________________________________
> freebsd-arch@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."
>

_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Bruce Evans-4]

On Thu, 1 Oct 2009, John Baldwin wrote:

> For the ithread name we are stuck with MAXCOMLEN.  I would like to overhaul
> the intrcnt stuff to possibly allow for longer names and be mostly MI (having
> all the longs shared across cache lines that might be shared by multiple CPUs
> isn't ideal either at this point).  I am considering having either an array or
> list of intrcnt structures with a name and a count with a sysctl that exports
> an array for vmstat to use (the kvm bits could always rebuild the list using
> the same logic as the sysctl handler for crash dumps).  The current design is
> very limiting and forces the weird space padding to make renaming intrcnt
> descriptions manageable when adding/removing interrupt handlers.

The design has few limits.  My old implementation had no limits on
name lengths.  It just had a limit on the combined length, due to it
allocating the string space statically.  String spaces without padding
require about 10 extra lines to create and zero extra lines to use
(since the users already expect a string space so they already have
the extra lines to walk over it).  Changing the descriptions, including
rebuilding the whole string space whenever an interrupt handler is
added/removed/renamed shouldn't be a problem since this is a rare
event.  Just lock out the sysctl and other accesses to the string space
while changing/reallocating the string space.  Counters are harder --
you can't move them without locking out interrupt handlers that
increment them.

You could also delay rebuilding until the sysctl (wouldn't be so easy
for kvm).  The sysctl would take longer, but perhaps no longer than
now since the data size should be much smaller -- instead of 100's of
padded strings there should be only a few unpadded ones (the strings
on ref8-i386 now consists of 49 "stray irqN" (padded), 42 unused "irqN:"
(padded), one "???" (padded; unused I think -- this "???" is for when
the table fills up and/or when an unexpected irq occurs, but the table
now covers everything and has zillions of stray slots for unexpected
ireq), and 15 lines for interrupts actually attached (8 lapic timer,
2 moderately used, 1 rarely used, 3 never used).

systat wouldn't like the tables' sizes or string contents changing
underneath it, but in the rare event that this happens it couldn't do
worse than crash, and it could detect the change using a new generation
count.  vmstat wouldn't have any problems since it only reads the tables
once (minor race between the independent sysctls).

The kvm interface doesn't support dynamic table allocation.  You would
probably have to change it eventually to support per-CPU counters if
you actually want to keep supporting interrupt stats in dead kernels.

Bruce
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[John Baldwin]

On Thursday 01 October 2009 1:54:15 pm Bruce Evans wrote:

> On Thu, 1 Oct 2009, John Baldwin wrote:
>
> > For the ithread name we are stuck with MAXCOMLEN.  I would like to overhaul
> > the intrcnt stuff to possibly allow for longer names and be mostly MI (having
> > all the longs shared across cache lines that might be shared by multiple CPUs
> > isn't ideal either at this point).  I am considering having either an array or
> > list of intrcnt structures with a name and a count with a sysctl that exports
> > an array for vmstat to use (the kvm bits could always rebuild the list using
> > the same logic as the sysctl handler for crash dumps).  The current design is
> > very limiting and forces the weird space padding to make renaming intrcnt
> > descriptions manageable when adding/removing interrupt handlers.
>
> The design has few limits.  My old implementation had no limits on
> name lengths.  It just had a limit on the combined length, due to it
> allocating the string space statically.  String spaces without padding
> require about 10 extra lines to create and zero extra lines to use
> (since the users already expect a string space so they already have
> the extra lines to walk over it).  Changing the descriptions, including
> rebuilding the whole string space whenever an interrupt handler is
> added/removed/renamed shouldn't be a problem since this is a rare
> event.  Just lock out the sysctl and other accesses to the string space
> while changing/reallocating the string space.  Counters are harder --
> you can't move them without locking out interrupt handlers that
> increment them.

I consider having to rebuild the entire namespace when changing one name
a design limitation.  I think it worked fine when you had machines without
hotplug hardware (or drivers) where the set of interrupts in use was
essentially built once during boot and static thereafter.  However, that
is no longer the machines we have.

--
John Baldwin
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[John Baldwin]

On Thursday 01 October 2009 1:14:01 pm Bruce Evans wrote:

> On Thu, 1 Oct 2009, John Baldwin wrote:
>
> > On Wednesday 30 September 2009 8:02:57 pm Bruce Evans wrote:
> >> On Wed, 30 Sep 2009, John Baldwin wrote:
> >>
> >>> A few folks have asked recently for the ability to add descriptive strings to
> >>> registered interrupt handlers.  This is especially true since the advent of
> >>> MSI with multiple interrupts per device.  I hacked up a prototype today that
> >>
> >> Interrupt names should be no longer than 4 (5 works sometimes) characters so
> >> that they can be displayed by systat -v.
> >
> > For the machines this is targeted at (i.e. ones with devices using MSI/MSI-X)
> > the number of interrupts is already well beyond what systat -v can currently
> > display anyway.
>
> This should be fixed before expanding the names into descriptions.

To me this change doesn't make systat any more or less broken than it was
before, so I consider systat an orthogonal problem.

> I wonder what uses all these interrupts.  Most FreeBSD cluster machines use
> a whole 3 interrupts apart from acpi_timer, with 1 barely active (1 disk,
> 1 net, and 1 serial i/o probably only used for a serial console).  My
> workstations use about 10 plus timers, but most are usually inactive
> except for net, keyboard and sometimes disk.  Does anyone have systems
> more than about 16 disk controllers and 16 network cards so as to need
> more than 32 interrupts?

As Jack mentioned, many newer devices are starting to take advantage of MSI
to use multiple interrupts for a single device.  A typical cxgb(4) adapter
will use up to 9 interrupts (1 for link/misc, and 1 per-CPU for a RX/TX
queueset).  The Intel 10gb driver takes this even farther by giving each
queueset separate RX/TX interrupts and it supports up to 16 queues giving
up to 33 total interrupts per device.  Even the igb(4) driver currently
uses 1 interrupt for link, 1 for each RX queue, and 1 for each TX queue and
it supports having at least 4 RX queues (possibly as many as 8?).

Devices of this nature using lots of interrupts is also what drove the change
to use per-CPU IDTs in 8.0 to increase the number of available IDT vectors
for these types of devices that want MSI interrupts tied to every CPU.

> > I believe most folks use vmstat -i or ithread stats in top
> > to measure real-time interrupts at this point.  Possibly systat could grow a
> > new interrupt-only mode that has sufficient room for full interrupt names.
>
> I don't find static info and thus vmstat -i very useful.  vmstat with a
> repeat interval is useful, but for some reason vmstat -i never supported
> the repeat interval (its multi-line display is certainly unsuitable for
> repetitions being readable), and with systat -v repetition in vmstat -i
> is rarely missed.  A repeating vmstat -i would have even more space
> problems than systat -v, since it would want to repeat single line output.

For certain tests simply doing a before and after of vmstat -i and figuring
out the diff manually (sometimes just having a diff at all is important
rather than the number (e.g. did some action I just did generate an
interrupt?)) is sufficient.

> systat and most things just want a unique tag, not a description or even
> a full name.  The full name could be looked up from a tag by looking at
> vmstat -i output which could show both.

This could be interesting perhaps.  I would probably want to change my
proposed intrcnt structure to contain three things then: a unique tag,
a full name, and a count.  One possible idea for the unique "tag" would
be to use the interrupt handler name if there is only a single handler
or to use the 'irqXXX' name if the interrupt is shared.

--
John Baldwin
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Bruce Evans-4]

On Thu, 1 Oct 2009, John Baldwin wrote:

> On Thursday 01 October 2009 1:54:15 pm Bruce Evans wrote:
>>
>> The design has few limits.  My old implementation had no limits on
>> name lengths....
>
> I consider having to rebuild the entire namespace when changing one name
> a design limitation.  I think it worked fine when you had machines without
> hotplug hardware (or drivers) where the set of interrupts in use was
> essentially built once during boot and static thereafter.

Those probably didn't have anything as sophisticated as building it at
boot time.  In 4.4BSD-Lite2 it is just hard-coded into locore for hp300,
sparc, luna64 and pmax, while for news3400 it is partially declared but
not defined.  It is missing even declarations for all other arches
including i386, and pmax has a comment saying that the counters aren't
used (perhaps they aren't used for any arch).  I fixed this for i386.

> However, that
> is no longer the machines we have.

Who cares if you have to rebuild the entire namespace occasionally.
Any sort of dynamic allocation would need to rebuild it to extend it.
String spaces are very easy to expand since they don't have any internal
pointers and shouldn't have many external ones.

Anyway, the design doesn't require rebuilding the namespace for all
changes.  My old implementation usually adds to the end of the string
space using statically allocated space.  There are external pointers
that keep the names indexed consistently with the counters.  New
interrupts normally get a new slot in intrcnt[].  Old slots are
intentionally not reused, except by devices with the same interrupt
name.  They preserve the names and interrupt counts of previously attached
devices.  Now, I think the intrcnt[] index is identical with the interrupt
number, and old driver names are lost and old interrupt counts are merged
(zeroed?) when a slot is reused.

The old implementation also handles sparsely allocated interrupt numbers
(which are just starting to become common) nicely without sparsely
allocating all the tables.  Now the tables use sparse allocation for
an enormous number of slots.  There seem to be 256 slots for non-MSI and
another 512 for MSI giving a basic 768.  This is then doubled to give
slots for stray interrupts, and there are a few special-purpose slots,
giving a total of a little more than 1536.  This gives a string space
size of about 32K.  To simplify the implementation, the space for MSI
and several APICs is allocated unconditionally.

Here is the 4.4BSD-Lite2 code for hp300:

% .globl _intrcnt,_eintrcnt,_intrnames,_eintrnames
% _intrnames:
% .asciz "spur"
% .asciz "hil"
% .asciz "lev2"
% .asciz "lev3"
% .asciz "lev4"
% .asciz "lev5"
% .asciz "dma"
% .asciz "clock"
% .asciz  "statclock"
% .asciz "nmi"
% _eintrnames:
% .even
% _intrcnt:
% .long 0,0,0,0,0,0,0,0,0,0
% _eintrcnt:

This takes 53 bytes (plus 1 or 3 for padding) for the string space.
Most machines still need about 53 bytes for the string space (don't
waste slots to count or name stray interrupts separately).  These 53
bytes can be built and processed by userland on every sysctl much
faster than the 32K of mostly-unused bytes can even be copied out.

Bruce
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."