Nabble - tech-smp

Re: >32 core support

2009-03-24T15:06:17Z

On Mon, 23 Mar 2009, Thor Lancelot Simon wrote:
> QEMU (or, at least, 'Q' on OS X, which uses QEMU as its guts) supports
> up to 128 CPUs, in 32 or 64 bit x86 emulation.

I've played with that some time ago, see
http://kerneltrap.org/index.php?q=mailarchive/netbsd-tech-kern/2005/12/22/292131

- Hubert

Re: >32 core support

2009-03-23T16:00:01Z

On Mon, Mar 23, 2009 at 10:46:00PM +0000, Andrew Doran wrote:
> For anyone interested in working on >32 core support here is a interesting
> link. It suggests that `bochs' can be patched to emulate a 48 core system.
>
> http://forum.osdev.org/viewtopic.php?f=1&t=19192

QEMU (or, at least, 'Q' on OS X, which uses QEMU as its guts) supports
up to 128 CPUs, in 32 or 64 bit x86 emulation.

We have some trouble with its ACPI implementation, or did last I checked
(a long time ago) which made it difficult but not impossible to get a
32 CPU system going, which was the most I tried.

Thor

>32 core support

2009-03-23T15:46:00Z

For anyone interested in working on >32 core support here is a interesting
link. It suggests that `bochs' can be patched to emulate a 48 core system.

http://forum.osdev.org/viewtopic.php?f=1&t=19192

x86 currently supports 32 cores/threads out of the box, but it may not be
enough for commodity servers within a couple of years. The low-level issues
I am aware of are:

- There are static arrays that will not scale (MAXCPUS). Some of these could
be folded into cpu_data, leaving us with a single array to worry about
sizing dynamically.

- There are 32-bit masks, the majority of which are in platform-specific
code. The kcpuset code needs work to do stuff that the kernel uses, like
atomic operations, scans and so on. A few masks are in MI code and some
will #error if MAXCPUS > 32.

- 32-bit systems don't have enough kernel address space to support large
numbers of cores effectively, although i386 could be interesting as a
bring-up target with an emulator like bochs.

- The x86 apic code currently supports 255 cores max (assuming 255 cores
plus 1 IOAPIC). x2apic support may be required in the future. As far as I
understand it, future systems may have cores that can only be addressed
with the x2apic's 32-bit IDs.

- The mutex/rwlock code will not scale to large numbers of cores because it
does a linear scan of the CPU list. There are memory order and data
structure visibility issues to be overcome.

- There are areas where the scheduler could use incremental improvement: it
does not know about NUMA, does periodic scans of the CPU list, etc.

Clearly this does not cover stuff like scalability of device drivers,
networking, vm system and whatnot, does not discuss virtualization /
partitioning, it's just the low-level issues that I am aware of given the
current set of stuff that we have in the kernel.

Contact List of Neurologists and much more

2009-02-24T13:26:03Z

Currently in Practice: Medical Doctors in the United States

788,952 in total <> 17,717 emails

Medical Doctor in over 34 specialties

Sort by over a dozen different fields

This week's special price = $395

{}{}{} If you order by the end of the week you can take all the items below for fr ee {}{}{}

+ Contact List of American Pharma Companies
47,000 personal emails and names of decision makers

+ Listing of US Hospitals
23,000 Admins in more than 7,000 hospitals {a $399 value]

+ Listing of US Dentists
More than half a million listings [worth $499 alone!]

+ American Chiropractors Contact List
Over than 100k chiropractors practicing in America

Email us at: Molina@...

valid until February 28

this is the link which will purge you from any futures please send an email to xyz@...

Physician List

2008-06-29T02:09:40Z

Here's what we're offering for this week:

Certified Physicians in the USA

788,465 in total * 17,638 emails

Coverage in many different areas of medicine such as Endocrinology, Pathology, Urology, Neurology, Plastic Surgery, Psychiatry, Cardiology and much more

Over a dozen sortable fields

List of American Pharma Companies
47,000 personal emails and names of decision makers

Hospital Facilities in the USA
Full data for all the major positions in more than 7k facilities

American Dentists
597,000 dentists and dental services ( a $350 value!)

Chiropractors in the USA
100,000 Chiropractors in the USA (worth $250 alone)

This week only you pay only:
$392 for all above data

send email to: richard.m.harris_md@...

this offer is only valid until July 05 2008

Uma frase e uma sugestao

2008-06-11T21:14:05Z

,

Saudaçoes. Tenho certeza que conhece a frase que diz: "Propaganda e a alma do
negocio". Por isso estou escrevendo para te lembrar da grande utilidade do email
marketing para a divulgaçao do seu negocio.

Os estao a sua disposiçao, mas estao tambem a disposiçao dos seus concorrentes.
Por isso conquiste primeiro este mercado antes que seja conquistado por outras
pessoas. Que tal iniciar a sua campanha esta semana?

Cordialmente,
Paula Grassi
http://www.divulgaemails.com
(0xx71)3491-9005 ou (0xx71)9932-0158(24h)
MSN e SKYPE: dvgmail

adicionar áudio Web Audio Plus

2008-05-21T15:53:45Z

adicionar áudio Web Audio Plus, som em flash Web Audio Plus, voz música áudio
marketing Web Audio Plus, mensagens de voz como colocar audio em sites, adicionar
áudio inserir som, Narração para sites Web Audio Plus, adicionar áudio Web Audio
Plus, site com som voz no site.

Visite:
http://www.audiomarketing10.com

inserir som Web Audio Plus, mensagens de voz Web Audio Plus, software para inserir
inserir som, narração fala como colocar audio em sites, adicionar áudio Áudio
em sites, narração fala Web Audio Plus, audio no site Narração para sites. narração
de audio em sites som em flash

Visite:
http://www.audiomarketing10.com

narração de audio em sites som em flash som em flash Web Audio Plus, voz música
áudio marketing Web Audio Plus, mensagens de voz como colocar audio em sites,
adicionar áudio inserir som, Narração para sites Web Audio Plus, adicionar áudio
Web Audio Plus, site com som voz no site. inserir som Web Audio Plus, mensagens
de voz Web Audio Plus, software para inserir inserir som, narração fala como
colocar audio em sites, adicionar áudio Áudio em sites, narração fala Web Audio
Plus, audio no site Narração para sites.

new xemacs segfault, can anyone reproduce this?

2008-01-13T12:21:15Z

Hi,

Since about 2-3 monthes, my xemacs is segfaulting from times to times
under -current SMP. Should I say this never (ever) happened before and I
did not change xemacs version, etc. (although everything has been
recompiled with current userland).

Today I finally isolated some reproducible behaviour, which disapear when
I put all but one CPU offline (thus my posting in this list).
I would like to know if some of you can also reproduce this problem (you
need to 'make extract' in pkgsrc/editor/xemacs):

ficus[~]# uname -a
NetBSD ficus 4.99.49 NetBSD 4.99.49 (FICUS) #96: Sun Jan 13 17:58:31 CET 2008 troot@ficus:/usr/obj/sys/arch/i386/compile/FICUS i386

ficus[~]# cpuctl list
ID Unbound LWPs Interrupts Last change
---- ------------ -------------- ----------------------------
0 online intr Sun Jan 13 19:50:51 2008
3 online intr Sun Jan 13 21:01:23 2008
2 online intr Sun Jan 13 21:01:24 2008
1 online intr Sun Jan 13 21:01:25 2008

And as a regular user with X display:
ficus[~] > xemacs -vanilla -eval '(setq progress-feedback-use-echo-area t)' -eval '(font-lock-mode)' /usr/pkgsrc/editors/xemacs/work/xemacs-21.4.17/src/extents.c

xemacs should segfault 9 times out of 10, and at least exhibit wrong
keyword decoration.

With 3 CPU offline the problem disapears:
ficus[~]# cpuctl list
ID Unbound LWPs Interrupts Last change
---- ------------ -------------- ----------------------------
0 online intr Sun Jan 13 19:50:51 2008
3 offline intr Sun Jan 13 21:11:42 2008
2 offline intr Sun Jan 13 21:11:43 2008
1 offline intr Sun Jan 13 21:11:44 2008

Since xemacs is single-threaded and this used to work fine, I would not
blame xemacs too fast. And I also tested with xemacs-21.4.21 and
xemacs-21.5.27, same thing ...

Do you think this can be a problem in the current kernel?

--
Cheers,
Anthony

cpu1: failed to become ready

2007-10-27T11:26:21Z

Problem:
cpu1 at mainbus0: apid 6 (application processor)
cpu1: starting
cpu1: failed to become ready

The system has 2 Xeon 2.66Mhz with HT. Randomly one cpu or another might
fail become ready between reboots. Anyone had found this issue before?

NetBSD moon 4.0_RC3 NetBSD 4.0_RC3 (GENERIC.MP) #0: Tue Oct 16 01:41:36 PDT 2007 builds@wb34:/home/builds/ab/netbsd-4-0-RC3/i386/200710160011Z-obj/home/builds/ab/netbsd-4-0-RC3/src/sys/arch/i386/compile/GENERIC.MP i386

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.

NetBSD 4.0_RC3 (GENERIC.MP) #0: Tue Oct 16 01:41:36 PDT 2007
builds@wb34:/home/builds/ab/netbsd-4-0-RC3/i386/200710160011Z-obj/home/builds/ab/netbsd-4-0-RC3/src/sys/arch/i386/compile/GENERIC.MP
total memory = 3071 MB
rbus: rbus_min_start set to 0xc0000000
avail memory = 3005 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
BIOS32 rev. 0 found at 0xfd6e0
mainbus0 (root)
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Xeon (686-class), 2657.98 MHz, id 0xf25
cpu0: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu0: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu0: features2 4400<CID,xTPR>
cpu0: "Intel(R) Xeon(TM) CPU 2.66GHz"
cpu0: I-cache 12K uOp cache 8-way, D-cache 8 KB 64B/line 4-way
cpu0: L2 cache 512 KB 64B/line 8-way
cpu0: ITLB 4K/4M: 64 entries
cpu0: DTLB 4K/4M: 64 entries
cpu0: enabling thermal monitor 1 ... enabled.
cpu0: calibrating local timer
cpu0: apic clock running at 132 MHz
cpu0: 16 page colors
cpu1 at mainbus0: apid 6 (application processor)
cpu1: starting
cpu1: failed to become ready
cpu2 at mainbus0: apid 1 (application processor)
cpu2: starting
cpu2: Intel Xeon (686-class), 2657.82 MHz, id 0xf25
cpu2: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu2: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu2: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu2: features2 4400<CID,xTPR>
cpu2: "Intel(R) Xeon(TM) CPU 2.66GHz"
cpu2: I-cache 12K uOp cache 8-way, D-cache 8 KB 64B/line 4-way
cpu2: L2 cache 512 KB 64B/line 8-way
cpu2: ITLB 4K/4M: 64 entries
cpu2: DTLB 4K/4M: 64 entries
cpu2: using thermal monitor 1
cpu3 at mainbus0: apid 7 (application processor)
cpu3: starting
cpu3: Intel Xeon (686-class), 2657.83 MHz, id 0xf25
cpu3: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu3: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu3: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu3: features2 4400<CID,xTPR>
cpu3: "Intel(R) Xeon(TM) CPU 2.66GHz"
cpu3: I-cache 12K uOp cache 8-way, D-cache 8 KB 64B/line 4-way
cpu3: L2 cache 512 KB 64B/line 8-way
cpu3: ITLB 4K/4M: 64 entries
cpu3: DTLB 4K/4M: 64 entries
cpu3: using thermal monitor 1
ioapic0 at mainbus0 apid 2 (I/O APIC)
ioapic0: pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0 apid 3 (I/O APIC)
ioapic1: pa 0xfec80000, version 20, 24 pins
ioapic2 at mainbus0 apid 4 (I/O APIC)
ioapic2: pa 0xfec80400, version 20, 24 pins
acpi0 at mainbus0: Advanced Configuration and Power Interface
acpi0: using Intel ACPI CA subsystem version 20060217
acpi0: X/RSDT: OemId <PTLTD , RSDT ,06040000>, AslId < LTP,00000000>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
ACPI-Fast 24-bit timer
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
PNP0A03 [PCI/PCI-X Host Bridge] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
INT0800 [Intel FWH Random Number Generator] at acpi0 not configured
PNP0C02 [Plug and Play motherboard register resources] at acpi0 not configured
PNP0200 [AT DMA Controller] at acpi0 not configured
npx1 at acpi0 (PNP0C04)
npx1: io 0xf0-0xfe irq 13
npx1: reported by CPUID; using exception 16
PNP0000 [AT Interrupt Controller] at acpi0 not configured
PNP0B00 [AT Real-Time Clock] at acpi0 not configured
pcppi1 at acpi0 (PNP0800)
pcppi1: io 0x61
pcppi1: children must have an explicit unit
midi0 at pcppi1: PC speaker (CPU-intensive output)
sysbeep0 at pcppi1
attimer1 at acpi0 (PNP0100): AT Timer
attimer1: io 0x40-0x43 irq 0
PNP0A06 [Generic Container Device] at acpi0 not configured
pckbc1 at acpi0 (PNP0303): kbd port
pckbc1: io 0x60,0x64 irq 1
com3 at acpi0 (PNP0501-1)
com3: io 0x3f8-0x3ff irq 4
com3: ns16550a, working fifo
com3: console
com4 at acpi0 (PNP0501-2)
com4: io 0x2f8-0x2ff irq 3
com4: ns16550a, working fifo
acpibut0 at acpi0 (PNP0C0C): ACPI Power Button
pcppi1: attached to attimer1
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel E7501 MCH Host (rev. 0x01)
Intel E7500 MCH DRAM Controller (undefined subclass 0x00, revision 0x01) at pci0 dev 0 function 1 not configured
ppb0 at pci0 dev 2 function 0: Intel E7500 MCH HI_B vppb 1 (rev. 0x01)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci1 dev 28 function 0 not configured
ppb1 at pci1 dev 29 function 0: Intel 82870P2 P64H2 PCI-PCI Bridge (rev. 0x04)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci1 dev 30 function 0 not configured
ppb2 at pci1 dev 31 function 0: Intel 82870P2 P64H2 PCI-PCI Bridge (rev. 0x04)
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled
twe0 at pci3 dev 1 function 0: 3ware Escalade
twe0: interrupting at ioapic2 pin 0 (irq 10)
twe0: 4 ports, Firmware FE7X 1.05.00.056, BIOS BE7X 1.08.00.046
twe0: Monitor ME7X 1.01.00.038, PCB Rev4 , Achip 3.20 , Pchip 1.30-66
twe0: port 0: Maxtor 6Y200P0 194481 MB
twe0: port 1: Maxtor 6Y200P0 194481 MB
twe0: port 2: Maxtor 6Y200P0 194481 MB
twe0: port 3: Maxtor 6Y200P0 194481 MB
ld0 at twe0 unit 0: TwinStor, status: Normal
ld0: 189 GB, 24792 cyl, 255 head, 63 sec, 512 bytes/sect x 398295040 sectors
ld1 at twe0 unit 2: TwinStor, status: Normal
ld1: 189 GB, 24792 cyl, 255 head, 63 sec, 512 bytes/sect x 398295040 sectors
Intel E7500 MCH HI_B vppb 2 (undefined subclass 0x00, revision 0x01) at pci0 dev 2 function 1 not configured
uhci0 at pci0 dev 29 function 0: Intel 82801CA USB Controller (rev. 0x02)
uhci0: interrupting at ioapic0 pin 16 (irq 10)
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1: Intel 82801CA USB Controller (rev. 0x02)
uhci1: interrupting at ioapic0 pin 19 (irq 5)
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2 at pci0 dev 29 function 2: Intel 82801CA USB Controller (rev. 0x02)
uhci2: interrupting at ioapic0 pin 18 (irq 10)
usb2 at uhci2: USB revision 1.0
uhub2 at usb2
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ppb3 at pci0 dev 30 function 0: Intel 82801BA Hub-PCI Bridge (rev. 0x42)
pci4 at ppb3 bus 4
pci4: i/o space, memory space enabled
vga1 at pci4 dev 3 function 0: ATI Technologies Rage XL (rev. 0x27)
wsdisplay0 at vga1 kbdmux 1
wsmux1: connecting to wsdisplay0
fxp0 at pci4 dev 4 function 0: i82550 Ethernet, rev 13
fxp0: interrupting at ioapic0 pin 20 (irq 10)
fxp0: Ethernet address 00:02:b3:d8:b8:9c
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
wm0 at pci4 dev 5 function 0: Intel i82540EM 1000BASE-T Ethernet, rev. 2
wm0: interrupting at ioapic0 pin 23 (irq 11)
wm0: 32-bit 33MHz PCI bus
wm0: 64 word (6 address bits) MicroWire EEPROM
wm0: Ethernet address 00:02:b3:d8:b9:1d
makphy0 at wm0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 3
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
pcib0 at pci0 dev 31 function 0
pcib0: Intel 82801CA LPC Interface (rev. 0x02)
piixide0 at pci0 dev 31 function 1
piixide0: Intel 82801CA IDE Controller (ICH3) (rev. 0x02)
piixide0: bus-master DMA support present
piixide0: primary channel configured to compatibility mode
piixide0: primary channel interrupting at ioapic0 pin 14 (irq 14)
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to compatibility mode
piixide0: secondary channel interrupting at ioapic0 pin 15 (irq 15)
atabus1 at piixide0 channel 1
Intel 82801CA SMBus Controller (SMBus serial bus, revision 0x02) at pci0 dev 31 function 3 not configured
isa0 at pcib0
isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
isapnp0: no ISA Plug 'n Play devices found
ioapic0: enabling
ioapic1: enabling
ioapic2: enabling
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
Kernelized RAIDframe activated
boot device: ld0
root on ld0a dumps on ld0b
root file system type: ffs
cpu2: CPU 1 running
cpu3: CPU 7 running
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)

--
-m

==========================================================================
pub 2048R/D149C1B5 2006-05-08
Key fingerprint = 4DD5 DEBA 28BF FDDA 09EC 3C6B 8504 4FB7 D149 C1B5
uid Marcelo R. Schmidt (SavvyByte LLC) <mschmidt@...>
gpg --keyserver pgp.mit.edu --recv-keys 0xD149C1B5
==========================================================================

attachment0 (489 bytes) Download Attachment

Our price $129 Autodesk Autocad 2008

2007-10-07T08:59:20Z

save -save $6590.05 our price $129.95 autocad 2008
http://www.makosh.cn

obtain a macro body part Humberto

2007-10-07T06:50:55Z

largish phallus for Lilian
http://rebanell.com

softintr running on boot CPU only?

2007-10-03T13:17:36Z

I am running 3.0.1 kernel on Dell SC430(two pentium cores) and run
vmstats saw softinterrupt only on boot CPU. Is it the way that the 3.0.1
kernel code supported to do? How about 4.0 and upcoming 5.0?

There is really not much documents on SMP/APCI implementation on x86
besides a few email threads. Does someone know more can share with me?

Thanks
Ming

Looking for help with high volume MySQL

2007-08-17T11:48:10Z

Hi, all,

I'm looking for help with setting up NetBSD machines in a MySQL cluster. I
have lots of questions, such as:

Is SMP efficient enough with NetBSD that a single 4, 6, or 8 processor
system would be as capable or perhaps more capable than, say, 2, 3, or 4
separate dual processor systems?

Why, on an amd64 system with 4 gigs of memory and MySQL ulimited as such:

ulimit -n 4096
ulimit -d 1572864
ulimit -n 600
ulimit -s 32768

do I sometimes get out of memory errors from MySQL? The total size of
MySQL seems to grow to about 1 gig, but never any larger, and I've had to
lower memory usage in my.cnf.

Why is my.cnf in /usr/local/ and not in /usr/local/etc/?

Does anyone have recommendations for setting up an arbitrary number of
replication slaves easily? Experience and suggestions? What's the best way
to benchmark two machines against each other? AMD or Intel?

Any suggestions or help would be most welcome.

Thanks,
John Klos
--
The good Christian should beware of mathematicians and all those who
make empty prophecies. The danger already exists that mathematicians
have made a covenant with the devil to darken the spirit and confine
man in the bonds of Hell.
-- St. Augustine

Re: MP safe syscall changes

2007-02-05T15:29:54Z

Andrew Doran <ad@...> writes:
> I just made a bunch more syscalls MP safe on the newlock2 branch, and ran a
> quick test on this machine: 4 x 700MHz P-III Xeon, 1MB L2 cache per CPU, 2GB
> RAM. Cleaning out a clean source tree with make -j16 cleandir:

I'm happy (and surprised) that there is a speedup here. Consider that
this is not an embarrassingly parallel task: removing lots of files is
ultimately bottlenecked on the disk and not something where we would
expect much parallel speedup.

> real system
>
> MP -9% -14%
> UP -0.5% +3.5%

I would be quite interested in seeing figures for building the world
in both the current MP and newlock2 MP cases. *That* might have some
interesting opportunities for parallelism on a machine with 2G of
ram. Do knock up the number of vnodes to around 2^17 or 2^18th before
starting, though, or you might end up a bit i/o bound with that much
parallelism.

Perry

MP safe syscall changes

2007-02-05T08:14:18Z

Hi,

I just made a bunch more syscalls MP safe on the newlock2 branch, and ran a
quick test on this machine: 4 x 700MHz P-III Xeon, 1MB L2 cache per CPU, 2GB
RAM. Cleaning out a clean source tree with make -j16 cleandir:

real user system

MP, newlock2 103.55s 174.51s 246.26s
MP, HEAD 114.21s 175.46s 287.01s
UP, newlock2 198.34s 152.67s 61.30s
UP, HEAD 199.18s 150.60s 59.20s

The results aren't astounding and are not particularly accurate (system and
user time are subject to sampling error), but the indicated difference
relative to HEAD is:

real system

MP -9% -14%
UP -0.5% +3.5%

At a guess, part of the additional two seconds system time in the UP case is
a result of all the jumping through hoops that needs to be done to support
both locking against interrupt handlers and LWPs. With interrupts as threads
that cost can be eliminated..

Cheers,
Andrew

Video Aulas, cursos em vídeo. Aprenda a fazer tudo com video cursos

2007-01-20T06:20:31Z

Video Aulas para download, cursos em vídeo. Vários assuntos. Aprenda a
fazer sushi, curso de Yoga em vído, video aula de sedução, video curso de
massagem sensual:

http://www.gueb.de/videocursosbrasil

Video cursos de informática, ginástica, artes marciais. Tudo em cursos
para download. Vídeos com entrega imediata. Aulas práticas e
demonstrativas. Plataforma para e learning. Ensino a distancia.
Treinamentos, cursos, congressos, seminarios, video digital com
transmissão ao vivo, aulas.

http://www.gueb.de/videocursosbrasil

Comercializa vídeos de aulas de música, incluindo guitarra, baixo,
bateria, flauta, gaita, pandeiro, canto, dança e harmonia. Encontre no
nosso site cursos de Várias áreas para sua escolha e aprimoramento:

http://www.gueb.de/videocursosbrasil

Re: lock debugging (was Re: Just for fun)

2007-01-16T14:07:16Z

On Tue, Jan 16, 2007 at 07:53:51AM -0500, Perry E. Metzger wrote:
>
> Andrew Doran <ad@...> writes:
> > The branch adds one more piece of the puzzle (locking around process state).
> > That it works comparatively well is a product of all of the locks that have
> > been trickling into the system for the last .. well, since 4.4BSD I think.
>
> What if we did something similar but instead taught the emulator about
> locks and traced unlocked touches of global data structures?

Coverity Prevent also has a set of concurrency checkers that can
identify issues with locking.

Those checkers haven't been turned on yet on scan.coverity.com, but we
are working towards making the additional results available. I just
wanted to let you know that you'll have additional information to call
on when that happens.

--
David Maxwell
Coverity Open Source Strategist
http://scan.coverity.com/

Re: lock debugging

2007-01-16T09:20:24Z

Pavel Cahyna <pavel@...> writes:

> On Tue, Jan 16, 2007 at 07:53:51AM -0500, Perry E. Metzger wrote:
>>
>> Andrew Doran <ad@...> writes:
>> > The branch adds one more piece of the puzzle (locking around
>> > process state). That it works comparatively well is a product of
>> > all of the locks that have been trickling into the system for the
>> > last .. well, since 4.4BSD I think.
>>
>> BTW, I wonder whether we can modify an x86 emulator into a lock
>> debugger.
>
> Yes: http://www.valgrind.org/info/tools.html#helgrind

Can that be used for running a kernel?

Perry
--
Perry E. Metzger perry@...

Re: lock debugging (was Re: Just for fun)

2007-01-16T09:13:58Z

lock debugging (was Re: Just for fun)

2007-01-16T04:53:51Z

Andrew Doran <ad@...> writes:
> The branch adds one more piece of the puzzle (locking around process state).
> That it works comparatively well is a product of all of the locks that have
> been trickling into the system for the last .. well, since 4.4BSD I think.

BTW, I wonder whether we can modify an x86 emulator into a lock
debugger.

Some years ago I read a paper about a system called "taintbochs" that
modified the Bochs emulator to track the flow of password and other
sensitive data inside a kernel.

What if we did something similar but instead taught the emulator about
locks and traced unlocked touches of global data structures?

Perry

Re: Just for fun

2007-01-16T02:56:54Z

On Mon, Jan 15, 2007 at 11:44:21AM -0500, Perry E. Metzger wrote:

> Andrew Doran <ad@...> writes:
> > A few years back pk@ locked a few more kernel subsystems and tried running
> > the kernel unlocked to see what would happen. I repeated the test with a
> > kernel from the newlock2 branch and to my amazement it completes a kernel
> > build on a 4 way system.
> >
> > build.sh isn't so good - once it hits 'make objdir' and there's a lot of
> > intensive file system activity going it will either panic, or lock up in
> > biowait -> vnlock, somewhere around 30 to 60 seconds in.
>
> That's pretty amazing. Should we attribute this to the low probability
> of contention in the real world,

The kernel build doesn't spend a huge amount of time in the kernel, but it
definitely produces contention. I think the majority of it is likely to
result from handling traps for zero fill and pageins.

> or to the quality of the stuff on newlock2? (I suspect a bit of both,
> which is, as you say, encouraging.)

The branch adds one more piece of the puzzle (locking around process state).
That it works comparatively well is a product of all of the locks that have
been trickling into the system for the last .. well, since 4.4BSD I think.

Andrew

Re: Just for fun

2007-01-15T08:44:21Z

Andrew Doran <ad@...> writes:
> A few years back pk@ locked a few more kernel subsystems and tried running
> the kernel unlocked to see what would happen. I repeated the test with a
> kernel from the newlock2 branch and to my amazement it completes a kernel
> build on a 4 way system.
>
> build.sh isn't so good - once it hits 'make objdir' and there's a lot of
> intensive file system activity going it will either panic, or lock up in
> biowait -> vnlock, somewhere around 30 to 60 seconds in.

That's pretty amazing. Should we attribute this to the low probability
of contention in the real world, or to the quality of the stuff on
newlock2? (I suspect a bit of both, which is, as you say, encouraging.)

> Still, I think it's pretty encouraging that things work as well as they do.
> I put dmesg and some lockstat output from a couple of runs at the URL below.
> Note that the timings for the sleep locks are wrong.
>
> http://www.netbsd.org/~ad/biglock/

Perry

Just for fun

2007-01-14T11:32:08Z

Hi,

A few years back pk@ locked a few more kernel subsystems and tried running
the kernel unlocked to see what would happen. I repeated the test with a
kernel from the newlock2 branch and to my amazement it completes a kernel
build on a 4 way system.

build.sh isn't so good - once it hits 'make objdir' and there's a lot of
intensive file system activity going it will either panic, or lock up in
biowait -> vnlock, somewhere around 30 to 60 seconds in.

Still, I think it's pretty encouraging that things work as well as they do.
I put dmesg and some lockstat output from a couple of runs at the URL below.
Note that the timings for the sleep locks are wrong.

http://www.netbsd.org/~ad/biglock/

Cheers,
Andrew

Re: Interrupts as threads

2007-01-14T11:16:07Z

Andrew Doran <ad@...> writes:
> The problem as I see it is: getting it right all at once would be a big
> effort, and I don't see that we have the resources to do it that way.

For what it is worth, I think an implementation path that allows
incremental steps that each are worthwhile (more or less what I see
you saying) is better than one that requires a massive application of
resources we don't have.

As for the particular decisions made, I have a bias towards letting
the person actually doing work have the biggest say unless they're
acting crazy, and you aren't acting crazy. You are clearly leading
this right now by doing the actual programming.

Perry

Re: Interrupts as threads

2007-01-12T06:38:54Z

Hi Bill,

On Tue, Dec 19, 2006 at 04:05:24PM -0800, Bill Studenmund wrote:

> On Sun, Dec 03, 2006 at 10:36:30PM +0000, Andrew Doran wrote:

> > The problem is that at the moment, acquiring any given spin lock is a
> > deadlock waiting to happen unless you are at ~IPL_SCHED or above when
> > acquiring it, or are certain that it will only ever be acquired with the
> > kernel lock unheld. Ensuring that the kernel lock is always held when you
> > acquire the spin lock means that the spin lock is useless :-).
>
> I've been thinking about this, and I think you are not correct. Well. In
> the long-term, you are. But I think as a transition step, we may have to
> accept it.

I'm not sure I follow. We did discuss this briefly and I think that you were
proposing a scheme where we have the indivdual interrupt handlers acquire
the kernel lock, although I might have completely misunderstood. :-)

> All we have to do is define a correct locking hierarcy. It's ok to aquire
> a given spin lock if you have the kernel lock. It's ok to aquire said
> spinlock w/o the kernel lock. It's just NOT ok to aquire the kernel lock
> while holding said lock. Yes, this could make interupt handling routines
> painful (when they hand things off to the main kernel), but we can do it.

The problem as I see it is: getting it right all at once would be a big
effort, and I don't see that we have the resources to do it that way.

I said that I wanted to use interrupts as threads, and process context locks
(the Solaris style mutexes) for MP safety. The two basic problems I want to
solve by doing that are:

o The one mentioned above - if the locks are difficult to use (potential
deadlocks against the kernel_lock) then that's a problem. For places
where we want MP safety (not MT safety) the mutexes can be thought of as
spin locks, but with additional property that they can block in order to
avoid deadlock. That might be against the current CPU already holding the
mutex (preemption), or against the CPU that already holds the mutex
wanting the big lock. Essentially, they remove the ordering constraint
against the big lock.

o If we keep the distinction between interrupt and process context across
the board, then we're likely to increase the number of locks in the in
kernel: one set for interrupt access, and one for process access. I've
spent a lot of time trying to make process and LWP state MP safe for
signalling. Even though it works well enough I'm not particularly happy
with the end result, because there's a mix of locks where we only need
~half the number. It's not good to use spinlocks with a raised SPL in a
lot of places, because we might need to block briefly on a process context
lock, or in the overall picture we end up holding interrupts for much
longer.

I did some simple profiling on the SPL operations and here is example output
from a run that I did. It's from a single CPU machine serving up files at
line rate using 8 ftpds over 100Mbps Ethernet, and doing some disk I/O
locally. I wasn't looking for any specific kind of behaviour other than
'doing I/O'. The machine is set up so that there is 1:1 mapping between each
symbolic interrupt level and each hardware IRQ line.

1 2 3 4 5

alevel | intrs persec | blkself persec | blkother persec | splraise persec
---------+-------------------+-------------------+-------------------+--------------------
softnet | 274094 6334 | 47549 1098 | 0 0 | 572849 13239
bio | 17939 414 | 252 5 | 0 0 | 1415581 32717
net | 373121 8623 | 29857 690 | 1760 40 | 631179 14588
tty | 0 0 | 0 0 | 3 0 | 428 9
vm | 0 0 | 0 0 | 11646 269 | 5211031 120438
audio | 0 0 | 0 0 | 0 0 | 0 0
clock | 4328 100 | 353 8 | 33913 783 | 1156023 26718
high | 0 0 | 0 0 | 112 2 | 291410 6735
ipi | 0 0 | 0 0 | 8 0 | 435 10

The columns are:

1. Interrupt level.
2. Number of interrupts at that level.
3. Number of splfoo() operations that blocked an interrupt at level foo.
This doesn't include the SPL adjustment that MD code does as part of
taking the interrupt.
4. Number of splfoo() operations that blocked an interrupt below level
foo. Note that this does not count blocked soft interrupts.
5. Number of splfoo() operations.

Given the amount of contention that this _one_ test shows it's clear that we
can't just use interrupts as threads and replace the SPL system with locks,
or we will really suffer from all the additional context switching. At
least, we can't do that without a major engineering effort to change how
work is passed from one level to the next.

I think that we can also cut the number of spl calls significantly. For one,
the number of splvm() calls indicated above seems a bit excessive.

What I propose is using interrupts as threads for IPL_VM and below, but also
preserving the SPL system. We would certianly need to use it for networking.
In other areas (like block I/O) where there aren't so many chokepoints I'm
inclined to believe that we can get away without it.

I don't think interrupts as threads have to be expensive: typically just a
stack switch and some additional accounting in the interrupt path. In
addition to the interrupt coming in, there would need to be defined
premption points, like lock release or splx(). That's in contrast to (to
use an example that people have cited) FreeBSD, where the initial approach
was a lot more costly as I understand it: in addition to taking the
interrupt, a thread had to be picked and scheduled to run at a later time,
for example on return to user space. I don't know how their system works
these days.

Thoughts?

Cheers,
Andrew

Re: Interrupts as threads

2006-12-19T16:05:24Z

On Sun, Dec 03, 2006 at 10:36:30PM +0000, Andrew Doran wrote:

> On Sat, Dec 02, 2006 at 10:25:42AM +0000, David Laight wrote:
>
> > On Fri, Dec 01, 2006 at 11:31:19PM +0000, Andrew Doran wrote:
> > >
> > > o There is no easy solution to the lock order problem with the kernel_lock
> > > when using spin locks.
> >
> > My brief peruse at the biglock code did make it clear how it worked when
> > an IRQ came in when the 2nd cpu had the biglock...
>
> The problem is that at the moment, acquiring any given spin lock is a
> deadlock waiting to happen unless you are at ~IPL_SCHED or above when
> acquiring it, or are certain that it will only ever be acquired with the
> kernel lock unheld. Ensuring that the kernel lock is always held when you
> acquire the spin lock means that the spin lock is useless :-).

I've been thinking about this, and I think you are not correct. Well. In
the long-term, you are. But I think as a transition step, we may have to
accept it.

All we have to do is define a correct locking hierarcy. It's ok to aquire
a given spin lock if you have the kernel lock. It's ok to aquire said
spinlock w/o the kernel lock. It's just NOT ok to aquire the kernel lock
while holding said lock. Yes, this could make interupt handling routines
painful (when they hand things off to the main kernel), but we can do it.

Take care,

Bill

attachment0 (193 bytes) Download Attachment

Re: Interrupts as threads

2006-12-03T14:36:30Z

On Sat, Dec 02, 2006 at 10:25:42AM +0000, David Laight wrote:

> On Fri, Dec 01, 2006 at 11:31:19PM +0000, Andrew Doran wrote:
> >
> > o There is no easy solution to the lock order problem with the kernel_lock
> > when using spin locks.
>
> My brief peruse at the biglock code did make it clear how it worked when
> an IRQ came in when the 2nd cpu had the biglock...

The problem is that at the moment, acquiring any given spin lock is a
deadlock waiting to happen unless you are at ~IPL_SCHED or above when
acquiring it, or are certain that it will only ever be acquired with the
kernel lock unheld. Ensuring that the kernel lock is always held when you
acquire the spin lock means that the spin lock is useless :-).

It would not be a major undertaking to push that down to IPL_VM by making
other interrupt handlers MP safe, the bulk of which are audio drivers I
think. Once you move beyond those drivers, things start to get more
difficult and I'm not convinced that we have the resources to tackle the
issue that way.

> > o Using spin locks we will have to keep the SPL above IPL_NONE for longer
> > that before, or accept (in non-trivial cases) the undesirable cost of
> > having both interrupt and process context locks around some objects.
>
> SMP code does tend to need mutex protection for longer than the non-SMP
> code required IPL protection anyway...

Yup.

> > o Raising and lowering the SPL is expensive, especially on machines that
> > need to talk with the hardware on SPL operation. The spin lock path also
> > has more test+branch pairs / conditional moves and memory references
> > involved than process locks. For a process context lock, the minimum we
> > can get away with on entry and exit is one test+branch and two cache line
> > references.
>
> Can we do deferred SPL changes on all archs?
> ie don't frob the hardware, but assume it won't raise an IRQ. If an IRQ
> does occur, mask it then and return, re-enabling of the splx call.

We probably could, but it's still expensive to acquire spin locks that
modify the SPL. Have a look at arch/i386/i386/lock_stubs.S on the newlock2
branch and compare the paths for spin vs. adaptive mutexes. (I know that at
least _lock_set_waiters() is broken. Also, I haven't optimised x86 assembly
in a long time - please don't laugh. ;-)

> > o Every spin lock / unlock pair denotes a critical section where threads
> > running in the kernel can not be preempted. That's not currently an issue
> > but if we move to support real time threads it could become one; I'm not
> > sure.
>
> Once we have a proper SMP kernel, making processes pre-emptable in the kernel
> (while not holding a lock) becomes ~free - whereas the non-SMP kernel code
> will make assumptions that stop pre-empting.
> However you probably want to be able to disable pre-emption whithout
> holding a mutex.

Yup, agreed.

> > The cleanest way to deal with these issues that I can see is to use
> > lightweight threads to handle interrupts
>
> That just makes it worse! Every hardware ISR would have to disable the
> IRQ itself, then the 'low level' ISR would need to re-enable it.

I don't see why we would have to do that. The way I envisoned this working,
for a typical interrupt path that doesn't have to block on any locks, the
prologue and epliogue for the interrupt itself (not the individual ISRs)
would have a few additional steps. Something like:

entry: pick an idle interrupt LWP from curcpu
note which LWP we have preempted
set curlwp
switch onto new stack

exit: restore curlwp
switch back to old stack

If one of the ISRs needs to block, then we would have to switch back to the
LWP that was preempted and hold the SPL at the ISR's base level until the
ISR receives the lock. So in effect, things like splx() would become a
request to alter the SPL, but wouldn't be able to drop it below the base
level for the CPU (which would typically be IPL_NONE).

> > My initial thought is to have one
> > thread per level, per CPU. These would be able to preempt already running
> > threads, and would hold preempted threads in-situ until the interrupt thread
> > returns or switches away. In most cases, SPL operations would be replaced by
> > locks.
>
> You'd have to look very closely at priority inversion problems....

True.. The turnstile mechanism would handle this for locks, but that's
currently implemented.

> > Blocking would no longer be prohibited, but strongly discouraged - so
> > doing something like pool_get(foo, PR_WAITOK) should likely trigger an
> > assertion.
>
> If you allow blocking, then people will use it because it 'appears to work'
> then you find that one of your ISR threads is busy

I considered that but I feel that it's really a documentation and code
review issue. There are plenty of places we can put assertions though. For
instance around memory allocation, or direct use of the tsleep() / cv_wait()
interfaces. In any case there are plenty of inadvisable things that you can
do from an interrupt handler already, and we have drivers that already do (I
wrote one of them :-).

> > Assuming you subscribe to handling interrupts with threads, it raises the
> > question: where to draw the line between threaded and 'traditional'. It
> > certianly makes sense to run soft interrupts this way, and I would draw the
> > line at higher priority ISRs like network, audio, serial and clock.
>
> It may make sense to use kernel threads (running through the scheduler)
> for some driver activity, and quite probably some of the code scheduled
> via 'softint' is in this category.

Sure, but there is a cutoff point: that introduces the full overhead of the
scheduler in addition to taking the interrupt. If the softint handler is an
interrupt thread, then that doesn't happen unless absolutely necessary.

Thanks,
Andrew

Re: Interrupts as threads

2006-12-02T02:25:42Z

On Fri, Dec 01, 2006 at 11:31:19PM +0000, Andrew Doran wrote:
>
> o There is no easy solution to the lock order problem with the kernel_lock
> when using spin locks.

My brief peruse at the biglock code did make it clear how it worked when
an IRQ came in when the 2nd cpu had the biglock...

> o Using spin locks we will have to keep the SPL above IPL_NONE for longer
> that before, or accept (in non-trivial cases) the undesirable cost of
> having both interrupt and process context locks around some objects.

SMP code does tend to need mutex protection for longer than the non-SMP
code required IPL protection anyway...

> o Raising and lowering the SPL is expensive, especially on machines that
> need to talk with the hardware on SPL operation. The spin lock path also
> has more test+branch pairs / conditional moves and memory references
> involved than process locks. For a process context lock, the minimum we
> can get away with on entry and exit is one test+branch and two cache line
> references.

Can we do deferred SPL changes on all archs?
ie don't frob the hardware, but assume it won't raise an IRQ. If an IRQ
does occur, mask it then and return, re-enabling of the splx call.

> o Every spin lock / unlock pair denotes a critical section where threads
> running in the kernel can not be preempted. That's not currently an issue
> but if we move to support real time threads it could become one; I'm not
> sure.

Once we have a proper SMP kernel, making processes pre-emptable in the kernel
(while not holding a lock) becomes ~free - whereas the non-SMP kernel code
will make assumptions that stop pre-empting.
However you probably want to be able to disable pre-emption whithout
holding a mutex.

> o We are doing too much work from interrupt context.

possibly true... but the cpu cycles have to be done somewhen, and deferring
to a different context just takes time.

> The cleanest way to deal with these issues that I can see is to use
> lightweight threads to handle interrupts

That just makes it worse! Every hardware ISR would have to disable the
IRQ itself, then the 'low level' ISR would need to re-enable it.

> My initial thought is to have one
> thread per level, per CPU. These would be able to preempt already running
> threads, and would hold preempted threads in-situ until the interrupt thread
> returns or switches away. In most cases, SPL operations would be replaced by
> locks.

You'd have to look very closely at priority inversion problems....

> Blocking would no longer be prohibited, but strongly discouraged - so
> doing something like pool_get(foo, PR_WAITOK) should likely trigger an
> assertion.

If you allow blocking, then people will use it because it 'appears to work'
then you find that one of your ISR threads is busy - not a problem until
several interrupt routines/drivers do it at the same time and you run out
of threads to do the wakeup.

> Assuming you subscribe to handling interrupts with threads, it raises the
> question: where to draw the line between threaded and 'traditional'. It
> certianly makes sense to run soft interrupts this way, and I would draw the
> line at higher priority ISRs like network, audio, serial and clock.

It may make sense to use kernel threads (running through the scheduler)
for some driver activity, and quite probably some of the code scheduled
via 'softint' is in this category. But IMHO this really needs to be code
this isn't really related to 'interrupt processing'.

David

--
David Laight: david@...

Interrupts as threads

2006-12-01T15:31:19Z

Hi,

I have been thinking about the lock ordering problem with the kernel big
lock quite a bit and what it will take to lock the MI kernel down, and have
made some observations.

o There is no easy solution to the lock order problem with the kernel_lock
when using spin locks.

o Using spin locks we will have to keep the SPL above IPL_NONE for longer
that before, or accept (in non-trivial cases) the undesirable cost of
having both interrupt and process context locks around some objects.

o Raising and lowering the SPL is expensive, especially on machines that
need to talk with the hardware on SPL operation. The spin lock path also
has more test+branch pairs / conditional moves and memory references
involved than process locks. For a process context lock, the minimum we
can get away with on entry and exit is one test+branch and two cache line
references.

o Every spin lock / unlock pair denotes a critical section where threads
running in the kernel can not be preempted. That's not currently an issue
but if we move to support real time threads it could become one; I'm not
sure.

o We are doing too much work from interrupt context.

The cleanest way to deal with these issues that I can see is to use
lightweight threads to handle interrupts. My initial thought is to have one
thread per level, per CPU. These would be able to preempt already running
threads, and would hold preempted threads in-situ until the interrupt thread
returns or switches away. In most cases, SPL operations would be replaced by
locks. Blocking would no longer be prohibited, but strongly discouraged - so
doing something like pool_get(foo, PR_WAITOK) should likely trigger an
assertion.

On something like an x86 or MIPS CPU, we wouldn't need to do a full context
switch for interrupts, just switch onto another stack. For things that are
time critical like clock or audio ISRs I think the current scheme of
deferring the interrupts might be better. Although it should not be common,
the delay involved in switching away when trying to acquire a lock seems
undesirable in these cases. (As an aisde, I have been meaning to do some
profiling to see just how often the SPL operations serve their purpose in a
variety of cases but haven't gotten around to it yet.)

Assuming you subscribe to handling interrupts with threads, it raises the
question: where to draw the line between threaded and 'traditional'. It
certianly makes sense to run soft interrupts this way, and I would draw the
line at higher priority ISRs like network, audio, serial and clock.

Thoughts?

Cheers,
Andrew

Re: SMP stability issues

2006-11-22T15:24:29Z

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris Rendle-Short wrote:
> Anyway, thanks to those that helped and offered suggestions. I'll run the machine on one CPU until I can work out something further.

My dual amd box is crashing during a full os build about 1 in 10 times
now, with a kernel made a week ago.

A kernel made 3 months ago, running on another machine with the same
hardware, does not do this.

Worse, the same smp-kernel running on a single CPU box also crashes /
locks up, while a non-smp does not on the same hardware.

- --Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (MingW32)

iD8DBQFFZNwtuzMQWQwZDN0RAoeMAJ9yxYtuHIwfiyEMUp7rso7PeOtGegCfaEvL
3jrF9oCfWUwaH7yeXRTCPnc=
=UWax
-----END PGP SIGNATURE-----

Re: SMP stability issues

2006-11-22T01:07:19Z

Well, I've spent the last week and a bit trying different kernels etc, and nothing seems to be working. I've tried those Via kernel patches, both on with ACPI enabled and disabled in the kernel, but all to no avail.

Long story short, I've hit a brick wall. I think I'm going to have to put it down to an unfortunately-timed hardware failure. The machine was moved around a bit when it was making the transition from Linux to NetBSD, so there could be something there.

Anyway, thanks to those that helped and offered suggestions. I'll run the machine on one CPU until I can work out something further.

Thanks,
Chris.

Re: SMP stability issues

2006-11-12T13:08:38Z

On Sun, 12 Nov 2006 08:55:21 -0800, Byron Servies <bservies@...> wrote:

> Hi,
>
> I'm not saying this is your problem, but a few years ago I had a VP6
> that worked in single CPU mode but not dual with NetBSD 1.6.
> Eventually, after having it lock up periodically, I gave up and ran
> it as a single CPU machine. That lasted a couple of months before
> the CPU0 socket failed entirely and the machine stopped booting at all.
>
> No smoke. No noise. The board/cpu was just plain bad, and there was
> no indication before it let go for good that this was the case.
>
> Byron

I'm hoping it's not something like that. I don't think it is, because it would be a bit of an unfortunate coincidence that the motherboard started to fail at the same time as I switch it from Linux to NetBSD.

Currently running GENERIC.MPDEBUG, hasn't locked up yet.

Chris.

Re: SMP stability issues

2006-11-12T09:33:21Z

On Sun, Nov 12, 2006 at 08:55:21AM -0800, Byron Servies wrote:

I don't think it's a broken board in my case; all the apollo-pro based
dual-PIII motherboard I tried shows this behavior.

--
Manuel Bouyer <bouyer@...>
NetBSD: 26 ans d'experience feront toujours la difference
--

Re: SMP stability issues

2006-11-12T08:55:21Z

Hi,

I'm not saying this is your problem, but a few years ago I had a VP6
that worked in single CPU mode but not dual with NetBSD 1.6.
Eventually, after having it lock up periodically, I gave up and ran
it as a single CPU machine. That lasted a couple of months before
the CPU0 socket failed entirely and the machine stopped booting at all.

No smoke. No noise. The board/cpu was just plain bad, and there was
no indication before it let go for good that this was the case.

Byron

Re: SMP stability issues

2006-11-12T02:45:13Z

On Sun, Nov 12, 2006 at 01:23:34PM +1100, Chris Rendle-Short wrote:
> Well, I just tried running GENERIC.MPACPI like some of the others suggested,
> however it is still locking up. Here is the dmesg from GENERIC.MPACPI
> (although it looks like I might need to check my ACPI configuration in the
> BIOS.

It looks kike it's using ACPI

> I will also try a kernel with DIAGNOSTIC, DEBUG and LOCKDEBUG enabled
> as you suggested. Is it likely to matter whether or not ACPI is enabled in
> the test kernel?

Yes, these checks are independant from ACPI vs MPBIOS

> pchb0 at pci0 dev 0 function 0
> pchb0: VIA Technologies VT82C691 (Apollo Pro) Host-PCI (rev. 0xc4)

OK, this is the same motherboard as I have here (I have several of theses). I
also have issues with them, I guess the debug options
will show you that the CPU is missing IPI interrupts on occasion.
If so, the attached patch should help (my boxes are rock solid with this
patch). Note that it's only active if you have
options DIAGNOSTIC
in your kernel config.
Acutally I suspect this is a bug in the chipset; I have Intel-based dual-PIII
motherboards which don't have this issue, nor do P4 SMP systems.

--
Manuel Bouyer <bouyer@...>
NetBSD: 26 ans d'experience feront toujours la difference
--

Index: i386/pmap.c
===================================================================
RCS file: /cvsroot/src/sys/arch/i386/i386/pmap.c,v
retrieving revision 1.181.2.2
diff -u -r1.181.2.2 pmap.c
--- i386/pmap.c 26 Sep 2005 20:24:52 -0000 1.181.2.2
+++ i386/pmap.c 12 Nov 2006 10:42:15 -0000
@@ -3652,6 +3652,7 @@
int s;
#ifdef DIAGNOSTIC
int count = 0;
+ int ipi_retry = 0;
#endif
#endif

@@ -3672,6 +3673,9 @@
/*
* Send the TLB IPI to other CPUs pending shootdowns.
*/
+#ifdef DIAGNOSTIC
+ipi_again:
+#endif
for (CPU_INFO_FOREACH(cii, ci)) {
if (ci == self)
continue;
@@ -3683,9 +3687,20 @@

while (self->ci_tlb_ipi_mask != 0) {
#ifdef DIAGNOSTIC
- if (count++ > 10000000)
+ if (count++ > 10000000) {
+ for (CPU_INFO_FOREACH(cii, ci)) {
+ if (ci == self)
+ continue;
+ printf("CPU %ld interrupt level 0x%x pending "
+ "0x%x depth %d ci_ipis %d\n", ci->ci_cpuid,
+ ci->ci_ilevel, ci->ci_ipending,
+ ci->ci_idepth, ci->ci_ipis);
+ }
+ if (ipi_retry++ < 5)
+ goto ipi_again;
panic("TLB IPI rendezvous failed (mask %x)",
self->ci_tlb_ipi_mask);
+ }
#endif
x86_pause();
}
Index: isa/npx.c
===================================================================
RCS file: /cvsroot/src/sys/arch/i386/isa/npx.c,v
retrieving revision 1.107.4.1
diff -u -r1.107.4.1 npx.c
--- isa/npx.c 12 May 2006 15:41:46 -0000 1.107.4.1
+++ isa/npx.c 12 Nov 2006 10:42:16 -0000
@@ -752,6 +752,8 @@
} else {
#ifdef DIAGNOSTIC
int spincount;
+ int ipi_retry = 0;
+ipi_again:
#endif

IPRINTF(("%s: fp ipi to %s %s lwp %p\n",
@@ -770,6 +772,16 @@
#ifdef DIAGNOSTIC
spincount++;
if (spincount > 10000000) {
+ printf("CPU %ld interrupt level 0x%x pending "
+ "0x%x depth %d ci_ipis %d\n", ci->ci_cpuid,
+ ci->ci_ilevel, ci->ci_ipending,
+ ci->ci_idepth, ci->ci_ipis);
+ printf("CPU %ld interrupt level 0x%x pending "
+ "0x%x depth %d ci_ipis %d\n", oci->ci_cpuid,
+ oci->ci_ilevel, oci->ci_ipending,
+ oci->ci_idepth, oci->ci_ipis);
+ if (ipi_retry++ < 5)
+ goto ipi_again;
panic("fp_save ipi didn't");
}
#endif