|
View:
New views
12 Messages
—
Rating Filter:
Alert me
|
|
|
Segfault in Apache after upgrade to v4.6Hi
I've just upgraded a couple of (old-ish) servers to 4.6 and now Apache has started throwing intermittent segfaults on both. (dmesgs appended below) I previously upgraded from 4.4 to 4.5 without incident. Both of these machines have been running faultlessly for months without issue so something has gone wrong following the upgrade. Both machines are serving different web-sites so I don't believe there is anything specific to the php/html code on the sites. I have the following PHP5 modules loaded in /var/www/conf/php5/*.ini Server 1 extension=gd.so extension=mbstring.so extension=mcrypt.so extension=mysql.so extension=pdo_mysql.so Server2 extension=mcrypt.so extension=mysql.so extension=snmp.so Assuming this is a fault in a loaded module it has to be either mcrypt or mysql. I suspect this is mysql as one of the sites served is basically static HTML without mysql and this site does not appear to cause a segfault. Any ideas how I can track down what is causing the problem and more importantly, how to fix it. Thanks in advance Jeremy =============Excerpt from /var/www/logs/error_log================= [Sun Nov 1 16:45:30 2009] [notice] SIGHUP received. Attempting to restart [Sun Nov 1 16:45:30 2009] [notice] Initializing etag from /var/www/logs/etag- state [Sun Nov 1 16:45:30 2009] [notice] Apache configured -- resuming normal operations [Sun Nov 1 16:45:30 2009] [notice] Accept mutex: sysvsem (Default: sysvsem) [Sun Nov 1 16:45:51 2009] [notice] child pid 10330 exit signal Segmentation fault (11) [Sun Nov 1 16:45:59 2009] [notice] child pid 9658 exit signal Segmentation fault (11) [Sun Nov 1 16:46:42 2009] [notice] child pid 1604 exit signal Segmentation fault (11) [Sun Nov 1 16:47:24 2009] [notice] child pid 32289 exit signal Segmentation fault (11) [Sun Nov 1 16:50:34 2009] [notice] child pid 31089 exit signal Segmentation fault (11) == dmesg 1 ============================================================ OpenBSD 4.6 (GENERIC) #58: Thu Jul 9 21:24:42 MDT 2009 deraadt@...:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel Pentium III ("GenuineIntel" 686-class, 512KB L2 cache) 502 MHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,SER,MMX,FXSR,SSE real mem = 267939840 (255MB) avail mem = 250265600 (238MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 04/01/98, BIOS32 rev. 0 @ 0xf0210, SMBIOS rev. 2.1 @ 0xf88b0 (22 entries) bios0: vendor ACER version "V3.2 R03-B3 EN" date 04/06/1999 bios0: Acer AcerPower_4100_series apm0 at bios0: Power Management spec V1.2 apm0: AC on, battery charge unknown acpi at bios0 function 0x0 not configured pcibios0 at bios0: rev 2.1 @ 0xf0200/0xb00 pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf9e00/144 (7 entries) pcibios0: PCI Interrupt Router at 000:07:0 ("Intel 82371AB PIIX4 ISA" rev 0x00) pcibios0: PCI bus #1 is the last bus bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 cpu0 at mainbus0: (uniprocessor) cpu0: disabling processor serial number pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x03 intelagp0 at pchb0 agp0 at intelagp0: aperture at 0xe0000000, size 0x4000000 ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x03 pci1 at ppb0 bus 1 vga1 at pci1 dev 0 function 0 "ATI Rage Pro" rev 0x5c wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) piixpcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x02 pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility wd0 at pciide0 channel 0 drive 0: <Maxtor 90432D2> wd0: 16-sector PIO, LBA, 4121MB, 8440992 sectors wd1 at pciide0 channel 0 drive 1: <ST34342A> wd1: 16-sector PIO, LBA, 4103MB, 8404830 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 wd1(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 2 atapiscsi0 at pciide0 channel 1 drive 1 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: <ASUS, CRW-4012A, 1.0> ATAPI 5/cdrom removable cd0(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 2 uhci0 at pci0 dev 7 function 2 "Intel 82371AB USB" rev 0x01: irq 11 piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x02: polling iic0 at piixpm0 lm1 at iic0 addr 0x2d: W83781D spdmem0 at iic0 addr 0x50: 64MB SDRAM non-parity PC100CL3 spdmem1 at iic0 addr 0x51: 64MB SDRAM non-parity PC100CL3 spdmem2 at iic0 addr 0x52: 128MB SDRAM non-parity PC100CL3 "DPT SmartCache/Raid" rev 0x02 at pci0 dev 18 function 0 not configured xl0 at pci0 dev 19 function 0 "3Com 3c905B 100Base-TX" rev 0x30: irq 10, address 00:00:e2:15:ae:59 exphy0 at xl0 phy 24: 3Com internal media interface sv0 at pci0 dev 20 function 0 "S3 SonicVibes" rev 0x00: irq 11 audio0 at sv0 isa0 at piixpcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 midi0 at pcppi0: <PC speaker> spkr0 at pcppi0 lpt0 at isa0 port 0x378/4 irq 7 lm0 at isa0 port 0x290/8: W83781D lm1 detached npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec usb0 at uhci0: USB revision 1.0 uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 biomask fb65 netmask ff65 ttymask ffff mtrr: Pentium Pro MTRR support softraid0 at root root on wd0a swap on wd0b dump on wd0b === dmesg 2 ======================================================= OpenBSD 4.6 (GENERIC) #58: Thu Jul 9 21:24:42 MDT 2009 deraadt@...:/usr/src/sys/arch/i386/compile/GENERIC cpu0: AMD Athlon(tm) Processor ("AuthenticAMD" 686-class, 512KB L2 cache) 802 MHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR real mem = 536375296 (511MB) avail mem = 509837312 (486MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 07/26/00, BIOS32 rev. 0 @ 0xfdb50, SMBIOS rev. 2.3 @ 0xf04f0 (30 entries) bios0: vendor American Megatrends Inc. version "62710" date 07/15/97 bios0: Gigabyte Technology Co. Ltd. 7VX apm0 at bios0: Power Management spec V1.2 apm0: AC on, battery charge unknown acpi at bios0 function 0x0 not configured pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000 pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf7800/160 (8 entries) pcibios0: PCI Interrupt Router at 000:07:0 ("VIA VT82C686 ISA" rev 0x00) pcibios0: PCI bus #1 is the last bus bios0: ROM list: 0xc0000/0x10000 cpu0 at mainbus0: (uniprocessor) pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 "VIA VT8371 Host" rev 0x02 viaagp0 at pchb0: v2 agp0 at viaagp0: aperture at 0xe0000000, size 0x10000000 ppb0 at pci0 dev 1 function 0 "VIA VT8371 PCI-PCI" rev 0x00 pci1 at ppb0 bus 1 vga1 at pci1 dev 0 function 0 "NVIDIA Vanta" rev 0x15 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) pcib0 at pci0 dev 7 function 0 "VIA VT82C686 ISA" rev 0x22 pciide0 at pci0 dev 7 function 1 "VIA VT82C571 IDE" rev 0x10: ATA66, channel 0 configured to compatibility, channel 1 configured to compatibility wd0 at pciide0 channel 0 drive 0: <ST330630A> wd0: 16-sector PIO, LBA, 29188MB, 59777640 sectors wd1 at pciide0 channel 0 drive 1: <ST320414A> wd1: 16-sector PIO, LBA, 19092MB, 39102336 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4 wd1(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 4 atapiscsi0 at pciide0 channel 1 drive 0 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: <ASUS, CD-S500/A, 1.3C> ATAPI 5/cdrom removable cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2 uhci0 at pci0 dev 7 function 2 "VIA VT83C572 USB" rev 0x10: irq 11 uhci1 at pci0 dev 7 function 3 "VIA VT83C572 USB" rev 0x10: irq 11 viaenv0 at pci0 dev 7 function 4 "VIA VT82C686 SMBus" rev 0x30: 24-bit timer at 3579545Hz rl0 at pci0 dev 10 function 0 "Realtek 8139" rev 0x10: irq 9, address 00:e0:4c:30:61:22 rlphy0 at rl0 phy 0: RTL internal PHY eap0 at pci0 dev 14 function 0 "Ensoniq AudioPCI97" rev 0x07: irq 9 ac97: codec id 0x83847608 (SigmaTel STAC9708/11) ac97: codec features 18 bit DAC, 18 bit ADC, SigmaTel 3D audio0 at eap0 midi0 at eap0: <AudioPCI MIDI UART> isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 midi1 at pcppi0: <PC speaker> spkr0 at pcppi0 lpt0 at isa0 port 0x378/4 irq 7 npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 usb0 at uhci0: USB revision 1.0 uhub0 at usb0 "VIA UHCI root hub" rev 1.00/1.00 addr 1 usb1 at uhci1: USB revision 1.0 uhub1 at usb1 "VIA UHCI root hub" rev 1.00/1.00 addr 1 biomask fd65 netmask ff65 ttymask ffff mtrr: Pentium Pro MTRR support softraid0 at root root on wd0a swap on wd0b dump on wd0b |
|
|
Re: Segfault in Apache after upgrade to v4.6On 1/11/2009, at 5:27 PM, Jeremy Bowen wrote:
> Hi > > I've just upgraded a couple of (old-ish) servers to 4.6 and now > Apache has > started throwing intermittent segfaults on both. (dmesgs appended > below) > I previously upgraded from 4.4 to 4.5 without incident. > > Both of these machines have been running faultlessly for months > without issue > so something has gone wrong following the upgrade. Both machines > are serving > different web-sites so I don't believe there is anything specific > to the > php/html code on the sites. > > > I have the following PHP5 modules loaded in /var/www/conf/php5/*.ini > > Server 1 > extension=gd.so > extension=mbstring.so > extension=mcrypt.so > extension=mysql.so > extension=pdo_mysql.so > > Server2 > extension=mcrypt.so > extension=mysql.so > extension=snmp.so > > > Assuming this is a fault in a loaded module it has to be either > mcrypt or > mysql. I suspect this is mysql as one of the sites served is > basically static > HTML without mysql and this site does not appear to cause a segfault. > > > Any ideas how I can track down what is causing the problem and more > importantly, how to fix it. > > Thanks in advance > Jeremy > Can't help you directly, but there are a few questions that might help ... 1. Upgraded as per instructions - http://www.openbsd.org/faq/ upgrade46.html? 2. Upgrade included packages at the end of those instructions? 3. Applied errata patches (don't think they would make a difference here?) 4. What version of Apache? 1.3 in base, or 2.x from ports? 5. MySQL and PHP - from packages or ports? What versions? 6. Nothing in the archives can help? e.g. http://marc.info/? l=openbsd-misc&m=124516871013636&w=2 Thanks. |
|
|
Re: Segfault in Apache after upgrade to v4.6Richard Toohey wrote:
> [cut] > > Can't help you directly, but there are a few questions that might help > ... > > 1. Upgraded as per instructions - > http://www.openbsd.org/faq/upgrade46.html? Yes, followed the instructions to the letter. Everything else upgraded successfully and is working as expected. If I made a mistake, I duplicated the error on both upgrades. I think this is unlikely. > 2. Upgrade included packages at the end of those instructions? Yes, upgraded packages too. This seemed to go smoothly. I've since pkg_delete and pkg_add'ed the obvious suspect packages with no effect. > 3. Applied errata patches (don't think they would make a difference > here?) No, I haven't done that as I haven't built from source. Hadn't planned on having to rebuild packages for this upgrade. I could build from ports if this is likely to help.... > 4. What version of Apache? 1.3 in base, or 2.x from ports? Apache v1.3 standard install from base. Using package defaults. Not using ports. > 5. MySQL and PHP - from packages or ports? What versions? PHP v5.2.10 from packages (php5-core, php5-mysql) mysql-client-5.0.83 from packages. The two servers are drastically different hardware running completely different web applications. The affected web-sites also work intermittently. Some requests succeed. Others cause the segfault. Unfortunately they are production servers and are now unreliable. I think I have confirmed this is a php5-mysql problem as the server seems stable with the php5-mysql package removed. It's a little hard to tell as most of the sites on the servers make extensive use of MySQL however there is one that doesn't and I was unable to reproduce the segfault from this site with the mysql packages removed. Re-added the mysql package and the segfaults returned. |
|
|
Re: Segfault in Apache after upgrade to v4.6Maybe a Suhosin issue...
CVSROOT: /cvs Module name: ports Changes by: robert@... 2009/07/20 10:30:14 Modified files: www/php5 : Makefile.inc www/php5/core : Makefile www/php5/extensions: Makefile Added files: www/php5/core/patches: patch-ext_suhosin_suhosin_c Log message: Make suhosin.session.encrypt=off by default because there are some problems with php 5.2.10 and the suhosin patch. noted by william@ and people on ports@ It's going to be re-enabled when it gets fixed upstream. Just, put suhosin.session.encrypt=off in you php.ini and check out. 2009/11/1 Jeremy Bowen <jeremy@...>: > Richard Toohey wrote: >> >> [cut] >> >> Can't help you directly, but there are a few questions that might help ... >> >> 1. Upgraded as per instructions - >> http://www.openbsd.org/faq/upgrade46.html? > > Yes, followed the instructions to the letter. Everything else upgraded > successfully and is working as expected. > If I made a mistake, I duplicated the error on both upgrades. I think this > is unlikely. >> >> 2. Upgrade included packages at the end of those instructions? > > Yes, upgraded packages too. This seemed to go smoothly. I've since > pkg_delete and pkg_add'ed the obvious suspect packages with no effect. >> >> 3. Applied errata patches (don't think they would make a difference here?) > > No, I haven't done that as I haven't built from source. Hadn't planned on > having to rebuild packages for this upgrade. I could build from ports if > this is likely to help.... >> >> 4. What version of Apache? 1.3 in base, or 2.x from ports? > > Apache v1.3 standard install from base. Using package defaults. Not using > ports. > >> 5. MySQL and PHP - from packages or ports? What versions? > > PHP v5.2.10 from packages (php5-core, php5-mysql) > mysql-client-5.0.83 from packages. > > > The two servers are drastically different hardware running completely > different web applications. The affected web-sites also work > Some requests succeed. Others cause the segfault. Unfortunately they are > production servers and are now unreliable. > > I think I have confirmed this is a php5-mysql problem as the server seems > stable with the php5-mysql package removed. It's a little hard to tell as > most of the sites on the servers make extensive use of MySQL however there > is one that doesn't and I was unable to reproduce the segfault from this > site with the mysql packages removed. Re-added the mysql package and the > segfaults returned. |
|
|
[SOLVED] Re: Segfault in Apache after upgrade to v4.6On 11/1/2009, "Gonzalo Lionel Rodriguez" <gonzalo@...> wrote:
>Maybe a Suhosin issue... > >CVSROOT: /cvs >Module name: ports >Changes by: robert@... 2009/07/20 10:30:14 > >Modified files: >www/php5 : Makefile.inc www/php5/core : Makefile www/php5/extensions: >Makefile Added files: >www/php5/core/patches: patch-ext_suhosin_suhosin_c >Log message: >Make suhosin.session.encrypt=off by default because there are some >with php 5.2.10 and the suhosin patch. noted by william@ and people on ports@ >It's going to be re-enabled when it gets fixed upstream. > > >Just, put suhosin.session.encrypt=off in you php.ini and check out. Well amazingly (and to my great relief) this seems to have fixed things. Thank you so much Gonzalo. Despite not having "suhosin.session.encrypt=on" in my php.ini (i guess it's the default), explicitly setting this to "off" has fixed the segfault problem. Anyway, thanks to all who offered suggestions. It reinforces my decision to use OpenBSD. The support is amazing. Cheers Jeremy > >2009/11/1 Jeremy Bowen <jeremy@...>: >> Richard Toohey wrote: >>> >>> [cut] >>> >>> Can't help you directly, but there are a few questions that might help ... >>> >>> 1. Upgraded as per instructions - >>> http://www.openbsd.org/faq/upgrade46.html? >> >> Yes, followed the instructions to the letter. Everything else upgraded >> successfully and is working as expected. >> If I made a mistake, I duplicated the error on both upgrades. I think this >> is unlikely. >>> >>> 2. Upgrade included packages at the end of those instructions? >> >> Yes, upgraded packages too. This seemed to go smoothly. I've since >> pkg_delete and pkg_add'ed the obvious suspect packages with no effect. >>> >>> 3. Applied errata patches (don't think they would make a difference here?) >> >> No, I haven't done that as I haven't built from source. Hadn't planned on >> having to rebuild packages for this upgrade. I could build from ports if >> this is likely to help.... >>> >>> 4. What version of Apache? 1.3 in base, or 2.x from ports? >> >> Apache v1.3 standard install from base. Using package defaults. Not using >> ports. >> >>> 5. MySQL and PHP - from packages or ports? What versions? >> >> PHP v5.2.10 from packages (php5-core, php5-mysql) >> mysql-client-5.0.83 from packages. >> >> >> The two servers are drastically different hardware running completely >> different web applications. The affected web-sites also work >intermittently. >> Some requests succeed. Others cause the segfault. Unfortunately they are >> production servers and are now unreliable. >> >> I think I have confirmed this is a php5-mysql problem as the server >> stable with the php5-mysql package removed. It's a little hard to tell as >> most of the sites on the servers make extensive use of MySQL however there >> is one that doesn't and I was unable to reproduce the segfault from this >> site with the mysql packages removed. Re-added the mysql package and the >> segfaults returned. |
|
|
Re: Segfault in Apache after upgrade to v4.6On 2009-11-01, Gonzalo Lionel Rodriguez <gonzalo@...> wrote:
> Maybe a Suhosin issue... This is almost certainly the problem. > Just, put suhosin.session.encrypt=off in you php.ini and check out. That will get you going for now, but -stable ports has a newer version of PHP and the Suhosin patchset where this problem and some security- related bugs are fixed. |
|
|
Re: Segfault in Apache after upgrade to v4.6* Jeremy Bowen <jeremy@...> [2009-11-01 07:23]:
> I've just upgraded a couple of (old-ish) servers to 4.6 and now Apache has > started throwing intermittent segfaults on both. (dmesgs appended below) > I have the following PHP5 modules loaded in /var/www/conf/php5/*.ini yyou need to upgrade php to 5.2.11, from -stable. -- Henning Brauer, hb@..., henning@... BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting |
|
|
Where are ports changes for -stable?Henning Brauer wrote:
> yyou need to upgrade php to 5.2.11, from -stable. > > Sorry if I have missed something, but where would I find the ports changes for -stable? (Other than manually looking in each port's Makefile details.) Until Henning mentioned the new version, I had no idea php had been upgraded. -- -RSM http://www.erratic.ca |
|
|
Re: Where are ports changes for -stable?On Sun, 01 Nov 2009 21:05:28 -0500
Scott McEachern <scott@...> wrote: > Henning Brauer wrote: > > yyou need to upgrade php to 5.2.11, from -stable. > > > > > Sorry if I have missed something, but where would I find the ports > changes for -stable? (Other than manually looking in each port's > Makefile details.) Until Henning mentioned the new version, I had no > idea php had been upgraded. > First there are the commit messages on the ports-changes mailinglist. Look for those tagged OPENBSD_4_6. When you update your local cvs checkout, just ommit the -q option and you will see every changed file, so you don't have to manually dive into the tree. - Robert |
|
|
Re: Where are ports changes for -stable?On 2009-11-02, Scott McEachern <scott@...> wrote:
> Henning Brauer wrote: >> yyou need to upgrade php to 5.2.11, from -stable. >> >> > Sorry if I have missed something, but where would I find the ports > changes for -stable? (Other than manually looking in each port's > Makefile details.) Until Henning mentioned the new version, I had no > idea php had been upgraded. > here's one easy way, assuming you have a cvs checkout of the ports tree and CVSROOT set to a suitable mirror. $ cd /usr/ports $ cvs -d $CVSROOT -q up -Pd -r OPENBSD_4_6 <watch the output> or, you can run it from cron. |
|
|
Re: Where are ports changes for -stable?On Mon, Nov 2, 2009 at 5:31 AM, Robert <robert@...> wrote:
> When you update your local cvs checkout, just ommit the -q option and > you will see every changed file, so you don't have to manually dive into > the tree. cvs will print changed files even with -q. -q just tells it to omit the directory traversal crap nobody cares about. |
|
|
Re: Where are ports changes for -stable?Robert wrote:
>> First there are the commit messages on the ports-changes mailinglist. >> Look for those tagged OPENBSD_4_6. >> >> When you update your local cvs checkout, just ommit the -q option and >> you will see every changed file, so you don't have to manually dive into >> the tree. >> >> >> - Robert >> >> >> I have subscribed to the ports-changes list and watch for the OPENBSD_4_6 tag as that seems most appropriate for my situation. The only problem with watching the cvs output is a catch-22: I don't do cvs up and a fresh build until there is a change to -stable. Thanks for the many replies folks, as always, problem solved. -- -RSM http://www.erratic.ca |
| Free embeddable forum powered by Nabble | Forum Help |