Re: System crawls after Upgrade 7.0->7.2

View: New views
4 Messages — Rating Filter:   Alert me  

Parent Message unknown Re: System crawls after Upgrade 7.0->7.2

by tlott-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message



On Tue, 8 Sep 2009 12:19:45 +0200
Tobias Lott <tlott@...> wrote:

> Hey Everyone,
>
> I upgraded a Dual Core Machine to 7.2-Stable (2 Days ago), all OS
> related Stuff is located on an UFS Slice, Application is on a ZFS
> Volume.
>
> After the Upgrade everything seemed fine, but a User noticed one PHP
> Script which is basically loading a plain Textfile into Mysql times
> out. PHP Timeout was set to 60 secs, that was more then enough just
> one day before the upgrade.
>
> System-wise its like Mysql can't get get the data fast enough, process
> is max at 10% cpu usage most of the time in sbwait state.
>
> Haven't changed any sysctl (kern.maxvnodes="200000",
> vm.kmem_size="512M" vm.kmem_size_max="512M" vfs.zfs.arc_max="100M"
> but had those before already) or kernel parameter at that time. But
> according to the Wiki's ZFS Tuning Guide those aren't needed anymore
> so I tried without but no change there either.
>
> I checked the following (and the current used values each):
> maxproc
> maxfiles
> kern.ipc.somaxconn
> kern.ipc.nmbclusters
>
> Getting no error messages dmesg, syslog wise.
>
> Tried moving Mysql to the UFS Slice but same thing happens happens
> there.
>
> Some very basic testing like dd-ing:
> # dd if=/dev/da0 of=/dev/null bs=1024 count=1048576
> 1048576+0 records in
> 1048576+0 records out
> 1073741824 bytes transferred in 157.698602 secs (6808823 bytes/sec)
>
> FreeBSD hostname 7.2-STABLE FreeBSD 7.2-STABLE #3 r196954:
> Tue Sep  8 02:10:22 CEST 2009
> root@hostname:/usr/obj/usr/src/sys/SPIRIT  amd64
>
> Since I upgraded the Machines zpool to version 13 already I don't
> really wanna go back to 7.0, but it seems the only way out atm.
>
> Hopefully someone can give me a Hint where maybe I forgot to check.
>
>
> Somehow it feels like the last Versions, I'd say starting with 7.x
> doesn't really feel that Rock-Stable as Versions Prior used to
> be. Prolly cause I'm using an Experimental Feature like ZFS, so no
> Offense nor Blame and this is not intended as a Flame or whatsoever!
> So don't get me wrong, I always liked, used, recommended FreeBSD since
> 4.0 and will continue to do so!
>
> Best Regards
>

I've tested some more it seems the Bottleneck is the HD, Raid Array is
Optimal checked it.
Running bonnie++ takes like forever, couldn't finish it since its a
Productive Machine, cause it renders the Server useless, every try to
access the server while bonnie is running times out. Weird thing is
bonnie++ is using max 1-3% CPU. But according to gstat its only ~3mb/s
(displayed as 100% busy) during first 3 Tests that is.


savecore: reboot after panic: page fault

But had no luck with the coredumb

# kgdb kernel.debug  /var/crash/vmcore.12
<snip>
Cannot access memory at address 0x0

Or is there any other method I could try?

Cause this is getting nasty unstable

--
Tobias Lott
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

Re: System crawls after Upgrade 7.0->7.2

by Clifton Royston :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Wed, Sep 09, 2009 at 10:48:07AM +0200, Tobias Lott wrote:

> On Tue, 8 Sep 2009 12:19:45 +0200
> Tobias Lott <tlott@...> wrote:
> > Hey Everyone,
> >
> > I upgraded a Dual Core Machine to 7.2-Stable (2 Days ago), all OS
> > related Stuff is located on an UFS Slice, Application is on a ZFS
> > Volume.
> >
> > After the Upgrade everything seemed fine, but a User noticed one PHP
> > Script which is basically loading a plain Textfile into Mysql times
> > out. PHP Timeout was set to 60 secs, that was more then enough just
> > one day before the upgrade.
...

> > Since I upgraded the Machines zpool to version 13 already I don't
> > really wanna go back to 7.0, but it seems the only way out atm.
> >
> > Hopefully someone can give me a Hint where maybe I forgot to check.
> >
> > Somehow it feels like the last Versions, I'd say starting with 7.x
> > doesn't really feel that Rock-Stable as Versions Prior used to
> > be. Prolly cause I'm using an Experimental Feature like ZFS, so no
> > Offense nor Blame and this is not intended as a Flame or whatsoever!
> > So don't get me wrong, I always liked, used, recommended FreeBSD since
> > 4.0 and will continue to do so!
> >
> > Best Regards
> >
>
> I've tested some more it seems the Bottleneck is the HD, Raid Array is
> Optimal checked it.

  Probably you already checked this, but did you go through the dmesg
output relating to ata and drive detection closely?

  In the past from time to time I've had machines suddenly start
crawling after an upgrade, and it turned out it was because some change
in the driver detection caused the ata to fall back to pio mode.  The
symptoms are pretty much like you describe - all HD IO takes forever.

  -- Clifton

--
    Clifton Royston  --  cliftonr@... / cliftonr@...
       President  - I and I Computing * http://www.iandicomputing.com/
 Custom programming, network design, systems and network consulting services
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

Parent Message unknown Re: System crawls after Upgrade 7.0->7.2

by tlott-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message



On Wed, 9 Sep 2009 21:30:22 +0200
Tobias Lott <tlott@...> wrote:

>
>
> On Wed, 9 Sep 2009 07:02:13 -1000
> Clifton Royston <cliftonr@...> wrote:
>
> > On Wed, Sep 09, 2009 at 10:48:07AM +0200, Tobias Lott wrote:
> > > On Tue, 8 Sep 2009 12:19:45 +0200
> > > Tobias Lott <tlott@...> wrote:
> > > > Hey Everyone,
> > > >
> > > > I upgraded a Dual Core Machine to 7.2-Stable (2 Days ago), all
> > > > OS related Stuff is located on an UFS Slice, Application is on
> > > > a ZFS Volume.
> > > >
> > > > After the Upgrade everything seemed fine, but a User noticed one
> > > > PHP Script which is basically loading a plain Textfile into
> > > > Mysql times out. PHP Timeout was set to 60 secs, that was more
> > > > then enough just one day before the upgrade.
> > ...
> > > > Since I upgraded the Machines zpool to version 13 already I
> > > > don't really wanna go back to 7.0, but it seems the only way
> > > > out atm.
> > > >
> > > > Hopefully someone can give me a Hint where maybe I forgot to
> > > > check.
> > > >
> > > > Somehow it feels like the last Versions, I'd say starting with
> > > > 7.x doesn't really feel that Rock-Stable as Versions Prior used
> > > > to be. Prolly cause I'm using an Experimental Feature like ZFS,
> > > > so no Offense nor Blame and this is not intended as a Flame or
> > > > whatsoever! So don't get me wrong, I always liked, used,
> > > > recommended FreeBSD since 4.0 and will continue to do so!
> > > >
> > > > Best Regards
> > > >
> > >
> > > I've tested some more it seems the Bottleneck is the HD, Raid
> > > Array is Optimal checked it.
> >
> >   Probably you already checked this, but did you go through the
> > dmesg output relating to ata and drive detection closely?
> >
> >   In the past from time to time I've had machines suddenly start
> > crawling after an upgrade, and it turned out it was because some
> > change in the driver detection caused the ata to fall back to pio
> > mode.  The symptoms are pretty much like you describe - all HD IO
> > takes forever.
> >
> >   -- Clifton
> >
>
> Thanks for that Hint, checked again to be sure, but thats not the
> Case.
>
> mpt0:vol0(mpt0:0:0): Settings ( Hot-Plug-Spares High-Priority-ReSync )
> mpt0:vol0(mpt0:0:0): Using Spare Pool: 0
> mpt0:vol0(mpt0:0:0): 2 Members:
>       (mpt0:1:32:0): Primary Online
>       (mpt0:1:1:0): Secondary Online
> mpt0:vol0(mpt0:0:0): RAID-1 - Optimal
> mpt0:vol0(mpt0:0:0): Status ( Enabled )
> (mpt0:vol0:1): Physical (mpt0:0:1:0), Pass-thru (mpt0:1:0:0)
> (mpt0:vol0:1): Online
> (mpt0:vol0:0): Physical (mpt0:0:32:0), Pass-thru (mpt0:1:1:0)
> (mpt0:vol0:0): Online
> acd0: CDROM <HL-DT-ST GCR-8240N/1.10> at ata0-master UDMA33
> acd1: CDROM <VIRTUALCDROM DRIVE/> at ata2-slave PIO3
> da0 at mpt0 bus 0 target 0 lun 0
> da0: <Dell VIRTUAL DISK 1028> Fixed Direct Access SCSI-5 device
> da0: 300.000MB/s transfers
> da0: Command Queueing Enabled
> SdMaP0::  A1P5 1C6P3U4 M#B1  (L3a1u0n5c4h6e4d3!2
>  512 byte sectors: 255H 63S/T 19330C)
>
>

Oh well just did some more research and found:
http://www.mail-archive.com/freebsd-performance@.../msg02461.html

Gonna try it out later and gonna give a Report.

--
Tobias Lott
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

Re: System crawls after Upgrade 7.0->7.2

by Stefan Lambrev-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi Tobias,

This is the "pseudo" RAID controller that comes by default with most  
Dell, low cost HP servers and probably others?
If you depend on I/O I would recommend you to change this raid with  
PERC6/i (looks like this is Dell server)
which is little more expensive, but comes with 256MB battery backed  
cache.

In my experience mpt devices are crap, and their performance is worse  
then software mirror.
PERC6/i is something totally different :) It's again LSI product  
branded from Dell and it works with mfi driver.

The only problem in the past with them was the management utility  
which was linux binary, but today there is
official native tool in ports, and in 8.0 in base install there will  
be a tool to manage your controller.

Well this wasn't that big problem compared to mpt which cannot be  
managed under FreeBSD at all?

On Sep 9, 2009, at 10:59 PM, Tobias Lott wrote:

>
>
> On Wed, 9 Sep 2009 21:30:22 +0200
> Tobias Lott <tlott@...> wrote:
>
>>
>>
>> On Wed, 9 Sep 2009 07:02:13 -1000
>> Clifton Royston <cliftonr@...> wrote:
>>
>>> On Wed, Sep 09, 2009 at 10:48:07AM +0200, Tobias Lott wrote:
>>>> On Tue, 8 Sep 2009 12:19:45 +0200
>>>> Tobias Lott <tlott@...> wrote:
>>>>> Hey Everyone,
>>>>>
>>>>> I upgraded a Dual Core Machine to 7.2-Stable (2 Days ago), all
>>>>> OS related Stuff is located on an UFS Slice, Application is on
>>>>> a ZFS Volume.
>>>>>
>>>>> After the Upgrade everything seemed fine, but a User noticed one
>>>>> PHP Script which is basically loading a plain Textfile into
>>>>> Mysql times out. PHP Timeout was set to 60 secs, that was more
>>>>> then enough just one day before the upgrade.
>>> ...
>>>>> Since I upgraded the Machines zpool to version 13 already I
>>>>> don't really wanna go back to 7.0, but it seems the only way
>>>>> out atm.
>>>>>
>>>>> Hopefully someone can give me a Hint where maybe I forgot to
>>>>> check.
>>>>>
>>>>> Somehow it feels like the last Versions, I'd say starting with
>>>>> 7.x doesn't really feel that Rock-Stable as Versions Prior used
>>>>> to be. Prolly cause I'm using an Experimental Feature like ZFS,
>>>>> so no Offense nor Blame and this is not intended as a Flame or
>>>>> whatsoever! So don't get me wrong, I always liked, used,
>>>>> recommended FreeBSD since 4.0 and will continue to do so!
>>>>>
>>>>> Best Regards
>>>>>
>>>>
>>>> I've tested some more it seems the Bottleneck is the HD, Raid
>>>> Array is Optimal checked it.
>>>
>>>  Probably you already checked this, but did you go through the
>>> dmesg output relating to ata and drive detection closely?
>>>
>>>  In the past from time to time I've had machines suddenly start
>>> crawling after an upgrade, and it turned out it was because some
>>> change in the driver detection caused the ata to fall back to pio
>>> mode.  The symptoms are pretty much like you describe - all HD IO
>>> takes forever.
>>>
>>>  -- Clifton
>>>
>>
>> Thanks for that Hint, checked again to be sure, but thats not the
>> Case.
>>
>> mpt0:vol0(mpt0:0:0): Settings ( Hot-Plug-Spares High-Priority-
>> ReSync )
>> mpt0:vol0(mpt0:0:0): Using Spare Pool: 0
>> mpt0:vol0(mpt0:0:0): 2 Members:
>>      (mpt0:1:32:0): Primary Online
>>      (mpt0:1:1:0): Secondary Online
>> mpt0:vol0(mpt0:0:0): RAID-1 - Optimal
>> mpt0:vol0(mpt0:0:0): Status ( Enabled )
>> (mpt0:vol0:1): Physical (mpt0:0:1:0), Pass-thru (mpt0:1:0:0)
>> (mpt0:vol0:1): Online
>> (mpt0:vol0:0): Physical (mpt0:0:32:0), Pass-thru (mpt0:1:1:0)
>> (mpt0:vol0:0): Online
>> acd0: CDROM <HL-DT-ST GCR-8240N/1.10> at ata0-master UDMA33
>> acd1: CDROM <VIRTUALCDROM DRIVE/> at ata2-slave PIO3
>> da0 at mpt0 bus 0 target 0 lun 0
>> da0: <Dell VIRTUAL DISK 1028> Fixed Direct Access SCSI-5 device
>> da0: 300.000MB/s transfers
>> da0: Command Queueing Enabled
>> SdMaP0::  A1P5 1C6P3U4 M#B1  (L3a1u0n5c4h6e4d3!2
>> 512 byte sectors: 255H 63S/T 19330C)
>>
>>
>
> Oh well just did some more research and found:
> http://www.mail-archive.com/freebsd-performance@.../msg02461.html
>
> Gonna try it out later and gonna give a Report.
>
> --
> Tobias Lott
> _______________________________________________
> freebsd-stable@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@...
> "

--
Best Wishes,
Stefan Lambrev
ICQ# 24134177





_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."