bce(4) BCM5907 CTX write errors on 7.2 driver

[Tom Judge]

Hi,

Has anyone seen these errors before:

http://www.freebsd.org/cgi/query-pr.cgi?pr=135836&cat=

The system is a Dell R610 and it happens on both cold and warm boots.

I am about to check a second chassis, and test with 8, and will follow
up after my tests.

Tom


_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

Tom Judge wrote:

> Hi,
>
> Has anyone seen these errors before:
>
> http://www.freebsd.org/cgi/query-pr.cgi?pr=135836&cat=
>
> The system is a Dell R610 and it happens on both cold and warm boots.
>
> I am about to check a second chassis, and test with 8, and will follow
> up after my tests.
>

Here are my tests with 3 of the 13th Gen Dell chassis.

http://www.tomjudge.com/index.php/FreeBSD/Dell_13th_Gen_Servers

Summary:

    R710 is fine
   
    R410 is fine once you can get it to PXE load correctly

    R610 is no good at the moment.

Tom

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Steven Hartland]

M610 is also broken due to unsupported bge revision :(

    Regards
    Steve
----- Original Message -----
From: "Tom Judge" <tom@...>
To: <net@...>
Cc: <pyunyh@...>; "Xin LI" <delphij@...>; "David Christensen" <davidch@...>; <rwilliams@...>;
"Stanislav Sedov" <stas@...>
Sent: Tuesday, October 27, 2009 8:11 PM
Subject: Re: bce(4) BCM5907 CTX write errors on 7.2 driver

> Tom Judge wrote:
>> Hi,
>>
>> Has anyone seen these errors before:
>>
>> http://www.freebsd.org/cgi/query-pr.cgi?pr=135836&cat=
>>
>> The system is a Dell R610 and it happens on both cold and warm boots.
>>
>> I am about to check a second chassis, and test with 8, and will follow up after my tests.
>>
>
> Here are my tests with 3 of the 13th Gen Dell chassis.
>
> http://www.tomjudge.com/index.php/FreeBSD/Dell_13th_Gen_Servers
>
> Summary:
>
>    R710 is fine
>   R410 is fine once you can get it to PXE load correctly
>
>    R610 is no good at the moment.
>
> Tom
>
> _______________________________________________
> freebsd-net@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."
>

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone [phone]
or return the E.mail to postmaster@....

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

Steven Hartland wrote:
> M610 is also broken due to unsupported bge revision :(
>

Hi Steve,

This seems to be a missing PHY driver for bce(4)+SerDes rather than a
bge issue, see PR:kern/134658

You already refrenced this in a thread on current@ releated to an HP system.

I have added this info to my page.

Tom

> ----- Original Message ----- From: "Tom Judge" <tom@...>
> To: <net@...>
> Cc: <pyunyh@...>; "Xin LI" <delphij@...>; "David
> Christensen" <davidch@...>; <rwilliams@...>;
> "Stanislav Sedov" <stas@...>
> Sent: Tuesday, October 27, 2009 8:11 PM
> Subject: Re: bce(4) BCM5907 CTX write errors on 7.2 driver
>
>
>> Tom Judge wrote:
>>> Hi,
>>>
>>> Has anyone seen these errors before:
>>>
>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=135836&cat=
>>>
>>> The system is a Dell R610 and it happens on both cold and warm boots.
>>>
>>> I am about to check a second chassis, and test with 8, and will
>>> follow up after my tests.
>>>
>>
>> Here are my tests with 3 of the 13th Gen Dell chassis.
>>
>> http://www.tomjudge.com/index.php/FreeBSD/Dell_13th_Gen_Servers
>>
>> Summary:
>>
>>    R710 is fine
>>   R410 is fine once you can get it to PXE load correctly
>>
>>    R610 is no good at the moment.
>>
>> Tom
>>
>> _______________________________________________
>> freebsd-net@... mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."
>>
>
>
> ================================================
> This e.mail is private and confidential between Multiplay (UK) Ltd.
> and the person or entity to whom it is addressed. In the event of
> misdirection, the recipient is prohibited from using, copying,
> printing or otherwise disseminating it or any information contained in
> it.
> In the event of misdirection, illegible or incomplete transmission
> please telephone [phone]
> or return the E.mail to postmaster@....
>
> _______________________________________________
> freebsd-net@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[David Christensen-4]

> Has anyone seen these errors before:
>
> http://www.freebsd.org/cgi/query-pr.cgi?pr=135836&cat=
>
> The system is a Dell R610 and it happens on both cold and warm boots.
>
> I am about to check a second chassis, and test with 8, and
> will follow up after my tests.

Yes, I've seen these errors on a Dell r710.  The problem is related
to an interaction between the FreeBSD driver and the bootcode firmware
on the BCM5709 controller which can cause the bootcode firmware to
enable certain hardware blocks in the chip while the FreeBSD driver is
attempting to reset the device.  This allows RX traffic to enter the
controller before the device is fully initialized.

The fix is to upgrade the BCM5709 bootcode to version 5.02 or later.
I'm checking if Dell has posted updated firmware which can address
the problem.

Dave
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

David Christensen wrote:

>> Has anyone seen these errors before:
>>
>> http://www.freebsd.org/cgi/query-pr.cgi?pr=135836&cat=
>>
>> The system is a Dell R610 and it happens on both cold and warm boots.
>>
>> I am about to check a second chassis, and test with 8, and
>> will follow up after my tests.
>>    
>
> Yes, I've seen these errors on a Dell r710.  The problem is related
> to an interaction between the FreeBSD driver and the bootcode firmware
> on the BCM5709 controller which can cause the bootcode firmware to
> enable certain hardware blocks in the chip while the FreeBSD driver is
> attempting to reset the device.  This allows RX traffic to enter the
> controller before the device is fully initialized.
>
> The fix is to upgrade the BCM5709 bootcode to version 5.02 or later.
> I'm checking if Dell has posted updated firmware which can address
> the problem.
>
>  

Hi David,

Thanks for the rapid response.

Dell have firmware 5.0.9 on their website here:  http://tiny.cc/ex834

Will that work?

Tom


_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[David Christensen-4]

> Thanks for the rapid response.
>
> Dell have firmware 5.0.9 on their website here:  http://tiny.cc/ex834
>
> Will that work?

Yes, that release does include a good version of BCM5709 bootcode (v5.06).
I couldn't really tell until I downloaded the file and looked at the
temporary files inside.  Thanks for the link.

Dave
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

First sorry for the top post, the blackberry won't allow me to bottom post or I can't find the option.

I will try this update when I get to the office in the morning. Hopefully it will resolve the issue.

Thanks

Tom
------Original Message------
From: David Christensen
To: Tom Judge
Cc: net@...
Cc: rwilliams@...
Cc: Gideon Naim
Subject: RE: bce(4) BCM5907 CTX write errors on 7.2 driver
Sent: 27 Oct 2009 17:18

> Thanks for the rapid response.
>
> Dell have firmware 5.0.9 on their website here:  http://tiny.cc/ex834
>
> Will that work?

Yes, that release does include a good version of BCM5709 bootcode (v5.06).
I couldn't really tell until I downloaded the file and looked at the
temporary files inside.  Thanks for the link.

Dave


Sent via BlackBerry from T-Mobile
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Stanislav Sedov-4]

On Tue, 27 Oct 2009 20:30:38 -0000
"Steven Hartland" <killing@...> mentioned:

> M610 is also broken due to unsupported bge revision :(
>

What bge revision do you have?  I recently committed support
for new bge(4) chip revisions to HEAD, can you check, please,
if HEAD recognize this adapter correctly?  You can boot HEAD
kernel on your 7.x/8.x userland for test purposes.

Thanks!

--
Stanislav Sedov
ST4096-RIPE

[Steven Hartland]

Sorry I typo'ed there, should have said bce, is that still relevant?.

We haven't been able to install any version yet, so can't just
try a kernel, no way of getting data on to the machine without
a working netcard ;-)

Did this update include any updated PHY support for bce? As the
the precise error indicates that as the issue, as can be seen here:
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/134658

If it does will this be in an ISO snapshot yet, as that's the only
way of testing?

    Regards
    Steve
----- Original Message -----
From: "Stanislav Sedov" <stas@...>

> M610 is also broken due to unsupported bge revision :(
>

What bge revision do you have?  I recently committed support
for new bge(4) chip revisions to HEAD, can you check, please,
if HEAD recognize this adapter correctly?  You can boot HEAD
kernel on your 7.x/8.x userland for test purposes.

Thanks!

--
Stanislav Sedov
ST4096-RIPE



================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone [phone]
or return the E.mail to postmaster@....

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Stanislav Sedov-4]

On Tue, 27 Oct 2009 23:02:09 -0000
"Steven Hartland" <killing@...> mentioned:

> Sorry I typo'ed there, should have said bce, is that still relevant?.
>
> We haven't been able to install any version yet, so can't just
> try a kernel, no way of getting data on to the machine without
> a working netcard ;-)
>
> Did this update include any updated PHY support for bce? As the
> the precise error indicates that as the issue, as can be seen here:
> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/134658
>
> If it does will this be in an ISO snapshot yet, as that's the only
> way of testing?
>

If I understand the PR comments right, the code to support this PHY
should be present in 8.0.  So you can start by trying out 8.0-RC1
ISO image (or USB stick image, fwiw).

--
Stanislav Sedov
ST4096-RIPE

[Steven Hartland]

On Tue, 27 Oct 2009 23:02:09 -0000
"Steven Hartland" <killing@...> mentioned:

> If I understand the PR comments right, the code to support this PHY
> should be present in 8.0.  So you can start by trying out 8.0-RC1
> ISO image (or USB stick image, fwiw).

Just tried 8.0RC2 no go, PHY still not supported :(

    Regards
    Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone [phone]
or return the E.mail to postmaster@....

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

Steven Hartland wrote:

>
> On Tue, 27 Oct 2009 23:02:09 -0000
> "Steven Hartland" <killing@...> mentioned:
>
>> If I understand the PR comments right, the code to support this PHY
>> should be present in 8.0.  So you can start by trying out 8.0-RC1
>> ISO image (or USB stick image, fwiw).
>
> Just tried 8.0RC2 no go, PHY still not supported :(
>

Looking though the mii code it seems the SerDes PHY support was moved
into the brgphy code in 2007.
This driver however only seems to support the 5708S PHY and not the 5709.

Maybe David can shed some light on the support for this.


Tom
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

David Christensen wrote:

>> Thanks for the rapid response.
>>
>> Dell have firmware 5.0.9 on their website here:  http://tiny.cc/ex834
>>
>> Will that work?
>
> Yes, that release does include a good version of BCM5709 bootcode (v5.06).
> I couldn't really tell until I downloaded the file and looked at the
> temporary files inside.  Thanks for the link.
>
>

After fighting with the update process for a bit and then learning about
the live cycle manager I have managed to test the R610 with the 5.0.9
firmware.

On the face of it, it seems that this resolves the issue.

Tom
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

Tom Judge wrote:

> David Christensen wrote:
>>> Thanks for the rapid response.
>>>
>>> Dell have firmware 5.0.9 on their website here:  http://tiny.cc/ex834
>>>
>>> Will that work?
>>
>> Yes, that release does include a good version of BCM5709 bootcode
>> (v5.06).
>> I couldn't really tell until I downloaded the file and looked at the
>> temporary files inside.  Thanks for the link.
>>
>>
>
> After fighting with the update process for a bit and then learning about
> the live cycle manager I have managed to test the R610 with the 5.0.9
> firmware.
>
> On the face of it, it seems that this resolves the issue.
>

After a reboot this morning the error has resurfaced and the box is
again useless.


Tom
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[David Christensen-4]

> > After fighting with the update process for a bit and then learning
> > about the live cycle manager I have managed to test the
> R610 with the
> > 5.0.9 firmware.
> >
> > On the face of it, it seems that this resolves the issue.
> >
> After a reboot this morning the error has resurfaced and the
> box is again useless.

Can you try a different test?  Power-on the system with the network
cable attached to an idle switch (i.e. keep all network traffic from
being forwarded to the NIC during driver initialization).  Does the
system power up successfully?  Repeatedly?  The problem I saw
was caused by network traffic being handled by the NIC during driver
initialization.  If you still see the same behavior then this might
be a different issue.

Dave
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

David Christensen wrote:

>>> After fighting with the update process for a bit and then learning
>>> about the live cycle manager I have managed to test the
>> R610 with the
>>> 5.0.9 firmware.
>>>
>>> On the face of it, it seems that this resolves the issue.
>>>
>> After a reboot this morning the error has resurfaced and the
>> box is again useless.
>
> Can you try a different test?  Power-on the system with the network
> cable attached to an idle switch (i.e. keep all network traffic from
> being forwarded to the NIC during driver initialization).  Does the
> system power up successfully?  Repeatedly?  The problem I saw
> was caused by network traffic being handled by the NIC during driver
> initialization.  If you still see the same behavior then this might
> be a different issue.
>

Looks like we are heading in the right direction here.

Setup:

R610[bce1]->Netgear 100/10 unmanaged switch-[uplink]->Managed server
access switch.

The R610 is the only thing connected to the netgear switch.

When the uplink was severed during boot up it was reconnected after the
link up event and while the NFS mounts where being mounted.

Cold reboots are with the server shutdown with the psu's remaining
connected and then restarted.

Warm are just 'shutdown -r now'

Ice are from the psus disconnected from the mains supply.

These results are time ordered.

Type Uplink Result
Warm No Works
Warm No Works
Warm No Works
Cold No Works
Warm No Works
Cold No Works
Warm No Works
Ice No Works
Warn No Works
Ice Yes Works
Warm Yes Works
Cold Yes Works
Warm Yes Works
Ice Bybass Works
Warm Bypass Fails
Cold Bypass Works - paniced on shutdown spin lock (sched) held to long pid 0
Cold Bypass Works
Warm Bypass Fails



We can provide full remote access to the following hardware (via
enterprise drac) if required and on site hands if this will help get
this issue resolved.

3 * R610
4 * R410
7 * R710


Tom
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[David Christensen-4]

> > Can you try a different test?  Power-on the system with the network
> > cable attached to an idle switch (i.e. keep all network
> traffic from
> > being forwarded to the NIC during driver initialization).  Does the
> > system power up successfully?  Repeatedly?  The problem I saw was
> > caused by network traffic being handled by the NIC during driver
> > initialization.  If you still see the same behavior then
> this might be
> > a different issue.
> >
>
> Looks like we are heading in the right direction here.
>

The next test is to diable the LOM's management firmware but boot to
an active network.  Get the User Diag utility at the bottom of
http://www.broadcom.com/support/ethernet_nic/netxtremeii.php.
Run the uxdiag utility with the command line:
"C:\>uxdiag -c 0 -mfw 0 -c 1 -mfw 0 -c 2 -mfw 0 -c 3 mfw 0"
The "-c X" specifies with LOM to use while the "-mfw 0" disables
the firmware.  Use the appropriate number of "-c X" values for
the number of ports on your system (the r710 has 4 ports).
To re-enable the firmware do the following:
"C:\>uxdiag -c 0 -mfw 1 -c 1 -mfw 1 -c 2 -mfw 1 -c 3 mfw 1"

Finally, the routine bce_print_adapter_info() in HEAD prints out both
the bootcode and management firmware versions.  If you can get those
same changes into your release I'd like to see the versions reported
on your system.

Dave
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

David Christensen wrote:

>>> Can you try a different test?  Power-on the system with the network
>>> cable attached to an idle switch (i.e. keep all network
>> traffic from
>>> being forwarded to the NIC during driver initialization).  Does the
>>> system power up successfully?  Repeatedly?  The problem I saw was
>>> caused by network traffic being handled by the NIC during driver
>>> initialization.  If you still see the same behavior then
>> this might be
>>> a different issue.
>>>
>> Looks like we are heading in the right direction here.
>>
>
>
> The next test is to diable the LOM's management firmware but boot to
> an active network.  Get the User Diag utility at the bottom of
> http://www.broadcom.com/support/ethernet_nic/netxtremeii.php.
> Run the uxdiag utility with the command line:
> "C:\>uxdiag -c 0 -mfw 0 -c 1 -mfw 0 -c 2 -mfw 0 -c 3 mfw 0"
> The "-c X" specifies with LOM to use while the "-mfw 0" disables
> the firmware.  Use the appropriate number of "-c X" values for
> the number of ports on your system (the r710 has 4 ports).
> To re-enable the firmware do the following:
> "C:\>uxdiag -c 0 -mfw 1 -c 1 -mfw 1 -c 2 -mfw 1 -c 3 mfw 1"
>
> Finally, the routine bce_print_adapter_info() in HEAD prints out both
> the bootcode and management firmware versions.  If you can get those
> same changes into your release I'd like to see the versions reported
> on your system.
>

Here is the info from a boot of 8.0 RC2.

ASIC: 0x57092003
B/C: 5.0.6
Rev: C0
Bus: PCIe x4, 2.5Gb/s
Flags: MSI|MFW
MFW: NCS 2.0.3

And looking at this it seems dells update CD was not up to date enough
and I only got 5.0.6 firmware not 5.0.9.

I will update this now and retest.

TJ
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[David Christensen-4]

> > The next test is to diable the LOM's management firmware
> but boot to
> > an active network.  Get the User Diag utility at the bottom of
> > http://www.broadcom.com/support/ethernet_nic/netxtremeii.php.
> > Run the uxdiag utility with the command line:
> > "C:\>uxdiag -c 0 -mfw 0 -c 1 -mfw 0 -c 2 -mfw 0 -c 3 mfw 0"
> > The "-c X" specifies with LOM to use while the "-mfw 0"
> disables the
> > firmware.  Use the appropriate number of "-c X" values for
> the number
> > of ports on your system (the r710 has 4 ports).
> > To re-enable the firmware do the following:
> > "C:\>uxdiag -c 0 -mfw 1 -c 1 -mfw 1 -c 2 -mfw 1 -c 3 mfw 1"
> >
> > Finally, the routine bce_print_adapter_info() in HEAD
> prints out both
> > the bootcode and management firmware versions.  If you can
> get those
> > same changes into your release I'd like to see the versions
> reported
> > on your system.
> >
>
> Here is the info from a boot of 8.0 RC2.
>
> ASIC: 0x57092003
> B/C: 5.0.6
> Rev: C0
> Bus: PCIe x4, 2.5Gb/s
> Flags: MSI|MFW
> MFW: NCS 2.0.3
>
> And looking at this it seems dells update CD was not up to
> date enough and I only got 5.0.6 firmware not 5.0.9.

The package version and the bootcode version are similar but
they are not the same.  The bootcode you have (v5.0.6) is
sufficient to fix the problem.

Dave
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."