gigabit "benchmark"

[Klaus Heinz-27]

Hi,

while searching for something different I stumbled across this
comparison using, among others, NetBSD/i386 4.0RC3

  http://www.bluelife.at/articles/98/

IMO you can ignore the German text, the table below "Hardware" speaks for
itself.

Can anyone repeat those results for NetBSD?
Any opinions why NetBSD does not reach (near-) wirespeed with those Intel
NICs as the others do and at the same time needs 100% CPU?

ciao
     Klaus

[Manuel Bouyer]

On Wed, Dec 05, 2007 at 09:56:27PM +0100, Klaus Heinz wrote:

> Hi,
>
> while searching for something different I stumbled across this
> comparison using, among others, NetBSD/i386 4.0RC3
>
>   http://www.bluelife.at/articles/98/
>
> IMO you can ignore the German text, the table below "Hardware" speaks for
> itself.
>
> Can anyone repeat those results for NetBSD?
> Any opinions why NetBSD does not reach (near-) wirespeed with those Intel
> NICs as the others do and at the same time needs 100% CPU?

Maybe em enables hardware checkums automatically, while on wm it needs to
be turned on explicitely ?

--
Manuel Bouyer <bouyer@...>
     NetBSD: 26 ans d'experience feront toujours la difference
--

[Hubert Feyrer-4]

On Wed, 5 Dec 2007, Klaus Heinz wrote:
> Any opinions why NetBSD does not reach (near-) wirespeed with those Intel
> NICs as the others do and at the same time needs 100% CPU?

TCP buffers too small, and other values not tuned, see [1]? Though IIRC
there was something about auto-tuning those buffers, but I don't know if
that's enabled by default (or how to enable it if not).

[1] http://proj.sunet.se/LSR2/


  - Hubert

[Joerg Sonnenberger]

On Wed, Dec 05, 2007 at 10:24:56PM +0100, Hubert Feyrer wrote:
> TCP buffers too small, and other values not tuned, see [1]? Though IIRC
> there was something about auto-tuning those buffers, but I don't know if
> that's enabled by default (or how to enable it if not).

I don't think that code is in NetBSD 4.

[Joerg Sonnenberger]

On Wed, Dec 05, 2007 at 09:56:27PM +0100, Klaus Heinz wrote:
> IMO you can ignore the German text, the table below "Hardware" speaks for
> itself.

Well, given that FreeBSD and OpenBSD are using the Intel driver, it
could simply be some missing optimising setting in wm(4). Exactly what
revisions are those cards? Beside the checksum settings, I would also
take a look at the interrupt rate.

Joerg

[Klaus Heinz-27]

Hubert Feyrer wrote:

> TCP buffers too small, and other values not tuned, see [1]? Though IIRC
> there was something about auto-tuning those buffers, but I don't know if
> that's enabled by default (or how to enable it if not).

I would have expected NetBSD to achieve wirespeed with a single Gigabit
NIC out of the box, nowadays. The blog entry does not mention tuning any
of the other operating systems, so I suppose no tuning was done.

ciao
     Klaus

[Havard Eidnes]

> TCP buffers too small, and other values not tuned, see [1]?

It could also be differences in parameter settings for interrupt
coalescing.

> Though IIRC there was something about auto-tuning those buffers, but
> I don't know if that's enabled by default (or how to enable it if
> not).

On a sufficiently new kernel, you could put this in /etc/sysctl.conf

net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1

to enable TCP window auto-tuning.  This functionality arrived in
NetBSD-current on August 7 this year, and will not be in 4.0.

Regards,

- Håvard

[Havard Eidnes-2]

Replying to myself:

> On a sufficiently new kernel, you could put this in /etc/sysctl.conf
>
> net.inet.tcp.sendbuf_auto=1
> net.inet.tcp.recvbuf_auto=1
>
> to enable TCP window auto-tuning.  This functionality arrived in
> NetBSD-current on August 7 this year, and will not be in 4.0.

Err... August 2, but what's a small week between friends? :-)

- Håvard

[Pavel Cahyna-4]

On Wed, Dec 05, 2007 at 10:24:56PM +0100, Hubert Feyrer wrote:
> On Wed, 5 Dec 2007, Klaus Heinz wrote:
> >Any opinions why NetBSD does not reach (near-) wirespeed with those Intel
> >NICs as the others do and at the same time needs 100% CPU?
>
> TCP buffers too small, and other values not tuned, see [1]? Though IIRC there
> was something about auto-tuning those buffers, but I don't know if that's
> enabled by default (or how to enable it if not).
>
> [1] http://proj.sunet.se/LSR2/

The test is for routing and pf, so TCP buffers should not matter.

Pavel

[Klaus Heinz-27]

Joerg Sonnenberger wrote:

> Well, given that FreeBSD and OpenBSD are using the Intel driver, it
> could simply be some missing optimising setting in wm(4). Exactly what

It looks like the wm driver does not automatically enable supported
options like TCP segmentation and checksum computing. The author of the
comparison will probably take a look at that, although his main interest
lies with Free-/OpenBSD.

ciao
     Klaus

[Manuel Bouyer]

On Wed, Dec 05, 2007 at 11:35:45PM +0100, Klaus Heinz wrote:
> Hubert Feyrer wrote:
>
> > TCP buffers too small, and other values not tuned, see [1]? Though IIRC
> > there was something about auto-tuning those buffers, but I don't know if
> > that's enabled by default (or how to enable it if not).
>
> I would have expected NetBSD to achieve wirespeed with a single Gigabit
> NIC out of the box, nowadays. The blog entry does not mention tuning any
> of the other operating systems, so I suppose no tuning was done.

Maybe we should have offload turned on by default. It looks like all other
OSes have it on by default these days ... at last it's easier to turn
off with NetBSD than with linux :)

--
Manuel Bouyer <bouyer@...>
     NetBSD: 26 ans d'experience feront toujours la difference
--

[Pavel Cahyna-4]

On Fri, Dec 07, 2007 at 10:55:31PM +0100, Manuel Bouyer wrote:

> On Wed, Dec 05, 2007 at 11:35:45PM +0100, Klaus Heinz wrote:
> > Hubert Feyrer wrote:
> >
> > > TCP buffers too small, and other values not tuned, see [1]? Though IIRC
> > > there was something about auto-tuning those buffers, but I don't know if
> > > that's enabled by default (or how to enable it if not).
> >
> > I would have expected NetBSD to achieve wirespeed with a single Gigabit
> > NIC out of the box, nowadays. The blog entry does not mention tuning any
> > of the other operating systems, so I suppose no tuning was done.
>
> Maybe we should have offload turned on by default. It looks like all other
> OSes have it on by default these days ... at last it's easier to turn
> off with NetBSD than with linux :)

Does the "em" driver in Free/OpenBSD turn it on by default?

And is it likely to make a big difference since what matters here are only
IP header checksums (not TCP/UDP checksums as it is a test of forwarding,
not send/receive)?

Pavel

[Joerg Sonnenberger]

On Sat, Dec 08, 2007 at 12:41:00PM +0100, Pavel Cahyna wrote:
> And is it likely to make a big difference since what matters here are only
> IP header checksums (not TCP/UDP checksums as it is a test of forwarding,
> not send/receive)?

I was more thinking about RX/TX interrupt moderation and that can make a
huge difference in terms of load/packet.

Joerg