Test on 10GBE Intel based network card
|
View:
New views
20 Messages
—
Rating Filter:
Alert me
|
| < Prev | 1 - 2 | Next > |
 |
Test on 10GBE Intel based network card
by Invernizzi Fabrizio
::
Rate this Message:
Hi all
I am doing some tests on a BSD system with a 10gbe Intel based network card and I have some doubts about the configuration since the performance I am experiencing looks (very) poor. This is the system I am doing test on: - HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed: 1333 MHz, L2 cache size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core installed. - Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR) 10 Gigabit Ethernet PCI Express Server Adapter Intel® based (chip 82598EB). Driver ixgbe-1.8.6 - FreeBSD 7.2-RELEASE (64 bit) with this options compiled in the kernel options ZERO_COPY_SOCKETS # Turn on zero copy send code options HZ=1000 options BPF_JITTER I worked on the driver settings in order to have big TX/RX rings and low interrupt rate (traffic latency is not an issue). In order to tune up the system i started with some echo request tests. These are the maximum Bandwidths I can send without loss: - 64 byte packets: 312 Mbps (1,64% CPU idle) - 512 byte packets: 2117 Mbps (1,63% CPU idle) - 1492 byte packets: 5525 Mbps (1,93% CPU idle) Am I right considering these figures lower than expected? The system is just managing network traffic! Now I have started with netgraph tests, in particular with ng_bpf and the overall system is going even worst. I sent some HTTP traffic (597 bytes-long packets) and I configured an ng_bpf to filter TCP traffic out from the incoming interface (ix0). If I use the ngctl msg to see counters on the ng_bpf node, I see extremely poor performance: - Sending 96Mbps of this traffic I figured out 0.1% packet loss. This looks very strange. May be some counter bug? - Sending 5500Mbps, the netgraph (not the network card driver) is loosing 21% of the number of sent packets. See below a snapshot of the CPU load under traffic load CPU: 0.0% user, 0.0% nice, 87.0% system, 9.1% interrupt, 3.9% idle Mem: 16M Active, 317M Inact, 366M Wired, 108K Cache, 399M Buf, 7222M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root 1 171 ki31 0K 16K RUN 2 20.2H 68.80% idle: cpu2 11 root 1 171 ki31 0K 16K RUN 3 20.1H 64.70% idle: cpu3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 64.26% idle: cpu0 13 root 1 171 ki31 0K 16K RUN 1 20.2H 63.67% idle: cpu1 38 root 1 -68 - 0K 16K CPU1 1 1:28 34.67% ix0 rxq 40 root 1 -68 - 0K 16K CPU2 0 1:26 34.18% ix0 rxq 34 root 1 -68 - 0K 16K CPU3 3 1:27 34.08% ix0 rxq 36 root 1 -68 - 0K 16K RUN 2 1:26 34.08% ix0 rxq 33 root 1 -68 - 0K 16K WAIT 3 0:40 4.05% irq260: ix0 39 root 1 -68 - 0K 16K WAIT 2 0:41 3.96% irq263: ix0 35 root 1 -68 - 0K 16K WAIT 0 0:39 3.66% irq261: ix0 37 root 1 -68 - 0K 16K WAIT 1 0:42 3.47% irq262: ix0 16 root 1 -32 - 0K 16K WAIT 0 14:53 2.49% swi4: clock sio Am I missing something? Does someone know some (more) system tuning to have higher traffic rate supported? Any help is greatly appreciated. Fabrizio ------------------------------------------------------------------ Telecom Italia Fabrizio INVERNIZZI Technology - TILAB Accesso Fisso e Trasporto Via Reiss Romoli, 274 10148 Torino Tel. +39 011 2285497 Mob. +39 3316001344 Fax +39 06 41867287 Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
Re: Test on 10GBE Intel based network card
by Stefan Lambrev-2
::
Rate this Message:
Hi,
The limitation that you see is about the max number of packets that FreeBSD can handle - it looks like your best performance is reached at 64 byte packets? Am I correct that the maximum you can reach is around 639,000 packets per second? Also you are not routing the traffic, but instead the server handles the requests itself and eat CPU to reply? On Aug 3, 2009, at 11:46 AM, Invernizzi Fabrizio wrote: > Hi all > > I am doing some tests on a BSD system with a 10gbe Intel based > network card and I have some doubts about the configuration since > the performance I am experiencing looks (very) poor. > > This is the system I am doing test on: > > > > - HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed: 1333 MHz, L2 > cache size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core > installed. > > - Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR) 10 Gigabit > Ethernet PCI Express Server Adapter Intel® based (chip 82598EB). > Driver ixgbe-1.8.6 > > - FreeBSD 7.2-RELEASE (64 bit) with this options compiled in the > kernel > options ZERO_COPY_SOCKETS # Turn on zero > copy send code > options HZ=1000 > options BPF_JITTER > > > > I worked on the driver settings in order to have big TX/RX rings and > low interrupt rate (traffic latency is not an issue). > > > > In order to tune up the system i started with some echo request tests. > > These are the maximum Bandwidths I can send without loss: > > - 64 byte packets: 312 Mbps (1,64% CPU idle) > > - 512 byte packets: 2117 Mbps (1,63% CPU idle) > > - 1492 byte packets: 5525 Mbps (1,93% CPU idle) > > > > Am I right considering these figures lower than expected? > The system is just managing network traffic! > > > > Now I have started with netgraph tests, in particular with ng_bpf > and the overall system is going even worst. > > I sent some HTTP traffic (597 bytes-long packets) and I configured > an ng_bpf to filter TCP traffic out from the incoming interface (ix0). > > If I use the ngctl msg to see counters on the ng_bpf node, I see > extremely poor performance: > > > > - Sending 96Mbps of this traffic I figured out 0.1% packet loss. > This looks very strange. May be some counter bug? > > - Sending 5500Mbps, the netgraph (not the network card driver) is > loosing 21% of the number of sent packets. See below a snapshot of > the CPU load under traffic load > > > > CPU: 0.0% user, 0.0% nice, 87.0% system, 9.1% interrupt, 3.9% idle > > Mem: 16M Active, 317M Inact, 366M Wired, 108K Cache, 399M Buf, 7222M > Free > > Swap: 2048M Total, 2048M Free > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 68.80% > idle: cpu2 > > 11 root 1 171 ki31 0K 16K RUN 3 20.1H 64.70% > idle: cpu3 > > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 64.26% > idle: cpu0 > > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 63.67% > idle: cpu1 > > 38 root 1 -68 - 0K 16K CPU1 1 1:28 34.67% > ix0 rxq > > 40 root 1 -68 - 0K 16K CPU2 0 1:26 34.18% > ix0 rxq > > 34 root 1 -68 - 0K 16K CPU3 3 1:27 34.08% > ix0 rxq > > 36 root 1 -68 - 0K 16K RUN 2 1:26 34.08% > ix0 rxq > > 33 root 1 -68 - 0K 16K WAIT 3 0:40 4.05% > irq260: ix0 > > 39 root 1 -68 - 0K 16K WAIT 2 0:41 3.96% > irq263: ix0 > > 35 root 1 -68 - 0K 16K WAIT 0 0:39 3.66% > irq261: ix0 > > 37 root 1 -68 - 0K 16K WAIT 1 0:42 3.47% > irq262: ix0 > > 16 root 1 -32 - 0K 16K WAIT 0 14:53 2.49% > swi4: clock sio > > > > > > > > Am I missing something? > > Does someone know some (more) system tuning to have higher traffic > rate supported? > > > > Any help is greatly appreciated. > > > > Fabrizio > > > > > > ------------------------------------------------------------------ > Telecom Italia > Fabrizio INVERNIZZI > Technology - TILAB > Accesso Fisso e Trasporto > Via Reiss Romoli, 274 10148 Torino > Tel. +39 011 2285497 > Mob. +39 3316001344 > Fax +39 06 41867287 > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente > alle persone indicate. La diffusione, copia o qualsiasi altra azione > derivante dalla conoscenza di queste informazioni sono rigorosamente > vietate. Qualora abbiate ricevuto questo documento per errore siete > cortesemente pregati di darne immediata comunicazione al mittente e > di provvedere alla sua distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain > privileged information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is > unauthorised. If you are not the intended recipient, please delete > this message and any attachments and advise the sender by return e- > mail, Thanks. > > _______________________________________________ > freebsd-performance@... mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@... > " -- Best Wishes, Stefan Lambrev ICQ# 24134177 _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
RE: Test on 10GBE Intel based network card
by Invernizzi Fabrizio
::
Rate this Message:
Hi
> -----Original Message----- > From: Stefan Lambrev [mailto:stefan.lambrev@...] > Sent: lunedì 3 agosto 2009 11.22 > To: Invernizzi Fabrizio > Cc: freebsd-performance@... > Subject: Re: Test on 10GBE Intel based network card > > Hi, > > The limitation that you see is about the max number of packets that > FreeBSD can handle - it looks like your best performance is reached at > 64 byte packets? If you are meaning in term of Packet per second, you are right. These are the packet per second measured during tests: 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps > Am I correct that the maximum you can reach is around 639,000 packets > per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? > Also you are not routing the traffic, but instead the server handles > the requests itself and eat CPU to reply? Correct. In these first tests I want to "tune" the system, so I am using the (let me say) worst scenario. > > _______________________________________________ > > freebsd-performance@... mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance- > unsubscribe@... > > " > > -- > Best Wishes, > Stefan Lambrev > ICQ# 24134177 > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
Re: Test on 10GBE Intel based network card
by István-3
::
Rate this Message:
what about your sysctls?
I would like to see what have you done yet. Here is a previous conversation about that: http://www.mail-archive.com/freebsd-performance@.../msg02293.html <http://www.mail-archive.com/freebsd-performance@.../msg02293.html>Actually I am not aware of the changes in 7.2 or what could be the best value set for you, worth to try to fine tune these. Regards, Istvan On Mon, Aug 3, 2009 at 10:53 AM, Invernizzi Fabrizio < fabrizio.invernizzi@...> wrote: > Hi > > > > -----Original Message----- > > From: Stefan Lambrev [mailto:stefan.lambrev@...] > > Sent: lunedì 3 agosto 2009 11.22 > > To: Invernizzi Fabrizio > > Cc: freebsd-performance@... > > Subject: Re: Test on 10GBE Intel based network card > > > > Hi, > > > > The limitation that you see is about the max number of packets that > > FreeBSD can handle - it looks like your best performance is reached at > > 64 byte packets? > > If you are meaning in term of Packet per second, you are right. These are > the packet per second measured during tests: > > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > > > Am I correct that the maximum you can reach is around 639,000 packets > > per second? > > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? > > > Also you are not routing the traffic, but instead the server handles > > the requests itself and eat CPU to reply? > > Correct. In these first tests I want to "tune" the system, so I am using > the (let me say) worst scenario. > > > > > _______________________________________________ > > > freebsd-performance@... mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > > To unsubscribe, send any mail to "freebsd-performance- > > unsubscribe@... > > > " > > > > -- > > Best Wishes, > > Stefan Lambrev > > ICQ# 24134177 > > > > > > > > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora > abbiate ricevuto questo documento per errore siete cortesemente pregati di > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privileged > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the intended > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > _______________________________________________ > freebsd-performance@... mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@..." > -- the sun shines for all _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
Re: Test on 10GBE Intel based network card
by Stefan Lambrev-2
::
Rate this Message:
Hi,
On Aug 3, 2009, at 12:53 PM, Invernizzi Fabrizio wrote: > Hi > > >> -----Original Message----- >> From: Stefan Lambrev [mailto:stefan.lambrev@...] >> Sent: lunedì 3 agosto 2009 11.22 >> To: Invernizzi Fabrizio >> Cc: freebsd-performance@... >> Subject: Re: Test on 10GBE Intel based network card >> >> Hi, >> >> The limitation that you see is about the max number of packets that >> FreeBSD can handle - it looks like your best performance is reached >> at >> 64 byte packets? > > If you are meaning in term of Packet per second, you are right. > These are the packet per second measured during tests: > > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > >> Am I correct that the maximum you can reach is around 639,000 packets >> per second? > > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? I duno - the tests I did before were with SYN packets (random source) which was my worst scenario, and the server CPU were busy generating MD5 check sums for "syncache" (around 35% of the time). If I have to compare my results with your, you beat me with factor 2.5, may be because you use ICMP for the test and your processor is better then my test stations :) Also my experience is only with gigabit cards (em driver) and FreeBSD 7.something_before_1 where the em thread was eating 100% cpu. If you are lucky LOCK_PROFILING(9) will help you to see where the CPUs spend their time, if not you will see kernel panic :) Once problematic locks identified they can be reworked, but I think the first part is already done and work on the second already started. In my experience increasing hw.em.rxd and hw.em.txd yelled better results, but I think ixgb already comes tuned by default as it still doesn't have to support such a large number of different cards. Also at the time of my tests there were not support for multi queues in the OS even if they were supported by the HW, which is changed in 7.2 (?) > >> Also you are not routing the traffic, but instead the server handles >> the requests itself and eat CPU to reply? > > Correct. In these first tests I want to "tune" the system, so I am > using the (let me say) worst scenario. > > >>> _______________________________________________ >>> freebsd-performance@... mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>> To unsubscribe, send any mail to "freebsd-performance- >> unsubscribe@... >>> " >> >> -- >> Best Wishes, >> Stefan Lambrev >> ICQ# 24134177 >> >> >> >> > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente > alle persone indicate. La diffusione, copia o qualsiasi altra azione > derivante dalla conoscenza di queste informazioni sono rigorosamente > vietate. Qualora abbiate ricevuto questo documento per errore siete > cortesemente pregati di darne immediata comunicazione al mittente e > di provvedere alla sua distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain > privileged information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is > unauthorised. If you are not the intended recipient, please delete > this message and any attachments and advise the sender by return e- > mail, Thanks. > -- Best Wishes, Stefan Lambrev ICQ# 24134177 _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
RE: Test on 10GBE Intel based network card
by Invernizzi Fabrizio
::
Rate this Message:
HI
These are the sysctl vars you pointed out kern.ipc.somaxconn: 128 net.inet.tcp.recvspace: 65536 net.inet.tcp.sendspace: 32768 kern.ipc.shmmax: 33554432 kern.ipc.shmmni: 192 kern.ipc.shmseg: 128 kern.ipc.semmni: 10 net.local.stream.sendspace: 8192 net.local.stream.recvspace: 8192 net.inet.tcp.local_slowstart_flightsize: 4 net.inet.tcp.nolocaltimewait: 0 net.inet.tcp.hostcache.expire: 3600 kern.maxusers: 384 kern.ipc.nmbclusters: 65635 kern.ipc.maxsockets: 65635 kern.ipc.maxsockbuf: 262144 net.inet.tcp.tcbhashsize: 512 net.inet.tcp.hostcache.hashsize: 512 Fabrizio ________________________________________ From: István [mailto:leccine@...] Sent: lunedì 3 agosto 2009 12.02 To: Invernizzi Fabrizio Cc: Stefan Lambrev; freebsd-performance@... Subject: Re: Test on 10GBE Intel based network card what about your sysctls? I would like to see what have you done yet. Here is a previous conversation about that: http://www.mail-archive.com/freebsd-performance@.../msg02293.html Actually I am not aware of the changes in 7.2 or what could be the best value set for you, worth to try to fine tune these. Regards, Istvan On Mon, Aug 3, 2009 at 10:53 AM, Invernizzi Fabrizio <fabrizio.invernizzi@...> wrote: Hi > -----Original Message----- > From: Stefan Lambrev [mailto:stefan.lambrev@...] > Sent: lunedì 3 agosto 2009 11.22 > To: Invernizzi Fabrizio > Cc: freebsd-performance@... > Subject: Re: Test on 10GBE Intel based network card > > Hi, > > The limitation that you see is about the max number of packets that > FreeBSD can handle - it looks like your best performance is reached at > 64 byte packets? 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps > Am I correct that the maximum you can reach is around 639,000 packets > per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? > Also you are not routing the traffic, but instead the server handles > the requests itself and eat CPU to reply? Correct. In these first tests I want to "tune" the system, so I am using the (let me say) worst scenario. > > _______________________________________________ > > freebsd-performance@... mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance- > unsubscribe@... > > " > > -- > Best Wishes, > Stefan Lambrev > ICQ# 24134177 > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." -- the sun shines for all Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
Re: Test on 10GBE Intel based network card
by István-3
::
Rate this Message:
Nice, I guess to analyse the bottlenecks you might want to use dtrace....If
it is working with 7.2, I am not sure about that. Regards, Istvan On Mon, Aug 3, 2009 at 11:17 AM, Invernizzi Fabrizio < fabrizio.invernizzi@...> wrote: > HI > > These are the sysctl vars you pointed out > > kern.ipc.somaxconn: 128 > net.inet.tcp.recvspace: 65536 > net.inet.tcp.sendspace: 32768 > kern.ipc.shmmax: 33554432 > kern.ipc.shmmni: 192 > kern.ipc.shmseg: 128 > kern.ipc.semmni: 10 > net.local.stream.sendspace: 8192 > net.local.stream.recvspace: 8192 > net.inet.tcp.local_slowstart_flightsize: 4 > net.inet.tcp.nolocaltimewait: 0 > net.inet.tcp.hostcache.expire: 3600 > > kern.maxusers: 384 > kern.ipc.nmbclusters: 65635 > kern.ipc.maxsockets: 65635 > kern.ipc.maxsockbuf: 262144 > net.inet.tcp.tcbhashsize: 512 > net.inet.tcp.hostcache.hashsize: 512 > > > Fabrizio > > ________________________________________ > From: István [mailto:leccine@...] > Sent: lunedì 3 agosto 2009 12.02 > To: Invernizzi Fabrizio > Cc: Stefan Lambrev; freebsd-performance@... > Subject: Re: Test on 10GBE Intel based network card > > what about your sysctls? > > I would like to see what have you done yet. > > Here is a previous conversation about that: > http://www.mail-archive.com/freebsd-performance@.../msg02293.html > > Actually I am not aware of the changes in 7.2 or what could be the best > value set for you, worth to try to fine tune these. > > Regards, > Istvan > On Mon, Aug 3, 2009 at 10:53 AM, Invernizzi Fabrizio < > fabrizio.invernizzi@...> wrote: > Hi > > > > -----Original Message----- > > From: Stefan Lambrev [mailto:stefan.lambrev@...] > > Sent: lunedì 3 agosto 2009 11.22 > > To: Invernizzi Fabrizio > > Cc: freebsd-performance@... > > Subject: Re: Test on 10GBE Intel based network card > > > > Hi, > > > > The limitation that you see is about the max number of packets that > > FreeBSD can handle - it looks like your best performance is reached at > > 64 byte packets? > If you are meaning in term of Packet per second, you are right. These are > the packet per second measured during tests: > > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > > > Am I correct that the maximum you can reach is around 639,000 packets > > per second? > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? > > > Also you are not routing the traffic, but instead the server handles > > the requests itself and eat CPU to reply? > Correct. In these first tests I want to "tune" the system, so I am using > the (let me say) worst scenario. > > > > > _______________________________________________ > > > freebsd-performance@... mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > > To unsubscribe, send any mail to "freebsd-performance- > > unsubscribe@... > > > " > > > > -- > > Best Wishes, > > Stefan Lambrev > > ICQ# 24134177 > > > > > > > > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora > abbiate ricevuto questo documento per errore siete cortesemente pregati di > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privileged > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the intended > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > _______________________________________________ > freebsd-performance@... mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@..." > > > > -- > the sun shines for all > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora > abbiate ricevuto questo documento per errore siete cortesemente pregati di > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privileged > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the intended > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > -- the sun shines for all _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
RE: Test on 10GBE Intel based network card
by Invernizzi Fabrizio
::
Rate this Message:
> > If you are meaning in term of Packet per second, you are right.
> > These are the packet per second measured during tests: > > > > 64 byte: 610119 Pps > > 512 byte: 516917 Pps > > 1492 byte: 464962 Pps > > > > > >> Am I correct that the maximum you can reach is around 639,000 packets > >> per second? > > > > Yes, as you can see the maximum is 610119 Pps. > > Where does this limit come from? > > I duno - the tests I did before were with SYN packets (random source) > which was my worst scenario, > and the server CPU were busy generating MD5 check sums for > "syncache" (around 35% of the time). > > If I have to compare my results with your, you beat me with factor > 2.5, may be because you use ICMP for the test > and your processor is better then my test stations :) > Also my experience is only with gigabit cards (em driver) and FreeBSD > 7.something_before_1 where the em thread was eating 100% cpu. > If you are lucky LOCK_PROFILING(9) will help you to see where the CPUs > spend their time, if not you will see kernel panic :) I will check, thanks for the hint. > Once problematic locks identified they can be reworked, but I think > the first part is already done > and work on the second already started. > > In my experience increasing hw.em.rxd and hw.em.txd yelled better > results, but I think ixgb already comes tuned by default > as it still doesn't have to support such a large number of different > cards. I did some tuning in the code of the driver e recompiled the kernel in order to reduce context switching (interrupt mitigation) since the driver does not support POLLING. > Also at the time of my tests there were not support for multi queues > in the OS even if they were supported by the HW, which is changed in > 7.2 (?) It looks like multi queue is working since I can see the load distributed over the 4 cores. Fabrizio Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
Re: Test on 10GBE Intel based network card
by Jack Vogel
::
Rate this Message:
If you go to FreeBSD 8 you will get the improved stack code, and the RX/TX
queue pairs will be pinned to cpus. It should improve performance. Make sure you have enough mbuf memory allocated, try increasing the descriptors. Jack On Mon, Aug 3, 2009 at 3:26 AM, Invernizzi Fabrizio < fabrizio.invernizzi@...> wrote: > > > If you are meaning in term of Packet per second, you are right. > > > These are the packet per second measured during tests: > > > > > > 64 byte: 610119 Pps > > > 512 byte: 516917 Pps > > > 1492 byte: 464962 Pps > > > > > > > > >> Am I correct that the maximum you can reach is around 639,000 packets > > >> per second? > > > > > > Yes, as you can see the maximum is 610119 Pps. > > > Where does this limit come from? > > > > I duno - the tests I did before were with SYN packets (random source) > > which was my worst scenario, > > and the server CPU were busy generating MD5 check sums for > > "syncache" (around 35% of the time). > > > > If I have to compare my results with your, you beat me with factor > > 2.5, may be because you use ICMP for the test > > and your processor is better then my test stations :) > > Also my experience is only with gigabit cards (em driver) and FreeBSD > > 7.something_before_1 where the em thread was eating 100% cpu. > > If you are lucky LOCK_PROFILING(9) will help you to see where the CPUs > > spend their time, if not you will see kernel panic :) > > > I will check, thanks for the hint. > > > Once problematic locks identified they can be reworked, but I think > > the first part is already done > > and work on the second already started. > > > > In my experience increasing hw.em.rxd and hw.em.txd yelled better > > results, but I think ixgb already comes tuned by default > > as it still doesn't have to support such a large number of different > > cards. > > I did some tuning in the code of the driver e recompiled the kernel in > order to reduce context switching (interrupt mitigation) since the driver > does not support POLLING. > > > Also at the time of my tests there were not support for multi queues > > in the OS even if they were supported by the HW, which is changed in > > 7.2 (?) > > It looks like multi queue is working since I can see the load distributed > over the 4 cores. > > > Fabrizio > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora > abbiate ricevuto questo documento per errore siete cortesemente pregati di > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privileged > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the intended > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > _______________________________________________ > freebsd-performance@... mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@..." > freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
Re: Test on 10GBE Intel based network card
by Julian Elischer
::
Rate this Message:
Invernizzi Fabrizio wrote:
> Hi > > >> -----Original Message----- >> From: Stefan Lambrev [mailto:stefan.lambrev@...] >> Sent: lunedì 3 agosto 2009 11.22 >> To: Invernizzi Fabrizio >> Cc: freebsd-performance@... >> Subject: Re: Test on 10GBE Intel based network card >> >> Hi, >> >> The limitation that you see is about the max number of packets that >> FreeBSD can handle - it looks like your best performance is reached at >> 64 byte packets? > > If you are meaning in term of Packet per second, you are right. These are the packet per second measured during tests: > > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > >> Am I correct that the maximum you can reach is around 639,000 packets >> per second? > > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? ah that's the whole point of tuning :-) there are severalpossibities: 1/ the card's interrupts are probably attache dto aonly 1 cpu, so that cpu can do no more work 2/ if more than 1 cpu is working, it may be that there is a lock in heavy contention somewhere. is the machine still responsive to other networks while running at maximum capacity on this network? (make sure that the other networks are on a differnet CPU (hmm I can't remember how to do that :-). > >> Also you are not routing the traffic, but instead the server handles >> the requests itself and eat CPU to reply? > > Correct. In these first tests I want to "tune" the system, so I am using the (let me say) worst scenario. > > _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
Re: Test on 10GBE Intel based network card
by Ray Kinsella
::
Rate this Message:
Hi all,
cpuset is the command to set a cpu affinity, there are details @ http://bramp.net/blog/post vmstat -z is the command you need to determine whether there is contention for mbufs. although the cpu usage does not suggest the system is memory constrained. Regards Ray Kinsella On Mon, Aug 3, 2009 at 4:20 PM, Jack Vogel <jfvogel@...> wrote: > If you go to FreeBSD 8 you will get the improved stack code, and the RX/TX > queue pairs > will be pinned to cpus. It should improve performance. > > Make sure you have enough mbuf memory allocated, try increasing the > descriptors. > > Jack > > > On Mon, Aug 3, 2009 at 3:26 AM, Invernizzi Fabrizio < > fabrizio.invernizzi@...> wrote: > > > > > If you are meaning in term of Packet per second, you are right. > > > > These are the packet per second measured during tests: > > > > > > > > 64 byte: 610119 Pps > > > > 512 byte: 516917 Pps > > > > 1492 byte: 464962 Pps > > > > > > > > > > > >> Am I correct that the maximum you can reach is around 639,000 > packets > > > >> per second? > > > > > > > > Yes, as you can see the maximum is 610119 Pps. > > > > Where does this limit come from? > > > > > > I duno - the tests I did before were with SYN packets (random source) > > > which was my worst scenario, > > > and the server CPU were busy generating MD5 check sums for > > > "syncache" (around 35% of the time). > > > > > > If I have to compare my results with your, you beat me with factor > > > 2.5, may be because you use ICMP for the test > > > and your processor is better then my test stations :) > > > Also my experience is only with gigabit cards (em driver) and FreeBSD > > > 7.something_before_1 where the em thread was eating 100% cpu. > > > If you are lucky LOCK_PROFILING(9) will help you to see where the CPUs > > > spend their time, if not you will see kernel panic :) > > > > > > I will check, thanks for the hint. > > > > > Once problematic locks identified they can be reworked, but I think > > > the first part is already done > > > and work on the second already started. > > > > > > In my experience increasing hw.em.rxd and hw.em.txd yelled better > > > results, but I think ixgb already comes tuned by default > > > as it still doesn't have to support such a large number of different > > > cards. > > > > I did some tuning in the code of the driver e recompiled the kernel in > > order to reduce context switching (interrupt mitigation) since the driver > > does not support POLLING. > > > > > Also at the time of my tests there were not support for multi queues > > > in the OS even if they were supported by the HW, which is changed in > > > 7.2 (?) > > > > It looks like multi queue is working since I can see the load distributed > > over the 4 cores. > > > > > > Fabrizio > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > > dalla conoscenza di queste informazioni sono rigorosamente vietate. > Qualora > > abbiate ricevuto questo documento per errore siete cortesemente pregati > di > > darne immediata comunicazione al mittente e di provvedere alla sua > > distruzione, Grazie. > > > > This e-mail and any attachments is confidential and may contain > privileged > > information intended for the addressee(s) only. Dissemination, copying, > > printing or use by anybody else is unauthorised. If you are not the > intended > > recipient, please delete this message and any attachments and advise the > > sender by return e-mail, Thanks. > > > > _______________________________________________ > > freebsd-performance@... mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to " > > freebsd-performance-unsubscribe@..." > > > _______________________________________________ > freebsd-performance@... mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@..." > freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
|
|
|
RE: Test on 10GBE Intel based network card
by Invernizzi Fabrizio
::
Rate this Message:
> >> The limitation that you see is about the max number of packets that
> >> FreeBSD can handle - it looks like your best performance is reached at > >> 64 byte packets? > > > > If you are meaning in term of Packet per second, you are right. These > are the packet per second measured during tests: > > > > 64 byte: 610119 Pps > > 512 byte: 516917 Pps > > 1492 byte: 464962 Pps > > > > > >> Am I correct that the maximum you can reach is around 639,000 packets > >> per second? > > > > Yes, as you can see the maximum is 610119 Pps. > > Where does this limit come from? > > ah that's the whole point of tuning :-) > there are severalpossibities: > 1/ the card's interrupts are probably attache dto aonly 1 cpu, > so that cpu can do no more work This seems not to be the problem. See below a top snapshot during a 64byte-long packet storm last pid: 8552; load averages: 0.40, 0.09, 0.03 up 0+20:36:58 09:40:29 124 processes: 13 running, 73 sleeping, 38 waiting CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cpu3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cpu0 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cpu2 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cpu1 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq .... It looks like the 4 rxq processes are bound to the 4 available cores with equal distribution. > 2/ if more than 1 cpu is working, it may be that there is a lock in > heavy contention somewhere. This I think is the problem. I am trying to understand how to 1- see where the heavy contention is (context switching? Some limiting setting?) 2- mitigate it > > is the machine still responsive to other networks while > running at maximum capacity on this network? (make sure that > the other networks are on a differnet CPU (hmm I can't remember how to > do that :-). Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
RE: Test on 10GBE Intel based network card
by Invernizzi Fabrizio
::
Rate this Message:
Ray,
>To me it looks like interrupt coalescing is not switched on for some reason. >Are you passing any parameters to the driver in boot.conf. This is my loader.conf kern.ipc.nmbclusters=65635 kern.hz=1000 net.bpf_jitter.enable=1 # net.graph.threads=32 # if_em_load="YES" # NETGRAPH TUNING net.graph.maxdata=1024 kern.ipc.somaxconn=4096 net.inet.tcp.recvspace=78840 net.inet.tcp.sendspace=78840 kern.ipc.shmmax=67108864 kern.ipc.shmmni=200 kern.ipc.shmseg=128 kern.ipc.semmni=70 net.local.stream.sendspace=82320 net.local.stream.recvspace=82320 net.inet.tcp.local_slowstart_flightsize=10 net.inet.tcp.nolocaltimewait=1 net.inet.tcp.hostcache.expire=3900 kern.maxusers=512 kern.ipc.nmbclusters=32768 kern.ipc.maxsockets=81920 kern.ipc.maxsockbuf=1048576 net.inet.tcp.tcbhashsize=4096 net.inet.tcp.hostcache.hashsize=1024 >Could you retest with vmstat switched on "vmstat 3" and send us the output. >I expect we are going to see alot of interrupts. Sending 535714 pps (64bytes-long) INTRUDER-64# vmstat 3 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 0 0 0 95420K 7203M 19 0 0 0 17 0 0 0 642 66 1078 0 2 98 0 0 0 95420K 7203M 0 0 0 0 0 0 2 0 18 65 527 0 0 100 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 18 67 527 0 0 100 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 17 64 525 0 0 100 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 31526 64 31402 0 87 13 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 36767 64 33320 0 99 1 0 0 0 95420K 7203M 423 0 0 0 406 0 0 0 36174 384 28107 0 99 1 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 36706 64 27043 0 99 1 0 0 0 95420K 7203M 0 0 0 0 0 0 0 0 34006 64 13117 0 91 9 2 0 0 95420K 7203M 0 0 0 0 0 0 0 0 17 64 550 0 1 99 0 0 0 95420K 7203M 0 0 0 0 3 0 3 0 19 68 507 0 0 100 dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 1.7.4 dev.ix.0.%driver: ix dev.ix.0.%location: slot=0 function=0 dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10c6 subvendor=0x8086 subdevice=0xa15f class=0x020000 dev.ix.0.%parent: pci3 dev.ix.0.stats: -1 dev.ix.0.debug: -1 dev.ix.0.flow_control: 0 dev.ix.0.enable_lro: 1 Adaptive Interrupt Mitigation is enabled: dev.ix.0.enable_aim: 1 I did not changed AIM settings since is quite complex to tune them up. I tried to reverse engineering the algorithm of AIM (see attached picture) but I can't obtain tangible improvements playing with these paramenters. My understanding is that I should reduce the low_latency, but it seems not to work. dev.ix.0.low_latency: 128 dev.ix.0.ave_latency: 400 dev.ix.0.bulk_latency: 1200 Not sure about this dev.ix.0.hdr_split: 0 Not sure about the meaning of dev.ix.0.rx_processing_limit: 100 These are the settings I am using in the ixgbe driver: #define DEFAULT_TXD 1024 #define PERFORM_TXD 2048 #define MAX_TXD 4096 #define MIN_TXD 64 #define DEFAULT_RXD 1024 #define PERFORM_RXD 2048 #define MAX_RXD 4096 #define MIN_RXD 64 #define IXGBE_TX_CLEANUP_THRESHOLD (adapter->num_tx_desc / 1) #define IXGBE_TX_OP_THRESHOLD (adapter->num_tx_desc / 4) I see that I had a good performance improvement setting IXGBE_TX_CLEANUP_THRESHOLD to the tx queue size. This is clear to understand since this (greatly) minimizes context switching in send process reducing the number of time the txq function is called. (Of course with this settings send latency is increased, but this is not an issue). I can't understand the meaning of these parameters and if they could help: /* * This parameter controls the maximum no of times the driver will loop in * the isr. Minimum Value = 1 */ #define MAX_LOOP 10 /* * This parameter controls the duration of transmit watchdog timer. */ #define IXGBE_TX_TIMEOUT 5 /* set to 5 seconds */ Fabrizio Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
RE: Test on 10GBE Intel based network card
by Invernizzi Fabrizio
::
Rate this Message:
Hi all.
Going on with my tests. I noticed that I have always a (received) pps/interrupt ratio(per second) that is always about 14 (a very low ratio in my opinion). If I disable aim via sysctl sysctl -w dev.ix.0.enable_aim=0 dev.ix.0.enable_aim: 1 -> 0 this ration remain the same. Something going wrong with interrupt coalescing, but I can't find out what. Any idea? Fabrizio > -----Original Message----- > From: owner-freebsd-performance@... [mailto:owner-freebsd- > performance@...] On Behalf Of Invernizzi Fabrizio > Sent: martedì 4 agosto 2009 9.55 > To: Julian Elischer > Cc: freebsd-performance@...; Stefan Lambrev > Subject: RE: Test on 10GBE Intel based network card > > > >> The limitation that you see is about the max number of packets that > > >> FreeBSD can handle - it looks like your best performance is reached > at > > >> 64 byte packets? > > > > > > If you are meaning in term of Packet per second, you are right. These > > are the packet per second measured during tests: > > > > > > 64 byte: 610119 Pps > > > 512 byte: 516917 Pps > > > 1492 byte: 464962 Pps > > > > > > > > >> Am I correct that the maximum you can reach is around 639,000 packets > > >> per second? > > > > > > Yes, as you can see the maximum is 610119 Pps. > > > Where does this limit come from? > > > > ah that's the whole point of tuning :-) > > there are severalpossibities: > > 1/ the card's interrupts are probably attache dto aonly 1 cpu, > > so that cpu can do no more work > > This seems not to be the problem. See below a top snapshot during a > 64byte-long packet storm > > last pid: 8552; load averages: 0.40, 0.09, 0.03 > up 0+20:36:58 09:40:29 > 124 processes: 13 running, 73 sleeping, 38 waiting > CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle > Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: > cpu3 > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: > cpu0 > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: > cpu2 > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: > cpu1 > 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 > rxq > 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 > rxq > 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 > rxq > 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 > rxq > .... > > It looks like the 4 rxq processes are bound to the 4 available cores with > equal distribution. > > > > > 2/ if more than 1 cpu is working, it may be that there is a lock in > > heavy contention somewhere. > > This I think is the problem. I am trying to understand how to > 1- see where the heavy contention is (context switching? Some limiting > setting?) > 2- mitigate it > > > > > is the machine still responsive to other networks while > > running at maximum capacity on this network? (make sure that > > the other networks are on a differnet CPU (hmm I can't remember how to > > do that :-). > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. > Qualora abbiate ricevuto questo documento per errore siete cortesemente > pregati di darne immediata comunicazione al mittente e di provvedere alla > sua distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privileged > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the > intended recipient, please delete this message and any attachments and > advise the sender by return e-mail, Thanks. > > _______________________________________________ > freebsd-performance@... mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance- > unsubscribe@..." Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
Re: Test on 10GBE Intel based network card
by Julian Elischer
::
Rate this Message:
Invernizzi Fabrizio wrote:
>>>> The limitation that you see is about the max number of packets that >>>> FreeBSD can handle - it looks like your best performance is reached at >>>> 64 byte packets? >>> If you are meaning in term of Packet per second, you are right. These >> are the packet per second measured during tests: >>> 64 byte: 610119 Pps >>> 512 byte: 516917 Pps >>> 1492 byte: 464962 Pps >>> >>> >>>> Am I correct that the maximum you can reach is around 639,000 packets >>>> per second? >>> Yes, as you can see the maximum is 610119 Pps. >>> Where does this limit come from? >> ah that's the whole point of tuning :-) >> there are severalpossibities: >> 1/ the card's interrupts are probably attache dto aonly 1 cpu, >> so that cpu can do no more work > > This seems not to be the problem. See below a top snapshot during a 64byte-long packet storm > > last pid: 8552; load averages: 0.40, 0.09, 0.03 up 0+20:36:58 09:40:29 > 124 processes: 13 running, 73 sleeping, 38 waiting > CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle > Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cpu3 > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cpu0 > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cpu2 > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cpu1 > 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq > 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq > 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq > 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq > .... > > It looks like the 4 rxq processes are bound to the 4 available cores with equal distribution. > > > >> 2/ if more than 1 cpu is working, it may be that there is a lock in >> heavy contention somewhere. > > This I think is the problem. I am trying to understand how to > 1- see where the heavy contention is (context switching? Some limiting setting?) > 2- mitigate it > there ia a lock profiling tool that right now I can't remember the name of.. look it up with google :-) FreeBSD lock profiling tool ah, first hit... http://blogs.epfl.ch/article/23832 >> is the machine still responsive to other networks while >> running at maximum capacity on this network? (make sure that >> the other networks are on a differnet CPU (hmm I can't remember how to >> do that :-). > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
Re: Test on 10GBE Intel based network card
by Jack Vogel
::
Rate this Message:
Your nmbclusters is very low, you list it twice so I'm assuming the second
value is what it ends up being, 32K :( I would set it to: kern.ipc.nmbclusters=262144 Also, I thought you were using the current driver, but now it looks like you are using something fairly old, use my latest code which is 1.8.8 Jack On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer <julian@...> wrote: > Invernizzi Fabrizio wrote: > >> The limitation that you see is about the max number of packets that >>>>> FreeBSD can handle - it looks like your best performance is reached at >>>>> 64 byte packets? >>>>> >>>> If you are meaning in term of Packet per second, you are right. These >>>> >>> are the packet per second measured during tests: >>> >>>> 64 byte: 610119 Pps >>>> 512 byte: 516917 Pps >>>> 1492 byte: 464962 Pps >>>> >>>> >>>> Am I correct that the maximum you can reach is around 639,000 packets >>>>> per second? >>>>> >>>> Yes, as you can see the maximum is 610119 Pps. >>>> Where does this limit come from? >>>> >>> ah that's the whole point of tuning :-) >>> there are severalpossibities: >>> 1/ the card's interrupts are probably attache dto aonly 1 cpu, >>> so that cpu can do no more work >>> >> >> This seems not to be the problem. See below a top snapshot during a >> 64byte-long packet storm >> >> last pid: 8552; load averages: 0.40, 0.09, 0.03 >> up >> 0+20:36:58 09:40:29 >> 124 processes: 13 running, 73 sleeping, 38 waiting >> CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle >> Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free >> Swap: 2048M Total, 2048M Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND >> 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: >> cpu3 >> 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: >> cpu0 >> 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: >> cpu2 >> 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: >> cpu1 >> 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq >> 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq >> 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq >> 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq >> .... >> >> It looks like the 4 rxq processes are bound to the 4 available cores with >> equal distribution. >> >> >> >> 2/ if more than 1 cpu is working, it may be that there is a lock in >>> heavy contention somewhere. >>> >> >> This I think is the problem. I am trying to understand how to >> 1- see where the heavy contention is (context switching? Some limiting >> setting?) >> 2- mitigate it >> >> > > there ia a lock profiling tool that right now I can't remember the name > of.. > > look it up with google :-) FreeBSD lock profiling tool > > ah, first hit... > > http://blogs.epfl.ch/article/23832 > > > > is the machine still responsive to other networks while >>> running at maximum capacity on this network? (make sure that >>> the other networks are on a differnet CPU (hmm I can't remember how to >>> do that :-). >>> >> >> >> >> Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle >> persone indicate. La diffusione, copia o qualsiasi altra azione derivante >> dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora >> abbiate ricevuto questo documento per errore siete cortesemente pregati di >> darne immediata comunicazione al mittente e di provvedere alla sua >> distruzione, Grazie. >> >> This e-mail and any attachments is confidential and may contain privileged >> information intended for the addressee(s) only. Dissemination, copying, >> printing or use by anybody else is unauthorised. If you are not the intended >> recipient, please delete this message and any attachments and advise the >> sender by return e-mail, Thanks. >> > > _______________________________________________ > freebsd-performance@... mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@..." > freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
RE: Test on 10GBE Intel based network card
by Invernizzi Fabrizio
::
Rate this Message:
Hi Jack,
Now I have upgraded the driver to 1.8.6 (official intel last release), but no better performance. I am about to test with the settings your are suggesting. Using the same configuation i used before, i see a reduction on interrupts (from 1 for 14 packets in 1.7.6 to 1 for 40 packets in 1.8.6). This leades to a 5% to 10% of free CPU, but no higer packet rate. I tried working with the AIM settings, but no luck. Where can i download your 1.8.8 driver version? Fabrizio ________________________________ From: Jack Vogel [mailto:jfvogel@...] Sent: martedì 4 agosto 2009 18.42 To: Julian Elischer Cc: Invernizzi Fabrizio; freebsd-performance@...; Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card Your nmbclusters is very low, you list it twice so I'm assuming the second value is what it ends up being, 32K :( I would set it to: kern.ipc.nmbclusters=262144 Also, I thought you were using the current driver, but now it looks like you are using something fairly old, use my latest code which is 1.8.8 Jack On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer <julian@...<mailto:julian@...>> wrote: Invernizzi Fabrizio wrote: The limitation that you see is about the max number of packets that FreeBSD can handle - it looks like your best performance is reached at 64 byte packets? If you are meaning in term of Packet per second, you are right. These are the packet per second measured during tests: 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps Am I correct that the maximum you can reach is around 639,000 packets per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? ah that's the whole point of tuning :-) there are severalpossibities: 1/ the card's interrupts are probably attache dto aonly 1 cpu, so that cpu can do no more work This seems not to be the problem. See below a top snapshot during a 64byte-long packet storm last pid: 8552; load averages: 0.40, 0.09, 0.03 up 0+20:36:58 09:40:29 124 processes: 13 running, 73 sleeping, 38 waiting CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cpu3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cpu0 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cpu2 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cpu1 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq .... It looks like the 4 rxq processes are bound to the 4 available cores with equal distribution. 2/ if more than 1 cpu is working, it may be that there is a lock in heavy contention somewhere. This I think is the problem. I am trying to understand how to 1- see where the heavy contention is (context switching? Some limiting setting?) 2- mitigate it there ia a lock profiling tool that right now I can't remember the name of.. look it up with google :-) FreeBSD lock profiling tool ah, first hit... http://blogs.epfl.ch/article/23832 is the machine still responsive to other networks while running at maximum capacity on this network? (make sure that the other networks are on a differnet CPU (hmm I can't remember how to do that :-). Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@...<mailto:freebsd-performance@...> mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@...<mailto:freebsd-performance-unsubscribe@...>" Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. [cid:00000000000000000000000000000001@...]Rispetta l'ambiente. Non stampare questa mail se non è necessario. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
RE: Test on 10GBE Intel based network card
by Invernizzi Fabrizio
::
Rate this Message:
No improvement with kern.ipc.nmbclusters=262144 and 1.8.6 driver :<(((((
++fabrizio ------------------------------------------------------------------ Telecom Italia Fabrizio INVERNIZZI Technology - TILAB Accesso Fisso e Trasporto Via Reiss Romoli, 274 10148 Torino Tel. +39 011 2285497 Mob. +39 3316001344 Fax +39 06 41867287 ________________________________ From: Jack Vogel [mailto:jfvogel@...] Sent: martedì 4 agosto 2009 18.42 To: Julian Elischer Cc: Invernizzi Fabrizio; freebsd-performance@...; Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card Your nmbclusters is very low, you list it twice so I'm assuming the second value is what it ends up being, 32K :( I would set it to: kern.ipc.nmbclusters=262144 Also, I thought you were using the current driver, but now it looks like you are using something fairly old, use my latest code which is 1.8.8 Jack On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer <julian@...<mailto:julian@...>> wrote: Invernizzi Fabrizio wrote: The limitation that you see is about the max number of packets that FreeBSD can handle - it looks like your best performance is reached at 64 byte packets? If you are meaning in term of Packet per second, you are right. These are the packet per second measured during tests: 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps Am I correct that the maximum you can reach is around 639,000 packets per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? ah that's the whole point of tuning :-) there are severalpossibities: 1/ the card's interrupts are probably attache dto aonly 1 cpu, so that cpu can do no more work This seems not to be the problem. See below a top snapshot during a 64byte-long packet storm last pid: 8552; load averages: 0.40, 0.09, 0.03 up 0+20:36:58 09:40:29 124 processes: 13 running, 73 sleeping, 38 waiting CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cpu3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cpu0 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cpu2 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cpu1 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq .... It looks like the 4 rxq processes are bound to the 4 available cores with equal distribution. 2/ if more than 1 cpu is working, it may be that there is a lock in heavy contention somewhere. This I think is the problem. I am trying to understand how to 1- see where the heavy contention is (context switching? Some limiting setting?) 2- mitigate it there ia a lock profiling tool that right now I can't remember the name of.. look it up with google :-) FreeBSD lock profiling tool ah, first hit... http://blogs.epfl.ch/article/23832 is the machine still responsive to other networks while running at maximum capacity on this network? (make sure that the other networks are on a differnet CPU (hmm I can't remember how to do that :-). Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@...<mailto:freebsd-performance@...> mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@...<mailto:freebsd-performance-unsubscribe@...>" Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. [cid:00000000000000000000000000000001@...]Rispetta l'ambiente. Non stampare questa mail se non è necessario. _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
|
|
Re[2]: Test on 10GBE Intel based network card
by Anthony Pankov
::
Rate this Message:
Hello Invernizzi,
First of all I didn't catch for what task you are tuning. For forwarding or for transfer? In any case you should start with minimal configuration (no netgraph modules, no any firewall). Choose a test similiar to your typical load. Provide us with test result, cpu consumption and output of ifconfig netstat -s netstat -m debug values from driver (sysctl (?).debug)). Wednesday, August 05, 2009, 2:13:23 PM, you wrote: IF> No improvement with kern.ipc.nmbclusters=262144 and 1.8.6 driver :<((((( IF> ++fabrizio IF> ------------------------------------------------------------------ IF> Telecom Italia IF> Fabrizio INVERNIZZI IF> Technology - TILAB IF> Accesso Fisso e Trasporto IF> Via Reiss Romoli, 274 10148 Torino IF> Tel. +39 011 2285497 IF> Mob. +39 3316001344 IF> Fax +39 06 41867287 IF> ________________________________ IF> From: Jack Vogel [mailto:jfvogel@...] IF> Sent: martedì 4 agosto 2009 18.42 IF> To: Julian Elischer IF> Cc: Invernizzi Fabrizio; freebsd-performance@...; Stefan Lambrev IF> Subject: Re: Test on 10GBE Intel based network card IF> Your nmbclusters is very low, you list it twice so I'm assuming the second value is IF> what it ends up being, 32K :( IF> I would set it to: IF> kern.ipc.nmbclusters=262144 IF> Also, I thought you were using the current driver, but now it looks like you are IF> using something fairly old, use my latest code which is 1.8.8 IF> Jack IF> On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer <julian@...<mailto:julian@...>> wrote: IF> Invernizzi Fabrizio wrote: IF> The limitation that you see is about the max number of packets that IF> FreeBSD can handle - it looks like your best performance is reached at IF> 64 byte packets? IF> If you are meaning in term of Packet per second, you are right. These IF> are the packet per second measured during tests: IF> 64 byte: 610119 Pps IF> 512 byte: 516917 Pps IF> 1492 byte: 464962 Pps IF> Am I correct that the maximum you can reach is around 639,000 packets IF> per second? IF> Yes, as you can see the maximum is 610119 Pps. IF> Where does this limit come from? IF> ah that's the whole point of tuning :-) IF> there are severalpossibities: IF> 1/ the card's interrupts are probably attache dto aonly 1 cpu, IF> so that cpu can do no more work IF> This seems not to be the problem. See below a top snapshot during a 64byte-long packet storm IF> last pid: 8552; load averages: 0.40, 0.09, 0.03 up 0+20:36:58 09:40:29 IF> 124 processes: 13 running, 73 sleeping, 38 waiting IF> CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle IF> Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free IF> Swap: 2048M Total, 2048M Free IF> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND IF> 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cpu3 IF> 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cpu0 IF> 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cpu2 IF> 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cpu1 IF> 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq IF> 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq IF> 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq IF> 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq IF> .... IF> It looks like the 4 rxq processes are bound to the 4 available cores with equal distribution. IF> 2/ if more than 1 cpu is working, it may be that there is a lock in IF> heavy contention somewhere. IF> This I think is the problem. I am trying to understand how to IF> 1- see where the heavy contention is (context switching? Some limiting setting?) IF> 2- mitigate it IF> there ia a lock profiling tool that right now I can't remember the name of.. IF> look it up with google :-) FreeBSD lock profiling tool IF> ah, first hit... IF> http://blogs.epfl.ch/article/23832 IF> is the machine still responsive to other networks while IF> running at maximum capacity on this network? (make sure that IF> the other networks are on a differnet CPU (hmm I can't remember how to IF> do that :-). -- Best regards, Anthony mailto:ap00@... _______________________________________________ freebsd-performance@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..." |
| < Prev | 1 - 2 | Next > |
| Free embeddable forum powered by Nabble | Forum Help |
