dummynet dropping too many packets

[rihad]

Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell
PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP
users online limited by dummynet pipes of various speeds. According to
netstat -s output around 500-1000 packets are being dropped every second
(this accounts for wasting around 7-12 mbit/s worth of traffic according
to systat -ifstat):

# while :; do netstat -z -s 2>/dev/null | fgrep -w "output packets
dropped"; sleep 1; done
         16824 output packets dropped due to no bufs, etc.
         548 output packets dropped due to no bufs, etc.
         842 output packets dropped due to no bufs, etc.
         709 output packets dropped due to no bufs, etc.
         652 output packets dropped due to no bufs, etc.
^C

Pipes have been created like this:
ipfw pipe  1024 config bw 1024kbit/s mask dst-ip 0xffffffff queue 350KBytes
etc., and then assigned to users by application (ipfw tablearg).

I've tried playing with the queue setting, from as little as 1 slot to
as much as 4096KBytes - packets are still being dropped, more or less.
Should I somehow calculate the proper queue value for the given pipe
width? The manpage says 50 slots is typical for Ethernet devices (not
mentioning whether it's 10, 100 or 1000 mbit/s), and that's it.

sysctls:
kern.ipc.nmbclusters=50000
net.inet.ip.dummynet.io_fast=1

Polling can't be enabled with bce.

Any hints? Should I provide any further info?

Thanks.
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Luigi Rizzo-5]

On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:
> Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell
> PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP
> users online limited by dummynet pipes of various speeds. According to
> netstat -s output around 500-1000 packets are being dropped every second
> (this accounts for wasting around 7-12 mbit/s worth of traffic according
> to systat -ifstat):

what kind of packets are you seeing as dropped ?
please give the output of 'netstat -s output | grep drop'

At those speeds you might be hitting various limits with your
config (e.g. 50k nmbclusters is probably way too small for
4k users -- means you have an average of 10-15 buffers per user;
the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering,
which is quite high besides the fact that in order to scale to 4k users
you would need over 1GB of kernel memory just for the buffers).

I'd most likely suspect the nmbclusters argument.
netstat -m might give you some useful stats.

cheers
luigi

> # while :; do netstat -z -s 2>/dev/null | fgrep -w "output packets
> dropped"; sleep 1; done
>         16824 output packets dropped due to no bufs, etc.
>         548 output packets dropped due to no bufs, etc.
>         842 output packets dropped due to no bufs, etc.
>         709 output packets dropped due to no bufs, etc.
>         652 output packets dropped due to no bufs, etc.
> ^C
>
> Pipes have been created like this:
> ipfw pipe  1024 config bw 1024kbit/s mask dst-ip 0xffffffff queue 350KBytes
> etc., and then assigned to users by application (ipfw tablearg).
>
> I've tried playing with the queue setting, from as little as 1 slot to
> as much as 4096KBytes - packets are still being dropped, more or less.
> Should I somehow calculate the proper queue value for the given pipe
> width? The manpage says 50 slots is typical for Ethernet devices (not
> mentioning whether it's 10, 100 or 1000 mbit/s), and that's it.
>
> sysctls:
> kern.ipc.nmbclusters=50000
> net.inet.ip.dummynet.io_fast=1
>
> Polling can't be enabled with bce.
>
> Any hints? Should I provide any further info?
>
> Thanks.
> _______________________________________________
> freebsd-net@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[rihad]

Luigi Rizzo wrote:

> On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:
>> Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell
>> PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP
>> users online limited by dummynet pipes of various speeds. According to
>> netstat -s output around 500-1000 packets are being dropped every second
>> (this accounts for wasting around 7-12 mbit/s worth of traffic according
>> to systat -ifstat):
>
> what kind of packets are you seeing as dropped ?
> please give the output of 'netstat -s output | grep drop'
>

The output packets, like I said:
# netstat -s output | grep drop
         2 connections closed (including 2 drops)
         0 embryonic connections dropped
                 2 connections dropped by rexmit timeout
                 0 connections dropped by persist timeout
         0 Connections (fin_wait_2) dropped because of timeout
                 0 connections dropped by keepalive
                 0 dropped
         2 dropped due to no socket
         0 dropped due to full socket buffers
         0 fragments dropped (dup or out of space)
         2 fragments dropped after timeout
         7538 output packets dropped due to no bufs, etc.

The statistics are zeroed every 15 seconds in another window as I'm
investigating the issue, but the rate is around 500-1000 lost packets
every second at the current ~530 mbit/s load.

> At those speeds you might be hitting various limits with your
> config (e.g. 50k nmbclusters is probably way too small for

I bet it isn't:
1967/5009/6976/50000 mbuf clusters in use (current/cache/total/max)

> 4k users -- means you have an average of 10-15 buffers per user;
> the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering,
> which is quite high besides the fact that in order to scale to 4k users
> you would need over 1GB of kernel memory just for the buffers).
Aha. Can you be more specific about the kernel memory stuff? Which
setting needs tweaking?


I have another similar box with 2 em GigE interfaces working @220-230
mbit/s and virtually no out-of-bufs dropped packets as with bce @500-600
mbit. It too has 350KBytes dummynet queue sizes. And it too has adequate
mbuf load:
3071/10427/13498/25600 mbuf clusters in use (current/cache/total/max)
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[rihad]

Luigi Rizzo wrote:

> On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:
>> Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell
>> PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP
>> users online limited by dummynet pipes of various speeds. According to
>> netstat -s output around 500-1000 packets are being dropped every second
>> (this accounts for wasting around 7-12 mbit/s worth of traffic according
>> to systat -ifstat):
>
> At those speeds you might be hitting various limits with your
> config (e.g. 50k nmbclusters is probably way too small for
> 4k users -- means you have an average of 10-15 buffers per user;
> the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering,
> which is quite high besides the fact that in order to scale to 4k users
> you would need over 1GB of kernel memory just for the buffers).

top output:
Mem: 2037M Active, 1248M Inact, 450M Wired, 184M Cache, 214M Buf, 17M Free

I guess we're quite far from reaching 1GB of kernel memory.
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Luigi Rizzo-5]

On Sun, Oct 04, 2009 at 07:55:57PM +0500, rihad wrote:

> Luigi Rizzo wrote:
> >On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:
> >>Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell
> >>PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP
> >>users online limited by dummynet pipes of various speeds. According to
> >>netstat -s output around 500-1000 packets are being dropped every second
> >>(this accounts for wasting around 7-12 mbit/s worth of traffic according
> >>to systat -ifstat):
> >
> >At those speeds you might be hitting various limits with your
> >config (e.g. 50k nmbclusters is probably way too small for
> >4k users -- means you have an average of 10-15 buffers per user;
> >the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering,
> >which is quite high besides the fact that in order to scale to 4k users
> >you would need over 1GB of kernel memory just for the buffers).
>
>
> top output:
> Mem: 2037M Active, 1248M Inact, 450M Wired, 184M Cache, 214M Buf, 17M Free
>
> I guess we're quite far from reaching 1GB of kernel memory.

of course, you'd have to configure also 500k nmbclusters (and then
probably this would not fit)

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Luigi Rizzo-5]

On Sun, Oct 04, 2009 at 07:53:23PM +0500, rihad wrote:

> Luigi Rizzo wrote:
> >On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:
> >>Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell
> >>PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP
> >>users online limited by dummynet pipes of various speeds. According to
> >>netstat -s output around 500-1000 packets are being dropped every second
> >>(this accounts for wasting around 7-12 mbit/s worth of traffic according
> >>to systat -ifstat):
> >
> >what kind of packets are you seeing as dropped ?
> >please give the output of 'netstat -s output | grep drop'
> >
> The output packets, like I said:
> # netstat -s output | grep drop
>         2 connections closed (including 2 drops)
>         0 embryonic connections dropped
>                 2 connections dropped by rexmit timeout
>                 0 connections dropped by persist timeout
>         0 Connections (fin_wait_2) dropped because of timeout
>                 0 connections dropped by keepalive
>                 0 dropped
>         2 dropped due to no socket
>         0 dropped due to full socket buffers
>         0 fragments dropped (dup or out of space)
>         2 fragments dropped after timeout
>         7538 output packets dropped due to no bufs, etc.
>
> The statistics are zeroed every 15 seconds in another window as I'm
> investigating the issue, but the rate is around 500-1000 lost packets
> every second at the current ~530 mbit/s load.
>
> >At those speeds you might be hitting various limits with your
> >config (e.g. 50k nmbclusters is probably way too small for
>
> I bet it isn't:
> 1967/5009/6976/50000 mbuf clusters in use (current/cache/total/max)
>
> >4k users -- means you have an average of 10-15 buffers per user;
> >the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering,
> >which is quite high besides the fact that in order to scale to 4k users
> >you would need over 1GB of kernel memory just for the buffers).
> Aha. Can you be more specific about the kernel memory stuff? Which
> setting needs tweaking?
>
>
> I have another similar box with 2 em GigE interfaces working @220-230
> mbit/s and virtually no out-of-bufs dropped packets as with bce @500-600
> mbit. It too has 350KBytes dummynet queue sizes. And it too has adequate
> mbuf load:
> 3071/10427/13498/25600 mbuf clusters in use (current/cache/total/max)

I think a quick way to tell if the problem is in dummynet/ipfw or elsewhere
would be to reconfigure the pipes (for short times, e.g. 1-2 minutes
while you test things) as

        # first, try to remove the shaping to see if the drops
        # are still present or not
        ipfw pipe XX delete; ipfw pipe XX config // no buffering

        # second, do more traffic aggregation to see if the number of
        # pipes influences the drops. These are several different
        # configs to be tried.
        ipfw pipe XX delete; ipfw pipe XX config bw 500Mbits/s
        ipfw pipe XX delete; ipfw pipe XX config bw 50Mbits/s mask src-ip 0xffffff00
        ipfw pipe XX delete; ipfw pipe XX config bw 5Mbits/s mask src-ip 0xfffffff0

and see if things change. If losses persist even removing dummynet,
then of course it is a device problem.
Also note that dummynet introduces some burstiness in the output,
which might saturate the output queue in the card (no idea what is
used by bce). This particular phenomenon could be reduced by raising
HZ to 2000 or 4000.

cheers
luigi
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[rihad]

Luigi Rizzo wrote:

> On Sun, Oct 04, 2009 at 07:53:23PM +0500, rihad wrote:
>> Luigi Rizzo wrote:
>>> On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:
>>>> Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell
>>>> PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP
>>>> users online limited by dummynet pipes of various speeds. According to
>>>> netstat -s output around 500-1000 packets are being dropped every second
>>>> (this accounts for wasting around 7-12 mbit/s worth of traffic according
>>>> to systat -ifstat):
>>> what kind of packets are you seeing as dropped ?
>>> please give the output of 'netstat -s output | grep drop'
>>>
>> The output packets, like I said:
>> # netstat -s output | grep drop
>>         2 connections closed (including 2 drops)
>>         0 embryonic connections dropped
>>                 2 connections dropped by rexmit timeout
>>                 0 connections dropped by persist timeout
>>         0 Connections (fin_wait_2) dropped because of timeout
>>                 0 connections dropped by keepalive
>>                 0 dropped
>>         2 dropped due to no socket
>>         0 dropped due to full socket buffers
>>         0 fragments dropped (dup or out of space)
>>         2 fragments dropped after timeout
>>         7538 output packets dropped due to no bufs, etc.
>>
>> The statistics are zeroed every 15 seconds in another window as I'm
>> investigating the issue, but the rate is around 500-1000 lost packets
>> every second at the current ~530 mbit/s load.
>>
>>> At those speeds you might be hitting various limits with your
>>> config (e.g. 50k nmbclusters is probably way too small for
>> I bet it isn't:
>> 1967/5009/6976/50000 mbuf clusters in use (current/cache/total/max)
>>
>>> 4k users -- means you have an average of 10-15 buffers per user;
>>> the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering,
>>> which is quite high besides the fact that in order to scale to 4k users
>>> you would need over 1GB of kernel memory just for the buffers).
>> Aha. Can you be more specific about the kernel memory stuff? Which
>> setting needs tweaking?
>>
>>
>> I have another similar box with 2 em GigE interfaces working @220-230
>> mbit/s and virtually no out-of-bufs dropped packets as with bce @500-600
>> mbit. It too has 350KBytes dummynet queue sizes. And it too has adequate
>> mbuf load:
>> 3071/10427/13498/25600 mbuf clusters in use (current/cache/total/max)
>
> I think a quick way to tell if the problem is in dummynet/ipfw or elsewhere
> would be to reconfigure the pipes (for short times, e.g. 1-2 minutes
> while you test things) as
>
> # first, try to remove the shaping to see if the drops
> # are still present or not
> ipfw pipe XX delete; ipfw pipe XX config // no buffering
>
> # second, do more traffic aggregation to see if the number of
> # pipes influences the drops. These are several different
> # configs to be tried.
> ipfw pipe XX delete; ipfw pipe XX config bw 500Mbits/s
> ipfw pipe XX delete; ipfw pipe XX config bw 50Mbits/s mask src-ip 0xffffff00
> ipfw pipe XX delete; ipfw pipe XX config bw 5Mbits/s mask src-ip 0xfffffff0
>
> and see if things change. If losses persist even removing dummynet,
> then of course it is a device problem.
> Also note that dummynet introduces some burstiness in the output,
> which might saturate the output queue in the card (no idea what is
> used by bce). This particular phenomenon could be reduced by raising
> HZ to 2000 or 4000.
>

Thanks for the tip. although I took an easier route by simply doing
"ipfw add allow ip from any to any" before the pipe rules, and the buf
drop rate instantly became 0. So the problem is dummynet/ipfw. Should I
go to setting HZ to 2000? Mine is 1000. I somehow don't think the change
would improve things. Maybe there's another way not involving a machine
reboot? This is a production machine ;-(
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Eugene Grosbein]

On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:

> sysctls:
> kern.ipc.nmbclusters=50000
> net.inet.ip.dummynet.io_fast=1

I guess you should also try to increase pipes length:

net.inet.ip.dummynet.hash_size=65536
net.inet.ip.dummynet.pipe_slot_limit=1000

And reconfigure pipes like this:

ipfw pipe NNN config bw ... queue 1000

And default 'taildrop' policy of dummynet pipes may be guilty,
you'd use GRED to prevent excessive drops.

Eugene Grosbein
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Luigi Rizzo-5]

On Mon, Oct 05, 2009 at 10:55:21AM +0800, Eugene Grosbein wrote:

> On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:
>
> > sysctls:
> > kern.ipc.nmbclusters=50000
> > net.inet.ip.dummynet.io_fast=1
>
> I guess you should also try to increase pipes length:
>
> net.inet.ip.dummynet.hash_size=65536
> net.inet.ip.dummynet.pipe_slot_limit=1000

in fact, i forgot to ask, we'd need to know the output of
"ipfw pipe show" just to get the idea if there is any
known reason for the drop (e.g. queues too short, etc.)

cheers
luigi

> And reconfigure pipes like this:
>
> ipfw pipe NNN config bw ... queue 1000
>
> And default 'taildrop' policy of dummynet pipes may be guilty,
> you'd use GRED to prevent excessive drops.
>
> Eugene Grosbein
> _______________________________________________
> freebsd-net@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[rihad]

Luigi Rizzo wrote:

> On Mon, Oct 05, 2009 at 10:55:21AM +0800, Eugene Grosbein wrote:
>> On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:
>>
>>> sysctls:
>>> kern.ipc.nmbclusters=50000
>>> net.inet.ip.dummynet.io_fast=1
>> I guess you should also try to increase pipes length:
>>
>> net.inet.ip.dummynet.hash_size=65536
>> net.inet.ip.dummynet.pipe_slot_limit=1000
>
> in fact, i forgot to ask, we'd need to know the output of
> "ipfw pipe show" just to get the idea if there is any
> known reason for the drop (e.g. queues too short, etc.)
>

The pipes are fine, each normally having 100-120 concurrent consumers
(i.e. active users). (The max number of consumers allowed per pipe is
hash_size*max_chain_len, normally 64*16==1024.)

kern.ipc.nmbclusters=111111

And these are by default:
#net.inet.ip.dummynet.hash_size=64
#net.inet.ip.dummynet.max_chain_len=16



Yesterday I set up a cronjob logging the number of drops in the past 15
minutes:
*/15 * * * *    (echo -n "$(date) "; netstat -z -s 2>/dev/null | fgrep
-w "output packets dropped") >> /tmp/bufs.log

And here's bufs.log:
Sun Oct  4 21:45:00 AZST 2009 418869 output packets dropped due to no
bufs, etc.
Sun Oct  4 22:00:00 AZST 2009 851693 output packets dropped due to no
bufs, etc.
Sun Oct  4 22:15:01 AZST 2009 932885 output packets dropped due to no
bufs, etc.
Sun Oct  4 22:30:00 AZST 2009 890522 output packets dropped due to no
bufs, etc.
Sun Oct  4 22:45:00 AZST 2009 1065931 output packets dropped due to no
bufs, etc.
Sun Oct  4 23:00:00 AZST 2009 937863 output packets dropped due to no
bufs, etc.
Sun Oct  4 23:15:01 AZST 2009 1018822 output packets dropped due to no
bufs, etc.
Sun Oct  4 23:30:00 AZST 2009 981922 output packets dropped due to no
bufs, etc.
Sun Oct  4 23:45:00 AZST 2009 1015124 output packets dropped due to no
bufs, etc.
Mon Oct  5 00:00:01 AZST 2009 1123926 output packets dropped due to no
bufs, etc.
Mon Oct  5 00:15:01 AZST 2009 948161 output packets dropped due to no
bufs, etc.
Mon Oct  5 00:30:00 AZST 2009 937277 output packets dropped due to no
bufs, etc.
Mon Oct  5 00:45:00 AZST 2009 875218 output packets dropped due to no
bufs, etc.
Mon Oct  5 01:00:00 AZST 2009 803527 output packets dropped due to no
bufs, etc.
Mon Oct  5 01:15:00 AZST 2009 728639 output packets dropped due to no
bufs, etc.
Mon Oct  5 01:30:00 AZST 2009 626154 output packets dropped due to no
bufs, etc.
Mon Oct  5 01:45:00 AZST 2009 519441 output packets dropped due to no
bufs, etc.
Mon Oct  5 02:00:00 AZST 2009 371098 output packets dropped due to no
bufs, etc.
Mon Oct  5 02:15:00 AZST 2009 681243 output packets dropped due to no
bufs, etc.
Mon Oct  5 02:30:00 AZST 2009 562909 output packets dropped due to no
bufs, etc.
Mon Oct  5 02:45:00 AZST 2009 426734 output packets dropped due to no
bufs, etc.
Mon Oct  5 03:00:00 AZST 2009 344619 output packets dropped due to no
bufs, etc.
Mon Oct  5 03:15:00 AZST 2009 90006 output packets dropped due to no
bufs, etc.
Mon Oct  5 03:30:00 AZST 2009 17064 output packets dropped due to no
bufs, etc.
Mon Oct  5 03:45:00 AZST 2009 3851 output packets dropped due to no
bufs, etc.
Mon Oct  5 04:00:00 AZST 2009 1323 output packets dropped due to no
bufs, etc.
Mon Oct  5 04:15:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 04:30:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 04:45:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 05:00:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 05:15:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 05:30:01 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 05:45:01 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 06:00:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 06:15:01 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 06:30:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 06:45:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 07:00:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 07:15:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 07:30:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 07:45:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 08:00:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 08:15:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 08:30:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 08:45:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 09:00:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 09:15:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 09:30:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 09:45:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 10:00:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 10:15:00 AZST 2009 0 output packets dropped due to no bufs, etc.
Mon Oct  5 10:30:00 AZST 2009 177 output packets dropped due to no
bufs, etc.
Mon Oct  5 10:45:00 AZST 2009 1701 output packets dropped due to no
bufs, etc.
Mon Oct  5 11:00:01 AZST 2009 19933 output packets dropped due to no
bufs, etc.
Mon Oct  5 11:15:00 AZST 2009 30003 output packets dropped due to no
bufs, etc.
Mon Oct  5 11:30:00 AZST 2009 56712 output packets dropped due to no
bufs, etc.
Mon Oct  5 11:45:00 AZST 2009 78721 output packets dropped due to no
bufs, etc.
Mon Oct  5 12:00:01 AZST 2009 112518 output packets dropped due to no
bufs, etc.
Mon Oct  5 12:15:00 AZST 2009 7229 output packets dropped due to no
bufs, etc.
Mon Oct  5 12:30:01 AZST 2009 24965 output packets dropped due to no
bufs, etc.
Mon Oct  5 12:45:00 AZST 2009 75900 output packets dropped due to no
bufs, etc.
Mon Oct  5 13:00:00 AZST 2009 45002 output packets dropped due to no
bufs, etc.
Mon Oct  5 13:15:00 AZST 2009 67161 output packets dropped due to no
bufs, etc.
Mon Oct  5 13:30:00 AZST 2009 112591 output packets dropped due to no
bufs, etc.


As you can see the drops gradually went away completely at about 4:00
a.m., and started coming up at about 10:30 a.m., although at a lower
rate, probably thanks to me bumping "ipfw ... queue NNN" up to 5000 at
10a.m. this morning. The traffic flow between 4a.m. and 10:30a.m., the
"quiet" times, is about 200-330 mbit/s 5 minute average, without a
single drop. But after that, in come the drops, no matter how high I set
the queue. Should I try 10000 slots? 20000? I'm really sure there are
plenty of heavy downloaders between 4a.m. and 10a.m., but still without
a single drop before approx. 330 mbit/s! Strange, isn't it? This makes
me believe I'm hitting some other global memory limit, rather than
per-user limit, at around 340-350 mbit/s, causing drops. top and netstat
-m are OK right now, though:

Mem: 1870M Active, 1220M Inact, 481M Wired, 201M Cache, 214M Buf, 164M Free
15595/5385/20980/111112 mbuf clusters in use (current/cache/total/max)
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Eugene Grosbein]

On Mon, Oct 05, 2009 at 01:53:20PM +0500, rihad wrote:

> As you can see the drops gradually went away completely at about 4:00
> a.m., and started coming up at about 10:30 a.m., although at a lower
> rate, probably thanks to me bumping "ipfw ... queue NNN" up to 5000 at
> 10a.m. this morning. The traffic flow between 4a.m. and 10:30a.m., the
> "quiet" times, is about 200-330 mbit/s 5 minute average, without a
> single drop. But after that, in come the drops, no matter how high I set
> the queue. Should I try 10000 slots? 20000?

First switch from taildrop (default) to GRED, it is designed to fight
your problem.

Eugene Grosbein

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[rihad]

Eugene Grosbein wrote:

> On Mon, Oct 05, 2009 at 01:53:20PM +0500, rihad wrote:
>
>> As you can see the drops gradually went away completely at about 4:00
>> a.m., and started coming up at about 10:30 a.m., although at a lower
>> rate, probably thanks to me bumping "ipfw ... queue NNN" up to 5000 at
>> 10a.m. this morning. The traffic flow between 4a.m. and 10:30a.m., the
>> "quiet" times, is about 200-330 mbit/s 5 minute average, without a
>> single drop. But after that, in come the drops, no matter how high I set
>> the queue. Should I try 10000 slots? 20000?
>
> First switch from taildrop (default) to GRED, it is designed to fight
> your problem.
>

Oh, I almost forgot... Right now I've googled up and am reading this
intro: http://www-rp.lip6.fr/~sf/WebSF/PapersWeb/iscc01.ps

So turning to GRED would turn my FreeBSD router from dumb into a smart
router that knows TCP? I thought pushing bits around at a lower level,
and a sufficient queue size were enough.
Still not sure why increasing queue size as high as I want doesn't
completely eliminate drops.
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[rihad]

Eugene Grosbein wrote:

> On Mon, Oct 05, 2009 at 01:53:20PM +0500, rihad wrote:
>
>> As you can see the drops gradually went away completely at about 4:00
>> a.m., and started coming up at about 10:30 a.m., although at a lower
>> rate, probably thanks to me bumping "ipfw ... queue NNN" up to 5000 at
>> 10a.m. this morning. The traffic flow between 4a.m. and 10:30a.m., the
>> "quiet" times, is about 200-330 mbit/s 5 minute average, without a
>> single drop. But after that, in come the drops, no matter how high I set
>> the queue. Should I try 10000 slots? 20000?
>
> First switch from taildrop (default) to GRED, it is designed to fight
> your problem.
>

      red | gred w_q/min_th/max_th/max_p
            Make use of the RED (Random Early Detection) queue
management algo-
            rithm.  w_q and max_p are floating point numbers between 0
and 1 (0
            not included), while min_th and max_th are integer numbers
specify-
            ing thresholds for queue management (thresholds are computed in
            bytes if the queue has been defined in bytes, in slots
otherwise).
            The dummynet(4) also supports the gentle RED variant (gred).

Do you or someone else know what w_q and max_p are?

There's just too much info for me to grasp here:
http://www.icir.org/floyd/red.html
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Eugene Grosbein]

On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote:

> Oh, I almost forgot... Right now I've googled up and am reading this
> intro: http://www-rp.lip6.fr/~sf/WebSF/PapersWeb/iscc01.ps
>
> So turning to GRED would turn my FreeBSD router from dumb into a smart
> router that knows TCP? I thought pushing bits around at a lower level,
> and a sufficient queue size were enough.

No, it will still deal with IP packets but more clever.

> Still not sure why increasing queue size as high as I want doesn't
> completely eliminate drops.

The goal is to make sources of traffic to slow down, this is the only
way to descrease drops - any finite queue may be overhelmed with traffic.
Taildrop does not really help with this. GRED does much better.

Eugene Grosbein
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Eugene Grosbein]

On Mon, Oct 05, 2009 at 02:48:29PM +0500, rihad wrote:

> >First switch from taildrop (default) to GRED, it is designed to fight
> >your problem.
>
>      red | gred w_q/min_th/max_th/max_p
>            Make use of the RED (Random Early Detection) queue
> management algo-
>            rithm.  w_q and max_p are floating point numbers between 0
> and 1 (0
>            not included), while min_th and max_th are integer numbers
> specify-
>            ing thresholds for queue management (thresholds are computed in
>            bytes if the queue has been defined in bytes, in slots
> otherwise).
>            The dummynet(4) also supports the gentle RED variant (gred).
>
> Do you or someone else know what w_q and max_p are?
>
> There's just too much info for me to grasp here:
> http://www.icir.org/floyd/red.html

http://docs.freebsd.org/cgi/getmsg.cgi?fetch=126518+0+archive/2009/freebsd-net/20091004.freebsd-net

Eugene Grosbein
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Luigi Rizzo-5]

On Mon, Oct 05, 2009 at 05:56:00PM +0800, Eugene Grosbein wrote:

> On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote:
>
> > Oh, I almost forgot... Right now I've googled up and am reading this
> > intro: http://www-rp.lip6.fr/~sf/WebSF/PapersWeb/iscc01.ps
> >
> > So turning to GRED would turn my FreeBSD router from dumb into a smart
> > router that knows TCP? I thought pushing bits around at a lower level,
> > and a sufficient queue size were enough.
>
> No, it will still deal with IP packets but more clever.
>
> > Still not sure why increasing queue size as high as I want doesn't
> > completely eliminate drops.
>
> The goal is to make sources of traffic to slow down, this is the only
> way to descrease drops - any finite queue may be overhelmed with traffic.
> Taildrop does not really help with this. GRED does much better.

i think the first problem here is figure out _why_ we have
the drops, as the original poster said that queues are configured
with a very large amount of buffer (and i think there is a
misconfiguration somewhere because the mbuf stats do not make
sense)

cheers
luigi
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Eugene Grosbein]

On Mon, Oct 05, 2009 at 12:04:46PM +0200, Luigi Rizzo wrote:

> > The goal is to make sources of traffic to slow down, this is the only
> > way to descrease drops - any finite queue may be overhelmed with traffic.
> > Taildrop does not really help with this. GRED does much better.
>
> i think the first problem here is figure out _why_ we have
> the drops, as the original poster said that queues are configured
> with a very large amount of buffer (and i think there is a
> misconfiguration somewhere because the mbuf stats do not make
> sense)

That may be very simple, f.e. wide uplink channel and policy that
dictates slower client speeds. Any taildrop queue would drop lots
of packets.

Eugene Grosbein
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[rihad]

Eugene Grosbein wrote:

> On Mon, Oct 05, 2009 at 12:04:46PM +0200, Luigi Rizzo wrote:
>
>>> The goal is to make sources of traffic to slow down, this is the only
>>> way to descrease drops - any finite queue may be overhelmed with traffic.
>>> Taildrop does not really help with this. GRED does much better.
>> i think the first problem here is figure out _why_ we have
>> the drops, as the original poster said that queues are configured
>> with a very large amount of buffer (and i think there is a
>> misconfiguration somewhere because the mbuf stats do not make
>> sense)
>
> That may be very simple, f.e. wide uplink channel and policy that
> dictates slower client speeds. Any taildrop queue would drop lots
> of packets.
>

If uplink is e.g. 100 mbit/s, but data is fed to client by dummynet at 1
mbit/s, doesn't the _client's_ TCP software know to slow things down to
not overwhelm 1 mbit/s? Where has TCP slow-start gone? My router box
isn't some application proxy that starts downloading at full 100 mbit/s
thus quickly filling client's 1 mbit/s link. It's just a router.

Although it doesn't yet make sense to me, I'll try going to GRED soon.
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[rihad]

Eugene Grosbein wrote:
> On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote:
>
>> Still not sure why increasing queue size as high as I want doesn't
>> completely eliminate drops.
>
> The goal is to make sources of traffic to slow down, this is the only
> way to descrease drops - any finite queue may be overhelmed with traffic.
> Taildrop does not really help with this. GRED does much better.
>

Alright, so I changed to gred by adding to each config command:
ipfw ... gred 0.002/900/1000/0.1 queue 1000
and reconfigured. Still around 300-400 drops per second, which was
typical at this load level before with taildrop anyway. There are around
3-5 mbit/s being wasted according to systat -ifstat.

Should I now increase slots to 5-10-20k?
Very strange.

"ipfw pipe show" correctly shows that gred is at work. For example:
00512: 512.000 Kbit/s    0 ms  1000 sl. 79 queues (64 buckets)
           GRED w_q 0.001999 min_th 900 max_th 1000 max_p 0.099991
     mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
...

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Luigi Rizzo-5]

On Mon, Oct 05, 2009 at 03:52:39PM +0500, rihad wrote:

> Eugene Grosbein wrote:
> >On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote:
> >
> >>Still not sure why increasing queue size as high as I want doesn't
> >>completely eliminate drops.
> >
> >The goal is to make sources of traffic to slow down, this is the only
> >way to descrease drops - any finite queue may be overhelmed with traffic.
> >Taildrop does not really help with this. GRED does much better.
> >
>
> Alright, so I changed to gred by adding to each config command:
> ipfw ... gred 0.002/900/1000/0.1 queue 1000
> and reconfigured. Still around 300-400 drops per second, which was
> typical at this load level before with taildrop anyway. There are around
> 3-5 mbit/s being wasted according to systat -ifstat.
>
> Should I now increase slots to 5-10-20k?
> Very strange.
>
> "ipfw pipe show" correctly shows that gred is at work. For example:
> 00512: 512.000 Kbit/s    0 ms  1000 sl. 79 queues (64 buckets)
>           GRED w_q 0.001999 min_th 900 max_th 1000 max_p 0.099991
>     mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
> ...

you keep omitting the important info i.e. whether individual
pipes have drops, significant queue lenghts and so on.

i am giving up!
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."