Network transition complete + PF question

[Matthew Dillon]

    The network move is complete.

    I have a question for the PF/ALTQ masters out there.  I am trying to
    configure PF in a manner similar to what Cisco's fair-queue algorithm
    does.  Cisco's algorithm basically hashes TCP and UDP traffic based
    on the port/IP pairs, creating a bunch of lists of backlogged packets
    and then schedules the packets at the head of each list.

    I am trying to find something equivalent with PF and not having much
    luck.  Neither CBQ nor HFSC seem to work well.   I can separate certain
    types of traffic but the real problem is when there are multiple
    TCP connections that are essentially classified the same, and one is
    hogging the outgoing bandwidth.

    So the question is, is there a PF solution for that or do I need to
    write a new ALTQ mechanic to implement fair queueing ?

    If there is no current solution I have a pretty good idea how to
    implement it.  I can use PF's 'keep state' mechanism and then hash
    the state structure pointer and store it in the packet header, then
    implement a new ALTQ that takes that hash code and throws it into an
    array of queues from which it fair-dequeues packets for output.

                                        -Matt

[Chris Turner-4]

Matthew Dillon wrote:
  >     I have a question for the PF/ALTQ masters out there.  I am trying to
>     configure PF in a manner similar to what Cisco's fair-queue algorithm
>     does.  Cisco's algorithm basically hashes TCP and UDP traffic based
>     on the port/IP pairs, creating a bunch of lists of backlogged packets
>     and then schedules the packets at the head of each list.

Not sure if I fit in the former category, but I don't *think*
there's anything that does this .. I've thought about someday trying to
rig something up - it would be nice e.g. to lower the priority of HTTP
sessions after some amount of data is travelling - that way my
interactive bw would not be hogged by big downloads..

it might be possible to hook up pfctl or pfflowd into some kind of table
modifying script some how

If we had tagging support, this might be easier

http://www.openbsd.org/faq/pf/tagging.html

(e.g. monitor state table every N & retag high bandwidth traffic,
  using the tags to assign traffic different priorities)

ideally the 'pf' config would allow some kind of 'rate' designator

[Cédric Berger-2]

Matthew Dillon wrote:

>     I am trying to find something equivalent with PF and not having much
>     luck.  Neither CBQ nor HFSC seem to work well.   I can separate certain
>     types of traffic but the real problem is when there are multiple
>     TCP connections that are essentially classified the same, and one is
>     hogging the outgoing bandwidth.

I'm not an export about ALTQ, but how about creating N similar queues,
and assigning a new TCP connection to one of these queues at random
using the "probability X%" keyword?

Cedric

[Joerg Sonnenberger]

On Wed, Apr 02, 2008 at 11:08:40PM -0700, Matthew Dillon wrote:
>     I have a question for the PF/ALTQ masters out there.  I am trying to
>     configure PF in a manner similar to what Cisco's fair-queue algorithm
>     does.  Cisco's algorithm basically hashes TCP and UDP traffic based
>     on the port/IP pairs, creating a bunch of lists of backlogged packets
>     and then schedules the packets at the head of each list.

I don't think any of the OS solutions will do that. What you can do for
this kind of application is put all downloads into one class with borrow
property and limit the source connections to a small value. Use a second
rule to drop all the other connections into.

Joerg

[Matthew Dillon]

:Not sure if I fit in the former category, but I don't *think*
:there's anything that does this .. I've thought about someday trying to
:rig something up - it would be nice e.g. to lower the priority of HTTP
:sessions after some amount of data is travelling - that way my
:interactive bw would not be hogged by big downloads..

    That you can actually do, but it will effect *all* of the HTTP sessions
    and not just the one the download is occuring over.  In the section
    on queues HFSC supports 'realtime sc', 'upperlimit sc', and
    'linkshare sc', where 'sc' can be a single percentage or in
    a special (m1, d, m2) format:

        "The format for service curve specifications is (m1, d, m2).
        m2 controls the bandwidth assigned to the queue.  m1 and d are
        optional and can be used to control the initial bandwidth assignment.
        For the first d milliseconds the queue gets the bandwidth given as
        m1, afterwards the value given in m2."


:it might be possible to hook up pfctl or pfflowd into some kind of table
:modifying script some how
:
:If we had tagging support, this might be easier
:
:http://www.openbsd.org/faq/pf/tagging.html
:
:(e.g. monitor state table every N & retag high bandwidth traffic,
:  using the tags to assign traffic different priorities)
:
:ideally the 'pf' config would allow some kind of 'rate' designator

    I'm pretty sure we have tagging support, but as far as I can tell
    there is no way to associate a tag with the keep-state PF uses to
    track a connection.

    It would be a cool addition if one could specify a keep state option
    to have it automatically tag packets associated with high-bandwidth
    connections.  That plus a fair dequeue would be killer.

                                        -Matt
                                        Matthew Dillon
                                        <dillon@...>

[Matthew Dillon]

:
:Matthew Dillon wrote:
:
:>     I am trying to find something equivalent with PF and not having much
:>     luck.  Neither CBQ nor HFSC seem to work well.   I can separate certain
:>     types of traffic but the real problem is when there are multiple
:>     TCP connections that are essentially classified the same, and one is
:>     hogging the outgoing bandwidth.
:
:I'm not an export about ALTQ, but how about creating N similar queues,
:and assigning a new TCP connection to one of these queues at random
:using the "probability X%" keyword?
:
:Cedric

    That would help. I don't see a 'probability' keyword anywhere though.

                                        -Matt
                                        Matthew Dillon
                                        <dillon@...>

[Cédric Berger-2]

Matthew Dillon wrote:

> :
> :Matthew Dillon wrote:
> :
> :>     I am trying to find something equivalent with PF and not having much
> :>     luck.  Neither CBQ nor HFSC seem to work well.   I can separate certain
> :>     types of traffic but the real problem is when there are multiple
> :>     TCP connections that are essentially classified the same, and one is
> :>     hogging the outgoing bandwidth.
> :
> :I'm not an export about ALTQ, but how about creating N similar queues,
> :and assigning a new TCP connection to one of these queues at random
> :using the "probability X%" keyword?
> :
> :Cedric
>
>     That would help. I don't see a 'probability' keyword anywhere though.
>
> -Matt
> Matthew Dillon
> <dillon@...>

Ugh, I didn't realize that dragonfly PF version was so old...
That code was added on April 2004...
http://www.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c.diff?r1=1.435&r2=1.436

Now porting that code and the is probably a 15 minute job...
Cedric

[Matthew Dillon]

:Ugh, I didn't realize that dragonfly PF version was so old...
:That code was added on April 2004...
:http://www.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c.diff?r1=1.435&r2=1.436
:
:Now porting that code and the is probably a 15 minute job...
:Cedric

    That doesn't look too bad.  There's some additional code that has
    to be merged from pfctl to parse the probability keyword but it
    looks fairly straight forward.  I'll take a look at it later today.

    It won't solve the fair share scheduler problem but it looks like it
    could be useful for a number of other tasks.

                                        -Matt
                                        Matthew Dillon
                                        <dillon@...>

[Simon 'corecode' Schubert]

Cédric Berger wrote:
> Ugh, I didn't realize that dragonfly PF version was so old...
> That code was added on April 2004...
> http://www.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c.diff?r1=1.435&r2=1.436
>
>
> Now porting that code and the is probably a 15 minute job...

Porting pf to a new version is a tedious task.  Last time I started (1 year ago?), I wound up with a 1MB patch :/

cheers
  simon

[Chris Turner-4]

Matthew Dillon wrote:
>
>     I'm pretty sure we have tagging support, but as far as I can tell
>     there is no way to associate a tag with the keep-state PF uses to
>     track a connection.

based on pf.conf I was wrong here - I was getting generic tagging
support confused with ethernet address tagging from bridge ineterfaces..

http://www.openbsd.org/faq/pf/tagging.html#ethernet

which would have nothing to do with the problem here..

[Matthew Dillon]

:based on pf.conf I was wrong here - I was getting generic tagging
:support confused with ethernet address tagging from bridge ineterfaces..
:
:http://www.openbsd.org/faq/pf/tagging.html#ethernet
:
:which would have nothing to do with the problem here..

    That's cool.  I'm a bit surprised that nobody seems to have implemented
    a fair queue for ALTQ.  There are lots of essays on using CBQ, but they
    all assume that you can classify the hogs and that doesn't actually
    solve the problem when a single service may have a mix of interactive
    and batch streams.

                                        -Matt
                                        Matthew Dillon
                                        <dillon@...>

[Matthew Dillon]

:Porting pf to a new version is a tedious task.  Last time I started (1 ye=
:ar ago?), I wound up with a 1MB patch :/
:
:cheers
:  simon

    It's one of those things where you pretty much have to manually patch.

                                        -Matt
                                        Matthew Dillon
                                        <dillon@...>

[Matthew Dillon]

:Porting pf to a new version is a tedious task.  Last time I started (1 ye=
:ar ago?), I wound up with a 1MB patch :/
:
:cheers
:  simon

    There are some rather large differences between what OpenBSD has and what
    DragonFly and FreeBSD have.  I think the only way to do it is to go
    feature by feature, committing each one, until the diff set is small
    enough to bite off the rest.  PF has diverged somewhat in all the
    BSDs (and I'm adding DFly to the fray with these new keep state
    features).  It might be best to let it settle down a bit.

                                            -Matt

[Matthew Dillon]

:On Wed, Apr 02, 2008 at 11:08:40PM -0700, Matthew Dillon wrote:
:>     I have a question for the PF/ALTQ masters out there.  I am trying to
:>     configure PF in a manner similar to what Cisco's fair-queue algorithm
:>     does.  Cisco's algorithm basically hashes TCP and UDP traffic based
:>     on the port/IP pairs, creating a bunch of lists of backlogged packets
:>     and then schedules the packets at the head of each list.
:
:I don't think any of the OS solutions will do that. What you can do for
:this kind of application is put all downloads into one class with borrow
:property and limit the source connections to a small value. Use a second
:rule to drop all the other connections into.
:
:Joerg

    Heh.  Poor Joerg.  His email went through www.pkgsrc-box.org which
    must have been a linux box.  It's been stuck in a queue all this time
    due to the window scale issues on my new router.

    You must have been pulling your hair out :-)

                                        -Matt
                                        Matthew Dillon
                                        <dillon@...>

[Joerg Sonnenberger]

On Mon, Apr 07, 2008 at 11:32:39AM -0700, Matthew Dillon wrote:
>     Heh.  Poor Joerg.  His email went through www.pkgsrc-box.org which
>     must have been a linux box.  It's been stuck in a queue all this time
>     due to the window scale issues on my new router.

Well, I have window scaling active on all my systems -- helps a lot
when you have a big pipe. That the system is running Linx is related to
the (non-)scalability of DragonFly for bulk builds -- it was a lot
faster to run 4 instances of DragonFly under Vmware than to run it
natively...

Joerg