PPPoE server (high traffic in WDM network)

[Michelle Konzack-4]

Hello,

I am using since over  10 years  Debian  GNU/Linux  and  3 years  longer
NetBSD. Also I have a running PicoBSD box.

Now I have a problem more grave...

I am ongoing to install a CWDM (1GE) and DWDM  (10GE)  network  for  the
Alvarion BreezeACCESS VL (38 base stations) and more then  200  Iskratel
FTTH DSLAMS of 96 ports (each with 100MBit, but only one  1GE  Upstream)
each.

What I now need are a PPPoE Severs (round-robin and loadbalancing) which
must work using FreeRadius and PostgreSQL.

There was someone on the <debian-isp> which  has  suggested  me  to  use
FreeBSD, because the PPPoE it is already build to  authenticate  against
Radius.

So, what I like to know is, if I have a 1GE and 10GE network,  how  many
clients can  one  PPPoE  Server  handel  and  what  are  the  CPU/Memory
requirements?

There is a little problem to get small but  reliabel  Servers  with  TWO
10GE interfaces.

I think, consumer mainboards are not suitabel even someone told me under
Linux, I need 2 MHz CPU-Speed and 2 MByte of Memory per client...

Please note, that I am ongoing ISP with over  150.000  customers  in  DE
between Freiburg and Karlsruhe (Baden-Württemberg)  and  using  consumer
mainboards is NOT reliabel since in the last 6 years I lost at least  20
per year in 280 Low-Cost Servers.

A "Sun Fire X4100M2" would be more reliabel... but even the smallest CPU
would be overkill because the machine has only 1GE interfaces.

Any suggestions?

Note 1: Even if I use a Sun Fire, I would prefer a microBSD
        running from an industrial SD/CF card.

Note:  Please do NOT CC me, I am on the list and read it...

Thanks, Greetings and nice Day/Evening
    Michelle Konzack
    Systemadministrator
    Tamay Dogan Network
    Debian GNU/Linux Consultant

--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
<http://www.tamay-dogan.net/>                 Michelle Konzack
<http://www.can4linux.org/>                   c/o Vertriebsp. KabelBW
<http://www.flexray4linux.org/>               Blumenstrasse 2
Jabber linux4michelle@...           77694 Kehl/Germany
IRC #Debian (irc.icq.com)                     Tel. DE: [phone]
ICQ #328449886                                Tel. FR: [phone]
_______________________________________________
freebsd-isp@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-isp
To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..."

[Nikos Vassiliadis-5]

Michelle Konzack wrote:
> I am ongoing to install a CWDM (1GE) and DWDM  (10GE)  network  for  the
> Alvarion BreezeACCESS VL (38 base stations) and more then  200  Iskratel
> FTTH DSLAMS of 96 ports (each with 100MBit, but only one  1GE  Upstream)
> each.

So, you'll have 96*200 possible PPP clients. How many concurrent PPP
sessions do you care to support?
And more importantly, how much aggregate bandwidth?

> What I now need are a PPPoE Severs (round-robin and loadbalancing) which
> must work using FreeRadius and PostgreSQL.

Don't understand what you mean round-robin and loadbalancing?
Read below.

> There was someone on the <debian-isp> which  has  suggested  me  to  use
> FreeBSD, because the PPPoE it is already build to  authenticate  against
> Radius.

FreeBSD has a RADIUS library in base. The two notable users of libradius
are ppp and net/mpd. The only choice in a ISP environment I think is the
net/mpd5 port. Read the outline here:
http://www.freebsd.org/cgi/url.cgi?ports/net/mpd5/pkg-descr

It is very good and is actually used in large setups.

> So, what I like to know is, if I have a 1GE and 10GE network,  how  many
> clients can  one  PPPoE  Server  handel  and  what  are  the  CPU/Memory
> requirements?

Can't reply, but keep in mind that filling a 10GE pipe is
a hard task on its own.

I *think* having more low fidelity BRASs, will serve your
needs better that a few high fidelity ones.

>
[snipped]
>
> Note 1: Even if I use a Sun Fire, I would prefer a microBSD
>         running from an industrial SD/CF card.

MicroBSD seems OpenBSD based. Can't comment on this.

You can try NanoBSD and TinyBSD which are FreeBSD based and I
believe can fit the bill. These two run with their filesystems
read-only mounted which is ideal for flash memories.

Nikos
_______________________________________________
freebsd-isp@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-isp
To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..."

[Michelle Konzack-4]

Hello Nikos,

Thanks for your answer.

Am 2009-07-16 12:27:06, schrieb Nikos Vassiliadis:
> Michelle Konzack wrote:
>> I am ongoing to install a CWDM (1GE) and DWDM  (10GE)  network  for  the
>> Alvarion BreezeACCESS VL (38 base stations) and more then  200  Iskratel
>> FTTH DSLAMS of 96 ports (each with 100MBit, but only one  1GE  Upstream)
>> each.
>
> So, you'll have 96*200 possible PPP clients. How many concurrent PPP
> sessions do you care to support?
> And more importantly, how much aggregate bandwidth?

Because the customers are permanently On-Line du to the  VoIP-Telephone,
we count with the full number of clients...

The distance between the FTTH DSLAM and the customers can be up to 10km.

The idea is now, that we do not simply connect the FTTH DSLAM's  to  the
CISCO switches but building a redunant Ethernet Carrier Network.

This mean, we can install in each village there own FTTH DSLAM  even  if
there are 2500 hausholds and we install 26 FTTH DSLAM's there.

This mean in theorie 250 GBit Customer Downstream, 26 Gbit Upstream  but
we count with a 10 GE which is maybe used to 30-50%.

OK, if we switch to an "Ethernet Carrier Network" I could install one or
two PPPoE Servers in each village.  But if one goes down, the second has
to handel 2500 client connections.

Note:   This is ONLY the base installation  between  Kehl,  Rheinau,
        Renchen and Oberkirch (arround  35.000  hausholds)  and  the
        whole region has 150.000 hausholds.

> Don't understand what you mean round-robin and loadbalancing?
> Read below.
<snip>
> FreeBSD has a RADIUS library in base. The two notable users of libradius
> are ppp and net/mpd. The only choice in a ISP environment I think is the
> net/mpd5 port. Read the outline here:
> http://www.freebsd.org/cgi/url.cgi?ports/net/mpd5/pkg-descr
>
> It is very good and is actually used in large setups.

Thankyo for the link, I will red on if I am in Office...

> Can't reply, but keep in mind that filling a 10GE pipe is
> a hard task on its own.

It depends on how many customers you have and with an Internet access of
100 Mbit plus services like IPTV and VOD you can fill up a 10 GE pipe.

> I *think* having more low fidelity BRASs, will serve your
> needs better that a few high fidelity ones.

You mean, putting a bunch of small 1U Servers into a 19" 42RU?

> You can try NanoBSD and TinyBSD which are FreeBSD based and I
> believe can fit the bill. These two run with their filesystems
> read-only mounted which is ideal for flash memories.

Can you recomment it for an ISP setup?

Hmmm, I am right, that NanoBSD can be bootup over network?
(this would be another solution)

Thanks, Greetings and nice Day/Evening
    Michelle Konzack
    Systemadministrator
    Tamay Dogan Network
    Debian GNU/Linux Consultant

--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
<http://www.tamay-dogan.net/>                 Michelle Konzack
<http://www.can4linux.org/>                   c/o Vertriebsp. KabelBW
<http://www.flexray4linux.org/>               Blumenstrasse 2
Jabber linux4michelle@...           77694 Kehl/Germany
IRC #Debian (irc.icq.com)                     Tel. DE: [phone]
ICQ #328449886                                Tel. FR: [phone]
_______________________________________________
freebsd-isp@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-isp
To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..."

[Nikos Vassiliadis-5]

> Hello Nikos,

Hi, I just saw your answer while browsing. I am not on isp@...
Please CC questions@.

> Am 2009-07-16 12:27:06, schrieb Nikos Vassiliadis:
>> Michelle Konzack wrote:
>>> I am ongoing to install a CWDM (1GE) and DWDM  (10GE)  network  for  the
>>> Alvarion BreezeACCESS VL (38 base stations) and more then  200  Iskratel
>>> FTTH DSLAMS of 96 ports (each with 100MBit, but only one  1GE  Upstream)
>>> each.
>>
>> So, you'll have 96*200 possible PPP clients. How many concurrent PPP
>> sessions do you care to support?
>> And more importantly, how much aggregate bandwidth?
>
> Because the customers are permanently On-Line du to the  VoIP-Telephone,
> we count with the full number of clients...
>
> The distance between the FTTH DSLAM and the customers can be up to 10km.
>
> The idea is now, that we do not simply connect the FTTH DSLAM's  to  the
> CISCO switches but building a redunant Ethernet Carrier Network.
>
> This mean, we can install in each village there own FTTH DSLAM  even  if
> there are 2500 hausholds and we install 26 FTTH DSLAM's there.
>
> This mean in theorie 250 GBit Customer Downstream, 26 Gbit Upstream  but
> we count with a 10 GE which is maybe used to 30-50%.
>
> OK, if we switch to an "Ethernet Carrier Network" I could install one or
> two PPPoE Servers in each village.  But if one goes down, the second has
> to handel 2500 client connections.

I *think* the number of clients is doable. I don't know about
the bandwidth.

> Note:   This is ONLY the base installation  between  Kehl,  Rheinau,
>         Renchen and Oberkirch (arround  35.000  hausholds)  and  the
>         whole region has 150.000 hausholds.
>
>> Don't understand what you mean round-robin and loadbalancing?
>> Read below.
> <snip>
>> FreeBSD has a RADIUS library in base. The two notable users of libradius
>> are ppp and net/mpd. The only choice in a ISP environment I think is the
>> net/mpd5 port. Read the outline here:
>> http://www.freebsd.org/cgi/url.cgi?ports/net/mpd5/pkg-descr
>>
>> It is very good and is actually used in large setups.
>
> Thankyo for the link, I will red on if I am in Office...
>
>> Can't reply, but keep in mind that filling a 10GE pipe is
>> a hard task on its own.
>
> It depends on how many customers you have and with an Internet access of
> 100 Mbit plus services like IPTV and VOD you can fill up a 10 GE pipe.

I meant "filling a 10 Gbit pipe with a general purpose computer
architecture is a hard task". Packet forwarding at these rates is
tricky.

>> I *think* having more low fidelity BRASs, will serve your
>> needs better that a few high fidelity ones.
>
> You mean, putting a bunch of small 1U Servers into a 19" 42RU?

Yes, you may find that having two small boxes instead of bigger one
gives better results performance-wise. You also have to test if SMP
helps and how much. A beast with 16 cores is more powerful from a
regular computer with 2 cores, but does it help in your setup?

>> You can try NanoBSD and TinyBSD which are FreeBSD based and I
>> believe can fit the bill. These two run with their filesystems
>> read-only mounted which is ideal for flash memories.
>
> Can you recomment it for an ISP setup?

It's FreeBSD running from a read-only mounted medium.
No more, no less. Yes, it's fine for an ISP setup.

>
> Hmmm, I am right, that NanoBSD can be bootup over network?
> (this would be another solution)

NanoBSD is meant to run in embedded stand-alone devices.
So, I *guess* that is conceptually very far from net booting.

Nikos
_______________________________________________
freebsd-isp@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-isp
To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..."

[Michelle Konzack-4]

Hello Nikos,

sorry if I can not answer the next 3 or 5 days,  but  my  Server  has  a
hardware outage and I need to get a new one...  :-/

Am 2009-07-23 20:52:15, schrieb Nikos Vassiliadis:
> I *think* the number of clients is doable. I don't know about
> the bandwidth.

The main problem with the bandwidth is, that even the VOD/IPTV and  VoIP
traffic goes throug the PPPoE server which is very bad.

I do not want to count the traffic to a  specific  /25  which  hold  the
storage servers, mean, the VOD/IPTV and VoIP  traffic  must  bypass  the
PPPoE server.  and this reduce the traffic enormous...

> I meant "filling a 10 Gbit pipe with a general purpose computer
> architecture is a hard task". Packet forwarding at these rates is
> tricky.

Right, I would never try it...

My idea is/was, to put the PPPoE server diretly byside the FTTH DSLAM's,
which mean, each 96port DSLAM has an upstrem of 1 GE and even if  I  put
10 of them in a 42RU, it would normaly not fill the  10 GE  ports  of  a
professionel Server.  And of corse,  I  can  put  always  two  or  three
together parallel.

The problem is only, that I can not install 10 (or 20 redunant) 1U Sun
Fire X4100M2, even if I can get up to 60% rebat of the listprice.

I have not the place to put 20 additiona servers into, nor  I  like  the
power consumation ~70 Watt with the smalles CPU and only 4 GByte of RAM.

> Yes, you may find that having two small boxes

Your two small boxes are at lleast 10 servers with 1U supporting 10 GBit
in summary or 20 if redunant.

The villages we are cabeling are between 480 and 3200 hausholds.

> instead of bigger one
> gives better results performance-wise. You also have to test if SMP
> helps and how much. A beast with 16 cores is more powerful from a
> regular computer with 2 cores, but does it help in your setup?

If I go with 1 U Sun Fire X4100M2 the Opteron has 4 Cores and 4  threads
per core (AFAIK there is a 8 threads version too)

>> Can you recomment it for an ISP setup?
>
> It's FreeBSD running from a read-only mounted medium.
> No more, no less. Yes, it's fine for an ISP setup.

If I have 4 GByte of memory, I could run entirely from RAMDISK...
Memory is cheaper then the harddrives

Thanks, Greetings and nice Day/Evening
    Michelle Konzack
    Systemadministrator
    Tamay Dogan Network
    Debian GNU/Linux Consultant

--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
<http://www.tamay-dogan.net/>                 Michelle Konzack
<http://www.can4linux.org/>                   c/o Vertriebsp. KabelBW
<http://www.flexray4linux.org/>               Blumenstrasse 2
Jabber linux4michelle@...           77694 Kehl/Germany
IRC #Debian (irc.icq.com)                     Tel. DE: [phone]
ICQ #328449886                                Tel. FR: [phone]
_______________________________________________
freebsd-isp@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-isp
To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..."

[muhammad usman]

Hi,

I've little off topic suggestion regarding network design.


The solution you are implementing will mean to make hundreds of users share the same broadcast domain. As all your FTTH and other DSLAMS would be working in bridge mode.
This scenerio is not safe as if anyone of your clients will start his own pppoe server you will be in strange trouble, there can be other issues too.


I guess your DSLAMs must have built in pppoe support and radius client, if its there then every port of dslams can be separate broadcast domain. This will cause you extra routing management (depending upon your scsnerio)


In case your DSLAMS have no pppoe feature then i would suggest you to at least put every DSLAM's uplink port in deparate VLAN and connect pppoe server using trunk port or multiple single ports.


And before implementing this solution consult your DSLAM vendor cause ive observed problems in ipdslams when used in bridge/transperant mode.


Regards
usman


--- On Wed, 7/15/09, Michelle Konzack <bsd4michelle@...> wrote:

From: Michelle Konzack <bsd4michelle@...>
Subject: PPPoE server (high traffic in WDM network)
To: freebsd-questions@..., freebsd-isp@...
Date: Wednesday, July 15, 2009, 11:40 PM

Hello,

I am using since over  10 years  Debian  GNU/Linux  and  3 years  longer
NetBSD. Also I have a running PicoBSD box.

Now I have a problem more grave...

I am ongoing to install a CWDM (1GE) and DWDM  (10GE)  network  for  the
Alvarion BreezeACCESS VL (38 base stations) and more then  200  Iskratel
FTTH DSLAMS of 96 ports (each with 100MBit, but only one  1GE  Upstream)
each.

What I now need are a PPPoE Severs (round-robin and loadbalancing) which
must work using FreeRadius and PostgreSQL.

There was someone on the <debian-isp> which  has  suggested  me  to  use
FreeBSD, because the PPPoE it is already build to  authenticate  against
Radius.

So, what I like to know is, if I have a 1GE and 10GE network,  how  many
clients can  one  PPPoE  Server  handel  and  what  are  the  CPU/Memory
requirements?

There is a little problem to get small but  reliabel  Servers  with  TWO
10GE interfaces.

I think, consumer mainboards are not suitabel even someone told me under
Linux, I need 2 MHz CPU-Speed and 2 MByte of Memory per client...

Please note, that I am ongoing ISP with over  150.000  customers  in  DE
between Freiburg and Karlsruhe (Baden-Württemberg)  and  using  consumer
mainboards is NOT reliabel since in the last 6 years I lost at least  20
per year in 280 Low-Cost Servers.

A "Sun Fire X4100M2" would be more reliabel... but even the smallest CPU
would be overkill because the machine has only 1GE interfaces.

Any suggestions?

Note 1: Even if I use a Sun Fire, I would prefer a microBSD
        running from an industrial SD/CF card.

Note:  Please do NOT CC me, I am on the list and read it...

Thanks, Greetings and nice Day/Evening
    Michelle Konzack
    Systemadministrator
    Tamay Dogan Network
    Debian GNU/Linux Consultant

--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
<http://www.tamay-dogan.net/>                 Michelle Konzack
<http://www.can4linux.org/>                   c/o Vertriebsp. KabelBW
<http://www.flexray4linux.org/>               Blumenstrasse 2
Jabber linux4michelle@...           77694 Kehl/Germany
IRC #Debian (irc.icq.com)                     Tel. DE: [phone]
ICQ #328449886                                Tel. FR: [phone]
_______________________________________________
freebsd-isp@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-isp
To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..."




_______________________________________________
freebsd-isp@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-isp
To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..."

[Michelle Konzack-4]

Hello Muhammad,

Am 2009-08-21 16:36:52, schrieb muhammad usman:
> Hi,
>
> I've little off topic suggestion regarding network design.
>
> The solution you are implementing will mean to make hundreds of users
> share the same broadcast domain. As all your FTTH and other DSLAMS
> would be working in bridge mode.

Right.

> This scenerio is not safe as if anyone of your clients will start his
> own pppoe server you will be in strange trouble, there can be other
> issues too.

I was thinking about this problem to, but  AFAIK  this  does  not  work,
because the FTTH Modem is blocking such things from the customer.

> I guess your DSLAMs must have built in pppoe support and radius
> client, if its there then every port of dslams can be separate
> broadcast domain. This will cause you extra routing management
> (depending upon your scsnerio)

No, the "Iskratel SI3000 FTTH DSLAM" is only a  transport  system  which
require external PPPoE service.

> In case your DSLAMS have no pppoe feature then i would suggest you to
> at least put every DSLAM's uplink port in deparate VLAN and connect
> pppoe server using trunk port or multiple single ports.

The problem is, HOW many VLANS can you open?  If I use the Transmode TS
DWDM (GE) and/or CWDM (10GE) then I am limited to 4096 VLAN's.

So, if you have inside this network  business  customers  which  require
seperated VLAN routing you run out of VLAN's

Yes, I can use a biger system, but this  cost  at  least  twice  of  the
"Transmode TS" system.  Juniper is much more expensive.

> And before implementing this solution consult your DSLAM vendor cause
> ive observed problems in ipdslams when used in bridge/transperant
> mode.

I am already in contact with Iskratel  in  Kranj/Slovenia,  because  the
Distributor in Germany is a little bit to small for the  Project  and  I
get direct help from there engineers to implement this.

The only difference between is, that my  Servers  are  running  entirely
"Debian GNU/Linux" and where it fit better, FreeBSD.

Thanks, Greetings and nice Day/Evening
    Michelle Konzack
    Systemadministrator
    Tamay Dogan Network
    Debian GNU/Linux Consultant

--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
<http://www.tamay-dogan.net/>                 Michelle Konzack
<http://www.can4linux.org/>                   Apt. 917
<http://www.flexray4linux.org/>               50, rue de Soultz
Jabber linux4michelle@...           67100 Strabourg/France
IRC    #Debian (irc.icq.com)                  Tel. DE: [phone]
ICQ    #328449886                             Tel. FR: [phone]
_______________________________________________
freebsd-isp@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-isp
To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..."