Software load balancers for red5?

[bererton]

We're looking into load balancing our red5 application and I have a
couple of questions that the list might have some experience with.

Has anyone configured red5, or RTMP/RTMPT/RTPMS in general to work
with a software load balancer such as HAProxy, pound, nginx, or other
balancer.  Our current requirements make origin/edge clustering
unnecessary (we're mostly just talking to a database through red5 for
now). Obviously a DNS or client side round robin would work, but it
would be great to pass connections off to the least loaded server if
we could.

We don't have access to the physical machines, so a software load
balancer would be best.

Any ideas/ experience with software load balancers and RTMP*?

thanks in advance for any help,
~Curt


p.s. a little rough draft on how to secure red5's default
configuration (this is very much a work in progress, I still have to
read through all of the config files):
 * Take a look at jasypt for encrypting properties files.
 * When red5 server starts it starts a java management extension bean
(JMX) which can be used to monitor the memory, threads etc. of the
server.
  * For more info about jmx start here
http://www-128.ibm.com/developerworks/java/library/j-jtp09196/index.html?ca=drs
  * The Spring bean that controls this is in conf/red5-common.xml.
find the jmx section:
   * Note that JMX communications are not sent over ssl by default.
   * Note that there is an html adapter port on 8082, and an rmi
adapter port on 9999
   * Furthermore the JMX user and password are defined in
conf/access.properties, note that the red5user has both read and
write, this should probably be changed to readonly for more security
   * Whether or not you change the above to access.properties, you
should definitely open the conf/password.properties and change the
default red5user password
 * There is a password file in conf/realm.properties change the
password, and you should probably obfuscate it as well.
 * If you were messing around with the admin panel you may have added
some admin users via that to webapps/admin/WEB-INF/users.properties,
also in the same directory the salt for the admin panel password
creator would probably be a good idea to change.
 * If your web application doesn't use remoting you can comment out
the remoting servlet in the web.xml file of your webapps' WEB-INF
directory.

_______________________________________________
Red5 mailing list
Red5@...
http://osflash.org/mailman/listinfo/red5_osflash.org

[Walter Tak]

1. add in your application a function that returns the amount of
rooms/instances/scopes and connected users
2a. develop a (php-)script that calls that function on your red5 server(s)
using AMFPHP via cron(job) once every xx minutes
2b. calculate the result (e.g. server A has 70 users and server B has 120
users) and store it in a small status-file (or db)
3a. have your flash-client-application call the (php-)script
3b the script reads the status-file or db
3c the script returns 'go connect to server A'
4 the flash client connects to server A , thus offloading server B

It's pretty easy to expand the script to check if any of the servers is
responding at all, in effect creating a failover system

Total time to develop, test and implement ; less than a day if you know
about java, (php-)script, cron(jobs).

Regards,
Walter

----- Original Message -----
From: "Curt Bererton" <curt.bererton@...>
To: "red5" <Red5@...>
Sent: Friday, 04 April 2008 20:02
Subject: [Red5] Software load balancers for red5?

> We're looking into load balancing our red5 application and I have a
> couple of questions that the list might have some experience with.
>
> Has anyone configured red5, or RTMP/RTMPT/RTPMS in general to work
> with a software load balancer such as HAProxy, pound, nginx, or other
> balancer.  Our current requirements make origin/edge clustering
> unnecessary (we're mostly just talking to a database through red5 for
> now). Obviously a DNS or client side round robin would work, but it
> would be great to pass connections off to the least loaded server if
> we could.
>
> We don't have access to the physical machines, so a software load
> balancer would be best.
>
> Any ideas/ experience with software load balancers and RTMP*?
>
> thanks in advance for any help,
> ~Curt
>
>
> p.s. a little rough draft on how to secure red5's default
> configuration (this is very much a work in progress, I still have to
> read through all of the config files):
> * Take a look at jasypt for encrypting properties files.
> * When red5 server starts it starts a java management extension bean
> (JMX) which can be used to monitor the memory, threads etc. of the
> server.
>  * For more info about jmx start here
> http://www-128.ibm.com/developerworks/java/library/j-jtp09196/index.html?ca=drs
>  * The Spring bean that controls this is in conf/red5-common.xml.
> find the jmx section:
>   * Note that JMX communications are not sent over ssl by default.
>   * Note that there is an html adapter port on 8082, and an rmi
> adapter port on 9999
>   * Furthermore the JMX user and password are defined in
> conf/access.properties, note that the red5user has both read and
> write, this should probably be changed to readonly for more security
>   * Whether or not you change the above to access.properties, you
> should definitely open the conf/password.properties and change the
> default red5user password
> * There is a password file in conf/realm.properties change the
> password, and you should probably obfuscate it as well.
> * If you were messing around with the admin panel you may have added
> some admin users via that to webapps/admin/WEB-INF/users.properties,
> also in the same directory the salt for the admin panel password
> creator would probably be a good idea to change.
> * If your web application doesn't use remoting you can comment out
> the remoting servlet in the web.xml file of your webapps' WEB-INF
> directory.
>
> _______________________________________________
> Red5 mailing list
> Red5@...
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
>
> --
> No virus found in this incoming message.
> Checked by AVG.
> Version: 7.5.519 / Virus Database: 269.22.5/1359 - Release Date: 04-04-08
> 08:23
>
>

_______________________________________________
Red5 mailing list
Red5@...
http://osflash.org/mailman/listinfo/red5_osflash.org

[Curt Bererton-2]

Hi Walter,

Thanks for your response... I did think about doing something similar
to this, my plan looked more like:
 - have each app server record usage stats in the DB (# connections,
memory etc.)
  - ie every few minutes they send stats to the DB
 - have client connect to any app server and ask it which app server
was the least loaded
 - disconnect, and reconnect to the least loaded server

The advantage to the PHP script is that the client doesn't need to
connect to anything before going. One advantage to the above plan is
that any app server could run the php script which would just querey
the DB and get the stats and return which app server to connect to, so
there wouldn't be a single server that told you which app server to
connect to.

One extra problem is that the servers will be running on Amazon's EC2,
so they are more likely to fail than a regular machine.

I haven't worked extensively with PHP, but this seems like something
that would be pretty trivial. I was hoping to only run red5 (all java)
and serve static content using jetty, and have absolutely no php code
at all (one less language to maintain). However, seems like your plan
might be faster performance than the connection plan above.

My thought with using an existing load balancing solution would be
that I wouldn't have to write any code in the future to do the
balancing. In any case either of the plans will work, I'll figure
something out.

thanks,
~Curt

>Message: 1
>Date: Fri, 4 Apr 2008 21:37:55 +0200
>From: "Walter Tak" <walter@...>
>Subject: Re: [Red5] Software load balancers for red5?
>To: <red5@...>
>Message-ID: <001401c8968b$6119a3f0$1e0313ac@dc2140>
>Content-Type: text/plain; format=flowed; charset="iso-8859-1";
>       reply-type=original

>1. add in your application a function that returns the amount of
>rooms/instances/scopes and connected users
>2a. develop a (php-)script that calls that function on your red5 server(s)
>using AMFPHP via cron(job) once every xx minutes
>2b. calculate the result (e.g. server A has 70 users and server B has 120
>users) and store it in a small status-file (or db)
>3a. have your flash-client-application call the (php-)script
>3b the script reads the status-file or db
>3c the script returns 'go connect to server A'
>4 the flash client connects to server A , thus offloading server B

>It's pretty easy to expand the script to check if any of the servers is
>responding at all, in effect creating a failover system
>Total time to develop, test and implement ; less than a day if you know
>about java, (php-)script, cron(jobs).
>Regards,
>Walter

_______________________________________________
Red5 mailing list
Red5@...
http://osflash.org/mailman/listinfo/red5_osflash.org

[Andy Shaules]

Im agreeing with walter, and you about a simple php or java setup like
that. You could make it on a java applet and DAO and avoid php. I am
into the idea of php front end on on host servicing multiple red5
servers in the background.

Pretty solid I think.

-----Original Message-----
From: red5-bounces@... [mailto:red5-bounces@...] On
Behalf Of Curt Bererton
Sent: Sunday, April 06, 2008 2:24 PM
To: red5
Subject: Re: [Red5] Software load balancers for red5?


Hi Walter,

Thanks for your response... I did think about doing something similar to
this, my plan looked more like:
 - have each app server record usage stats in the DB (# connections,
memory etc.)
  - ie every few minutes they send stats to the DB
 - have client connect to any app server and ask it which app server was
the least loaded
 - disconnect, and reconnect to the least loaded server

The advantage to the PHP script is that the client doesn't need to
connect to anything before going. One advantage to the above plan is
that any app server could run the php script which would just querey the
DB and get the stats and return which app server to connect to, so there
wouldn't be a single server that told you which app server to connect
to.

One extra problem is that the servers will be running on Amazon's EC2,
so they are more likely to fail than a regular machine.

I haven't worked extensively with PHP, but this seems like something
that would be pretty trivial. I was hoping to only run red5 (all java)
and serve static content using jetty, and have absolutely no php code at
all (one less language to maintain). However, seems like your plan might
be faster performance than the connection plan above.

My thought with using an existing load balancing solution would be that
I wouldn't have to write any code in the future to do the balancing. In
any case either of the plans will work, I'll figure something out.

thanks,
~Curt

>Message: 1
>Date: Fri, 4 Apr 2008 21:37:55 +0200
>From: "Walter Tak" <walter@...>
>Subject: Re: [Red5] Software load balancers for red5?
>To: <red5@...>
>Message-ID: <001401c8968b$6119a3f0$1e0313ac@dc2140>
>Content-Type: text/plain; format=flowed; charset="iso-8859-1";
>       reply-type=original

>1. add in your application a function that returns the amount of
>rooms/instances/scopes and connected users 2a. develop a (php-)script
>that calls that function on your red5 server(s) using AMFPHP via
>cron(job) once every xx minutes 2b. calculate the result (e.g. server A

>has 70 users and server B has 120
>users) and store it in a small status-file (or db)
>3a. have your flash-client-application call the (php-)script 3b the
>script reads the status-file or db 3c the script returns 'go connect to

>server A' 4 the flash client connects to server A , thus offloading
>server B

>It's pretty easy to expand the script to check if any of the servers is

>responding at all, in effect creating a failover system Total time to
>develop, test and implement ; less than a day if you know about java,
>(php-)script, cron(jobs). Regards,
>Walter

_______________________________________________
Red5 mailing list
Red5@... http://osflash.org/mailman/listinfo/red5_osflash.org


_______________________________________________
Red5 mailing list
Red5@...
http://osflash.org/mailman/listinfo/red5_osflash.org

[Walter Tak]

You could deploy tomcat to serve jsp-pages instead of PHP if you want to use
the same language.
PHP can be replaced by merely anything, from a bash-script, perl, python,
ruby to dot net, C#, etc.

A very safe setup would be

two Red5 servers
two application servers (not round-robinned)

Client tries to connect to app-server A. If unsuccessful try to connect to
app-server B.
After a successful connect the client receives the url of an available Red5
server.
Client connects to the specified Red5 server.
Client is happy

T.b.h. you could even think about having 2 or 3 Red5-servers and use Jetty
for the JSP pages to avoid the need of seperate app-server(s).

This would turn your dedicated app-servers into Red5 servers as well and
you'd have a 'farm' of independent nodes that can act as an app-server AND a
Red5-streaming server. Very easy to maintain and very easy to expand.

The behaviour of that farm would be more like a Raid-5 disk-setup, where
each newly added discs adds security (failover) AND capacity (diskspace) ;
in contrast to Raid-1 (mirror) where each added disc doesn't not always
improve capacity and security ; you need to add 2 at least and one of them
will be doing 'nothing'.

If your app requires a lot of power then this farm fits well since it's
fully scalable ; you could deploy 5, 20 or 600 servers.
The client would connect to a random server, get the best Red5 server to
connect to and (re)connect to another one.

Just load the hostnames of all available servers from an xml-file from the
webserver that hosts the SWF file.

Imho this is the optimum loadbalanced situation without requiring expensive
hardware loadbalancers or any point of failure. Even if 19 of 20 servers are
down the client still can connect to number 20 (it just takes some time to
get there since the client will try to connect to one of the servers at a
time ofcourse). You don't have to wait 20 seconds to connect because if the
client cannot connect to a given server in 2 seconds than chance is high
that that server is very busy so it should be skipped asap.

> Hi Walter,
>
> Thanks for your response... I did think about doing something similar
> to this, my plan looked more like:
> - have each app server record usage stats in the DB (# connections,
> memory etc.)
>  - ie every few minutes they send stats to the DB
> - have client connect to any app server and ask it which app server
> was the least loaded
> - disconnect, and reconnect to the least loaded server
>
> The advantage to the PHP script is that the client doesn't need to
> connect to anything before going. One advantage to the above plan is
> that any app server could run the php script which would just querey
> the DB and get the stats and return which app server to connect to, so
> there wouldn't be a single server that told you which app server to
> connect to.
>
> One extra problem is that the servers will be running on Amazon's EC2,
> so they are more likely to fail than a regular machine.
>
> I haven't worked extensively with PHP, but this seems like something
> that would be pretty trivial. I was hoping to only run red5 (all java)
> and serve static content using jetty, and have absolutely no php code
> at all (one less language to maintain). However, seems like your plan
> might be faster performance than the connection plan above.
>
> My thought with using an existing load balancing solution would be
> that I wouldn't have to write any code in the future to do the
> balancing. In any case either of the plans will work, I'll figure
> something out.
>
> thanks,
> ~Curt

--
I am using the free version of SPAMfighter for private users.
It has removed 282 spam emails to date.
Paying users do not have this message in their emails.
Get the free SPAMfighter here: http://www.spamfighter.com/len



_______________________________________________
Red5 mailing list
Red5@...
http://osflash.org/mailman/listinfo/red5_osflash.org

[Daniel Rossi-2]

I think ive posted in the past my custom As3 connect code which  
incorporates fallback and sound robin features would that be of any  
use ?

On 07/04/2008, at 10:04 PM, Walter Tak wrote:

> You could deploy tomcat to serve jsp-pages instead of PHP if you  
> want to use
> the same language.
> PHP can be replaced by merely anything, from a bash-script, perl,  
> python,
> ruby to dot net, C#, etc.
>
> A very safe setup would be
>
> two Red5 servers
> two application servers (not round-robinned)
>
> Client tries to connect to app-server A. If unsuccessful try to  
> connect to
> app-server B.
> After a successful connect the client receives the url of an  
> available Red5
> server.
> Client connects to the specified Red5 server.
> Client is happy
>
> T.b.h. you could even think about having 2 or 3 Red5-servers and use  
> Jetty
> for the JSP pages to avoid the need of seperate app-server(s).
>
> This would turn your dedicated app-servers into Red5 servers as well  
> and
> you'd have a 'farm' of independent nodes that can act as an app-
> server AND a
> Red5-streaming server. Very easy to maintain and very easy to expand.
>
> The behaviour of that farm would be more like a Raid-5 disk-setup,  
> where
> each newly added discs adds security (failover) AND capacity  
> (diskspace) ;
> in contrast to Raid-1 (mirror) where each added disc doesn't not  
> always
> improve capacity and security ; you need to add 2 at least and one  
> of them
> will be doing 'nothing'.
>
> If your app requires a lot of power then this farm fits well since  
> it's
> fully scalable ; you could deploy 5, 20 or 600 servers.
> The client would connect to a random server, get the best Red5  
> server to
> connect to and (re)connect to another one.
>
> Just load the hostnames of all available servers from an xml-file  
> from the
> webserver that hosts the SWF file.
>
> Imho this is the optimum loadbalanced situation without requiring  
> expensive
> hardware loadbalancers or any point of failure. Even if 19 of 20  
> servers are
> down the client still can connect to number 20 (it just takes some  
> time to
> get there since the client will try to connect to one of the servers  
> at a
> time ofcourse). You don't have to wait 20 seconds to connect because  
> if the
> client cannot connect to a given server in 2 seconds than chance is  
> high
> that that server is very busy so it should be skipped asap.
>
>
>
>> Hi Walter,
>>
>> Thanks for your response... I did think about doing something similar
>> to this, my plan looked more like:
>> - have each app server record usage stats in the DB (# connections,
>> memory etc.)
>> - ie every few minutes they send stats to the DB
>> - have client connect to any app server and ask it which app server
>> was the least loaded
>> - disconnect, and reconnect to the least loaded server
>>
>> The advantage to the PHP script is that the client doesn't need to
>> connect to anything before going. One advantage to the above plan is
>> that any app server could run the php script which would just querey
>> the DB and get the stats and return which app server to connect to,  
>> so
>> there wouldn't be a single server that told you which app server to
>> connect to.
>>
>> One extra problem is that the servers will be running on Amazon's  
>> EC2,
>> so they are more likely to fail than a regular machine.
>>
>> I haven't worked extensively with PHP, but this seems like something
>> that would be pretty trivial. I was hoping to only run red5 (all  
>> java)
>> and serve static content using jetty, and have absolutely no php code
>> at all (one less language to maintain). However, seems like your plan
>> might be faster performance than the connection plan above.
>>
>> My thought with using an existing load balancing solution would be
>> that I wouldn't have to write any code in the future to do the
>> balancing. In any case either of the plans will work, I'll figure
>> something out.
>>
>> thanks,
>> ~Curt
>
>
> --
> I am using the free version of SPAMfighter for private users.
> It has removed 282 spam emails to date.
> Paying users do not have this message in their emails.
> Get the free SPAMfighter here: http://www.spamfighter.com/len
>
>
>
> _______________________________________________
> Red5 mailing list
> Red5@...
> http://osflash.org/mailman/listinfo/red5_osflash.org

_______________________________________________
Red5 mailing list
Red5@...
http://osflash.org/mailman/listinfo/red5_osflash.org

[Daniel Rossi-2]

On 07/04/2008, at 10:25 PM, Daniel Rossi wrote:

> I think ive posted in the past my custom As3 connect code which
> incorporates fallback and sound robin features would that be of any
> use ?

Hahaha im cutting up audio at the moment, fraudian slip. I meant round  
robin !

_______________________________________________
Red5 mailing list
Red5@...
http://osflash.org/mailman/listinfo/red5_osflash.org

[sledjama]

I am looking out for a load balancer for 3 red5 servers
While this technique seems fine.
The problem i have is using these 3 servers as if they are one.
for instance, the sharedobjects, i need then to appear as if they are on th samee server.
i will need to be accessing shared objects from another, which may be on another server.

Looks messy to me.

bererton wrote:

We're looking into load balancing our red5 application and I have a
couple of questions that the list might have some experience with.

Has anyone configured red5, or RTMP/RTMPT/RTPMS in general to work
with a software load balancer such as HAProxy, pound, nginx, or other
balancer.  Our current requirements make origin/edge clustering
unnecessary (we're mostly just talking to a database through red5 for
now). Obviously a DNS or client side round robin would work, but it
would be great to pass connections off to the least loaded server if
we could.

We don't have access to the physical machines, so a software load
balancer would be best.

Any ideas/ experience with software load balancers and RTMP*?

thanks in advance for any help,
~Curt


p.s. a little rough draft on how to secure red5's default
configuration (this is very much a work in progress, I still have to
read through all of the config files):
 * Take a look at jasypt for encrypting properties files.
 * When red5 server starts it starts a java management extension bean
(JMX) which can be used to monitor the memory, threads etc. of the
server.
  * For more info about jmx start here
http://www-128.ibm.com/developerworks/java/library/j-jtp09196/index.html?ca=drs
  * The Spring bean that controls this is in conf/red5-common.xml.
find the jmx section:
   * Note that JMX communications are not sent over ssl by default.
   * Note that there is an html adapter port on 8082, and an rmi
adapter port on 9999
   * Furthermore the JMX user and password are defined in
conf/access.properties, note that the red5user has both read and
write, this should probably be changed to readonly for more security
   * Whether or not you change the above to access.properties, you
should definitely open the conf/password.properties and change the
default red5user password
 * There is a password file in conf/realm.properties change the
password, and you should probably obfuscate it as well.
 * If you were messing around with the admin panel you may have added
some admin users via that to webapps/admin/WEB-INF/users.properties,
also in the same directory the salt for the admin panel password
creator would probably be a good idea to change.
 * If your web application doesn't use remoting you can comment out
the remoting servlet in the web.xml file of your webapps' WEB-INF
directory.

_______________________________________________
Red5 mailing list
[e-mail]
http://osflash.org/mailman/listinfo/red5_osflash.org

[H.R. Soltani]

The answer lies in third party solutions like Heart Beat and other high
availability  software.
Ubuntu 10.4 and the eucalyptus cloud solutions may be an option.
Also, we load balanced red5 servers on Amazon EC2 for a client.

Warmest Regards,
H.R. Soltani
Star Federal, Inc.
Management, Consulting, and Development
When you prosper, we succeed!
Mailing Address: 228 Park Avenue South, #24165, New York, NY 10003
New York City Phone: [phone]
Washington DC Phone: [phone]
Toll-Free Phone: [phone]
Fax: [phone]
Website: StarFederal.com

-----Original Message-----
From: red5-bounces@... [mailto:red5-bounces@...] On Behalf
Of sledjama
Sent: Sunday, April 03, 2011 11:09 AM
To: Red5@...
Subject: Re: [Red5] Software load balancers for red5?


I am looking out for a load balancer for 3 red5 servers
While this technique seems fine.
The problem i have is using these 3 servers as if they are one.
for instance, the sharedobjects, i need then to appear as if they are on th
samee server.
i will need to be accessing shared objects from another, which may be on
another server.

Looks messy to me.







bererton wrote:

>
> We're looking into load balancing our red5 application and I have a
> couple of questions that the list might have some experience with.
>
> Has anyone configured red5, or RTMP/RTMPT/RTPMS in general to work
> with a software load balancer such as HAProxy, pound, nginx, or other
> balancer.  Our current requirements make origin/edge clustering
> unnecessary (we're mostly just talking to a database through red5 for
> now). Obviously a DNS or client side round robin would work, but it
> would be great to pass connections off to the least loaded server if
> we could.
>
> We don't have access to the physical machines, so a software load
> balancer would be best.
>
> Any ideas/ experience with software load balancers and RTMP*?
>
> thanks in advance for any help,
> ~Curt
>
>
> p.s. a little rough draft on how to secure red5's default
> configuration (this is very much a work in progress, I still have to
> read through all of the config files):
>  * Take a look at jasypt for encrypting properties files.
>  * When red5 server starts it starts a java management extension bean
> (JMX) which can be used to monitor the memory, threads etc. of the
> server.
>   * For more info about jmx start here
>

http://www-128.ibm.com/developerworks/java/library/j-jtp09196/index.html?ca=
drs

>   * The Spring bean that controls this is in conf/red5-common.xml.
> find the jmx section:
>    * Note that JMX communications are not sent over ssl by default.
>    * Note that there is an html adapter port on 8082, and an rmi
> adapter port on 9999
>    * Furthermore the JMX user and password are defined in
> conf/access.properties, note that the red5user has both read and
> write, this should probably be changed to readonly for more security
>    * Whether or not you change the above to access.properties, you
> should definitely open the conf/password.properties and change the
> default red5user password
>  * There is a password file in conf/realm.properties change the
> password, and you should probably obfuscate it as well.
>  * If you were messing around with the admin panel you may have added
> some admin users via that to webapps/admin/WEB-INF/users.properties,
> also in the same directory the salt for the admin panel password
> creator would probably be a good idea to change.
>  * If your web application doesn't use remoting you can comment out
> the remoting servlet in the web.xml file of your webapps' WEB-INF
> directory.
>
> _______________________________________________
> Red5 mailing list
> Red5@...
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>

--
View this message in context:
http://old.nabble.com/Software-load-balancers-for-red5--tp16502311p31306293.
html
Sent from the Red5 - English mailing list archive at Nabble.com.


_______________________________________________
Red5 mailing list
Red5@...
http://osflash.org/mailman/listinfo/red5_osflash.org


_______________________________________________
Red5 mailing list
Red5@...
http://osflash.org/mailman/listinfo/red5_osflash.org

[sledjama]

pls contact me at
sledgahammer23401[at]yahooDotCom

H.R. Soltani wrote:

The answer lies in third party solutions like Heart Beat and other high
availability  software.
Ubuntu 10.4 and the eucalyptus cloud solutions may be an option.
Also, we load balanced red5 servers on Amazon EC2 for a client.

Warmest Regards,
H.R. Soltani
Star Federal, Inc.
Management, Consulting, and Development
When you prosper, we succeed!
Mailing Address: 228 Park Avenue South, #24165, New York, NY 10003
New York City Phone: [phone]
Washington DC Phone: [phone]
Toll-Free Phone: [phone]
Fax: [phone]
Website: StarFederal.com