Configuring virtual hosts on the fly

[Scott Gifford]

Hello,

I'm working on an Apache configuration for a cluster of machines
serving a variety of virtual hosts.

New virtual hosts are provisioned by a Web application, and all
information needed to set up the virtual hosts is in a database table.
I would like my Apache to set up the virtual hosts on the fly based on
information in the tables.

I have seen examples for generating the virtual hosts configuration
from a database when Apache starts.  That leaves me with the problem
of how the unprivileged Web application doing the provisioning can
reliably trigger a reload across an entire cluster, and also what to
do if something goes wrong so I don't end up with a whole cluster of
dead Web servers.

It seems much cleaner to me to let Apache look up the information for
a virtual host as it is needed.  Conceptually, when each request comes
in, I would like Apache to look at the Host: header, build a temporary
virtual host entry from that, and use it to service the request.  I
will use caching to make it reasonably fast.  Then the provisioning
script just writes to the database table and is done, and I can handle
any errors on a per-request basis without Apache failing to start.

I see some hooks in PerlTransHandler and PerlMapToStorageHandler that
seem like they can almost do what I want, but I don't see how to set
other virtual host parameters, like ServerAdmin, UseCanonicalName,
etc.

I'm just starting to work on this, and I thought I would see if
anybody had tried anything similar before and had some suggestions for
what to do, or what not to do.

Thanks!

----Scott.

[awarnier]

Scott Gifford wrote:
> Hello,
>
> I'm working on an Apache configuration for a cluster of machines
> serving a variety of virtual hosts.
>
> New virtual hosts are provisioned by a Web application, and all
> information needed to set up the virtual hosts is in a database table.
> I would like my Apache to set up the virtual hosts on the fly based on
> information in the tables.
Sounds intersting.

>
> I have seen examples for generating the virtual hosts configuration
> from a database when Apache starts.  That leaves me with the problem
> of how the unprivileged Web application doing the provisioning can
> reliably trigger a reload across an entire cluster, and also what to
> do if something goes wrong so I don't end up with a whole cluster of
> dead Web servers.
>
> It seems much cleaner to me to let Apache look up the information for
> a virtual host as it is needed.  Conceptually, when each request comes
> in, I would like Apache to look at the Host: header, build a temporary
> virtual host entry from that, and use it to service the request.  I
> will use caching to make it reasonably fast.  Then the provisioning
> script just writes to the database table and is done, and I can handle
> any errors on a per-request basis without Apache failing to start.
>
> I see some hooks in PerlTransHandler and PerlMapToStorageHandler that
> seem like they can almost do what I want, but I don't see how to set
> other virtual host parameters, like ServerAdmin

http://perl.apache.org/docs/2.0/api/Apache2/ServerRec.html#C_server_admin_

, UseCanonicalName,
> etc.
>
> I'm just starting to work on this, and I thought I would see if
> anybody had tried anything similar before and had some suggestions for
> what to do, or what not to do.
>
I have never tried this myself, but the mod_perl 2.x documentation
contains some hints that leads to believe that it should be possible.
Like
http://perl.apache.org/docs/2.0/api/Apache2/ServerRec.html
http://perl.apache.org/docs/2.0/api/Apache2/ServerUtil.html
http://perl.apache.org/docs/2.0/api/Apache2/PerlSections.html
...

I would start by defining a single VirtualHost (which will thus be the
default host), with a variety of configuration sections, and see if I
can *modify* them on-the-fly.  Assuming that the VirtualHosts which you
want to configure on-the-fly have some high degree of similarity, that
sounds easier to do than building a whole configuration each time from
scratch.
Like
<Location /some-highly-unlikely-prefix-001>
...
</Location>
<Location /some-highly-unlikely-prefix-002>
...
</Location>
and then modify the first Location to become
<Location /some-real-one>
...
</Location>
etc..

[Torsten Foertsch]

On Sun 11 Oct 2009, André Warnier wrote:

> > I see some hooks in PerlTransHandler and PerlMapToStorageHandler
> > that seem like they can almost do what I want, but I don't see how
> > to set other virtual host parameters, like ServerAdmin
>
> http://perl.apache.org/docs/2.0/api/Apache2/ServerRec.html#C_server_a
>dmin_
>
> , UseCanonicalName,
>
> > etc.
> >
> > I'm just starting to work on this, and I thought I would see if
> > anybody had tried anything similar before and had some suggestions
> > for what to do, or what not to do.
>
> I have never tried this myself, but the mod_perl 2.x documentation
> contains some hints that leads to believe that it should be possible.

Some things could work this way, others definitely will not. For example
you cannot create an IP based vhost on demand. You cannot create log
files on the fly. Well, you can but you cannot create them as root as
it is done during startup.

In general its not a good idea to mess with things that are not marked
as applicable in directory context for the following reason. Firstly,
you have to make sure to reset them when you request is done. See the
implementation of $r->document_root for example. And secondly, in a
threaded environment changing those values in one thread can affect the
behavior of other requests in other threads because they are shared.

UseCanonicalName is applyable in directory context. So, you can safely
set it via $r->add_config. It may however be useless to set it in a
translation or map2storage handler because if you have a directory or
location container or even a .htaccess file that sets the directive it
overwrites anything set in a translation handler. Best phase to do that
is probably fixup.

ServerAdmin is valid only in server config and virtual host context
(http://httpd.apache.org/docs/2.2/mod/core.html#serveradmin). So I
wouldn't mess with that. Better create custom error pages.

Torsten

--
Need professional mod_perl support?
Just hire me: torsten.foertsch@...

[Adam Prime]

Scott Gifford wrote:

> Hello,
>
> I'm working on an Apache configuration for a cluster of machines
> serving a variety of virtual hosts.
>
> New virtual hosts are provisioned by a Web application, and all
> information needed to set up the virtual hosts is in a database table.
> I would like my Apache to set up the virtual hosts on the fly based on
> information in the tables.
>
> I have seen examples for generating the virtual hosts configuration
> from a database when Apache starts.  That leaves me with the problem
> of how the unprivileged Web application doing the provisioning can
> reliably trigger a reload across an entire cluster, and also what to
> do if something goes wrong so I don't end up with a whole cluster of
> dead Web servers.
>
> It seems much cleaner to me to let Apache look up the information for
> a virtual host as it is needed.  Conceptually, when each request comes
> in, I would like Apache to look at the Host: header, build a temporary
> virtual host entry from that, and use it to service the request.  I
> will use caching to make it reasonably fast.  Then the provisioning
> script just writes to the database table and is done, and I can handle
> any errors on a per-request basis without Apache failing to start.
>
> I see some hooks in PerlTransHandler and PerlMapToStorageHandler that
> seem like they can almost do what I want, but I don't see how to set
> other virtual host parameters, like ServerAdmin, UseCanonicalName,
> etc.
>
> I'm just starting to work on this, and I thought I would see if
> anybody had tried anything similar before and had some suggestions for
> what to do, or what not to do.
>
> Thanks!

I did some cursory investigation into something sort of like this, at
least the idea that the application would be able to create new virtual
hosts, and that those vhosts should start 'working' (for some definition
of working) immediately.

The approach that I had intended to take with regards to getting the
name based vhosts to work immediately (without a restart) was to have a
default virtual host that would detect requests that were coming in for
vhosts that we should be able to handle, and then setting up the handler
stack for that request using push_handlers in an init handler.  There
are some problems with this approach (segmenting logs for one), but the
idea for me was to get the thing to work right away, and if it took
until the next restart to get the vhost actually configured as a vhost
properly, that wasn't a huge deal.

Not sure if that's any help or not, but...

Adam

[William T]

On Sun, Oct 11, 2009 at 11:54 AM, Scott Gifford
<sgifford@...> wrote:
> Hello,
>
> I'm working on an Apache configuration for a cluster of machines
> serving a variety of virtual hosts.

I would not try to unify disparate configs into one unless each server
is actually going to service all the virtual hosts your pulling in.
You would unnecessarily couple configs together that don't need to be.

> It seems much cleaner to me to let Apache look up the information for
> a virtual host as it is needed.  Conceptually, when each request comes
> in, I would like Apache to look at the Host: header, build a temporary
> virtual host entry from that, and use it to service the request.  I
> will use caching to make it reasonably fast.  Then the provisioning
> script just writes to the database table and is done, and I can handle
> any errors on a per-request basis without Apache failing to start.

Sounds like you might be pushing the envelope on what Apache can
actually do.  If you cannot solve the problem in Apache you could
consider relying on Apache default vhost as a way to funnel all
requests into a Perl "dynamic vhost" implementation.  You get exactly
what your looking for, but you tradeoff the speed and maturity of
Apache vhost code for yours.

-wjt

[Scott Gifford]

Thanks William, comments inline...

William T <dietbuddha@...> writes:

> On Sun, Oct 11, 2009 at 11:54 AM, Scott Gifford
> <sgifford@...> wrote:
>> Hello,
>>
>> I'm working on an Apache configuration for a cluster of machines
>> serving a variety of virtual hosts.
>
> I would not try to unify disparate configs into one unless each server
> is actually going to service all the virtual hosts your pulling in.
> You would unnecessarily couple configs together that don't need to
> be.

Each server will actually service all of the virtual hosts, so there
won't be anything unnecessary there.

[...]

> Sounds like you might be pushing the envelope on what Apache can
> actually do.  If you cannot solve the problem in Apache you could
> consider relying on Apache default vhost as a way to funnel all
> requests into a Perl "dynamic vhost" implementation.  You get exactly
> what your looking for, but you tradeoff the speed and maturity of
> Apache vhost code for yours.

Thanks, I will definitely take a look!  Do you have any recommended
reading on this, or just what Google turns up for "dynamic vhost"?

Thanks again!

---Scott.

[William T]

On Tue, Oct 13, 2009 at 8:08 AM, Scott Gifford
<sgifford@...> wrote:
>> Sounds like you might be pushing the envelope on what Apache can
>> actually do.  If you cannot solve the problem in Apache you could
>> consider relying on Apache default vhost as a way to funnel all
>> requests into a Perl "dynamic vhost" implementation.  You get exactly
>> what your looking for, but you tradeoff the speed and maturity of
>> Apache vhost code for yours.
>
> Thanks, I will definitely take a look!  Do you have any recommended
> reading on this, or just what Google turns up for "dynamic vhost"?

I don't know of anything that specifically does this, then again I
havn't looked.  I was mainly thinking about pushing the problem into a
different layer since I wasn't sure the Apache layer would work.  If
it were me I would probably start by looking for through "Mass Dynamic
Virtual Hosting" and make sure you can't solve your problem in the
Apache layer, or with a ModPerl Apache module to extend Apache (either
one already written, or one you write yourself).  Then move on to Perl
code that does webserving.

-wjt

[Ryan Yagatich-2]

What about mod_vhost_alias? (
http://httpd.apache.org/docs/2.0/mod/mod_vhost_alias.html )

<quote>
Summary

This module creates dynamically configured virtual hosts, by allowing the IP
address and/or the Host: header of the HTTP request to be used as part of
the pathname to determine what files to serve. This allows for easy use of a
huge number of virtual hosts with similar configurations.
</quote>




-----Original Message-----
From: Scott Gifford [mailto:sgifford@...]
Sent: Tuesday, October 13, 2009 11:09 AM
To: William T
Cc: modperl@...
Subject: Re: Configuring virtual hosts on the fly

Thanks William, comments inline...

William T <dietbuddha@...> writes:

> On Sun, Oct 11, 2009 at 11:54 AM, Scott Gifford
> <sgifford@...> wrote:
>> Hello,
>>
>> I'm working on an Apache configuration for a cluster of machines
>> serving a variety of virtual hosts.
>
> I would not try to unify disparate configs into one unless each server
> is actually going to service all the virtual hosts your pulling in.
> You would unnecessarily couple configs together that don't need to
> be.

Each server will actually service all of the virtual hosts, so there
won't be anything unnecessary there.

[...]

> Sounds like you might be pushing the envelope on what Apache can
> actually do.  If you cannot solve the problem in Apache you could
> consider relying on Apache default vhost as a way to funnel all
> requests into a Perl "dynamic vhost" implementation.  You get exactly
> what your looking for, but you tradeoff the speed and maturity of
> Apache vhost code for yours.

Thanks, I will definitely take a look!  Do you have any recommended
reading on this, or just what Google turns up for "dynamic vhost"?

Thanks again!

---Scott.

[Michael Peters]

Looking at this from a different perspective, have you tried writing a
monitoring program that looks for updates to the database and then would
restart the appropriate apache servers on the various machines. It would
do them one at a time (taking them out of rotation from your load
balancer). It wouldn't be immediate response, could definitely be done
in under a minute.

--
Michael Peters
Plus Three, LP

[Scott Gifford]

Michael Peters <mpeters@...> writes:

> Looking at this from a different perspective, have you tried writing a
> monitoring program that looks for updates to the database and then
> would restart the appropriate apache servers on the various
> machines. It would do them one at a time (taking them out of rotation
> from your load balancer). It wouldn't be immediate response, could
> definitely be done in under a minute.

Thanks Michael,

I thought about that.  It is probably what I will do if I can't get
something a little cleaner.

I have had mixed experiences in the past with automatically restarting
Apache after a configuration change.  It is very easy to end up with
something unexpected in the configuration, which causes the
configuration to fail, which causes apache to stop.  And with a
cluster, that would happen on all hosts.

Now, ideally, I wouldn't make any errors in my configuration code and
this would never be an issue.  But I have learned the hard way that
making things as foolproof as possible is the best way to protect
myself from my own foolishness.  :-)

When I have done this in the past, I have done it with generating
configuration files, so of course one misplaced newline or
angle-bracket will kill the server.  Maybe generating the
configuration directly from a <Perl> section is more robust?  Maybe
there are ways to catch configuration errors in that layer and handle
them cleanly without preventing Apache from starting?  Any suggestions
in this area would be appreciated.

Thanks again!

-----Scott.

[Scott Gifford]

"Ryan Yagatich" <ryagatich@...> writes:

> What about mod_vhost_alias? (
> http://httpd.apache.org/docs/2.0/mod/mod_vhost_alias.html )
>
> <quote>
> Summary
>
> This module creates dynamically configured virtual hosts, by allowing the IP
> address and/or the Host: header of the HTTP request to be used as part of
> the pathname to determine what files to serve. This allows for easy use of a
> huge number of virtual hosts with similar configurations.
> </quote>

Thanks!  That looks very close to what I want.  I wonder if there is a
way to do a database lookup and substitute that?...

----Scott.

[Joel-3]

I thought I'd weigh in on two items of note....

On Oct 13, 2009, at 12:17 PM, Scott Gifford wrote:

> When I have done this in the past, I have done it with generating
> configuration files, so of course one misplaced newline or
> angle-bracket will kill the server.  Maybe generating the
> configuration directly from a <Perl> section is more robust?  Maybe
> there are ways to catch configuration errors in that layer and handle
> them cleanly without preventing Apache from starting?  Any suggestions
> in this area would be appreciated.
>


It seems to me that you are assuming that your script must apply the  
new configuration to the live apache server.

Why not have your script simply apply a potential new configuration to  
a "test" apache installation (separate from all others), have it  
restart apache there, hit a few critical URLs to test functionality  
and then report to you if there's an error. If it's successful, it can  
then apply that config to the live cluster, keeping fingers crossed,  
of course. (and making sure your monitoring software is going to pull  
the fire alarm if the cluster goes down.)

If there's an error, you have some time to gracefully solve it and  
your system continues to run in the meantime. Of course, this means  
that the new configuration wasn't applied, but in the grand scheme of  
things, a missing update to a config is far, far better than taking  
down an entire cluster because your automatic configurator has a screw  
loose.


On another note, regarding mod_vhost_alias and a database lookup to  
dynamically do something...

> "Ryan Yagatich" <ryagatich@...> writes:
>
>> What about mod_vhost_alias? (
>> http://httpd.apache.org/docs/2.0/mod/mod_vhost_alias.html )
>>
>> <quote>
>> Summary
>>
>> This module creates dynamically configured virtual hosts, by  
>> allowing the IP
>> address and/or the Host: header of the HTTP request to be used as  
>> part of
>> the pathname to determine what files to serve. This allows for easy  
>> use of a
>> huge number of virtual hosts with similar configurations.
>> </quote>
>
> Thanks!  That looks very close to what I want.  I wonder if there is a
> way to do a database lookup and substitute that?...
>

In my experience, you really want to minimize the hits on your  
database and placing a lookup of any kind for -every- connection to  
apache would be murderous regardless of which database you're using.

--Joel

[Michael Peters]

On 10/13/2009 12:17 PM, Scott Gifford wrote:

> I have had mixed experiences in the past with automatically restarting
> Apache after a configuration change.  It is very easy to end up with
> something unexpected in the configuration, which causes the
> configuration to fail, which causes apache to stop.  And with a
> cluster, that would happen on all hosts.

If you run apache with a -t option it will check that the configuration
is sane. I'm assuming that your configuration is auto-generated by this
monitor somehow, then you can create a new config file and test it with

   httpd -t -f /path/to/new/file

Then if it's ok, you can move it to the real location and do the restart.

> When I have done this in the past, I have done it with generating
> configuration files, so of course one misplaced newline or
> angle-bracket will kill the server.

I generally prefer generated config files, especially if you have
multiple sites with basically the same config. I also use templated
config files so that the template I'm working on looks a lot like the
finished product. It's easier to prevent typos this way, IMO.

--
Michael Peters
Plus Three, LP

[Scott Gifford]

Joel Richard <perl@...> writes:

> I thought I'd weigh in on two items of note....
>
> On Oct 13, 2009, at 12:17 PM, Scott Gifford wrote:
>
>> When I have done this in the past, I have done it with generating
>> configuration files, so of course one misplaced newline or
>> angle-bracket will kill the server.  Maybe generating the
>> configuration directly from a <Perl> section is more robust?  Maybe
>> there are ways to catch configuration errors in that layer and handle
>> them cleanly without preventing Apache from starting?  Any suggestions
>> in this area would be appreciated.
>>
>
>
> It seems to me that you are assuming that your script must apply the
> new configuration to the live apache server.
>
> Why not have your script simply apply a potential new configuration to
> a "test" apache installation (separate from all others), have it
> restart apache there, hit a few critical URLs to test functionality
> and then report to you if there's an error. If it's successful, it can
> then apply that config to the live cluster, keeping fingers crossed,
> of course. (and making sure your monitoring software is going to pull
> the fire alarm if the cluster goes down.)

That's a good idea, though a bit involved to implement.

> If there's an error, you have some time to gracefully solve it and

Yeah, good point, thanks.

[...]

>> Thanks!  That looks very close to what I want.  I wonder if there is a
>> way to do a database lookup and substitute that?...
>>
>
> In my experience, you really want to minimize the hits on your
> database and placing a lookup of any kind for -every- connection to
> apache would be murderous regardless of which database you're using.

Right, absolutely the result would have to be cached.  Then I think it
would be no problem.

----Scott.

[Scott Gifford]

Scott Gifford <sgifford@...> writes:

[...]

> I see some hooks in PerlTransHandler and PerlMapToStorageHandler that
> seem like they can almost do what I want, but I don't see how to set
> other virtual host parameters, like ServerAdmin, UseCanonicalName,
> etc.

I was able to get something I like working in PerlTransHandler.  I
have it configured like this:

    PerlTransHandler My::VirtualHost::TransHandler

Then in My::VirtualHost::TransHandler:

    our %sitepath;
    sub handler {
      my $r = shift;
      my $hn = $r->hostname();
      if (!$hn) {
          return Apache2::Const::DECLINED;
      }
      if (!exists($sitepath{$hn})) {
        $sitepath{$hn} = db_lookup($hn);
      }
      if ($sitepath{$hn}) {
        $r->filename($sitepath{$hn}.$r->uri());
        return Apache2::Const::OK;
      } else {
        return Apache2::Const::DECLINED;
    }

Thanks for all the suggestions, they were very helpful!

----Scott.