A dirty patch for serving static data with fcgi

[Ozgur]

Hi

It's been a few days with erlang, and one day with yaws, so please warn me if i'm digging
the wrong hole :)

Considering replacing my lighttpd+django setups with yaws+django, i began playing with yaws yesterday.
By following the rails on yaws example setting at http://wiki.github.com/klacke/yaws/run-rails-under-yaws
managed to run django with fcgi on yaws. One thing that bugged me was, all the requests are passed to  
fcgi instance, whether it's dynamic or static content request, causing unnecessary load on fcgi app.
see snippet below

out(Arg) ->
    case yaws_cgi:call_fcgi_responder(Arg) of
        R when is_list(R) ->
            case proplists:get_value(status, R) of
                404 -> {page, Arg#arg.server_path};  % Let Yaws try to serve static files
                _   -> R
            end;

        X -> X
    end.

i've added a per-server "static_paths" parameter to yaws.conf. sample usage below
 <server blah>
 ...
 static_paths = /media/, /files/
 </server

which enables some more control over the static content handling in my django appmod. snippet below

...
out(Arg) ->
    case lists:filter(fun(Y) -> string:rstr(Arg#arg.server_path, Y) =/= 0 end, Arg#arg.static_paths) of
        [_|_] ->
        %% serve static
        error_logger:info_msg("static serving ~p~n",[Arg#arg.server_path]),
        {page, Arg#arg.server_path};
        _ ->
           
            error_logger:info_msg("dynamic serving ~p~n",[Arg#arg.server_path]),
            case yaws_cgi:call_fcgi_responder(Arg) of
                R when is_list(R) ->
                    case proplists:get_value(status, R) of
                    404 ->
                        {page, Arg#arg.server_path}; % Let Yaws try to serve static files
                    _  -> R
                    end;

                X -> X
            end
    end.

Regards

--
Ozgur <hinoglu@...>


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Erlyaws-list mailing list
Erlyaws-list@...
https://lists.sourceforge.net/lists/listinfo/erlyaws-list

[Steve Vinoski-2]

On Sun, Sep 13, 2009 at 10:58 AM, Ozgur <hinoglu@...> wrote:

Hi

It's been a few days with erlang, and one day with yaws, so please warn me if i'm digging
the wrong hole :)

Considering replacing my lighttpd+django setups with yaws+django, i began playing with yaws yesterday.
By following the rails on yaws example setting at http://wiki.github.com/klacke/yaws/run-rails-under-yaws
managed to run django with fcgi on yaws. One thing that bugged me was, all the requests are passed to
fcgi instance, whether it's dynamic or static content request, causing unnecessary load on fcgi app.
see snippet below

out(Arg) ->
   case yaws_cgi:call_fcgi_responder(Arg) of
       R when is_list(R) ->
           case proplists:get_value(status, R) of
               404 -> {page, Arg#arg.server_path};  % Let Yaws try to serve static files
               _   -> R
           end;

       X -> X
   end.

i've added a per-server "static_paths" parameter to yaws.conf. sample usage below
 <server blah>
 ...
 static_paths = /media/, /files/
 </server

which enables some more control over the static content handling in my django appmod. snippet below

...
out(Arg) ->
   case lists:filter(fun(Y) -> string:rstr(Arg#arg.server_path, Y) =/= 0 end, Arg#arg.static_paths) of
       [_|_] ->
       %% serve static
       error_logger:info_msg("static serving ~p~n",[Arg#arg.server_path]),
       {page, Arg#arg.server_path};
       _ ->

           error_logger:info_msg("dynamic serving ~p~n",[Arg#arg.server_path]),
           case yaws_cgi:call_fcgi_responder(Arg) of
               R when is_list(R) ->
                   case proplists:get_value(status, R) of
                   404 ->
                       {page, Arg#arg.server_path}; % Let Yaws try to serve static files
                   _  -> R
                   end;

               X -> X
           end
   end.

I don't think it's necessary to add a new field to the server conf for this. An easy way to do this is to examine the URI in the Arg parameter of your out() function to see if the request is targeting one of your static paths, and take appropriate action if it is. If you'd rather keep the list of static paths out of your code so it can be easily changed, you could use the opaque field of the server conf and set it to whatever you like. You could also use a yapp instead of an appmod and keep the list of static paths as part of the environment in your .app file.

Can you try one of these approaches instead and see how you get on, then let us know?

--steve

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Erlyaws-list mailing list
Erlyaws-list@...
https://lists.sourceforge.net/lists/listinfo/erlyaws-list

[Ozgur]

> and take appropriate action if it is. If you'd rather keep the list of
> static paths out of your code so it can be easily changed, you could use the
> opaque field of the server conf and set it to whatever you like. You could
> also use a yapp instead of an appmod and keep the list of static paths as
> part of the environment in your .app file.
>
> Can you try one of these approaches instead and see how you get on, then let
> us know?


Steve thanks for pointers. following is my final mod_django.erl with vanilla yaws :)
seems to work good

out(Arg) ->
    case proplists:get_value("static_paths", Arg#arg.opaque) of
        P when is_list(P) ->
            Sp = lists:filter(fun(Y) -> Y =/= [] end,
                lists:map(fun(X)-> string:strip(X) end, string:tokens(P, ","))),
           
            case lists:filter(fun(Y) -> string:rstr(Arg#arg.server_path, Y) == 1 end, Sp) of
                [_|_] ->
               
                    %% serve static
                    error_logger:info_msg("static serving ~p~n",[Arg#arg.server_path]),
                    {page, Arg#arg.server_path};
           
                _ ->
               
                error_logger:info_msg("dynamic serving ~p~n",[Arg#arg.server_path]),
                case yaws_cgi:call_fcgi_responder(Arg) of
                    R when is_list(R) ->
                        case proplists:get_value(status, R) of
                        404 ->
                            {page, Arg#arg.server_path}; % Let Yaws try to serve static files
                        _  -> R
                        end;

                    X -> X
                end
            end;
        _ ->
       
            {page, Arg#arg.server_path} % Let Yaws try to serve static files
    end.





--
Ozgur <hinoglu@...>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Erlyaws-list mailing list
Erlyaws-list@...
https://lists.sourceforge.net/lists/listinfo/erlyaws-list

[Steve Vinoski-2]

On Sun, Sep 13, 2009 at 4:56 PM, Ozgur <hinoglu@...> wrote:

Steve thanks for pointers. following is my final mod_django.erl with vanilla yaws :)
seems to work good

out(Arg) ->
   case proplists:get_value("static_paths", Arg#arg.opaque) of
       P when is_list(P) ->
           Sp = lists:filter(fun(Y) -> Y =/= [] end,
               lists:map(fun(X)-> string:strip(X) end, string:tokens(P, ","))),

           case lists:filter(fun(Y) -> string:rstr(Arg#arg.server_path, Y) == 1 end, Sp) of

               [_|_] ->

                   %% serve static
                   error_logger:info_msg("static serving ~p~n",[Arg#arg.server_path]),
                   {page, Arg#arg.server_path};

               _ ->

               error_logger:info_msg("dynamic serving ~p~n",[Arg#arg.server_path]),
               case yaws_cgi:call_fcgi_responder(Arg) of
                   R when is_list(R) ->
                       case proplists:get_value(status, R) of
                       404 ->
                           {page, Arg#arg.server_path}; % Let Yaws try to serve static files
                       _  -> R
                       end;

                   X -> X
               end

           end;

       _ ->

           {page, Arg#arg.server_path} % Let Yaws try to serve static files

   end.

Great, glad to hear it works for you. You might consider adding an entry to the Yaws wiki about your work.

There's one other part of your original patch that caught my eye: the increase on the FCGI timeout. Should that be boosted? Both your patch and the Rails integration wiki entry mention that the timeout is currently too low. I can make that change if needed, just let me know.

--steve

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Erlyaws-list mailing list
Erlyaws-list@...
https://lists.sourceforge.net/lists/listinfo/erlyaws-list

[Ozgur]

> Great, glad to hear it works for you. You might consider adding an entry to
> the Yaws wiki about your work.

I will update the wiki in a few days.

>
> There's one other part of your original patch that caught my eye: the
> increase on the FCGI timeout. Should that be boosted? Both your patch and

Yes, 1 sec is too short for fcgi application to return a response. All the pages of my application required local
database interaction and i did not try the response time by calling a test page that just returns a fixed string, so i'm not sure
what is the real bottleneck here, yet increasing the read timeout value fixes "fcgi failed" problem.

Regards,
Özgür

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Erlyaws-list mailing list
Erlyaws-list@...
https://lists.sourceforge.net/lists/listinfo/erlyaws-list

[Ozgur]

> > Great, glad to hear it works for you. You might consider adding an entry to
> > the Yaws wiki about your work.

I've updated wiki http://wiki.github.com/klacke/yaws/run-django-under-yaws

Regards,
Özgür


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Erlyaws-list mailing list
Erlyaws-list@...
https://lists.sourceforge.net/lists/listinfo/erlyaws-list