http firewall: modsecurity excessive logging.. how to manage?

[David Taveras]

Hello,

We have a site with about 2000 visits per day, and now the logging is
getting extremely hard to review, as security is number one the ideal
situation for me would be to be able to classify the output into
groups so that I as a sysadmin can be aware of all, know if there is a
increase of hits for a particular rule, and most important is to know
when Iam getting (or tried to) getting SQL/PHP injected.

Is there a way without using commercial add-ons to classify all this
output and actually make sense of it, possibly by sending important
alerts?  How do other people do this?

Sure: best practice is to have secure PHP code.. but in an environment
where you cannot trust the code. This is my only path.


Thank you.

David Taveras

[CPB]

David Taveras wrote:
> Hello,
>
> We have a site with about 2000 visits per day, and now the logging is
> getting extremely hard to review, as security is number one the ideal
> situation for me would be to be able to classify the output into
> groups so that I as a sysadmin can be aware of all, know if there is a
> increase of hits for a particular rule, and most important is to know
> when Iam getting (or tried to) getting SQL/PHP injected.
>  
I have a script that goes through my error_log and when it finds
entries  I class as bad, blocks that IP through pfctl additions to a
block list.

You could do something similiar to scan  for relevant entries  in error
and access logs.
Those entries could be written into appropriate  log files for each
"category".
Only you will be able to determine what is of interest and  what to ignore.

I only needed a few weeks (but little actual time) to finish tweaking my
'scanner', as I watched the logs.

[Joachim Schipper-2]

On Tue, Nov 03, 2009 at 11:50:25AM -0600, David Taveras wrote:

> Hello,
>
> We have a site with about 2000 visits per day, and now the logging is
> getting extremely hard to review, as security is number one the ideal
> situation for me would be to be able to classify the output into
> groups so that I as a sysadmin can be aware of all, know if there is a
> increase of hits for a particular rule, and most important is to know
> when Iam getting (or tried to) getting SQL/PHP injected.
>
> Is there a way without using commercial add-ons to classify all this
> output and actually make sense of it, possibly by sending important
> alerts?  How do other people do this?
>
> Sure: best practice is to have secure PHP code.. but in an environment
> where you cannot trust the code. This is my only path.

As a general rule, reviewing stuff that your firewall/filter has stopped
isn't terribly useful. After all, it's only the stuff that it lets
through that you care about...

However, to answer your question, I have had good success with using
sysutils/sec, the Simple Event Correlator (for syslog, but it's fairly
generic.) I use a hackish sed script to allow some macros (like __IP__
for a regex matching IP addresses), and a generic preamble/post-amble to
make sure that a log record is matched by exactly one rule in all files
(one of the last rules matches and reports everything not matched by an
earlier rule). With a bit of scripting, it can send mail (and presumably
pager notifications).

Be warned, though, that it's very generic and hence you'll have to write
most of this stuff yourself.

                Joachim

[Gonzalo Lionel Rodriguez]

If you need some 'hardcore' pf rule, you can do something like this:

pass in log quick proto tcp from $ext_if to any port 80 \
        flags S/SA keep state (max-src-conn 50, max-src-conn-rate 10/8, \
         overload <shit> flush global)

Adjust (max-src-conn and max-src-conn-rate) to your needs.

2009/11/3 David Taveras <d3taveras38d3@...>:

> Hello,
>
> We have a site with about 2000 visits per day, and now the logging is
> getting extremely hard to review, as security is number one the ideal
> situation for me would be to be able to classify the output into
> groups so that I as a sysadmin can be aware of all, know if there is a
> increase of hits for a particular rule, and most important is to know
> when Iam getting (or tried to) getting SQL/PHP injected.
>
> Is there a way without using commercial add-ons to classify all this
> output and actually make sense of it, possibly by sending important
> alerts?  How do other people do this?
>
> Sure: best practice is to have secure PHP code.. but in an environment
> where you cannot trust the code. This is my only path.
>
>
> Thank you.
>
> David Taveras
>
>