Re: Re: Snort with an expert system

[Tomas Olsson-3]

Hi,
Coming late into this conversation, but what about using statistical learning filtering instead of an expert system? We have done it using an anomaly detection algorithm we have developed:

http://eprints.sics.se/3591/ 

(link to paper https://daisy.dsv.su.se/fil/visa?id=24833)

/Tomas
http://www.sics.se

[Stefano Zanero-2]

tol@... wrote:
> Hi,
> Coming late into this conversation, but what about using statistical learning filtering instead of an expert system?

Sorry to be blunt, but I don't really see how this can possibly lead to
finding or filtering false positives in a misuse detector. And no, the
experiments on the cited paper fail to convince me.

Best,
Stefano

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Tomas Olsson-3]

Stefan,
I appreciate your feedback. I am aware that the DARPA dataset is not
looked upon with favor in the security community, so I can understand
that that using it is not enough. But, how would I convince you? By
applying the method on real data and letting a security professional
tell me if it is performing OK?

/Tomas


Stefano Zanero wrote:

> tol@... wrote:
>  
>> Hi,
>> Coming late into this conversation, but what about using statistical learning filtering instead of an expert system?
>>    
>
> Sorry to be blunt, but I don't really see how this can possibly lead to
> finding or filtering false positives in a misuse detector. And no, the
> experiments on the cited paper fail to convince me.
>
> Best,
> Stefano
>  

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Stefano Zanero-2]

Tomas Olsson wrote:
> Stefan,
> I appreciate your feedback. I am aware that the DARPA dataset is not
> looked upon with favor in the security community, so I can understand
> that that using it is not enough. But, how would I convince you? By
> applying the method on real data and letting a security professional
> tell me if it is performing OK?

Usually, extraordinary claims need extraordinary proof. If there was any
reason to believe that clustering data in the way you describe would
lead to spotting false positives (which, in the case of Snort, would
rather be noncontextual alerts which you do not care about), testing it
over IDEVAL may be sufficient.

Since there is no reason why this should work, you need much more
convincing experiments to show that it actually does. And it's not just
a matter of the dataset, it's also a matter of what you define as a
false positive: in fact, the term "false positive" has a different
meaning for misuse detectors and anomaly detectors.

Best,
Stefano

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Tomas Olsson-3]

Stefano,

According to this book about Snort
(http://www.amazon.com/Snort-Toolkit-Beales-Source-Security/dp/1597490997):

"A false positive is an alert that triggers on normal traffic where no
intrusion or attack is underway"

That is how we used the term in the paper. Is that not how it is used
with an anomaly detector with respect to the use as an intrusion detector?

In addition, I don't understand why there would be no reason that this
algorithm would work. Could you explain? The algorithm is developed by
experts in Bayesian statistics and has been applied in other fields as well.

But I agree, we have to show that this algorithm works with more
experiments.

If somebody would be willing to let us test the algorithm on real data,
we would be very happy... :)

/Tomas

Stefano Zanero wrote:

> Usually, extraordinary claims need extraordinary proof. If there was any
> reason to believe that clustering data in the way you describe would
> lead to spotting false positives (which, in the case of Snort, would
> rather be noncontextual alerts which you do not care about), testing it
> over IDEVAL may be sufficient.
>
> Since there is no reason why this should work, you need much more
> convincing experiments to show that it actually does. And it's not just
> a matter of the dataset, it's also a matter of what you define as a
> false positive: in fact, the term "false positive" has a different
> meaning for misuse detectors and anomaly detectors.
>
> Best,
> Stefano
>  

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Stefano Zanero-2]

> "A false positive is an alert that triggers on normal traffic where no
> intrusion or attack is underway"

That's a good definition, but not really complete. Under that
definition, if you place a rule that flags IRC connections, and it
fires, is that a false positive?

Is it a false positive a case where there is no rule, or the traffic
does not match with the rule, and the engine still fires?

Is it a false positive a case where a rule correctly matches, but the
user didn't want to be alerted to that traffic ?

> In addition, I don't understand why there would be no reason that this
> algorithm would work. Could you explain? The algorithm is developed by
> experts in Bayesian statistics and has been applied in other fields as
> well.

The algorithm type has apparently no relevance to the problem. Why
should a false positive be statistically different, in the sense you are
considering, from a true positive?

Best,
Stefano

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Tomas Olsson-3]

My comments in the text below.

Stefano Zanero wrote:
>> "A false positive is an alert that triggers on normal traffic where no
>> intrusion or attack is underway"
>>    
>
> That's a good definition, but not really complete. Under that
> definition, if you place a rule that flags IRC connections, and it
> fires, is that a false positive?
>  
Well, then you do not use snort as an automated intrusion detector but
as a  generic log monitor. No problem with that,
but I would say it is a false positive if the IRC connection is not
considered an intrusion.

> Is it a false positive a case where there is no rule, or the traffic
> does not match with the rule, and the engine still fires?
>
>  
This does not fit with the above definition since the alert must be
triggered by the traffic.
> Is it a false positive a case where a rule correctly matches, but the
> user didn't want to be alerted to that traffic ?
>  

Yes, if there was no attack or intrusion triggering the alert. But, why
would the user not want to be alerted if it is a real intrusion?

With respect to using the alerts as input to our algorithm, no of these
objections are important. We just use the type of alerts as sensor data
that we want to analyze to see when the frequencies of each type of
alert diverge from what previously has been observed.

>  
>> In addition, I don't understand why there would be no reason that this
>> algorithm would work. Could you explain? The algorithm is developed by
>> experts in Bayesian statistics and has been applied in other fields as
>> well.
>>    
>
> The algorithm type has apparently no relevance to the problem. Why
> should a false positive be statistically different, in the sense you are
> considering, from a true positive?
>  

Well, there is nothing that says that there must be any difference
between a false and a true alert. However, assume that there are
legitimate traffic that  triggers false alerts on a regular basis. With
our algorithm, we learn to recognize the frequency pattern of these
alerts. Later, suddenly there appears malicious traffic triggering true
alerts, maybe for a more limited time. Then would not the frequency
pattern of the generated alerts change in some way?   We can detect that
change, but also the collective change of different alert types not
shown from a single alert type.

Kind regards
Tomas

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Stefano Zanero-2]

>> Is it a false positive a case where there is no rule, or the traffic
>> does not match with the rule, and the engine still fires?

> This does not fit with the above definition since the alert must be
> triggered by the traffic.

You would be surprised in knowing that this is the only case where
you're pretty sure it IS a false positive that you are looking at (a
false positive of the engine itself, whereas the other examples are
noncontextual alerts caused by careless configuration by the user)

> Yes, if there was no attack or intrusion triggering the alert. But, why
> would the user not want to be alerted if it is a real intrusion?

Because maybe it is a rule firing for a real attack on a vulnerability
that is not present. By the way: is this a false positive or not? :-)

Do you see why I say that "false positive" is a dangerous beast to define?

> With respect to using the alerts as input to our algorithm, no of these
> objections are important. We just use the type of alerts as sensor data
> that we want to analyze to see when the frequencies of each type of
> alert diverge from what previously has been observed.

And what does that imply ? Do you filter out what diverges, or do you
filter out what does not diverge? How "diverging statistically" with the
specific algorithm which you chose actually have any relationship with
an alert being a false positive or not?

> Well, there is nothing that says that there must be any difference
> between a false and a true alert.

That's the point, exactly.

> However, assume that there are
> legitimate traffic that  triggers false alerts on a regular basis.

Here you are: you are detecting misconfigurations and noncontextuals,
not false positives ;-)

As I said, it's a matter of definition.

And "artificial ignorance" (as dubbed by Marcus Ranum) works using the
principle you stated, but with a much simpler apparatus. If this is all
you're looking for, then probably the algorithm you are using is an
overkill.

(and, in IDEVAL, there's probably no such traffic, unless you severely
misconfigure Snort)

Best,
Stefano

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Tomas Olsson-3]

Stefano Zanero wrote:

>>> Is it a false positive a case where there is no rule, or the traffic
>>> does not match with the rule, and the engine still fires?
>>>      
>
>  
>> This does not fit with the above definition since the alert must be
>> triggered by the traffic.
>>    
>
> You would be surprised in knowing that this is the only case where
> you're pretty sure it IS a false positive that you are looking at (a
> false positive of the engine itself, whereas the other examples are
> noncontextual alerts caused by careless configuration by the user)
>
>  
>> Yes, if there was no attack or intrusion triggering the alert. But, why
>> would the user not want to be alerted if it is a real intrusion?
>>    
>
> Because maybe it is a rule firing for a real attack on a vulnerability
> that is not present. By the way: is this a false positive or not? :-)
>
> Do you see why I say that "false positive" is a dangerous beast to define?
>
>  
>> With respect to using the alerts as input to our algorithm, no of these
>> objections are important. We just use the type of alerts as sensor data
>> that we want to analyze to see when the frequencies of each type of
>> alert diverge from what previously has been observed.
>>    
>
> And what does that imply ? Do you filter out what diverges, or do you
> filter out what does not diverge? How "diverging statistically" with the
> specific algorithm which you chose actually have any relationship with
> an alert being a false positive or not?
>
>  
>> Well, there is nothing that says that there must be any difference
>> between a false and a true alert.
>>    
>
> That's the point, exactly.
>
>  
>> However, assume that there are
>> legitimate traffic that  triggers false alerts on a regular basis.
>>    
>
> Here you are: you are detecting misconfigurations and noncontextuals,
> not false positives ;-)
>
> As I said, it's a matter of definition.
>
> And "artificial ignorance" (as dubbed by Marcus Ranum) works using the
> principle you stated, but with a much simpler apparatus. If this is all
> you're looking for, then probably the algorithm you are using is an
> overkill.
>
> (and, in IDEVAL, there's probably no such traffic, unless you severely
> misconfigure Snort)
>
> Best,
> Stefano
>  

OK, I think I've got your point. The "misconfiguration" of snort was
that we did not tune the signature rules. We used the default rules.

Maybe, I just have been thinking of this wrongly? If we instead see the
IDS as a sensor that triggers alerts on interesting "events"/patterns in
the traffic that we think is interesting. Thereafter, we can monitor the
alerts and signal when ever something "unusual" happens using our
algorithm. The we do not filter false positives, but we have created
another type of IDS based on anomaly detection in combination with
rules, if we assume that the signal correlates to intrusions.
/Tomas



-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Joel Esler]

On Thu, Jun 25, 2009 at 10:04 AM, Tomas Olsson <tol@...> wrote:

>
> Stefano Zanero wrote:
>>>>
>>>> Is it a false positive a case where there is no rule, or the traffic
>>>> does not match with the rule, and the engine still fires?
>>>>
>>
>>
>>>
>>> This does not fit with the above definition since the alert must be
>>> triggered by the traffic.
>>>
>>
>> You would be surprised in knowing that this is the only case where
>> you're pretty sure it IS a false positive that you are looking at (a
>> false positive of the engine itself, whereas the other examples are
>> noncontextual alerts caused by careless configuration by the user)

 Here's a topic for discussion, just to fan the flame, and basically
just to get the discussion further along.

<I work for Sourcefire>

"There are no false positives in pure signature based intrusion
detection". (Note I said signature based, not rule based, there is a
difference, anyway....)  If a false positive is defined as you have it
above, then there are no false positives.  If you have a rule that
alerted on a piece of traffic that the rule should NOT have alerted
on, whose fault is it?  You for writing the rule? or the engine's
fault?  Its only doing what you told it to do. Confused?  Let me back
up.

For example, a rule fires because an IIS exploit is destined for your
Apache server.  Is that a false positive?  In the pure IDS sense, no,
because the traffic took place.  But when you put the alert in
context, then yes, it is a false positive.  The rule should not have
triggered because the end application base is incorrect as it pertains
to the rule.  Put that scenario on a real network where IPs change and
applications get installed all the time, and OSes come and go, ports
open and close, services are on those ports, and on non-standard
ports, and lets face it you don't know where or what those ports are
etc.. and you see the problem.

Which is why context given to Snort is so important, which is why
Sourcefire developed things like RNA (
http://www.sourcefire.com/products/3D/rna ) in order to solve that
problem.  Which is also why things like Snort 3.0 are being developed,
to be able to deal with adjustments in a more real-time fashion.

That being said.  False positives do happen.  Which is why there are
false positive reporting methods.  If someone ELSE wrote the rule,
then its a duty to report those FPs, so those FPs can be corrected as
much as possible. If you wrote the rule, then it's time to go back to
the drawing board.

An IPS should only alert when you need to go DO something.  IMO.  I
hate having superfluous alerts.  Alerts = work = time = money = more
work.

These are my opinions and not the opinion of my company, but basically
just to fuel the conversation a bit.   Sorry if it seemed like I
plugged a bit.

--
Joel

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Greg Shipley]

I respect the spirited and intelligent conversation here, but at the
risk of sounding like a) an old guy that's been following this stuff
for too long and b) a complete jerk:

1. IDS vendor / IDS software engineer / uber-geek view: "it's not
   technically a false-positive because if signature/ rule /
   pattern-match/ neugent/ whatever fired on x and it was programmed
   to identify q but you have to factor in y, and z, and..."

   <bang head here -----> X

2. Infosec operational person trying to do his job: "Was I attacked
   and was the attack successful?  Yes or NO will suffice, thank you."

I submit that for the vast majority of consumers of IDS technology we
really only give a crap about #2.  So if the device can give us a
reasonably accurate answers to #2 we are happy.  And if it can't we
are unhappy.

I think the fact we've been discussing these topics for close to
twenty years now suggests that we aren't happy, but maybe I'm too old
and being a jerk.  :)

My .01,

-Greg



On Thu, 25 Jun 2009, Joel Esler wrote:

> On Thu, Jun 25, 2009 at 10:04 AM, Tomas Olsson <tol@...> wrote:
> >
> > Stefano Zanero wrote:
> >>>>
> >>>> Is it a false positive a case where there is no rule, or the traffic
> >>>> does not match with the rule, and the engine still fires?
> >>>>
> >>
> >>
> >>>
> >>> This does not fit with the above definition since the alert must be
> >>> triggered by the traffic.
> >>>
> >>
> >> You would be surprised in knowing that this is the only case where
> >> you're pretty sure it IS a false positive that you are looking at (a
> >> false positive of the engine itself, whereas the other examples are
> >> noncontextual alerts caused by careless configuration by the user)
>
>  Here's a topic for discussion, just to fan the flame, and basically
> just to get the discussion further along.
>
> <I work for Sourcefire>
>
> "There are no false positives in pure signature based intrusion
> detection". (Note I said signature based, not rule based, there is a
> difference, anyway....)  If a false positive is defined as you have it
> above, then there are no false positives.  If you have a rule that
> alerted on a piece of traffic that the rule should NOT have alerted
> on, whose fault is it?  You for writing the rule? or the engine's
> fault?  Its only doing what you told it to do. Confused?  Let me back
> up.
>
> For example, a rule fires because an IIS exploit is destined for your
> Apache server.  Is that a false positive?  In the pure IDS sense, no,
> because the traffic took place.  But when you put the alert in
> context, then yes, it is a false positive.  The rule should not have
> triggered because the end application base is incorrect as it pertains
> to the rule.  Put that scenario on a real network where IPs change and
> applications get installed all the time, and OSes come and go, ports
> open and close, services are on those ports, and on non-standard
> ports, and lets face it you don't know where or what those ports are
> etc.. and you see the problem.
>
> Which is why context given to Snort is so important, which is why
> Sourcefire developed things like RNA (
> http://www.sourcefire.com/products/3D/rna ) in order to solve that
> problem.  Which is also why things like Snort 3.0 are being developed,
> to be able to deal with adjustments in a more real-time fashion.
>
> That being said.  False positives do happen.  Which is why there are
> false positive reporting methods.  If someone ELSE wrote the rule,
> then its a duty to report those FPs, so those FPs can be corrected as
> much as possible. If you wrote the rule, then it's time to go back to
> the drawing board.
>
> An IPS should only alert when you need to go DO something.  IMO.  I
> hate having superfluous alerts.  Alerts = work = time = money = more
> work.
>
> These are my opinions and not the opinion of my company, but basically
> just to fuel the conversation a bit.   Sorry if it seemed like I
> plugged a bit.
>
> --
> Joel
>
> -----------------------------------------------------------------
> Securing Your Online Data Transfer with SSL.
> A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>
>
>
>

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Martin Roesch]

Not for nothing but #2 is exactly what Sourcefire's been doing since
2004.  Sorry for the commercial but I think I've been pretty outspoken
on this topic since 2000 or so...

Marty


On Thu, Jun 25, 2009 at 2:55 PM, Greg Shipley<gshipley@...> wrote:

>
> I respect the spirited and intelligent conversation here, but at the
> risk of sounding like a) an old guy that's been following this stuff
> for too long and b) a complete jerk:
>
> 1. IDS vendor / IDS software engineer / uber-geek view: "it's not
>   technically a false-positive because if signature/ rule /
>   pattern-match/ neugent/ whatever fired on x and it was programmed
>   to identify q but you have to factor in y, and z, and..."
>
>   <bang head here -----> X
>
> 2. Infosec operational person trying to do his job: "Was I attacked
>   and was the attack successful?  Yes or NO will suffice, thank you."
>
> I submit that for the vast majority of consumers of IDS technology we
> really only give a crap about #2.  So if the device can give us a
> reasonably accurate answers to #2 we are happy.  And if it can't we
> are unhappy.
>
> I think the fact we've been discussing these topics for close to
> twenty years now suggests that we aren't happy, but maybe I'm too old
> and being a jerk.  :)
>
> My .01,
>
> -Greg
>
>
>
> On Thu, 25 Jun 2009, Joel Esler wrote:
>
>> On Thu, Jun 25, 2009 at 10:04 AM, Tomas Olsson <tol@...> wrote:
>> >
>> > Stefano Zanero wrote:
>> >>>>
>> >>>> Is it a false positive a case where there is no rule, or the traffic
>> >>>> does not match with the rule, and the engine still fires?
>> >>>>
>> >>
>> >>
>> >>>
>> >>> This does not fit with the above definition since the alert must be
>> >>> triggered by the traffic.
>> >>>
>> >>
>> >> You would be surprised in knowing that this is the only case where
>> >> you're pretty sure it IS a false positive that you are looking at (a
>> >> false positive of the engine itself, whereas the other examples are
>> >> noncontextual alerts caused by careless configuration by the user)
>>
>>  Here's a topic for discussion, just to fan the flame, and basically
>> just to get the discussion further along.
>>
>> <I work for Sourcefire>
>>
>> "There are no false positives in pure signature based intrusion
>> detection". (Note I said signature based, not rule based, there is a
>> difference, anyway....)  If a false positive is defined as you have it
>> above, then there are no false positives.  If you have a rule that
>> alerted on a piece of traffic that the rule should NOT have alerted
>> on, whose fault is it?  You for writing the rule? or the engine's
>> fault?  Its only doing what you told it to do. Confused?  Let me back
>> up.
>>
>> For example, a rule fires because an IIS exploit is destined for your
>> Apache server.  Is that a false positive?  In the pure IDS sense, no,
>> because the traffic took place.  But when you put the alert in
>> context, then yes, it is a false positive.  The rule should not have
>> triggered because the end application base is incorrect as it pertains
>> to the rule.  Put that scenario on a real network where IPs change and
>> applications get installed all the time, and OSes come and go, ports
>> open and close, services are on those ports, and on non-standard
>> ports, and lets face it you don't know where or what those ports are
>> etc.. and you see the problem.
>>
>> Which is why context given to Snort is so important, which is why
>> Sourcefire developed things like RNA (
>> http://www.sourcefire.com/products/3D/rna ) in order to solve that
>> problem.  Which is also why things like Snort 3.0 are being developed,
>> to be able to deal with adjustments in a more real-time fashion.
>>
>> That being said.  False positives do happen.  Which is why there are
>> false positive reporting methods.  If someone ELSE wrote the rule,
>> then its a duty to report those FPs, so those FPs can be corrected as
>> much as possible. If you wrote the rule, then it's time to go back to
>> the drawing board.
>>
>> An IPS should only alert when you need to go DO something.  IMO.  I
>> hate having superfluous alerts.  Alerts = work = time = money = more
>> work.
>>
>> These are my opinions and not the opinion of my company, but basically
>> just to fuel the conversation a bit.   Sorry if it seemed like I
>> plugged a bit.
>>
>> --
>> Joel
>>
>> -----------------------------------------------------------------
>> Securing Your Online Data Transfer with SSL.
>> A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
>> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>>
>>
>>
>>
>
> -----------------------------------------------------------------
> Securing Your Online Data Transfer with SSL.
> A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>
>

--
Martin Roesch - Founder/CTO, Sourcefire Inc. - [phone]
Sourcefire - Security for the Real World - http://www.sourcefire.com
Snort: Open Source IDP - http://www.snort.org

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Richard Bejtlich]

On Thu, Jun 25, 2009 at 2:55 PM, Greg Shipley<gshipley@...> wrote:

>
> I respect the spirited and intelligent conversation here, but at the
> risk of sounding like a) an old guy that's been following this stuff
> for too long and b) a complete jerk:
>
> 1. IDS vendor / IDS software engineer / uber-geek view: "it's not
>   technically a false-positive because if signature/ rule /
>   pattern-match/ neugent/ whatever fired on x and it was programmed
>   to identify q but you have to factor in y, and z, and..."
>
>   <bang head here -----> X
>
> 2. Infosec operational person trying to do his job: "Was I attacked
>   and was the attack successful?  Yes or NO will suffice, thank you."
>
> I submit that for the vast majority of consumers of IDS technology we
> really only give a crap about #2.  So if the device can give us a
> reasonably accurate answers to #2 we are happy.  And if it can't we
> are unhappy.
>
> I think the fact we've been discussing these topics for close to
> twenty years now suggests that we aren't happy, but maybe I'm too old
> and being a jerk.  :)
>
> My .01,
>
> -Greg
>

Hi everyone,

This is a cool debate.  I submit that it is technically impossible to
build anything that will not 100% avoid "#2" false positives.  I'm a
#1 guy myself; the only real "false positive" is the system telling
you it saw something, when that something actually never happened,
e.g., "IDS: I saw ICMP!  User: There was no ICMP; your engine isn't
working properly."

For any case you develop that you think is absolutely, positively,
without a doubt an "intrusion" that you could identify with an IDS, I
can probably develop a case where that activity could turn out to be
legitimate, and therefore, in the eyes of the organization, a "false
positive."

I think the "IDS" has been misnamed from the beginning.  (Blame Mr.
Anderson?)  It should have been Attack Indication System or something
similar.  After all "If you can detect it, why can't you prevent it?"
Now it's really time to "bang head here."  :)

Sincerely,

Richard

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Gary Halleen (ghalleen)]

On 6/25/09 3:26 AM, "Stefano Zanero" <s.zanero@...> wrote:

>> "A false positive is an alert that triggers on normal traffic where no
>> intrusion or attack is underway"
>
> That's a good definition, but not really complete. Under that
> definition, if you place a rule that flags IRC connections, and it
> fires, is that a false positive?

GH:  No.  If a rule or signature fires on traffic you asked it to fire on,
then it is not a false positive, regardless of whether or not it is an
attack or intrusion.


>
> Is it a false positive a case where there is no rule, or the traffic
> does not match with the rule, and the engine still fires?

GH:  Yes.

 
> Is it a false positive a case where a rule correctly matches, but the
> user didn't want to be alerted to that traffic ?

GH:  Some say yes, some say no.  I am one of those who says this is not a
false positive.  Perhaps this is a misconfiguration of the sensor, but it is
still doing what it was asked to do.  I wrote "Security Monitoring with
CS-MARS" for Cisco Press, and that product considers this to be a false
positive.

Stay Secure!
Gary

Gary Halleen, CISSP-ISSAP, CHP
Author, Security Monitoring with CS-MARS, ISBN: 1587052709



-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Gary Halleen (ghalleen)]

You're not being a jerk, Greg.

To the security or network operations people, all noise is a false positive.
They want the noise to go away.  Marty's discussions on target-based IDS are
dead on.

This is an area where IDS/IPS products are evolving, and so are the
monitoring consoles.

You reduce the noise by:
1.  Gain information about the target.  Is it potentially vulnerable to an
attack based on its operating system and software installed on it?  How
valuable is the asset?

2.  Gain information about the attacker.  Does it have a reputation of
attacking other systems, hosting malware, etc?

3.  Correlate information on the monitoring console.  Did an attack actually
reach the destination?  Did it cause any damage?  Did other devices between
the attacker and victim also see the attack?  Did anything stop it?

Gary



On 6/25/09 11:55 AM, "Greg Shipley" <gshipley@...> wrote:

>
> I respect the spirited and intelligent conversation here, but at the
> risk of sounding like a) an old guy that's been following this stuff
> for too long and b) a complete jerk:
>
> 1. IDS vendor / IDS software engineer / uber-geek view: "it's not
>    technically a false-positive because if signature/ rule /
>    pattern-match/ neugent/ whatever fired on x and it was programmed
>    to identify q but you have to factor in y, and z, and..."
>
>    <bang head here -----> X
>
> 2. Infosec operational person trying to do his job: "Was I attacked
>    and was the attack successful?  Yes or NO will suffice, thank you."
>
> I submit that for the vast majority of consumers of IDS technology we
> really only give a crap about #2.  So if the device can give us a
> reasonably accurate answers to #2 we are happy.  And if it can't we
> are unhappy.
>
> I think the fact we've been discussing these topics for close to
> twenty years now suggests that we aren't happy, but maybe I'm too old
> and being a jerk.  :)
>
> My .01,
>
> -Greg

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Gary Halleen (ghalleen)]

Well, I guess I have to pipe in also, then.  Cisco is doing the same.  Read
my book "Security Monitoring with CS-MARS" for more info.

Gary


On 6/25/09 1:29 PM, "Martin Roesch" <roesch@...> wrote:

> Not for nothing but #2 is exactly what Sourcefire's been doing since
> 2004.  Sorry for the commercial but I think I've been pretty outspoken
> on this topic since 2000 or so...
>
> Marty


-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Martin Roesch]

Inline...

On Thu, Jun 25, 2009 at 5:12 PM, Richard Bejtlich<taosecurity@...> wrote:

> On Thu, Jun 25, 2009 at 2:55 PM, Greg Shipley<gshipley@...> wrote:
>>
>> I respect the spirited and intelligent conversation here, but at the
>> risk of sounding like a) an old guy that's been following this stuff
>> for too long and b) a complete jerk:
>>
>> 1. IDS vendor / IDS software engineer / uber-geek view: "it's not
>>   technically a false-positive because if signature/ rule /
>>   pattern-match/ neugent/ whatever fired on x and it was programmed
>>   to identify q but you have to factor in y, and z, and..."
>>
>>   <bang head here -----> X
>>
>> 2. Infosec operational person trying to do his job: "Was I attacked
>>   and was the attack successful?  Yes or NO will suffice, thank you."
>>
>> I submit that for the vast majority of consumers of IDS technology we
>> really only give a crap about #2.  So if the device can give us a
>> reasonably accurate answers to #2 we are happy.  And if it can't we
>> are unhappy.
>>
>> I think the fact we've been discussing these topics for close to
>> twenty years now suggests that we aren't happy, but maybe I'm too old
>> and being a jerk.  :)
>>
>> My .01,
>>
>> -Greg
>>
>
> Hi everyone,
>
> This is a cool debate.  I submit that it is technically impossible to
> build anything that will not 100% avoid "#2" false positives.  I'm a
> #1 guy myself; the only real "false positive" is the system telling
> you it saw something, when that something actually never happened,
> e.g., "IDS: I saw ICMP!  User: There was no ICMP; your engine isn't
> working properly."

I think the #2 case is about improving the signal to noise ratio.  I
had a group in the office a couple weeks ago who were getting 1M+
events a day from their legacy IDS deployment and that had rendered
the system effectively useless because they had no tools to assess the
impact of the detects against their deployed infrastructure.  If you
look at the Verizon report you can pretty clearly see that raw
uncontextualized detection data serves virtually no purpose in the
vast majority of deployments.  If you really want to build a useful
IDS you have to figure out how to perform that front line
contextualization in a way that's both correct and useful.  You'll
still get false positives but if you've removed 99% of the noise first
you'll have a useful detection capability anyway.

> For any case you develop that you think is absolutely, positively,
> without a doubt an "intrusion" that you could identify with an IDS, I
> can probably develop a case where that activity could turn out to be
> legitimate, and therefore, in the eyes of the organization, a "false
> positive."

This is always true but lining up what's being detected vs what an
organization can actually be vulnerable to is always going to be
useful.

> I think the "IDS" has been misnamed from the beginning.  (Blame Mr.
> Anderson?)  It should have been Attack Indication System or something
> similar.  After all "If you can detect it, why can't you prevent it?"
> Now it's really time to "bang head here."  :)

Oh man, don't get me started...


--
Martin Roesch - Founder/CTO, Sourcefire Inc. - [phone]
Sourcefire - Security for the Real World - http://www.sourcefire.com
Snort: Open Source IDP - http://www.snort.org

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Stuart Staniford]

On Jun 25, 2009, at 5:18 PM, Gary Halleen wrote:

> On 6/25/09 3:26 AM, "Stefano Zanero" <s.zanero@...>  
> wrote:
>
>>> "A false positive is an alert that triggers on normal traffic  
>>> where no
>>> intrusion or attack is underway"
>>
>> That's a good definition, but not really complete. Under that
>> definition, if you place a rule that flags IRC connections, and it
>> fires, is that a false positive?
>
> GH:  No.  If a rule or signature fires on traffic you asked it to  
> fire on,
> then it is not a false positive, regardless of whether or not it is an
> attack or intrusion.

To echo what Greg said - from a customer perspective, it's all the  
same.  Customers generally buy both an engine and a set of rules as a  
single package, and if the combination is reporting things that aren't  
actual attacks, then it's making them unhappy.  Few customers are  
writing their own rules.

Distinguishing between whether the problem is in the engine or the  
rule is useful internally at the vendor to decide what needs to get  
fixed, but customers are not likely to care that much.

The way our (FireEye's) technology reduces false positives is to  
replay the traffic in an instrumented virtual machine, to see if it  
really is an attack or not.  We have a lot fewer false positives than  
traditional IDS products (we don't ship a release with any that are  
known to us, though a few still pop up in the field unfortunately -  
you can never test against everything that will show up on a  
customer's network)

Stuart Staniford.

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Stefano Zanero-2]

>> Not for nothing but #2 is exactly what Sourcefire's been doing since
>> 2004.  Sorry for the commercial but I think I've been pretty outspoken
>> on this topic since 2000 or so...

> Well, I guess I have to pipe in also, then.  Cisco is doing the same.  Read
> my book "Security Monitoring with CS-MARS" for more info.

Sorry Marty, sorry Gary, I love both products, but they are not even
close to realizing what Greg asked for :)

Of course, they do reduce "false positives/noncontextual
alerts/whatevers", and so they are to be commended, but knowing "if the
attack has been successful" is actually way beyond anybody's capability,
short of a crystal sphere :)

Stefano

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Gary Halleen (ghalleen)]

I don't disagree with you.  In fact, in an earlier message I said that for
operations people (security or network), all noise is a false positive, even
if, technically, it is not really.

Gary



On 6/26/09 12:30 PM, "Stuart Staniford" <sstaniford@...> wrote:

>
> On Jun 25, 2009, at 5:18 PM, Gary Halleen wrote:
>
>> On 6/25/09 3:26 AM, "Stefano Zanero" <s.zanero@...>
>> wrote:
>>
>>>> "A false positive is an alert that triggers on normal traffic
>>>> where no
>>>> intrusion or attack is underway"
>>>
>>> That's a good definition, but not really complete. Under that
>>> definition, if you place a rule that flags IRC connections, and it
>>> fires, is that a false positive?
>>
>> GH:  No.  If a rule or signature fires on traffic you asked it to
>> fire on,
>> then it is not a false positive, regardless of whether or not it is an
>> attack or intrusion.
>
> To echo what Greg said - from a customer perspective, it's all the
> same.  Customers generally buy both an engine and a set of rules as a
> single package, and if the combination is reporting things that aren't
> actual attacks, then it's making them unhappy.  Few customers are
> writing their own rules.
>
> Distinguishing between whether the problem is in the engine or the
> rule is useful internally at the vendor to decide what needs to get
> fixed, but customers are not likely to care that much.
>
> The way our (FireEye's) technology reduces false positives is to
> replay the traffic in an instrumented virtual machine, to see if it
> really is an attack or not.  We have a lot fewer false positives than
> traditional IDS products (we don't ship a release with any that are
> known to us, though a few still pop up in the field unfortunately -
> you can never test against everything that will show up on a
> customer's network)
>
> Stuart Staniford.
>
> -----------------------------------------------------------------
> Securing Your Online Data Transfer with SSL.
> A guide to understanding SSL certificates, how they operate and their
> application. By making use of an SSL certificate on your web server, you can
> securely collect sensitive information online, and increase business by giving
> your customers confidence that their transactions are safe.
> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>
>

The Hacker only has to be right once...

Stay Secure!

Gary Halleen, CISSP-ISSAP, CHP
Author, Security Monitoring with CS-MARS, ISBN: 1587052709



-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194