Making my own IDS... how to detect packet loss ?

[junkJon]

I am writing my own IDS.  I have a packet sniffer + TCP reassembler
that I've written.  For each stream I detect, I have to keep some
state + a buffer of the packets for the stream.  However, a problem
that I've run into is that sometimes packets could get lost (meaning I
do not capture them, but they were actually sent/received between two
hosts on the network).  If I do drop the packets, my current
implementation gets all messed up since I never end up freeing the
resources I've allocated for that stream.  So, lost packets seem to be
the bain of my current implementation.

My question is (assuming all TCP streams), how can I determine whether
a packet is lost by just looking at packets that I currently have in
my buffer?

The packets could come in any order so just because I receive a packet
with seq #N, doesn't mean that a packet with seq #N-1 was sent.

I hope this is the appropriate mailing list for my question.

Thanks
J

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Aaron Turner-2]

On Wed, Jul 15, 2009 at 6:09 PM, Jonathon<thejunkjon@...> wrote:

> I am writing my own IDS.  I have a packet sniffer + TCP reassembler
> that I've written.  For each stream I detect, I have to keep some
> state + a buffer of the packets for the stream.  However, a problem
> that I've run into is that sometimes packets could get lost (meaning I
> do not capture them, but they were actually sent/received between two
> hosts on the network).  If I do drop the packets, my current
> implementation gets all messed up since I never end up freeing the
> resources I've allocated for that stream.  So, lost packets seem to be
> the bain of my current implementation.
>
> My question is (assuming all TCP streams), how can I determine whether
> a packet is lost by just looking at packets that I currently have in
> my buffer?
>
> The packets could come in any order so just because I receive a packet
> with seq #N, doesn't mean that a packet with seq #N-1 was sent.
>
> I hope this is the appropriate mailing list for my question.

If you track the values of the Ack's in packets then you'll know when
you've dropped one or more TCP segments which were received by the
other end.

Example:

Assume server sends 3 packets with Sequence numbers of 10, 50 and 100.
 But you only see 10 & 100.  If the client only Ack's 10, then you
know it didn't see 50 either.  But if it Ack's 100 then you know it
saw data you missed.

--
Aaron Turner
http://synfin.net/
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
    -- Benjamin Franklin

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[junkJon]

Thanks all for your replies.  I think the general consensus was to
look at the ACKs.  However the problem with that solution is that I
could get packets in any order.  So, for example, I could receive an
ACK for packet N, which doesn't necessarily mean that I've dropped
packet N.  It could possibly mean that packet N has not come to me.
If I did get the packets though in order, that would definitely make
things much easier :)

Thanks

J


On Thu, Jul 16, 2009 at 9:15 AM, Aaron Turner<synfinatic@...> wrote:

> On Wed, Jul 15, 2009 at 6:09 PM, Jonathon<thejunkjon@...> wrote:
>> I am writing my own IDS.  I have a packet sniffer + TCP reassembler
>> that I've written.  For each stream I detect, I have to keep some
>> state + a buffer of the packets for the stream.  However, a problem
>> that I've run into is that sometimes packets could get lost (meaning I
>> do not capture them, but they were actually sent/received between two
>> hosts on the network).  If I do drop the packets, my current
>> implementation gets all messed up since I never end up freeing the
>> resources I've allocated for that stream.  So, lost packets seem to be
>> the bain of my current implementation.
>>
>> My question is (assuming all TCP streams), how can I determine whether
>> a packet is lost by just looking at packets that I currently have in
>> my buffer?
>>
>> The packets could come in any order so just because I receive a packet
>> with seq #N, doesn't mean that a packet with seq #N-1 was sent.
>>
>> I hope this is the appropriate mailing list for my question.
>
> If you track the values of the Ack's in packets then you'll know when
> you've dropped one or more TCP segments which were received by the
> other end.
>
> Example:
>
> Assume server sends 3 packets with Sequence numbers of 10, 50 and 100.
>  But you only see 10 & 100.  If the client only Ack's 10, then you
> know it didn't see 50 either.  But if it Ack's 100 then you know it
> saw data you missed.
>
> --
> Aaron Turner
> http://synfin.net/
> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
>    -- Benjamin Franklin
>
> -----------------------------------------------------------------
> Securing Your Online Data Transfer with SSL.
> A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>
>
>

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Paul Palmer]

The short answer is that you correlate the SEQ# from one side of the
connection with the ACK# you see on the other side of the connection. If
you are seeing SEQ# from after the lost data and the other host is
acknowledging receipt past the lost data, it is highly unlikely that you
will ever see the lost data come across the wire.

Paul




From:
Jonathon <thejunkjon@...>
To:
focus-ids@...
Date:
07/16/2009 12:58 PM
Subject:
Making my own IDS... how to detect packet loss ?



I am writing my own IDS.  I have a packet sniffer + TCP reassembler
that I've written.  For each stream I detect, I have to keep some
state + a buffer of the packets for the stream.  However, a problem
that I've run into is that sometimes packets could get lost (meaning I
do not capture them, but they were actually sent/received between two
hosts on the network).  If I do drop the packets, my current
implementation gets all messed up since I never end up freeing the
resources I've allocated for that stream.  So, lost packets seem to be
the bain of my current implementation.

My question is (assuming all TCP streams), how can I determine whether
a packet is lost by just looking at packets that I currently have in
my buffer?

The packets could come in any order so just because I receive a packet
with seq #N, doesn't mean that a packet with seq #N-1 was sent.

I hope this is the appropriate mailing list for my question.

Thanks
J

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their
application. By making use of an SSL certificate on your web server, you
can securely collect sensitive information online, and increase business
by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194







-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Francis, Shannon]

I actually think that's one of the shortcomings of TCP. I could be wrong but I don't think there is a way to tell if a packet has been dropped if they are sent out of order, because the whole reason it's a TCP connection is that it's error-checking, meaning it will wait until either it has received all packets and reassembled them or the connection times out. Please correct me if I'm wrong though :-D.

Regards,
 
Shannon Francis

-----Original Message-----
From: listbounce@... [mailto:listbounce@...] On Behalf Of Jonathon
Sent: Thursday, July 16, 2009 1:44 PM
To: focus-ids@...
Subject: Re: Making my own IDS... how to detect packet loss ?

Thanks all for your replies.  I think the general consensus was to
look at the ACKs.  However the problem with that solution is that I
could get packets in any order.  So, for example, I could receive an
ACK for packet N, which doesn't necessarily mean that I've dropped
packet N.  It could possibly mean that packet N has not come to me.
If I did get the packets though in order, that would definitely make
things much easier :)

Thanks

J


On Thu, Jul 16, 2009 at 9:15 AM, Aaron Turner<synfinatic@...> wrote:

> On Wed, Jul 15, 2009 at 6:09 PM, Jonathon<thejunkjon@...> wrote:
>> I am writing my own IDS.  I have a packet sniffer + TCP reassembler
>> that I've written.  For each stream I detect, I have to keep some
>> state + a buffer of the packets for the stream.  However, a problem
>> that I've run into is that sometimes packets could get lost (meaning I
>> do not capture them, but they were actually sent/received between two
>> hosts on the network).  If I do drop the packets, my current
>> implementation gets all messed up since I never end up freeing the
>> resources I've allocated for that stream.  So, lost packets seem to be
>> the bain of my current implementation.
>>
>> My question is (assuming all TCP streams), how can I determine whether
>> a packet is lost by just looking at packets that I currently have in
>> my buffer?
>>
>> The packets could come in any order so just because I receive a packet
>> with seq #N, doesn't mean that a packet with seq #N-1 was sent.
>>
>> I hope this is the appropriate mailing list for my question.
>
> If you track the values of the Ack's in packets then you'll know when
> you've dropped one or more TCP segments which were received by the
> other end.
>
> Example:
>
> Assume server sends 3 packets with Sequence numbers of 10, 50 and 100.
>  But you only see 10 & 100.  If the client only Ack's 10, then you
> know it didn't see 50 either.  But if it Ack's 100 then you know it
> saw data you missed.
>
> --
> Aaron Turner
> http://synfin.net/
> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
>    -- Benjamin Franklin
>
> -----------------------------------------------------------------
> Securing Your Online Data Transfer with SSL.
> A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>
>
>

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194



-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194

[Martin Roesch]

You should be calculating the next expected ack number based on the
packets that you observe coming from a transmitter.  If the returning
ack numbers are higher than the calculated next expected ack you've
missed something (barring spoofing, of course).  Once you've missed
something in a state-machine-based traffic analyzer you have a couple
choices:

1) If you're tracking flows you can mark the flow as desynchronized
and ignore it until it times out or goes through a shutdown sequence.

2) You can try to resynchronize on the protocol based on known
sequences that are intrinsic and recognizable to that protocol.

3) You can just log everything (the packets) and post-process the data
if you think that it could be important.

If you get packets out of order the acks will still come in order
(assuming no drops), your job will be to queue them until the point
where you're confident you can do a proper reassembly and processing
pass.  The acks tell the story and if you're in purely passive-mode
they're your sole indicator of what each end-point has actually seen.
If the packets are coming out of order and there are large windows
involved (and even when there are not) you're going to have to have
fail-safe strategies for saving yourself from memory exhaustion and
computational complexity attacks against your engine.

TCP stream analyzers in pure-passive technologies are complex animals,
don't be surprised at the complexity required to do it properly.

Marty




On Thu, Jul 16, 2009 at 2:49 PM, Francis,
Shannon<Shannon.Francis@...> wrote:

> I actually think that's one of the shortcomings of TCP. I could be wrong but I don't think there is a way to tell if a packet has been dropped if they are sent out of order, because the whole reason it's a TCP connection is that it's error-checking, meaning it will wait until either it has received all packets and reassembled them or the connection times out. Please correct me if I'm wrong though :-D.
>
> Regards,
>
> Shannon Francis
>
> -----Original Message-----
> From: listbounce@... [mailto:listbounce@...] On Behalf Of Jonathon
> Sent: Thursday, July 16, 2009 1:44 PM
> To: focus-ids@...
> Subject: Re: Making my own IDS... how to detect packet loss ?
>
> Thanks all for your replies.  I think the general consensus was to
> look at the ACKs.  However the problem with that solution is that I
> could get packets in any order.  So, for example, I could receive an
> ACK for packet N, which doesn't necessarily mean that I've dropped
> packet N.  It could possibly mean that packet N has not come to me.
> If I did get the packets though in order, that would definitely make
> things much easier :)
>
> Thanks
>
> J
>
>
> On Thu, Jul 16, 2009 at 9:15 AM, Aaron Turner<synfinatic@...> wrote:
>> On Wed, Jul 15, 2009 at 6:09 PM, Jonathon<thejunkjon@...> wrote:
>>> I am writing my own IDS.  I have a packet sniffer + TCP reassembler
>>> that I've written.  For each stream I detect, I have to keep some
>>> state + a buffer of the packets for the stream.  However, a problem
>>> that I've run into is that sometimes packets could get lost (meaning I
>>> do not capture them, but they were actually sent/received between two
>>> hosts on the network).  If I do drop the packets, my current
>>> implementation gets all messed up since I never end up freeing the
>>> resources I've allocated for that stream.  So, lost packets seem to be
>>> the bain of my current implementation.
>>>
>>> My question is (assuming all TCP streams), how can I determine whether
>>> a packet is lost by just looking at packets that I currently have in
>>> my buffer?
>>>
>>> The packets could come in any order so just because I receive a packet
>>> with seq #N, doesn't mean that a packet with seq #N-1 was sent.
>>>
>>> I hope this is the appropriate mailing list for my question.
>>
>> If you track the values of the Ack's in packets then you'll know when
>> you've dropped one or more TCP segments which were received by the
>> other end.
>>
>> Example:
>>
>> Assume server sends 3 packets with Sequence numbers of 10, 50 and 100.
>>  But you only see 10 & 100.  If the client only Ack's 10, then you
>> know it didn't see 50 either.  But if it Ack's 100 then you know it
>> saw data you missed.
>>
>> --
>> Aaron Turner
>> http://synfin.net/
>> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
>> Those who would give up essential Liberty, to purchase a little temporary
>> Safety, deserve neither Liberty nor Safety.
>>    -- Benjamin Franklin
>>
>> -----------------------------------------------------------------
>> Securing Your Online Data Transfer with SSL.
>> A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
>> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>>
>>
>>
>
> -----------------------------------------------------------------
> Securing Your Online Data Transfer with SSL.
> A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>
>
>
> -----------------------------------------------------------------
> Securing Your Online Data Transfer with SSL.
> A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>
>
>

--
Martin Roesch - Founder/CTO, Sourcefire Inc. - [phone]
Sourcefire - Security for the Real World - http://www.sourcefire.com
Snort: Open Source IDP - http://www.snort.org

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194