Measuring Bandwidth usage by Application Protocol

[jm-13]

It's a little quiet on the list today. May be everyone's taking an
extended "Community Day" or something. Anyway, I'm looking at generating
some network usage statistics on an open network with a large number of
users. The data I'm looking to collect is

  Source IP
  Source Port
  Destination IP
  Destination Port
  Application Protocol
  Packets
  Octets
  Duration


of each session or flow on a 15 minute basis. NetFlow is close to giving
me with this, but lacks the application level protocol information that
I'm after. Any thought, or suggestions on how to collect this data?

Jeff.
--
linux mailing list
linux@...
https://lists.samba.org/mailman/listinfo/linux

[Steve Walsh-2]

nethogs mebbe?

http://nethogs.sourceforge.net/

gives you traffic per pid..

On 11/02/2009 03:53 PM, jm wrote:

>
> It's a little quiet on the list today. May be everyone's taking an
> extended "Community Day" or something. Anyway, I'm looking at
> generating some network usage statistics on an open network with a
> large number of users. The data I'm looking to collect is
>
>  Source IP
>  Source Port
>  Destination IP
>  Destination Port
>  Application Protocol
>  Packets
>  Octets
>  Duration
>
>
> of each session or flow on a 15 minute basis. NetFlow is close to
> giving me with this, but lacks the application level protocol
> information that I'm after. Any thought, or suggestions on how to
> collect this data?
>
> Jeff.

--
--==--
Steve Walsh
RHCE
Vice President / Sysadmin team member - Linux Australia
Networks and Technology - Linux.conf.au 2008
Evil Network Bunny - Linux.conf.au 2009
Networking alpha geek - Linux.conf.au 2010

--
linux mailing list
linux@...
https://lists.samba.org/mailman/listinfo/linux

[Dale Shaw]

Hi Jeff,

On Mon, Nov 2, 2009 at 3:53 PM, jm <jeffm@...> wrote:

>
> I'm looking at generating some network
> usage statistics on an open network with a large number of users. The data
> I'm looking to collect is
>
>  Source IP
>  Source Port
>  Destination IP
>  Destination Port
>  Application Protocol
>  Packets
>  Octets
>  Duration

OK.

> of each session or flow on a 15 minute basis. NetFlow is close to giving me
> with this, but lacks the application level protocol information that I'm
> after. Any thought, or suggestions on how to collect this data?

What application-layer detail do you need that a NetFlow-based
solution doesn't give you? Usually (from what I've seen) this is done
in the reporting engine, rather than the collector itself -- sometimes
these functions are combined into one system/product. It's usually
limited (again, based on my experience) to application recognition
based on network and transport layer information (e.g. src/dst IP
addresses, IP protocol numbers, TCP/UDP port numbers).

Digging deeper into the packet gets pretty expensive in terms of
network device resources, so perhaps a port mirror or TAP solution
utilising a passive probe would give you the insight you need.

Anyway, which collector are you using? Are you strictly looking for a
$free solution?

Have you had a play with 'ntop'?

cheers,
Dale
--
linux mailing list
linux@...
https://lists.samba.org/mailman/listinfo/linux

[jm-13]

Dale Shaw wrote:

>
> What application-layer detail do you need that a NetFlow-based
> solution doesn't give you? Usually (from what I've seen) this is done
> in the reporting engine, rather than the collector itself -- sometimes
> these functions are combined into one system/product. It's usually
> limited (again, based on my experience) to application recognition
> based on network and transport layer information (e.g. src/dst IP
> addresses, IP protocol numbers, TCP/UDP port numbers).
>
>  

At a minimum the applications I'd like to identifiy are

Bit Torrent and other peer to peer
games
ftp and other file transfer protocols
http (grouped by well known URLs)
VPNs including PPTP, IP-SEC, etc
SMTP, POP, IMAP

and quite a few I've mostly likely left off this list, plus the
forgotten unknown category.
> Digging deeper into the packet gets pretty expensive in terms of
> network device resources, so perhaps a port mirror or TAP solution
> utilising a passive probe would give you the insight you need.
>  
True. I already have all traffic mirrored to a dedicated box which is
used for netflow.

> Anyway, which collector are you using? Are you strictly looking for a
> $free solution?
>  
fprobe and flow-tools.

Lets see budget....ZERO dollars which is the usual problem faced when
doing this sort of thing. For the sake of discussion feel free to
mention the non-free.

A couple of people have suggested Cisco NBR. The only problem is that I
don't have a cisco capable of this at the right location in the network.

Jeff.

--
linux mailing list
linux@...
https://lists.samba.org/mailman/listinfo/linux

On Mon, 2009-11-02 at 15:53 +1100, jm wrote:

> It's a little quiet on the list today. May be everyone's taking an
> extended "Community Day" or something. Anyway, I'm looking at generating
> some network usage statistics on an open network with a large number of
> users. The data I'm looking to collect is
>
>   Source IP
>   Source Port
>   Destination IP
>   Destination Port
>   Application Protocol
>   Packets
>   Octets
>   Duration
>
>
> of each session or flow on a 15 minute basis. NetFlow is close to giving
> me with this, but lacks the application level protocol information that
> I'm after. Any thought, or suggestions on how to collect this data?

have you come across argus?

http://www.qosient.com/argus/

Is used in quite a few places around town.

I've used it to get summary info and then pull that into R for detailed
statistical analysis.


Digging into the stuff in the data portion of the packet tho... that
gets harder (read $$$, in time if not actual $$$).


HTH?
Cheers!
- Steve


--
linux mailing list
linux@...
https://lists.samba.org/mailman/listinfo/linux

[jm-13]

Steve McInerney wrote:
> have you come across argus?
>
> http://www.qosient.com/argus/
>
>  
No, I haven't. I've started reading about it now though.

> Is used in quite a few places around town.
>
> I've used it to get summary info and then pull that into R for detailed
> statistical analysis.
>  

This is what I'm trying to do. Get raw numbers to drag into R where I
can manipulate/play with them. So far I'm only been able to do this with
numbers exported from cacti and was hoping for something a little more
detailed so as to investigate individual users, protocol, autonomous
systems, etc. Can you comment further on how your using R or can you
point to any useful information on networks traffic and analysis with R.
> Digging into the stuff in the data portion of the packet tho... that
> gets harder (read $$$, in time if not actual $$$).
>  
Yeap. Definitately.

Jeff.

--
linux mailing list
linux@...
https://lists.samba.org/mailman/listinfo/linux

On Thu, 2009-11-05 at 09:07 +1100, jm wrote:

> > I've used it to get summary info and then pull that into R for detailed
> > statistical analysis.
> >  
>
> This is what I'm trying to do. Get raw numbers to drag into R where I
> can manipulate/play with them. So far I'm only been able to do this with
> numbers exported from cacti and was hoping for something a little more
> detailed so as to investigate individual users, protocol, autonomous
> systems, etc. Can you comment further on how your using R or can you
> point to any useful information on networks traffic and analysis with R.

Sure. Sorry for the delay, life intruded as it so often does.

Keeping in mind it's been about 2 years since I've last done this in
anger, so I haven't got any notes on the detail unf.


The general gist is reduce the data down to something "interesting" and
fiddle from there. Export to a format that R can deal with; import;
display as a percentile graph - I've found for this stuff percentiles
FTW - emphatically over avg etc.

Use various filters to exclude noise. eg within argus, awk! or R itself.

http://www.stedee.id.au/2007/09-29/advanced_sysadmin_troubleshooting_slow_websites

is a very very high level "howto" of ... how to. :-)

http://www.stedee.id.au/Apache_WebSite_Responsiveness

is a more detailed one of futzing around with apache logs to extract
info.


Argus can enable you to dump raw columns of data; that can in turn be
imported into R and analysed in some way shape or form. But you'll, in
general, need to apply some level of filtering - ala tcpdump style
expressions - first.

Sorry this is a tad light on detail. But tbh, I just don't recall
them :-)


Cheers!
- Steve


--
linux mailing list
linux@...
https://lists.samba.org/mailman/listinfo/linux

[jm-13]

Steve McInerney wrote:

> On Thu, 2009-11-05 at 09:07 +1100, jm wrote:
>  
>>> I've used it to get summary info and then pull that into R for detailed
>>> statistical analysis.
>>>  
>>>      
>> This is what I'm trying to do. Get raw numbers to drag into R where I
>> can manipulate/play with them. So far I'm only been able to do this with
>> numbers exported from cacti and was hoping for something a little more
>> detailed so as to investigate individual users, protocol, autonomous
>> systems, etc. Can you comment further on how your using R or can you
>> point to any useful information on networks traffic and analysis with R.
>>    
>
>
> Sure. Sorry for the delay, life intruded as it so often does.
>  

Life's a bit like that.

> Keeping in mind it's been about 2 years since I've last done this in
> anger, so I haven't got any notes on the detail unf.
>
>
> The general gist is reduce the data down to something "interesting" and
> fiddle from there. Export to a format that R can deal with; import;
> display as a percentile graph - I've found for this stuff percentiles
> FTW - emphatically over avg etc.
>
> Use various filters to exclude noise. eg within argus, awk! or R itself.
>
> http://www.stedee.id.au/2007/09-29/advanced_sysadmin_troubleshooting_slow_websites
>
> is a very very high level "howto" of ... how to. :-)
>
> http://www.stedee.id.au/Apache_WebSite_Responsiveness
>
> is a more detailed one of futzing around with apache logs to extract
> info.
>
>  

Interesting read. On the latter one (Apache Website Responsiveness) I
would have though you add box plots and standard deviation on a per
period basis, eg day, into the mix. That's just me though and you've got
to remember your audience. Lead to a couple of extra tools I'd not heard
of as well, but nothing to be surprised at there I suppose as there's
more tools out there than one can be familiar with.

Jeff.

--
linux mailing list
linux@...
https://lists.samba.org/mailman/listinfo/linux

On Wed, 2009-11-11 at 14:33 +1100, jm wrote:

>
> Steve McInerney wrote:
> > http://www.stedee.id.au/Apache_WebSite_Responsiveness
> >
> > is a more detailed one of futzing around with apache logs to extract
> > info.
> >
> >  
> Interesting read. On the latter one (Apache Website Responsiveness) I
> would have though you add box plots and standard deviation on a per
> period basis, eg day, into the mix. That's just me though and you've got
> to remember your audience.

No, that's damn good advice.
I didn't do that, because my statistics knowledge is fairly low; so was
working within what I knew.

I guess at the end of the day, the idea from this sort of analysis is to
wind up with a series of "data" that in turn drives actions to take to
fix whatever.

In the inital case I did this for apache, it was "ZOMG! SLOW!!!" so we
have a massive effort to make pages faster. We achieved that goal, so it
wasn't such a big deal to go on.

Keeping in mind we also had (still have?) several additional monitoring
point, nagios into rrd etc so we had continual monitoring of page speed,
above and beyond this "every few months" analysis.


Cheers!
- Steve


--
linux mailing list
linux@...
https://lists.samba.org/mailman/listinfo/linux