adding session id to entries in access log

[Douglas Sims-2]

We've just launched the first mod_perl site I've ever designed.  It's all going very well so far but I'm sure there are some things worth improving.  I wonder if anyone might have suggestions about this scenario:

I want to add the session id to the access log entries.  This example: http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler shows how to write to a different logfile but I want to write to whatever would be the normal access log for whatever VirtualHost it's in.   We've only got a PerlResponseHandler now but I think this should probably go in a PerlLogHander.  What's the best way to go about this?

We're very interested in tracking long-term user browsing behavior and so we set one persistent cookie with a session key at each request if there's no cookie or if the existing cookie is obsolete (user logged in, logged out, more than 1 hour since last access, 12 since last visit, IP changed, or user agent changed.)  If, when a new session id is created there is an existing (but obsolete) session cookie then the obsolete one is stored in the sessions table as the previous session key.

I've tried to follow the philosophy that Randal Schwartz described in a recent thread here - a cookie is just a serial number for a browser.  By rotating the cookies often we're hoping to avoid problems with stolen or leaked sessions and by storing the previous session id (if there is one) with every new session we're planning to be able to build a linked list of session activity which we can correlate with specific users who log in at any part of that linked list.

[Cosimo Streppone]

Douglas wrote:

> I want to add the session id to the access log entries.

We just added "%{session_id}C" as an additional field to
our CustomLog directive, and that worked fine for us.

http://httpd.apache.org/docs/2.0/mod/mod_log_config.html

--
Cosimo

[Randal L. Schwartz]

>>>>> "Douglas" == Douglas Sims <ratsbane@...> writes:

Douglas> I've tried to follow the philosophy that Randal Schwartz described in
Douglas> a recent thread here - a cookie is just a serial number for a
Douglas> browser.  By rotating the cookies often we're hoping to avoid
Douglas> problems with stolen or leaked sessions and by storing the previous
Douglas> session id (if there is one) with every new session we're planning to
Douglas> be able to build a linked list of session activity which we can
Douglas> correlate with specific users who log in at any part of that linked
Douglas> list.

That's an interesting idea... brand the browser, but rotate it from time to
time, maintaining a list.  Thanks for suggesting that... I'll have to explore
that in some future project.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - [phone]
<merlyn@...> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

[Douglas Sims-2]

Thanks, Paul and Cosimo.

That module is just what I was looking for.  Logging the note instead of the cookie is probably better for what we're doing.

On Fri, Oct 2, 2009 at 6:56 AM, Paul Silevitch <paul@...> wrote:

You can use apache's custom log (http://httpd.apache.org/docs/2.0/mod/mod_log_config.html) to log cookie values into the access logs:

%...{Foobar}C The contents of cookie Foobar in the request sent to the server.

The above will not log a value for the first request by a new visitor (since the cookie hasn't been set yet).  Instead, create a note that gets set on every request in your handler and log that:

%...{Foobar}n The contents of note Foobar from another module.

HTH,

Paul

On Fri, Oct 2, 2009 at 2:21 AM, Douglas Sims <ratsbane@...> wrote:

We've just launched the first mod_perl site I've ever designed.  It's all going very well so far but I'm sure there are some things worth improving.  I wonder if anyone might have suggestions about this scenario:

I want to add the session id to the access log entries.  This example: http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler shows how to write to a different logfile but I want to write to whatever would be the normal access log for whatever VirtualHost it's in.   We've only got a PerlResponseHandler now but I think this should probably go in a PerlLogHander.  What's the best way to go about this?

We're very interested in tracking long-term user browsing behavior and so we set one persistent cookie with a session key at each request if there's no cookie or if the existing cookie is obsolete (user logged in, logged out, more than 1 hour since last access, 12 since last visit, IP changed, or user agent changed.)  If, when a new session id is created there is an existing (but obsolete) session cookie then the obsolete one is stored in the sessions table as the previous session key.

I've tried to follow the philosophy that Randal Schwartz described in a recent thread here - a cookie is just a serial number for a browser.  By rotating the cookies often we're hoping to avoid problems with stolen or leaked sessions and by storing the previous session id (if there is one) with every new session we're planning to be able to build a linked list of session activity which we can correlate with specific users who log in at any part of that linked list.

[Douglas Sims-2]

We're starting to get some data now and I'm seeing a few minor problems with the implementation of the idea. 

Here's one: if a request arrives with a cookie that is associated with a different IP address then we create a new session entry and send a new cookie.  I'm noticing that some users have IP addreses that change very frequently.  This seems particularly likely with mobile devices.  We're going to have to tweak the algorithm a bit to track activity across those requests - or use some attribute of the IP address instead of just the IP address - to trigger rotation of the session identity.

On Sun, Oct 11, 2009 at 10:19 AM, Randal L. Schwartz <merlyn@...> wrote:

>>>>> "Douglas" == Douglas Sims <ratsbane@...> writes:

Douglas> I've tried to follow the philosophy that Randal Schwartz described in
Douglas> a recent thread here - a cookie is just a serial number for a
Douglas> browser.  By rotating the cookies often we're hoping to avoid
Douglas> problems with stolen or leaked sessions and by storing the previous
Douglas> session id (if there is one) with every new session we're planning to
Douglas> be able to build a linked list of session activity which we can
Douglas> correlate with specific users who log in at any part of that linked
Douglas> list.

That's an interesting idea... brand the browser, but rotate it from time to
time, maintaining a list.  Thanks for suggesting that... I'll have to explore
that in some future project.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - [phone]
<merlyn@...> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

[Douglas Sims-2]

Thanks!  That's a good idea.

Just as an example, here are several IPs that seemed to be the same client.  The user agent, referer, etc. were all the same and the IPs resolve to the same top-level domain.

205.228.12.236
205.228.12.151
205.228.12.254

On Mon, Oct 12, 2009 at 5:52 PM, Paul Silevitch <paul@...> wrote:

Also, requests that go through a proxy can appear to come from different IP addresses from hit to hit (or visit to visit).  Usually, proxies will put the real IP as the first IP in the X-Forwarded-For header.

HTH,

Paul

On Mon, Oct 12, 2009 at 6:43 PM, Douglas Sims <ratsbane@...> wrote:

We're starting to get some data now and I'm seeing a few minor problems with the implementation of the idea. 

Here's one: if a request arrives with a cookie that is associated with a different IP address then we create a new session entry and send a new cookie.  I'm noticing that some users have IP addreses that change very frequently.  This seems particularly likely with mobile devices.  We're going to have to tweak the algorithm a bit to track activity across those requests - or use some attribute of the IP address instead of just the IP address - to trigger rotation of the session identity.

On Sun, Oct 11, 2009 at 10:19 AM, Randal L. Schwartz <merlyn@...> wrote:

>>>>> "Douglas" == Douglas Sims <ratsbane@...> writes:

Douglas> I've tried to follow the philosophy that Randal Schwartz described in
Douglas> a recent thread here - a cookie is just a serial number for a
Douglas> browser.  By rotating the cookies often we're hoping to avoid
Douglas> problems with stolen or leaked sessions and by storing the previous
Douglas> session id (if there is one) with every new session we're planning to
Douglas> be able to build a linked list of session activity which we can
Douglas> correlate with specific users who log in at any part of that linked
Douglas> list.

That's an interesting idea... brand the browser, but rotate it from time to
time, maintaining a list.  Thanks for suggesting that... I'll have to explore
that in some future project.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - [phone]
<merlyn@...> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion