TLS renegotiation disabling : mod_ssl and OpenSSL 0.9.8l

[Jean-Marc Desperrier-2]

Hi,

So when Apache is compiled with openssl 0.9.8l, TLS renegotiation will
be fully disabled.

But the problem with that if that some comments of the discussion inside
https://issues.apache.org/bugzilla/show_bug.cgi?id=39243 are true, this
change will unexpectedly break very badly a *lot* of sites.

Those comments suggest Apache currently requests TLS renegotiation in
quite a few cases where it should not be needed, and where it won't be
expected.

First there's the short SSLSessionCacheTimeout problem :
https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c23

In my understanding, SSLSessionCacheTimeout should never cause
renegociation. If the client tries to do a SSL session resume, and the
server does not have the SSL info for that SSL id anymore, the result is
a brand new SSL Session, *not* a SSL renegotiation.

If they actually are resumes caused by SSLSessionCacheTimeout, then it
seems SSLSessionCacheTimeout times out sessions that are currently
active at the TCP level, and where the user is just trying to send more
data. Or there's a bug in the resume code that first says yes, then
finds the session id should have been timed out, so forces a
renegotiation. Anyway, this means this was already broken in some way
before, but it used to be of little consequences and will now be a huge
problem.

Second there's the SSLVerifyClient optional problem :
https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c21

When SSLVerifyClient optional is actually used to allow various
authentication level, it's expected that the change will break the site.

But what this comment report is that simply having SSLVerifyClient
optional set, not having any different value anywhere, not even *using*
client certificates, will cause renegotiation to happen and therefore
sites to break when TLS renegotiation is disabled.
And Peter Gutmann just reported on oss-sec that they are many servers
configured like this (see http://seclists.org/oss-sec/2009/q4/138 ) with
their admin not even knowing that they require client certificates.

According to the following comment, it might be that a very simple patch
would correct this problems :
https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c16
(I've checked the svn code for apache 2.2.x, that piece of code is still
exactly the same, but I don't understand enough to be sure that just
handling the "optional" case in addition to the "none" case would really
be enough to solve the problem)

[Stefan Fritsch]

Hi,

On Tuesday 10 November 2009, Jean-Marc Desperrier wrote:

> So when Apache is compiled with openssl 0.9.8l, TLS renegotiation
>  will be fully disabled.
>
> But the problem with that if that some comments of the discussion
>  inside https://issues.apache.org/bugzilla/show_bug.cgi?id=39243
>  are true, this change will unexpectedly break very badly a *lot*
>  of sites.
>
> Those comments suggest Apache currently requests TLS renegotiation
>  in quite a few cases where it should not be needed, and where it
>  won't be expected.
>
> First there's the short SSLSessionCacheTimeout problem :
> https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c23
>
> In my understanding, SSLSessionCacheTimeout should never cause
> renegociation. If the client tries to do a SSL session resume, and
>  the server does not have the SSL info for that SSL id anymore, the
>  result is a brand new SSL Session, *not* a SSL renegotiation.
>
> If they actually are resumes caused by SSLSessionCacheTimeout, then
>  it seems SSLSessionCacheTimeout times out sessions that are
>  currently active at the TCP level, and where the user is just
>  trying to send more data. Or there's a bug in the resume code that
>  first says yes, then finds the session id should have been timed
>  out, so forces a renegotiation. Anyway, this means this was
>  already broken in some way before, but it used to be of little
>  consequences and will now be a huge problem.
>
> Second there's the SSLVerifyClient optional problem :
> https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c21
>
> When SSLVerifyClient optional is actually used to allow various
> authentication level, it's expected that the change will break the
>  site.
>
> But what this comment report is that simply having SSLVerifyClient
> optional set, not having any different value anywhere, not even
>  *using* client certificates, will cause renegotiation to happen
>  and therefore sites to break when TLS renegotiation is disabled.
> And Peter Gutmann just reported on oss-sec that they are many
>  servers configured like this (see
>  http://seclists.org/oss-sec/2009/q4/138 ) with their admin not
>  even knowing that they require client certificates.
>
> According to the following comment, it might be that a very simple
>  patch would correct this problems :
> https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c16
> (I've checked the svn code for apache 2.2.x, that piece of code is
>  still exactly the same, but I don't understand enough to be sure
>  that just handling the "optional" case in addition to the "none"
>  case would really be enough to solve the problem)
>

I cannot reproduce the problems. With an openssl that rejects all
renegotiations, both reconnections after ssl session timeout and
connections to a host with sslverifyclient optional work fine (with
openssl s_client).

Cheers,
Stefan

[Jean-Marc Desperrier-2]

Stefan Fritsch wrote:
> On Tuesday 10 November 2009, Jean-Marc Desperrier wrote:
>> [ Apache + openssl 0.9.8l = TLS renegotiation fully disabled ]
>> First there's the short SSLSessionCacheTimeout problem :
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c23
>> [...] If they actually are renegotiation caused by SSLSessionCacheTimeout,
 >> [...], this means this was

>>   already broken in some way before, but it used to be of little
>>   consequences and will now be a huge problem.
>>
>> Second there's the SSLVerifyClient optional problem :
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c21
>> [...]  what this comment report is that simply having SSLVerifyClient
>>  optional set, [...], will cause renegotiation to happen
>>  and therefore sites to break when TLS renegotiation is disabled.
>
> I cannot reproduce the problems. With an openssl that rejects all
> renegotiations, both reconnections after ssl session timeout and
> connections to a host with sslverifyclient optional work fine (with
> openssl s_client).

Thank you for your interest on that problem.

One thing still : Everyone who uses client certificate authentication
knows that they are many apache configurations around that will force
the user to repeatedly reauthenticate himself for apparently no good reason.

It's hard to believe the explanation is only that all of the concerned
sites forgot to activate the "session resume" option.
SSLVerifyClient and SSLSessionCacheTimeout forcing unnecessary
renegotiation did seem like a very plausible alternative explanation.

This fact is the very reason why the vilified "remember client
certificate" option is there in Firefox 3.5 (wasn't there in 3.0),
there's a large number of bugs opened on the subject in their bugzilla :
https://bugzilla.mozilla.org/show_bug.cgi?id=510820
https://bugzilla.mozilla.org/show_bug.cgi?id=453802
https://bugzilla.mozilla.org/show_bug.cgi?id=428744
https://bugzilla.mozilla.org/show_bug.cgi?id=474445
https://bugzilla.mozilla.org/show_bug.cgi?id=395399
https://bugzilla.mozilla.org/show_bug.cgi?id=32010

I'll try to find out more about this, with so many users reporting that
problem, there should be a way to get some more detailed info about what
causes it, if it's related with erroneous renegotiation or not.

[Jean-Marc Desperrier-2]

Jean-Marc Desperrier wrote:

> Everyone who uses client certificate authentication knows that they are
> many apache configurations around that will force the user to repeatedly
> reauthenticate himself for apparently no good reason.
>
> It's hard to believe the explanation is only that all of the concerned
> sites forgot to activate the "session resume" option.
> SSLVerifyClient and SSLSessionCacheTimeout forcing unnecessary
> renegotiation did seem like a very plausible alternative explanation.
> [...]
>
> I'll try to find out more about this, with so many users reporting that
> problem, there should be a way to get some more detailed info about what
> causes it, if it's related with erroneous renegotiation or not.

Ok, so in fact I have one apache instance available locally with a
problem of this kind. It's configured to not require client
authentication by defaut, but to require it on the /authentication url

So what happens truly is that after firefox does a request to
/authentication, mod_ssl does renegociation 6 times in a row for
apparently no reason.

An interesting point is that firefox is *not* reusing the ssl session in
that case, for some reason it sends a SessionID of 0 after the "Hello
Request" from the server. I'll forward that to the NSS team, because if
that behavior continues, it will be no use to implement the IETF draft
for secure renegociation, it will fail all the same.

In that specific case, there's not much useful to learn, we get several
re-authentication where only one was needed, but after that it's OK, no
more needless renegociation requests. But I hear that in some
configuration there's a reauthentication everytime the user requires a
resource. I'll check if I can find more, if it's not just a case of
SSLSessionCache not being enabled.

Here's the wireshark captured exchange between the client and server,
note that "Hello Request" always *immediatly* follows the end of the
renegotiation. This is with Apache 2.2.11/Openssl 0.9.8i (not a
production server) :

> 217 19:30:50.745606 client_ip server_ip HTTP GET /authentication/ HTTP/1.1
> 218 19:30:50.747473 server_ip client_ip TLSv1 Hello Request
> 219 19:30:50.747896 client_ip server_ip TLSv1 Client Hello
> 220 19:30:50.749114 server_ip client_ip TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
> 257 19:30:59.267340 client_ip server_ip TLSv1 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
> 259 19:30:59.288262 server_ip client_ip TLSv1 Change Cipher Spec, Finished
> 260 19:30:59.289066 server_ip client_ip TLSv1 Hello Request
> 262 19:30:59.289511 client_ip server_ip TLSv1 Client Hello
> 263 19:30:59.290741 server_ip client_ip TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
> 282 19:31:07.747697 client_ip server_ip TLSv1 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
> 284 19:31:07.768765 server_ip client_ip TLSv1 Change Cipher Spec, Finished
> 285 19:31:07.769681 server_ip client_ip TLSv1 Hello Request
> 287 19:31:07.770128 client_ip server_ip TLSv1 Client Hello
> 288 19:31:07.771261 server_ip client_ip TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
> 309 19:31:12.237699 client_ip server_ip TLSv1 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
> 311 19:31:12.258634 server_ip client_ip TLSv1 Change Cipher Spec, Finished
> 312 19:31:12.259583 server_ip client_ip TLSv1 Hello Request
> 314 19:31:12.260044 client_ip server_ip TLSv1 Client Hello
> 315 19:31:12.261187 server_ip client_ip TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
> 441 19:31:17.613690 client_ip server_ip TLSv1 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
> 443 19:31:17.634661 server_ip client_ip TLSv1 Change Cipher Spec, Finished
> 444 19:31:17.635574 server_ip client_ip TLSv1 Hello Request
> 446 19:31:17.636005 client_ip server_ip TLSv1 Client Hello
> 447 19:31:17.638787 server_ip client_ip TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
> 467 19:31:21.725473 client_ip server_ip TLSv1 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
> 469 19:31:21.746555 server_ip client_ip TLSv1 Change Cipher Spec, Finished
> 470 19:31:21.747127 server_ip client_ip TLSv1 Hello Request
> 472 19:31:21.747552 client_ip server_ip TLSv1 Client Hello
> 473 19:31:21.748911 server_ip client_ip TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
> 507 19:31:37.237284 client_ip server_ip TLSv1 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
> 509 19:31:37.258081 server_ip client_ip TLSv1 Change Cipher Spec, Finished
> 510 19:31:37.260057 server_ip client_ip HTTP HTTP/1.1 200 OK  (text/html)

[Torsten Foertsch]

On Mon 16 Nov 2009, Jean-Marc Desperrier wrote:

> Here's the wireshark captured exchange between the client and server,
> note that "Hello Request" always *immediatly* follows the end of the
> renegotiation. This is with Apache 2.2.11/Openssl 0.9.8i (not a
>
> production server) :
> > 217   19:30:50.745606 client_ip       server_ip       HTTP    GET
> > /authentication/ HTTP/1.1
> > 218   19:30:50.747473 server_ip       client_ip       TLSv1   Hello
> > Request
> > 219   19:30:50.747896 client_ip       server_ip       TLSv1   Clien
> >t Hello
> > 220   19:30:50.749114 server_ip       client_ip       TLSv1   Serve
> >r Hello, Certificate, Certificate Request, Server Hello Done
> > 257   19:30:59.267340 client_ip       server_ip       TLSv1   Certi
> >ficate, Client Key Exchange, Certificate Verify, Change Cipher Spec,
> > Finished
> > 259   19:30:59.288262 server_ip       client_ip       TLSv1   Chang
> >e Cipher Spec, Finished
> > 260   19:30:59.289066 server_ip       client_ip       TLSv1   Hello
> > Request
> > 262   19:30:59.289511 client_ip       server_ip       TLSv1   Clien
> >t Hello

...
> > 510   19:31:37.260057 server_ip       client_ip       HTTP    HTTP/
> >1.1 200 OK  (text/html)

I have noticed something similar. Don't know if it applies to you. If
your /authentication/ is a resource that generates a directory listing
via mod_autoindex then apache issues a subrequest for each directory
entry. Now, if only /authentication/ requires a client certificate but
the VHost or base server does not then each entry leads to a
renegotiation. Correct me if I am wrong but that is how I have
explained the behavior for me.

Torsten

--
Need professional mod_perl support?
Just hire me: torsten.foertsch@...

[Joe Orton]

On Mon, Nov 16, 2009 at 08:21:20PM +0100, Jean-Marc Desperrier wrote:
> Ok, so in fact I have one apache instance available locally with a  
> problem of this kind. It's configured to not require client  
> authentication by defaut, but to require it on the /authentication url
>
> So what happens truly is that after firefox does a request to  
> /authentication, mod_ssl does renegociation 6 times in a row for  
> apparently no reason.

Please file a bug and attach all of:

a) error_log output at "LogLevel debug" for that case
b) the config snipping that you're using for /authentication
c) the mod_ssl configuration

Regards, Joe

[Rainer Jung-3]

On 16.11.2009 20:21, Jean-Marc Desperrier wrote:
> Jean-Marc Desperrier wrote:
> An interesting point is that firefox is *not* reusing the ssl session in
> that case, for some reason it sends a SessionID of 0 after the "Hello
> Request" from the server. I'll forward that to the NSS team, because if
> that behavior continues, it will be no use to implement the IETF draft
> for secure renegociation, it will fail all the same.

Only concerning empty session id: during testing the renegotiation fix I
was also looking at session IDs and got aware, that the TLS session
ticket extension will move the session status to the client. In this
case the requests belonging to the initial connection to the server will
not carry a session id. Later connections from the same client will
provide some client generated id to the server that the server replaces
it by a new one. The session ticket extension allows the server to not
maintain the session data in its own cache. In the presence of the
session ticket extension, session IDs observed on the server are no
longer a good measurement for session reuse.

In Firefox's about:config you can filter fot "tls" and switch session
tickets off.

Regards,

Rainer

[Jean-Marc Desperrier-2]

Rainer Jung wrote:
>  In the presence of the
> session ticket extension, session IDs observed on the server are no
> longer a good measurement for session reuse.

Nice remark, except it's not that, it's really broken. With "session
tickets off" (confirmed by the absence of the session ticket extension
in the client hello), it's still the same behaviour. Apache
2.2.11/openssl 0.9.8i does not have session tickets enabled in my setup.

This being said :  The idea of using non-constant SSL session ID in the
specification of the session ticket extension was really *bad*.

[Jean-Marc Desperrier-2]

Joe Orton wrote:
> Please file a bug and attach all of:
>
> a) error_log output at "LogLevel debug" for that case
> b) the config snipping that you're using for /authentication
> c) the mod_ssl configuration

This is now done in bug
https://issues.apache.org/bugzilla/show_bug.cgi?id=48215

error.log might have enough info to understand what happens, but I
included everything else needed to repro from scratch.

[Dr Stephen Henson-2]

Jean-Marc Desperrier wrote:

> Joe Orton wrote:
>> Please file a bug and attach all of:
>>
>> a) error_log output at "LogLevel debug" for that case
>> b) the config snipping that you're using for /authentication
>> c) the mod_ssl configuration
>
> This is now done in bug
> https://issues.apache.org/bugzilla/show_bug.cgi?id=48215
>
> error.log might have enough info to understand what happens, but I
> included everything else needed to repro from scratch.
>

What happens with the latest 0.9.8-stable version of OpenSSL?

Steve.
--
Dr Stephen N. Henson. Senior Technical/Cryptography Advisor,
Open Source Software Institute: www.oss-institute.org
OpenSSL Core team: www.openssl.org

[Jean-Marc Desperrier-2]

Dr Stephen Henson wrote:

> Jean-Marc Desperrier wrote:
>> Joe Orton wrote:
>>> Please file a bug and attach all of:
>>>
>>> a) error_log output at "LogLevel debug" for that case
>>> b) the config snipping that you're using for /authentication
>>> c) the mod_ssl configuration
>>
>> This is now done in bug
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=48215
>>
>> error.log might have enough info to understand what happens, but I
>> included everything else needed to repro from scratch.
>
> What happens with the latest 0.9.8-stable version of OpenSSL?

Stephen, what result do you expect from this ?
Does the latest 0.9.8-stable already implement safe renegociation ? But
I'd need a version of Firefox that implement it for testing (I'll try to
get that from Nelson).

If renegociation is simply disabled, this case will simply fail as expected.
It's not a case of mod_ssl starting renegotiation where *none* is required.
Some comments imply that one also happens sometimes but I don't know if
it's true as I don't know how precisely to reproduce it.
But I won't exclude it given how easy it is to fall into the problem of
mod_ssl requiring more renegotiations than really needed.

[Jean-Marc Desperrier-2]

Torsten Foertsch wrote:
> If
> your/authentication/  is a resource that generates a directory listing
> via mod_autoindex then apache issues a subrequest for each directory
> entry.

This is not what I was testing, but you are *very right* that there is
also that problem. I'll open a bug for it, maybe analyzes will show it's
just a duplicate of 48215, but for now technically it might be a
different issue.

 > Now, if only/authentication/  requires a client certificate but
 > the VHost or base server does not then each entry leads to a
 > renegotiation.
> Correct me if I am wrong but that is how I have
> explained the behavior for me.

I don't know but anyway it's still a bug.

If the resolution of the SSL vulnerability had been to remove
renegociation altogether, it would not matter. But as renegociation will
still be there, bugs that affect renegociation should be solved.

[Jean-Marc Desperrier-2]

Stefan Fritsch wrote:
> I cannot reproduce the problems. With an openssl that rejects all
> renegotiations, both reconnections after ssl session timeout and
> connections to a host with sslverifyclient optional work fine (with
> openssl s_client).

I have now succeeded in reproducing at least partially the
"SSLVerifyClient optional" problem, though what I'm testing in not
exactly the same as you.

I'm testing that with a server where the vhost context has
"SSLVerifyClient None" and a /authentication directory has
"SSLVerifyClient optional", requests that alternate between these two
directory will repeatedly require authentication even when you have
already authenticated yourself inside the same SSL session.

The setup is the same as in
https://issues.apache.org/bugzilla/show_bug.cgi?id=48215 , with
SSLVerifyDepth 0 moved to vhost context in order to work around the
initial bug 48215 problem.

I just needs the following change in the reproduction steps to reproduce
from firefox :
- load /
- load /authentification
- just click enter to accept authentication
- press back to return to /
- press F5 to force reload
- load /authentification
- press F5 to force reload
- authentication is requested again by a server initiated renegotiation

Doing this fast enough (less than 4 or 5 seconds between each step),
Firefox doesn't close the TCP connexion, so it happens inside the same
already authenticated ssl session, with no ssl session resume involved.

I think that in bug 48228, when I describe that reloading the page
causes renegotiation, it's the same bug.

So I don't know if Joe prefers that I open another bug for this, or
wants to handle this in bug 48228.

I have validated that the behavior is the same when the client doesn't
provide a certificate after the renegotiation. It's not really
surprising but it's interesting in the light of the case of people
having set "SSLVerifyClient optional" by error, and not actually using
client certificates.

[Ruediger Pluem]

On 11/18/2009 08:32 PM, Jean-Marc Desperrier wrote:

> Stefan Fritsch wrote:
>> I cannot reproduce the problems. With an openssl that rejects all
>> renegotiations, both reconnections after ssl session timeout and
>> connections to a host with sslverifyclient optional work fine (with
>> openssl s_client).
>
> I have now succeeded in reproducing at least partially the
> "SSLVerifyClient optional" problem, though what I'm testing in not
> exactly the same as you.
>
> I'm testing that with a server where the vhost context has
> "SSLVerifyClient None" and a /authentication directory has
> "SSLVerifyClient optional", requests that alternate between these two
> directory will repeatedly require authentication even when you have
> already authenticated yourself inside the same SSL session.

Have you tried if this goes away when you set

OptRenegotiate for SSLOptions

http://httpd.apache.org/docs/2.2/en/mod/mod_ssl.html#ssloptions

?

Regards

Rüdiger

[Dr Stephen Henson-2]

Jean-Marc Desperrier wrote:

> Dr Stephen Henson wrote:
>> Jean-Marc Desperrier wrote:
>>> Joe Orton wrote:
>>>> Please file a bug and attach all of:
>>>>
>>>> a) error_log output at "LogLevel debug" for that case
>>>> b) the config snipping that you're using for /authentication
>>>> c) the mod_ssl configuration
>>>
>>> This is now done in bug
>>> https://issues.apache.org/bugzilla/show_bug.cgi?id=48215
>>>
>>> error.log might have enough info to understand what happens, but I
>>> included everything else needed to repro from scratch.
>>
>> What happens with the latest 0.9.8-stable version of OpenSSL?
>
> Stephen, what result do you expect from this ?
>

Basically to rule out ticket issues other things fixed in 0.9.8-stable. A number
of session ticket fixes are in 0.9.8-stable but not in 0.9.8l.

Steve.
--
Dr Stephen N. Henson. Senior Technical/Cryptography Advisor,
Open Source Software Institute: www.oss-institute.org
OpenSSL Core team: www.openssl.org