Is full-duplex socket use possible with OpenSSL?

[Jason Pettiss-3]

I have a server which reads/writes a socket independently; that is to say, at the same time (not a request-response model).  I note in the FAQ it says I must not allow multiple threads to use an SSL connection, so clearly if my sockets are blocking I cannot support full-duplex traffic (because I cannot call SSL_write while an SSL_read is blocking, for instance).

It's important that I be able to read a packet as soon as one is available, and at the same time, send a packet as soon as I have one to send... I would not want to delay the send until a pending read were complete for example.

I'm uncertain whether placing the socket into non-blocking mode will actually help here: if an SSL_read returns telling me I need to call it again later, is it alright to go ahead and start a new SSL_write operation?

Also I'm wondering if the limitation of not being able to write/read at the same time in blocking mode is easily overcome, for example by preventing re-negotiation (my application is on both ends of the pipe here), or by replacing the read/write BIOs, or by supplying some magical mutex callback function or something.

Thanks for any tips,

Jason Pettiss
jpettiss@...



     
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[JoelKatz]

Jason Pettiss wrote:

> I have a server which reads/writes a socket independently; that is to
> say, at the same time (not a request-response model).  I note in the
> FAQ it says I must not allow multiple threads to use an SSL connection,
> so clearly if my sockets are blocking I cannot support full-duplex
> traffic (because I cannot call SSL_write while an SSL_read is blocking,
> for instance).

> It's important that I be able to read a packet as soon as one is
> available, and at the same time, send a packet as soon as I have one to
> send... I would not want to delay the send until a pending read were
> complete for example.
 
> I'm uncertain whether placing the socket into non-blocking mode will
> actually help here: if an SSL_read returns telling me I need to call it
> again later, is it alright to go ahead and start a new SSL_write
> operation?

That's not what SSL_read will tell you. SSL_read will tell you that it
cannot make further forward progress until something happens. You can call
SSL_read at any later time you wish. The report that it cannot make forward
progress is just a hint.

The only quirks are with SSL_write. You must set
SSL_ACCEPT_MOVING_WRITE_BUFFER (unless you are sure your write buffer will
never move). And you must present a consistent data stream to SSL_write. (So
you can't try to send 'FOO', get 1 back, and later try to send anything that
doesn't start with 'OO'.)
 
> Also I'm wondering if the limitation of not being able to write/read at
> the same time in blocking mode is easily overcome, for example by
> preventing re-negotiation (my application is on both ends of the pipe
> here), or by replacing the read/write BIOs, or by supplying some
> magical mutex callback function or something.

Blocking mode is way more trouble than it's worth. I would just ditch it,
and all the problems it causes, once and for all. Then never look back.

DS



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Darryl Miles]

David Schwartz wrote:

> Jason Pettiss wrote:
>
>> I have a server which reads/writes a socket independently; that is to
>> say, at the same time (not a request-response model).  I note in the
>> FAQ it says I must not allow multiple threads to use an SSL connection,
>> so clearly if my sockets are blocking I cannot support full-duplex
>> traffic (because I cannot call SSL_write while an SSL_read is blocking,
>> for instance).
>
>> It's important that I be able to read a packet as soon as one is
>> available, and at the same time, send a packet as soon as I have one to
>> send... I would not want to delay the send until a pending read were
>> complete for example.
>  
>> I'm uncertain whether placing the socket into non-blocking mode will
>> actually help here: if an SSL_read returns telling me I need to call it
>> again later, is it alright to go ahead and start a new SSL_write
>> operation?
>
> That's not what SSL_read will tell you. SSL_read will tell you that it
> cannot make further forward progress until something happens. You can call
> SSL_read at any later time you wish. The report that it cannot make forward
> progress is just a hint.
>
> The only quirks are with SSL_write. You must set
> SSL_ACCEPT_MOVING_WRITE_BUFFER (unless you are sure your write buffer will
> never move). And you must present a consistent data stream to SSL_write. (So
> you can't try to send 'FOO', get 1 back, and later try to send anything that
> doesn't start with 'OO'.)

But this flag (while documented to the contrary) does nothing inside
libssl.  So yes the documentation says you should set it, prove to me
that OpenSSL behaves in a different way because you set it.

A hint to DS: grep the source tree of OpenSSL and follow all the
code-paths determined by this flag to their conclusion.


>> Also I'm wondering if the limitation of not being able to write/read at
>> the same time in blocking mode is easily overcome, for example by
>> preventing re-negotiation (my application is on both ends of the pipe
>> here), or by replacing the read/write BIOs, or by supplying some
>> magical mutex callback function or something.
>
> Blocking mode is way more trouble than it's worth. I would just ditch it,
> and all the problems it causes, once and for all. Then never look back.


My own thoughts on the Original Posters comments are:
  * The OpenSSL API does indicate the threading issues with your
proposed usage.  It is true to say that if you serialize the usage of
any 'SSL *' instance with respect to itself then you will never
experience a usage/threading problem.  This is to say that two (or more)
threads can each independently operate the OpenSSL API with _DIFFERENT_
'SSL *' instances at the same time (without regard for one another).

  * Now the next question you might want to ask, "is it allowed for
exactly two threads to operate specifically the SSL_read() and
SSL_write() on the _SAME_ 'SSL *' instance at the same time ?"  My
understanding would be that the answer is NO.   This is a limitation in
the OpenSSL library, since some of the shared parts of 'SSL *' have no
protection and the SSL_read() and SSL_write() code-paths have not been
audited/reworked to minimize the contention/data-race issues.



However this does not exclude the use of OpenSSL for full-duplex operations.

You need to separate your 3 concerns:

  * The desire to process incoming data as soon as possible.
  * The desire to send outgoing data as soon as possible.
  * The desire to have your application go to sleep when neither of the
above is possible and the desire for your operating system to wake up
your application as soon as some condition changes which _MIGHT_ make it
possible for one of the first 2 points (read/write) to take place now.


The 'read' case)
  Well this is already covered in both blocking and non-blocking usage,
your application gets back control (to process data) as soon as data can
be processed.

The 'write' case)
  Well this is already covered in both blocking and non-blocking usage,
your application gets back control (to create more data to send) as soon
as the layer below OpenSSL (usually the OS kernel buffering) has stored.

The 'sleep/wakeup' mechanism)
  Well this is clearly an issue of blocking verses non-blocking.  There
is a clear case that you _MUST_ use blocking IO here (this is despite Mr
Schwartz's comments otherwise).   The reason you must use non-blocking
is that in order to satisfy concerns 1 and 2 you can-not possibly let
the operating system block your application from having control of the
'SSL *' because (if you remember from the comment 2 I made right at the
start) the OpenSSL API does not let you operate SSL_read() and
SSL_write() on the _SAME_ 'SSL *' instance at the same time.  So if some
  other thread is stuck and asleep in the middle of using 'SSL *' then
it is unsafe for you to use it from another (unblocked) thread.
  So to me there is no clear way to use blocking IO once all the facts
are considered with your intended usage and your design criteria.



The only other comment I can make is that both the SSL_read() and
SSL_write() calls have a soft-error return for when no further work
(progress) can be made.  It is at this point you perform your 'sleep'
function and indicate to the OS which events you want that sleep to be
woken by.

This is based on your applications intent, usually an application is
always ready to read in more data (but internal buffering and memory
exhaustion considerations should be made), so it usually indicates to
the OS to wake me up if more data is available to read.

Your application then also has to evaluate its intent to send data, you
don't always have something more to send.  If you do then you need to
indicate to the OS to wake me up if I can push more data down into the
kernel buffer.

You then call your OS sleep function with the appropriate wakeup events
(and possible maximum timeout).

You can then keep looping around this basic IO sleep/wake cycle.




Darryl
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[JoelKatz]

Darryl Miles wrote:

> But this flag (while documented to the contrary) does nothing inside
> libssl.  So yes the documentation says you should set it, prove to me
> that OpenSSL behaves in a different way because you set it.

One of the biggest downsides of open source software is that encourages
people to code to what something happens to do rather than what it's
guaranteed to do.
 
> A hint to DS: grep the source tree of OpenSSL and follow all the
> code-paths determined by this flag to their conclusion.

Software development doesn't work that way. That's how you produce code that
suddenly fails mysteriously when you upgrade an unrelated component. The
first rule of software development is "thou shall not assume that something
that happens a particular way is guaranteed to do so, especially when the
documentation specifically warns that it is not".

> Now the next question you might want to ask, "is it allowed for
> exactly two threads to operate specifically the SSL_read() and
> SSL_write() on the _SAME_ 'SSL *' instance at the same time ?"  My
> understanding would be that the answer is NO.   This is a limitation in
> the OpenSSL library, since some of the shared parts of 'SSL *' have no
> protection and the SSL_read() and SSL_write() code-paths have not been
> audited/reworked to minimize the contention/data-race issues.

This is how everything else works, it's odd to say it's somehow a limitation
of OpenSSL that it works the same way everything else works. Try to read to
a string in one thread while you write to it from another. The general rule
of thread synchronization is that it is your responsibility to serialize
access to the same object from concurrent threads and the library's job to
synchronize accesses to distinct objects. OpenSSL follows this general rule.

Kernel objects are the exception, only because we cannot allow a program
(broken or valid) to screw up kernel objects. So the kernel has no choice
but to "overserialize".

> Your application then also has to evaluate its intent to send data, you
> don't always have something more to send.  If you do then you need to
> indicate to the OS to wake me up if I can push more data down into the
> kernel buffer.

No, that is not how OpenSSL works. When you want to send data, you simply
call SSL_write. You only check if I/O is possible if OpenSSL specifically
tells you to. (OpenSSL may need to do something other than write to send the
data, for example, it may need to read renegotiation data.)

The other gotcha is that if you use separate read and write threads, you
*must* remember that an SSL connection only has one state. You cannot
independently maintain your own state in each thread, or you can deadlock.
This is a major cause of SSL deadlocks in "two thread" applications that run
their threads independently.

Here's the nightmare scenario:

1) You are in a point in the protocol where the other side will not send
anything unless we send something first. However, we try to read just in
case it sends something.

2) You call SSL_write from your write thread trying to send the data that
will keep the application protocol going, but a renegotiation is in progress
and no data has been received yet. You get WANT_READ.

(At this point, the SSL connection's one and only status is "want read to
send".)

2) The renegotiation data is received, but no application data is received.

3) You call SSL_read from your read thread (either just to try it, or
because you get a 'select' hit from the renegotiation data being received,
it doesn't matter. The OpenSSL library reads the renegotiation data, but no
application data is available. You get WANT_READ, since application data
needs to be received to make forward progress.

(At this point, the SSL connection's one and only status is "want read to
receive". Note that the read thread's actions *invalidate* the state the
write thread thinks it's in.)

4) Your write thread, having no idea that the read thread received a
different status, stupidly thinks it cannot make forward progress based on
the state it got from step 1 (since that's the last thing *it* did).
However, it *can* make forward progress (because another thread changed the
SSL state).

5) Now the other end is waiting for you to send data, and you are waiting to
receive the renegotiation data you already received.

You see, in step 4, the write thread *must* know that the read thread
changed the SSL connection's status. Otherwise you deadlock.

DS



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Darryl Miles]

David Schwartz wrote:
> Darryl Miles wrote:
>
>> But this flag (while documented to the contrary) does nothing inside
>> libssl.  So yes the documentation says you should set it, prove to me
>> that OpenSSL behaves in a different way because you set it.
>
> One of the biggest downsides of open source software is that encourages
> people to code to what something happens to do rather than what it's
> guaranteed to do.

Can I please see your "working" (i.e. white paper) on your conclusion
(or that of someone elses conclusion you are merely relaying here); that
this issue is more dominant when involving "open source" ?  My gut says
it doesn't agree with you on that statement.

Sure, such an issue exists but just because things are "open source"
doesn't increase it.    Lets call it "code by observation".



>> A hint to DS: grep the source tree of OpenSSL and follow all the
>> code-paths determined by this flag to their conclusion.
>
> Software development doesn't work that way. That's how you produce code that
> suddenly fails mysteriously when you upgrade an unrelated component. The
> first rule of software development is "thou shall not assume that something
> that happens a particular way is guaranteed to do so, especially when the
> documentation specifically warns that it is not".

But there is no "master grand plan for the future" on implementing this
point.  At best there was once was but that plan was then found
unnecessary, or was abandoned.

So this is a call to all active developers on OpenSSL what exactly is
your plan for the future with SSL_ACCEPT_MOVING_WRITE_BUFFER.  Can the
Open Source community please have an online document outlining the idea,
the concept and the timescales of your intentions.

Lets give a month to come up with such a plan (or at least pipe up that
more time is needed to produce a plan) before this relic should be
earmarked to removal.  Especially in the shadow of OpenSSL version 1.0


The existing documentation isn't very clear, it doesn't sufficiently
cover what this flags means:
  * by citing a good example and a bad example with explanation as to
which rule(s) is/are broken
  * a detailed statement of rules (when to use and when not to use)
  * a detailed explanation of scope (the things this flag can not fix
but users might think it fixes)

Anybody wishing to write up such documentation I can assist which some
unclear situations which I do not think the existing documentation
adequately covers.

But first lets hear the future plan to code something actually needing
it, otherwise I would like to see my
SSL_ACCEPT_MANNED_SPACE_FLIGHT_TO_PLUTO added to OpenSSL please.


Since there is no one in the OpenSSL developer community which is
standing up for this flag (from a specification and coding point of view
with an intention to finish the work relating to it).  It should be
removed to make users life easier.  This does not mean such a flag can
never go in, but the merits of it can be discussed at a later date (once
a body of code is available to require it).


Darryl

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Darryl Miles]

David Schwartz wrote:

> Darryl Miles wrote:
>
> This is how everything else works, it's odd to say it's somehow a limitation
> of OpenSSL that it works the same way everything else works. Try to read to
> a string in one thread while you write to it from another. The general rule
> of thread synchronization is that it is your responsibility to serialize
> access to the same object from concurrent threads and the library's job to
> synchronize accesses to distinct objects. OpenSSL follows this general rule.
>
> Kernel objects are the exception, only because we cannot allow a program
> (broken or valid) to screw up kernel objects. So the kernel has no choice
> but to "overserialize".

FYI modern kernel's do not need to serialize (let alone "overserialize",
whatever that means, is that a computer science term?).  I.e. the read()
write() code paths for the same file-descriptor/handle can be called
simultaneously from two or more threads without any harm to the kernel.
  Sure fine grained serialization of the workings inside the kernel
might take place, but thats is implementation detail, irrelevant to the
contract the kernel API provides its users.

This is merely a result of prudent multi-threaded coding inside the
kernel presumably as a result of a "performance centric usage case" that
customers/users want.

I advocate that some users would find it useful to be able to invoke
SSL_read() and SSL_write() from exactly two threads on the same 'SSL *'
simultaneously.  There is merit in this and as things stands OpenSSL
does not allow it due to a design choice (aka "design limitation").

I do not advocate that expanding the above scope to allowing more than
two threads or two threads both SSL_write() or both SSL_read() would be
useful.  I see no merit in that (one factor in this is that the nature
of SSL/TLS is that a sequence of packets are serialized on the wire and
that checksums/state from the last packet influence the encoding of the
next, this is part of the "tamper proof security" provided by SSL/TLS,
so there is no case to parallelize).

There is no reason why OpenSSL can not allow two threaded operation if
it were designed differently.  So I stand by my usage of the word
"limitation".



>> Your application then also has to evaluate its intent to send data, you
>> don't always have something more to send.  If you do then you need to
>> indicate to the OS to wake me up if I can push more data down into the
>> kernel buffer.
>
> No, that is not how OpenSSL works.

Who was talking about OpenSSL here ?  "Your application ...." is the
clue here, see if you can get a clue, try reading it again in context
was the topic being discussed.


Darryl
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Victor Duchovni]

On Fri, Oct 23, 2009 at 03:47:51PM +0100, Darryl Miles wrote:

> I advocate that some users would find it useful to be able to invoke
> SSL_read() and SSL_write() from exactly two threads on the same 'SSL *'
> simultaneously.  There is merit in this and as things stands OpenSSL does
> not allow it due to a design choice (aka "design limitation").

You are mistaken. There are no message boundaries, and multiple threads
reading and writing the same SSL session would get random fragments of
the remote data on read, and emit random fragments of data on write.

There is no sensible use-case for concurrent multiple thread access
to an SSL object. All access must be serialized to ensure remotely
reasonable semantics.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Jason Pettiss-3]

> > Now the next question you might want to ask, "is it
> allowed for
> > exactly two threads to operate specifically the
> SSL_read() and
> > SSL_write() on the _SAME_ 'SSL *' instance at the same
> time ?"  My
> > understanding would be that the answer is
> NO.   This is a limitation in
> > the OpenSSL library, since some of the shared parts of
> 'SSL *' have no
> > protection and the SSL_read() and SSL_write()
> code-paths have not been
> > audited/reworked to minimize the contention/data-race
> issues.
>
> This is how everything else works, it's odd to say it's
> somehow a limitation
> of OpenSSL that it works the same way everything else
> works. Try to read to
> a string in one thread while you write to it from another.

I think we've lost the point: if I write to a socket from more than one thread at a time, clearly I've messed up.  Even if the operating system doesn't complain, my stream is nonsense (unless I only ever write a single byte at a time).

However, it's clearly alright to read a socket from one thread while writing a socket from another: indeed, this is the purpose of a socket.  That OpenSSL doesn't allow this usage seems like a limitation of the library.  (Although maybe it's actually of the TLS protocol itself...?)

> The other gotcha is that if you use separate read and write
> threads, you
> *must* remember that an SSL connection only has one state.
> You cannot
> independently maintain your own state in each thread, or
> you can deadlock.

> You see, in step 4, the write thread *must* know that the
> read thread
> changed the SSL connection's status. Otherwise you
> deadlock.

Your explanation here is excellent.  If I understand it correctly it's not really the problem of multiple access to a shared buffer which would understandably cause corruption, it's that there's a single flag which indicates the 'direction' if you will of the SSL structure itself:

#define SSL_ERROR_WANT_READ 2
#define SSL_ERROR_WANT_WRITE 3

And since these are defined in such a way that you can't have both READ|WRITE at the same time, if I don't somehow externally remember this information and share it between my threads I could run into trouble.

Ok so to summarize you Dave and Darryl: Blocking sockets + OpenSSL will only work for a request-response model without redesigning the library itself, because external synchronization deadlocks (for obvious reasons) and no synchronization deadlocks because the library/application no longer know what needs to happen to make forward progress.

Forgive me if I misunderstand either of you, but it sounds like if I use non-blocking sockets, I'll be able to use but a single thread to both push & pull independent streams of data, and I don't have to wait for an interrupted write to complete in order to begin a new read, or vice versa, so long as I remember the actual WANT_* state of each stream.

I'd been warned away from non-blocking socket use in OpenSSL from the varies searches I did across this mailing list, but honestly I'd actually prefer to use them.

To make sure I'm clear on this: if I myself don't have any data to read and an SSL_write returns WANT_READ, that doesn't mean I myself need to call SSL_read-- what it means is I need to wait until the socket is readable, and then call SSL_write again (with the same args of course).

It'd be awesome if there was a 'canonical' example for this... I've read through several different applications using OpenSSL (stunnel, Ice, curl) but they're so heavily hacked up to overcome various system limitations / implementation needs that it's not entirely obvious what's going on.

Guess I'll go make that example now.  :)

Thanks much,

--jason



     
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Victor Duchovni]

On Fri, Oct 23, 2009 at 08:50:38AM -0700, Jason Pettiss wrote:

> However, it's clearly alright to read a socket from one thread while
> writing a socket from another: indeed, this is the purpose of a socket.
> That OpenSSL doesn't allow this usage seems like a limitation of the
> library.  (Although maybe it's actually of the TLS protocol itself...?)

SSL is a state-machine, not a pipe. Reading data may require writes, and
writing data may require reads (e.g. when re-negotiating). If you want
to write and read as data arrives in either direction, don't block, and
enter the state machine to move data in either direction as data arrives.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Jason Pettiss-3]

> > I advocate that some users would find it useful to be
> able to invoke
> > SSL_read() and SSL_write() from exactly two threads on
> the same 'SSL *'
> > simultaneously.  There is merit in this and as
> things stands OpenSSL does
> > not allow it due to a design choice (aka "design
> limitation").
>
> You are mistaken. There are no message boundaries, and
> multiple threads
> reading and writing the same SSL session would get random
> fragments of
> the remote data on read, and emit random fragments of data
> on write.
>
> There is no sensible use-case for concurrent multiple
> thread access
> to an SSL object. All access must be serialized to ensure
> remotely
> reasonable semantics.

Alright, here's a simple use case: I have a large file here, you have a large file there.  We'd like to trade them.  We have two independent streams available (one from me to you, one from you to me).  A socket, in other words.

We could take turns sending discrete pieces of each file but that's silly and slow.

Assuming we can load these gigantic files into memory to make the example simpler, we could both do this to write:

char* p = entire_file_buffer;
char* e = p + size_of_file;
while (p!=e) {
  int n = send(sock_fd, p, e-p);
  if (n<0) return ERR;
  p += n;
}

And we both do this to read:

char* p = entire_file_buffer;
char* e = p + size_of_file;
while (p!=e) {
  int n = recv(sock_fd, p, e-p);
  if (n<0) return ERR;
  p += n;
}

It's simple, uses two threads, one socket, and makes the best use of our bandwidth.

So I'm hoping it is your misunderstanding actually, that you thought we were suggesting two different threads should be able to write the same SSL* at the same time, or that two different threads be able to read the same SSL* at the same time, which clearly doesn't make sense for a stream-based protocol.  We weren't suggesting that.

We were suggesting that it would be really, really nice if the example above could have send replaced with SSL_write and recv replaced with SSL_read and it would just work.  :)

--jason




______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Victor Duchovni]

On Fri, Oct 23, 2009 at 09:15:35AM -0700, Jason Pettiss wrote:

> We could take turns sending discrete pieces of each file but that's silly and slow.
>
> Assuming we can load these gigantic files into memory to make the example simpler, we could both do this to write:

It is possible to use non-blocking SSL_read() SSL_write() calls that
are interleaved, but not without a mutex or a separate thread that
owns all SSL I/O that consumes requests to read/write.

It is simpler to use two SSL connections. SSL is a state-machine, not a pipe.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Jason Pettiss-3]

> It is possible to use non-blocking SSL_read() SSL_write()
> calls that
> are interleaved, but not without a mutex or a separate
> thread that
> owns all SSL I/O that consumes requests to read/write.
>
> It is simpler to use two SSL connections. SSL is a
> state-machine, not a pipe.

Awesome the former suggestion fits my needs exactly: I have one thread that's gotta manage N sockets for both read & write and it's pretty agnostic about the data itself: just wants to push it along.  I wasn't sure if it was ok to interleave but the confirmation is very nice to have.

Can I use two SSL connections over a single socket?  That doesn't seem possible.  How are the SSL connections going to synchronize use of that socket?

Two unidirectional sockets is my last resort here... in my experience unidirectional traffic is horrible for latency and without disabling TCP_NODELAY, it kills your throughput (assuming you're passing smallish messages).

--jason



     
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Victor Duchovni]

On Fri, Oct 23, 2009 at 09:34:22AM -0700, Jason Pettiss wrote:

> > It is possible to use non-blocking SSL_read() SSL_write() calls that
> > are interleaved, but not without a mutex or a separate thread that
> > owns all SSL I/O that consumes requests to read/write.
> >
> > It is simpler to use two SSL connections. SSL is a
> > state-machine, not a pipe.

Two SSL connections over two sockets of course. Unless you want to
implement a stream multiplexor between TCP and SSL. Then you could
indeed build two SSL objects one for each logical direction of
data transfer. You can do nifty things with bio_pairs(), but building
multiple streams over TCP is probably too much complexity for what you want.
 
> Two unidirectional sockets is my last resort here... in my
> experience unidirectional traffic is horrible for latency
> and without disabling TCP_NODELAY,
> it kills your throughput (assuming you're passing smallish messages).

If you are proxying an inter-active protocol, you need to do it over a
single socket to avoid Nagle delays (or set TCP_NODELAY, which is fine
if you never send small packets unnecessarily).

If you are moving large files in two directions, just avoid writes that
don't fill the socket buffer.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[JoelKatz]

Darryl Miles wrote:

> > Kernel objects are the exception, only because we cannot allow a
> > program
> > (broken or valid) to screw up kernel objects. So the kernel has no
> > choice
> > but to "overserialize".

> FYI modern kernel's do not need to serialize (let alone
> "overserialize",
> whatever that means, is that a computer science term?).  I.e. the
> read()
> write() code paths for the same file-descriptor/handle can be called
> simultaneously from two or more threads without any harm to the kernel.

The kernel must be designed such that a non-privileged application can do
anything, even things that don't make logical sense, without harm to the
kernel. So the kernel has to handle even cases that make no sense at all,
such as two concurrent multi-byte 'write' operations to the same TCP socket.
It does this by extensive internal synchronization code that would normally
not be required.

Because OpenSSL doesn't have this issue, there is no reason it should have
that type of synchronization. As has already been pointed out in this
thread, it is perfectly fine if OpenSSL crashes if there are two concurrent
SSL_write calls to the same SSL connection. There is no sensible reason to
do that, and OpenSSL has nothing to defend (like the kernel does). So making
this work would be overserialization -- locking just to "permit" what is not
sane anyway.

> Sure fine grained serialization of the workings inside the kernel
> might take place, but thats is implementation detail, irrelevant to the
> contract the kernel API provides its users.

The contract the kernel API provides is that nothing the user does can mess
the kernel up, even if the user does something insane.
 
> This is merely a result of prudent multi-threaded coding inside the
> kernel presumably as a result of a "performance centric usage case"
> that
> customers/users want.

No, it's kernel self-defense. User-space libraries generally do not have
that kind of self defense. Try to read from a string in one thread while you
write to it in another and see what happens.
 
> I advocate that some users would find it useful to be able to invoke
> SSL_read() and SSL_write() from exactly two threads on the same 'SSL *'
> simultaneously.  There is merit in this and as things stands OpenSSL
> does not allow it due to a design choice (aka "design limitation").

Right, but it's due to the fact that OpenSSL is like pretty much every other
thread-safe library. It doesn't permit concurrent access to the same object
from multiple threads unless that's a pure read access that doesn't change
any state. The lack of a useful feature that is atypical of libraries is not
a design flaw or unusual quirk.

> There is no reason why OpenSSL can not allow two threaded operation if
> it were designed differently.  So I stand by my usage of the word
> "limitation".

Fine, it's a limitation of OpenSSL that it's like pretty much every other
thread-safe, user-space library.
 
> >> Your application then also has to evaluate its intent to send data,
> you
> >> don't always have something more to send.  If you do then you need
> to
> >> indicate to the OS to wake me up if I can push more data down into
> the
> >> kernel buffer.

> > No, that is not how OpenSSL works.
 
> Who was talking about OpenSSL here ?  "Your application ...." is the
> clue here, see if you can get a clue, try reading it again in context
> was the topic being discussed.

You were talking about how an application interacts with OpenSSL (look back
two paragraphs from the one you quoted). And that's not how an application
interacts with OpenSSL. You do not go to the OS when you want to do
something, like you would with TCP.

An application that wants to write data to an SSL connection calls SSL_write
whether or not it is possible to send data on the underlying SSL connection.
An application that wants to read data from an SSL connection calls SSL_read
whether or not there's data available to be read on the socket.

As I explained, operating as you describe will cause deadlocks. The data you
are waiting for may have already arrived and been processed by OpenSSL. An
OpenSSL-using application should not try to "look through" the SSL state
machine except when told to look at the socket by OpenSSL (by
WANT_READ/WANT_WRITE indications).

And, to be helpful, I would suggest that the simplest solution for your
application, assuming it doesn't need to handle large numbers of SSL
connections, would be to wrap the SSL connection in a service thread. That
service thread would have its own read/write state machine that tracks the
SSL state machine, issues SSL_read/SSL_write operations, blocks on the
socket when told to do so by OpenSSL, and so on. That way, you can emulate
blocking read/write operations if you want (blocking until the service
thread wakes you).

DS



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Darryl Miles]

Victor Duchovni wrote:
> SSL is a state-machine, not a pipe. Reading data may require writes, and
> writing data may require reads (e.g. when re-negotiating). If you want
> to write and read as data arrives in either direction, don't block, and
> enter the state machine to move data in either direction as data arrives.

"not a pipe" is a little ambiguous.  The generally accepted meaning of a
pipe is a single direction of data flow.

What we are talking about is a bidirectional-pipe (other people just
call this a 'socket' to differentiate it from a "pipe").


I don't interpret Jason's comments as implying that "SSL is a pipe".  At
no point has Jason's problem been about only wanting a single direction
of data flow (without requirement for data to be flowing in the other
direction).  Please read the original post again.


Hey did you know that TCP is a state-machine too.  I bet you did.  Hey
reading data might require writes too, in TCP that is, for example I
can't read any more new application data because the other end keeps
sending me the same data block over and over, so I must write an ACK so
that it sends me some new application data to process.

These matters have absolutely nothing to do with how application threads
of execution are provided an API to do the business.  This is all down
to design rules and implementation.


Darryl
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Darryl Miles]

The issue is down to the OpenSSL API thread-safety rules (which are
dictated to by the internal design of OpenSSL).

I covered those thread-safety rules in a previous posting.



Yes the common application design pattern for full-duplex SSL streams is
to only ever have one thread doing the work on a specific instance of
'SSL *' at any one time.

Given your application design requirements you indicated in your
original posting then in order to achieve all your goals you must use
the kernel socket in non-blocking mode.

The reasons why were explained in a previous reply of mine.



There is no reason why you should be warned away from using OpenSSL with
non-blocking sockets.  But you have to understand multi-threaded
programming is hard and therefore more programmers will have difficulty
previously in simply not understanding the concepts correctly.








Jason Pettiss wrote:
> To make sure I'm clear on this: if I myself don't have any data to read and an SSL_write returns WANT_READ, that doesn't mean I myself need to call SSL_read-- what it means is I need to wait until the socket is readable, and then call SSL_write again (with the same args of course).

Okay this is a new question.  Yes if you call SSL_write() and get back
-1/WANT_READ then yes you do need to call SSL_read() to unstall that
situation.  In-fact SSL_peek() might be better to use if you have no
where to put the application data right now but you want to attempt to
see if the condition can be resolved by the next SSL protocol packet to
be processed.  Obviously if application data exists and is in the way
calling SSL_peek() won't clear the SSL_write() stall.  You must
SSL_read() that data so the OpenSSL library can get to the final part of
the renegotiation handshake packet.

I do not believe the SSL_write() call is allowed to access the
underlying BIO/kernel-socket to read in more data.  I think SSL_write()
is allowed to process any data already read into buffer (from kernel to
OpenSSL library internal buffer) in an attempt to unstall the situation
itself.  But it can't invoke read() on the kernel for it.

Due to this you have to call SSL_read()|SSL_peek() at least once, since
these calls are allowed to access the underlying BIO/kernel-socket to
attempt to read() in more data.


So once you observe an SSL_write() returning -1/WANT_READ you should
immediately attempt to SSL_read()|SSL_peek() and if that also returns
-1/WANT_READ then you can go to sleep and wait for more data to come in
(wait until the socket is readable).

When that data comes in you call SSL_read()|SSL_peek() and if that
doesn't return -1/WANT_READ then should give your SSL_write() another
try.  From memory I think SSL_read()|SSL_peek() return 0 (i.e. no new
application data) at least once to eat up the final part of the
renegotiation handshake process.

But it doesn't hurt to always call SSL_write() after every
SSL_read()|SSL_peek() if you know that you are under this special
condition (that SSL_write() previously returned -1/WANT_READ).  Once
your SSL_write() returns something other than -1/WANT_READ you can clear
this special condition.


Now the same is true in reverse.  The special condition that is
SSL_read() returning -1/WANT_WRITE.  I'm sure you can work out the
details on this.


Darryl
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[JoelKatz]

Darryl Miles wrote:

> I do not believe the SSL_write() call is allowed to access the
> underlying BIO/kernel-socket to read in more data.  I think SSL_write()
> is allowed to process any data already read into buffer (from kernel to
> OpenSSL library internal buffer) in an attempt to unstall the situation
> itself.  But it can't invoke read() on the kernel for it.

If SSL_write has to read from the socket to make forward progress, there is
absolutely no reason it shouldn't just do so. There is no reason it should
compel the application to do it.

My documentation says:

       [T]he return value of SSL_write() will yield SSL_ERROR_WANT_READ or
       SSL_ERROR_WANT_WRITE. As at any time a re-negotiation is possible, a
       call to SSL_write() can also cause read operations! The calling
process
       then must repeat the call after taking appropriate action to satisfy
       the needs of SSL_write(). The action depends on the underlying BIO.
       When using a non-blocking socket, nothing is to be done, but select()
       can be used to check for the required condition. When using a
buffering
       BIO, like a BIO pair, data must be written into or retrieved out of
the
       BIO before being able to continue.

This suggests the exact opposite of what you said. One of these sources is
right and the other is wrong, and it makes a huge difference which!

My understanding, for many years, coincides with this documentation.
However, I can't think of any specific case where this difference would have
affected me, as my coding is extremely defensive and would tolerate either
mechanism without a problem.

DS



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Darryl Miles]

David Schwartz wrote:

> Darryl Miles wrote:
>
>> I do not believe the SSL_write() call is allowed to access the
>> underlying BIO/kernel-socket to read in more data.  I think SSL_write()
>> is allowed to process any data already read into buffer (from kernel to
>> OpenSSL library internal buffer) in an attempt to unstall the situation
>> itself.  But it can't invoke read() on the kernel for it.
>
> If SSL_write has to read from the socket to make forward progress, there is
> absolutely no reason it shouldn't just do so. There is no reason it should
> compel the application to do it.
>
> My documentation says:
>
>        [T]he return value of SSL_write() will yield SSL_ERROR_WANT_READ or
>        SSL_ERROR_WANT_WRITE. As at any time a re-negotiation is possible, a
>        call to SSL_write() can also cause read operations! The calling
> process
>        then must repeat the call after taking appropriate action to satisfy
>        the needs of SSL_write(). The action depends on the underlying BIO.
>        When using a non-blocking socket, nothing is to be done, but select()
>        can be used to check for the required condition. When using a
> buffering
>        BIO, like a BIO pair, data must be written into or retrieved out of
> the
>        BIO before being able to continue.
>
> This suggests the exact opposite of what you said. One of these sources is
> right and the other is wrong, and it makes a huge difference which!
>
> My understanding, for many years, coincides with this documentation.
> However, I can't think of any specific case where this difference would have
> affected me, as my coding is extremely defensive and would tolerate either
> mechanism without a problem.

"One of these sources is right and the other is wrong" ... Yes, no,
maybe...  You maybe correct in the detail here, I am going on my
hazy-memory of experimenting with this situation and the observable
behavior.  But I never wrote up notes on the matter not saw fit to
improve the documentation.

My conclusions on it were that an SSL_write() can cause a packet decode
to complete but only:
  * If the data for the entire packet has already been read() into the
SSL user-space buffer (i.e. no longer in the BIO/kernel).  The
read-ahead optimization makes it possible for this to happen.
  * If there is no application data waiting to be destructively removed
ahead of the re-negotiation packet.  i.e. SSL_read().  Until all
application data has been sunk/removed from OpenSSL it won't decode the
next packet.

My memory on this was that SSL_write() itself won't call on the BIO to
perform a read() but it will attempt to decode the next incoming packet
from the data it may already have, this is in the hope that it turns out
to be the re-negotiation response (in many situations it gets lucky!).
If it decodes the next packet and it turns out to be incoming
application data then SSL_write() is stuffed!  No amount of calling it
again will clear the -1/WANT_READ condition.



The largest part of my previous post was explaining how to handle the
situation generalls and calling SSL_read() and then re-trying
SSL_write() to see if the condition has cleared it the way to deal with
it.  You can not rely on repeatedly calling SSL_write() alone to clear
the problem.  Which was my interpretation of what Jason was asking.



To re-express the same thing another way:

SSL_write() calls can not by-pass the already in-progress inbound
application data (to get at the re-negotiation response packet
immediately).  There is a possibility there is still some application
data waiting to be SSL_read() before the re-negotiation SSL protocol
packet can be seen, decoded and processed.

Imagine the re-negotiation SSL protocol packet is actually still inside
the kernel buffering (waiting for user-space to read() to pull it).  Now
image that there are at least 2 large full-size application data packets
also spanned across the user-space and kernel buffers (ahead of the SSL
re-negotiation packet).

SSL_write() has no where to put the data once it has decoded a large
full-sized application data packet.  Inside OpenSSL there is a rigid
buffering scheme, there is a decode buffer into which the encrypted
packet is read in from BIO/kernel.  There is also a clear-text buffer
into which the resultant application data from a single packet decode
can be stored.  The decode/decyption process only takes place if the
clear-text buffer is empty (i.e. user-space has SSL_read() all the
previous data from it, so it will attempt to pull in more data and
re-fill it).

It is for sure that OpenSSL doesn't have an infinite expandable memory
buffer to keep holding application data to allow SSL_write() to find the
re-negotiation packet.  So it is the worst-case scenario I have in mind
when explaining how to handle the matter in my previous post.



The documentation could certainly be improved no matter what the correct
way to express the situation is.  The docs were written to support the
implementation (not the other way around).


Darryl
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Konstantin Ivanov-3]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi all, 

I am developing a server application which is based on Windows IO Completion ports which basically means that the reads and write to the socket are asynchronous. This also means that I cannot use the SSL_read and SSL_write functions which are tied to the socket fd if I am correct. So I tried to use the BIO_read and BIO_write, but I am having difficulty in using it. Basically what I would like to do is to read the content passed from the client over SSL connection into the buffer, which I can decrypt using, parse, and then issue another read command on the completion port. For send, I would like to write data into an encrypted buffer and then post a send command to the completion port with the pointer to encrypted data. Can someone please comment on how I could implement such functionality as I believe I am suing the BIO_read and BIO_write incorrect (this was the tutorial that I referred to: http://h71000.www7.hp.com/doc/83final/ba554_90007/ch04s03.html) 

Thanks, 

[wolfoftheair]

My understanding is that if SSL_ERROR_WANT_WRITE happened with
SSL_read(), the next SSL_read() would actually call write() to make
the forward progress.

-Kyle H

On Sun, Oct 25, 2009 at 11:03 PM, Darryl Miles
<darryl-mailinglists@...> wrote:

> David Schwartz wrote:
>>
>> Darryl Miles wrote:
>>
>>> I do not believe the SSL_write() call is allowed to access the
>>> underlying BIO/kernel-socket to read in more data.  I think SSL_write()
>>> is allowed to process any data already read into buffer (from kernel to
>>> OpenSSL library internal buffer) in an attempt to unstall the situation
>>> itself.  But it can't invoke read() on the kernel for it.
>>
>> If SSL_write has to read from the socket to make forward progress, there
>> is
>> absolutely no reason it shouldn't just do so. There is no reason it should
>> compel the application to do it.
>>
>> My documentation says:
>>
>>       [T]he return value of SSL_write() will yield SSL_ERROR_WANT_READ or
>>       SSL_ERROR_WANT_WRITE. As at any time a re-negotiation is possible, a
>>       call to SSL_write() can also cause read operations! The calling
>> process
>>       then must repeat the call after taking appropriate action to satisfy
>>       the needs of SSL_write(). The action depends on the underlying BIO.
>>       When using a non-blocking socket, nothing is to be done, but
>> select()
>>       can be used to check for the required condition. When using a
>> buffering
>>       BIO, like a BIO pair, data must be written into or retrieved out of
>> the
>>       BIO before being able to continue.
>>
>> This suggests the exact opposite of what you said. One of these sources is
>> right and the other is wrong, and it makes a huge difference which!
>>
>> My understanding, for many years, coincides with this documentation.
>> However, I can't think of any specific case where this difference would
>> have
>> affected me, as my coding is extremely defensive and would tolerate either
>> mechanism without a problem.
>
> "One of these sources is right and the other is wrong" ... Yes, no, maybe...
>  You maybe correct in the detail here, I am going on my hazy-memory of
> experimenting with this situation and the observable behavior.  But I never
> wrote up notes on the matter not saw fit to improve the documentation.
>
> My conclusions on it were that an SSL_write() can cause a packet decode to
> complete but only:
>  * If the data for the entire packet has already been read() into the SSL
> user-space buffer (i.e. no longer in the BIO/kernel).  The read-ahead
> optimization makes it possible for this to happen.
>  * If there is no application data waiting to be destructively removed ahead
> of the re-negotiation packet.  i.e. SSL_read().  Until all application data
> has been sunk/removed from OpenSSL it won't decode the next packet.
>
> My memory on this was that SSL_write() itself won't call on the BIO to
> perform a read() but it will attempt to decode the next incoming packet from
> the data it may already have, this is in the hope that it turns out to be
> the re-negotiation response (in many situations it gets lucky!). If it
> decodes the next packet and it turns out to be incoming application data
> then SSL_write() is stuffed!  No amount of calling it again will clear the
> -1/WANT_READ condition.
>
>
>
> The largest part of my previous post was explaining how to handle the
> situation generalls and calling SSL_read() and then re-trying SSL_write() to
> see if the condition has cleared it the way to deal with it.  You can not
> rely on repeatedly calling SSL_write() alone to clear the problem.  Which
> was my interpretation of what Jason was asking.
>
>
>
> To re-express the same thing another way:
>
> SSL_write() calls can not by-pass the already in-progress inbound
> application data (to get at the re-negotiation response packet immediately).
>  There is a possibility there is still some application data waiting to be
> SSL_read() before the re-negotiation SSL protocol packet can be seen,
> decoded and processed.
>
> Imagine the re-negotiation SSL protocol packet is actually still inside the
> kernel buffering (waiting for user-space to read() to pull it).  Now image
> that there are at least 2 large full-size application data packets also
> spanned across the user-space and kernel buffers (ahead of the SSL
> re-negotiation packet).
>
> SSL_write() has no where to put the data once it has decoded a large
> full-sized application data packet.  Inside OpenSSL there is a rigid
> buffering scheme, there is a decode buffer into which the encrypted packet
> is read in from BIO/kernel.  There is also a clear-text buffer into which
> the resultant application data from a single packet decode can be stored.
>  The decode/decyption process only takes place if the clear-text buffer is
> empty (i.e. user-space has SSL_read() all the previous data from it, so it
> will attempt to pull in more data and re-fill it).
>
> It is for sure that OpenSSL doesn't have an infinite expandable memory
> buffer to keep holding application data to allow SSL_write() to find the
> re-negotiation packet.  So it is the worst-case scenario I have in mind when
> explaining how to handle the matter in my previous post.
>
>
>
> The documentation could certainly be improved no matter what the correct way
> to express the situation is.  The docs were written to support the
> implementation (not the other way around).
>
>
> Darryl
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    openssl-users@...
> Automated List Manager                           majordomo@...
>

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...