ssl_read() hangs after wakeup from sleep in OSX 10.5.8

[Parimal Das]

Hello people

I am getting a hang by doing the following
IF i put my OS X laptop in sleep, when ssl_getbuffer() is getting performed,
On waking up, the thread hangs.

The call trace is as follows
                  562 mail_fetch_message
                    562 imap_msgdata
                      562 imap_send
                        562 imap_sout
                          562 imap_reply
                            562 imap_parse_unsolicited
                              562 imap_parse_string             

               
                                562 ssl_getbuffer

                                  562 ssl_getdata
                                    562 ssl3_read_internal
                                      562 ssl3_read_bytes
                                        562 ssl3_read_n
                                          562 BIO_read

                                            562 read$UNIX2003
                                              562 read$UNIX2003

I am using OpenSSL 0.9.7 included in IMAP-2009 (http://www.panda.com/imap/) on OS X 10.5.8 (Leopard)

Anyone has some idea about this problem??
Any suggestions??

Advance Thanks
-Parimal Das


--
--
Warm Regards,

Parimal Das
Webyog Softworks

[Darryl Miles]

"after wakeup from sleep" ?   What do you mean ?


Do you mean you used an API like poll() or select() to put the thread to
sleep and then it wakes up and the socket readability was indicated ?
If so please post an outline of the code your sleep/wake mechanism you
are using.

Do you mean your operating system put the host computer (possibly a
laptop) to sleep, as in low power mode ?  Not sure I can help you on
this one, maybe someone else can.


Darryl
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Parimal Das]

Its the second case Darry,
Here the 'sleep' is Operating System Sleep mode induced by closing the lid of laptop.
After opening the laptop, when the system wakes up,
My application is always hanging at the same place.

        562 ssl_getbuffer

               562 ssl_getdata
                      562 ssl3_read_internal
                             562 ssl3_read_bytes
                                   562 ssl3_read_n
                                          562 BIO_read
                                                 562 read$UNIX2003


Thanks Again

-Parimal Das

On Wed, Oct 28, 2009 at 12:32 AM, Darryl Miles <darryl-mailinglists@...> wrote:

"after wakeup from sleep" ?   What do you mean ?


Do you mean you used an API like poll() or select() to put the thread to sleep and then it wakes up and the socket readability was indicated ? If so please post an outline of the code your sleep/wake mechanism you are using.

Do you mean your operating system put the host computer (possibly a laptop) to sleep, as in low power mode ?  Not sure I can help you on this one, maybe someone else can.


Darryl

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Darryl Miles]

Parimal Das wrote:
>
> Its the second case Darry,
> Here the 'sleep' is Operating System Sleep mode induced by closing the
> lid of laptop.
> After opening the laptop, when the system wakes up,
> My application is always hanging at the same place.

It is possible there is something specific OSX does in relation to
restarting the application.


Knowing the call function trace is a start but you need to rebuild the
application with full debugging and insert some logging.  What you want
to be able to do to this mailing list is describe the sequence of event
that take place.

So you need to describe the entire situation:
  * From the context of the kernel system calls the application is
making, including return codes and data.  On linux strace is a command
line tool that can do this.
  * Dig into the code and describe the sleep/wake mechanism used.


You then need to work out from the code base of the application which
party is taking responsibility for what matters relating to socket IO.
I.e. the application or the OpenSSL library.  Its possible to delegate
most of the work to the OpenSSL library but its also possible to
override and provide custom handling inside the application.


Can you write a simple test case, this is the smallest possible program
that can reliably demonstrate the problem.


If you are not yourself a developer (software engineer) then it is
unlikely you will be able to make further progress on this matter.  It
is not even clear at this time the problem is with the OpenSSL library
so the onus is on your to provide test cases to prove that.  In doing
that then it maybe possible for non-OSX users to assist with the problem
and provide patches for you to test.


Darryl
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[JoelKatz]

Parimal Das wrote:

> Its the second case Darry,
> Here the 'sleep' is Operating System Sleep mode induced by closing the lid
of laptop.
> After opening the laptop, when the system wakes up,
> My application is always hanging at the same place.

Bug is in your code. It is doing what you asked it do -- waiting up to
forever for data from the other side. The other side will never send
anything because it has long forgotten about the connection. Your
application will never send anything because it is blocked in a read
function. TCP and UDP will do the same thing if you call 'read' or 'recv'
and block for data that will never arrive.

DS



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Parimal Das]

Hello,

Here is my test code. I am downloading a file with https connection.
This is compiled as  $g++ -lssl -lcrypto sslShow.cpp.  on OS X 10.5.8
Using default OS X libs (libcrypto 0.9.7  and libssl 0.9.7)

When it has downloaded some 2MB data, I closed my laptop lid (OSX induced sleep)
After 5 minutes when i open my laptop, the process hangs at the same place as before.

I have reproduced the same with latest 0.9.8k version also.

Please suggest.
1. What i should include in this code to correct this hang?
2. How to set read/write timeouts?

Thanks a lot guys.
(the Test Code & Call Trace is as follows )

============CALL TRACE=======
Call graph:
    811 Thread_2507
      811 start
        811 main
          811 BIO_read
            811 ssl_read
              811 ssl3_read_internal
                811 ssl3_read_bytes
                  811 ssl3_read_n
                    811 BIO_read
                      811 read$UNIX2003
                        811 read$UNIX2003

============================MY TEST CODE =====================
#include <openssl/ssl.h>
#include <openssl/err.h>
#include <openssl/bio.h>
#include <iostream>
#define MAX_PACKET_SIZE 10000

int main() {
   
    BIO * bio;       
    SSL * ssl;       
    SSL_CTX * ctx;
   
    /* Initializing OpenSSL */   
    SSL_load_error_strings();     

  
    ERR_load_BIO_strings();       
    OpenSSL_add_all_algorithms();
    SSL_library_init(); //mandatory and missing from some examples       

    ctx = SSL_CTX_new(SSLv23_client_method());
   
    if (ctx == NULL) {               
        std::cout << "Ctx is null" << std::endl;               
        ERR_print_errors_fp(stderr);       
    }
   
    //using a store from examples       
    if(! SSL_CTX_load_verify_locations(ctx, "/Users/pd/workspace/openssl/TrustStore.pem", NULL))       
    {            /* Handle failed load here */               
        std::cout << "Faild load verify locations" << std::endl;   
    }

    bio = BIO_new_ssl_connect(ctx);       
    BIO_get_ssl(bio, & ssl);       
    SSL_set_mode(ssl, SSL_MODE_AUTO_RETRY);

    //replace with your own test server   
    BIO_set_conn_hostname(bio, "www.myDomain.com:https");
   
    if(BIO_do_connect(bio) <= 0) {               
        std::cout<<"Failed connection" << std::endl;                               
    } else {               
        std::cout<<"Connected" << std::endl;       
    }

if(SSL_get_verify_result(ssl) != X509_V_OK)       
    {  
        /* Handle the failed verification */                   
        std::cout << "Failed get verify result " << std::endl;
   
        fprintf(stderr, "Certificate verification error: %i\n", SSL_get_verify_result(ssl));       
        //do not exit here (but some more verification would not hurt) because if you are using a self-signed certificate you will receive 18       
        //18 X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT which is not an error        }

        char *write_buf = "GET /downloads/goodApp.exe / HTTP/1.0\n\n";
       
        if(BIO_write(bio, write_buf, strlen(write_buf)) <= 0){               
            if(! BIO_should_retry(bio)){                                       
                /* Handle failed write here */                               
            }                       
            /* Do something to handle the retry */                           
            std::cout << "Failed write" << std::endl;        
        }
   
        char buf[MAX_PACKET_SIZE];   
        int p;       
        char r[1024];       

        FILE *fp;
        fp = fopen("something.abc", "a+");
       
        for(;;){              
            p = BIO_read(bio, r, 1023);              
            if(p <= 0) break;              
            r[p] = 0;                          
            fprintf(fp, "%s", r);                  
        }
       
        fclose(fp);

        std::cout << "Done reading" << std::endl;
   
        /* To free it from memory, use this line */   
        ERR_print_errors_fp(stderr);       
        BIO_free_all(bio);   
    }
   
    return 0;
}

On Thu, Oct 29, 2009 at 4:57 PM, David Schwartz <davids@...> wrote:

Parimal Das wrote:

> Its the second case Darry,
> Here the 'sleep' is Operating System Sleep mode induced by closing the lid
of laptop.
> After opening the laptop, when the system wakes up,
> My application is always hanging at the same place.

Bug is in your code. It is doing what you asked it do -- waiting up to
forever for data from the other side. The other side will never send
anything because it has long forgotten about the connection. Your
application will never send anything because it is blocked in a read
function. TCP and UDP will do the same thing if you call 'read' or 'recv'
and block for data that will never arrive.

DS

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

--
--
Warm Regards,
Parimal Das

[JoelKatz]

Parimal Das wrote:

> Please suggest.
> 1. What i should include in this code to correct this hang?

It depends on what your code should do in this case. Do you want to wait a
limited amount of time for the other side to reply? Or do you want to wait
possibly forever? Your current code specifically elects to wait possibly
forever, but then you complain when it waits possibly forever. If the user
should interrupt it, then this is sensible. If not, then it's not.

> 2. How to set read/write timeouts?

Well, 'alarm' is the easiest way in toy code like this. You can reset the
'alarm' every time you receive a certain amount of data. It depends how you
want to handle cases like where the other side 'dribbles' data at you.
Decide how long is reasonable to wait, and code that.

But this is another reason that blocking socket operations are difficult to
use. In what realistic situation do you want to wait forever? So you make a
blocking operation and then, surprise, have to work around the annoying fact
that it blocks.

DS



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...

[Graham-9]

google: TCP OPTION KEEPALIVE
http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/

You would be best with an application level timeout,
which would send an application enquiry (heartbeat)
from the laptop to the server.

Waking up from sleeping, the application would send the next heartbeat.
(any data has the same effect, the problem was that the laptop was silent).

That would get an immediate response, from the remote OS, that the TCP connection was dismantled ages ago, whilst the laptop was comatose.

That would appear to the SSL BIO as select_says_fd_is_readable
and when it reads from the socket-fd, an error eof.
Then SSL knows and lets you know too.

If you dont want to change the application protocol,
you can have the OS do exactly that for that socket.
No DATA.data will appear on the application side of the socket-fd,
just hidden TCP control messages, on the network side.
That will also keep any NAT router happy.

In an ideal world, the wake-from-sleep should do that for each TCP
connection. Maybe it leaves it for each app to figure it out on its own.

Graham

2009/10/29 Parimal Das <parimal.das@...>

When it has downloaded some 2MB data, I closed my laptop lid (OSX induced sleep)
After 5 minutes when i open my laptop, the process hangs at the same place as before.

[Parimal Das]

Hi

The c-client library/API does its own socket I/O for non-SSL sessions, but in SSL the socket I/O is delegated to OpenSSL.

When c-client does its own socket I/O, it sets a timeout (normally 15 seconds) on a select() call prior to doing any read() or write() calls. Thus, c-client never does a read() or write() that would block.

If the select() timeout hits, there is a callback to the application to enquire whether the application wants to continue waiting.  If the application chooses to continue waiting, the select() is restarted with an updated timeout.  Otherwise, the socket is closed and the application receives a corresponding I/O error.

The net effect is that a non-SSL I/O can wait forever as long as the application consents.  c-client does not unilaterally disconnect.

My problem is that this doesn't happen with SSL sessions because the socket I/O has been delegated to OpenSSL.  There is no obvious way to instruct OpenSSL to timeout its socket I/O, much less do the mechanism described above.

So, the questions are:
 (1) Is there a way to set a timeout for OpenSSL's socket I/O (given that it has been delegated to OpenSSL)?  If so, how?
 (2) If the answer is "yes", is there a way to do the "query" type timeout described above?  If so, how?
 (3) If the answer to either (1) or (2) is "no", then how would we go about altering the OpenSSL consumer (which, in this case, is c-client) so that OpenSSL uses the consumer's socket I/O code instead of OpenSSL's socket I/O code?  I'm hoping that you will tell me that there's some callback function pointer that can be passed.

-Parimal

--
--
Warm Regards,

Parimal Das

[Parimal Das]

Sorry, I forgot to mention that c-client library/API is part of IMAP-2009 library (http://www.panda.com/imap/)
I am using IMAP library, which in turn using OpenSSL.

I am re-posting my last post here.

The IMAP(2009) c-client library/API does its own socket I/O for non-SSL sessions, but in SSL the socket I/O is delegated to OpenSSL.
When c-client does its own socket I/O, it sets a timeout (normally 15 seconds) on a select() call prior to doing any read() or write() calls. Thus, c-client never does a read() or write() that would block.

If the select() timeout hits, there is a callback to the application to enquire whether the application wants to continue waiting.  If the application chooses to continue waiting, the select() is restarted with an updated timeout.  Otherwise, the socket is closed and the application receives a corresponding I/O error.

The net effect is that a non-SSL I/O can wait forever as long as the application consents.  c-client does not unilaterally disconnect.

My problem is that this doesn't happen with SSL sessions because the socket I/O has been delegated to OpenSSL.  There is no obvious way to instruct OpenSSL to timeout its socket I/O, much less do the mechanism described above.

So, the questions are:
 (1) Is there a way to set a timeout for OpenSSL's socket I/O (given that it has been delegated to OpenSSL)?  If so, how?
 (2) If the answer is "yes", is there a way to do the "query" type timeout described above?  If so, how?
 (3) If the answer to either (1) or (2) is "no", then how would we go about altering the OpenSSL consumer (which, in this case, is c-client) so that OpenSSL uses the consumer's socket I/O code instead of OpenSSL's socket I/O code?  I'm hoping that you will tell me that there's some callback function pointer that can be passed.

I again apologies for reposting.

Parimal Das

[JoelKatz]

Parimal Das wrote:

> The IMAP(2009) c-client library/API does its own socket I/O for
> non-SSL sessions, but in SSL the socket I/O is delegated to OpenSSL.
> When c-client does its own socket I/O, it sets a timeout (normally
> 15 seconds) on a select() call prior to doing any read() or write()
> calls. Thus, c-client never does a read() or write() that would block.

Actually, they still can block. Suppose 'select' gets a 'write' hit because
X bytes can be written without blocking, but then you call 'write' to send
X+1 bytes. Boom, you block. So unless the library actually uses non-blocking
socket operations, it can always block due to various types of ambush.

> If the select() timeout hits, there is a callback to the application
> to enquire whether the application wants to continue waiting.
> If the application chooses to continue waiting, the select()
> is restarted with an updated timeout.  Otherwise, the socket is
> closed and the application receives a corresponding I/O error.

> The net effect is that a non-SSL I/O can wait forever as long as
> the application consents.  c-client does not unilaterally disconnect.

You can code the same thing with OpenSSL. Use non-blocking sockets and
non-blocking BIOs. Really, the only difference is that with TCP, you can
'select' before or after you try to 'read' or 'write'. (You can call 'read'
and then only call 'select' if the read would block, or you can call
'select' first and only call 'read' if 'select' tells you to; same for
write.) With SSL, you must try to SSL_read or SSL_write *before* you
'select'. Otherwise, you may wind up waiting for something that has already
happened. (And how would you know whether to 'select' for read or write
anyway? You may need to write to the socket to read decrypted data, and you
may need to read from the socket to send encrypted data depending on  what's
going on with the SSL engine.)

> My problem is that this doesn't happen with SSL sessions because
> the socket I/O has been delegated to OpenSSL.  There is no obvious way
> to instruct OpenSSL to timeout its socket I/O, much less do the
> mechanism described above.

Set the sockets and BIOs non-blocking. If SSL_read/SSL_write tells you they
will block, then 'select' in the appropriate direction. If 'select' times
out, then do your callback thing.

> So, the questions are:
> (1) Is there a way to set a timeout for OpenSSL's socket I/O
> (given that it has been delegated to OpenSSL)?  If so, how?

No, but you don't need to.

> (3) If the answer to either (1) or (2) is "no", then how would
> we go about altering the OpenSSL consumer (which, in this case,
> is c-client) so that OpenSSL uses the consumer's socket I/O code
> instead of OpenSSL's socket I/O code?  I'm hoping that you will
> tell me that there's some callback function pointer that can be passed.

You can do that, but it's a lot more work. Just call 'select' when OpenSSL
tells you to and callback the application if the 'select' times out.

DS



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@...
Automated List Manager                           majordomo@...