all webserver threads blocking on db.open()

[Claudiu Saftoiu]

Hello all,

I'm using Repoze.BFG, with paster to launch the webserver. This is a 

similar issue to the one I emailed about before titled

"server stops handling requests - nowhere near 100% CPU or Memory

used"

The situation is the same. I used z3c.deadlockdebugger , and what

I notice is that, when the server is blocked, there are about 100

threads running (as opposed to 15 or so when the server has

just started), and all their stack traces look like this:

Thread 140269004887808:

  File "/usr/lib/python2.6/threading.py", line 504, in __bootstrap

    self.__bootstrap_inner()

  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner

    self.run()

  File "/usr/lib/python2.6/threading.py", line 484, in run

    self.__target(*self.__args, **self.__kwargs)

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 878, in worker_thread_callback

    runnable()

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 1052, in <lambda>

    lambda: self.process_request_in_thread(request, client_address))

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 1068, in process_request_in_thread

    self.finish_request(request, client_address)

  File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request

    self.RequestHandlerClass(request, client_address, self)

  File "/usr/lib/python2.6/SocketServer.py", line 617, in __init__

    self.handle()

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 442, in handle

    BaseHTTPRequestHandler.handle(self)

  File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle

    self.handle_one_request()

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 437, in handle_one_request

    self.wsgi_execute()

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 287, in wsgi_execute

    self.wsgi_start_response)

  File "/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/connector.py", line 18, in __call__

    conn = self.db.open()

  File "/home/tsa/env/lib/python2.6/site-packages/ZODB/DB.py", line 729, in open

    self._a()

  File "/usr/lib/python2.6/threading.py", line 123, in acquire

    rc = self.__block.acquire(blocking)

The server gets to a blocked state every 24 hours or so. Simply restarting

the webserver works fine; however, i'd like to know what the problem is so

restarting won't be necessary, and to prevent it from getting worse. Any ideas/

suggestions?

Thanks,

- Claudiu

_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

[Hanno Schlichting-4]

I think you might get better help on one of the Pyramid support channels.

Your problems all seem to be related to configuring a web server in
production mode, rather than database issues.

From what I can tell you are dealing with hung requests. I'd look at
either the paster configuration options for anything related to
timeouts, thread pools, handling of incomplete requests and so on. Or
use a more production quality web server like Apache (mod_wsgi), Nginx
(gevent/gunicon) which likely has better default configuration values
for these things.

Hanno

On Mon, May 7, 2012 at 5:38 PM, Claudiu Saftoiu <csaftoiu@...> wrote:

> Hello all,
>
> I'm using Repoze.BFG, with paster to launch the webserver. This is a
> similar issue to the one I emailed about before titled
> "server stops handling requests - nowhere near 100% CPU or Memory
> used"
>
> The situation is the same. I used z3c.deadlockdebugger , and what
> I notice is that, when the server is blocked, there are about 100
> threads running (as opposed to 15 or so when the server has
> just started), and all their stack traces look like this:
>
> Thread 140269004887808:
>   File "/usr/lib/python2.6/threading.py", line 504, in __bootstrap
>     self.__bootstrap_inner()
>   File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
>     self.run()
>   File "/usr/lib/python2.6/threading.py", line 484, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line
> 878, in worker_thread_callback
>     runnable()
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line
> 1052, in <lambda>
>     lambda: self.process_request_in_thread(request, client_address))
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line
> 1068, in process_request_in_thread
>     self.finish_request(request, client_address)
>   File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request
>     self.RequestHandlerClass(request, client_address, self)
>   File "/usr/lib/python2.6/SocketServer.py", line 617, in __init__
>     self.handle()
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line
> 442, in handle
>     BaseHTTPRequestHandler.handle(self)
>   File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
>     self.handle_one_request()
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line
> 437, in handle_one_request
>     self.wsgi_execute()
>   File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line
> 287, in wsgi_execute
>     self.wsgi_start_response)
>   File
> "/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/connector.py",
> line 18, in __call__
>     conn = self.db.open()
>   File "/home/tsa/env/lib/python2.6/site-packages/ZODB/DB.py", line 729, in
> open
>     self._a()
>   File "/usr/lib/python2.6/threading.py", line 123, in acquire
>     rc = self.__block.acquire(blocking)
>
> The server gets to a blocked state every 24 hours or so. Simply restarting
> the webserver works fine; however, i'd like to know what the problem is so
> restarting won't be necessary, and to prevent it from getting worse. Any
> ideas/
> suggestions?
>
> Thanks,
> - Claudiu
>
> _______________________________________________
> For more information about ZODB, see http://zodb.org/
>
> ZODB-Dev mailing list  -  ZODB-Dev@...
> https://mail.zope.org/mailman/listinfo/zodb-dev
>

_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

[Claudiu Saftoiu]

On Mon, May 7, 2012 at 12:13 PM, Hanno Schlichting <hanno@...> wrote:

I think you might get better help on one of the Pyramid support channels.

Your problems all seem to be related to configuring a web server in
production mode, rather than database issues.

Thanks, I'll check out the Repoze.BFG IRC channel as well.

From what I can tell you are dealing with hung requests. I'd look at
either the paster configuration options for anything related to
timeouts, thread pools, handling of incomplete requests and so on. Or
use a more production quality web server like Apache (mod_wsgi), Nginx
(gevent/gunicon) which likely has better default configuration values
for these things.

The problem from my previous email was indeed hung requests. However, the

stack trace for those looked different:

Thread 140605868680960:

  File "/usr/lib/python2.6/threading.py", line 504, in __bootstrap

    self.__bootstrap_inner()

  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner

    self.run()

  File "/usr/lib/python2.6/threading.py", line 484, in run

    self.__target(*self.__args, **self.__kwargs)

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 878, in worker_thread_callback

    runnable()

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 1052, in <lambda>

    lambda: self.process_request_in_thread(request, client_address))

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 1068, in process_request_in_thread

    self.finish_request(request, client_address)

  File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request

    self.RequestHandlerClass(request, client_address, self)

  File "/usr/lib/python2.6/SocketServer.py", line 617, in __init__

    self.handle()

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 442, in handle

    BaseHTTPRequestHandler.handle(self)

  File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle

    self.handle_one_request()

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 437, in handle_one_request

    self.wsgi_execute()

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 287, in wsgi_execute

    self.wsgi_start_response)

  File "/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/connector.py", line 21, in __call__

    result = self.next_app(environ, start_response)

  File "/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/cachecleanup.py", line 25, in __call__

    return self.next_app(environ, start_response)

  File "/home/tsa/env/lib/python2.6/site-packages/repoze/retry/__init__.py", line 65, in __call__

    chunk = original_wsgi_input.read(rest)

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 474, in read

    data = self.file.read(length)

  File "/usr/lib/python2.6/socket.py", line 377, in read

    data = self._sock.recv(left)

Note the line it blocks on is "self._sock.recv(left)", well after the response started. 

In the trace I just provided, the block was on opening the DB connection *at the start of the request*:

  ...

  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py", line 287, in wsgi_execute

    self.wsgi_start_response)

  File "/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/connector.py", line 18, in __call__

    conn = self.db.open()

  File "/home/tsa/env/lib/python2.6/site-packages/ZODB/DB.py", line 729, in open

    self._a()

  File "/usr/lib/python2.6/threading.py", line 123, in acquire

    rc = self.__block.acquire(blocking)

Why would the database start blocking on opening a new database connection? The

issue does indeed seem to be with ZODB. 

Thanks,

- Claudiu

_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev