instrumenting a ZEO server?

[Chris Withers]

Hi All,

I'm wondering if there's any way to get a ZEO server to log about how
it's performing?
(eg: how many connections, how many clients are waiting, where they're
waiting, etc)

I'm trying to debug a problem where app servers are being dropped out of
a load balancer pool for being unresponsive, and I think they're waiting
on the ZEO server so I'm trying to get some stats on the ZEO server side
as to what's going on...

As far as the clients go, DeadlockDebugger is providing some insights:

   File "/opt/Zope-2.9/lib/python/ZEO/ClientStorage.py", line 749, in loadEx
     self._lock.acquire()    # for atomic processing of invalidations

I assume this is the client waiting for the storage server's lock?
Is this the ZEO server's single threadedness or is this a lock between
threads on the app server?

   File "/opt/Zope-2.9/lib/python/ZEO/ClientStorage.py", line 769, in loadEx
     data, tid, ver = self._server.loadEx(oid, version)
   File "/opt/Zope-2.9/lib/python/ZEO/ServerStub.py", line 192, in loadEx
     return self.rpc.call("loadEx", oid, version)
   File "/opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py", line 531, in call
     r_flags, r_args = self.wait(msgid)
   File "/opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py", line 638, in wait
     asyncore.poll(delay, self._singleton)
   File "/usr/local/lib/python2.4/asyncore.py", line 122, in poll
     r, w, e = select.select(r, w, e, timeout)

What does this mean? I'm guessing it's the actual transfer of data from
the storage server to the app server, right?

cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
http://mail.zope.org/mailman/listinfo/zodb-dev

[Alan Runyan-3]

On Wed, Aug 26, 2009 at 8:08 AM, Chris Withers<chris@...> wrote:

> Hi All,
>
> I'm wondering if there's any way to get a ZEO server to log about how
> it's performing?
> (eg: how many connections, how many clients are waiting, where they're
> waiting, etc)
>
> I'm trying to debug a problem where app servers are being dropped out of
> a load balancer pool for being unresponsive, and I think they're waiting
> on the ZEO server so I'm trying to get some stats on the ZEO server side
> as to what's going on...
>
> As far as the clients go, DeadlockDebugger is providing some insights:
>
>   File "/opt/Zope-2.9/lib/python/ZEO/ClientStorage.py", line 749, in loadEx
>     self._lock.acquire()    # for atomic processing of invalidations
>
> I assume this is the client waiting for the storage server's lock?
> Is this the ZEO server's single threadedness or is this a lock between
> threads on the app server?
>   File "/opt/Zope-2.9/lib/python/ZEO/ClientStorage.py", line 769, in loadEx
>     data, tid, ver = self._server.loadEx(oid, version)
>   File "/opt/Zope-2.9/lib/python/ZEO/ServerStub.py", line 192, in loadEx
>     return self.rpc.call("loadEx", oid, version)
>   File "/opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py", line 531, in call
>     r_flags, r_args = self.wait(msgid)
>   File "/opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py", line 638, in wait
>     asyncore.poll(delay, self._singleton)
>   File "/usr/local/lib/python2.4/asyncore.py", line 122, in poll
>     r, w, e = select.select(r, w, e, timeout)
>
> What does this mean? I'm guessing it's the actual transfer of data from
> the storage server to the app server, right?

What version of ZODB are you using?  If you have zeo on in debugging
what you can see is if there are outstanding clients waiting to write
while a transaction is already writing. It says something like "# clients
blocked while transaction in progress" in zeo server log files.

How many connections:
iirc zrpc is connection oriented so you should have # of application clients
connected to ZEO server.  Or it may be # of clients * # of ZODB threads.

How many clients are waiting:
You should be able to see this in zeo log files

Where they're waiting:
deadlockdebugger or products.signalstack will show what the clients are doing.

may want to strace the client and server to see if there is long
pauses of inactivity.
maybe your running up against a real ZODB bug that was fixed in later releases?
look at ZODB 3.8 branch NEWS.txt

good luck
alan
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
http://mail.zope.org/mailman/listinfo/zodb-dev

[Chris Withers]

Alan Runyan wrote:
> What version of ZODB are you using?

Whatever ships with Zope 2.9.8

> If you have zeo on in debugging

What does this mean?

> How many clients are waiting:
> You should be able to see this in zeo log files

Not currently... maybe this relates to your mystical "zeo on in
debugging" comment? ;-)

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
http://mail.zope.org/mailman/listinfo/zodb-dev

[Alan Runyan-3]

>> What version of ZODB are you using?
>
> Whatever ships with Zope 2.9.8

I think that is ZODB 3.6 which is on the older side.
I was simply noting that there are several invalidation
fixes in ZODB 3.8; whether they fix your problem I do
not know.

>> If you have zeo on in debugging
>
> What does this mean?

Turn up logging in zeo.conf to DEBUG
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
http://mail.zope.org/mailman/listinfo/zodb-dev