ZEO Cache inconsistency

[Vincent Pelletier-3]

Hi.

I'm facing a cache consistency problem in an old Zeo running on 2.8.8
products.

Minimal required setup:
- Zeo sharing a ZODB
- MySQL
- Zope instance 1 mounting the shared ZODB and connected to MySQL (Z1)
- Zope instance 2 mounting the shared ZODB and connected to MySQL (Z2)

Scenario:
- Z1 & Z22 access some object to be used as container in next step, so both
  have it in cache
- Z1 creates an object in ZODB, and stored its path in MySQL
- Z2 fetches the path from MySQL and tries to fetch the object from the
  ZODB and gets a KeyError

Traced cause:
Z2 didn't process the cache invalidation request sent to it by Zeo upon
transaction beginning (well, upon joining the database connector).
If I access the new subobject by its OID from Z2 during the 3rd scenario step,
it works (object is found in Zeo).
If I peek at pending invalidations from Z2 during the 3rd scenario step, I
found an invalidation for the container.

Note: in my test case, the container is a BTreeFolder2, but this should in
theory happen with any kind of container. BTreeFolder2 makes the "invalidation
peek" check harder, as one must find the OID of the bucket which contains the
new object.

Note2: this bug of course does not happen each time. It was triggered with an
approx. 1 per 5000 ratio.

Could you tell me if there is a fix for this problem in more recent versions
of Zeo ?

Do you have any idea of a possible fix ?

I believe
- this can only be avoided by adding an exchange with Zeo servers so Zope can
  check if it processed all invalidations prior to joining the storage, so
  storage would "force" Zope to process pending invalidations
- it is not acceptable to do such check upon object read access (too heavy)
- it might be hard to do that exchange at the end of the transaction (the last
  invalidation number will have been long forgotten by Zeo)
- it might slow down transaction to add an exchange at storage join (if there
  is none at the moment, which I don't know)

--
Vincent Pelletier
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

[Benji York-2]

On Tue, Oct 20, 2009 at 5:41 AM, Vincent Pelletier <vincent@...> wrote:
> I'm facing a cache consistency problem in an old Zeo running on 2.8.8
> products.

I don't know what version of ZODB those Zope versions use, but version
3.8.1 of ZODB included several bug fixes in the ZEO cache code.
--
Benji York
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

[Vincent Pelletier-3]

On Thursday 29 October 2009 15:26:32 Benji York wrote:
> I don't know what version of ZODB those Zope versions use,

It used ZODB 3.4.4 (as of __init__.py:__version__).

> but version 3.8.1 of ZODB included several bug fixes in the ZEO cache code.

Very good to know. I'll dig into this, thanks a lot.

--
Vincent Pelletier
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev