Nabble - Zope - ZODB-Dev

Re: Locating backrefs of persistent objects

2009-12-17T16:03:25Z

Marius Gedminas wrote:
> I'd like to implement the following feature for zodbbrowser: given a
> persistent object X, find all other persistent objects that have
> references to it.

RelStorage has this functionality, but I don't know how we ought to
expose it. Start a pack to update the graph, then query the object_ref
table. Turn on pack-dry-run if you only want to update the table and
don't want to pack.

I'd like to make RelStorage implement some Python API for listing
backreferences, but no such API has been defined.

Shane

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Locating backrefs of persistent objects

2009-12-17T13:30:58Z

On Thu, Dec 17, 2009 at 4:06 PM, Marius Gedminas <marius@...> wrote:

> I'd like to implement the following feature for zodbbrowser: given a
> persistent object X, find all other persistent objects that have
> references to it.
>
> My use case for this is debugging: e.g. given a log entry that mentions
> a conflict error about an OOBTreeBucket, I'd like to know which OOBTree
> object contains it, and what object wraps the OOBTree, and at that point
> I usually see a container with a __name__ and __parent__ and can figure
> out what's going on.
>
> I realize that ZODB doesn't keep track of backreferences, so the only
> way to get that is to do a full object traversal, which is expensive in
> both CPU and memory.

...

> ? Is there existing code I could use for inspiration? zodbex? zc.zodbgc?
> Bits and pieces of ZODB itself?

zc.zodbdgc's multi-zodb-check-refs script will optionally produce a database
of reverse references that your browser could use.

Note that building this database can take a very long time.

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Locating backrefs of persistent objects

2009-12-17T13:06:27Z

I'd like to implement the following feature for zodbbrowser: given a
persistent object X, find all other persistent objects that have
references to it.

My use case for this is debugging: e.g. given a log entry that mentions
a conflict error about an OOBTreeBucket, I'd like to know which OOBTree
object contains it, and what object wraps the OOBTree, and at that point
I usually see a container with a __name__ and __parent__ and can figure
out what's going on.

I realize that ZODB doesn't keep track of backreferences, so the only
way to get that is to do a full object traversal, which is expensive in
both CPU and memory.

Given all that, what's the best (most efficient) way of computing the
backreferences either for a single object, or for _all_ objects in the
ZODB (that I could then cache for later use)?

* implement something by hand, by using storage iterators and
extracting forward references from each pickle

or

* implement something by hand, by doing graph traversal (BFS/DFS)
starting from the root object

or

* reuse existing object graph traversal routines of the ZODB garbage
collector

? Is there existing code I could use for inspiration? zodbex? zc.zodbgc?
Bits and pieces of ZODB itself?

The wishlist bug for this feature is here:
https://bugs.launchpad.net/zodbbrowser/+bug/497758

Marius Gedminas
--
Linux don't need no steenkin' viruses. The users can destroy the
system all by themselves....
-- Peter Dalgaard in comp.os.linux.misc

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

signature.asc (196 bytes) Download Attachment

Re: ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)

2009-12-17T10:26:31Z

On Thu, Dec 10, 2009 at 8:12 PM, Marius Gedminas <marius@...> wrote:

On Thu, Dec 10, 2009 at 02:07:06PM -0500, Jim Fulton wrote:
> On Thu, Dec 10, 2009 at 1:50 PM, Marius Gedminas <marius@...> wrote:
> > On Thu, Dec 10, 2009 at 12:41:11PM -0500, Jim Fulton wrote:
> >> On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas <marius@...> wrote:
> >> In 3.9 for FileStorage, if you give a starting tid that is toward the
> >> end of the file,
> >> it will scan backward, saving a lot of time.
> >
> > Ah, but how can I get a valid tid that is toward the end of the file?
> > I assume that if I give a tid that doesn't exist, I'll get an error --
> > which is the behavior oldstate() exhibits -- rather than the next
> > existing transaction after that timestamp, or I could fudge by getting
> > the current timestamp and subtracting 24 hours.
>
> No, the iterator starts at the first transactions who's tid is >= the
> start tid.
>
> So, guess a time and use repr(ZODB.TimeStamp.TimeStamp(...))

Cool! Thank you,

Marius Gedminas
--

another way is to use the fstools.prev_txn starting at the end of the file to grab the newest transactions, the script i send to the list last week uses it to poke a large zodb to just get the contents of the last 20 transactions without iterating through the whole thing or guessing a timestamp.

https://mail.zope.org/pipermail/zodb-dev/attachments/20091207/531fc1e4/attachment.obj

cheers,

kapil

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Python properties on Persistent objects

2009-12-17T08:10:21Z

>
> Properties work with persistent objects with one limitation that I'm aware of.
> The __setattribute__ method used by the persistent base class assumes
> that any attribute assignment modifies the object's persistent state.
> A property
> that doesn't modify state or that modifies volatile state will fool
> it. Conceivably,
> one could override __setattribute__ to handle such properties, but doing so
> would require great care. I don't recommend it.

Great, thanks! I already assumed so, but now this little feature is
officially confirmed.

-Mikko
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Python properties on Persistent objects

2009-12-17T04:21:40Z

On Wed, Dec 16, 2009 at 7:12 PM, Mikko Ohtamaa <mikko@...> wrote:
> I need to have little clarification should properties work on
> Persistent objects.

Properties work with persistent objects with one limitation that I'm aware of.
The __setattribute__ method used by the persistent base class assumes
that any attribute assignment modifies the object's persistent state.
A property
that doesn't modify state or that modifies volatile state will fool
it. Conceivably,
one could override __setattribute__ to handle such properties, but doing so
would require great care. I don't recommend it.

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Python properties on Persistent objects

2009-12-17T03:10:50Z

> That code is insane. Really insane. It's generating classes and doing
> custom adapter factories that inject a _v_ variable into a persistent
> object temporarily. I wouldn't try to copy it at all. I would run away.

But I thought this was *the* Zope way and the brightest minds in the
community have obviously written the code :)

In any case, the persistent + property issue has received some
awareness until it bits someone's ass again. I am going back to the
drawing board and my behaviors.

property set = always starts transaction even though nothing is being written

-Mikko
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Python properties on Persistent objects

2009-12-17T01:18:57Z

Mikko Ohtamaa wrote:
>> This isn't right: a z3c.form form is just a view like any other. It is
>> looked up on the context and the request. In Dexterity (which I assume
>> you're using?), that's going to be the content object.
>
> Sorry, I think I mixed with zope.formlib. In any case I hope to find
> the answer to the orignal question.
>
> The original pattern which I copied is is here:
> http://code.google.com/p/getpaid/source/browse/Products.PloneGetPaid/trunk/Products/PloneGetPaid/preferences.py

That code is insane. Really insane. It's generating classes and doing
custom adapter factories that inject a _v_ variable into a persistent
object temporarily. I wouldn't try to copy it at all. I would run away.

I *think* what may be happening here is that most of the vocabularies in
plone.app.vocabularies have code like this:

context = getattr(context, 'context', context)

This hack is there to support the IAdding view use case where the
context of the add form is another view and the first "real" content
object from which you can acquire stuff is self.context.context. We're
moving away from that in favour of not acquiring tools and not using
views-on-views, which are evil anyway. Hence, the 'context' property in
StoreSettings, which I presume is used as the context for a form, is
being cajoled into being the Plone site root. But that's using a hack to
trick a hack into doing different hackery.

This is getting far off topic for the ZODB list, but if you can explain
what problem you're actually trying to solve, I'm sure we can come up
with a suggestion that is less of an architecture spacewalk.

When you start having to ask the ZODB list, maybe it's time to think
whether you could find a simpler solution. :-)

> I don't think I am doing it incorrectly. The other reason of using
> this pattern is that the behavior factory does not need to have a
> write or (an extra) object load if the behavior is never set on the
> context object. The default object returned by factory acts similar
> than the persistent object and I can set the behavior defaults using
> zope.schema - I don't need to do "if not set" logic anywhere else in
> my code.

There is no reason you can't get this "don't-write-unless-necessary"
behaviour in a much more sane way. In fact, it ought to be the default.
To help you, I'd need a more comprehensive understanding of what you're
doing and why, though.

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Python properties on Persistent objects

2009-12-17T00:43:18Z

>
> This isn't right: a z3c.form form is just a view like any other. It is
> looked up on the context and the request. In Dexterity (which I assume
> you're using?), that's going to be the content object.

Sorry, I think I mixed with zope.formlib. In any case I hope to find
the answer to the orignal question.

The original pattern which I copied is is here:
http://code.google.com/p/getpaid/source/browse/Products.PloneGetPaid/trunk/Products/PloneGetPaid/preferences.py

>
> I think you need to take a step back and look at how your form is set
> up, not dive into deep magic of persistence.

I don't think I am doing it incorrectly. The other reason of using
this pattern is that the behavior factory does not need to have a
write or (an extra) object load if the behavior is never set on the
context object. The default object returned by factory acts similar
than the persistent object and I can set the behavior defaults using
zope.schema - I don't need to do "if not set" logic anywhere else in
my code.

=)

Cheers,
-Mikko
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Python properties on Persistent objects

2009-12-16T16:49:40Z

Laurence Rowe wrote:

> 2009/12/17 Mikko Ohtamaa<mikko@...>:
>> Hi,
>>
>> I need to have little clarification should properties work on
>> Persistent objects. I am running ZODB 3.8.4 on Plone 3.3.
>>
>> I am using plone.behavior and adapters to retrofit objects with a new
>> behavior (HeaderBehavior object). This object is also editable through
>> z3c.form interface. z3c.form requires a context variable on the object
>> e.g. to look up dynamic vocabularies. To avoid having this
>> object.context attribute to be peristent (as it's known every time by
>> the factory method of the adapter which creates/look-ups
>> HeaderBehavior) I tried to spoof context variable using properties and
>> internal volatile variable. This was a trick I learnt somewhere
>> (getpaid.core?)
>
> This sounds like you are passing context somewhere where view is expected.

Yeah, to me as well it sounds like you're barking up the wrong tree
here. You shouldn't need any kind of faking like this.

"z3c.form requires a context variable on the object
e.g. to look up dynamic vocabularies"

This isn't right: a z3c.form form is just a view like any other. It is
looked up on the context and the request. In Dexterity (which I assume
you're using?), that's going to be the content object.

I think you need to take a step back and look at how your form is set
up, not dive into deep magic of persistence.

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Python properties on Persistent objects

2009-12-16T16:24:05Z

2009/12/17 Mikko Ohtamaa <mikko@...>:

> Hi,
>
> I need to have little clarification should properties work on
> Persistent objects. I am running ZODB 3.8.4 on Plone 3.3.
>
> I am using plone.behavior and adapters to retrofit objects with a new
> behavior (HeaderBehavior object). This object is also editable through
> z3c.form interface. z3c.form requires a context variable on the object
> e.g. to look up dynamic vocabularies. To avoid having this
> object.context attribute to be peristent (as it's known every time by
> the factory method of the adapter which creates/look-ups
> HeaderBehavior) I tried to spoof context variable using properties and
> internal volatile variable. This was a trick I learnt somewhere
> (getpaid.core?)

This sounds like you are passing context somewhere where view is expected.

Laurence
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Python properties on Persistent objects

2009-12-16T16:12:00Z

Hi,

I need to have little clarification should properties work on
Persistent objects. I am running ZODB 3.8.4 on Plone 3.3.

I am using plone.behavior and adapters to retrofit objects with a new
behavior (HeaderBehavior object). This object is also editable through
z3c.form interface. z3c.form requires a context variable on the object
e.g. to look up dynamic vocabularies. To avoid having this
object.context attribute to be peristent (as it's known every time by
the factory method of the adapter which creates/look-ups
HeaderBehavior) I tried to spoof context variable using properties and
internal volatile variable. This was a trick I learnt somewhere
(getpaid.core?)

It didn't work. Looks like Persistent class is not aware of properties
and interprets property set as a transaction start. So even though I
thought I was having a volatile context, my Undo log kept having new
write entries on every page read for pages having HeaderBehavior
objects in their annotations. Data.fs grew steadily with small writes
and there was a whole bunch of ConflictErrors.

I think I found a workaround. By overriding __setattr__ and not
letting through properties and volatile attribute set to Persistent
__setattr__ you can avoid the problem.

Is my rationale correct? Should Persistent behave well with
properties? I found only one discussion in Google, circa 2004.

Sample code below:

from persistent import Persistent
class VolatileContext(Persistent):
""" Mix-in class to provide non-persistent context attribute to
persistent classes.

Some subsystems (e.g. z3c.forms) expect objects to have a context
reference to parent/site/whatever.
However, storing this back-reference persistenly is not needed, as
the factory
method will always know the context.

This helper class creates a context property which is volatile
(never persistent),
but can be still set on the object after creation or after database load.
"""

zope.interface.implements(IVolatileContext)

def _set_context(self, context):
self._v_context = context

def _get_context(self):
return self._v_context

def _set_factory(self, factory):
self._v_factory = factory

def _get_factory(self):
return self._v_factory

# http://docs.python.org/library/functions.html#property
context = property(_get_context, _set_context)

factory = property(_get_factory, _set_factory)

def save(self):
""" """
self.factory.makePersistent(self)

def __setattr__(self, name, value):
if name not in ("context", "factory", "_v_context", "_v_factory"):
Persistent.__setattr__(self, name, value)
print "leaking " + name + " " + str(value)
else:
print "property set " + name + " " + str(value)
object.__setattr__(self, name, value)

class AnnotationPersistentFactory(object):
""" A factory pattern to manufacture persistent objects stored
within the parent object annotations.

Until the first write, the default (non-persistent) object is
return. This prevents
possible situations where database read could cause write.

The first write must call
AnnotationPersistentFactory.makePersistent(object).
Alternative, you can call AnnotationPersistentFactory.makePersistent(object)
when entering the editing interface for the first time.

After the first write, the saved persistent object is return.
"""

def __init__(self, persistent_class, key):
"""
@param persistent_class: Class reference / factory method
which will create new objects.
Created classes must conform VolatileContext interface

@param key: ASCII string, Key name used with IAnnotations
"""
self.persistent_class = persistent_class
self.key = key
self._assertProperlySetUp()

def _assertProperlySetUp(self):
"""
Check that the framework is properly set up
"""
assert callable(self.persistent_class), "Factory is missing"

assert hasattr(self.persistent_class, "context"), "The
persistent object must support volatile context interface"

assert self.key is not None, "You must give the annotations key"

def makePersistent(self, object):
""" Write created persistent object to the database.

This will store the object on the annotations of its context.
"""
assert isinstance(object, self.persistent_class), "Object %s
was not type of %s" % (str(object), str(self.persistent_class))
annotations = IAnnotations(object.context)
annotations[self.key] = object

def __call__(self, context):
""" Called by Zope framework when doing a factory call.

Usually this class is refered as <adapter factory=""> and
this method creates a new, read-only, persistent object.
"""

annotations = IAnnotations(context)

if not self.key in annotations:
# Construct a new (default) instance
print "Created"
object = self.persistent_class()
else:
# Return the object stored previously
print "Found"
object = annotations[self.key]

# Set volatile context reference
object.context = context
object.factory = self

return object

-Mikko
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

ZODB 3.8.5 released

2009-12-16T09:46:29Z

I've released 3.8.5, which has one important bug fix and some minor
forward-compatibility changes we discussed here recently.

http://pypi.python.org/pypi/ZODB3/3.8.5

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: zc.demostorage2 assumes too much?

2009-12-15T09:02:13Z

On Thu, Nov 26, 2009 at 7:33 AM, Wolfgang Schnerring <ws@...> wrote:

> Hello,
>
> zc.demostorage2 assumes that the tpc_begin() of its underlying
> storages takes three parameters (transaction, tid, status), while both
> MappingStorage and IStorage only know about the first two.
>
> This strikes me as risky, and it does indeed break, for example when
> trying to use a DemoStorage as the 'changes' backend of a
> DemoStorage2.
>
> I've tried and removed the status parameter from zc.demostorage2 and
> its tests still pass, but I must confess I have no clue what this
> status parameter actually does, so I don't really know what (if
> anything) to do about this issue.

The tid and status parameters are used when copying already committed
transactions. zc.demostprage2 should check whether tid is None
and only pass tid and status when it isn't.

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

ZODB 3.9.4 released

2009-12-14T15:03:35Z

http://pypi.python.org/pypi/ZODB3/3.9.4

Bugs Fixed

* A ZEO threading bug could cause transactions to read
inconsistent data. (This sometimes caused an AssertionError in
Connection._setstate_noncurrent.)
* DemoStorage.loadBefore sometimes returned invalid data which
would trigger AssertionErrors in ZODB.Connection.
* History support was broken when using stprages that work with
ZODB 3.8 and 3.9.
* zope.testing was an unnecessary non-testing dependency.
* Internal ZEO errors were logged at the INFO level, rather than
at the error level.
* The FileStorage backup and restore script, repozo, gave a
deprecation warning under Python 2.6.
* C Header files weren't installed correctly.
* The repozo backup utility gave deprecation warnings when run.

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: BTrees and id mapping (was Re: Data.fs size grows non-stop)

2009-12-14T13:49:23Z

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pedro Ferreira wrote:

> David Glick wrote:
>> Pedro Ferreira wrote:
>>
>>> Hello all,
>>>
>>> So, I've been experimenting with a new structure for our date indexes,
>>> and I was considering using an IOBTree that would map integer timestamps
>>> to BTree-based sets. As for the sets, I considered two options:
>>>
>>>
>> Be careful with using timestamps as IOBTree keys. By default this type
>> of BTree uses the system integer, which on many systems is 32 bits.
>> That means that you'll overflow the maximum allowed key for timestamps
>> beyond the year 2038. It won't even give an error; it will just
>> silently overflow and overwrite existing values in the BTree.
>>
> Yes, I know. That's one of the reasons I'm using YYYYmmdd integers
> instead (the other one being i only need 1 day resolution).
>
> Thanks a lot for the tip!

An LOBTree uses a guaranteed-to-be-long (64 bits) integer for the keys,
if you need that.

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver@...
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksmsuMACgkQ+gerLs4ltQ6mNgCgwBpVRaUli7a7Ij1NQYYHsxif
OD0AnjgAGY1TB6GAadx1qSmwW/8w5c1a
=ZJRe
-----END PGP SIGNATURE-----

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: BTrees and id mapping (was Re: Data.fs size grows non-stop)

2009-12-14T02:17:10Z

David Glick wrote:

> Pedro Ferreira wrote:
>
>> Hello all,
>>
>> So, I've been experimenting with a new structure for our date indexes,
>> and I was considering using an IOBTree that would map integer timestamps
>> to BTree-based sets. As for the sets, I considered two options:
>>
>>
> Be careful with using timestamps as IOBTree keys. By default this type
> of BTree uses the system integer, which on many systems is 32 bits.
> That means that you'll overflow the maximum allowed key for timestamps
> beyond the year 2038. It won't even give an error; it will just
> silently overflow and overwrite existing values in the BTree.
>

Yes, I know. That's one of the reasons I'm using YYYYmmdd integers
instead (the other one being i only need 1 day resolution).

Thanks a lot for the tip!

Pedro
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

BTrees and id mapping (was Re: Data.fs size grows non-stop)

2009-12-11T03:09:10Z

Hello all,

So, I've been experimenting with a new structure for our date indexes,
and I was considering using an IOBTree that would map integer timestamps
to BTree-based sets. As for the sets, I considered two options:

1. An IITreeSet based on the OID and an additional IOBTree that maps
back the OIDs to objects;
2. An OOTreeSet;

As expected 1 was light years faster when I tried to query for large
timespans (doing normal union operations using OOTreeSets seems to be a
very slow operation). Still, for single days (and generally timespans
smaller than 7 days), the OOTreeSet seems to be much faster (apparently,
iterating through the all the ids in the resulting set and fetching the
corresponding object takes some time, even for sets with only one
member, so, IITreeSet seems to be in disadvantage here).
Does anyone have any advice on optimizing the first case? I believe the
IITreeSet is the way to go, but if only there was some way to make the
int<->object conversion operation less expensive...

Thanks in advance,

Pedro
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)

2009-12-10T17:12:21Z

On Thu, Dec 10, 2009 at 02:07:06PM -0500, Jim Fulton wrote:

> On Thu, Dec 10, 2009 at 1:50 PM, Marius Gedminas <marius@...> wrote:
> > On Thu, Dec 10, 2009 at 12:41:11PM -0500, Jim Fulton wrote:
> >> On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas <marius@...> wrote:
> >> In 3.9 for FileStorage, if you give a starting tid that is toward the
> >> end of the file,
> >> it will scan backward, saving a lot of time.
> >
> > Ah, but how can I get a valid tid that is toward the end of the file?
> > I assume that if I give a tid that doesn't exist, I'll get an error --
> > which is the behavior oldstate() exhibits -- rather than the next
> > existing transaction after that timestamp, or I could fudge by getting
> > the current timestamp and subtracting 24 hours.
>
> No, the iterator starts at the first transactions who's tid is >= the
> start tid.
>
> So, guess a time and use repr(ZODB.TimeStamp.TimeStamp(...))

Cool! Thank you,

Marius Gedminas
--
This message has been ROT-13 encrypted twice for higher security.

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

signature.asc (196 bytes) Download Attachment

Re: ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)

2009-12-10T11:07:06Z

On Thu, Dec 10, 2009 at 1:50 PM, Marius Gedminas <marius@...> wrote:

> On Thu, Dec 10, 2009 at 12:41:11PM -0500, Jim Fulton wrote:
>> On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas <marius@...> wrote:
>> In 3.9 for FileStorage, if you give a starting tid that is toward the
>> end of the file,
>> it will scan backward, saving a lot of time.
>
> Ah, but how can I get a valid tid that is toward the end of the file?
> I assume that if I give a tid that doesn't exist, I'll get an error --
> which is the behavior oldstate() exhibits -- rather than the next
> existing transaction after that timestamp, or I could fudge by getting
> the current timestamp and subtracting 24 hours.

No, the iterator starts at the first transactions who's tid is >= the start tid.

So, guess a time and use repr(ZODB.TimeStamp.TimeStamp(...))

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)

2009-12-10T10:50:27Z

On Thu, Dec 10, 2009 at 12:41:11PM -0500, Jim Fulton wrote:

> On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas <marius@...> wrote:
> ...
> > Well, there is IStorage, and I think all storages (except for
> > FileStorage) implemented it faithfully
>
> I guess that depends on what you mean by faithfully. MappingStorage
> and Demostorage
> in 3.8 inherited a null implementation from BaseStorage and it's
> arguable how faithful that
> implementation is.
>
> > (oid,
> > version-that-got-removed-in-3.9, size=1). FileStorage in 3.8 violated
> > the interface by renaming size=1 to length=1, but it's fixed in 3.9.
>
> and in 3.8.

*checks*

FileStorage in 3.8.1 violated the interface, to be precise.

ZODB 3.8.1 is mandated by
http://download.zope.org/zope3.4/3.4.0/versions.cfg, which our
production app currently uses.

(Kinda offtopic: Is there a newer 3.4 KGS release?
http://download.zope.org/ tells me "Welcome" and nothing more.
http://download.zope.org/zope3.4/ tells me in great big joyful letters
at the top of the page "zope 3.4.0 Has Been Released!")

> > What I miss _a lot_ is a way to look at the last several transactions.
> > There's the iterator API, but it goes from oldest-to-latest, and when
> > you've got a large DB, that takes a while (85 seconds for a Data.fs with
> > 922737 transactions and 8 GB of data, with 100 MB/s read throughput).
>
> In 3.9 for FileStorage, if you give a starting tid that is toward the
> end of the file,
> it will scan backward, saving a lot of time.

Ah, but how can I get a valid tid that is toward the end of the file?
I assume that if I give a tid that doesn't exist, I'll get an error --
which is the behavior oldstate() exhibits -- rather than the next
existing transaction after that timestamp, or I could fudge by getting
the current timestamp and subtracting 24 hours.

> I've attached an analysis script that illustrates iterating backward.

Thank you!

(Mutt is inconvenient: there's no way to look at attachments while you're
replying to a message.)

> Someday, someone should get around to adding an option to the file-storage
> iterator to go backward.

Marius Gedminas
--
Life begins when you can spend your spare time programming instead of
watching television.
-- Cal Keegan

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

signature.asc (196 bytes) Download Attachment

Re: ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)

2009-12-10T09:41:11Z

On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas <marius@...> wrote:
...
> Well, there is IStorage, and I think all storages (except for
> FileStorage) implemented it faithfully

I guess that depends on what you mean by faithfully. MappingStorage
and Demostorage
in 3.8 inherited a null implementation from BaseStorage and it's
arguable how faithful that
implementation is.

> (oid,
> version-that-got-removed-in-3.9, size=1). FileStorage in 3.8 violated
> the interface by renaming size=1 to length=1, but it's fixed in 3.9.

and in 3.8.

...

> What I miss _a lot_ is a way to look at the last several transactions.
> There's the iterator API, but it goes from oldest-to-latest, and when
> you've got a large DB, that takes a while (85 seconds for a Data.fs with
> 922737 transactions and 8 GB of data, with 100 MB/s read throughput).

In 3.9 for FileStorage, if you give a starting tid that is toward the
end of the file,
it will scan backward, saving a lot of time.

I've attached an analysis script that illustrates iterating backward.
Someday, someone should get around to adding an option to the file-storage
iterator to go backward.

Jim

--
Jim Fulton

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

transaction_rates.py (2K) Download Attachment

Re: ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)

2009-12-10T08:54:57Z

On Thu, Dec 10, 2009 at 10:08:36AM -0500, Jim Fulton wrote:

> On Thu, Dec 10, 2009 at 5:58 AM, Marius Gedminas <marius@...> wrote:
> > On Wed, Dec 09, 2009 at 01:04:03PM -0500, Jim Fulton wrote:
> >> On Wed, Dec 9, 2009 at 12:06 PM, Marius Gedminas <marius@...> wrote:
> >> ...
> >> > (Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very
> >> > ugly hacks I managed.)
> >>
> >> This sounds like something that needs to be fixed. Can you share some of the
> >> issues you ran into? (Or maybe file bugs reports.)
> >
> > Two issues only:
> >
> > * In ZODB 3.8 PersistentDict and PersistentMapping are unrelated
> > classes, so if I want to have adapters for both, I have to register
> > them separately in ZCML. In ZODB 3.9, PersistentDict is an alias
> > for PersistentMapping, so I get a ZCML configuration conflict error
> > if I have adapter directives for both.
>
> OK, it's impractical to do anything about this.
>
> >
> > Solved by defining a decoy class for one of the adapters so they
> > don't conflict.
>
> Hm, I'm not sure I follow this.

Code:

if PersistentMapping is PersistentDict:
# <...snip large comment explaining this...>
class DecoyPersistentDict(PersistentMapping):
"""Decoy to avoid ZCML errors while supporting both ZODB 3.8 and 3.9."""
class PersistentDictState(PersistentMappingState):
"""Decoy to avoid ZCML errors while supporting both ZODB 3.8 and 3.9."""
adapts(DecoyPersistentDict, dict, None)
else:
class PersistentDictState(PersistentMappingState):
"""Convenient access to a persistent dict's items."""
adapts(PersistentDict, dict, None)

ZCML:

<adapter factory=".state.PersistentDictState" />
<adapter factory=".state.PersistentMappingState" />

> You could have used a different
> approach of providing the second adapter registration as an override.

Oooh, neat hack. But I'm not sure I like it better than the one I
implemented.

> > * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well
> > as other kinds of storages, I suppose) is mandatory. In ZODB 3.9
> > it's gone.

I went back and corrected my statement (originally it referred to
FileStorage.history, and I was thinking about ClientStorage in my
parenthetical statement), and then I of course forgot to remove the
now-obsolete parenthetical statement itself.

> It's mandatory in ClientStorage and optional in FileStorage. (MappingStorage and
> DemoStorage don't have a history method in 3.8.
>
> I suggest we treat this as a bug in 3.8 and make the version argument
> to ClientStorage's history method optional in a 3.8 bug fix release.

+1

> > Solved by peeking into the method signature with inspect.getargspec()
> > and supplying a version only if it's needed.
>
> Ick. :)

:)

> >> In particular, I think it should be a goal that it isn't too hard to
> >> write code that works with both ZODB 3.8 and 3.9.
> >
> > Normal code doesn't usually play with storage.history, I'd say, so I'm
> > not too peeved by having to do various contortions.
>
> Zope 2 had (has?) a standard view for looking at an object's history
> that I found very useful. I wish there was one for Zope3.

Well, there is IStorage, and I think all storages (except for
FileStorage) implemented it faithfully (oid,
version-that-got-removed-in-3.9, size=1). FileStorage in 3.8 violated
the interface by renaming size=1 to length=1, but it's fixed in 3.9.

I personally would like to be able to have size=None imply "unlimited,
give me as much as you've got"), but 9999999999999999 works just as well
in practice.

What I miss _a lot_ is a way to look at the last several transactions.
There's the iterator API, but it goes from oldest-to-latest, and when
you've got a large DB, that takes a while (85 seconds for a Data.fs with
922737 transactions and 8 GB of data, with 100 MB/s read throughput).

> History can be very useful in my experience.

Oh yes. This is why I stopped packing my production Data.fs'es and
first started using storage.history() with a couple of nice wrappers
from debugzope shell, and, later, convinced a coworker to write
ZODBBrowser.

Marius Gedminas
--
I used to think I was indecisive, but now I'm not so sure.

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

signature.asc (196 bytes) Download Attachment

Re: Data.fs size grows non-stop

2009-12-10T08:12:48Z

Hi Pedro

> Betreff: Re: AW: [ZODB-Dev] Data.fs size grows non-stop
>
>
> > If you need performance, take a look at the z3c.indexer
> package which
> > offers a much faster indexing concept. I implemented this package
> > because my app coudn't handle the implicit default indexing
> handling.
> > The README.txt explains what is different and how to use it. It
> > defently needs some work to implement a z3c.indexer based indexing
> > concept but if you do it right, it can handle a lot more indexing
> > tasks then the generic zope or zc.catalog concept.
> >
> >
> Thanks a lot. But is it easy to use this from within a
> non-Zope application? In the examples I could see references
> to several zope.site classes, such as LocalSiteManager...
> since this is a standalone app, I wonder if it won't force us
> to import some other zope packages...

The site is only used as a sample setup and for provide
a local persistent IIntIds utility.

Every zope catalog implementation needs an object id
normaly an id given from IIntIds.

But if your app provides another unique integer id
for each object, you can use the z3c.indexer package
with small modifications.

But of corse it will bring in some dependencies,
but not more then the zope or zc.catalog will.

Let me know if you could use the package with less
dependencies then we have now and we can try to cut
the dependencies down or probably defined them as test
dependencies

Regards
Roger Ineichen

> Pedro
> >> Pedro
> >> _______________________________________________
> >> For more information about ZODB, see the ZODB Wiki:
> >> http://www.zope.org/Wikis/ZODB/
> >>
> >> ZODB-Dev mailing list - ZODB-Dev@...
> >> https://mail.zope.org/mailman/listinfo/zodb-dev
> >>
> >>
> >
> >
>
>

Re: ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)

2009-12-10T07:08:36Z

On Thu, Dec 10, 2009 at 5:58 AM, Marius Gedminas <marius@...> wrote:

> On Wed, Dec 09, 2009 at 01:04:03PM -0500, Jim Fulton wrote:
>> On Wed, Dec 9, 2009 at 12:06 PM, Marius Gedminas <marius@...> wrote:
>> ...
>> > (Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very
>> > ugly hacks I managed.)
>>
>> This sounds like something that needs to be fixed. Can you share some of the
>> issues you ran into? (Or maybe file bugs reports.)
>
> Two issues only:
>
> * In ZODB 3.8 PersistentDict and PersistentMapping are unrelated
> classes, so if I want to have adapters for both, I have to register
> them separately in ZCML. In ZODB 3.9, PersistentDict is an alias
> for PersistentMapping, so I get a ZCML configuration conflict error
> if I have adapter directives for both.

OK, it's impractical to do anything about this.

>
> Solved by defining a decoy class for one of the adapters so they
> don't conflict.

Hm, I'm not sure I follow this. You could have used a different
approach of providing the second adapter registration as an override.

> * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well
> as other kinds of storages, I suppose) is mandatory. In ZODB 3.9
> it's gone.

It's mandatory in ClientStorage and optional in FileStorage. (MappingStorage and
DemoStorage don't have a history method in 3.8.

I suggest we treat this as a bug in 3.8 and make the version argument
to ClientStorage's history method optional in a 3.8 bug fix release.

> Solved by peeking into the method signature with inspect.getargspec()
> and supplying a version only if it's needed.

Ick. :)

>> In particular, I think it should be a goal that it isn't too hard to
>> write code that works with both ZODB 3.8 and 3.9.
>
> Normal code doesn't usually play with storage.history, I'd say, so I'm
> not too peeved by having to do various contortions.

Zope 2 had (has?) a standard view for looking at an object's history
that I found very useful. I wish there was one for Zope3.

History can be very useful in my experience.

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Data.fs size grows non-stop

2009-12-10T06:58:51Z

On Thu, Dec 10, 2009 at 4:06 AM, Pedro Ferreira
<jose.pedro.ferreira@...> wrote:

> Jim Fulton wrote:
>>
>> On Wed, Dec 9, 2009 at 10:54 AM, Pedro Ferreira
>> <jose.pedro.ferreira@...> wrote:
>> ...
>>
>>>
>>> Well, at least we won't have to rewrite the whole bucket... but still, it
>>> would be much nicer to fragment the list
>>> in smaller chunks. We could use an OOBTree instead... but something less
>>> complex would suffice... any suggestions?
>>>
>>
>> It's hard for people to suggest something without knowing your usage
>> pattern.
>> Do you want something that is effectively a set?
>
> The situation is simple: we have events (objects) that have to be indexed by
> date. We currently use an OOBTree with the days as keys and lists of objects
> as values. The objective is to avoid rewriting these lists each time
> something changes, since we can have tens of thousands of events in the same
> day. Some kind of btree-based list or set would work, I guess.
>>
>> Is integer indexing important?
>>
>
> Well, it's an index of dates, so, I guess we could use integers as keys
> instead of strings, as we are now.

I wasn't referring to indexing at that level. You were using lists
rather than sets
and I was wondering whether there was a reason for that.

It sounds like a data structure with minimal container semantics
would be fine to store the events for a day.

If you can arrange that the events have a stable ordering such that
no 2 events are equal, then you can use OOTreeSets rather than
lists.

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Data.fs size grows non-stop

2009-12-10T06:38:09Z

Hello,
> Are the values in your BTree implemented as persistent lists or some
> other subclass of Persistent? That makes a big difference in the size
> of each commit to the BTree.
>
Yes, that is exactly the problem. We are using simple Python lists, and
that implies rewriting the whole bucket.
Still, rewriting only the list won't be the best solution, since they
can be pretty big sometimes. That's why I was looking for something like
a BTree-based list (that actually seems to exist).

Regards,

Pedro

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Data.fs size grows non-stop

2009-12-10T06:36:15Z

> If you need performance, take a look at the z3c.indexer package
> which offers a much faster indexing concept. I implemented
> this package because my app coudn't handle the implicit default
> indexing handling. The README.txt explains what is different and how
> to use it. It defently needs some work to implement a z3c.indexer
> based indexing concept but if you do it right, it can handle a
> lot more indexing tasks then the generic zope or zc.catalog
> concept.
>
>

Thanks a lot. But is it easy to use this from within a non-Zope
application? In the examples I could see references to several zope.site
classes, such as LocalSiteManager... since this is a standalone app, I
wonder if it won't force us to import some other zope packages...

Pedro

>> Pedro
>> _______________________________________________
>> For more information about ZODB, see the ZODB Wiki:
>> http://www.zope.org/Wikis/ZODB/
>>
>> ZODB-Dev mailing list - ZODB-Dev@...
>> https://mail.zope.org/mailman/listinfo/zodb-dev
>>
>>
>
>

Re: Data.fs size grows non-stop

2009-12-10T05:29:38Z

Hi Pedro

> Betreff: Re: [ZODB-Dev] Data.fs size grows non-stop
>
>
> > Why not store the events on some BTree or similar structure with a
> > simple unique integer id, and use a separate BTree to map dates to
> > these events? You can even use zc.catalog or similar tools
> to create
> > that mapping - you would get a lot of query options for
> free with those as well.
> >
> >
> Yeah, that would require a lot of changes, but it would be
> definitely great. What does zc.catalog have that zope.catalog
> doesn't, by the way?

If you need performance, take a look at the z3c.indexer package
which offers a much faster indexing concept. I implemented
this package because my app coudn't handle the implicit default
indexing handling. The README.txt explains what is different and how
to use it. It defently needs some work to implement a z3c.indexer
based indexing concept but if you do it right, it can handle a
lot more indexing tasks then the generic zope or zc.catalog
concept.

Here are some performance data:

z3c.indexer
indexer based indexing time: 2.47 s

zope.catalog
catalog based indexing time: 14.41 s

btw,
You can run the performance test by yourself
located in the z3c.indexer package.

Regards
Roger Ineichen

> Pedro
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list - ZODB-Dev@...
> https://mail.zope.org/mailman/listinfo/zodb-dev
>

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)

2009-12-10T03:57:25Z

On Thu, Dec 10, 2009 at 11:08:25AM +0000, Chris Withers wrote:

> Marius Gedminas wrote:
> > * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well
> > as other kinds of storages, I suppose) is mandatory. In ZODB 3.9
> > it's gone.
> >
> > Solved by peeking into the method signature with inspect.getargspec()
> > and supplying a version only if it's needed.
>
> I believe this may be fixed with 3.9.4 is released.
> ...but it sounds like your code isn't ending up using the Z303 adapter,
> which is why you're running into the problem.

What's Z303?

> How are you mixing versions? 3.8 on client and 3.9 on server or?

I'm not mixing versions at runtime; my intent is to write code that
works with either ZODB 3.8.x or 3.9.x.

You can see it here in all its glory:
http://bazaar.launchpad.net/~zodbbrowser-dev/zodbbrowser/trunk/annotate/head:/src/zodbbrowser/history.py#L30

http://bazaar.launchpad.net/~zodbbrowser-dev/zodbbrowser/trunk/annotate/head:/src/zodbbrowser/state.py#L115

> (sorry, I lost the previous thread and saw this as a new one when Jim
> changed the subject line...)

This is a new topic, completely unrelated to the previous thread.

Marius Gedminas
--
I'm a shareware signature! Send $2 if you use me, $10 for a manual.

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

signature.asc (196 bytes) Download Attachment

Re: ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)

2009-12-10T03:08:25Z

Marius Gedminas wrote:
> * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well
> as other kinds of storages, I suppose) is mandatory. In ZODB 3.9
> it's gone.
>
> Solved by peeking into the method signature with inspect.getargspec()
> and supplying a version only if it's needed.

I believe this may be fixed with 3.9.4 is released.
...but it sounds like your code isn't ending up using the Z303 adapter,
which is why you're running into the problem.

How are you mixing versions? 3.8 on client and 3.9 on server or?
(sorry, I lost the previous thread and saw this as a new one when Jim
changed the subject line...)

cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)

2009-12-10T02:58:11Z

On Wed, Dec 09, 2009 at 01:04:03PM -0500, Jim Fulton wrote:
> On Wed, Dec 9, 2009 at 12:06 PM, Marius Gedminas <marius@...> wrote:
> ...
> > (Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very
> > ugly hacks I managed.)
>
> This sounds like something that needs to be fixed. Can you share some of the
> issues you ran into? (Or maybe file bugs reports.)

Two issues only:

* In ZODB 3.8 PersistentDict and PersistentMapping are unrelated
classes, so if I want to have adapters for both, I have to register
them separately in ZCML. In ZODB 3.9, PersistentDict is an alias
for PersistentMapping, so I get a ZCML configuration conflict error
if I have adapter directives for both.

Solved by defining a decoy class for one of the adapters so they
don't conflict.

* In ZODB 3.8, the 'version' argument of ClientStorage.history (as well
as other kinds of storages, I suppose) is mandatory. In ZODB 3.9
it's gone.

Solved by peeking into the method signature with inspect.getargspec()
and supplying a version only if it's needed.

> In particular, I think it should be a goal that it isn't too hard to
> write code that works with both ZODB 3.8 and 3.9.

Normal code doesn't usually play with storage.history, I'd say, so I'm
not too peeved by having to do various contortions.

Marius Gedminas
--
Win32-API is wonderful : Full of things to wonder about...

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

signature.asc (196 bytes) Download Attachment

Re: Data.fs size grows non-stop

2009-12-10T01:52:23Z

On 2009-12-10 10:48, Pedro Ferreira wrote:
>
>> Why not store the events on some BTree or similar structure with a
>> simple unique integer id, and use a separate BTree to map dates to
>> these events? You can even use zc.catalog or similar tools to create
>> that mapping - you would get a lot of query options for free with
>> those as well.
>>
> Yeah, that would require a lot of changes, but it would be definitely
> great. What does zc.catalog have that zope.catalog doesn't, by the way?

I'm not sure to be honest. There is a whole plethora of options:
zope.catalog, zc.catalog, repoze.catalog, hurry.query, and possibly
others as well.

Wichert.

--
Wichert Akkerman <wichert@...> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Data.fs size grows non-stop

2009-12-10T01:48:01Z

> Why not store the events on some BTree or similar structure with a
> simple unique integer id, and use a separate BTree to map dates to these
> events? You can even use zc.catalog or similar tools to create that
> mapping - you would get a lot of query options for free with those as well.
>
>
Yeah, that would require a lot of changes, but it would be definitely
great. What does zc.catalog have that zope.catalog doesn't, by the way?

Pedro
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list - ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Data.fs size grows non-stop

2009-12-10T01:11:11Z

Thanks!
But I still get an error:

pferreir
/usr/lib/python2.6/site-packages/ZODB3-3.9.0-py2.6-linux-i686.egg$
zodbbrowser ~/indico/db/last.fs
Traceback (most recent call last):
File "/usr/bin/zodbbrowser", line 8, in <module>
load_entry_point('zodbbrowser==0.6.1', 'console_scripts',
'zodbbrowser')()
File
"/usr/lib/python2.6/site-packages/zodbbrowser-0.6.1-py2.6.egg/zodbbrowser/standalone.py",
line 179, in main
configure(options)
File
"/usr/lib/python2.6/site-packages/zodbbrowser-0.6.1-py2.6.egg/zodbbrowser/standalone.py",
line 87, in configure
context = xmlconfig.string(options.site_definition, context=context)
File
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py",
line 664, in string
processxmlfile(f, context)
File
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py",
line 378, in processxmlfile
parser.parse(src)
File "/usr/lib/python2.6/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python2.6/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/lib/python2.6/xml/sax/expatreader.py", line 207, in feed
self._parser.Parse(data, isFinal)
File "/usr/lib/python2.6/xml/sax/expatreader.py", line 349, in
end_element_ns
self._cont_handler.endElementNS(pair, None)
File
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py",
line 357, in endElementNS
self.context.end()
File
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/config.py",
line 537, in end
self.stack.pop().finish()
File
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/config.py",
line 684, in finish
args = toargs(context, *self.argdata)
File
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/config.py",
line 1376, in toargs
args[str(name)] = field.fromUnicode(s)
File
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/fields.py",
line 141, in fromUnicode
raise schema.ValidationError(v)
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "<string>",
line 4.10-4.72
ConfigurationError: ('Invalid value for', 'package', 'ImportError:
Module zope.app has no global securitypolicy')

weird...

Marius Gedminas wrote:

> On Wed, Dec 09, 2009 at 06:09:36PM +0200, Marius Gedminas wrote:
>
>> On Wed, Dec 09, 2009 at 02:42:58PM +0100, Pedro Ferreira wrote:
>>
>>> Hello,
>>>
>>>> Just zodbbrowser with no prefix:
>>>>
>>>> http://pypi.python.org/pypi/zodbbrowser
>>>> https://launchpad.net/zodbbrowser
>>>>
>>>> It's a web-app: it can connect to your ZEO server so you can inspect the
>>>> DB while it's being used.
>>>>
>>>>
>>> We tried this, but we currently get an error related with the general
>>> security policy for zope.app. Maybe we need to install Zope?
>>>
>> No, it should be self-contained.
>>
>> I'm usually testing with the Zope 3.4 KGS.
>>
>
> I've just released zodbbrowser 0.6.1 which works with ZODB 3.9.x and
> latest versions of other packages too.
>
> (Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very
> ugly hacks I managed.)
>
> Marius Gedminas
>