|
»
Problems packing Zope 2.9.8 storage
|
View:
New views
11 Messages
—
Rating Filter:
Alert me
|
 |
Problems packing Zope 2.9.8 storage
by Chris Withers
::
Rate this Message:
Hi All,
I've got a ZODB that's ballooned and I'm now having problems packing with zeopack. I tried with zeopack, which appeared to just hang (still in top, but no cpu usage for an hour or two), so I ctrl-c'ed it. Now when I try, I just get the text "Already packing" back. There is a .old file there, but it was last modified over a week ago. Looking in FileStorage.py, it appears there's a possibility for an exception to be raised after self._pack_is_in_progress has been set to True but before the try/finally clause that clears self._pack_is_in_progress has been entered. Has anyone experienced this? If so, what should I look to do now? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
|
|
Re: Problems packing Zope 2.9.8 storage
by Shane Hathaway
::
Rate this Message:
Chris Withers wrote:
> I've got a ZODB that's ballooned and I'm now having problems packing > with zeopack. > > I tried with zeopack, which appeared to just hang (still in top, but no > cpu usage for an hour or two), so I ctrl-c'ed it. > > Now when I try, I just get the text "Already packing" back. The ZEO server does all the packing work. The zeopack utility only tells the ZEO server to get started, then waits for it to finish. It probably wasn't hung. > There is a .old file there, but it was last modified over a week ago. The analysis phase of packing can take a long time. A week is too much, though. :-) > If so, what should I look to do now? If I were solving it, I would see what happens if I copy the storage to RelStorage and then pack it. Maybe that's just me though. Shane _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
|
|
Re: Problems packing Zope 2.9.8 storage
by Chris Withers
::
Rate this Message:
Shane Hathaway wrote:
>> There is a .old file there, but it was last modified over a week ago. > > The analysis phase of packing can take a long time. A week is too much, > though. :-) Yeah, this is off the back of an automated pack/backup/unpack/verify to keep two storages in sync on two database servers, one live, one hot standby. The process went wrong because one of the storages has ballooned in size... >> If so, what should I look to do now? > > If I were solving it, I would see what happens if I copy the storage to > RelStorage and then pack it. Maybe that's just me though. I thought RelStorage was slow at packing? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
|
|
Re: Problems packing Zope 2.9.8 storage
by Shane Hathaway
::
Rate this Message:
Chris Withers wrote:
> I thought RelStorage was slow at packing? Not anymore. I optimized packing last December/January and it has been plenty fast ever since, AFAIK. Shane _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
|
|
Re: Problems packing Zope 2.9.8 storage
by Chris Withers
::
Rate this Message:
Shane Hathaway wrote:
> Chris Withers wrote: >> I thought RelStorage was slow at packing? > > Not anymore. I optimized packing last December/January and it has been > plenty fast ever since, AFAIK. Interesting. Got any metrics on its speed compared to FileStorage, including packing? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
|
|
Re: Problems packing Zope 2.9.8 storage
by Shane Hathaway
::
Rate this Message:
Chris Withers wrote:
> Shane Hathaway wrote: >> Chris Withers wrote: >>> I thought RelStorage was slow at packing? >> Not anymore. I optimized packing last December/January and it has been >> plenty fast ever since, AFAIK. > > Interesting. Got any metrics on its speed compared to FileStorage, > including packing? I don't have numbers on pack speed, but I've eliminated all the speed issues people have brought up. People have been looking at these performance charts for quite a while: http://shane.willowrise.com/archives/how-to-fix-the-mysql-write-speed/ http://shane.willowrise.com/archives/relstorage-111-coming-soon/ http://shane.willowrise.com/archives/relstorage-10-and-measurements/ Look at this chart in particular, which suggests that RelStorage with PostgreSQL or MySQL is several times faster than ZEO for heavy read loads: http://shane.willowrise.com/wp-content/uploads/2009/03/relstorage-113-speed-10k.png I am waiting for someone to dispute these results. I *want* ZEO to be faster, since its code paths are simpler than any relational database, but the performance measurements suggest that under pressure, RelStorage with MySQL or PostgreSQL can read 3 objects in the time it takes ZEO with FileStorage to read 1. Shane _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
|
|
Re: Problems packing Zope 2.9.8 storage
by Benji York-2
::
Rate this Message:
On Mon, Sep 14, 2009 at 2:44 PM, Shane Hathaway <shane@...> wrote:
> People have been looking at these performance charts for quite a while: Are the benchmarks publicly available? I looked through the blog posts and svn.zope.org but couldn't find them. -- Benji York Senior Software Engineer Zope Corporation _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
|
|
Re: Problems packing Zope 2.9.8 storage
by Shane Hathaway
::
Rate this Message:
Benji York wrote:
> On Mon, Sep 14, 2009 at 2:44 PM, Shane Hathaway <shane@...> wrote: >> People have been looking at these performance charts for quite a while: > > Are the benchmarks publicly available? I looked through the blog posts > and svn.zope.org but couldn't find them. http://svn.zope.org/relstorage/trunk/relstorage/tests/speedtest.py?view=markup That script generates CSV, which I paste into this spreadsheet to produce the graphs: http://svn.zope.org/relstorage/trunk/relstorage/tests/comparison.ods Shane _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
|
|
Re: Problems packing Zope 2.9.8 storage
by Benji York-2
::
Rate this Message:
On Mon, Sep 14, 2009 at 3:52 PM, Shane Hathaway <shane@...> wrote:
> Benji York wrote: >> >> On Mon, Sep 14, 2009 at 2:44 PM, Shane Hathaway <shane@...> >> wrote: >>> >>> People have been looking at these performance charts for quite a while: >> >> Are the benchmarks publicly available? I looked through the blog posts >> and svn.zope.org but couldn't find them. [snip answer] Thanks! -- Benji York Senior Software Engineer Zope Corporation _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
|
|
Re: Problems packing Zope 2.9.8 storage
by Chris Withers
::
Rate this Message:
Shane Hathaway wrote:
> Look at this chart in particular, which suggests that RelStorage with > PostgreSQL or MySQL is several times faster than ZEO for heavy read loads: Cool, now all I need is something like zeoraid for postgres (or, at a push, mysql)... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
|
|
Re: Problems packing Zope 2.9.8 storage
by Shane Hathaway
::
Rate this Message:
Chris Withers wrote:
> Shane Hathaway wrote: >> Look at this chart in particular, which suggests that RelStorage with >> PostgreSQL or MySQL is several times faster than ZEO for heavy read loads: > > Cool, now all I need is something like zeoraid for postgres (or, at a > push, mysql)... There are existing replication solutions for PostgreSQL and I am fairly confident at least one of them will work with RelStorage. It's a simple schema, after all. I know RelStorage replication works with MySQL. Shane _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@... https://mail.zope.org/mailman/listinfo/zodb-dev |
| Free embeddable forum powered by Nabble | Forum Help |
