zidestore 100%

View: New views
9 Messages — Rating Filter:   Alert me  

zidestore 100%

by buzzdeee :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi ogo users,      
     
retrieving the contacts list from ogo via zidestore in the kaddressbook takes  
 
about 5 up to ten minutes for 1500 contacts. The zidestore occupies nearly the  
 
whole processing power, running with 95+ percent. This horribly slows down the
whole machine, including the webui. Especially in the morning, when the users
start to work.
   
I am using ogo-zidestore-1.0beta.2-r1412.0  
 
Any hints on how to investigate this issue?
 
 
kind regards
Sebastian
 
--      
Sebastian Reitenbach            Tel.: ++49-(0)3381-8904-305          
RapidEye AG                     Fax: ++49-(0)3381-8904-101          
Friedrich-Franz-Str. 19         e-mail:reitenbach@...          
D-14770 Brandenburg             web:http://www.rapideye.de       

--
OpenGroupware.org Users
users@...
http://mail.opengroupware.org/mailman/listinfo/users

Re: zidestore 100%

by Helge Hess :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Nov 22, 2005, at 12:08, Sebastian Reitenbach wrote:
> retrieving the contacts list from ogo via zidestore in the
> kaddressbook takes about 5 up to ten minutes for 1500 contacts. The
> zidestore occupies nearly the  whole processing power, running with
> 95+ percent. This horribly slows down the whole machine, including the
> webui. Especially in the morning, when the users start to work.
>
> I am using ogo-zidestore-1.0beta.2-r1412.0
>
> Any hints on how to investigate this issue?

The main problem is that Kontact doesn't seem to cache the contacts
properly.

The initial load of the cache can indeed be that high with GroupDAV
because the client will issue 1 PROPFIND to discover the contacts and
then 1500 individual HTTP GET requests.
Of course this is supposed to happen only *once* per client and
therefore considered reasonable (subsequent starts of the application
are supposed to fetch changed records only based on the etag
information).

ZideLook is a bit more efficient here, it fetches up to 64? contacts in
one GET (so ~25 GETs insetad of 1500). But it uses a non-standard HTTP
hack which is why it isn't in GroupDAV.

Greets,
   Helge
--
http://docs.opengroupware.org/Members/helge/
OpenGroupware.org

--
OpenGroupware.org Users
users@...
http://mail.opengroupware.org/mailman/listinfo/users

Parent Message unknown Re: zidestore 100%

by buzzdeee :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi,  
 
 
users@... wrote:  

> On Nov 22, 2005, at 12:08, Sebastian Reitenbach wrote:  
> > retrieving the contacts list from ogo via zidestore in the  
> > kaddressbook takes about 5 up to ten minutes for 1500 contacts. The  
> > zidestore occupies nearly the  whole processing power, running with  
> > 95+ percent. This horribly slows down the whole machine, including the  
> > webui. Especially in the morning, when the users start to work.  
> >  
> > I am using ogo-zidestore-1.0beta.2-r1412.0  
> >  
> > Any hints on how to investigate this issue?  
>  
> The main problem is that Kontact doesn't seem to cache the contacts  
> properly.  
>  
kontact (kaddressbook) doesn't cache the contacts at all, after deactivating  
and reactivating, the whole list is retrieved again.  
created a bug report a while ago, please vote if you have some time...  
http://bugs.kde.org/votes.cgi?action=show_bug&bug_id=114920 
 
> The initial load of the cache can indeed be that high with GroupDAV  
> because the client will issue 1 PROPFIND to discover the contacts and  
> then 1500 individual HTTP GET requests.  
> Of course this is supposed to happen only *once* per client and  
> therefore considered reasonable (subsequent starts of the application  
> are supposed to fetch changed records only based on the etag  
> information).  
yes, makes sense, unfortunately kaddressbook doesn't cache,...  
 
>  
> ZideLook is a bit more efficient here, it fetches up to 64? contacts in  
> one GET (so ~25 GETs insetad of 1500). But it uses a non-standard HTTP  
> hack which is why it isn't in GroupDAV.  
what is this for a hack, why is it not working with HTTP?  
so the propfind is first going to retrieve the list of (new/changed) contacts  
and will then download them. What is the problem of requesting and downloading  
64 or howmany ever, contacts in a row?  
 
kind regards
Sebastian
 
 
--  
Sebastian Reitenbach            Tel.: ++49-(0)3381-8904-305      
RapidEye AG                     Fax: ++49-(0)3381-8904-101      
Friedrich-Franz-Str. 19         e-mail:reitenbach@...      
D-14770 Brandenburg             web:http://www.rapideye.de   
 
 

--
OpenGroupware.org Users
users@...
http://mail.opengroupware.org/mailman/listinfo/users

Re: zidestore 100%

by Helge Hess :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On 22. Nov 2005, at 16:59 Uhr, Sebastian Reitenbach wrote:

>> ZideLook is a bit more efficient here, it fetches up to 64?  
>> contacts in
>> one GET (so ~25 GETs insetad of 1500). But it uses a non-standard  
>> HTTP
>> hack which is why it isn't in GroupDAV.
> what is this for a hack, why is it not working with HTTP?
> so the propfind is first going to retrieve the list of (new/
> changed) contacts
> and will then download them. What is the problem of requesting and  
> downloading
> 64 or howmany ever, contacts in a row?

Thats simple: HTTP doesn't have a method to request the content of  
multiple URLs in one step.
One would need to introduce additional non-base-HTTP methods to  
support that (eg CalDAV does this with REPORTs), which is against the  
idea of GroupDAV.

ZideLook just encodes multiple objects in a single URL, eg
   PROPFIND /Calendar/10203_10238_12837_1277_19271

Greets,
   Helge
--
http://docs.opengroupware.org/Members/helge/
OpenGroupware.org

--
OpenGroupware.org Users
users@...
http://mail.opengroupware.org/mailman/listinfo/users

Parent Message unknown Re: zidestore 100%

by buzzdeee :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi,  
 
at least I found a reason why the webui was so horribly slow the last days.  
The zidestore was not really the problem. There seems to be a memory leak in  
OGo1.0Beta2 webui, which I am using right now.  
I updated two or three weeks ago, and it ran since then. Today I found the  
memory usage of the webui was at around 45% of 2GB RAM. After restarting it,  
it was back to at about 2-3 % of RAM.  
 
 
users@... wrote:  

> On 22. Nov 2005, at 16:59 Uhr, Sebastian Reitenbach wrote:  
> >> ZideLook is a bit more efficient here, it fetches up to 64?    
> >> contacts in  
> >> one GET (so ~25 GETs insetad of 1500). But it uses a non-standard    
> >> HTTP  
> >> hack which is why it isn't in GroupDAV.  
> > what is this for a hack, why is it not working with HTTP?  
> > so the propfind is first going to retrieve the list of (new/  
> > changed) contacts  
> > and will then download them. What is the problem of requesting and    
> > downloading  
> > 64 or howmany ever, contacts in a row?  
>  
> Thats simple: HTTP doesn't have a method to request the content of    
> multiple URLs in one step.  
> One would need to introduce additional non-base-HTTP methods to    
> support that (eg CalDAV does this with REPORTs), which is against the    
> idea of GroupDAV.  
>  
> ZideLook just encodes multiple objects in a single URL, eg  
>    PROPFIND /Calendar/10203_10238_12837_1277_19271  
>  
thanks for the insights, does this is allowed/specified in the GroupDav
protocol? As I now now that it was the webui slowing down itself, and not the
zidestore, i can live with it as it is (well, a caching mechanism in the
kaddressbook would be really useful, but that's another story...).  
 
 
kind regards
Sebastian
 
--  
Sebastian Reitenbach            Tel.: ++49-(0)3381-8904-305      
RapidEye AG                     Fax: ++49-(0)3381-8904-101      
Friedrich-Franz-Str. 19         e-mail:reitenbach@...      
D-14770 Brandenburg             web:http://www.rapideye.de   

--
OpenGroupware.org Users
users@...
http://mail.opengroupware.org/mailman/listinfo/users

Re: zidestore 100%

by Olivier Hallot :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Yes...  there are bugs related to this.

We use to rotate our logs nightly, and in the meantime we restart the
service (pre-rotate to stop and post rotate to start).

See http://docs.opengroupware.org/Members/olivier/ogo-log.html/view

You may need to do some adjustments wrt your distro.

Olivier

Sebastian Reitenbach wrote:

>Hi,  
>  
>at least I found a reason why the webui was so horribly slow the last days.  
>The zidestore was not really the problem. There seems to be a memory leak in  
>OGo1.0Beta2 webui, which I am using right now.  
>I updated two or three weeks ago, and it ran since then. Today I found the  
>memory usage of the webui was at around 45% of 2GB RAM. After restarting it,  
>it was back to at about 2-3 % of RAM.  
>
>  
>users@... wrote:  
>  
>
>>On 22. Nov 2005, at 16:59 Uhr, Sebastian Reitenbach wrote:  
>>    
>>
>>>>ZideLook is a bit more efficient here, it fetches up to 64?    
>>>>contacts in  
>>>>one GET (so ~25 GETs insetad of 1500). But it uses a non-standard    
>>>>HTTP  
>>>>hack which is why it isn't in GroupDAV.  
>>>>        
>>>>
>>>what is this for a hack, why is it not working with HTTP?  
>>>so the propfind is first going to retrieve the list of (new/  
>>>changed) contacts  
>>>and will then download them. What is the problem of requesting and    
>>>downloading  
>>>64 or howmany ever, contacts in a row?  
>>>      
>>>
>>  
>>Thats simple: HTTP doesn't have a method to request the content of    
>>multiple URLs in one step.  
>>One would need to introduce additional non-base-HTTP methods to    
>>support that (eg CalDAV does this with REPORTs), which is against the    
>>idea of GroupDAV.  
>>  
>>ZideLook just encodes multiple objects in a single URL, eg  
>>   PROPFIND /Calendar/10203_10238_12837_1277_19271  
>>  
>>    
>>
>thanks for the insights, does this is allowed/specified in the GroupDav
>protocol? As I now now that it was the webui slowing down itself, and not the
>zidestore, i can live with it as it is (well, a caching mechanism in the
>kaddressbook would be really useful, but that's another story...).  
>
>
>kind regards
>Sebastian
>  
>--  
>Sebastian Reitenbach            Tel.: ++49-(0)3381-8904-305      
>RapidEye AG                     Fax: ++49-(0)3381-8904-101      
>Friedrich-Franz-Str. 19         e-mail:reitenbach@...      
>D-14770 Brandenburg             web:http://www.rapideye.de   
>
>  
>

--
Olivier Hallot
Scinergy Consulting
Tel (021) 8822-8812
Rio de Janeiro, Brasil
http://www.scinergy.com.br


--
OpenGroupware.org Users
users@...
http://mail.opengroupware.org/mailman/listinfo/users

Parent Message unknown Re: zidestore 100%

by buzzdeee :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi,  
users@... wrote:  
> Yes...  there are bugs related to this.  
>  
> We use to rotate our logs nightly, and in the meantime we restart the  
> service (pre-rotate to stop and post rotate to start).  
>  
> See http://docs.opengroupware.org/Members/olivier/ogo-log.html/view 
>  
> You may need to do some adjustments wrt your distro.  
 
thanks a lot, thought about restarting the webui with a cron job, but while
log rotating is also a great idea. Well, not a perfect solution, but a usable
workaround.  
 
kind regards  
Sebastian  
 
--  
Sebastian Reitenbach            Tel.: ++49-(0)3381-8904-305      
RapidEye AG                     Fax: ++49-(0)3381-8904-101      
Friedrich-Franz-Str. 19         e-mail:reitenbach@...      
D-14770 Brandenburg             web:http://www.rapideye.de   

--
OpenGroupware.org Users
users@...
http://mail.opengroupware.org/mailman/listinfo/users

Re: zidestore 100%

by Olivier Hallot :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

I got the same issue

http://mail.opengroupware.org/pipermail/users/2005-November/014901.html

I hope we get a beta3 with that fixed soon.

Olivier

Sebastian Reitenbach wrote:

>Hi ogo users,      
>      
>retrieving the contacts list from ogo via zidestore in the kaddressbook takes  
>  
>about 5 up to ten minutes for 1500 contacts. The zidestore occupies nearly the  
>  
>whole processing power, running with 95+ percent. This horribly slows down the
>whole machine, including the webui. Especially in the morning, when the users
>start to work.
>  
>I am using ogo-zidestore-1.0beta.2-r1412.0  
>  
>Any hints on how to investigate this issue?
>
>
>kind regards
>Sebastian
>
>--      
>Sebastian Reitenbach            Tel.: ++49-(0)3381-8904-305          
>RapidEye AG                     Fax: ++49-(0)3381-8904-101          
>Friedrich-Franz-Str. 19         e-mail:reitenbach@...          
>D-14770 Brandenburg             web:http://www.rapideye.de       
>
>  
>

--
Olivier Hallot
Scinergy Consulting
Tel (021) 8822-8812
Rio de Janeiro, Brasil
http://www.scinergy.com.br


--
OpenGroupware.org Users
users@...
http://mail.opengroupware.org/mailman/listinfo/users

Re: zidestore 100%

by Olivier Hallot :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

sorry, not meant for this thread.

Olivier Hallot wrote:

> I got the same issue
>
> http://mail.opengroupware.org/pipermail/users/2005-November/014901.html
>
> I hope we get a beta3 with that fixed soon.
>
> Olivier
>
> Sebastian Reitenbach wrote:
>
>> Hi ogo users,           retrieving the contacts list from ogo via
>> zidestore in the kaddressbook takes    
>> about 5 up to ten minutes for 1500 contacts. The zidestore occupies
>> nearly the  
>> whole processing power, running with 95+ percent. This horribly slows
>> down the whole machine, including the webui. Especially in the
>> morning, when the users start to work.   I am using
>> ogo-zidestore-1.0beta.2-r1412.0  
>> Any hints on how to investigate this issue?
>>
>> kind regards Sebastian
>> --       Sebastian Reitenbach            Tel.:
>> ++49-(0)3381-8904-305          RapidEye AG                     Fax:
>> ++49-(0)3381-8904-101          Friedrich-Franz-Str. 19        
>> e-mail:reitenbach@...           D-14770
>> Brandenburg             web:http://www.rapideye.de     
>>  
>>
>

--
Olivier Hallot
Scinergy Consulting
Tel (021) 8822-8812
Rio de Janeiro, Brasil
http://www.scinergy.com.br


--
OpenGroupware.org Users
users@...
http://mail.opengroupware.org/mailman/listinfo/users