OGo needs CTag Support

[Helge Hess]

Hi,

to work well with large xyzDAV folders, OGo ZideStore should implement  
the ctag.

Everytime the ctag changes, the client will perform the big and slow  
PROPFIND, but only then.


I wonder whether returning the current value of the global pkey  
sequence is a sufficient start. The theory being, that any change in  
OGo triggers a log entry, which in turn bumps the pkey sequence value  
(is this true when we delete items?).


Obviously a per-folder counter would be better. (one for tasks,  
appointments, persons and contacts).

Like:
   CREATE TABLE ctags (
     table VARCHAR NOT NULL,
     ctag  INTEGER NOT NULL DEFAULT 0
   );

and then on each and every change (do not forget phone/adr/extattr  
tables)

   UPDATE ctags SET ctag = ctag + 1 WHERE table = 'person';

Could be done by a trigger I guess? Hm. Putting that into the code  
would be better.

Greets,
   Helge
--
Helge Hess
http://helgehess.eu/
--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Wed, 2009-06-10 at 17:43 +0200, Helge Hess wrote:
> Hi,
> to work well with large xyzDAV folders, OGo ZideStore should implement  
> the ctag.
> Everytime the ctag changes, the client will perform the big and slow  
> PROPFIND, but only then.
> I wonder whether returning the current value of the global pkey  
> sequence is a sufficient start. The theory being, that any change in  
> OGo triggers a log entry, which in turn bumps the pkey sequence value  
> (is this true when we delete items?).

It does if tombstone is enabled;  at least for company objects and
tasks.  Adding another operation is pretty easy.

> Obviously a per-folder counter would be better. (one for tasks,  
> appointments, persons and contacts).
> Like:
>    CREATE TABLE ctags (
>      table VARCHAR NOT NULL,
>      ctag  INTEGER NOT NULL DEFAULT 0
>    );
> and then on each and every change (do not forget phone/adr/extattr  
> tables)
>    UPDATE ctags SET ctag = ctag + 1 WHERE table = 'person';
> Could be done by a trigger I guess? Hm. Putting that into the code  
> would be better.

It would seem easy enough.  I think it would be easy to implement a
command that just bumped the ctag based on the entity.

[ctx runCommand:@"object::increment-ctag", @"object", obj, nil]

Then just
UPDATE ctags SET ctag = ctag + 1 WHERE table = [obj entityName];

?

The "ctag" is unique to the folder correct (not across all folders)?  So
the same ctag could be used for /Contacts and /public/Contacts without
an issue?  It just merely needs to change (increment) local to the
folder?

--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Helge Hess]

On 10.06.2009, at 18:58, Adam Tauno WIlliams wrote:
> The "ctag" is unique to the folder correct (not across all  
> folders)?  So
> the same ctag could be used for /Contacts and /public/Contacts without
> an issue?

Yes. Well, the issue is that both will be resynced even if only one of  
them changed. But that should be OK.

> It just merely needs to change (increment) local to the folder?


Yes.

Greets,
   Helge
--
Helge Hess
http://helgehess.eu/
--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Wed, 2009-06-10 at 19:17 +0200, Helge Hess wrote:
> On 10.06.2009, at 18:58, Adam Tauno WIlliams wrote:
> > The "ctag" is unique to the folder correct (not across all  
> > folders)?  So
> > the same ctag could be used for /Contacts and /public/Contacts without
> > an issue?
> Yes. Well, the issue is that both will be resynced even if only one of  
> them changed. But that should be OK.

Better than no ctag support;  and if there is a better idea later...

> > It just merely needs to change (increment) local to the folder?
> Yes.



--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Wed, 2009-06-10 at 13:26 -0400, Adam Tauno Williams wrote:

> On Wed, 2009-06-10 at 19:17 +0200, Helge Hess wrote:
> > On 10.06.2009, at 18:58, Adam Tauno WIlliams wrote:
> > > The "ctag" is unique to the folder correct (not across all  
> > > folders)?  So
> > > the same ctag could be used for /Contacts and /public/Contacts without
> > > an issue?
> > Yes. Well, the issue is that both will be resynced even if only one of  
> > them changed. But that should be OK.
> Better than no ctag support;  and if there is a better idea later...
> > > It just merely needs to change (increment) local to the folder?
> > Yes.

CREATE TABLE ctags (
  entity VARCHAR NOT NULL,
  ctag  INTEGER NOT NULL DEFAULT 0
);

INSERT INTO ctags (entity) VALUES ('Person');
INSERT INTO ctags (entity) VALUES ('Enterprise');
INSERT INTO ctags (entity) VALUES ('Date');
INSERT INTO ctags (entity) VALUES ('Job');
INSERT INTO ctags (entity) VALUES ('Team');

And something like -

#import "common.h"
#import "LSIncrementCTagCommand.h"

@implementation LSIncrementCTagCommand

- (void)_executionInContext:(id)_context {
  EOAdaptorChannel    *eoChannel;
  NSString            *sql;

  if ([[self object] isNotNull])
  {
    sql = [[NSString alloc] initWithFormat:@"UPDATE ctags"
                                           @" SET ctag = (ctag + 1)"
                                           @" WHERE entity = '%@';",
                                             [[self object]
entityName]];

    eoChannel = [[self databaseChannel] adaptorChannel];
    if (![eoChannel evaluateExpression:sql]) {
      [self logWithFormat:@"failed to update ctag for entity %@",
                        [[self object] entityName]];
    }
    [sql release];
  }
}

@end


--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Helge Hess]

On 10.06.2009, at 19:49, Adam Tauno Williams wrote:

> On Wed, 2009-06-10 at 13:26 -0400, Adam Tauno Williams wrote:
>> On Wed, 2009-06-10 at 19:17 +0200, Helge Hess wrote:
>>> On 10.06.2009, at 18:58, Adam Tauno WIlliams wrote:
>>>> The "ctag" is unique to the folder correct (not across all
>>>> folders)?  So
>>>> the same ctag could be used for /Contacts and /public/Contacts  
>>>> without
>>>> an issue?
>>> Yes. Well, the issue is that both will be resynced even if only  
>>> one of
>>> them changed. But that should be OK.
>> Better than no ctag support;  and if there is a better idea later...
>>>> It just merely needs to change (increment) local to the folder?
>>> Yes.
>
> CREATE TABLE ctags (
>  entity VARCHAR NOT NULL,
>  ctag  INTEGER NOT NULL DEFAULT 0
> );
>
> INSERT INTO ctags (entity) VALUES ('Person');
> INSERT INTO ctags (entity) VALUES ('Enterprise');
> INSERT INTO ctags (entity) VALUES ('Date');
> INSERT INTO ctags (entity) VALUES ('Job');
> INSERT INTO ctags (entity) VALUES ('Team');
>
> And something like -
>
> #import "common.h"
> #import "LSIncrementCTagCommand.h"
>
> @implementation LSIncrementCTagCommand
>
> - (void)_executionInContext:(id)_context {
>  EOAdaptorChannel    *eoChannel;
>  NSString            *sql;
>
>  if ([[self object] isNotNull])
>  {
>    sql = [[NSString alloc] initWithFormat:@"UPDATE ctags"
>                                           @" SET ctag = (ctag + 1)"
>                                           @" WHERE entity = '%@';",
>                                             [[self object]
> entityName]];
>
>    eoChannel = [[self databaseChannel] adaptorChannel];
>    if (![eoChannel evaluateExpression:sql]) {
>      [self logWithFormat:@"failed to update ctag for entity %@",
>                        [[self object] entityName]];
>    }
>    [sql release];
>  }
> }
>
> @end

Looks good.

Helge
--
Helge Hess
http://helgehess.eu/
--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Wed, 2009-06-10 at 19:58 +0200, Helge Hess wrote:
> On 10.06.2009, at 19:49, Adam Tauno Williams wrote:
> > @end
> Looks good.

In r2256.  And added to the company::set command for testing;  obviously
needs to be added other places, will take care of that.

--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Wed, 2009-06-10 at 14:18 -0400, Adam Tauno Williams wrote:
> On Wed, 2009-06-10 at 19:58 +0200, Helge Hess wrote:
> > On 10.06.2009, at 19:49, Adam Tauno Williams wrote:
> > > @end
> > Looks good.
> In r2256.  And added to the company::set command for testing;  obviously
> needs to be added other places, will take care of that.

Would it be preferred to add -

  LSRunCommandV(_context, @"object", @"increment-ctag",
                @"object",      [self object],
                nil);

- to all necessary commands or to add a -

- (void)incrementCTag:(id)_object
                  inContext:(LSCommandContext *)_ctx
{
  LSRunCommandV(_ctx, @"object", @"increment-ctag",
                @"object", _object,
                nil);
}

- to LSDBObjectDeleteCommand, LSDBObjectNewCommand, and
LSDBObjectSetCommand and just call [self incrementCTag] in the necessary
commands. ?

--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Helge Hess]

On 10.06.2009, at 22:21, Adam Tauno Williams wrote:

> Would it be preferred to add -
>
>  LSRunCommandV(_context, @"object", @"increment-ctag",
>                @"object",      [self object],
>                nil);
>
> - to all necessary commands or to add a -
>
> - (void)incrementCTag:(id)_object
>                  inContext:(LSCommandContext *)_ctx
> {
>  LSRunCommandV(_ctx, @"object", @"increment-ctag",
>                @"object", _object,
>                nil);
> }
>
> - to LSDBObjectDeleteCommand, LSDBObjectNewCommand, and
> LSDBObjectSetCommand and just call [self incrementCTag] in the  
> necessary
> commands. ?

I guess I wouldn't even make it a command but rather make it a regular  
method on LSDBObjectBaseCommand.

'increment' is no good name though, it should be 'recalculate' or  
something like that, in case we want to have non-sequential ctags later.

Thanks,
   Helge
--
Helge Hess
http://helgehess.eu/



--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Wed, 2009-06-10 at 22:57 +0200, Helge Hess wrote:
> On 10.06.2009, at 22:21, Adam Tauno Williams wrote:
> > - to LSDBObjectDeleteCommand, LSDBObjectNewCommand, and
> > LSDBObjectSetCommand and just call [self incrementCTag] in the  
> > necessary
> > commands. ?
> I guess I wouldn't even make it a command but rather make it a regular  
> method on LSDBObjectBaseCommand.
> 'increment' is no good name though, it should be 'recalculate' or  
> something like that, in case we want to have non-sequential ctags later.

Done in r2259.  Company objects (Person/Enterprise) should update ctags
value for create/update/delete.

--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Wed, 2009-06-10 at 22:23 -0400, Adam Tauno WIlliams wrote:

> On Wed, 2009-06-10 at 22:57 +0200, Helge Hess wrote:
> > On 10.06.2009, at 22:21, Adam Tauno Williams wrote:
> > > - to LSDBObjectDeleteCommand, LSDBObjectNewCommand, and
> > > LSDBObjectSetCommand and just call [self incrementCTag] in the  
> > > necessary
> > > commands. ?
> > I guess I wouldn't even make it a command but rather make it a regular  
> > method on LSDBObjectBaseCommand.
> > 'increment' is no good name though, it should be 'recalculate' or  
> > something like that, in case we want to have non-sequential ctags later.
> Done in r2259.  Company objects (Person/Enterprise) should update ctags
> value for create/update/delete.

r2260: Job and Date ctags update


--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Dirk Bartley]

Did the subversion url change??


[dbartley@mta1 opengroupware]$ svn info
http://svn.opengroupware.org/OpenGroupware.org/trunk
Path: trunk
URL: http://svn.opengroupware.org/OpenGroupware.org/trunk
Repository Root: http://svn.opengroupware.org/OpenGroupware.org
Repository UUID: fda58e92-02e2-0310-b1e4-b2b32738dd10
Revision: 2249

2249 < 2260

Dirk

On Thu, 2009-06-11 at 06:38 -0400, Adam Tauno WIlliams wrote:

> On Wed, 2009-06-10 at 22:23 -0400, Adam Tauno WIlliams wrote:
> > On Wed, 2009-06-10 at 22:57 +0200, Helge Hess wrote:
> > > On 10.06.2009, at 22:21, Adam Tauno Williams wrote:
> > > > - to LSDBObjectDeleteCommand, LSDBObjectNewCommand, and
> > > > LSDBObjectSetCommand and just call [self incrementCTag] in the  
> > > > necessary
> > > > commands. ?
> > > I guess I wouldn't even make it a command but rather make it a regular  
> > > method on LSDBObjectBaseCommand.
> > > 'increment' is no good name though, it should be 'recalculate' or  
> > > something like that, in case we want to have non-sequential ctags later.
> > Done in r2259.  Company objects (Person/Enterprise) should update ctags
> > value for create/update/delete.
>
> r2260: Job and Date ctags update
>
>

--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Thu, 2009-06-11 at 08:16 -0400, Dirk Bartley wrote:
> Did the subversion url change??
> [dbartley@mta1 opengroupware]$ svn info
> http://svn.opengroupware.org/OpenGroupware.org/trunk
> Path: trunk
> URL: http://svn.opengroupware.org/OpenGroupware.org/trunk
> Repository Root: http://svn.opengroupware.org/OpenGroupware.org
> Repository UUID: fda58e92-02e2-0310-b1e4-b2b32738dd10
> Revision: 2249
> 2249 < 2260

The public SVN lags the developer SVN by some period of time.

--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Wed, 2009-06-10 at 22:57 +0200, Helge Hess wrote:

> On 10.06.2009, at 22:21, Adam Tauno Williams wrote:
> > Would it be preferred to add -
> >  LSRunCommandV(_context, @"object", @"increment-ctag",
> >                @"object",      [self object],
> >                nil);
> > - to all necessary commands or to add a -
> > - (void)incrementCTag:(id)_object
> >                  inContext:(LSCommandContext *)_ctx
> > {
> >  LSRunCommandV(_ctx, @"object", @"increment-ctag",
> >                @"object", _object,
> >                nil);
> > }
> > - to LSDBObjectDeleteCommand, LSDBObjectNewCommand, and
> > LSDBObjectSetCommand and just call [self incrementCTag] in the  
> > necessary
> > commands. ?
> I guess I wouldn't even make it a command but rather make it a regular  
> method on LSDBObjectBaseCommand.
> 'increment' is no good name though, it should be 'recalculate' or  
> something like that, in case we want to have non-sequential ctags later.

As for ctag [presentation] support in ZideStore, is there a recommended
implementation strategy?  Should ctag be a property of SxFolder+DAV?

--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Helge Hess]

On 18.06.2009, at 13:44, Adam Tauno WIlliams wrote:
>> I guess I wouldn't even make it a command but rather make it a  
>> regular
>> method on LSDBObjectBaseCommand.
>> 'increment' is no good name though, it should be 'recalculate' or
>> something like that, in case we want to have non-sequential ctags  
>> later.
> As for ctag [presentation] support in ZideStore, is there a  
> recommended
> implementation strategy?  Should ctag be a property of SxFolder+DAV?

Yes, I think so. It should already have a mapped name in SOPE  
(DAVPropMap.plist), probably davCTag.

Thanks,
   Helge
--
Helge Hess
http://helgehess.eu/
--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Thu, 2009-06-18 at 14:20 +0200, Helge Hess wrote:

> On 18.06.2009, at 13:44, Adam Tauno WIlliams wrote:
> >> I guess I wouldn't even make it a command but rather make it a  
> >> regular
> >> method on LSDBObjectBaseCommand.
> >> 'increment' is no good name though, it should be 'recalculate' or
> >> something like that, in case we want to have non-sequential ctags  
> >> later.
> > As for ctag [presentation] support in ZideStore, is there a  
> > recommended
> > implementation strategy?  Should ctag be a property of SxFolder+DAV?
>
> Yes, I think so. It should already have a mapped name in SOPE  
> (DAVPropMap.plist), probably davCTag.

So a -
- (NSString *)davCTag {
  return @"BLAH";
}
- will 'magically' fill a request for prop ctag (assuming the tag is
mapped)?

But:

$  grep -i ctag sope/sope-appserver/NGObjWeb/DAVPropMap.plist
  "{http://calendarserver.org/ns/}getctag" = davCollectionTag;

That is in the /* Apple CalServer */ section.

That is the only DAVPropMap.plist I can find.

I'd expect a
"{DAV:}ctag"       = "davCTag";

yes?
--
OpenGroupware developer: awilliam@...
<http://whitemiceconsulting.blogspot.com/>
OpenGroupare & Cyrus IMAPd documenation @
<http://docs.opengroupware.org/Members/whitemice/wmogag/file_view>

--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Helge Hess]

On 18.06.2009, at 14:31, Adam Tauno Williams wrote:
> So a -
> - (NSString *)davCTag {
>  return @"BLAH";
> }
> - will 'magically' fill a request for prop ctag (assuming the tag is
> mapped)?

Unless some method overrides the request, yes. It would be -
davCollectionTag, see below.

> But:
>
> $  grep -i ctag sope/sope-appserver/NGObjWeb/DAVPropMap.plist
>  "{http://calendarserver.org/ns/}getctag" = davCollectionTag;
>
> That is in the /* Apple CalServer */ section.
>
> That is the only DAVPropMap.plist I can find.
>
> I'd expect a
> "{DAV:}ctag"       = "davCTag";
>
> yes?

No. ctag is an Apple extension.

Greets,
   Helge
--
Helge Hess
http://helgehess.eu/
--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

Should I be able to
propget Contacts getctag
- or -
propget Contacts {http://calendarserver.org/ns/}getctag
 - and see the value of SxFolder.davCollectionTag

It just says -
Fetching properties for `Contacts':
Server did not return result for getctag
- in either case.

I tried adding davCollectionTag to "davCollectionTag" and
"CadaverListSet" to ZideStore/ZSFrontend/DAVPropSets.plist (which gets
installed to /usr/share/zidestore-5.5/DAVPropSets.plist)

--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Helge Hess]

On 18.06.2009, at 15:35, Adam Tauno Williams wrote:
> Should I be able to
> propget Contacts getctag
> - or -
> propget Contacts {http://calendarserver.org/ns/}getctag
> - and see the value of SxFolder.davCollectionTag

On Darwin CalendarServer:
---snip---
dav:/calendars/users/helge/> set namespace http://calendarserver.org/ns/
dav:/calendars/users/helge/> propget calendar/ getctag
Fetching properties for `calendar/':
Value of getctag is: 2009-06-18 13:08:52.544120
---snap---

> I tried adding davCollectionTag to "davCollectionTag" and
> "CadaverListSet" to ZideStore/ZSFrontend/DAVPropSets.plist (which gets
> installed to /usr/share/zidestore-5.5/DAVPropSets.plist)

Hm. I'm not 100% sure this is a good idea. Might be :-)

ZideStore attempts to match certain property sets and then use a  
specialized ObjC method to return just that. This might be exactly not  
the thing you want :-) Not 100% sure of the relevant codepath.

Anyways, getctag is part of the regular GroupDAVv2 request, so its  
always requested anyways.

Greets,
   Helge
--
Helge Hess
http://helgehess.eu/
--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer

[Adam Tauno Williams]

On Thu, 2009-06-18 at 09:35 -0400, Adam Tauno Williams wrote:

> Should I be able to
> propget Contacts getctag
> - or -
> propget Contacts {http://calendarserver.org/ns/}getctag
>  - and see the value of SxFolder.davCollectionTag
> It just says -
> Fetching properties for `Contacts':
> Server did not return result for getctag
> - in either case.
> I tried adding davCollectionTag to "davCollectionTag" and
> "CadaverListSet" to ZideStore/ZSFrontend/DAVPropSets.plist (which gets
> installed to /usr/share/zidestore-5.5/DAVPropSets.plist)

Er.  In the first case, according to log, it is looking for the property
in a different namespace.  Which makes sense.

Jun 18 13:41:54 ogo-zidestore-5.5 [15249]:
<<0x0x8f96da4[OLDavPropMapper]>>D query cadaver key:
{http://webdav.org/cadaver/custom-properties/}getctag

And the second just pukes with "<data>: not well formed "

But this works -
dav:/zidestore/dav/adam/> set namespace http://calendarserver.org/ns/
dav:/zidestore/dav/adam/> propget Contacts getctag
Fetching properties for `Contacts':
Value of getctag is: BLAH

Nice.

--
OpenGroupware.org Developer
developer@...
http://mail.opengroupware.org/mailman/listinfo/developer