« Return to Thread: [RFC] Plugin specifc capabilities II
Re: [RFC] Plugin specifc capabilities II
by Daniel Gollub-2
::
Rate this Message:
On Wednesday 01 April 2009 08:51:32 am Patrick Ohly wrote:
[...]
> > We have OSyncArchive - which cover the API to access the database which
> > archive/cache the full/entire entries of a format.
> >
> > CREATE TABLE tbl_archive (objtype VARCHAR(64), mappingid INTEGER, data
> > BLOB, PRIMARY KEY (objtype, mappingid) )
> >
> > This is how the internal table looks like to store the full/entire
> > entries.
> >
> > Reading this - SQL query i just realize that my previous statements were
> > wrong. For each mapping there is only one "cache". Does this answer your
> > question?
>
> Not quite. What is the "mappingid"?
A "mapping" is an unique map of UIDs of one "logical" entry. (UIDs are only
valid/unique per Peer/Member.)
Example:
A Group of 3 Members have one "logcal" (and same) entry - their native UIDs
are:
Member #1: UID:0001
Member #2: UID:pas@foobar1
Member #3: UID:104
Since all members have native and different UIDs for the same "logical" entry
there is need internally to keep (persistent) a mapping between does different
UIDs. Since you can't lookup/modify/delete UID:pas@foobar1 on Member#1 and
expect that the right entry get touched.
So all this 3 UIDs, which represent one "logical" entry, are mapped. And this
mapping has it's own ID -> "mappingid". Internally we can lookup the mappingid
by just supplying the Member ID, objtype and an UID.
> In the statement above it serves as
> the key which is used to identify an item, together with the type. Who
> chooses the mappingid and what is its namespace? How many instances of
> the table above are created?
long long int osync_mapping_table_get_next_id(OSyncMappingTable *table)
This function gets the next free mappingid.
I hope this answers your question?
There is just one mapping-table (for the entire Group).
The table looks like:
CREATE TABLE tbl_changes (objtype VARCHAR(64) NOT NULL, id INTEGER PRIMARY KEY
AUTOINCREMENT, uid VARCHAR NOT NULL, memberid INTEGER NOT NULL, mappingid
INTEGER NOT NULL, objengine VARCHAR(64) NOT NULL );
INSERT INTO "tbl_changes" VALUES('contact',17,'foo4',2,1,'contact');
INSERT INTO "tbl_changes" VALUES('contact',18,'foo7',2,2,'contact');
INSERT INTO "tbl_changes" VALUES('contact',19,'foo5',2,3,'contact');
INSERT INTO "tbl_changes" VALUES('contact',20,'foo1',2,4,'contact');
INSERT INTO "tbl_changes" VALUES('contact',25,'bar4',1,1,'contact');
INSERT INTO "tbl_changes" VALUES('contact',26,'bar7',1,2,'contact');
INSERT INTO "tbl_changes" VALUES('contact',27,'bar5',1,3,'contact');
INSERT INTO "tbl_changes" VALUES('contact',28,'bar1',1,4,'contact');
[...]
>
> > > At
> > > least according to the Wiki page above, merge plugin's only work with
> > > one particular format, so I suppose conversion plugins are the ones
> > > which deal with two different formats.
> >
> > Right - this is how the current implementation looks like.
> >
> > > How are conversion plugins parameterized?
> >
> > osync_bool conv_func(char *input, unsigned int inpsize,
> > char **output, unsigned int *outpsize,
> > osync_bool *free_input,
> > const char *config,
> > void *userdata,
> > OSyncError **error)
>
> You said elsewhere that this "config" parameter is "free defineable
> config strings which get set in the plugin config in the section about
> format configuration (e.g. <Resource><Formats>). This part of the plugin
> config can be set during the discover phase()."
>
> If it is "free defineable" (which I take as "defined by the conversion
> function" as the user of the data), how can anyone else fill in
> meaningful data? The sync UI wouldn't know the format, nor would the
> sync plugin that discovers a device.
Some sync plugins do. There are only few plugins which really have to deal
with all kind of formats - e.g. SyncML and file plugin. The file plugin just
encapsulates all the different format and doesn't need to understand it at
all.
So today there is only the SyncML plugin/protocol which is really completely
format independet. Most other plugins really now there are only 2 or 4
different objtypes/content-types in one or two different format each. And they
don't have any interface to get more information about the used format. Since
it's hardcoded in the device/application. So those also hardcode it in their
default plugin configuration.
Example evolution plugin:
<Resource>
<Enabled>1</Enabled>
<Formats>
<Format>
<Config>VCARD_EXTENSION=Evolution</Config>
<Name>vcard21</Name>
</Format>
<Format>
<Config>VCARD_EXTENSION=Evolution</Config>
<Name>vcard30</Name>
</Format>
</Formats>
<ObjType>contact</ObjType>
<Url>default</Url>
</Resource>
<Config/> would get supplied to the conversion function as config parameter.
Today only the vformat plugin - which provdes conversion of vformats to
xmlformat - only interprets VCARD_EXTENSION=.
This config can be set also during runtime - in the discover phase.
E.g. we _could_ (just a bad example) also introduce an option
"DEVICE_CLOCK=LOCALTIME;" to deal with floating timestamp conversion or stuff
like that. Or "HOOK_WORKAROUND_VENDOR_BUG=vendor_3rd_address_gets_lost".
Who can set this?
Plugin Authors by shipping a default plugin config which already sets this.
The plugin discover function can set this config option during dynamically
(e.g. if it detects a certain device).
The format plugins and their converters needs to be patched to handle those
special config option.
>
> But there's a bigger disconnect between the current design and
> real-world systems which use capabilities: in SyncML, capabilities are
> based on the format transmitted via SyncML, for example vCard 2.1.
Right - but there are tons of SyncML mobiles which don't provide detailed
capabilities information. Often it's just the fieldname . No parameter, no
maxoccurs, no size, no encoding, no ...
I could imagine - and i hope - SyncML server do more heavy use of CTCap. But
again - SyncML is one of many protocols/interfaces which provide this nice
features. So we still need provide fixed interfaces for other plugins to deal
with less luxurious protocols.
> A
> merger/demerger which only knows about the common internal format cannot
> do anything with that information.
Right - it cannot deal directly deal with it. Some translation would be
required.
>
> You would have to write a merger/demerger for every "foreign" format
Right. But so far i see no possibility to avoid this. Since the merging and
demerging only can be done if the those entry is in the "forgein"/"common"
fomrat - and not in the native format [1].
I fully agree that it would be easier and more helpful if their would be a
possibility to merge/demerge if the format would stay in the "native" format.
But currently i'm not seeing how this could be done without dataloss. I
revisit this again - maybe i missed something...
Do you have any idea how merge/demerge in the "native" format would be
possible without dataloss?
Do you have any other better proposal?
> and
> then merging/demerging might as well be done as part of the format
> conversion - except that the current APIs don't allow that.
No. In current proposal the converter would already get a stripped/demerged
entry. So the converter should not do any further stripping/demerging of
information.
Or do you have any certain example in mind - where merging and demerging in
the conversion is required?
>
> > Do you want to add OSyncCapabilities pointer as well?
>
> Yes, but in the current design the sync plugins don't provide that
> information.
Which information? The capabilities? Sure they do - who else assembles the
capabilities?
> They only provide capabilities for the merger/demerger.
Yes - not quite sure if you read the last paragraphs of my previous reply. But
we could add to the OSyncCapabilities*, which so far only stored format
capabilities, also device specific capabilities (e.g. UTC, NumberOfChanges,
...), and pass a pointer to the conversion function as well.
>
> > Two OSyncCapabilities* - for the source and destination peer?
>
> No. Only for the less capable "foreign" format. By design the internal
> format must be as capable as all formats it is converted from/into.
Right.
[1] http://opensync.org/wiki/trunk/features/mergerFormatConversion
Best Regards,
Daniel
------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel
[...]
> > We have OSyncArchive - which cover the API to access the database which
> > archive/cache the full/entire entries of a format.
> >
> > CREATE TABLE tbl_archive (objtype VARCHAR(64), mappingid INTEGER, data
> > BLOB, PRIMARY KEY (objtype, mappingid) )
> >
> > This is how the internal table looks like to store the full/entire
> > entries.
> >
> > Reading this - SQL query i just realize that my previous statements were
> > wrong. For each mapping there is only one "cache". Does this answer your
> > question?
>
> Not quite. What is the "mappingid"?
A "mapping" is an unique map of UIDs of one "logical" entry. (UIDs are only
valid/unique per Peer/Member.)
Example:
A Group of 3 Members have one "logcal" (and same) entry - their native UIDs
are:
Member #1: UID:0001
Member #2: UID:pas@foobar1
Member #3: UID:104
Since all members have native and different UIDs for the same "logical" entry
there is need internally to keep (persistent) a mapping between does different
UIDs. Since you can't lookup/modify/delete UID:pas@foobar1 on Member#1 and
expect that the right entry get touched.
So all this 3 UIDs, which represent one "logical" entry, are mapped. And this
mapping has it's own ID -> "mappingid". Internally we can lookup the mappingid
by just supplying the Member ID, objtype and an UID.
> In the statement above it serves as
> the key which is used to identify an item, together with the type. Who
> chooses the mappingid and what is its namespace? How many instances of
> the table above are created?
long long int osync_mapping_table_get_next_id(OSyncMappingTable *table)
This function gets the next free mappingid.
I hope this answers your question?
There is just one mapping-table (for the entire Group).
The table looks like:
CREATE TABLE tbl_changes (objtype VARCHAR(64) NOT NULL, id INTEGER PRIMARY KEY
AUTOINCREMENT, uid VARCHAR NOT NULL, memberid INTEGER NOT NULL, mappingid
INTEGER NOT NULL, objengine VARCHAR(64) NOT NULL );
INSERT INTO "tbl_changes" VALUES('contact',17,'foo4',2,1,'contact');
INSERT INTO "tbl_changes" VALUES('contact',18,'foo7',2,2,'contact');
INSERT INTO "tbl_changes" VALUES('contact',19,'foo5',2,3,'contact');
INSERT INTO "tbl_changes" VALUES('contact',20,'foo1',2,4,'contact');
INSERT INTO "tbl_changes" VALUES('contact',25,'bar4',1,1,'contact');
INSERT INTO "tbl_changes" VALUES('contact',26,'bar7',1,2,'contact');
INSERT INTO "tbl_changes" VALUES('contact',27,'bar5',1,3,'contact');
INSERT INTO "tbl_changes" VALUES('contact',28,'bar1',1,4,'contact');
[...]
>
> > > At
> > > least according to the Wiki page above, merge plugin's only work with
> > > one particular format, so I suppose conversion plugins are the ones
> > > which deal with two different formats.
> >
> > Right - this is how the current implementation looks like.
> >
> > > How are conversion plugins parameterized?
> >
> > osync_bool conv_func(char *input, unsigned int inpsize,
> > char **output, unsigned int *outpsize,
> > osync_bool *free_input,
> > const char *config,
> > void *userdata,
> > OSyncError **error)
>
> You said elsewhere that this "config" parameter is "free defineable
> config strings which get set in the plugin config in the section about
> format configuration (e.g. <Resource><Formats>). This part of the plugin
> config can be set during the discover phase()."
>
> If it is "free defineable" (which I take as "defined by the conversion
> function" as the user of the data), how can anyone else fill in
> meaningful data? The sync UI wouldn't know the format, nor would the
> sync plugin that discovers a device.
Some sync plugins do. There are only few plugins which really have to deal
with all kind of formats - e.g. SyncML and file plugin. The file plugin just
encapsulates all the different format and doesn't need to understand it at
all.
So today there is only the SyncML plugin/protocol which is really completely
format independet. Most other plugins really now there are only 2 or 4
different objtypes/content-types in one or two different format each. And they
don't have any interface to get more information about the used format. Since
it's hardcoded in the device/application. So those also hardcode it in their
default plugin configuration.
Example evolution plugin:
<Resource>
<Enabled>1</Enabled>
<Formats>
<Format>
<Config>VCARD_EXTENSION=Evolution</Config>
<Name>vcard21</Name>
</Format>
<Format>
<Config>VCARD_EXTENSION=Evolution</Config>
<Name>vcard30</Name>
</Format>
</Formats>
<ObjType>contact</ObjType>
<Url>default</Url>
</Resource>
<Config/> would get supplied to the conversion function as config parameter.
Today only the vformat plugin - which provdes conversion of vformats to
xmlformat - only interprets VCARD_EXTENSION=.
This config can be set also during runtime - in the discover phase.
E.g. we _could_ (just a bad example) also introduce an option
"DEVICE_CLOCK=LOCALTIME;" to deal with floating timestamp conversion or stuff
like that. Or "HOOK_WORKAROUND_VENDOR_BUG=vendor_3rd_address_gets_lost".
Who can set this?
Plugin Authors by shipping a default plugin config which already sets this.
The plugin discover function can set this config option during dynamically
(e.g. if it detects a certain device).
The format plugins and their converters needs to be patched to handle those
special config option.
>
> But there's a bigger disconnect between the current design and
> real-world systems which use capabilities: in SyncML, capabilities are
> based on the format transmitted via SyncML, for example vCard 2.1.
Right - but there are tons of SyncML mobiles which don't provide detailed
capabilities information. Often it's just the fieldname . No parameter, no
maxoccurs, no size, no encoding, no ...
I could imagine - and i hope - SyncML server do more heavy use of CTCap. But
again - SyncML is one of many protocols/interfaces which provide this nice
features. So we still need provide fixed interfaces for other plugins to deal
with less luxurious protocols.
> A
> merger/demerger which only knows about the common internal format cannot
> do anything with that information.
Right - it cannot deal directly deal with it. Some translation would be
required.
>
> You would have to write a merger/demerger for every "foreign" format
Right. But so far i see no possibility to avoid this. Since the merging and
demerging only can be done if the those entry is in the "forgein"/"common"
fomrat - and not in the native format [1].
I fully agree that it would be easier and more helpful if their would be a
possibility to merge/demerge if the format would stay in the "native" format.
But currently i'm not seeing how this could be done without dataloss. I
revisit this again - maybe i missed something...
Do you have any idea how merge/demerge in the "native" format would be
possible without dataloss?
Do you have any other better proposal?
> and
> then merging/demerging might as well be done as part of the format
> conversion - except that the current APIs don't allow that.
No. In current proposal the converter would already get a stripped/demerged
entry. So the converter should not do any further stripping/demerging of
information.
Or do you have any certain example in mind - where merging and demerging in
the conversion is required?
>
> > Do you want to add OSyncCapabilities pointer as well?
>
> Yes, but in the current design the sync plugins don't provide that
> information.
Which information? The capabilities? Sure they do - who else assembles the
capabilities?
> They only provide capabilities for the merger/demerger.
Yes - not quite sure if you read the last paragraphs of my previous reply. But
we could add to the OSyncCapabilities*, which so far only stored format
capabilities, also device specific capabilities (e.g. UTC, NumberOfChanges,
...), and pass a pointer to the conversion function as well.
>
> > Two OSyncCapabilities* - for the source and destination peer?
>
> No. Only for the less capable "foreign" format. By design the internal
> format must be as capable as all formats it is converted from/into.
Right.
[1] http://opensync.org/wiki/trunk/features/mergerFormatConversion
Best Regards,
Daniel
------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel
« Return to Thread: [RFC] Plugin specifc capabilities II
| Free embeddable forum powered by Nabble | Forum Help |
