Re: [RFC] Plugin specifc capabilities II

On Tuesday 31 March 2009 04:14:32 pm Patrick Ohly wrote:
> On Tue, 2009-03-31 at 13:22 +0200, Daniel Gollub wrote:
> > [1] http://opensync.org/wiki/trunk/features/mergerFormatConversion
> >
> >From a didactic point of view I suggest to structure the page
>
> differently: describe the desired solution first, then for those who
> disagree with it explain why other solutions don't work.

It's a wiki - feel free to change it ;)

>
> I can't say I understand all of the tables that explain why they don't
> work, but as the proposed solution mirrors what I believe is right (all
> comparison/merging must be done in a common format) I won't argue about
> that ;-)

Sorry - i didn't had more time to explain why the others are not work. I'll
add later why they're working. Thanks for the feedback.

>
> So let's focus on the final solution. Can you add information and/or
> explain more clearly what kind of input is needed by the "merge" and
> "demerge" steps? What is their output besides the "format C" items?


typedef osync_bool (* OSyncFormatMergeFunc) (char **data, unsigned int *size,
const char *entire, unsigned int entsize, OSyncCapabilities *caps, void
*user_data, OSyncError **error);

The merge get with parameter "data" the original reported change - which get
modified. This change is already in the format which the merger is registered.
Parameter "entire" is the cached entry, from previous sync which is complete,
in the same format. OSyncCapabilities* are the plugin specific reported
capabilities. The merger  uses this information to determine if a field is
support and if it should get merged or not.

The result is in "data" - a merged entry which has the entire set of fields -
which weren't supported by the member/device.

Today the OSyncCapabilities would be only field with "xmlformat" capabilities
format.

typedef osync_bool (* OSyncFormatDemergeFunc) (char **data, unsigned int
*size, OSyncCapabilities *caps, void *user_data, OSyncError **error);

Similar as merge - parameter "data" contains the original reported change data
in the demerger specific format. All fields not mentioned in
OSyncCapabilities* get deleted. (The original entry got already cached in teh
Archive before the format plugin fucntion "demerge" gets called - plugin is
not responsible for caching - no dataloss!). Result is in "data" - a stripped
entry which only contains fields which are supported by the target
member/peer/device.

>
> In particular: how many caches are needed?
What do you mean by "caches" in detail?

We have OSyncArchive - which cover the API to access the database which
archive/cache the full/entire entries of a format.

CREATE TABLE tbl_archive (objtype VARCHAR(64), mappingid INTEGER, data BLOB,
PRIMARY KEY (objtype, mappingid) )

This is how the internal table looks like to store the full/entire entries.

Reading this - SQL query i just realize that my previous statements were
wrong. For each mapping there is only one "cache". Does this answer your
question?

This one cache for all members should be fine - since the changes from all
members get first converted into the "common format". So the cache is usable
for all members as well. Without a "common format" between all members
synchronization wouldn't complete anyway.

>
> > [2] http://opensync.org/wiki/trunk/features/mergerCapabilities
>
> Allowing arbitrary combinations of OSyncObjFormat and capabilities
> description looks nice in theory; in practice I don't expect that there
> will be many mergers that really support many arbitrary combinations
> The API can be left like it is,

Why API the current? Or the proposed one from [2]?

> but sync plugin authors need to know
> which combinations they can count on.

The idea actually is that sync plugin use the one which already exist (e.g.
xmlformat). Or they write their own format plugin with merger/demerge
functions. (Like in the example from the wiki [2])

> In particular, both item format
> and the corresponding capabilities format must be defined well because
> many people will use and depend on them.

Right - and here i see the problem. That for instance the "xmlforamt-caps" is
not defined well enough for all cases. For that reason there is now the
possibility to write plugins for their "native capabilities format".

>
> Merge plugins and conversion plugins are different beasts, right?

Yes and no ;)

So actually we have one thing called "format plugin" - this does multiple jobs
at the same time:
 - registering an objformat
   * setting objformat name and objtype
   * setting compare func
   * setting revision func
   * setting merge/demerge func
   * ...
 - conversion of objformat A to objformat B
   * just setting conversion functions

You can create format-plugins which just do one of the two tasks. E.g. vformat
plugin is actually splitted in serveral shared objects. vcard.so, vevent.so,
.. and vformat-xmlformat.so. The first shared objects just register the
objformat "vcard21", "vcard30", "vevent20" and so on. The later one - vformat-
xmlformat.so - consists only of the converters from/to xmlformats from/to the
vformats.

Today the merge/demerge function is part of the "regtistering an objformat".
In my proposal i thought about adding an interface to seperate this so there
would be the possibility just to write a "merge/demerge" format-plugin.
(Without registering a new objformat - which would require a conversion
function to be able to convert to other common formats)

> At
> least according to the Wiki page above, merge plugin's only work with
> one particular format, so I suppose conversion plugins are the ones
> which deal with two different formats.

Right - this is how the current implementation looks like.


>
> How are conversion plugins parameterized?



osync_bool conv_func(char *input, unsigned int inpsize,
                    char **output, unsigned int *outpsize,
                    osync_bool *free_input,
                    const char *config,
                    void *userdata,
                    OSyncError **error)

Do you want to add OSyncCapabilities pointer as well?
Two OSyncCapabilities* - for the source and destination peer?
So plugins could optional look up certain capabilities which have impact on
conversion....

>
> Like merging they need information about both sides of the conversion.
> To give just one example, in SyncML the server is told whether its
> client supports UTC or relies on times in local time. This information
> would be important when converting between the client's format
> (vCalendar 1.0) and the internal, common format (some kind of more
> general event description).

Right. But this is something different. This are "device specific
capabilities" which have influence on the conversion. The merger/demerger
tasks are just handling of "format capabilities" - e.g. in SyncML speech
CTCaps. CTCaps is on the same level as UTC, NumberOfChanges, ...

So far it was planmed to use the "const char *config" parameter to pass this
UTC information to the conversion option (or format extensions). The config
parameter are free defineable config strings which get set in the plugin
config in the section about format configuration (e.g. <Resource><Formats>).
This part of the plugin config can be set during the discover phase().

For the syncml plugin we planned to _add_ an interface so the plugin can
report the CTCaps and other DevInfs during the regular sync. In detail we
would add osync_context_report_capabilities() which would be handled inside
connect(), connect_done() and get_changes() context. As soon the  SyncML DS
package contains the DevInf it should report OSyncCapabilities*.

Actually we could really mix OSyncCapabilies* with Format capabilities (e.g.
CTCap) and device inf capabilities (UTC, NumberOfChanges, ...)

But this would require that we do this "arbitrary combinations of
OSyncObjFormat and capabilities description" change. The current
implementation require that OSyncCapabilities* get only filled with XMLFormat
field names.


Hope this helps.

Best Regards,
Daniel

------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel