[RFC] Plugin specifc capabilities II

[Daniel Gollub-2]

Hi all,

as requested from me several times - i reviewed the entire Merger and
Capabilities process.

First thing which caused quite a lot of confusion: XMLFormat will not be
discontinued! We just try to support several "common formats" at the same
time.

"Common Formats " (like XMLFormat) are a really critical think to OpenSync.
Without a "common format" it's not easily possible to do comparing to
"changes" with different capabilities.

Regarding the current capabilities handling in OpenSync and the implementation
of Merger/Demerger and where the conversion has to happen. I did some design
proposal [1] which should cover all possibilities and comes up with a final
implementation proposal (which is not the current implementation in /trunk).

Coming back to the initial topic: Supporting several "common formats"

The current design only would allow to support one format per objtype (which
are currently obviously the XMLFormats). I hope i found a solution for this
and hopefully documented this clear enough in the wiki [2].


Please give me feedback on this! This is one of the last things i want to
change for 0.39 - since this would require some tiny API changes. Once this is
set i'm more then happy to tag and release 0.39.

Since everyone wants really to see 0.40 - please make sure that we come very
quickly on working solution for everyone ASAP.

Thanks a lot for reading!

(And sorry for my super bad writing - it's 1pm and i haven't had breakfast
yet.... )

[1] http://opensync.org/wiki/trunk/features/mergerFormatConversion
[2] http://opensync.org/wiki/trunk/features/mergerCapabilities

Best Regards,
Daniel

------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Patrick Ohly]

On Tue, 2009-03-31 at 13:22 +0200, Daniel Gollub wrote:
> [1] http://opensync.org/wiki/trunk/features/mergerFormatConversion

>From a didactic point of view I suggest to structure the page
differently: describe the desired solution first, then for those who
disagree with it explain why other solutions don't work.

I can't say I understand all of the tables that explain why they don't
work, but as the proposed solution mirrors what I believe is right (all
comparison/merging must be done in a common format) I won't argue about
that ;-)

So let's focus on the final solution. Can you add information and/or
explain more clearly what kind of input is needed by the "merge" and
"demerge" steps? What is their output besides the "format C" items?

In particular: how many caches are needed?

> [2] http://opensync.org/wiki/trunk/features/mergerCapabilities

Allowing arbitrary combinations of OSyncObjFormat and capabilities
description looks nice in theory; in practice I don't expect that there
will be many mergers that really support many arbitrary combinations.
The API can be left like it is, but sync plugin authors need to know
which combinations they can count on. In particular, both item format
and the corresponding capabilities format must be defined well because
many people will use and depend on them.

Merge plugins and conversion plugins are different beasts, right? At
least according to the Wiki page above, merge plugin's only work with
one particular format, so I suppose conversion plugins are the ones
which deal with two different formats.

How are conversion plugins parameterized?

Like merging they need information about both sides of the conversion.
To give just one example, in SyncML the server is told whether its
client supports UTC or relies on times in local time. This information
would be important when converting between the client's format
(vCalendar 1.0) and the internal, common format (some kind of more
general event description).

--
Bye, Patrick Ohly
--  
Patrick.Ohly@...
http://www.estamos.de/



------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Daniel Gollub-2]

On Tuesday 31 March 2009 04:14:32 pm Patrick Ohly wrote:
> On Tue, 2009-03-31 at 13:22 +0200, Daniel Gollub wrote:
> > [1] http://opensync.org/wiki/trunk/features/mergerFormatConversion
> >
> >From a didactic point of view I suggest to structure the page
>
> differently: describe the desired solution first, then for those who
> disagree with it explain why other solutions don't work.

It's a wiki - feel free to change it ;)

>
> I can't say I understand all of the tables that explain why they don't
> work, but as the proposed solution mirrors what I believe is right (all
> comparison/merging must be done in a common format) I won't argue about
> that ;-)

Sorry - i didn't had more time to explain why the others are not work. I'll
add later why they're working. Thanks for the feedback.

>
> So let's focus on the final solution. Can you add information and/or
> explain more clearly what kind of input is needed by the "merge" and
> "demerge" steps? What is their output besides the "format C" items?


typedef osync_bool (* OSyncFormatMergeFunc) (char **data, unsigned int *size,
const char *entire, unsigned int entsize, OSyncCapabilities *caps, void
*user_data, OSyncError **error);

The merge get with parameter "data" the original reported change - which get
modified. This change is already in the format which the merger is registered.
Parameter "entire" is the cached entry, from previous sync which is complete,
in the same format. OSyncCapabilities* are the plugin specific reported
capabilities. The merger  uses this information to determine if a field is
support and if it should get merged or not.

The result is in "data" - a merged entry which has the entire set of fields -
which weren't supported by the member/device.

Today the OSyncCapabilities would be only field with "xmlformat" capabilities
format.

typedef osync_bool (* OSyncFormatDemergeFunc) (char **data, unsigned int
*size, OSyncCapabilities *caps, void *user_data, OSyncError **error);

Similar as merge - parameter "data" contains the original reported change data
in the demerger specific format. All fields not mentioned in
OSyncCapabilities* get deleted. (The original entry got already cached in teh
Archive before the format plugin fucntion "demerge" gets called - plugin is
not responsible for caching - no dataloss!). Result is in "data" - a stripped
entry which only contains fields which are supported by the target
member/peer/device.

>
> In particular: how many caches are needed?
What do you mean by "caches" in detail?

We have OSyncArchive - which cover the API to access the database which
archive/cache the full/entire entries of a format.

CREATE TABLE tbl_archive (objtype VARCHAR(64), mappingid INTEGER, data BLOB,
PRIMARY KEY (objtype, mappingid) )

This is how the internal table looks like to store the full/entire entries.

Reading this - SQL query i just realize that my previous statements were
wrong. For each mapping there is only one "cache". Does this answer your
question?

This one cache for all members should be fine - since the changes from all
members get first converted into the "common format". So the cache is usable
for all members as well. Without a "common format" between all members
synchronization wouldn't complete anyway.

>
> > [2] http://opensync.org/wiki/trunk/features/mergerCapabilities
>
> Allowing arbitrary combinations of OSyncObjFormat and capabilities
> description looks nice in theory; in practice I don't expect that there
> will be many mergers that really support many arbitrary combinations
> The API can be left like it is,

Why API the current? Or the proposed one from [2]?

> but sync plugin authors need to know
> which combinations they can count on.

The idea actually is that sync plugin use the one which already exist (e.g.
xmlformat). Or they write their own format plugin with merger/demerge
functions. (Like in the example from the wiki [2])

> In particular, both item format
> and the corresponding capabilities format must be defined well because
> many people will use and depend on them.

Right - and here i see the problem. That for instance the "xmlforamt-caps" is
not defined well enough for all cases. For that reason there is now the
possibility to write plugins for their "native capabilities format".

>
> Merge plugins and conversion plugins are different beasts, right?

Yes and no ;)

So actually we have one thing called "format plugin" - this does multiple jobs
at the same time:
 - registering an objformat
   * setting objformat name and objtype
   * setting compare func
   * setting revision func
   * setting merge/demerge func
   * ...
 - conversion of objformat A to objformat B
   * just setting conversion functions

You can create format-plugins which just do one of the two tasks. E.g. vformat
plugin is actually splitted in serveral shared objects. vcard.so, vevent.so,
.. and vformat-xmlformat.so. The first shared objects just register the
objformat "vcard21", "vcard30", "vevent20" and so on. The later one - vformat-
xmlformat.so - consists only of the converters from/to xmlformats from/to the
vformats.

Today the merge/demerge function is part of the "regtistering an objformat".
In my proposal i thought about adding an interface to seperate this so there
would be the possibility just to write a "merge/demerge" format-plugin.
(Without registering a new objformat - which would require a conversion
function to be able to convert to other common formats)

> At
> least according to the Wiki page above, merge plugin's only work with
> one particular format, so I suppose conversion plugins are the ones
> which deal with two different formats.

Right - this is how the current implementation looks like.


>
> How are conversion plugins parameterized?



osync_bool conv_func(char *input, unsigned int inpsize,
                    char **output, unsigned int *outpsize,
                    osync_bool *free_input,
                    const char *config,
                    void *userdata,
                    OSyncError **error)

Do you want to add OSyncCapabilities pointer as well?
Two OSyncCapabilities* - for the source and destination peer?
So plugins could optional look up certain capabilities which have impact on
conversion....

>
> Like merging they need information about both sides of the conversion.
> To give just one example, in SyncML the server is told whether its
> client supports UTC or relies on times in local time. This information
> would be important when converting between the client's format
> (vCalendar 1.0) and the internal, common format (some kind of more
> general event description).

Right. But this is something different. This are "device specific
capabilities" which have influence on the conversion. The merger/demerger
tasks are just handling of "format capabilities" - e.g. in SyncML speech
CTCaps. CTCaps is on the same level as UTC, NumberOfChanges, ...

So far it was planmed to use the "const char *config" parameter to pass this
UTC information to the conversion option (or format extensions). The config
parameter are free defineable config strings which get set in the plugin
config in the section about format configuration (e.g. <Resource><Formats>).
This part of the plugin config can be set during the discover phase().

For the syncml plugin we planned to _add_ an interface so the plugin can
report the CTCaps and other DevInfs during the regular sync. In detail we
would add osync_context_report_capabilities() which would be handled inside
connect(), connect_done() and get_changes() context. As soon the  SyncML DS
package contains the DevInf it should report OSyncCapabilities*.

Actually we could really mix OSyncCapabilies* with Format capabilities (e.g.
CTCap) and device inf capabilities (UTC, NumberOfChanges, ...)

But this would require that we do this "arbitrary combinations of
OSyncObjFormat and capabilities description" change. The current
implementation require that OSyncCapabilities* get only filled with XMLFormat
field names.


Hope this helps.

Best Regards,
Daniel

------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Patrick Ohly]

On Tue, 2009-03-31 at 17:41 +0200, Daniel Gollub wrote:
> On Tuesday 31 March 2009 04:14:32 pm Patrick Ohly wrote:
> > In particular: how many caches are needed?
> What do you mean by "caches" in detail?

The entity where items are "cached". In other words, a set of ID+data
pairs where the ID is used to extract the data stored there earlier.

> We have OSyncArchive - which cover the API to access the database which
> archive/cache the full/entire entries of a format.
>
> CREATE TABLE tbl_archive (objtype VARCHAR(64), mappingid INTEGER, data BLOB,
> PRIMARY KEY (objtype, mappingid) )
>
> This is how the internal table looks like to store the full/entire entries.
>
> Reading this - SQL query i just realize that my previous statements were
> wrong. For each mapping there is only one "cache". Does this answer your
> question?

Not quite. What is the "mappingid"? In the statement above it serves as
the key which is used to identify an item, together with the type. Who
chooses the mappingid and what is its namespace? How many instances of
the table above are created?

> > > [2] http://opensync.org/wiki/trunk/features/mergerCapabilities
> >
> > Allowing arbitrary combinations of OSyncObjFormat and capabilities
> > description looks nice in theory; in practice I don't expect that there
> > will be many mergers that really support many arbitrary combinations
> > The API can be left like it is,
>
> Why API the current? Or the proposed one from [2]?

The one from [2].

> > At
> > least according to the Wiki page above, merge plugin's only work with
> > one particular format, so I suppose conversion plugins are the ones
> > which deal with two different formats.
>
> Right - this is how the current implementation looks like.
>
>
> >
> > How are conversion plugins parameterized?
>
>
>
> osync_bool conv_func(char *input, unsigned int inpsize,
>                     char **output, unsigned int *outpsize,
>                     osync_bool *free_input,
>                     const char *config,
>                     void *userdata,
>                     OSyncError **error)

You said elsewhere that this "config" parameter is "free defineable
config strings which get set in the plugin config in the section about
format configuration (e.g. <Resource><Formats>). This part of the plugin
config can be set during the discover phase()."

If it is "free defineable" (which I take as "defined by the conversion
function" as the user of the data), how can anyone else fill in
meaningful data? The sync UI wouldn't know the format, nor would the
sync plugin that discovers a device.

But there's a bigger disconnect between the current design and
real-world systems which use capabilities: in SyncML, capabilities are
based on the format transmitted via SyncML, for example vCard 2.1. A
merger/demerger which only knows about the common internal format cannot
do anything with that information.

You would have to write a merger/demerger for every "foreign" format and
then merging/demerging might as well be done as part of the format
conversion - except that the current APIs don't allow that.

> Do you want to add OSyncCapabilities pointer as well?

Yes, but in the current design the sync plugins don't provide that
information. They only provide capabilities for the merger/demerger.

> Two OSyncCapabilities* - for the source and destination peer?

No. Only for the less capable "foreign" format. By design the internal
format must be as capable as all formats it is converted from/into.

--
Bye, Patrick Ohly
--  
Patrick.Ohly@...
http://www.estamos.de/



------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Daniel Gollub-2]

On Wednesday 01 April 2009 08:51:32 am Patrick Ohly wrote:
[...]

> > We have OSyncArchive - which cover the API to access the database which
> > archive/cache the full/entire entries of a format.
> >
> > CREATE TABLE tbl_archive (objtype VARCHAR(64), mappingid INTEGER, data
> > BLOB, PRIMARY KEY (objtype, mappingid) )
> >
> > This is how the internal table looks like to store the full/entire
> > entries.
> >
> > Reading this - SQL query i just realize that my previous statements were
> > wrong. For each mapping there is only one "cache". Does this answer your
> > question?
>
> Not quite. What is the "mappingid"?

A "mapping" is an unique map of UIDs of one "logical" entry. (UIDs are only
valid/unique per Peer/Member.)

Example:

A Group of 3 Members have one "logcal" (and same) entry - their native UIDs
are:
Member #1: UID:0001
Member #2: UID:pas@foobar1
Member #3: UID:104

Since all members have native and different UIDs for the same "logical" entry
there is need internally to keep (persistent) a mapping between does different
UIDs. Since you can't lookup/modify/delete UID:pas@foobar1 on Member#1 and
expect that the right entry get touched.

So all this 3 UIDs, which represent one "logical" entry, are mapped. And this
mapping has it's own ID -> "mappingid". Internally we can lookup the mappingid
by just supplying the Member ID, objtype and an UID.

> In the statement above it serves as
> the key which is used to identify an item, together with the type. Who
> chooses the mappingid and what is its namespace? How many instances of
> the table above are created?

long long int osync_mapping_table_get_next_id(OSyncMappingTable *table)

This function gets the next free mappingid.
I hope this answers your question?

There is just one mapping-table (for the entire Group).

The table looks like:

CREATE TABLE tbl_changes (objtype VARCHAR(64) NOT NULL, id INTEGER PRIMARY KEY
AUTOINCREMENT, uid VARCHAR NOT NULL, memberid INTEGER NOT NULL, mappingid
INTEGER NOT NULL, objengine VARCHAR(64) NOT NULL );

INSERT INTO "tbl_changes" VALUES('contact',17,'foo4',2,1,'contact');
INSERT INTO "tbl_changes" VALUES('contact',18,'foo7',2,2,'contact');
INSERT INTO "tbl_changes" VALUES('contact',19,'foo5',2,3,'contact');
INSERT INTO "tbl_changes" VALUES('contact',20,'foo1',2,4,'contact');
INSERT INTO "tbl_changes" VALUES('contact',25,'bar4',1,1,'contact');
INSERT INTO "tbl_changes" VALUES('contact',26,'bar7',1,2,'contact');
INSERT INTO "tbl_changes" VALUES('contact',27,'bar5',1,3,'contact');
INSERT INTO "tbl_changes" VALUES('contact',28,'bar1',1,4,'contact');

[...]

>
> > > At
> > > least according to the Wiki page above, merge plugin's only work with
> > > one particular format, so I suppose conversion plugins are the ones
> > > which deal with two different formats.
> >
> > Right - this is how the current implementation looks like.
> >
> > > How are conversion plugins parameterized?
> >
> > osync_bool conv_func(char *input, unsigned int inpsize,
> >                     char **output, unsigned int *outpsize,
> >                     osync_bool *free_input,
> >                     const char *config,
> >                     void *userdata,
> >                     OSyncError **error)
>
> You said elsewhere that this "config" parameter is "free defineable
> config strings which get set in the plugin config in the section about
> format configuration (e.g. <Resource><Formats>). This part of the plugin
> config can be set during the discover phase()."
>
> If it is "free defineable" (which I take as "defined by the conversion
> function" as the user of the data), how can anyone else fill in
> meaningful data? The sync UI wouldn't know the format, nor would the
> sync plugin that discovers a device.

Some sync plugins do. There are only few plugins which really have to deal
with all kind of formats - e.g. SyncML and file plugin. The file plugin just
encapsulates all the different format and doesn't need to understand it at
all.

So today there is only the SyncML plugin/protocol which is really completely
format independet. Most other plugins really now there are only 2 or 4
different objtypes/content-types in one or two different format each. And they
don't have any interface to get more information about the used format. Since
it's hardcoded in the device/application. So those also hardcode it in their
default plugin configuration.

Example evolution plugin:

    <Resource>
      <Enabled>1</Enabled>
      <Formats>
        <Format>
          <Config>VCARD_EXTENSION=Evolution</Config>
          <Name>vcard21</Name>
        </Format>
        <Format>
          <Config>VCARD_EXTENSION=Evolution</Config>
          <Name>vcard30</Name>
        </Format>
      </Formats>
      <ObjType>contact</ObjType>
      <Url>default</Url>
    </Resource>

<Config/> would get supplied to the conversion function as config parameter.
Today only the vformat plugin - which provdes conversion of vformats to
xmlformat - only interprets VCARD_EXTENSION=.

This config can be set also during runtime - in the discover phase.

E.g. we _could_ (just a bad example) also introduce an option
"DEVICE_CLOCK=LOCALTIME;" to deal with floating timestamp conversion or stuff
like that. Or "HOOK_WORKAROUND_VENDOR_BUG=vendor_3rd_address_gets_lost".

Who can set this?
Plugin Authors by shipping a default plugin config which already sets this.
The plugin discover function can set this config option during dynamically
(e.g. if it detects a certain device).

The format plugins and their converters needs to be patched to handle those
special config option.

>
> But there's a bigger disconnect between the current design and
> real-world systems which use capabilities: in SyncML, capabilities are
> based on the format transmitted via SyncML, for example vCard 2.1.

Right - but there are tons of SyncML mobiles which don't provide detailed
capabilities information. Often it's just the fieldname . No parameter, no
maxoccurs, no size, no encoding, no ...

I could imagine - and i hope - SyncML server do more heavy use of CTCap. But
again - SyncML is one of many protocols/interfaces which provide this nice
features. So we still need provide fixed interfaces for other plugins to deal
with less luxurious protocols.


> A
> merger/demerger which only knows about the common internal format cannot
> do anything with that information.

Right - it cannot deal directly deal with it. Some translation would be
required.

>
> You would have to write a merger/demerger for every "foreign" format

Right. But so far i see no possibility to avoid this. Since the merging and
demerging only can be done if the those entry is in the "forgein"/"common"
fomrat - and not in the native format [1].

I fully agree that it would be easier and more helpful if their would be a
possibility to merge/demerge if the format would stay in the "native" format.

But currently i'm not seeing how this could be done without dataloss. I
revisit this again - maybe i missed something...

Do you have any idea how merge/demerge in the "native" format would be
possible without dataloss?
Do you have any other better proposal?

> and
> then merging/demerging might as well be done as part of the format
> conversion - except that the current APIs don't allow that.

No. In current proposal the converter would already get a stripped/demerged
entry. So the converter should not do any further stripping/demerging of
information.

Or do you have any certain example in mind - where merging and demerging in
the conversion is required?

>
> > Do you want to add OSyncCapabilities pointer as well?
>
> Yes, but in the current design the sync plugins don't provide that
> information.

Which information? The capabilities? Sure they do - who else assembles the
capabilities?

> They only provide capabilities for the merger/demerger.

Yes - not quite sure if you read the last paragraphs of my previous reply. But
we could add to the OSyncCapabilities*, which so far only stored format
capabilities, also device specific capabilities (e.g. UTC, NumberOfChanges,
...), and pass a pointer to the conversion function as well.

>
> > Two OSyncCapabilities* - for the source and destination peer?
>
> No. Only for the less capable "foreign" format. By design the internal
> format must be as capable as all formats it is converted from/into.

Right.


[1] http://opensync.org/wiki/trunk/features/mergerFormatConversion


Best Regards,
Daniel

------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Patrick Ohly]

On Wed, 2009-04-01 at 10:33 +0200, Daniel Gollub wrote:

> On Wednesday 01 April 2009 08:51:32 am Patrick Ohly wrote:
> [...]
> > In the statement above it serves as
> > the key which is used to identify an item, together with the type. Who
> > chooses the mappingid and what is its namespace? How many instances of
> > the table above are created?
>
> long long int osync_mapping_table_get_next_id(OSyncMappingTable *table)
>
> This function gets the next free mappingid.
> I hope this answers your question?

Yes. The mappingid corresponds to the GUID in a SyncML, with a name
space that is common among all peers synchronizing against this
particular instance of OpenSync.

> > You said elsewhere that this "config" parameter is "free defineable
> > config strings which get set in the plugin config in the section about
> > format configuration (e.g. <Resource><Formats>). This part of the plugin
> > config can be set during the discover phase()."
> >
> > If it is "free defineable" (which I take as "defined by the conversion
> > function" as the user of the data), how can anyone else fill in
> > meaningful data? The sync UI wouldn't know the format, nor would the
> > sync plugin that discovers a device.
>
> Some sync plugins do.

... but they have to agree with the format plugin on the content of the
config. Therefore the config format has to be defined for each format
type, otherwise we could end up with a situation where a config A for
format AF is passed to a format plugin which can deal with AF, but only
config B.

> > A
> > merger/demerger which only knows about the common internal format cannot
> > do anything with that information.
>
> Right - it cannot deal directly deal with it. Some translation would be
> required.

Which might not be possible.

> > You would have to write a merger/demerger for every "foreign" format
>
> Right. But so far i see no possibility to avoid this. Since the merging and
> demerging only can be done if the those entry is in the "forgein"/"common"
> fomrat - and not in the native format [1].
>
> I fully agree that it would be easier and more helpful if their would be a
> possibility to merge/demerge if the format would stay in the "native" format.
>
> But currently i'm not seeing how this could be done without dataloss. I
> revisit this again - maybe i missed something...
>
> Do you have any idea how merge/demerge in the "native" format would be
> possible without dataloss?
> Do you have any other better proposal?

Drop the separate merger/demerger concept. Move that functionality into
the format conversion.

When converting towards OpenSync, the format plugin works with an item
in the "foreign", limited format, a capabilities description that
describes the format used by the peer, and a complete item in the
internal format. The result is an updated internal item with information
from the peer mixed in, or a complete copy of the incoming item when
there is no internal item yet.

When sending towards a peer, the format plugin only needs to pick
properties from the internal item according to the peers capabilities.
"Only read what you need" looks more natural to me than the current
concept of "strip an item, then convert the stripped down version". The
additional advantage is that this conversion has full access to all
available information, which may be an advantage when going beyond a
static capabilities-based conversion towards a scripted one which also
uses dynamic configuration.

> > > Do you want to add OSyncCapabilities pointer as well?
> >
> > Yes, but in the current design the sync plugins don't provide that
> > information.
>
> Which information? The capabilities? Sure they do - who else assembles the
> capabilities?

They can only provide capabilities targeted at either the merger or the
converter, but not both. In the current design they are asked to provide
information for the merger, so conversion cannot use the capabilities
provided e.g. by the SyncML protocol.

--
Best Regards

Patrick Ohly
Senior Software Engineer

Intel GmbH
Software & Solutions Group                
Hermuelheimer Strasse 8a                  Phone: [phone]
50321 Bruehl                              Fax: [phone]
Germany


------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Daniel Gollub-2]

On Wednesday 01 April 2009 01:52:49 pm Patrick Ohly wrote:
[...]

> > > You said elsewhere that this "config" parameter is "free defineable
> > > config strings which get set in the plugin config in the section about
> > > format configuration (e.g. <Resource><Formats>). This part of the
> > > plugin config can be set during the discover phase()."
> > >
> > > If it is "free defineable" (which I take as "defined by the conversion
> > > function" as the user of the data), how can anyone else fill in
> > > meaningful data? The sync UI wouldn't know the format, nor would the
> > > sync plugin that discovers a device.
> >
> > Some sync plugins do.
>
> ... but they have to agree with the format plugin on the content of the
> config. Therefore the config format has to be defined for each format
> type, otherwise we could end up with a situation where a config A for
> format AF is passed to a format plugin which can deal with AF, but only
> config B.

Right.

>
> > > A
> > > merger/demerger which only knows about the common internal format
> > > cannot do anything with that information.
> >
> > Right - it cannot deal directly deal with it. Some translation would be
> > required.
>
> Which might not be possible.

Since there would be the possibility to write sync plugin specific merger
functions i don't see the problem to handle the translation to various common
formats at the same time.

>
> > > You would have to write a merger/demerger for every "foreign" format
> >
> > Right. But so far i see no possibility to avoid this. Since the merging
> > and demerging only can be done if the those entry is in the
> > "forgein"/"common" fomrat - and not in the native format [1].
> >
> > I fully agree that it would be easier and more helpful if their would be
> > a possibility to merge/demerge if the format would stay in the "native"
> > format.
> >
> > But currently i'm not seeing how this could be done without dataloss. I
> > revisit this again - maybe i missed something...
> >
> > Do you have any idea how merge/demerge in the "native" format would be
> > possible without dataloss?
> > Do you have any other better proposal?
>
> Drop the separate merger/demerger concept. Move that functionality into
> the format conversion.

Ok - interesting idea.

>
> When converting towards OpenSync, the format plugin works with an item
> in the "foreign", limited format, a capabilities description that
> describes the format used by the peer, and a complete item in the
> internal format. The result is an updated internal item with information
> from the peer mixed in, or a complete copy of the incoming item when
> there is no internal item yet.
>
> When sending towards a peer, the format plugin only needs to pick
> properties from the internal item according to the peers capabilities.
> "Only read what you need" looks more natural to me than the current
> concept of "strip an item, then convert the stripped down version". The
> additional advantage is that this conversion has full access to all
> available information, which may be an advantage when going beyond a
> static capabilities-based conversion towards a scripted one which also
> uses dynamic configuration.

[...]

Sounds fine to me. I try to prepare a Proof of Concept implementation.
Unfortunately this would require again changes on all format plugins which
have converter. But if this really works out, then this is definitely worth to
delay 0.39 for one more day.

Best Regards,
Daniel

------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Graham Cobb-4]

On Tue, March 31, 2009 12:22, Daniel Gollub wrote:

> as requested from me several times - i reviewed the entire Merger and
> Capabilities process.
>
> Please give me feedback on this! This is one of the last things i want to
> change for 0.39 - since this would require some tiny API changes. Once
> this is
> set i'm more then happy to tag and release 0.39.
>
> Since everyone wants really to see 0.40 - please make sure that we come
> very
> quickly on working solution for everyone ASAP.

Daniel,

I am away on holiday and will not have a chance to review this in any
detail until after Easter. I did glance at the Wiki pages and I am not
sure if this addresses the problems I found while trying to create the
capabilities for GPE: the merger process seems to need to be different for
data being sent to the device from OpenSYnc and data being received from
the device and sent to OpenSync. In particular, there is data which should
be included in the update sent to the device but which should not be
trusted when it is received from the device. This seems to mean that there
are three states neded for each attribute: supported, untrusted (specify
on send, ignore on receive), excluded (do not specify on send, ignore on
receive).

The second problem was granularity: how detailed the capabilities can be
(e.g. if the device can store a home and work address but not the various
different types of address which can be defined using the RFC).  The only
short term API impact of this isue is that it shoud be possible to make
the capability definition files handle more granular capabilities in the
future without breaking the capabilities we define today. So, for example,
the format should allow us to define today that address is untrusted but
tomorrow we may extend the merger capabilities to specify that up to two
addresses are supported.

Sorry I am not in a position to review the proposals.  Also, I do not know
when I will next have access to my mail while I am away, but I will leave
you to think about these!

Graham


------------------------------------------------------------------------------
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Daniel Gollub-3]

On Wednesday 01 April 2009 01:52:49 pm Patrick Ohly wrote:
[...]
> > But currently i'm not seeing how this could be done without dataloss. I
> > revisit this again - maybe i missed something...
> >
> > Do you have any idea how merge/demerge in the "native" format would be
> > possible without dataloss?
> > Do you have any other better proposal?
>
> Drop the separate merger/demerger concept. Move that functionality into
> the format conversion.

I looked into that very very close and actually fully implemented this
approach in OpenSync, vformat plugin and the xmlformat plugin. Now i have some
doubts that this approach has only benefits to the current draft in the wiki
... there are also some downsides:

> When converting towards OpenSync, the format plugin works with an item
> in the "foreign", limited format, a capabilities description that
> describes the format used by the peer, and a complete item in the
> internal format. The result is an updated internal item with information
> from the peer mixed in, or a complete copy of the incoming item when
> there is no internal item yet.

I don't see the reason why this couldn't be done with separating the merge and
conversion function. When i tried to implement/port the format plugins to your
proposal i realized that i would just have to call the already implemented
XMLFormat merge code inside the conversion function - or just do the function
call of it at the very end of the conversion.

Simplified example:

osync_bool convert_a_to_b(char **entry_data, unsigned int *entry_size,
                    const char *cached_data, unsigned int cached_size,
                    OSyncCapabilities *caps, const char *config, OSyncError**)
{

  conversion_routine(entry_data, entry_size);
  merge_routine(entry_data, entry_size, cached_data, cached_size, caps);
  return TRUE;
}

Since merge_routine() would always deal with data from the internal format
this would be redundant code - since the merge routine would be
copied/reimplemented in every format plugin which provides format conversion
towards this particular internal format.

If we would keep it separated this would look internally something like this:
(again very simplified)

void incoming_item_from_plugin(OSyncEngine *engine, OSyncChange *change)
{
  OSyncConverter *converter;
  OSyncObjFormat *a = osync_change_get_objformat(change);
  OSyncObjFormat *b = osync_format_env_find_objformat("common");

  converter = osync_format_env_find_converter(engine->formatenv, a, b);


  /* Conversion to forgein/peer format to common format */
  converter->conv_func(change->data, change->size);
 
  /* Get the cached item (which is in the common format already) */
  char *cached_data;
  unsigned int cached_size;
  osync_archive_load_item(engine->archive, &cached_data, &cached_size);

  /* Merge - based on common format data and caps */
  b->merge_func(change->data, change->size, caps, cached_data, cached_size);

  [.... report it to the engine for mapping ...]
}

The advantage here would be that only the "common format" plugin would need to
implement a merge function. (Which is what we have currently only inside the
"xmlformat" plugin with the current approach)

But i see your point with passing along some further capabilities information
to influence the conversion. Like SyncML DevInf UTC or so. As i stated earlier
mails we could have done this with the "const char *config" parameter. But
yeah there are some arguments against it  ...


> When sending towards a peer, the format plugin only needs to pick
> properties from the internal item according to the peers capabilities.
> "Only read what you need" looks more natural to me than the current
> concept of "strip an item, then convert the stripped down version".

Actually this could be done also with separating merger from conversion
function - but completely drop the "demerge" function. So there would be no
demerge call in advance when the format plugin's conversion function get
called.

I guess the initial idea of the "demerge" plugin was to make the conversion
function very simple and generic. The current conversion function doesn't have
to deal with capabilities and just convert everything what they are able to
convert.

> The
> additional advantage is that this conversion has full access to all
> available information, which may be an advantage when going beyond a
> static capabilities-based conversion towards a scripted one which also
> uses dynamic configuration.

Right - that's a real advantage i can't argue against (at least not yet ;)).

If we would keep the "merge" (not demerge!) function separated from conversion
the function - and just drop entirely the "demerge" function from the concept
- we could also have full access and go beyond static caps conversion.

We would just need to pass the OSyncCapabilities* to the conversion function
argument list - additionally. So the conversion functions to and from the
internal/common format have access to device specific information.

Thoughts? Comments?

Best Regards,
Daniel

--
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: [phone]         Handelsregister:   HRB 3537
EMail: gollub@...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D


------------------------------------------------------------------------------

_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Daniel Gollub-3]

On Thursday 02 April 2009 01:11:06 am Graham Cobb wrote:
[...]

> I am away on holiday and will not have a chance to review this in any
> detail until after Easter. I did glance at the Wiki pages and I am not
> sure if this addresses the problems I found while trying to create the
> capabilities for GPE: the merger process seems to need to be different for
> data being sent to the device from OpenSYnc and data being received from
> the device and sent to OpenSync. In particular, there is data which should
> be included in the update sent to the device but which should not be
> trusted when it is received from the device. This seems to mean that there
> are three states neded for each attribute: supported, untrusted (specify
> on send, ignore on receive), excluded (do not specify on send, ignore on
> receive).

Could you give me an vcard/gpe example for: untrusted and excluded scenario?

>
> The second problem was granularity: how detailed the capabilities can be
> (e.g. if the device can store a home and work address but not the various
> different types of address which can be defined using the RFC).  The only
> short term API impact of this isue is that it shoud be possible to make
> the capability definition files handle more granular capabilities in the
> future without breaking the capabilities we define today. So, for example,
> the format should allow us to define today that address is untrusted but
> tomorrow we may extend the merger capabilities to specify that up to two
> addresses are supported.

I hope we address this with a SyncML like capabilities schema - vcard-Based-
capabilities description:

<Caps objtype="contact">
<Cap>
<DisplayName>Name</DisplayName>
<MaxOccurs>1</MaxOccurs>
<Max>64</Max>
<Min>1</Min>
<Name>N</Name>
<Parameter>
  <Name>ENCODING</Name>
  <Type>string</Type>
  <ValEnum>ACSII</ValEnum>
  <ValEnum>UTF-8</ValEnum>
  <ValEnum>UTF-16</ValEnum>
</Parameter>
<Type>string</Type>
</Cap>
<Cap>
<DisplayName>Telephone</DisplayName>
<MaxOccurs>3</MaxOccurs>
<Max>128</Max>
<Min>1</Min>
<Name>TEL</Name>
<Parameter>
  <Name>ENCODING</Name>
  <Type>string</Type>
  <ValEnum>ASCII</ValEnum>
</Parameter>
<Parameter>
  <Name>TYPE</Name>
  <Type>string</Type>
  <ValEnum>HOME</ValEnum>
  <ValEnum>WORK</ValEnum>
  <ValEnum>CELL</ValEnum>
  <ValEnum>FAX</ValEnum>
  <ValEnum>CELL,WORK</ValEnum>
  <ValEnum>FAX,WORK</ValEnum>
  <ValEnum>PREFERED</ValEnum>
</Parameter>
<Type>string</Type>
</Cap>
</Caps>

(This XML dump will get assembled via an osync_capabilit(ies|
y)_get/set/add_xxx() interface - in the discover function of a sync-plugin)

Best Regards,
Daniel

--
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: [phone]         Handelsregister:   HRB 3537
EMail: gollub@...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Graham Cobb-4]

On Sunday 10 May 2009 20:58:15 Daniel Gollub wrote:

> > I am away on holiday and will not have a chance to review this in any
> > detail until after Easter. I did glance at the Wiki pages and I am not
> > sure if this addresses the problems I found while trying to create the
> > capabilities for GPE: the merger process seems to need to be different
> > for data being sent to the device from OpenSYnc and data being received
> > from the device and sent to OpenSync. In particular, there is data which
> > should be included in the update sent to the device but which should not
> > be trusted when it is received from the device. This seems to mean that
> > there are three states neded for each attribute: supported, untrusted
> > (specify on send, ignore on receive), excluded (do not specify on send,
> > ignore on receive).
>
> Could you give me an vcard/gpe example for: untrusted and excluded
> scenario?

One real example: until recently there was a bug in the GPE contact
processing.  The N: line was received, parsed, stored and displayed to the
user perfectly correctly.  However, when generating the N: line for export,
the bug dropped one of the fields!  So, for example,

N:Last ; First ; Middle ; Nprefix ; Suffix

might come back as:

N:Last ; First ; Middle ; ; Suffix

So, for this software version, I wanted to configure the capabiltiies such
that OpenSync sends the N: line but ignores any received N: line because it
has been corrupted.

Another case is TYPE parameters.  GPE does not support them so they get
dropped from fields which include them (e.g. phone numbers, email addresses,
etc.).  Until we have a capabiltiies specification format which allows me to
specify this, I wanted to configure these fields such that they get sent to
GPE but the returned value is ignored.  

These are examples of what I called "untrusted": the data should be sent to
the device but it should be ignored when received from the device. Note that
in these cases, it isn't good enough to remove them from the outbound update
completely because these are important fields that the GPE user wants to be
able to see.

The other case is not currently a problem for GPE but there (may be) some
devices which cannot handle receiving a particular field at all -- it needs
to be removed from the update before sending it to the device.  This is the
way the current capabilities code works which is (presumably) desired for
some devices (and could be desired for GPE in future if a bug meant that
receiving a particular attribute caused an error).

I hope these examples clarify what I meant in these two cases, let me know if
I need to explain further.

One way to solve this problem is to have two separate capabilities files: one
for filtering outbound updates, and a separate one for filtering inbound
reports and merging.

Graham

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Graham Cobb-4]

On Tuesday 12 May 2009 15:06:14 Graham Cobb wrote:
> One way to solve this problem is to have two separate capabilities files:
> one for filtering outbound updates, and a separate one for filtering
> inbound reports and merging.

I have just realised that this would be a very bad idea: the inbound filtering
must always be a superset of the outbound (if the attribute was not sent to
the device then clearly any value received from the device must be ignored).

So, if the decision was to have two files they would have to be: one
for "supported", the other for "untrusted" (and, by implication, anything not
mentioned in either file would be "excluded").  It would not be an error to
have the same attribute mentioned in both: "supported" would take priority.  
This is so that you could do things like have a phone number that just has a
TYPE=HOME and one that just has TYPE=WORK supported while all other phone
numbers would be untrusted (that is exactly what GPE's capabilities are).

The outbound filter code would remove attributes that are not mentioned in
either file (i.e. "excluded").  The inbound filter code would remove
attributes not mentioned in the "supported" file (i.e. excluded and
untrusted).  The merger code would ignore any changes or additions or
deletions if the attribute was not mentioned in the "supported" file.
(Inbound filter and merger might be combined into one step -- I am not sure).

Graham

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel