[RFC] Different libraries for OSYNC_START_TYPE_EXTERNAL

[Henrik /KaarPoSoft-2]

Dear all,

I am working on porting mozilla-sync to use OSYNC_START_TYPE_EXTERNAL.

I already send a mail about this on the list ("[RFC] Using
OSYNC_START_TYPE_EXTERNAL") on which I still hope to get comments.

However, one more conceptual question has come up, and I would like to
discuss this with the community.

Currently both osynctool (or, in a production environment: your own sync
function in your own GUI) and the external process are using the exact
same plugin module (such as mozilla-sync).

In fact, the engine (e.g. in osynctool) will initialize a proxy, and
this proxy will send the name of the plugin and the path to the plugin
directory to the external process.

The external process will then set up an identical environment in  
_osync_client_handle_initialize (at least if the external process is
coded in the simplest possible way I could find: calling
osync_client_run_and_block and letting the OpenSync libraries take care
of the rest).

This means that the engine (e.g. in osynctool) will load all plugin
modules, which in turn leads to loading all the libraries the module
depends on.

In the mozilla-sync case that means xpcom libraries etc.

It also means that the mozilla-sync plugin will need to be compiled the
Mozilla development files.

All of this totally unnecesary, as all the Mozilla-relevant stuff is
taking place in the external process.

The external process will also load all plugin modules - again totally
unnecessary since it will only need a single plugin.

So I think we need a mechanism which allows a plugin (of type
OSYNC_START_TYPE_EXTERNAL) to have TWO different libraries:
- one for the engine
- one for the external process

A very simple way to accomplish this would be:

1)
Add a field to struct OSyncClient:
char *overriding_plugin_dir

2)
Add corresponding getters and setters in osync_client.c

3)
In _osync_client_handle_initialize (which gets passed the OSyncClient)
If overriding_plugin_dir<>NULL then call
osync_plugin_env_load(client->plugin_env, overriding_plugin_dir, error)
instead of
osync_plugin_env_load(client->plugin_env, plugindir, error)

4)
Now the external process can set overriding_plugin_dir after it has
created the client with osync_client_new.

This would have the additional benefit of keeping the peer-specific (in
this case: Mozilla-specific) code out of the OpenSync codebase.

So even though Thunderbird 2 and Thunderbird 3 are so different, that it
seems I will have to create two different plugins, all this would be
handled inside a plugin for Thunderbird, while keeping the mozilla-sync
in OpenSync very slim.

Any comments and thoughts on this would be most appreciated.

/Henrik

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Graham Cobb-4]

On Tuesday 27 October 2009 17:07:50 Henrik /KaarPoSoft wrote:
> Currently both osynctool (or, in a production environment: your own sync
> function in your own GUI) and the external process are using the exact
> same plugin module (such as mozilla-sync).

Henrik,

I haven't looked at the code (maybe a good thing so we can discuss how it
**should** work).

Do we need to have the plugin running at all in the engine process?  Can the
engine just be told (somehow) to create the proxy and talk to the real plugin
in the external process?

> The external process will then set up an identical environment in
> _osync_client_handle_initialize (at least if the external process is
> coded in the simplest possible way I could find: calling
> osync_client_run_and_block and letting the OpenSync libraries take care
> of the rest).

Similarly, we don't really need the engine (let alone the other plugins)
running at all in the external process -- just the other end of the proxy
connection.  In fact, I thought I remembered that there was a way to do that
(but I would have to look at the code!).  I do remember that when I was
looking at the timeout stuff I found the client/proxy stuff confusing!

> So I think we need a mechanism which allows a plugin (of type
> OSYNC_START_TYPE_EXTERNAL) to have TWO different libraries:
> - one for the engine
> - one for the external process

Why does it need one for the engine at all?

The rough way I had thought this would work...

The external plugin is NOT installed in the usual place for Opensync plugins.

The GUI (or osynctool) creates the engine as normal, which loads all the other
plugins.

The GUI (or osynctool) calls a special osync function to tell it that there is
a remote plugin accessible over such-and-such a pipe.  The engine creates a
proxy for this.  At this point the engine believes the plugin exists even
though there is only a proxy.  It can be added to groups, etc.

Meanwhile, the external process has created the other end of the proxy NOT by
creating the whole engine but just by creating a client (is that the right
terminology -- I might have to go and look at the code).

Either as part of creating that client, or with a separate function, the
external process tells the client where to find the plugin (which is not in a
standard place) and connects the plugin to the client, with no engine.  It
may be that the plugin is not even loaded, in the normal opensync meaning of
that term -- it could be a static part of the external process and
its "get_sync_info" address is passed to the client.

The external process then runs the client.

I am sure, from your description, that this is not how it actually works
today, but this seems easier than having plugin directory overides.

Graham

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Henrik /KaarPoSoft-2]

Thanks for your comments, Graham.

Here is a bit of "replies" / new comments.

Graham Cobb wrote:
> Do we need to have the plugin running at all in the engine process?  Can the
> engine just be told (somehow) to create the proxy and talk to the real plugin
> in the external process?
>  
If I understand correctly, today the engine is loading up all the plugin
modules, but for an OSYNC_START_TYPE_EXTERNAL plugin, the engine just
calls a few functions to get info. So we need "something", that the
engine can call to know how to deal with the external process. I.e. a
plugin, but a very lightweight one.
> Similarly, we don't really need the engine (let alone the other plugins)
> running at all in the external process -- just the other end of the proxy
> connection.  In fact, I thought I remembered that there was a way to do that
> (but I would have to look at the code!).  I do remember that when I was
> looking at the timeout stuff I found the client/proxy stuff confusing!
>  
There might be a different way, but I have not found it. But with very
little documentation it is kind of hard to look through the whole
codebase and see if there is a bit that fits is quite hard.
But if there is a better way than what I have proposed I am all ears!
One of the reasons for my mails about this is indeed to see if anyone
has a better way.
>> So I think we need a mechanism which allows a plugin (of type
>> OSYNC_START_TYPE_EXTERNAL) to have TWO different libraries:
>> - one for the engine
>> - one for the external process
>>    
>
> Why does it need one for the engine at all?
>  
See above: We need at least a slim plugin to tell the engine how to
contact the external process.
> The rough way I had thought this would work...
>
> The external plugin is NOT installed in the usual place for Opensync plugins.
>
> The GUI (or osynctool) creates the engine as normal, which loads all the other
> plugins.
>
> The GUI (or osynctool) calls a special osync function to tell it that there is
> a remote plugin accessible over such-and-such a pipe.
I think this info should come from the slim plugin, not the GUI/osynctool.
>  The engine creates a
> proxy for this.  At this point the engine believes the plugin exists even
> though there is only a proxy.  It can be added to groups, etc.
>
> Meanwhile, the external process has created the other end of the proxy NOT by
> creating the whole engine but just by creating a client (is that the right
> terminology -- I might have to go and look at the code).
>  
Yes, that is also how it would work in my approach. The external process
just creates a client (but using the opensync lib, which also includes
the engine etc, which will not be called).
> Either as part of creating that client, or with a separate function, the
> external process tells the client where to find the plugin (which is not in a
> standard place) and connects the plugin to the client, with no engine.
Exactly.
>   It
> may be that the plugin is not even loaded, in the normal opensync meaning of
> that term -- it could be a static part of the external process and
> its "get_sync_info" address is passed to the client.
>  
Sure. My reason for making it a "real" opensync plugin was simply that
then I could use the opensync client code with minimal modifications.
> The external process then runs the client.
>
> I am sure, from your description, that this is not how it actually works
> today, but this seems easier than having plugin directory overides.
>
>  
First of all it seems that we agree on the overall principles, which is
great.

The only question is: How to implement it, and here we have slightly
different approaches.

However, the difference is not very big!

You suggest the GUI tells the engine about the external process; I
suggest a "slim" plugin.

You suggest the external process loads the "fat" plugin from a different
location; you suggest basically the same or that no plugin is loaded at all.


Graham, I really appreciate the feedback you have given!

Comments, anybody ?


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Daniel Gollub-3]

Hi Henrik,


On Tuesday 27 October 2009 06:07:50 pm Henrik /KaarPoSoft wrote:
[...]
> I am working on porting mozilla-sync to use OSYNC_START_TYPE_EXTERNAL.

Awesome!

[...]
>
> In fact, the engine (e.g. in osynctool) will initialize a proxy, and
> this proxy will send the name of the plugin and the path to the plugin
> directory to the external process.


By "plugin directory" you mean the member directory of the group - right?

E.g. ~/.opensync/groupX/Y/

This is at what the engine uses as argument to spawn for proxy. The function
osync_client_proxy_spawn() is then creating two pipes in this member
directory: pluginpipe and enginepipe

>
> The external process will then set up an identical environment in
> _osync_client_handle_initialize (at least if the external process is
> coded in the simplest possible way I could find: calling
> osync_client_run_and_block and letting the OpenSync libraries take care
> of the rest).

Right.

>
> This means that the engine (e.g. in osynctool) will load all plugin
> modules, which in turn leads to loading all the libraries the module
> depends on.
>
> In the mozilla-sync case that means xpcom libraries etc.

Right, but ...

>
> It also means that the mozilla-sync plugin will need to be compiled the
> Mozilla development files.
>
> All of this totally unnecesary, as all the Mozilla-relevant stuff is
> taking place in the external process.

.. you don't have to load this.

>
> The external process will also load all plugin modules - again totally
> unnecessary since it will only need a single plugin.


Why don't you try this:

Just write a very tiny mozilla-sync which just contains the function
"get_sync_info"?

It just registers the plugin and set the start-type to
OSYNC_START_TYPE_EXTERNAL.

So this plugin (stub) is not using any mozilla API -> no mozilla dependecies
.... Engine doesn't load those libraries.


The plugin stub only does one thing: register the plugin to the OSyncPluginEnv
... and set the name and description of the plugin, as well as that the plugin
is an external one and needs to be started via OSYNC_START_TYPE_EXTERNAL ...


And your real "plugin" which is acutally a complete different process - or
even an own application is implementing a full OSyncClient and OSyncPlugin
environment? Completeyl without that get_sync_info() stuff ...

I'll provide for that a proof-of-concepüt ...

Or do you see already anything why this can't work?

[...]

Best Regards,
Daniel

--
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: [phone]         Handelsregister:   HRB 3537
EMail: gollub@...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Henrik /KaarPoSoft-2]

Daniel Gollub wrote:

> On Tuesday 27 October 2009 06:07:50 pm Henrik /KaarPoSoft wrote:
> [...]
>  
>> I am working on porting mozilla-sync to use OSYNC_START_TYPE_EXTERNAL.
>>    
>
> Awesome!
>
> [...]
>  
>> In fact, the engine (e.g. in osynctool) will initialize a proxy, and
>> this proxy will send the name of the plugin and the path to the plugin
>> directory to the external process.
>>    
>
>
> By "plugin directory" you mean the member directory of the group - right?
>
> E.g. ~/.opensync/groupX/Y/
>
> This is at what the engine uses as argument to spawn for proxy. The function
> osync_client_proxy_spawn() is then creating two pipes in this member
> directory: pluginpipe and enginepipe
>
>  

No, I do mean the PLUGIN directory.
static osync_bool _osync_client_handle_initialize(OSyncClient *client,
OSyncMessage *message, OSyncError **error):
    osync_message_read_string(message, &enginepipe, error);
    osync_message_read_string(message, &formatdir, error);
    osync_message_read_string(message, &plugindir, error);
    osync_message_read_string(message, &pluginname, error);
    osync_message_read_string(message, &groupname, error);
    osync_message_read_string(message, &configdir, error);
    osync_message_read_int(message, &haspluginconfig, error);

>
> Why don't you try this:
>
> Just write a very tiny mozilla-sync which just contains the function
> "get_sync_info"?
>
> It just registers the plugin and set the start-type to
> OSYNC_START_TYPE_EXTERNAL.
>
> So this plugin (stub) is not using any mozilla API -> no mozilla dependecies
> .... Engine doesn't load those libraries.
>
>
> The plugin stub only does one thing: register the plugin to the OSyncPluginEnv
> ... and set the name and description of the plugin, as well as that the plugin
> is an external one and needs to be started via OSYNC_START_TYPE_EXTERNAL ...
>  

This is exactly what I did.
> And your real "plugin" which is acutally a complete different process - or
> even an own application is implementing a full OSyncClient and OSyncPlugin
> environment? Completeyl without that get_sync_info() stuff ...
>  
I could do that, but it would take a lot of coding to make my own
OSyncClient. That's why I simply use the full opensync library. That's
only 10 lines of code!

But if you have a proof-of-concept with a full stand-alone OSyncClient
without to much code, I will certainly have a look!


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Graham Cobb-4]

On Thursday 29 October 2009 15:55:40 Henrik /KaarPoSoft wrote:
> Daniel Gollub wrote:
> > And your real "plugin" which is acutally a complete different process -
> > or even an own application is implementing a full OSyncClient and
> > OSyncPlugin environment? Completeyl without that get_sync_info() stuff
> > ...
>
> I could do that, but it would take a lot of coding to make my own
> OSyncClient. That's why I simply use the full opensync library. That's
> only 10 lines of code!

I think that is what I have been trying to suggest (badly) -- thanks for the
clarification Daniel.

But, of course, Henrik, you shouldn't have to implement your own OSyncClient.  
What we have to do is to make the client code and plugin support code usable
from your process, without bringing in all the rest of OpenSync.  That may be
harder than it looks because some functions that plugins call may assume the
whole OpenSync environment is present.

Graham

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Henrik /KaarPoSoft-2]

Graham Cobb wrote:

> On Thursday 29 October 2009 15:55:40 Henrik /KaarPoSoft wrote:
>  
>> Daniel Gollub wrote:
>>    
>>> And your real "plugin" which is acutally a complete different process -
>>> or even an own application is implementing a full OSyncClient and
>>> OSyncPlugin environment? Completeyl without that get_sync_info() stuff
>>> ...
>>>      
>> I could do that, but it would take a lot of coding to make my own
>> OSyncClient. That's why I simply use the full opensync library. That's
>> only 10 lines of code!
>>    
>
> I think that is what I have been trying to suggest (badly) -- thanks for the
> clarification Daniel.
>
> But, of course, Henrik, you shouldn't have to implement your own OSyncClient.  
> What we have to do is to make the client code and plugin support code usable
> from your process, without bringing in all the rest of OpenSync.  That may be
> harder than it looks because some functions that plugins call may assume the
> whole OpenSync environment is present.
>  

Any hints on how to do that would be appreciated!

/Henrik

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel