A split build for OpenOffice.org

[Mathias Bauer]

Hi,

since several weeks I have been thinking about a split build(*) for
OpenOffice.org that fulfils the following criteria:

* the split parts should be buildable in an acceptable time
* the split should group modules together that have something in common
* the boundaries between the groups should be along stable APIs where
  they exist
* code and non-code modules should be separated
* there should be no cross build dependencies between groups (module A
  in group 1 depends on module B in group 2 and module C in group 2
  depends on module D in group 1)
* there should be no cross runtime dependencies between groups (module A
  in group 1 depends on module B in group 2 and module C in group 2
  depends on module D in group 1)

You can find the status quo of the results I got here:

http://wiki.services.openoffice.org/wiki/Build_Environment_Effort/Split_Build

Background information:

http://blogs.sun.com/GullFOSS/entry/a_split_build_for_openoffice
http://wiki.services.openoffice.org/wiki/Build_Environment_Effort/Dependencies
http://wiki.services.openoffice.org/wiki/Build_Environment_Effort/Module_Problem

As mentioned in the blog, this is just a first step. The "code
architecture" can still be optimized, but it already looks good wrt. to
the important criteria of acceptable build times and missing cross
dependencies. How many packages each of these groups shall contains is
open for discussion. My POV is that each group should be one package.

One of the next steps will be to find a way to pack single packages and
integrate that into the build process.

Input, comments and help :-) will be appreciated.

Regards,
Mathias

----------------------------------------------------------------------------------
(*) means: defining groups of modules that can be built separately and
making the OOo build just the sum of all of these partial builds
----------------------------------------------------------------------------------

--
Mathias Bauer (mba) - Project Lead OpenOffice.org Writer
OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS
Please don't reply to "nospamformba@...".
I use it for the OOo lists and only rarely read other mails sent to it.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Jan Nieuwenhuizen-2]

Op donderdag 05-11-2009 om 10:04 uur [tijdzone +0100], schreef Mathias
Bauer:

Hi Mathias,

> Input, comments and help :-) will be appreciated.

> One of the next steps will be to find a way to pack single packages and
> integrate that into the build process.

For the external category this is quite easy: remove them,
ad PKG_CHECK_MODULES () checks in configure.in and remove
corresponding --with-system-foo options.

That would clean things up a bit!

Greetings,
Jan.

--
Jan Nieuwenhuizen <janneke@...> | GNU LilyPond - The music typesetter
Avatar®: http://AvatarAcademy.nl    | http://lilypond.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Thorsten Behrens-3]

Hi Mathias,

really happy about progress here, especially with the vcl/svtools
changes you made.

You wrote:
> * the split parts should be buildable in an acceptable time
> * the split should group modules together that have something in common
> * the boundaries between the groups should be along stable APIs where
>   they exist
> * code and non-code modules should be separated
>
The latter one is most beneficial; at least the classical roles for
people doing changes usually don't touch both. For the former, the
wording is a bit fuzzy - and except for ure, is there any stable api
at all inside OOo?

> * there should be no cross build dependencies between groups (module A
>   in group 1 depends on module B in group 2 and module C in group 2
>   depends on module D in group 1)
>
Yep.

> * there should be no cross runtime dependencies between groups (module A
>   in group 1 depends on module B in group 2 and module C in group 2
>   depends on module D in group 1)
>
Hm - doesn't that go against the dependency inversion/injection
principle so beneficial to large code bases?

And in addition to the aforementioned goals, maybe also optimizing
the dependency graph for minimal cuts would be valuable (barring
other criteria, like stable api)?

Other than that, really loved to read about progress in this
specific area, but would suggest to seriously think about more
fine-grained repository splits. As this would much encourage people
to not go cross-boundary too often ... ;)

Cheers,

-- Thorsten

[Rene Engelhard-7]

Hi,

On Thu, Nov 05, 2009 at 03:44:20PM +0100, Jan Nieuwenhuizen wrote:
> For the external category this is quite easy: remove them,
> ad PKG_CHECK_MODULES () checks in configure.in and remove
> corresponding --with-system-foo options.
>
> That would clean things up a bit!

This would require that OOo stopped applying huge patches to the libs it
uses which make  them not usable as they are from the system. Or (like
with graphite) you have a incompatibiity of the used STL.

Nice idea (sys /me as supporter and implementor of most of the system-*)
but the way is far to reach this - unless all changes OOo does are upstreamed
and *THEN* OOo relies on them (as the project expects from us GNU/Linux
distributions, too)

Grüße/Regards,

Rene

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Mathias Bauer]

Hi Thorsten,

Thorsten Behrens wrote:

> Hi Mathias,
>
> really happy about progress here, especially with the vcl/svtools
> changes you made.

It was tough but also fun. I hope that we can do similar things in svx
etc. I already took the low hanging fruits msfilter and cui, but even
this took more than a week. What I have learned is that our way to
handle resources makes it hard to reorganize code and that HIDs are a
PITA. So one of the next projects I would like to start is "get rid of
the hid.lst".

> You wrote:
>> * the split parts should be buildable in an acceptable time
>> * the split should group modules together that have something in common
>> * the boundaries between the groups should be along stable APIs where
>>   they exist
>> * code and non-code modules should be separated
>>
> The latter one is most beneficial; at least the classical roles for
> people doing changes usually don't touch both. For the former, the
> wording is a bit fuzzy - and except for ure, is there any stable api
> at all inside OOo?

I assume that "ure" means the group I have suggested, not what it has
been as a product, so it includes the offapi/offuh modules that also has
a stable API.

"Stable" doesn't mean that it should be stable "for ever". As an
example, the APIs of the external libraries are also what I consider
stable in the context of OOo.

Just to avoid misunderstandings: I didn't say that stable APIs are a
necessary precondition for a split position, but *if* there are such
stable APIs they should be preferred points for a split.

We also have a lot of libraries where the API is "fairly stable", like
tools, comphelper, toolkit, even vcl. That *could* be enough.

>> * there should be no cross build dependencies between groups (module A
>>   in group 1 depends on module B in group 2 and module C in group 2
>>   depends on module D in group 1)
>>
> Yep.
>
>> * there should be no cross runtime dependencies between groups (module A
>>   in group 1 depends on module B in group 2 and module C in group 2
>>   depends on module D in group 1)
>>
> Hm - doesn't that go against the dependency inversion/injection
> principle so beneficial to large code bases?

I think that there is no problem with having runtime dependencies of
"low build level" modules on "high build level" modules, but IMHO it
should not be a cross dependency in both directions as explained in my
example, in that case one of the two blocks should be split. But I admit
that this is more a personal preference than a technical requirement.

> And in addition to the aforementioned goals, maybe also optimizing
> the dependency graph for minimal cuts would be valuable (barring
> other criteria, like stable api)?

Can you explain what "minimal cuts" means?

> Other than that, really loved to read about progress in this
> specific area, but would suggest to seriously think about more
> fine-grained repository splits. As this would much encourage people
> to not go cross-boundary too often ... ;)

Heeee, you know that you can't solve social problems by technical means. :-)

I think that repository boundaries can help to avoid the erosion of code
separation. But OTOH if a code separation and its implementation as a
repository split turns out to be a bad choice, the split will be a PITA.
So we should not use repository splits to force people to work in a
certain way but rather find out what is the best or desired way to work
and support that by a repository structure that fits best to that.

Regards,
Mathias

--
Mathias Bauer (mba) - Project Lead OpenOffice.org Writer
OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS
Please don't reply to "nospamformba@...".
I use it for the OOo lists and only rarely read other mails sent to it.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Thorsten Behrens-3]

Mathias Bauer wrote:
> > And in addition to the aforementioned goals, maybe also optimizing
> > the dependency graph for minimal cuts would be valuable (barring
> > other criteria, like stable api)?
>
> Can you explain what "minimal cuts" means?
>
Hi Mathias,

sorry, was referring to this:
http://en.wikipedia.org/wiki/Cut_(graph_theory)

> I think that repository boundaries can help to avoid the erosion of code
> separation. But OTOH if a code separation and its implementation as a
> repository split turns out to be a bad choice, the split will be a PITA.
> So we should not use repository splits to force people to work in a
> certain way but rather find out what is the best or desired way to work
> and support that by a repository structure that fits best to that.
>
Admittedly, constantly moving code across repository boundaries is a
royal PITA. But I don't see a reason not to do that, after the split
build setup has stabilized.

Cheers,

-- Thorsten