[ApacheDS] [BigBang] Status Report

[Alex Karasulu]

Hi all,


Overview
======

In the bigbang branches (BBB) [0] we're conducting a massive code refactoring of ApacheDS 1.5 in preparation
for the 2.0 road map [1].  There are three major phases to the big bang effort:

Phase I:
   Remove configuration bean hierarchy to directly configure service objects with dependency injection.  At
   the same time we decouple configuration via JNDI but not the use of JNDI for in-vm based ldap operations
   against the directory core.

Phase II:
   Decouple JNDI completely from the core and from protocol providers for solid state operations: i.e. add, del
   modify, search etc.  JNDI classes and interfaces will no longer be used within the server.  Consolidate all
   JNDI initial context factories into a single factory implementation and move this wrapper provider outside of
   the core into it's own module.

Phase III:
   Cleanup after refactoring and implement additional high impact changes such as:
       - virtual attributes
       - instrumentation
       - a better interceptor mechanism.
       - finalize facades for schema and other subsystems
       - code security to sandbox stored procedures

After this last phase the BBBs will replace their respective trunks for shared, daemon, apacheds,
and installers.


Status
=====

I just finished a major part of phase I in the BBBs.  Specifically I have removed the top level object for the
configuration as well as everything in the configuration bean hierarchy except the protocol providers (PP).
Those still remain but they should be trivial to remove and configure the PP's themselves.  After this change
I had to deal with several broken tests which I fixed just tonight.  They still use JNDI for solid state operations
which is fine for now.  David if you're ready you can now start working those protocol provider configuration
beans.  I'll start on the LDAP one for now which will perhaps take the longest time to cleanup.

There are still some residual things remaining however I'll clean these scars up with some more refactoring
as we work on the protocol providers.  Just to see the impact of these changes up until this point you might
want to look at some test cases and the server.xml file.  It's really looking good and a slew (not even a 3rd)
of the JNDI garbage has been removed.  There's much more clarity now and I'm certain this is the right path
to take.  Here are links to those files:

http://svn.apache.org/repos/asf/directory/apacheds/branches/bigbang/server-xml/src/main/resources/server.xml

As you can see the server.xml has become much more concise with greater clarity.  This is thanks to David's
recommendation to use XBean.  Furthermore notice how the components themselves are being assembled
instead of the configuration classes that were used before.

One important thing to note is that there is no longer an environment section used before for JNDI environment
properties.  This used to store among other things the username/password of the administrative user to start
up the server.  This will no longer be needed.  All embedded code is trusted, including the protocol providers,
until we build a sandbox for code added to the server as stored procedures to run with the permissions of the
owner in phase III.

Finally here are some test cases whose setUp() methods show how to construct and start the core and the
server with all protocols respectively:

http://iedia.notlong.com

Notice that JNDI and the configuration beans are not used any longer to start up the core.

http://agace.notlong.com

Again notice the same thing with the ApacheDS object. Also note that by default everything is configured to
work properly via smart defaults however note that LDAP is the only protocol active by default.  All others
including LDAPS needs to be configured.


Plan for the Near Future
=================

The next step I guess is still up to David and I.  We'll be removing all the protocol configuration beans to
directly configure these beans by themselves.  Perhaps other components like MINA thread pools etc
will be configured and injected as well.  There's a lot of cleanup that will still take place once these objects
have been removed.

After that we can begin working on phase II.  Some or all of the work for this phase can happen in parallel.
Emmanuel and I discussed this to some degree. Also Ersin those changes to properly enable virtual/computed
attributes can now start as well as some sandboxing infrastructure for stored procedures.


Thanks,
Alex


-------------------------------
[0] - http://svn.apache.org/repos/asf/directory/apacheds/branches/bigbang-with-dependencies
[1] - http://cwiki.apache.org/confluence/display/DIRxPMGT/2.0+Roadmap

[Emmanuel Lecharny-3]

Thanks a lot Alex for this repport ! It gives us a lot of visibility,
and will help to get in synch when you will be done with all those
configuration cleanup.

In the same time, - beside breaking the trunk with stupid commits ;)
-, I'm finishing to define a new version of JNDI classes used
internally to the server, like :
- ServerEntry, a replacement for Attributes
- ServerAttribute, a replacement for Attribute
- Value, a handler for attributes' values (JNDI uses Object)
- ServerModification, a replacement for ModificationItem

This is also done, tests included. Some more classes and interfaces
have to be replaced, and I have listed all those classes and
interfaces here :
http://cwiki.apache.org/confluence/display/DIRxPMGT/JNDI+removing

The biggest part will be to replace the NamingEnumeration which is
used everywhere, and the SearchResult. The other classes are used from
place to place, and can be removed quite easily.

Another task I'm working on, and I saw that Stefan Szörner has also
working on it,  is to speed up integration tests. It some point, tests
are getting soooo long that we are reluctant to launch them before
committing (I did it a couple of time, and it was a disaster, but
having to wait 20 minutes each time is just overkilling). I really
think we can have tests running in less than 5 minutes, and I will
work on that this week, waiting for Alex to finish his cleanup.

Thanks to all of you, and many thanks to Alex and David for this great cleanup !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

[Alex Karasulu]

Hi Emmanuel,

On 10/8/07, Emmanuel Lecharny <elecharny@...> wrote:

Thanks a lot Alex for this repport ! It gives us a lot of visibility,
and will help to get in synch when you will be done with all those
configuration cleanup.

NP this is a must as the team grows.  We're not in Kansas anymore toto!

In the same time, - beside breaking the trunk with stupid commits ;)
-, I'm finishing to define a new version of JNDI classes used
internally to the server, like :
- ServerEntry, a replacement for Attributes
- ServerAttribute, a replacement for Attribute
- Value, a handler for attributes' values (JNDI uses Object)
- ServerModification, a replacement for ModificationItem

This is also done, tests included. Some more classes and interfaces
have to be replaced, and I have listed all those classes and
interfaces here :
http://cwiki.apache.org/confluence/display/DIRxPMGT/JNDI+removing

The biggest part will be to replace the NamingEnumeration which is
used everywhere, and the SearchResult. The other classes are used from
place to place, and can be removed quite easily.

Yes we need a ServerResult object as well as a Cursor interface which should
be more like a jdbc ResultSet.  I have some ideas on this from past experiences.
Just quickly the Cursor should not be pre-fetching in the constructor as we do
now with NamingEnumerations.  This is why JDBC has getObject() and next()
but next returns a boolean.  It does not prefetch but drives the initial fetch in the
loop.  Anyways we can discuss this in more detail.  However this work can start
happening soon in the JDBM implementation as long as we have tools to butress
the changes by converting back to what higher levels of the server expect in terms
of JNDI objects like NamingEnumeration and SearchResult.

Another task I'm working on, and I saw that Stefan Szörner has also
working on it,  is to speed up integration tests. It some point, tests
are getting soooo long that we are reluctant to launch them before
committing (I did it a couple of time, and it was a disaster, but
having to wait 20 minutes each time is just overkilling). I really
think we can have tests running in less than 5 minutes, and I will
work on that this week, waiting for Alex to finish his cleanup.

Actually this can start happening now.  You have most of what you need to do this.  We might
step on each others toes on occasion but these tests are killing us.  Having them out of the
way ASAP before Phase II might help accelerate our big bang effort.

Thanks,
Alex