Geronimo Clustering with WADI?

[Matthew Jording]

Hello Jules and community.

   I will need to have a production quality deployment of Geronimo ready
within the next couple of months. One of the requirements is to have
some level of cluster node management facility. Initially my thoughts
are web service and WSDM/JMX mixes. If you are currently working on such
a facility perhaps I can just start contributing to it. At the very
least I will keep you appraised of my progress. From responses on the
Geronimo Developer lists I assume I should start with the clustering
GBeans in the various tiers and work my way up.

I know that this is also the WADI/ActiveCluster plan for Geronimo and
wanted to see if there is any work being done on this that I could build
upon.


Thanks
Matthew Jording

[Jules Gosnell-4]

Matthew Jording wrote:

> Hello Jules and community.
>
Hi,

>   I will need to have a production quality deployment of Geronimo
> ready within the next couple of months. One of the requirements is to
> have some level of cluster node management facility.

Management is a big area :-)

> Initially my thoughts are web service and WSDM/JMX mixes.

Can you be a little more specific about exactly the sort of fn-ality
that you are looking for. We are concentrating on scalable and
performant clustered state (sessions) at the moment. WS sessions are
under development. Individual WADI nodes are JMX manageable. I have
ideas about how monitoring info might be aggregated for the whole
cluster on each WADI node, but this is NYI... There are many other areas
of clustering fn-ality.

> If you are currently working on such a facility perhaps I can just
> start contributing to it. At the very least I will keep you appraised
> of my progress. From responses on the Geronimo Developer lists I
> assume I should start with the clustering GBeans in the various tiers
> and work my way up.

Lets get a clear idea of what you are after and take it from there. Any
contribution would be very welcome :-) A lot of thought has gone into
how a Geronimo integration should look, but work in this direction has
stalled. Some fresh enthusiasm in this area would be great.

>
> I know that this is also the WADI/ActiveCluster plan for Geronimo and
> wanted to see if there is any work being done on this that I could
> build upon.

This part of WADI is undergoing a lot of thought at the moment - the
more the merrier.

Thanks for your interest in WADI,


Jules

>
>
> Thanks
> Matthew Jording



--
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/

[Matthew Jording]

Thanks for the prompt reply Jules,

    Here is a clarification of fn-ality: I need to be able to receive
information from each node, Session load per node (collected for
histogram display), node uptime, pre-failure conditions and failover
node. Secondly I need to manage the cluster via manual load balancing
and master-chain relations for failover, and of course stateful node
shutdown & startup. I'm concentrating on the web tier first and use the
Tomcat clustering GBeans in Geronimo and create proper mbeans that can
interact with a WSDM webservice which can provide the bridge to any
outside management

Its a bit to chew on, and Geronimo is the obvious choice of app server.
As a matter of course I wanted to see how far along these requirements
are 'out of the box' and build on any existing work to give back to the
Geronimo and WADI crew. From what I understand, although wadi is
available in Geronimo v1.0 the integration is not there. which is why I
am starting from the gbean up. Obviously if there is something I could
bridge between Geronimo and WADI even at an experimental level that
would help me understand the proposed relationship.

Thanks again
Matthew Jording

Jules Gosnell wrote:

> Matthew Jording wrote:
>
>> Hello Jules and community.
>>
> Hi,
>
>>   I will need to have a production quality deployment of Geronimo
>> ready within the next couple of months. One of the requirements is to
>> have some level of cluster node management facility.
>
> Management is a big area :-)
>
>> Initially my thoughts are web service and WSDM/JMX mixes.
>
> Can you be a little more specific about exactly the sort of fn-ality
> that you are looking for. We are concentrating on scalable and
> performant clustered state (sessions) at the moment. WS sessions are
> under development. Individual WADI nodes are JMX manageable. I have
> ideas about how monitoring info might be aggregated for the whole
> cluster on each WADI node, but this is NYI... There are many other
> areas of clustering fn-ality.
>
>> If you are currently working on such a facility perhaps I can just
>> start contributing to it. At the very least I will keep you appraised
>> of my progress. From responses on the Geronimo Developer lists I
>> assume I should start with the clustering GBeans in the various tiers
>> and work my way up.
>
> Lets get a clear idea of what you are after and take it from there.
> Any contribution would be very welcome :-) A lot of thought has gone
> into how a Geronimo integration should look, but work in this
> direction has stalled. Some fresh enthusiasm in this area would be great.
>
>>
>> I know that this is also the WADI/ActiveCluster plan for Geronimo and
>> wanted to see if there is any work being done on this that I could
>> build upon.
>
> This part of WADI is undergoing a lot of thought at the moment - the
> more the merrier.
>
> Thanks for your interest in WADI,
>
>
> Jules
>
>>
>>
>> Thanks
>> Matthew Jording
>
>
>

[Jules Gosnell-4]

Matthew Jording wrote:

> Thanks for the prompt reply Jules,
>
>    Here is a clarification of fn-ality: I need to be able to receive
> information from each node, Session load per node (collected for
> histogram display), node uptime, pre-failure conditions and failover node.

num sessions per node is OK
node uptime - OK
pre-failure conditions ? you mean you want a snapshot of what was going
on on a node just before it died ?
failover node - this would depend upon the replication scheme - but
should be possible...

> Secondly I need to manage the cluster via manual load balancing

you mean that you want to manually move x sessions from node a->b if a
looks overloaded ? - why manually, wouldn't it be better if this
situation never arose ? This would be WADI's aim,

> and master-chain relations for failover,

you'll have to expand here - not sure what you mean - our current
replication scheme uses a pluggable algorithm to decide which nodes to
use as bckups - so perhaps ou couls plug something in here ?

> and of course stateful node shutdown & startup.

we will have this. When a node shutd down it distributes its state
direct to other peers...

> I'm concentrating on the web tier first and use the Tomcat clustering
> GBeans in Geronimo and create proper mbeans that can interact with a
> WSDM webservice which can provide the bridge to any outside management

We use Spring's JMX integration to register interesting WADI pojos with
JMX, from there they can be queried via remote or local JMX - should be
possible to attach your webservice if you want.

>
> Its a bit to chew on, and Geronimo is the obvious choice of app
> server. As a matter of course I wanted to see how far along these
> requirements are 'out of the box' and build on any existing work to
> give back to the Geronimo and WADI crew. From what I understand,
> although wadi is available in Geronimo v1.0 the integration is not
> there. which is why I am starting from the gbean up. Obviously if
> there is something I could bridge between Geronimo and WADI even at an
> experimental level that would help me understand the proposed
> relationship.

It sounds like pretty much everything that you want is on our immediate
todo list or done... I don't see any real requirement for this to be
integrated with the Geronimo kernel (since you monitor via JMX), just as
long as it can be deployed into a webcontainer within Geronimo ... If
this is not working at the moment, then it is only a small patch. A
complete integration with Geronimo is a larger proposition, that has
been discussed a number of times. If you are interested in getting
involved in this, that would be great.

I'll start thinking about your requirements. We should prioritise them
and talk about them in more detail.

A good start :-)

Jules

>
> Thanks again
> Matthew Jording
>
> Jules Gosnell wrote:
>
>> Matthew Jording wrote:
>>
>>> Hello Jules and community.
>>>
>> Hi,
>>
>>>   I will need to have a production quality deployment of Geronimo
>>> ready within the next couple of months. One of the requirements is
>>> to have some level of cluster node management facility.
>>
>>
>> Management is a big area :-)
>>
>>> Initially my thoughts are web service and WSDM/JMX mixes.
>>
>>
>> Can you be a little more specific about exactly the sort of fn-ality
>> that you are looking for. We are concentrating on scalable and
>> performant clustered state (sessions) at the moment. WS sessions are
>> under development. Individual WADI nodes are JMX manageable. I have
>> ideas about how monitoring info might be aggregated for the whole
>> cluster on each WADI node, but this is NYI... There are many other
>> areas of clustering fn-ality.
>>
>>> If you are currently working on such a facility perhaps I can just
>>> start contributing to it. At the very least I will keep you
>>> appraised of my progress. From responses on the Geronimo Developer
>>> lists I assume I should start with the clustering GBeans in the
>>> various tiers and work my way up.
>>
>>
>> Lets get a clear idea of what you are after and take it from there.
>> Any contribution would be very welcome :-) A lot of thought has gone
>> into how a Geronimo integration should look, but work in this
>> direction has stalled. Some fresh enthusiasm in this area would be
>> great.
>>
>>>
>>> I know that this is also the WADI/ActiveCluster plan for Geronimo
>>> and wanted to see if there is any work being done on this that I
>>> could build upon.
>>
>>
>> This part of WADI is undergoing a lot of thought at the moment - the
>> more the merrier.
>>
>> Thanks for your interest in WADI,
>>
>>
>> Jules
>>
>>>
>>>
>>> Thanks
>>> Matthew Jording
>>
>>
>>
>>
>

--
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/

[Matthew Jording]

Wow, thats great Jules,

  You guys have clearly been busier then the site at first indicates. I
clearly need to start playing with with WADI and see what I can glean.

So as far as pre-failure conditions, I just need to save node state and
preserve logs and make all of that available in some manner to the
'cluster manager'. Manual load balancing would be a requirement only to
evacuate sessions from a node for any potential reason other then
failover. What I meant by Master-chain relationships is the ability to
prioritize the replication scheme.

Priorities of requirements:
1. sessions per node
2. node uptime
3. failover node / replication scheme
4. node state preservation

Jules Gosnell wrote:

> Matthew Jording wrote:
>
>> Thanks for the prompt reply Jules,
>>
>>    Here is a clarification of fn-ality: I need to be able to receive
>> information from each node, Session load per node (collected for
>> histogram display), node uptime, pre-failure conditions and failover
>> node.
>
> num sessions per node is OK
> node uptime - OK
> pre-failure conditions ? you mean you want a snapshot of what was
> going on on a node just before it died ?
> failover node - this would depend upon the replication scheme - but
> should be possible...
>
>> Secondly I need to manage the cluster via manual load balancing
>
> you mean that you want to manually move x sessions from node a->b if a
> looks overloaded ? - why manually, wouldn't it be better if this
> situation never arose ? This would be WADI's aim,
>
>> and master-chain relations for failover,
>
> you'll have to expand here - not sure what you mean - our current
> replication scheme uses a pluggable algorithm to decide which nodes to
> use as bckups - so perhaps ou couls plug something in here ?
>
>> and of course stateful node shutdown & startup.
>
> we will have this. When a node shutd down it distributes its state
> direct to other peers...
>
>> I'm concentrating on the web tier first and use the Tomcat clustering
>> GBeans in Geronimo and create proper mbeans that can interact with a
>> WSDM webservice which can provide the bridge to any outside management
>
> We use Spring's JMX integration to register interesting WADI pojos
> with JMX, from there they can be queried via remote or local JMX -
> should be possible to attach your webservice if you want.
>
>>
>> Its a bit to chew on, and Geronimo is the obvious choice of app
>> server. As a matter of course I wanted to see how far along these
>> requirements are 'out of the box' and build on any existing work to
>> give back to the Geronimo and WADI crew. From what I understand,
>> although wadi is available in Geronimo v1.0 the integration is not
>> there. which is why I am starting from the gbean up. Obviously if
>> there is something I could bridge between Geronimo and WADI even at
>> an experimental level that would help me understand the proposed
>> relationship.
>
> It sounds like pretty much everything that you want is on our
> immediate todo list or done... I don't see any real requirement for
> this to be integrated with the Geronimo kernel (since you monitor via
> JMX), just as long as it can be deployed into a webcontainer within
> Geronimo ... If this is not working at the moment, then it is only a
> small patch. A complete integration with Geronimo is a larger
> proposition, that has been discussed a number of times. If you are
> interested in getting involved in this, that would be great.
>
> I'll start thinking about your requirements. We should prioritise them
> and talk about them in more detail.
>
> A good start :-)
>
> Jules
>
>>
>> Thanks again
>> Matthew Jording
>>
>> Jules Gosnell wrote:
>>
>>> Matthew Jording wrote:
>>>
>>>> Hello Jules and community.
>>>>
>>> Hi,
>>>
>>>>   I will need to have a production quality deployment of Geronimo
>>>> ready within the next couple of months. One of the requirements is
>>>> to have some level of cluster node management facility.
>>>
>>>
>>> Management is a big area :-)
>>>
>>>> Initially my thoughts are web service and WSDM/JMX mixes.
>>>
>>>
>>> Can you be a little more specific about exactly the sort of fn-ality
>>> that you are looking for. We are concentrating on scalable and
>>> performant clustered state (sessions) at the moment. WS sessions are
>>> under development. Individual WADI nodes are JMX manageable. I have
>>> ideas about how monitoring info might be aggregated for the whole
>>> cluster on each WADI node, but this is NYI... There are many other
>>> areas of clustering fn-ality.
>>>
>>>> If you are currently working on such a facility perhaps I can just
>>>> start contributing to it. At the very least I will keep you
>>>> appraised of my progress. From responses on the Geronimo Developer
>>>> lists I assume I should start with the clustering GBeans in the
>>>> various tiers and work my way up.
>>>
>>>
>>> Lets get a clear idea of what you are after and take it from there.
>>> Any contribution would be very welcome :-) A lot of thought has gone
>>> into how a Geronimo integration should look, but work in this
>>> direction has stalled. Some fresh enthusiasm in this area would be
>>> great.
>>>
>>>>
>>>> I know that this is also the WADI/ActiveCluster plan for Geronimo
>>>> and wanted to see if there is any work being done on this that I
>>>> could build upon.
>>>
>>>
>>> This part of WADI is undergoing a lot of thought at the moment - the
>>> more the merrier.
>>>
>>> Thanks for your interest in WADI,
>>>
>>>
>>> Jules
>>>
>>>>
>>>>
>>>> Thanks
>>>> Matthew Jording
>>>
>>>
>>>
>>>
>>
>
>

[Matthew Jording]

Jules,
   I forgot to mention I would be happy to contribute opinion, code, or
documentation to the effort as needed.

Thanks
Matthew Jording


Jules Gosnell wrote:

> Matthew Jording wrote:
>
>> Thanks for the prompt reply Jules,
>>
>>    Here is a clarification of fn-ality: I need to be able to receive
>> information from each node, Session load per node (collected for
>> histogram display), node uptime, pre-failure conditions and failover
>> node.
>
> num sessions per node is OK
> node uptime - OK
> pre-failure conditions ? you mean you want a snapshot of what was
> going on on a node just before it died ?
> failover node - this would depend upon the replication scheme - but
> should be possible...
>
>> Secondly I need to manage the cluster via manual load balancing
>
> you mean that you want to manually move x sessions from node a->b if a
> looks overloaded ? - why manually, wouldn't it be better if this
> situation never arose ? This would be WADI's aim,
>
>> and master-chain relations for failover,
>
> you'll have to expand here - not sure what you mean - our current
> replication scheme uses a pluggable algorithm to decide which nodes to
> use as bckups - so perhaps ou couls plug something in here ?
>
>> and of course stateful node shutdown & startup.
>
> we will have this. When a node shutd down it distributes its state
> direct to other peers...
>
>> I'm concentrating on the web tier first and use the Tomcat clustering
>> GBeans in Geronimo and create proper mbeans that can interact with a
>> WSDM webservice which can provide the bridge to any outside management
>
> We use Spring's JMX integration to register interesting WADI pojos
> with JMX, from there they can be queried via remote or local JMX -
> should be possible to attach your webservice if you want.
>
>>
>> Its a bit to chew on, and Geronimo is the obvious choice of app
>> server. As a matter of course I wanted to see how far along these
>> requirements are 'out of the box' and build on any existing work to
>> give back to the Geronimo and WADI crew. From what I understand,
>> although wadi is available in Geronimo v1.0 the integration is not
>> there. which is why I am starting from the gbean up. Obviously if
>> there is something I could bridge between Geronimo and WADI even at
>> an experimental level that would help me understand the proposed
>> relationship.
>
> It sounds like pretty much everything that you want is on our
> immediate todo list or done... I don't see any real requirement for
> this to be integrated with the Geronimo kernel (since you monitor via
> JMX), just as long as it can be deployed into a webcontainer within
> Geronimo ... If this is not working at the moment, then it is only a
> small patch. A complete integration with Geronimo is a larger
> proposition, that has been discussed a number of times. If you are
> interested in getting involved in this, that would be great.
>
> I'll start thinking about your requirements. We should prioritise them
> and talk about them in more detail.
>
> A good start :-)
>
> Jules
>
>>
>> Thanks again
>> Matthew Jording
>>
>> Jules Gosnell wrote:
>>
>>> Matthew Jording wrote:
>>>
>>>> Hello Jules and community.
>>>>
>>> Hi,
>>>
>>>>   I will need to have a production quality deployment of Geronimo
>>>> ready within the next couple of months. One of the requirements is
>>>> to have some level of cluster node management facility.
>>>
>>>
>>> Management is a big area :-)
>>>
>>>> Initially my thoughts are web service and WSDM/JMX mixes.
>>>
>>>
>>> Can you be a little more specific about exactly the sort of fn-ality
>>> that you are looking for. We are concentrating on scalable and
>>> performant clustered state (sessions) at the moment. WS sessions are
>>> under development. Individual WADI nodes are JMX manageable. I have
>>> ideas about how monitoring info might be aggregated for the whole
>>> cluster on each WADI node, but this is NYI... There are many other
>>> areas of clustering fn-ality.
>>>
>>>> If you are currently working on such a facility perhaps I can just
>>>> start contributing to it. At the very least I will keep you
>>>> appraised of my progress. From responses on the Geronimo Developer
>>>> lists I assume I should start with the clustering GBeans in the
>>>> various tiers and work my way up.
>>>
>>>
>>> Lets get a clear idea of what you are after and take it from there.
>>> Any contribution would be very welcome :-) A lot of thought has gone
>>> into how a Geronimo integration should look, but work in this
>>> direction has stalled. Some fresh enthusiasm in this area would be
>>> great.
>>>
>>>>
>>>> I know that this is also the WADI/ActiveCluster plan for Geronimo
>>>> and wanted to see if there is any work being done on this that I
>>>> could build upon.
>>>
>>>
>>> This part of WADI is undergoing a lot of thought at the moment - the
>>> more the merrier.
>>>
>>> Thanks for your interest in WADI,
>>>
>>>
>>> Jules
>>>
>>>>
>>>>
>>>> Thanks
>>>> Matthew Jording
>>>
>>>
>>>
>>>
>>
>
>

[Jules Gosnell-4]

Matthew Jording wrote:

> Wow, thats great Jules,
>
>  You guys have clearly been busier then the site at first indicates. I
> clearly need to start playing with with WADI and see what I can glean.
>
> So as far as pre-failure conditions, I just need to save node state
> and preserve logs and make all of that available in some manner to the
> 'cluster manager'.

two issues here...

by 'failure' as opposed to 'shutdown', I understand a situation where
there is no opportunity to take evasive or mitigating actions - so
whatever state/logs you wish to save must be incrementally backed up
during normal operation - an expensive overhead. Initial node
configuration is injected via a Spring config. All of the affected
fields (or certainly most of these) will be final - i.e. read-only - so,
I am not sure that we have much dynamically modifiable state. What sort
of state did you have in mind (other than sessions) ? As far as logs go
(unless you are talking transaction logs - and HttpSessions are
transactionless), these are written out incrementally to the destination
of your choice, so I'm not sure what further action you would want here
- can you elucidate?

regarding your 'cluster manager' - if this is a client, then, no
problem. If it is a node/peer, thenit sounds like your arch differs from
the one I have in mind for WADI. I am thinking in terms of an arch where
detailed and timely data about a node is only available directly from
this node, but smaller and less timely data about each node is published
to the rest of the cluster and may be aggregated upon any node. This
data may be used to drive e.g. automatic load/state-balancing algorithms
or a remote view of the cluster's aggregated stats... How does this
compare with the functions of your 'cluster manager'. Is it likely that
any peer in a WADI cluster could take on this role (my preferred solution).

> Manual load balancing would be a requirement only to evacuate sessions
> from a node for any potential reason other then failover.

So, not a shutdown, but a 'slimdown' ? I guess we could expose the
top-level call for this to the management API. There may be issues with
incoming requests (which would have been switched off in a shutdown
situation) pulling sessions towards a node as we are trying to push them
away to other nodes - I need to understand the context in which you are
likely to use this fn-ality. Can you give a concrete example ?

> What I meant by Master-chain relationships is the ability to
> prioritize the replication scheme.

This sounds like our pluggable replication-partner algorithm. Gianny is
the man for this - he wrote it. My understanding is that its API is
sufficient to allow the automatic selection of replication partner[s]
(the number of backups is parameterisable on a per-cluster basis) on a
per-session basis, based on any criteria that the deployer might be able
to inject into the plugin. e.g. You might wish to ensure that session
backups are always sent to a machine in another rack, room etc, to
further reduce the chance of compound failure of all your backups....
Does this sound like the sort of thing that you are after ? I believe
that you may also configure whether replication is done sync or
asynchronously etc...

>
> Priorities of requirements:
> 1. sessions per node

we have this already

http://wadi.codehaus.org/getting-started/jmxmonitoring.html

> 2. node uptime

this could be made available trivially - via the same route as (1)

> 3. failover node / replication scheme

this could be made available by registering the replication-partner
plugin with JMX as it is injected into WADI by Spring.

> 4. node state preservation

I'm still not clear on exact requirements here - unless state=sessions,
in which case this is one of our primary requirements.

Having said all of this, WADI is still someway from being enterprise
strength, but we are moving in this direction as fast as we can. Your
contribution wll help speed our progress and focus our efforts on real
world cases - very important.

If you haven't already, you should take a look at our 'Getting Started'
doc - on the 'User' menu on left hand side of home page - this will
start to give you an idea of how WADI works, and ask lots of questions
on the lists. The more questions that are asked, the better - the lists
are archived and user driven question/answers are probably better than
developer-driven ones...

Lets keep talking :-)


Jules

>
> Jules Gosnell wrote:
>
>> Matthew Jording wrote:
>>
>>> Thanks for the prompt reply Jules,
>>>
>>>    Here is a clarification of fn-ality: I need to be able to receive
>>> information from each node, Session load per node (collected for
>>> histogram display), node uptime, pre-failure conditions and failover
>>> node.
>>
>>
>> num sessions per node is OK
>> node uptime - OK
>> pre-failure conditions ? you mean you want a snapshot of what was
>> going on on a node just before it died ?
>> failover node - this would depend upon the replication scheme - but
>> should be possible...
>>
>>> Secondly I need to manage the cluster via manual load balancing
>>
>>
>> you mean that you want to manually move x sessions from node a->b if
>> a looks overloaded ? - why manually, wouldn't it be better if this
>> situation never arose ? This would be WADI's aim,
>>
>>> and master-chain relations for failover,
>>
>>
>> you'll have to expand here - not sure what you mean - our current
>> replication scheme uses a pluggable algorithm to decide which nodes
>> to use as bckups - so perhaps ou couls plug something in here ?
>>
>>> and of course stateful node shutdown & startup.
>>
>>
>> we will have this. When a node shutd down it distributes its state
>> direct to other peers...
>>
>>> I'm concentrating on the web tier first and use the Tomcat
>>> clustering GBeans in Geronimo and create proper mbeans that can
>>> interact with a WSDM webservice which can provide the bridge to any
>>> outside management
>>
>>
>> We use Spring's JMX integration to register interesting WADI pojos
>> with JMX, from there they can be queried via remote or local JMX -
>> should be possible to attach your webservice if you want.
>>
>>>
>>> Its a bit to chew on, and Geronimo is the obvious choice of app
>>> server. As a matter of course I wanted to see how far along these
>>> requirements are 'out of the box' and build on any existing work to
>>> give back to the Geronimo and WADI crew. From what I understand,
>>> although wadi is available in Geronimo v1.0 the integration is not
>>> there. which is why I am starting from the gbean up. Obviously if
>>> there is something I could bridge between Geronimo and WADI even at
>>> an experimental level that would help me understand the proposed
>>> relationship.
>>
>>
>> It sounds like pretty much everything that you want is on our
>> immediate todo list or done... I don't see any real requirement for
>> this to be integrated with the Geronimo kernel (since you monitor via
>> JMX), just as long as it can be deployed into a webcontainer within
>> Geronimo ... If this is not working at the moment, then it is only a
>> small patch. A complete integration with Geronimo is a larger
>> proposition, that has been discussed a number of times. If you are
>> interested in getting involved in this, that would be great.
>>
>> I'll start thinking about your requirements. We should prioritise
>> them and talk about them in more detail.
>>
>> A good start :-)
>>
>> Jules
>>
>>>
>>> Thanks again
>>> Matthew Jording
>>>
>>> Jules Gosnell wrote:
>>>
>>>> Matthew Jording wrote:
>>>>
>>>>> Hello Jules and community.
>>>>>
>>>> Hi,
>>>>
>>>>>   I will need to have a production quality deployment of Geronimo
>>>>> ready within the next couple of months. One of the requirements is
>>>>> to have some level of cluster node management facility.
>>>>
>>>>
>>>>
>>>> Management is a big area :-)
>>>>
>>>>> Initially my thoughts are web service and WSDM/JMX mixes.
>>>>
>>>>
>>>>
>>>> Can you be a little more specific about exactly the sort of
>>>> fn-ality that you are looking for. We are concentrating on scalable
>>>> and performant clustered state (sessions) at the moment. WS
>>>> sessions are under development. Individual WADI nodes are JMX
>>>> manageable. I have ideas about how monitoring info might be
>>>> aggregated for the whole cluster on each WADI node, but this is
>>>> NYI... There are many other areas of clustering fn-ality.
>>>>
>>>>> If you are currently working on such a facility perhaps I can just
>>>>> start contributing to it. At the very least I will keep you
>>>>> appraised of my progress. From responses on the Geronimo Developer
>>>>> lists I assume I should start with the clustering GBeans in the
>>>>> various tiers and work my way up.
>>>>
>>>>
>>>>
>>>> Lets get a clear idea of what you are after and take it from there.
>>>> Any contribution would be very welcome :-) A lot of thought has
>>>> gone into how a Geronimo integration should look, but work in this
>>>> direction has stalled. Some fresh enthusiasm in this area would be
>>>> great.
>>>>
>>>>>
>>>>> I know that this is also the WADI/ActiveCluster plan for Geronimo
>>>>> and wanted to see if there is any work being done on this that I
>>>>> could build upon.
>>>>
>>>>
>>>>
>>>> This part of WADI is undergoing a lot of thought at the moment -
>>>> the more the merrier.
>>>>
>>>> Thanks for your interest in WADI,
>>>>
>>>>
>>>> Jules
>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>> Matthew Jording
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>

--
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/

[Matthew Jording]

Jules Gosnell wrote:

> Matthew Jording wrote:
>
>> Wow, thats great Jules,
>>
>>  You guys have clearly been busier then the site at first indicates.
>> I clearly need to start playing with with WADI and see what I can glean.
>>
>> So as far as pre-failure conditions, I just need to save node state
>> and preserve logs and make all of that available in some manner to
>> the 'cluster manager'.
>
> two issues here...
>
> by 'failure' as opposed to 'shutdown', I understand a situation where
> there is no opportunity to take evasive or mitigating actions - so
> whatever state/logs you wish to save must be incrementally backed up
> during normal operation - an expensive overhead. Initial node
> configuration is injected via a Spring config. All of the affected
> fields (or certainly most of these) will be final - i.e. read-only -
> so, I am not sure that we have much dynamically modifiable state. What
> sort of state did you have in mind (other than sessions) ? As far as
> logs go (unless you are talking transaction logs - and HttpSessions
> are transactionless), these are written out incrementally to the
> destination of your choice, so I'm not sure what further action you
> would want here - can you elucidate?

I have to flesh this out. May take a bit, and some morning coffee

>
> regarding your 'cluster manager' - if this is a client, then, no
> problem. If it is a node/peer, thenit sounds like your arch differs
> from the one I have in mind for WADI. I am thinking in terms of an
> arch where detailed and timely data about a node is only available
> directly from this node, but smaller and less timely data about each
> node is published to the rest of the cluster and may be aggregated
> upon any node. This data may be used to drive e.g. automatic
> load/state-balancing algorithms or a remote view of the cluster's
> aggregated stats... How does this compare with the functions of your
> 'cluster manager'. Is it likely that any peer in a WADI cluster could
> take on this role (my preferred solution).

The 'cluster manager' could be a client of node/peer services, so the
WADI solution seems to fit.

>
>> Manual load balancing would be a requirement only to evacuate
>> sessions from a node for any potential reason other then failover.
>
> So, not a shutdown, but a 'slimdown' ? I guess we could expose the
> top-level call for this to the management API. There may be issues
> with incoming requests (which would have been switched off in a
> shutdown situation) pulling sessions towards a node as we are trying
> to push them away to other nodes - I need to understand the context in
> which you are likely to use this fn-ality. Can you give a concrete
> example ?

There are instances where a full shutdown of an application node is
undesirable during maintenance builds. If for instance our application
is not shutting down cleanly, the node shutdown bringing that instance
down can have adverse effects upon shared data stores. It can therefore
be helpful to have a mechanism to move sessions off of a node without
shutting that node down. There are other instances where I have wished
for this that will again require coffee to bring to mind.

>
>> What I meant by Master-chain relationships is the ability to
>> prioritize the replication scheme.
>
> This sounds like our pluggable replication-partner algorithm. Gianny
> is the man for this - he wrote it. My understanding is that its API is
> sufficient to allow the automatic selection of replication partner[s]
> (the number of backups is parameterisable on a per-cluster basis) on a
> per-session basis, based on any criteria that the deployer might be
> able to inject into the plugin. e.g. You might wish to ensure that
> session backups are always sent to a machine in another rack, room
> etc, to further reduce the chance of compound failure of all your
> backups.... Does this sound like the sort of thing that you are after
> ? I believe that you may also configure whether replication is done
> sync or asynchronously etc...

Yes, that sounds exactly what I need to be able to prioritize nodes for
failover. If you can call out the hooks to the algorithm and where it is
being used, that may get me started. Is Gianny listening to this list? I
would like to discuss this further and see what I can pull off.
>
>>
>> Priorities of requirements:
>> 1. sessions per node
>
> we have this already
>
> http://wadi.codehaus.org/getting-started/jmxmonitoring.html
great
>
>> 2. node uptime
>
> this could be made available trivially - via the same route as (1)
excellent
>
>> 3. failover node / replication scheme
>
> this could be made available by registering the replication-partner
> plugin with JMX as it is injected into WADI by Spring.
Sweet, looking forward to finding hooks and talking more.
>
>> 4. node state preservation
>
> I'm still not clear on exact requirements here - unless
> state=sessions, in which case this is one of our primary requirements.
state >= sessions. There are other stateful bits I wanted that again
require coffee.
>
> Having said all of this, WADI is still someway from being enterprise
> strength, but we are moving in this direction as fast as we can. Your
> contribution wll help speed our progress and focus our efforts on real
> world cases - very important.
So I am going to see if I can pull off a web service example of the
first two reqs: sessions per node, and uptime this morning. If it all
works out I'll send you the resultant build.

>
> If you haven't already, you should take a look at our 'Getting
> Started' doc - on the 'User' menu on left hand side of home page -
> this will start to give you an idea of how WADI works, and ask lots of
> questions on the lists. The more questions that are asked, the better
> - the lists are archived and user driven question/answers are probably
> better than developer-driven ones...
>
> Lets keep talking :-)
>
>
> Jules
>
>>
>> Jules Gosnell wrote:
>>
>>> Matthew Jording wrote:
>>>
>>>> Thanks for the prompt reply Jules,
>>>>
>>>>    Here is a clarification of fn-ality: I need to be able to
>>>> receive information from each node, Session load per node
>>>> (collected for histogram display), node uptime, pre-failure
>>>> conditions and failover node.
>>>
>>>
>>> num sessions per node is OK
>>> node uptime - OK
>>> pre-failure conditions ? you mean you want a snapshot of what was
>>> going on on a node just before it died ?
>>> failover node - this would depend upon the replication scheme - but
>>> should be possible...
>>>
>>>> Secondly I need to manage the cluster via manual load balancing
>>>
>>>
>>> you mean that you want to manually move x sessions from node a->b if
>>> a looks overloaded ? - why manually, wouldn't it be better if this
>>> situation never arose ? This would be WADI's aim,
>>>
>>>> and master-chain relations for failover,
>>>
>>>
>>> you'll have to expand here - not sure what you mean - our current
>>> replication scheme uses a pluggable algorithm to decide which nodes
>>> to use as bckups - so perhaps ou couls plug something in here ?
>>>
>>>> and of course stateful node shutdown & startup.
>>>
>>>
>>> we will have this. When a node shutd down it distributes its state
>>> direct to other peers...
>>>
>>>> I'm concentrating on the web tier first and use the Tomcat
>>>> clustering GBeans in Geronimo and create proper mbeans that can
>>>> interact with a WSDM webservice which can provide the bridge to any
>>>> outside management
>>>
>>>
>>> We use Spring's JMX integration to register interesting WADI pojos
>>> with JMX, from there they can be queried via remote or local JMX -
>>> should be possible to attach your webservice if you want.
>>>
>>>>
>>>> Its a bit to chew on, and Geronimo is the obvious choice of app
>>>> server. As a matter of course I wanted to see how far along these
>>>> requirements are 'out of the box' and build on any existing work to
>>>> give back to the Geronimo and WADI crew. From what I understand,
>>>> although wadi is available in Geronimo v1.0 the integration is not
>>>> there. which is why I am starting from the gbean up. Obviously if
>>>> there is something I could bridge between Geronimo and WADI even at
>>>> an experimental level that would help me understand the proposed
>>>> relationship.
>>>
>>>
>>> It sounds like pretty much everything that you want is on our
>>> immediate todo list or done... I don't see any real requirement for
>>> this to be integrated with the Geronimo kernel (since you monitor
>>> via JMX), just as long as it can be deployed into a webcontainer
>>> within Geronimo ... If this is not working at the moment, then it is
>>> only a small patch. A complete integration with Geronimo is a larger
>>> proposition, that has been discussed a number of times. If you are
>>> interested in getting involved in this, that would be great.
>>>
>>> I'll start thinking about your requirements. We should prioritise
>>> them and talk about them in more detail.
>>>
>>> A good start :-)
>>>
>>> Jules
>>>
>>>>
>>>> Thanks again
>>>> Matthew Jording
>>>>
>>>> Jules Gosnell wrote:
>>>>
>>>>> Matthew Jording wrote:
>>>>>
>>>>>> Hello Jules and community.
>>>>>>
>>>>> Hi,
>>>>>
>>>>>>   I will need to have a production quality deployment of Geronimo
>>>>>> ready within the next couple of months. One of the requirements
>>>>>> is to have some level of cluster node management facility.
>>>>>
>>>>>
>>>>>
>>>>> Management is a big area :-)
>>>>>
>>>>>> Initially my thoughts are web service and WSDM/JMX mixes.
>>>>>
>>>>>
>>>>>
>>>>> Can you be a little more specific about exactly the sort of
>>>>> fn-ality that you are looking for. We are concentrating on
>>>>> scalable and performant clustered state (sessions) at the moment.
>>>>> WS sessions are under development. Individual WADI nodes are JMX
>>>>> manageable. I have ideas about how monitoring info might be
>>>>> aggregated for the whole cluster on each WADI node, but this is
>>>>> NYI... There are many other areas of clustering fn-ality.
>>>>>
>>>>>> If you are currently working on such a facility perhaps I can
>>>>>> just start contributing to it. At the very least I will keep you
>>>>>> appraised of my progress. From responses on the Geronimo
>>>>>> Developer lists I assume I should start with the clustering
>>>>>> GBeans in the various tiers and work my way up.
>>>>>
>>>>>
>>>>>
>>>>> Lets get a clear idea of what you are after and take it from
>>>>> there. Any contribution would be very welcome :-) A lot of thought
>>>>> has gone into how a Geronimo integration should look, but work in
>>>>> this direction has stalled. Some fresh enthusiasm in this area
>>>>> would be great.
>>>>>
>>>>>>
>>>>>> I know that this is also the WADI/ActiveCluster plan for Geronimo
>>>>>> and wanted to see if there is any work being done on this that I
>>>>>> could build upon.
>>>>>
>>>>>
>>>>>
>>>>> This part of WADI is undergoing a lot of thought at the moment -
>>>>> the more the merrier.
>>>>>
>>>>> Thanks for your interest in WADI,
>>>>>
>>>>>
>>>>> Jules
>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Matthew Jording
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>

[Jules Gosnell-4]

Matthew Jording wrote:

> Jules Gosnell wrote:
>
>> Matthew Jording wrote:
>>
>>> Wow, thats great Jules,
>>>
>>>  You guys have clearly been busier then the site at first indicates.
>>> I clearly need to start playing with with WADI and see what I can
>>> glean.
>>>
>>> So as far as pre-failure conditions, I just need to save node state
>>> and preserve logs and make all of that available in some manner to
>>> the 'cluster manager'.
>>
>>
>> two issues here...
>>
>> by 'failure' as opposed to 'shutdown', I understand a situation where
>> there is no opportunity to take evasive or mitigating actions - so
>> whatever state/logs you wish to save must be incrementally backed up
>> during normal operation - an expensive overhead. Initial node
>> configuration is injected via a Spring config. All of the affected
>> fields (or certainly most of these) will be final - i.e. read-only -
>> so, I am not sure that we have much dynamically modifiable state.
>> What sort of state did you have in mind (other than sessions) ? As
>> far as logs go (unless you are talking transaction logs - and
>> HttpSessions are transactionless), these are written out
>> incrementally to the destination of your choice, so I'm not sure what
>> further action you would want here - can you elucidate?
>
> I have to flesh this out. May take a bit, and some morning coffee

OK - I'll hang on...

>>
>> regarding your 'cluster manager' - if this is a client, then, no
>> problem. If it is a node/peer, thenit sounds like your arch differs
>> from the one I have in mind for WADI. I am thinking in terms of an
>> arch where detailed and timely data about a node is only available
>> directly from this node, but smaller and less timely data about each
>> node is published to the rest of the cluster and may be aggregated
>> upon any node. This data may be used to drive e.g. automatic
>> load/state-balancing algorithms or a remote view of the cluster's
>> aggregated stats... How does this compare with the functions of your
>> 'cluster manager'. Is it likely that any peer in a WADI cluster could
>> take on this role (my preferred solution).
>
> The 'cluster manager' could be a client of node/peer services, so the
> WADI solution seems to fit.

excellent :-)

>>
>>> Manual load balancing would be a requirement only to evacuate
>>> sessions from a node for any potential reason other then failover.
>>
>>
>> So, not a shutdown, but a 'slimdown' ? I guess we could expose the
>> top-level call for this to the management API. There may be issues
>> with incoming requests (which would have been switched off in a
>> shutdown situation) pulling sessions towards a node as we are trying
>> to push them away to other nodes - I need to understand the context
>> in which you are likely to use this fn-ality. Can you give a concrete
>> example ?
>
> There are instances where a full shutdown of an application node is
> undesirable during maintenance builds. If for instance our application
> is not shutting down cleanly, the node shutdown bringing that instance
> down can have adverse effects upon shared data stores. It can
> therefore be helpful to have a mechanism to move sessions off of a
> node without shutting that node down. There are other instances where
> I have wished for this that will again require coffee to bring to mind.

I see - so you might force sessions to evacuate a 'sick' node, before
performing maintenance on it - interesting idea - I will give it some
thought..

>>
>>> What I meant by Master-chain relationships is the ability to
>>> prioritize the replication scheme.
>>
>>
>> This sounds like our pluggable replication-partner algorithm. Gianny
>> is the man for this - he wrote it. My understanding is that its API
>> is sufficient to allow the automatic selection of replication
>> partner[s] (the number of backups is parameterisable on a per-cluster
>> basis) on a per-session basis, based on any criteria that the
>> deployer might be able to inject into the plugin. e.g. You might wish
>> to ensure that session backups are always sent to a machine in
>> another rack, room etc, to further reduce the chance of compound
>> failure of all your backups.... Does this sound like the sort of
>> thing that you are after ? I believe that you may also configure
>> whether replication is done sync or asynchronously etc...
>
> Yes, that sounds exactly what I need to be able to prioritize nodes
> for failover. If you can call out the hooks to the algorithm and where
> it is being used, that may get me started. Is Gianny listening to this
> list? I would like to discuss this further and see what I can pull off.

He's registered with the list, but may not be following this thread.
I've cc-ed him directly to draw his attention to it... It will be
interesting to see exactly what you are going to do with this ability...
It was obvious that we needed it, but we were not sure what would be
useful default fn-ality. I think that Gianny has provided a plugin which
does a fairly basic 'round-robin' approach so that all replicant
services end up carrying the same sort of numbers of replicants... This
is an example of where the non-timely data that I was talking about
above may come in useful....

>>
>>>
>>> Priorities of requirements:
>>> 1. sessions per node
>>
>>
>> we have this already
>>
>> http://wadi.codehaus.org/getting-started/jmxmonitoring.html
>
> great
>
>>
>>> 2. node uptime
>>
>>
>> this could be made available trivially - via the same route as (1)
>
> excellent
>
>>
>>> 3. failover node / replication scheme
>>
>>
>> this could be made available by registering the replication-partner
>> plugin with JMX as it is injected into WADI by Spring.
>
> Sweet, looking forward to finding hooks and talking more.
>
>>
>>> 4. node state preservation
>>
>>
>> I'm still not clear on exact requirements here - unless
>> state=sessions, in which case this is one of our primary requirements.
>
> state >= sessions. There are other stateful bits I wanted that again
> require coffee.

ohoh ! I know that other approaches distribute e.g. Context attributes.
I have avoided doing this as it is not mentioned by the spec and
probably brings a swathe of problems of its own. However, I am always
open to persuasion :-), especially if you are volunteering to do the work !

>>
>> Having said all of this, WADI is still someway from being enterprise
>> strength, but we are moving in this direction as fast as we can. Your
>> contribution wll help speed our progress and focus our efforts on
>> real world cases - very important.
>
> So I am going to see if I can pull off a web service example of the
> first two reqs: sessions per node, and uptime this morning. If it all
> works out I'll send you the resultant build.

uptime can be calculated on a per-node basis from the 'birthTime'
attribute carried in the Map that is distributed by each node. The code
around this area is being refactored at the moment. If you are happy to
work from CVS, I could stick a method in for you today to return this
data...  If you concentrate on getting a build out of CVS up and running
and walking through the Getting Started doc, I will look at this. Bring
any issues straight back to the list and we will get you up and running
as  painlessly as possible.

regards,


Jules
 

>>
>> If you haven't already, you should take a look at our 'Getting
>> Started' doc - on the 'User' menu on left hand side of home page -
>> this will start to give you an idea of how WADI works, and ask lots
>> of questions on the lists. The more questions that are asked, the
>> better - the lists are archived and user driven question/answers are
>> probably better than developer-driven ones...
>>
>> Lets keep talking :-)
>>
>>
>> Jules
>>
>>>
>>> Jules Gosnell wrote:
>>>
>>>> Matthew Jording wrote:
>>>>
>>>>> Thanks for the prompt reply Jules,
>>>>>
>>>>>    Here is a clarification of fn-ality: I need to be able to
>>>>> receive information from each node, Session load per node
>>>>> (collected for histogram display), node uptime, pre-failure
>>>>> conditions and failover node.
>>>>
>>>>
>>>>
>>>> num sessions per node is OK
>>>> node uptime - OK
>>>> pre-failure conditions ? you mean you want a snapshot of what was
>>>> going on on a node just before it died ?
>>>> failover node - this would depend upon the replication scheme - but
>>>> should be possible...
>>>>
>>>>> Secondly I need to manage the cluster via manual load balancing
>>>>
>>>>
>>>>
>>>> you mean that you want to manually move x sessions from node a->b
>>>> if a looks overloaded ? - why manually, wouldn't it be better if
>>>> this situation never arose ? This would be WADI's aim,
>>>>
>>>>> and master-chain relations for failover,
>>>>
>>>>
>>>>
>>>> you'll have to expand here - not sure what you mean - our current
>>>> replication scheme uses a pluggable algorithm to decide which nodes
>>>> to use as bckups - so perhaps ou couls plug something in here ?
>>>>
>>>>> and of course stateful node shutdown & startup.
>>>>
>>>>
>>>>
>>>> we will have this. When a node shutd down it distributes its state
>>>> direct to other peers...
>>>>
>>>>> I'm concentrating on the web tier first and use the Tomcat
>>>>> clustering GBeans in Geronimo and create proper mbeans that can
>>>>> interact with a WSDM webservice which can provide the bridge to
>>>>> any outside management
>>>>
>>>>
>>>>
>>>> We use Spring's JMX integration to register interesting WADI pojos
>>>> with JMX, from there they can be queried via remote or local JMX -
>>>> should be possible to attach your webservice if you want.
>>>>
>>>>>
>>>>> Its a bit to chew on, and Geronimo is the obvious choice of app
>>>>> server. As a matter of course I wanted to see how far along these
>>>>> requirements are 'out of the box' and build on any existing work
>>>>> to give back to the Geronimo and WADI crew. From what I
>>>>> understand, although wadi is available in Geronimo v1.0 the
>>>>> integration is not there. which is why I am starting from the
>>>>> gbean up. Obviously if there is something I could bridge between
>>>>> Geronimo and WADI even at an experimental level that would help me
>>>>> understand the proposed relationship.
>>>>
>>>>
>>>>
>>>> It sounds like pretty much everything that you want is on our
>>>> immediate todo list or done... I don't see any real requirement for
>>>> this to be integrated with the Geronimo kernel (since you monitor
>>>> via JMX), just as long as it can be deployed into a webcontainer
>>>> within Geronimo ... If this is not working at the moment, then it
>>>> is only a small patch. A complete integration with Geronimo is a
>>>> larger proposition, that has been discussed a number of times. If
>>>> you are interested in getting involved in this, that would be great.
>>>>
>>>> I'll start thinking about your requirements. We should prioritise
>>>> them and talk about them in more detail.
>>>>
>>>> A good start :-)
>>>>
>>>> Jules
>>>>
>>>>>
>>>>> Thanks again
>>>>> Matthew Jording
>>>>>
>>>>> Jules Gosnell wrote:
>>>>>
>>>>>> Matthew Jording wrote:
>>>>>>
>>>>>>> Hello Jules and community.
>>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>>   I will need to have a production quality deployment of
>>>>>>> Geronimo ready within the next couple of months. One of the
>>>>>>> requirements is to have some level of cluster node management
>>>>>>> facility.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Management is a big area :-)
>>>>>>
>>>>>>> Initially my thoughts are web service and WSDM/JMX mixes.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Can you be a little more specific about exactly the sort of
>>>>>> fn-ality that you are looking for. We are concentrating on
>>>>>> scalable and performant clustered state (sessions) at the moment.
>>>>>> WS sessions are under development. Individual WADI nodes are JMX
>>>>>> manageable. I have ideas about how monitoring info might be
>>>>>> aggregated for the whole cluster on each WADI node, but this is
>>>>>> NYI... There are many other areas of clustering fn-ality.
>>>>>>
>>>>>>> If you are currently working on such a facility perhaps I can
>>>>>>> just start contributing to it. At the very least I will keep you
>>>>>>> appraised of my progress. From responses on the Geronimo
>>>>>>> Developer lists I assume I should start with the clustering
>>>>>>> GBeans in the various tiers and work my way up.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Lets get a clear idea of what you are after and take it from
>>>>>> there. Any contribution would be very welcome :-) A lot of
>>>>>> thought has gone into how a Geronimo integration should look, but
>>>>>> work in this direction has stalled. Some fresh enthusiasm in this
>>>>>> area would be great.
>>>>>>
>>>>>>>
>>>>>>> I know that this is also the WADI/ActiveCluster plan for
>>>>>>> Geronimo and wanted to see if there is any work being done on
>>>>>>> this that I could build upon.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> This part of WADI is undergoing a lot of thought at the moment -
>>>>>> the more the merrier.
>>>>>>
>>>>>> Thanks for your interest in WADI,
>>>>>>
>>>>>>
>>>>>> Jules
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Matthew Jording
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>

--
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/

[Jules Gosnell-4]

spot the deliberate mistake :-)

When I said CVS, I meant SVN...

All repo details etc shoul be on the website (wad.codehaus.org) along
with Developer resources (menu on left hand side).

If anything is missing or out of date, just shout,

Jules



Jules Gosnell wrote:

> Matthew Jording wrote:
>
>> Jules Gosnell wrote:
>>
>>> Matthew Jording wrote:
>>>
>>>> Wow, thats great Jules,
>>>>
>>>>  You guys have clearly been busier then the site at first
>>>> indicates. I clearly need to start playing with with WADI and see
>>>> what I can glean.
>>>>
>>>> So as far as pre-failure conditions, I just need to save node state
>>>> and preserve logs and make all of that available in some manner to
>>>> the 'cluster manager'.
>>>
>>>
>>>
>>> two issues here...
>>>
>>> by 'failure' as opposed to 'shutdown', I understand a situation
>>> where there is no opportunity to take evasive or mitigating actions
>>> - so whatever state/logs you wish to save must be incrementally
>>> backed up during normal operation - an expensive overhead. Initial
>>> node configuration is injected via a Spring config. All of the
>>> affected fields (or certainly most of these) will be final - i.e.
>>> read-only - so, I am not sure that we have much dynamically
>>> modifiable state. What sort of state did you have in mind (other
>>> than sessions) ? As far as logs go (unless you are talking
>>> transaction logs - and HttpSessions are transactionless), these are
>>> written out incrementally to the destination of your choice, so I'm
>>> not sure what further action you would want here - can you elucidate?
>>
>>
>> I have to flesh this out. May take a bit, and some morning coffee
>
>
> OK - I'll hang on...
>
>>>
>>> regarding your 'cluster manager' - if this is a client, then, no
>>> problem. If it is a node/peer, thenit sounds like your arch differs
>>> from the one I have in mind for WADI. I am thinking in terms of an
>>> arch where detailed and timely data about a node is only available
>>> directly from this node, but smaller and less timely data about each
>>> node is published to the rest of the cluster and may be aggregated
>>> upon any node. This data may be used to drive e.g. automatic
>>> load/state-balancing algorithms or a remote view of the cluster's
>>> aggregated stats... How does this compare with the functions of your
>>> 'cluster manager'. Is it likely that any peer in a WADI cluster
>>> could take on this role (my preferred solution).
>>
>>
>> The 'cluster manager' could be a client of node/peer services, so the
>> WADI solution seems to fit.
>
>
> excellent :-)
>
>>>
>>>> Manual load balancing would be a requirement only to evacuate
>>>> sessions from a node for any potential reason other then failover.
>>>
>>>
>>>
>>> So, not a shutdown, but a 'slimdown' ? I guess we could expose the
>>> top-level call for this to the management API. There may be issues
>>> with incoming requests (which would have been switched off in a
>>> shutdown situation) pulling sessions towards a node as we are trying
>>> to push them away to other nodes - I need to understand the context
>>> in which you are likely to use this fn-ality. Can you give a
>>> concrete example ?
>>
>>
>> There are instances where a full shutdown of an application node is
>> undesirable during maintenance builds. If for instance our
>> application is not shutting down cleanly, the node shutdown bringing
>> that instance down can have adverse effects upon shared data stores.
>> It can therefore be helpful to have a mechanism to move sessions off
>> of a node without shutting that node down. There are other instances
>> where I have wished for this that will again require coffee to bring
>> to mind.
>
>
> I see - so you might force sessions to evacuate a 'sick' node, before
> performing maintenance on it - interesting idea - I will give it some
> thought..
>
>>>
>>>> What I meant by Master-chain relationships is the ability to
>>>> prioritize the replication scheme.
>>>
>>>
>>>
>>> This sounds like our pluggable replication-partner algorithm. Gianny
>>> is the man for this - he wrote it. My understanding is that its API
>>> is sufficient to allow the automatic selection of replication
>>> partner[s] (the number of backups is parameterisable on a
>>> per-cluster basis) on a per-session basis, based on any criteria
>>> that the deployer might be able to inject into the plugin. e.g. You
>>> might wish to ensure that session backups are always sent to a
>>> machine in another rack, room etc, to further reduce the chance of
>>> compound failure of all your backups.... Does this sound like the
>>> sort of thing that you are after ? I believe that you may also
>>> configure whether replication is done sync or asynchronously etc...
>>
>>
>> Yes, that sounds exactly what I need to be able to prioritize nodes
>> for failover. If you can call out the hooks to the algorithm and
>> where it is being used, that may get me started. Is Gianny listening
>> to this list? I would like to discuss this further and see what I can
>> pull off.
>
>
> He's registered with the list, but may not be following this thread.
> I've cc-ed him directly to draw his attention to it... It will be
> interesting to see exactly what you are going to do with this
> ability... It was obvious that we needed it, but we were not sure what
> would be useful default fn-ality. I think that Gianny has provided a
> plugin which does a fairly basic 'round-robin' approach so that all
> replicant services end up carrying the same sort of numbers of
> replicants... This is an example of where the non-timely data that I
> was talking about above may come in useful....
>
>>>
>>>>
>>>> Priorities of requirements:
>>>> 1. sessions per node
>>>
>>>
>>>
>>> we have this already
>>>
>>> http://wadi.codehaus.org/getting-started/jmxmonitoring.html
>>
>>
>> great
>>
>>>
>>>> 2. node uptime
>>>
>>>
>>>
>>> this could be made available trivially - via the same route as (1)
>>
>>
>> excellent
>>
>>>
>>>> 3. failover node / replication scheme
>>>
>>>
>>>
>>> this could be made available by registering the replication-partner
>>> plugin with JMX as it is injected into WADI by Spring.
>>
>>
>> Sweet, looking forward to finding hooks and talking more.
>>
>>>
>>>> 4. node state preservation
>>>
>>>
>>>
>>> I'm still not clear on exact requirements here - unless
>>> state=sessions, in which case this is one of our primary requirements.
>>
>>
>> state >= sessions. There are other stateful bits I wanted that again
>> require coffee.
>
>
> ohoh ! I know that other approaches distribute e.g. Context
> attributes. I have avoided doing this as it is not mentioned by the
> spec and probably brings a swathe of problems of its own. However, I
> am always open to persuasion :-), especially if you are volunteering
> to do the work !
>
>>>
>>> Having said all of this, WADI is still someway from being enterprise
>>> strength, but we are moving in this direction as fast as we can.
>>> Your contribution wll help speed our progress and focus our efforts
>>> on real world cases - very important.
>>
>>
>> So I am going to see if I can pull off a web service example of the
>> first two reqs: sessions per node, and uptime this morning. If it all
>> works out I'll send you the resultant build.
>
>
> uptime can be calculated on a per-node basis from the 'birthTime'
> attribute carried in the Map that is distributed by each node. The
> code around this area is being refactored at the moment. If you are
> happy to work from CVS, I could stick a method in for you today to
> return this data...  If you concentrate on getting a build out of CVS
> up and running and walking through the Getting Started doc, I will
> look at this. Bring any issues straight back to the list and we will
> get you up and running as  painlessly as possible.
>
> regards,
>
>
> Jules
>
>
>>>
>>> If you haven't already, you should take a look at our 'Getting
>>> Started' doc - on the 'User' menu on left hand side of home page -
>>> this will start to give you an idea of how WADI works, and ask lots
>>> of questions on the lists. The more questions that are asked, the
>>> better - the lists are archived and user driven question/answers are
>>> probably better than developer-driven ones...
>>>
>>> Lets keep talking :-)
>>>
>>>
>>> Jules
>>>
>>>>
>>>> Jules Gosnell wrote:
>>>>
>>>>> Matthew Jording wrote:
>>>>>
>>>>>> Thanks for the prompt reply Jules,
>>>>>>
>>>>>>    Here is a clarification of fn-ality: I need to be able to
>>>>>> receive information from each node, Session load per node
>>>>>> (collected for histogram display), node uptime, pre-failure
>>>>>> conditions and failover node.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> num sessions per node is OK
>>>>> node uptime - OK
>>>>> pre-failure conditions ? you mean you want a snapshot of what was
>>>>> going on on a node just before it died ?
>>>>> failover node - this would depend upon the replication scheme -
>>>>> but should be possible...
>>>>>
>>>>>> Secondly I need to manage the cluster via manual load balancing
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> you mean that you want to manually move x sessions from node a->b
>>>>> if a looks overloaded ? - why manually, wouldn't it be better if
>>>>> this situation never arose ? This would be WADI's aim,
>>>>>
>>>>>> and master-chain relations for failover,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> you'll have to expand here - not sure what you mean - our current
>>>>> replication scheme uses a pluggable algorithm to decide which
>>>>> nodes to use as bckups - so perhaps ou couls plug something in here ?
>>>>>
>>>>>> and of course stateful node shutdown & startup.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> we will have this. When a node shutd down it distributes its state
>>>>> direct to other peers...
>>>>>
>>>>>> I'm concentrating on the web tier first and use the Tomcat
>>>>>> clustering GBeans in Geronimo and create proper mbeans that can
>>>>>> interact with a WSDM webservice which can provide the bridge to
>>>>>> any outside management
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> We use Spring's JMX integration to register interesting WADI pojos
>>>>> with JMX, from there they can be queried via remote or local JMX -
>>>>> should be possible to attach your webservice if you want.
>>>>>
>>>>>>
>>>>>> Its a bit to chew on, and Geronimo is the obvious choice of app
>>>>>> server. As a matter of course I wanted to see how far along these
>>>>>> requirements are 'out of the box' and build on any existing work
>>>>>> to give back to the Geronimo and WADI crew. From what I
>>>>>> understand, although wadi is available in Geronimo v1.0 the
>>>>>> integration is not there. which is why I am starting from the
>>>>>> gbean up. Obviously if there is something I could bridge between
>>>>>> Geronimo and WADI even at an experimental level that would help
>>>>>> me understand the proposed relationship.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> It sounds like pretty much everything that you want is on our
>>>>> immediate todo list or done... I don't see any real requirement
>>>>> for this to be integrated with the Geronimo kernel (since you
>>>>> monitor via JMX), just as long as it can be deployed into a
>>>>> webcontainer within Geronimo ... If this is not working at the
>>>>> moment, then it is only a small patch. A complete integration with
>>>>> Geronimo is a larger proposition, that has been discussed a number
>>>>> of times. If you are interested in getting involved in this, that
>>>>> would be great.
>>>>>
>>>>> I'll start thinking about your requirements. We should prioritise
>>>>> them and talk about them in more detail.
>>>>>
>>>>> A good start :-)
>>>>>
>>>>> Jules
>>>>>
>>>>>>
>>>>>> Thanks again
>>>>>> Matthew Jording
>>>>>>
>>>>>> Jules Gosnell wrote:
>>>>>>
>>>>>>> Matthew Jording wrote:
>>>>>>>
>>>>>>>> Hello Jules and community.
>>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>>   I will need to have a production quality deployment of
>>>>>>>> Geronimo ready within the next couple of months. One of the
>>>>>>>> requirements is to have some level of cluster node management
>>>>>>>> facility.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Management is a big area :-)
>>>>>>>
>>>>>>>> Initially my thoughts are web service and WSDM/JMX mixes.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Can you be a little more specific about exactly the sort of
>>>>>>> fn-ality that you are looking for. We are concentrating on
>>>>>>> scalable and performant clustered state (sessions) at the
>>>>>>> moment. WS sessions are under development. Individual WADI nodes
>>>>>>> are JMX manageable. I have ideas about how monitoring info might
>>>>>>> be aggregated for the whole cluster on each WADI node, but this
>>>>>>> is NYI... There are many other areas of clustering fn-ality.
>>>>>>>
>>>>>>>> If you are currently working on such a facility perhaps I can
>>>>>>>> just start contributing to it. At the very least I will keep
>>>>>>>> you appraised of my progress. From responses on the Geronimo
>>>>>>>> Developer lists I assume I should start with the clustering
>>>>>>>> GBeans in the various tiers and work my way up.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Lets get a clear idea of what you are after and take it from
>>>>>>> there. Any contribution would be very welcome :-) A lot of
>>>>>>> thought has gone into how a Geronimo integration should look,
>>>>>>> but work in this direction has stalled. Some fresh enthusiasm in
>>>>>>> this area would be great.
>>>>>>>
>>>>>>>>
>>>>>>>> I know that this is also the WADI/ActiveCluster plan for
>>>>>>>> Geronimo and wanted to see if there is any work being done on
>>>>>>>> this that I could build upon.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This part of WADI is undergoing a lot of thought at the moment -
>>>>>>> the more the merrier.
>>>>>>>
>>>>>>> Thanks for your interest in WADI,
>>>>>>>
>>>>>>>
>>>>>>> Jules
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Matthew Jording
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>

--
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/

[Gianny Damour-2]

Jules Gosnell wrote:

> Matthew Jording wrote:
>
<snip>

>>>
>>> This sounds like our pluggable replication-partner algorithm. Gianny
>>> is the man for this - he wrote it. My understanding is that its API
>>> is sufficient to allow the automatic selection of replication
>>> partner[s] (the number of backups is parameterisable on a
>>> per-cluster basis) on a per-session basis, based on any criteria
>>> that the deployer might be able to inject into the plugin. e.g. You
>>> might wish to ensure that session backups are always sent to a
>>> machine in another rack, room etc, to further reduce the chance of
>>> compound failure of all your backups.... Does this sound like the
>>> sort of thing that you are after ? I believe that you may also
>>> configure whether replication is done sync or asynchronously etc...
>>
>>
>> Yes, that sounds exactly what I need to be able to prioritize nodes
>> for failover. If you can call out the hooks to the algorithm and
>> where it is being used, that may get me started. Is Gianny listening
>> to this list? I would like to discuss this further and see what I can
>> pull off.
>
>
> He's registered with the list, but may not be following this thread.
> I've cc-ed him directly to draw his attention to it... It will be
> interesting to see exactly what you are going to do with this
> ability... It was obvious that we needed it, but we were not sure what
> would be useful default fn-ality. I think that Gianny has provided a
> plugin which does a fairly basic 'round-robin' approach so that all
> replicant services end up carrying the same sort of numbers of
> replicants... This is an example of where the non-timely data that I
> was talking about above may come in useful....

Actually, I was following this thread.

There are a couple of information that can be exposed :
1. count measurements of backing storage election: number of times that
the algorithm has been executed to identify where secondaries should be
stored;
2. count measurements of backing storage re-election: number of times
that the algorithm has been executed to re-organize secondaries;
3. count measurements of secondaries managed by a given node; and
4. nodes currently considered as potential secondaries for a given node.
This information could be set by a management console (as long as you
remove nodes from this potential set and do not try to add new ones).

Regarding the replication algorithm, it is easy to plug-in whatever
strategy you want.

Also, when would you like to have some "level of cluster node management
facility" for Geronimo with WADI? As Jules stated it, WADI is nearly
feature complete; yet, some efforts are under-way to complete it and
make it production ready. So, if your delivery calendar is aggressive,
then, at this time, WADI may not be the safest choice for your project.

Thanks,
Gianny

[Jules Gosnell-4]

>
>>
>>> 2. node uptime
>>
>>
>> this could be made available trivially - via the same route as (1)
>
> excellent

I've just added birthTime and upTime accessors to StandardManager - they
should be inherited and therefore available on Distributable and
ClusteredManager as well. Check out how I hook pojos into the Spring JMX
integration in the demo webapp's WEB-INF/wadi-web.xml - you may have to
uncomment some stuff...

Any problems, just give drop a mail to the list.

Jules


--
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/

[Jules Gosnell-4]

>
> Also, when would you like to have some "level of cluster node
> management facility" for Geronimo with WADI? As Jules stated it, WADI
> is nearly feature complete; yet, some efforts are under-way to
> complete it and make it production ready. So, if your delivery
> calendar is aggressive, then, at this time, WADI may not be the safest
> choice for your project.
>
nicely put, Gianny :-)

Matthew mentioned a timeline of 'a couple of months'. I guess once we
know exactly what he needs and he knows exactly what we have, we can put
together a roadmap and decide if this is going to fly or not ?

Jules

> Thanks,
> Gianny
>


--
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/

[Gianny Damour-2]

Jules Gosnell wrote:

>
>>
>> Also, when would you like to have some "level of cluster node
>> management facility" for Geronimo with WADI? As Jules stated it, WADI
>> is nearly feature complete; yet, some efforts are under-way to
>> complete it and make it production ready. So, if your delivery
>> calendar is aggressive, then, at this time, WADI may not be the
>> safest choice for your project.
>>
> nicely put, Gianny :-)
>
> Matthew mentioned a timeline of 'a couple of months'. I guess once we
> know exactly what he needs and he knows exactly what we have, we can
> put together a roadmap and decide if this is going to fly or not ?

I agree. If a couple of months is at least two months, then, a priori,
this is going to fly.

Thanks,
Gianny

>
> Jules
>
>> Thanks,
>> Gianny
>>
>
>

[Matthew Jording]

Gianny Damour wrote:

> Jules Gosnell wrote:
>
>> Matthew Jording wrote:
>>
> <snip>
>
>>>>
>>>> This sounds like our pluggable replication-partner algorithm.
>>>> Gianny is the man for this - he wrote it. My understanding is that
>>>> its API is sufficient to allow the automatic selection of
>>>> replication partner[s] (the number of backups is parameterisable on
>>>> a per-cluster basis) on a per-session basis, based on any criteria
>>>> that the deployer might be able to inject into the plugin. e.g. You
>>>> might wish to ensure that session backups are always sent to a
>>>> machine in another rack, room etc, to further reduce the chance of
>>>> compound failure of all your backups.... Does this sound like the
>>>> sort of thing that you are after ? I believe that you may also
>>>> configure whether replication is done sync or asynchronously etc...
>>>
>>>
>>> Yes, that sounds exactly what I need to be able to prioritize nodes
>>> for failover. If you can call out the hooks to the algorithm and
>>> where it is being used, that may get me started. Is Gianny listening
>>> to this list? I would like to discuss this further and see what I
>>> can pull off.
>>
>>
>> He's registered with the list, but may not be following this thread.
>> I've cc-ed him directly to draw his attention to it... It will be
>> interesting to see exactly what you are going to do with this
>> ability... It was obvious that we needed it, but we were not sure
>> what would be useful default fn-ality. I think that Gianny has
>> provided a plugin which does a fairly basic 'round-robin' approach so
>> that all replicant services end up carrying the same sort of numbers
>> of replicants... This is an example of where the non-timely data that
>> I was talking about above may come in useful....
>
> Actually, I was following this thread.
>
> There are a couple of information that can be exposed :
> 1. count measurements of backing storage election: number of times
> that the algorithm has been executed to identify where secondaries
> should be stored;
> 2. count measurements of backing storage re-election: number of times
> that the algorithm has been executed to re-organize secondaries;
> 3. count measurements of secondaries managed by a given node; and
> 4. nodes currently considered as potential secondaries for a given
> node. This information could be set by a management console (as long
> as you remove nodes from this potential set and do not try to add new
> ones).
>
> Regarding the replication algorithm, it is easy to plug-in whatever
> strategy you want.
>
> Also, when would you like to have some "level of cluster node
> management facility" for Geronimo with WADI? As Jules stated it, WADI
> is nearly feature complete; yet, some efforts are under-way to
> complete it and make it production ready. So, if your delivery
> calendar is aggressive, then, at this time, WADI may not be the safest
> choice for your project.
>
> Thanks,
> Gianny
>
>

Thanks Gianny,
    As soon as I am up to speed with my basic functionality I will be
asking how to hook into the features of the algorithm. My delivery
schedule is aggressive, but we are committed to the Geronimo platform.
If our efforts will be spent ensuring our requirements, it sounds like
the time is best spent standing on the shoulders of the work you have
done thus far.

[Matthew Jording]

Doesn't look like maven is finding the wadi axis2 dependencies via
ibiblio. I will find a copy and install it to complete the build, but I
figured you should know.

[ERROR] BUILD ERROR
[INFO]
----------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

required artifacts missing:
  org.codehaus.wadi:wadi-axis2:jar:2.0M2-SNAPSHOT

for the artifact:
  org.codehaus.wadi:wadi-assembly:pom:2.0M2-SNAPSHOT

from the specified remote repositories:
  central (http://repo1.maven.org/maven2),
  wadi-dependencies (http://dist.codehaus.org/wadi/dependencies/maven2),
  maven2-ibiblio (http://ibiblio.org/maven2)


Jules Gosnell wrote:

> spot the deliberate mistake :-)
>
> When I said CVS, I meant SVN...
>
> All repo details etc shoul be on the website (wad.codehaus.org) along
> with Developer resources (menu on left hand side).
>
> If anything is missing or out of date, just shout,
>
> Jules
>
>
>
> Jules Gosnell wrote:
>
>> Matthew Jording wrote:
>>
>>> Jules Gosnell wrote:
>>>
>>>> Matthew Jording wrote:
>>>>
>>>>> Wow, thats great Jules,
>>>>>
>>>>>  You guys have clearly been busier then the site at first
>>>>> indicates. I clearly need to start playing with with WADI and see
>>>>> what I can glean.
>>>>>
>>>>> So as far as pre-failure conditions, I just need to save node
>>>>> state and preserve logs and make all of that available in some
>>>>> manner to the 'cluster manager'.
>>>>
>>>>
>>>>
>>>> two issues here...
>>>>
>>>> by 'failure' as opposed to 'shutdown', I understand a situation
>>>> where there is no opportunity to take evasive or mitigating actions
>>>> - so whatever state/logs you wish to save must be incrementally
>>>> backed up during normal operation - an expensive overhead. Initial
>>>> node configuration is injected via a Spring config. All of the
>>>> affected fields (or certainly most of these) will be final - i.e.
>>>> read-only - so, I am not sure that we have much dynamically
>>>> modifiable state. What sort of state did you have in mind (other
>>>> than sessions) ? As far as logs go (unless you are talking
>>>> transaction logs - and HttpSessions are transactionless), these are
>>>> written out incrementally to the destination of your choice, so I'm
>>>> not sure what further action you would want here - can you elucidate?
>>>
>>>
>>> I have to flesh this out. May take a bit, and some morning coffee
>>
>>
>> OK - I'll hang on...
>>
>>>>
>>>> regarding your 'cluster manager' - if this is a client, then, no
>>>> problem. If it is a node/peer, thenit sounds like your arch differs
>>>> from the one I have in mind for WADI. I am thinking in terms of an
>>>> arch where detailed and timely data about a node is only available
>>>> directly from this node, but smaller and less timely data about
>>>> each node is published to the rest of the cluster and may be
>>>> aggregated upon any node. This data may be used to drive e.g.
>>>> automatic load/state-balancing algorithms or a remote view of the
>>>> cluster's aggregated stats... How does this compare with the
>>>> functions of your 'cluster manager'. Is it likely that any peer in
>>>> a WADI cluster could take on this role (my preferred solution).
>>>
>>>
>>> The 'cluster manager' could be a client of node/peer services, so
>>> the WADI solution seems to fit.
>>
>>
>> excellent :-)
>>
>>>>
>>>>> Manual load balancing would be a requirement only to evacuate
>>>>> sessions from a node for any potential reason other then failover.
>>>>
>>>>
>>>>
>>>> So, not a shutdown, but a 'slimdown' ? I guess we could expose the
>>>> top-level call for this to the management API. There may be issues
>>>> with incoming requests (which would have been switched off in a
>>>> shutdown situation) pulling sessions towards a node as we are
>>>> trying to push them away to other nodes - I need to understand the
>>>> context in which you are likely to use this fn-ality. Can you give
>>>> a concrete example ?
>>>
>>>
>>> There are instances where a full shutdown of an application node is
>>> undesirable during maintenance builds. If for instance our
>>> application is not shutting down cleanly, the node shutdown bringing
>>> that instance down can have adverse effects upon shared data stores.
>>> It can therefore be helpful to have a mechanism to move sessions off
>>> of a node without shutting that node down. There are other instances
>>> where I have wished for this that will again require coffee to bring
>>> to mind.
>>
>>
>> I see - so you might force sessions to evacuate a 'sick' node, before
>> performing maintenance on it - interesting idea - I will give it some
>> thought..
>>
>>>>
>>>>> What I meant by Master-chain relationships is the ability to
>>>>> prioritize the replication scheme.
>>>>
>>>>
>>>>
>>>> This sounds like our pluggable replication-partner algorithm.
>>>> Gianny is the man for this - he wrote it. My understanding is that
>>>> its API is sufficient to allow the automatic selection of
>>>> replication partner[s] (the number of backups is parameterisable on
>>>> a per-cluster basis) on a per-session basis, based on any criteria
>>>> that the deployer might be able to inject into the plugin. e.g. You
>>>> might wish to ensure that session backups are always sent to a
>>>> machine in another rack, room etc, to further reduce the chance of
>>>> compound failure of all your backups.... Does this sound like the
>>>> sort of thing that you are after ? I believe that you may also
>>>> configure whether replication is done sync or asynchronously etc...
>>>
>>>
>>> Yes, that sounds exactly what I need to be able to prioritize nodes
>>> for failover. If you can call out the hooks to the algorithm and
>>> where it is being used, that may get me started. Is Gianny listening
>>> to this list? I would like to discuss this further and see what I
>>> can pull off.
>>
>>
>> He's registered with the list, but may not be following this thread.
>> I've cc-ed him directly to draw his attention to it... It will be
>> interesting to see exactly what you are going to do with this
>> ability... It was obvious that we needed it, but we were not sure
>> what would be useful default fn-ality. I think that Gianny has
>> provided a plugin which does a fairly basic 'round-robin' approach so
>> that all replicant services end up carrying the same sort of numbers
>> of replicants... This is an example of where the non-timely data that
>> I was talking about above may come in useful....
>>
>>>>
>>>>>
>>>>> Priorities of requirements:
>>>>> 1. sessions per node
>>>>
>>>>
>>>>
>>>> we have this already
>>>>
>>>> http://wadi.codehaus.org/getting-started/jmxmonitoring.html
>>>
>>>
>>> great
>>>
>>>>
>>>>> 2. node uptime
>>>>
>>>>
>>>>
>>>> this could be made available trivially - via the same route as (1)
>>>
>>>
>>> excellent
>>>
>>>>
>>>>> 3. failover node / replication scheme
>>>>
>>>>
>>>>
>>>> this could be made available by registering the replication-partner
>>>> plugin with JMX as it is injected into WADI by Spring.
>>>
>>>
>>> Sweet, looking forward to finding hooks and talking more.
>>>
>>>>
>>>>> 4. node state preservation
>>>>
>>>>
>>>>
>>>> I'm still not clear on exact requirements here - unless
>>>> state=sessions, in which case this is one of our primary requirements.
>>>
>>>
>>> state >= sessions. There are other stateful bits I wanted that again
>>> require coffee.
>>
>>
>> ohoh ! I know that other approaches distribute e.g. Context
>> attributes. I have avoided doing this as it is not mentioned by the
>> spec and probably brings a swathe of problems of its own. However, I
>> am always open to persuasion :-), especially if you are volunteering
>> to do the work !
>>
>>>>
>>>> Having said all of this, WADI is still someway from being
>>>> enterprise strength, but we are moving in this direction as fast as
>>>> we can. Your contribution wll help speed our progress and focus our
>>>> efforts on real world cases - very important.
>>>
>>>
>>> So I am going to see if I can pull off a web service example of the
>>> first two reqs: sessions per node, and uptime this morning. If it
>>> all works out I'll send you the resultant build.
>>
>>
>> uptime can be calculated on a per-node basis from the 'birthTime'
>> attribute carried in the Map that is distributed by each node. The
>> code around this area is being refactored at the moment. If you are
>> happy to work from CVS, I could stick a method in for you today to
>> return this data...  If you concentrate on getting a build out of CVS
>> up and running and walking through the Getting Started doc, I will
>> look at this. Bring any issues straight back to the list and we will
>> get you up and running as  painlessly as possible.
>>
>> regards,
>>
>>
>> Jules
>>
>>
>>>>
>>>> If you haven't already, you should take a look at our 'Getting
>>>> Started' doc - on the 'User' menu on left hand side of home page -
>>>> this will start to give you an idea of how WADI works, and ask lots
>>>> of questions on the lists. The more questions that are asked, the
>>>> better - the lists are archived and user driven question/answers
>>>> are probably better than developer-driven ones...
>>>>
>>>> Lets keep talking :-)
>>>>
>>>>
>>>> Jules
>>>>
>>>>>
>>>>> Jules Gosnell wrote:
>>>>>
>>>>>> Matthew Jording wrote:
>>>>>>
>>>>>>> Thanks for the prompt reply Jules,
>>>>>>>
>>>>>>>    Here is a clarification of fn-ality: I need to be able to
>>>>>>> receive information from each node, Session load per node
>>>>>>> (collected for histogram display), node uptime, pre-failure
>>>>>>> conditions and failover node.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> num sessions per node is OK
>>>>>> node uptime - OK
>>>>>> pre-failure conditions ? you mean you want a snapshot of what was
>>>>>> going on on a node just before it died ?
>>>>>> failover node - this would depend upon the replication scheme -
>>>>>> but should be possible...
>>>>>>
>>>>>>> Secondly I need to manage the cluster via manual load balancing
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> you mean that you want to manually move x sessions from node a->b
>>>>>> if a looks overloaded ? - why manually, wouldn't it be better if
>>>>>> this situation never arose ? This would be WADI's aim,
>>>>>>
>>>>>>> and master-chain relations for failover,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> you'll have to expand here - not sure what you mean - our current
>>>>>> replication scheme uses a pluggable algorithm to decide which
>>>>>> nodes to use as bckups - so perhaps ou couls plug something in
>>>>>> here ?
>>>>>>
>>>>>>> and of course stateful node shutdown & startup.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> we will have this. When a node shutd down it distributes its
>>>>>> state direct to other peers...
>>>>>>
>>>>>>> I'm concentrating on the web tier first and use the Tomcat
>>>>>>> clustering GBeans in Geronimo and create proper mbeans that can
>>>>>>> interact with a WSDM webservice which can provide the bridge to
>>>>>>> any outside management
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> We use Spring's JMX integration to register interesting WADI
>>>>>> pojos with JMX, from there they can be queried via remote or
>>>>>> local JMX - should be possible to attach your webservice if you
>>>>>> want.
>>>>>>
>>>>>>>
>>>>>>> Its a bit to chew on, and Geronimo is the obvious choice of app
>>>>>>> server. As a matter of course I wanted to see how far along
>>>>>>> these requirements are 'out of the box' and build on any
>>>>>>> existing work to give back to the Geronimo and WADI crew. From
>>>>>>> what I understand, although wadi is available in Geronimo v1.0
>>>>>>> the integration is not there. which is why I am starting from
>>>>>>> the gbean up. Obviously if there is something I could bridge
>>>>>>> between Geronimo and WADI even at an experimental level that
>>>>>>> would help me understand the proposed relationship.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> It sounds like pretty much everything that you want is on our
>>>>>> immediate todo list or done... I don't see any real requirement
>>>>>> for this to be integrated with the Geronimo kernel (since you
>>>>>> monitor via JMX), just as long as it can be deployed into a
>>>>>> webcontainer within Geronimo ... If this is not working at the
>>>>>> moment, then it is only a small patch. A complete integration
>>>>>> with Geronimo is a larger proposition, that has been discussed a
>>>>>> number of times. If you are interested in getting involved in
>>>>>> this, that would be great.
>>>>>>
>>>>>> I'll start thinking about your requirements. We should prioritise
>>>>>> them and talk about them in more detail.
>>>>>>
>>>>>> A good start :-)
>>>>>>
>>>>>> Jules
>>>>>>
>>>>>>>
>>>>>>> Thanks again
>>>>>>> Matthew Jording
>>>>>>>
>>>>>>> Jules Gosnell wrote:
>>>>>>>
>>>>>>>> Matthew Jording wrote:
>>>>>>>>
>>>>>>>>> Hello Jules and community.
>>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>>   I will need to have a production quality deployment of
>>>>>>>>> Geronimo ready within the next couple of months. One of the
>>>>>>>>> requirements is to have some level of cluster node management
>>>>>>>>> facility.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Management is a big area :-)
>>>>>>>>
>>>>>>>>> Initially my thoughts are web service and WSDM/JMX mixes.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Can you be a little more specific about exactly the sort of
>>>>>>>> fn-ality that you are looking for. We are concentrating on
>>>>>>>> scalable and performant clustered state (sessions) at the
>>>>>>>> moment. WS sessions are under development. Individual WADI
>>>>>>>> nodes are JMX manageable. I have ideas about how monitoring
>>>>>>>> info might be aggregated for the whole cluster on each WADI
>>>>>>>> node, but this is NYI... There are many other areas of
>>>>>>>> clustering fn-ality.
>>>>>>>>
>>>>>>>>> If you are currently working on such a facility perhaps I can
>>>>>>>>> just start contributing to it. At the very least I will keep
>>>>>>>>> you appraised of my progress. From responses on the Geronimo
>>>>>>>>> Developer lists I assume I should start with the clustering
>>>>>>>>> GBeans in the various tiers and work my way up.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Lets get a clear idea of what you are after and take it from
>>>>>>>> there. Any contribution would be very welcome :-) A lot of
>>>>>>>> thought has gone into how a Geronimo integration should look,
>>>>>>>> but work in this direction has stalled. Some fresh enthusiasm
>>>>>>>> in this area would be great.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I know that this is also the WADI/ActiveCluster plan for
>>>>>>>>> Geronimo and wanted to see if there is any work being done on
>>>>>>>>> this that I could build upon.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This part of WADI is undergoing a lot of thought at the moment
>>>>>>>> - the more the merrier.
>>>>>>>>
>>>>>>>> Thanks for your interest in WADI,
>>>>>>>>
>>>>>>>>
>>>>>>>> Jules
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Matthew Jording
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
>

[Filip Hanik - Dev Lists]

Jules Gosnell wrote:

> Matthew Jording wrote:
>
> excellent :-)
>
>>>
>>>> Manual load balancing would be a requirement only to evacuate
>>>> sessions from a node for any potential reason other then failover.
>>>
>>>
>>> So, not a shutdown, but a 'slimdown' ? I guess we could expose the
>>> top-level call for this to the management API. There may be issues
>>> with incoming requests (which would have been switched off in a
>>> shutdown situation) pulling sessions towards a node as we are trying
>>> to push them away to other nodes - I need to understand the context
>>> in which you are likely to use this fn-ality. Can you give a
>>> concrete example ?
>>
>> There are instances where a full shutdown of an application node is
>> undesirable during maintenance builds. If for instance our
>> application is not shutting down cleanly, the node shutdown bringing
>> that instance down can have adverse effects upon shared data stores.
>> It can therefore be helpful to have a mechanism to move sessions off
>> of a node without shutting that node down. There are other instances
>> where I have wished for this that will again require coffee to bring
>> to mind.
>
> I see - so you might force sessions to evacuate a 'sick' node, before
> performing maintenance on it - interesting idea - I will give it some
> thought..

this might seem useful, but in most cases it is useless. evacuating a
session, means that the load balancer knows where the session has been
evacuated to. and in enterprise env, this is almost never the case. In
tomcat for example, the only known location is the backup session, so
you can fail over to any node.

what you are talking about is called "draining", and that means that the
"sick" node simply wont accept new connections but will finish the
existing ones. This is already a feature on many load balancing
platforms. And with draining, no need to evacuate sessions, as another
node will become primary.

Filip

[Jules Gosnell-4]

Filip Hanik - Dev Lists wrote:

> Jules Gosnell wrote:
>
>> Matthew Jording wrote:
>>
>> excellent :-)
>>
>>>>
>>>>> Manual load balancing would be a requirement only to evacuate
>>>>> sessions from a node for any potential reason other then failover.
>>>>
>>>>
>>>>
>>>> So, not a shutdown, but a 'slimdown' ? I guess we could expose the
>>>> top-level call for this to the management API. There may be issues
>>>> with incoming requests (which would have been switched off in a
>>>> shutdown situation) pulling sessions towards a node as we are
>>>> trying to push them away to other nodes - I need to understand the
>>>> context in which you are likely to use this fn-ality. Can you give
>>>> a concrete example ?
>>>
>>>
>>> There are instances where a full shutdown of an application node is
>>> undesirable during maintenance builds. If for instance our
>>> application is not shutting down cleanly, the node shutdown bringing
>>> that instance down can have adverse effects upon shared data stores.
>>> It can therefore be helpful to have a mechanism to move sessions off
>>> of a node without shutting that node down. There are other instances
>>> where I have wished for this that will again require coffee to bring
>>> to mind.
>>
>>
>> I see - so you might force sessions to evacuate a 'sick' node, before
>> performing maintenance on it - interesting idea - I will give it some
>> thought..
>
> this might seem useful, but in most cases it is useless. evacuating a
> session, means that the load balancer knows where the session has been
> evacuated to. and in enterprise env, this is almost never the case. In
> tomcat for example, the only known location is the backup session, so
> you can fail over to any node.
>
> what you are talking about is called "draining", and that means that
> the "sick" node simply wont accept new connections but will finish the
> existing ones. This is already a feature on many load balancing
> platforms. And with draining, no need to evacuate sessions, as another
> node will become primary.
>

I know where you are coming from Filip - and these were my immediate
thoughts - however there is another possibility, much less efficient,
but much more immediate....

I'm assuming that you can't simply close the http listening port,
because you are servicing more than one web context and your loadbalacer
does not support the ability to remove a single context from an http
port - otherwise things become a lot easier.

Without a way to preemptively communicate the session's evacuation and
new location to the load-balancer, Filip is right, the subsequent
request for the session will be dispatched by the LB onto the 'sick'
node from which the session has just been evacuated.

WADI then has a choice.

1) Relocate the session back to the request/sick-node, in which case you
are just back to square one having wasted a lot of bandwidth and cycles
- the conversation will continue until its conclusion and the context
will need to remain available until all extant conversations have
concluded - Filip's 'draining'.

2). If WADI had a way of retrospectively communicating a change of
session location (or 'resticking') to the LB, a second possibility
arises - this is the case with e.g. httpd/mod_jk where you can do some
cookie-fiddling. WADI can choose to relocate the request arriving at the
sick node to the session's new host, either by redirecting or proxying
it. The request is rendered at the session's new location and the client
is 'restuck' via some hint to the load-balancer, so that subsequent
requests are dispatched to the new location. Load-balancers are becoming
more full-featured and I expect that we will be able to squeeze this
sort of fn-ality out of an increasing number...

The code for (2) was working a while ago, but since some refactoring of
the session relocation code has become disconnected. I hope to have it
reconnected soon....

(2)'s advantage over one is that you can evacuate a node in a matter of
seconds rather than a much longer indefinite (because users may extend
their sessions) period. In a large scale deployment the  time saving
made in shutting down each node can reap substantial benefits.


Jules

> Filip
>


--
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/

[Jules Gosnell-4]

Matthew Jording wrote:

> Doesn't look like maven is finding the wadi axis2 dependencies via
> ibiblio. I will find a copy and install it to complete the build, but
> I figured you should know

wadi-axis2.jar is an artefact that is built by WADI itself...

have you done a 'mvn install' from the top-level, to build an install
all artefacts in your local repo, before doing the assembly stage ? look
at the bin/build.sh script and see if you can get that to run.

Jules

>
> [ERROR] BUILD ERROR
> [INFO]
> ----------------------------------------------------------------------------
>
> [INFO] Failed to resolve artifact.
>
> required artifacts missing:
>  org.codehaus.wadi:wadi-axis2:jar:2.0M2-SNAPSHOT
>
> for the artifact:
>  org.codehaus.wadi:wadi-assembly:pom:2.0M2-SNAPSHOT
>
> from the specified remote repositories:
>  central (http://repo1.maven.org/maven2),
>  wadi-dependencies (http://dist.codehaus.org/wadi/dependencies/maven2),
>  maven2-ibiblio (http://ibiblio.org/maven2)
>
>
> Jules Gosnell wrote:
>
>> spot the deliberate mistake :-)
>>
>> When I said CVS, I meant SVN...
>>
>> All repo details etc shoul be on the website (wad.codehaus.org) along
>> with Developer resources (menu on left hand side).
>>
>> If anything is missing or out of date, just shout,
>>
>> Jules
>>
>>
>>
>> Jules Gosnell wrote:
>>
>>> Matthew Jording wrote:
>>>
>>>> Jules Gosnell wrote:
>>>>
>>>>> Matthew Jording wrote:
>>>>>
>>>>>> Wow, thats great Jules,
>>>>>>
>>>>>>  You guys have clearly been busier then the site at first
>>>>>> indicates. I clearly need to start playing with with WADI and see
>>>>>> what I can glean.
>>>>>>
>>>>>> So as far as pre-failure conditions, I just need to save node
>>>>>> state and preserve logs and make all of that available in some
>>>>>> manner to the 'cluster manager'.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> two issues here...
>>>>>
>>>>> by 'failure' as opposed to 'shutdown', I understand a situation
>>>>> where there is no opportunity to take evasive or mitigating
>>>>> actions - so whatever state/logs you wish to save must be
>>>>> incrementally backed up during normal operation - an expensive
>>>>> overhead. Initial node configuration is injected via a Spring
>>>>> config. All of the affected fields (or certainly most of these)
>>>>> will be final - i.e. read-only - so, I am not sure that we have
>>>>> much dynamically modifiable state. What sort of state did you have
>>>>> in mind (other than sessions) ? As far as logs go (unless you are
>>>>> talking transaction logs - and HttpSessions are transactionless),
>>>>> these are written out incrementally to the destination of your
>>>>> choice, so I'm not sure what further action you would want here -
>>>>> can you elucidate?
>>>>
>>>>
>>>>
>>>> I have to flesh this out. May take a bit, and some morning coffee
>>>
>>>
>>>
>>> OK - I'll hang on...
>>>
>>>>>
>>>>> regarding your 'cluster manager' - if this is a client, then, no
>>>>> problem. If it is a node/peer, thenit sounds like your arch
>>>>> differs from the one I have in mind for WADI. I am thinking in
>>>>> terms of an arch where detailed and timely data about a node is
>>>>> only available directly from this node, but smaller and less
>>>>> timely data about each node is published to the rest of the
>>>>> cluster and may be aggregated upon any node. This data may be used
>>>>> to drive e.g. automatic load/state-balancing algorithms or a
>>>>> remote view of the cluster's aggregated stats... How does this
>>>>> compare with the functions of your 'cluster manager'. Is it likely
>>>>> that any peer in a WADI cluster could take on this role (my
>>>>> preferred solution).
>>>>
>>>>
>>>>
>>>> The 'cluster manager' could be a client of node/peer services, so
>>>> the WADI solution seems to fit.
>>>
>>>
>>>
>>> excellent :-)
>>>
>>>>>
>>>>>> Manual load balancing would be a requirement only to evacuate
>>>>>> sessions from a node for any potential reason other then failover.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> So, not a shutdown, but a 'slimdown' ? I guess we could expose the
>>>>> top-level call for this to the management API. There may be issues
>>>>> with incoming requests (which would have been switched off in a
>>>>> shutdown situation) pulling sessions towards a node as we are
>>>>> trying to push them away to other nodes - I need to understand the
>>>>> context in which you are likely to use this fn-ality. Can you give
>>>>> a concrete example ?
>>>>
>>>>
>>>>
>>>> There are instances where a full shutdown of an application node is
>>>> undesirable during maintenance builds. If for instance our
>>>> application is not shutting down cleanly, the node shutdown
>>>> bringing that instance down can have adverse effects upon shared
>>>> data stores. It can therefore be helpful to have a mechanism to
>>>> move sessions off of a node without shutting that node down. There
>>>> are other instances where I have wished for this that will again
>>>> require coffee to bring to mind.
>>>
>>>
>>>
>>> I see - so you might force sessions to evacuate a 'sick' node,
>>> before performing maintenance on it - interesting idea - I will give
>>> it some thought..
>>>
>>>>>
>>>>>> What I meant by Master-chain relationships is the ability to
>>>>>> prioritize the replication scheme.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> This sounds like our pluggable replication-partner algorithm.
>>>>> Gianny is the man for this - he wrote it. My understanding is that
>>>>> its API is sufficient to allow the automatic selection of
>>>>> replication partner[s] (the number of backups is parameterisable
>>>>> on a per-cluster basis) on a per-session basis, based on any
>>>>> criteria that the deployer might be able to inject into the
>>>>> plugin. e.g. You might wish to ensure that session backups are
>>>>> always sent to a machine in another rack, room etc, to further
>>>>> reduce the chance of compound failure of all your backups.... Does
>>>>> this sound like the sort of thing that you are after ? I believe
>>>>> that you may also configure whether replication is done sync or
>>>>> asynchronously etc...
>>>>
>>>>
>>>>
>>>> Yes, that sounds exactly what I need to be able to prioritize nodes
>>>> for failover. If you can call out the hooks to the algorithm and
>>>> where it is being used, that may get me started. Is Gianny
>>>> listening to this list? I would like to discuss this further and
>>>> see what I can pull off.
>>>
>>>
>>>
>>> He's registered with the list, but may not be following this thread.
>>> I've cc-ed him directly to draw his attention to it... It will be
>>> interesting to see exactly what you are going to do with this
>>> ability... It was obvious that we needed it, but we were not sure
>>> what would be useful default fn-ality. I think that Gianny has
>>> provided a plugin which does a fairly basic 'round-robin' approach
>>> so that all replicant services end up carrying the same sort of
>>> numbers of replicants... This is an example of where the non-timely
>>> data that I was talking about above may come in useful....
>>>
>>>>>
>>>>>>
>>>>>> Priorities of requirements:
>>>>>> 1. sessions per node
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> we have this already
>>>>>
>>>>> http://wadi.codehaus.org/getting-started/jmxmonitoring.html
>>>>
>>>>
>>>>
>>>> great
>>>>
>>>>>
>>>>>> 2. node uptime
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> this could be made available trivially - via the same route as (1)
>>>>
>>>>
>>>>
>>>> excellent
>>>>
>>>>>
>>>>>> 3. failover node / replication scheme
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> this could be made available by registering the
>>>>> replication-partner plugin with JMX as it is injected into WADI by
>>>>> Spring.
>>>>
>>>>
>>>>
>>>> Sweet, looking forward to finding hooks and talking more.
>>>>
>>>>>
>>>>>> 4. node state preservation
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I'm still not clear on exact requirements here - unless
>>>>> state=sessions, in which case this is one of our primary
>>>>> requirements.
>>>>
>>>>
>>>>
>>>> state >= sessions. There are other stateful bits I wanted that
>>>> again require coffee.
>>>
>>>
>>>
>>> ohoh ! I know that other approaches distribute e.g. Context
>>> attributes. I have avoided doing this as it is not mentioned by the
>>> spec and probably brings a swathe of problems of its own. However, I
>>> am always open to persuasion :-), especially if you are volunteering
>>> to do the work !
>>>
>>>>>
>>>>> Having said all of this, WADI is still someway from being
>>>>> enterprise strength, but we are moving in this direction as fast
>>>>> as we can. Your contribution wll help speed our progress and focus
>>>>> our efforts on real world cases - very important.
>>>>
>>>>
>>>>
>>>> So I am going to see if I can pull off a web service example of the
>>>> first two reqs: sessions per node, and uptime this morning. If it
>>>> all works out I'll send you the resultant build.
>>>
>>>
>>>
>>> uptime can be calculated on a per-node basis from the 'birthTime'
>>> attribute carried in the Map that is distributed by each node. The
>>> code around this area is being refactored at the moment. If you are
>>> happy to work from CVS, I could stick a method in for you today to
>>> return this data...  If you concentrate on getting a build out of
>>> CVS up and running and walking through the Getting Started doc, I
>>> will look at this. Bring any issues straight back to the list and we
>>> will get you up and running as  painlessly as possible.
>>>
>>> regards,
>>>
>>>
>>> Jules
>>>
>>>
>>>>>
>>>>> If you haven't already, you should take a look at our 'Getting
>>>>> Started' doc - on the 'User' menu on left hand side of home page -
>>>>> this will start to give you an idea of how WADI works, and ask
>>>>> lots of questions on the lists. The more questions that are asked,
>>>>> the better - the lists are archived and user driven
>>>>> question/answers are probably better than developer-driven ones...
>>>>>
>>>>> Lets keep talking :-)
>>>>>
>>>>>
>>>>> Jules
>>>>>
>>>>>>
>>>>>> Jules Gosnell wrote:
>>>>>>
>>>>>>> Matthew Jording wrote:
>>>>>>>
>>>>>>>> Thanks for the prompt reply Jules,
>>>>>>>>
>>>>>>>>    Here is a clarification of fn-ality: I need to be able to
>>>>>>>> receive information from each node, Session load per node
>>>>>>>> (collected for histogram display), node uptime, pre-failure
>>>>>>>> conditions and failover node.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> num sessions per node is OK
>>>>>>> node uptime - OK
>>>>>>> pre-failure conditions ? you mean you want a snapshot of what
>>>>>>> was going on on a node just before it died ?
>>>>>>> failover node - this would depend upon the replication scheme -
>>>>>>> but should be possible...
>>>>>>>
>>>>>>>> Secondly I need to manage the cluster via manual load balancing
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> you mean that you want to manually move x sessions from node
>>>>>>> a->b if a looks overloaded ? - why manually, wouldn't it be
>>>>>>> better if this situation never arose ? This would be WADI's aim,
>>>>>>>
>>>>>>>> and master-chain relations for failover,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> you'll have to expand here - not sure what you mean - our
>>>>>>> current replication scheme uses a pluggable algorithm to decide
>>>>>>> which nodes to use as bckups - so perhaps ou couls plug
>>>>>>> something in here ?
>>>>>>>
>>>>>>>> and of course stateful node shutdown & startup.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> we will have this. When a node shutd down it distributes its
>>>>>>> state direct to other peers...
>>>>>>>
>>>>>>>> I'm concentrating on the web tier first and use the Tomcat
>>>>>>>> clustering GBeans in Geronimo and create proper mbeans that can
>>>>>>>> interact with a WSDM webservice which can provide the bridge to
>>>>>>>> any outside management
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We use Spring's JMX integration to register interesting WADI
>>>>>>> pojos with JMX, from there they can be queried via remote or
>>>>>>> local JMX - should be possible to attach your webservice if you
>>>>>>> want.
>>>>>>>
>>>>>>>>
>>>>>>>> Its a bit to chew on, and Geronimo is the obvious choice of app
>>>>>>>> server. As a matter of course I wanted to see how far along
>>>>>>>> these requirements are 'out of the box' and build on any
>>>>>>>> existing work to give back to the Geronimo and WADI crew. From
>>>>>>>> what I understand, although wadi is available in Geronimo v1.0
>>>>>>>> the integration is not there. which is why I am starting from
>>>>>>>> the gbean up. Obviously if there is something I could bridge
>>>>>>>> between Geronimo and WADI even at an experimental level that
>>>>>>>> would help me understand the proposed relationship.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> It sounds like pretty much everything that you want is on our
>>>>>>> immediate todo list or done... I don't see any real requirement
>>>>>>> for this to be integrated with the Geronimo kernel (since you
>>>>>>> monitor via JMX), just as long as it can be deployed into a
>>>>>>> webcontainer within Geronimo ... If this is not working at the
>>>>>>> moment, then it is only a small patch. A complete integration
>>>>>>> with Geronimo is a larger proposition, that has been discussed a
>>>>>>> number of times. If you are interested in getting involved in
>>>>>>> this, that would be great.
>>>>>>>
>>>>>>> I'll start thinking about your requirements. We should
>>>>>>> prioritise them and talk about them in more detail.
>>>>>>>
>>>>>>> A good start :-)
>>>>>>>
>>>>>>> Jules
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks again
>>>>>>>> Matthew Jording
>>>>>>>>
>>>>>>>> Jules Gosnell wrote:
>>>>>>>>
>>>>>>>>> Matthew Jording wrote:
>>>>>>>>>
>>>>>>>>>> Hello Jules and community.
>>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>>   I will need to have a production quality deployment of
>>>>>>>>>> Geronimo ready within the next couple of months. One of the
>>>>>>>>>> requirements is to have some level of cluster node management
>>>>>>>>>> facility.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Management is a big area :-)
>>>>>>>>>
>>>>>>>>>> Initially my thoughts are web service and WSDM/JMX mixes.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you be a little more specific about exactly the sort of
>>>>>>>>> fn-ality that you are looking for. We are concentrating on
>>>>>>>>> scalable and performant clustered state (sessions) at the
>>>>>>>>> moment. WS sessions are under development. Individual WADI
>>>>>>>>> nodes are JMX manageable. I have ideas about how monitoring
>>>>>>>>> info might be aggregated for the whole cluster on each WADI
>>>>>>>>> node, but this is NYI... There are many other areas of
>>>>>>>>> clustering fn-ality.
>>>>>>>>>
>>>>>>>>>> If you are currently working on such a facility perhaps I can
>>>>>>>>>> just start contributing to it. At the very least I will keep
>>>>>>>>>> you appraised of my progress. From responses on the Geronimo
>>>>>>>>>> Developer lists I assume I should start with the clustering
>>>>>>>>>> GBeans in the various tiers and work my way up.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Lets get a clear idea of what you are after and take it from
>>>>>>>>> there. Any contribution would be very welcome :-) A lot of
>>>>>>>>> thought has gone into how a Geronimo integration should look,
>>>>>>>>> but work in this direction has stalled. Some fresh enthusiasm
>>>>>>>>> in this area would be great.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I know that this is also the WADI/ActiveCluster plan for
>>>>>>>>>> Geronimo and wanted to see if there is any work being done on
>>>>>>>>>> this that I could build upon.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This part of WADI is undergoing a lot of thought at the moment
>>>>>>>>> - the more the merrier.
>>>>>>>>>
>>>>>>>>> Thanks for your interest in WADI,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Jules
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Matthew Jording
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>

--
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/

[Matthew Jording]

That did it thanks.
Now I have to figure out why the webapp will not deploy to geronimo.


Jules Gosnell wrote:

> Matthew Jording wrote:
>
>> Doesn't look like maven is finding the wadi axis2 dependencies via
>> ibiblio. I will find a copy and install it to complete the build, but
>> I figured you should know
>
> wadi-axis2.jar is an artefact that is built by WADI itself...
>
> have you done a 'mvn install' from the top-level, to build an install
> all artefacts in your local repo, before doing the assembly stage ?
> look at the bin/build.sh script and see if you can get that to run.
>
> Jules
>
>>
>> [ERROR] BUILD ERROR
>> [INFO]
>> ----------------------------------------------------------------------------
>>
>> [INFO] Failed to resolve artifact.
>>
>> required artifacts missing:
>>  org.codehaus.wadi:wadi-axis2:jar:2.0M2-SNAPSHOT
>>
>> for the artifact:
>>  org.codehaus.wadi:wadi-assembly:pom:2.0M2-SNAPSHOT
>>
>> from the specified remote repositories:
>>  central (http://repo1.maven.org/maven2),
>>  wadi-dependencies (http://dist.codehaus.org/wadi/dependencies/maven2),
>>  maven2-ibiblio (http://ibiblio.org/maven2)
>>
>>
>> Jules Gosnell wrote:
>>
>>> spot the deliberate mistake :-)
>>>
>>> When I said CVS, I meant SVN...
>>>
>>> All repo details etc shoul be on the website (wad.codehaus.org)
>>> along with Developer resources (menu on left hand side).
>>>
>>> If anything is missing or out of date, just shout,
>>>
>>> Jules
>>>
>>>
>>>
>>> Jules Gosnell wrote:
>>>
>>>> Matthew Jording wrote:
>>>>
>>>>> Jules Gosnell wrote:
>>>>>
>>>>>> Matthew Jording wrote:
>>>>>>
>>>>>>> Wow, thats great Jules,
>>>>>>>
>>>>>>>  You guys have clearly been busier then the site at first
>>>>>>> indicates. I clearly need to start playing with with WADI and
>>>>>>> see what I can glean.
>>>>>>>
>>>>>>> So as far as pre-failure conditions, I just need to save node
>>>>>>> state and preserve logs and make all of that available in some
>>>>>>> manner to the 'cluster manager'.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> two issues here...
>>>>>>
>>>>>> by 'failure' as opposed to 'shutdown', I understand a situation
>>>>>> where there is no opportunity to take evasive or mitigating
>>>>>> actions - so whatever state/logs you wish to save must be
>>>>>> incrementally backed up during normal operation - an expensive
>>>>>> overhead. Initial node configuration is injected via a Spring
>>>>>> config. All of the affected fields (or certainly most of these)
>>>>>> will be final - i.e. read-only - so, I am not sure that we have
>>>>>> much dynamically modifiable state. What sort of state did you
>>>>>> have in mind (other than sessions) ? As far as logs go (unless
>>>>>> you are talking transaction logs - and HttpSessions are
>>>>>> transactionless), these are written out incrementally to the
>>>>>> destination of your choice, so I'm not sure what further action
>>>>>> you would want here - can you elucidate?
>>>>>
>>>>>
>>>>>
>>>>> I have to flesh this out. May take a bit, and some morning coffee
>>>>
>>>>
>>>>
>>>> OK - I'll hang on...
>>>>
>>>>>>
>>>>>> regarding your 'cluster manager' - if this is a client, then, no
>>>>>> problem. If it is a node/peer, thenit sounds like your arch
>>>>>> differs from the one I have in mind for WADI. I am thinking in
>>>>>> terms of an arch where detailed and timely data about a node is
>>>>>> only available directly from this node, but smaller and less
>>>>>> timely data about each node is published to the rest of the
>>>>>> cluster and may be aggregated upon any node. This data may be
>>>>>> used to drive e.g. automatic load/state-balancing algorithms or a
>>>>>> remote view of the cluster's aggregated stats... How does this
>>>>>> compare with the functions of your 'cluster manager'. Is it
>>>>>> likely that any peer in a WADI cluster could take on this role
>>>>>> (my preferred solution).
>>>>>
>>>>>
>>>>>
>>>>> The 'cluster manager' could be a client of node/peer services, so
>>>>> the WADI solution seems to fit.
>>>>
>>>>
>>>>
>>>> excellent :-)
>>>>
>>>>>>
>>>>>>> Manual load balancing would be a requirement only to evacuate
>>>>>>> sessions from a node for any potential reason other then failover.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> So, not a shutdown, but a 'slimdown' ? I guess we could expose
>>>>>> the top-level call for this to the management API. There may be
>>>>>> issues with incoming requests (which would have been switched off
>>>>>> in a shutdown situation) pulling sessions towards a node as we
>>>>>> are trying to push them away to other nodes - I need to
>>>>>> understand the context in which you are likely to use this
>>>>>> fn-ality. Can you give a concrete example ?
>>>>>
>>>>>
>>>>>
>>>>> There are instances where a full shutdown of an application node
>>>>> is undesirable during maintenance builds. If for instance our
>>>>> application is not shutting down cleanly, the node shutdown
>>>>> bringing that instance down can have adverse effects upon shared
>>>>> data stores. It can therefore be helpful to have a mechanism to
>>>>> move sessions off of a node without shutting that node down. There
>>>>> are other instances where I have wished for this that will again
>>>>> require coffee to bring to mind.
>>>>
>>>>
>>>>
>>>> I see - so you might force sessions to evacuate a 'sick' node,
>>>> before performing maintenance on it - interesting idea - I will
>>>> give it some thought..
>>>>
>>>>>>
>>>>>>> What I meant by Master-chain relationships is the ability to
>>>>>>> prioritize the replication scheme.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> This sounds like our pluggable replication-partner algorithm.
>>>>>> Gianny is the man for this - he wrote it. My understanding is
>>>>>> that its API is sufficient to allow the automatic selection of
>>>>>> replication partner[s] (the number of backups is parameterisable
>>>>>> on a per-cluster basis) on a per-session basis, based on any
>>>>>> criteria that the deployer might be able to inject into the
>>>>>> plugin. e.g. You might wish to ensure that session backups are
>>>>>> always sent to a machine in another rack, room etc, to further
>>>>>> reduce the chance of compound failure of all your backups....
>>>>>> Does this sound like the sort of thing that you are after ? I
>>>>>> believe that you may also configure whether replication is done
>>>>>> sync or asynchronously etc...
>>>>>
>>>>>
>>>>>
>>>>> Yes, that sounds exactly what I need to be able to prioritize
>>>>> nodes for failover. If you can call out the hooks to the algorithm
>>>>> and where it is being used, that may get me started. Is Gianny
>>>>> listening to this list? I would like to discuss this further and
>>>>> see what I can pull off.
>>>>
>>>>
>>>>
>>>> He's registered with the list, but may not be following this
>>>> thread. I've cc-ed him directly to draw his attention to it... It
>>>> will be interesting to see exactly what you are going to do with
>>>> this ability... It was obvious that we needed it, but we were not
>>>> sure what would be useful default fn-ality. I think that Gianny has
>>>> provided a plugin which does a fairly basic 'round-robin' approach
>>>> so that all replicant services end up carrying the same sort of
>>>> numbers of replicants... This is an example of where the non-timely
>>>> data that I was talking about above may come in useful....
>>>>
>>>>>>
>>>>>>>
>>>>>>> Priorities of requirements:
>>>>>>> 1. sessions per node
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> we have this already
>>>>>>
>>>>>> http://wadi.codehaus.org/getting-started/jmxmonitoring.html
>>>>>
>>>>>
>>>>>
>>>>> great
>>>>>
>>>>>>
>>>>>>> 2. node uptime
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> this could be made available trivially - via the same route as (1)
>>>>>
>>>>>
>>>>>
>>>>> excellent
>>>>>
>>>>>>
>>>>>>> 3. failover node / replication scheme
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> this could be made available by registering the
>>>>>> replication-partner plugin with JMX as it is injected into WADI
>>>>>> by Spring.
>>>>>
>>>>>
>>>>>
>>>>> Sweet, looking forward to finding hooks and talking more.
>>>>>
>>>>>>
>>>>>>> 4. node state preservation
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I'm still not clear on exact requirements here - unless
>>>>>> state=sessions, in which case this is one of our primary
>>>>>> requirements.
>>>>>
>>>>>
>>>>>
>>>>> state >= sessions. There are other stateful bits I wanted that
>>>>> again require coffee.
>>>>
>>>>
>>>>
>>>> ohoh ! I know that other approaches distribute e.g. Context
>>>> attributes. I have avoided doing this as it is not mentioned by the
>>>> spec and probably brings a swathe of problems of its own. However,
>>>> I am always open to persuasion :-), especially if you are
>>>> volunteering to do the work !
>>>>
>>>>>>
>>>>>> Having said all of this, WADI is still someway from being
>>>>>> enterprise strength, but we are moving in this direction as fast
>>>>>> as we can. Your contribution wll help speed our progress and
>>>>>> focus our efforts on real world cases - very important.
>>>>>
>>>>>
>>>>>
>>>>> So I am going to see if I can pull off a web service example of
>>>>> the first two reqs: sessions per node, and uptime this morning. If
>>>>> it all works out I'll send you the resultant build.
>>>>
>>>>
>>>>
>>>> uptime can be calculated on a per-node basis from the 'birthTime'
>>>> attribute carried in the Map that is distributed by each node. The
>>>> code around this area is being refactored at the moment. If you are
>>>> happy to work from CVS, I could stick a method in for you today to
>>>> return this data...  If you concentrate on getting a build out of
>>>> CVS up and running and walking through the Getting Started doc, I
>>>> will look at this. Bring any issues straight back to the list and
>>>> we will get you up and running as  painlessly as possible.
>>>>
>>>> regards,
>>>>
>>>>
>>>> Jules
>>>>
>>>>
>>>>>>
>>>>>> If you haven't already, you should take a look at our 'Getting
>>>>>> Started' doc - on the 'User' menu on left hand side of home page
>>>>>> - this will start to give you an idea of how WADI works, and ask
>>>>>> lots of questions on the lists. The more questions that are
>>>>>> asked, the better - the lists are archived and user driven
>>>>>> question/answers are probably better than developer-driven ones...
>>>>>>
>>>>>> Lets keep talking :-)
>>>>>>
>>>>>>
>>>>>> Jules
>>>>>>
>>>>>>>
>>>>>>> Jules Gosnell wrote:
>>>>>>>
>>>>>>>> Matthew Jording wrote:
>>>>>>>>
>>>>>>>>> Thanks for the prompt reply Jules,
>>>>>>>>>
>>>>>>>>>    Here is a clarification of fn-ality: I need to be able to
>>>>>>>>> receive information from each node, Session load per node
>>>>>>>>> (collected for histogram display), node uptime, pre-failure
>>>>>>>>> conditions and failover node.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> num sessions per node is OK
>>>>>>>> node uptime - OK
>>>>>>>> pre-failure conditions ? you mean you want a snapshot of what
>>>>>>>> was going on on a node just before it died ?
>>>>>>>> failover node - this would depend upon the replication scheme -
>>>>>>>> but should be possible...
>>>>>>>>
>>>>>>>>> Secondly I need to manage the cluster via manual load balancing
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> you mean that you want to manually move x sessions from node
>>>>>>>> a->b if a looks overloaded ? - why manually, wouldn't it be
>>>>>>>> better if this situation never arose ? This would be WADI's aim,
>>>>>>>>
>>>>>>>>> and master-chain relations for failover,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> you'll have to expand here - not sure what you mean - our
>>>>>>>> current replication scheme uses a pluggable algorithm to decide
>>>>>>>> which nodes to use as bckups - so perhaps ou couls plug
>>>>>>>> something in here ?
>>>>>>>>
>>>>>>>>> and of course stateful node shutdown & startup.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> we will have this. When a node shutd down it distributes its
>>>>>>>> state direct to other peers...
>>>>>>>>
>>>>>>>>> I'm concentrating on the web tier first and use the Tomcat
>>>>>>>>> clustering GBeans in Geronimo and create proper mbeans that
>>>>>>>>> can interact with a WSDM webservice which can provide the
>>>>>>>>> bridge to any outside management
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> We use Spring's JMX integration to register interesting WADI
>>>>>>>> pojos with JMX, from there they can be queried via remote or
>>>>>>>> local JMX - should be possible to attach your webservice if you
>>>>>>>> want.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Its a bit to chew on, and Geronimo is the obvious choice of
>>>>>>>>> app server. As a matter of course I wanted to see how far
>>>>>>>>> along these requirements are 'out of the box' and build on any
>>>>>>>>> existing work to give back to the Geronimo and WADI crew. From
>>>>>>>>> what I understand, although wadi is available in Geronimo v1.0
>>>>>>>>> the integration is not there. which is why I am starting from
>>>>>>>>> the gbean up. Obviously if there is something I could bridge
>>>>>>>>> between Geronimo and WADI even at an experimental level that
>>>>>>>>> would help me understand the proposed relationship.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> It sounds like pretty much everything that you want is on our
>>>>>>>> immediate todo list or done... I don't see any real requirement
>>>>>>>> for this to be integrated with the Geronimo kernel (since you
>>>>>>>> monitor via JMX), just as long as it can be deployed into a
>>>>>>>> webcontainer within Geronimo ... If this is not working at the
>>>>>>>> moment, then it is only a small patch. A complete integration
>>>>>>>> with Geronimo is a larger proposition, that has been discussed
>>>>>>>> a number of times. If you are interested in getting involved in
>>>>>>>> this, that would be great.
>>>>>>>>
>>>>>>>> I'll start thinking about your requirements. We should
>>>>>>>> prioritise them and talk about them in more detail.
>>>>>>>>
>>>>>>>> A good start :-)
>>>>>>>>
>>>>>>>> Jules
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks again
>>>>>>>>> Matthew Jording
>>>>>>>>>
>>>>>>>>> Jules Gosnell wrote:
>>>>>>>>>
>>>>>>>>>> Matthew Jording wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello Jules and community.
>>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>>   I will need to have a production quality deployment of
>>>>>>>>>>> Geronimo ready within the next couple of months. One of the
>>>>>>>>>>> requirements is to have some level of cluster node
>>>>>>>>>>> management facility.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Management is a big area :-)
>>>>>>>>>>
>>>>>>>>>>> Initially my thoughts are web service and WSDM/JMX mixes.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you be a little more specific about exactly the sort of
>>>>>>>>>> fn-ality that you are looking for. We are concentrating on
>>>>>>>>>> scalable and performant clustered state (sessions) at the
>>>>>>>>>> moment. WS sessions are under development. Individual WADI
>>>>>>>>>> nodes are JMX manageable. I have ideas about how monitoring
>>>>>>>>>> info might be aggregated for the whole cluster on each WADI
>>>>>>>>>> node, but this is NYI... There are many other areas of
>>>>>>>>>> clustering fn-ality.
>>>>>>>>>>
>>>>>>>>>>> If you are currently working on such a facility perhaps I
>>>>>>>>>>> can just start contributing to it. At the very least I will
>>>>>>>>>>> keep you appraised of my progress. From responses on the
>>>>>>>>>>> Geronimo Developer lists I assume I should start with the
>>>>>>>>>>> clustering GBeans in the various tiers and work my way up.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Lets get a clear idea of what you are after and take it from
>>>>>>>>>> there. Any contribution would be very welcome :-) A lot of
>>>>>>>>>> thought has gone into how a Geronimo integration should look,
>>>>>>>>>> but work in this direction has stalled. Some fresh enthusiasm
>>>>>>>>>> in this area would be great.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I know that this is also the WADI/ActiveCluster plan for
>>>>>>>>>>> Geronimo and wanted to see if there is any work being done
>>>>>>>>>>> on this that I could build upon.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This part of WADI is undergoing a lot of thought at the
>>>>>>>>>> moment - the more the merrier.
>>>>>>>>>>
>>>>>>>>>> Thanks for your interest in WADI,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Jules
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Matthew Jording
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
>