Associating projects with distributed build agents

[Wendy Smoak-3]

One of the goals at the top of the Distributed Builds wiki page is the
ability to build on multiple platforms.

As we've discussed in some other threads and on irc, we need some way
of telling Continuum which agent(s) to execute builds on.

To me, it seems like this belongs as a field on the build environment,
which is then selected on the build definition.

That would be a good next step IMO -- to pin a particular build def to
a particular agent through the environment.

However, I don't want to lose the effect of distributed builds in
parallel that we're getting from the current 'next available'
selection.

So, how does having groups of agents sound?  Then you could associate
a build environment with a pool of agents, and the next available one
of those would be chosen.

You might have a large group of identically configured Linux boxes
that most of your builds can be distributed to.  However, maybe there
is one special project group that needs their own private pool to work
on a top secret project.  Another team might be targeting Solaris and
want to do their builds on a pool of three Solaris boxes.  And of
course the project that only builds on Windows needs to always build
on the one Windows agent you have reluctantly configured for them. ;)

There's an additional requirement that the system admin must be able
to limit which environments a project team is allowed to choose.
While a project developer can modify build definitions and add new
ones, he should not have free choice of *all* the available
environments.  This probably means adding an "Allowed Build
Environments" field to the project group.

How does that sound?  Does anyone have a different idea of how this should work?

--
Wendy

[Christian Edward Gruber]

Rather than agent groups, could we go with a criteria approach?  That  
is to say, you can describe and agent's capabilities, then associate  
criteria for a project group or project, and then let the system sort  
out which agents can fulfill which criteria?

Christian.

On 17-Jan-09, at 12:28 , Wendy Smoak wrote:

> One of the goals at the top of the Distributed Builds wiki page is the
> ability to build on multiple platforms.
>
> As we've discussed in some other threads and on irc, we need some way
> of telling Continuum which agent(s) to execute builds on.
>
> To me, it seems like this belongs as a field on the build environment,
> which is then selected on the build definition.
>
> That would be a good next step IMO -- to pin a particular build def to
> a particular agent through the environment.
>
> However, I don't want to lose the effect of distributed builds in
> parallel that we're getting from the current 'next available'
> selection.
>
> So, how does having groups of agents sound?  Then you could associate
> a build environment with a pool of agents, and the next available one
> of those would be chosen.
>
> You might have a large group of identically configured Linux boxes
> that most of your builds can be distributed to.  However, maybe there
> is one special project group that needs their own private pool to work
> on a top secret project.  Another team might be targeting Solaris and
> want to do their builds on a pool of three Solaris boxes.  And of
> course the project that only builds on Windows needs to always build
> on the one Windows agent you have reluctantly configured for them. ;)
>
> There's an additional requirement that the system admin must be able
> to limit which environments a project team is allowed to choose.
> While a project developer can modify build definitions and add new
> ones, he should not have free choice of *all* the available
> environments.  This probably means adding an "Allowed Build
> Environments" field to the project group.
>
> How does that sound?  Does anyone have a different idea of how this  
> should work?
>
> --
> Wendy
>

Christian E. Gruber - President / Senior Consultant                
email:  cgruber@...
Isráfíl Consulting Services Corporation                            
mobile: [phone]
"Keenness of understanding is due to keenness of vision..."        
phone:  [phone]

[Wendy Smoak-3]

I can see that working, if I can define arbitrary criteria like a
property name/value.

I need a way to make sure that Top Secret Project X builds on agents
3, 5 and 14 only, and that no other projects ever build on those
agents.

--
Wendy

On Sat, Jan 17, 2009 at 10:47 AM, Christian Edward Gruber
<cgruber@...> wrote:
> Rather than agent groups, could we go with a criteria approach?  That is to
> say, you can describe and agent's capabilities, then associate criteria for
> a project group or project, and then let the system sort out which agents
> can fulfill which criteria?

[Christian Edward Gruber]

I guess then having agent groups as a logical group, but make group be  
part of the criteria would help with that then, group effectively  
being a property of the agent.

The key for me is to have the whole thing specified as constraints,  
not explicit solutions, so that new agents can be launched and they  
will automatically fit into the constraints based on their  
configuration, rather than having to add them to this or that build  
definition.  Filter/constraint configuration is a great way to  
simplify config, especially with large banks of agents.

Christian

On 17-Jan-09, at 16:22 , Wendy Smoak wrote:

> I can see that working, if I can define arbitrary criteria like a
> property name/value.
>
> I need a way to make sure that Top Secret Project X builds on agents
> 3, 5 and 14 only, and that no other projects ever build on those
> agents.
>
> --
> Wendy
>
> On Sat, Jan 17, 2009 at 10:47 AM, Christian Edward Gruber
> <cgruber@...> wrote:
>> Rather than agent groups, could we go with a criteria approach?  
>> That is to
>> say, you can describe and agent's capabilities, then associate  
>> criteria for
>> a project group or project, and then let the system sort out which  
>> agents
>> can fulfill which criteria?
>

Christian E. Gruber - President / Senior Consultant                
email:  cgruber@...
Isráfíl Consulting Services Corporation                            
mobile: [phone]
"Keenness of understanding is due to keenness of vision..."        
phone:  [phone]

[brettporter]

+1

I think selection needs to be a combination of capability, permission  
and availability. I would mostly prefer the user is in full control of  
selecting these rather than having too much magic (automated selection  
might be a nice feature, but the basic starting point should be a  
fully configured system).

- Brett

On 17/01/2009, at 1:36 PM, Christian Edward Gruber wrote:

> I guess then having agent groups as a logical group, but make group  
> be part of the criteria would help with that then, group effectively  
> being a property of the agent.
>
> The key for me is to have the whole thing specified as constraints,  
> not explicit solutions, so that new agents can be launched and they  
> will automatically fit into the constraints based on their  
> configuration, rather than having to add them to this or that build  
> definition.  Filter/constraint configuration is a great way to  
> simplify config, especially with large banks of agents.
>
> Christian
>
> On 17-Jan-09, at 16:22 , Wendy Smoak wrote:
>
>> I can see that working, if I can define arbitrary criteria like a
>> property name/value.
>>
>> I need a way to make sure that Top Secret Project X builds on agents
>> 3, 5 and 14 only, and that no other projects ever build on those
>> agents.
>>
>> --
>> Wendy
>>
>> On Sat, Jan 17, 2009 at 10:47 AM, Christian Edward Gruber
>> <cgruber@...> wrote:
>>> Rather than agent groups, could we go with a criteria approach?  
>>> That is to
>>> say, you can describe and agent's capabilities, then associate  
>>> criteria for
>>> a project group or project, and then let the system sort out which  
>>> agents
>>> can fulfill which criteria?
>>
>
> Christian E. Gruber - President / Senior Consultant                
> email:  cgruber@...
> Isráfíl Consulting Services Corporation                            
> mobile: [phone]
> "Keenness of understanding is due to keenness of vision..."        
> phone:  [phone]
>
>
>
>
>

--
Brett Porter
brett@...
http://blogs.exist.com/bporter/

[Christian Edward Gruber]

I think the "magic" can be made visible, in that if it is criteria  
based, the server can display those agents which would apply to the  
current criteria, so that a user can see the implications of his  
selection.  Since the display will be the same logic as the criteria  
selection (it's really just straight set theory) then the user is not  
going to be surprised.

Christian

On 19-Jan-09, at 02:44 , Brett Porter wrote:

> +1
>
> I think selection needs to be a combination of capability,  
> permission and availability. I would mostly prefer the user is in  
> full control of selecting these rather than having too much magic  
> (automated selection might be a nice feature, but the basic starting  
> point should be a fully configured system).
>
> - Brett
>
> On 17/01/2009, at 1:36 PM, Christian Edward Gruber wrote:
>
>> I guess then having agent groups as a logical group, but make group  
>> be part of the criteria would help with that then, group  
>> effectively being a property of the agent.
>>
>> The key for me is to have the whole thing specified as constraints,  
>> not explicit solutions, so that new agents can be launched and they  
>> will automatically fit into the constraints based on their  
>> configuration, rather than having to add them to this or that build  
>> definition.  Filter/constraint configuration is a great way to  
>> simplify config, especially with large banks of agents.
>>
>> Christian
>>
>> On 17-Jan-09, at 16:22 , Wendy Smoak wrote:
>>
>>> I can see that working, if I can define arbitrary criteria like a
>>> property name/value.
>>>
>>> I need a way to make sure that Top Secret Project X builds on agents
>>> 3, 5 and 14 only, and that no other projects ever build on those
>>> agents.
>>>
>>> --
>>> Wendy
>>>
>>> On Sat, Jan 17, 2009 at 10:47 AM, Christian Edward Gruber
>>> <cgruber@...> wrote:
>>>> Rather than agent groups, could we go with a criteria approach?  
>>>> That is to
>>>> say, you can describe and agent's capabilities, then associate  
>>>> criteria for
>>>> a project group or project, and then let the system sort out  
>>>> which agents
>>>> can fulfill which criteria?
>>>
>>
>> Christian E. Gruber - President / Senior Consultant                
>> email:  cgruber@...
>> Isráfíl Consulting Services Corporation                            
>> mobile: [phone]
>> "Keenness of understanding is due to keenness of vision..."        
>> phone:  [phone]
>>
>>
>>
>>
>>
>
> --
> Brett Porter
> brett@...
> http://blogs.exist.com/bporter/
>

Christian E. Gruber - President / Senior Consultant                
email:  cgruber@...
Isráfíl Consulting Services Corporation                            
mobile: [phone]
"Keenness of understanding is due to keenness of vision..."        
phone:  [phone]

[Wendy Smoak-3]

The more I think about this in light of the other discussion about the
problem of figuring out whether or not there have been scm changes
(because the last-used scm checkout might be anywhere) I'm leaning
towards:

1. associate a build environment with a single build agent [1]
2. get parallel builds working on the build agent (CONTINUUM-2045)

It's the effect of concurrent builds that I'm attached to, not
necessarily selecting an agent at build time.  That is, I'd love to
see permission- and capability-based selection of agents at some
point, but I can live with something less than that for the moment.

What do you think?

[1] I still need to make sure only an admin controls which projects
build on which agents...

--
Wendy

On Mon, Jan 19, 2009 at 3:43 PM, Christian Edward Gruber
<cgruber@...> wrote:

> I think the "magic" can be made visible, in that if it is criteria based,
> the server can display those agents which would apply to the current
> criteria, so that a user can see the implications of his selection.  Since
> the display will be the same logic as the criteria selection (it's really
> just straight set theory) then the user is not going to be surprised.
>
> Christian
>
> On 19-Jan-09, at 02:44 , Brett Porter wrote:
>
>> +1
>>
>> I think selection needs to be a combination of capability, permission and
>> availability. I would mostly prefer the user is in full control of selecting
>> these rather than having too much magic (automated selection might be a nice
>> feature, but the basic starting point should be a fully configured system).

...

[Christian Edward Gruber]

I really don't like moving this way.  I think that determining whether  
there have been changes isn't so freaky if you use repository  
commands, not workspace commands to do this.  This should work with  
almost every client/server SCM system except maybe CVS, and with that  
you can still effectively do it with "cvs log -r X,X" (or diff)  
commands.  What worries me is a shift in the approach of continuum  
away from a really useful system based on a limitation of the SCMs.  
I'd rather have a more functional continuum that has fall-back  
approaches for less feature-rich SCM systems.

Christian.

On 28-Jan-09, at 19:08 , Wendy Smoak wrote:

> The more I think about this in light of the other discussion about the
> problem of figuring out whether or not there have been scm changes
> (because the last-used scm checkout might be anywhere) I'm leaning
> towards:
>
> 1. associate a build environment with a single build agent [1]
> 2. get parallel builds working on the build agent (CONTINUUM-2045)
>
> It's the effect of concurrent builds that I'm attached to, not
> necessarily selecting an agent at build time.  That is, I'd love to
> see permission- and capability-based selection of agents at some
> point, but I can live with something less than that for the moment.
>
> What do you think?
>
> [1] I still need to make sure only an admin controls which projects
> build on which agents...
>
> --
> Wendy
>
> On Mon, Jan 19, 2009 at 3:43 PM, Christian Edward Gruber
> <cgruber@...> wrote:
>> I think the "magic" can be made visible, in that if it is criteria  
>> based,
>> the server can display those agents which would apply to the current
>> criteria, so that a user can see the implications of his  
>> selection.  Since
>> the display will be the same logic as the criteria selection (it's  
>> really
>> just straight set theory) then the user is not going to be surprised.
>>
>> Christian
>>
>> On 19-Jan-09, at 02:44 , Brett Porter wrote:
>>
>>> +1
>>>
>>> I think selection needs to be a combination of capability,  
>>> permission and
>>> availability. I would mostly prefer the user is in full control of  
>>> selecting
>>> these rather than having too much magic (automated selection might  
>>> be a nice
>>> feature, but the basic starting point should be a fully configured  
>>> system).
> ...
>

Christian E. Gruber - President / Senior Consultant
Isráfíl Consulting Services Corporation
email:  cgruber@...
mobile: [phone]
web:    http://www.israfil.net/
"...keenness of understanding is due to keenness of vision."

[Wendy Smoak-3]

Does anyone know if the necessary functionality is available through
Maven SCM?  As I understand it, that's what all the scm access goes
through.

--
Wendy

On Wed, Jan 28, 2009 at 5:15 PM, Christian Edward Gruber
<cgruber@...> wrote:
> I really don't like moving this way.  I think that determining whether there
> have been changes isn't so freaky if you use repository commands, not
> workspace commands to do this.  This should work with almost every
> client/server SCM system except maybe CVS, and with that you can still
> effectively do it with "cvs log -r X,X" (or diff) commands.  What worries me
> is a shift in the approach of continuum away from a really useful system
> based on a limitation of the SCMs.  I'd rather have a more functional
> continuum that has fall-back approaches for less feature-rich SCM systems.