Thoughts on build environment selection when releasing in distributed build setup

[Jevica Arianne B. Zurbano-3]

Hi!

I suggest that for CONTINUUM-2386 (Build environment selection is
ignored when releasing with distributed build enabled)

The build environment selection during release will be disabled/removed.

As mentioned in CONTINUUM-2403 (Release should happen on a build agent
rather than on the master), that the release will happen in the
buildagent where the project was last built since the working copy is in
there.

wdyt?

--

Thanks,

Jev

[Wendy Smoak-3]

On Sun, Jan 31, 2010 at 8:57 PM, Jevica Arianne B. Zurbano
<jevica.arianne@...> wrote:

> I suggest that for CONTINUUM-2386 (Build environment selection is ignored
> when releasing with distributed build enabled)
>
> The build environment selection during release will be disabled/removed.

How does that solve the problem?

I need to be able to select the build environment to be used during
the release, so that I can control things like what JDK, Maven version
and environment variables are used.

> As mentioned in CONTINUUM-2403 (Release should happen on a build agent
> rather than on the master), that the release will happen in the buildagent
> where the project was last built since the working copy is in there.

I think you mean http://jira.codehaus.org/browse/CONTINUUM-2043 which
was already fixed.  What does it have to do with 2386?

--
Wendy

[brettporter]

I'm a bit confused by the sequence - does the release do a new checkout? If so, why does the working copy matter?

Even if not, I think that you should be able to select the build environment (after all, you can if you aren't using dist. builds). It should do a checkout there (if there isn't one), and default to the default build agent if no other is selected rather than just disabling it...

- Brett

On 01/02/2010, at 2:57 PM, Jevica Arianne B. Zurbano wrote:

> Hi!
>
> I suggest that for CONTINUUM-2386 (Build environment selection is ignored when releasing with distributed build enabled)
>
> The build environment selection during release will be disabled/removed.
>
> As mentioned in CONTINUUM-2403 (Release should happen on a build agent rather than on the master), that the release will happen in the buildagent where the project was last built since the working copy is in there.
>
> wdyt?
>
> --
>
> Thanks,
>
> Jev
>

--
Brett Porter
brett@...
http://brettporter.wordpress.com/

[Marica Tan-2]

On Mon, Feb 1, 2010 at 12:12 PM, Brett Porter <brett@...> wrote:

> I'm a bit confused by the sequence - does the release do a new checkout? If
> so, why does the working copy matter?
>
>
The working copy is needed during the release prepare.


> Even if not, I think that you should be able to select the build
> environment (after all, you can if you aren't using dist. builds). It should
> do a checkout there (if there isn't one), and default to the default build
> agent if no other is selected rather than just disabling it...
>
>
+1

--
Marica

> - Brett
>
> On 01/02/2010, at 2:57 PM, Jevica Arianne B. Zurbano wrote:
>
> > Hi!
> >
> > I suggest that for CONTINUUM-2386 (Build environment selection is ignored
> when releasing with distributed build enabled)
> >
> > The build environment selection during release will be disabled/removed.
> >
> > As mentioned in CONTINUUM-2403 (Release should happen on a build agent
> rather than on the master), that the release will happen in the buildagent
> where the project was last built since the working copy is in there.
> >
> > wdyt?
> >
> > --
> >
> > Thanks,
> >
> > Jev
> >
>
> --
> Brett Porter
> brett@...
> http://brettporter.wordpress.com/
>
>
>
>
>

[Jevica Arianne B. Zurbano-3]

Here's my approach regarding the buildagent used for release.
(http://jira.codehaus.org/browse/CONTINUUM-2386)

Scenarios:
* multiple buildagents in BUILDAGENT_GROUP configured in a BUILD_ENVIRONMENT
   - The first enabled buildagent encountered will be used.
* no BUILDAGENT_GROUP configured in the selected BUILD_ENVIRONMENT, and
buildagents in BUILDAGENT_GROUP are disabled
   - The default buildagent will be used.

wdyt?


Marica Tan wrote:

> On Mon, Feb 1, 2010 at 12:12 PM, Brett Porter <brett@...> wrote:
>
>  
>> I'm a bit confused by the sequence - does the release do a new checkout? If
>> so, why does the working copy matter?
>>
>>
>>    
> The working copy is needed during the release prepare.
>
>
>  
>> Even if not, I think that you should be able to select the build
>> environment (after all, you can if you aren't using dist. builds). It should
>> do a checkout there (if there isn't one), and default to the default build
>> agent if no other is selected rather than just disabling it...
>>
>>
>>    
> +1
>
> --
> Marica
>
>
>
>  
>> - Brett
>>    
>>
>> --
>> Brett Porter
>> brett@...
>> http://brettporter.wordpress.com/
>>
>>
>>
>>
>>
>>    
>
>  

--

Thanks,

Jev

[Wendy Smoak-3]

On Tue, Feb 2, 2010 at 5:41 AM, Jevica Arianne B. Zurbano
<jevica.arianne@...> wrote:
> Here's my approach regarding the buildagent used for release.
> (http://jira.codehaus.org/browse/CONTINUUM-2386)
>
> Scenarios:
> * multiple buildagents in BUILDAGENT_GROUP configured in a BUILD_ENVIRONMENT
>  - The first enabled buildagent encountered will be used.

Is there a reason to change the current behavior, which is that it
prefers the last-used build agent?  (Fixing it so that it checks out a
new working copy as Brett mentioned seems out of scope for this issue,
but maybe it's easier.  I think right now you have to have a previous
successful build for the release to work.)

> * no BUILDAGENT_GROUP configured in the selected BUILD_ENVIRONMENT, and
> buildagents in BUILDAGENT_GROUP are disabled
>  - The default buildagent will be used.

Not sure about the "and" here, isn't this two separate cases?  (How
can they be disabled if there is no group?)

How is the default build agent defined?

What happens if there is no BUILD_ENVIRONMENT chosen at all?  (Or is
it required in the distributed build case?)

Maybe a wiki page would be better to sort this out, it needs to get
into the docs eventually. :)

--
Wendy

[Wendy Smoak-3]

I had a long conversation with myself about this on the drive home,
and I keep going in circles.  I don't know how much of the existing
behavior is intentional vs. incidental.

In non-distributed build, it will not let you do a release unless the
projects are in 'build success' state. Obviously, the release happens
on that working copy.

Carrying that over to distributed build, you still have to have a
previous successful build.  IF the release has to happen on that
working copy, then you don't get to pick the build agent.  And since
the build agent is part of the build environment...

So, can someone explain why you have to have a previous successful
build in order to do a release?  It's not a requirement at the maven
command line.  The release might fail, but you are not prevented from
trying it.

Is this a real requirement, or can we go with Brett's idea that
Continuum Release should do a checkout, which would mean it could
happen anywhere?

Honestly I think this work should be delayed until the architecture
changes are sorted out.  It shouldn't be this complicated.

--
Wendy

[Marica Tan-2]

I think the reasons why it needs to have a previous successful build are:

1. to make sure there is a working copy
2. to make sure we have a successful build before we tag. there is no sense
to release a project if the build failed because the release will fail as
well.


Thanks,
-
Marica

On Wed, Feb 3, 2010 at 9:14 AM, Wendy Smoak <wsmoak@...> wrote:

> I had a long conversation with myself about this on the drive home,
> and I keep going in circles.  I don't know how much of the existing
> behavior is intentional vs. incidental.
>
> In non-distributed build, it will not let you do a release unless the
> projects are in 'build success' state. Obviously, the release happens
> on that working copy.
>
> Carrying that over to distributed build, you still have to have a
> previous successful build.  IF the release has to happen on that
> working copy, then you don't get to pick the build agent.  And since
> the build agent is part of the build environment...
>
> So, can someone explain why you have to have a previous successful
> build in order to do a release?  It's not a requirement at the maven
> command line.  The release might fail, but you are not prevented from
> trying it.
>
> Is this a real requirement, or can we go with Brett's idea that
> Continuum Release should do a checkout, which would mean it could
> happen anywhere?
>
> Honestly I think this work should be delayed until the architecture
> changes are sorted out.  It shouldn't be this complicated.
>
> --
> Wendy
>

[Wendy Smoak-3]

On Tue, Feb 2, 2010 at 6:58 PM, Marica Tan <marica.tan@...> wrote:
> I think the reasons why it needs to have a previous successful build are:
>
> 1. to make sure there is a working copy

You don't need to build to have a working copy, just check out.

> 2. to make sure we have a successful build before we tag. there is no sense
> to release a project if the build failed because the release will fail as
> well.

There's no relationship between a successful build in the past and a
successful release now.  A commit in between could have fixed (or
broken) the build, giving a different result when you try to release.

This feels like an arbitrary restriction that could be removed.

--
Wendy

[Marica Tan-2]

On Wed, Feb 3, 2010 at 10:04 AM, Wendy Smoak <wsmoak@...> wrote:

> On Tue, Feb 2, 2010 at 6:58 PM, Marica Tan <marica.tan@...> wrote:
> > I think the reasons why it needs to have a previous successful build are:
> >
> > 1. to make sure there is a working copy
>
> You don't need to build to have a working copy, just check out.
>
>
We could add a check to see if the working copy exists. If it doesn't then
do a checkout.

> > 2. to make sure we have a successful build before we tag. there is no
> sense
> > to release a project if the build failed because the release will fail as
> > well.
>
> There's no relationship between a successful build in the past and a
> successful release now.  A commit in between could have fixed (or
> broken) the build, giving a different result when you try to release.
>
> This feels like an arbitrary restriction that could be removed.
>
> Yes, I agree.

> --
> Wendy
>

--
Marica

[Jevica Arianne B. Zurbano-3]

 From what I have investigated, the current behavior is that it only
updates the working copy (UpdateWorkingCopyPhase.java code snippet
below) that is why the working copy is needed and attempting to release
it on another build agent might fail if the working copy is not found.
IMO, the checking out of the working copy could be applied in this case.

    try
    {
        result = provider.update( repository, new ScmFileSet( new File(
releaseDescriptor.getWorkingDirectory() ) ), (ScmVersion) null );
    }
    catch ( ScmException e )

The build agent to be used is determined or taken from the last project
build result (DefaultDistributedReleaseManager.java code snippet below).
This is why it always result to using the last or default build agent.
However, it is possible to set it to use the build agent from the
selected environment.

    BuildResult buildResult =
buildResultDao.getLatestBuildResultForProject( project.getId() );
    String buildAgentUrl = buildResult.getBuildUrl();



Marica Tan wrote:

> I think the reasons why it needs to have a previous successful build are:
>
> 1. to make sure there is a working copy
> 2. to make sure we have a successful build before we tag. there is no sense
> to release a project if the build failed because the release will fail as
> well.
>
>
> Thanks,
> -
> Marica
>
> On Wed, Feb 3, 2010 at 9:14 AM, Wendy Smoak <wsmoak@...> wrote:
>
>  
>> I had a long conversation with myself about this on the drive home,
>> and I keep going in circles.  I don't know how much of the existing
>> behavior is intentional vs. incidental.
>>
>> In non-distributed build, it will not let you do a release unless the
>> projects are in 'build success' state. Obviously, the release happens
>> on that working copy.
>>
>> Carrying that over to distributed build, you still have to have a
>> previous successful build.  IF the release has to happen on that
>> working copy, then you don't get to pick the build agent.  And since
>> the build agent is part of the build environment...
>>
>> So, can someone explain why you have to have a previous successful
>> build in order to do a release?  It's not a requirement at the maven
>> command line.  The release might fail, but you are not prevented from
>> trying it.
>>
>> Is this a real requirement, or can we go with Brett's idea that
>> Continuum Release should do a checkout, which would mean it could
>> happen anywhere?
>>    

+1 on doing a checkout
I am not sure if this is a requirement though.

>> Honestly I think this work should be delayed until the architecture
>> changes are sorted out.  It shouldn't be this complicated.
>> --
>> Wendy
>>
>>    
>
>  

--

Thanks,

Jev

[Marica Tan-2]

On Wed, Feb 3, 2010 at 10:58 AM, Jevica Arianne B. Zurbano <
jevica.arianne@...> wrote:

> From what I have investigated, the current behavior is that it only updates
> the working copy (UpdateWorkingCopyPhase.java code snippet below) that is
> why the working copy is needed and attempting to release it on another build
> agent might fail if the working copy is not found. IMO, the checking out of
> the working copy could be applied in this case.
>
>   try
>   {
>       result = provider.update( repository, new ScmFileSet( new File(
> releaseDescriptor.getWorkingDirectory() ) ), (ScmVersion) null );
>   }
>   catch ( ScmException e )
>
> This is already out of the scope for this issue :)

We can create another issue that will handle this.

> The build agent to be used is determined or taken from the last project
> build result (DefaultDistributedReleaseManager.java code snippet below).
> This is why it always result to using the last or default build agent.
> However, it is possible to set it to use the build agent from the selected
> environment.
>
>   BuildResult buildResult = buildResultDao.getLatestBuildResultForProject(
> project.getId() );
>   String buildAgentUrl = buildResult.getBuildUrl();
>
>
> IMHO, we could have a simple fix for now, like being able to set an

environment variable (e.g. heap size) or a build tool (jdk, maven or ant)
during release and then ignore the build agent group in the build
environment for the meantime.


Thanks,
--
Marica

>
>
> Marica Tan wrote:
>
>> I think the reasons why it needs to have a previous successful build are:
>>
>> 1. to make sure there is a working copy
>> 2. to make sure we have a successful build before we tag. there is no
>> sense
>> to release a project if the build failed because the release will fail as
>> well.
>>
>>
>> Thanks,
>> -
>> Marica
>>
>> On Wed, Feb 3, 2010 at 9:14 AM, Wendy Smoak <wsmoak@...> wrote:
>>
>>
>>
>>> I had a long conversation with myself about this on the drive home,
>>> and I keep going in circles.  I don't know how much of the existing
>>> behavior is intentional vs. incidental.
>>>
>>> In non-distributed build, it will not let you do a release unless the
>>> projects are in 'build success' state. Obviously, the release happens
>>> on that working copy.
>>>
>>> Carrying that over to distributed build, you still have to have a
>>> previous successful build.  IF the release has to happen on that
>>> working copy, then you don't get to pick the build agent.  And since
>>> the build agent is part of the build environment...
>>>
>>> So, can someone explain why you have to have a previous successful
>>> build in order to do a release?  It's not a requirement at the maven
>>> command line.  The release might fail, but you are not prevented from
>>> trying it.
>>>
>>> Is this a real requirement, or can we go with Brett's idea that
>>> Continuum Release should do a checkout, which would mean it could
>>> happen anywhere?
>>>
>>>
>>
> +1 on doing a checkout
> I am not sure if this is a requirement though.
>
>
>  Honestly I think this work should be delayed until the architecture
>>> changes are sorted out.  It shouldn't be this complicated.
>>> --
>>> Wendy
>>>
>>>
>>>
>>
>>
>>
>
> --
>
> Thanks,
>
> Jev
>
>

[Wendy Smoak-3]

On Tue, Feb 2, 2010 at 11:30 PM, Marica Tan <marica.tan@...> wrote:

>> IMHO, we could have a simple fix for now, like being able to set an
> environment variable (e.g. heap size) or a build tool (jdk, maven or ant)
> during release and then ignore the build agent group in the build
> environment for the meantime.

Have you verified that it ignores the entire build environment?  I
only noticed the problem with the build agent.  I have not tested
whether the tools/envars are respected.

For 1.3.6 we need to clearly document whatever the actual behavior is.

For 1.4.x I would like to see the ability to select a build
environment, including the associated build agent group, and have the
release environment include those selections.  It should prefer the
last-used build agent if it is present in the group, to save
time/space by not having to do another checkout.  But if there is no
checkout of the project being released on the build agent that gets
selected, then it should do a checkout and proceed, just as it would
for a scheduled or forced build.

But first I would like to see if the architectural changes that anyone
has in mind touch this area of the code, possibly invalidating this
entire discussion.

Thanks,
--
Wendy

--
Wendy

[Jevica Arianne B. Zurbano-3]

I have created
http://cwiki.apache.org/confluence/display/CONTINUUM/Build+Environment+Selection+During+Release

Kindly review.

Thanks in advance!

--


Jev

[Wendy Smoak-3]

On Mon, Feb 8, 2010 at 10:11 PM, Jevica Arianne B. Zurbano
<jevica.arianne@...> wrote:
> I have created
> http://cwiki.apache.org/confluence/display/CONTINUUM/Build+Environment+Selection+During+Release

Thanks, that makes it easier to work with.  Initial thoughts:

Selection of Build Agent - Part A:  I don't think the
default/last-used build agent should ever be used if it is not part of
the selected build environment.

That is, if I select a build environment for the release, I would be
unpleasantly surprised to have Continuum decide to use an agent that
is not part of that build env.  (not included in the agent group
assigned to that build env.)

It should prefer the last-used agent *if* it is in the build env, to
save time by not having to do another checkout.

How does the 'selected agent' get selected from the agent group?
(This may be documented elsewhere as it's probably the same as for
scheduled builds.)

The definition of the default build agent as the one that was last
used to build the project doesn't work if we accept that a previous
successful build is not necessary to do a release.  (But I see that's
under future enhancements, so okay.)

Thanks,
--
Wendy

[Marica Tan-2]

On Wed, Feb 10, 2010 at 12:19 AM, Wendy Smoak <wsmoak@...> wrote:

> On Mon, Feb 8, 2010 at 10:11 PM, Jevica Arianne B. Zurbano
> <jevica.arianne@...> wrote:
> > I have created
> >
> http://cwiki.apache.org/confluence/display/CONTINUUM/Build+Environment+Selection+During+Release
>
> Thanks, that makes it easier to work with.  Initial thoughts:
>
> Selection of Build Agent - Part A:  I don't think the
> default/last-used build agent should ever be used if it is not part of
> the selected build environment.
>
> If the user didn't select a build environment, how will continuum know

which agent to perform the release?
Are we requiring the user to select a build environment when releasing with
distributed builds?

With non-distributed, I can opt not to select a build environment and still
be able to do a release.

> That is, if I select a build environment for the release, I would be
> unpleasantly surprised to have Continuum decide to use an agent that
> is not part of that build env.  (not included in the agent group
> assigned to that build env.)
>
> It should prefer the last-used agent *if* it is in the build env, to
> save time by not having to do another checkout.
>
> How does the 'selected agent' get selected from the agent group?
> (This may be documented elsewhere as it's probably the same as for
> scheduled builds.)
>
>

+1

I think the selection should go like this:

1. last-used agent
2. agent with the least build load.

The definition of the default build agent as the one that was last
> used to build the project doesn't work if we accept that a previous
> successful build is not necessary to do a release.  (But I see that's
> under future enhancements, so okay.)
>
>
If we implement this, we could just add a check if there's a previous build
and if none then do a checkout in the selected build agent, which is the
agent with the least build load. And I think this could work on both
with/without selected build environment.

Thanks,
--
Marica

Thanks,
> --
> Wendy
>

[Wendy Smoak-3]

On Tue, Feb 9, 2010 at 3:39 PM, Marica Tan <marica.tan@...> wrote:
> On Wed, Feb 10, 2010 at 12:19 AM, Wendy Smoak <wsmoak@...> wrote:

>> Selection of Build Agent - Part A:  I don't think the
>> default/last-used build agent should ever be used if it is not part of
>> the selected build environment.
>>
>> If the user didn't select a build environment, how will continuum know
> which agent to perform the release?
> Are we requiring the user to select a build environment when releasing with
> distributed builds?
>
> With non-distributed, I can opt not to select a build environment and still
> be able to do a release.

In non-distributed build, not selecting a build environment means it
will use the "default"  environment, whatever Continuum is running in.

That doesn't seem to make sense in a distributed environment, since
nothing builds on the master where you are pressing the buttons.

Therefore, I think selection of a build environment should be
mandatory in distributed builds.

Any arguments to the contrary?

[This decision will have effects on the scheduled/forced build
features as well as configuration and UI.]

--
Wendy

[Marica Tan-2]

On Wed, Feb 10, 2010 at 6:50 AM, Wendy Smoak <wsmoak@...> wrote:

> On Tue, Feb 9, 2010 at 3:39 PM, Marica Tan <marica.tan@...> wrote:
> > On Wed, Feb 10, 2010 at 12:19 AM, Wendy Smoak <wsmoak@...> wrote:
>
> >> Selection of Build Agent - Part A:  I don't think the
> >> default/last-used build agent should ever be used if it is not part of
> >> the selected build environment.
> >>
> >> If the user didn't select a build environment, how will continuum know
> > which agent to perform the release?
> > Are we requiring the user to select a build environment when releasing
> with
> > distributed builds?
> >
> > With non-distributed, I can opt not to select a build environment and
> still
> > be able to do a release.
>
> In non-distributed build, not selecting a build environment means it
> will use the "default"  environment, whatever Continuum is running in.
>
> That doesn't seem to make sense in a distributed environment, since
> nothing builds on the master where you are pressing the buttons.
>
> Therefore, I think selection of a build environment should be
> mandatory in distributed builds.
>
> Any arguments to the contrary?
>
> [This decision will have effects on the scheduled/forced build
> features as well as configuration and UI.]
>

+1

We should create another issue for making it mandatory in distributed builds
and document this too.

Thanks,
--
Marica

>

--
> Wendy
>

[Jevica Arianne B. Zurbano-3]

Marica Tan wrote:

> On Wed, Feb 10, 2010 at 6:50 AM, Wendy Smoak <wsmoak@...> wrote:
>
>  
>> On Tue, Feb 9, 2010 at 3:39 PM, Marica Tan <marica.tan@...> wrote:
>>    
>> In non-distributed build, not selecting a build environment means it
>> will use the "default"  environment, whatever Continuum is running in.
>>
>> That doesn't seem to make sense in a distributed environment, since
>> nothing builds on the master where you are pressing the buttons.
>>
>> Therefore, I think selection of a build environment should be
>> mandatory in distributed builds.
>>
>> Any arguments to the contrary?
>>
>> [This decision will have effects on the scheduled/forced build
>> features as well as configuration and UI.]
>>
>>    
>
> +1
>
> We should create another issue for making it mandatory in distributed builds
> and document this too.
>  

+1
for making the build environment selection mandatory and for documentation


Marica Tan wrote:

> On Wed, Feb 10, 2010 at 12:19 AM, Wendy Smoak <wsmoak@...> wrote:
>  
>> That is, if I select a build environment for the release, I would be
>> unpleasantly surprised to have Continuum decide to use an agent that
>> is not part of that build env.  (not included in the agent group
>> assigned to that build env.)
>>
>> It should prefer the last-used agent *if* it is in the build env, to
>> save time by not having to do another checkout.
>>
>> How does the 'selected agent' get selected from the agent group?
>> (This may be documented elsewhere as it's probably the same as for
>> scheduled builds.)
>>
>>
>>    
> +1
>
> I think the selection should go like this:
>
> 1. last-used agent
> 2. agent with the least build load.
>  

wiki updated.

> The definition of the default build agent as the one that was last
>  
>> used to build the project doesn't work if we accept that a previous
>> successful build is not necessary to do a release.  (But I see that's
>> under future enhancements, so okay.)
>>
>>
>>    
> If we implement this, we could just add a check if there's a previous build
> and if none then do a checkout in the selected build agent, which is the
> agent with the least build load. And I think this could work on both
> with/without selected build environment.
>  

This is added in the "Future Enhancements". IMHO, this should be
discussed separately since it concerns a different feature of Continuum.
And, this might affect some other features.
>
>  

--

Thanks,

Jev

[Jevica Arianne B. Zurbano-3]

Updated wiki for CONTINUUM-2386 (Build environment selection is ignored
when releasing with distributed build enabled)
http://cwiki.apache.org/confluence/display/CONTINUUM/Build+Environment+Selection+During+Release

These are the issues to be fixed/implemented: (under the References
section from the link above)
<http://jira.codehaus.org/browse/CONTINUUM-2386>CONTINUUM-2368 - Build
environment selection is ignored when releasing with distributed build
enabled
<http://jira.codehaus.org/browse/CONTINUUM-2458>CONTINUUM-2458 -
Continuum Release should do a checkout if there is no working copy
CONTINUUM-2464 - Build environment should be required when releasing in
distributed build setup

--

Thanks,

Jev