Distributed Builds Always Use Same Agent

[George Snyder]

Hello --

I am trying to set up Continuum 1.3.4 to run certain builds on SunOS machines and other builds on Linux machines.  However, when I enable Distributed Builds, all builds seem to run on the first enabled build agent.  My distributed builds configuration looks like:

Build Definition: "SunOS 9 Install"
    Environment: "SunOS 9 Builds"
        Build Agent Group: "SunOS 9 Agents"
            Build Agent: "BldSun1" = http://bldsun1:8181/continuum-buildagent/xmlrpc

Build Definition: "Linux RH Install"
    Environment: "Linux RH Builds"
        Build Agent Group: "Linux RH Agents"
            Build Agent: "BldLnx1" = http://bldlnx1:8181/continuum-buildagent/xmlrpc

If both agents are enabled, then both build definitions always run on the SunOS agent (BldSun1).  If I disable that agent, then both build definitions always run on the Linux agent (BldLnx1).

Is this the same problem as described in CONTINUUM-2349?

Also, http://continuum.apache.org/docs/1.3.4/installation/build-agent.html says:

  "Note that all of your agents need to be identical, as there is no way to
  control which agent Continuum chooses.."

It's not clear to me whether this means all the agents in a build agent group (seems most likely), or all the agents running on a host, or all agents known to a master.  Can someone please clarify?

-- Thanks,
-- George Snyder

[Wendy Smoak-3]

On Thu, Oct 1, 2009 at 1:30 PM, George Snyder <George.Snyder@...> wrote:

> I am trying to set up Continuum 1.3.4 to run certain builds on SunOS
> machines and other builds on Linux machines.  However, when I enable
> Distributed Builds, all builds seem to run on the first enabled build agent.

I wasn't able to reproduce this.  I set up two build defs with
different environments (which have different agent groups) and forced
builds on each definition went to the correct agent.

One thing I will suggest is re-starting.  We recently found that the
continuum.xml file isn't getting updated when you change things
through the UI, so depending on what you were doing, it's possible
that the config file is out of sync.  Stopping Continuum will cause it
to update that file, and it will be read in on startup.

> Also, http://continuum.apache.org/docs/1.3.4/installation/build-agent.html
> says:
>  "Note that all of your agents need to be identical, as there is no way to
>  control which agent Continuum chooses.."

I think that's old info, from before the 'Build Agent Group' feature
was introduced.  Prior to groups, you had no control over what agent
your project would build on.  Can you open an issue to get that
corrected?

--
Wendy

[George Snyder]

I narrowed down the symptoms a bit.  My new observation is:

If a Build Environment specifies a Build Agent Group but NO Installations, then associated Build Definitions run on some other Build Agent.

(This seems consistent with CONTINUUM-2349, which mentions creating a Build Environment, but does not mention an Installation).

Workaround: Add at least one Installation to each Build Environment which specifies a Build Agent Group.

(We have been using Continuum 1.1 for a couple of years with no need for Installations or Profiles.  For the workaround, I added a "Maven 2" path, since we have it in the same place on all machines).

It seems odd to require Installations for distributed builds.  The corresponding paths are defined in the build agent's 'continuum-buildagent.xml'.  The dialog for creating an Installation seems to require that "Tool" paths exist on the master, which may be different.  (The dialog does not seem to check environment variable paths, so presumably one could set JAVA_HOME instead of a JDK).

(This is my first encounter with this area in Continuum, so I may not be understanding it completely).

I submitted CONTINUUM-2380 for the documentation.

-- George Snyder

[Wendy Smoak-3]

On Fri, Oct 2, 2009 at 9:17 AM, George Snyder <George.Snyder@...> wrote:

> I narrowed down the symptoms a bit.  My new observation is:
>
> If a Build Environment specifies a Build Agent Group but NO Installations,
> then associated Build Definitions run on some other Build Agent.

My Build Environments did not have any Installations, I only defined
the agent group, and it still built on the correct agent.  Something
else is going on...

--
Wendy

[George Snyder]

You're right -- I'm still getting builds sent to a build agent which is not in the group for the build definition.  Any suggestions on how to debug this?  I don't see any messages about agent selection in 'continuum.log'.

-- George Snyder

Wendy Smoak-3 wrote:

My Build Environments did not have any Installations, I only defined
the agent group, and it still built on the correct agent.  Something
else is going on...

[Wendy Smoak-3]

On Fri, Oct 2, 2009 at 12:34 PM, George Snyder <George.Snyder@...> wrote:
> You're right -- I'm still getting builds sent to a build agent which is not
> in the group for the build definition.  Any suggestions on how to debug
> this?  I don't see any messages about agent selection in 'continuum.log'.

I opened an issue to improve the logging on the master.  Right now you
can't tell what's building or where it's going...

Did you try re-starting or did you make sure continuum.xml is correct?
 What you see in the Web UI *might* not be the info Continuum is
actually using.  (That, at least, has been fixed on trunk.)

Let me see if I can do something about the log messages.  If you're
interested in trying out a snapshot with these fixes, please come join
us on the dev list so we can talk about using the latest and greatest
unreleased stuff.

--
Wendy

[George Snyder]

Wendy Smoak-3 wrote:

I opened an issue to improve the logging on the master.  Right now you
can't tell what's building or where it's going...

Thanks.

Wendy Smoak-3 wrote:

Did you try re-starting or did you make sure continuum.xml is correct?
 What you see in the Web UI *might* not be the info Continuum is
actually using.  (That, at least, has been fixed on trunk.)

I have been restarting the master after each configuration change.  

My build logs print out environment info, including host name.  The ones I checked all matched the name in the agent URL shown on the Continuum Queues page.

Wendy Smoak-3 wrote:

Let me see if I can do something about the log messages.  If you're
interested in trying out a snapshot with these fixes, please come join
us on the dev list so we can talk about using the latest and greatest
unreleased stuff.

I'll look into the dev list.

-- George Snyder

[George Snyder]

Another symptom which may be related: E-mail notifications for distributed builds are not getting variables substituted correctly. It feels like an interface mismatch between master and server. Both are running on the same machine (version info is below). In the example below, I copied the "actually" values from the on-line report (build output).

---

From: Continuum@DevBld1 [mailto:[e-mail]]
Sent: Friday, October 02, 2009 8:09 PM
To: Snyder, George
Subject: [continuum] BUILD ERROR: Unknown build state 0: [Inactive Components] - Build System Test Project - ${build.buildDefinition.description}
                                        # Actually "Default Maven 2 Build Definition"

Online report : http://devbld1:8085/continuum/buildResult.action?buildId=181&projectId=21

Build statistics:
  State: Unknown project state '0'      # Actually "Success"
  Previous State: Ok
  Started at: $formatter.formatTimestamp( $build.startTime ) 
                                        # Actually Oct 02, 2009 08:08:16 PM EDT
  Finished at: $formatter.formatTimestamp( $build.endTime )
                                        # Actually Oct 02, 2009 08:08:38 PM EDT
  Total time: 0s                        # Actually 21 sec
  Build Trigger: Schedule               # Actually "Forced"
  Build Number: 0                       # Actually 7
  Exit code: 0
  Building machine hostname: devbld1
  Operating system : Linux(unknown)
  Java Home version :
          java version "1.6.0_11"
          Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
          Java HotSpot(TM) Server VM (build 11.0-b16, mixed mode)

  Builder version :
          Maven version: 2.0.8
          Java version: 1.6.0_11
          OS name: "linux" version: "2.6.18-8.1.10.el5" arch: "i386" Family: "unix"

---

-- George Snyder

[Marica Tan-2]

On Tue, Oct 6, 2009 at 8:19 AM, George Snyder <George.Snyder@...>wrote:

>
> Another symptom which may be related: E-mail notifications for distributed
> builds are not getting variables substituted correctly.  It feels like an
> interface mismatch between master and server.  Both are running on the same
> machine (version info is above).  In the example below, I copied the
> "actually" values from the on-line report (build output).
>
>
> This has been fixed by CONTINUUM-2348.

>
> From: Continuum@DevBld1 [mailto:continuum@...]
> Sent: Friday, October 02, 2009 8:09 PM
> To: Snyder, George
> Subject: [continuum] BUILD ERROR: Unknown build state 0: [Inactive
> Components] - Build System Test Project -
> ${build.buildDefinition.description}
>                                        # Actually "Default Maven 2 Build
> Definition"
>
> Online report :
> http://devbld1:8085/continuum/buildResult.action?buildId=181&projectId=21
>
> Build statistics:
>  State: Unknown project state '0'      # Actually "Success"
>  Previous State: Ok
>  Started at: $formatter.formatTimestamp( $build.startTime )
>                                        # Actually Oct 02, 2009 08:08:16 PM
> EDT
>  Finished at: $formatter.formatTimestamp( $build.endTime )
>                                        # Actually Oct 02, 2009 08:08:38 PM
> EDT
>  Total time: 0s                        # Actually 21 sec
>  Build Trigger: Schedule               # Actually "Forced"
>  Build Number: 0                       # Actually 7
>  Exit code: 0
>  Building machine hostname: devbld1
>  Operating system : Linux(unknown)
>  Java Home version :
>          java version "1.6.0_11"
>          Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
>          Java HotSpot(TM) Server VM (build 11.0-b16, mixed mode)
>
>  Builder version :
>          Maven version: 2.0.8
>          Java version: 1.6.0_11
>          OS name: "linux" version: "2.6.18-8.1.10.el5" arch: "i386" Family:
> "unix"
>
>
>
> -- George Snyder
>
> --
> View this message in context:
> http://www.nabble.com/Distributed-Builds-Always-Use-Same-Agent-tp25705964p25757480.html
> Sent from the Continuum - Users mailing list archive at Nabble.com.
>

[Marica Tan-2]

On Sat, Oct 3, 2009 at 3:34 AM, George Snyder <George.Snyder@...>wrote:

>
> You're right -- I'm still getting builds sent to a build agent which is not
> in the group for the build definition.


This might be related to CONTINUUM-2349.

> Any suggestions on how to debug
> this?  I don't see any messages about agent selection in 'continuum.log'.
>
> -- George Snyder
>
>
> Wendy Smoak-3 wrote:
> >
> > My Build Environments did not have any Installations, I only defined
> > the agent group, and it still built on the correct agent.  Something
> > else is going on...
> >
>
> --
> View this message in context:
> http://www.nabble.com/Distributed-Builds-Always-Use-Same-Agent-tp25705964p25721342.html
> Sent from the Continuum - Users mailing list archive at Nabble.com.
>
>

[George Snyder]

I finally got my Build Environments to use the right build agents.  The major factors seem to have been:

- Carefully correcting config/continuum.xml by hand-editing;

- Killing off rogue build agent processes on my Solaris 9 machine.  I don't know how much of a problem they were causing, but they certainly weren't helping.  I submitted CONTINUUM-2382.

Thanks for the help,
-- George Snyder