SCons Future Directions and Thoughts

[Ken Deeter]

These are a few notes from a conversation I had with Steve in person.

We've been using scons at VMware for over a year now, and we've been pretty
happy with it. We've converted the builds of one of our largest products and
realized significant build performance improvements, as well as a much more
concise and maintainable build infrastructure.

That being said, there are always improvements to be made, and the purpose of
this message is to get the discussion started so that we can hopefully
clarify and focus scons development efforts.

-----

The first issue I want to bring up is that of the release cycle. Currently
scons doesn't really have much of a defined release process. Steve and I have
chatted about it in the past, we've never really done anything to effectively
change the status quo. From what I have seen, the main implication of this
has been that there hasn't been much discipline exercised in deciding when
and where to accept major destabilizing features. There is no game plan, and
thus everything is done on an ad-hoc, case-by-case process.

Because of this, companies that use scons, like VMware, have a difficult time
contributing code back. If we're stuck on an old version, our potential
patches to that old version are more likely to be irrelevant against the
upstream version. We also get reverse situations, where it is difficult for a
company to import a new version of scons because not enough work has been
done between the introduction of major disruptive features to end up with a
solid relatively bug-free product. It pretty much comes down to a crapshoot
of versions, where you pick one and hope that it works.

I propose that the solution is to move to time-based releases, with
stabilization periods (and perhaps release candidates) before every release.
I think the GNOME project has shown that this can work well. What this
amounts to is essentially that we say up front that we will aim to release
every 6 months. Perhaps 4 months or so into each release cycle, we can branch
the code to create a release branch. Stabilization work can happen on this
branch, and no major features will be added to this branch. We can cut
regular snapshot release candidate releases from this branch, and release a
final tarball off of the branch at the end of the six months. If needed we
can also release critical point releases to fix major flaws found after the
release.

There are several benefits to this approach:
 - users of a release can know how much time the code has had to stabilize
 - patch contributors can target a 'stable' code base
 - the algorithm for whether a new feature makes it in becomes simpler. It
becomes a question of 'might it cause instability?' instead of 'what set of
features do we want for the next release' (and anyone who's worked at a large
software company knows that the latter question cannot easily be answered
even by a large group of dedicated people)
 - contributors will see their new code released sooner because the release
is bound by time, not by someone's arbitrary idea of readiness.

There are some downsides:
 - branching of the code always creates a few headaches
 - more release 'work', though hopefully this will be automatable over time.

As scons continues to make inroads into large software development
organizations, having a predictable and reliable release process becomes
increasingly important. I would hate to see people lose interest in using the
project because of any perceived unpredictability.

-----

The second issue I'd like to bring up is about focus of the project. The one
thing I've seen time and again is that open source projects get pulled in
every which direction, and begin to lose focus of their core competencies.
With the few resources that open source projects have, spreading developer
time over more and more features causes not only developer stress, but a
stagnation in the progress of the final product.

I'd like to first present a way to look at different parts of scons that I've
developed over the time I've worked with it at VMware. In this context, I'd
like to suggest what I consider is the core value that scons provides as a
tool, and some suggestions to really help us concentrate on providing more
core value.

Before we started with scons, we evaluated a bunch of different systems. We
picked scons because it has several properties that provide a lot of value:
  1. the abstraction of construction environments and the representation of
the dependency graph
  2. the fact that we could 'program' the depenendency tree.
  3. It can walk the dependency tree and manage parallel build jobs.

1) gives us a solid foundation that we can say with reasonable confidence
that we will be able to adapt to any situation that might arise. Then 2)
gives us a really flexible and concise way to specify a huge and complex
dependency graph. 3), while it can be done by make, one can argue that scons
can do it better.

What scons has managed to do is to separate the usefulness of a dependency
graph solver, with the usual inflexible specification format of the
dependency graph itself.

Scons as a system can be divided into two large parts. (again, these are how
I happen to view them)
 - The "core": includes construction environments, nodes, actions, builders,
and the infrastructure to do the dag walk. As well as the various features
that are directly related to these entities.
 - The "conveniences": the tools, the various support for version control
repositories, build directory management through BuildDir(), etc.

Now it's no coincidence that these two parts also map directly to how we use
scons at VMware. We use the 'core' part, but we don't use the 'conveniences'
part. However, I also believe that this separation likely applies for other
large organizations that want to use scons.

I'll briefly describe why we don't use each item I listed in the conveniences
part:

 * Tools: the scons builtin tools have a general approach that involves
trying to detect toolchains installed on the host doing to the build. While
this is exactly what build systems for open source projects need to do, it
turns out this is _exactly the opposite_ of what build systems for
organizations that try to ship binary software want to do. At VMware, we've
essentially replace all of the Tools system with an inhouse version that has
the same concept of mix and match tools, but are written to specify exact
versions of various tools in precise locations. The toolchain is maintained
centrally, and picking up 'your version of gcc' is exactly the last thing one
wants a build to do.

 * version control repositories: having a build system go and fetch an
individual file whenever it needs it simply does not scale. We're talking
about potentially checking out 40k files in a build.. having to do a request
for each one is insane. In fact, even large open source projects have a model
where all the source is checked out before a build is run.

 * build directory management: our build directory structure is simply too
complicated to be managed by a simple mechanism like BuildDir().

Another, more technical way to look at this: Everything in 'conveniences' can
be achieved in a user's scons scripts. At VMware, we've done tools.
Repository support could be reduced to some kind of clever builder that is
attached to source files. BuildDir can be done by writing a library to map
sourcefiles to targetfiles. Architecturally, these can be pretty well
separated from the core scons 'engine' and api.

So where does this leave us? I've found that in using scons for VMware, I
really want to see the "core" part advance. At times, I feel that this is
almost hindered by having to support  the "conveniences". What I would
personally like to see is more focus on the core part, and better
"management" of the conveniences part.

By this I mean that more time is spent on stability, correctness, and
performance, and we define clear interfaces so that users can achieve the
"convenience" features without "polluting" the core. I think that there is a
lot of potential simplification of the core that can be achieved if the
project accepts this division.

The core is what really provides value, and what separates scons from other
systems. Making this part of the system as good as it can be can really
benefit everyone.

I won't go so far as to suggest immediate removal of some of the
'convenience' features.. I don't think that's appropriate. But I'd like to
start the discussion of whether this classification and focus makes sense. I
have no illusions about open source projects.. I realize that I can't force
anyone to work on something that they don't care to or that is not in their
interest. But I believe we have something to gain from a shared perspective.
A shared vision can help us to make future decisions on architecture and
features.

-----

I'm personally willing to try to spend some of my cycles to make these ideas
a reality. But I wanted to see if these ideas resonated with the crowd here
before I proceed.

Thanks,
Ken

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[cdavid]

Ken Deeter wrote:

> These are a few notes from a conversation I had with Steve in person.
>
> We've been using scons at VMware for over a year now, and we've been pretty
> happy with it. We've converted the builds of one of our largest products and
> realized significant build performance improvements, as well as a much more
> concise and maintainable build infrastructure.
>
> That being said, there are always improvements to be made, and the purpose of
> this message is to get the discussion started so that we can hopefully
> clarify and focus scons development efforts.
>
> -----
>
> The first issue I want to bring up is that of the release cycle. Currently
> scons doesn't really have much of a defined release process. Steve and I have
> chatted about it in the past, we've never really done anything to effectively
> change the status quo. From what I have seen, the main implication of this
> has been that there hasn't been much discipline exercised in deciding when
> and where to accept major destabilizing features. There is no game plan, and
> thus everything is done on an ad-hoc, case-by-case process.
>
> Because of this, companies that use scons, like VMware, have a difficult time
> contributing code back. If we're stuck on an old version, our potential
> patches to that old version are more likely to be irrelevant against the
> upstream version. We also get reverse situations, where it is difficult for a
> company to import a new version of scons because not enough work has been
> done between the introduction of major disruptive features to end up with a
> solid relatively bug-free product. It pretty much comes down to a crapshoot
> of versions, where you pick one and hope that it works.
>
> I propose that the solution is to move to time-based releases, with
> stabilization periods (and perhaps release candidates) before every release.
> I think the GNOME project has shown that this can work well. What this
> amounts to is essentially that we say up front that we will aim to release
> every 6 months. Perhaps 4 months or so into each release cycle, we can branch
> the code to create a release branch. Stabilization work can happen on this
> branch, and no major features will be added to this branch. We can cut
> regular snapshot release candidate releases from this branch, and release a
> final tarball off of the branch at the end of the six months. If needed we
> can also release critical point releases to fix major flaws found after the
> release.
>
> There are several benefits to this approach:
>  - users of a release can know how much time the code has had to stabilize
>  - patch contributors can target a 'stable' code base
>  - the algorithm for whether a new feature makes it in becomes simpler. It
> becomes a question of 'might it cause instability?' instead of 'what set of
> features do we want for the next release' (and anyone who's worked at a large
> software company knows that the latter question cannot easily be answered
> even by a large group of dedicated people)
>  - contributors will see their new code released sooner because the release
> is bound by time, not by someone's arbitrary idea of readiness.
>
> There are some downsides:
>  - branching of the code always creates a few headaches
>  - more release 'work', though hopefully this will be automatable over time.
>
> As scons continues to make inroads into large software development
> organizations, having a predictable and reliable release process becomes
> increasingly important. I would hate to see people lose interest in using the
> project because of any perceived unpredictability.

I personally think it would be good too. Let me present myself, because
I think the background and where I am coming from matters for this
discussion: I have used scons for several projects, and am about to
"push" it for a relatively big open source one (numpy/scipy). Lack of
releases is as much an hindrance for open source projects as it is for
you (and other companies who primary ship binaries); the reason is
somewhat different, I think (long time to get bug fixes in scons; scons
is a dependency of the distributed bits for open source projects), but
that does not really matter.

>
> -----
>
> The second issue I'd like to bring up is about focus of the project. The one
> thing I've seen time and again is that open source projects get pulled in
> every which direction, and begin to lose focus of their core competencies.
> With the few resources that open source projects have, spreading developer
> time over more and more features causes not only developer stress, but a
> stagnation in the progress of the final product.
>

> Another, more technical way to look at this: Everything in 'conveniences' can
> be achieved in a user's scons scripts. At VMware, we've done tools.
> Repository support could be reduced to some kind of clever builder that is
> attached to source files. BuildDir can be done by writing a library to map
> sourcefiles to targetfiles. Architecturally, these can be pretty well
> separated from the core scons 'engine' and api.
I have the exact opposite thinking about scons (what is missing is a
good set of convenience tools), but the exact same conclusion :) Scons
is kind of a strange piece of software to me: it is easier, more
manageable, more modern version of one of the most hated piece of
software in open source (auto-tools). Still, it is hard to say that it
had gained any 'market share', and I think the reasons are speed and
convenience. Nowadays, except for big projects, I would say convenience
is the main culprit.

Now, to use scons in an open source project, you need to deal with
tools, update them, etc... And this is difficult to do:
    - code-wise there are some cases where the only way it to do changes
in the source (SConf capabilities: the public api provided by scons is
way too simplistic for the vast majority of non trivial open source
projects).
    - because of the above, it is difficult to contribute. I contributed
some trivial patches to scons for convenience tools that almost any open
source software need, but it was not so easy to 'get in'.

For those reasons, a more clear separation between what you call
convenience and core, both at the code and at the participation level
would be really good. It would make easier for core developers (and you
:) ) to focus on the core, and easier for people which think scons is
almost there, but not quite, because of many basic functionalities
missing (from autoconf, libtool, etc...).

cheers,

David


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[behle]

On Jan 29, 2008 10:18 PM, Ken Deeter <ken@...> wrote:

>
> I won't go so far as to suggest immediate removal of some of the
> 'convenience' features.. I don't think that's appropriate. But I'd like to
> start the discussion of whether this classification and focus makes sense. I
> have no illusions about open source projects.. I realize that I can't force
> anyone to work on something that they don't care to or that is not in their
> interest. But I believe we have something to gain from a shared perspective.
> A shared vision can help us to make future decisions on architecture and
> features.
>

I also use SCons in a method very similar to you, I usually download
the *-local packages and strip the tools directory out of package with
only the core left as well.  Even if the source repository itself
didn't maintain the distinction between the two packages, it might be
useful to release a *-core and a *-full version of the package (sorta
like what the *-local package is now).

Even when I do have projects where I use the SCons full package, I
tend to use the built in env.Program() builders less and less because
of a number of limitations in them with cross compiling.  Even for
source code compilations, I typically use a difference set of builders
for programs and static / shared libraries than what comes in the
default tools distribution.  And there has been well documented high
profile potential adopters of SCons who have run into similar issues
that didn't feel like completely reinventing the wheel with respect to
setting up their own builders to handle more complex compilation than
what is already available.

In addition, I think there is also an informal 3rd part of the project
in the area of contributed tools on the wiki as well.  I think it
would be nice if there a version controlled repository for contributed
tools (that is easier to gain write access too than the main
repository) to be able to easily fetch all the contributed tools to
see if any of them already do what you are looking for.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Erling D. Andersen]

Hi,

>Now it's no coincidence that these two parts also map directly to how we use
>scons at VMware. We use the 'core' part, but we don't use the 'conveniences'
>part. However, I also believe that this separation likely applies for other
>large organizations that want to use scons.

We are small organization but nevertheless agree.

> * Tools: the scons builtin tools have a general approach that involves
>trying to detect toolchains installed on the host doing to the build. While
>this is exactly what build systems for open source projects need to do, it
>turns out this is _exactly the opposite_ of what build systems for
>organizations that try to ship binary software want to do. At VMware, we've
>essentially replace all of the Tools system with an inhouse version that has
>the same concept of mix and match tools, but are written to specify exact
>versions of various tools in precise locations. The toolchain is maintained
>centrally, and picking up 'your version of gcc' is exactly the last thing one
>wants a build to do.

We have also added a layer to Scons that:
 
 -  Turns autodetection off. This is precisely what we not want.
 - We tell on each build host what is available of tools (in an XML formatted file).
 - On build time we setup the appropriate tools that we want to use.

So we want  exactly the same as you.


>I won't go so far as to suggest immediate removal of some of the
>'convenience' features.. I don't think that's appropriate. But I'd like to
>start the discussion of whether this classification and focus makes sense. I
>have no illusions about open source projects.. I realize that I can't force
>anyone to work on something that they don't care to or that is not in their
>interest. But I believe we have something to gain from a shared perspective.
>A shared vision can help us to make future decisions on architecture and
>features.

Based on my 5 year experience with Scons I agree completely with your opinion(s) about Scons.

Erling





---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Benoit Belley]

Hi Brandon,

Here's an alternative solution to stripping out the Tool sub-directory of an SCons installation. You just have to include the following file in your project source tree:

$ cat site_scons/site_tools/default.py
"""Default tool for the Toxik SCons environment.

This overrides SCons' default tool.
"""
import SCons.Tool

def generate(env, **kw):
    # Make sure that the default environment can not be used to build
    # anything. We do this to make sure that people use the imported
    # environment in their SConscript instead of creating a new one.
    env['BUILDERS'] = {}

def exists(env):
    return True

That way only the tools that you explicitly include will be part of your construction environments.

Benoit

Brandon Ehle wrote:

On Jan 29, 2008 10:18 PM, Ken Deeter ken@... wrote:
  

I won't go so far as to suggest immediate removal of some of the
'convenience' features.. I don't think that's appropriate. But I'd like to
start the discussion of whether this classification and focus makes sense. I
have no illusions about open source projects.. I realize that I can't force
anyone to work on something that they don't care to or that is not in their
interest. But I believe we have something to gain from a shared perspective.
A shared vision can help us to make future decisions on architecture and
features.

    

I also use SCons in a method very similar to you, I usually download
the *-local packages and strip the tools directory out of package with
only the core left as well.  Even if the source repository itself
didn't maintain the distinction between the two packages, it might be
useful to release a *-core and a *-full version of the package (sorta
like what the *-local package is now).

Even when I do have projects where I use the SCons full package, I
tend to use the built in env.Program() builders less and less because
of a number of limitations in them with cross compiling.  Even for
source code compilations, I typically use a difference set of builders
for programs and static / shared libraries than what comes in the
default tools distribution.  And there has been well documented high
profile potential adopters of SCons who have run into similar issues
that didn't feel like completely reinventing the wheel with respect to
setting up their own builders to handle more complex compilation than
what is already available.

In addition, I think there is also an informal 3rd part of the project
in the area of contributed tools on the wiki as well.  I think it
would be nice if there a version controlled repository for contributed
tools (that is easier to gain write access too than the main
repository) to be able to easily fetch all the contributed tools to
see if any of them already do what you are looking for.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

  

--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@...

[Ken Deeter]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Benoit.

 

I think this actually isn't sufficient for our purposes. Clearing out the builders table does not give you a pristine environment. We like to define everything down to CCCOM equivalents, so we want an environment that is truly empty (no variables defined).

 

We also had several other problems with the builtin in tool mechanism. One was in terms of how a particular tool module is found. We couldn't find a way to easily tack on a set of directories to look in for tools, and have this set dynamically change based on what kind of environment we were trying to create.

 

Ken

 

From: Benoit Belley [mailto:benoit.belley@...]
Sent: Wednesday, January 30, 2008 5:25 AM
To: dev@...
Subject: Re: [scons-dev] SCons Future Directions and Thoughts

 

Hi Brandon,

Here's an alternative solution to stripping out the Tool sub-directory of an SCons installation. You just have to include the following file in your project source tree:

$ cat site_scons/site_tools/default.py
"""Default tool for the Toxik SCons environment.

This overrides SCons' default tool.
"""
import SCons.Tool

def generate(env, **kw):
    # Make sure that the default environment can not be used to build
    # anything. We do this to make sure that people use the imported
    # environment in their SConscript instead of creating a new one.
    env['BUILDERS'] = {}

def exists(env):
    return True

That way only the tools that you explicitly include will be part of your construction environments.

Benoit

Brandon Ehle wrote:

On Jan 29, 2008 10:18 PM, Ken Deeter ken@... wrote:

I won't go so far as to suggest immediate removal of some of the

'convenience' features.. I don't think that's appropriate. But I'd like to

start the discussion of whether this classification and focus makes sense. I

have no illusions about open source projects.. I realize that I can't force

anyone to work on something that they don't care to or that is not in their

interest. But I believe we have something to gain from a shared perspective.

A shared vision can help us to make future decisions on architecture and

features.

I also use SCons in a method very similar to you, I usually download

the *-local packages and strip the tools directory out of package with

only the core left as well.  Even if the source repository itself

didn't maintain the distinction between the two packages, it might be

useful to release a *-core and a *-full version of the package (sorta

like what the *-local package is now).

Even when I do have projects where I use the SCons full package, I

tend to use the built in env.Program() builders less and less because

of a number of limitations in them with cross compiling.  Even for

source code compilations, I typically use a difference set of builders

for programs and static / shared libraries than what comes in the

default tools distribution.  And there has been well documented high

profile potential adopters of SCons who have run into similar issues

that didn't feel like completely reinventing the wheel with respect to

setting up their own builders to handle more complex compilation than

what is already available.

In addition, I think there is also an informal 3rd part of the project

in the area of contributed tools on the wiki as well.  I think it

would be nice if there a version controlled repository for contributed

tools (that is easier to gain write access too than the main

repository) to be able to easily fetch all the contributed tools to

see if any of them already do what you are looking for.

---------------------------------------------------------------------

To unsubscribe, e-mail: dev-unsubscribe@...

For additional commands, e-mail: dev-help@...

--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@...

[Sohail Somani-2]

On Tue, 29 Jan 2008 22:18:10 -0800, Ken Deeter wrote:

> The first issue I want to bring up is that of the release cycle.
[snip suggestion of time-based releases]

I thought this is already being done.
 
> The second issue I'd like to bring up is about focus of the project. The
> one thing I've seen time and again is that open source projects get
> pulled in every which direction, and begin to lose focus of their core
> competencies. With the few resources that open source projects have,
> spreading developer time over more and more features causes not only
> developer stress, but a stagnation in the progress of the final product.
[snip discussion about SCons core vs conveniences like Tool and
repositories]

Having used SCons for a while now, I completely agree and saw myself
going in this direction before I pursued other opportunities. However,
the large majority of development shops (closed-source or open-source) do
not have competent full-time people dedicated to supporting builds.
Therefore, to dismiss support of these as a distraction would cut the
legs off of SCons while it is running.

Perhaps what is needed is an Enterprisey SCons.

/me wonders if there is a business model in that...

--
Sohail Somani
http://uint32t.blogspot.com


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[behle]

The other thing relevant to this is that we like to use py2exe or
equivalent to package up scons into a single executable so that people
don't need to install python, and the way the it scans for builtin
tools (and platforms too I think), does so in a way that doesn't work
with most of the package modules.


On Jan 30, 2008 9:48 AM, Ken Deeter <ken@...> wrote:

>
>
>
>
> Benoit.
>
>
>
> I think this actually isn't sufficient for our purposes. Clearing out the
> builders table does not give you a pristine environment. We like to define
> everything down to CCCOM equivalents, so we want an environment that is
> truly empty (no variables defined).
>
>
>
> We also had several other problems with the builtin in tool mechanism. One
> was in terms of how a particular tool module is found. We couldn't find a
> way to easily tack on a set of directories to look in for tools, and have
> this set dynamically change based on what kind of environment we were trying
> to create.
>
>
>
> Ken
>
>
>
>
>
> From: Benoit Belley [mailto:benoit.belley@...]
>  Sent: Wednesday, January 30, 2008 5:25 AM
>  To: dev@...
>  Subject: Re: [scons-dev] SCons Future Directions and Thoughts
>
>
>
>
>
> Hi Brandon,
>
>  Here's an alternative solution to stripping out the Tool sub-directory of
> an SCons installation. You just have to include the following file in your
> project source tree:
>
>  $ cat site_scons/site_tools/default.py
>  """Default tool for the Toxik SCons environment.
>
>  This overrides SCons' default tool.
>  """
>  import SCons.Tool
>
>  def generate(env, **kw):
>      # Make sure that the default environment can not be used to build
>      # anything. We do this to make sure that people use the imported
>      # environment in their SConscript instead of creating a new one.
>      env['BUILDERS'] = {}
>
>  def exists(env):
>      return True
>
>  That way only the tools that you explicitly include will be part of your
> construction environments.
>
>  Benoit
>
>  Brandon Ehle wrote: On Jan 29, 2008 10:18 PM, Ken Deeter <ken@...>
> wrote:
>
>
> I won't go so far as to suggest immediate removal of some of the
> 'convenience' features.. I don't think that's appropriate. But I'd like to
> start the discussion of whether this classification and focus makes sense. I
> have no illusions about open source projects.. I realize that I can't force
> anyone to work on something that they don't care to or that is not in their
> interest. But I believe we have something to gain from a shared perspective.
> A shared vision can help us to make future decisions on architecture and
> features.
>
>
>
> I also use SCons in a method very similar to you, I usually download
> the *-local packages and strip the tools directory out of package with
> only the core left as well.  Even if the source repository itself
> didn't maintain the distinction between the two packages, it might be
> useful to release a *-core and a *-full version of the package (sorta
> like what the *-local package is now).
>
> Even when I do have projects where I use the SCons full package, I
> tend to use the built in env.Program() builders less and less because
> of a number of limitations in them with cross compiling.  Even for
> source code compilations, I typically use a difference set of builders
> for programs and static / shared libraries than what comes in the
> default tools distribution.  And there has been well documented high
> profile potential adopters of SCons who have run into similar issues
> that didn't feel like completely reinventing the wheel with respect to
> setting up their own builders to handle more complex compilation than
> what is already available.
>
> In addition, I think there is also an informal 3rd part of the project
> in the area of contributed tools on the wiki as well.  I think it
> would be nice if there a version controlled repository for contributed
> tools (that is easier to gain write access too than the main
> repository) to be able to easily fetch all the contributed tools to
> see if any of them already do what you are looking for.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>
>
>
>
> --------------------------------------------------------------------- To
> unsubscribe, e-mail: dev-unsubscribe@... For additional
> commands, e-mail: dev-help@...

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Benoit Belley]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Ken,

Maybe I should given more explanation because it does clear out pretty much everything.  For example, CCCOM is not defined in the environment with my trick.

The important point is not so much that the default.py tool flushes the BUILDERS construction variable. The important point is that since SCons looks in site_scons/site_tools before looking in SCons/Tool, it is site_scons/site_tools/default.py that gets loaded instead of SCons/Tool/default.py. This bypasses the automatic loading of any other tools in the SCons/Tool directory (apart from the stuff defined SCons/Tool/__init__.py).

If you print-out the environment, you get only harmless construction variables (IMO):

$ cat SConstruct
print Environment.Dump()
$ cat site_scons/site_tools/default.py
import SCons.Tool

def generate(env, **kw):
    # Make sure that the default environment can not be used to build
    # anything. We do this to make sure that people use the imported
    # environment in their SConscript instead of creating a new one.
    env['BUILDERS'] = {}

def exists(env):
    return True

$ scons
{ 'BUILDERS': {},
  'CONFIGUREDIR': '#/.sconf_temp',
  'CONFIGURELOG': '#/config.log',
  'CPPSUFFIXES': [ '.c',
                   '.C',
                   '.cxx',
                   '.cpp',
                   '.c++',
                   '.cc',
                   '.h',
                   '.H',
                   '.hxx',
                   '.hpp',
                   '.hh',
                   '.F',
                   '.fpp',
                   '.FPP',
                   '.m',
                   '.mm',
                   '.S',
                   '.spp',
                   '.SPP'],
  'DSUFFIXES': ['.d'],
  'Dir': <SCons.Defaults.Variable_Method_Caller instance at 0x2a99a04128>,
  'Dirs': <SCons.Defaults.Variable_Method_Caller instance at 0x2a99a04170>,
  'ENV': {'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin'},
  'ESCAPE': <function escape at 0x2a99f34e60>,
  'File': <SCons.Defaults.Variable_Method_Caller instance at 0x2a99a041b8>,
  'IDLSUFFIXES': ['.idl', '.IDL'],
  'LATEXSUFFIXES': ['.tex', '.ltx', '.latex'],
  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'MAXLINELENGTH': 128072,
  'OBJPREFIX': '',
  'OBJSUFFIX': '.o',
  'PLATFORM': 'posix',
  'PROGPREFIX': '',
  'PROGSUFFIX': '',
  'PSPAWN': <function piped_env_spawn at 0x2a99f41b18>,
  'RDirs': <SCons.Defaults.Variable_Method_Caller instance at 0x2a99a04200>,
  'SCANNERS': [],
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.so',
  'SHOBJPREFIX': '$OBJPREFIX',
  'SHOBJSUFFIX': '$OBJSUFFIX',
  'SPAWN': <function spawnvpe_spawn at 0x2a99f41140>,
  'TEMPFILE': <class SCons.Platform.TempFileMunge at 0x2a99a0aa10>,
  'TEMPFILEPREFIX': '@',
  'TOOLS': ['default'],
  '_CPPDEFFLAGS': '${_defines(CPPDEFPREFIX, CPPDEFINES, CPPDEFSUFFIX, __env__)}',
  '_CPPINCFLAGS': '$( ${_concat(INCPREFIX, CPPPATH, INCSUFFIX, __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBDIRFLAGS': '$( ${_concat(LIBDIRPREFIX, LIBPATH, LIBDIRSUFFIX, __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBFLAGS': '${_concat(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX, __env__)}',
  '__RPATH': '$_RPATH',
  '_concat': <function _concat at 0x2a99a03668>,
  '_defines': <function _defines at 0x2a99a037d0>,
  '_stripixes': <function _stripixes at 0x2a99a03758>}

Benoit

Ken Deeter wrote:

Benoit.

 

I think this actually isn't sufficient for our purposes. Clearing out the builders table does not give you a pristine environment. We like to define everything down to CCCOM equivalents, so we want an environment that is truly empty (no variables defined).

 

We also had several other problems with the builtin in tool mechanism. One was in terms of how a particular tool module is found. We couldn't find a way to easily tack on a set of directories to look in for tools, and have this set dynamically change based on what kind of environment we were trying to create.

 

Ken

 

From: Benoit Belley [benoit.belley@...]
Sent: Wednesday, January 30, 2008 5:25 AM
To: dev@...
Subject: Re: [scons-dev] SCons Future Directions and Thoughts

 

Hi Brandon,

Here's an alternative solution to stripping out the Tool sub-directory of an SCons installation. You just have to include the following file in your project source tree:

$ cat site_scons/site_tools/default.py
"""Default tool for the Toxik SCons environment.

This overrides SCons' default tool.
"""
import SCons.Tool

def generate(env, **kw):
    # Make sure that the default environment can not be used to build
    # anything. We do this to make sure that people use the imported
    # environment in their SConscript instead of creating a new one.
    env['BUILDERS'] = {}

def exists(env):
    return True

That way only the tools that you explicitly include will be part of your construction environments.

Benoit

Brandon Ehle wrote:

On Jan 29, 2008 10:18 PM, Ken Deeter ken@... wrote:

I won't go so far as to suggest immediate removal of some of the

'convenience' features.. I don't think that's appropriate. But I'd like to

start the discussion of whether this classification and focus makes sense. I

have no illusions about open source projects.. I realize that I can't force

anyone to work on something that they don't care to or that is not in their

interest. But I believe we have something to gain from a shared perspective.

A shared vision can help us to make future decisions on architecture and

features.

I also use SCons in a method very similar to you, I usually download

the *-local packages and strip the tools directory out of package with

only the core left as well.  Even if the source repository itself

didn't maintain the distinction between the two packages, it might be

useful to release a *-core and a *-full version of the package (sorta

like what the *-local package is now).

Even when I do have projects where I use the SCons full package, I

tend to use the built in env.Program() builders less and less because

of a number of limitations in them with cross compiling.  Even for

source code compilations, I typically use a difference set of builders

for programs and static / shared libraries than what comes in the

default tools distribution.  And there has been well documented high

profile potential adopters of SCons who have run into similar issues

that didn't feel like completely reinventing the wheel with respect to

setting up their own builders to handle more complex compilation than

what is already available.

In addition, I think there is also an informal 3rd part of the project

in the area of contributed tools on the wiki as well.  I think it

would be nice if there a version controlled repository for contributed

tools (that is easier to gain write access too than the main

repository) to be able to easily fetch all the contributed tools to

see if any of them already do what you are looking for.

---------------------------------------------------------------------

To unsubscribe, e-mail: dev-unsubscribe@...

For additional commands, e-mail: dev-help@...

--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@...

--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@...

[Ken Deeter]

Some parts of this message have been removed. Learn more about Nabble's security policy.

 

Ah I see what you're saying. Sure that would work, but we actually don't even want some of those variables that are left ;-). Paranoia I guess.

 

Ken

 

From: Benoit Belley [mailto:benoit.belley@...]
Sent: Wednesday, January 30, 2008 11:01 AM
To: dev@...
Subject: Re: [scons-dev] SCons Future Directions and Thoughts

 

Hi Ken,

Maybe I should given more explanation because it does clear out pretty much everything.  For example, CCCOM is not defined in the environment with my trick.

The important point is not so much that the default.py tool flushes the BUILDERS construction variable. The important point is that since SCons looks in site_scons/site_tools before looking in SCons/Tool, it is site_scons/site_tools/default.py that gets loaded instead of SCons/Tool/default.py. This bypasses the automatic loading of any other tools in the SCons/Tool directory (apart from the stuff defined SCons/Tool/__init__.py).

If you print-out the environment, you get only harmless construction variables (IMO):

$ cat SConstruct
print Environment.Dump()
$ cat site_scons/site_tools/default.py
import SCons.Tool

def generate(env, **kw):
    # Make sure that the default environment can not be used to build
    # anything. We do this to make sure that people use the imported
    # environment in their SConscript instead of creating a new one.
    env['BUILDERS'] = {}

def exists(env):
    return True

$ scons
{ 'BUILDERS': {},
  'CONFIGUREDIR': '#/.sconf_temp',
  'CONFIGURELOG': '#/config.log',
  'CPPSUFFIXES': [ '.c',
                   '.C',
                   '.cxx',
                   '.cpp',
                   '.c++',
                   '.cc',
                   '.h',
                   '.H',
                   '.hxx',
                   '.hpp',
                   '.hh',
                   '.F',
                   '.fpp',
                   '.FPP',
                   '.m',
                   '.mm',
                   '.S',
                   '.spp',
                   '.SPP'],
  'DSUFFIXES': ['.d'],
  'Dir': <SCons.Defaults.Variable_Method_Caller instance at 0x2a99a04128>,
  'Dirs': <SCons.Defaults.Variable_Method_Caller instance at 0x2a99a04170>,
  'ENV': {'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin'},
  'ESCAPE': <function escape at 0x2a99f34e60>,
  'File': <SCons.Defaults.Variable_Method_Caller instance at 0x2a99a041b8>,
  'IDLSUFFIXES': ['.idl', '.IDL'],
  'LATEXSUFFIXES': ['.tex', '.ltx', '.latex'],
  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'MAXLINELENGTH': 128072,
  'OBJPREFIX': '',
  'OBJSUFFIX': '.o',
  'PLATFORM': 'posix',
  'PROGPREFIX': '',
  'PROGSUFFIX': '',
  'PSPAWN': <function piped_env_spawn at 0x2a99f41b18>,
  'RDirs': <SCons.Defaults.Variable_Method_Caller instance at 0x2a99a04200>,
  'SCANNERS': [],
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.so',
  'SHOBJPREFIX': '$OBJPREFIX',
  'SHOBJSUFFIX': '$OBJSUFFIX',
  'SPAWN': <function spawnvpe_spawn at 0x2a99f41140>,
  'TEMPFILE': <class SCons.Platform.TempFileMunge at 0x2a99a0aa10>,
  'TEMPFILEPREFIX': '@',
  'TOOLS': ['default'],
  '_CPPDEFFLAGS': '${_defines(CPPDEFPREFIX, CPPDEFINES, CPPDEFSUFFIX, __env__)}',
  '_CPPINCFLAGS': '$( ${_concat(INCPREFIX, CPPPATH, INCSUFFIX, __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBDIRFLAGS': '$( ${_concat(LIBDIRPREFIX, LIBPATH, LIBDIRSUFFIX, __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBFLAGS': '${_concat(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX, __env__)}',
  '__RPATH': '$_RPATH',
  '_concat': <function _concat at 0x2a99a03668>,
  '_defines': <function _defines at 0x2a99a037d0>,
  '_stripixes': <function _stripixes at 0x2a99a03758>}

Benoit

Ken Deeter wrote:

Benoit.

 

I think this actually isn't sufficient for our purposes. Clearing out the builders table does not give you a pristine environment. We like to define everything down to CCCOM equivalents, so we want an environment that is truly empty (no variables defined).

 

We also had several other problems with the builtin in tool mechanism. One was in terms of how a particular tool module is found. We couldn't find a way to easily tack on a set of directories to look in for tools, and have this set dynamically change based on what kind of environment we were trying to create.

 

Ken

 

From: Benoit Belley [benoit.belley@...]
Sent: Wednesday, January 30, 2008 5:25 AM
To: dev@...
Subject: Re: [scons-dev] SCons Future Directions and Thoughts

 

Hi Brandon,

Here's an alternative solution to stripping out the Tool sub-directory of an SCons installation. You just have to include the following file in your project source tree:

$ cat site_scons/site_tools/default.py
"""Default tool for the Toxik SCons environment.

This overrides SCons' default tool.
"""
import SCons.Tool

def generate(env, **kw):
    # Make sure that the default environment can not be used to build
    # anything. We do this to make sure that people use the imported
    # environment in their SConscript instead of creating a new one.
    env['BUILDERS'] = {}

def exists(env):
    return True

That way only the tools that you explicitly include will be part of your construction environments.

Benoit

Brandon Ehle wrote:

On Jan 29, 2008 10:18 PM, Ken Deeter ken@... wrote:

I won't go so far as to suggest immediate removal of some of the

'convenience' features.. I don't think that's appropriate. But I'd like to

start the discussion of whether this classification and focus makes sense. I

have no illusions about open source projects.. I realize that I can't force

anyone to work on something that they don't care to or that is not in their

interest. But I believe we have something to gain from a shared perspective.

A shared vision can help us to make future decisions on architecture and

features.

I also use SCons in a method very similar to you, I usually download

the *-local packages and strip the tools directory out of package with

only the core left as well.  Even if the source repository itself

didn't maintain the distinction between the two packages, it might be

useful to release a *-core and a *-full version of the package (sorta

like what the *-local package is now).

Even when I do have projects where I use the SCons full package, I

tend to use the built in env.Program() builders less and less because

of a number of limitations in them with cross compiling.  Even for

source code compilations, I typically use a difference set of builders

for programs and static / shared libraries than what comes in the

default tools distribution.  And there has been well documented high

profile potential adopters of SCons who have run into similar issues

that didn't feel like completely reinventing the wheel with respect to

setting up their own builders to handle more complex compilation than

what is already available.

In addition, I think there is also an informal 3rd part of the project

in the area of contributed tools on the wiki as well.  I think it

would be nice if there a version controlled repository for contributed

tools (that is easier to gain write access too than the main

repository) to be able to easily fetch all the contributed tools to

see if any of them already do what you are looking for.

---------------------------------------------------------------------

To unsubscribe, e-mail: dev-unsubscribe@...

For additional commands, e-mail: dev-help@...

--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@...

--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@... For additional commands, e-mail: dev-help@...

[William Deegan]

[... lots removed ...]
[.. Comments about 6month release cycle ...]

Perhaps it would make sense to have a similar release pattern to fedora core/red hat enterprise.

More frequent developer releases, every N gets super stabilized into an enterprise release?

[garyo]

Bill Deegan wrote:
> [... lots removed ...]
> [.. Comments about 6month release cycle ...]
>
> Perhaps it would make sense to have a similar release pattern to fedora
> core/red hat enterprise.
>
> More frequent developer releases, every N gets super stabilized into an
> enterprise release?

That's pretty much exactly what the checkpoint/minor/major schedule
we've recently started on does.  Checkpoints average a month or two,
point releases (hopefully) six monthsish, major releases, who knows.
After all we're not at 1.0 yet!

As for enterprise-grade releases (with backported fixes etc.), we'd need
more test and release infrastructure and bodies to do that.  More
release and testing help would help regularize this process.  Please
volunteer!

--
Gary Oberbrunner

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Ken Deeter]

>
> That's pretty much exactly what the checkpoint/minor/major schedule
> we've recently started on does.  Checkpoints average a month or two,
> point releases (hopefully) six monthsish, major releases, who knows.
> After all we're not at 1.0 yet!
>

A couple questions.. do we branch the code before releases?

Are there decisions along the lines of, "a new release is soon, so lets hold
off on accepting this major patch" being made?

I think what's telling is the "(hopefully)" bit. With a time-based release,
there is no "hopefully", because it's the time, not the features that define
the release. The fact that you can say "major releases, who knows" will make
any potential enterprise user very uncomfortable because you're essentially
saying that there is no predictability at all. With a time-based release, we
can say that in N months, maximum, you will see a release with some effort
put into stability and a holding off of new features.

> As for enterprise-grade releases (with backported fixes etc.), we'd need
> more test and release infrastructure and bodies to do that.  More
> release and testing help would help regularize this process.  Please
> volunteer!
>

I agree.. talking about full enterprise release is fun, but without the
manpower, it's just talk. I'd rather just see regular branches of the code,
with some stabilization work on those branches, and releases made from those
branches. That's not too much more work, and provides a clear benefit.

Ken

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[garyo]

Ken Deeter wrote:
>> That's pretty much exactly what the checkpoint/minor/major schedule
>> we've recently started on does.  Checkpoints average a month or two,
>> point releases (hopefully) six monthsish, major releases, who knows.
>> After all we're not at 1.0 yet!
>>
>
> A couple questions.. do we branch the code before releases?

Check the dev mailing list archives for May 2007; the subject of the
thread was "RELEASE:  branching strategy" (especially Steven's messages
which are authoritative.)

So far we have been following that model.  It probably should have
gotten written up into the wiki (maybe it did and I don't know where).

--
Gary Oberbrunner

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Ken Deeter]

Gary,

I went through the old messages.. seems like a bunch of schemes were
proposed, but I couldn't seem to find any final consensus. I think the last
one Steven proposed was the 'golden thread' model.

In terms of the ideas that were proposed, I think the optimal scheme is the
'branch before release' plus the 'golden-thread' model + time based releases.

I'll illustrate what I think is wrong with the pure golden-thread model by
example. At vmware, we would have probably liked to see a release done before
the interactive mode changes went in (mostly because there hasn't been a
stable release in a while, and there's a lot of stuff we had Steve work on
that's in the trunk but hasn't been released, mainly the sig refactoring). So
while there was a quiet period between when sig refactor went in, and when
interactive mode went in, there was no predictability or transparency. Even
if we had been pretty confident about the state of the tree before
interactive mode, now that that's in, it in effect 'resets' our confidence.
While scons does have a test suite, in our experience, we still find problems
with each release.

I think the fundamental observation is that while you can call the trunk
'golden', it is not provably 'golden' until a bunch of people have used it in
their business critical scenarios and have given the thumbs up. While scons
does have a test suite, in practice, we have found small problems in each
release that we have had to hack around. Also, it's hard to really make any
claims about performance until you've measured it against a huge source tree
like ours. Really saying that the code is stable simply takes time, and it
requires the ability to limit new checkins to those that are low-risk and fix
specific small issues. If you use the trunk as the place to receive these
checkins, then now you've made it so that features completed on other
branches now have to "wait" until a release made from the trunk.

Also, because the project didn't have a "wait" period (without a time-based
cycle, these wait periods can become infinite), there were instances where we
would have liked to contribute patches back up, but because the trunk had
moved forward, they were meaningless. And even if they did apply, the trunk
would have already incorporated some new feature that made it impossible for
us to update to the new version just to get our new fix.

The part of the golden-thread model about doing major feature work on
separate branches is fine. But as long as that work is done on separate
branches, they will not get nearly as much testing as they need to. Most
users will only use stable branches or  releases off the trunk. That means
those are the only branches that will see "real" testing. So it is somewhat
of a fallacy to think that the trunk is always releasable in this model. If a
feature lands on the trunk from a branch that hasn't seen real testing, then
I'll put my money on the fact that something will be broken.

So let me propose in detail, what I mean by a time-based, golden-thread +
branch before release strategy.

-- big features are developed on branches, as in the golden-thread model.
they are merged to the trunk when they are 'ready.' and by ready, we probably
mean 'not obviously broken'

-- periodically, on a regular time basis, branches are made from the trunk,
which become release branches. When to make this branch is purely a matter of
time. It should not "wait" for features to land on the trunk. The only
exception I can think of is if the trunk has not changed at all since the
last branch, but that's sort of a pathological case, and we've probably got
other problems if nothing's changed in 6 months ;-)

-- We can make regular snapshots off the trunk as we have been doing, for
people who want to follow the trunk. If people working on the trunk want to
have regular stabilization periods before these snapshots, that's up to them.

-- Once a release branch has been created, we pick a version number for it,
and start making releases. We can start with betas, and then rc's, etc. But
personally, I don't care about the naming. I would be fine with a 'build
number' a la the windows releases. For example scons-0.98-12345, where the
number can be the revision number of the latest change made on the release
branch. In fact, this makes it really easy, since it relieves us of the duty
of trying to decide 'what is stable enough' or what is 'tested enough'. All
we're saying is "here's the newest release from are stabilized branch", and
it's up to the user to decide if it's suitable. As long as we're disciplined
about only introducing low-risk fixes, the higher the "build version", the
more confident a user can be about it. Or we could just make very frequent
releases and call them 0.98.0, 0.98.1, 0.98.2, and so on.

<tangent>
The real point is that, release numbers in open source don't mean what they
do in the closed source world. They only serve to establish temporal and
branching relationships between different releases. In the commercial world
'1.0' could mean something [which is still arbitrary: for companies it means,
"we've got most of our features, and we need to make some money off of this
soon"] but as can be seen from the fact that many companies are using scons
at version 0.96, not being 1.0 has not stopped anyone. Furthermore, I don't
think the way we've defined 1.0 is attainable. If nobody has a clear vision
of even 1.0 is, then how is it ever going to happen? I personally think its
better to just consistently make releases that people can use. The more
releases you make, the more likely they'll be 'good enough' for an increasing
number of people.
</tangent>

-- The merging of features from feature branches to the trunk can be timed
appropriately with the release period. That is, if we agree to make a release
branch 4 months into every 6 month period, it's probably not a good idea to
merge a major feature on the last day of the 4th month.

-- We can take patches on the release branches to fix specific small issues.
Crashes, incorrectness, performance problems, etc. If these fixes are useful
to the trunk, they can be cherry picked and merged to the trunk. Note that if
a feature is broken on a release branch due to a design flaw, we do _not_
re-design the feature on the release branch. We simply release and say 'this
code exists, but it has design flaws.. use trunk, or wait until the next
release branch'.


Thoughts, Comments?

Ken

> Ken Deeter wrote:
> > > That's pretty much exactly what the checkpoint/minor/major schedule
> > > we've recently started on does.  Checkpoints average a month or two,
> > > point releases (hopefully) six monthsish, major releases, who knows.
> > > After all we're not at 1.0 yet!
> > >
> >
> > A couple questions.. do we branch the code before releases?
>
> Check the dev mailing list archives for May 2007; the subject of the
> thread was "RELEASE:  branching strategy" (especially Steven's messages
> which are authoritative.)
>
> So far we have been following that model.  It probably should have
> gotten written up into the wiki (maybe it did and I don't know where).
>
> --
> Gary Oberbrunner
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[gregnoel]

OK, the Superbowl is over, the out-of-town guests have departed (I  
love them, but they take up lots of time), I'm getting over the flu  
they left behind, and I'm finally ready to get to this topic.

On Jan 29, 2008, at 10:18 PM, Ken Deeter started a thread to discuss  
aspects of what SCons should be doing in the future.  There was some  
interesting discussion, but unfortunately, I think the thread was  
trying to juggle too many topics.

As I see it, the thread had at least these topics:
         -  Release cycle
         -  Configuring tools (compilers, utilities, etc.)
         -  Aggregating fetches from (remote) repositories
         -  Variant build directories
         -  Performance
         -  Core v. "conveniences" and packaging
         -  Formalizing contributed tools
Rather than try to respond to each of them in this thread, thereby  
diffusing the thread too much, I'm going to start a thread for each  
of the topics as a follow-up to this one.  I'll put "FUTURE" in each  
of the subjects so that they are readily identified.

I'll get the first couple of responses out this evening; with luck,  
I'll have all of them out by the end of the weekend.

On a personal note, I'd appreciate anyone who responds to those  
threads to cut the quoted material down to the context needed to  
understand your reply.  After all, if you're not willing to spend  
some effort to make your reply as clear as possible, why should I go  
to the trouble to figure out what you meant?

Hope this helps,
-- Greg Noel, retired UNIX guru

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[gregnoel]

On Jan 29, 2008, at 10:18 PM, Ken Deeter took exception to the lack  
of a formal model for managing releases.  He expressed the worry that  
without such a scheme, production users wouldn't be able to upgrade  
reliably and safely.   After reading the discussion from last April,  
he proposed an alternative that is basically a 'golden' trunk but  
with branches for releases; releases are triggered purely on time.  
(This is simplified, but if I've misstated the basic premise, please  
correct me.)

First, it's not as bad as you seem to fear.  Extensive testing means  
that there are almost no regressions that can cause production code  
to break.  Backward compatibility is a mania with Steven; I've never  
seen a project that did regression testing to this extent.  A  
checkpoint release is completely stable with regard to the features  
of the prior full release.

Next, as Gary said, the strategy discussed last April seems to cover  
your original points.  However, instead of a single golden trunk,  
there are three trunks, each of which is golden for a different  
function.

I could swear that there was a wiki page describing this process, but  
I can't find it (the text-based search seems to be broken, so it may  
be there, but under some name that's not obvious to me).  This should  
be remedied; I'll see about creating a suitable page.  Enhancements  
and fixes move from the initial trunk into checkpoints in the second  
trunk and then into larger releases in the third trunk.

The intent of the scheme is to give changes however much time they  
need to mature while still providing machinery for regular, time-
based releases.  I suspect it's closer to what you want than you  
realize.

I see two major problems with this model: one major and one minor.  
The minor problem is that it takes more time to manage the movement  
from trunk to trunk.  This needs to be solved by the application of  
some bodies other than Steven to keep track of the patchsets and  
perform the actual moves.

However, the major problem is that I don't think Steven has  
completely bought in to the proposed scheme.  He's still using two  
trunks rather than three and preparing checkpoint releases off the  
main trunk instead of having a trunk dedicated to the purpose.  I  
don't blame him for this, since the community support that we were  
trying to rally last April has never materialized.

Hope this helps,
-- Greg Noel, retired UNIX guru

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[gregnoel]

On Jan 29, 2008, at 10:18 PM, Ken Deeter pointed out some flaws in  
the way SCons choses the default set of tools when an Environment is  
set up.

This has been discussed in many guises, the most-recent was only a  
week ago (see the thread "Retrieve tools info in SConstruct" started  
by David Cournapeau).  It's pretty clear that this is an area where  
SCons is lacking and could use improvement.

You have one end of it: you want to separate the tools from the core  
so that they are easier to set up.  Instead, I argue that there needs  
to be a single configuration scheme that has at a minimum these  
seventeen features:

         - Be backward compatible
         - Determine if the tool is present
         - Be backward compatible
         - Select one tool if multiple are present
         - Be backward compatible
         - Do cross-complilation
         - Be backward compatible
         - Accept configuration from "external" information
         - Be backward compatible
         - Set the default tools (if not overridden by Environment)
         - Be backward compatible
         - Autodetect tools if no manual selection
         - Be backward compatible
         - Be interrogatable for what was selected
         - Be backward compatible
         - Retain configuration across invocations
         - Be backward compatible

Now, some of those may seem to be a tad redundant, but that's where  
the problem lies.  If we had license to design a completely new API,  
then life would be simpler, but we don't, so we have to design  
something that will work unchanged for existing SConscripts, and this  
complicates things.

I don't think there's a simple solution.  I think Configure tests  
need to be extended to test for tools and that the set of tools  
provided by the 'default' tool needs to be exposed for modification.  
And on top of that, there needs to be some way to probe what was set  
up.  For example, there's already a secret flag that tells if  
relocatable and static binaries are the same; this type of  
information should be gathered into a single mechanism.  (For  
example, the discussion last week was whether the compiler accepted  
GNU-style flags or not.)

I think I could write a specification for this, but I haven't thought  
much about exactly what the API would look like, so I'm open to any  
suggestions you (or anybody) may have.

Hope this helps,
-- Greg Noel, retired UNIX guru

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[cdavid]

Greg Noel wrote:
>
> I think I could write a specification for this, but I haven't thought
> much about exactly what the API would look like, so I'm open to any
> suggestions you (or anybody) may have.
That's something waf does pretty well in my experience. So I think
looking at how waf does it may be a good start.

cheers,

David

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Thomas Nagy]

> I think I could write a specification for this, but I haven't thought
> much about exactly what the API would look like, so I'm open to any
> suggestions you (or anybody) may have.

The model is quite simple actually:
* input variables (operating system, etc)
* a state (the environment)
* a set of rules of moderate size, where rules are added/removed and
do not change too much (if win32 then add '.exe')

As a conclusion, an expert system looks like the best choice.

Thomas

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...