SCons Future Directions and Thoughts

[Ken Deeter]

Sorry, I think this discussion is useful, but I'd like to point out that this
is somewhat deviating from the initial point I was trying to make.

I think we can spend *forever* talking about what a good model for tools
would be. But to me, this is entirely irrelevant.

What I was trying to say is that the that the tools in their current form
provides is entirely supplementary to the core value of scons as a build
system.

I don't think we'll ever be able to define a model for how tools should look,
or how command lines should be generically specified, that will satisfy
everyone. It's just not possible. After I wrote my original email, I got a
lot of feedback (some through the list, some outside the list) from people
who use the tools exactly in the way that we at VMware do. That is, we don't
use them.

We wrote our own generic command lines, and our own module loading mechanism.
The part of scons we reuse is the machinery to convert the environment
variables into actual command lines, but we don't use any of the built in
variable schema at all. We write our own scanners, and our own builders.

If you think about it, it's exactly the same with make. make does not dictate
anything about how your command line should look. The user is free to
completely specify it, because it many situations, that's exactly what you
want. Make does provide some simple built in rules, but they only work in the
simplest of cases.

autoconf and automake came along later, to address a specific need. That
build environments varied widely when it came to building open source code,
so a separate configuration mechanism was needed.  

It is of my opinion that scons should follow this model. The configuration
mechanism should be a separate project, that builds on top of very minimal
standard interfaces that scons provides. There could be one project to keep
going with the tools we have now. There could be another that tries to
implement a new model that may work better for others. It would also let
companies like ours, share the same basic low level interface, but completely
roll our own mechanism that is suited to our needs. Maybe someone will come
along and write a 'autoscons' that can detect local toolchains and spit out a
bunch of tool files.

My original email was lamenting the fact that sometimes, issues with the
tools was holding back improvements in the core system from being released. I
think if the tools and core scons have more independent paths, then we could
end up with a situation that is much more beneficial to both open source
projects and closed source commercial users.

Ken

> On Feb 11, 2008, at 1:18 AM, Thomas Nagy writes that any variable can
> be used for any function and mean whatever one wants.
>
> I'm sorry; that's not very useful.  It's not a model, it's magic.  If
> you can't identify a dozen or more sample variables and describe
> exactly how you anticipate they will be used or created, then you're
> simply not communicating what you see.
>
> The engine sets a few reserved variables?  OK, what?  When?  How?
> This is a _minimum_ you should be giving in a model.
>
> The user can insert some variables?  OK, what?  When?  How?  You need
> to provide at least a few representative samples to give the flavor.
>
> The user can specify rules, and I'd imagine that some are provided by
> the engine as well.  OK, what?  When?  How?  I'd expect some sort of
> representative samples to give an idea of how the transformations
> take place.
>
> I believe you know what these should look like, but so far you
> haven't communicated it to us (or to me, at least).  I know it's
> _hard_ to take a concept in your head and describe it in words, but
> until you can do that, I guarantee that you don't understand it as
> well as you think you do.
>
> It's even harder for you, since your native language isn't English.
> If it works better for you, write the description first in German,
> then translate it.  My German skills are incredibly rusty, but if
> there's a rough translation, I can work with you to polish it.
>
> Hope this helps,
> -- Greg Noel, retired UNIX guru
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Ken Deeter]

 
> To start, "convenience" is a bad name.  What's convenience to one
> person is core to another.  You need a better word that implies a set
> of add-on features without implying an "extension" (which is
> something only a few would use).  I looked in my thesaurus but I
> didn't find any candidates, so I'm afraid I can't offer any suggestions.
>

Heh, yea I didn't mean for the 'convenience' term to stick. It was just a
place holder.
 

> Proceeding to the issue, I'm sympathetic with the motivation.
> Without constraints, SCons will simply grow without limit.  It's
> already doubled in size (based on the *-local packaging) since I
> started using it, and the increase is almost all in features I never
> use.  And I can only see it getting larger.  Some way of cutting the
> feature set down to just what one needs would be a blessing.
>
> On the other hand, I don't see any way of separating SCons into just
> two pieces that doesn't gore someone's ox.  Even if you divide it
> into a core piece and some number of conveniences, I don't see how to
> divide the conveniences into non-overlapping sets.  If you're serious
> about this, you need to come up with a more precise separation.
>

I saw Steve mentioned this too, but how about splitting tools into a separate
package? I agree I don't have too many concrete suggestions here as well, but
I was more exploring whether such future suggestions would be met with a
positive response.

> If it's any help, one of the motivations of issue 1526 is to split
> the processing into independent stages, so that one could replace the
> parser that reads SConscripts with a different front-end that, say,
> read Jam files and created a DAG.  The DAG walker could still be used
> to run the DAG and execute the commands.  In that case, one might
> wish to package the Jam front-end and not include the SConscript
> front-end.  In any event, issue 1526 can be considered a first step
> in dividing SCons into pieces.
>

Why? Why can't you just have SConstruct that uses python to call into some
other mechanism that creates the dependency tree. I think part of the point
of scons is that your scripts are written in a programming language, so
you're free to do whatever you want. If someone wants to write a layer on top
of scons that knows how to read jam-scripts, that's fine. But again, this is
not the core point of scons, and I think trying to make things this generic
is only adding complexity.

In general, I think there are a lot of features that people try to bring into
core, that can be just as well accomplished by a little bit of code in the
user's scripts. We should work hard to keep these features from getting into
the core so that the core stays cleaner and easier to maintain. If there is
some feature that a bunch of people want, we can maybe set something up so
that it's easier for them to share their work, but in many cases it still
doesn't justify getting into the core scons code.

I think what I'm trying to espouse a little is that we follow more of the
unix approach of having more small tools that are combinable. I think of
scons as a runtime environment for scripts with an api to specify a DAG. It
provides the basic primitives of an Environment, Actions, Builders, and
Nodes, and everything on top of that is just as easily done in a user's
script as it is done in the core, the latter being preferable because it
makes the core stuff easier to maintain.

> So, bottom line, if you have a more detailed proposal for separating
> SCons into pieces, with a way of creating the appropriate packages,
> I'd be happy to tear it apart, er, critique it.  But, honestly, I
> think there are better places to be using cycles, at least for now.
>

Ok, well I'll try to come up with more concrete proposals. Separating the
tools out is probably the most crystalized suggestion I have for now. While I
feel like I could do without repository support and builddir(), those are
probably a bit trickier.

Ken

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Thomas Nagy]

> It's even harder for you, since your native language isn't English.
> If it works better for you, write the description first in German,
> then translate it.  My German skills are incredibly rusty, but if
> there's a rough translation, I can work with you to polish it.

You seem to be rushing on the code and the examples instead of
spending enough efforts on the analysis.

I have provided a formal model corresponding to your constraints
(evolution in the time, backward compatibility, etc), now it is up to
you to prove it wrong in finding an example for rejecting the model,
for extending it, or demonstrating how to simplify it.

Even with the platform and the CXXFLAGS variables as examples the
description seems to remain unclear to you. Have you spent more than 5
minutes trying to understand why this suggestion was made ? Have you
made the effort of looking at expert systems definitions and examples
? Have you actually tried to find a scenario contradicting the model
given ?

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[gregnoel]

On Feb 11, 2008, at 11:21 AM, Ken Deeter wrote a very thoughtful,  
albeit lengthly, message.  I'm going to summarize ruthlessly, but  
before I get to the point-by-point issues, I have two comments.

The first seems to be a matter of perspective.  You apparently assume  
that there's basically one audience for SCons: a commercial company  
that's using it for production (or within fuzz of this  
categorization).  I don't agree.  I see at least three audiences: the  
SCons developer, the friendly user, and the vast unwashed.  Each of  
the branches I propose caters to one audience.  A developer can  
subscribe to the bleeding edge, the friendly user can try out  
checkpoints, and everyone else waits for the formal releases.  Right  
now there's not a lot of distinction between the last two categories,  
but I would hope as time goes on that the majority of users wouldn't  
even be subscribed to scons-users.

The second is the the SCons release process is still missing one key  
attribute: an active release team.  I see it as their responsibility  
to rule on whether a feature is mature enough and to set the dates  
for releases.  Right now Steven does this, and I don't believe it  
should be his sole responsibility.  The problem is that nobody has  
been able to kick-start the team and get it to start making the  
decisions, but when that happens, I would hope that many of your  
concerns will evaporate.  (Maybe the wiki page should be edited to  
emphasize the role of the release team for these decisions?)

> ... the more branches you have, the more you spread out real world  
> testing. ... it is difficult (and I would argue unrealistic) to  
> expect people or companies to spend a lot of time testing [earlier]  
> branches.

But then, I don't expect the vast unwashed to test the alpha or beta  
branches.  I do expect that there are smaller communities that are  
interested in using (and testing) those branches.

> ... What is 'stable enough' to allow a merge from the development  
> branch to the stable branch? What is stable enough to move from  
> stable checkpoint branch to product release branch? ...

Whatever is the judgement of the release team.  I would expect  
"small" features to be promoted quickly and "large" features to be  
promoted more slowly; I would imagine that most features would be  
"small" and get into a release quickly.  On the other hand, the big  
signature refactoring (BSR) took more than a year from the time that  
it first appeared until it got into a release.  I would like to  
formalize that ability so a potentially destabilizing feature like  
BSR could get more exposure before committing it to production.

> ... Major releases are done on a time basis, by branching core into  
> a release branch. There is *no* other criteria ...

The problem with this is that if you decide to make a time-based  
release at noon and someone checks in an ill-conceived patch at  
11:59, even if it passes the regression tests there's no chance to  
revert it.  SCons takes backward compatibility seriously: that  
feature, warts and all, would live for a long time.  I've seen this  
happen with time-based releases, so I think the decoupling is worth  
the effort.

You can try to get around this by instituting draconian rules, such  
as freezing the branch before making the fork, requiring reviews,  
etc., but I've _never_ known them to work unless there was an actual  
institution behind the project that could enforce sanctions.  No  
matter how often you warn people, someone will make a last-minute  
change that will be regretted.

If the responsibility for promoting features lies with a release  
team, they can choose which features will go in a given release.  
They set the date for the release and then pick the features that are  
ready.  (The major exception would be release candidates on the  
checkpoint branch: only bug fixes would be accepted.)  This provides  
the same decoupling as code freezes, timing features, extensive  
reviews, and so forth, but without the draconian restrictions.

> ...  We explicitly state that a .0 release has seen very little  
> real world testing. As people test the release, we take fixes on  
> the release branch. ... We release .1, .2, .3 releases on a  
> frequent basis. ...

I've used products that had this scheme.  I _never_ implemented the .
0 release; I'd let the suckers, er, early adopters take the arrows.  
And I'm not alone in this policy.  In reality, it's no different than  
a checkpoint, so why not call it that?

And I've noticed that if a product gets a reputation for being buggy,  
and requires lots of releases to get it right, people won't trust it  
as much as a product that is mostly bug-free.  I'd rather get the  
major bugs out in checkpoints before someone depends on them in a  
production release and they become features.

> 4.  Major features are developed on branches off of core. ...

We agree here, but I'd move the feature into the alpha branch  
sooner.  In essence, I'd trade the hassle of keeping all the  
development branches in sync with the core for the hassle of  
promoting features individually.

On Feb 11, 2008, at 11:33 AM, Ken Deeter wrote:

> ... I think it is *impossible* to avoid pushing out buggy releases.  
> Every release will have bugs. Period. ...

Let me restate this: I believe it is impossible to avoid pushing out  
releases with bugs, but it's possible (or at least desirable) to  
avoid pushing out buggy releases.  The difference is huge.

> I'm saying that it's better to get something out, even if it has  
> bugs, and effectively crowdsource the testing work. ...

(Crowdsource, huh? {;-} Interesting expression...)

Maybe you're simply willing to have a release with more bugs than I  
am, but I know that if I saw a project that released .0, .1, .2, and .
3 in a couple of months, I wouldn't trust it, and I'd spend a lot  
more time testing it than I would a project that rarely released a .1.

Hope this helps,
-- Greg Noel, retired UNIX guru

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Thomas Nagy]

> I don't think we'll ever be able to define a model for how tools should look,

Why do you think so ? Just because you have not seen such a model
(yet), it does not mean that someone will not invent it or that it is
not worth trying.

> or how command lines should be generically specified, that will satisfy
> everyone. It's just not possible.

It is possible to make the command-lines modifiable easily (but not in
Scons for backward-compatibility issues).

> we don't use any of the built in
> variable schema at all. We write our own scanners, and our own builders.

The same (abstract) model which was invented a long time ago is used,
this means it was not too bad after all.

> My original email was lamenting the fact that sometimes, issues with the
> tools was holding back improvements in the core system from being released. I
> think if the tools and core scons have more independent paths, then we could
> end up with a situation that is much more beneficial to both open source
> projects and closed source commercial users.

Creating new apis is also a way to ensure further separation between
the core and the extensions (project management is another one).

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[gregnoel]

On Feb 11, 2008, at 12:17 AM, Steven Knight wrote:

> This sounds like it would be worthwhile to explore integrating the  
> Node refactoring you did separately from the TaskmasterNG work.  Is  
> that feasible?

Um, probably not, but I'm not sure.  TaskmasterNG creates a dual of  
the DAG containing only "builder" objects (i.e., actions), walking  
the DAG as it's creating the dual.  That collapses a lot of the  
effort required to do the walk, since the dual is used for  
backtracking.  I'd think the idea could be used, but not the code  
directly.  It would be a lot of work.

> Aha.  Maybe that's the rub.  I have a blind spot about thinking of  
> anything that saves the DAG structure in the same paragraph as the  
> phrase "simplest case."

It's the case that's simple, not the implementation.  Splitting it  
into two parts guarantees that nothing is accidently passed through  
memory.

> Yeah, that could make it work.  I keep thinking about the big  
> picture and all the corner cases, when the more fruitful approach  
> here would be more like XP, start by making the simplest possible  
> "Hello, world!" project work and build from there.

Yes, start simple and iterate until it handles an arbitrary project.

> I think a Node refactoring like what it sounds like you've started  
> with TaskmasterNG would be a good step towards simplifying [nodes].

It removes all the backtracking clutter, which doesn't have to be  
saved or restored with the DAG, so that would help.  Maybe what I  
should do is get an SVN account and put what I've done in a branch.

>> ... issue 1526 is number two in the voting sweepstakes ...
> True.  Half those cast votes are yours, though...  :_)

Yeah, because I consider it important.  But the other half of the  
votes _aren't_ mine, so I'm not the only one who considers it valuable.

Hope this helps,
-- Greg Noel, retired UNIX guru

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Benoit Belley]

Hi Everyone,

We have done a bit of profiling at Autodesk regarding the performance of
our SCons builds. I'd like to share some of my results with you. We are
using SCons to build a moderate size commercial application. I hope that
it is representative of what others are experiencing.

Here are the results that the Python profiler gives me on a null Build,
i.e. a build where all the targets are up-to-date. The dependency DAG
for this run has around 1100 nodes.  To put things in perspective, a
full build of our product requires the evaluation of around 19000 nodes
for a single build variant.

   Build Options:  -j32 --implicit-cache --max-drift=5
                            CacheDir('somepath')
                            SourceCode('.', None)
                            Decider( 'MD5-timestamp' )
                         

Mon Feb 11 14:43:07 2008    /var/tmp/profile

         1767271 function calls (1668632 primitive calls) in 14.002 CPU
seconds

   Ordered by: cumulative time
   List reduced from 1346 to 60 due to restriction <60>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   14.002   14.002 profile:0(<function _main
at 0x2a99c91cf8>)
        1    0.000    0.000   14.002   14.002
/usr/lib/scons/SCons/Script/Main.py:749(_main)
        2    0.000    0.000   10.323    5.162
/usr/lib/scons/SCons/Job.py:89(run)
        1    0.000    0.000   10.266   10.266
/usr/lib/scons/SCons/Script/Main.py:990(_build_targets)
        1    0.042    0.042    9.515    9.515
/usr/lib/scons/SCons/Job.py:353(start)
     1113    0.029    0.000    8.312    0.007
/usr/lib/scons/SCons/Taskmaster.py:652(next_task)
     1113    0.125    0.000    7.141    0.006
/usr/lib/scons/SCons/Taskmaster.py:481(_find_next_ready_node)
1941/1674    0.026    0.000    5.688    0.003
/usr/lib/scons/SCons/Node/__init__.py:932(children)
26869/19892    0.514    0.000    5.650    0.000 :0(map)
1902/1635    0.021    0.000    5.648    0.003
/usr/lib/scons/SCons/Node/__init__.py:599(scan)
      536    0.006    0.000    4.896    0.009
/usr/lib/scons/SCons/Node/FS.py:2732(is_up_to_date)
      536    0.073    0.000    4.849    0.009
/usr/lib/scons/SCons/Node/__init__.py:978(changed)
8985/4728    0.115    0.000    4.084    0.001
/usr/lib/scons/SCons/Environment.py:419(subst)
8985/4728    0.147    0.000    3.962    0.001
/usr/lib/scons/SCons/Subst.py:363(scons_subst)
21190/5930    0.160    0.000    3.762    0.001 :0(sub)
19722/1655    0.313    0.000    3.752    0.002
/usr/lib/scons/SCons/Subst.py:473(substitute)
      4/1    0.019    0.005    3.731    3.731
/usr/lib/scons/SCons/Script/SConscript.py:163(_SConscript)
        1    0.001    0.001    3.727    3.727
/vobs/toxik/src/SConstruct:3(<module>)
      808    0.009    0.000    3.706    0.005
/usr/lib/scons/SCons/Executor.py:188(get_contents)
7243/1697    0.123    0.000    3.704    0.002
/usr/lib/scons/SCons/Subst.py:482(sub_match)
      270    0.001    0.000    3.690    0.014
/usr/lib/scons/SCons/Executor.py:198(<lambda>)
6122/1948    0.049    0.000    3.676    0.002 :0(apply)
11057/1698    0.343    0.000    3.616    0.002
/usr/lib/scons/SCons/Subst.py:390(expand)
      248    0.003    0.000    3.592    0.014
/usr/lib/scons/SCons/Action.py:717(get_contents)
      254    0.005    0.000    3.524    0.014
/usr/lib/scons/SCons/Action.py:648(get_contents)
      249    0.004    0.000    3.473    0.014
/usr/lib/scons/SCons/Action.py:777(get_contents)
    57/16    0.000    0.000    3.096    0.193
/usr/lib/scons/SCons/Script/SConscript.py:593(__call__)
      3/1    0.001    0.000    3.089    3.089
/usr/lib/scons/SCons/Script/SConscript.py:510(SConscript)
        1    0.000    0.000    3.086    3.086
/vobs/toxik/src/SConscript:3(<module>)
        1    0.000    0.000    3.074    3.074
/vobs/toxik/src/base/SConscript:3(<module>)
  667/311    0.005    0.000    2.860    0.009
/usr/lib/scons/SCons/Environment.py:176(__call__)
  330/306    0.014    0.000    2.830    0.009
/vobs/toxik/src/site_scons/site_tools/tx_wrappers.py:323(wrapper)
      360    0.002    0.000    2.437    0.007
/usr/lib/scons/SCons/Builder.py:586(__call__)
  676/360    0.051    0.000    2.354    0.007
/usr/lib/scons/SCons/Builder.py:545(_execute)
      333    0.005    0.000    2.343    0.007
/usr/lib/scons/SCons/Environment.py:206(__call__)
11549/5343    0.149    0.000    1.573    0.000
/usr/lib/scons/SCons/Subst.py:451(func)
      642    0.048    0.000    1.559    0.002
/usr/lib/scons/SCons/Builder.py:482(_create_nodes)
    16696    0.151    0.000    1.089    0.000
/usr/lib/scons/SCons/Util.py:132(to_String_for_signature)
     1102    0.009    0.000    1.047    0.001
/usr/lib/scons/SCons/Script/Main.py:293(make_ready)
     1104    0.020    0.000    1.039    0.001
/usr/lib/scons/SCons/Taskmaster.py:287(make_ready_current)
       46    0.000    0.000    0.969    0.021
/vobs/toxik/src/site_scons/site_tools/tx_wrappers.py:516(TxSharedObject)
29312/17763    0.280    0.000    0.966    0.000
/usr/lib/scons/SCons/Util.py:140(to_String_for_subst)
13313/10864    0.208    0.000    0.913    0.000
/usr/lib/scons/SCons/Node/FS.py:1964(_lookup_abs)
     3685    0.120    0.000    0.897    0.000
/usr/lib/scons/SCons/Node/FS.py:1087(_lookup)
1653/1133    0.062    0.000    0.887    0.001 :0(eval)
      267    0.006    0.000    0.868    0.003
/usr/lib/scons/SCons/Node/FS.py:2390(get_stored_implicit)
     3096    0.091    0.000    0.865    0.000
/usr/lib/scons/SCons/Environment.py:367(arg2nodes)
1653/1133    0.014    0.000    0.840    0.001 <string>:1(<module>)
    13842    0.154    0.000    0.836    0.000
/usr/lib/scons/SCons/Node/FS.py:2722(decide_source)
        1    0.000    0.000    0.751    0.751
/usr/lib/scons/SCons/Script/Main.py:1157(jobs_postfunc)
        1    0.000    0.000    0.751    0.751
/usr/lib/scons/SCons/SConsign.py:99(write)
     8200    0.217    0.000    0.734    0.000 :0(filter)
       68    0.011    0.000    0.707    0.010
/usr/lib/scons/SCons/SConsign.py:229(write)
   195093    0.680    0.000    0.680    0.000 :0(isinstance)
    14868    0.102    0.000    0.653    0.000
/usr/lib/scons/SCons/Environment.py:1201(_changed_timestamp_then_content)
      123    0.000    0.000    0.651    0.005
/vobs/toxik/src/site_scons/site_tools/tx_wrappers.py:989(TxUnitTest)
      123    0.010    0.000    0.651    0.005
/vobs/toxik/src/site_scons/site_tools/tx_wrappers.py:926(MyUnitTest)
      267    0.105    0.000    0.627    0.002
/usr/lib/scons/SCons/Node/FS.py:2090(prepare_dependencies)
     1103    0.020    0.000    0.588    0.001
/usr/lib/scons/SCons/Node/FS.py:2544(make_ready)
    10244    0.057    0.000    0.585    0.000
/usr/lib/scons/SCons/Node/FS.py:565(__str__)


Already, we can make a few observations:

a) 3/4 of the time is spent traversing the DAG (jobs.run()-jobs_postfunc)

   Reading SConscript:   3.731 sec.     (26%)
   Building:             9.572 sec.     (68%)
   Writting sconssign:   0.751 sec.     (5.4%)
   Total:               14.002 sec.


b) Time Reading of the SConscript is dominated by the builder invocation
(Builder.py:__call__):

   Builder invocation:   2.437 sec.  (65%)
   Reading SConscript:   3.731 sec.

   * It is still mystery to me why Builder invocations cost so much...


c) A large part of the building time is actually spent is
Subst.py(scons_subst):


   Substs:   3.962   (41%)
   Building: 9.572

   * We had significant speed-up by reducing the amount of string
substitution that was necessary to build each target. For example,
adding the following lines at the end of our SConstruct file gave an
overall 5% speed-up of null builds:

            # Baking these provides a 5% speed-up of null builds.
            for p in ['CPPPATH', 'LIBPATH', 'RPATH']:
                if env.has_key(p):
                    env[p] = [env.Dir(x) for x in env[p]]
 
   * It seems to me that it should be possible to implement a
substitution cache for a given Environment to automate this. Of course,
one would need to be careful with the handling of target keywords
overriding construction variables.

d)  Determining whether a given file changed on disk amounts only for a
small part of the Building phase (changed_timestamp_then_content())
 
   MD5-timestamp: 0.836 (8.7%)
   Building:      9.572

   * Note how the MD5 checksum does not show-up at all in the list. The
timestamp matching seems to be triggered all the time.

   * Still the function changed_timestamp_then_content() was invoked
195093 on a build counting only 1100 nodes. Caching the result of
File.decide_source and File.decide_target might give some noticeable
speed-up.

     * The dominating part of checking whether a target is up-to-date is
the string substitution required for computing the action signature.

e) There is still some unexplained times. For example, I don't
understand yet what Taskmaster.next_task() might be doing apart from
mainly invoking node.children() which invokes node.is_up_to_date()

  8.312 sec.  /usr/lib/scons/SCons/Taskmaster.py:652(next_task)
  4.896 sec.  /usr/lib/scons/SCons/Node/FS.py:2732(is_up_to_date)

Benoit

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[gregnoel]

On Feb 11, 2008, at 11:49 AM, Ken Deeter wrote:

> Sorry, I think this discussion is useful, but I'd like to point out  
> that this is somewhat deviating from the initial point I was trying  
> to make. ... What I was trying to say is that the that the tools in  
> their current form provides is entirely supplementary to the core  
> value of scons as a build system.

This is my fault.  Some of the followups touched on the topic, and  
it's a discussion I've wanted to have, so I created a thread for it  
to see if anyone was interested.  There's another thread that's  
discussing what's the core value of SCons (although it's more about  
how to separate SCons into useful pieces); if that's not focused  
enough for you, create a fork of the conversation with a new subject  
and go for it.

Start it by answering this question: Aside from the Taskmaster and  
the Node subsystem, what is it that you see as the core of SCons?  
Those are the ones you've mentioned so far, but there must be more.  
Be prepared to defend your answer.

Hope this helps,
-- Greg Noel, retired UNIX guru

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Ken Deeter]

On Feb 11, 2008, at 3:27 PM, Greg Noel wrote:
> The first seems to be a matter of perspective.  You apparently  
> assume that there's basically one audience for SCons: a commercial  
> company that's using it for production (or within fuzz of this  
> categorization).  I don't agree.

That couldn't be farther from the truth. But what I'm curious about is  
how big is each user base, and where are the likely contributions  
going to come from? I am just as interested in seeing scons spread as  
widely as possible. But I'm writing here to talk about some of the  
issues a commercial company has with scons in its current form, issues  
that I think are representative of those faced by other companies  
trying to use it. I think failing to deal with those issues would be a  
great detriment to scons as a whole, since it's when companies start  
relying on your tools that they might be willing to contribute more  
time and people.


> I see at least three audiences: the SCons developer, the friendly  
> user, and the vast unwashed.  Each of the branches I propose caters  
> to one audience.  A developer can subscribe to the bleeding edge,  
> the friendly user can try out checkpoints, and everyone else waits  
> for the formal releases.  Right now there's not a lot of distinction  
> between the last two categories, but I would hope as time goes on  
> that the majority of users wouldn't even be subscribed to scons-users.
>

How many 'friendly users' do you estimate are out there? Who are these  
people? It's hard for me to believe that there are that many people  
out there who play around with scons just for the fun of it. Usually  
they are using it to try to solve some other problem. If you can show  
me that there is indeed a large community of such people, then I'd  
certainly give more though of doing something to cater to them. But  
after watching scons-dev and scons-users for a year or so, I didn't  
get the sense that there was this 'in between' set of people. My sense  
was that there were 1) developers, who are either on core or a feature  
branch, and 2) users who need a stable version to solve whatever other  
problem they're working on.


> The second is the the SCons release process is still missing one key  
> attribute: an active release team.  I see it as their responsibility  
> to rule on whether a feature is mature enough and to set the dates  
> for releases.  Right now Steven does this, and I don't believe it  
> should be his sole responsibility.  The problem is that nobody has  
> been able to kick-start the team and get it to start making the  
> decisions, but when that happens, I would hope that many of your  
> concerns will evaporate.  (Maybe the wiki page should be edited to  
> emphasize the role of the release team for these decisions?)

Well, I've been considering spending more of my time on helping out  
with releases, but it's difficult for me to commit to that if I don't  
really agree with the release procedure of philosophy. I think part of  
making a volunteer-based project successful is to reduce the amount of  
volunteer work that needs to be done. Having lots of branches to go  
through, and complex decisions on what goes in a release results in a  
lot of work for each release. I think a simple, iterative, lower-
overhead process can produce the same result in less time.

>
>> ... the more branches you have, the more you spread out real world  
>> testing. ... it is difficult (and I would argue unrealistic) to  
>> expect people or companies to spend a lot of time testing [earlier]  
>> branches.
>
> But then, I don't expect the vast unwashed to test the alpha or beta  
> branches.  I do expect that there are smaller communities that are  
> interested in using (and testing) those branches.
>

Sorry if I was unclear, I didn't mean to propose creating separate  
branches for alpha and beta releases. Instead, for example, all 0.98.x  
release would come from one branch. We could branch from core and call  
the first release alpha. then beta, then rc, then call it a .0. In my  
example I called the first release a .0, but that was just for the  
sake of illustration. The point is there needs to a place where code  
that's been released publicly can evolve and get fixes without mixing  
them with changes from branches/core.

>> ... What is 'stable enough' to allow a merge from the development  
>> branch to the stable branch? What is stable enough to move from  
>> stable checkpoint branch to product release branch? ...
>
> Whatever is the judgement of the release team.  I would expect  
> "small" features to be promoted quickly and "large" features to be  
> promoted more slowly; I would imagine that most features would be  
> "small" and get into a release quickly.  On the other hand, the big  
> signature refactoring (BSR) took more than a year from the time that  
> it first appeared until it got into a release.

Since the release team is mostly just Steve, that means whatever steve  
thinks, right? ;-)


> I would like to formalize that ability so a potentially  
> destabilizing feature like BSR could get more exposure before  
> committing it to production.
>

While there's nothing wrong with testing pre-release features and big  
changes, I think you might be overestimating users' willingness to  
test such a release. I can tell you that as a commercial developer  
using scons for really large projects, testing releases from non-
mainline branches is not one of my highest priorities, unless that  
feature has something that I desperately need. Whereas, if you say,  
"Hey, here's a set of code that is going to be released. It passes all  
our tests. Please check it out and make sure it works in your  
configuration before we send it out", I'm much more likely to find  
cycles to test that, because I know I will have to deal with it in the  
future.

>> ... Major releases are done on a time basis, by branching core into  
>> a release branch. There is *no* other criteria ...
>
> The problem with this is that if you decide to make a time-based  
> release at noon and someone checks in an ill-conceived patch at  
> 11:59, even if it passes the regression tests there's no chance to  
> revert it.  SCons takes backward compatibility seriously: that  
> feature, warts and all, would live for a long time.  I've seen this  
> happen with time-based releases, so I think the decoupling is worth  
> the effort.
>
> You can try to get around this by instituting draconian rules, such  
> as freezing the branch before making the fork, requiring reviews,  
> etc., but I've _never_ known them to work unless there was an actual  
> institution behind the project that could enforce sanctions.  No  
> matter how often you warn people, someone will make a last-minute  
> change that will be regretted.
>

Well, I agree you do have to have a freeze period before the branch  
happens. That being said, if someone does check in at 11:59, clearly  
they are not following the rules, and the change just needs to be  
backed out. SCons doesn't seem to have that high of a commit volume,  
so I don't see why enforcing a freeze would be that difficult.


> If the responsibility for promoting features lies with a release  
> team, they can choose which features will go in a given release.  
> They set the date for the release and then pick the features that  
> are ready.  (The major exception would be release candidates on the  
> checkpoint branch: only bug fixes would be accepted.)  This provides  
> the same decoupling as code freezes, timing features, extensive  
> reviews, and so forth, but without the draconian restrictions.
>

I think the vision that you're implying here, of being able to cherry  
pick features that are 'ready' and combine them into one reliable  
codebase with minimal testing, is somewhat over-optimistic. In  
practice, I think you need a codebase that has been tested as one set,  
because interactions between features are always non-trivial and  
difficult to predict.

Also, I again question the ability of the release team to really  
decide what is ready and what is not. They can test as much as they  
want, but I still won't trust it until I see it run against our vmware  
tree of many millions of lines of code. And this is not to question  
the capability of people who make up the release team (hi Steve! ;-).  
I think it's just dealing with the reality of software development.  
It's just really hard to do.

Besides, I don't consider having a freeze period so draconian if it  
means we can get a release out regularly. Getting  a whole bunch of  
users to test something provides a lot more value that's worth the  
tradeoff of having to have a developer or two wait a week or two  
before checking something in to the core branch.

>> ...  We explicitly state that a .0 release has seen very little  
>> real world testing. As people test the release, we take fixes on  
>> the release branch. ... We release .1, .2, .3 releases on a  
>> frequent basis. ...
>
> I've used products that had this scheme.  I _never_ implemented the .
> 0 release; I'd let the suckers, er, early adopters take the arrows.  
> And I'm not alone in this policy.  In reality, it's no different  
> than a checkpoint, so why not call it that?
>

Because it's not the same as a checkpoint. A checkpoint implies that  
it's some arbitrary snapshot from a development branch, where anything  
could have happened right before or and anything can happen right  
after. Testing a checkpoint release might be totally pointless,  
because some huge change might come in right afterwards that makes a  
bunch of testing I just did irrelevant.

Having a release branch to test is a promise to users their testing  
won't get thrown away but some big change that lands later. Rather,  
only low-risk "stabilization" fixes will be accepted, and the  
community that uses the stable release can be allowed to cooperate and  
evolve a more and more solid codebase.

> And I've noticed that if a product gets a reputation for being  
> buggy, and requires lots of releases to get it right, people won't  
> trust it as much as a product that is mostly bug-free.  I'd rather  
> get the major bugs out in checkpoints before someone depends on them  
> in a production release and they become features.
>

I think a project is much more evaluated by the quality of their  
latest stable release. That implies two things: a) it has a good  
releases process for stabilized code, and b) it has a release process  
that allows it to get stable code out the door as often as possible. I  
think it's fine for people to learn that they need to wait 'til  
version .2 before they should really try to rely on it. I mean that's  
the same with so many other things. Windows releases, you wait until  
SP2, etc.

As someone who works at a large software company, I can tell you that  
you can have tons of QA and testing releases, and you'll still  
sometimes end up with crappy .0 releases. It's just the nature of  
things. The scons project clearly doesn't have the same resources. In  
this situation, employing a release model that still requires a lot of  
'internal' testing (i.e. specifically testing of code that hasn't been  
pushed out as a 'release') doesn't make sense to me.


>> 4.  Major features are developed on branches off of core. ...
>
> We agree here, but I'd move the feature into the alpha branch  
> sooner.  In essence, I'd trade the hassle of keeping all the  
> development branches in sync with the core for the hassle of  
> promoting features individually.
>

That's mostly fine. As long as I have a stable branch that I can work  
with, I don't mind


> On Feb 11, 2008, at 11:33 AM, Ken Deeter wrote:
>
>> ... I think it is *impossible* to avoid pushing out buggy releases.  
>> Every release will have bugs. Period. ...
>
> Let me restate this: I believe it is impossible to avoid pushing out  
> releases with bugs, but it's possible (or at least desirable) to  
> avoid pushing out buggy releases.  The difference is huge.
>

How do you know something is a buggy release?

>> I'm saying that it's better to get something out, even if it has  
>> bugs, and effectively crowdsource the testing work. ...
>
> (Crowdsource, huh? {;-} Interesting expression...)
>
> Maybe you're simply willing to have a release with more bugs than I  
> am, but I know that if I saw a project that released .0, .1, .2,  
> and .3 in a couple of months, I wouldn't trust it, and I'd spend a  
> lot more time testing it than I would a project that rarely released  
> a .1.
>

So then we call it alpha, beta, beta 2, and rc. The name doesn't  
matter, and we can name things conservatively if we think it helps  
users understand what we expect to be the quality of each release.  
The point is, open source only works if you get code into as many  
hands as possible. So whether the code has bugs or not, I think it  
will improve faster the more people are testing it.

Ken

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Ken Deeter]

On Feb 11, 2008, at 3:36 PM, Thomas Nagy wrote:

>> I don't think we'll ever be able to define a model for how tools  
>> should look,
>
> Why do you think so ? Just because you have not seen such a model
> (yet), it does not mean that someone will not invent it or that it is
> not worth trying.
>

Pessimism, perhaps. Maybe it's not impossible. But I think it's  
exceedingly difficult. So much so I think it's more productive to come  
up with a mechanism that allow for different options, including  
allowing a user to roll their own.

>
>> we don't use any of the built in
>> variable schema at all. We write our own scanners, and our own  
>> builders.
>
> The same (abstract) model which was invented a long time ago is used,
> this means it was not too bad after all.

Which model are you talking about? I think the idea of scanners and  
builders and nodes is sufficiently generic. I don't think the builtin  
set of construction environment variables that scons uses is  
sufficiently generic though. We've essentially had to re-do most of it  
for our purposes. We probably wouldn't have decided to use scons at  
all if we didn't think this was possible.

>
>> My original email was lamenting the fact that sometimes, issues  
>> with the
>> tools was holding back improvements in the core system from being  
>> released. I
>> think if the tools and core scons have more independent paths, then  
>> we could
>> end up with a situation that is much more beneficial to both open  
>> source
>> projects and closed source commercial users.
>
> Creating new apis is also a way to ensure further separation between
> the core and the extensions (project management is another one).
>

I agree. I just wanted to say that the scope of those api's should be  
very limited, and really only address the features that we know for  
sure everyone wants.

Ken

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Ken Deeter]

> I don't think this is right, when I first refactored the Parallel jobs
> class (this was many, many years ago) I changed it from N threads that
> all ran the taskmaster, dispatched jobs, etc. (this led to very bad
> behavior on some machines because Python doesn't provide "fairness"
> guarentees wrt. its synchronization primitives.  What would happen is
> that the results of a "key" job would be starved, which would prevent
> new jobs that depended on it from being dispatched) to a worker thread
> pool/ consumer/producer model where the main thread dispatched/
> enqueued up to N jobs to the worker threads, and then processed up to
> N results.  If the results queue wasn't full, it would then go back to
> dispatch the next job(s).  I just checked and while the code has
> changed somewhat since then, it still appears to use the same general
> algorithm.
>
> I now recall that once I experiemented with unbounding the queue
> between the master and the worker threads (since the total number of
> active jobs is still bound by the number of worker threads) and found
> SCons was able to get much better performance because it was able to
> keep the queue full by running when long compiles were running so it
> could keep dispatching when a string of shorter compiles were running.
> I had to abandon this approach because it increased SCons' memory
> footprint, but now that I've got 8GB in our main build machine it's
> probably worth revisiting.  Perhaps not totally unbounding that queue,
> but perhaps some multiple of max_jobs.
>

We at vmware have found essentially the same thing. SCons is clearly  
the bottleneck in being able to dispatch more jobs, and the more time  
it spends 'waiting' because it thinks it has dispatched enough jobs,  
the worse the overall build time.

I also did try your hack (setting the queue to an unlimited length)  
and it seemed to work pretty well. It appeared to produce a level-
order walk of the dependency graph (i.e, it'd do all the .o's first,  
then all the .a's, then all the executables). My only concern with it  
was perhaps that it might cause file cache locality issues. The  
shorter the queue, scons concentrates on an increasingly related set  
of files. The longer the queue, scons goes all over the place,  
potentially blowing away file cache space by issuing "unrelated" jobs  
close to each other.

Ken

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Ken Deeter]

>
>
> For the cases with many sources and targets (>10,000 sources and
> targets), calculating dependencies got a little slow unless you used a
> trick where you combine all the sources into a single source that
> contained a checksum of the other ones (an approach like this was
> recently suggested on the list).
>

Yea, we didn't hit this until recently too. But it sounds like Greg's  
approach might help this aspect of it out.

When a builder has lots of outputs, I never really understood why each  
of them has to be processed sort of "individually".

Ken

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Ken Deeter]

On Feb 11, 2008, at 4:52 PM, Greg Noel wrote:

>
> Start it by answering this question: Aside from the Taskmaster and  
> the Node subsystem, what is it that you see as the core of SCons?  
> Those are the ones you've mentioned so far, but there must be more.  
> Be prepared to defend your answer.
>

Well, I suppose there is my personal opinion, but what matters to me  
most probably reflects how I use it at my job.

I'm going through the man page and trying to come up with a list of  
things:

Environments
Actions
Builders (the concept of, not the set of builders provided by the  
builtin tools)
DAG walk (taskmaster) and parallel job dispatcher
Nodes (File nodes, Dir nodes, Value nodes, and Alias Nodes)
Environment substitution mechanism (essentially env.Subst())


This is essentially everything that we use at VMware. The things we  
also want to use eventually are CacheDir and interactive mode. These  
two are inherently tied into the entities above.

Some of the things that I think can be separated out:

BuildDir: this seems like a somewhat simple attempt to support variant  
builds, but is too simplistic, and muddles some code due to the weird  
functionality it provides. I think this mechanism makes major  
assumptions about how source code and build output is organized. It is  
also pretty easily done in user's scripts in an arbitrarily complex way.

Tools: this is mostly because in our environment, we don't care about  
detecting tools on the build host. We want the opposite: we want  
exactly the tools we specify, and in the exact location we expect them  
to be. We've created a similar approach to tools, where pieces of the  
toolchain are expressed in python modules with a generate() function.

Repository: we don't use this. I'd be surprised if a lot of people do.  
Checking files out one by one just seems inefficient. We have 42000  
files in our main source tree, and checking out files one by one would  
just kill our server. I think it also adds some weird twists to the  
core code because everything has to be checking constantly if it  
talking about a repository file or not. I also think this code can be  
provided as a add on library, that user's scripts can call. You could  
probably attach builders to nodes for .c files and friends that know  
how to pull files from repositories.

Ken

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[gregnoel]

On Feb 11, 2008, at 12:04 PM, Ken Deeter wrote:

>> If it's any help, one of the motivations of issue 1526 is to split  
>> the processing into independent stages, ...
>
> ... Why can't you just have SConstruct that uses python to call  
> into some other mechanism that creates the dependency tree. [sic -  
> '?' not '.'] ...

Be my guest.  My point was that *if you wanted to* you could split  
off the front-end from (some flavor of) the core.  You said that you  
wanted just the core functionality; by separating the processing into  
stages that can be invoked independently, you have the option of only  
running the pieces that make sense for you, or replacing pieces with  
new ones specialized for your environment.  You may not want to  
replace the option scanner or the front-end, but somebody else might.

> ... I think of scons as a runtime environment for scripts with an  
> api to specify a DAG. It provides the basic primitives of an  
> Environment, Actions, Builders, and Nodes, and everything on top of  
> that is just as easily done in a user's script ...

Hmmm...  What is there in SCons that _isn't_ in that set?  BuildDir  
and Repository?  CacheDir?  Configure contexts?  Command-line options  
and flags?  Tools and scanners?  Looking at the SCons directory,  
that's about all I see.

The problem with them is that they're pretty integrated with the  
elements you consider core.  BuildDir and Repository change how nodes  
are located.  CacheDir has to be called at the right point in the  
flow to look for cached objects and save new ones.  Configure  
contexts and option objects have to be able to pass their results to  
an Environment.

Tools and scanners are designed to be modular, so indeed those could  
be factored out, but I'm not sure how much that saves you.

And I'm pretty sure that I don't want every single user to have to  
implement and maintain the same set of functions, even if they're  
"easily done" there.

> ... Separating the tools out is probably the most crystalized  
> suggestion I have for now. While I feel like I could do without  
> repository support and builddir(), those are probably a bit trickier.

BuildDir and Repository are implemented in a single small routine  
that creates a search list.  If the search list is one element long  
(i.e., no build dir or repository specified), there's probably a tiny  
bit of loop overhead that could be avoided in the various routines  
that use the search list, but it's probably not worth replacing it  
with a simplified version.

Maybe all that can be factored out are the tools and scanners.  
There's already a mechanism to replace them (via --site_dir), so I'm  
not sure how much you're gaining by getting rid of them.

Now, we can talk about dividing the _maintenance_ of various pieces  
if you don't want to eliminate them from a distribution.  In that  
case, I'd look at tools and scanners as one piece, for sure.  Other  
candidates are configure contexts, command-line options, and the  
Taskmaster.  (Yes, the Taskmaster is core, but it's really separate  
from the DAG-building functions.)

This sort of division is trickier, since there are interactions, but  
maybe that will encourage a better API between the pieces.

Hope this helps,
-- Greg Noel, retired UNIX guru

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[gregnoel]

On Feb 12, 2008, at 4:41 PM, Ken Deeter wrote:

> ... But what I'm curious about is how big is each user base, and  
> where are the likely contributions going to come from? ...

As a point of reference, there are about 150 people on the dev list  
and about 450 on the users list (numbers a year old).  Even assuming  
that all of the dev list also subscribe to the users list and only  
ten percent of them are willing to do a little testing, that's 15  
alpha testers and 30 beta testers.  That seems like enough to shake  
out the worst of the bugs before they are enshrined in production.

> ... I'm writing here to talk about some of the issues a commercial  
> company has with scons in its current form, ... I think failing to  
> deal with those issues would be a great detriment to scons as a  
> whole, ...

Yes, but I don't think either of our proposals would solve that.  
Both of them need some form of active release team to decide on the  
release dates, drive the features into the releases, and push them  
out the door.  Steven can't do it by himself, but until we can  
coerce, er, draft, er, convince others to help, we're pretty much stuck.

If you want to have a big effect on SCons, find some way to get the  
release team up and running.

> ... It's hard for me to believe that there are that many people out  
> there who play around with scons just for the fun of it. Usually  
> they are using it to try to solve some other problem. ...

Well, _I_ play with SCons for the fun of it.  I also have some  
personal projects that use SCons that I also play with for the fun of  
it.  The great advantage of being retired is that you can do things  
like that...

> ... I didn't get the sense that there was this 'in between' set of  
> people. My sense was that there were 1) developers, who are either  
> on core or a feature branch, and 2) users who need a stable version  
> to solve whatever other problem they're working on.

I'd suggest that the people on the mailing lists are pretty much all  
the in-between crowd.  For the most part, you're talking about the  
portion that _isn't_ on the mailing lists: the people who install the  
package provided by their vendor and then live with it.  At most,  
they're on the announcement list (hmmm, Steven, how many people are  
on that list?) and would only subscribe to the users list if they  
have a problem (and drop off as soon as it's solved).

Currently, I'd imagine that the community that lives on vendor  
packages is pretty small, no more than a thousand.  But I'll point  
out that the reason we don't hear from them is because we emphasize  
bug-free releases so they have no need to contact us.

> Well, I've been considering spending more of my time on helping out  
> with releases, but it's difficult for me to commit to that if I  
> don't really agree with the release procedure ...

We'd be sorry to lose you; your opinions are interesting, even if I  
disagree.  And if you did volunteer, you could be setting hard dates  
for releases and promoting features as quickly as possible; you could  
see how fast and responsive you could make the process. {;-}

> Sorry if I was unclear, I didn't mean to propose creating separate  
> branches for alpha and beta releases. ...

Uh, it's clear that I was unclear.  I'm proposing branches for alpha  
and beta, otherwise known as development and checkpoint.  That  
corresponds pretty much to the state of the branch.

> Since the release team is mostly just Steve, that means whatever  
> steve thinks, right? ;-)

Yes, currently.  He's probably too close to the code to make reasoned  
judgments all the time.  He also likes to stuff in "just one more  
fix" to releases, so he never gets the release out on schedule.  If  
we take that off his shoulders, then he could still try to get as  
many fixes in as he can, but others would take the responsibility to  
cut the release on time, possibly leaving out some of the last-minute  
fixes.

Again, I think the decoupling is important since it gives the effect  
of a code freeze (and the like) without preventing development.

> ... if you say, "Hey, here's a set of code that is going to be  
> released. ... Please check it out and make sure it works in your  
> configuration before we send it out", I'm much more likely to find  
> cycles to test that...

Those would be release candidates, off of the checkpoint/beta  
branch.  They would be advertised to a wider audience than the  
straight checkpoints, but probably not to as wide an audience as the  
mainline releases.

> Well, I agree you do have to have a freeze period before the branch  
> happens. ... SCons doesn't seem to have that high of a commit  
> volume, so I don't see why enforcing a freeze would be that difficult.

I've never liked freezes; it makes me queue up multiple fixes in my  
local tree (or a personal branch), and slows development because  
others can't take it for a spin.  I'd rather decouple the development  
branch from the release branches, which gives the effect of a  
freeze.  The fact that SCons doesn't currently have a high commit  
volume doesn't mean that it might not be higher in the future.

> I think the vision that you're implying here, of being able to  
> cherry pick features ... with minimal testing, is somewhat over-
> optimistic. In practice, I think you need a codebase that has been  
> tested as one set...

Maybe.  I think it has as good a shot as trying to debug a release  
after it's out the door.  The checkpoints provide a way of  
incrementally checking on new features; the release candidate  
checkpoints even more so.

> Also, I again question the ability of the release team to really  
> decide what is ready and what is not. ... I still won't trust it  
> until I see it run against our vmware tree ...

If you're going to test it, no matter what they choose, it doesn't  
make any difference how good their decisions are.  It can't be any  
worse than uncritically accepting all the features.

> Because it's not the same as a checkpoint. A checkpoint implies  
> that it's some arbitrary snapshot from a development branch, ...  
> Testing a checkpoint release might be totally pointless, ...

Steven is always after me about mantras, so here's one for you: "A  
checkpoint is _not_ a snapshot.  In particular, a checkpoint is not  
an _arbitrary_ snapshot."  A snapshot is taken at a particular time,  
no matter the state of the branch; a checkpoint is managed like a  
release, so there's some control of what goes into it.

If you think a checkpoint, particularly a release candidate  
checkpoint, is worthless for testing, then don't bother.  Wait for  
the mainstream release.  I see it as no different than a .0 release  
in your scheme, but if you choose not to see that, it's OK with me.

> How do you know something is a buggy release?

You said it was, not me.  I try to avoid them.  I don't want users to  
depend on bugs and then complain when they are fixed, something that  
would happen in your scheme.

> ... we call it alpha, beta, beta 2, and rc. The name doesn't  
> matter...  open source only works if you get code into as many  
> hands as possible...

If the name doesn't matter, then alpha is the development branch,  
beta is the checkpoint branch, RC are bug-fixes-only checkpoint  
releases.

You think it will get into more hands if it's pushed out as a  
mainstream release; I think it will get into a sufficient number of  
hands as checkpoints.  We'll have to agree to disagree.

Hope this helps,
-- Greg Noel, retired UNIX guru

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[gregnoel]

I really didn't want to have to write this message.

On Feb 11, 2008, at 3:19 PM, Thomas Nagy wrote:

> You seem to be rushing on the code and the examples instead of  
> spending enough efforts on the analysis.

I'm spending my effort trying to understand what inputs are needed  
and what outputs can be expected.

> I have provided a formal model corresponding to your constraints  
> (evolution in the time, backward compatibility, etc),

I'm sorry, but what formal model was that?  A formal model involves  
mathematics; I _surely_ haven't seen any of that.

And a model implies more than "There are variables.  There are  
rules.  The rules cause variables to be created or changed.  The  
resultant variables are what you want."  That's all you've given us  
so far; if you've got more, you haven't shown it.

> now it is up to you to prove it wrong in finding an example for  
> rejecting the model, for extending it, or demonstrating how to  
> simplify it. Even with the platform and the CXXFLAGS variables as  
> examples the description seems to remain unclear to you.

No, first it's up to you to explain your model.  Yes, it's unclear to  
me.  If someone doesn't understand you, it's your responsibility to  
explain it more clearly.  If you can't do that, if you aren't even  
willing to try, then I must believe you don't understand it yourself.

> Have you spent more than 5 minutes trying to understand why this  
> suggestion was made ? Have you made the effort of looking at expert  
> systems definitions and examples ? Have you actually tried to find  
> a scenario contradicting the model given ?

This is a rude and insulting ad-hominem attack.  I have a graduate  
degree in Mathematics and forty years of experience implementing all  
kinds of complicated systems, including expert systems, fuzzy logic  
controllers, JIT schedulers, more than a dozen "little languages" in  
various problem domains, and many others.  Almost certainly I know  
more about expert systems than you do.  You have no call to insinuate  
that it's my fault; you owe me an apology.

So apologize.  And then explain your model without hand-waving.

-- Greg Noel, retired UNIX guru

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[David Cournapeau-4]

On Mon, 2008-02-11 at 23:06 -0800, Greg Noel wrote:

>
> I've never liked freezes; it makes me queue up multiple fixes in my  
> local tree (or a personal branch), and slows development because  
> others can't take it for a spin.  I'd rather decouple the development  
> branch from the release branches, which gives the effect of a  
> freeze.  The fact that SCons doesn't currently have a high commit  
> volume doesn't mean that it might not be higher in the future.
>

(hope I got the quoting right this time)

Well, to be peaky, those are not consequences of freeze, but a
limitation of the tool (subversion).

As an extremely light contributor to scons, let me just express my
feeling, which seems to be shared by other, and is linked to this whole
discussion I think: the entry bar to scons contribution is extremely
high. This is seen at many levels: not just the code, but the whole
infrastructure. For example, this is a detail but telling, it is hard to
just get the code from svn. On all other projects I have contributed to
(well, not that many, I am neither a guru or retired :) ), you get it in
ten seconds from google. For scons, you need an account with a password,
which is public, but hard to find from the webpage; the main public
webpage for development itself is daunting and not really clear for a
newcommer.

For me, that would be the main point of splitting in some sense the code
of scons "core" from scons "tools": making the bar of entry lower.
Because scons "core" demands anyway a good understanding of scons to
contribute to, which is not the case at all for scons "tools": this one,
many people can contribute to (and it has right now many small bugs:
some of them do not work right now, like fortran tools, and this is
often easier to fix for people using the tool than for scons
contributors unfamiliar with the tool; IOW, your arguments for the
current situation wrt regression testing, releases, etc... may not apply
to this part of the code).

How many users contribute to scons wrt number of users ? My impression -
I may be totally wrong here, so I apologize if this is wrong - is that
not that many people contribute things to scons, and the ratio is low,
at least compared to other open source projects I am contributing to. I
know *I* almost gave up (I actually gave up using scons at some point
for one year), and only persevered because the main project I am
contributing to (numpy) uses a totally f*-up build system which cause
all kind of problems (distutils), and I wanted to improve the situation
for this another project.

This is an argument for what Ken Deeter suggested, but from a totally
different point of view. Does this make sense ?

cheers,

David


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Thomas Nagy]

> > Have you spent more than 5 minutes trying to understand why this
> > suggestion was made ? Have you made the effort of looking at expert
> > systems definitions and examples ? Have you actually tried to find
> > a scenario contradicting the model given ?
>
> This is a rude and insulting ad-hominem attack.  I have a graduate
> degree in Mathematics and forty years of experience implementing all
> kinds of complicated systems, including expert systems, fuzzy logic
> controllers, JIT schedulers, more than a dozen "little languages" in
> various problem domains, and many others.  Almost certainly I know
> more about expert systems than you do.  You have no call to insinuate
> that it's my fault; you owe me an apology.
>
> So apologize.  And then explain your model without hand-waving.

Wow, I did not even think you would take it as a personal attack, but
now that you insist:
* I am really surprised you have not come with a model or a
specification in the first place. After all, such a problem cannot be
that difficult for such a skilled person as you ?
* How can you even take this as a personal attack ? If you are
confident in your skills, you should not even need to mention them.
* You insist on reasoning by cases and examples, which is obviously
the wrong approach: the cases cannot be all foreseen, so the scheme is
supposed to have as few constraints as possible to evolve. The best
approach is to start from a generic model, and to refine it when
additional constraints are found. So far you have provided generic
constraints and i have provided a generic model.

I will now provide examples, which should be obvious for anyone with a
little knowledge in Scons and project configuration:

An instance of Environment is initialized (empty or not), and a set or
rules is applied on that object.
* A rule called by the user copies the platform in the Environment
* A rule is declared for setting the variable DATADIR in the
Environment instance, using the environment as input (using the
variable PREFIX) and writing right into the Environment instance
* Rules for detecting a compiler for a particular language are loaded
from a file, they perform the following operations:
  * finding a program in the system, using the PATH variable in the
environment instance
  * checking the program version using COMPILER_MIN_VERSION in the
environment instance
* New configuration rules are declared in a user script and they are
called immediately on an Environment instance
* A rule for detecting a particular compiler flag is inserted (from a
user script) in the list of rules for detecting a compiler. It is
inserted before the rule used for detecting a library. Afterwards the
rules are applied in order on an environment instance.

Here are a few rules examples:
* if variable is not set, set it to value
* remove element x if present in environment[var]
* find program x and assign to environment[var]
* run check y and assign to env[var] or raise a configuration error

These examples make various assumptions on how the code should look
like, they should not be taken for designing apis. In any case, the
environment changes should be made as small units of change instead of
huge if/else trees.

Hope this helps,
Thomas

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Ken Deeter]

>
> (hope I got the quoting right this time)
>
> Well, to be peaky, those are not consequences of freeze, but a
> limitation of the tool (subversion).
>
> As an extremely light contributor to scons, let me just express my
> feeling, which seems to be shared by other, and is linked to this whole
> discussion I think: the entry bar to scons contribution is extremely
> high. This is seen at many levels: not just the code, but the whole
> infrastructure. For example, this is a detail but telling, it is hard to
> just get the code from svn. On all other projects I have contributed to
> (well, not that many, I am neither a guru or retired :) ), you get it in
> ten seconds from google. For scons, you need an account with a password,
> which is public, but hard to find from the webpage; the main public
> webpage for development itself is daunting and not really clear for a
> newcommer.
>

FWIW, I tend to agree with this. I forget if you're the one that suggested
moving to a DVCS, but I also think that has the potential to help scons out a
lot. It would have certainly helped us in maintaining a 'local' version at
vmware in a manner that was easy to feed back up stream, and it lets anyone
create a 'feature branch' and experiment. Right now we have a somewhat
cumbersome svn to perforce and back mechanism that Steve wrote for us.

Given that you need windows support, I'd say mercurial is a logical choice
here.. has the developer community seriously considered this?
Ken

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Russel Winder-4]

On Tue, 2008-02-12 at 09:50 -0800, Ken Deeter wrote:

> > As an extremely light contributor to scons, let me just express my
> > feeling, which seems to be shared by other, and is linked to this whole
> > discussion I think: the entry bar to scons contribution is extremely
> > high. This is seen at many levels: not just the code, but the whole
> > infrastructure. For example, this is a detail but telling, it is hard to
> > just get the code from svn. On all other projects I have contributed to
> > (well, not that many, I am neither a guru or retired :) ), you get it in
> > ten seconds from google. For scons, you need an account with a password,
> > which is public, but hard to find from the webpage; the main public
> > webpage for development itself is daunting and not really clear for a
> > newcommer.

Isn't this really a criticism of Tigris rather than SCons?  As far as I
am aware anonymous access to a repository at Tigris is always via the
user name guest with no password.  I just checked out SCons:

|> svn checkout http://scons.tigris.org/svn/scons/trunk SCons
Authentication realm: <http://scons.tigris.org:80> CollabNet Subversion
Repository
Username: guest
Password for 'guest':
A    SCons/test
. . .

> FWIW, I tend to agree with this. I forget if you're the one that suggested
> moving to a DVCS, but I also think that has the potential to help scons out a
> lot. It would have certainly helped us in maintaining a 'local' version at
> vmware in a manner that was easy to feed back up stream, and it lets anyone
> create a 'feature branch' and experiment. Right now we have a somewhat
> cumbersome svn to perforce and back mechanism that Steve wrote for us.

I like Bazaar myself since it is a DVCS that can have Subversion-like
features, i.e. a central master branch that can be checked out or
branched.  I hear lots of good things about Mercurial as well, but I
don't think it can handle the centralized workflow model.

(No I am not a plant put here by Canonical :-)

I am using Bazaar with the bzr-svn plugin as a way of checking out a
Subversion repository as a Bazaar checkout.  This works well as you get
the branch of a DVCS with the checkout of a Subversion checkout.

SVK is another alternative for doing Subversion store mirroring with
local branches.  However, I find it a bit cumbersome whilst Bazaar is
more straightforward.  I think Mercurial also has a Subversion
repository branch capability, but I have not tried it.

> Given that you need windows support, I'd say mercurial is a logical choice
> here.. has the developer community seriously considered this?

I think Mercurial and Bazaar are equal on this platform issue, both
being Python applications.

--
Russel.
====================================================
Dr Russel Winder                 Partner

Concertant LLP                   t: [phone]
41 Buckmaster Road,              f: [phone]
London SW11 1EN, UK.             m: [phone]