Thinking towards 7.6 katamari, including xcb

[Alan Coopersmith]

While we should hopefully have the 7.5 katamari wrapped up this
weekend once xserver 1.7.1 is out, I've been thinking ahead towards
the next round of this.   (I'll need time to recover from this round
before I'm foolish enough to consider agreeing to help drive that one,
maybe in a few months...)

The last few katamari's have been pretty much annual releases, and
that seems to work - we've also shown that we can release the X
server & driver bundles between katamaris without much problem.

The current Xserver 1.8 schedule calls for it to release on 2010-3-31.
If we stick to the 6 month cadence well, then 1.9 should be released
on 2010-9-31.  (Though I still think it should be called 2.0 if the
drivers do merge back into the xserver tarball.)

I'd suggest then we plan on the 7.6 katamari releasing in early
October 2010, close to one year after 7.5.

Due to the recent change to xlsclients (and suggested similar changes to
other clients that still need to be written), the 7.6 katamari will probably
be the first to have a hard requirement on XCB support - sometime during
the next year we should decide if we want XCB to remain a separate thing
we depend on or whether we want to work with the XCB maintainers to include
it in our release process as a core katamari component.    (So far we've
kept Mesa, xkeyboard-config, pixman and libpciaccess as sister projects
that we work closely with, depend on, and even build in some of our build
scripts, but which we don't release in our release tree or apply our module
standards (like the xorg autoconf macros) to - whether any of those should
also be pulled into the main X.Org release process is something else people
keep asking.)

--
        -Alan Coopersmith-           alan.coopersmith@...
         Sun Microsystems, Inc. - X Window System Engineering

_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Keith Packard]

Excerpts from Alan Coopersmith's message of Thu Oct 22 05:36:30 +0900 2009:

> The current Xserver 1.8 schedule calls for it to release on 2010-3-31.
> If we stick to the 6 month cadence well, then 1.9 should be released
> on 2010-9-31.  (Though I still think it should be called 2.0 if the
> drivers do merge back into the xserver tarball.)

I'm hoping we can pull the 1.9 release in by three months and having a
very short merge window that just adds the drivers back without other
significant changes. To make this work, we'd need to see trees with
drivers merged back well before the 1.8 release date. Driver
maintainers with an interest in having their driver integrated in 1.9
should consider posting merged trees for review in the next few months.

> I'd suggest then we plan on the 7.6 katamari releasing in early
> October 2010, close to one year after 7.5.

That seems like a good schedule. One thing I'd like to see is far
fewer packages released just before the katamari though; can you say
what kind of issues you've seen resolved by the updated modules?

> Due to the recent change to xlsclients (and suggested similar changes to
> other clients that still need to be written), the 7.6 katamari will probably
> be the first to have a hard requirement on XCB support - sometime during
> the next year we should decide if we want XCB to remain a separate thing
> we depend on or whether we want to work with the XCB maintainers to include
> it in our release process as a core katamari component.

I'd say xcb should be part of the katamari; I don't see any
significant issues here, we can just package it up and include it.

--
keith.packard@...


_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Daniel Stone]

Hi,

On Thu, Oct 22, 2009 at 10:44:35AM +0900, Keith Packard wrote:

> Excerpts from Alan Coopersmith's message of Thu Oct 22 05:36:30 +0900 2009:
> > The current Xserver 1.8 schedule calls for it to release on 2010-3-31.
> > If we stick to the 6 month cadence well, then 1.9 should be released
> > on 2010-9-31.  (Though I still think it should be called 2.0 if the
> > drivers do merge back into the xserver tarball.)
>
> I'm hoping we can pull the 1.9 release in by three months and having a
> very short merge window that just adds the drivers back without other
> significant changes. To make this work, we'd need to see trees with
> drivers merged back well before the 1.8 release date. Driver
> maintainers with an interest in having their driver integrated in 1.9
> should consider posting merged trees for review in the next few months.

What? Why?

If 7.6 in December 2010 seems like a good idea, then what's the point of
doing 1.9 in September 2010? Is the thinking to ram all the features we
need for the next year in to 1.8, do a short 1.9, seriously[0] maintain
it as a stable branch and keep it going and ship 7.6 with a more
plausible 1.9.5 or thereabouts, and then do the feature dance again for
1.10?

If so, is this something we want to think about doing long-term? (If so,
we might want to invert our cycles to stick with the x.y : y ->
odd/unstable, even/stable convention used by pretty much every other
open source project ever.)

> > I'd suggest then we plan on the 7.6 katamari releasing in early
> > October 2010, close to one year after 7.5.
>
> That seems like a good schedule. One thing I'd like to see is far
> fewer packages released just before the katamari though; can you say
> what kind of issues you've seen resolved by the updated modules?

Is there any advantage to leaving them sit in git with no release
forever? In an ideal world, they'd all be maintained, but realistically
they're pretty much all maintained by the current release manager.

> > Due to the recent change to xlsclients (and suggested similar changes to
> > other clients that still need to be written), the 7.6 katamari will probably
> > be the first to have a hard requirement on XCB support - sometime during
> > the next year we should decide if we want XCB to remain a separate thing
> > we depend on or whether we want to work with the XCB maintainers to include
> > it in our release process as a core katamari component.
>
> I'd say xcb should be part of the katamari; I don't see any
> significant issues here, we can just package it up and include it.

+1

(And +1 to the Dec 2010 schedule as well.  Not only does it sound
 plausible and give us more time to get our user[1]-facing APIs as good
 as possible while letting us rip the server to shreds, I don't see
 anyone else stepping up to do the thankless katamari work.)

Cheers,
Daniel

[0]: _Seriously_.
[1]: External libraries/programs.


_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Keith Packard]

Excerpts from Daniel Stone's message of Thu Oct 22 12:09:24 +0900 2009:

> What? Why?

Doing more frequent releases will obviously reduce the lag between
implementation and deployment; this should do lots of good for
everyone involved. I get constant requests for shorter X server
release intervals, enough so that I'm willing to do the work to make
it happen if that's OK with the X.org community.

The Intel driver is on a quarterly release schedule at this point, and
we've been getting good feedback on this process from Linux OSVs and
other users of the driver. Obviously sticking to schedules is a key
part of any benefit to the OSVs here, and in my experience, shorter
schedules are easier to hit than longer ones. Yes, it's a lot of
release management, but as I said, I'm willing to make that happen.

In any case, I've already committed to quarterly Intel driver
releases, so either the X server as a whole gets released on this
schedule or I'll end up doing a separate 'old X server + new intel
driver' release every other quarter. That would suck.

Independent of the release schedule, getting people thinking about how
drivers will get integrated should improve the chances of a successful
1.9 release.

> If 7.6 in December 2010 seems like a good idea, then what's the point of
> doing 1.9 in September 2010?

Few OSVs track just the katamari anyway; with drivers integrated into
the server, we've got a complete and consistent X server that is
deliverable at all times.

Getting 1.9 done with the drivers integrated lets me get started
merging API/ABI changes in the server, something which I find long
overdue. I'll be posting patches for review in this area shortly.

> Is the thinking to ram all the features we
> need for the next year in to 1.8, do a short 1.9, seriously[0] maintain
> it as a stable branch and keep it going and ship 7.6 with a more
> plausible 1.9.5 or thereabouts, and then do the feature dance again for
> 1.10?

Nope; my plan is strictly time-based releases that contain whatever
features are ready at that time. I'll push my requirements to match
the merge window, and I'd hope that other people would be willing to
do the same. Doing these frequently reduces the cost of missing a
particular release, letting us push things off that aren't quite
ready. Of course, OSVs will be able to pull such features into their
packages as necessary to hit their customer requirements where said
features are not yet released upstream.

The key here is to get features ready for the merge before the merge
window opens. We can merge huge changes in a short time if they've
seen review and general agreement on the design. The release schedule
shouldn't in any way drive broad feature development schedules, other
than the final alignment with a suitable merge window.

Frankly, I see the katamari process as less urgent once we merge the
server and drivers back together -- we've always had an extremely
strong API/ABI guarantee across the wire/kernel interface, and that
isn't changing anytime soon. That means that the promised testing and
integration offered by the katamari will be less important as the less
stable driver API/ABI is removed as an external interface.

> If so, is this something we want to think about doing long-term? (If so,
> we might want to invert our cycles to stick with the x.y : y ->
> odd/unstable, even/stable convention used by pretty much every other
> open source project ever.)

I'd like to say that all of our releases are stable releases --
unstable work should occur in feature branches or separate repositories.

> (And +1 to the Dec 2010 schedule as well.  Not only does it sound
>  plausible and give us more time to get our user[1]-facing APIs as good
>  as possible while letting us rip the server to shreds, I don't see
>  anyone else stepping up to do the thankless katamari work.)

Agreed -- building the client-side pieces is still too much work to
even contemplate on a more frequent basis. Thanks again to Alan for
stepping up this time around!

--
keith.packard@...


_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Jesse Barnes-2]

On Thu, 22 Oct 2009 14:09:24 +1100
Daniel Stone <daniel@...> wrote:

> Hi,
>
> On Thu, Oct 22, 2009 at 10:44:35AM +0900, Keith Packard wrote:
> > Excerpts from Alan Coopersmith's message of Thu Oct 22 05:36:30
> > +0900 2009:
> > > The current Xserver 1.8 schedule calls for it to release on
> > > 2010-3-31. If we stick to the 6 month cadence well, then 1.9
> > > should be released on 2010-9-31.  (Though I still think it should
> > > be called 2.0 if the drivers do merge back into the xserver
> > > tarball.)
> >
> > I'm hoping we can pull the 1.9 release in by three months and
> > having a very short merge window that just adds the drivers back
> > without other significant changes. To make this work, we'd need to
> > see trees with drivers merged back well before the 1.8 release
> > date. Driver maintainers with an interest in having their driver
> > integrated in 1.9 should consider posting merged trees for review
> > in the next few months.
>
> What? Why?
>
> If 7.6 in December 2010 seems like a good idea, then what's the point
> of doing 1.9 in September 2010? Is the thinking to ram all the
> features we need for the next year in to 1.8, do a short 1.9,
> seriously[0] maintain it as a stable branch and keep it going and
> ship 7.6 with a more plausible 1.9.5 or thereabouts, and then do the
> feature dance again for 1.10?
>
> If so, is this something we want to think about doing long-term? (If
> so, we might want to invert our cycles to stick with the x.y : y ->
> odd/unstable, even/stable convention used by pretty much every other
> open source project ever.)

Well, Linux actually moved away from that model, and so far I think
it's been very beneficial.

To summarize some off-list discussion:

Pros of a shorter release cycle (e.g. 3 months)
  - shorter lag time between new features & invasive bug fixes and
    those bits landing in distros
  - tighter development practices, i.e. narrow merge window, necessary
    focus on stabilizing things (well stabilizing design at least,
    and probably most bugs) before then
  - less work to maintain stable branch than a long cycle

Cons:
  - shorter development cycle might make landing big, invasive changes
    harder (e.g. perpetually unready for the merge window can make for
    a vicious cycle)
  - less incentive to maintain a long lived and complete stable branch
  - shocking to users who are accustomed to randomly timed major releases

I think we already went over the pros & cons of integrated drivers, but
I think that discussion is largely orthogonal to this one.

Jesse
_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Daniel Stone]

On Thu, Oct 22, 2009 at 04:48:36PM +0900, Keith Packard wrote:
> Excerpts from Daniel Stone's message of Thu Oct 22 12:09:24 +0900 2009:
> > What? Why?
>
> Doing more frequent releases will obviously reduce the lag between
> implementation and deployment; this should do lots of good for
> everyone involved.

That would quite obviously be good, but we can still be smart about
how we time our six-month cycles.  I believe that timing it right
would result in a very short lag for Fedora, Ubuntu, and SUSE; given
that they're on six-month release cycles, a well-timed six-month
release cycle would have the same lag (or less than) than a three-month
cycle.

> I get constant requests for shorter X server
> release intervals, enough so that I'm willing to do the work to make
> it happen if that's OK with the X.org community.

Only if it doesn't stifle the server development.

> The Intel driver is on a quarterly release schedule at this point, and
> we've been getting good feedback on this process from Linux OSVs and
> other users of the driver. Obviously sticking to schedules is a key
> part of any benefit to the OSVs here, and in my experience, shorter
> schedules are easier to hit than longer ones. Yes, it's a lot of
> release management, but as I said, I'm willing to make that happen.

Sure, but OSVs and device enables are a reasonably special case.  Having
six-month cycles means that maintaining a stable branch is relatively
easy, since within a one-year timeframe, we only really have two
releases we care about, which surely shouldn't make most backporting
that difficult.

Given that everyone's on six-month cycles, having three-month cycles
means that we'd need to support four branches instead of two.  After
all, we can't really leave all the distros out in the cold (and
OSVs/OEMs certainly can't), so that's two branches we need to support.
And people would quite fairly be pissed if a stable tree was abandoned
after only three months' support, so we really need to support the other
two.

This is assuming that people aren't going to push new major releases
into their stable trees which, well, they aren't.  Probably not even if
our QA was up to the standard we hope it'll be at in two years, which it
isn't.

So, er, which part of this is easier?

(Also, assuming that not every driver will get merged in -- who's going
to be the one to make sure that every released driver builds with every
released server? Seriously, pciaccess was bad enough, and that was once,
not once every three months.)

> In any case, I've already committed to quarterly Intel driver
> releases, so either the X server as a whole gets released on this
> schedule or I'll end up doing a separate 'old X server + new intel
> driver' release every other quarter. That would suck.

I'm sorry that you made that commitment. ;) Maemo is shipping with a
server that's a git snapshot from February, but everyone involved
understands that we've just got to suck it up.

> Independent of the release schedule, getting people thinking about how
> drivers will get integrated should improve the chances of a successful
> 1.9 release.

Well, you're arguably slightly biased on this front because you maintain
a driver, and I'm arguably slightly biased on this front because I don't
touch any drivers.  But I'd say that there's probably a lot more to a
useful 1.9 than drivers, but merging them in in order to fix our API
would definitely be a huge step forward.

> > If 7.6 in December 2010 seems like a good idea, then what's the point of
> > doing 1.9 in September 2010?
>
> Few OSVs track just the katamari anyway; with drivers integrated into
> the server, we've got a complete and consistent X server that is
> deliverable at all times.

And then they're left out in the cold, because in the eight months since
they picked up that stable version, we've shipped three new major
versions? That doesn't really seem like the definition of
OSV-friendliness to me.

> Getting 1.9 done with the drivers integrated lets me get started
> merging API/ABI changes in the server, something which I find long
> overdue. I'll be posting patches for review in this area shortly.

Cool; no argument here.

> > Is the thinking to ram all the features we
> > need for the next year in to 1.8, do a short 1.9, seriously[0] maintain
> > it as a stable branch and keep it going and ship 7.6 with a more
> > plausible 1.9.5 or thereabouts, and then do the feature dance again for
> > 1.10?
>
> Nope; my plan is strictly time-based releases that contain whatever
> features are ready at that time. I'll push my requirements to match
> the merge window, and I'd hope that other people would be willing to
> do the same. Doing these frequently reduces the cost of missing a
> particular release, letting us push things off that aren't quite
> ready. Of course, OSVs will be able to pull such features into their
> packages as necessary to hit their customer requirements where said
> features are not yet released upstream.
>
> The key here is to get features ready for the merge before the merge
> window opens. We can merge huge changes in a short time if they've
> seen review and general agreement on the design. The release schedule
> shouldn't in any way drive broad feature development schedules, other
> than the final alignment with a suitable merge window.

That's fine if we've already decided on a three-month window; assuming
that our testing base for features which have not yet been merged is
small enough that we can go from first proposed for merge to shippable
in a .0 release in three or four months, I think having the compressed
schedule will actually cripple us.

Put it this way: it took, what, a year and a half to get MPX into shape?
Even when it was into master, so everyone was forced to deal with it.  I
don't disagree that it'd be a great position to be in, but I don't think
we're there yet.  If we get there and we feel like maintaining four
trees rather than two, then sure, let's sit down and look at three
months.  Right now though, it feels like either insanity (in putting out
four stable server releases per year -- which means supporting them, as
you well know from having to backport the world to server 1.2), or sheer
dishonesty (in that we're not going to be able to support what we've
only just shipped).

> Frankly, I see the katamari process as less urgent once we merge the
> server and drivers back together -- we've always had an extremely
> strong API/ABI guarantee across the wire/kernel interface, and that
> isn't changing anytime soon. That means that the promised testing and
> integration offered by the katamari will be less important as the less
> stable driver API/ABI is removed as an external interface.

I'm not even thinking about the katamari: everything I've said in this
and previous mails applies to the server only (and the drivers to some
extent).

> > If so, is this something we want to think about doing long-term? (If so,
> > we might want to invert our cycles to stick with the x.y : y ->
> > odd/unstable, even/stable convention used by pretty much every other
> > open source project ever.)
>
> I'd like to say that all of our releases are stable releases --
> unstable work should occur in feature branches or separate repositories.

Think of it as stable/LTS if that helps.

Cheers,
Daniel


_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Daniel Stone]

Hi,

On Thu, Oct 22, 2009 at 06:32:03PM +0900, Jesse Barnes wrote:

> On Thu, 22 Oct 2009 14:09:24 +1100
> Daniel Stone <daniel@...> wrote:
> > If 7.6 in December 2010 seems like a good idea, then what's the point
> > of doing 1.9 in September 2010? Is the thinking to ram all the
> > features we need for the next year in to 1.8, do a short 1.9,
> > seriously[0] maintain it as a stable branch and keep it going and
> > ship 7.6 with a more plausible 1.9.5 or thereabouts, and then do the
> > feature dance again for 1.10?
>
> Well, Linux actually moved away from that model, and so far I think
> it's been very beneficial.

And to the three-month (give or take) model, no?

> To summarize some off-list discussion:
>
> Pros of a shorter release cycle (e.g. 3 months)
>   - shorter lag time between new features & invasive bug fixes and
>     those bits landing in distros

Not really.  Distros ship when they ship, which right now is six months.
And the ones people care about tend to freeze within an acceptable
distance of each other, so thinking about it and timing your server
releases properly means you can get a very short time-to-distro.

>   - tighter development practices, i.e. narrow merge window, necessary
>     focus on stabilizing things (well stabilizing design at least,
>     and probably most bugs) before then

No doubt.  I think to some extent this depends on merging the drivers
though, unless we stop offering the guarantee (of sorts) that drivers
will build against every previously released server.

>   - less work to maintain stable branch than a long cycle

Sure.  It's less work to maintain _a_ stable branch.  However, the cost
of maintaining four stable branches, the oldest of which dates back a
year, is greater than the cost of maintaining two stable branches, the
oldest of which dates back a year.  Unless our plan is to abandon
certain releases, which is what I was getting at with the odd/even
numbering.

If our support cycle is one year, then we need to maintain four
branches; I strongly doubt that will happen to an acceptable standard.
Someone will always be left out in the cold.  If that happens, then we
need to be honest about that up front: we need to say 'sure, you can use
this, but in nine months or less, you'll be on your own'.

Which puts a bit of a dampener on the whole lower-time-to-distro thing,
no?

> Cons:
>   - shorter development cycle might make landing big, invasive changes
>     harder (e.g. perpetually unready for the merge window can make for
>     a vicious cycle)

Yep.  MPX would be, er, challenging.

>   - less incentive to maintain a long lived and complete stable branch

As above.

>   - shocking to users who are accustomed to randomly timed major releases

Phoronix won't even have anything to write about!

> I think we already went over the pros & cons of integrated drivers, but
> I think that discussion is largely orthogonal to this one.

It's mostly orthogonal, but I think that having integrated drivers is
one of the things that makes 3 month cycles possible.  Merge all the
drivers (at least all the ones we care about) and have a decent story
for updating the others to new APIs and making sure they build and run
with every released server version we have, and then I'll drop most of
my objections.

Cheers,
Daniel


_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Eric Anholt-4]

On Thu, 2009-10-22 at 22:48 +1100, Daniel Stone wrote:

> On Thu, Oct 22, 2009 at 04:48:36PM +0900, Keith Packard wrote:
> > Excerpts from Daniel Stone's message of Thu Oct 22 12:09:24 +0900 2009:
> > > What? Why?
> >
> > Doing more frequent releases will obviously reduce the lag between
> > implementation and deployment; this should do lots of good for
> > everyone involved.
>
> That would quite obviously be good, but we can still be smart about
> how we time our six-month cycles.  I believe that timing it right
> would result in a very short lag for Fedora, Ubuntu, and SUSE; given
> that they're on six-month release cycles, a well-timed six-month
> release cycle would have the same lag (or less than) than a three-month
> cycle.
>
> > I get constant requests for shorter X server
> > release intervals, enough so that I'm willing to do the work to make
> > it happen if that's OK with the X.org community.
>
> Only if it doesn't stifle the server development.
>
> > The Intel driver is on a quarterly release schedule at this point, and
> > we've been getting good feedback on this process from Linux OSVs and
> > other users of the driver. Obviously sticking to schedules is a key
> > part of any benefit to the OSVs here, and in my experience, shorter
> > schedules are easier to hit than longer ones. Yes, it's a lot of
> > release management, but as I said, I'm willing to make that happen.
>
> Sure, but OSVs and device enables are a reasonably special case.  Having
> six-month cycles means that maintaining a stable branch is relatively
> easy, since within a one-year timeframe, we only really have two
> releases we care about, which surely shouldn't make most backporting
> that difficult.

In Mesa, we've been doing 3-month cycles, with developers pushing fixes
back to the previous 2 stable releases.  It's been working out pretty
well, other than the particular merging procedure (in my opinion).  I
haven't seen complaints about rushing forward on new development and
leaving people behind, and beyond about 2 cycles back (6 months) things
get different enough that I don't think you'll see developers support it
anyway.

server-1.5-branch has been awfully idle as of late.

--
Eric Anholt
eric@...                         eric.anholt@...




_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Peter Hutterer-3]

On Thu, Oct 22, 2009 at 04:48:36PM +0900, Keith Packard wrote:
> Excerpts from Daniel Stone's message of Thu Oct 22 12:09:24 +0900 2009:
>
> > What? Why?
>
> Doing more frequent releases will obviously reduce the lag between
> implementation and deployment; this should do lots of good for
> everyone involved. I get constant requests for shorter X server
> release intervals, enough so that I'm willing to do the work to make
> it happen if that's OK with the X.org community.

A few questions:

How many of these requests were driven by our permanently late release
cycle? i.e. would an actual 6 month release cycle classify as "shorter
release interval"?

Is there enough churn in the server to warrant more frequent releases?
The larger changes are mostly complete now and looking at my schedule I
doubt XI2.1 will land for 1.8.

If there aren't enough features, having extra releases and the associated
branches increases the time spent release-managering. looking at how close
1.7 is to master I wonder if we'd end up with multiple branches that are the
same except for a few patch sets.
since we need to cherry-pick, you get the chance to reproduce bugs in
interesting ways - cherry-picked to 1.7 but not to 1.8, but it is on 1.9,
etc.
How long do you want to maintain stable branches? Or do we just keep
maintaining _some_ stable branches and let the others rot? If so, which
ones?

Deployment is -largely- distro driven. with our past track record regarding
QA I'm not sure how many distros are willing to deploy a new server update
during their stable cycle. At which point you end with server releases being
skipped by distros altogether.

IMO we don't have the testers yet to make all that happen. As it is, I'm
permanently switching from master to 1.7 to my devel branches to make sure I
get half-decent coverage. I'm not sure how much exposure the various
personal trees get right now.

tbh, I'm not convinced yet of the benefits of shorter release cycles
(shorter than 6 months, that is).

Cheers,
  Peter
_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Keith Packard]

Excerpts from Peter Hutterer's message of Thu Oct 22 23:15:36 -0700 2009:

> How many of these requests were driven by our permanently late release
> cycle? i.e. would an actual 6 month release cycle classify as "shorter
> release interval"?

Well, 1.6 was released 6 months after 1.5, and 1.8 will be released 6
months after 1.7, so we're actually not doing too badly on average
here.

> Is there enough churn in the server to warrant more frequent releases?
> The larger changes are mostly complete now and looking at my schedule I
> doubt XI2.1 will land for 1.8.

Once we merge the drivers back into the server, the level of change in
the overall package will go up dramatically. Yes, I'm hoping that the
feature set in the core server will stabilize, but we've got a pile of
API/ABI changes to work through that we can only realistically do with
integrated drivers.

> If there aren't enough features, having extra releases and the associated
> branches increases the time spent release-managering. looking at how close
> 1.7 is to master I wonder if we'd end up with multiple branches that are the
> same except for a few patch sets.

My hope is that our need for backporting changes will be reduced by
shrinking the release window. We had several 1.6.x releases mostly
because 1.7 was a delayed -- there were features pending on master
that OSVs needed to ship before 1.7 did. I back-ported more than just
bug-fixes here, which I wasn't entirely happy about.

> How long do you want to maintain stable branches? Or do we just keep
> maintaining _some_ stable branches and let the others rot? If so, which
> ones?

I'd be willing to ship two versions and back-port patches from master
to both of those (so, 1.7.x and 1.8.x while 1.9 is in
development). That's more than we've done in the past, and provides a
6-9 month window given 3 month release cycles.

> Deployment is -largely- distro driven. with our past track record regarding
> QA I'm not sure how many distros are willing to deploy a new server update
> during their stable cycle. At which point you end with server releases being
> skipped by distros altogether.

I doubt we'd manage to skip all of the distros, and in any case,
re-integrating the drivers into the server should help bring back
people willing to try a new X server/driver combo.

> tbh, I'm not convinced yet of the benefits of shorter release cycles
> (shorter than 6 months, that is).

I can't support a 6 month cycle in my video driver, and I doubt other
video drivers could either; hardware changes too fast. If the video
drivers are to be re-integrated into the server, we'll need some
compromise in how often the X server is released.

--
keith.packard@...


_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Jesse Barnes-2]

On Thu, 22 Oct 2009 22:53:52 +1100
Daniel Stone <daniel@...> wrote:

> Hi,
>
> On Thu, Oct 22, 2009 at 06:32:03PM +0900, Jesse Barnes wrote:
> > On Thu, 22 Oct 2009 14:09:24 +1100
> > Daniel Stone <daniel@...> wrote:
> > > If 7.6 in December 2010 seems like a good idea, then what's the
> > > point of doing 1.9 in September 2010? Is the thinking to ram all
> > > the features we need for the next year in to 1.8, do a short 1.9,
> > > seriously[0] maintain it as a stable branch and keep it going and
> > > ship 7.6 with a more plausible 1.9.5 or thereabouts, and then do
> > > the feature dance again for 1.10?
> >
> > Well, Linux actually moved away from that model, and so far I think
> > it's been very beneficial.
>
> And to the three-month (give or take) model, no?

Right.  That's also been a big help.

> > To summarize some off-list discussion:
> >
> > Pros of a shorter release cycle (e.g. 3 months)
> >   - shorter lag time between new features & invasive bug fixes and
> >     those bits landing in distros
>
> Not really.  Distros ship when they ship, which right now is six
> months. And the ones people care about tend to freeze within an
> acceptable distance of each other, so thinking about it and timing
> your server releases properly means you can get a very short
> time-to-distro.

Distros have to coordinate several packages of software though, and
there's always going to be conflicts between ones you care about.
Having frequent releases can mitigate that.

> >   - tighter development practices, i.e. narrow merge window,
> > necessary focus on stabilizing things (well stabilizing design at
> > least, and probably most bugs) before then
>
> No doubt.  I think to some extent this depends on merging the drivers
> though, unless we stop offering the guarantee (of sorts) that drivers
> will build against every previously released server.

Yeah, having the drivers there would probably help...

> >   - less work to maintain stable branch than a long cycle
>
> Sure.  It's less work to maintain _a_ stable branch.  However, the
> cost of maintaining four stable branches, the oldest of which dates
> back a year, is greater than the cost of maintaining two stable
> branches, the oldest of which dates back a year.  Unless our plan is
> to abandon certain releases, which is what I was getting at with the
> odd/even numbering.

Realistically though, upstream doesn't maintain older stable branches;
distros always have specific patches they've backported into their
released package, independent of what upstream does.  If you think we
need to maintain 4 stable branches going back a year though, then yeah
I see your point.

> If our support cycle is one year, then we need to maintain four
> branches; I strongly doubt that will happen to an acceptable standard.
> Someone will always be left out in the cold.  If that happens, then we
> need to be honest about that up front: we need to say 'sure, you can
> use this, but in nine months or less, you'll be on your own'.
>
> Which puts a bit of a dampener on the whole lower-time-to-distro
> thing, no?

Not really; if you're relying totally on upstream to handle your bugs,
then as a distro you're in big trouble.  Distros are supposed to be
where projects get polished into a real product, as opposed to just
being rough cut building blocks (except perhaps in the case of actual
end-user applications).  I don't think that means we can ignore bugs,
but it should mean we don't have to handle all of the logistics of
tracking bug fixes back into N stable release branches.

--
Jesse Barnes, Intel Open Source Technology Center
_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Rémi Cardona-3]

Le 26/10/2009 21:28, Keith Packard a écrit :
>> tbh, I'm not convinced yet of the benefits of shorter release cycles
>> (shorter than 6 months, that is).
>
> I can't support a 6 month cycle in my video driver, and I doubt other
> video drivers could either; hardware changes too fast. If the video
> drivers are to be re-integrated into the server, we'll need some
> compromise in how often the X server is released.

If the server+drivers packages becomes as self-contained as the kernel,
then I think it's fine.

However, if the server "forces" changes like the Xext lib/proto
overhaul, then the benefits are going to be much less.

As for the release cycles, how about having 3 month cycles globally but
the server core stays feature-frozen every other release? At least maybe
for the first release or 2?

Food for thoughts

Cheers,

Rémi
_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Keith Packard]

Excerpts from Rémi Cardona's message of Mon Oct 26 13:57:00 -0700 2009:

> However, if the server "forces" changes like the Xext lib/proto
> overhaul, then the benefits are going to be much less.

We may require newer versions of the protocol headers as extensions
are integrated into the server, but we will not require newer versions
of any libraries.

> As for the release cycles, how about having 3 month cycles globally but
> the server core stays feature-frozen every other release? At least maybe
> for the first release or 2?

We might offer some level of ABI/API stability as drivers are merged
into the server. Perhaps making the first 3 month release API/ABI
compatible with the 1.8 release?

Then we can see how things look at that point; perhaps doing API/ABI
changes only on even releases.

--
keith.packard@...


_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Alan Coopersmith]

Keith Packard wrote:
> Excerpts from Rémi Cardona's message of Mon Oct 26 13:57:00 -0700 2009:
>
>> However, if the server "forces" changes like the Xext lib/proto
>> overhaul, then the benefits are going to be much less.
>
> We may require newer versions of the protocol headers as extensions
> are integrated into the server, but we will not require newer versions
> of any libraries.

Except when fixing the protocol/library splits like happened in 7.5?

Or if Xephyr/Xdmx/Xnest start using a new protocol extension version
in their client sides?

--
        -Alan Coopersmith-           alan.coopersmith@...
         Sun Microsystems, Inc. - X Window System Engineering

_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Bryce Harrington-5]

On Mon, Oct 26, 2009 at 01:28:40PM -0700, Keith Packard wrote:
> > Deployment is -largely- distro driven. with our past track record regarding
> > QA I'm not sure how many distros are willing to deploy a new server update
> > during their stable cycle. At which point you end with server releases being
> > skipped by distros altogether.
>
> I doubt we'd manage to skip all of the distros, and in any case,
> re-integrating the drivers into the server should help bring back
> people willing to try a new X server/driver combo.

If it helps at all to hear from one of the distros, I can outline some
of the process and problems in the decision making we've gone through
for X versions in Ubuntu.

The main constraints I have for selection of X components in Ubuntu are
these freeze dates:

                      .04 release           .10 release
                     
  FeatureFreeze:      End of February       End of August

  BetaFreeze:         End of March          End of September

If the X release happens before FeatureFreeze, it's a no-brainer, we
ship everything from the X release (we can pull from Debian).  Upstream
can be happy since bug reports and patches are against their latest
release.

If the X release happens after BetaFreeze, it's also easy since it's
just too late to update stuff.  Upstream would be less happy in this
scenario since Ubuntu bug reports and patches will be against the
previous version.

But most commonly, the release comes between FF and Beta, and things are
tricky.  We have to pick on a package-by-package basis between staying
with the old version and risk not having the newest features, or
shipping a new version and risking having a buggy X.


In a perfect world, the stable X.org release would hit just prior to
FeatureFreeze - one in mid-Feb, the other mid-Aug.  In this world,
Ubuntu would be shipping the latest X.org release all the time, and bug
reports and patches would always be against the version of things that
upstream wants to support.

Bryce
_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Peter Hutterer-3]

On Mon, Oct 26, 2009 at 01:28:40PM -0700, Keith Packard wrote:
> Excerpts from Peter Hutterer's message of Thu Oct 22 23:15:36 -0700 2009:
>
> > How many of these requests were driven by our permanently late release
> > cycle? i.e. would an actual 6 month release cycle classify as "shorter
> > release interval"?
>
> Well, 1.6 was released 6 months after 1.5, and 1.8 will be released 6
> months after 1.7, so we're actually not doing too badly on average
> here.

1.4.1 was planned for November, released in June.
1.5 was planned for March, released September.
1.7 was planned for July, released October.

If you ask around which release dates ppl remember, my bet is on those that
were late, not those that were on time. In the same way as one only
remembers the trains being late, but never the ones on time. 1.8 is still in
the future, so it's risky to use that as a reference point.

> > Is there enough churn in the server to warrant more frequent releases?
> > The larger changes are mostly complete now and looking at my schedule I
> > doubt XI2.1 will land for 1.8.
>
> Once we merge the drivers back into the server, the level of change in
> the overall package will go up dramatically. Yes, I'm hoping that the
> feature set in the core server will stabilize, but we've got a pile of
> API/ABI changes to work through that we can only realistically do with
> integrated drivers.
>
> > If there aren't enough features, having extra releases and the associated
> > branches increases the time spent release-managering. looking at how close
> > 1.7 is to master I wonder if we'd end up with multiple branches that are the
> > same except for a few patch sets.
>
> My hope is that our need for backporting changes will be reduced by
> shrinking the release window. We had several 1.6.x releases mostly
> because 1.7 was a delayed -- there were features pending on master
> that OSVs needed to ship before 1.7 did. I back-ported more than just
> bug-fixes here, which I wasn't entirely happy about.

My worry is less about active backporting but about missing out on
cherry-picks. If the branches are close together, 1.8, 1.9 and 1.10 may
needd patches [A-Z], but 1.9 forgot to cherry-pick [K-M] and 1.8 didn't
cherry-pick E, M, N and O. So although they're close enough, you're suddenly
running different combinations of patches and bugs.

I don't know of a tool that helps with this.
 
> > How long do you want to maintain stable branches? Or do we just keep
> > maintaining _some_ stable branches and let the others rot? If so, which
> > ones?
>
> I'd be willing to ship two versions and back-port patches from master
> to both of those (so, 1.7.x and 1.8.x while 1.9 is in
> development). That's more than we've done in the past, and provides a
> 6-9 month window given 3 month release cycles.

Who's going to nominate the patches for backporting though? Nominating a
patch means the developer has to know that the other branch has the same
issue and test this locally. That requires a lot of time, expecting this for
3 branches only really works if those haven't changed much between releases.
Which I guess will be the case with 3-monthly releases, but that's a bit of
a loop there then.
 
> > Deployment is -largely- distro driven. with our past track record regarding
> > QA I'm not sure how many distros are willing to deploy a new server update
> > during their stable cycle. At which point you end with server releases being
> > skipped by distros altogether.
>
> I doubt we'd manage to skip all of the distros, and in any case,
> re-integrating the drivers into the server should help bring back
> people willing to try a new X server/driver combo.

Provided our pre-commit QA goes up. I've been running git master since the
MPX merge and my biggest worry was always having to update the video driver.
I usually lost a day to get things running again. I'm sure others had
similar issues with input, except that I knew how to fix those on my box so
I wasn't worried about them.
If I have to drag down the whole lot each time, I'd be seriously worried
about running git again.

> > tbh, I'm not convinced yet of the benefits of shorter release cycles
> > (shorter than 6 months, that is).
>
> I can't support a 6 month cycle in my video driver, and I doubt other
> video drivers could either; hardware changes too fast. If the video
> drivers are to be re-integrated into the server, we'll need some
> compromise in how often the X server is released.

How many of these hardware changes require a new server?

The dependency here seems to be
- we pull the drivers into the server
- drivers need faster updates
- the server now needs more frequent releases

If the drivers aren't pulled in, then the server can trot along slower.
So the real question is - does the benefit of pulling the drivers into the
server outweigh the costs of releasing and maintaining multiple server
branches?
The decision to merge seems to have been made already, so I guess that
question is answered.

Cheers,
  Peter
_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Keith Packard]

Excerpts from Peter Hutterer's message of Tue Oct 27 23:52:23 -0700 2009:

> If the drivers aren't pulled in, then the server can trot along slower.

And that's what's been happening to date; the server loafs along at a
6-month to 1-year release cycle. And we get few people running recent
servers because they've got a lot of changes in them, and they have to
rebuild and install several modules get the new server working.

> So the real question is - does the benefit of pulling the drivers into the
> server outweigh the costs of releasing and maintaining multiple server
> branches?

At this point, we've got a huge pile of legacy garbage in our driver
API which is bloating the server and making it difficult to fix some
problems. We could fix that in the current environment, but that would
mean getting all of the drivers to follow along while still providing
backwards compatibilities with old server APIs. Anyone want to do the
whole RandR 1.2 transition again?

I'm also unconvinced that X.org needs to maintain a million (or even
three) server branches, even with a 3-month server release cycle. I
know an OSV will need to maintain anything they release for long term
support, but that really isn't something X.org can do for them --
we're only talking about 6 months vs 1 year in the case of a
two-release window. A drop in the bucket of difficulty compared with
back porting patches for seven years. Anyone is welcome to pick up an
old release and maintain it; I know Greg K-H is still maintaining
Linux 2.6.27 as it's useful for Novell as a company, after all.

But, if doing 3 month releases of the whole server tree means that
we'll scare OSVs away from our project, then I wonder how they manage
the Linux kernel today. Is the three month cycle a nightmare there? Or
is it helping them?

> The decision to merge seems to have been made already, so I guess that
> question is answered.

Until we actually merge stuff, we can always revisit this
decision. Not that I really want to; I'm looking forward to deleting
more code from the intel driver and the X server. But, I admit to not
really thinking through the desire to keep shipping drivers frequently.

There are significant benefits to merging in drivers, the two that I'm
looking forward to are:

 1) Bisect server and driver together to find bugs.
 2) Fixing the API/ABI aggressively.

--
keith.packard@...


_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Daniel Stone]

Hi,

On Wed, Oct 28, 2009 at 12:42:40AM -0700, Keith Packard wrote:

> Excerpts from Peter Hutterer's message of Tue Oct 27 23:52:23 -0700 2009:
> > So the real question is - does the benefit of pulling the drivers into the
> > server outweigh the costs of releasing and maintaining multiple server
> > branches?
>
> At this point, we've got a huge pile of legacy garbage in our driver
> API which is bloating the server and making it difficult to fix some
> problems. We could fix that in the current environment, but that would
> mean getting all of the drivers to follow along while still providing
> backwards compatibilities with old server APIs. Anyone want to do the
> whole RandR 1.2 transition again?

No doubt -- as I said, this is one of the best arguments for merging the
drivers into the server.  You can't even call our collection of rubbish
header files an API with a straight face, and that needs to be fixed at
some stage, so.

> I'm also unconvinced that X.org needs to maintain a million (or even
> three) server branches, even with a 3-month server release cycle. I
> know an OSV will need to maintain anything they release for long term
> support, but that really isn't something X.org can do for them --
> we're only talking about 6 months vs 1 year in the case of a
> two-release window. A drop in the bucket of difficulty compared with
> back porting patches for seven years. Anyone is welcome to pick up an
> old release and maintain it; I know Greg K-H is still maintaining
> Linux 2.6.27 as it's useful for Novell as a company, after all.
>
> But, if doing 3 month releases of the whole server tree means that
> we'll scare OSVs away from our project, then I wonder how they manage
> the Linux kernel today. Is the three month cycle a nightmare there? Or
> is it helping them?

OTOH, we are not the kernel.  OSVs currently throw large amounts of
development time at the kernel because they know how it works and they
expect that they have to put in that much work.  Right now, they take
stable X trees, and expect them to work and at least be loosely
maintained into the future.

Red Hat are the exception to the rule here.

If we want to change this and say that once a release is made, you can
use it if you want and we guarantee that it won't bitrot or actively get
worse, but beyond that you're on your own ... well, we can do it.  But
it's a huge change from what we have now (releases we make are actively
supported for bug fixes etc well beyond their release date), and I think
it's one that should be discussed, given that it was never even
mentioned at XDC, nor on the list.

The sole motivation for this seems to be making driver maintainers'
lives easier (which is no bad thing), and giving hardware vendors a
shorter time-to-market for hardware enables.  Is there anything stopping
you from backporting your hardware-enable code to at least the last
stable release (i.e. 6 months old), and shipping a point release? The
hardware-enable diffs I've seen from the Intel driver seem to be quite
small indeed, so I get the feeling this wouldn't be a huge burden on you
guys, and it would also prevent the detrimental effects a lot of the
server hackers feel we'll see with a vastly reduced cycle.

It would be nice to see what the distros think about this, too.

Cheers,
Daniel


_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Bryce Harrington-5]

On Wed, Oct 28, 2009 at 07:39:41PM +1100, Daniel Stone wrote:

> On Wed, Oct 28, 2009 at 12:42:40AM -0700, Keith Packard wrote:
> > But, if doing 3 month releases of the whole server tree means that
> > we'll scare OSVs away from our project, then I wonder how they manage
> > the Linux kernel today. Is the three month cycle a nightmare there? Or
> > is it helping them?
>
> OTOH, we are not the kernel.  OSVs currently throw large amounts of
> development time at the kernel because they know how it works and they
> expect that they have to put in that much work.
>
> The sole motivation for this seems to be making driver maintainers'
> lives easier (which is no bad thing), and giving hardware vendors a
> shorter time-to-market for hardware enables.
>
> It would be nice to see what the distros think about this, too.

For Ubuntu, it is more the timing and predictability of the releases
than their frequency.  If it were timed right, a 6-month cycle that
releases right before our feature freeze date would be about perfect.

But the problem is that what may be perfect for one distro may be too
early or too late for another distro.  This is where a 3-month cycle
could have some advantage.

We've seen some of this benefit in Intel's release cycle.  We get a
release N which is early in our development cycle, and another N+1 which
typically comes in well past feature freeze.  Since the difference
between N and N+1 is only 3 months, we could just ship N and not feel
guilty about it.  Or, if N+1 is mostly a bug-fix release, and in our own
testing we find it super solid, we can pull it in even really late
without feeling we're taking too big of a risk.

Bryce



_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb

[Stephane Marchesin]

On Wed, Oct 28, 2009 at 11:04, Bryce Harrington <bryce@...> wrote:

> On Wed, Oct 28, 2009 at 07:39:41PM +1100, Daniel Stone wrote:
>> On Wed, Oct 28, 2009 at 12:42:40AM -0700, Keith Packard wrote:
>> > But, if doing 3 month releases of the whole server tree means that
>> > we'll scare OSVs away from our project, then I wonder how they manage
>> > the Linux kernel today. Is the three month cycle a nightmare there? Or
>> > is it helping them?
>>
>> OTOH, we are not the kernel.  OSVs currently throw large amounts of
>> development time at the kernel because they know how it works and they
>> expect that they have to put in that much work.
>>
>> The sole motivation for this seems to be making driver maintainers'
>> lives easier (which is no bad thing), and giving hardware vendors a
>> shorter time-to-market for hardware enables.
>>
>> It would be nice to see what the distros think about this, too.
>
> For Ubuntu, it is more the timing and predictability of the releases
> than their frequency.  If it were timed right, a 6-month cycle that
> releases right before our feature freeze date would be about perfect.
>
> But the problem is that what may be perfect for one distro may be too
> early or too late for another distro.  This is where a 3-month cycle
> could have some advantage.
>
> We've seen some of this benefit in Intel's release cycle.  We get a
> release N which is early in our development cycle, and another N+1 which
> typically comes in well past feature freeze.  Since the difference
> between N and N+1 is only 3 months, we could just ship N and not feel
> guilty about it.  Or, if N+1 is mostly a bug-fix release, and in our own
> testing we find it super solid, we can pull it in even really late
> without feeling we're taking too big of a risk.
>

Or, as others suggested, you (distro vendors) could hire people
knowledgeable about X/driver internals and maintain/backport your own
stuff. This is the core of the problem here, there is no such thing as
a free lunch...
I'm afraid here more releases will shift even more burden on
understaffed X.Org teams.

Stephane
_______________________________________________
Xcb mailing list
Xcb@...
http://lists.freedesktop.org/mailman/listinfo/xcb