Version control systems

[Ian Lynagh]

On Mon, Aug 11, 2008 at 04:17:59PM +0100, Simon Marlow wrote:

>
> One way we could create the forks would be to create a git repo for each
> package with two branches: the master branch that GHC builds, and a
> separate branch that tracks the main darcs repository, and is synced
> automatically whenever patches are pushed to the main darcs repo.  We'd
> have to explicitly merge the tracking branch into the master branch from
> time to time.  When we want to make changes locally, we can just commit
> them to the GHC branch and push the changes upstream in a batch later (and
> then we'd end up having to merge them back in to the GHC branch... but
> hopefully git's merge is clever enough to avoid manual intervention here).
>  This is complicated and ugly of course; better suggestions welcome.

I don't think that this will work well. At some point we'll have to
resolve conflicts that accumulate (by rebasing GHC-only patches after
conflicting upstream patches come in? This may not be trivial, as the
rebaser may not be familiar with the patches), and I suspect that the
forks will end up diverging for significant periods of time, as there's
little impetus to merge them.

Even the current situation with Cabal is a bit of a pain, as it's easy
to forget to push patches upstream as well as GHC's repo, and that's
just with 2 repos of the same VCS.

I actually think the original plan, where only the ghc repo (plus one or
two others) is in git is preferable. You may have to use a different VCS
for different subprojects, but after that it's downhill all the way.
You don't have to worry about patches being converted from one VCS to
another, moved to another repo, converted back and merged back into the
first repo.

I expect people will be using different VCSs for different /projects/,
if not subprojects, anyway.


Thanks
Ian

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Johan Henriksson-2]

Don Stewart wrote:

> kili:
>  
>> On Mon, Aug 11, 2008 at 04:17:59PM +0100, Simon Marlow wrote:
>> [...]
>>    
>>> As for Cabal - we had a thread on cvs-ghc last week, and as I said there
>>> we'd love to hear suggestions for how to improve things, including wild
>>> and crazy ideas for throwing it all away and starting again.  However, as
>>> I explained, there are good reasons for the way things are done now, the
>>> main one being that the build system for packages is not written twice.
>>>      
>> Well, at least the Makefile creation was a step (the first step?)
>> into the wrong direction, IMHO. I'll run a GHC build to get some
>> of those generated Makefiles and followup on cvs-ghc, but for a
>> starter, Cabal shouldn't know anything about implementation-specific
>> internal build systems; instead it should rely only on it's own
>> metadata.  Implementation-specific stuff (such as how to run the
>> compiler) should be supplied by the implementation, not by Cabal.

if we're going to kick on cabal I might throw in my two cents.

I see an increasing problem in that every community comes up with their
own package system, instead of reusing existing frameworks. dependencies
to other non-haskell libraries has to be addressed for every other
coexisting package system (such as apt-get), if it is addressed at all.
likewise, other languages depending on haskell will have trouble
resolving dependencies.

so my point is, if there will be any bigger reworking of cabal, I think
one should consider how it could work as a module in a bigger (maybe
future) meta-packaging framework, lifting up binaries to for example
.deb, .exe-installer, .dmg or whatever is the most native for the
platform. I see a point in language specific package systems as they
have more insight into the build process, but the current
implementations assume a very ideal world in which there are no other
dependencies involved.

/Johan
_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Manuel M T Chakravarty]

Matthias Kilian:

> On Mon, Aug 11, 2008 at 04:17:59PM +0100, Simon Marlow wrote:
> [...]
>> As for Cabal - we had a thread on cvs-ghc last week, and as I said  
>> there
>> we'd love to hear suggestions for how to improve things, including  
>> wild
>> and crazy ideas for throwing it all away and starting again.  
>> However, as
>> I explained, there are good reasons for the way things are done  
>> now, the
>> main one being that the build system for packages is not written  
>> twice.
>
> Well, at least the Makefile creation was a step (the first step?)
> into the wrong direction, IMHO. I'll run a GHC build to get some
> of those generated Makefiles and followup on cvs-ghc, but for a
> starter, Cabal shouldn't know anything about implementation-specific
> internal build systems; instead it should rely only on it's own
> metadata.  Implementation-specific stuff (such as how to run the
> compiler) should be supplied by the implementation, not by Cabal.
>
> I see more and more workarounds for workarounds for an unmaintainable
> (and unusable) build system, and after the latest discussions about
> git vs. darcs, maintaining GHC-specific branches of libraries etc.,
> I think I'll just drop maintainership from all GHC-related OpenBSD
> ports until the GHC build system chaos settles down a little bit.

Thanks for demonstrating my point...

Complicated build infrastructure and lack of portability used to be a  
big problem for GHC in the past.  Over the last years, the situation  
got much better (to a large extent due to SimonM sanitising the  
makefile-based build system).  Why are we so keen to throw it all away  
now?

Manuel

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Manuel M T Chakravarty]

Simon Marlow:

> Manuel M T Chakravarty wrote:
>
>> I think all *core* libraries must switch.  Seriously, requiring GHC  
>> developer to use a mix of two vcs during development is a Very Bad  
>> Idea.  Don was excited about getting more people to look at the  
>> source when it is in git (see the comments he posted from reddit).  
>> By requiring two vcs you will get *less* people to look at the  
>> source.
>> This is not only to get the sources to hack them, but you  
>> effectively require developers to learn the commands for two vcs  
>> (when they are already reluctant to learn one).  For example, often  
>> enough somebody who changes something in GHC will modify the base  
>> package, too.  Then, to commit the overall work, you need to commit  
>> using both vcs.  If you need to branch for your work, you need to  
>> create branches in two vcs (no idea whether the semantics of a  
>> branch in git and darcs is anywhere similar).  When you merge your  
>> branch, you need to merge in both vcs.  You can't seriously propose  
>> such a set up!
>
> I completely agree this is a problem.  The main obstacle with just  
> switching the core libraries is that they are shared by other  
> implementations and other maintainers.  So I see no alternative but  
> to create forks of those repositories for use by GHC, unless/until  
> the other projects/maintainers want to migrate to git.  Some of the  
> repositories are not shared - for example ghc-prim, integer-gmp,  
> template-haskell, and these don't need to be forked.
>
> One way we could create the forks would be to create a git repo for  
> each package with two branches: the master branch that GHC builds,  
> and a separate branch that tracks the main darcs repository, and is  
> synced automatically whenever patches are pushed to the main darcs  
> repo.  We'd have to explicitly merge the tracking branch into the  
> master branch from time to time.  When we want to make changes  
> locally, we can just commit them to the GHC branch and push the  
> changes upstream in a batch later (and then we'd end up having to  
> merge them back in to the GHC branch... but hopefully git's merge is  
> clever enough to avoid manual intervention here).  This is  
> complicated and ugly of course; better suggestions welcome.

Yes, it's a pain.  However, it is better than two vcs for one project.

>> I *strongly* object to moving to git before this isn't sorted out.  
>> As Roman said before, GHC is heading into a dangerous direction.  
>> It gets progressively harder to contribute to the project at the  
>> moment.  First, changing the build system to Cabal.  Now, proposing  
>> to use two vcs.  Somebody who is new to the project not only has to  
>> learn the internals of GHC, but they also have to learn two new  
>> vcs, and if they need to change the build system, they need to  
>> learn a new build tool.  Raising the bar for developers to  
>> contribute to a project has been proven to be a very bad idea many  
>> times.  Let's not take GHC down that path.
>
> I'm not completely convinced we need to have this all worked out  
> before GHC switches, although it would be nice of course.  We  
> currently have infastructure in place for the build to work with a  
> mixture of darcs and git repositories, and existing developers  
> already have to learn git anyway.  They just need to remember to use  
> darcs for libraries and git for the main GHC repo, and this is only  
> a temporary situation.

As far as I am concerned, building GHC is turning into a big mess.  We  
discussed ways to improve it again, BUT I'd rather not see it getting  
any messier before it gets better.  Hence, please let's have a  
complete plan that we are convinced will work before making any more  
changes.

> As for Cabal - we had a thread on cvs-ghc last week, and as I said  
> there we'd love to hear suggestions for how to improve things,  
> including wild and crazy ideas for throwing it all away and starting  
> again.  However, as I explained, there are good reasons for the way  
> things are done now, the main one being that the build system for  
> packages is not written twice.

Yes, we need cabal for packages because we don't want two build  
systems.  However, this does not justify the use of Cabal outside of  
libraries/.  Nobody explained to me why that was necessary.  Why  
change all the rest of the build system.  What is the benefit for the  
ghc project?

To be honest, if you ask me, I'd go back to the old makefile based  
system and remove Cabal from everywhere except building of the library  
packages.

Manuel

PS: Just for some more collateral damage.  Did anybody check whether  
the Mac OS installer support and the -unfortunately, only partially  
working- support to compile for older OS X versions that I added to  
the *makefiles* still works with the Cabal-based system?  I doubt it.  
Took me quite a while to get all this going, and I am not very well  
motivated to spend a lot of time to figure out how it might work with  
Cabal.  IMHO using Cabal for anything but the libraries was a step  
back for no good reason.
_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Manuel M T Chakravarty]

Ross Paterson:

> On Mon, Aug 11, 2008 at 04:17:59PM +0100, Simon Marlow wrote:
>> The main obstacle with just switching the core libraries is that they
>> are shared by other implementations and other maintainers.  So I  
>> see no
>> alternative but to create forks of those repositories for use by GHC,
>> unless/until the other projects/maintainers want to migrate to git.
>
> Forking is much worse than using multiple vcs's, and if we don't fork,
> anyone working on those libraries will have to use git at least to  
> get GHC
> HEAD to check that they're not breaking it.  And clearly GHC  
> developers
> outnumber developers of other implementations.  (I don't think a move
> to git will lead to more GHC developers, but I buy the interns  
> argument.)

Ah, good point!  Changing ghc to git means *all* developers of boot  
libraries need to use git *regardless* of what repo format the boot  
libraries are in.  After all, they need to validate against the  
current ghc head before pushing.

In other words, the decision to move the ghc repo affects all core  
library developers anyway.  No use pretenting that changing only the  
ghc repo (and leaving the rest in darcs) would make anything simpler  
for anybody.

> My concern is that there are rather more developers of libraries and
> assorted other packages, and this will place an arbitrary divide  
> across
> those.  Unless everyone moves to git, of course.

There are surely more developers of libraries in general than there  
are GHC developers.  However, I doubt that there are more developers  
of boot libraries, who are not also ghc developers, than there are ghc  
developers.  The change doesn't have to affect anybody, but ghc  
developers and *core* library developers.

Manuel

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Manuel M T Chakravarty]

Ian Lynagh:
> Even the current situation with Cabal is a bit of a pain, as it's easy
> to forget to push patches upstream as well as GHC's repo, and that's
> just with 2 repos of the same VCS.

As I said before, IMHO it is a big mistake for ghc to depend on  
development versions of Cabal.  GHC should only depend on stable Cabal  
versions.

> I actually think the original plan, where only the ghc repo (plus  
> one or
> two others) is in git is preferable. You may have to use a different  
> VCS
> for different subprojects, but after that it's downhill all the way.
> You don't have to worry about patches being converted from one VCS to
> another, moved to another repo, converted back and merged back into  
> the
> first repo.

Having two vcs for one project is bad.  One reason to switch to git (I  
am told) is that people had problems with darcs on some platforms  
(windows and Solaris, for example).  How is that going to be any  
better if part of the project is still in darcs?  So, can we please  
make up our mind?  If darcs has problems on some platforms, then we  
should not use darcs at all for ghc.  If darcs does not have problems  
on some platforms, then there is one less reason to switch.

All core library developers need to use git anyway to validate their  
core library patches.  So, let's just move the ghc repo and *all core  
libraries* over to git.  If git is good enough for the ghc repo, it  
should be good enough for the core library repos as well, shouldn't it.

Manuel

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Duncan Coutts]

On Tue, 2008-08-12 at 01:15 +0200, Johan Henriksson wrote:

> I see an increasing problem in that every community comes up with
> their own package system, instead of reusing existing frameworks.

That's because there are no usable existing frameworks. It would be
wonderful of course if there were some standard language neutral build
and packaging system where each language just wrote some lib and could
integrate nicely into multi-language systems.

> dependencies to other non-haskell libraries has to be addressed for
> every other coexisting package system (such as apt-get), if it is
> addressed at all. likewise, other languages depending on haskell will
> have trouble resolving dependencies.
>
> so my point is, if there will be any bigger reworking of cabal, I think
> one should consider how it could work as a module in a bigger (maybe
> future) meta-packaging framework, lifting up binaries to for example
> .deb, .exe-installer, .dmg or whatever is the most native for the
> platform.

There are tools to convert Cabal packages to native packages for rpm,
deb, ebuild and arch. The Cabal format was designed to allow this
translation. This includes dependencies on C libs and external programs.

Note that this is in contrast to existing frameworks like autoconf
which do not allow the automatic extraction of dependencies to allow
automatic conversion into native packages.

> I see a point in language specific package systems as they
> have more insight into the build process, but the current
> implementations assume a very ideal world in which there are no other
> dependencies involved.

I don't think this is true.

Duncan

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Donnie Jones]

Hello,

On Mon, Aug 11, 2008 at 8:20 PM, Manuel M T Chakravarty <chak@...> wrote:

Simon Marlow:

Manuel M T Chakravarty wrote:

I think all *core* libraries must switch.  Seriously, requiring GHC developer to use a mix of two vcs during development is a Very Bad Idea.  Don was excited about getting more people to look at the source when it is in git (see the comments he posted from reddit).  By requiring two vcs you will get *less* people to look at the source.
This is not only to get the sources to hack them, but you effectively require developers to learn the commands for two vcs (when they are already reluctant to learn one).  For example, often enough somebody who changes something in GHC will modify the base package, too.  Then, to commit the overall work, you need to commit using both vcs.  If you need to branch for your work, you need to create branches in two vcs (no idea whether the semantics of a branch in git and darcs is anywhere similar).  When you merge your branch, you need to merge in both vcs.  You can't seriously propose such a set up!

I completely agree this is a problem.  The main obstacle with just switching the core libraries is that they are shared by other implementations and other maintainers.  So I see no alternative but to create forks of those repositories for use by GHC, unless/until the other projects/maintainers want to migrate to git.  Some of the repositories are not shared - for example ghc-prim, integer-gmp, template-haskell, and these don't need to be forked.

One way we could create the forks would be to create a git repo for each package with two branches: the master branch that GHC builds, and a separate branch that tracks the main darcs repository, and is synced automatically whenever patches are pushed to the main darcs repo.  We'd have to explicitly merge the tracking branch into the master branch from time to time.  When we want to make changes locally, we can just commit them to the GHC branch and push the changes upstream in a batch later (and then we'd end up having to merge them back in to the GHC branch... but hopefully git's merge is clever enough to avoid manual intervention here).  This is complicated and ugly of course; better suggestions welcome.

Yes, it's a pain.  However, it is better than two vcs for one project.

I *strongly* object to moving to git before this isn't sorted out.  As Roman said before, GHC is heading into a dangerous direction.  It gets progressively harder to contribute to the project at the moment.  First, changing the build system to Cabal.  Now, proposing to use two vcs.  Somebody who is new to the project not only has to learn the internals of GHC, but they also have to learn two new vcs, and if they need to change the build system, they need to learn a new build tool.  Raising the bar for developers to contribute to a project has been proven to be a very bad idea many times.  Let's not take GHC down that path.

I'm not completely convinced we need to have this all worked out before GHC switches, although it would be nice of course.  We currently have infastructure in place for the build to work with a mixture of darcs and git repositories, and existing developers already have to learn git anyway.  They just need to remember to use darcs for libraries and git for the main GHC repo, and this is only a temporary situation.

As far as I am concerned, building GHC is turning into a big mess.  We discussed ways to improve it again, BUT I'd rather not see it getting any messier before it gets better.  Hence, please let's have a complete plan that we are convinced will work before making any more changes.

As for Cabal - we had a thread on cvs-ghc last week, and as I said there we'd love to hear suggestions for how to improve things, including wild and crazy ideas for throwing it all away and starting again.  However, as I explained, there are good reasons for the way things are done now, the main one being that the build system for packages is not written twice.

Yes, we need cabal for packages because we don't want two build systems.  However, this does not justify the use of Cabal outside of libraries/.  Nobody explained to me why that was necessary.  Why change all the rest of the build system.  What is the benefit for the ghc project?

To be honest, if you ask me, I'd go back to the old makefile based system and remove Cabal from everywhere except building of the library packages.

Manuel

I am a new developer with GHC and most of my background is with C programming and Makefile based build systems, such as with the Linux Kernel.  Thus, it was much easier for me to get started hacking on GHC to only need to modify Makefiles as compared to learning an entirely different build system; therefore, I think you lower the barrier to entry for gaining new GHC developers if you stick with the Makefile build system, which is far more common, stable, robust, and definitely taught/used in many university projects.  (Some of these reasons may be the same reasons GHC repo is switching to git.) 

As I said, I am new to hacking on GHC, so I am not sure what reasons there are to switch to Cabal for the build system; but, I am not an expert in build systems and I was able to figure out the GHC Makefiles to add static and run-time flags to GHC, etc.  I definitely think the current Makefile build system could be improved, but overall I did find it quite manageable for my needs.

Disclaimer: I really have very little experience with Cabal other than using it for installing packages, so take everything I have said with a grain of salt.
__
Donnie Jones

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Gour-3]

>>>>> "Manuel" == Manuel M T Chakravarty <chak@...> writes:

Manuel> In other words, the decision to move the ghc repo affects all
Manuel> core library developers anyway.  No use pretenting that changing
Manuel> only the ghc repo (and leaving the rest in darcs) would make
Manuel> anything simpler for anybody.

I'd say that moving GHC to git affects the WHOLE haskell community and
we can already think about having all the HackageDB running git.

I'm not at all pleased with it (personally I switched from darcs to
bzr), bet better to face what the future brings instead of putting the
head in the sand. :-)


Sincerely,
Gour

--

Gour  | Zagreb, Croatia  | GPG key: C6E7162D
----------------------------------------------------------------


_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Simon Marlow-7]

Manuel M T Chakravarty wrote:

> As far as I am concerned, building GHC is turning into a big mess.  We
> discussed ways to improve it again, BUT I'd rather not see it getting
> any messier before it gets better.  Hence, please let's have a complete
> plan that we are convinced will work before making any more changes.
>
>> As for Cabal - we had a thread on cvs-ghc last week, and as I said
>> there we'd love to hear suggestions for how to improve things,
>> including wild and crazy ideas for throwing it all away and starting
>> again.  However, as I explained, there are good reasons for the way
>> things are done now, the main one being that the build system for
>> packages is not written twice.
>
> Yes, we need cabal for packages because we don't want two build
> systems.  However, this does not justify the use of Cabal outside of
> libraries/.  Nobody explained to me why that was necessary.  Why change
> all the rest of the build system.  What is the benefit for the ghc project?

GHC is a package, just like any other.  The GHC package was the main reason
we still had a lot of the old infrastructure for building packages still in
the build system, so there was a compelling reason to switch the compiler
itself to Cabal, at least.

It's true that this change wasn't all win.  We gained in some places and
lost in others - the build system is more unfriendly to developers now, as
opposed to people just building GHC, and that really is something we need
to address.

> To be honest, if you ask me, I'd go back to the old makefile based
> system and remove Cabal from everywhere except building of the library
> packages.

I wouldn't object to dropping the use of Cabal for other tools in the build
tree; the reasons for using it elsewhere are certainly not as compelling as
for packages.

Ian, I realise this means backing out a lot of the work you've been doing
recently, and it would mean that we'd lose a lot of time in the runup to
6.10.1, but perhaps it's a step that we need to take to get us back on the
right track again?

Cheers,
        Simon
_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Johan Tibell-2]

On Tue, Aug 12, 2008 at 2:46 AM, Manuel M T Chakravarty
<chak@...> wrote:
> Ian Lynagh:
> Having two vcs for one project is bad.  One reason to switch to git (I am
> told) is that people had problems with darcs on some platforms (windows and
> Solaris, for example).  How is that going to be any better if part of the
> project is still in darcs?  So, can we please make up our mind?  If darcs
> has problems on some platforms, then we should not use darcs at all for ghc.
>  If darcs does not have problems on some platforms, then there is one less
> reason to switch.

I switched all my Haskell projects over to Git, as a developer on
Linux, is that I wasted way too much time fighting with Darcs that I
should have spent programming. It's way too slow. I've run in to
exponential merges with only two developers commiting to the same
repository. It randomly freezes -- exponential merge, general
sluggishness, who knows! -- or crashes. And merging, perhaps the most
important operation in a DVCS, is a pain. I don't trust Darcs to keep
my source code safe anymore.

Cheers,

Johan
_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Malcolm Wallace]

On 12 Aug 2008, at 01:35, Manuel M T Chakravarty wrote:
> Ah, good point!  Changing ghc to git means *all* developers of boot  
> libraries need to use git *regardless* of what repo format the boot  
> libraries are in.  After all, they need to validate against the  
> current ghc head before pushing.

It is worth pointing out that I *never* validate against ghc head when  
I commit to the core libraries. (Actually, I don't even keep any  
checkout of ghc head.)  Generally I'm fixing something that has  
unintentionally broken the nhc98 build of the libraries, *despite* the  
breaking-patch being validated against ghc.  To be honest I don't  
particularly care if my fixing patch then breaks ghc again.  Why not?  
Because the "chain of blame" effectively leads back past me to the  
earlier patch.  (In practice, re-breaking ghc is very rare.)

Now, there is only one person taking care of nhc98 (me), and probably  
I'm its only user as well, but I do still think it is worth the 30  
secs or so every day it takes to check the nightly build logs and the  
30mins it occasionally takes to fix breakage when necessary.  Building  
a full Haskell'98 compiler is a significant undertaking, and it would  
be a great shame to simply discard it because the libraries are no  
longer available in a shared format.  Who knows, maybe someone will  
find it easier to port to their iPhone than ghc.  :-)

What I'm not really prepared to do is to extend the fixing time by an  
extra 30mins just to validate against ghc.  I might be prepared to  
learn a new VCS, but from what I've seen so far, git looks rather  
complex and difficult to use.

It is also worth noting that where a larger community of developers  
has gathered around a core library (e.g. Cabal), ghc has found it  
necessary to branch off a ghc-only version of that library, so that  
commits to the library head do not need to be validated against ghc  
head.  Igloo takes care of merging across a large bunch of patches  
every once in a while.  This model seems to work well.  In theory, the  
core library head could remain in darcs, with the ghc branch of it in  
git.  All the pain of merging would be dumped on one person (sorry  
Igloo!) but everyone else gets the benefit.

Regards,
     Malcolm

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Simon Peyton-Jones]

| It is worth pointing out that I *never* validate against ghc head when
| I commit to the core libraries.

I think that's perfectly reasonable for the reasons you explain.

Simon
_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Simon Marlow-7]

Matthias Kilian wrote:

> On Mon, Aug 11, 2008 at 04:17:59PM +0100, Simon Marlow wrote:
> [...]
>> As for Cabal - we had a thread on cvs-ghc last week, and as I said there
>> we'd love to hear suggestions for how to improve things, including wild
>> and crazy ideas for throwing it all away and starting again.  However, as
>> I explained, there are good reasons for the way things are done now, the
>> main one being that the build system for packages is not written twice.
>
> Well, at least the Makefile creation was a step (the first step?)
> into the wrong direction, IMHO. I'll run a GHC build to get some
> of those generated Makefiles and followup on cvs-ghc, but for a
> starter, Cabal shouldn't know anything about implementation-specific
> internal build systems; instead it should rely only on it's own
> metadata.

I'm not completely sure, but I think you may have misunderstood how Cabal's
makefile generation currently works.  It has no specific knowledge of GHC's
build system, and it does rely on its own metadata.  (in my other message
I'm suggesting moving the Makefile generation into GHC's build system so
that it could be made specific to GHC, though).

>  Implementation-specific stuff (such as how to run the
> compiler) should be supplied by the implementation, not by Cabal.

This is what makes me unsure.  Implementation of what?  Are you suggesting
a redesign of Cabal, or just changing the way something works?

Cheers,
        Simon

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Ian Lynagh]

On Sun, Aug 10, 2008 at 08:17:50PM -0400, Norman Ramsey wrote:
>  > On Sat, Aug 09, 2008 at 06:56:23PM -0400, Norman Ramsey wrote:
>  > >
>  > personally I would much prefer to see money spent on making darcs
>  > better, for reasons I won't repeat again.
>
> I missed them and wouldn't mind receiving a private note.

OK, I'll send to the list so that I have somewhere convenient to point
people if this comes up in the future:

* A lot of darcs's functionality could be refactored into generally
  usable Haskell libraries, e.g. LCS-finding, downloading-with-libcurl.

* darcs was once a flagship Haskell application, supporting the idea
  that Haskell can be used in the real world. That image has mostly
  faded away now due to the problems it has, but I think we can get it
  back if we can get a high quality darcs out there. That would be good
  for the community's image.

* darcs has (in my opinion, at least) a much simpler, more intuitive
  interface than the other version control systems. I don't think I'm
  alone here, as I think this is where a lot of the resistance against
  moving to git is coming from.

* I think darcs is the Obvious, Right way to do version control.
  Phil Wadler (at least, I think it was him; and probably many others
  too) has said that the lambda calculus is universal, in the sense that
  if we were to meet a sufficiently advanced alien culture, it is almost
  inconceivable that they would not have also discovered the lambda
  calculus. Darcs-style patch theory, before conflicting patches are
  introduced, falls into the same category in my opinion. (I'm not yet
  sure if it can be extended to include some definition of conflictors
  too). By contrast, the heuristics and multiple merge algorithms of
  other systems feels very ad-hoc.


Thanks
Ian

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Ian Lynagh]

On Tue, Aug 12, 2008 at 10:10:31AM +0100, Malcolm Wallace wrote:
>
> On 12 Aug 2008, at 01:35, Manuel M T Chakravarty wrote:
> >Ah, good point!  Changing ghc to git means *all* developers of boot  
> >libraries need to use git *regardless* of what repo format the boot  
> >libraries are in.  After all, they need to validate against the  
> >current ghc head before pushing.
>
> It is worth pointing out that I *never* validate against ghc head when  
> I commit to the core libraries.

Also, all of the people who send us patches don't need to validate (and
I suspect most of them don't), as we validate the patches before pushing
them.


Thanks
Ian

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Duncan Coutts]

On Mon, 2008-08-11 at 14:29 +0200, Thomas Schilling wrote:

> On 11 Aug 2008, at 13:00, Duncan Coutts wrote:
> > It's not clear to me that we've really bothered to find out. The last
> > evaluation in relation to ghc that I'm aware of was prior to the 2.0
> > release. My impression is that we've all complained about the darcs v1
> > problems (justly) but spent the most effort investigating things other
> > than darcs v2 which would be the easiest to upgrade to and not have  
> > the
> > problems of using two different systems for ghc vs other libs.
>
> I converted the ghc repo to darcs2 (locally):
>
> Getting file local history:
>
>   * darcs changes --last 20 compiler/main/HscTypes.lhs
>
>     very quick but prints only two patches
>
>   * darcs changes compiler/hsSyn/HsTypes.lhs
>
>     1m22s  (16s for the second time)

Interesting that you get so much variance between runs. I get 32s user
time first time and 30s the second.

In this test darcs 2 is faster that darcs 1 on v1 format repos and darcs
2 is faster on v2 format repos than on v1 format, though only by a few
seconds.

At a guess, the issue here is that darcs is not indexing those changes
per-file, which is why --last 20 doesn't give the last 20 for that file
and why asking for all changes takes so long. Perhaps if it did cache
this info per-file it'd help with annotate too.

>     Git <1s
>
>   * darcs get ghc2  ghc-test  (creating a *local* branch)
>
>     real    13m25.365s
>     user    0m14.677s
>     sys     0m29.541s
>
>     (at least it seems it actually worked, though)

That's an order of magnitude different to what I see:

$ time darcs2 get ghc2 ghc-test
Copying patches, to get lazy repository hit ctrl-C...
Finished getting.                      
                           
real 0m21.428s
user 0m11.221s
sys 0m1.380s

Note that this is much faster in the darcs v2 format than darcs 2 using
the darcs v1 format:

$ time darcs2 get ghc ghc-test1
Finished getting.                                                  
                         
real 1m51.959s
user 1m15.449s
sys 0m11.877s

However darcs v1 is faster still:

$ time darcs1 get ghc ghc-test1_

Copying patch 19084 of 19084... done.
Finished getting.

real 0m8.851s
user 0m3.668s
sys 0m0.708s

It doesn't seem to spend any time applying the patches, unlike what
darcs 2 is doing for v1 or v2 formats.

Though in any case, one doesn't need to darcs get locally since one can
use cp -a right?

>     git clone ghc g2  (the slow method of creating a local branch)
>
>     real    0m6.742s
>     user    0m0.335s
>     sys     0m0.652s
>
>   * I haven't tested a remote pull yet.  At 80 Kb/s, it should take  
> about 15min to clone via Git (70 MB).  A test of darcs would be  
> interesting.

We'll be testing this for the code.h.o conversion. We'll keep you
posted.

Duncan

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Ian Lynagh]

On Tue, Aug 12, 2008 at 10:20:14AM +1000, Manuel M T Chakravarty wrote:

>
> To be honest, if you ask me, I'd go back to the old makefile based  
> system and remove Cabal from everywhere except building of the library  
> packages.
>
> Manuel
>
> PS: Just for some more collateral damage.  Did anybody check whether  
> the Mac OS installer support and the -unfortunately, only partially  
> working- support to compile for older OS X versions that I added to  
> the *makefiles* still works with the Cabal-based system?  I doubt it.  
> Took me quite a while to get all this going, and I am not very well  
> motivated to spend a lot of time to figure out how it might work with  
> Cabal.  IMHO using Cabal for anything but the libraries was a step  
> back for no good reason.

Do you mean the "rebuilding the tools with stage2" stuff? If so, that's
an interesting example to pick, as that was the impetus behind changing
how the build system worked for all the non-libraries/ghc.

Those changes made the build non-idempotent: we would build something
with the bootstrapping compiler, build some other stuff, then come back,
clean it, and build it again with the in-tree compiler. This was a
little annoying at the best of times, as e.g. rerunning make at the top
level would needlessly rebuild some stuff.

However, when my local changes meant that programs built by GHC
segfaulted, it was especially irritating to find that after (hopefully)
fixing the bug I couldn't just run make in compiler/ or rts/, because
ghc-pkg etc now just segfaulted!

It was at that point that I half-reverted the changes, and later I
reimplemented something similar using Cabal. Now we make, for example,
ghc-pkg with the bootstrapping compiler in utils/ghc-pkg/dist-inplace,
and then later on we make it with the stage1 compiler in
utils/ghc-pkg/dist-install.


To answer your actual question: No, not having OS X yet I haven't tested
it, but I did make an effort to keep it working. In mk/cabal-flags.mk we
say:

USE_STAGE_CONFIGURE_FLAGS = \
    ...
    $(addprefix --cc-option=,$(MACOSX_DEPLOYMENT_CC_OPTS)) \
    $(addprefix --ld-option=,$(MACOSX_DEPLOYMENT_LD_OPTS))

which will hopefully do the trick, and (IMO) in a much cleaner, more
maintainable way than would have been possible with the old build
system.


Thanks
Ian

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Simon Peyton-Jones]

Friends

| > I see more and more workarounds for workarounds for an unmaintainable
| > (and unusable) build system, and after the latest discussions about
| > git vs. darcs, maintaining GHC-specific branches of libraries etc.,
| > I think I'll just drop maintainership from all GHC-related OpenBSD
| > ports until the GHC build system chaos settles down a little bit.
|
| Ian, please read this.
| The inability to build GHC reliably is a problem.
|
| Can someone with a plan please describe what measures are in place
| to ensure GHC emerges buildable, and the tree regains the status of a
| tree that *does not break*?

I don't think we should over-react here. There's been lots of email on
this thread, some of which IMHO makes things sound rather worse than
they really are.

Let me say how it looks to me. There are two separate but
loosely-related conversations going on.

1.  Changes to GHC's build system.

Cabal is used to build Haskell libraries.  We started to use it to
build the libraries that come with GHC; and we recently moved over to
Cabal to build GHC itself (which is, these days, just another library).
The old makefile-based system was essentially duplicating much of
the functionality of Cabal, and that duplication was painful.

In retrospect, we should have made this change in a branch, and tested
it thoroughly before applying it to the HEAD.  Build systems tend to
be platform dependent, so testing on one platform isn't enough.  Nor
did we consult, or even communicate, enough before going ahead.  And
we need more Wiki documentation about how to drive the new system.
The net effect of these omissions has been a lot of pain to our
collaborators.  I am very sorry about that.

But I think it'd be a pity to confuse the pain of transition with the
destination.  The build system is settling down.  For the moment, it
probably makes sense not to aggressively pull patches from the GHC
repo if you don't have to, but we absolutely do not expect that
situation to persist.  We'll make an announcement when we're ready for
you to give it a try.  The clear goal is: it simply builds flawlessly.

There is an element of "dogfoooding" here.  GHC is a stress test for
what Cabal can do, and is itself not fully mature.  But the pain we
experience thereby leads to bug-fixes and significant features for
Cabal that are useful for everyone.

Perhaps we made the move too early though!  The new design is not
set in stone, and we are actively thinking about ways to improve it,
*including* backing off from Cabal in places where it appears too
inflexible.  Of course, any such further changes would extend the
period of upheaval, but (a) we'll publish a design before executing,
and (b) we'll do it on a branch.



2.  The version control system (VCS)

At the same time, we had an extended conversation about changing the
version control system we use for GHC.  There was a lot of
consultation here, as a result of which we chose git. I won't rehearse
again the reasons we are unhappy with darcs, except to say that darcs
is a thing of beauty, but the scale of GHC's repository seems to flush
out many darcs bugs and performance problems that have proved difficult
to fix.

Unlike the build system, we have not yet executed this decision.  In
particular, the earlier discussions focused mainly on the relative
merits of the various systems.  But it's more complicated than that.

GHC needs "core libraries" without which it cannot be built.  It is
obviously highly desirable that a developer can build GHC with just
one VCS, which suggests that the core libraries should be in git too.
But those same core libraries are used by nhc98 and Hugs (I think
that's all), and the last thing we want to do is to impose new costs
on other implementations.  Diversity in implementation is a Good Thing.

It's unclear exactly what to do about this.  The most plausible
possibility is to keep the core libraries that are shared with other
implementations in darcs as now, and mirror them in git for GHC
developers.  That will impose pain on GHC developers to keep the git
stuff in sync with the darcs master copies; but at least other
developers would be unaffected.

It's a hard judgement call to say which pain is greatest: the pain of
staying with darcs, or the pain of managing the two-VCS problem.
Regardless, though, if all you want to do is build GHC from scratch,
it absolutely will be a question of getting the relevant VCS,
installing support software (Happy, Alex, an earlier GHC), and typing
'make'. You won't have to know about funny branches.


We (GHC HQ) are still learning the transition to wider participation
in building and hacking on GHC, which we *very much* welcome.  Bear
with us if we don't get it right first time.  We're trying!

Simon
_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

[Eric-182]

A metacomment:

As a lurker, and a reader of other languages' mail groups, I just wanted to complement you GHC folks on the quality of your discussion.  You're discussing an issue that people clearly have strong opinions about, yet you've all remained polite and respectful and kept the signal-to-noise ratio of the discussion high, unlike what I see on some the mail groups of some other languages.

It's a pleasure reading even your disagreements.

--Eric




     
_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@...
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users