about GNU Hurd

[Richard Stallman]

       GNU Mach is not fundamentally obsolete or bad, I really do not
       understand where people get this idea from.

    That's what is said about GNU Mach...

According to what Marcos told me, GNU Mach is fundamentally incapable
of properly doing the job we had in mind for it to do.  This does not mean
it is fundamentally useless for all uses.  It might be fine on a computer
only used by one person, according to my memories of the issues.
But that won't really excite people, I think.  So we need Hurd-NG
for a real success.

[Richard Stallman]

    Once again, I totally agree with you and I repeat, till we
    fix/focus on our organizational problems, nothing good can emerge
    from current development (sorry to think/to say that, it is not
    targeted at current active Hurd developers).

In a volunteer project, the only way to improve the organization is if
a person shows up who can, and wants to, play a role in the
leadership.

Hurd-NG consists of some new programs, plus most of the existing Hurd
servers with small changes.  For the long term, that's our roadmap to
a system that might really excite people and thus achieve a big
success.

If there is one of you who would like more responsibility in improving
the existing Hurd servers, that could be a way to improve progress
here, using the resources we have, in a way that advances the existing
Hurd as well as contributing to Hurd-NG.

[Richard Stallman]

      Those flaws could also be fixed without rewritting everything,
      as was shown by several people including the people who actually
      wrote the Hurd.  Sadly, nobody wishes to waste time working on
      something that will (not might, but will) be discarded.

They never showed this to me.

[Richard Stallman]

    Marcus said this for Hurd/L4 at the start, Hurd-NG is a completely
    different beast and completely different.

Marcus told me this week that some low-level parts of the Hurd need to
be redesigned for Hurd-NG, but lots of servers would just be ported.

[Alfred M. Szmidt]

       Marcus said this for Hurd/L4 at the start, Hurd-NG is a
       completely different beast and completely different.

   Marcus told me this week that some low-level parts of the Hurd need
   to be redesigned for Hurd-NG, but lots of servers would just be
   ported.

I'd really like to see that thread, could you forward it to me?
Marcus seems to be changing things back and forth to fast for people
to follow, last time I heard anything about the topic (about a month
ago), Hurd-NG would be a complete redesign of the Hurd, discarding all
code.

Olaf, you tend to be up to par on this topic, could you elaborate?

[Alfred M. Szmidt]

          GNU Mach is not fundamentally obsolete or bad, I really do
          not understand where people get this idea from.

       That's what is said about GNU Mach...

   According to what Marcos told me, GNU Mach is fundamentally
   incapable of properly doing the job we had in mind for it to do.

This begs the question, what do we have in mind?  What should GNU
strive towards?

   This does not mean it is fundamentally useless for all uses.  It
   might be fine on a computer only used by one person, according to
   my memories of the issues.  But that won't really excite people, I
   think.  So we need Hurd-NG for a real success.

There are a couple multi-users systems out there that run GNU/Hurd,
and they run suprisingly well.  Barry deFreese maintains them, he
share his insights on that topic.

[olafBuddenhagen]

Hi,

On Sun, Sep 02, 2007 at 10:05:48PM -0600, Michael Heath wrote:

> I think the problem people face is: Why work on something that is
> already fundamentally obsolete or bad?

It is not any more fundamentally obsolete or bad than most other
software out there.

We *know* that there are some fundamental issues with the current
implementation. We *know* for example that performance will never quite
match modern monolithic systems; that there are problems that can not be
fully solved within the current design.

However, we do not know how to solve these. Nobody knows. We don't have
any working design for the Hurd better than the current one. Marcus and
Neal at some point believed that they have, and unfortunately were
pretty vocal about it -- resulting in the false believe that the current
implementation is useless and any work on it wasted :-( But that's not
the case.

The current implementation has issues, but nobody know how to solve them
-- not Marcus, not Neal, Shapiro, nor anyone else on this planet. To
some of the issues there are know solutions in specific contexts, but
it's uncertain how they can be applied to a Hurd design. Others are
totally open questions. What Neal and Marcus are currently doing is
bleeding edge research work. They have some ideas to test, but no
complete working design.

We know the current Hurd has issues; yet it is still the best known
working design for the Hurd -- in fact, it's the only working Hurd
design that we know. You think we want to abandon that?

> Why not focus on the reimplementation you know has to come?
>
> And because questions of what that reimplementation is and how people
> can help with it keep being answered with "I don't knows", "Don't you
> worry about that", or half answers, people aren't excited to help with
> anything.

Well, these are about the best answers you can expect. We really hope
that Neal's and Marcus' research will result in something that will
prove useful for the Hurd; will result in improvments to the Hurd
desingn and implementation at some point in the future. But it's
research. There is not some concrete system being worked on, or even a
complete design for one; there is nothing really people could help with
-- unless you want to seriously go into microkernel-related operating
system research.

Seriously means: Working on it more or less full time, for many months
at least, more likely years; to read hudreds of research papers; to
spend hours and hours, months and months contemplating known as well as
possible new approaches; trying to come up with useful designs; testing
new ideas...

(People who really are into this, can get an idea what Neal and Marcus
are presently working on from some of the more recent posts on the
l4-hurd mailing list.)

But even then, even if you really want to help with this research work,
you can hardly contribute anything useful, unless you already spent a
considerable time working on the current implementation -- understanding
what it's about, how it works, where the shortcomings are. You can't
come up with an improved design without really knowing the previous one.

Regarding the reimplementation you are talking about, I don't really
believe the current Hurd will ever be replaced by a total
reimplementation; not if I can prevent it. We must expect; in fact hope,
that sooner or later there will be some major changes to the lower-level
parts of the system. That doesn't mean all existing code (especially
higher-level code) gets thrown away; and more importantly, it doesn't
mean all knowledge aquired, all experience gained with the existing
implementation gets thrown away.

If you want to help the Hurd, work on the existing system -- that's
crucial not only for the current implementation, but also to any
possible future improvements. We have a working system with a small but
active community around it. I'm sure anyone can see how incredibly silly
it would be to abandon that on the prospect of some possible future
design improvements.

> Other projects have a clear road map, clear set of design goals, and a
> clear structure on who is in charge of what - why doesn't the Hurd and
> associated projects?
>
> Road maps are important.

I totally agree that a roadmap would be immensly helpful: A roadmap for
improving the existing implementation. But not a roadmap for research on
possible future design options.

-antrik-

[olafBuddenhagen]

Hi,

On Mon, Sep 03, 2007 at 08:29:55PM -0600, Michael Heath wrote:

> That isn't the case with the Hurd. Not many people use it, and not
> many people will use it until the improvements come that will only
> come with major progress.

The reason people are not using the Hurd, is not because it runs on
Mach, or any other shortcoming in the current design.

The reasons are that it is not complete and polished enough to make it a
viable system for people to use seriously; and that it lacks most of the
infrastructure necessary to enable users to actually benefit from the
advantage of the Hurd architecture -- there is not really much incentive
for people to switch.

Both of these aspects can only be addressed by improving the existing
implementation, not by musing about possible future design changes.

-antrik-

[Michael Heath]

Thanks for the response, antrik.

You need to realize that I was speaking mainly in my e-mail about the combination of Hurd + Mach, not just one or the other. I am well aware that continued work on higher-level components of the Hurd is not wasted time or energy, and, as Richard suggested in his e-mail earlier, work on those components would be greatly appreciated.

It's understandable that the issues here and how to fix them are so complex and tedious that there are not any clear answers yet. What I was mainly saying, however, in my previous e-mails is that very little real progress can be made until we find such answers, because the current problems both directly limit progress and limit interest in the project.

I'm going to disagree with you on your statement that nothing useful can be done on new work or research except for those few who are intimately familiar with the current internal workings of Mach and the Hurd. There is ALWAYS lots of helpful work that people of various talents can do to help a project.

I don't know what needs to be done, or exactly what it is that people can help with. I'm just suggesting that we step back and take a closer look at the current status of the project, and what needs to and can be done to improve it.

If it is truly the consensus that this is indeed going to take a long, long time before it can be fundamentally fixed, perhaps the GNU project SHOULD investigate using another kernel, at least until these problems are fixed.

Michael Heath

On 9/5/07, olafBuddenhagen@... <olafBuddenhagen@...> wrote:

Hi,

On Sun, Sep 02, 2007 at 10:05:48PM -0600, Michael Heath wrote:

> I think the problem people face is: Why work on something that is
> already fundamentally obsolete or bad?

It is not any more fundamentally obsolete or bad than most other
software out there.

We *know* that there are some fundamental issues with the current
implementation. We *know* for example that performance will never quite
match modern monolithic systems; that there are problems that can not be
fully solved within the current design.

However, we do not know how to solve these. Nobody knows. We don't have
any working design for the Hurd better than the current one. Marcus and
Neal at some point believed that they have, and unfortunately were
pretty vocal about it -- resulting in the false believe that the current
implementation is useless and any work on it wasted :-( But that's not
the case.

The current implementation has issues, but nobody know how to solve them
-- not Marcus, not Neal, Shapiro, nor anyone else on this planet. To
some of the issues there are know solutions in specific contexts, but
it's uncertain how they can be applied to a Hurd design. Others are
totally open questions. What Neal and Marcus are currently doing is
bleeding edge research work. They have some ideas to test, but no
complete working design.

We know the current Hurd has issues; yet it is still the best known
working design for the Hurd -- in fact, it's the only working Hurd
design that we know. You think we want to abandon that?

> Why not focus on the reimplementation you know has to come?
>
> And because questions of what that reimplementation is and how people
> can help with it keep being answered with "I don't knows", "Don't you
> worry about that", or half answers, people aren't excited to help with
> anything.

Well, these are about the best answers you can expect. We really hope
that Neal's and Marcus' research will result in something that will
prove useful for the Hurd; will result in improvments to the Hurd
desingn and implementation at some point in the future. But it's
research. There is not some concrete system being worked on, or even a
complete design for one; there is nothing really people could help with
-- unless you want to seriously go into microkernel-related operating
system research.

Seriously means: Working on it more or less full time, for many months
at least, more likely years; to read hudreds of research papers; to
spend hours and hours, months and months contemplating known as well as
possible new approaches; trying to come up with useful designs; testing
new ideas...

(People who really are into this, can get an idea what Neal and Marcus
are presently working on from some of the more recent posts on the
l4-hurd mailing list.)

But even then, even if you really want to help with this research work,
you can hardly contribute anything useful, unless you already spent a
considerable time working on the current implementation -- understanding
what it's about, how it works, where the shortcomings are. You can't
come up with an improved design without really knowing the previous one.

Regarding the reimplementation you are talking about, I don't really
believe the current Hurd will ever be replaced by a total
reimplementation; not if I can prevent it. We must expect; in fact hope,
that sooner or later there will be some major changes to the lower-level
parts of the system. That doesn't mean all existing code (especially
higher-level code) gets thrown away; and more importantly, it doesn't
mean all knowledge aquired, all experience gained with the existing
implementation gets thrown away.

If you want to help the Hurd, work on the existing system -- that's
crucial not only for the current implementation, but also to any
possible future improvements. We have a working system with a small but
active community around it. I'm sure anyone can see how incredibly silly
it would be to abandon that on the prospect of some possible future
design improvements.

> Other projects have a clear road map, clear set of design goals, and a
> clear structure on who is in charge of what - why doesn't the Hurd and
> associated projects?
>
> Road maps are important.

I totally agree that a roadmap would be immensly helpful: A roadmap for
improving the existing implementation. But not a roadmap for research on
possible future design options.

-antrik-

[Richard Stallman]

    > Is there anyone here who has the skills to install the large file
    > system support, and wants to do it?

    I will take care of this issue.

Thank you very much.

[Richard Stallman]

    You can just merge ams-branch to HEAD, will save you alot of work.  It
    still contains a slew of patches and bug fixes that have not been
    commited.

These patches and fixes have to be considered one by one.
If they have been put together in one branch, that means
he has to disentangle them in order to consider them.

[olafBuddenhagen]

Hi,

On Wed, Sep 05, 2007 at 05:47:56PM +0200, Wolfgang Jährling wrote:

> > And as Olaf pointed out, nobody, not even Marcus, knows what Hurd-NG
> > will look like, there is no goal, not even an idea behind it.
>
> I don't think this is true.  The documents I found here[0] and here[1]
> gave me a lot of information about the direction of Hurd-NG.
>
> [0] http://lists.gnu.org/archive/html/l4-hurd/2007-01/msg00007.html
> [1] http://www.bddebian.com/~wiki/hurd/ng/philosophy/

Unfortunately, these documents no longer describe the current
direction... Especially the wiki. (The position paper is somewhat
newer.)

The wiki piece was created at the time when ngHurd on Coyotos was
considered the way forward by some (not me) -- but this is no longer the
case. I wonder whether this part of the wiki wouldn't be better
discarded alltogether; at this point it is pretty misleading I think.

In the meantime, Marcus admitted that he got carried away with Coyotos.
He realized that the changes necessary to the Coyotos design to match
Hurd goals would be much bigger than he originally hoped; and that it
must be so, as not only specific goals of Coyotos are different, but in
fact the whole mindeset behind the system is a totally different one
from GNU and the Hurd.

He also realized that while security is important, the kind of security
offered by Coyotos goes much beyond what the Hurd really needs. And last
but not least, he realized that Coyotos doesn't really solve many of the
existing problems either.

The immediate result is that the original ngHurd plans based on Coyotos,
as described in the wiki, are no longer pursued. However, there are
other results as well. Besides of the more or less technical conclusions
described above, Marcus also learned some other things in the process:
For one, he seems to have realised now that it's not really possible to
come up with a perfect design by brooding long enough and then just
implementing that... So he changed his strategy: Now he works on a
prototype to test his current ideas (which are mostly based on Coyotos
and seL4 I understand), consciously postponing the other open design
questions, not trying to build a complete new Hurd at once.

Another thing he seems to have understood is that it's not a good idea
to go public with untested or even unfinished designs, claiming they are
better than the existing one. Rather, he tries to test his new ideas
himself first. Which is why you hear little about his present work. And
this is a good thing. This whole discussion shows that the way he used
to push his ideas turned out disasterous for the Hurd community, leaving
people interested in the Hurd pretty much in the void...

-antrik-

[olafBuddenhagen]

Hi,

On Wed, Sep 05, 2007 at 08:15:37PM +0200, Alfred M. Szmidt wrote:

> Well, these documents are old, and this is what
> http://www.bddebian.com/~wiki/hurd/ng/ says in the first paragraphs:
>
> | There is an effort to create a completely new system design (for now
> | called ngHurd, or Hurd-ng or HurdNG), which originated from the
> | Hurd/L4 port.
>
> | The original Hurd/L4, which was meant to be mostly a direct port of
> | the existing Hurd design to a new microkernel, has been abandoned by
> | it's main developers, because some technical issues with the L4
> | Pistachio kernel turned out to be very fundamental. While
> | reeveluating the design (and upcoming new L4 variants), the
> | developers picked up some new ideas, and decided that now they
> | rather want to work on a completely different design, which combines
> | some of the original Hurd ideas with concepts from Jonathan
> | Shapiro's high security EROS and Coyotos systems.

Note that these paragraphs were written by me. They represent my view on
the Coyotos-based ngHurd efforts; not the view of the active
participants. But as you said yourself, this stuff is outdated anyways.

> I guess some sorting out what HurdNG actually is needs to be done...

Originally it referred to the Coyotos-based design, which would have
been a totally different system from the existing Hurd, only remotely
based on some Hurd ideas. I prefer reserving this term only for such a
totally different design, and using Hurd/<whatever> to refer to any port
of the existing Hurd to a new microkernel, changing mostly the low-level
bits and introducing only minor refinements otherwise, but preserving
the rest of the code, part of the existing architecture, and -- most
importantly -- the spirit of the existing Hurd.

However, pretty much anyone else seems to use ngHurd to refer to any new
design Marcus happens to come up with, making the term pretty
meaningless.

I really don't know in which direction Marcus currently tends. But IMHO
it doesn't really matter in regards to the ongoing work on the exiting
Hurd. If Marcus comes up with something close to the current
Implementation, most the existing stuff can be moved over. If he comes
up with a totally different system: Well, it will be a totally different
system -- not likely ever to replace the existing Hurd as GNU's official
kernel. In either case, work on the current Hurd implementation won't be
wasted.

-antrik-

[olafBuddenhagen]

Hi,

On Tue, Sep 04, 2007 at 07:38:24PM +0200, Alfred M. Szmidt wrote:

> Both L4 and Coyotos use synchrony message passing, while Mach uses
> asynchronous message passing.  The Hurd servers depend on the
> functionality of async. message passing so just rewriting them to
> become synchronous would be quite a big problem.  There are many other
> differences between Mach, L4, Coyotos, ...  So it is not at all
> trivial, and will require huge changes.

Well, Shapiro abandoned the attempt to implement a partially-async IPC
mechanism in Coyotos; but Neal and Marcus are sticking with the idea,
and it is part of Marucs' prototype kernel.

The thread model of the Hurd servers will probably need to change sooner
or later for better performance; but this is pretty orthogonal to a
possible microkernel change. The advantage of writing our own kernel is
that we can implement the features that *we* think most useful, instead
of building a system around some existing kernel.

So, async vs. sync IPC is not really an issue. How much the existing
code will need to be adapted depends on different questions.

-antrik-

[olafBuddenhagen]

Hi,

On Wed, Sep 05, 2007 at 10:36:23PM +0200, Alfred M. Szmidt wrote:

>    Marcus told me this week that some low-level parts of the Hurd need
>    to be redesigned for Hurd-NG, but lots of servers would just be
>    ported.
>
> I'd really like to see that thread, could you forward it to me? Marcus
> seems to be changing things back and forth to fast for people to
> follow, last time I heard anything about the topic (about a month
> ago), Hurd-NG would be a complete redesign of the Hurd, discarding all
> code.
>
> Olaf, you tend to be up to par on this topic, could you elaborate?

I don't know much details, and haven't heard a direct statement on this
from Marcus for ages. I'm pretty sure Richard's information is more
correct than anything I could write.

All I can say is what I told you already several months ago: That Marcus
seems to have become more "conservative" again. My impression is that he
no longer wants to rewrite everything to create a perfect system, but
rather focuses on the areas that most need improvement, and otherwise
tries to keep as much as possible.

-antrik-

[olafBuddenhagen]

Hi,

On Wed, Sep 05, 2007 at 09:10:13PM -0600, Michael Heath wrote:

> It's understandable that the issues here and how to fix them are so
> complex and tedious that there are not any clear answers yet. What I
> was mainly saying, however, in my previous e-mails is that very little
> real progress can be made until we find such answers, because the
> current problems both directly limit progress and limit interest in
> the project.

No, they do not. The fundamental issues with Mach are *not* the reason
why we are stuck with drivers from Linux 2.0. They are *not* the reason
why all disk I/O is like ten times slower due to lack of optimization.
(They are responsible for bad performance in *some* areas, but not the
bulk of it.) They are *not* the reason why we have no sound support, no
USB support, no SATA support etc. And they are *not* the reason why
nobody bothered to write software that allows making use of Hurd's
interesting features. They are not the reason for any of the major
problems of the current Hurd implementation.

All of these things can and should be fixed within the current
implementation. None of these things will get magically fixed by an
improved design that might come up some day. This is not a matter of
perfect design; this is a matter of actually doing the grunt work.

> I'm going to disagree with you on your statement that nothing useful
> can be done on new work or research except for those few who are
> intimately familiar with the current internal workings of Mach and the
> Hurd. There is ALWAYS lots of helpful work that people of various
> talents can do to help a project.

Like what?

There *are* endless things that people can do to help the Hurd though:
By working on the current implementation. (Coding, mentoring, porting
packages, testing, documenting, communicating to the outside,
managing...)

But none of this is in demand for Marcus' research work. Really, I can't
think of anything that people could do -- unless they have a very good
understanding of the Hurd and/or of microkernel design -- to help his
efforts at the present stage.

And to be honest, I don't see why most people would even want to. To
advance the Hurd, it's so much more important to help fixing the
pressing issues in the existing implementation... Give people something
that they can actually use, instead of dwelling on the prospect of an
even more advanced architecture.

-antrik-

[Alfred M. Szmidt]

   Hi,

   On Sun, Sep 02, 2007 at 10:05:48PM -0600, Michael Heath wrote:

   > I think the problem people face is: Why work on something that is
   > already fundamentally obsolete or bad?

   It is not any more fundamentally obsolete or bad than most other
   software out there.

   We *know* that there are some fundamental issues with the current
   implementation. We *know* for example that performance will never quite
   match modern monolithic systems; that there are problems that can not be
   fully solved within the current design.

This depends on what issues you are refering to.  The two
(semi-serious) problems that Marcus pointed out wrt to passive
translators and firmlinks can be solved.  As was pointed out by
Bushenell, and someone else.

[Alfred M. Szmidt]

       You can just merge ams-branch to HEAD, will save you alot of
       work.  It still contains a slew of patches and bug fixes that
       have not been commited.

   These patches and fixes have to be considered one by one.  If they
   have been put together in one branch, that means he has to
   disentangle them in order to consider them.

They were already considered one by one, so that job is already done.

[olafBuddenhagen]

Hi,

On Thu, Sep 06, 2007 at 09:12:02PM +0200, Alfred M. Szmidt wrote:

>    We *know* that there are some fundamental issues with the current
>    implementation. We *know* for example that performance will never
>    quite match modern monolithic systems; that there are problems that
>    can not be fully solved within the current design.
>
> This depends on what issues you are refering to.  The two
> (semi-serious) problems that Marcus pointed out wrt to passive
> translators and firmlinks can be solved.  As was pointed out by
> Bushenell, and someone else.

I was referring to more general issues.

There is no accounting of resources used by applications, and Mach lacks
the facilities to implement that. IPC is slow, and can't be fixed
without simplifying the semantics. Passive translators are way too
limited. (The specific issues pointed out with passive translators can
all be solved by specific corrections or workarounds I believe; but in
the long run we need a more powerful mechanism IMHO.) Some of the Hurd
server interfaces are too complicated, and unnecessarily store
per-client state in the server, making interposition very hard.
Userspace drivers are not possible for most hardware because of missing
kernel facilities. The thread model of Hurd servers (one thread per
request) is fundamentally wrong, making for poor performance and
resource utilisation. (And making resource accounting much harder...)
Some functionality is unnecessarily implemented in complicated and
inefficient servers that handle multiple connections, although no
coordination is necessary between the individual connections.

All of these issues are real, and we will have to address them sooner or
later. Only I don't think any of them is very urgent -- none of them
prevents the existing Hurd from being a useful system, once the more
pressing implementation issues are fixed.

-antrik-

[Richard Stallman]

    The reasons are that it is not complete and polished enough to make it a
    viable system for people to use seriously; and that it lacks most of the
    infrastructure necessary to enable users to actually benefit from the
    advantage of the Hurd architecture

The second sound like something people could work on now
that would carry over easily to Hurd-NG when that is ready.
A lot of the first point also could do that.