Coda newbie

[Matias Surdi]

Hi,

I'm going to test some distributed filesystems, and by now my two
options are OpenAFS and Coda.

Is coda in active development?

Would you recommend to use it under FreeBSD or Debian?


Thanks a lot.

[Jan Harkes]

On Mon, Jul 21, 2008 at 07:03:50PM +0200, Matias Surdi wrote:
> I'm going to test some distributed filesystems, and by now my two  
> options are OpenAFS and Coda.
>
> Is coda in active development?

There is still active development, but it isn't going terribly fast.
OpenAFS definitely has a larger community behind it.

> Would you recommend to use it under FreeBSD or Debian?

There was a good push to get a working Coda release out for the most
recent FreeBSD-stable release. I think it works pretty well now. I tend
to mostly use a Debian based system myself, but haven't heard complaints
from people using Coda on Fedora or Gentoo either. I think there are
also at least two people using NetBSD.

Jan

[u+codalist-wk5r]

Hi,

On Fri, Jul 25, 2008 at 03:56:14PM -0400, Jan Harkes wrote:
> > Would you recommend to use it under FreeBSD or Debian?
>
> There was a good push to get a working Coda release out for the most
> recent FreeBSD-stable release. I think it works pretty well now. I tend

IIRC there still were some issues under Linux ABI on FreeBSD, namely
that directories on Coda could not be read beyond the first block/page
(opening files works even if their names are not on the first page in
the directory).
I guess the glitch is not in Coda but in the ABI layer.

> to mostly use a Debian based system myself, but haven't heard complaints
> from people using Coda on Fedora or Gentoo either. I think there are

If the distribution does not cripple the kernel, Coda will work fine.

Some distros' packages can be old or broken. (e.g. Gentoo ebuild has been
broken for a long time, placing Coda mount at some other place than /coda.
2008-07-12 I got a reply to my bug report from 2006-08-17 - note the years.
Now there is an experimental ebuild which adds a symlink /coda -> ....
so it _may_ work on Gentoo now)

The universal installer (http://www.aetey.se/index.php?Static&pg=CodaInstHowto)
is your friend on any distro.

Regards,
Rune

[Andy Valencia-3]

--------
[u+codalist-wk5r@... writes:]

> If the distribution does not cripple the kernel, Coda will work fine.

FYI, for FreeBSD 7.0 I found that the latest Coda stuff was in the ports
tree, but this didn't include the kernel module.  All I could find on
the kernel side was back an entire revision.  I was in an environment
which already had Coda running, so I didn't want to drop back a rev.

I queried here, and didn't get a response.  So I'm assuming that
tracking FreeBSD is not much of a priority.

Thus, FreeBSD didn't work out for me, so I stayed with Linux.

> Some distros' packages can be old or broken. (e.g. Gentoo ebuild has been
> broken for a long time, placing Coda mount at some other place than /coda.
> 2008-07-12 I got a reply to my bug report from 2006-08-17 - note the years.
> Now there is an experimental ebuild which adds a symlink /coda -> ....
> so it _may_ work on Gentoo now)

Note also I had to punt entirely on Coda after some files dropped out to
un-readability even with my greatest care to cleanly shut down clients
and servers.  I was able to dredge the basic contents out of the raw
storage files, but I decided that even with its potential capabilities,
it was going to be a net loss of my time to try and use Coda.  This was
the latest Coda from CMU's site, compiled for Kubuntu Gutsy (which was
the latest at that time, as I recollect).

So proceed with caution.  As far as I can tell, Coda doesn't scale to
modern media sizes.  And you will want a non-Coda backup system which is
run often enough to stay very current.  And I wouldn't recommend
deploying it except where it's OK to have unscheduled 1-2 hour outages
while somebody goes and tinkers with Coda when something bad/unexpected
happens.

Sorry to sound so negative, but my experiences really didn't lead me to
believe that Coda is ready for a production environment.

$0.02,
Andy Valencia

[Jan Harkes]

On Fri, Jul 25, 2008 at 02:41:43PM -0700, Andy Valencia wrote:
> FYI, for FreeBSD 7.0 I found that the latest Coda stuff was in the ports
> tree, but this didn't include the kernel module.  All I could find on
> the kernel side was back an entire revision.  I was in an environment
> which already had Coda running, so I didn't want to drop back a rev.

I just checked and the Coda kernel module in FreeBSD 7.0 is correct and
current. There is a COMPAT_CODA_5 define to revert back to the older
interface, but if that is not defined it uses the current kernel-venus
api.

> I queried here, and didn't get a response.  So I'm assuming that
> tracking FreeBSD is not much of a priority.

I missed the mail, but even then it took me a while to figure out which
release FreeBSD 7.0 was the most current stable release, Coda support
was broken in the previous stable.

> So proceed with caution.  As far as I can tell, Coda doesn't scale to
> modern media sizes.

Depends on what you want to store. Coda doesn't scale well to millions
of files, but if the files are of moderate length (several megabytes
each, f.i. single track music files or digital photos) it has no problem
filling up a few hundred GB of disk space.

>       And you will want a non-Coda backup system which is
> run often enough to stay very current.

I've been running Amanda myself, and just this week set up BackupPC,
which uses content hashes to avoid storing duplicates of the same data.
This is really nice when you backup up all available replicas of
replicated volumes because you only end up storing one copy as long as
the replicas are in sync. Of course it also revealed an issue where a
volume dump gets aborted when the socket buffers fill up and the volutil
process blocks while trying to write more.

> Sorry to sound so negative, but my experiences really didn't lead me to
> believe that Coda is ready for a production environment.

That's ok. I happen to use Coda for our web server, my email, and
various other people have found uses as well. But there are still
situations where it may not be as reliable. Partly this is because Coda
uses optimistic replication, and when that optimism fails it falls back
on heuristics to fix up any resulting conflicts. Those heuristics are
not all encompassing, directory resolution has gotten reasonably usable.

But there are still corner cases, one I know of and am working on is
when a file is created on one replica, we then store the data and try to
resolve the store operation. Because the other replica(s) have not yet
seen the create they don't recognise the file and we get a conflict.
Most of the time it works, because the parent directory (create
operation) is resolved first, but once in a while things happen in the
wrong order and we get a conflict.

So hopefully at some point we will be able to handle whatever problems
are triggered by your usage models as well.

Jan

[u+codalist-wk5r]

Hi Andy,

On Fri, Jul 25, 2008 at 02:41:43PM -0700, Andy Valencia wrote:
> So proceed with caution.  As far as I can tell, Coda doesn't scale to
> modern media sizes.  And you will want a non-Coda backup system which is
> run often enough to stay very current.  And I wouldn't recommend
 ...
> Sorry to sound so negative, but my experiences really didn't lead me to
> believe that Coda is ready for a production environment.

Thanks for sharing the experience, it is a pity Coda didn't quite work for you.

Coda suitability depends a lot on the usage pattern,
it is not universal (nor is any other distributed file system)
nor bug-free (same for other systems, but Coda for a very long time
had very limited developer resources compared e.g. to AFS).

For a "highly motivated and loyal user" Coda is certainly very suitable
and useful. Checked for oldest files in my directories on Coda, including
the one normally used as "homedir" and some dates are from before 2003.
This means I dwell on Coda since realms came into existence.

No, this was not for a fainthearted, but nowadays 1. Coda is a lot better
2. I know the limits better :)
so it works pretty well (touching wood :)

Given a knowledgeable person in proximity, even mere mortals survive
on Coda, if they learn how to reconize the signs of trouble and call for help.

I would not risk moving more end users to Coda than I can provide
"immediate" support to, i.e. support which they percept as immediate,
Conflict resolution by a guru within five minutes is a good starting point.

The highest threshold while moving to Coda is education, both for the users,
for the helpdesk staff and for the admins.
The next one is tuning the applications so that they do not cause unnoying
and unnecessary conflicts ("mozilla", "bash", "less" come to mind immediately).

Regards,
Rune

[Robert Watson]

On Fri, 25 Jul 2008, Andy Valencia wrote:

> --------
> [u+codalist-wk5r@... writes:]
>
>> If the distribution does not cripple the kernel, Coda will work fine.
>
> FYI, for FreeBSD 7.0 I found that the latest Coda stuff was in the ports
> tree, but this didn't include the kernel module.  All I could find on the
> kernel side was back an entire revision.  I was in an environment which
> already had Coda running, so I didn't want to drop back a rev.
>
> I queried here, and didn't get a response.  So I'm assuming that tracking
> FreeBSD is not much of a priority.

(Joining this thread late due to a busy August)

As far as I'm aware, the kernel module shipped in FreeBSD 7.0 should speak
fine to the latest Coda user components.  I don't believe there have been
significant changes in the protocol between Coda kernel and user parts in
quite a long time.

I have made significant improvements to the kernel module that will be shipped
with FreeBSD 7.1 in a couple of months.  It should be possible to compile the
new version of the kernel module on a 7.0 system as they have basically the
same KPI for VFS.  It may even be possible to simply use the 7.1 Coda kernel
module on 7.0, although I would discourage that.

I've been thinking about removing support for older Coda versions in the
FreeBSD module, since they're done at compile-time, and this list is probably
a good a list as any to ask about it.

Finally, as Rune reports, there is a reported issue with the Linux ABI
emulation and the Coda kernel modules.  It's unclear to me whether this is a
bug in the ABI emulation layer or the Coda kernel module, and unfortunately
I've not had time to track down that issue.  It would be good to get that
fixed before FreeBSD 7.1 ships.

Robert N M Watson
Computer Laboratory
University of Cambridge

>
> Thus, FreeBSD didn't work out for me, so I stayed with Linux.
>
>> Some distros' packages can be old or broken. (e.g. Gentoo ebuild has been
>> broken for a long time, placing Coda mount at some other place than /coda.
>> 2008-07-12 I got a reply to my bug report from 2006-08-17 - note the years.
>> Now there is an experimental ebuild which adds a symlink /coda -> ....
>> so it _may_ work on Gentoo now)
>
> Note also I had to punt entirely on Coda after some files dropped out to
> un-readability even with my greatest care to cleanly shut down clients
> and servers.  I was able to dredge the basic contents out of the raw
> storage files, but I decided that even with its potential capabilities,
> it was going to be a net loss of my time to try and use Coda.  This was
> the latest Coda from CMU's site, compiled for Kubuntu Gutsy (which was
> the latest at that time, as I recollect).
>
> So proceed with caution.  As far as I can tell, Coda doesn't scale to
> modern media sizes.  And you will want a non-Coda backup system which is
> run often enough to stay very current.  And I wouldn't recommend
> deploying it except where it's OK to have unscheduled 1-2 hour outages
> while somebody goes and tinkers with Coda when something bad/unexpected
> happens.
>
>
> $0.02,
> Andy Valencia
>
>

[Greg Troxel]

Robert Watson <rwatson@...> writes:

> I have made significant improvements to the kernel module that will be
> shipped with FreeBSD 7.1 in a couple of months.  It should be possible
> to compile the new version of the kernel module on a 7.0 system as
> they have basically the same KPI for VFS.  It may even be possible to
> simply use the 7.1 Coda kernel module on 7.0, although I would
> discourage that.

Thanks for mentioning it here - I should take a look and see what of
those changes I should port to NetBSD.

> I've been thinking about removing support for older Coda versions in
> the FreeBSD module, since they're done at compile-time, and this list
> is probably a good a list as any to ask about it.

I have not yet removed support for the old venus protocol from NetBSD,
but I think it's time.

> Finally, as Rune reports, there is a reported issue with the Linux ABI
> emulation and the Coda kernel modules.  It's unclear to me whether
> this is a bug in the ABI emulation layer or the Coda kernel module,
> and unfortunately I've not had time to track down that issue.  It
> would be good to get that fixed before FreeBSD 7.1 ships.

That's interesting - there is a directory reading problem with linux
programs on NetBSD, and it seems to be a disagreement about how
directories do or don't cross 4K boundaries, but I'm not clear on what's
going on.

[Robert Watson]

On Tue, 2 Sep 2008, Greg Troxel wrote:

> Robert Watson <rwatson@...> writes:
>
>> I have made significant improvements to the kernel module that will be
>> shipped with FreeBSD 7.1 in a couple of months.  It should be possible to
>> compile the new version of the kernel module on a 7.0 system as they have
>> basically the same KPI for VFS.  It may even be possible to simply use the
>> 7.1 Coda kernel module on 7.0, although I would discourage that.
>
> Thanks for mentioning it here - I should take a look and see what of those
> changes I should port to NetBSD.

A quick summary of the changes since 7.0, almost entirely bug fixes, cleanup,
and general infrastructure improvement:

- TODO list updated
- Remove unused code
- Revise (improve?) vnode locking of cache nodes
- Trim old NetBSD- or Mach-specific comments
- Remove implementations of UFS-internal VOPs
- Remove devtomp()
- Maintain statfs() stats
- Style, comment, namespace cleanups throughout, except coda.h
- Remove Coda-specific name cache, use global FreeBSD name cache
- Implement rudimentary access cache along the lines of the NFS and smbfs
   access caches, and not unlike the access cache in the Linux coda module; we
   do slightly more granular invalidation than the Linux one though, since that
   existed in the previous Coda-specific name cache.  Note that this is a
   single-uid access cache, which has its limitations, but is better than none
   at all.
- Don't flush the namecache on CODA_PURGEUSER, just access control state
- Don't need to flush the name cache on reintegrated files due to fid
   renumber, since name cache no longer knows about fids
- Do flush the cnode attribute cache when a fid is renumbered, so that the new
   inode number will be generated on next stat()
- Remove unused OLD_DIAGNOSTIC code
- Fix up make_coda_node(), but there are still race conditions under load
- Disable interruption of coda_call() sleep, as it's not done right (still) --
   need to look more at the Linux coda module to see how it handles this
- Use queue(9) instead of custom queue macros

Your review of these changes would be most welcome :-).  These changes were
pretty much all made to the 7.x branch in March, but I'm not sure if that
means they're getting much exercise.  I should probably add a copyright claim
somewher,e I suppose.

>> I've been thinking about removing support for older Coda versions in the
>> FreeBSD module, since they're done at compile-time, and this list is
>> probably a good a list as any to ask about it.
>
> I have not yet removed support for the old venus protocol from NetBSD, but I
> think it's time.

OK.  I'll make that change in FreeBSD as well then.

>> Finally, as Rune reports, there is a reported issue with the Linux ABI
>> emulation and the Coda kernel modules.  It's unclear to me whether this is
>> a bug in the ABI emulation layer or the Coda kernel module, and
>> unfortunately I've not had time to track down that issue.  It would be good
>> to get that fixed before FreeBSD 7.1 ships.
>
> That's interesting - there is a directory reading problem with linux
> programs on NetBSD, and it seems to be a disagreement about how directories
> do or don't cross 4K boundaries, but I'm not clear on what's going on.

OK.  I probably won't have a chance to look at this for a bit, but if someone
else figures something out I'm happy to merge the results into FreeBSD.  If it
were in the next two weeks, then that would make it pretty likely we could
include the fix in 7.1, which would be great.

Robert N M Watson
Computer Laboratory
University of Cambridge