Re: SoC VFS Redesign

[Res-3]

On 31.05.2006 12:24, Brandon Hamilton wrote:
> I have been looking over the current Virtual File System implemented in Crystal Space and studying the documentation, getting to know a bit more about how CS actually works.
>
> As for the VFS, my proposal includes the following things (I in the process of prioritizing and fleshing out the details - and scrapping ideas that will not be used):
>
> The current iVFS will be rewritten to become the VFS manager, handling the management of plugins and paths, and providing an interface for access to the VFS.
>
> The new File System will implement the features of the VFS. Files on disk will be accessed in file blocks of a fixed size (within iFile interface) – this will help support features such as efficient memory allocation and faster file access, as well as real-time compression decompression as discussed below. The block size will be determined to during experimentation and must facilitate fast disk access.

WRT file access, the current iFile provides to ways to get data:
- Sequential access via Read()/SetPos()/....
- All-at-once access by obtaining the contents as One Big Buffer.
CS code itself uses sometimes this, sometimes that. Each requires
somewhat different access optimizations.
I'm not sure if it's already necessary to discuss exact optimizations;
after all, those are just "implementation details". Perhaps get the
design of the new interfaces, plugin architecture finished and working
before considering optimizations on such a low level as file access?

> The new File System with its plugins will be designed and developed to implement the following features and performance enhancements:
>
> File Access and Management:
> The new File System will implement and handle all the necessary functionality of opening/accessing/closing/reading/writing files through the VFS interface.
>
> Caching:
> The new File System will implement caching algorithms efficiently to ensure optimal performance and minimum hard drive reads (cache misses). Various algorithms, e.g LANDLORD based algorithms, will be explored to determine the method most suited to the use and application of the Crystal Space engine.

Question is how far this is needed. I'm not sure how often or whether at
all a file is read multiple times by a typical CS application. Caching
of individual files may or may not be worthwhile - perhaps investigate
the file usage behaviour of some CS app (e.g. Planeshift comes to mind).
OTOH doing some archive-related caching is probably a good idea, since
multiple reads from an archive are likely rather common. This is also
done by the current VFS implementation.

>
> Memory Placements/Allocation:
> Placement of files in memory is important so that memory fragmentation does not occur to avoid latency in the loading and allocation times of new files. This will attempt to provide better memory placement strategies for files to minimize the fragmentation problem. Methods such as Address-Ordered? Good Fit will be looked at to address this issue.

Note that recently the CS::Memory::Heap class
(http://crystalspace3d.org/docs/online/api/classCS_1_1Memory_1_1Heap.php)
was added to CS; it provides a heap separate from the main heap for
allocations (and thus in some scenarios can prevent fragmentation
there). It is backed by dlmalloc
(http://g.oswego.edu/dl/html/malloc.html), but general-purpose, so
perhaps check out its suitability, may or may not save you implementing
some custom allocator.

> Archives and Compression:
> The new File System will provide transparent management of archive files, including recognition, loading, decompression, compression and saving. The files will be efficiently mounted into the VFS tree and will be accessible as ordinary file within the VFS. Compression and decompression will be performed in parallel to file loading, performing on each block as it is available, in order to provide minimum latency in the loading/saving of compressed files and archives.

Some things to consider, what would be the syntax?
Let's say you have an archive foo.zip. Do you access it like
.../foo.zip/somefile? Or treat it like a directory, ie .../foo/somefile?
If the latter what happens if there is also a directory "foo" next to
foo.zip, does the dir shadow the zip, the zip shadow the dir, or do they
get merged?

> Dynamic Namespace:
> The new File System will provide dynamic path and namespace management. This plugin would provide efficient path searching and namespace mapping based on priority (conflict rules for user, system or app configuration domains).
>
> Plugin Mounts:
> Support for mounting plugins at any location within the VFS tree, and automatic managing of static and dynamic links to plugins and paths within the file system.

You could probably make everything a plugin; access to physical files
could be a "plugin", access to contents of an archive could be a
"plugin" (albeit such basic functionality like physical file access and
zip file access should be built into VFS itself, but using the "plugin"
interface.)
Another interesting idea might be to let the plugins access their data
via plugins, too... e.g. the zip plugin itself operates by reading an
iFile provided by some other plugins.
On the implementation side it probably just boils down to utilizing the
to-be-defined "iFileSystem" for the tasks above; VFS namespace/physical
FS/zip files could just be different implementations for that.

> Asynchronous I/O:
> The new File System could be extended to provide support for asynchronous IO, which can be used to facilitate data streaming and loading. Other plugins with CS would be able to use this interface to efficiently manage the streaming of data.

That would probably make the "Streaming Loader" student happy ;)

> A new interface will probably be defined (iFileSystem) which will be utilized by the iVFS implementation.

I assume iFileSystem would be a source for both getting a hierarchy
(e.g. directories, listing contents) as well as iFile objects (ie
opening a file).

> Constraints
> The current iVFS interface will not be broken and file system access will attempt to be as transparent as possible

Something to consider might be some permission system; ie so that you
can specify certain paths as read-only or so. This might be relevant for
clients that support user scripts, mods etc. but want to have some level
of security and prevent overwriting of arbitrary files.

Filenames: CS strives to use Unicode where output and input are
concerned; wrt file input/output, file names would be affected.
Currently, the filenames passed around are in the host character set;
this is obviously not that good - though it was not much of a problem so
far since probably most filenames are ASCII. Nevertheless, filenames
should be UTF-8 as well.
On the implementation side it probably boils down to using
Unicode-enabled functions where filenames are involved (e.g. wfopen()
instead of fopen()), hopefully not terribly difficult, but something to
keep in mind.

What platform do you develop on?
Some features (e.g. async I/O) have APIs that differ (usually wildly)
between platforms, so for such stuff wrapper classes are used, you only
work with the CS wrapper class in your code and the platform-specific
details are hidden for you. (csMemoryMappedIO would be an example for
such a wrapper.)
So when implementing something platform-specific, you should (a) provide
an implementation at least for the platform you work on (b) provide a
fallback for platforms that don't support this feature or for which the
wrapper hasn't been implemented yet (doesn't have to be very elaborate;
e.g. an async I/O fallback could just do a synchronous read).

-f.r.




_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Amir Taaki-2]

> > Archives and Compression:
> > The new File System will provide transparent management of archive files,
> > including recognition, loading, decompression, compression and saving.
> > The files will be efficiently mounted into the VFS tree and will be
> > accessible as ordinary file within the VFS. Compression and decompression
> > will be performed in parallel to file loading, performing on each block
> > as it is available, in order to provide minimum latency in the
> > loading/saving of compressed files and archives.
>
> Some things to consider, what would be the syntax?
> Let's say you have an archive foo.zip. Do you access it like
> .../foo.zip/somefile? Or treat it like a directory, ie .../foo/somefile?
> If the latter what happens if there is also a directory "foo" next to
> foo.zip, does the dir shadow the zip, the zip shadow the dir, or do they
> get merged?
>

Some good stuff, Just want to add that I prefer ../foo.zip/somefile as opposed
to ../foo/somefile since I can find all the zips in a directory (like user
installed levels) and just access them with having to perform any name
mangling.
IMHO I should be able to put a zip file with a missing extension (foo not
foo.zip) and it should detect foo as being a zip by reading its magic bytes.
I don't think the extension should affect its mount name.

Also what about support for other protocols? I'm not saying implement them,
just adding support for them so that in the future if someone wanted to code
for reading data from that source its there

http://xyz.com/a/b/cde/somefile
zip://(http://xyz.com/a/b/cde/file.zip)/somedir/somefile

Amir Taaki


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Amir Taaki-2]

Just realised you're using crystal-develop not crystal-main (resending):

> > Archives and Compression:
> > The new File System will provide transparent management of archive files,
> > including recognition, loading, decompression, compression and saving.
> > The files will be efficiently mounted into the VFS tree and will be
> > accessible as ordinary file within the VFS. Compression and decompression
> > will be performed in parallel to file loading, performing on each block
> > as it is available, in order to provide minimum latency in the
> > loading/saving of compressed files and archives.
>
> Some things to consider, what would be the syntax?
> Let's say you have an archive foo.zip. Do you access it like
> .../foo.zip/somefile? Or treat it like a directory, ie .../foo/somefile?
> If the latter what happens if there is also a directory "foo" next to
> foo.zip, does the dir shadow the zip, the zip shadow the dir, or do they
> get merged?

Some good stuff, Just want to add that I prefer ../foo.zip/somefile as
 opposed to ../foo/somefile since I can find all the zips in a directory
 (like user installed levels) and just access them with having to perform any
 name mangling.
IMHO I should be able to put a zip file with a missing extension (foo not
foo.zip) and it should detect foo as being a zip by reading its magic bytes.
I don't think the extension should affect its mount name.

Also what about support for other protocols? I'm not saying implement them,
just adding support for them so that in the future if someone wanted to code
for reading data from that source its there

http://xyz.com/a/b/cde/somefile
zip://(http://xyz.com/a/b/cde/file.zip)/somedir/somefile

Amir Taaki


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Res-3]

On 01.06.2006 12:08, Amir Taaki wrote:
> Also what about support for other protocols? I'm not saying implement them,
> just adding support for them so that in the future if someone wanted to code
> for reading data from that source its there
>
> http://xyz.com/a/b/cde/somefile
> zip://(http://xyz.com/a/b/cde/file.zip)/somedir/somefile

The ideas list specifically contains "Plugin Mounts", so one could
imagine to access a http server by mounting in a http plugin somewhere.

-f.r.




_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Res-3]

On 01.06.2006 12:08, Amir Taaki wrote:
> Some good stuff, Just want to add that I prefer ../foo.zip/somefile as opposed
> to ../foo/somefile since I can find all the zips in a directory (like user
> installed levels) and just access them with having to perform any name
> mangling.

Well, archive automounting should go all the way of course. So if you
list all the files in a (VFS) directory, you should see the automounted
directories, just like normal directories (and the accompanying zip
files probably hidden, if we end up with mangling).

-f.r.




_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Amir Taaki-2]

On Thursday 01 June 2006 14:37, res wrote:

> On 01.06.2006 12:08, Amir Taaki wrote:
> > Some good stuff, Just want to add that I prefer ../foo.zip/somefile as
> > opposed to ../foo/somefile since I can find all the zips in a directory
> > (like user installed levels) and just access them with having to perform
> > any name mangling.
>
> Well, archive automounting should go all the way of course. So if you
> list all the files in a (VFS) directory, you should see the automounted
> directories, just like normal directories (and the accompanying zip
> files probably hidden, if we end up with mangling).
>
> -f.r.

Ahh I see. Can't really see anything wrong there. Although I do think that
there should be still a way to identify which dirs are zips and there
shouldn't be name mangling I don't think (hiding is good, but forcefully
hiding everything can sometimes cause problems).

Amir Taaki


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[JorritTyb]

On 6/1/06, res <resqu@...> wrote:

> Well, archive automounting should go all the way of course. So if you
> list all the files in a (VFS) directory, you should see the automounted
> directories, just like normal directories (and the accompanying zip
> files probably hidden, if we end up with mangling).

I'm not in favor of that. If you have both bla.zip and bla then it
should be possible to get a file from bla.zip and from bla too. I
think the best way is to use bla.zip/xxx and bla/xxx (as different
files in different locations).

Greetings,

>
> -f.r.
>
>
>
>
>
> _______________________________________________
> Crystal-develop mailing list
> Crystal-develop@...
> https://lists.sourceforge.net/lists/listinfo/crystal-develop
>
>
>
>

--
Project Manager of Crystal Space (http://www.crystalspace3d.org)
and CEL (http://cel.crystalspace3d.org)
Support Crystal Space. Donate at
https://sourceforge.net/donate/index.php?group_id=649


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Christopher-15]

Just wanted to throw a few things into this conversation.

Firstly, If this is being developed in a windows environment, I'm willing to offer aid, testing, and even do some implimentation work (for async IO) for linux.

Secondly, has anythought been given to the idea of making the native filesystem viewable/writeable by VFS? For example, if plainshift were to allow the local saving of character files, it would be very nice to be able to prompt the user, and let them choose "My Documents\Characters\Plainshift\foo.char", and for VFS to be able to handle that correctly. (Basically, it would mean being able to report the native file structure, and being able to read/write to it. Some handling for permissions might be required.)

All in al, it seems like a nice setup.

--Chris


--
Christopher S. Case
Project Head
The Seventh Game: Precursors
http://games.g33xnexus.com/precursors
(716) 785 - 5553(Cellphone)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"To err is human. To forgive, divine.
To fix mistakes, now that's an Engineer."
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Res-3]

On 01.06.2006 23:09, Chris Case wrote:
> Secondly, has anythought been given to the idea of making the native
> filesystem viewable/writeable by VFS?

It is. iVFS::Mount() takes a native path as the path to mount from. You
can even mount the FS root with MountRoot().

> For example, if plainshift were to
> allow the local saving of character files, it would be very nice to be
> able to prompt the user, and let them choose "My
> Documents\Characters\Plainshift\foo.char", and for VFS to be able to
> handle that correctly. (Basically, it would mean being able to report
> the native file structure, and being able to read/write to it. Some
> handling for permissions might be required.)

Well you can do that, albeit the proper course of action would perhaps
be to not ask the user anything and just store user-specific data in the
right place, which would be the user's HOME directory (or equivalent).
CS has facilities for that, too (csGetPlatformConfig(), and less
specific functionality, csGetPlatformConfigPath()), you just got to use
them.

-f.r.





_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Michael D. Adams]

On 6/1/06, Jorrit Tyberghein <jorrit.tyberghein@...> wrote:

> On 6/1/06, res <resqu@...> wrote:
>
> > Well, archive automounting should go all the way of course. So if you
> > list all the files in a (VFS) directory, you should see the automounted
> > directories, just like normal directories (and the accompanying zip
> > files probably hidden, if we end up with mangling).
>
> I'm not in favor of that. If you have both bla.zip and bla then it
> should be possible to get a file from bla.zip and from bla too. I
> think the best way is to use bla.zip/xxx and bla/xxx (as different
> files in different locations).

To clarify, you are saying you prefer an auto-mounted archives to
*not* do any name mangling?  Is that correct?  (For the record, I
would agree with that position.)

Any one too in love with name mangling can always just manually mount
the archive or add a sim-link to the auto-mounted archive.

...

Hmm, <evil grin>.  Open as file universe.zip/earth.zip/europe.map.
Open as file universe.zip.  Write from universe.zip/mars.zip to
universe.zip/earth.zip.  (Read from universe.zip half way through.)
Read from universe.zip/earth.zip/europe.map. (A worm hole? The end of
the world?)  ... and then add streaming on top.

Michael D. Adams
mdmkolbe@...


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Res-3]

On 01.06.2006 21:01, Jorrit Tyberghein wrote:

> On 6/1/06, res <resqu@...> wrote:
>
>> Well, archive automounting should go all the way of course. So if you
>> list all the files in a (VFS) directory, you should see the automounted
>> directories, just like normal directories (and the accompanying zip
>> files probably hidden, if we end up with mangling).
>
> I'm not in favor of that. If you have both bla.zip and bla then it
> should be possible to get a file from bla.zip and from bla too. I
> think the best way is to use bla.zip/xxx and bla/xxx (as different
> files in different locations).

You could use some shadowing, so the bla/ you see in VFS would be a
combination of the contents if bla/ and bla.zip. You would have to have
some order of preference for files of the same name, though.

-f.r.




_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[JorritTyb]

On 6/2/06, res <resqu@...> wrote:

> > I'm not in favor of that. If you have both bla.zip and bla then it
> > should be possible to get a file from bla.zip and from bla too. I
> > think the best way is to use bla.zip/xxx and bla/xxx (as different
> > files in different locations).
>
> You could use some shadowing, so the bla/ you see in VFS would be a
> combination of the contents if bla/ and bla.zip. You would have to have
> some order of preference for files of the same name, though.

This is of course possible but I rather strongly think we should NOT
do this. It is best to be explicit here and not to have name mangling
at all. That way you don't need any priority system and there is no
confusion about what file will be read.

Greetings,

--
Project Manager of Crystal Space (http://www.crystalspace3d.org)
and CEL (http://cel.crystalspace3d.org)
Support Crystal Space. Donate at
https://sourceforge.net/donate/index.php?group_id=649


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[JorritTyb]

> To clarify, you are saying you prefer an auto-mounted archives to
> *not* do any name mangling?  Is that correct?  (For the record, I
> would agree with that position.)

Yes that's right. Auto-mounting is a great feature to have but I would
prefer not any confusing name mangling.

Greetings,

--
Project Manager of Crystal Space (http://www.crystalspace3d.org)
and CEL (http://cel.crystalspace3d.org)
Support Crystal Space. Donate at
https://sourceforge.net/donate/index.php?group_id=649


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Amir Taaki-2]

On Friday 02 June 2006 01:44, Michael D. Adams wrote:

> On 6/1/06, Jorrit Tyberghein <jorrit.tyberghein@...> wrote:
> > On 6/1/06, res <resqu@...> wrote:
> > > Well, archive automounting should go all the way of course. So if you
> > > list all the files in a (VFS) directory, you should see the automounted
> > > directories, just like normal directories (and the accompanying zip
> > > files probably hidden, if we end up with mangling).
> >
> > I'm not in favor of that. If you have both bla.zip and bla then it
> > should be possible to get a file from bla.zip and from bla too. I
> > think the best way is to use bla.zip/xxx and bla/xxx (as different
> > files in different locations).
>
> To clarify, you are saying you prefer an auto-mounted archives to
> *not* do any name mangling?  Is that correct?  (For the record, I
> would agree with that position.)
>
> Any one too in love with name mangling can always just manually mount
> the archive or add a sim-link to the auto-mounted archive.

I would agree with that too. Are we all agreeing then against name mangling?

> ...
>
> Hmm, <evil grin>.  Open as file universe.zip/earth.zip/europe.map.
> Open as file universe.zip.  Write from universe.zip/mars.zip to
> universe.zip/earth.zip.  (Read from universe.zip half way through.)
> Read from universe.zip/earth.zip/europe.map. (A worm hole? The end of
> the world?)  ... and then add streaming on top.

Nice. Just what I'd do everyday ;)

> Michael D. Adams
> mdmkolbe@...
>
>
> _______________________________________________
> Crystal-develop mailing list
> Crystal-develop@...
> https://lists.sourceforge.net/lists/listinfo/crystal-develop

Amir Taaki


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Mark Sanders]

res wrote:

> On 01.06.2006 21:01, Jorrit Tyberghein wrote:
>  
>> I'm not in favor of that. If you have both bla.zip and bla then it
>> should be possible to get a file from bla.zip and from bla too. I
>> think the best way is to use bla.zip/xxx and bla/xxx (as different
>> files in different locations).
>>    
>
> You could use some shadowing, so the bla/ you see in VFS would be a
> combination of the contents if bla/ and bla.zip. You would have to have
> some order of preference for files of the same name, though.
>
> -f.r.
>  

I would prefer no name mangling at all. It is much clearer and simpler
to just mount a zip as /bla.zip/ instead of /bla/ ...

1. You know what is a zip and what not.
2. You don't have to do any magic when the file xxx exists in both
/bla/xxx and in the zip file called bla.zip.
3. You already need to change the VFS mounts of you change a directory
to a zip file so there is no advantage in name mangling here.

The only real reason I can think of at this moment for name mangling is
to obfuscate what you mount.


Best regards,

Mark Sanders (CyaNox)


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Eric Sunshine]

On Jun 2, 2006, at 4:49 AM, Mark Sanders wrote:

> I would prefer no name mangling at all. It is much clearer and simpler
> to just mount a zip as /bla.zip/ instead of /bla/ ...
> 1. You know what is a zip and what not.
> 2. You don't have to do any magic when the file xxx exists in both
> /bla/xxx and in the zip file called bla.zip.
> 3. You already need to change the VFS mounts of you change a directory
> to a zip file so there is no advantage in name mangling here.
> The only real reason I can think of at this moment for name  
> mangling is
> to obfuscate what you mount.

There is at least one relatively strong reason in favor of name-
mangling, and that is to provide a uniform way for higher-priority  
files / resources to override or shadow lower-priority ones. For  
instance, a user may download a replacement file for some in-game  
model which should shadow / replace the model which shipped with the  
game. In the original VFS re-design proposal by Greg Block, the user  
could do this simply by dumping the new model in his user-local game  
resource directory; for example, $HOME/.planeshift/models/characters/
wizard.model. VFS would automatically identify and use this  
replacement file instead of the factory-shipped wizard.model, which  
might happen to reside within $PLANESHIFT/models/characters.zip.

The idea was that the source of a VFS directory / resource should be  
transparent. As a client of the directory / resource, you shouldn't  
have to know anything about its "physical" manifestation; whether it  
reside on a local filesystem, or within a zip file, or some other  
type of archive file, or on some remote server accessed via HTTP or  
some other protocol. The prioritized domain-based architecture is  
supposed to be transparent to clients, and insulate them from these  
low-level details of physical manifestation. Stated more precisely, a  
person wanting to install a new wizard.model should not be forced to  
duplicate precisely the low-level path of manifestation tied to the  
original model. That is, you should merely be able to dump your new  
model into your own user-specific game-resource directory and have it  
recognized automatically. You should not be forced to place it in a  
zip file within a .tar.gz file vended from an HTTP server merely  
because the original model shipped with the game was in a zip file  
within a .tar.gz file vended from an HTTP server.

Point #3 in the quoted post is somewhat moot because one key idea of  
the VFS re-design is to do away with ugly, hard-coded, difficult-to-
maintain configuration files, and instead have VFS configure itself  
dynamically and automatically.

-- ES



_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[JorritTyb]

> There is at least one relatively strong reason in favor of name-
> mangling, and that is to provide a uniform way for higher-priority
> files / resources to override or shadow lower-priority ones. For
> instance, a user may download a replacement file for some in-game
> model which should shadow / replace the model which shipped with the
> game. In the original VFS re-design proposal by Greg Block, the user
> could do this simply by dumping the new model in his user-local game
> resource directory; for example, $HOME/.planeshift/models/characters/
> wizard.model. VFS would automatically identify and use this
> replacement file instead of the factory-shipped wizard.model, which
> might happen to reside within $PLANESHIFT/models/characters.zip.

While this is theoretically a nice feature I wonder how desired this
is in practice:
  - First of all most game makers actually don't want users to replace
their models and would try their best to prevent this kind of usage.
  - Secondly, if a user wants to replace a model he/she can also
substitute the model in the zip file itself.
  - Thirdly this kind of name mangling would not solve situations
where the game data didn't happen to be stored in a zip file anyway.
In that case the user would have no way to override it (except again
by replacing the original model).

I think the disadvantages of automatic name mangling are bigger then
the advantages. Also the automatic name mangling would only partially
solve the problem of letting a user replace game models so it is not
perfect for that either.


> Point #3 in the quoted post is somewhat moot because one key idea of
> the VFS re-design is to do away with ugly, hard-coded, difficult-to-
> maintain configuration files, and instead have VFS configure itself
> dynamically and automatically.

I have strong doubts on how practical this will be. Especially in the
case of vfs automatically trying to discover where a file is located.
How would this work in cases where files happen to have the same name
(which is not uncommon if you are using textures made by different
people and used for different models). Also many files in CS have
exactly the same name. It is common for people to name the world file
'world'. Even if they have multiple map files every world file in
those map files will be called 'world'. So I'm not certain how such an
automatic system would work? Can you elaborate on this?

Greetings,

--
Project Manager of Crystal Space (http://www.crystalspace3d.org)
and CEL (http://cel.crystalspace3d.org)
Support Crystal Space. Donate at
https://sourceforge.net/donate/index.php?group_id=649


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Res-3]

On 02.06.2006 15:05, Eric Sunshine wrote:
> There is at least one relatively strong reason in favor of name-
> mangling, and that is to provide a uniform way for higher-priority  
> files / resources to override or shadow lower-priority ones.

Though, such functionality may not necessarily have to go into the
"core" of VFS.
The ideas so far specifically contain "plugin mounts". So if easy
modifiability is desired, one could write a "shadowing file system" that
wraps a path in the "normal" VFS namespace, but scans for archives in a
directory and performs the shadowing.

-f.r.




_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Eric Sunshine]

On 6/2/06, Jorrit Tyberghein <jorrit.tyberghein@...> wrote:

> > For instance, a user may download a replacement file for some in-game
> > model which should shadow / replace the model which shipped with the
> > game. In the original VFS re-design proposal by Greg Block, the user
> > could do this simply by dumping the new model in his user-local game
> > resource directory; for example, $HOME/.planeshift/models/characters/
> > wizard.model. VFS would automatically identify and use this
> > replacement file instead of the factory-shipped wizard.model, which
> > might happen to reside within $PLANESHIFT/models/characters.zip.
> While this is theoretically a nice feature I wonder how desired this
> is in practice:

It's very desirable to have a priority-based domain architecture. We
use it elsewhere in CS for other facilities. It's a useful concept and
handy abstraction mechanism which simplifies usage and implementation,
and improves flexibility and promotes modularity.

Anyone familiar with the 'chrome' system in Mozilla products can
understand how important this type of abstraction can be. In fact, it
would be useful for the implementer of the re-designed VFS to keep
Mozilla 'chrome' facility in mind since the two share many concepts.

>  - First of all most game makers actually don't want users to replace
> their models and would try their best to prevent this kind of usage.

This flexibility isn't restricted to user extensions. It can be, and
should be used by game designers themselves. Consider a stock sound
which is used when some event -- which is not specific to any
level/map -- occurs in a game. Now consider that a level designer
needs a to use a different sound within his map for this common event.
Provided that his map is mounted into VFS with higher priority than
the "standard sound library", the sound he includes in his map can
easily and automatically shadow the standard sound for the event. The
sound might appear at VFS path /sounds/explosions/rocket.ogg in both
the standard sound library and the map archive, but because the map
archive has higher priority, the sound in the map archive prevails
when the map is mounted. When the map is unloaded, the standard sound
again becomes visible.

>  - Secondly, if a user wants to replace a model he/she can also
> substitute the model in the zip file itself.

Don't be trapped thinking along lines of monolithic resource
management. What we are aiming for with this re-design is a flexible
and modular system (a key point). Also, if the game runs from CD-ROM
or DVD, users would have no opportunity to modify factory-supplied zip
files (which is a bad idea anyhow).

>  - Thirdly this kind of name mangling would not solve situations
> where the game data didn't happen to be stored in a zip file anyway.
> In that case the user would have no way to override it (except again
> by replacing the original model).

I have no idea what you mean. The plan as proposed by Greg Block
handles this case as easily as any other case. Don't forget that this
system aims to incorporate a layered, prioritized set of bindings. A
file in a higher-priority layer automatically causes it to shadow the
same file in the same path at a lower-priority layer. This works
equally well regardless of the physical source of the file (zip, raw
directory, HTTP, etc.); and it allows higher-priority resources to
shadow lower-priority resources without forcing them to occupy the
same sort of back-end storage. A raw file can shadow a entry from a
zip just as easily as an entry from a zip can shadow a raw file. The
entire idea is to make this as transparent as possible. The path
/sounds/explosions/rocket.ogg should refer to exactly that resource
without regard to its physical manifestation (whether it is just a raw
file, or an entry in a zip, or a resource grabbed via HTTP, etc.).

> I think the disadvantages of automatic name mangling are bigger then
> the advantages. Also the automatic name mangling would only partially
> solve the problem of letting a user replace game models so it is not
> perfect for that either.

In what way does it only partially solve the problem? I haven't seen
any evidence suggesting this to be the case. The original proposal
definitely takes this case into consideration.

> > Point #3 in the quoted post is somewhat moot because one key idea of
> > the VFS re-design is to do away with ugly, hard-coded, difficult-to-
> > maintain configuration files, and instead have VFS configure itself
> > dynamically and automatically.
> I have strong doubts on how practical this will be. Especially in the
> case of vfs automatically trying to discover where a file is located.
> How would this work in cases where files happen to have the same name
> (which is not uncommon if you are using textures made by different
> people and used for different models).

You never can have two files with same name in a single directory
anyhow, so I don't understand your question. Textures with the same
name can co-exist just as easily within VFS as they can outside of VFS
as long as they occupy different directories.

> Also many files in CS have
> exactly the same name. It is common for people to name the world file
> 'world'. Even if they have multiple map files every world file in
> those map files will be called 'world'.

Again, you never have two 'world' files in the same directory. As with
physical storage, your distinct 'world' files would live in separate
VFS subdirectories. For example, VFS paths /map1/world, /map2/world,
etc.

> So I'm not certain how such an
> automatic system would work? Can you elaborate on this?

I suggest referencing Greg Block's original proposal. It is both
simple and relatively elegant:

https://sourceforge.net/tracker/?func=detail&aid=823451&group_id=649&atid=350649

-- ES


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop

[Michael D. Adams]

On 6/2/06, Eric Sunshine <sunshine@...> wrote:

> On 6/2/06, Jorrit Tyberghein <jorrit.tyberghein@...> wrote:
> > > For instance, a user may download a replacement file for some in-game
> > > model which should shadow / replace the model which shipped with the
> > > game. In the original VFS re-design proposal by Greg Block, the user
> > > could do this simply by dumping the new model in his user-local game
> > > resource directory; for example, $HOME/.planeshift/models/characters/
> > > wizard.model. VFS would automatically identify and use this
> > > replacement file instead of the factory-shipped wizard.model, which
> > > might happen to reside within $PLANESHIFT/models/characters.zip.
> > While this is theoretically a nice feature I wonder how desired this
> > is in practice:
>
> It's very desirable to have a priority-based domain architecture. We
> use it elsewhere in CS for other facilities. It's a useful concept and
> handy abstraction mechanism which simplifies usage and implementation,
> and improves flexibility and promotes modularity.
>
> Anyone familiar with the 'chrome' system in Mozilla products can
> understand how important this type of abstraction can be. In fact, it
> would be useful for the implementer of the re-designed VFS to keep
> Mozilla 'chrome' facility in mind since the two share many concepts.
>
> >  - First of all most game makers actually don't want users to replace
> > their models and would try their best to prevent this kind of usage.
>
> This flexibility isn't restricted to user extensions. It can be, and
> should be used by game designers themselves. Consider a stock sound
> which is used when some event -- which is not specific to any
> level/map -- occurs in a game. Now consider that a level designer
> needs a to use a different sound within his map for this common event.
> Provided that his map is mounted into VFS with higher priority than
> the "standard sound library", the sound he includes in his map can
> easily and automatically shadow the standard sound for the event. The
> sound might appear at VFS path /sounds/explosions/rocket.ogg in both
> the standard sound library and the map archive, but because the map
> archive has higher priority, the sound in the map archive prevails
> when the map is mounted. When the map is unloaded, the standard sound
> again becomes visible.

Question:
How does Mozilla's chrome deal with the override issue?  Name
mangling?  Config/install files?

Side note:
A number of Mac games already use the resource fork to do tricks like
this (ask me if you want details) so there is a use and demand for
this.

Main point:
I find the arguments that Eric is making to be compelling.  But there
is a middle ground.  Zip files don't have to have a ".zip" extension
and not every file/directory with a ".zip" extension has to be a Zip
file.  VFS can tell the difference by looking at signature bytes.

What I mean is this.  Use no name mangling, but don't trust the
extensions for file type information.  Suppose the game references the
paths "/sounds.zip/ping.wav" and "/textures/wave.tga" and the user
wants to over ride them.  The user can make a *directory* names
"sounds.zip" or a *zip*file* named "textures" containing the
appropriate files.

I know what you're thinking.  "Now I have a bunch of miss-named
files/directories running around."  And yes, you do.  If you're on
Linux remember that file extensions are a social convention and don't
mean anything to the computer.  If you're on Windows, learn that file
extensions are a social conventions and are only needed by the
computer for file associations (use the open dialog or get really good
at renaming).  If you're on Mac, you have types/creator codes, so what
are you complaining about? (For OSX see also Linux.)

I don't really like this kind of solution because it does feel like a
work-around, but I think it achives the desired goals.

Michael D. Adams
mdmkolbe@...


_______________________________________________
Crystal-develop mailing list
Crystal-develop@...
https://lists.sourceforge.net/lists/listinfo/crystal-develop