Limits for mapping fs files

[tsmiller]

Zodb developers,
When I open my database I am mapping individual files in the following way. I have about 20 fs files mapped in this fashion.  My concern is that I am going to hit some kind of limit.  Is accessing these fs files going to slow down as the number grows?  Am I going to hit an upper limit - say 256 or something like that?  

thanks,

tom

    def open( self ):
        self.base = {}
        dbmap = {}
        tm = transaction.TransactionManager()
        self.tm = tm
        #grfind
        db = DB( FileStorage.FileStorage( os.path.join(currentDirectory, "zodb", "grfind.fs") ),  database_name='root', databases=dbmap)
        root = db.open(transaction_manager=tm)
        self.dbRoot = root.root()
       #system
        db = DB( FileStorage.FileStorage( os.path.join(currentDirectory, "zodb", "system.fs") ), database_name='system',  databases=dbmap)
        self.systemConn = root.get_connection('system').root()
        #accounts
        db = DB( FileStorage.FileStorage( os.path.join(currentDirectory, "zodb", "accounts.fs") ), database_name='accounts',  databases=dbmap)
        self.accountsConn = root.get_connection('accounts').root()
       #dialogs
        db = DB( FileStorage.FileStorage(os.path.join(currentDirectory, "zodb", "dialogs.fs")), database_name='dialogs',  databases=dbmap)
        self.dialogsConn = root.get_connection('dialogs').root()
       #im
        db = DB( FileStorage.FileStorage(os.path.join(currentDirectory, "zodb", "im.fs")), database_name='im',  databases=dbmap)
        self.imConn = root.get_connection('im').root()

[Jim Fulton]

On Tue, Oct 27, 2009 at 12:17 PM, tsmiller <tsmiller@...> wrote:
>
> Zodb developers,
> When I open my database I am mapping individual files in the following way.

Why?

> I have about 20 fs files mapped in this fashion.  My concern is that I am
> going to hit some kind of limit.  Is accessing these fs files going to slow
> down as the number grows?  Am I going to hit an upper limit - say 256 or
> something like that?

There collection of databases are managed with a Python dictionary, so
you should be able to have many thousands of databases without feeling
pain due to the number. You'll probably run out of file descriptors
first.

Why are you splitting your database up so much (or even at all)?

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

[tsmiller]

Jim,
Thanks for the fast response.  As to why.  Part of the reason is ignorance.  When I started working with web applications I had to learn javascript, python, qooxdoo, cherrypy, zodb, etc.  So when things worked they stayed as simple as possible.  I would think that there are better ways to accomplish what I am doing, but what I am doing seems to work and works quite well.

My application is an online marketplace consisting of distinct stores ( books, cars, fabric, music, etc.) in which the client is constantly reading and writing to the zodb database.  I am using cherryPy on the server side.

1) Actually my app is about to be a collection of marketplaces, each with a set of stores.  It is feasible that I could have 20 marketplaces and hundreds of stores in the future.  It makes management very easy by putting each store in its own file.
 
2) I write alot to my fs files so I worry about size.  The end users create and manage their stores and their data and they work on the live database when they do so.  I think that it may be necessary to change a store to a (forgive me!) relational database if my files get too big.  For reading and writing.  Having them broken up into separate files makes this easier also.  

3) It is easier to manage.  I can move each store to separate marketplaces if I wish.  I am trying to make it to where I can change any part of my system without affecting another.

I have been using this database for about three years now.  Mostly part time when not doing my real job, but I have recently started to work on software fulltime.  This database has NEVER corrupted any data or as far as I can tell screwed up in any way.  It is downright amazing.  You did one hell of a job.  

tom

Jim Fulton wrote:

On Tue, Oct 27, 2009 at 12:17 PM, tsmiller <[e-mail]> wrote:
>
> Zodb developers,
> When I open my database I am mapping individual files in the following way.

Why?

> I have about 20 fs files mapped in this fashion.  My concern is that I am
> going to hit some kind of limit.  Is accessing these fs files going to slow
> down as the number grows?  Am I going to hit an upper limit - say 256 or
> something like that?

There collection of databases are managed with a Python dictionary, so
you should be able to have many thousands of databases without feeling
pain due to the number. You'll probably run out of file descriptors
first.

Why are you splitting your database up so much (or even at all)?

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  [e-mail]
https://mail.zope.org/mailman/listinfo/zodb-dev