I am working on a digital video recorder. The system is linux based and there are 16 video sources. The video sources write data the data disk syncronously. Once it is filled up, there is a recycle mechanism which will remove the old video files and free up new space. As you can imagined, there will be serious external fragmentation problem as time passes.
I was told that jfs and xfs can do much better than ext3, to tackle the fragment problem, so I conducted a few benchmark tests and found that jfs is doing excellent job. The findings is not strong enough to persuade my boss to change, and hence I've been reading the rationale and source code behind jfs. Here I summarized two major questions that can help me explain jfs's magic:
1. when I write the first byte or open a file, what will jfs do? cuz my findings is that, the 16 channels create files of size around 32MB. They grow in size of course, but majority fragment or number of extents I found is only ONE...
according to ur disscussion with Peter, jfs allocates one page to a file at a time. and this allocation is locked under one allocation group. the page size according to jfs_filesys.h is 4096. You said the allocation would be allocated but not recorded (ABNR), which raised two subquestions:
1a. is those ABNR blocks stored temporary in memory, 16 files on grow and towards 32MB, it is a huge memory requirement. is it really that everyone stored in memory and flushed to the disk at file close?? what is the jfs memory requirement then?
1b. since only one file is allocated in one allocation group (AG), then how many AG is there in ur disk when it's formated (mkfs)? and is there an upper bound for the maximum number of files which can be opened and written at the same moment in jfs??
2. jfs is so called extent-based allocation. how jfs knows the right size of extent to allocate to a fixed file? and growing file? the stat i found shows that majority of my files ( <= 32MB ) are single fragment file (number of extent = 1). I would really like to understand the "magic" how it can be achieved.
The findings and rationale behind will lead us to a filesystem change. I would be very gladful if anyone can help me. Thankyou very much!