reiserfs - recommendations for small files

[Viji V Nair-2]

Hi,

System : Fedora 11 x86_64
Current Filesystem: 150G ext4 (formatted with "-T small" option)
Number of files: 50 Million, 1 to 30K png images

We are generating these files using a python programme and getting
very slow IO performance. While generation there are only writes, no
read. After generation there are heavy read and no write. These files
are not in a single directory, this is a pyramid structure. There are
total 15 pyramids and coming down from top to bottom the sub
directories and files  are multiplied by a factor of 4

I am looking for an alternate file system and best
practices/recommendation to get a better performance. What the
optimisation methods available in reiserfs for small files?

Any suggestions of the above are greatly appreciated.

Viji
--
To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Toby Thain-2]

On 17-Oct-09, at 2:11 PM, Viji V Nair wrote:

> Hi,
>
> System : Fedora 11 x86_64
> Current Filesystem: 150G ext4 (formatted with "-T small" option)
> Number of files: 50 Million, 1 to 30K png images
>
> We are generating these files using a python programme and getting
> very slow IO performance. While generation there are only writes, no
> read. After generation there are heavy read and no write. These files
> are not in a single directory, this is a pyramid structure. There are
> total 15 pyramids and coming down from top to bottom the sub
> directories and files  are multiplied by a factor of 4
>
> I am looking for an alternate file system and best
> practices/recommendation to get a better performance. What the
> optimisation methods available in reiserfs for small files?

You could benchmark this on reiser3 today (it's perfectly stable &  
small file handling is supposedly a sweet spot).

--Toby

>
> Any suggestions of the above are greatly appreciated.
>
> Viji
> --
> To unsubscribe from this list: send the line "unsubscribe reiserfs-
> devel" in
> the body of a message to majordomo@...
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Viji V Nair-2]

I

On Tue, Oct 20, 2009 at 12:46 AM, Toby Thain <toby@...> wrote

>
> On 17-Oct-09, at 2:11 PM, Viji V Nair wrote:
>
>> Hi,
>>
>> System : Fedora 11 x86_64
>> Current Filesystem: 150G ext4 (formatted with "-T small" option)
>> Number of files: 50 Million, 1 to 30K png images
>>
>> We are generating these files using a python programme and getting
>> very slow IO performance. While generation there are only writes, no
>> read. After generation there are heavy read and no write. These files
>> are not in a single directory, this is a pyramid structure. There are
>> total 15 pyramids and coming down from top to bottom the sub
>> directories and files  are multiplied by a factor of 4
>>
>> I am looking for an alternate file system and best
>> practices/recommendation to get a better performance. What the
>> optimisation methods available in reiserfs for small files?
>
> You could benchmark this on reiser3 today (it's perfectly stable & small
> file handling is supposedly a sweet spot).

I am creating a fedora 11 kernel rpm with reiserfs3 and reiser4
support, planning to bench mark it on multiple hardware. Will update
the results.

>
> --Toby
>
>>
>> Any suggestions of the above are greatly appreciated.
>>
>> Viji
>> --
>> To unsubscribe from this list: send the line "unsubscribe reiserfs-devel"
>> in
>> the body of a message to majordomo@...
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

--
To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Edward Shishkin-2]

Viji V Nair wrote:

> Hi,
>
> System : Fedora 11 x86_64
> Current Filesystem: 150G ext4 (formatted with "-T small" option)
> Number of files: 50 Million, 1 to 30K png images
>
> We are generating these files using a python programme and getting
> very slow IO performance. While generation there are only writes, no
> read. After generation there are heavy read and no write. These files
> are not in a single directory, this is a pyramid structure. There are
> total 15 pyramids and coming down from top to bottom the sub
> directories and files  are multiplied by a factor of 4
>
> I am looking for an alternate file system and best
> practices/recommendation to get a better performance. What the
> optimisation methods available in reiserfs for small files?
>  

I think that default tail packing policy would be the best for you.
Also use tea hash.

Thanks,
Edward.

> Any suggestions of the above are greatly appreciated.
>
> Viji
> --
> To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
> the body of a message to majordomo@...
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>  

--
To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html