mnesia vs ets

[Roberto Ostinelli]

dear all,

i am populating a mnesia disc_copies table with 20,000 records like this:

WriteFun = fun() -> [mnesia:dirty_write({test_table, K, {}}) || K <-
lists:seq(1, 20000)] end,
mnesia:activity(sync_dirty, WriteFun).

then i try reading from the table using mnesia:

[mnesia:dirty_read(test_table, K) || K <- lists:seq(1, 20000)].

and then using ets:

[ets:lookup(test_table, K) || K <- lists:seq(1, 20000)].

benchmarking the two ways of reading gives the following results:

MNESIA: 29473.89 rows / min.
ETS: 93312.60 rows / min.

this means that with ets reading speed is 3x than dirty_read with MNESIA.

am i missing something here, or is this normal?

r.

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[G.S.-2]

It's normal.

But you can if you wish, access the Mnesia's ets tables directly, which will
give you extra speed in reading.

Regards,
-Gene

On Tue, Oct 27, 2009 at 5:49 PM, Roberto Ostinelli <roberto@...>wrote:

> dear all,
>
> i am populating a mnesia disc_copies table with 20,000 records like this:
>
> WriteFun = fun() -> [mnesia:dirty_write({test_table, K, {}}) || K <-
> lists:seq(1, 20000)] end,
> mnesia:activity(sync_dirty, WriteFun).
>
> then i try reading from the table using mnesia:
>
> [mnesia:dirty_read(test_table, K) || K <- lists:seq(1, 20000)].
>
> and then using ets:
>
> [ets:lookup(test_table, K) || K <- lists:seq(1, 20000)].
>
> benchmarking the two ways of reading gives the following results:
>
> MNESIA: 29473.89 rows / min.
> ETS: 93312.60 rows / min.
>
> this means that with ets reading speed is 3x than dirty_read with MNESIA.
>
> am i missing something here, or is this normal?
>
> r.
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
>

[Roberto Ostinelli]

> If you wish you could directly access the ets tables within mnesia, that will give you the fastest read possible.

isn't this what i'm doing with ets:lookup(test_table, K)?

also, what's the point in doing a key/value mnesia:dirty_read on
non-fragmented disc_copy tables instead of a 3x fasters ETS lookup?

r.

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Ulf Wiger-3]

Roberto Ostinelli wrote:
> also, what's the point in doing a key/value mnesia:dirty_read on
> non-fragmented disc_copy tables instead of a 3x fasters ETS lookup?

Dirty read is location- and storage type-agnostic. This means that
you will have greater flexibility to change the storage type or
distribution pattern of the table without necessarily breaking
your code.

In some cases, this flexibility is not worth as much as the raw
speed. Personally, I always start out using transactions, and
always using mnesia:activity(Type, F). I have very rarely had to
conclude that transactions weren't fast enough for my needs.

BTW, dirty reads to _not_ support fragmented tables out of the
box. Neither does the standard mnesia:transaction(F).
You need to use mnesia:activity(Type, F, mnesia_frag).

BR,
Ulf W
--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Roberto Ostinelli]

> dirty_read adds 2 other things (two additional ets lookups) where
> is table located (which node) and the stoarge type, ets or dets.
>
> So dirty_read abstracts away the location and storage type which you
> implicit knows
> when calling ets:lookup/2.

thank you dan, that clears it up.


2009/10/28 Ulf Wiger <ulf.wiger@...>:
> In some cases, this flexibility is not worth as much as the raw
> speed. Personally, I always start out using transactions, and
> always using mnesia:activity(Type, F). I have very rarely had to
> conclude that transactions weren't fast enough for my needs.
>
> BTW, dirty reads to _not_ support fragmented tables out of the
> box. Neither does the standard mnesia:transaction(F).
> You need to use mnesia:activity(Type, F, mnesia_frag).

hi ulf,

i do not get lost in premature optimization :) i have clear specifics
that state a need for me to read 10,000 key/value pairs/sec.
therefore, i'm benchmarking to see various speeds, so that i can then
decide for the best speed/distribution/transaction tradeoff.

i thought i might share the results with you:


1. mnesia table in 6 fragments distributed over 2 erlang nodes, created with:

FragProps = [{node_pool, ['1@...', '2@...']}, {n_fragments,
6}, {n_disc_copies, 1}],
mnesia:create_table(?TABLENAME, [{frag_properties, FragProps},
{attributes, record_info(fields, test)}]).

- write [speed: 370.19 rows/min]:

WriteFun = fun() -> [mnesia:dirty_write({?TABLENAME, K, {}}) || K <-
lists:seq(1, Num)] end,
mnesia:activity(sync_dirty, WriteFun, mnesia_frag).

- read with transaction [speed: 361.10 rows/min]:

ReadFun = fun(Key) -> mnesia:read({?TABLENAME, Key}) end,
[mnesia:activity(transaction, ReadFun, [K], mnesia_frag) || K <-
lists:seq(1, ?NUM)].

- read dirty [speed: 552.25 rows/min]:

ReadFun = fun(Key) -> mnesia:read({?TABLENAME, Key}) end,
[mnesia:activity(sync_dirty, ReadFun, [K], mnesia_frag) || K <-
lists:seq(1, ?NUM)].


2. mnesia table in 6 fragments over a single erlang node, created with:

FragProps = [{node_pool, ['1@...']}, {n_fragments, 6}, {n_disc_copies, 1}],
mnesia:create_table(?TABLENAME, [{frag_properties, FragProps},
{attributes, record_info(fields, test)}]).

- write [speed: 3451.64 rows/min]:

WriteFun = fun() -> [mnesia:dirty_write({?TABLENAME, K, {}}) || K <-
lists:seq(1, Num)] end,
mnesia:activity(sync_dirty, WriteFun).

- read with transaction [speed: 947.47 rows/min]:

ReadFun = fun(Key) -> mnesia:read({?TABLENAME, Key}) end,
[mnesia:activity(transaction, ReadFun, [K], mnesia_frag) || K <-
lists:seq(1, ?NUM)].

- read dirty [speed: 10900.37 rows/min rows/min]:

ReadFun = fun(Key) -> mnesia:read({?TABLENAME, Key}) end,
[mnesia:activity(sync_dirty, ReadFun, [K], mnesia_frag) || K <-
lists:seq(1, ?NUM)].


3. mnesia table no fragments on single node, created with:

mnesia:create_table(?TABLENAME, [{disc_copies, ['1@...',
'2@...']}, {attributes, record_info(fields, test)}]).

- write [speed: 3545.29 rows/min]:

WriteFun = fun() -> [mnesia:dirty_write({?TABLENAME, K, {}}) || K <-
lists:seq(1, Num)] end,
mnesia:activity(sync_dirty, WriteFun).

- read with transaction [speed: 2148.44 rows/min]:

[mnesia:transaction(fun(K) -> mnesia:read({?TABLENAME, K}) end) || K
<- lists:seq(1, ?NUM)].

- read dirty [speed: 29384.40 rows/min]:

[mnesia:dirty_read(?TABLENAME, K) || K <- lists:seq(1, ?NUM)].


4. mnesia table with no fragments on single node.

- read [speed: 91975.16 rows/min]


interestingly enough:

a. write on mnesia table no fragments on single node is same speed
than with fragments.
b. read [transaction/dirty] on mnesia table without fragments is 3x
faster than with fragments.
c. using ets to read a disc_copy not fragmented mnesia table is 3x
faster than using mnesia dirty_write.


does this meet your expectations? any comments always welcome :)

r.

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org