Corrupt database in mnesia

[Yogish Baliga]

Hello all,

  On our production box, I have seen mnesia getting corrupt once in a while. LATEST.LOG, PREVIOUS.LOG becomes 150G or so and 1st fragment of .DCL file reaches 6G or so. Restarting mnesia again will take forever to recover. But after deleting LATEST.LOG, PREVIOUS.LOG file and restarting mnesia will recover pretty fast and the data is intact in the database.

  2 questions regarding this issue

    1. What is causing the corruption of .LOG and .DCL files?
    2. Why is that removing LATEST.LOG, PREVIOUS.LOG and restarting mnesia recovers pretty fast and data is not lost?
    3. How to prevent this from happening again?

Thanx,
-- baliga

 
"Point of view is worth 80 IQ points" --Alan Kay

http://dudefrommangalore.blogspot.com/

[Kenneth Lundin]

Hi,

If you have a previous log with size 150G I suppose you have run with
mensia overloaded for
quite some time.

What is your mnesia setup ? disk_copy tables or what?
How big entries in the tables?
The PREVIOUS.LOG is the transactions whose result is not yet written
into the tables. I.e
a backlog of things to do.
When you delete the PREVIOUS and LATEST you will miss all the
transactions not yet
reflected in the tables.

/Kenneth Erlang/OTP Ericsson

On Sun, Nov 1, 2009 at 10:07 PM, Yogish Baliga <yogishb@...> wrote:

> Hello all,
>
>  On our production box, I have seen mnesia getting corrupt once in a while. LATEST.LOG, PREVIOUS.LOG becomes 150G or so and 1st fragment of .DCL file reaches 6G or so. Restarting mnesia again will take forever to recover. But after deleting LATEST.LOG, PREVIOUS.LOG file and restarting mnesia will recover pretty fast and the data is intact in the database.
>
>  2 questions regarding this issue
>
>    1. What is causing the corruption of .LOG and .DCL files?
>    2. Why is that removing LATEST.LOG, PREVIOUS.LOG and restarting mnesia recovers pretty fast and data is not lost?
>    3. How to prevent this from happening again?
>
> Thanx,
> -- baliga
>
>
> "Point of view is worth 80 IQ points" --Alan Kay
>
> http://dudefrommangalore.blogspot.com/

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Yogish Baliga]

Hello Kenneth,

  Thanx for the response.

  Mnesia is setup as disk_copy tables.

  These tables store the logs. So loosing them was not a major issues. I did not see Mnesia overloaded message in the Erlang VM log file.

  In any case, I figured out the root cause of huge transaction log file. I was emulating the bag semantics of the table at application level by storing the the log entries as list. Every write was fetching the existing list, adding the current log to it and writing it back to table. :-(. This has dramatic effect only after having a large number of entries in the list. The current implementation started as a hack for log storage and ended up in production environment. I am in the process of changing the table type of bag.

Cheers,
-- baliga
 
"Point of view is worth 80 IQ points" --Alan Kay

http://dudefrommangalore.blogspot.com/





________________________________
From: Kenneth Lundin <kenneth.lundin@...>
To: Yogish Baliga <yogishb@...>
Cc: Erlang Questions <erlang-questions@...>
Sent: Mon, November 2, 2009 11:21:31 PM
Subject: Re: [erlang-questions] Corrupt database in mnesia

Hi,

If you have a previous log with size 150G I suppose you have run with
mensia overloaded for
quite some time.

What is your mnesia setup ? disk_copy tables or what?
How big entries in the tables?
The PREVIOUS.LOG is the transactions whose result is not yet written
into the tables. I.e
a backlog of things to do.
When you delete the PREVIOUS and LATEST you will miss all the
transactions not yet
reflected in the tables.

/Kenneth Erlang/OTP Ericsson

On Sun, Nov 1, 2009 at 10:07 PM, Yogish Baliga <yogishb@...> wrote:

> Hello all,
>
>  On our production box, I have seen mnesia getting corrupt once in a while. LATEST.LOG, PREVIOUS.LOG becomes 150G or so and 1st fragment of .DCL file reaches 6G or so. Restarting mnesia again will take forever to recover. But after deleting LATEST.LOG, PREVIOUS.LOG file and restarting mnesia will recover pretty fast and the data is intact in the database.
>
>  2 questions regarding this issue
>
>    1. What is causing the corruption of .LOG and .DCL files?
>    2. Why is that removing LATEST.LOG, PREVIOUS.LOG and restarting mnesia recovers pretty fast and data is not lost?
>    3. How to prevent this from happening again?
>
> Thanx,
> -- baliga
>
>
> "Point of view is worth 80 IQ points" --Alan Kay
>
> http://dudefrommangalore.blogspot.com/

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org