Missing Content

[Michael Havard]

We're missing 16 days worth of content.

Python 2.3.5, Zope 2.8.4, Plone 2.1.2, and a host of other products. ZEO
configuration. Persistent disk caching is on.

Yesterday everything was fine. All the content was there and accessible.
Today I try to log in and the server doesn't respond. I go out and restart
the zope client, still no response. I restart the zeo server and the site
delivers content. Problem! The content is from two weeks ago. All of the
content from the last 16 days is missing. Thinking maybe someone restored
from an older backup I went out to our daily backups and pulled the previous
days data.fs backup. A restore didn't work. It still shows the outdated
content. I went out to look at the backups again and noticed that the date
on the data.fs was 4/14 while the date on data.fs.tmp was 4/26. data.fs.tmp
is a 9 meg file.

Is the missing content somehow captured in data.fs.tmp or is it somewhere
else that I can retrieve it? If it is how do I get it back?



_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
http://mail.zope.org/mailman/listinfo/zodb-dev

[Jim Fulton]

Michael Havard wrote:

> We're missing 16 days worth of content.
>
> Python 2.3.5, Zope 2.8.4, Plone 2.1.2, and a host of other products. ZEO
> configuration. Persistent disk caching is on.
>
> Yesterday everything was fine. All the content was there and accessible.
> Today I try to log in and the server doesn't respond. I go out and restart
> the zope client, still no response. I restart the zeo server and the site
> delivers content. Problem! The content is from two weeks ago. All of the
> content from the last 16 days is missing. Thinking maybe someone restored
> from an older backup I went out to our daily backups and pulled the previous
> days data.fs backup. A restore didn't work. It still shows the outdated
> content. I went out to look at the backups again and noticed that the date
> on the data.fs was 4/14 while the date on data.fs.tmp was 4/26. data.fs.tmp
> is a 9 meg file.

If I had to guess, I would guess that, at some point, someone removed
Data.fs and restored it from backup without restarting the process.
If that happened, the process would still be writing to the removed
database.  That's just a wild guess.  I wonder if the Data.fs file matches
a backup made on the 14th.

> Is the missing content somehow captured in data.fs.tmp or is it somewhere
> else that I can retrieve it?

Unlikely.

Jim

--
Jim Fulton           mailto:jim@...       Python Powered!
CTO                  [phone]            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
http://mail.zope.org/mailman/listinfo/zodb-dev

[Dennis Allison]

What about permissions on the Data.fs?  If ZEO could not write the Data.fs
but could maintain the persistent cache, that would explain what you
observed.  


On Mon, 1 May 2006, Jim Fulton wrote:

> Michael Havard wrote:
> > We're missing 16 days worth of content.
> >
> > Python 2.3.5, Zope 2.8.4, Plone 2.1.2, and a host of other products. ZEO
> > configuration. Persistent disk caching is on.
> >
> > Yesterday everything was fine. All the content was there and accessible.
> > Today I try to log in and the server doesn't respond. I go out and restart
> > the zope client, still no response. I restart the zeo server and the site
> > delivers content. Problem! The content is from two weeks ago. All of the
> > content from the last 16 days is missing. Thinking maybe someone restored
> > from an older backup I went out to our daily backups and pulled the previous
> > days data.fs backup. A restore didn't work. It still shows the outdated
> > content. I went out to look at the backups again and noticed that the date
> > on the data.fs was 4/14 while the date on data.fs.tmp was 4/26. data.fs.tmp
> > is a 9 meg file.
>
> If I had to guess, I would guess that, at some point, someone removed
> Data.fs and restored it from backup without restarting the process.
> If that happened, the process would still be writing to the removed
> database.  That's just a wild guess.  I wonder if the Data.fs file matches
> a backup made on the 14th.
>
> > Is the missing content somehow captured in data.fs.tmp or is it somewhere
> > else that I can retrieve it?
>
> Unlikely.
>
> Jim
>
>

--

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
http://mail.zope.org/mailman/listinfo/zodb-dev

[Dieter Maurer]

Michael Havard wrote at 2006-5-1 06:57 -0500:
>Python 2.3.5, Zope 2.8.4, Plone 2.1.2, and a host of other products. ZEO
>configuration. Persistent disk caching is on.
>
>Yesterday everything was fine. All the content was there and accessible.
>Today I try to log in and the server doesn't respond. I go out and restart
>the zope client, still no response. I restart the zeo server and the site
>delivers content. Problem! The content is from two weeks ago. All of the
>content from the last 16 days is missing.

I have seen such a behaviour when a colleague moved the storage file
without stopping the ZEO server and then copied a new storage file
at the old place.

  Under *nix, moving/deleting a file does not affect processes
  which have this file opened. Such processes continue to
  work with the open file.


In the case described above, the ZEO server continued to use
the old storage file and ignored the copy. Of course, as soon
as the ZEO server was restarted, the new ZEO server used the copy --
and did not see anything that went into the moved away storage file.

In our case, the moved file was still present and we could recover
all modifications. If we had removed the file instead of moving it,
then only experts able to restore deleted files would have been able
to get our modifications back.

--
Dieter
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
http://mail.zope.org/mailman/listinfo/zodb-dev

[Jim Fulton]

Dennis Allison wrote:
> What about permissions on the Data.fs?  If ZEO could not write the Data.fs
> but could maintain the persistent cache, that would explain what you
> observed.  

No.  If the storage server couldn't write to the file, it would fail.
Only the server writes to Data.fs.  The client and it's cache are not
involved.  In fact, this has nothing to do with ZEO.,

Jim

--
Jim Fulton           mailto:jim@...       Python Powered!
CTO                  [phone]            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
http://mail.zope.org/mailman/listinfo/zodb-dev

[Michael Havard]

 The backup data.fs for the last 16 days has a timestamp of 4/14/2006 and
has not changed in size. Noone has restored from backup during the last 16
days. And, that I'm aware of, noone has moved files. The data.fs.index and
data.fs.lock are both timestamped at 4/17 and haven't changed in any of the
backups. data.fs.tmp is the only file listed in the backups that has
consistently changed day to day, both in timestamp and size.


"Jim Fulton" <jim@...> wrote in message
news:44565AE4.1060305@......

> Dennis Allison wrote:
>> What about permissions on the Data.fs?  If ZEO could not write the
>> Data.fs
>> but could maintain the persistent cache, that would explain what you
>> observed.
>
> No.  If the storage server couldn't write to the file, it would fail.
> Only the server writes to Data.fs.  The client and it's cache are not
> involved.  In fact, this has nothing to do with ZEO.,
>
> Jim
>
> --
> Jim Fulton           mailto:jim@...       Python Powered!
> CTO                  [phone]            http://www.python.org
> Zope Corporation     http://www.zope.com       http://www.zope.org
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev@...
> http://mail.zope.org/mailman/listinfo/zodb-dev
>

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@...
http://mail.zope.org/mailman/listinfo/zodb-dev