Migrating from Plone 4.1.4 ZEO data.fs and blobs to Relstorage

[Sam Wilson-4]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi List,

We need to migrate an existing test environment running Plone 4.1.4
with ZEO to a new environment on RelStorage.

I have configured zodbconvert to import the data.fs however I cannot
for the life of me find doco on how to migrate the blob (bushy layout)
into RelStorage. I am working with content exported via
collective.recipe.backup

If anyone could give an assist with this one would be a life saver!

Cheers,

Sam
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iF4EAREIAAYFAk+HcmYACgkQFdt86iEfl/cgUAD/Z8Rl8ju30WPvrQWPGoAXHlOc
l1SNV9cEhlgImApeufQBAKs9O840XuVRpldMyt37YtQ/nomXca1hy4yyZJp6IlqO
=PpMq
-----END PGP SIGNATURE-----



_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

[Maurits van Rees-3]

Op 13-04-12 02:25, Sam Wilson schreef:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Hi List,
>
> We need to migrate an existing test environment running Plone 4.1.4
> with ZEO to a new environment on RelStorage.
>
> I have configured zodbconvert to import the data.fs however I cannot
> for the life of me find doco on how to migrate the blob (bushy layout)
> into RelStorage. I am working with content exported via
> collective.recipe.backup
>
> If anyone could give an assist with this one would be a life saver!

Look at the blob related options, if you have not done so already:
http://pypi.python.org/pypi/RelStorage#relstorage-options
I do not look at relstorage and blobstorage often enough to remember how
everything fits together. :-)

I am not aware of a method to migrate filesystem-based blobstorage to a
relational database.

The current filesystem-based blobs should just work, as long as you have
enabled the shared-blob-dir option and your blob-dir is shared among all
clients via NFS or something similar.

Cheers,

--
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl

_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

[Sam Wilson-4]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi All,

Hanno gave me a hand off list with a few bits of config that I
couldn't find listed in the official documentation on
http://pypi.python.org/pypi/RelStorage#zodbconvert

For the benefit of others finding this, my final zodbconvert.cfg ended
up like;

<filestorage source>
  # ZEO Data.fs
  path var/filestorage/Data.fs
  # ZEO bushy blobstorage content
  blob-dir var/blobstorage.bushy
</filestorage>

<relstorage destination>
  shared-blob-dir false
  # ZODB Cache Dir
  blob-dir ./var/blobstorage
  blob-cache-size 10mb
  <mysql>
    db zodb
    user dbuser
    passwd dbpass
  </mysql>
</relstorage>

This seems to have pulled all the content into MySQL.

Cheers,

Sam
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iF4EAREIAAYFAk+LaYIACgkQFdt86iEfl/dlBQEAm4qftwIYWWAuyf2adp8MwEty
kmUG9IaeoRut8TvBBNIBAJ3xvndbdGeZRvKxTFYBruHaGLwOasbE5qUGReyZfMnG
=9UEa
-----END PGP SIGNATURE-----
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

[Sean Upton]

On Sun, Apr 15, 2012 at 6:36 PM, Sam Wilson <sam@...> wrote:
> Hanno gave me a hand off list with a few bits of config that I
> couldn't find listed in the official documentation on
> http://pypi.python.org/pypi/RelStorage#zodbconvert

Once you figure out the configuration options, zodbconvert is indeed a
gem -- should you ever feel inclined, it also makes a decent way to
backup RelStorage to a FileStorage should that be useful for your
backup processes.

I use this to rsync gzipped daily FileStorage snapshots of my database
from production RelStorage to development on my workstation (which I
load into RelStorage via another zodbconvert step), and it also
supplements my pgdump backups from PostgreSQL.

Sean
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev

[M.J. Pieters]

On Fri, Apr 13, 2012 at 02:25, Sam Wilson <sam@...> wrote:
> I have configured zodbconvert to import the data.fs however I cannot
> for the life of me find doco on how to migrate the blob (bushy layout)
> into RelStorage.

Note that I cannot recommend putting ZODB blobs into any relstorage
database apart from Oracle. PostgreSQL doesn't handle large blobs very
well (storing them internally as a series of chunks in a dedicated
table), while MySQL doesn't really support blobs at all.

We have one customer on RelStorage on a Oracle cluster (RelStorage was
commissioned for them, in fact) where all ZODB blobs also live in the
same database. Performance is quite decent but the customer doesn't
try to store more than a few MBs per file.

Better keep them in a NFS setup for most cases.

--
Martijn Pieters
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@...
https://mail.zope.org/mailman/listinfo/zodb-dev