Any NGS Database Dumps ?

[Paul Taylor-9]

Hi

I just tried to run the upgrade.sh script to convert my 20090524 data to
NGS, unsuprisingly it didn't work, I assume because I don't have a full
Musicbrainz server, just a database running on OSX, and I really don't
fancy trying to get a server working. If somebody has converted the db
successfully it would be great if db dumps could be made available to
load  into a new database created as described at
http://musicbrainz.org/doc/Database_Installation

Paul/ijabz

_______________________________________________
MusicBrainz-devel mailing list
MusicBrainz-devel@...
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

[Paul Taylor-9]

Paul Taylor wrote:

> Hi
>
> I just tried to run the upgrade.sh script to convert my 20090524 data to
> NGS, unsuprisingly it didn't work, I assume because I don't have a full
> Musicbrainz server, just a database running on OSX, and I really don't
> fancy trying to get a server working. If somebody has converted the db
> successfully it would be great if db dumps could be made available to
> load  into a new database created as described at
> http://musicbrainz.org/doc/Database_Installation
>
> Paul/ijabz
>  

Me again.

I've spent a good few hours trying to setup mb_server on a mac without
success, Ive had help from quite a few different people but it still
doesn't work - which is fustrating. The other option is to try an use a
virtual machine, which Ive also found very awkward in the past, and I
expect I'll hit other problems if I go down this route including
performance. The INSTALL instructions have clearly been written with
only Linux in mind, and I think it would be of benefit if we had a
seperate INSTALL file for the Mac which clearly states how various parts
such as  Postgres should be installed, after all half the core
developers seem to actually use Macs as their native os.

When installing something like mb_server I don't mind following the
installations instructions exactly and installing things in places I
might not as a first choice, if I'm confident that the installation is
going to work, but the current instructions are very wooly in places
with lines like

''Before you start, you need to install a C extension on your database
server. You will
    need libunac (https://gna.org/projects/unac/) and PostgreSQL
development headers. '

If someone has actually run upgrade.sh successfully couldn't that data
to be uploaded to musicbrainz in the form of the existing data dumps to
allow those interested in the db but not mb_server to use it.


Paul



_______________________________________________
MusicBrainz-devel mailing list
MusicBrainz-devel@...
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

[Oliver Charles-2]

On Fri, Sep 25, 2009 at 11:41 AM, Paul Taylor <paul_t100@...> wrote:

> I've spent a good few hours trying to setup mb_server on a mac without
> success, Ive had help from quite a few different people but it still
> doesn't work - which is fustrating. The other option is to try an use a
> virtual machine, which Ive also found very awkward in the past, and I
> expect I'll hit other problems if I go down this route including
> performance. The INSTALL instructions have clearly been written with
> only Linux in mind, and I think it would be of benefit if we had a
> seperate INSTALL file for the Mac which clearly states how various parts
> such as  Postgres should be installed, after all half the core
> developers seem to actually use Macs as their native os.

No, I do not agree. It is not MusicBrainz responsibility to tell
people how to setup Postgres. The INSTALL file is not really Linux
specific, the process is the same on all systems -- run the
Makefile.PL, make sure all the dependencies it complains about are
installed, and then proceed to setup a database dump. This latter step
is complex at the moment, because there are no official database dumps
for NGS out, so you have to setup another mb_server version, import on
that, and then upgrade it.

> When installing something like mb_server I don't mind following the
> installations instructions exactly and installing things in places I
> might not as a first choice, if I'm confident that the installation is
> going to work, but the current instructions are very wooly in places
> with lines like
>
> ''Before you start, you need to install a C extension on your database
> server. You will
>    need libunac (https://gna.org/projects/unac/) and PostgreSQL
> development headers. '

And how is this wooly? It's perfectly clear - we need a custom
Postgres extension, and this mentions what the requirements to build
that are.

> If someone has actually run upgrade.sh successfully couldn't that data
> to be uploaded to musicbrainz in the form of the existing data dumps to
> allow those interested in the db but not mb_server to use it.

I'd love to see some database dumps too, as my upgrade seems to take
years to complete for some reason

--
    Oliver Charles / aCiD2

_______________________________________________
MusicBrainz-devel mailing list
MusicBrainz-devel@...
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

[Paul Taylor-9]

Oliver Charles wrote:

> On Fri, Sep 25, 2009 at 11:41 AM, Paul Taylor <paul_t100@...> wrote:
>  
>> I've spent a good few hours trying to setup mb_server on a mac without
>> success, Ive had help from quite a few different people but it still
>> doesn't work - which is fustrating. The other option is to try an use a
>> virtual machine, which Ive also found very awkward in the past, and I
>> expect I'll hit other problems if I go down this route including
>> performance. The INSTALL instructions have clearly been written with
>> only Linux in mind, and I think it would be of benefit if we had a
>> seperate INSTALL file for the Mac which clearly states how various parts
>> such as  Postgres should be installed, after all half the core
>> developers seem to actually use Macs as their native os.
>>    
>
> No, I do not agree. It is not MusicBrainz responsibility to tell
> people how to setup Postgres.

It is if Musicbrainz expects Postgres setup in a certain way. Ive had
Postgres setup fro months working perfectly well, but whether it is
setup as required for the mb_server install is another matter.... The
instructions for setting up the Java Lucene server explicitly mention
all the issues found and how to work round them (the downside is the
process appears more complex than it actually is) , whereas for
mb_server some of these issues are glossed over.

> The INSTALL file is not really Linux
> specific, the process is the same on all systems -- run the
> Makefile.PL, make sure all the dependencies it complains about are
> installed, and then proceed to setup a database dump. This latter step
> is complex at the moment, because there are no official database dumps
> for NGS out, so you have to setup another mb_server version, import on
> that, and then upgrade it.
>
>  
>> When installing something like mb_server I don't mind following the
>> installations instructions exactly and installing things in places I
>> might not as a first choice, if I'm confident that the installation is
>> going to work, but the current instructions are very wooly in places
>> with lines like
>>
>> ''Before you start, you need to install a C extension on your database
>> server. You will
>>    need libunac (https://gna.org/projects/unac/) and PostgreSQL
>> development headers. '
>>    
>
> And how is this wooly? It's perfectly clear - we need a custom
> Postgres extension, and this mentions what the requirements to build
> that are.
>
>  

Its woolly, because its been suggssted to build Postgres from source,
install the package , install using MacPorts . It doesnt define what
version of libunac to install and what to do with it, its not clear
where the Postgrs dev headers come from and whether they are installed
as part of the postgres install.
>> If someone has actually run upgrade.sh successfully couldn't that data
>> to be uploaded to musicbrainz in the form of the existing data dumps to
>> allow those interested in the db but not mb_server to use it.
>>    
>
> I'd love to see some database dumps too, as my upgrade seems to take
> years to complete for some reason
>
>  
Heres hoping ;)

_______________________________________________
MusicBrainz-devel mailing list
MusicBrainz-devel@...
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

[Oliver Charles-2]

Gah, my last reply went off list. Here is my email and I'll reply with
Paul's below, sorry about the mess!

-----

On Fri, Sep 25, 2009 at 1:05 PM, Paul Taylor <paul_t100@...> wrote:

>> No, I do not agree. It is not MusicBrainz responsibility to tell
>> people how to setup Postgres.
>
> It is if Musicbrainz expects Postgres setup in a certain way. Ive had
> Postgres setup fro months working perfectly well, but whether it is setup as
> required for the mb_server install is another matter.... The instructions
> for setting up the Java Lucene server explicitly mention all the issues
> found and how to work round them (the downside is the process appears more
> complex than it actually is) , whereas for mb_server some of these issues
> are glossed over.

Like what? The only difference I am aware of with mb_server is that
you need to install a custom Postgres extension, and that is
documented in the INSTALL. What is it missing?

>>> When installing something like mb_server I don't mind following the
>>> installations instructions exactly and installing things in places I
>>> might not as a first choice, if I'm confident that the installation is
>>> going to work, but the current instructions are very wooly in places
>>> with lines like
>>>
>>> ''Before you start, you need to install a C extension on your database
>>> server. You will
>>>   need libunac (https://gna.org/projects/unac/) and PostgreSQL
>>> development headers. '
>>>
>>
>> And how is this wooly? It's perfectly clear - we need a custom
>> Postgres extension, and this mentions what the requirements to build
>> that are.
>>
>>
>
> Its woolly, because its been suggssted to build Postgres from source,
> install the package , install using MacPorts . It doesnt define what version
> of libunac to install and what to do with it, its not clear where the
> Postgrs dev headers come from and whether they are installed as part of the
> postgres install.

Dev headers come if you install from source not from binary, so
MacPorts would give you that, as would just downloading the source
from the site. If you run Ubuntu, I would imagine you can bring those
headers in by installing postgresql-dev or something similar. There
are a many different ways, and we can't cover every single setup, we
have to assume that people understand how their system works.

--
   Oliver Charles / aCiD2

_______________________________________________
MusicBrainz-devel mailing list
MusicBrainz-devel@...
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

[Oliver Charles-2]

And this is Paul's reply...

---

Oliver Charles wrote:

> On Fri, Sep 25, 2009 at 1:05 PM, Paul Taylor <paul_t100@...> wrote:
>
>>> No, I do not agree. It is not MusicBrainz responsibility to tell
>>> people how to setup Postgres.
>>>
>> It is if Musicbrainz expects Postgres setup in a certain way. Ive had
>> Postgres setup fro months working perfectly well, but whether it is setup as
>> required for the mb_server install is another matter.... The instructions
>> for setting up the Java Lucene server explicitly mention all the issues
>> found and how to work round them (the downside is the process appears more
>> complex than it actually is) , whereas for mb_server some of these issues
>> are glossed over.
>>
>
> Like what? The only difference I am aware of with mb_server is that
> you need to install a custom Postgres extension, and that is
> documented in the INSTALL. What is it missing?
>
>
>>>> When installing something like mb_server I don't mind following the
>>>> installations instructions exactly and installing things in places I
>>>> might not as a first choice, if I'm confident that the installation is
>>>> going to work, but the current instructions are very wooly in places
>>>> with lines like
>>>>
>>>> ''Before you start, you need to install a C extension on your database
>>>> server. You will
>>>>   need libunac (https://gna.org/projects/unac/) and PostgreSQL
>>>> development headers. '
>>>>
>>>>
>>> And how is this wooly? It's perfectly clear - we need a custom
>>> Postgres extension, and this mentions what the requirements to build
>>> that are.
>>>
>>>
>>>
>> Its woolly, because its been suggssted to build Postgres from source,
>> install the package , install using MacPorts . It doesnt define what version
>> of libunac to install and what to do with it, its not clear where the
>> Postgrs dev headers come from and whether they are installed as part of the
>> postgres install.
>>
>
> Dev headers come if you install from source not from binary, so
> MacPorts would give you that, as would just downloading the source
> from the site.

But installing the package wouldn't (which is what I did) , can't you
see what I mean.
> If you run Ubuntu, I would imagine you can bring those
> headers in by installing postgresql-dev or something similar.
'imagining' , installing stuff is always a pain, but would be easier if
you didnt have to 'imagine' what to do wouldn't it.
> There
> are a many different ways, and we can't cover every single setup, we
> have to assume that people understand how their system works.
>
You can't cover every single way - if people want to have their own
setup thats cool  but providing one way per system that actually works
would mean more people getting involved. I don't need to understand the
intracasies of how my Mac works on other projects , why should I have to
do on this. To give another example I never use Perl, the instructions
for installing Perl modules in
http://bugs.musicbrainz.org/browser/mb_server/branches/RELEASE_20090524-BRANCH/INSTALL
are not clear, if you are  used to Perl this is trivial, but if youre
not it isn't, the latest version of DBD:Pg doesnt won't install on my
Mac so I'm trying to install DBD::Pg version 1.32  but I cannot find the
syntax to specify the version. (I think the assumption is that if you
are downloading mb_server you are a developer familar with Perl )

_______________________________________________
MusicBrainz-devel mailing list
MusicBrainz-devel@...
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

[Oliver Charles-2]

On Fri, Sep 25, 2009 at 1:56 PM, Paul Taylor <paul_t100@...> wrote:
>> Dev headers come if you install from source not from binary, so
>> MacPorts would give you that, as would just downloading the source
>> from the site.

> But installing the package wouldn't (which is what I did) , can't you
> see what I mean.

I understand that, and that's why I later mentioned that we have to
assume people understand how their system works. OSX packages don't
give you source code, so I'd know that if I was told I'd need
development headers, I'd need to build from source.

>> If you run Ubuntu, I would imagine you can bring those
>> headers in by installing postgresql-dev or something similar.

> 'imagining' , installing stuff is always a pain, but would be easier if
> you didnt have to 'imagine' what to do wouldn't it.

I'm only imagining here because I don't use Ubuntu :)

>> There
>> are a many different ways, and we can't cover every single setup, we
>> have to assume that people understand how their system works.

> You can't cover every single way - if people want to have their own
> setup thats cool  but providing one way per system that actually works
> would mean more people getting involved. I don't need to understand the
> intracasies of how my Mac works on other projects , why should I have to
> do on this.

Because MusicBrainz is a complicated piece of software. You're not
installing an isolated application, you are installing a product with
many different parts of functionality. It's simply unfeasible for us
to create something that you can just roll out in one line (at the
moment, though it's certainly a nice goal).

> To give another example I never use Perl, the instructions
> for installing Perl modules in
> http://bugs.musicbrainz.org/browser/mb_server/branches/RELEASE_20090524-BRANCH/INSTALL
> are not clear, if you are  used to Perl this is trivial, but if youre
> not it isn't, the latest version of DBD:Pg doesnt won't install on my
> Mac so I'm trying to install DBD::Pg version 1.32  but I cannot find the
> syntax to specify the version. (I think the assumption is that if you
> are downloading mb_server you are a developer familar with Perl )

Well this isn't INSTALL specific, we specific the minimum versions we
require of just about everything now - it's up to you how you install
them, but going through CPAN is generally the easiest way.

If you're really unhappy with INSTALL, I'm sure we'll happily accept
patches or wiki pages to explain how to set it up on OSX, but I'm
afraid I don't have time to go out of my way and write stuff like this
up :(

--
    Oliver Charles / aCiD2

_______________________________________________
MusicBrainz-devel mailing list
MusicBrainz-devel@...
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

[Robert Kaye]

I've made an NGS dump here:

    ftp://ftp.musicbrainz.org/pub/musicbrainz/data/ngs

The data is quite old already, but this should serve as a starting  
point to get data loaded into an NGS install. If everything works as  
predicted, you should be able to import the file using the InitDb.pl  
command in the NGS codebase (/mb_server/trunk in SVN) just as you  
would in the older mason codebase.

Please let me know if it works or not.

On Sep 23, 2009, at 9:10 AM, Paul Taylor wrote:

> Hi
>
> I just tried to run the upgrade.sh script to convert my 20090524  
> data to
> NGS, unsuprisingly it didn't work, I assume because I don't have a  
> full
> Musicbrainz server, just a database running on OSX, and I really don't
> fancy trying to get a server working. If somebody has converted the db
> successfully it would be great if db dumps could be made available to
> load  into a new database created as described at
> http://musicbrainz.org/doc/Database_Installation
>
> Paul/ijabz
>
> _______________________________________________
> MusicBrainz-devel mailing list
> MusicBrainz-devel@...
> http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

--

--ruaok      "Vague, but exciting..." -- Mike Sendall on the WWW  
proposal.

Robert Kaye     --     rob@...     --    http://mayhem-chaos.net






_______________________________________________
MusicBrainz-devel mailing list
MusicBrainz-devel@...
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel

[Paul Taylor-9]

Robert Kaye wrote:

> I've made an NGS dump here:
>
>    ftp://ftp.musicbrainz.org/pub/musicbrainz/data/ngs
>
> The data is quite old already, but this should serve as a starting
> point to get data loaded into an NGS install. If everything works as
> predicted, you should be able to import the file using the InitDb.pl
> command in the NGS codebase (/mb_server/trunk in SVN) just as you
> would in the older mason codebase.
>
> Please let me know if it works or not.

Yes thanks it did work, (sort of)

I have been unable to get the Perl module DBD:Pg installed so InitDb.pl
didnt work but the instructions at
http://musicbrainz.org/doc/Database_Installation did, this doesn't
mention database_raw (thats an extra step) and it causes the new
database to be created in the default schema rather than musicbrainz but
thats not a problem. Also create indexs fails because I dont have any
function  page_index(character varying) so I comment that out.

To try and get the full ystem working I even removed my working Postgres
Server and reinstalled using MacPorts and it still didn't allow DBD:Pg
to be installed, so I'll have to revisit this some time but for now I
have a setup that alows me do some work on the search server, the only
thing I need now is more time !

BTW the notion of a schema that  is distinct  to that of the user seems
to be postgres concept. If you are now using  a musicbrainz schema  it
would make more sense for the default user  to be  musicbrainz rather
than musicbrainz_user  then  by default they would look in the
musicbrainz  schema before  the public schema without needing to modify
search_path.


Paul

_______________________________________________
MusicBrainz-devel mailing list
MusicBrainz-devel@...
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-devel