meta store.

[Bas Driessen]

Hello,

Question regarding meta store. There is a function to update the meta store as follows:

gda_connection_update_meta_store

When I run this, it waits about 10 seconds or so, so I assume it is building the meta store. However, when it is finished, I don't see any output. I would expect files to show up in etc/libgda-4.0, but nothing is added there. Is there meant to be any output?

Also if this generates a structure in memory only, does that mean that I have to call this function every time when I start my application?

My understanding was, that the meta store was written to an XML file and that the libgda would reference that. Only when a table layout has changed, you would run the gda_connection_update_meta_store function to update the XML files.

Anyone who can shed some light on this?

Thank you,
Bas.


_______________________________________________
gnome-db-list mailing list
gnome-db-list@...
http://mail.gnome.org/mailman/listinfo/gnome-db-list

[Murray Cumming]

On Fri, 2009-07-17 at 21:26 +1000, Bas Driessen wrote:
> Hello,
>
> Question regarding meta store. There is a function to update the meta
> store as follows:
>
> gda_connection_update_meta_store
>
> When I run this, it waits about 10 seconds or so,

It was slow for Glom too, so we found a way to optimize it:
http://bugzilla.gnome.org/show_bug.cgi?id=575235#c9

I still don't understand how the optimization works, but at least we
have actual code to do it:
http://git.gnome.org/cgit/libgdamm/tree/libgda/src/connection.ccg#n367
and here you can see the "public" (PostgreSQL) and "main" schema_name
strings that we used:
http://git.gnome.org/cgit/glom/commit/?id=087d27f48f080a3609142514ed5bf54ff0472ff2

>  so I assume it is building the meta store. However, when it is
> finished, I don't see any output. I would expect files to show up in
> etc/libgda-4.0, but nothing is added there. Is there meant to be any
> output?
>
> Also if this generates a structure in memory only, does that mean that
> I have to call this function every time when I start my application?
>
> My understanding was, that the meta store was written to an XML file
> and that the libgda would reference that. Only when a table layout has
> changed, you would run the gda_connection_update_meta_store function
> to update the XML files.
>
> Anyone who can shed some light on this?
>
> Thank you,
> Bas.
>
> _______________________________________________
> gnome-db-list mailing list
> gnome-db-list@...
> http://mail.gnome.org/mailman/listinfo/gnome-db-list

--
murrayc@...
www.murrayc.com
www.openismus.com

_______________________________________________
gnome-db-list mailing list
gnome-db-list@...
http://mail.gnome.org/mailman/listinfo/gnome-db-list

[Bas Driessen]

On Fri, 2009-07-17 at 13:35 +0200, Murray Cumming wrote:

On Fri, 2009-07-17 at 21:26 +1000, Bas Driessen wrote:
> Hello,
> 
> Question regarding meta store. There is a function to update the meta
> store as follows:
> 
> gda_connection_update_meta_store
> 
> When I run this, it waits about 10 seconds or so,

It was slow for Glom too, so we found a way to optimize it:
http://bugzilla.gnome.org/show_bug.cgi?id=575235#c9

I still don't understand how the optimization works, but at least we
have actual code to do it:
http://git.gnome.org/cgit/libgdamm/tree/libgda/src/connection.ccg#n367
and here you can see the "public" (PostgreSQL) and "main" schema_name
strings that we used:
http://git.gnome.org/cgit/glom/commit/?id=087d27f48f080a3609142514ed5bf54ff0472ff2

>  so I assume it is building the meta store. However, when it is
> finished, I don't see any output. I would expect files to show up in
> etc/libgda-4.0, but nothing is added there. Is there meant to be any
> output?
> 
> Also if this generates a structure in memory only, does that mean that
> I have to call this function every time when I start my application? 
> 
> My understanding was, that the meta store was written to an XML file
> and that the libgda would reference that. Only when a table layout has
> changed, you would run the gda_connection_update_meta_store function
> to update the XML files.
> 
> Anyone who can shed some light on this?
> 

Thanks Murray. This answered partially my questions.

Where is the meta data stored? You say that it is stored in a sqlite database in memory. So nothing is written to disk (as in a file) correct? This means that as soon as I close the application all is gone and therefore I have to call the gda_connection_update_meta_store again when I start the application, correct?

Further question I have reading your links. There is some talk about caching the meta store to speed up things. I don't believe there is a section of how that exactly works in the libgda documentation. How would this work?

Thanks again,
Bas.


_______________________________________________
gnome-db-list mailing list
gnome-db-list@...
http://mail.gnome.org/mailman/listinfo/gnome-db-list

[Murray Cumming]

On Fri, 2009-07-17 at 21:55 +1000, Bas Driessen wrote:
> Further question I have reading your links. There is some talk about
> caching the meta store to speed up things. I don't believe there is a
> section of how that exactly works in the libgda documentation. How
> would
> this work?

That's just some code in Glom. We just avoid requesting an update of the
meta store too often, when we are sure that the database structure has
not changed.

--
murrayc@...
www.murrayc.com
www.openismus.com

_______________________________________________
gnome-db-list mailing list
gnome-db-list@...
http://mail.gnome.org/mailman/listinfo/gnome-db-list

[Vivien Malerba]

2009/7/17 Bas Driessen <bas.driessen@...>

Hello,

Question regarding meta store. There is a function to update the meta store as follows:

gda_connection_update_meta_store

When I run this, it waits about 10 seconds or so, so I assume it is building the meta store. However, when it is finished, I don't see any output. I would expect files to show up in etc/libgda-4.0, but nothing is added there. Is there meant to be any output?

Also if this generates a structure in memory only, does that mean that I have to call this function every time when I start my application?

My understanding was, that the meta store was written to an XML file and that the libgda would reference that. Only when a table layout has changed, you would run the gda_connection_update_meta_store function to update the XML files.

Meta data storage has changed since V3. In V3, getting the meta data was quicker and created an XML file which was loaded in memory even if unused. There were problems with that approach:
* it used memory even when not required
* not much meta data was being fetched (which is why it was so fast), the main missing feature was that database schemas were not handled.
* the loading time would have increased and the consummed memory as well if I had added the missing meta data
* it was almost impossible to the API used to add some personnal data to the dictionary.

So in V4:
* the meta data is now stored in a database instead of an XML file
* the data is not loaded in main memory if not needed
* the meta data being fetched now includes schemas, triggers, ...
* the database used to store meta data from a connection can be configured, see http://library.gnome.org/devel/libgda/stable/gda-dict.html#GdaMetaStoreSetup. By default a temporary database is created, destroyed when the connection is closed (more exactly when the GdaMetaStore object associated to the connection is destroyed).

it's up to the user to handle the meta data updates, probably one time to get the complete meta data the first time, and then incremental updates when database objects are created or removed. This is what Libgda's tools (the gda-sql console program and the upcomming gda-browser UI program) do: then create an SQLite database to store meta data in $HOME/.local/share/libgda

Vivien


_______________________________________________
gnome-db-list mailing list
gnome-db-list@...
http://mail.gnome.org/mailman/listinfo/gnome-db-list

[Bas Driessen]

On Fri, 2009-07-17 at 14:08 +0200, Vivien Malerba wrote:

2009/7/17 Bas Driessen <bas.driessen@...>

Hello,

Question regarding meta store. There is a function to update the meta store as follows:

gda_connection_update_meta_store

When I run this, it waits about 10 seconds or so, so I assume it is building the meta store. However, when it is finished, I don't see any output. I would expect files to show up in etc/libgda-4.0, but nothing is added there. Is there meant to be any output?

Also if this generates a structure in memory only, does that mean that I have to call this function every time when I start my application?

My understanding was, that the meta store was written to an XML file and that the libgda would reference that. Only when a table layout has changed, you would run the gda_connection_update_meta_store function to update the XML files.

Meta data storage has changed since V3. In V3, getting the meta data was quicker and created an XML file which was loaded in memory even if unused. There were problems with that approach:
* it used memory even when not required
* not much meta data was being fetched (which is why it was so fast), the main missing feature was that database schemas were not handled.
* the loading time would have increased and the consummed memory as well if I had added the missing meta data
* it was almost impossible to the API used to add some personnal data to the dictionary.

So in V4:
* the meta data is now stored in a database instead of an XML file
* the data is not loaded in main memory if not needed
* the meta data being fetched now includes schemas, triggers, ...
* the database used to store meta data from a connection can be configured, see http://library.gnome.org/devel/libgda/stable/gda-dict.html#GdaMetaStoreSetup. By default a temporary database is created, destroyed when the connection is closed (more exactly when the GdaMetaStore object associated to the connection is destroyed).

it's up to the user to handle the meta data updates, probably one time to get the complete meta data the first time, and then incremental updates when database objects are created or removed. This is what Libgda's tools (the gda-sql console program and the upcomming gda-browser UI program) do: then create an SQLite database to store meta data in $HOME/.local/share/libgda

Thanks Vivien. So if I use a postgresql database, will the meta data be stored in a postgresql table? If so where is it, as I can not see it in the psql command line tool.

In order to cache, I need to prevent that this meta data is removed upon exiting my application, correct? Is that all explained of how to do that in the link you sent me?  Well, I probably better read it first :)

Bas.
_______________________________________________
gnome-db-list mailing list
gnome-db-list@...
http://mail.gnome.org/mailman/listinfo/gnome-db-list

[Vivien Malerba]

2009/7/17 Bas Driessen <bas.driessen@...>

On Fri, 2009-07-17 at 14:08 +0200, Vivien Malerba wrote:

2009/7/17 Bas Driessen <bas.driessen@...>

Hello,

Question regarding meta store. There is a function to update the meta store as follows:

gda_connection_update_meta_store

When I run this, it waits about 10 seconds or so, so I assume it is building the meta store. However, when it is finished, I don't see any output. I would expect files to show up in etc/libgda-4.0, but nothing is added there. Is there meant to be any output?

Also if this generates a structure in memory only, does that mean that I have to call this function every time when I start my application?

My understanding was, that the meta store was written to an XML file and that the libgda would reference that. Only when a table layout has changed, you would run the gda_connection_update_meta_store function to update the XML files.

Meta data storage has changed since V3. In V3, getting the meta data was quicker and created an XML file which was loaded in memory even if unused. There were problems with that approach:
* it used memory even when not required
* not much meta data was being fetched (which is why it was so fast), the main missing feature was that database schemas were not handled.
* the loading time would have increased and the consummed memory as well if I had added the missing meta data
* it was almost impossible to the API used to add some personnal data to the dictionary.

So in V4:
* the meta data is now stored in a database instead of an XML file
* the data is not loaded in main memory if not needed
* the meta data being fetched now includes schemas, triggers, ...
* the database used to store meta data from a connection can be configured, see http://library.gnome.org/devel/libgda/stable/gda-dict.html#GdaMetaStoreSetup. By default a temporary database is created, destroyed when the connection is closed (more exactly when the GdaMetaStore object associated to the connection is destroyed).

it's up to the user to handle the meta data updates, probably one time to get the complete meta data the first time, and then incremental updates when database objects are created or removed. This is what Libgda's tools (the gda-sql console program and the upcomming gda-browser UI program) do: then create an SQLite database to store meta data in $HOME/.local/share/libgda

Thanks Vivien. So if I use a postgresql database, will the meta data be stored in a postgresql table? If so where is it, as I can not see it in the psql command line tool.

It's up to you to decide where you want the meta data to be stored:
* it will be by default in a temporary SQLite database which is not what you want
* it can be stored in your PostgreSQL database along with your data
* it can be in a separate SQLite file

In order to cache, I need to prevent that this meta data is removed upon exiting my application, correct? Is that all explained of how to do that in the link you sent me?  Well, I probably better read it first :)

There some sample code there, and you can then have a look at the gda-sql.c's code.
 

Vivien

_______________________________________________
gnome-db-list mailing list
gnome-db-list@...
http://mail.gnome.org/mailman/listinfo/gnome-db-list