A "terms" table

[Matt Mullenweg]

WordPress is like a sandwich.

Assuming we've scared off all the vegetarians with all the talk of BBQ,
the core is the meat. Our meat is the wp_posts table, which stores what
I would refer to as the primary points of content. Currently for us this
is posts, pages, and attachments, though in the future I could see it
expanding to support new post types such as externals, galleries, and
hopefully things we can't even imagine yet.

On the side you have chips (good comments), vegetables (idiot comments),
and that funny stuff your cousin brought that you're going to move
around on the plate but never eat (spam comments). I think comments are
okay right now, maybe they could use a meta table but we can talk about
that later.

Meat alone is only a real meal at rboren's house, so most people put
things on the sandwich to add flavor and spice it up. Some add other
types of meat, in the WP world this is postmeta, which we call custom
fields in polite company.

We also havae condiments which are currently handled by two tables:
wp_categories and wp_post2cat. On the taxonomy/condiment side, right now
we really only allow ketchup aka categories, and users for at least a
year have been asking for more. In 2.2 we decided to satiate their
appetites.

Everyone agrees that ketchup and mayonnaise are totally different, even
though they're both condiments and you put them both on sandwiches. No
one is trying to create some horrible pink mixture of the two tastes.

However there are currently two schools of thought on how we should
store the data for categories and tags at a very low level in our DB.

Let me do my best to make the case for putting category data and tag
data in separate tables, and feel free to chime in if you think I've
missed any points.

* We shouldn't ship anything with a data schema people disagree on,
because plugins and themes will be written against it.
* They're different things, so we should have them in different tables.
* Tags can have things like synonyms, and don't need things like hierarchy.
* There are ugly legacy field names in the category table like
category_nicename, cat_name, cat_ID (wtf capitals) and we can clean
those up in new tables
* With separate tables our queries on the admin side become WAY easier
and cleaner to do, with no bitwise or _count nonsense
* Plugins for tagging have implemented it this way.

The code currently in SVN does something different. It uses the
categories table for names of the tags and then adds fields to hint how
those names are being used for the admin section. If I wanted to make
everyone happy and be popular I would just go with the above since there
seems to be good consensus there, but I think this is an important
long-term decision for WP so let me spell out some reasons why I think
the current design has legs not just for 2.2 but beyond.

1. It performs faster.

On front-end display, we have added ZERO QUERIES to support tags. The
query that grabs categories is also grabbing tags and we're sorting them
out in the code.

In the dashboard some of the queries are more complicated (though not
really any different than what we deal with for link categories) and a
few milliseconds slower than the old ones. However, that really doesn't
matter because 1) we only need to write them once and more importantly
2) they're run several orders of magnitude fewer times than the ones
that display the blog on the front-end. A mantra has always been that
user time is more important than developer time.

A separate tag naming table and post2tag table would require at least 2
additional queries and/or joins to the front page, which already think
does too many queries and is too heavy.

2. It's a better long-term foundation.

I think there are a lot of benefits to having a single ID that maps to a
term and a slug. Let's pretend we had perfect foresight 5 years ago and
instead of wp_categories we had wp_terms.

Regardless of the UI and philosophy behind categories, tags, and ooga
booga, on a data level they're still mapping a set of terms to an item
in post_content.

In WP a term has three important things: an ID, a human-entered name,
and a URL-friendly slug. We use the ID in our relations instead of the
slug because it's more efficient and slugs are not necessarily unique
(because of hierarchy).

Having "dogs" in a category table have one ID and "dogs" in a tag table
have a different ID is a long-term deck of cards that we will seriously
regret later. It's MUCH harder to reconcile items with internally
different IDs than it is to split out unique IDs into different tables.

As for some of the bit and count fields currently causing grief, I would
argue the solution for that isn't a separate tags table, but a separate
table specifically for that type of data. In Drupal for this
infrastructure they have a term_data, term_hiercharchy, term_node,
term_relation, term_synonym, vocabulary, and vocabulary_node_types
tables. I think that might be a little more than we need, but there are
some concepts there we could pretty cleanly combine into a single extra
table that isn't called categories or tags, and will provide a good and
scalable foundation for years to come.

3. There should be no user- or plugin-facing problems with how it's
currently implemented, or if we decide to change it.

Now this isn't to suggest for a second there aren't bugs, many have been
fixed already and I'm sure there are many still left, but that is going
to be true of ANY code we put in WP and anyone who suggests otherwise is
not very familiar with software development. From a point of view of
plugin authors, they shouldn't have to think or care if we're storing it
in a categories table or a turkey, the function they use should remain
consistent no matter what we change or gymnastics we do behind the
curtain. No matter what we do in 2.2 or 2.3, that's not going to change.

I do think there is something intrinsically better about shipping and
iterating than noodling without release in search of the "perfect"
implementation.

More importantly from a user's point of view, all that really matters is
that they have a box they can type tags in and that their host doesn't
tell them not to upgrade to 2.2 because it does more queries.

4. I'm open

I'm not personally tied to any code written thus far and if I think the
best thing is.

There is a separate but related decision around what to do about the
release date. Based on the discussion here I'm going to make go/no-go
decision on Tuesday.

If we do delay I think we should laser-focus on tags and now allow other
pet-issues to creep in, and I will fully expect people to put in as much
time writing code and fixing bugs as they have arguing points on mailing
lists, IRC, and trac. At the very least I hope we've learned a bit more
about getting these things out of the way early rather than a week or
two before a release. Also if something is sitting in trac, take it to
the hackers list early.

I think if we stick with the current implementation we can hit it with a
very stable release next Monday, but if we decide to replace it we need
to push it back at least into mid-May.

--
Matt Mullenweg
  http://photomatt.net | http://wordpress.org
http://automattic.com | http://akismet.com
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Robert Deaton]

On 4/15/07, Matt Mullenweg <m@...> wrote:
> WordPress is like a sandwich.

I would swear you were highly overweight if I didn't know any better.

I will be snipping heavily.

> We also havae condiments which are currently handled by two tables:
> wp_categories and wp_post2cat. On the taxonomy/condiment side, right now
> we really only allow ketchup aka categories, and users for at least a
> year have been asking for more. In 2.2 we decided to satiate their
> appetites.

I wouldn't say we only "allow" ketchup, I'd say in the past we only
provided ketchup on our sandwhich, and if you want mayonnaise aka
tags, you needed to put them on yourself. I still don't necessarily
think this is a bad idea, but I won't argue this point, as removing
tags entirely is almost certainly not going to happen.

> Everyone agrees that ketchup and mayonnaise are totally different, even
> though they're both condiments and you put them both on sandwiches. No
> one is trying to create some horrible pink mixture of the two tastes.

You could've fooled me in the past. :)

> Let me do my best to make the case for putting category data and tag
> data in separate tables, and feel free to chime in if you think I've
> missed any points.
>
> * We shouldn't ship anything with a data schema people disagree on,
> because plugins and themes will be written against it.
> * They're different things, so we should have them in different tables.
> * Tags can have things like synonyms, and don't need things like hierarchy.
> * There are ugly legacy field names in the category table like
> category_nicename, cat_name, cat_ID (wtf capitals) and we can clean
> those up in new tables
> * With separate tables our queries on the admin side become WAY easier
> and cleaner to do, with no bitwise or _count nonsense
> * Plugins for tagging have implemented it this way.
>
> The code currently in SVN does something different. It uses the
> categories table for names of the tags and then adds fields to hint how
> those names are being used for the admin section. If I wanted to make
> everyone happy and be popular I would just go with the above since there
> seems to be good consensus there, but I think this is an important
> long-term decision for WP so let me spell out some reasons why I think
> the current design has legs not just for 2.2 but beyond.
>
> 1. It performs faster.
>
> On front-end display, we have added ZERO QUERIES to support tags. The
> query that grabs categories is also grabbing tags and we're sorting them
> out in the code.

One of the first rules of SQL has always been to not fetch more than
you need, to allow sorting and searching to be done on the SQL side,
etc. This breaks that fundamental rule.

The bitfield also introduces new things. Let's face it, the APIs for
fetching things from the database are horribly incomplete and will
never ever possibly cover all the things that plugins will want to do.
So, how about a plugin that wants to fetch all the tags to do some
magic? Well, with a bitfield, that plugin now has to fetch all the
categories and all the tags, and then sort through with bitwise
operations and discard the categories which they don't care about.
Efficiency? iffy.

> In the dashboard some of the queries are more complicated (though not
> really any different than what we deal with for link categories) and a
> few milliseconds slower than the old ones. However, that really doesn't
> matter because 1) we only need to write them once and more importantly
> 2) they're run several orders of magnitude fewer times than the ones
> that display the blog on the front-end. A mantra has always been that
> user time is more important than developer time.

> A separate tag naming table and post2tag table would require at least 2
> additional queries and/or joins to the front page, which already think
> does too many queries and is too heavy.

I think you missed a noun in the last clause of the sentence. Who
thinks it is? I think that the number of queries shouldn't necessarily
make a _huge_ difference, its the queries themselves, how much they
fetch, how much is actually used, etc. WP takes a "fetch everything
with the assumption it will be used approach", perhaps we need to
rethink that.

> 2. It's a better long-term foundation.
>
> I think there are a lot of benefits to having a single ID that maps to a
> term and a slug. Let's pretend we had perfect foresight 5 years ago and
> instead of wp_categories we had wp_terms.

I still think we'd be having this conversation today.

> Regardless of the UI and philosophy behind categories, tags, and ooga
> booga, on a data level they're still mapping a set of terms to an item
> in post_content.

Are you gradually moving us toward dropping this table altogether and
moving it all into postmeta?

> In WP a term has three important things: an ID, a human-entered name,
> and a URL-friendly slug. We use the ID in our relations instead of the
> slug because it's more efficient and slugs are not necessarily unique
> (because of hierarchy).
>
> Having "dogs" in a category table have one ID and "dogs" in a tag table
> have a different ID is a long-term deck of cards that we will seriously
> regret later. It's MUCH harder to reconcile items with internally
> different IDs than it is to split out unique IDs into different tables.

But like you said earlier, ketchup and mayonnaise are two different
things. Why would we be trying to put them together?

> As for some of the bit and count fields currently causing grief, I would
> argue the solution for that isn't a separate tags table, but a separate
> table specifically for that type of data. In Drupal for this
> infrastructure they have a term_data, term_hiercharchy, term_node,
> term_relation, term_synonym, vocabulary, and vocabulary_node_types
> tables. I think that might be a little more than we need, but there are
> some concepts there we could pretty cleanly combine into a single extra
> table that isn't called categories or tags, and will provide a good and
> scalable foundation for years to come.
>

> 3. There should be no user- or plugin-facing problems with how it's
> currently implemented, or if we decide to change it.

I disagree. Especially because the implementation is lacking so many
features that other tagging plugins have done well for some time now,
new plugins are bound to be forced to write their own queries to
interface with this table. And when they don't work anymore in the
next version, someone's gonna be pissed.

>
> Now this isn't to suggest for a second there aren't bugs, many have been
> fixed already and I'm sure there are many still left, but that is going
> to be true of ANY code we put in WP and anyone who suggests otherwise is
> not very familiar with software development. From a point of view of
> plugin authors, they shouldn't have to think or care if we're storing it
> in a categories table or a turkey, the function they use should remain
> consistent no matter what we change or gymnastics we do behind the
> curtain. No matter what we do in 2.2 or 2.3, that's not going to change.
>
> I do think there is something intrinsically better about shipping and
> iterating than noodling without release in search of the "perfect"
> implementation.

I agree, there is something better about shipping and iterating, as
long as the initial shipment has something that is not fundamentally
broken.

I snipped the rest cause I have more to say but I have to run out for
lunch. I'll come back later.


--
--Robert Deaton
http://lushlab.com
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Ryan Boren]

On 4/15/07, Matt Mullenweg <m@...> wrote:

> Regardless of the UI and philosophy behind categories, tags, and ooga
> booga, on a data level they're still mapping a set of terms to an item
> in post_content.
>
> In WP a term has three important things: an ID, a human-entered name,
> and a URL-friendly slug. We use the ID in our relations instead of the
> slug because it's more efficient and slugs are not necessarily unique
> (because of hierarchy).
>
> Having "dogs" in a category table have one ID and "dogs" in a tag table
> have a different ID is a long-term deck of cards that we will seriously
> regret later. It's MUCH harder to reconcile items with internally
> different IDs than it is to split out unique IDs into different tables.

I agree with these points.  These points are the reason I tried to
make the current design work.

> As for some of the bit and count fields currently causing grief, I would
> argue the solution for that isn't a separate tags table, but a separate
> table specifically for that type of data. In Drupal for this
> infrastructure they have a term_data, term_hiercharchy, term_node,
> term_relation, term_synonym, vocabulary, and vocabulary_node_types
> tables. I think that might be a little more than we need, but there are
> some concepts there we could pretty cleanly combine into a single extra
> table that isn't called categories or tags, and will provide a good and
> scalable foundation for years to come.

The presence of category_parent and the count fields are the crux of
my problem with the unified table.  They cause a lot of grief and kill
any future-proofing you think you're getting by having a unified terms
table.  If you're open to a new table for this information, I think I
could get behind that.  I'll give it some thought.

Ryan
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Chris-212]

You're the boss in this I guess - fwiw why the rush? because some other
blogging systems put out releases more often? bad reason!
The documentation needs sorting for version 2.1 - stability is good - DB
changes are bad unless very carefully implemented and documented so  =
Keep tags out of the way, and don't rush to another release - May/June,
later.. whatever!
Chris

Matt Mullenweg wrote:

> WordPress is like a sandwich.
>
> Assuming we've scared off all the vegetarians with all the talk of
> BBQ, the core is the meat. Our meat is the wp_posts table, which
> stores what I would refer to as the primary points of content.
> Currently for us this is posts, pages, and attachments, though in the
> future I could see it expanding to support new post types such as
> externals, galleries, and hopefully things we can't even imagine yet.
>
> On the side you have chips (good comments), vegetables (idiot
> comments), and that funny stuff your cousin brought that you're going
> to move around on the plate but never eat (spam comments). I think
> comments are okay right now, maybe they could use a meta table but we
> can talk about that later.
>
> Meat alone is only a real meal at rboren's house, so most people put
> things on the sandwich to add flavor and spice it up. Some add other
> types of meat, in the WP world this is postmeta, which we call custom
> fields in polite company.
>
> We also havae condiments which are currently handled by two tables:
> wp_categories and wp_post2cat. On the taxonomy/condiment side, right
> now we really only allow ketchup aka categories, and users for at
> least a year have been asking for more. In 2.2 we decided to satiate
> their appetites.
>
> Everyone agrees that ketchup and mayonnaise are totally different,
> even though they're both condiments and you put them both on
> sandwiches. No one is trying to create some horrible pink mixture of
> the two tastes.
>
> However there are currently two schools of thought on how we should
> store the data for categories and tags at a very low level in our DB.
>
> Let me do my best to make the case for putting category data and tag
> data in separate tables, and feel free to chime in if you think I've
> missed any points.
>
> * We shouldn't ship anything with a data schema people disagree on,
> because plugins and themes will be written against it.
> * They're different things, so we should have them in different tables.
> * Tags can have things like synonyms, and don't need things like
> hierarchy.
> * There are ugly legacy field names in the category table like
> category_nicename, cat_name, cat_ID (wtf capitals) and we can clean
> those up in new tables
> * With separate tables our queries on the admin side become WAY easier
> and cleaner to do, with no bitwise or _count nonsense
> * Plugins for tagging have implemented it this way.
>
> The code currently in SVN does something different. It uses the
> categories table for names of the tags and then adds fields to hint
> how those names are being used for the admin section. If I wanted to
> make everyone happy and be popular I would just go with the above
> since there seems to be good consensus there, but I think this is an
> important long-term decision for WP so let me spell out some reasons
> why I think the current design has legs not just for 2.2 but beyond.
>
> 1. It performs faster.
>
> On front-end display, we have added ZERO QUERIES to support tags. The
> query that grabs categories is also grabbing tags and we're sorting
> them out in the code.
>
> In the dashboard some of the queries are more complicated (though not
> really any different than what we deal with for link categories) and a
> few milliseconds slower than the old ones. However, that really
> doesn't matter because 1) we only need to write them once and more
> importantly 2) they're run several orders of magnitude fewer times
> than the ones that display the blog on the front-end. A mantra has
> always been that user time is more important than developer time.
>
> A separate tag naming table and post2tag table would require at least
> 2 additional queries and/or joins to the front page, which already
> think does too many queries and is too heavy.
>
> 2. It's a better long-term foundation.
>
> I think there are a lot of benefits to having a single ID that maps to
> a term and a slug. Let's pretend we had perfect foresight 5 years ago
> and instead of wp_categories we had wp_terms.
>
> Regardless of the UI and philosophy behind categories, tags, and ooga
> booga, on a data level they're still mapping a set of terms to an item
> in post_content.
>
> In WP a term has three important things: an ID, a human-entered name,
> and a URL-friendly slug. We use the ID in our relations instead of the
> slug because it's more efficient and slugs are not necessarily unique
> (because of hierarchy).
>
> Having "dogs" in a category table have one ID and "dogs" in a tag
> table have a different ID is a long-term deck of cards that we will
> seriously regret later. It's MUCH harder to reconcile items with
> internally different IDs than it is to split out unique IDs into
> different tables.
>
> As for some of the bit and count fields currently causing grief, I
> would argue the solution for that isn't a separate tags table, but a
> separate table specifically for that type of data. In Drupal for this
> infrastructure they have a term_data, term_hiercharchy, term_node,
> term_relation, term_synonym, vocabulary, and vocabulary_node_types
> tables. I think that might be a little more than we need, but there
> are some concepts there we could pretty cleanly combine into a single
> extra table that isn't called categories or tags, and will provide a
> good and scalable foundation for years to come.
>
> 3. There should be no user- or plugin-facing problems with how it's
> currently implemented, or if we decide to change it.
>
> Now this isn't to suggest for a second there aren't bugs, many have
> been fixed already and I'm sure there are many still left, but that is
> going to be true of ANY code we put in WP and anyone who suggests
> otherwise is not very familiar with software development. From a point
> of view of plugin authors, they shouldn't have to think or care if
> we're storing it in a categories table or a turkey, the function they
> use should remain consistent no matter what we change or gymnastics we
> do behind the curtain. No matter what we do in 2.2 or 2.3, that's not
> going to change.
>
> I do think there is something intrinsically better about shipping and
> iterating than noodling without release in search of the "perfect"
> implementation.
>
> More importantly from a user's point of view, all that really matters
> is that they have a box they can type tags in and that their host
> doesn't tell them not to upgrade to 2.2 because it does more queries.
>
> 4. I'm open
>
> I'm not personally tied to any code written thus far and if I think
> the best thing is.
>
> There is a separate but related decision around what to do about the
> release date. Based on the discussion here I'm going to make go/no-go
> decision on Tuesday.
>
> If we do delay I think we should laser-focus on tags and now allow
> other pet-issues to creep in, and I will fully expect people to put in
> as much time writing code and fixing bugs as they have arguing points
> on mailing lists, IRC, and trac. At the very least I hope we've
> learned a bit more about getting these things out of the way early
> rather than a week or two before a release. Also if something is
> sitting in trac, take it to the hackers list early.
>
> I think if we stick with the current implementation we can hit it with
> a very stable release next Monday, but if we decide to replace it we
> need to push it back at least into mid-May.
>

_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Peter Westwood]

Ryan Boren wrote:

> On 4/15/07, Matt Mullenweg <m@...> wrote:
>> Regardless of the UI and philosophy behind categories, tags, and ooga
>> booga, on a data level they're still mapping a set of terms to an item
>> in post_content.
>>
>> In WP a term has three important things: an ID, a human-entered name,
>> and a URL-friendly slug. We use the ID in our relations instead of the
>> slug because it's more efficient and slugs are not necessarily unique
>> (because of hierarchy).
>>
>> Having "dogs" in a category table have one ID and "dogs" in a tag table
>> have a different ID is a long-term deck of cards that we will seriously
>> regret later. It's MUCH harder to reconcile items with internally
>> different IDs than it is to split out unique IDs into different tables.
>
> I agree with these points.  These points are the reason I tried to
> make the current design work.
>

I also agree with this.

I do feel we need a super light terms table that just stores the three
important things - ID, name and slug this would be wp_terms.

We then would have one table per term usage type with the linkages in
them. i.e post2cat post2tag link2cat etc. - post2cat and post2tag could
live as postmeta data although this would make them more expensive to
access.

>> As for some of the bit and count fields currently causing grief, I would
>> argue the solution for that isn't a separate tags table, but a separate
>> table specifically for that type of data. In Drupal for this
>> infrastructure they have a term_data, term_hiercharchy, term_node,
>> term_relation, term_synonym, vocabulary, and vocabulary_node_types
>> tables. I think that might be a little more than we need, but there are
>> some concepts there we could pretty cleanly combine into a single extra
>> table that isn't called categories or tags, and will provide a good and
>> scalable foundation for years to come.
>
> The presence of category_parent and the count fields are the crux of
> my problem with the unified table.  They cause a lot of grief and kill
> any future-proofing you think you're getting by having a unified terms
> table.  If you're open to a new table for this information, I think I
> could get behind that.  I'll give it some thought.
>

It would be nice to be able to restore the lost functionality for links
in this process too - i.e. the ability to customise the lists in the
admin ui that was lost in 2.0 - a lot of people used this and we just
dropped it!

This probably means we need tables like: link_cat_extras and
post_cat_extras (these need better names and maybe they can be merged or
can be extra columns in the wp_terms table ????)

westi
--
Peter Westwood
http://blog.ftwr.co.uk
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Ryan Boren]

On 4/15/07, Ryan Boren <ryan@...> wrote:

> > As for some of the bit and count fields currently causing grief, I would
> > argue the solution for that isn't a separate tags table, but a separate
> > table specifically for that type of data. In Drupal for this
> > infrastructure they have a term_data, term_hiercharchy, term_node,
> > term_relation, term_synonym, vocabulary, and vocabulary_node_types
> > tables. I think that might be a little more than we need, but there are
> > some concepts there we could pretty cleanly combine into a single extra
> > table that isn't called categories or tags, and will provide a good and
> > scalable foundation for years to come.
>
> The presence of category_parent and the count fields are the crux of
> my problem with the unified table.  They cause a lot of grief and kill
> any future-proofing you think you're getting by having a unified terms
> table.  If you're open to a new table for this information, I think I
> could get behind that.  I'll give it some thought.

For everyone's perusal, the drupal schema can be found here (scroll down):

http://cvs.drupal.org/viewcvs/drupal/drupal/modules/system/system.install?rev=1.92&view=markup

Taxonomy code:

http://cvs.drupal.org/viewcvs/drupal/drupal/modules/taxonomy/taxonomy.module?rev=1.351&view=markup
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Robert Deaton]

On 4/15/07, Matt Mullenweg <m@...> wrote:

> 4. I'm open
>
> I'm not personally tied to any code written thus far and if I think the
> best thing is.
>
> There is a separate but related decision around what to do about the
> release date. Based on the discussion here I'm going to make go/no-go
> decision on Tuesday.
>
> If we do delay I think we should laser-focus on tags and now allow other
> pet-issues to creep in, and I will fully expect people to put in as much
> time writing code and fixing bugs as they have arguing points on mailing
> lists, IRC, and trac. At the very least I hope we've learned a bit more
> about getting these things out of the way early rather than a week or
> two before a release. Also if something is sitting in trac, take it to
> the hackers list early.

I really wish we would've learned something about checking in heavy
changesets within a week of release time, but every major release
proves otherwise. I personally think if we're going to try to stick to
a fixed release schedule for this release and future releases, that
throwing major things like tagging into the core within the last month
of the cycle is a big step backwards.


> I think if we stick with the current implementation we can hit it with a
> very stable release next Monday, but if we decide to replace it we need
> to push it back at least into mid-May.



--
--Robert Deaton
http://lushlab.com
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Ryan Boren]

On 4/15/07, Ryan Boren <ryan@...> wrote:

> On 4/15/07, Ryan Boren <ryan@...> wrote:
> > > As for some of the bit and count fields currently causing grief, I would
> > > argue the solution for that isn't a separate tags table, but a separate
> > > table specifically for that type of data. In Drupal for this
> > > infrastructure they have a term_data, term_hiercharchy, term_node,
> > > term_relation, term_synonym, vocabulary, and vocabulary_node_types
> > > tables. I think that might be a little more than we need, but there are
> > > some concepts there we could pretty cleanly combine into a single extra
> > > table that isn't called categories or tags, and will provide a good and
> > > scalable foundation for years to come.
> >
> > The presence of category_parent and the count fields are the crux of
> > my problem with the unified table.  They cause a lot of grief and kill
> > any future-proofing you think you're getting by having a unified terms
> > table.  If you're open to a new table for this information, I think I
> > could get behind that.  I'll give it some thought.
>
> For everyone's perusal, the drupal schema can be found here (scroll down):
>
> http://cvs.drupal.org/viewcvs/drupal/drupal/modules/system/system.install?rev=1.92&view=markup
>
> Taxonomy code:
>
> http://cvs.drupal.org/viewcvs/drupal/drupal/modules/taxonomy/taxonomy.module?rev=1.351&view=markup

I'll borrow and corrupt Drupal's vocabulary terminology.  We have
three vocabularies: link categories, post categories, and post tags.
We want a term with a given name to always have the same id but be
able to participate independently in each of these vocabularies.  If
we pare down the tables to the least we can get away with, we end with
with three.

CREATE TABLE $wpdb->terms (
  term_ID bigint(20) NOT NULL auto_increment,
  term_name varchar(55) NOT NULL default '',
  term_nicename varchar(200) NOT NULL default '',
  term_description longtext NOT NULL,  // Maybe put this in term_vocabulary
  PRIMARY KEY  (term_ID),
  KEY term_nicename (term_nicename)
);
CREATE TABLE $wpdb->term_vocabulary (
  term_ID bigint(20) NOT NULL default '0',
  vocabulary_ID bigint(20) NOT NULL default '0',
  parent bigint(20) NOT NULL default '0',
  count bigint(20) NOT NULL default '0'
);
CREATE TABLE $wpdb->object2term (
  object_ID bigint(20) NOT NULL default '0',
  term_ID bigint(20) NOT NULL default '0',
  vocabulary_ID bigint(20) NOT NULL default '0',
  PRIMARY KEY  (object_ID),
  KEY object_ID (object_ID,term_ID),
  KEY term_ID (term_ID,object_ID)
);

A term is created within the context of a vocabulary.  When creating,
if the term doesn't exist in the terms table it is added there and to
term_vocabulary with the appropriate term ID.  If the term already
exists in terms, it is just added to term_vocabulary.   Adding the
term to a post adds an entry to object2term with the appropriate
vocabulary id.  The count and parent relationship are per vocabulary.
Deleting a term removes its term_vocabulary entry.  If the term is no
longer used in any vocabulary, it is removed from terms.

That's the rough idea. Thoughts?

Ryan
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Ryan Boren]

On 4/15/07, Robert Deaton <false.hopes@...> wrote:
> I really wish we would've learned something about checking in heavy
> changesets within a week of release time, but every major release
> proves otherwise. I personally think if we're going to try to stick to
> a fixed release schedule for this release and future releases, that
> throwing major things like tagging into the core within the last month
> of the cycle is a big step backwards.

Tagging was supposed to go in at the beginning of the 2.2 cycle, but
this debate delayed its entry until late in the cycle and is delaying
things still.  Things were supposed to be locked down long ago, but
here we are.

Ryan
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Robert Deaton]

On 4/15/07, Ryan Boren <ryan@...> wrote:

> On 4/15/07, Robert Deaton <false.hopes@...> wrote:
> > I really wish we would've learned something about checking in heavy
> > changesets within a week of release time, but every major release
> > proves otherwise. I personally think if we're going to try to stick to
> > a fixed release schedule for this release and future releases, that
> > throwing major things like tagging into the core within the last month
> > of the cycle is a big step backwards.
>
> Tagging was supposed to go in at the beginning of the 2.2 cycle, but
> this debate delayed its entry until late in the cycle and is delaying
> things still.  Things were supposed to be locked down long ago, but
> here we are.

I am not opposed to ripping it all out and starting fresh as soon as
2.2 is out the door.

--
--Robert Deaton
http://lushlab.com
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Steve Lewis-6]

On 4/15/07, Robert Deaton <false.hopes@...> wrote:

> I really wish we would've learned something about checking in heavy
> changesets within a week of release time, but every major release
> proves otherwise. I personally think if we're going to try to stick to
> a fixed release schedule for this release and future releases, that
> throwing major things like tagging into the core within the last month
> of the cycle is a big step backwards.


I couldn't agree more.  If there is an existential crisis with the
implementation of tags, particularly at such a low level, then roll Tagging
out of 2.2 NOW and give it the time it needs to be resolved, or blow the
release schedule for 2.2.  As has been pointed out, releasing tags in core,
and then changing the DB structure for storing it, is a recipe for
disaster.

That said, I think tags should be in their own table, but I am still
flabbergasted over what was done over links/blogroll.

--
SteveL
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Matt Mullenweg]

Robert Deaton wrote:
> I would swear you were highly overweight if I didn't know any better.

I'm working on it.

> One of the first rules of SQL has always been to not fetch more than
> you need, to allow sorting and searching to be done on the SQL side,
> etc. This breaks that fundamental rule.

In the context of displaying a post the categories and tags for that
post aren't more than we need, they are exactly what we need. It's an
index lookup.

> The bitfield also introduces new things. Let's face it, the APIs for
> fetching things from the database are horribly incomplete and will
> never ever possibly cover all the things that plugins will want to do.

If our APIs are incomplete, that's a separate and unrelated bug to the
underlying data structure. A plugin should never talk to the core DB
tables and fields directly.

> So, how about a plugin that wants to fetch all the tags to do some
> magic? Well, with a bitfield, that plugin now has to fetch all the
> categories and all the tags, and then sort through with bitwise
> operations and discard the categories which they don't care about.

You can get one or the other on the query level, no reason to select
both. It's also something we could pretty easily create a wrapper
function for, if there isn't one already.

> I think you missed a noun in the last clause of the sentence. Who
> thinks it is?

Oh sorry, I think there are too many. I agree that it's better to have
more fast and light queries than fewer heavy ones, but I don't think WP
is yet fully optimized in this regard, though 2.1 was a step in the
right direction.

> Are you gradually moving us toward dropping this table altogether and
> moving it all into postmeta?

I thought about this a lot, but that would make it pretty difficult to
do a query against posts in category or tag X, which I think is a pretty
common use case and one we should weight the schema toward.

> But like you said earlier, ketchup and mayonnaise are two different
> things. Why would we be trying to put them together?

I'm not going to try and stretch the metaphor too far, but I think at
the base level they do something that can be described and stored in
exactly the same way. I guess you could think of it as the ingredients
that go into each. Perhaps a better analogy would be ketchup and
spaghetti sauce.

> I disagree. Especially because the implementation is lacking so many
> features that other tagging plugins have done well for some time now,
> new plugins are bound to be forced to write their own queries to
> interface with this table. And when they don't work anymore in the
> next version, someone's gonna be pissed.

I think if we cover the basic CRUD operations, every missing feature I
can imagine is doable. If people want to talk to the DB directly, they
should be smart enough to know that means they'll have to update their
plugin accordingly. Fortunately, with a regular release schedule and an
Open Source codebase like ours there aren't really surprises like that
you won't know pretty far in advance.

> I agree, there is something better about shipping and iterating, as
> long as the initial shipment has something that is not fundamentally
> broken.

I don't think anything we've talked about on either is fundamentally
broken, especially from a user point of view.

Historically we've found that the things that are actual issues are very
different from the things we've anticipated to be issues. I don't think
this is unique to us, but is a general tenet of software design and
development in general.

--
Matt Mullenweg
  http://photomatt.net | http://wordpress.org
http://automattic.com | http://akismet.com
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Matt Freedman]

Or, we could pull the Tags out of Trunk, then release what's in Trunk as
2.1.4, then carry on resolving and working on all these problems that are
coming up for 2.2, and release it in a month or 2...

On 4/15/07, Robert Deaton <false.hopes@...> wrote:

>
> On 4/15/07, Ryan Boren <ryan@...> wrote:
> > On 4/15/07, Robert Deaton <false.hopes@...> wrote:
> > > I really wish we would've learned something about checking in heavy
> > > changesets within a week of release time, but every major release
> > > proves otherwise. I personally think if we're going to try to stick to
> > > a fixed release schedule for this release and future releases, that
> > > throwing major things like tagging into the core within the last month
> > > of the cycle is a big step backwards.
> >
> > Tagging was supposed to go in at the beginning of the 2.2 cycle, but
> > this debate delayed its entry until late in the cycle and is delaying
> > things still.  Things were supposed to be locked down long ago, but
> > here we are.
>
> I am not opposed to ripping it all out and starting fresh as soon as
> 2.2 is out the door.
>
> --
> --Robert Deaton
> http://lushlab.com
> _______________________________________________
> wp-hackers mailing list
> wp-hackers@...
> http://lists.automattic.com/mailman/listinfo/wp-hackers
>

--
Matt (speedboxer@...)
http://mattsblog.ca/ | http://livemp.net/
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Matt Mullenweg]

Matt wrote:
> Or, we could pull the Tags out of Trunk, then release what's in Trunk as
> 2.1.4, then carry on resolving and working on all these problems that are
> coming up for 2.2, and release it in a month or 2...

I don't think we should change the semantics of a minor point release
(2.x.x) to mean adding features.

Likewise if tagging isn't in 2.2, I'm not sure if it has enough sizzle
to drive upgrades. The other sexy features were permalink redirection
and widgets, and we axed both of those for 2.2 already.

--
Matt Mullenweg
  http://photomatt.net | http://wordpress.org
http://automattic.com | http://akismet.com
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Matt Mullenweg]

Ryan Boren wrote:
> For everyone's perusal, the drupal schema can be found here (scroll down):

Here's a neat representation of the same:

http://webdevgeeks.com/schemaspy/index.html

--
Matt Mullenweg
  http://photomatt.net | http://wordpress.org
http://automattic.com | http://akismet.com
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Matt Mullenweg]

Ryan Boren wrote:
> The presence of category_parent and the count fields are the crux of
> my problem with the unified table.  They cause a lot of grief and kill
> any future-proofing you think you're getting by having a unified terms
> table.  If you're open to a new table for this information, I think I
> could get behind that.  I'll give it some thought.

Let's not lose counts, they can be super useful.

--
Matt Mullenweg
  http://photomatt.net | http://wordpress.org
http://automattic.com | http://akismet.com
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Matt Freedman]

I meant to put that their wasn't any reason to upgrade. :P

Well, then, we should delay the release of 2.2. Otherwise, quite a lot of
people are going to be unhappy, and WP might start to get a bad name.


On 4/15/07, Matt Mullenweg <m@...> wrote:

>
> Matt wrote:
> > Or, we could pull the Tags out of Trunk, then release what's in Trunk as
> > 2.1.4, then carry on resolving and working on all these problems that
> are
> > coming up for 2.2, and release it in a month or 2...
>
> I don't think we should change the semantics of a minor point release
> (2.x.x) to mean adding features.
>
> Likewise if tagging isn't in 2.2, I'm not sure if it has enough sizzle
> to drive upgrades. The other sexy features were permalink redirection
> and widgets, and we axed both of those for 2.2 already.
>
> --
> Matt Mullenweg
> http://photomatt.net | http://wordpress.org
> http://automattic.com | http://akismet.com
> _______________________________________________
> wp-hackers mailing list
> wp-hackers@...
> http://lists.automattic.com/mailman/listinfo/wp-hackers
>

--
Matt (speedboxer@...)
http://mattsblog.ca/ | http://livemp.net/
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[spencerp]

Matt wrote:

> Or, we could pull the Tags out of Trunk, then release what's in Trunk as
> 2.1.4, then carry on resolving and working on all these problems that are
> coming up for 2.2, and release it in a month or 2...
>
> On 4/15/07, Robert Deaton <false.hopes@...> wrote:
>>
>> On 4/15/07, Ryan Boren <ryan@...> wrote:
>> > On 4/15/07, Robert Deaton <false.hopes@...> wrote:
>> > > I really wish we would've learned something about checking in heavy
>> > > changesets within a week of release time, but every major release
>> > > proves otherwise. I personally think if we're going to try to
>> stick to
>> > > a fixed release schedule for this release and future releases, that
>> > > throwing major things like tagging into the core within the last
>> month
>> > > of the cycle is a big step backwards.
>> >
>> > Tagging was supposed to go in at the beginning of the 2.2 cycle, but
>> > this debate delayed its entry until late in the cycle and is delaying
>> > things still.  Things were supposed to be locked down long ago, but
>> > here we are.
>>
>> I am not opposed to ripping it all out and starting fresh as soon as
>> 2.2 is out the door.
>>

Oh Jesus! Stop the crazy talk people... :P Like Ryan said basically,
"we're" already behind on that and just ripping it all out now would be
crazy. I just manually adjusted TONS of my posts from that UTW tag
import crap deal the other day. And made a tag cloud fer it and all that
good mushy stuff...

WP 2.2 has some more changes since 2.1.X, but shite, not by much. With
Tags in now, and the talk going on about splitting db tables, just move
the release date forward. It's easier to do that, then rip out what's
already been added... At the rate folks are talking about.. that SC2
integration into the core will make it in around WP 2.7 or greater lmao!
*sarcasm* of course! :P

spencerp
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Rob Miller-4]

On 16/04/07, Matt Mullenweg <m@...> wrote:
> Likewise if tagging isn't in 2.2, I'm not sure if it has enough sizzle
> to drive upgrades. The other sexy features were permalink redirection
> and widgets, and we axed both of those for 2.2 already.

Why not push back 2.2, sort out this debate over tagging, *and*
include the permalink redirection stuff and widgets?

It really doesn't seem like there's enough in 2.2 to justify a
release, at the moment, and I really don't see the point in releasing
for the sake of it.

--
Rob

-----------------------
Rob Miller
Senior Programmer

email - rob@...
web - http://www.graphics.net/ | http://robm.me.uk/
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Matt Mullenweg]

Rob wrote:
> Why not push back 2.2, sort out this debate over tagging, *and*
> include the permalink redirection stuff and widgets?

Because down that road lies a year in between 2.0 and 2.1. A small
feature release is better than no release.

--
Matt Mullenweg
  http://photomatt.net | http://wordpress.org
http://automattic.com | http://akismet.com
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers