JSON Syndication

[Matt Patenaude]

Hey there,

For more than a year now, I've been developing my own content  
management system called Tux CMS (http://blogwithtux.org), which  
takes a lot of its feature ideas and concepts from WordPress (I've  
also ported a few WP themes over to it). Extremely little borrowed  
code, but a lot of borrowed inspiration, so first of all, I'd like to  
say thanks to the WordPress team for an awesome product! :)

Second, though, I would like to suggest a feature. For the past few  
months I've been getting fed up with XML, and getting to know JSON.  
XML can be quite clunky, and really isn't suited for syndication when  
you think about it. JSON, on the other hand, is a very lightweight  
data serialization language, which greatly improves size and  
simplicity of use of syndicated content (it has direct relationships  
to programming structures, so it's easier to access programmatically  
than XML through the DOM).

I'm trying to get a movement going of a syndication format made  
completely in JSON. The first step, I figured, was to develop an  
informal specification: http://howto.blogwithtux.org/index.php?
title=JSON_feed.

The next step was to create a working implementation of my  
specification: http://www.mattpat.net/json.php. But then, when trying  
to decide on the third step, I wondered, hmm, what would get this the  
most exposure? Why, implementation by one of the best and most widely  
used blogging engines, of course! I'm a big fan of WordPress (despite  
the fact that I write my own CMS... of course, it's open source, so I  
would call it more friendly competition), so I thought I might start  
here.

Is this something WordPress developers might be interested in? I want  
to make this a large movement... I even wrote an article on A List  
Apart for it (still waiting for a response on it).

Let me know!

Sincerely,
Matt Patenaude
http://www.mattpat.net 
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Mike Purvis]

It's a neat idea, but I have two problems with it:

1. It's too late. There's already a format war between Atom and RSS,
not to mention podcasting enclosures, GeoRSS, and everything else.

2. The #1 thing that makes JSON cool and useful is the &callback=x
parameter, which allows it to be called cross-domain using dynamic
script tags. As an example, see how Google's geocoder works:

http://maps.google.com/maps/geo?output=json&callback=dostuff

However, it be disasterously irresponsible to implement something like
this in Wordpress and encourage people to use it. It's an open
invitation to run arbitrary JavaScript on random domains, a weakness
that XML simply does not have.

At any rate, the first step in this process would be to develop the
feed as a plugin, and demonstrate that it offers something not
available in existing syndication schemes. If this were truly the
case, then perhaps the leet hax0rs would consider adding it to the
core, down the road.

Mike


On 2/15/07, Matt Patenaude <MattPat@...> wrote:

> Hey there,
>
> For more than a year now, I've been developing my own content
> management system called Tux CMS (http://blogwithtux.org), which
> takes a lot of its feature ideas and concepts from WordPress (I've
> also ported a few WP themes over to it). Extremely little borrowed
> code, but a lot of borrowed inspiration, so first of all, I'd like to
> say thanks to the WordPress team for an awesome product! :)
>
> Second, though, I would like to suggest a feature. For the past few
> months I've been getting fed up with XML, and getting to know JSON.
> XML can be quite clunky, and really isn't suited for syndication when
> you think about it. JSON, on the other hand, is a very lightweight
> data serialization language, which greatly improves size and
> simplicity of use of syndicated content (it has direct relationships
> to programming structures, so it's easier to access programmatically
> than XML through the DOM).
>
> I'm trying to get a movement going of a syndication format made
> completely in JSON. The first step, I figured, was to develop an
> informal specification: http://howto.blogwithtux.org/index.php?
> title=JSON_feed.
>
> The next step was to create a working implementation of my
> specification: http://www.mattpat.net/json.php. But then, when trying
> to decide on the third step, I wondered, hmm, what would get this the
> most exposure? Why, implementation by one of the best and most widely
> used blogging engines, of course! I'm a big fan of WordPress (despite
> the fact that I write my own CMS... of course, it's open source, so I
> would call it more friendly competition), so I thought I might start
> here.
>
> Is this something WordPress developers might be interested in? I want
> to make this a large movement... I even wrote an article on A List
> Apart for it (still waiting for a response on it).
>
> Let me know!
>
> Sincerely,
> Matt Patenaude
> http://www.mattpat.net
> _______________________________________________
> wp-hackers mailing list
> wp-hackers@...
> http://lists.automattic.com/mailman/listinfo/wp-hackers
>

--
http://uwmike.com // http://googlemapsbook.com
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Matt Patenaude]

Thanks for the responses guys! Couple of points I'd like to bring up.

> In effort to not just be a talking head:
>   - http://jrm.cc/wp-json.php
>   - http://jrm.cc/wp-json.phps
>
> And it even passes a basic "validaton":
>   - http://jrm.cc/extras/json-valid.html

Awesome stuff, man! This was just what I was looking at.

> The biggest gotchas:
>   - Content-Type, which to use?
>   - Escaping, making sure it's "right"

Well, as for content-type, application/json is the MIME-type  
specified by the JSON spec, but I'd imagine our own feed format would  
take its own content-type (for example, Atom is application/atom+xml,  
not just application/xml)... maybe application/feed+json, or  
something similar? It would all depend on how mature the  
specification became.

As for escaping, I handle it on my website by using a JSON library:  
http://tuxcms.cvs.sourceforge.net/tuxcms/tuxcms/json.php?view=markup

Note how, in the above file, my entire feed is constructed using PHP  
data structures, and then one line converts it into the JSON feed.  
Cool, huh? :)

> It's a neat idea, but I have two problems with it:
>
> 1. It's too late. There's already a format war between Atom and RSS,
> not to mention podcasting enclosures, GeoRSS, and everything else.

True, but in my opinion, they both have the fundamental problem of  
using complex XML. I'm not saying by any means to replace XML feeds  
by JSON feeds, but JSON feeds are definitely the new movement.  
Del.icio.us and Yahoo! have both started offering their web service  
outputs in JSON in addition to XML, because it's so simple to use  
with almost any programming language (which is largely the goal of  
any syndicated content). Every movement has to start somewhere, and  
in computers, I don't think there such a thing as too late. (OK, yes  
there is, but I don't think this is an instant of it :P).

> 2. The #1 thing that makes JSON cool and useful is the &callback=x
> parameter, which allows it to be called cross-domain using dynamic
> script tags.

Well, callbacks might be cool, but I don't think they're the #1 thing  
that makes JSON cool-- I'd say it's the fact that it's simple,  
lightweight, and supported almost everywhere.

FIrst of all, callbacks wouldn't have to necessarily be implemented  
by WordPress. In fact, I don't even think there is much use for a  
feed specification. If the format caught on, it would probably be  
included using a link element, not a script element. Besides, the  
inclusion would be done by a plug-in, so we would control how it was  
included on the page. The plug-in would allow for the syndication of  
JSON, not necessarily its use on a web page.

Also, note the above implementation's (by Jeff Minard) use of  
callback (pass a random callback to the first URL). No matter what  
you pass, it wraps the feed in wp_jason_feed(eval()). This would  
provide a little extra control for feeds used on other WordPress sites.

Anyway, you both said the best way to go is to make a plug-in, and I  
think I'll do just that. Thanks for the help!

-Matt

On Feb 16, 2007, at 11:51 AM, Mike Purvis wrote:

> It's a neat idea, but I have two problems with it:
>
> 1. It's too late. There's already a format war between Atom and RSS,
> not to mention podcasting enclosures, GeoRSS, and everything else.
>
> 2. The #1 thing that makes JSON cool and useful is the &callback=x
> parameter, which allows it to be called cross-domain using dynamic
> script tags. As an example, see how Google's geocoder works:
>
> http://maps.google.com/maps/geo?output=json&callback=dostuff
>
> However, it be disasterously irresponsible to implement something like
> this in Wordpress and encourage people to use it. It's an open
> invitation to run arbitrary JavaScript on random domains, a weakness
> that XML simply does not have.
>
> At any rate, the first step in this process would be to develop the
> feed as a plugin, and demonstrate that it offers something not
> available in existing syndication schemes. If this were truly the
> case, then perhaps the leet hax0rs would consider adding it to the
> core, down the road.
>
> Mike
>
>
> On 2/15/07, Matt Patenaude <MattPat@...> wrote:
>> Hey there,
>>
>> For more than a year now, I've been developing my own content
>> management system called Tux CMS (http://blogwithtux.org), which
>> takes a lot of its feature ideas and concepts from WordPress (I've
>> also ported a few WP themes over to it). Extremely little borrowed
>> code, but a lot of borrowed inspiration, so first of all, I'd like to
>> say thanks to the WordPress team for an awesome product! :)
>>
>> Second, though, I would like to suggest a feature. For the past few
>> months I've been getting fed up with XML, and getting to know JSON.
>> XML can be quite clunky, and really isn't suited for syndication when
>> you think about it. JSON, on the other hand, is a very lightweight
>> data serialization language, which greatly improves size and
>> simplicity of use of syndicated content (it has direct relationships
>> to programming structures, so it's easier to access programmatically
>> than XML through the DOM).
>>
>> I'm trying to get a movement going of a syndication format made
>> completely in JSON. The first step, I figured, was to develop an
>> informal specification: http://howto.blogwithtux.org/index.php?
>> title=JSON_feed.
>>
>> The next step was to create a working implementation of my
>> specification: http://www.mattpat.net/json.php. But then, when trying
>> to decide on the third step, I wondered, hmm, what would get this the
>> most exposure? Why, implementation by one of the best and most widely
>> used blogging engines, of course! I'm a big fan of WordPress (despite
>> the fact that I write my own CMS... of course, it's open source, so I
>> would call it more friendly competition), so I thought I might start
>> here.
>>
>> Is this something WordPress developers might be interested in? I want
>> to make this a large movement... I even wrote an article on A List
>> Apart for it (still waiting for a response on it).
>>
>> Let me know!
>>
>> Sincerely,
>> Matt Patenaude
>> http://www.mattpat.net
>> _______________________________________________
>> wp-hackers mailing list
>> wp-hackers@...
>> http://lists.automattic.com/mailman/listinfo/wp-hackers
>>
>
>
> --
> http://uwmike.com // http://googlemapsbook.com
> _______________________________________________
> wp-hackers mailing list
> wp-hackers@...
> http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Elliotte Harold]

Matt Patenaude wrote:

> Second, though, I would like to suggest a feature. For the past few
> months I've been getting fed up with XML, and getting to know JSON. XML
> can be quite clunky, and really isn't suited for syndication when you
> think about it.

I call shenanigans on this. XML is exactly suited for serialization, and
JSON isn't. You cannot easily pass syndication feeds in simple
name-value maps. There's too much structure there that you lose when
doing that. If what you have is a name-value map, then JSON may be
simpler. Syndication data is not made up of name-value pairs though.

The problems RSS had were the result of avoiding XML and trying to make
it more like JSON. We've been down that road already. It didn't work.
That's why there's now Atom.

> JSON, on the other hand, is a very lightweight data
> serialization language, which greatly improves size and simplicity of
> use of syndicated content (it has direct relationships to programming
> structures, so it's easier to access programmatically than XML through
> the DOM).

Well I certainly wouldn't suggest anybody use the DOM, but there are
alternatives. The problem is DOM, not XML. Try XQuery, XSLT, XPath, or
XOM to name just a few.

> I'm trying to get a movement going of a syndication format made
> completely in JSON. The first step, I figured, was to develop an
> informal specification:
> http://howto.blogwithtux.org/index.php?title=JSON_feed.

The syndication format I see there is much simpler than what people are
sending today with Atom. It is essentially a variation of the original
RSS specs which were massively oversimple for the use case, resulting in
truly ugly hacks to account for things people needed to do that RSS
couldn't. Please consider the needs of full text feeds with embedded
markup. Once you do that JSON's no longer so simple.

And I'm not even talking about the inevitable security bugs that are
going to open up if people begin executing JavaScript code that
syndicators send them.

--
Elliotte Rusty Harold  elharo@...
Java I/O 2nd Edition Just Published!
http://www.cafeaulait.org/books/javaio2/
http://www.amazon.com/exec/obidos/ISBN=0596527500/ref=nosim/cafeaulaitA/
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Charles-50]

>> Second, though, I would like to suggest a feature. For the past
>> few months I've been getting fed up with XML, and getting to know
>> JSON. XML can be quite clunky, and really isn't suited for
>> syndication when you think about it.
>
> I call shenanigans on this.

Yeah, I think he's probably confusing a particular schema with XML.  Well, at least a plug-in is a good way to create a solution-in-search-of-a-problem exercise coming several years too late to be meaningful.

-- Charles


_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Matt Patenaude]

I'm not trying to knock XML as a language. XML is very powerful, and  
I believe that there are many places (particularly in rich documents)  
where its uses are plentiful. However, I don't think syndication is  
one of them.

> Please consider the needs of full text feeds with embedded markup.  
> Once you do that JSON's no longer so simple.

And what exactly are the needs of full text feeds? I think that  
matter is up to interpretation, but the way I see it, the entire  
purpose of online syndication is to distribute a snippet of a larger  
work in a format that piques the reader's interest, and encourages  
them to visit the original source to read more. I do not see  
syndication having the need to convey all of the semantic markup of  
the document being syndicated. When I read an Atom feed, I honestly  
don't really care where the author put emphasis. If I want that much  
info, I'll click the link and read the whole article.

The format that's doing the syndication is only a container-- RSS is  
a container, Atom is a container, and this JSON feed format would  
also be a container. What would go in the content section could be  
anything you want, just like Atom. I would probably add a type  
property to each entry, so that you could specify if the content was  
text/html, text/plain, or even something a little fancier. Find me an  
Atom feed that can't be represented by JSON... bet it won't be easy.  
There's no saying a JSON feed couldn't handle embedded markup (in  
fact, I had always intended it to).

Of course, syndication is no use if it's ridiculously hard to  
implement. I'm not saying that XML is all that bad, but JSON is much  
simpler. Since JSON relates directly to programming data structures,  
it's easier to handle and deliver to the people who want to read it.

> The syndication format I see there is much simpler than what people  
> are sending today with Atom.

Well see, that's interesting, because I copied the most commonly used  
elements from the Atom standard and just turned it into JSON.

> Yeah, I think he's probably confusing a particular schema with XML.

For the record, no, I was not. If I simply had a distaste for an XML  
schema, I would have proposed a new schema, not a new method. ;)

> And I'm not even talking about the inevitable security bugs that  
> are going to open up if people begin executing JavaScript code that  
> syndicators send them.

A JSON feed shouldn't contain any JavaScript code whatsoever. In  
fact, I think the callback feature of JSON should not be implemented  
by scripts generating JSON feeds. Further, it will be encouraged (in  
big bold letters ;)) that people using the feeds use a JSON  
interpreter, rather than just eval()'ing it. That eliminates pretty  
much every problem.

I'm not giving up just yet, I still think JSON has a lot of promise  
as a syndication container. Thanks a lot for the input though! :)

-Matt
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[cori]

I'm not a WP hacker, so perhaps I should keep my mouth shut, but as a
not-hacker I disagree with this:
> And what exactly are the needs of full text feeds? I think that matter
> is up to interpretation, but the way I see it, the entire purpose of
> online syndication is to distribute a snippet of a larger work in a
> format that piques the reader's interest, and encourages them to visit
> the original source to read more. I do not see syndication having the
> need to convey all of the semantic markup of the document being
> syndicated. When I read an Atom feed, I honestly don't really care
> where the author put emphasis. If I want that much info, I'll click
> the link and read the whole article.
I don't agree with that assessment of syndication or the importance of
full-content at all.  I want the full content in the feed, preferably
with the author's emphasis.  In fact, a lack of full content often means
I won't subscribe at all.

This is not a commentary on the concept of JSON feeds - I don't know
enough about JSON to comment on that, but full-content is definitely an
important aspect of syndication as a whole, and I would argue that the
interpretation I detail above is at least as common as the one you
outline, Matt.

_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Elliotte Harold]

Matt Patenaude wrote:
> I'm not trying to knock XML as a language. XML is very powerful, and I
> believe that there are many places (particularly in rich documents)
> where its uses are plentiful. However, I don't think syndication is one
> of them.

You're wrong. It is one of them. In fact, syndication is exactly the
sort of application that XML was designed for. It fits that particular
problem extremely well.

> And what exactly are the needs of full text feeds? I think that matter
> is up to interpretation, but the way I see it, the entire purpose of
> online syndication is to distribute a snippet of a larger work in a
> format that piques the reader's interest, and encourages them to visit
> the original source to read more.

That's one purpose. It's hardly the only one. Many people prefer reading
their entire feeds in the feed reader rather than a browser. The market
has already determined that they want this. The original RSS tried to
limit itself to what yo;re doing and the audience quickly hacked round
it because they wanted full text feeds with embedded markup. Your
proposal. like the original RSS, only solves part of the problem.

> I do not see syndication having the
> need to convey all of the semantic markup of the document being
> syndicated. When I read an Atom feed, I honestly don't really care where
> the author put emphasis. If I want that much info, I'll click the link
> and read the whole article.

That's fine for you. You don't have to use it. However many authors and
readers do want this. Your format will not succeed without providing it.


> The format that's doing the syndication is only a container-- RSS is a
> container, Atom is a container, and this JSON feed format would also be
> a container.

RSS is a broken container. Your format is a broken container.

> What would go in the content section could be anything you
> want, just like Atom. I would probably add a type property to each
> entry, so that you could specify if the content was text/html,
> text/plain, or even something a little fancier.

You're going to reinvent double escaped markup, just like RSS did. This
has already been shown to be a major flaw in RSS and it will be no less
of one in JSON. Atom does not have this problem. (Well, it shouldn't.
Some people and plug-ins persist in sending double escaped markup in
Atom too, but at least it's not required there.)

Those who forget history are doomed to repeat it, I guess.

> Find me an Atom feed
> that can't be represented by JSON... bet it won't be easy. There's no
> saying a JSON feed couldn't handle embedded markup (in fact, I had
> always intended it to).

The question is not, "Can it do it?" The question is it, "Can it do it
well enough?" More properly, ""Can it do it better or even equally to
Atom?" The answer to that question is no, it can't.


> Of course, syndication is no use if it's ridiculously hard to implement.
> I'm not saying that XML is all that bad, but JSON is much simpler. Since
> JSON relates directly to programming data structures, it's easier to
> handle and deliver to the people who want to read it.

Syndication isn't about programming data structures. It's about
publishing document structures. Stop thinking like a programmer and
start thinking like an author.

Even if it were true that using XML made life harder for the programmer
(and it doesn't) XML would still be the right answer because it makes
life easier for the publisher and the reader. The programmers in this
space are working to satisfy the needs of the authors and the readers,
not the other way around.

>> The syndication format I see there is much simpler than what people
>> are sending today with Atom.

True, because people are sending more complex data than your format can
support.

> A JSON feed shouldn't contain any JavaScript code whatsoever. In fact, I
> think the callback feature of JSON should not be implemented by scripts
> generating JSON feeds. Further, it will be encouraged (in big bold
> letters ;)) that people using the feeds use a JSON interpreter, rather
> than just eval()'ing it. That eliminates pretty much every problem.

And it's encouraged that programmers always use a good garbage
collection library and check the results of each malloc before using it.
That's why we really don't have any problems with buffer overflows any
more. :-)

--
Elliotte Rusty Harold  elharo@...
Java I/O 2nd Edition Just Published!
http://www.cafeaulait.org/books/javaio2/
http://www.amazon.com/exec/obidos/ISBN=0596527500/ref=nosim/cafeaulaitA/
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Doug Stewart]

On 2/21/07, Elliotte Harold <elharo@...> wrote:
>
> Those who forget history are doomed to repeat it, I guess.

A small aside -
My two favorite takes on the above quote have to be:
"History doesn't repeat itself, but it sure does rhyme a lot."

"History doesn't repeat itself, it just stutters."

*grin*

--
-Doug

http://literalbarrage.org/blog/
_______________________________________________
wp-hackers mailing list
wp-hackers@...
http://lists.automattic.com/mailman/listinfo/wp-hackers