MediaWiki is getting a new programming language

[Apoc 2400]

The current template code is ugly, but it does work. Few people can
understand the source code of the more intricate templates, which is a
shame, but we are getting very good use out of the ugly wikicode. We have a
huge number of templates that do a lot of good with the limited
ParserFunctions we have.

I understand the developers want to replace it with a new template
programming language, but we should be aware of the risks. Starting over
from scratch and making it clean this time is one of the most common ways to
begin a spectacular failure in software development. Joel Spolsky write a
good column on it in 2000:
http://www.joelonsoftware.com/articles/fog0000000069.html

Is it even possible to sandbox Lua to guard against any exploits, even the
kind that just overloads the servers? Will it integrate well with articles
written in Wikicode? How long will it take until we can use it?

Most people don't need to understand advanced templates, just use it. There
is actually no lack of powerful templates. We have a lot of them. The
problem is that they are too hard to use.

{{Birth date and age}} is a rather popular template we use to show a persons
birth date and automatically calculate the current age. It allows us to
provide this without having to update it every year. I guess some developers
hate us for doing something like that in wikicode, but hey, it works. The
only problem it that it's hard to read in the article code. It is used as
{{Birth date and age|1993|2|4|mf=yes}}. If we had the StringFunctions
extension enabled it could be simplified to take dates like 1993-02-04
rather than 1993|2|4. Perhaps it could even take plain text dates like
"February 4, 1993". If we had the variables extension one could specify the
dateformat (month or day first) in one place in the article and template
like {{Birth date and age}} would follow.

We are stuck with templates that are hard to use because the developers are
against allowing useful programming constructs in wikicode. I really
wouldn't mind a good "real" programming language for templates, but should
we really but all eggs in one basket?

/Apoc2400
_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[Carcharoth]

On Fri, Jul 3, 2009 at 9:28 PM, Apoc 2400<apoc2400@...> wrote:

<snip>

> {{Birth date and age}} is a rather popular template we use to show a persons
> birth date and automatically calculate the current age. It allows us to
> provide this without having to update it every year. I guess some developers
> hate us for doing something like that in wikicode, but hey, it works. The
> only problem it that it's hard to read in the article code. It is used as
> {{Birth date and age|1993|2|4|mf=yes}}. If we had the StringFunctions
> extension enabled it could be simplified to take dates like 1993-02-04
> rather than 1993|2|4. Perhaps it could even take plain text dates like
> "February 4, 1993". If we had the variables extension one could specify the
> dateformat (month or day first) in one place in the article and template
> like {{Birth date and age}} would follow.

I think we *already* have templates that can take February 4, 1993 (or
something similar). They had some weird name, similar to the current
templates, but slightly different. I can't find them right now, but
maybe someone else remembers them?

Carcharoth

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[Aryeh Gregor]

On Fri, Jul 3, 2009 at 4:28 PM, Apoc 2400<apoc2400@...> wrote:
> The current template code is ugly, but it does work.

It uses up an enormous amount of CPU resources, and there have been
some cases where it's spectacularly failed.  Complicated templates
have been responsible for bringing down sites more than once.  If
[[Michael Jackson]] had been written in simple wikitext with no
templates, the rush to read it probably wouldn't have brought down
Wikipedia, for instance -- it was only so severe because the page took
many seconds to parse.

There have been previous incidents too.  In one case I can't recall
the specifics of, a sysadmin had to delete a template on eswiki after
it crashed at least that site (I think also affecting other sites on
the same cluster, can't recall).

Not to mention, of course, that there are plenty of people (especially
programmers) who would be happy to help out with some template coding
-- but don't because the code is so horrifying.

> I understand the developers want to replace it with a new template
> programming language, but we should be aware of the risks. Starting over
> from scratch and making it clean this time is one of the most common ways to
> begin a spectacular failure in software development. Joel Spolsky write a
> good column on it in 2000:
> http://www.joelonsoftware.com/articles/fog0000000069.html

No one is suggesting that ParserFunctions will be disabled.  The new
system will be added in parallel, and assuming it's really superior,
it will replace ParserFunctions over time.  (That essay is irrelevant
here anyway.  We're not talking about rewriting MediaWiki or any other
project from scratch.)

> Is it even possible to sandbox Lua to guard against any exploits, even the
> kind that just overloads the servers?

That's one of the major problems with the suggestion.  It's part of
why nothing has been finalized yet.  (The other key problem is making
sure Wikipedia content is easily usable by third parties on shared
hosting.)

> Will it integrate well with articles
> written in Wikicode?

That would depend on the implementation details, which are very far
from being worked out.

> How long will it take until we can use it?

This is all still purely theoretical at this point.  We haven't even
worked out if a solution is possible, let alone started setting even
the most preliminary timetables.

> Most people don't need to understand advanced templates, just use it. There
> is actually no lack of powerful templates. We have a lot of them. The
> problem is that they are too hard to use.

Partly because some things are too difficult to do in ParserFunctions.
 More intelligent parameter handling would be possible using a real
language.  Of course, StringFunctions would work as well, to a lesser
extent.

> It is used as
> {{Birth date and age|1993|2|4|mf=yes}}. If we had the StringFunctions
> extension enabled it could be simplified to take dates like 1993-02-04
> rather than 1993|2|4. Perhaps it could even take plain text dates like
> "February 4, 1993".

This should already work just fine using {{#time}}.  I'd say most
likely, nobody's bothered to do it because template programming is so
incredibly obnoxious that only a dedicated few can muster the
willpower to struggle through it.  A real language with normal syntax
would mean that someone like me -- a programmer -- could go over right
now and fix it up in a couple of minutes.  But I'm not even going to
try messing with such a mess of curly braces.

> We are stuck with templates that are hard to use because the developers are
> against allowing useful programming constructs in wikicode.

Not all are.

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[geni]

2009/7/3 Aryeh Gregor <Simetrical+wikilist@...>:

> On Fri, Jul 3, 2009 at 4:28 PM, Apoc 2400<apoc2400@...> wrote:
>> The current template code is ugly, but it does work.
>
> It uses up an enormous amount of CPU resources, and there have been
> some cases where it's spectacularly failed.  Complicated templates
> have been responsible for bringing down sites more than once.  If
> [[Michael Jackson]] had been written in simple wikitext with no
> templates, the rush to read it probably wouldn't have brought down
> Wikipedia, for instance -- it was only so severe because the page took
> many seconds to parse.
>
> There have been previous incidents too.  In one case I can't recall
> the specifics of, a sysadmin had to delete a template on eswiki after
> it crashed at least that site (I think also affecting other sites on
> the same cluster, can't recall).

And with the exception of the Jackson issue I understand those issues
have been fixed. The jackson issue is somewhat questionable if it can
be solved by a new code setup. Give people a new code setup and they
will find a way to do unexpected things with that to. Remember the
days when VFD made made a significant dent in server load?

Heh issue long predates wikipedia in any case. Back in the early 70s
Southampton university created SOFOR because standard FORTRAN didn't
compile fast enough.

> Not to mention, of course, that there are plenty of people (especially
> programmers) who would be happy to help out with some template coding
> -- but don't because the code is so horrifying.

Programmers are not who we are primarily interested in with regards to
template code. There are far fewer programmers than non programmers
and they are more in demand by open source groups.

It's far more important that our template code be accessible to non
programmers than it be something programmers like. It also needs to be
accessible within a wikipedia environment something a traditional
language is unlikely to be. Template code may look bad but in practice
it's fairly resistant to people copying and pasting then playing with
to get what they want.

>No one is suggesting that ParserFunctions will be disabled.  The new
>system will be added in parallel, and assuming it's really superior,
>it will replace ParserFunctions over time.

Questionable. Check out the templates on
http://en.wikipedia.org/wiki/Matthew_Pinsent

Old code unless bot eliminated (msg:stub for example) tends not to die.

--
geni

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[Ian Woollard]

On 03/07/2009, geni <geniice@...> wrote:
> And with the exception of the Jackson issue I understand those issues
> have been fixed. The jackson issue is somewhat questionable if it can
> be solved by a new code setup. Give people a new code setup and they
> will find a way to do unexpected things with that to. Remember the
> days when VFD made made a significant dent in server load?

Maybe, although if there's particular bottlenecks, you're usually
better off just optimising those; hardcoding a few common templates in
some non user accessible language is likely to be a lot less work than
an entirely new programming language. Actually there doubtless are
already multiple languages available.

> Programmers are not who we are primarily interested in with regards to
> template code. There are far fewer programmers than non programmers
> and they are more in demand by open source groups.
>
> It's far more important that our template code be accessible to non
> programmers than it be something programmers like. It also needs to be
> accessible within a wikipedia environment something a traditional
> language is unlikely to be. Template code may look bad but in practice
> it's fairly resistant to people copying and pasting then playing with
> to get what they want.

The other way to go is just to optimise the heck out of the existing
template system, with enough caching and hashing or run time
compilation most things run quickly enough. Still, doing this is
usually complex.

> Old code unless bot eliminated (msg:stub for example) tends not to die.

Yup. Conspiracy theories and programming languages. Both can be fun.

> --
> geni

--
-Ian Woollard

"All the world's a stage... but you'll grow out of it eventually."

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l