Special:All pages broken at about 03:00 UTC this morning

[Robert Ullmann]

No longer displays pages from a title. In some cases it will display a list
of index pages.

For example,
http://en.wiktionary.org/w/index.php?title=Special:AllPages&from=Hillsborough!&namespace=0displays
a list of index pages from Hilma, instead of pages titles from
Hilma.

http://en.wiktionary.org/wiki/Special:AllPages/Hiffe displays no pages at
all, and clicking on "Go" switches to Prefix index.

Not useful. Can we put back the working version please?
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Voice of All]

The format changed to avoid excessively long output. I've looked at the form and those links on wiktionary and don't see anything incorrect or not "working".

-Aaron Schulz

> Date: Wed, 20 Aug 2008 08:06:29 +0300
> From: rlullmann@...
> To: wikitech-l@...
> Subject: [Wikitech-l] Special:All pages broken at about 03:00 UTC this morning
>
> No longer displays pages from a title. In some cases it will display a list
> of index pages.
>
> For example,
> http://en.wiktionary.org/w/index.php?title=Special:AllPages&from=Hillsborough!&namespace=0displays
> a list of index pages from Hilma, instead of pages titles from
> Hilma.
>
> http://en.wiktionary.org/wiki/Special:AllPages/Hiffe displays no pages at
> all, and clicking on "Go" switches to Prefix index.
>
> Not useful. Can we put back the working version please?
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@...
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_________________________________________________________________
Get ideas on sharing photos from people like you.  Find new ways to share.
http://www.windowslive.com/explore/photogallery/posts?ocid=TXT_TAGLM_WL_Photo_Gallery_082008
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Robert Ullmann]

Like this:

http://en.wiktionary.org/wiki/Special:AllPages/Hill

used to display 480 titles starting at "Hill". Now it does what PrefixIndex
does. If I wanted that, I'd use Prefixindex. How do we get access to the
original function? (And I don't understand the bit about "excessively long
output" at all, it always displayed 480 titles (except at the end of the
wiki).

It is now broken, and should be put back. (In particular, the python
wikipedia framework has been broken.)

Two things in particular are now broken: the ability to display a section of
the title index from an arbitrary starting point, and the the ability to
display titles starting with at a prefix that doesn't exist in the database.
(Also the undesirable  switch from Allpages to Prefixindex.)

How do we do those now?

Robert

(also a meta question: how do we find out about these things before waking
up to find them broken?)

On Wed, Aug 20, 2008 at 8:30 AM, Aaron Schulz <jschulz_4587@...> wrote:

>
> The format changed to avoid excessively long output. I've looked at the
> form and those links on wiktionary and don't see anything incorrect or not
> "working".
>
> -Aaron Schulz
>
> > Date: Wed, 20 Aug 2008 08:06:29 +0300
> > From: rlullmann@...
> > To: wikitech-l@...
> > Subject: [Wikitech-l] Special:All pages broken at about 03:00 UTC this
>      morning
> >
> > No longer displays pages from a title. In some cases it will display a
> list
> > of index pages.
> >
> > For example,
> >
> http://en.wiktionary.org/w/index.php?title=Special:AllPages&from=Hillsborough!&namespace=0displays<http://en.wiktionary.org/w/index.php?title=Special:AllPages&from=Hillsborough%21&namespace=0displays>
> > a list of index pages from Hilma, instead of pages titles from
> > Hilma.
> >
> > http://en.wiktionary.org/wiki/Special:AllPages/Hiffe displays no pages
> at
> > all, and clicking on "Go" switches to Prefix index.
> >
> > Not useful. Can we put back the working version please?
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l@...
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> _________________________________________________________________
> Get ideas on sharing photos from people like you.  Find new ways to share.
>
> http://www.windowslive.com/explore/photogallery/posts?ocid=TXT_TAGLM_WL_Photo_Gallery_082008
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@...
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On Wed, Aug 20, 2008 at 3:58 PM, Robert Ullmann <rlullmann@...> wrote:
> It is now broken, and should be put back. (In particular, the python
> wikipedia framework has been broken.)

The User Interface is designed for users. The python Wikipedia
framework, therefore, should use the API, and then it won't have these
problems.

--
Andrew Garrett

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Robert Ullmann]

Quite so. However, it doesn't. And a *very* large amount of stuff is built
on it. Would be good to fix it. Before changing the UI it relies on.

Question: How would the people maintaining it have known that it was going
to be broken this morning?

Is easy to fix, took me < 3 minutes to fix my copy of wikipedia.py (mostly
deleting code ;-).

But everyone else using the allpages generator will be stuck, and almost all
will have no clue what is wrong or how to fix it.

On Wed, Aug 20, 2008 at 10:17 AM, Andrew Garrett <andrew@...> wrote:

> On Wed, Aug 20, 2008 at 3:58 PM, Robert Ullmann <rlullmann@...>
> wrote:
> > It is now broken, and should be put back. (In particular, the python
> > wikipedia framework has been broken.)
>
> The User Interface is designed for users. The python Wikipedia
> framework, therefore, should use the API, and then it won't have these
> problems.
>
> --
> Andrew Garrett
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@...
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Daniel Friesen]

Anyone developing automated software which interfaces with MediaWiki
should be subscribed to this list. And for any such person, this thread
should be enough of a red-flag for them.
The API has existed for awhile, and the UI should not be depended on for
any automated things.

~Daniel Friesen(Dantman, Nadir-Seen-Fire) of:
-The Nadir-Point Group (http://nadir-point.com)
--It's Wiki-Tools subgroup (http://wiki-tools.com)
--The ElectronicMe project (http://electronic-me.org)
--Games-G.P.S. (http://ggps.org)
-And Wikia ACG on Wikia.com (http://wikia.com/wiki/Wikia_ACG)
--Animepedia (http://anime.wikia.com)
--Narutopedia (http://naruto.wikia.com)

Robert Ullmann wrote:

> Quite so. However, it doesn't. And a *very* large amount of stuff is built
> on it. Would be good to fix it. Before changing the UI it relies on.
>
> Question: How would the people maintaining it have known that it was going
> to be broken this morning?
>
> Is easy to fix, took me < 3 minutes to fix my copy of wikipedia.py (mostly
> deleting code ;-).
>
> But everyone else using the allpages generator will be stuck, and almost all
> will have no clue what is wrong or how to fix it.
>
> On Wed, Aug 20, 2008 at 10:17 AM, Andrew Garrett <andrew@...> wrote:
>
>  
>> On Wed, Aug 20, 2008 at 3:58 PM, Robert Ullmann <rlullmann@...>
>> wrote:
>>    
>>> It is now broken, and should be put back. (In particular, the python
>>> wikipedia framework has been broken.)
>>>      
>> The User Interface is designed for users. The python Wikipedia
>> framework, therefore, should use the API, and then it won't have these
>> problems.
>>
>> --
>> Andrew Garrett
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l@...
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>>    

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Aryeh Gregor]

On Wed, Aug 20, 2008 at 1:58 AM, Robert Ullmann <rlullmann@...> wrote:
> Like this:
>
> http://en.wiktionary.org/wiki/Special:AllPages/Hill
>
> used to display 480 titles starting at "Hill". Now it does what PrefixIndex
> does. If I wanted that, I'd use Prefixindex. How do we get access to the
> original function? (And I don't understand the bit about "excessively long
> output" at all, it always displayed 480 titles (except at the end of the
> wiki).

As far as I can tell, formerly the URL you gave did nothing at all.
r39427 <http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=39427>
appears to have allowed subpage syntax to be used for AllPages; if I
undo that commit then AllPages/Foo returns the same page as plain old
AllPages.  Your suggested behavior appears more sensible in any event,
though.

I'm not really sympathetic to bot authors who screen-scrape special
pages instead of using the API, though.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Lars Aronsson]

Aaron Schulz wrote:

> The format changed to avoid excessively long output. I've looked
> at the form and those links on wiktionary and don't see anything
> incorrect or not "working".

Changing Allpages to Prefindex doesn't accomplish this.  Much
shorter lists than before are now returned for unusual prefixes,
such as "Kinematic". But I can still request a Prefindex for "A"
and get 350 (?) links listed.  Apparently there is both a change
in behaviour (from Allpages to Prefindex) and a limit (to 350
links), and the desired result is achieved by the limit, not by
the changed behaviour.

A better solution would be to provide a limit of 200 or 350 links
in the output, but keeping the old behaviour of Allpages.

Especially, the ability to step *back* as well as forward in that
list can be very useful, when you're looking for alternative
spellings of a non-existing page name.  After searching for
Kinematics, I need to step backwards to find Kinematic diagram.



--
  Lars Aronsson (lars@...)
  Aronsson Datateknik - http://aronsson.se

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Brion Vibber-3]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Ullmann wrote:
> Quite so. However, it doesn't. And a *very* large amount of stuff is built
> on it. Would be good to fix it. Before changing the UI it relies on.
>
> Question: How would the people maintaining it have known that it was going
> to be broken this morning?

They have known for the past SEVERAL YEARS that UI COMPONENTS FREQUENTLY
CHANGE and they should use MACHINE-READABLE APIS, as they have been told
repeatedly OVER THE YEARS.

- -- brion
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkisQBsACgkQwRnhpk1wk45GVwCgx0FBY2oVOaTZfeyt8zqCoNwg
SOAAoK+oEqOpq2x6ZHqwh9zJ511m0+Df
=hB4h
-----END PGP SIGNATURE-----

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Brion Vibber-3]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Aryeh Gregor wrote:

> On Wed, Aug 20, 2008 at 1:58 AM, Robert Ullmann <rlullmann@...> wrote:
>> Like this:
>>
>> http://en.wiktionary.org/wiki/Special:AllPages/Hill
>>
>> used to display 480 titles starting at "Hill". Now it does what PrefixIndex
>> does. If I wanted that, I'd use Prefixindex. How do we get access to the
>> original function? (And I don't understand the bit about "excessively long
>> output" at all, it always displayed 480 titles (except at the end of the
>> wiki).
>
> As far as I can tell, formerly the URL you gave did nothing at all.

You're incorrect.

- -- brion
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkisQG4ACgkQwRnhpk1wk466rwCeNFPGtvNgq6+W1PMAjhsxnzmL
kN8AoMrN0zASrL8CjDmXrlT0ZxUZdEYw
=OC+D
-----END PGP SIGNATURE-----

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Brion Vibber-3]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lars Aronsson wrote:
> Especially, the ability to step *back* as well as forward in that
> list can be very useful, when you're looking for alternative
> spellings of a non-existing page name.  After searching for
> Kinematics, I need to step backwards to find Kinematic diagram.

Indeed, we've had regressions in the paging links (they used to be
there, in both directions) as well as the starting point behavior.

Reverted in r39708.

- -- brion
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkisQsoACgkQwRnhpk1wk44H3ACdGpkU5fLkIVR3vU4uwi2WxTNV
EwwAoI/tmd47EwkUqFQKmwAIA2LNdzps
=8gxC
-----END PGP SIGNATURE-----

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Ilmari Karonen]

Brion Vibber wrote:

> Robert Ullmann wrote:
>> Quite so. However, it doesn't. And a *very* large amount of stuff is built
>> on it. Would be good to fix it. Before changing the UI it relies on.
>>
>> Question: How would the people maintaining it have known that it was going
>> to be broken this morning?
>
> They have known for the past SEVERAL YEARS that UI COMPONENTS FREQUENTLY
> CHANGE and they should use MACHINE-READABLE APIS, as they have been told
> repeatedly OVER THE YEARS.

...so where's the edit API?

(Yeah, I know.  But my real point is, you'll have a hard time convincing
people that the API is actually a finished product ready for prime time
as long as its feature set, as enabled on Wikimedia sites, has a huge
gaping hole like that.)

--
Ilmari Karonen

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Brion Vibber-3]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ilmari Karonen wrote:

> Brion Vibber wrote:
>> Robert Ullmann wrote:
>>> Quite so. However, it doesn't. And a *very* large amount of stuff is built
>>> on it. Would be good to fix it. Before changing the UI it relies on.
>>>
>>> Question: How would the people maintaining it have known that it was going
>>> to be broken this morning?
>> They have known for the past SEVERAL YEARS that UI COMPONENTS FREQUENTLY
>> CHANGE and they should use MACHINE-READABLE APIS, as they have been told
>> repeatedly OVER THE YEARS.
>
> ...so where's the edit API?

Why is an edit API needed for reading a list of page names? Oh wait, it
isn't! :)

The edit API is enabled on test.wikipedia.org for testing. If someday I
hear from the API developers that it's ready to go, we'll see about
turning it on elsewhere.

- -- brion
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkisRc4ACgkQwRnhpk1wk47M/ACdEqmHERiPwAHJe99bqdU6nMTr
dkwAnAjlzFiHoIslJglBZ+ZaqxyGHmSO
=nWvQ
-----END PGP SIGNATURE-----

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Russell Blau-2]

"Brion Vibber" <brion@...> wrote in
message news:48AC45CE.8060308@......

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ilmari Karonen wrote:
>> Brion Vibber wrote:
>>> Robert Ullmann wrote:
>>>> Quite so. However, it doesn't. And a *very* large amount of stuff is
>>>> built
>>>> on it. Would be good to fix it. Before changing the UI it relies on.
>>>>
>>>> Question: How would the people maintaining it have known that it was
>>>> going
>>>> to be broken this morning?
>>> They have known for the past SEVERAL YEARS that UI COMPONENTS FREQUENTLY
>>> CHANGE and they should use MACHINE-READABLE APIS, as they have been told
>>> repeatedly OVER THE YEARS.
>>
>> ...so where's the edit API?
>
> Why is an edit API needed for reading a list of page names? Oh wait, it
> isn't! :)

That, of course, is not the point at all.  The point is, why invest large
amounts of time and effort in developing a bot framework to use an API that
isn't yet complete?




_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Brion Vibber-3]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Russell Blau wrote:
>> Why is an edit API needed for reading a list of page names? Oh wait, it
>> isn't! :)
>
> That, of course, is not the point at all.  The point is, why invest large
> amounts of time and effort in developing a bot framework to use an API that
> isn't yet complete?

The API will never be complete -- it will always be under development.

MediaWiki will never be complete -- it will always be under development.

Wikipedia will never be complete -- it will always be under development.

Yet, I don't think we're going to back off and not use any of them until
they're done?

Screen-scraping constantly-changing UI is like repeatedly banging
yourself in the head with a bowling ball. It's painful and doesn't
accomplish much, but it feels SO GOOD when you stop!

It's not as though nobody uses the API -- it's used by lots of bot
frameworks and custom JS gadgets on the site, and has been FOR YEARS.
It's time for pywikipediabot to wake up and stop hitting itself in the
head...

- -- brion
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkisTSoACgkQwRnhpk1wk45rMgCgqm2FtUbEBCnkd3P4uY8dR+nm
oVAAoIDpoNCy/skMlP8Mai2bfPQwk/Us
=4sZR
-----END PGP SIGNATURE-----

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[SQL-6]

Russell Blau wrote:

> "Brion Vibber" <brion@...> wrote in
> message news:48AC45CE.8060308@......
>  
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Ilmari Karonen wrote:
>>    
>>> Brion Vibber wrote:
>>>      
>>>> Robert Ullmann wrote:
>>>>        
>>>>> Quite so. However, it doesn't. And a *very* large amount of stuff is
>>>>> built
>>>>> on it. Would be good to fix it. Before changing the UI it relies on.
>>>>>
>>>>> Question: How would the people maintaining it have known that it was
>>>>> going
>>>>> to be broken this morning?
>>>>>          
>>>> They have known for the past SEVERAL YEARS that UI COMPONENTS FREQUENTLY
>>>> CHANGE and they should use MACHINE-READABLE APIS, as they have been told
>>>> repeatedly OVER THE YEARS.
>>>>        
>>> ...so where's the edit API?
>>>      
>> Why is an edit API needed for reading a list of page names? Oh wait, it
>> isn't! :)
>>    
>
> That, of course, is not the point at all.  The point is, why invest large
> amounts of time and effort in developing a bot framework to use an API that
> isn't yet complete?
>
>
>
>  

I've been using the read API for at least a year, in a custom framework
of my own design. It's easier to use than screen-scraping, uses less
bandwidth for both the host and, client, it's stable, and it's faster.
There's every reason in the world to use it, including that you don't
have to worry about the UI changing by surprise at 3AM.

SQL

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Aryeh Gregor]

On Wed, Aug 20, 2008 at 12:03 PM, Brion Vibber <brion@...> wrote:
> Aryeh Gregor wrote:
>> As far as I can tell, formerly the URL you gave did nothing at all.
>
> You're incorrect.

Yeah, looks like I only reverted one revision to test when it was a
whole series.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Victor Vasiliev]

SQL writes:
> There's every reason in the world to use it, including that you don't
> have to worry about the UI changing by surprise at 3AM.
>
> SQL
Yes, you just have to worry that query.php is replaced with api.php,
which may be then replaced with something else. And also, I went through
the mediawiki-api archives and found about 9 breaking changes for API.
Pretty stable, isn't it? :)
--vvv

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Aryeh Gregor]

On Wed, Aug 20, 2008 at 1:58 PM, Victor Vasiliev <vasilvv@...> wrote:
> Yes, you just have to worry that query.php is replaced with api.php,
> which may be then replaced with something else. And also, I went through
> the mediawiki-api archives and found about 9 breaking changes for API.
> Pretty stable, isn't it? :)

I don't understand why there are ever breaking changes in the API.
THE API SHOULD NEVER EVER EVER CHANGE.  IT SHOULD BE A STABLE
INTERFACE.  There needs to be a policy, set in stone, not negotiable,
that behavior for a given API query should NEVER change in an
incompatible fashion.  If something is inconsistent it should be
deprecated and a new function instituted to replace it.  The old
deprecated way could then be marked for a few years (say, three years,
or at least one year) before finally being removed.  Authors of bot
frameworks should be encouraged to write code checking for
<deprecated> or whatever and having their framework raise a warning of
some kind to inform them that they should update it.

This really seems like a no-brainer to me.  But, I'm not an API
developer, so . . .

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Roan Kattouw]

Russell Blau schreef:
>
> That, of course, is not the point at all.  The point is, why invest large
> amounts of time and effort in developing a bot framework to use an API that
> isn't yet complete?
Because it can replace *most* (not all, yet) stuff screenscraping is
currently still used for, and it'll be easier to migrate edit functions
to the API once they're available.

About enabling the edit API on Wikipedia: I think it's ready to go. The
only real problem it currently has is that action=edit doesn't handle
aborts by hooks too well; the edit will be aborted, but the client will
only get a vague "A hook aborted the action" (or something along those
lines) error. A hook that allows for aborting the edit with a meaningful
error message has been added (APIEditBeforeSave), but AFAIK it's
currently only used by ConfirmEdit (and only because I migrated it
myself). Similarly, extensions influencing other actions may rely on UI
hooks that the API bypasses altogether.

We probably want testers' input here too. Also, we'll probably want to
investigate which of the extensions enabled on Wikipedia use 'wrong'
hooks and fix them.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l