No-indexing of project-space pages

[David Gerard-2]

2008/7/25 Newyorkbrad (Wikipedia) <newyorkbrad@...>:

> Any thoughts following up on the status of our ability to create improved
> internal searching as discussed here?  Implementation of the no-index
> proposal should not await (and I gather is not awaiting) improvements in
> this feature, but a commitment to proceed with such improvements would
> eliminate virtually the only serious objection that has been raised to it.


Our search has improved markedly in the past year and continues to get
better. If people can give examples of where our internal search just
isn't up to the task, those will be of great use to the devs.


- d.

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[geni]

2008/7/25 David Gerard <dgerard@...>:

> 2008/7/25 Newyorkbrad (Wikipedia) <newyorkbrad@...>:
>
>> Any thoughts following up on the status of our ability to create improved
>> internal searching as discussed here?  Implementation of the no-index
>> proposal should not await (and I gather is not awaiting) improvements in
>> this feature, but a commitment to proceed with such improvements would
>> eliminate virtually the only serious objection that has been raised to it.
>
>
> Our search has improved markedly in the past year and continues to get
> better. If people can give examples of where our internal search just
> isn't up to the task, those will be of great use to the devs.

serious
No spell checking.
less serious
I can't enter chemical structures.

--
geni

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[WJhonson]

Considering as how this issue of noindexing user-space and user-talk by default only seems an issue with a very small handful of editors, I would think the community at-large, who has been having their space indexed for years, would fall on the side of continuing withthe *least* amount of disruption to the status quo.


??? That would be to no-index *on request* user pages and their associated talk pages, project pages and their associated talk pages.? That is, the default would be, as it is today, to index.? But a project or user can opt-in to not index.


?


Will Johnson













-----Original Message-----

From: Newyorkbrad (Wikipedia) <newyorkbrad@...>

To: English Wikipedia <wikien-l@...>

Sent: Thu, 24 Jul 2008 3:40 pm

Subject: Re: [WikiEN-l] No-indexing of project-space pages





On 7/23/08, WJhonson@... <WJhonson@...> wrote:  >  > Killing a mosquito with a hammer is not the proper approach however.  > Most of, if not all the major issues NYB brought up, were  > addressed already.      I appreciate from Steve Bain's and other posts that the issue I raised in my  first post has been addressed with regard to XfD, RfA, RfAr, RFC,  and BLP/N pages (that is, at least the archives of these pages; I think the  current pages should be no-indexed as well).  However, the same principles  that called for no-indexing these pages would also apply to DRV (which is  indistinguishable in principle, for these purposes, from XfD; the only  reason those pages are still indexed, as I understand it, has to do with  page layout issues), as well as SSP, RfCU, the former PAIN and CSN, WQA, AN,  AN/I, and AN3.    Pending issues include (1) do we agree to no-indexing all of these; (2)  should project space be no-index with an "index this page" opt-in (my  preference and how do we set that up); (3) how do we treat userspace (my  preference would be a no-index default with users allowed to opt into being  indexed if they wish); and (4) other namespaces.    Could someone post a link to any discussion that might be taking place  on-wiki or on Bugzilla?    Thanks,  Newyorkbrad  _______________________________________________  WikiEN-l mailing list  WikiEN-l@...  To unsubscribe from this mailing list, visit:  https://lists.wikimedia.org/mailman/listinfo/wikien-l 

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[Chris Howie]

On Thu, Jul 24, 2008 at 11:07 PM,  <wjhonson@...> wrote:
> Considering as how this issue of noindexing user-space and user-talk by default only seems an issue with a very small handful of editors, I would think the community at-large, who has been having their space indexed for years, would fall on the side of continuing withthe *least* amount of disruption to the status quo.
>
> ??? That would be to no-index *on request* user pages and their associated talk pages, project pages and their associated talk pages.? That is, the default would be, as it is today, to index.? But a project or user can opt-in to not index.

I'd say a reasonable middle-ground would be to upgrade the software to
support __INDEX__ and __NOINDEX__ then give users a week or two to tag
the stuff they want indexed with __INDEX__ before flipping the noindex
switch on.  This would allow us to keep indexing the useful stuff
without having to hunt down every userfied BLP and tag it.  Also, new
userspace pages would be no-indexed by default, which seems like a
good thing to me.

--
Chris Howie
http://www.chrishowie.com
http://en.wikipedia.org/wiki/User:Crazycomputers

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[geni]

2008/7/25 Chris Howie <cdhowie@...>:

> On Thu, Jul 24, 2008 at 11:07 PM,  <wjhonson@...> wrote:
>> Considering as how this issue of noindexing user-space and user-talk by default only seems an issue with a very small handful of editors, I would think the community at-large, who has been having their space indexed for years, would fall on the side of continuing withthe *least* amount of disruption to the status quo.
>>
>> ??? That would be to no-index *on request* user pages and their associated talk pages, project pages and their associated talk pages.? That is, the default would be, as it is today, to index.? But a project or user can opt-in to not index.
>
> I'd say a reasonable middle-ground would be to upgrade the software to
> support __INDEX__ and __NOINDEX__ then give users a week or two to tag
> the stuff they want indexed with __INDEX__ before flipping the noindex
> switch on.  This would allow us to keep indexing the useful stuff
> without having to hunt down every userfied BLP and tag it.  Also, new
> userspace pages would be no-indexed by default, which seems like a
> good thing to me.
>

With the amount of stuff that it is useful to index about the only
sane response to putting such a procedure in place would be to bot tag
everything with __INDEX__ .

--
geni

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[David Katz-5]

A reasonable compromise

On Thu, Jul 24, 2008 at 11:23 PM, Chris Howie <cdhowie@...> wrote:

> On Thu, Jul 24, 2008 at 11:07 PM,  <wjhonson@...> wrote:
>> Considering as how this issue of noindexing user-space and user-talk by default only seems an issue with a very small handful of editors, I would think the community at-large, who has been having their space indexed for years, would fall on the side of continuing withthe *least* amount of disruption to the status quo.
>>
>> ??? That would be to no-index *on request* user pages and their associated talk pages, project pages and their associated talk pages.? That is, the default would be, as it is today, to index.? But a project or user can opt-in to not index.
>
> I'd say a reasonable middle-ground would be to upgrade the software to
> support __INDEX__ and __NOINDEX__ then give users a week or two to tag
> the stuff they want indexed with __INDEX__ before flipping the noindex
> switch on.  This would allow us to keep indexing the useful stuff
> without having to hunt down every userfied BLP and tag it.  Also, new
> userspace pages would be no-indexed by default, which seems like a
> good thing to me.
>
> --
> Chris Howie
> http://www.chrishowie.com
> http://en.wikipedia.org/wiki/User:Crazycomputers
>
> _______________________________________________
> WikiEN-l mailing list
> WikiEN-l@...
> To unsubscribe from this mailing list, visit:
> https://lists.wikimedia.org/mailman/listinfo/wikien-l
>

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[Risker]

2008/7/25 Chris Howie <cdhowie@...>:

> <snip>
>

>
> I'd say a reasonable middle-ground would be to upgrade the software to
> support __INDEX__ and __NOINDEX__ then give users a week or two to tag
> the stuff they want indexed with __INDEX__ before flipping the noindex
> switch on.  This would allow us to keep indexing the useful stuff
> without having to hunt down every userfied BLP and tag it.  Also, new
> userspace pages would be no-indexed by default, which seems like a
> good thing to me.
>
>

Concur that this sounds entirely reasonable, although it might be worthwhile
for some BLP-sensitive folk to trawl through the ones that people tag as
__INDEX__ at a future date.

Risker
_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[Gregory Maxwell]

On Fri, Jul 25, 2008 at 1:28 AM, Risker <risker.wp@...> wrote:
> Concur that this sounds entirely reasonable, although it might be worthwhile
> for some BLP-sensitive folk to trawl through the ones that people tag as
> __INDEX__ at a future date.

If it gets it implemented and lets us change the default... GREAT!

May well be that __INDEX__ becomes Wikipediaspeak for __SPAM__, if so,
then so be it... at least it wouldn't be too hard to watch for.

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[John Vandenberg]

On Fri, Jul 25, 2008 at 1:50 PM, geni <geniice@...> wrote:

> 2008/7/25 Chris Howie <cdhowie@...>:
>> On Thu, Jul 24, 2008 at 11:07 PM,  <wjhonson@...> wrote:
>>> Considering as how this issue of noindexing user-space and user-talk by default only seems an issue with a very small handful of editors, I would think the community at-large, who has been having their space indexed for years, would fall on the side of continuing withthe *least* amount of disruption to the status quo.
>>>
>>> ??? That would be to no-index *on request* user pages and their associated talk pages, project pages and their associated talk pages.? That is, the default would be, as it is today, to index.? But a project or user can opt-in to not index.
>>
>> I'd say a reasonable middle-ground would be to upgrade the software to
>> support __INDEX__ and __NOINDEX__ then give users a week or two to tag
>> the stuff they want indexed with __INDEX__ before flipping the noindex
>> switch on.  This would allow us to keep indexing the useful stuff
>> without having to hunt down every userfied BLP and tag it.  Also, new
>> userspace pages would be no-indexed by default, which seems like a
>> good thing to me.
>>
>
>
> With the amount of stuff that it is useful to index about the only
> sane response to putting such a procedure in place would be to bot tag
> everything with __INDEX__ .

We would definitely need bots to tag large slabs of pages that are
useful, but I think this is a sensitive enough issue that we should do
it right.  A project page listing all pages to be categorised created
to collate reasonable classes of pages, time allowed for a little
discussion, and only then should operators man their bots.

--
John

I do not agree.   We have immense amount of crap in our other
namespaces that nobody will actively find and tag/untag.

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[John Vandenberg]

On Fri, Jul 25, 2008 at 3:52 PM, John Vandenberg <jayvdb@...> wrote:

> On Fri, Jul 25, 2008 at 1:50 PM, geni <geniice@...> wrote:
>> With the amount of stuff that it is useful to index about the only
>> sane response to putting such a procedure in place would be to bot tag
>> everything with __INDEX__ .
>
> We would definitely need bots to tag large slabs of pages that are
> useful, but I think this is a sensitive enough issue that we should do
> it right.  A project page listing all pages to be categorised created
> to collate reasonable classes of pages, time allowed for a little
> discussion, and only then should operators man their bots.
>
> --
> John
>
> I do not agree.   We have immense amount of crap in our other
> namespaces that nobody will actively find and tag/untag.

This part tacked on the end is the reason why we should not let bots
tag _everything_ as you suggested.

--
John

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[Stephen Bain]

On Fri, Jul 25, 2008 at 9:31 AM, geni <geniice@...> wrote:
>
> serious
> No spell checking.

If memory serves me correctly, spell checking is a feature currently
available in the MWSearch system, though last I heard it had
performance issues, so if it exists it's probably not turned on yet on
the Wikimedia projects.

--
Stephen Bain
stephen.bain@...

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[Luna-4]

Perhaps the ideal solution is one that omits these pages from search
results UNLESS the user specifically requests them in some way (for
example, a search term including "site:en.wikipedia.org")... but I
kinda doubt that's realistic, on my part.

I'll be sorry to lose some ability to find information easily, but
it's an unfortunate conundrum that articles and discussion pages are
given the same weight, and absent some better way to fix the resulting
problems... at least our internal search capabilities have gradually
been improving, as I think has consistently been the main worry when
this has come up in the past.

Should policy/guideline/essay/help pages continue to be indexed? It
primarily seems to be process/activity pages which are problematic in
this regard.

Are user pages uniformly problematic, or mainly those for
banned/blocked users? If it's an isolated group at all, we could just
as easily use the __NOINDEX__ keyword in relevant templates.

-Luna

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[geni]

2008/7/26 Luna <lunasantin@...>:
> Perhaps the ideal solution is one that omits these pages from search
> results UNLESS the user specifically requests them in some way (for
> example, a search term including "site:en.wikipedia.org")... but I
> kinda doubt that's realistic, on my part.
>
> I'll be sorry to lose some ability to find information easily, but
> it's an unfortunate conundrum that articles and discussion pages are
> given the same weight,

Very unlikely. Notice how rarely a talk page beats out it's article page.

--
geni

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[David Katz-5]

Would this prevent mirror sites from 'scraping' noindex pages? This
would be a definite improvement as talk pages are often a depository
of libel and whilst we can remove and oversight material on wp pages
we're powerless to do anything on a mirror

On 7/25/08, geni <geniice@...> wrote:

> 2008/7/26 Luna <lunasantin@...>:
>> Perhaps the ideal solution is one that omits these pages from search
>> results UNLESS the user specifically requests them in some way (for
>> example, a search term including "site:en.wikipedia.org")... but I
>> kinda doubt that's realistic, on my part.
>>
>> I'll be sorry to lose some ability to find information easily, but
>> it's an unfortunate conundrum that articles and discussion pages are
>> given the same weight,
>
> Very unlikely. Notice how rarely a talk page beats out it's article page.
>
> --
> geni
>
> _______________________________________________
> WikiEN-l mailing list
> WikiEN-l@...
> To unsubscribe from this mailing list, visit:
> https://lists.wikimedia.org/mailman/listinfo/wikien-l
>

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[Chris Howie]

On Wed, Jul 30, 2008 at 3:12 AM, David Katz <dkatz2001@...> wrote:
> Would this prevent mirror sites from 'scraping' noindex pages? This
> would be a definite improvement as talk pages are often a depository
> of libel and whilst we can remove and oversight material on wp pages
> we're powerless to do anything on a mirror

I believe most mirrors use an actual database dump (though some might
scrape).  In that case they would get all the content on Wikipedia,
wherever it is.

--
Chris Howie
http://www.chrishowie.com
http://en.wikipedia.org/wiki/User:Crazycomputers

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[geni]

2008/7/30 David Katz <dkatz2001@...>:
> Would this prevent mirror sites from 'scraping' noindex pages? This
> would be a definite improvement as talk pages are often a depository
> of libel and whilst we can remove and oversight material on wp pages
> we're powerless to do anything on a mirror


It wouldn't and mirrors generally use database dumps anyway.

--
geni

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[David Katz-5]

On Wed, Jul 30, 2008 at 7:45 AM, Chris Howie <cdhowie@...> wrote:
> On Wed, Jul 30, 2008 at 3:12 AM, David Katz <dkatz2001@...> wrote:
>> Would this prevent mirror sites from 'scraping' noindex pages? This
>> would be a definite improvement as talk pages are often a depository
>> of libel and whilst we can remove and oversight material on wp pages
>> we're powerless to do anything on a mirror
>
> I believe most mirrors use an actual database dump (though some might
> scrape).  In that case they would get all the content on Wikipedia,
> wherever it is.

Is there anyway we can set things so that mirrors only mirror the
actual namespace content and not talk, user and WP administrative
pages?

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[David Gerard-2]

2008/7/30 David Katz <dkatz2001@...>:

> Is there anyway we can set things so that mirrors only mirror the
> actual namespace content and not talk, user and WP administrative
> pages?


Yeah, not put it in the dumps. However, effectively proprietising
Wikipedia in such a manner (remember, the project space is GFDL too)
doesn't really sit well with the "free content" thing.

Most mirrors wouldn't want that stuff to mirror Wikipedia anyway.
Those that do, are either Doing It Wrong or really do want to mirror
the project space too.


- d.

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[geni]

2008/7/30 David Katz <dkatz2001@...>:

> On Wed, Jul 30, 2008 at 7:45 AM, Chris Howie <cdhowie@...> wrote:
>> On Wed, Jul 30, 2008 at 3:12 AM, David Katz <dkatz2001@...> wrote:
>>> Would this prevent mirror sites from 'scraping' noindex pages? This
>>> would be a definite improvement as talk pages are often a depository
>>> of libel and whilst we can remove and oversight material on wp pages
>>> we're powerless to do anything on a mirror
>>
>> I believe most mirrors use an actual database dump (though some might
>> scrape).  In that case they would get all the content on Wikipedia,
>> wherever it is.
>
> Is there anyway we can set things so that mirrors only mirror the
> actual namespace content and not talk, user and WP administrative
> pages?

Nope. Tthere are a couple of ways you could do it in in theory but the
side effects would be unacceptable.


--
geni

_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l

[Jussi-Ville Heiskanen]

David Gerard wrote:

> 2008/7/30 David Katz <dkatz2001@...>:
>
>  
>> Is there anyway we can set things so that mirrors only mirror the
>> actual namespace content and not talk, user and WP administrative
>> pages?
>>    
>
>
> Yeah, not put it in the dumps. However, effectively proprietising
> Wikipedia in such a manner (remember, the project space is GFDL too)
> doesn't really sit well with the "free content" thing.
>
> Most mirrors wouldn't want that stuff to mirror Wikipedia anyway.
> Those that do, are either Doing It Wrong or really do want to mirror
> the project space too.
>
>
> - d.
>  

Umh, this might be a stupid question, but since I don't know any
better, I'll ask anyway...

What would be wrong with having sectioned dumps? With
mainspace and article talk pages in one dump section, and
Userpages, userspace subpages and user talk pages in
in another section of the dump, and a third dump for all
the wonderful "Wikipedia" namespace pages and their
talk pages.

That way everyone can pick and mix...

Yours,

Jussi-Ville Heiskanen


_______________________________________________
WikiEN-l mailing list
WikiEN-l@...
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l