Nabble - Wikipedia Developers

Re: Failed to Download any pages-articles.xml.bz2 file

2009-12-15T13:26:07Z

Rob Giberson wrote:
> hmm... I was just able to download another file from the enwiki list which
> is 2.4GB without any problem. Thoughts?

enwiki-20091128-pages-articles.xml.bz2 5795591893 bytes
Store that in a long int variable of 4 bytes. Now, retrieve it:
1500624597. Which is ~1.5GB (1,39 GiB).

Your company proxy doesn't allow downloading files bigger than 4 GB.
It will only take as size the remainder.

Perhaps you could fool it doing Range requests for the remainder (you
can use wget -c ).
Anyway, your company should get its proxy fixed.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Failed to Download any pages-articles.xml.bz2 file

2009-12-15T11:22:43Z

hmm... I was just able to download another file from the enwiki list which
is 2.4GB without any problem. Thoughts?

On Tue, Dec 15, 2009 at 10:37 AM, Marco Schuster <
marco@...> wrote:

> This is likely your company proxy not supporting huge downloads. Might be
> its cache partition got filled up.
>
> Marco
>
> 2009/12/15 Rob Giberson <ajaxdwr@...>
>
> > I failed several times and ended up downloading exactly the same number
> of
> > bytes: 1,465,454KB (1.46GB), even for different versions of that file.
> >
> > Is it possible that the files themselves are corrupted? Anyone
> successfully
> > downloaded one of these big files recently?
> >
> > Rob
> >
> > On Mon, Dec 14, 2009 at 8:11 PM, Rob Giberson <ajaxdwr@...> wrote:
> >
> > > You got that right. I am behind a company proxy. Any idea how to get
> this
> > > around?
> > >
> > >
> > > On Mon, Dec 14, 2009 at 8:09 PM, K. Peachey <p858snake@...
> > >wrote:
> > >
> > >> On Tue, Dec 15, 2009 at 12:25 PM, Rob Giberson <ajaxdwr@...>
> > >> wrote:
> > >> > Guess this problem might be asked several (many) times before...but
> > >> >
> > >> > I tried to download the pages-articles.xml.bz2 file which is
> > >> approximately
> > >> > 5.xGB. However, all versions I tried failed at about 1.5GB. I
> noticed
> > >> > someone posted a Python code online as a workaround, but it did not
> > work
> > >> for
> > >> > me.
> > >> >
> > >> > My machine is Windows Server 2008 64 bit. Any idea how to get this
> > huge
> > >> > file?
> > >> >
> > >> > Appreciated.
> > >> > Rob
> > >> You wouldn't happen to have proxies between you and your internet
> > >> connection would you?
> > >>
> > >> -Peachey
> > >>
> > >> _______________________________________________
> > >> Wikitech-l mailing list
> > >> Wikitech-l@...
> > >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >>
> > >
> > >
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l@...
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
>
>
>
> --
> VMSoft GbR
> Nabburger Str. 15
> 81737 München
> Geschäftsführer: Marco Schuster, Volker Hemmert
> http://vmsoft-gbr.de
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@...
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Failed to Download any pages-articles.xml.bz2 file

2009-12-15T10:37:06Z

This is likely your company proxy not supporting huge downloads. Might be
its cache partition got filled up.

Marco

2009/12/15 Rob Giberson <ajaxdwr@...>

> I failed several times and ended up downloading exactly the same number of
> bytes: 1,465,454KB (1.46GB), even for different versions of that file.
>
> Is it possible that the files themselves are corrupted? Anyone successfully
> downloaded one of these big files recently?
>
> Rob
>
> On Mon, Dec 14, 2009 at 8:11 PM, Rob Giberson <ajaxdwr@...> wrote:
>
> > You got that right. I am behind a company proxy. Any idea how to get this
> > around?
> >
> >
> > On Mon, Dec 14, 2009 at 8:09 PM, K. Peachey <p858snake@...
> >wrote:
> >
> >> On Tue, Dec 15, 2009 at 12:25 PM, Rob Giberson <ajaxdwr@...>
> >> wrote:
> >> > Guess this problem might be asked several (many) times before...but
> >> >
> >> > I tried to download the pages-articles.xml.bz2 file which is
> >> approximately
> >> > 5.xGB. However, all versions I tried failed at about 1.5GB. I noticed
> >> > someone posted a Python code online as a workaround, but it did not
> work
> >> for
> >> > me.
> >> >
> >> > My machine is Windows Server 2008 64 bit. Any idea how to get this
> huge
> >> > file?
> >> >
> >> > Appreciated.
> >> > Rob
> >> You wouldn't happen to have proxies between you and your internet
> >> connection would you?
> >>
> >> -Peachey
> >>
> >> _______________________________________________
> >> Wikitech-l mailing list
> >> Wikitech-l@...
> >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >>
> >
> >
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@...
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

--
VMSoft GbR
Nabburger Str. 15
81737 München
Geschäftsführer: Marco Schuster, Volker Hemmert
http://vmsoft-gbr.de
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Failed to Download any pages-articles.xml.bz2 file

2009-12-15T10:32:30Z

I failed several times and ended up downloading exactly the same number of
bytes: 1,465,454KB (1.46GB), even for different versions of that file.

Is it possible that the files themselves are corrupted? Anyone successfully
downloaded one of these big files recently?

Rob

On Mon, Dec 14, 2009 at 8:11 PM, Rob Giberson <ajaxdwr@...> wrote:

> You got that right. I am behind a company proxy. Any idea how to get this
> around?
>
>
> On Mon, Dec 14, 2009 at 8:09 PM, K. Peachey <p858snake@...>wrote:
>
>> On Tue, Dec 15, 2009 at 12:25 PM, Rob Giberson <ajaxdwr@...>
>> wrote:
>> > Guess this problem might be asked several (many) times before...but
>> >
>> > I tried to download the pages-articles.xml.bz2 file which is
>> approximately
>> > 5.xGB. However, all versions I tried failed at about 1.5GB. I noticed
>> > someone posted a Python code online as a workaround, but it did not work
>> for
>> > me.
>> >
>> > My machine is Windows Server 2008 64 bit. Any idea how to get this huge
>> > file?
>> >
>> > Appreciated.
>> > Rob
>> You wouldn't happen to have proxies between you and your internet
>> connection would you?
>>
>> -Peachey
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l@...
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Failed to Download any pages-articles.xml.bz2 file

2009-12-14T20:11:05Z

You got that right. I am behind a company proxy. Any idea how to get this
around?

On Mon, Dec 14, 2009 at 8:09 PM, K. Peachey <p858snake@...> wrote:

> On Tue, Dec 15, 2009 at 12:25 PM, Rob Giberson <ajaxdwr@...> wrote:
> > Guess this problem might be asked several (many) times before...but
> >
> > I tried to download the pages-articles.xml.bz2 file which is
> approximately
> > 5.xGB. However, all versions I tried failed at about 1.5GB. I noticed
> > someone posted a Python code online as a workaround, but it did not work
> for
> > me.
> >
> > My machine is Windows Server 2008 64 bit. Any idea how to get this huge
> > file?
> >
> > Appreciated.
> > Rob
> You wouldn't happen to have proxies between you and your internet
> connection would you?
>
> -Peachey
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@...
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Failed to Download any pages-articles.xml.bz2 file

2009-12-14T20:09:32Z

On Tue, Dec 15, 2009 at 12:25 PM, Rob Giberson <ajaxdwr@...> wrote:

> Guess this problem might be asked several (many) times before...but
>
> I tried to download the pages-articles.xml.bz2 file which is approximately
> 5.xGB. However, all versions I tried failed at about 1.5GB. I noticed
> someone posted a Python code online as a workaround, but it did not work for
> me.
>
> My machine is Windows Server 2008 64 bit. Any idea how to get this huge
> file?
>
> Appreciated.
> Rob

You wouldn't happen to have proxies between you and your internet
connection would you?

-Peachey

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Failed to Download any pages-articles.xml.bz2 file

2009-12-14T19:43:00Z

Guess this problem might be asked several (many) times before...but

I tried to download the pages-articles.xml.bz2 file which is approximately
5.xGB. However, all versions I tried failed at about 1.5GB. I noticed
someone posted a Python code online as a workaround, but it did not work for
me.

My machine is Windows Server 2008 64 bit. Any idea how to get this huge
file?

Appreciated.
Rob
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Failed to Download any pages-articles.xml.bz2 file

2009-12-14T18:25:26Z

Re: how to add custom details in all sidebar (skins)

2009-12-14T14:20:01Z

Prince Gerald Albert - ERS, HCL Tech wrote:

> I'm creating a plugin for mediawiki and would like to display details in all plugins.
>
> To my understanding I need to write details in all the skins or is there a way I can write in one place and each skin renders and displays.
>
> I understand for each skin the design would be different and little tweak required.
>
> Can you please suggest the best way to do it or some reference articles which will help?
>
> Thanks in advance
> Prince Gerald

Take a look at SkinBuildSidebar hook, or perhaps SkinTemplateToolboxEnd.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

how to add custom details in all sidebar (skins)

2009-12-14T02:26:46Z

I'm creating a plugin for mediawiki and would like to display details in all plugins.

To my understanding I need to write details in all the skins or is there a way I can write in one place and each skin renders and displays.

I understand for each skin the design would be different and little tweak required.

Can you please suggest the best way to do it or some reference articles which will help?

Thanks in advance
Prince Gerald

DISCLAIMER:
-----------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of
this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have
received this email in error please delete it and notify the sender immediately. Before opening any mail and
attachments please check them for viruses and defect.

-----------------------------------------------------------------------------------------------------------------------
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Bugzilla Weekly Report

2009-12-13T19:00:03Z

MediaWiki Bugzilla Report for December 07, 2009 - December 14, 2009

Status changes this week

Bugs NEW : 90
Bugs ASSIGNED : 2
Bugs REOPENED : 7
Bugs RESOLVED : 73

Total bugs still open: 4191

Resolutions for the week:

Bugs marked FIXED : 54
Bugs marked REMIND : 1
Bugs marked INVALID : 8
Bugs marked DUPLICATE : 2
Bugs marked WONTFIX : 5
Bugs marked WORKSFORME : 1
Bugs marked LATER : 2
Bugs marked MOVED : 0

Specific Product/Component Resolutions & User Metrics

New Bugs Per Component

Translate 6
SemanticForms 4
Internationalization 3
Special pages 3
server 2

New Bugs Per Product

MediaWiki 18
Wikimedia 6
MediaWiki extensions 25
Wikipedia Mobile 3

Top 5 Bug Resolvers

roan.kattouw [AT] gmail.com 38
agarrett [AT] wikimedia.org 10
innocentkiller [AT] gmail.com 7
alex.emsenhuber [AT] bluewin.ch 5
JSchulz_4587 [AT] msn.com 2

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Flagged revs on en:wp?

2009-12-13T15:28:30Z

On 12/13/2009 05:59 AM, David Gerard wrote:
> Would it be possible for someone to write a techblog post about what's
> in the way of flagged revs on en:wp? Quite a few people are wondering
> what's up and why someone hasn't pulled the switch.
>

That seems like a fine thing to do. I've already promised to post an
update here once we have a clearer picture; I'll write up a more
general-audience version for the blog, too.

William

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Flagged revs on en:wp?

2009-12-13T08:47:08Z

Hi!

> Are you saying this hardware is already procured and deployed? That it'll
> be procured and deployed before FR is ready? Something else?

It will be procured and deployed once FR is actively used. We can tag few millions or tens of millions of revisions until then.

Domas
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Flagged revs on en:wp?

2009-12-13T07:55:42Z

On Sun, Dec 13, 2009 at 9:15 AM, Domas Mituzas <midom.lists@...>wrote:

> Why do people treat regular capacity planning as "problems".
>

Well, your original post implied that lack of new hardware could hold things
up.

"Still, we need new hardware, and as long as it is procured and deployed,
we'd be safe deploying FR."

Are you saying this hardware is already procured and deployed? That it'll
be procured and deployed before FR is ready? Something else?
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Flagged revs on en:wp?

2009-12-13T06:15:56Z

Hi!

> Would it be possible for someone to write a techblog post about what's
> in the way of flagged revs on en:wp? Quite a few people are wondering
> what's up and why someone hasn't pulled the switch.

There're no hardware problems today, and defragmenting data has prolonged the time-to-resource-crisis stuff ;-)
Why do people treat regular capacity planning as "problems".

Domas
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Flagged revs on en:wp?

2009-12-13T05:59:28Z

2009/12/13 Domas Mituzas <midom.lists@...>:

> We did defragment our databases during the switch, so diskspace concerns are not that big anymore. Still, we need new hardware, and as long as it is procured and deployed, we'd be safe deploying FR.

I hadn't realised hardware problems were in the way too ...

Would it be possible for someone to write a techblog post about what's
in the way of flagged revs on en:wp? Quite a few people are wondering
what's up and why someone hasn't pulled the switch.

- d.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Flagged revs on en:wp?

2009-12-13T05:41:23Z

Hi!

> Domas' concerns about hardware support are also illuminating ([1]). Has the
> switch to MySQL-5.1 alleviated those at all?

The switch hasn't improved anything (oh well, except crash recovery, and we used that feature already once! ;-) - maybe made developers a bit happier that they will be able to write less efficient queries in future.

We did defragment our databases during the switch, so diskspace concerns are not that big anymore. Still, we need new hardware, and as long as it is procured and deployed, we'd be safe deploying FR.

Domas
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Flagged revs on en:wp?

2009-12-13T05:15:17Z

"David Gerard" <dgerard@...> wrote in message
news:fbad4e140912101314m6da7284fv4d149b3a362c667d@......
> Do we have any time frame for flagged revisions on en:wp?
>

Domas' concerns about hardware support are also illuminating ([1]). Has the
switch to MySQL-5.1 alleviated those at all?

--HM

[1] -
http://lists.wikimedia.org/pipermail/wikitech-l/2009-November/046001.html

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: mysql changes

2009-12-12T16:48:45Z

On Sat, Dec 12, 2009 at 7:33 PM, Domas Mituzas <midom.lists@...> wrote:
> Hmmm, I think that replication can be defined as (from)master(to)slave, so you can replicate from lower versions to higher versions. Not opposite, unless you're using trainwreck ;-)

Er, right, I obviously meant "You can't replicate from higher versions
to lower versions."

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: mysql changes

2009-12-12T16:37:00Z

Hi!

> ...if it has v4 slaves

I know how v4 slave will break. It will break entirely. Binlogs are not compatible.

> Perhaps someone that wants to be able to switch the master back to a mysql4.

Such steps are one direction, usually :) And why would anyone, except me, want to run on mysql4? :-)

By the way, we're having huge stalls now, buffer pool dirty page flushing sometimes starves when is not able to find any dirty pages to flush.

> Was that a wild idea that if there's no 'upgraded to mysql 5.1' entry in
> the log, the old mysql4 master was kept as a slave?

Well, it is useless pretty soon, as applying binlogs is pain, and it gets old very very very fast.

> mysql replicates the sql statements, so I wouldn't be surprised if a
> mysql5 which only received mysql4 compatible statements worked as master
> for mysql4s.

Yes, it is possible, if there's binlog conversion going on, or external replication agent is used (such as trainwreck).

Domas
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: mysql changes

2009-12-12T16:33:27Z

Yo,

> You can't replicate from lower versions to higher versions, in
> general. Only the reverse is safe. So if the master is 5.1, we can
> assume all the slaves are probably (at least) 5.1.

Hmmm, I think that replication can be defined as (from)master(to)slave, so you can replicate from lower versions to higher versions. Not opposite, unless you're using trainwreck ;-)

Domas
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: mysql changes

2009-12-12T16:07:21Z

Domas Mituzas wrote:
> Hi!
>
>> Don't be so eager.
>>
>> Replication will break in funny ways when a 5.1 master will get a v5
>> event.
>
> ??? what do you mean?

...if it has v4 slaves

>> In the best scenario, all mysql4 slaves stop replicating at once,
>> overloading the 5.1 ones and leading the wiki to a halt.
>
> We can run the site off a single slave, usually (well, probably need to send watchlists to blackhole for a while, but.. ;-)
>
>> Domas ends up
>> removing the offending statement with a hex editor or upgrading all the
>> slaves in a hurry.
>
> Who in the sane mind upgrades masters before slaves?

Perhaps someone that wants to be able to switch the master back to a mysql4.

Was that a wild idea that if there's no 'upgraded to mysql 5.1' entry in
the log, the old mysql4 master was kept as a slave?

Note that I wasn't assuming a setup with the master being the only mysql
5.1, just that still were _some_ v4 slaves.
werna has already cleared up that all slaves are also 5.1 so all is good.

>> Do we have at least one 5.1 slave per 5.1-mastered pool ?
>
> Did we have ever any non-5.1 slave per 5.1-mastered pool?
>
> Domas

When answering, I looked at wikitech but was unable to find the mysql
version each slave was running. Consider it a bug #1 issue :)

Aryeh wrote:
> You can't replicate from lower versions to higher versions, in
> general. Only the reverse is safe. So if the master is 5.1, we can
> assume all the slaves are probably (at least) 5.1.

mysql replicates the sql statements, so I wouldn't be surprised if a
mysql5 which only received mysql4 compatible statements worked as master
for mysql4s.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: mysql changes

2009-12-12T15:47:58Z

On Fri, Dec 11, 2009 at 5:18 PM, Platonides <Platonides@...> wrote:
> Do we have at least one 5.1 slave per 5.1-mastered pool ?

You can't replicate from lower versions to higher versions, in
general. Only the reverse is safe. So if the master is 5.1, we can
assume all the slaves are probably (at least) 5.1.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: mysql changes

2009-12-12T14:30:23Z

Hi!

> Don't be so eager.
>
> Replication will break in funny ways when a 5.1 master will get a v5
> event.

??? what do you mean?

> In the best scenario, all mysql4 slaves stop replicating at once,
> overloading the 5.1 ones and leading the wiki to a halt.

We can run the site off a single slave, usually (well, probably need to send watchlists to blackhole for a while, but.. ;-)

> Domas ends up
> removing the offending statement with a hex editor or upgrading all the
> slaves in a hurry.

Who in the sane mind upgrades masters before slaves?

> Do we have at least one 5.1 slave per 5.1-mastered pool ?

Did we have ever any non-5.1 slave per 5.1-mastered pool?

Domas
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: mysql changes

2009-12-12T08:29:39Z

Andrew Garrett wrote:
> Domas said that each cluster was "fully 5.1", meaning all masters and
> slaves are running 5.1.

Domas only said in its message that s2 was fully 5.1.
Without further data, the old master and probably some slaves would
continue replicating s1.
Thanks for the information.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Extracting main titles from enwiki-latest-all-titles-in-ns0.gz

2009-12-12T08:24:54Z

> 2009/12/11 Behrang Saeedzadeh <behrangsa@...>:
>> Hi,
>>
>> I have downloaded enwiki-latest-all-titles-in-ns0.gz and I want to extract
>> main titles and store them in another file. For example, some titles have
>> meta information (e.g. disambiguation etc.) and I want these to be removed.
>> Can I remove all the text between parentheses from the titles to achieve
>> this?
>>
>> Also some titles start with the "!" character. and some are enclosed between
>> two or three of them such as !Adiso_Amigos!. What is the purpose of "!" in
>> such cases? Also why some titles are enclosed between two double quotes such
>> as "400_Years_of_Telescope"?
>>
>> Finally, is there a document describing all these conventions?
>>
>> P.S: Is this the right place to ask such questions?
>>
>> Cheers,
>> Behrang Saeedzadeh

They are already complete.
The titles which contain "(disambiguation)" is because there is a page
whose title say that.
And there is also a page whose name is "!Adios_Amigos!" (in fact a
redirect to ¡Adios_Amigos!).

If you want to filter out redirects and disambiguation pages, just
enwiki-latest-all-titles-in-ns0 is not enough.
On the other hand, enwiki-latest-all-titles-in-ns0 has already filtered
all non-article titles, like templates, talks, user pages, pages about
wikipedia...

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Visualization of Wikipedia content

2009-12-12T08:21:34Z

I'm not sure I'd like that (dreamed) library. Seeing the right side is
nice, but it doesn't fit with the left one.

Lars, wikibooks uses a square filled in several levels to denote the
book completeness see the top right section (Niveles de desarrollo) at
http://es.wikibooks.org/wiki/Portada

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [WikiEN-l] Extracting main titles from enwiki-latest-all-titles-in-ns0.gz

2009-12-12T07:45:41Z

Hi,

On Sat, Dec 12, 2009 at 4:35 PM, David Gerard <dgerard@...> wrote:

>
> 2009/12/11 Behrang Saeedzadeh <behrangsa@...>:
> > Hi,
> >
> > I have downloaded enwiki-latest-all-titles-in-ns0.gz and I want to
> extract
> > main titles and store them in another file. For example, some titles have
> > meta information (e.g. disambiguation etc.) and I want these to be
> removed.
> > Can I remove all the text between parentheses from the titles to achieve
> > this?
> >
>

You have to parse it by hand.

> > Also some titles start with the "!" character. and some are enclosed
> between
> > two or three of them such as !Adiso_Amigos!. What is the purpose of "!"
> in
> > such cases?

It's part of the topic's name (in case of <
http://en.wikipedia.org/wiki/%C2%A1Adios_Amigos!>, the band's name). The
reverse exclamation mark is part of the Spanish language.

> > Also why some titles are enclosed between two double quotes such
> > as "400_Years_of_Telescope"?
>
Same case: The " are part of the topic's name (e.g. <
http://en.wikipedia.org/wiki/%22Weird_Al%22_Yankovic>).

Marco

PS: Next time, please do correct copy&paste so people have a chance to see
what you want. Both your supplied examples had to be corrected, the second
one was missing a "the": <http://en.wikipedia.org/wiki/
"400_Years_of_the_Telescope">

--
VMSoft GbR
Nabburger Str. 15
81737 München
Geschäftsführer: Marco Schuster, Volker Hemmert
http://vmsoft-gbr.de
_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [WikiEN-l] Extracting main titles from enwiki-latest-all-titles-in-ns0.gz

2009-12-12T07:35:47Z

This is probably not the right place, you would want wikitech-l (where
I've cc'ed this reply).

- d.

2009/12/11 Behrang Saeedzadeh <behrangsa@...>:

> Hi,
>
> I have downloaded enwiki-latest-all-titles-in-ns0.gz and I want to extract
> main titles and store them in another file. For example, some titles have
> meta information (e.g. disambiguation etc.) and I want these to be removed.
> Can I remove all the text between parentheses from the titles to achieve
> this?
>
> Also some titles start with the "!" character. and some are enclosed between
> two or three of them such as !Adiso_Amigos!. What is the purpose of "!" in
> such cases? Also why some titles are enclosed between two double quotes such
> as "400_Years_of_Telescope"?
>
> Finally, is there a document describing all these conventions?
>
> P.S: Is this the right place to ask such questions?
>
> Cheers,
> Behrang Saeedzadeh
> -------------------------------
> http://my.opera.com/behrangsa
> http://twitter.com/behrangsa
> _______________________________________________
> WikiEN-l mailing list
> WikiEN-l@...
> To unsubscribe from this mailing list, visit:
> https://lists.wikimedia.org/mailman/listinfo/wikien-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: mysql changes

2009-12-12T02:59:25Z

On 11/12/09 10:18 PM, Platonides wrote:

> Dmitriy Sintsov wrote:
>
>> Good to hear!
>> My extension was broken with 4.0.40 (nested selects, perhaps). It was
>> very outdated version - any linux distribution has a 5.x for a long
>> time. Probably MediaWiki core will drop the compatibility, too.
>> Dmitriy
>>
> Don't be so eager.
>
> Replication will break in funny ways when a 5.1 master will get a v5
> event. In the best scenario, all mysql4 slaves stop replicating at once,
> overloading the 5.1 ones and leading the wiki to a halt. Domas ends up
> removing the offending statement with a hex editor or upgrading all the
> slaves in a hurry.
>
> Do we have at least one 5.1 slave per 5.1-mastered pool ?
>

Domas said that each cluster was "fully 5.1", meaning all masters and
slaves are running 5.1.

--
Andrew Garrett
agarrett@...
http://werdn.us

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Visualization of Wikipedia content

2009-12-12T02:22:19Z

Erik Zachte wrote:
> Lars, I just saw this amazing bookshelf, from your home town even, needs to
> be shared.
>
> http://features.cgsociety.org/stories/2009_05/2009_05_stockholmlibrary/18-final-image.jpg
>
Despite the photo realism, that is a rendering of one
architect's dream of what the future extension of the
Stockholm Public Library might look like. The old
library was built in 1928 and is much loved, despite
being somewhat unmodern and limited. After it was
found that the new extension would become very
expensive, and risk to modify the old loved building,
the planning of the extension was cancelled earlier
this fall.

The text to that picture is here,
http://features.cgsociety.org/story_custom.php?story_id=5097

The existing library is here,
http://en.wikipedia.org/wiki/Stockholm_Public_Library

And this is the real, existing interior,
http://commons.wikimedia.org/wiki/File:Stadsbiblioteket_2008e.jpg

--
Lars Aronsson (lars@...)
Aronsson Datateknik - http://aronsson.se/

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Visualization of Wikipedia content

2009-12-11T18:33:33Z

Ah image link broken, here it is in shorter form:

http://tinyurl.com/y87aosk

Erik Zachte

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: Visualization of Wikipedia content

2009-12-11T18:29:58Z

Lars:
> The various (humorous) Size-of-Wikipedia pictures
> come to mind,
> http://commons.wikimedia.org/wiki/File:Size_of_English_Wikipedia_broken
> _down.png
> http://commons.wikimedia.org/wiki/File:Size_of_Wikipedia_broken_down.sv
> g
>

Lars, I just saw this amazing bookshelf, from your home town even, needs to
be shared.

http://features.cgsociety.org/stories/2009_05/2009_05_stockholmlibrary/18-fi
nal-image.jpg

Cheers, Erik Zachte

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: mysql changes

2009-12-11T14:18:07Z

Dmitriy Sintsov wrote:
> Good to hear!
> My extension was broken with 4.0.40 (nested selects, perhaps). It was
> very outdated version - any linux distribution has a 5.x for a long
> time. Probably MediaWiki core will drop the compatibility, too.
> Dmitriy

Don't be so eager.

Replication will break in funny ways when a 5.1 master will get a v5
event. In the best scenario, all mysql4 slaves stop replicating at once,
overloading the 5.1 ones and leading the wiki to a halt. Domas ends up
removing the offending statement with a hex editor or upgrading all the
slaves in a hurry.

Do we have at least one 5.1 slave per 5.1-mastered pool ?

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Visualization of Wikipedia content

2009-12-11T13:28:35Z

What ideas have been tried or suggested in using
visualization as an aid in Wikipedia? For example,
could the articles of a category be summarized as
an image that would provide new insights to the
user, either encyclopedia reader or writer?

When scanned books are proofread in Project Runeberg
(runeberg.org, a website similar to Wikisource),
a horizontal bar graph is used to indicate which
pages have been proofread or not, with one pixel
for each page of the book (so 500 pixels wide for
a 500 page book), with green pixels for proofread
pages and red pixels for not yet proofread pages.
Here is an example of a partially proofread 2 volume
work, at the bottom, http://runeberg.org/bokogbib/

Could something similar be useful in Wikipedia?
Perhaps for overview of maintenance categories?
Are there any tools at the toolserver that do
such things? Has that been tried?

The various (humorous) Size-of-Wikipedia pictures
come to mind,
http://commons.wikimedia.org/wiki/File:Size_of_English_Wikipedia_broken_down.png
http://commons.wikimedia.org/wiki/File:Size_of_Wikipedia_broken_down.svg

--
Lars Aronsson (lars@...)
Aronsson Datateknik - http://aronsson.se

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l