Quick analysis of the Python dist-packages transition

[Josselin Mouette]

Hi,

starting from Python 2.6, the Debian packages look for modules in a
different directory: /usr/lib/python2.6/dist-packages instead
of /usr/lib/python2.X/site-packages. This is handled transparently by
python-central and python-support, but at install time, distutils (the
thingy behind “python setup.py”) installs modules in another directory
by default, and the packaging has to cope with it.

Therefore, a number of packages have to be fixed before they can work
with python2.6. Practically speaking, this is the only thing that
prevents python2.6 from entering unstable. This is a first attempt at
listing packages needing to be fixed.

There are 1396 source packages using python-central or python-support in
Debian. (The analysis excludes packages not using them since they are
already broken.)

      * 505 of these packages do not use distutils and should not be
        affected, still shipping files to site-packages/. However,
        according to Scott Kimmermann (who handled parts of this
        transition in Ubuntu), python-central does not look for modules
        in /usr/lib/python2.6/site-packages, so most modules using it
        are broken. If this is the case, python-central needs to be
        NMUed to handle such packages.
      * 73 packages don’t use the shipped setup.py and use a
        Debian-specific installation system (e.g. to install modules in
        a private directory).
      * 818 packages use distutils/setuptools for installation.


I - CDBS: 310 packages

CDBS needs updating to work with python2.6. A patch was proposed by
Martin Pitt in #537373 and the maintainers have already agreed for a
NMU, so it’s just a matter of uploading it. In the meantime, Piotr
Ożarowski proposed another idea (setting --install-lib instead of
--install-layout) which looks much cleaner, so we’ll probably use that
approach instead. In all cases this will be done soon.

      * 269 CDBS packages should not be affected.
      * 41 packages fiddle with site-packages. If either Martin’s or
        Piotr’s approach is used, they won’t need updating.


II - DH: 143 packages

Debhelper has already been updated so that dh uses --install-layout=deb.

      * 141 DH packages should already work.
      * 2 packages fiddle with site-packages and need updating.


III - Debhelper: 438 packages

      * 52 packages already use --install-layout=deb and don’t play with
        site-packages.
      * 246 packages don’t, but should work as well provided that we
        ensure python-central is fixed.
      * 73 packages fiddle with site-packages and need updating.


Overall summary:

      * CDBS needs to be updated (should be done in a week at most).
      * python-central needs a NMU to
        handle /usr/lib/python2.6/site-packages as a source directory.
      * 75 Python packages need to be updated, the dd-list is attached.

If there are no objections, I will submit a MBF for those 75 packages in
a few days.

Cheers,
--
 .''`.      Josselin Mouette
: :' :
`. `'   “I recommend you to learn English in hope that you in
  `-     future understand things”  -- Jörg Schilling

Daniel Leidert (dale) <daniel.leidert@...>
   pymol (U)

Adam Cécile (Le_Vert) <gandalf@...>
   hellanzb

Nicolas FRANCOIS (Nekral) <nicolas.francois@...>
   translate-toolkit

Marco Presi (Zufus) <zufus@...>
   matplotlib (U)

Francesc Altet <faltet@...>
   pytables

Kumar Appaiah <akumar@...>
   harvestman (U)
   python-goopy (U)

Nacho Barrientos Arias <nacho@...>
   rdflib

Ernesto Nadir Crespo Avila <ecrespo@...>
   pythoncard
   pyx

Michael Banck <mbanck@...>
   pymol (U)

Julien BLACHE <jblache@...>
   eikazo

Jérémy Bobbio <lunar@...>
   python-clientform (U)
   python-mechanize (U)

W. Martin Borgert <debacle@...>
   trac (U)

A. Maitland Bottoms <bottoms@...>
   mayavi

Giacomo Catenazzi <cate@...>
   bauble

Ondrej Certik <ondrej@...>
   matplotlib (U)

Jesus Climent <jesus.climent@...>
   trac (U)

Kevin Coyner <kcoyner@...>
   kodos

LI Daobing <lidaobing@...>
   pymol (U)

Debian Bazaar Maintainers <pkg-bazaar-maint@...>
   bzr-builddeb

Debian Games Team <pkg-games-devel@...>
   libtpclient-py

Debian Python Modules Team <python-modules-team@...>
   constraint
   ctypes (U)
   inotifyx (U)
   logilab-constraint
   matplotlib
   pastedeploy (U)
   pastewebkit (U)
   pyscard (U)
   python-docutils
   python-goopy
   python-kinterbasdb
   python-memcache (U)
   python-pyglew
   python-pytils (U)
   python-reportlab (U)
   sqlobject (U)
   webhelpers (U)

Debian X Strike Force <debian-x@...>
   ccsm

Debian/Ubuntu Zope Team <pkg-zope-developers@...>
   python-clientform
   python-mechanize
   python-tz
   zope.interface

Debichem Team <debichem-devel@...>
   pymol

Barry deFreese <bdefreese@...>
   libtpclient-py (U)

Cédric Delfosse <cedric@...>
   gaphor

Benjamin Drung <bdrung@...>
   matplotlib (U)

Alexandre Fayolle <afayolle@...>
   constraint (U)
   logilab-constraint (U)
   matplotlib (U)
   pyqonsole
   xmldiff

Sean Finney <seanius@...>
   ccsm (U)

Gustavo Franco <stratus@...>
   gdebi
   gdebi (U)

John Goerzen <jgoerzen@...>
   pygopherd

Debian QA Group <packages@...>
   kphotobymail
   synopsis

Mikhail Gusarov <dottedmag@...>
   python-pytils

Anders Hammarquist <iko@...>
   python-meld3
   supervisor

Magnus Holmgren <holmgren@...>
   pyscrabble

Adam C. Powell, IV <hazelsct@...>
   pysparse

Michael Janssen <jamuraa@...>
   bittorrent

Matthias Klose <doko@...>
   gadfly
   lxml
   python-gnuplot
   python-imaging
   python-reportlab
   python-scientific
   python-setuptools
   slides
   translate-toolkit (U)

martin f. krafft <madduck@...>
   python-docutils (U)

Jonny Lamb <jonny@...>
   synce-kpm
   synce-sync-engine

Chris Lawrence <lawrencc@...>
   foomatic-gui

lirc Maintainer Team <pkg-lirc-maint@...>
   pylirc

Luis Matos <gass@...>
   trac (U)

Simon McVittie <smcv@...>
   python-docutils (U)

Ricardo Javier Cardenes Medina <rcardenes@...>
   fnorb

Loic Minier <lool@...>
   pylirc (U)

Oleksandr Moskalenko <malex@...>
   webhelpers

Piotr OÅŒarowski <piotr@...>
   emma (U)
   pastedeploy
   pastewebkit

Anand Pillai <abpillai@...>
   harvestman (U)

Python Applications Packaging Team <python-apps-team@...>
   emma
   harvestman
   trac

Arnaud Quette <aquette@...>
   pylirc (U)

Ganesan Rajagopal <rganesan@...>
   clientcookie
   ctypes

Marco Rodrigues <gothicx@...>
   emma (U)

Joel Rosdahl <joel@...>
   egenix-mx-base
   python-pysqlite2

Alexandre Rossi <alexandre.rossi@...>
   deejayd

Ludovic Rousseau <rousseau@...>
   pyscard

Andres Salomon <dilinger@...>
   trac (U)

Otavio Salvador <otavio@...>
   trac (U)

Ritesh Raj Sarraf <rrs@...>
   inotifyx

Christopher Schmidt <crschmidt@...>
   python-memcache
   tilecache

Tristan Seligmann <mithrandi@...>
   quodlibet

Gustavo Noronha Silva <kov@...>
   sqlobject (U)

Christine Spang <christine@...>
   quodlibet (U)

Igor Stroh <jenner@...>
   python-reportlab (U)

Brian Sutherland <jinty@...>
   python-clientform (U)
   python-mechanize (U)
   python-pullparser
   python-tz (U)
   smart-notifier

Brian Sutherland <brian@...>
   zope.interface (U)

Reinhard Tartler <siretart@...>
   bzr-builddeb (U)

Sandro Tosi <morph@...>
   logilab-constraint (U)
   matplotlib (U)
   python-kinterbasdb (U)
   python-pyglew (U)

Fabio Tranchitella <kobold@...>
   python-clientform (U)
   python-mechanize (U)
   python-tz (U)
   sqlobject
   zope.interface (U)

Matthias Urlichs <smurf@...>
   yapps2

Matej Vela <vela@...>
   python-ldap

Michael Vogt <mvo@...>
   gdebi (U)

David Watson <dwatson@...>
   pybridge

James Westby <jw+debian@...>
   bzr-builddeb (U)

Paul Wise <pabs@...>
   fonttools

Alexander Zangerl <az@...>
   duplicity

[Mikhail Gusarov-3]

Twas brillig at 21:18:16 18.09.2009 UTC+02 when joss@... did gyre and gimble:

 JM> Mikhail Gusarov <dottedmag@...>
 JM>    python-pytils

Fixed and passed to Piotr for sponsoring.

--
  http://fossarchy.blogspot.com/

[Josselin Mouette]

Le vendredi 18 septembre 2009 à 21:18 +0200, Josselin Mouette a écrit :
> * 246 packages don’t, but should work as well provided that we
>         ensure python-central is fixed.

I forgot to explain how exactly it needs to be fixed.

> * python-central needs a NMU to
>         handle /usr/lib/python2.6/site-packages as a source directory.

It needs to support /usr/local/lib/python2.6/{dist,site}-packages as
well. Otherwise, 83 more packages will have to be changed.

--
 .''`.      Josselin Mouette
: :' :
`. `'   “I recommend you to learn English in hope that you in
  `-     future understand things”  -- Jörg Schilling

[Ben Finney-5]

Josselin Mouette <joss@...> writes:

> Therefore, a number of packages have to be fixed before they can work
> with python2.6. Practically speaking, this is the only thing that
> prevents python2.6 from entering unstable. This is a first attempt at
> listing packages needing to be fixed.

Thank you for this work, and for communicating the results in these
forums. This also allows us to have a better idea how far Python 2.6 is
From unstable.

--
 \           “A thorough reading and understanding of the Bible is the |
  `\                           surest path to atheism.” —Donald Morgan |
_o__)                                                                  |
Ben Finney

[Raphael Hertzog-3]

On Fri, 18 Sep 2009, Josselin Mouette wrote:
> If there are no objections, I will submit a MBF for those 75 packages in
> a few days.

Go ahead, we have waited too much for python 2.6 already.

Cheers,
--
Raphaël Hertzog


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Alexandre Rossi]

Hi,

>   deejayd

Fixed package is awaiting sponsorship at mentors.debian.net

Alexandre


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Luca Falavigna-4]

Il giorno Sat, 19 Sep 2009 20:20:16 +0200
Alexandre Rossi <alexandre.rossi@...> ha scritto:

> Fixed package is awaiting sponsorship at mentors.debian.net

To speed things up, if you haven't a regular sponsor or if he/she is
busy ATM, please join #debian-python on OFTC and add pointers to your
packages to the channel topic, they will be processed quicker.

Regards,

--
  .''`.
 :  :' :   Luca Falavigna <dktrkranz@...>
 `.  `'
   `-

[Steve Langasek]

On Fri, Sep 18, 2009 at 09:18:16PM +0200, Josselin Mouette wrote:
>       * 505 of these packages do not use distutils and should not be
>         affected, still shipping files to site-packages/. However,
>         according to Scott Kimmermann (who handled parts of this
>         transition in Ubuntu), python-central does not look for modules
>         in /usr/lib/python2.6/site-packages, so most modules using it
>         are broken.

What do you mean, "look for" modules there?  Are you proposing that these
packages ship files under /usr/lib/python2.6/site-packages, or that
python-central find modules under this directory at build-time and move them
to /usr/lib/python2.6/dist-packages?

AIUI, the former would be contrary to upstream's wishes regarding the
organization of distro-provided modules.

>         If this is the case, python-central needs to be NMUed to handle
>         such packages.

There is no bug filed against python-central for this.  An NMU would be out
of order.

--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@...                                     vorlon@...

[Piotr Ożarowski-2]

[Steve Langasek, 2009-09-20]

> On Fri, Sep 18, 2009 at 09:18:16PM +0200, Josselin Mouette wrote:
> >       * 505 of these packages do not use distutils and should not be
> >         affected, still shipping files to site-packages/. However,
> >         according to Scott Kimmermann (who handled parts of this
> >         transition in Ubuntu), python-central does not look for modules
> >         in /usr/lib/python2.6/site-packages, so most modules using it
> >         are broken.
>
> What do you mean, "look for" modules there?  Are you proposing that these
> packages ship files under /usr/lib/python2.6/site-packages, or that
> python-central find modules under this directory at build-time and move them
> to /usr/lib/python2.6/dist-packages?

no, move them to /usr/share/pyshared, just like it does with
dist-packages

> There is no bug filed against python-central for this.

I will file a bug in a minute (not that it will change much)
--
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[Josselin Mouette]

Le vendredi 18 septembre 2009 à 21:18 +0200, Josselin Mouette a écrit :
> If there are no objections, I will submit a MBF for those 75 packages in
> a few days.

Done, omitting a reported false positive and a few packages fixed in the
meantime.

http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-python@...;tag=python2.6

--
 .''`.      Josselin Mouette
: :' :
`. `'   “I recommend you to learn English in hope that you in
  `-     future understand things”  -- Jörg Schilling

[Stefano Zacchiroli]

On Tue, Sep 22, 2009 at 12:36:09PM +0200, Josselin Mouette wrote:
> Done, omitting a reported false positive and a few packages fixed in the
> meantime.
> http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-python@...;tag=python2.6

Thanks.

Question: why the severity is only "important"?

Shouldn't we aim at uploading python 2.6 to unstable really soon now? In
that respect, if I understand the problems you mentioned in the first
post correctly, those bugs should be better made RC.

Cheers.

--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..|  .  |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...........| ..: |.... Je dis tu à tous ceux que j'aime

[Emilio Pozuelo Monfort-4]

Stefano Zacchiroli wrote:

> On Tue, Sep 22, 2009 at 12:36:09PM +0200, Josselin Mouette wrote:
>> Done, omitting a reported false positive and a few packages fixed in the
>> meantime.
>> http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-python@...;tag=python2.6
>
> Thanks.
>
> Question: why the severity is only "important"?
>
> Shouldn't we aim at uploading python 2.6 to unstable really soon now? In
> that respect, if I understand the problems you mentioned in the first
> post correctly, those bugs should be better made RC.

I guess they will be RC once python2.6 is uploaded to unstable (and made the
default) since they will FTBFS. That won't impede python2.6 being uploaded soon.

Cheers,
Emilio