new dh_python proposal
|
View:
New views
9 Messages
—
Rating Filter:
Alert me
|
 |
new dh_python proposal
by Piotr Ożarowski-2
::
Rate this Message:
My attempt to merge python-central and python-support failed few months ago, so
here's another proposal to improve the Python situation in Debian. It's nothing new actually, just a compilation of previous Matthias' and Joss' ideas and few of my own. The main idea is to ship symlinks (not .pyc files!) in binary packages as Matthias proposed back in February and thus get rid of both helper tools (most problems with current tools occur at install / upgrade time). I want to keep it as simple as possible and let maintainers customize it in maintainer or rtinstall/rtupdate/rtremove scripts (if really needed). Advantages: * a lot less opportunities to break a system while installing / upgrading Python packages, * no more problems with packages that provide the same namespace and use different helper tool, * Python modules available out of the box (think about daemons), * no more "which tool should I use?" or "how do they differ?" questions ;-) The main disadvantage of this approach is that architecture independent packages will have to be rebuilt once new Python version will be added to the supported ones (I think we can avoid a rebuild if a version is removed from the supported ones). Luk (our release manager) told me that binNMUs for arch:all packages will be possible in Debian at some point, but it's a matter or months or even years, so for now - manual NMUs will be needed. I think it will be easy to detect which packages need one basing on Contents file and Depends field. Another disadvantage is the fact that pysupport's namespace feature will not be supported. If we want to keep it simple and let dpkg know about all the files - new *-common package with all common __init__.py (and other if needed) files will have to be provided, just like we did before pysupport 0.7. Otherwise removing .pyc files will not be a trivial thing to do. Note that there are lots of problems with this feature anyway (upstreams tend to use site-packages for things that don't belong there), so removing it will improve the situation at the end, IMHO. I'll try to make dh_python a drop in replacement for dh_pycentral and dh_pysupport (i.e. tools used at build time) for most packages, but I don't consider it to be a main goal. Some packages will have to be manually updated (f.e. the ones that use pysupport's namespace feature). I want the tool to be team maintained, I didn't decide yet if new Alioth project should be created or if we should use DPMT or PAPT repo for this, though. Short overview of how things will work with new dh_python: ========================================================== build time: ^^^^^^^^^^^ * files installed into the standard location[1] or into private directory * dh_python will: - move .py files from SL[1] into /usr/share/py{,3}shared, - move .so files to /usr/lib/pythonX.Y/*-packages if /usr/local was used, - create symlinks for all py{,3}shared content in /usr/lib/pythonX.Y/*-packages *if* files are the same for all Python versions in SL[1] (if not, leave original files) where X.Y are requested Python versions, - create simple maintainer scripts with pycompile and pyclean commands and (if needed) rtupdate script (for private directories that use default Python version). Private modules that cannot be used with default Python version will get additional pycompile command with all needed arguments, like minimum required Python version or hardcoded one in both, maintainer script and rtupdate one [1] SL = standard distutils/setuptools location: /usr/{,local}/lib/pythonX.Y/{site,dist}-packages/ installation time: ^^^^^^^^^^^^^^^^^^ * maintainer script should bytecompile pyc files for all provided symlinks / private directories if given Python version is installed * user installs new pythonX.Y package: - bytecompile related symlinks (pycompile -V X.Y) * user removes pythonX.Y package: - remove all pythonX.Y's .pyc files (pyclean -V X.Y) - all packages with private directories that use this Python version will be removed by dpkg (Depends field will do its job) * default Python version changes: - rtupdate scripts for packages with private modules that support it (i.e. the ones without hardcoded Python version and without private extensions) invoked examples: ========= example 1 - public modules/extensions only ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ $ python2.4 ./setup.py install --root=debian/python-foo/ debian/python-foo/usr/lib/python2.4/site-packages/foo/__init__.py debian/python-foo/usr/lib/python2.4/site-packages/foo/_foo.so debian/python-foo/usr/lib/python2.4/site-packages/foo/bar.py debian/python-foo/usr/lib/python2.4/site-packages/spam.py $ python2.5 ./setup.py install --root=debian/python-foo/ debian/python-foo/usr/lib/python2.5/site-packages/foo/__init__.py debian/python-foo/usr/lib/python2.5/site-packages/foo/_foo.so debian/python-foo/usr/lib/python2.5/site-packages/foo/bar.py debian/python-foo/usr/lib/python2.5/site-packages/spam.py $ python2.6 ./setup.py install --root=debian/python-foo/ debian/python-foo/usr/local/lib/python2.6/dist-packages/foo/__init__.py debian/python-foo/usr/local/lib/python2.6/dist-packages/foo/_foo.so debian/python-foo/usr/local/lib/python2.6/dist-packages/foo/bar.py debian/python-foo/usr/local/lib/python2.6/dist-packages/spam.py $ python3.1 ./3.x/setup.py install --root=debian/python3-foo/ debian/python3-foo/usr/lib/python3.1/dist-packages/foo/__init__.py debian/python3-foo/usr/lib/python3.1/dist-packages/foo/_foo.so debian/python3-foo/usr/lib/python3.1/dist-packages/foo/bar.py $ dh_python # gets minimum required Python version from debian/pyversions or from # XS-Python-Version or (this part is new) from Build-Depends(-Indep); # ignores private dirs debian/python-foo/usr/lib/python2.4/site-packages/foo/_foo.so debian/python-foo/usr/lib/python2.4/site-packages/foo/__init__.py -> /usr/share/pyshared/foo/__init__.py debian/python-foo/usr/lib/python2.4/site-packages/foo/bar.py -> /usr/share/pyshared/foo/bar.py debian/python-foo/usr/lib/python2.4/site-packages/foo/spam.py -> /usr/share/pyshared/foo/spam.py debian/python-foo/usr/lib/python2.5/site-packages/foo/_foo.so debian/python-foo/usr/lib/python2.5/site-packages/foo/__init__.py -> /usr/share/pyshared/foo/__init__.py debian/python-foo/usr/lib/python2.5/site-packages/foo/bar.py -> /usr/share/pyshared/foo/bar.py debian/python-foo/usr/lib/python2.5/site-packages/foo/spam.py -> /usr/share/pyshared/foo/spam.py debian/python-foo/usr/lib/python2.6/dist-packages/foo/_foo.so debian/python-foo/usr/lib/python2.6/dist-packages/foo/__init__.py -> /usr/share/pyshared/foo/__init__.py debian/python-foo/usr/lib/python2.6/dist-packages/foo/bar.py -> /usr/share/pyshared/foo/bar.py debian/python-foo/usr/lib/python2.6/dist-packages/foo/spam.py # not a symlink as it differes from Python2.4 and 2.5's version debian/python-foo/usr/share/pyshared/foo/__init__.py debian/python-foo/usr/share/pyshared/foo/bar.py debian/python-foo/usr/share/pyshared/spam.py debian/python3-foo/usr/lib/python3.1/dist-packages/foo/_foo.so debian/python3-foo/usr/lib/python3.1/dist-packages/foo/__init__.py -> /usr/share/py3shared/foo/__init__.py debian/python3-foo/usr/lib/python3.1/dist-packages/foo/bar.py -> /usr/share/py3shared/foo/bar.py debian/python3-foo/usr/lib/python3.1/dist-packages/foo/spam.py -> /usr/share/py3shared/foo/spam.py debian/python3-foo/usr/share/py3shared/foo/__init__.py debian/python3-foo/usr/share/py3shared/foo/bar.py debian/python3-foo/usr/share/py3shared/spam.py $ grep foo debian/python-foo.postinst pycompile foo spam $ grep foo debian/python3-foo.postinst py3compile foo spam $ grep foo debian/python-foo.prerm pyclean foo spam $ grep foo debian/python3-foo.prerm py3clean foo spam example 2 - private modules/scripts ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # bar uses default Python version $ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir1/ --install-scripts=/usr/share/privatedir1 debian/foo/usr/share/privatedir1/bar/__init__.py debian/foo/usr/share/privatedir1/run.py # spam uses Python 2.4 $ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir2/ --install-scripts=/usr/share/privatedir2 debian/foo/usr/share/privatedir2/spam/__init__.py debian/foo/usr/share/privatedir2/run.py # egg uses Python >= 2.6 $ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir3/ --install-scripts=/usr/share/privatedir3 debian/foo/usr/lib/privatedir3/egg/__init__.py debian/foo/usr/lib/privatedir3/run.py # baz uses Python >= 2.5 and has private extension $ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir4/ --install-scripts=/usr/share/privatedir4 debian/foo/usr/lib/privatedir4/baz/__init__.py debian/foo/usr/lib/privatedir4/baz/baz.so debian/foo/usr/lib/privatedir4/run.py $ dh_python /usr/share/privatedir1 # ignores SL; reads debian/pyversions or XS-Python-Version as -V is not given debian/foo/usr/share/privatedir1/bar/__init__.py debian/foo/usr/share/privatedir1/run.py # Python not hardcoded in shebang and -V not used so recompile once default # Python version will change, minimum required version hardcoded in the script debian/foo/usr/share/python/runtime.d/python-foo.rtupdate $ grep python:Depends debian/python-foo.substvars python:Depends=python $ dh_python /usr/share/privatedir2 -V 2.4 debian/foo/usr/share/privatedir2/bar/__init__.py debian/foo/usr/share/privatedir2/run.py # note that rtupdate script is not touched, python2.4 is added to the Depends # instead and maintainer script contains hardcoded python2.4 debian/foo/usr/share/python/data/python-foo $ grep python:Depends debian/python-foo.substvars python:Depends=python, python2.4 $ grep privatedir debian/python-foo.postinst pycompile /usr/share/privatedir1 pycompile /usr/share/privatedir2 -V 2.4 $ grep privatedir debian/python-foo.prerm pyclean /usr/share/privatedir1 /usr/share/privatedir2 $ dh_python /usr/share/privatedir3 -V 2.6+ debian/foo/usr/share/privatedir3/egg/__init__.py debian/foo/usr/share/privatedir3/run.py debian/foo/usr/share/python/runtime.d/python-foo.rtupdate $ grep python:Depends debian/python-foo.substvars python:Depends=python, python2.4, python (>=2.6) | python2.6 $ grep privatedir debian/python-foo.postinst pycompile /usr/share/privatedir1 pycompile /usr/share/privatedir2 -V 2.4 pycompile /usr/share/privatedir3 -V 2.6+ $ grep privatedir debian/python-foo.prerm pyclean /usr/share/privatedir1 /usr/share/privatedir2 /usr/share/privatedir3 $ dh_python /usr/share/privatedir4 -V 2.5+ # note the "+" here and lack of it later debian/foo/usr/lib/privatedir4/baz/__init__.py debian/foo/usr/lib/privatedir4/baz/baz.so debian/foo/usr/lib/privatedir4/run.py $ grep python:Depends debian/python-foo.substvars python:Depends=python, python2.4, python (>=2.6) | python2.6, python2.5 $ grep privatedir debian/python-foo.postinst pycompile /usr/share/privatedir1 pycompile /usr/share/privatedir2 -V 2.4 pycompile /usr/share/privatedir3 -V 2.6+ pycompile /usr/lib/privatedir4 -V 2.5 # note that there's no "+" here (due to private extension) $ grep privatedir debian/python-foo.prerm pyclean /usr/share/privatedir1 /usr/share/privatedir2 /usr/share/privatedir3 /usr/lib/privatedir4 -- -=[ Piotr Ożarowski ]=- -=[ http://www.ozarowski.pl ]=- |
|
|
Re: new dh_python proposal
by Piotr Ożarowski-2
::
Rate this Message:
Just a follow-up the clear some things up:
* about the -common package (i.e. pysupport namespace issue): - it's not a must, if one package can act as namespace provider, there's no need to provide another one, of course, - being able to list all files provided by packages is something we really want to have * dh_python[1] will try to avoid moving files to pyshared if package supports only one Python version (f.e. the latest one) * py{,3}compile will support -X/--exclude option I maintain one module that uses site-packages to keep templates with .py files inside and although I patched it to move these files outside site-packages, I know that not every maintainer will want to do that, so -X will disable byte compilation of files for given directory/file * about lack of XS-Python-Version and debian/pyversions - if available, both previous places will be used to get minimum/maximum required Python version, it will complicate detection of packages that need binNMU, so I want to drop both of them at some point, - dh/cdbs/dh_python will get minimum/maximum required Python versions from Build-Depends{,-Indep} and/or python-all's Depends. If one will need to build depend on newer version of python{,-all,-dev} (due to some Debian specific changes) - tools will ignore versioned dependencies that include dash sign or use the lower one if two different dependencies are provided (f.e. "Build-Depends: python-all-dev (>= 2.5.8), python-all-dev (>= 2.4)" will be equivalent of "XS-Python-Version: >=2.4") * how to detect which package needs binNMU? I think parsing Build-Depends{,-Indep}, Contents file and Depends header will be enough to detect such packages, i.e. packages that: - Build-Depends{,-Indep} on python-all{,-dev} AND there's no <<X.Y build dependency (where X.Y is the one to be introduced) - Build-Depends on python-dev ("python-dev (>=X.Y) | pythonX.Y-dev" or "python-dev (<<X.Y)" might help to detect some false positives) AND provide private extension (/usr/lib/*/*.so) - Build-Depends{,-Indep} on "python (>=X.Y) | pythonX.Y" and Depend on pythonX.Y (i.e. "python (>=X.Y) | pythonX.Y" will NOT be in Depends) [arch:all packages with hardcoded shebang due to default python not being good enough] - Build-Depends{,-Indep} on "python" or "python (>=X.Y) | pythonX.Y" and there's no rtupdate script in binary package [private modules without hardcoded shebang] will need binNMU once new Python version will be added to the supported ones [1] or "dh_pypython" - I'm still not sure if using the same name is a good idea as I want to write it in Python and maybe at some point let someone rewrite it in Perl so that it could be included in debhelper package. [Piotr Ożarowski, 2009-08-02] > $ python3.1 ./3.x/setup.py install --root=debian/python3-foo/ > debian/python3-foo/usr/lib/python3.1/dist-packages/foo/__init__.py > debian/python3-foo/usr/lib/python3.1/dist-packages/foo/_foo.so > debian/python3-foo/usr/lib/python3.1/dist-packages/foo/bar.py will be debian/python3-foo/usr/local/lib... here by default, of course [...] > # baz uses Python >= 2.5 and has private extension > $ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir4/ --install-scripts=/usr/share/privatedir4 s,/usr/share/,/usr/lib/,g > debian/foo/usr/lib/privatedir4/baz/__init__.py > debian/foo/usr/lib/privatedir4/baz/baz.so > debian/foo/usr/lib/privatedir4/run.py -- -=[ Piotr Ożarowski ]=- -=[ http://www.ozarowski.pl ]=- |
|
|
Re: new dh_python proposal
by anatoly techtonik
::
Rate this Message:
|
|
|
Re: new dh_python proposal
by Kumar Appaiah-2
::
Rate this Message:
|
|
|
Re: new dh_python proposal
by Piotr Ożarowski-2
::
Rate this Message:
[anatoly techtonik, 2009-08-04]
> Before discussing your proposal I would really appreciate if somebody > from insiders could describe situation in Debian Python in all > possible detail including the history of development. I believe that > this piece of work is absolutely necessary, because the problem of > repackaging Python packages for Debian and maintaining integrity > between installed packets, packages and Python versions is too complex > to keep every detail in mind unless you can devote your full time to > the problem. Short? Our goal is to support more than one Python version at the same time if having one Python in the archive is not possible (f.e. when there are applications that still require older Python). We do that currently by shipping Python extensions for all supported Python versions in the same package and byte compiling Python modules at install time for all installed (and supported) Python versions (sharing .py files whenever possible). Read [1] and then [2] if you want more details (yeah, [1] is not official[3] as there were some differences of opinion :-) - I hope it will be easier to finally update the official policy if this proposal will be accepted) [1] http://people.debian.org/~srivasta/manoj-policy/ [2] http://lists.debian.org/debian-python/2009/02/msg00061.html [3] http://www.debian.org/doc/packaging-manuals/python-policy/ > You say that attempt to merge -cental and -support failed, but didn't > mention why. attempt is here[4,5], the reason why it failed is here[6] [4] http://lists.debian.org/debian-python/2009/02/msg00083.html [5] http://lists.debian.org/debian-python/2009/02/msg00099.html [6] http://lists.debian.org/debian-python/2009/03/msg00015.html > You say that current tools have problems that occur at > install/upgrade time, but do not mention these problems explicitly. So > it is impossible to say whatever your ideas solve current problems and > won't add new. see [7] and [8] for some of them. I tried to help many users (including co-workers and #debian-python channel guests) that had ImportError problems I couldn't reproduce on my machine and usually the solution was to reinstall related Python packages (due to missing symlinks or symlinks from older package versions not properly removed at upgrade). I had that problem once on my own system as well, couldn't reproduce it later (and thus couldn't really write a sensible bug report). Right now upgrading from Lenny's pycentral based packages will fail if list of files will be different in Squeeze' version (due to moving to pysupport or dropping "nomove" feature or simply by providing different module names) unless you provide a preinst script that will remove old files. Another issue is the fact that packages with common namespace have to use the same tool (search f.e. for twisted issues in the mailing list or BTS) as otherwise they will end in different directories (i.e. will be reachable from different locations of sys.path) and Python will not recognize that. Yet another issue is the pysupport namespace feature that actually causes more problems than it solves (see f.e. #459446 or #536739) Also, during upgrades, daemons downtime should be as short as possible and since Python modules are right now not available after unpacking debs, it's hard to accomplish it. You also have to do some extra work to be be able to start them during installation. [7] http://bugs.debian.org/python-central [8] http://bugs.debian.org/python-support > Referencing rtinstall/rtupdate/rtremove scripts is cool for a seasoned > Debian developer, but for a Python developer it means nothing. In > other words - not only Python things are obscure for Debian > developers, but Debian stuff is also a mystery for Python masters, so > these things should be explained symmetrically for both communities. > The chances for Debian Python Packaging experts to pop up are > magnitude less if we won't explain the situation in detail. see chapter 6[9] of "Manoj's policy"[1] [9] http://people.debian.org/~srivasta/manoj-policy/x673.html > Now about the proposal (from newcomer's point of view): > dh_python is a shell script -- I have a strong belief that Python > package automation scripts should be written in Python, there is no > need to learn bashes when you program Python - do not expect that > package maintainers will be able to debug their scripts in shell > language. I will write them in Python, old dh_python is written in Perl /bin/sh as dh_python's shebang is not that bad idea, BTW :-) [...] > May I ask a question - is there a difference between installation of > Python Modules and Python Applications? yes, Python applications should be installed in private locations (i.e. outside site-packages) whenever possible (to avoid name conflicts and unnecessary symlink burden). > Does the problem set you are > trying to resolve with this proposal include the difference? Have you > considered distributing Python Applications via virtualenv? How this > proposal handles virtualenv installations? virtualenv is great solution if you want to use different module versions than the ones provided in debs and don't want to break other parts of your system[10] (upstreams usually don't know how important it is to keep the API stable and Python doesn't have sonames equivalent[11])... but security team would hate us if we'd use it in Debian packages (fixing bugs in one package is much easier than fixing the same bug in dozens of them) [10] all these guys that suggest "sudo ez_install ..." should be forced to work as sysadmins until they'll apologize (a week will probably be enough ;) [11] you can rename the module of course (like jinja -> jinja2), but only one upstream author did that in packages that I maintain - thanks Armin!). Search for python-cherrypy{,3} issues for more details. > In conclusion my opinion is that problem set is not defined well > enough to propose a solution or estimate if it will be effective both > for current flow and for future ideas. I would start from wiki. The main reason for my proposal is to end policy wars, make maintaining Python packages easier and have Python 2.6 and 3.1 finally in unstable. -- -=[ Piotr Ożarowski ]=- -=[ http://www.ozarowski.pl ]=- |
|
|
Re: new dh_python proposal
by Matthias Klose
::
Rate this Message:
|
|
|
Re: new dh_python proposal
by Piotr Ożarowski-2
::
Rate this Message:
[Matthias Klose, 2009-08-20]
> On 03.08.2009 19:16, Piotr Ożarowski wrote: > >* about lack of XS-Python-Version and debian/pyversions > > - if available, both previous places will be used to get > > minimum/maximum required Python version, it will complicate > > detection of packages that need binNMU, so I want to drop both of > > them at some point, [...] > Why move this attribute from an explicit place to an implicit one? > Encoding all stuff in dependencies isn't that obvious. Indeed we do > create new attributes like Homepage, which were included before in > the description. [...] > Is it really worth adding semantics to the build > dependency/dependency fields and instead removing the explicit > information about version information? * to ease detecting packages that need binNMU? * to make it simpler for package maintainers ("my module needs Python >=2.4 - lets build depend on it" instead of "where the heck is XS-P-V / debian/pyversion's syntax described?") * to make it easier for dh/cdbs maintainers? Anyway, I don't care about it that much, it will not make any difference if we decide to support all 3 approaches or only existing two (both previous ones has to be supported at the beginning anyway). -- -=[ Piotr Ożarowski ]=- -=[ http://www.ozarowski.pl ]=- |
|
|
Re: new dh_python proposal
by Bernd Zeimetz
::
Rate this Message:
|
|
|
Re: new dh_python proposal
by Piotr Ożarowski-2
::
Rate this Message:
FTR: Joss and few other maintainers (whose opinion I care about) didn't
like my proposal (mainly due to binNMUs for arch:all packages) so I'm not working on this new tool. I planed to start working on it once we'll agree how it should look like. There's no consensus so I'm focusing on preparing all current tools for python2.6 upload to unstable. Today I uploaded cdbs NMU to delayed/7 (that fixes #537373) and reported a bug against python-central (#547565) which I plan to fix in NMU this week as well. -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 |
signature.asc (204 bytes) | Free embeddable forum powered by Nabble | Forum Help |
