Lintian warnings for Python packaging?

[Luca Falavigna-4]

Currently, Lintian supports dozen of tags [1], but very few strictly
related to Python packaging. I think maintainers and sponsors would
benefit a lot if some common mistakes and suggestions are automatically
displayed by Lintian.

I propose a non comprehensive list of tags I'd like to see available:
* E: Don't hard-code {site,dist}-packages
* E: Python files in incorrect python2.?/{site,dist}-packages directory
* W: Build extension for every supported Python version
* W: Arch: all package but build-dependency on python*-dev
* W: #!/usr/bin/env python as shebang for Python scripts
* I: python-support and XB-Python-Version field
* I: No dh_python but pycompat file available
* I: Place Python applications in private directory
* P: Python extension but no -dbg package

If there is consensus on this, I could work on these tags and
eventually submitting to Lintian maintainers for review. Thoughts?

[1] http://lintian.debian.org/tags.html

--
  .''`.
 :  :' :   Luca Falavigna <dktrkranz@...>
 `.  `'
   `-

[Scott Kitterman-5]

On Mon, 2 Nov 2009 00:54:04 +0100 Luca Falavigna <dktrkranz@...>
wrote:

>Currently, Lintian supports dozen of tags [1], but very few strictly
>related to Python packaging. I think maintainers and sponsors would
>benefit a lot if some common mistakes and suggestions are automatically
>displayed by Lintian.
>
>I propose a non comprehensive list of tags I'd like to see available:
>* E: Don't hard-code {site,dist}-packages
>* E: Python files in incorrect python2.?/{site,dist}-packages directory
>* W: Build extension for every supported Python version
>* W: Arch: all package but build-dependency on python*-dev
>* W: #!/usr/bin/env python as shebang for Python scripts
>* I: python-support and XB-Python-Version field
>* I: No dh_python but pycompat file available
>* I: Place Python applications in private directory
>* P: Python extension but no -dbg package
>
>If there is consensus on this, I could work on these tags and
>eventually submitting to Lintian maintainers for review. Thoughts?
>
>[1] http://lintian.debian.org/tags.html

Since we currently lack anything like a maintained Python policy, I think
this is putting the cart before the horse.  Particularly any Error level
tags should, IMO, have some basis in policy.

Scott K


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Ben Finney-5]

Luca Falavigna <dktrkranz@...> writes:

> I propose a non comprehensive list of tags I'd like to see available:

Thank you, I think this is a good start.

> * E: Don't hard-code {site,dist}-packages
> * E: Python files in incorrect python2.?/{site,dist}-packages
> directory
> * W: Build extension for every supported Python version
> * I: Place Python applications in private directory

What would be the test for these cases? That is, what would Lintian
actually check in the package to determine whether these should be
emitted?

> * W: Arch: all package but build-dependency on python*-dev

Is there a converse for this, for ‘Architecture: any’ packages?

> * I: python-support and XB-Python-Version field

As I understand it, many people are dealing with the lack of clear
policy in this matter by having *both* ‘debian/pyversions’ file and the
‘XB-Python-Version’ field.

Until there is clear consensus that ‘XB-Python-Version’ is *not*
required, I would not want Lintian to check this.

> * W: #!/usr/bin/env python as shebang for Python scripts
> * I: No dh_python but pycompat file available
> * P: Python extension but no -dbg package

Good.

--
 \         “In any great organization it is far, far safer to be wrong |
  `\          with the majority than to be right alone.” —John Kenneth |
_o__)                                            Galbraith, 1989-07-28 |
Ben Finney

[Jakub Wilk]

* Luca Falavigna <dktrkranz@...>, 2009-11-02, 00:54:
>I propose a non comprehensive list of tags I'd like to see available:
>* E: Don't hard-code {site,dist}-packages

Uhm, what do you mean?

>* E: Python files in incorrect python2.?/{site,dist}-packages directory

We need this check as soon as possible.

>* I: No dh_python but pycompat file available

Well, we should have:
W: dh_python (use dh_pysupport instead)
in the first place.

>* I: Place Python applications in private directory

And how do you define "applications"?

>* W: Arch: all package but build-dependency on python*-dev

I've already filed a bug about this:
http://bugs.debian.org/551793

Other ideas for lintian checks:

- Python source mixes tabs with spaces.
- *-dbg package contains *_d.so, but no detached symbols in
/usr/lib/debug/.

--
Jakub Wilk

[Steve Langasek]

On Mon, Nov 02, 2009 at 12:54:04AM +0100, Luca Falavigna wrote:
> Currently, Lintian supports dozen of tags [1], but very few strictly
> related to Python packaging. I think maintainers and sponsors would
> benefit a lot if some common mistakes and suggestions are automatically
> displayed by Lintian.

> I propose a non comprehensive list of tags I'd like to see available:
> * E: Don't hard-code {site,dist}-packages

hard-coded where, and how will you detect this?

> * W: Build extension for every supported Python version

how will you detect this?

> * W: #!/usr/bin/env python as shebang for Python scripts

AFAIK this is allowed by the existing python policy; and if it's allowed,
then I think such a warning is just noise.  We should either forbid this
shebang line completely (with a clear rationale), or not bother maintainers
about it, because if it's not mandatory then it's not worth modifying
upstream code over.

> * I: python-support and XB-Python-Version field

Why bother?  XB-Python-Version was intended as archive-side annotation of
packages; if it's noteworthy at all, I think it's noteworthy whether or not
using python-support.

> * I: No dh_python but pycompat file available
> * I: Place Python applications in private directory

Likewise, I don't see the point for something that's only informational.

> * P: Python extension but no -dbg package

Well, this fits the definition of "pedantic" (people - e.g., me - don't
agree that this check is correct).  But I don't see the point in adding
pedantic tags to lintian. :)

The rest seem ok to me.

Cheers,
--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@...                                     vorlon@...

[Luca Falavigna-4]

Scott Kitterman ha scritto:
>> * E: Don't hard-code {site,dist}-packages
>> * E: Python files in incorrect python2.?/{site,dist}-packages directory
>
> Since we currently lack anything like a maintained Python policy, I think
> this is putting the cart before the horse.  Particularly any Error level
> tags should, IMO, have some basis in policy.

Speaking of the error tags, this is not policy, but rather how things
are handled packaging side. Packages won't work properly if files are
stored in a wrong directory, and since we probably release with both 2.5
and 2.6, chances are high to make some errors not easily discoverable at
build time. That is worth the severity, IMHO.

For the rest, I agree they're not based on policy but on a wiki page, we
could wait for the new policy to be drafted, I'm not sure when this will
happen, though.

--

  .''`.
 : :' :   Luca Falavigna <dktrkranz@...>
 `. `'
   `-

[Luca Falavigna-4]

Ben Finney ha scritto:
>> * I: Place Python applications in private directory
>
> What would be the test for these cases? That is, what would Lintian
> actually check in the package to determine whether these should be
> emitted?

This is quite complex to say, I initially thought of checking for files
under /usr/bin and related modules under /usr/share/pyshared, but there
are some cases where this is not enough.


>> * W: Arch: all package but build-dependency on python*-dev
>
> Is there a converse for this, for ‘Architecture: any’ packages?

Do you mean the case where there are Arch: any packages with Python
scripts that requires no compilation? That's very hard to tell, and it
will probably result in many false positives, so I'd check for the "all"
case only.


>> * I: python-support and XB-Python-Version field
>
> As I understand it, many people are dealing with the lack of clear
> policy in this matter by having *both* ‘debian/pyversions’ file and the
> ‘XB-Python-Version’ field.
>
> Until there is clear consensus that ‘XB-Python-Version’ is *not*
> required, I would not want Lintian to check this.

You was probably referring to XS-Python-Version field, while I was
referring to XB-P-V. It is currently unmanaged by python-support, so I
think a notice to maintainers about this could be interesting. I don't
dare to touch XS-P-V or pyversions at all, given that there are several
implications for those files.


--

  .''`.
 : :' :   Luca Falavigna <dktrkranz@...>
 `. `'
   `-

[Luca Falavigna-4]

Jakub Wilk ha scritto:
>> * E: Don't hard-code {site,dist}-packages
>
> Uhm, what do you mean?

That's what we are trying to fix with
http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-python@...;tag=dist-packages

In case we missed some packages, or some NEW ones were included in the
archives since we last checked, we give maintainers a big notice. This
could be similar to "Python files in incorrect directory", but this
would catch the case of empty packages.


>> * I: No dh_python but pycompat file available
>
> Well, we should have:
> W: dh_python (use dh_pysupport instead)
> in the first place.

I thought this was already defined, but probably not in Lintian.


>> * W: Arch: all package but build-dependency on python*-dev
>
> I've already filed a bug about this:
> http://bugs.debian.org/551793

Thanks.


--

  .''`.
 : :' :   Luca Falavigna <dktrkranz@...>
 `. `'
   `-

[Luca Falavigna-4]

Steve Langasek ha scritto:
>> * E: Don't hard-code {site,dist}-packages
>
> hard-coded where, and how will you detect this?

debian/rules and other control files such as debian/install and
debian/links.


>> * W: Build extension for every supported Python version
>
> how will you detect this?

Parsing XS-P-V or pyversions, comparing it with supported versions and
then checking for files under /usr/lib/py*


>> * W: #!/usr/bin/env python as shebang for Python scripts
>
> AFAIK this is allowed by the existing python policy; and if it's allowed,
> then I think such a warning is just noise.  We should either forbid this
> shebang line completely (with a clear rationale), or not bother maintainers
> about it, because if it's not mandatory then it's not worth modifying
> upstream code over.

I agree it should be clarified and defined properly in Python policy.

As a side note, some other *NIXes (e.g. FreeBSD, IIRC) need
#!/usr/bin/env as shebang due to how they distribute Python.


>> * I: python-support and XB-Python-Version field
>
> Why bother?  XB-Python-Version was intended as archive-side annotation of
> packages; if it's noteworthy at all, I think it's noteworthy whether or not
> using python-support.

Rationale is "if it's not useful, why use it then?"


>> * I: No dh_python but pycompat file available
>> * I: Place Python applications in private directory
>
> Likewise, I don't see the point for something that's only informational.

To give maintainers a note about boilerplate files which can be removed
or how to avoid polluting global namespace without a valid reason.


>> * P: Python extension but no -dbg package
>
> Well, this fits the definition of "pedantic" (people - e.g., me - don't
> agree that this check is correct).  But I don't see the point in adding
> pedantic tags to lintian. :)

-dbg packages are useful for debugging purposes, so it's not strictly
necessary (hence the "pedantic"). There are plans to provide automatic
dbgsym packages, but I don't think they can handle Python ones. I'll
have a look at this matter.


--

  .''`.
 : :' :   Luca Falavigna <dktrkranz@...>
 `. `'
   `-

[Ben Finney-5]

Luca Falavigna <dktrkranz@...> writes:

> Scott Kitterman ha scritto:
> > Since we currently lack anything like a maintained Python policy, I
> > think this is putting the cart before the horse. […]

> […] we could wait for the new policy to be drafted, I'm not sure when
> this will happen, though.

I don't know if anyone has even taken the reins for this recently.

The last time I knew someone was actually developing a Debian Python
policy was when Manoj Srivastava was drafting a document to help record
some of the ad hoc practices he observed, and that work appears to have
ceased sometime in 2006.

The Debian wiki page on the topic, though no doubt useful to some
extent, seems more a collection of tips than an attempt at a policy.

Is there a silent Debian Python policy drafter out there who would like
to step forward? Or is this work now moribund?

--
 \             “The greater the artist, the greater the doubt; perfect |
  `\       confidence is granted to the less talented as a consolation |
_o__)                                           prize.” —Robert Hughes |
Ben Finney

[Scott Kitterman-5]

On Mon, 02 Nov 2009 21:22:47 +1100 Ben Finney <ben+debian@...>
wrote:

>Luca Falavigna <dktrkranz@...> writes:
>
>> Scott Kitterman ha scritto:
>> > Since we currently lack anything like a maintained Python policy, I
>> > think this is putting the cart before the horse. [ &]
>
>> [ &] we could wait for the new policy to be drafted, I'm not sure when
>> this will happen, though.
>
>I don't know if anyone has even taken the reins for this recently.
>
>The last time I knew someone was actually developing a Debian Python
>policy was when Manoj Srivastava was drafting a document to help record
>some of the ad hoc practices he observed, and that work appears to have
>ceased sometime in 2006.
>
>The Debian wiki page on the topic, though no doubt useful to some
>extent, seems more a collection of tips than an attempt at a policy.
>
>Is there a silent Debian Python policy drafter out there who would like
>to step forward? Or is this work now moribund?
>

I'm not aware of any ongoing work.  I would be willing to help work on such
a thing, but we currently lack a good mechanism for developing/approving
such a policy.

Scott K


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[anatoly techtonik]

On Mon, Nov 2, 2009 at 2:27 PM, Scott Kitterman <debian@...> wrote:
>
> I'm not aware of any ongoing work.  I would be willing to help work on such
> a thing, but we currently lack a good mechanism for developing/approving
> such a policy.

With clear policy and precise goal you won't need approving mechanism
to see if they work for defined set of cases or not.

While everybody want policy just to know how to do thing properly,
there are in fact very few people who really understand how
complicated is the task of maintaining python code, modules and
applications. When there is precise goal, next action is to collect
scenarios for the whole install/update/remove lifecycle of Python code
in Debian. Only after this step is complete it is possible to start
drafting self-explanatory architecture that will be capable to support
all these scenarios.


There is no need in mechanism for developing a policy - in wiki
everybody can start contributing immediately with a full history of
changes. There can be a sprint though to force the progress and keep
work focused. To make it easier to contribute scenarios a template can
come handy.


I've edited http://wiki.debian.org/DebianPython to be concise
introduction into the problems with Python code packaging, summarized
issues with the current policy, but still can't provide vision for a
new policy. That's why I'd like to see
http://wiki.debian.org/DebianPython/Tutorial with step-by-step
instructions and explanations of the reasons why things should be done
in some particular way, what problems arise if they won't done as
requested, and how it makes maintenance easier. There can be a series
of tutorials starting with most basic packaging scenario (one module)
and gradually move to most complicated (application with several
C-modules installed in virtualenv).

There is a difference in Scenario and Tutorial in that Tutorial is
based on some policy draft while Scenario concentrates on a very-very
source of the problem. I.e. scenario is "As a user, I want some stable
version of that Python module to be present for my scripts in my
Debian installation" or "As an admin, I want to install Trac in
isolated environment and upgrade it separately as security fixes are
coming out".


--
anatoly t.


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Scott Kitterman-5]

On Mon, 2 Nov 2009 16:50:00 +0300 anatoly techtonik <techtonik@...> wrote:
>On Mon, Nov 2, 2009 at 2:27 PM, Scott Kitterman <debian@...> wrote:
>>
>> I'm not aware of any ongoing work.  I would be willing to help work on such
>> a thing, but we currently lack a good mechanism for developing/approving
>> such a policy.
>
>With clear policy and precise goal you won't need approving mechanism
>to see if they work for defined set of cases or not.
>

...

Yes and we have neither right now.  Writing stuff on a wiki won't change that.

Until we have a legitimate Python policy, all we have to base decisions on is running code.  All the code doesn't agree.

Scott K

P.S.  I am subscribed to the list, so no need to cc me.


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Josselin Mouette]

Le lundi 02 novembre 2009 à 21:22 +1100, Ben Finney a écrit :
> Is there a silent Debian Python policy drafter out there who would like
> to step forward? Or is this work now moribund?

Bug reports concerning the Python policy have been silently ignored. I’m
afraid this will last as long as the reference version is in the
python-defaults package.

Cheers,
--
 .''`.      Josselin Mouette
: :' :
`. `'   “I recommend you to learn English in hope that you in
  `-     future understand things”  -- Jörg Schilling

[Scott Kitterman-5]

On Tue, 03 Nov 2009 19:02:21 +0100 Josselin Mouette <joss@...> wrote:
>Le lundi 02 novembre 2009 à 21:22 +1100, Ben Finney a écrit :
>> Is there a silent Debian Python policy drafter out there who would like
>> to step forward? Or is this work now moribund?
>
>Bug reports concerning the Python policy have been silently ignored. I m
>afraid this will last as long as the reference version is in the
>python-defaults package.
>
I'm inclined to agree.  How would you suggest such a document be managed?

Scott K


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Ben Finney-5]

Josselin Mouette <joss@...> writes:

> Le lundi 02 novembre 2009 à 21:22 +1100, Ben Finney a écrit :
> > Is there a silent Debian Python policy drafter out there who would
> > like to step forward? Or is this work now moribund?
>
> Bug reports concerning the Python policy have been silently ignored.
> I’m afraid this will last as long as the reference version is in the
> python-defaults package.

(I am reading this to mean “the reference version of the Debian Python
policy is in the python-defaults package”.)

Okay. Clearly one way for this to improve would be for some of those bug
reports to be responded to by the maintainer.

In the absence of that, though, what other way forward is there? What
would need to change for the reference version of the Python policy to
be somewhere else? Just start referring to a different document, or
something more?

--
 \      “The most merciful thing in the world… is the inability of the |
  `\        human mind to correlate all its contents.” —Howard Philips |
_o__)                                                        Lovecraft |
Ben Finney

[Ben Finney-5]

Josselin Mouette <joss@...> writes:

> Le lundi 02 novembre 2009 à 21:22 +1100, Ben Finney a écrit :
> > Is there a silent Debian Python policy drafter out there who would
> > like to step forward? Or is this work now moribund?
>
> Bug reports concerning the Python policy have been silently ignored.
> I’m afraid this will last as long as the reference version is in the
> python-defaults package.

(I am reading this to mean “the reference version of the Debian Python
policy is in the python-defaults package”.)

Okay. Clearly one way for this to improve would be for some of those bug
reports to be responded to by the maintainer.

In the absence of that, though, what other way forward is there? What
would need to change for the reference version of the Python policy to
be somewhere else? Just start referring to a different document, or
something more?

--
 \      “The most merciful thing in the world… is the inability of the |
  `\        human mind to correlate all its contents.” —Howard Philips |
_o__)                                                        Lovecraft |
Ben Finney

[anatoly techtonik]

On Wed, Nov 4, 2009 at 2:29 AM, Ben Finney <ben+debian@...> wrote:
> (I am reading this to mean “the reference version of the Debian Python
> policy is in the python-defaults package”.)
>
> Okay. Clearly one way for this to improve would be for some of those bug
> reports to be responded to by the maintainer.
>
> In the absence of that, though, what other way forward is there?

Here.

> What
> would need to change for the reference version of the Python policy to
> be somewhere else?

Remove it from http://www.debian.org/doc/packaging-manuals/python-policy/

Remove this page http://python-modules.alioth.debian.org/ (wiki is
more up to date)

Replace this one with wiki too and remove links.
http://python-apps.alioth.debian.org/policy.html

> Just start referring to a different document, or
> something more?

Point everything to wiki.
Update MoinMoin to remove bugs and enable latest features that can
become useful for collaboration. In particular:

 * Add "subscription by default" feature to policy pages so that
everybody who edited page automatically receives updates. This will
increase collaboration rate. You may forward these edits here as well.

 * Upgrade also fixes
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=546905 so that
inserting table of content will not be a compromise between layout and
technical limitations

--
anatoly t.


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...