python shebang, and other interpreters.

[Andrea Gasparini]

Hi,
I'm searching for some pointers of discussions about python shebang.
My main consideration is that if I want to install another python
installation or another interpreter (say for example jython, ironpython, or
unladenswallow ... ) I can't use it for all python programs, because every
update of python will rewrite /usr/bin/python.

The only report I found is:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=189473
but the motivation seems quite weak to me. I intended it this way: "prevent
users do local installations because they can both^W ...report bugs more
difficult to understand and fix".

There are perhaps security reasons? What could happen with an ENV installed
python?

Thanks, and sorry if the question will result dumb.
--
-gaspa-
-----------------------------------------------
-------- https://launchpad.net/~gaspa ---------
------ HomePage: iogaspa.altervista.org -------
-Il lunedi'dell'arrampicatore: www.lunedi.org -


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Wolodja Wentland-2]

On Thu, Jul 09, 2009 at 13:08 +0200, Andrea Gasparini wrote:
> Hi,
> I'm searching for some pointers of discussions about python shebang.
> My main consideration is that if I want to install another python
> installation or another interpreter (say for example jython,
> ironpython, or
> unladenswallow ... ) I can't use it for all python programs, because
> every
> update of python will rewrite /usr/bin/python.

I would like to raise this topic again because I think that it should be
decided upon.

The current python-policy contains a "do what you seem fit, but we
*prefer* /usr/bin/python" policy regarding "/usr/bin/python" vs
"/usr/bin/env python". This is not acceptable because it renders the
behaviour of Python programs totally unpredictable *if* the Python
environment is changed.

Statistics
==========

I can not generate an overview over all python applications, but the
following numbers give an overview on the schemes employed on *my*
system:

Programs in /usr/bin
--------------------

#!/usr/bin/env python       43 programs
#!/usr/bin/env python2.4     1 program
#!/usr/bin/env python2.4     2 programs

#!/usr/bin/python           50 programs
#!/usr/bin/python2.4         1 program
#!/usr/bin/python2.5         4 programs

As you can see the policy lead to a wild potpourri of shebang schemes
used for python packages.

Discussion
==========

I want to propose a policy change which enforces the usage of either
"/usr/bin/python{X.Y}" *or* "/usr/bin/env python{X.Y}" and MBF against all
packages that do not comply with the new policy. I believe that deciding
upon one scheme is much better than the situation right now. There will
be arguments against and in favour of both, so go forth and discuss. ;-)

I recently filed #547003 [1] but feel that it is not a bug as long as
the policy is not clear on the subject. Depending on the outcome of this
discussion i will either tag it with the tag used to monitor this policy
transition or "wontfix".

The bug report however is a perfect example for the problems users run
into with the current scheme. "Recent" developments within the python
developers community like virtualenv [2] will lead to many very
differently configured Python environments being present on a Debian
system. This is a development Debian will have to face in the future, so
let's get squeeze ready for it.

I use virtualenv extensively to test my software against different
python and library versions and am always wondering if an particular
/usr/bin/* program will work within that environment or use the standard
one.

Note that i am arguing as a Python *developer*, the situation for
*users* of Python software might be different.

#!/usr/bin/env python{X.Y}
--------------------------

I prefer if "#!/usr/bin/env python" would become the default policy
- except for a few packages that *have* to run within the Python
environment distributed with Debian - for the following reasons:

1. It works as expected if the administrator just installs the current
   Python environment distributed with Debian.

2. Python developers and users making use of virtualenv can expect
   programs installed with aptitude to work regardless of whether they
   are within a virtualenv or not.

   Programs that modify the Python environment (easy_install, pip, ...)
   use the currently used Python environment, which is IMHO the expected
   behaviour.

   If a virtual environment is initialised with "--no-site-packages" the
   user explicitly states that packages present in the Python environment
   used to set up the virtual environment should not be used, and will
   have to take care of missing libraries. The breaking of dependencies
   is wanted and expected in this case and this should be the only case
   in which breakage occurs.

3. If an administrator decides to install an additional Python
   environment from source into /usr/local it is *their* responsibility
   and stated goal to administer a Python environment not supported by
   Debian and the breakage that might occur is *their* responsibility.


Final Remarks
-------------

Please note that IANDD and am therefore not aware of the intricacies
associated with distributing Python within Debian. I am sure I
overlooked plenty of arguments and problems of either policy, but I
think that deciding upon *one* scheme is much better than the situation
we have right now.

If the Debian community decides to go with "#!/usr/bin/python{X.Y}" I am
left in a situation where the tools I am working with do not behave in
the way *I* expect them to. Is there a way to automatically patch every
Python program to use the scheme preferred by me unless it contains a
"no no" flag set by the package maintainer?

with kind regards

    Wolodja Wentland

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=547003
[2] http://pypi.python.org/pypi/virtualenv

[Piotr Ożarowski-2]

[Wolodja Wentland, 2009-09-20]

> Statistics
> ==========
>
> I can not generate an overview over all python applications, but the
> following numbers give an overview on the schemes employed on *my*
> system:
>
> Programs in /usr/bin
> --------------------
>
> #!/usr/bin/env python       43 programs
> #!/usr/bin/env python2.4     1 program
> #!/usr/bin/env python2.4     2 programs
>
> #!/usr/bin/python           50 programs
> #!/usr/bin/python2.4         1 program
> #!/usr/bin/python2.5         4 programs

on my system:

#!/usr/bin/python* 209 programs
#!/usr/bin/env python* 37 programs

IMHO /usr/bin/python should be a rule and "/usr/bin/env python" - (very
rare) exception (ipython or paster might qualify here)

BTW: If you install Python scripts via "python setup.py ..." (most
packages do that), it will not use env in shebang.
--
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[Wolodja Wentland-2]

On Sun, Sep 20, 2009 at 18:42 +0200, Piotr Ożarowski wrote:
> IMHO /usr/bin/python should be a rule and "/usr/bin/env python" - (very
> rare) exception (ipython or paster might qualify here)

Could you elaborate on the reasons for that? I am really interested and
it is my impression that enforcing the "/usr/bin/env python" scheme
provides significant advantages over "/usr/bin/python".

I am, however, not aware of the problems that might entail because
IANADD/pkg-python/python-apps member. What are these?

with kind regards

    Wolodja Wentland

[Piotr Ożarowski-2]

[Wolodja Wentland, 2009-09-20]
> On Sun, Sep 20, 2009 at 18:42 +0200, Piotr Ożarowski wrote:
> > IMHO /usr/bin/python should be a rule and "/usr/bin/env python" - (very
> > rare) exception (ipython or paster might qualify here)
>
> Could you elaborate on the reasons for that? I am really interested and
> it is my impression that enforcing the "/usr/bin/env python" scheme
> provides significant advantages over "/usr/bin/python".

I tested my package(s) with specific interpreter, if administrator/user
wants to use different one, he can run "his_own_python /usr/bin/application",
I will not fix bugs reported by users/administrators who use custom
interpreter. "/usr/bin/env python" might be a good solution for local
scripts but not for system ones.
--
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[Wolodja Wentland-2]

On Sun, Sep 20, 2009 at 19:05 +0200, Piotr Ożarowski wrote:

> [Wolodja Wentland, 2009-09-20]
> > On Sun, Sep 20, 2009 at 18:42 +0200, Piotr Ożarowski wrote:
> > > IMHO /usr/bin/python should be a rule and "/usr/bin/env python" - (very
> > > rare) exception (ipython or paster might qualify here)
> >
> > Could you elaborate on the reasons for that? I am really interested and
> > it is my impression that enforcing the "/usr/bin/env python" scheme
> > provides significant advantages over "/usr/bin/python".
>
> I tested my package(s) with specific interpreter, if administrator/user
> wants to use different one, he can run "his_own_python /usr/bin/application",
> I will not fix bugs reported by users/administrators who use custom
> interpreter. "/usr/bin/env python" might be a good solution for local
> scripts but not for system ones.

I can see your need as a python application maintainer to be *sure* that
the python version distributed with debian is used to run that program.
But the '/usr/bin/env python' scheme will result in exactly that
behaviour if the administrator/user has not intentionally installed a
*unsupported* Python environment.

I always had the impression that Debian relies on the fact that users
install software via the package manegment system provided by Debian,
but are still given the freedom to make adjustments to their environment
for which they take full responsibilty. IMHO installing a new Python
environment not provided (and supported) by Debian falls into this
freedom. But this comes with responsibility ...

To give a somewhat extreme example. A user could decide to install a new
Python version within /usr/local - which i think is commonly done with
Python 2.6 these days - and could link /usr/bin/python to
/usr/local/bin/python2.6 . Which would - of course - be stupid but the
administrator has the freedom to do so and has to live with the
consequences, ie errors in the execution of programs installed with
apt*.

To alleviate this problem a little Debian could give precise
instructions on how to install external Python environments. Which would
for example mean, that if a user installs Python 2.6 / 3.1 / whatever
she will delete /usr/local/bin python and can use the new version by
explicitly calling python{2.6,3.1}. If this is done '/usr/bin/env
python' still runs the program with the standard Debian version.

I am aware of the fact that this might cause a lot of bug reports by
users who don't know what they are doing, and expect their system to
"just work" whatever stupid decisions they take as an administrator.
This could be solved by providing feedback to the user that they are
using a unsupported Python version whenever they try to report a bug
with reportbug and it is found that '/usr/bin/env python' is indeed not
the Python environment as distributed by Debian.

If however the *env python scheme is enforced in the policy the problems
I outlined in my original post are solved without additional problems
(?). If there are any i would like to know about them!

I don't want to stress this point though. I am happy with more or less
anything the Debian python team decides upon as long as the policy
enforces *one* scheme, so I as a user know exactly how Python
application distributed by Debian behave. This would mean a MBF against
all packages that break that scheme, which might mean a lot of little
fixes to existing packages but at least ensures consistency. If I can do
anything to help with that I am happy to do so, although I think
applying a patch is more work than just changing existing quilt/dpatch
directly.

with kind regards

    Wolodja Wentland

[Piotr Ożarowski-2]

[Wolodja Wentland, 2009-09-20]

> I can see your need as a python application maintainer to be *sure* that
> the python version distributed with debian is used to run that program.
> But the '/usr/bin/env python' scheme will result in exactly that
> behaviour if the administrator/user has not intentionally installed a
> *unsupported* Python environment.
>
> I always had the impression that Debian relies on the fact that users
> install software via the package manegment system provided by Debian,
> but are still given the freedom to make adjustments to their environment
> for which they take full responsibilty. IMHO installing a new Python
> environment not provided (and supported) by Debian falls into this
> freedom. But this comes with responsibility ...
>
> To give a somewhat extreme example. A user could decide to install a new
> Python version within /usr/local - which i think is commonly done with
> Python 2.6 these days - and could link /usr/bin/python to
> /usr/local/bin/python2.6 . Which would - of course - be stupid but the
> administrator has the freedom to do so and has to live with the
> consequences, ie errors in the execution of programs installed with
> apt*.

If user/administrator is not following FHS and touching files outside
/usr/local, it's his problem. /usr/local and /etc is where administrator
can do his changes/improvements.

Sure there are *many* insane Python developers who suggest
"sudo easy_install Foo"[0] to their users, but then, when these users
come to me[1] all I'm proposing is to reinstall[2] their system (which you
should almost[3] never do on Debian).

[0] which installs eggs system wide (sic!)
[1] believe me, after tracking 10th problem that was caused by local
    modifications, first thing you check next time is:
    `grep \.egg bug_report_with_traceback`
[2] think twice before you will try to convince me you can uninstall
    eggs
[3] last time I reinstalled my system was when I changed architecture,
    and believe me - my system is one huge playground
--
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[Wolodja Wentland-2]

On Sun, Sep 20, 2009 at 20:52 +0200, Piotr Ożarowski wrote:
> If user/administrator is not following FHS and touching files outside
> /usr/local, it's his problem. /usr/local and /etc is where administrator
> can do his changes/improvements.

I completely agree! Did I give you reason to believe that I actually
propose such insane actions? All I did is to give a very *extreme*
example of insane actions that could be taken by admin managing their
Python environments.

> Sure there are *many* insane Python developers who suggest
> "sudo easy_install Foo"[0] to their users, but then, when these users
> come to me[1] all I'm proposing is to reinstall[2] their system (which you
> should almost[3] never do on Debian).
>
> [0] which installs eggs system wide (sic!)
> [1] believe me, after tracking 10th problem that was caused by local
>     modifications, first thing you check next time is:
>     `grep \.egg bug_report_with_traceback`
> [2] think twice before you will try to convince me you can uninstall
>     eggs
> [3] last time I reinstalled my system was when I changed architecture,
>     and believe me - my system is one huge playground

I am totally with you on that. Which is why I try to avoid setuptools
as hard as I can and use plain distutils to package my software and
urge users to install them to PREFIX /usr/local or use
--install-layout=deb. And if they insist on using python package
managers i suggest them to use pip w/ virtualenv  instead of
easy_install which luckily does not install eggs and will not
contaminate the global environment.

I don't see how this is related to the intention of my original post
which was the *suggestion* for a policy change to require *either*
'/usr/bin/python' or '/usr/bin/env python' so that Python programs
installed with apt* show consistent behaviour.

And to reiterate another point: If I as a user change my Python
environment is it unreasonable to assume that all Python software will
run in that environment?

Are there *any* reasons that I am not aware of to use '/usr/bin/python'
instead of '/usr/bin/env python' ?

with kind regards
   
    Wolodja Wentland

[Bernd Zeimetz]

Wolodja Wentland wrote:

> I can see your need as a python application maintainer to be *sure* that
> the python version distributed with debian is used to run that program.

Exactly.

> But the '/usr/bin/env python' scheme will result in exactly that
> behaviour if the administrator/user has not intentionally installed a
> *unsupported* Python environment.

If the behaviour is the same, why use '/usr/bin/env python' then. If the
admin/user/fool wants to install a different python version, they're able to
modify/reinstall the application in a sane way, and probably under the new
Python environment (which is the only proper way, mixing stuff likes to break
things).

>
> I always had the impression that Debian relies on the fact that users
> install software via the package manegment system provided by Debian,
> but are still given the freedom to make adjustments to their environment
> for which they take full responsibilty. IMHO installing a new Python
> environment not provided (and supported) by Debian falls into this
> freedom. But this comes with responsibility ...

Exactly. You're not forced to install something via apt. There is /usr/local and
$PATH.So if you install a new Python environment, please install all the
necessary stuff for it on your own. Thats the only way that makes sense for a
distribution, otherwise there is absolutely no sane way to react on bug reports
and other issues.

> [..]

> To alleviate this problem a little Debian could give precise
> instructions on how to install external Python environments. Which would
> for example mean, that if a user installs Python 2.6 / 3.1 / whatever
> she will delete /usr/local/bin python and can use the new version by
> explicitly calling python{2.6,3.1}. If this is done '/usr/bin/env
> python' still runs the program with the standard Debian version.

Just use http://pypi.python.org/pypi/virtualenv


> I am aware of the fact that this might cause a lot of bug reports by
> users who don't know what they are doing, and expect their system to
> "just work" whatever stupid decisions they take as an administrator.

Something which should be avoided clearly.

> This could be solved by providing feedback to the user that they are
> using a unsupported Python version whenever they try to report a bug
> with reportbug and it is found that '/usr/bin/env python' is indeed not
> the Python environment as distributed by Debian.

Which will only work as long as reportbug is run in the same environment... and
which will probably just fail again, as reportbug is using Python.

> If however the *env python scheme is enforced in the policy the problems
> I outlined in my original post are solved without additional problems
> (?). If there are any i would like to know about them!

There are probably not more problems than the few which were mentioned already,
but they are such large issues that thinking about enforcing the use ov env is
absolutely insane.


--
 Bernd Zeimetz                            Debian GNU/Linux Developer
 http://bzed.de                                http://www.debian.org
 GPG Fingerprints: 06C8 C9A2 EAAD E37E 5B2C BE93 067A AD04 C93B FF79
                   ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Piotr Ożarowski-2]

[Wolodja Wentland, 2009-09-20]
> On Sun, Sep 20, 2009 at 20:52 +0200, Piotr Ożarowski wrote:
>>[Wolodja Wentland, 2009-09-20]
>>> To give a somewhat extreme example. A user could decide to install a new
>>> Python version within /usr/local - which i think is commonly done with
>>> Python 2.6 these days - and could link /usr/bin/python to
>>> /usr/local/bin/python2.6
    ^^^^^^^^^^^^^^^^^^^^^^^^          ^^^^^^^^^^^^^^^^^^^^^^^
>>> . Which would - of course - be stupid but the
>>> administrator has the freedom to do so and has to live with the
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> consequences, ie errors in the execution of programs installed with
>>> apt*.
> > If user/administrator is not following FHS and touching files outside
> > /usr/local, it's his problem. /usr/local and /etc is where administrator
> > can do his changes/improvements.
>
> I completely agree! Did I give you reason to believe that I actually
> propose such insane actions?

See above. Sane administrator will never change /usr/bin/python symlink.
(too many tools depend on it)

[...]
> Which is why I try to avoid setuptools

setuptools it not bad per se, it's just how people use it

> as hard as I can and use plain distutils to package my software and
> urge users to install them to PREFIX /usr/local or use
> --install-layout=deb.

--install-layout=deb should be used by package maintainers only, if
you're encouraging users to install files in /usr and not /usr/local
(--install-layout=deb is just a shortcut for --prefix=/usr and
--install-lib=/usr/lib/pythonX.Y/dist-packages) you're doing exactly the
same what "sudo easy_install"-people do (it doesn't matter that these
modules were not installed via Eggs)

--install-layout=deb was introduced due to another (*very desired*)
change: Python >= 2.6 now installs to /usr/local by default (finally!)

[...]
> I don't see how this is related to the intention of my original post
> which was the *suggestion* for a policy change to require *either*
                                                            ^^^^^^^^
> '/usr/bin/python' or '/usr/bin/env python' so that Python programs
> installed with apt* show consistent behaviour.

See my initial reply (/usr/bin/python* by default, with exceptions).

There's no "either". No permanent changes system wide, never! Some
applications can be used in local env.[0] and some cannot. These that
can be used in such env. should have '/usr/bin/env python' in shebang,
these that cannot - '/usr/bin/python*'.

[0] by local env. I mean something like python-virtualenv, not changing
    `env python` output system wide or replacing /usr/bin/python
    symlink. No one except python package maintainer can change
    /usr/bin/python*

> And to reiterate another point: If I as a user change my Python
> environment is it unreasonable to assume that all Python software will
> run in that environment?

you can try to run in this new env. whatever application you want, even
packaged one with shebang set to /usr/bin/python* (see one of my
previous mails to see how), most of them will fail, though.
--
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[Wolodja Wentland-2]

On Sun, Sep 20, 2009 at 22:18 +0200, Bernd Zeimetz wrote:
> Wolodja Wentland wrote:

[ placed on top because this is the main point ]

> > If however the *env python scheme is enforced in the policy the problems
> > I outlined in my original post are solved without additional problems
> > (?). If there are any i would like to know about them!
>
> There are probably not more problems than the few which were mentioned already,
> but they are such large issues that thinking about enforcing the use ov env is
> absolutely insane.

I really don't get it. How could it possibly be better to have a wild
mix of programs that use '/usr/bin/python' and those using '/usr/bin/env
python' than having *one* predictable, consistent standard enforced in
the policy with *very* few exceptions. A situation in which packages not
using the enforced scheme are considered buggy and in violation of
policy and only *very* few packages are granted the right to use the
alternative scheme?

As I said in the last reply to Piotr's mail: I don't insist on making
'/usr/bin/env python' the default scheme and if you thought that this
was the intention of raising this discussion you must have misunderstood
me/my mails. (or more likely: I failed to convey my point)

I said many times that I lack the experience to see all aspects
associated with packaging Python for Debian which is why I discuss the
topic with people that do that on a daily basis. I just had the impression
that it is a good idea to have *one* scheme used by programs in Debian.

Could you please enumerate the "large" issues associated with
'/usr/bin/env python' for me? If the issues are that large why are
programs that use '/usr/bin/env python' not (considered) completely
broken?

I just had the, maybe naïve, impression that enforcing '/usr/bin/env
python' has some advantages while still providing the same
user/maintainer experience if *only* the Debian packaged Python is
installed. This would for example allow users of python-virtualenv as
packaged by Debian to run programs installed with apt* while they are in
their virtual environment and not having to install all these programs
*again* with a package management solution from Python (preferrably
pip).

This is use case 1 as described below.

It still won't break if other Python versions are installed
in a way sanctioned by Debian, by making sure that "python" always
refers to the one shipped with Debian.

The only problem I see here is that people will have programs in their
$PATH that *might* not be executable because there is a mismatch between
$PATH and $PYTHONPATH. I don't have a solution for that, but this is
clearly outside of the "supported by Debian" realm.


> > To alleviate this problem a little Debian could give precise
> > instructions on how to install external Python environments. Which would
> > for example mean, that if a user installs Python 2.6 / 3.1 / whatever
> > she will delete /usr/local/bin python and can use the new version by
> > explicitly calling python{2.6,3.1}. If this is done '/usr/bin/env
> > python' still runs the program with the standard Debian version.
> Just use http://pypi.python.org/pypi/virtualenv

I already do, which is actually the reason why I raised this topic here,
because I saw incoherent and unpredictable behaviour of programs that i
execute while being in the virtual environment.

The problem is that *some* programs I execute while being in the
environment work with the Python interpreter installed within the virtual
environment (copy of /usr/bin/python in .virtualenvs/foo/bin) while
others use /usr/bin/python. This is inconsistent behaviour which forces
the user to constantly check if the program they want to run uses
'/usr/bin/python' or '/usr/bin/env python'.

I have basically two use cases:

    1. Install not yet packaged software into a virtual environment so i
       can use it without polluting the system environment. The virtual
       environment is initially a clone of the system one.

    2. Create virtual environments with not yet packaged Python versions
       (2.6, 3.1) in order to test software with new and upcoming Python
       versions.

For 2. I install the Python environment in /usr/local, delete
/usr/local/bin/python (so that if i type python I *always* get the
system /usr/python/bin one and create virtual environments by executing
python{2.6,3.1} (mk)virtualenv.

If the standard policy is changed to '/usr/bin/python' i will just
install another version of python in /usr/local which matches the one
installed by Debian and basically re-create the system environment so
*I* can be sure that if I run software in the used environment all
programs will work with that environment.

Your statement "Just use ..." seems to imply that the creation of
virtual environments for Python versions not yet shipped by Debian is
possible without installing the respective Python version to /usr/local
first. Could you tell me how?

> > I am aware of the fact that this might cause a lot of bug reports by
> > users who don't know what they are doing, and expect their system to
> > "just work" whatever stupid decisions they take as an administrator.
> Something which should be avoided clearly.

Exactly!

> > This could be solved by providing feedback to the user that they are
> > using a unsupported Python version whenever they try to report a bug
> > with reportbug and it is found that '/usr/bin/env python' is indeed not
> > the Python environment as distributed by Debian.
> Which will only work as long as reportbug is run in the same environment... and
> which will probably just fail again, as reportbug is using Python.

OK, that is true. I can't really provide a solution for that right now
(if there is one) but let me say that it is quite likely (so not
certain!) that if the user runs reportbug that reportbug itself is
executed within the very same Python environment the bug is reported on.

But let's not discuss this now as it has nothing to do with the
intention of this thread (-> at the beginning of this mail/thread).

with kind regards

    Wolodja Wentland

[Wolodja Wentland-2]

On Sun, Sep 20, 2009 at 23:49 +0200, Piotr Ożarowski wrote:
> [Wolodja Wentland, 2009-09-20]
> > On Sun, Sep 20, 2009 at 20:52 +0200, Piotr Ożarowski wrote:
> >>[Wolodja Wentland, 2009-09-20]

> >>> To give a somewhat extreme example. A user could decide to install a new
> >>> Python version within /usr/local - which i think is commonly done with
> >>> Python 2.6 these days - and could link /usr/bin/python to
> >>> /usr/local/bin/python2.6
>     ^^^^^^^^^^^^^^^^^^^^^^^^          ^^^^^^^^^^^^^^^^^^^^^^^
> >>> . Which would - of course - be stupid but the
> >>> administrator has the freedom to do so and has to live with the
>     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >>> consequences, ie errors in the execution of programs installed with
> >>> apt*.
> > > If user/administrator is not following FHS and touching files outside
> > > /usr/local, it's his problem. /usr/local and /etc is where administrator
> > > can do his changes/improvements.

> > I completely agree! Did I give you reason to believe that I actually
> > propose such insane actions?

> See above. Sane administrator will never change /usr/bin/python symlink.
> (too many tools depend on it)

Yes! Did you interpreted what I said differently? I gave that example to
exemplify in "a somewhat extreme example" which I consider to "be
stupid" (to actually do) what I consider *supported* actions taken by an
administrator and what actions are *unsupported* . The example is
clearly considered unsupported as is installing additional python
versions.

> > Which is why I try to avoid setuptools
> setuptools it not bad per se, it's just how people use it

Hmm, i had the impression after reading:

http://www.b-list.org/weblog/2008/dec/14/packaging/

that distutils is preferred over setuptools. And that package maintainer
have a lot of work to get application data out of the egg, making sure
that a (Python) package is installed with
"--single-version-externally-managed", patching code to work without
relying on setuptools application data disceovery API and and and ...

> > as hard as I can and use plain distutils to package my software and
> > urge users to install them to PREFIX /usr/local or use
> > --install-layout=deb.
>
> --install-layout=deb should be used by package maintainers only, if
> you're encouraging users to install files in /usr and not /usr/local
> (--install-layout=deb is just a shortcut for --prefix=/usr and
> --install-lib=/usr/lib/pythonX.Y/dist-packages) you're doing exactly the
> same what "sudo easy_install"-people do (it doesn't matter that these
> modules were not installed via Eggs)

I *should* have added " ... if they *insist* on installing to /usr",
sorry that i forgot to do so. I encourage to install to /usr/local or to
use a virtual environment.

> > I don't see how this is related to the intention of my original post
> > which was the *suggestion* for a policy change to require *either*
> > '/usr/bin/python' or '/usr/bin/env python' so that Python programs
> > installed with apt* show consistent behaviour.
> There's no "either". No permanent changes system wide, never!
What do you mean by "permanent changes system wide"?

> Some applications can be used in local env.[0] and some cannot. These
> that can be used in such env. should have '/usr/bin/env python' in
> shebang, these that cannot - '/usr/bin/python*'.

> [0] by local env. I mean something like python-virtualenv, not changing

The problem is that, IPython for example, does not work with
python-virtualenv right now. Which is why i filed a bug and thought:

    * "What else will not work?"

    * "Wouldn't enforcing */env python on all packages mean that all
       packages work with python-virtualenv?"
   
    * "Wouldn't that make python programs on Debian much more
       predictable, because they all (except few) follow the same
       scheme?"
   
    * "Is this *really* a good idea?"
   
    * "Let's discuss this on debian-python"

and I wrote my initial mail (which you might want to read again) ...

>     `env python` output system wide or replacing /usr/bin/python
>     symlink. No one except python package maintainer can change
>     /usr/bin/python*

Yes, exactly! Never said anything else. I you take this from the
"extreme" example let me say that users/admins also have the freedom to
delete their harddrive or throw their computer out of the window. Which
just like violating the FHS cause in effects *unsupported* by Debian and
would be just as stupid and insane.

> > And to reiterate another point: If I as a user change my Python
> > environment is it unreasonable to assume that all Python software will
> > run in that environment?
> you can try to run in this new env. whatever application you want, even
> packaged one with shebang set to /usr/bin/python* (see one of my
> previous mails to see how), most of them will fail, though.

This is just not true. Only (some) packages with '/usr/bin/python' will
fail because the virtual environment will initially be an almost (see
below) exact clone of the original environment used to create the
virtual environment. It will even use the same modules and not copy it
to the new environment. (Or did *I* completely misunderstood the way
virtualenv works? (possible))

It *only* true iff (note the two 'f') the user creates an virtual
environment with '--no-site-packages' which is done deliberately and
with the *explicitly stated goal* to accept breakage of software that
relies on libraries in $PYTHONPATH but which is still present in $PATH.

.. which is IMHO acceptable.

I am arguing mainly as a developer who cares about Debian being a great
platform for software development in Python (which it is!) and Python
applications with '/usr/bin/python' force me to install every single
application I would like to use into every single virtual environment
if i want to use the applications in them. This also forces me to use
Python package managers (pip) to install that software and means I might
miss some nice patches from Debian not yet incorporated into upstream.

This essentially renders python-virtualenv unusable with the system
environment and somehow renders the python-virtualenv package
meaningless, because it would be much easier to just create the
virtual environment with /usr/local/bin/python and not having to worry
about "contamination" of the environment with modules installed system
wide. The issue that i'll still have broken applications in my $PATH
remains but that's OK with me because I'll know that I have to install
all applications that i would like to use into the virtualenvironment
again.

If my expectations are wrong I sincerely apologise for taking so much of
your time (for which I am *very* grateful) it's just that I would like
to point out things that stand in my way when developing Python on
Debian.

with kind regards

    Wolodja Wentland

with kind regards

    Wolodja Wentland

[Floris Bruynooghe-2]

On Mon, Sep 21, 2009 at 12:35:19AM +0200, Wolodja Wentland wrote:

> On Sun, Sep 20, 2009 at 22:18 +0200, Bernd Zeimetz wrote:
> > Wolodja Wentland wrote:
>
> [ placed on top because this is the main point ]
>
> > > If however the *env python scheme is enforced in the policy the problems
> > > I outlined in my original post are solved without additional problems
> > > (?). If there are any i would like to know about them!
> >
> > There are probably not more problems than the few which were mentioned already,
> > but they are such large issues that thinking about enforcing the use ov env is
> > absolutely insane.
>
> I really don't get it. How could it possibly be better to have a wild
> mix of programs that use '/usr/bin/python' and those using '/usr/bin/env
> python' than having *one* predictable, consistent standard enforced in
> the policy with *very* few exceptions. A situation in which packages not
> using the enforced scheme are considered buggy and in violation of
> policy and only *very* few packages are granted the right to use the
> alternative scheme?

This is indeed desired and the policy says to use /usr/bin/python, for
reasons explained in this thread already.  Maybe the applications
using /usr/bin/env should get a bug filled, I agree with that.

And I would even argue that /usr/bin/env is not allowed as an
exception, the tools needing this could provide several versions of
their app for each suppored python version, e.g. py.test, py.test2.4,
py.test2.5.  And if you want to use it with a non-debian environment
you can either install the tool in that environment or set the
environment up so you can do "/usr/local/bin/python2.6
/usr/bin/py.test", which is no different from now.

Regards
Floris


--
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Wolodja Wentland-2]

On Mon, Sep 21, 2009 at 08:36 +0100, Floris Bruynooghe wrote:
> On Mon, Sep 21, 2009 at 12:35:19AM +0200, Wolodja Wentland wrote:
> > On Sun, Sep 20, 2009 at 22:18 +0200, Bernd Zeimetz wrote:
> > > Wolodja Wentland wrote:

>> I really don't get it. How could it possibly be better to have a
>> wild mix of programs that use '/usr/bin/python' and those using
>> '/usr/bin/env python' than having *one* predictable, consistent
>> standard enforced in the policy with *very* few exceptions. A
>> situation in which packages not using the enforced scheme are
>> considered buggy and in violation of policy and only *very* few
>> packages are granted the right to use the alternative scheme?

> This is indeed desired and the policy says to use /usr/bin/python, for
> reasons explained in this thread already.  Maybe the applications
> using /usr/bin/env should get a bug filled, I agree with that.

Thank you!

> And I would even argue that /usr/bin/env is not allowed as an
> exception, the tools needing this could provide several versions of
> their app for each suppored python version, e.g. py.test, py.test2.4,
> py.test2.5.  

Hmmm, no. Python applications that need a specific version of Python
should use

    #!/usr/bin/env pythonX.Y

or

    #!/usr/bin/pythonX.Y

stating that they need this specific version of Python. A lot of Python
software runs with different Python versions and if there are
differences the application is buggy and should get fixed.

> And if you want to use it with a non-debian environment
> you can either install the tool in that environment or set the
> environment up so you can do "/usr/local/bin/python2.6
> /usr/bin/py.test", which is no different from now.

The problem is not really the application but rather it's dependencies.
If I try to run /usr/local/bin/python2.6 /usr/bin/foo and foo depends on
module 'foo' not installed for 2.6 the application will fail. Although
it runs perfectly fine under (Debian maintained) python2.5 .

with kind regards

    Wolodja

[Josselin Mouette]

Le lundi 21 septembre 2009 à 08:36 +0100, Floris Bruynooghe a écrit :
> This is indeed desired and the policy says to use /usr/bin/python, for
> reasons explained in this thread already.  Maybe the applications
> using /usr/bin/env should get a bug filled, I agree with that.

Given their number, this should probably be a lintian warning instead.

Cheers,
--
 .''`.      Josselin Mouette
: :' :
`. `'   “I recommend you to learn English in hope that you in
  `-     future understand things”  -- Jörg Schilling

[Siegfried Gevatter (RainCT)-2]

2009/9/21 Josselin Mouette <joss@...>:
>>  Maybe the applications
>> using /usr/bin/env should get a bug filled, I agree with that.

Can someone explain again what is it that we are gong to gain by
patching like 50% of all applications?

--
Siegfried-Angel Gevatter Pujals (RainCT)
Free Software Developer       363DEAE3


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Wolodja Wentland-2]

On Mon, Sep 21, 2009 at 14:07 +0200, Siegfried Gevatter wrote:
> 2009/9/21 Josselin Mouette <joss@...>:
> >>  Maybe the applications
> >> using /usr/bin/env should get a bug filled, I agree with that.

> Can someone explain again what is it that we are gong to gain by
> patching like 50% of all applications?

First and foremost consistency among and predictable behaviour of *all*
Python application in Debian.

I can't say much to the specific merits of adopting '/usr/bin/python'
as default scheme, others may want to do, but the most important point
is that the "do what you see fit, but we prefer *bin/python" lead to a
wild hotchpotch of implemented schemes [1] (1-2 and 3-4)  causing
different behaviour of Python applications in certain situations.

thanks for all the fish

    Wolodja

[1] Schemes:

    1. /usr/bin/python
    2. /usr/bin/pythonX.Y (2.4 and 2.5 right now)

    3. /usr/bin/env python
    4. /usr/bin/env pythonX.Y

[Toni Mueller-5]

Hi,

On Mon, 21.09.2009 at 12:13:46 +0200, Wolodja Wentland <wentland@...> wrote:

> Hmmm, no. Python applications that need a specific version of Python
> should use
>
>     #!/usr/bin/env pythonX.Y
>
> or
>
>     #!/usr/bin/pythonX.Y
>
> stating that they need this specific version of Python.

how else is an application supposed to state that it needs a specific
Python version?

> A lot of Python software runs with different Python versions and if
> there are differences the application is buggy and should get fixed.

*LOL*

You might want to look into "fixing" Zope that way. Good luck, and I
guess that some fame will come your way, should you succeed.

However, I agree that it's generally desirable to be able to run
applications on "any" Python version, but, as with probably all other
bigger systems (Java, ...), it is just not realistic to say "If app X
doesn't run with all environments Y, then the app is broken", simply
because no-one will be inclined, or even able, to "fix" it, usually for
marginal benefit.


Kind regards,
--Toni++


--
To UNSUBSCRIBE, email to debian-python-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Wolodja Wentland-2]

On Mon, Sep 21, 2009 at 21:51 +0200, Toni Mueller wrote:
> On Mon, 21.09.2009 at 12:13:46 +0200, Wolodja Wentland <wentland@...> wrote:

----
Note
----

Let me first say one thing.

I actually planned to write a big "thank you for the discussion" mail
this evening, because I really had the feeling that I as a Debian *user*
can help to improve Debian by discussing problems I have in my daily life
with the distribution I love on the appropriate lists. If the points I
make in the discussion are valid they are brought to the attention of
the right people who might act accordingly and the world is a better
place.

If I made mistakes or had misunderstandings I will quickly be corrected
and I will have learned something valuable about Debian. People might
change the documentation so other users don't face the same problems
again. No harm done here as well ...

Your mail however changed that feeling significantly and I think that
this gives a wrong example ...

------
Thanks
------

To everybody who contributed to this thread:

THANK YOU! I really appreciate the time you spend on reading my mails
and crafting your own.

------------------------------------------------------------------------

OFF TOPIC:

> > Hmmm, no. Python applications that need a specific version of Python
> > should use
> >     #!/usr/bin/env pythonX.Y
> > or
> >     #!/usr/bin/pythonX.Y
> > stating that they need this specific version of Python.
>
> how else is an application supposed to state that it needs a specific
> Python version?

Huh?! Why should there be another way? They could however just use
/usr/bin/python without specifying a Python version although they fail
to run if they are executed with one of the "wrong" ones.

> > A lot of Python software runs with different Python versions and if
> > there are differences the application is buggy and should get fixed.

> *LOL*

?

> You might want to look into "fixing" Zope that way. Good luck, and I
> guess that some fame will come your way, should you succeed.

I would consider specifying the specific Python version a program has to
be run with a possible fix, which is why i started the reply with
"Python applications that need a specific [...] version should use [...]
[to] state they need this specific version of Python"

> However, I agree that it's generally desirable to be able to run
> applications on "any" Python version, but, as with probably all other
> bigger systems (Java, ...), it is just not realistic to say "If app X
> doesn't run with all environments Y, then the app is broken", simply
> because no-one will be inclined, or even able, to "fix" it, usually for
> marginal benefit.

Do you *really* believe I think that every Python program should run
with every Python release that was ever released? As in running a Python
program developed using 2.5 with 1.0?

To make sure that I am not misunderstood and the words are not twisted
in my mouth:

--- snip ---
1.3.1 Interpreter Name

Python scripts depending on the default Python version (see Main
package, Section 1.2) or not depending on a specific Python version
should use python (unversioned) as the interpreter name.

Python scripts that only work with a specific Python version must
explicitly use the versioned interpreter name (pythonX.Y).
--- snip ---

I would consider programs not following this to be buggy.

Toni, let's stop the discussion here or take it off-list because this
has almost nothing to do with the topic of this thread.

Kind regards

    Wolodja