Nabble - Python - distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T18:14:13Z

On Wed, 23 Dec 2009 02:58:10 +0100, "Martin v. Löwis" <martin@...>
wrote:
>> However, under Windows 7 Enterprise, with a high security policy. It's
>> totally impossible now to install the same package, even if python has
>> been installed by the system administrator. It won't let you run .exe
>> files.
>
> I don't think that's the case - Windows 7 most certainly lets you run
> exe files.
>
> IOW, I don't know what the problem might be that you are experiencing.

Sorry for the noise. It's something to do with the new security policy that
I don't understand. It won't run any program but the installed programs.

Thanks for asking - Merry Christmas

David
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T17:58:10Z

> However, under Windows 7 Enterprise, with a high security policy. It's
> totally impossible now to install the same package, even if python has
> been installed by the system administrator. It won't let you run .exe
> files.

I don't think that's the case - Windows 7 most certainly lets you run
exe files.

IOW, I don't know what the problem might be that you are experiencing.

Regards,
Martin

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T17:56:13Z

On Wed, 23 Dec 2009 01:01:11 +0100, Lennart Regebro <regebro@...>
wrote:
> OK, so in the Perl community there is apparently a lot of confusion on
> what CPAN is.

CPAN is plain and simple. There is no confusion, because there is
just one 'brand-name' for the whole kit and caboodle. "Packaging"
in the perl world just goes under the name CPAN.

> And in the Python community there is a lot of confusion
> about installation.

Relatively speaking, yes.

> And although CPAN does't install and has nothing
> to do with installation, people in the Perl community think it does.

That's right. Because of the bundling.

> So according to you, when Perl people say "Python need CPAN" what they
> actually mean, according to you, is "We can't convert one package
> format to another", because they don't know that CPAN is the archive,
> not the installation program.
>
> Is this correct?

The question is a little too complicated to answer directly on
that wording.

However, a Perl user (speaking for myself) would typically
know nothing about conversion of different package formats,
because as far as I am aware the user isn't exposed to that level
of complexity.

There is no .tar.gz, .zip, .bz2, .exe, .msi or .egg concept of
packages in perl. And having to pick one.. that may or may not
be right for your configuration.

Call a perl user a pampered pooch by all means. But all they
know is that if they need a module.. then they use CPAN.

Really, some of these questions should go back up the management
chain for answering. We shouldn't be having expectations that
we can write code when drunk, fix it in the morning, and be
merry and listen to ump-pah-pah music (like CPAN people).
Especially if the development process is to be reading and
commenting on PEPs.

Please.. lets just go back to the work at hand..

This is serious..

David

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T16:01:11Z

On Tue, Dec 22, 2009 at 23:34, David Lyon <david.lyon@...> wrote:
> Oh come on... nobody is saying pypi sucks. Only that the package
> installation experience in python does. In python, these are two
> separate and entirely different things.
>
> Under Perl, the whole thing is just called CPAN.

OK, so in the Perl community there is apparently a lot of confusion on
what CPAN is. And in the Python community there is a lot of confusion
about installation. And although CPAN does't install and has nothing
to do with installation, people in the Perl community think it does.
So according to you, when Perl people say "Python need CPAN" what they
actually mean, according to you, is "We can't convert one package
format to another", because they don't know that CPAN is the archive,
not the installation program.

Is this correct?

--
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T14:34:07Z

On Tue, 22 Dec 2009 19:04:00 +0100, Lennart Regebro <regebro@...>
wrote:

> 2009/12/22 David Cournapeau :
>>
>> ..
>>
>> The devil is really in the details for packaging, which is why
>> explicit is a very desirable feature. Someone noticed earlier that
>> distutils was the exact contrary of the python zen on some core
>> issues: more than one way to do it, lack of simplicity, often
>> implicit. I think this describe the problem quite well, and I am
>> deeply convinced that that's the root cause of the failure of Pypi for
>> a subset of the community.
>
> Well, you have completely failed in explaining why CPAN is fantastic
> and PyPI sucks, probably because your answer isn't about CPAN at all,
> but just the same old packaging rant. It really grows old you know.

Oh come on... nobody is saying pypi sucks. Only that the package
installation experience in python does. In python, these are two
separate and entirely different things.

Under Perl, the whole thing is just called CPAN. There's one CPAN
client that you use (btw - that gets included in your perl
distribution) and it works.

Guido is being harassed about by users from the scientific community
as I understand it. They are after a 'nice' user experience like
they get in Perl.

So let me translate as best as I can ...

"Python people want _NICE_USER_EXPERIENCE_ and how
the latter came about"

David

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T14:02:35Z

On Tue, 22 Dec 2009 19:48:38 +0100, "Martin v. Löwis" <martin@...>
wrote:

>> ..
>
> There is most certainly a reason. The binary distribution may be lacking
> pieces of source code that would be needed to build another (different)
> binary distribution.

Actually, I have a question about that. I'll just ask here because
you may be able to answer it in 30 seconds but it might take me
a few hours to find out the answer myself.

Under Windows, through the build-msi and build-exe procedure, we
have windows GUI package installers. When I was using one built by Mark
Hammond the other day, I noticed that it worked pretty fabulously.

However, under Windows 7 Enterprise, with a high security policy. It's
totally impossible now to install the same package, even if python has
been installed by the system administrator. It won't let you run .exe
files.

Since this whole approach now seems broken going forward.. what
is going to happen? Can that code be moved somewhere so that it
can install regular (source) packages?

David

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T11:24:09Z

On 2009-12-22 13:20 PM, "Martin v. Löwis" wrote:
>> Yes, that use case is rare (and bizarre), but the desire to convert
>> Windows-platform eggs to .msi or .exe installers is not so rare and does
>> not present the same level of difficulty. The latter is the use case
>> David was referring to, not the one you outlined.
>
> Why do people want to do that? Can't they build the type of package they
> want from the source package?

Not necessarily on Windows.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T11:20:17Z

> Yes, that use case is rare (and bizarre), but the desire to convert
> Windows-platform eggs to .msi or .exe installers is not so rare and does
> not present the same level of difficulty. The latter is the use case
> David was referring to, not the one you outlined.

Why do people want to do that? Can't they build the type of package they
want from the source package?

Regards,
Martin
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T10:54:42Z

On 2009-12-22 12:48 PM, "Martin v. Löwis" wrote:

>> By installer, I mean things produced by bdist_*. A significant portion
>> of windows users don't like eggs, and prefer .exe-based (or
>> .msi-based) installers. Currently, it is not possible to (reliably)
>> convert from one to the other (e.g. eggs->wininst), but there is no
>> reason why this is so.
>
> There is most certainly a reason. The binary distribution may be lacking
> pieces of source code that would be needed to build another (different)
> binary distribution.
>
> For example, if you have a Linux RPM or .deb, it is, in general, not
> possible to create a Windows installer out of this, as you may need
> to recompile extension modules. Including the full source code in the
> binary distribution just to support this rare use case is unreasonable:
> people who want a different package format can build from the source
> package.

Yes, that use case is rare (and bizarre), but the desire to convert
Windows-platform eggs to .msi or .exe installers is not so rare and does not
present the same level of difficulty. The latter is the use case David was
referring to, not the one you outlined.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T10:48:38Z

> By installer, I mean things produced by bdist_*. A significant portion
> of windows users don't like eggs, and prefer .exe-based (or
> .msi-based) installers. Currently, it is not possible to (reliably)
> convert from one to the other (e.g. eggs->wininst), but there is no
> reason why this is so.

There is most certainly a reason. The binary distribution may be lacking
pieces of source code that would be needed to build another (different)
binary distribution.

For example, if you have a Linux RPM or .deb, it is, in general, not
possible to create a Windows installer out of this, as you may need
to recompile extension modules. Including the full source code in the
binary distribution just to support this rare use case is unreasonable:
people who want a different package format can build from the source
package.

Regards,
Martin
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T10:15:30Z

On Tue, Dec 22, 2009 at 10:14, Steffen Mueller <smueller@...> wrote:

> Hi Lennart,
>
> Lennart Regebro <regebro <at> gmail.com> writes:
>> Interesting. I generally don't find automatically generated
>> documentation useful, but if it works for PERL I guess that's a
>> personal taste.
>
> Let me add two nits here:
> It's Perl, not PERL. The name of the language is *not* an acronym. Some people
> are really picky about that.

In a language where you can spell any functionality in millions of
ways, you are not allowed to spell the language however you want!? ;-)
Ok, Pörl. I mean Perl. :-)

> More importantly, it's not "auto-generated" documentation per se. Not something
> like using doxygen to generate an API reference purely from the function/method
> name and signature. Instead, search.cpan.org (and kobesearch) extract the
> inlined documentation from the Perl modules. This documentation was written by
> the authors. It was *not* generated from the code itself. Here's what that looks
> like for a random one of my modules:
> http://search.cpan.org/~smueller/PAR-Repository-Client/lib/PAR/Repository/
> Client.pm

OK, so that would be docstrings and stuff then. It's true, if we had
something like that it would work as an incentive for making more such
documentation.

> You could get the same page via
> http://search.cpan.org/perldoc?PAR::Repository::Client. This is an example of
> how CPAN works on namespace and distribution level. After a new file is
> uploaded, its contained namespaces (packages/class names) are added to the
> index if they're not already there.

How does it handle if two modules implement the same namespace?

> The index always contains a reference to
> the distribution that contains the newest *authorized* version of any
> namespace.

Who authorizes it?

> We have a band of PAUSE admins with super cow powers to save the day. It's not
> like there's a lot of us, but so far, we've been able to handle authorization
> requests reasonably. Our policy in case an author goes missing who has
> permissions for a certain namespace are documented here:
> http://search.cpan.org/CPAN/modules/04pause.html#takeover

I have to say that I vastly prefer not to have any authorization and
allow anyone to release anything in any namespace. But then I am
getting fanatically anarchical in these issues. You can not organize
freedom.

>> So far we have identified a "show all versions" link, and
>> automatically generated documentation. Thanks! Finally something
>> concrete!
>
> In a very concrete context: I really like how search.cpan.org will show you the
> latest stable version by default and provide drop-downs to select any developer
> (alpha) or old releases. It also has nice, consistent URLs as demonstrated
> above. ...~author will get me to author's overview page, .../dist/XXX will get
> me to the XXX distribution's overview, etc. Accessing any files within each
> distribution is possible via similar URLs.

The only thing I can see there that's missing is the "drop down", or
link to a list of all versions or similar. That would also make it
easier to make alpha versions, they would be easily viewable, but not
shown and listed by default.
We should suggest that to Martin v Löwis.

> Another point that I really like about the service is that the distribution
> pages provide links to many other related services that are run by other
> volunteers. Take for example http://search.cpan.org/dist/PAR-Repository-Client/
> There is a link to cpanforum, specifically the relevant discussion forum for the
> module at hand. It shows a link to the bug/request tracker with the number of
> open bugs, the one next to it will display a nice hierarchical (recursive) list
> of dependencies.

Right, those third-party services doesn't exist for PyPI.

--
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T10:04:00Z

2009/12/22 David Cournapeau <cournape@...>:
> By installer, I mean things produced by bdist_*. A significant portion
> of windows users don't like eggs, and prefer .exe-based (or
> .msi-based) installers. Currently, it is not possible to (reliably)
> convert from one to the other (e.g. eggs->wininst), but there is no
> reason why this is so.

You are right, I can't see any reason why not. Why can't you?
Obviously you can do it from a source distribution, but what
information is missing from the binary distributions?

> You are not wrong, but we are not talking about the same scenario. The
> scenario I care the most is user A build or install a package, and
> user B wants to install it; A may know very little about python.

In which case no sane user B wants to install it. ;)

How is this a different scenario? What scenario was I talking about?

On Tue, Dec 22, 2009 at 14:02, David Cournapeau <cournape@...> wrote:
> From the top of my head:
> - how data files are included

Well, there are rules for that, just as with python files, you dont'
have to explicitly specify every file that should be included, thank
god, that would be really annoying.

And this has WHAT to do with CPAN? Just to try to keep even remotely
on topic? Isn't this turning into the usual "distutils sucks and
should be rewritten but I can't explain why and how" rant? And isn't
the answer still: "Well, do it then?"

> Features only help as much as long as you agree on the purpose (or
> "state of mind" as I put it). A lot of things I consider misfeatures
> in the whole packaging ecosystem in python are justified by being
> convenience to the developer, at the cost of reliability (at least in
> my opinion).
>
> Data files is my favorite example, as it is deceptively simple
> feature-wise, and yet so confusing. We had 2 emails in the recent days
> on this ML on the topic, and even people very familiar with distutils
> got it wrong. Just considering numpy.distutils/distutils/setuptools,
> there are 5 ways to handle data files at least (data_files,
> package_data, MANIFEST[.in], VCS, add_data_dir). None of them makes it
> easy to install a data file in arbitrary location. I think
> distutils/setuptools lacks a robust way of including data files, but
> it is not as much a feature problem (which could be solved relatively
> easily), as more a problem of what people want.
>
> The devil is really in the details for packaging, which is why
> explicit is a very desirable feature. Someone noticed earlier that
> distutils was the exact contrary of the python zen on some core
> issues: more than one way to do it, lack of simplicity, often
> implicit. I think this describe the problem quite well, and I am
> deeply convinced that that's the root cause of the failure of Pypi for
> a subset of the community.

Well, you have completely failed in explaining why CPAN is fantastic
and PyPI sucks, probably because your answer isn't about CPAN at all,
but just the same old packaging rant. It really grows old you know.

--
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Swig build_py ran before build_ext

2009-12-22T06:36:52Z

Here is feature request: http://bugs.python.org/issue7562

2009/12/22 Tarek Ziadé <ziade.tarek@...>:

> On Tue, Dec 22, 2009 at 10:49 AM, Jari Pennanen <jari.pennanen@...> wrote:
>> Well, if that is so, I will paste my workaround for the sake of
>> completeness if other people come here to wonder:
>>
>> #!/usr/bin/env python
>> from distutils.core import setup, Extension
>> from distutils.command.build_py import build_py
>>
>> dist = setup(...)
>>
>> # Rerun the build_py to ensure that swig generated py files are also copied
>> build_py = build_py(dist)
>> build_py.ensure_finalized()
>> build_py.run()
>>
>> Btw, this could be mentioned in
>> http://docs.python.org/distutils/setupscript.html since SWIG is rather
>> common tool... it took me a notable time to figure this out.
>
> Having an option to define explicitely a custom order for the
> subcommands for the build command in setup.py sounds like a feature we
> could add.
>
> Please add an issue in bugs.python.org, I'll add Marc-André in cc.
> there and we can investigate in this,
>
> Regards
> Tarek
>

--
Tmi: Jari Pennanen
Y-tunnus: 1867270-2
Puh: 050 4911400
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T05:35:15Z

On Tue, Dec 22, 2009 at 2:02 PM, David Cournapeau <cournape@...> wrote:
[..]

>
> Data files is my favorite example, as it is deceptively simple
> feature-wise, and yet so confusing. We had 2 emails in the recent days
> on this ML on the topic, and even people very familiar with distutils
> got it wrong. Just considering numpy.distutils/distutils/setuptools,
> there are 5 ways to handle data files at least (data_files,
> package_data, MANIFEST[.in], VCS, add_data_dir). None of them makes it
> easy to install a data file in arbitrary location. I think
> distutils/setuptools lacks a robust way of including data files, but
> it is not as much a feature problem (which could be solved relatively
> easily), as more a problem of what people want.

I'll speak about distutils only here.

That topic is a good example indeed.

See: http://bugs.python.org/issue2279

1/ Someone added an issue tracker in bugs.python.org because those
data files are
not added unless you add an explicit MANIFEST.in

2/ I worked to change that

3/ it is now included in Python 2.7

So now I am asking you: what do *you* want ?

When you say "which could be solved relatively easily" I suggest that
you take the time to add concise and precise proposals in
bugs.python.org so I can work on them.

And when you say "even people very familiar with distutils got it
wrong" I suggest that you also take the time to propose some doc
fixes, or points its lacking in the tracker.

Distutils has now a maintainer, and that would be me. In two days, it
will be one year exactly since I've taken over the maintenance and
I've done so far 185 commits in the distutils package,
and 15 commits in its documentation and this rate is going to increase
once the PEPs we are building are accepted. Unfortunately I can't
make it a great package in just one year, and there is a huge amount
of work left to be done.

So I suggest that you take the chance to help me fixing the stuff you
don't like in that package by feeding the tracker.

Patches welcome ! ;)

Cheers,
Tarek

--
Tarek Ziadé | http://ziade.org
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T05:02:12Z

On Tue, Dec 22, 2009 at 4:43 PM, Lennart Regebro <regebro@...> wrote:
> On Mon, Dec 21, 2009 at 23:48, David Cournapeau <cournape@...> wrote:
>> mind". Reliable packaging requires explicit handling, where the whole
>> python stack for packaging relies a lot on implicit behavior.
>
> Can you give examples of this implicit behaviour?

>From the top of my head:
- how data files are included
- how sdist is produced (a lot of sdist-generated tarballs contain
lot of junk like *.pyc or egg-info). I think we already had this
discussion, and that I was the only one who thought this was a
problem.
- how files are installed: distutils just copy whatever is in the
build directory for at least some files, so you cannot easily check
which files are installed - hence my proposal on a build manifest to
make this step explicit. This is maybe the number one issue we have in
numpy/scipy where people unfamiliar with distutils warts try to
build/install numpy/scipy.

> It's not worse, it's the same.But it's admittedly a big lack of
> feature in the current Python situation, yes. But it's also not a
> question of "CPAN", is it? CPAN doesn't uninstall things.

Yes, CPAN does not uninstall things, which is why I was making the
difference betwen explicitely uninstally things and (implicitely)
uninstalling beforere reinstalling things. The former would not be
done though CPAN, the later would.

>
> Up until this, you wore concrete and could explain exactly what you
> didn't like. You are partly incorrect and for some reason very angry,
> which is nonconstructive but beside the point. Your claim that it's
> not of features and concrete problems but a "state of mind" was
> countered by your concrete answers. ;-)

Features only help as much as long as you agree on the purpose (or
"state of mind" as I put it). A lot of things I consider misfeatures
in the whole packaging ecosystem in python are justified by being
convenience to the developer, at the cost of reliability (at least in
my opinion).

Data files is my favorite example, as it is deceptively simple
feature-wise, and yet so confusing. We had 2 emails in the recent days
on this ML on the topic, and even people very familiar with distutils
got it wrong. Just considering numpy.distutils/distutils/setuptools,
there are 5 ways to handle data files at least (data_files,
package_data, MANIFEST[.in], VCS, add_data_dir). None of them makes it
easy to install a data file in arbitrary location. I think
distutils/setuptools lacks a robust way of including data files, but
it is not as much a feature problem (which could be solved relatively
easily), as more a problem of what people want.

The devil is really in the details for packaging, which is why
explicit is a very desirable feature. Someone noticed earlier that
distutils was the exact contrary of the python zen on some core
issues: more than one way to do it, lack of simplicity, often
implicit. I think this describe the problem quite well, and I am
deeply convinced that that's the root cause of the failure of Pypi for
a subset of the community.

David
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Swig build_py ran before build_ext

2009-12-22T02:19:46Z

On Tue, Dec 22, 2009 at 10:49 AM, Jari Pennanen <jari.pennanen@...> wrote:

> Well, if that is so, I will paste my workaround for the sake of
> completeness if other people come here to wonder:
>
> #!/usr/bin/env python
> from distutils.core import setup, Extension
> from distutils.command.build_py import build_py
>
> dist = setup(...)
>
> # Rerun the build_py to ensure that swig generated py files are also copied
> build_py = build_py(dist)
> build_py.ensure_finalized()
> build_py.run()
>
> Btw, this could be mentioned in
> http://docs.python.org/distutils/setupscript.html since SWIG is rather
> common tool... it took me a notable time to figure this out.

Having an option to define explicitely a custom order for the
subcommands for the build command in setup.py sounds like a feature we
could add.

Please add an issue in bugs.python.org, I'll add Marc-André in cc.
there and we can investigate in this,

Regards
Tarek
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Swig build_py ran before build_ext

2009-12-22T01:49:22Z

Well, if that is so, I will paste my workaround for the sake of
completeness if other people come here to wonder:

#!/usr/bin/env python
from distutils.core import setup, Extension
from distutils.command.build_py import build_py

dist = setup(...)

# Rerun the build_py to ensure that swig generated py files are also copied
build_py = build_py(dist)
build_py.ensure_finalized()
build_py.run()

Btw, this could be mentioned in
http://docs.python.org/distutils/setupscript.html since SWIG is rather
common tool... it took me a notable time to figure this out.

(Sorry about duplicate mail)

2009/12/22 David Cournapeau <cournape@...>:

> On Tue, Dec 22, 2009 at 7:16 AM, Jari Pennanen <jari.pennanen@...> wrote:
>> Hi!
>>
>> I have old/new problem, once mentioned in 2004 at this same mailing
>> list ( http://mail.python.org/pipermail/distutils-sig/2004-March/003807.html
>> ) but surely that implementation is bad... But the problem persists, I
>> use SWIG and had to Google all over to only find out build order
>> cannot be changed.
>>
>> Someone should implement build_order keyword argument to setup, e.g.:
>>
>> setup(..., build_order=['build_ext', 'build_py'])
>>
>> If some orders are missing, like in above example: build_clib,
>> build_scripts, they would be appended as last. Anyways, the point is
>> make the build order customizable through setup keyword.
>
> This would need to be tested/confirmed on real packages, but I am
> afraid it would break a lot of packages, especially the ones extending
> distutils.
>
> cheers,
>
> David
>

Re: Python people want CPAN and how the latter came about

2009-12-22T01:14:38Z

Hi Lennart,

Lennart Regebro <regebro <at> gmail.com> writes:
> Interesting. I generally don't find automatically generated
> documentation useful, but if it works for PERL I guess that's a
> personal taste.

Let me add two nits here:
It's Perl, not PERL. The name of the language is *not* an acronym. Some people
are really picky about that.

More importantly, it's not "auto-generated" documentation per se. Not something
like using doxygen to generate an API reference purely from the function/method
name and signature. Instead, search.cpan.org (and kobesearch) extract the
inlined documentation from the Perl modules. This documentation was written by
the authors. It was *not* generated from the code itself. Here's what that looks
like for a random one of my modules:
http://search.cpan.org/~smueller/PAR-Repository-Client/lib/PAR/Repository/
Client.pm

You could get the same page via
http://search.cpan.org/perldoc?PAR::Repository::Client. This is an example of
how CPAN works on namespace and distribution level. After a new file is
uploaded, its contained namespaces (packages/class names) are added to the
index if they're not already there. The index always contains a reference to
the distribution that contains the newest *authorized* version of any
namespace. Thus, you can access the distribution directly like in the first
URL or have search.cpan.org go from package name to the most recent
authorized version automatically (via the perldoc link).

> > 4. Namespaces and some way of reserving them. There are likely many
> > modules named postgresql on PyPI, but there's only one DBD::Pg (although
> > there are other PostgreSQL modules that implement the perl DBI driver
> > interface). This also helps with specifying dependencies.
>
> There is no reserving in the Python world, that's true. I'm skeptical.
> There's enough problems with packages not being updated, we don't want
> that for namespaces too, that would be terrible.

We have a band of PAUSE admins with super cow powers to save the day. It's not
like there's a lot of us, but so far, we've been able to handle authorization
requests reasonably. Our policy in case an author goes missing who has
permissions for a certain namespace are documented here:
http://search.cpan.org/CPAN/modules/04pause.html#takeover

> > It definitely appears to have the framework, but lacks some finishing
> > touches that would enormously enhance usability.
>
> So far we have identified a "show all versions" link, and
> automatically generated documentation. Thanks! Finally something
> concrete!

In a very concrete context: I really like how search.cpan.org will show you the
latest stable version by default and provide drop-downs to select any developer
(alpha) or old releases. It also has nice, consistent URLs as demonstrated
above. ...~author will get me to author's overview page, .../dist/XXX will get
me to the XXX distribution's overview, etc. Accessing any files within each
distribution is possible via similar URLs.

Another point that I really like about the service is that the distribution
pages provide links to many other related services that are run by other
volunteers. Take for example http://search.cpan.org/dist/PAR-Repository-Client/
There is a link to cpanforum, specifically the relevant discussion forum for the
module at hand. It shows a link to the bug/request tracker with the number of
open bugs, the one next to it will display a nice hierarchical (recursive) list
of dependencies. Again. Run by another volunteer. The line below is dedicated to
CPAN testers results. It shows the number of FAIL/PASS/UNKNOWN results followed
by a link to the page with the raw reports and -- this is extremely useful as an
author -- a colourful matrix that shows the test results depending on perl
version and operating system. My example in this mail is not a very widely used
module. Thus, the matrix is somewhat sparse, too:
http://matrix.cpantesters.org/?dist=PAR-Repository-Client+0.25. But it clearly
shows my module isn't compatible to really old versions of Perl (5.6). In some
other cases, I might learn that it isn't working well on some operating system
because its column is red or yellow. Here's an example of a more widely used and
installed distribution: http://matrix.cpantesters.org/?dist=Data-Dumper+2.125.
As an author, these tools help me identify weak points or portability problems
in my software. As a user, this can help me understand whether I really want to
rely on this piece of software for my business. Everyone wins.

Please don't take any of this the wrong way. I'm not trying to say "look what we
can do and you don't" at all. Not only do I have no idea what the python tools
look like, but in the previous two paragraphs, I am really just trying to point
out a few technical strong points of various CPAN-related services that are very
valuable to me as a user, developer and contributor.

Best regards,
Steffen

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T00:11:38Z

2009/12/22 Lennart Regebro <regebro@...>:
> On Tue, Dec 22, 2009 at 02:07, David Cournapeau <cournape@...> wrote:
>> But there is much more than what PEP 314 defines. Think about
>> everything which is in an egg, for example. How are files locations
>> are described, how are files outside site-packages described, etc...
>
> What do you mean "how"?

This is answered in my another email (build manifest and eggs).

>
>> As a concrete example, there is not enough metadata in every installer
>> so that you can convert from one to the other, even though this would
>> be very useful and definitely possible.
>
> This I don't understand. What do you mean with "installer"? What kind
> of conversions do you refer to?

By installer, I mean things produced by bdist_*. A significant portion
of windows users don't like eggs, and prefer .exe-based (or
.msi-based) installers. Currently, it is not possible to (reliably)
convert from one to the other (e.g. eggs->wininst), but there is no
reason why this is so.

>
>> For example, you can upload a distribution which does not even have a
>> name, or a version, easy_install tries to find tarballs by scrapping
>> webpages, etc...
>
> Well, if you like your distribution to be called UNKNOWN-0.0, then I
> guess you can. But you also get warnings like this:
>
> warning: sdist: standard file not found: should have one of README, README.txt
> warning: sdist: missing required meta-data: name, url
> warning: sdist: missing meta-data: either (author and author_email) or
> (maintainer and maintainer_email) must be supplied
>
> In my opinion that's your problem of you ignore those

You are not wrong, but we are not talking about the same scenario. The
scenario I care the most is user A build or install a package, and
user B wants to install it; A may know very little about python.

David
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-22T00:02:28Z

On Tue, Dec 22, 2009 at 02:07, David Cournapeau <cournape@...> wrote:
> But there is much more than what PEP 314 defines. Think about
> everything which is in an egg, for example. How are files locations
> are described, how are files outside site-packages described, etc...

What do you mean "how"?

> As a concrete example, there is not enough metadata in every installer
> so that you can convert from one to the other, even though this would
> be very useful and definitely possible.

This I don't understand. What do you mean with "installer"? What kind
of conversions do you refer to?

> For example, you can upload a distribution which does not even have a
> name, or a version, easy_install tries to find tarballs by scrapping
> webpages, etc...

Well, if you like your distribution to be called UNKNOWN-0.0, then I
guess you can. But you also get warnings like this:

warning: sdist: standard file not found: should have one of README, README.txt
warning: sdist: missing required meta-data: name, url
warning: sdist: missing meta-data: either (author and author_email) or
(maintainer and maintainer_email) must be supplied

In my opinion that's your problem of you ignore those, but maybe I'm
wrong. Convince me.

--
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-21T23:43:05Z

On Mon, Dec 21, 2009 at 23:48, David Cournapeau <cournape@...> wrote:
> mind". Reliable packaging requires explicit handling, where the whole
> python stack for packaging relies a lot on implicit behavior.

Can you give examples of this implicit behaviour?

> - distutils (and most tools on top) throw away metadata, and then
> other tools try to guess them back.

What metadata is thrown away?

> tools try to add another level of complexity to circumvent it. The
> fact that python has something like 5 or 6 tools to deploy things
> where most other languages have only one or two is quite striking.

Can you list these 5 or 6 tools to deploy, because I'm only aware of
two, easy_install and pip. And in the current refactoring of
Distribute, easy_install is going away.

> - Linked to the above, metadata are not enforced in pypi. This just
> cannot work. Most other systems in existence enforce strict rules.

It is working, right now, so that's a strange claim. Obviously, you
are less likely to use a package with less metadata, but that's the
package authors problem, not your, or?

> For example, when you
> install a package, the previous installation is not removed. Worse,
> there is no way to correctly uninstall things

It's not worse, it's the same. But it's admittedly a big lack of
feature in the current Python situation, yes. But it's also not a
question of "CPAN", is it? CPAN doesn't uninstall things. It's a
package repository. In any case this is a well known problem being
looked at and in the process of being solved.

> - etc...

Up until this, you wore concrete and could explain exactly what you
didn't like. You are partly incorrect and for some reason very angry,
which is nonconstructive but beside the point. Your claim that it's
not of features and concrete problems but a "state of mind" was
countered by your concrete answers. ;-)

Now if you cool down a bit we probably can find the actual problems. :-)

--
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Specifying eggs and build manifest ?

2009-12-21T19:21:07Z

On Tue, Dec 22, 2009 at 11:06 AM, Tarek Ziadé <ziade.tarek@...> wrote:

> If you are referring to the installation format, Distribute wants to
> drop any extra format in the future and stick with a unique, standard
> format described in PEP 376 to avoid having an heterogeneous
> site-packages.

Does that mean that eggs would be "deprecated" ?

> I like the idea

Cool :)

Here is what I got so far. I have my small own packaging solution, and
a small utility called toymaker, which works as steps (toymaker
configure, toymaker build, etc...). For this discussion, the only
thing to know is that toymaker build produces this build manifest, and
both toymaker install/build_egg read this build manifest to produce
the install target.

Right now, the format I use is ad-hoc, what matters at this point is
the information it describes, and anything is open to changes if
anyone has a better idea on a particular point. There are three
sections:
- A copy of PKG-INFO data in a simple form for easy parsing
- A variable section, containing paths (bindir, libdir, etc..) and executables
- A files section, which contains subsections of files to install.

For example, for sphinx, toymaker build produces this

name=Sphinx
license=BSD
author=Georg Brandl
url=http://sphinx.pocoo.org/
top_levels=sphinx
download_url=http://pypi.python.org/pypi/Sphinx
summary=Python documentation generator
author_email=georg@...
version=0.6.3
platforms=any
install_requires=Pygments>=0.8
install_requires=Jinja2>=2.1
install_requires=docutils>=0.4
classifiers=Development Status :: 4 - Beta
classifiers=Environment :: Console
classifiers=Environment :: Web Environment
classifiers=Intended Audience :: Developers
classifiers=License :: OSI Approved :: BSD License
classifiers=Operating System :: OS Independent
classifiers=Programming Language :: Python
classifiers=Topic :: Documentation
classifiers=Topic :: Utilities
description=Sphinx is a tool that makes it easy to create intelligent
and beautiful
description=documentation for Python projects (or other documents consisting of
description=multiple reStructuredText sources), written by Georg Brandl.
description=It was originally created to translate the new Python documentation,
description=but has now been cleaned up in the hope that it will be
useful to many
description=other projects.
description=
description=Sphinx uses reStructuredText as its markup language, and
many of its strengths
description=come from the power and straightforwardness of
reStructuredText and its
description=parsing and translating suite, the Docutils.
description=
description=Although it is still under constant development, the
following features
description=are already present, work fine and can be seen "in action"
in the Python docs:
description=
description=* Output formats: HTML (including Windows HTML Help),
plain text and LaTeX,
description= for printable PDF versions
description=* Extensive cross-references: semantic markup and automatic links
description= for functions, classes, glossary terms and similar
pieces of information
description=* Hierarchical structure: easy definition of a document
tree, with automatic
description= links to siblings, parents and children
description=* Automatic indices: general index as well as a module index
description=* Code handling: automatic highlighting using the Pygments
highlighter
description=* Various extensions are available, e.g. for automatic
testing of snippets
description= and inclusion of appropriately formatted docstrings.
description=
description=A development egg can be found `here
description=<http://bitbucket.org/birkenfeld/sphinx/get/tip.gz#egg=Sphinx-dev>`_.
!- VARIABLES
paths
pkgname=Sphinx
datarootdir=$prefix/share
eprefix=$prefix
sitedir=$libdir/python$py_version_short/site-packages
libdir=$eprefix/lib
prefix=/usr/local
py_version_short=2.6
gendatadir=$sitedir
datadir=$datarootdir
includedir=$prefix/include
sbindir=$eprefix/sbin
pkgdatadir=$datadir/$pkgname
bindir=$eprefix/bin
mandir=$datarootdir/man
executables
sphinx-build=sphinx:main
sphinx-quickstart=sphinx.quickstart:main
sphinx-autogen=sphinx.ext.autosummary.generate:main
!- FILELIST
executables:sphinx-build
srcdir=build/scripts-2.6
target=$bindir
sphinx-build
executables:sphinx-quickstart
srcdir=build/scripts-2.6
target=$bindir
sphinx-quickstart
executables:sphinx-autogen
srcdir=build/scripts-2.6
target=$bindir
sphinx-autogen
datafiles:gendata_sphinx_locale
...

The file contains just enough data to build egg and install on disk
ATM, and I include the build manifest in the produced egg so that
conversion to wininst-based (or other types of) installers is
possible. For the file list, each section has the same structure:
- type:name -> type would be used for post-install (byte compile
python files, chmod binaries, etc...), name is not really important,
but it enables having several subsections of the same type (so that
different target/srcdir are possible)
- the next two lines are srcdir and target: target can be a variable
as defined in the path section (this is important to "relocate"
installs in some cases), srcdir is such as every file of the section
can be found in srcdir/file.

Although I have not thought about the problem yet, it should also be
usable for uninstall, and for on-disk query of installed packages. I
should note at this point that it is mostly a rip-off of one concept
in cabal I really like, the installed package information, which is
used for those purposes as well:

http://www.haskell.org/cabal/release/cabal-latest/doc/API/Cabal/Distribution-InstalledPackageInfo.html

> Distribute 0.7 is still under heavy construction, and the goals we have
> about the installation features are not fully defined yet. What we want for
> sure is to make it the laboratory of Distutils to improve things in smaller
> cycles than python. And having a better build description sounds like
> something we could use/have.

I could start implementing it within distutils/Distribute, at least
for the parts which I guess will be the least controversial (the FILE
section and the path variables), and specify the exact file format.

I think it should be possible for the install command of distribute to
only rely on this file - this would actually be a good test, comparing
installs with the old and the new way.

David
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Specifying eggs and build manifest ?

2009-12-21T18:48:39Z

On Tue, Dec 22, 2009 at 11:42 AM, P.J. Eby <pje@...> wrote:
> At 10:24 AM 12/22/2009 +0900, David Cournapeau wrote:
>>
>> 1 Formally specifying the egg format (and versioning it !) - or is
>> egg format outside distribute goal ?
>
> FYI: http://peak.telecommunity.com/DevCenter/EggFormats

I know about this page, but it says

"Instead, this is internal documentation for how those concepts and
features are implemented in concrete terms. It is intended for people
who are working on the setuptools code base, who want to be able to
troubleshoot setuptools problems, want to write code that reads the
file formats involved, or want to otherwise tinker with
setuptools-generated files and directories."

So unless this description is not accurate anymore, it is not a
specification. I want to build eggs, without having distutils (and
setuptools) involved at all. I would like to be able to do so while
staying compatible with the existing infrastructure, so that if
someone use my own tool instead of setuptools to produce an egg, it
still works as expected with virtualenv, buildout, etc...

David
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Specifying eggs and build manifest ?

2009-12-21T18:42:49Z

At 10:24 AM 12/22/2009 +0900, David Cournapeau wrote:
> 1 Formally specifying the egg format (and versioning it !) - or is
>egg format outside distribute goal ?

FYI: http://peak.telecommunity.com/DevCenter/EggFormats

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-21T18:16:26Z

On Tue, Dec 22, 2009 at 2:58 AM, David Cournapeau <cournape@...> wrote:

> On Tue, Dec 22, 2009 at 10:50 AM, Tarek Ziadé <ziade.tarek@...> wrote:
>
>> Two things:
>>
>> 1/ you get warnings if you try to build a distribution with missing metadata
>> and there's now a command in 2.7 called "check" so you can set a
>> strict control
>> on client side
>
> This is nice, I missed this new check feature. This is definitely a
> step in the right direction.
>

more on this:
http://docs.python.org/dev/distutils/examples.html#checking-a-package
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Specifying eggs and build manifest ?

2009-12-21T18:06:58Z

On Tue, Dec 22, 2009 at 2:24 AM, David Cournapeau <cournape@...> wrote:
> Hi,
>
> I wonder if there is any interest in the current distribute effort to
> specify some low-level commonalities outside the distutils code for
> interoperation. In particular, I had two things in mind:
> 1 Formally specifying the egg format (and versioning it !) - or is
> egg format outside distribute goal ?

If you are referring to the installation format, Distribute wants to
drop any extra format in the future and stick with a unique, standard
format described in PEP 376 to avoid having an heterogeneous
site-packages.

> 2 Formally specifying something like a build manifest which would be
> used for installation.
>
> I think the first point is self-explanatory. Concerning the second
> point, the current way of installing things in distutils is a bit too
> ad-hoc to my taste, and could be fixed inside distutils to make it
> easier to interoperate with other methods (be it native OS methods or
> other python-related tools). Right now, distutils install things from
> a list of files, but this only contains the target location for some
> kind of files. What about the following kind of format instead (or in
> complement for backward-compatibility):
>
> - split the flat list into a list of "typed" sections, where type
> would be e.g. data, python files, extensions, scripts, etc...
> - for each section, specify both the source and target directory.
> - the format should be easy to parse/produce, and versioned.

I like the idea

>
> The rationale is that such a format would enable the following:
> - it becomes easier to post-process files at install time, with
> different methods for each type
> - such a format could form a basis to produce "conventional" install,
> eggs and other kind of installers, including convertion between them.
>
> Assuming this idea fits into what people involved in distribute are
> doing, I would be willing to propose a more formal description - I
> already have the implementation in my own project toydist, where I use
> build manifest to do explicit install and eggs/wininst conversion.

Distribute 0.7 is still under heavy construction, and the goals we have
about the installation features are not fully defined yet. What we want for
sure is to make it the laboratory of Distutils to improve things in smaller
cycles than python. And having a better build description sounds like
something we could use/have.
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Swig build_py ran before build_ext

2009-12-21T18:04:05Z

On Tue, Dec 22, 2009 at 7:16 AM, Jari Pennanen <jari.pennanen@...> wrote:

> Hi!
>
> I have old/new problem, once mentioned in 2004 at this same mailing
> list ( http://mail.python.org/pipermail/distutils-sig/2004-March/003807.html
> ) but surely that implementation is bad... But the problem persists, I
> use SWIG and had to Google all over to only find out build order
> cannot be changed.
>
> Someone should implement build_order keyword argument to setup, e.g.:
>
> setup(..., build_order=['build_ext', 'build_py'])
>
> If some orders are missing, like in above example: build_clib,
> build_scripts, they would be appended as last. Anyways, the point is
> make the build order customizable through setup keyword.

This would need to be tested/confirmed on real packages, but I am
afraid it would break a lot of packages, especially the ones extending
distutils.

cheers,

David
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-21T17:58:48Z

On Tue, Dec 22, 2009 at 10:50 AM, Tarek Ziadé <ziade.tarek@...> wrote:

> Two things:
>
> 1/ you get warnings if you try to build a distribution with missing metadata
> and there's now a command in 2.7 called "check" so you can set a
> strict control
> on client side

This is nice, I missed this new check feature. This is definitely a
step in the right direction.

> Are you referring to the install schemes ? these schemes are described
> in the stdlib itself, but are not as detailed as what you could find
> in Debian for sure.

I have started a new thread with a more details explanation, since
this is not directly related to PyPi (although it is important to make
installing from Pypi more robust IMHO).

cheers,

David
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-21T17:50:38Z

On Tue, Dec 22, 2009 at 2:07 AM, David Cournapeau <cournape@...> wrote:
[..]
>
>> The "metadata", as defined in PEP 314,
>> are not thrown away by Distutils, but properly uploaded at PyPI.
>
> But there is much more than what PEP 314 defines.

Then that's not the metadata as defined in python packaging, but some
other information
you are talking about.

>Think about
> everything which is in an egg, for example. How are files locations
> are described, how are files outside site-packages described, etc...
> As a concrete example, there is not enough metadata in every installer
> so that you can convert from one to the other, even though this would
> be very useful and definitely possible.
> So it is not just metadata at the "input" stage (although there is
> certainly a lot of things lost there too), but how they are
> represented internally, and how they are represented in the various
> installers. This matters a lot for people who extend distutils.

I think you are talking about internal representation of elements in an archive
built by distutils here, which is different from the metadata that describes the
project/release and pushed at PyPI.

There's no need to publish and share those extra information unless you want
to create some kind of converter.

What's important is to clearly define the metadata of a project, as published
at PyPI and installed on your system. Not the internal cooking of each
installer.

PEP 376 as a matter of fact, provides API to read/write the metadata
of installed
distributions.

Now, for extending distutils, that's another problem: we already said
that distutils
was hard to extend, and we have proposed some changes for this (for example
to have a configure/build/install scheme), but this has nothing to do
with the published
metadata.

As a matter of fact, I am working on a configure command for
distutils, looking at 4suite's work
I've started a prototype here :
http://bitbucket.org/tarek/distutils-configure (early stage)
you are welcome to join :)

>
>>
>>> - Linked to the above, metadata are not enforced in pypi. This just
>>> cannot work. Most other systems in existence enforce strict rules.
>>
>> I am not sure what you mean by enforcing strict rules. You need to be
>> more precise.
>
> For example, you can upload a distribution which does not even have a
> name, or a version, easy_install tries to find tarballs by scrapping
> webpages, etc...

Two things:

1/ you get warnings if you try to build a distribution with missing metadata
and there's now a command in 2.7 called "check" so you can set a
strict control
on client side

2/ easy_install doesn't "scrap" webpages but uses a protocol implemented
by PyPI - see
http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api

>
>>
>> What do you want to do ? reject any package that doesn't meet some QA tests ?
>> Is PyPI is a community repository, or an entreprise repository ?
>
> That's common sense to reject packages which do not define a basic set
> of metadata, and this has nothing to do with enterprise vs community.
> Debian, for example, enforces metadata in its package, and it does not
> qualify as enterprise software to me if the term is meant as the
> opposite of community.

I am even opposing PyPI to debian packages because PyPI allows
anyone to upload some python code to share, even if it doesn't have
all the required
metadata or QA. Installers are not the only PyPI end users right now.

IOW, you don't have to be a professional packager to publish and share
code at PyPI.
That's how I understand its philosophy and I think it goes pretty well
with Python's own
philosophy.

>
>> I would have understood this frustration a year ago, but now, that's
>> just a free rant/flame :)
>
> I have tried to explain several times about describing how files are
> installed where, and everytime I was shut down (not by you). In
> particular, I think I have explained already why I believe PEP 376
> does not solve the issue of interoperation with other installers
> because it loses source/target directory information.

Are you referring to the install schemes ? these schemes are described
in the stdlib itself, but are not as detailed as what you could find
in Debian for sure.

And these scheme only concerns Python locations, because other locations
were a bit out of distutils scope. But some people are working on extending
that in PEP 376 so we can point stuff like man pages etc.

But distutils will never ever compete with debian's dpkg or fedora's
rpm if you need
a full blown packaging system for a given platform. Rather, the best thing we
can do is to make sure those packagers can easily extract info out of
distutils distributions;

if you think this can be improved, if you have examples of info loss,
as you said
that's the perfect thing to throw in the PEP discussions. And if you
already did some months
ago, I am sorry, I think you will have to do it again because that
just means imho that
the "distutils community" was not ready to hear it back then. But it
grew up a little bit since then
so I think it worth a try.

Tarek

--
Tarek Ziadé | http://ziade.org
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Specifying eggs and build manifest ?

2009-12-21T17:33:28Z

Just use buildout...

On Tue, 22 Dec 2009 10:24:07 +0900, David Cournapeau <cournape@...>
wrote:

> Hi,
>
> I wonder if there is any interest in the current distribute effort to
> specify some low-level commonalities outside the distutils code for
> interoperation. In particular, I had two things in mind:
> 1 Formally specifying the egg format (and versioning it !) - or is
> egg format outside distribute goal ?
> 2 Formally specifying something like a build manifest which would be
> used for installation.
>
> I think the first point is self-explanatory. Concerning the second
> point, the current way of installing things in distutils is a bit too
> ad-hoc to my taste, and could be fixed inside distutils to make it
> easier to interoperate with other methods (be it native OS methods or
> other python-related tools). Right now, distutils install things from
> a list of files, but this only contains the target location for some
> kind of files. What about the following kind of format instead (or in
> complement for backward-compatibility):
>
> - split the flat list into a list of "typed" sections, where type
> would be e.g. data, python files, extensions, scripts, etc...
> - for each section, specify both the source and target directory.
> - the format should be easy to parse/produce, and versioned.
>
> The rationale is that such a format would enable the following:
> - it becomes easier to post-process files at install time, with
> different methods for each type
> - such a format could form a basis to produce "conventional" install,
> eggs and other kind of installers, including convertion between them.
>
> Assuming this idea fits into what people involved in distribute are
> doing, I would be willing to propose a more formal description - I
> already have the implementation in my own project toydist, where I use
> build manifest to do explicit install and eggs/wininst conversion.
>
> cheers,
>
> David
> _______________________________________________
> Distutils-SIG maillist - Distutils-SIG@...
> http://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Specifying eggs and build manifest ?

2009-12-21T17:24:07Z

Hi,

I wonder if there is any interest in the current distribute effort to
specify some low-level commonalities outside the distutils code for
interoperation. In particular, I had two things in mind:
1 Formally specifying the egg format (and versioning it !) - or is
egg format outside distribute goal ?
2 Formally specifying something like a build manifest which would be
used for installation.

I think the first point is self-explanatory. Concerning the second
point, the current way of installing things in distutils is a bit too
ad-hoc to my taste, and could be fixed inside distutils to make it
easier to interoperate with other methods (be it native OS methods or
other python-related tools). Right now, distutils install things from
a list of files, but this only contains the target location for some
kind of files. What about the following kind of format instead (or in
complement for backward-compatibility):

- split the flat list into a list of "typed" sections, where type
would be e.g. data, python files, extensions, scripts, etc...
- for each section, specify both the source and target directory.
- the format should be easy to parse/produce, and versioned.

The rationale is that such a format would enable the following:
- it becomes easier to post-process files at install time, with
different methods for each type
- such a format could form a basis to produce "conventional" install,
eggs and other kind of installers, including convertion between them.

Assuming this idea fits into what people involved in distribute are
doing, I would be willing to propose a more formal description - I
already have the implementation in my own project toydist, where I use
build manifest to do explicit install and eggs/wininst conversion.

cheers,

David
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Finishing PEP 345

2009-12-21T17:13:56Z

> FYI. This Distutils-SIG thread is about proposing PEP 345 and impacts PyPI.
> So if there's anything that look suspicious to anyone, please join the
> discussion at Distutils-SIG

In a number of places (Requires-Dist, Requires-Python), comma-separated
lists of version constraints are used. The PEP needs to specify what the
collective constraint means that gets specified.

Most likely, the intended meaning is that the constraints get
and-combined, in which case the example

Requires-Python: 2.5, 2.6

is non-sensical (no Python release meets that constraint). (else,
if it was meant to denote or-combination, then ">1.0, !=1.3.4, <2.0"
would be non-sensical, specifying no constrating at all)

For the Copyright field, I'm not sure what the purpose of it is.
If it is what I think it is (listing attribution), it should probably
be specified as "multiple use".

For Documentation, I think the entire field must be reconsidered.
If it is really meant to be reStructuredText, then the spec should
explain how to do leading spaces/indentation.

For Platform, I fail to see the point of supporting both multiple
use, and comma-separated lists.

For Metadata-Version, I think formally, the only legal value according
to the PEP is 1.2. If 1.0 and 1.1 are also conforming values, the PEP
should elaborate what it means to put a different version number into
the field.

Regards,
Martin
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-21T17:07:07Z

On Tue, Dec 22, 2009 at 9:04 AM, Tarek Ziadé <ziade.tarek@...> wrote:

> On Mon, Dec 21, 2009 at 11:48 PM, David Cournapeau <cournape@...> wrote:
>> On Mon, Dec 21, 2009 at 7:13 PM, Lennart Regebro <regebro@...> wrote:
>>> What nobody still fails to explain in this discussion is what CPAN
>>> "is" and Why Python doesn't already have it.
>>
>> That's not the right question to ask. The problem is not much a
>> feature problem as much as a fundamental implementation and "state of
>> mind". Reliable packaging requires explicit handling, where the whole
>> python stack for packaging relies a lot on implicit behavior. I don't
>> know much about CPAN, but both CRAN ("CPAN for R") and hackage ("CPAN
>> for haskell") work reliably where Pypi often fails, and the reasons
>> are easy to see:
>>
>> - distutils (and most tools on top) throw away metadata, and then
>> other tools try to guess them back. That alone is source of numerous
>> errors, weird hacks, and unexplainable issues. Generally, when there
>> is a design error at some level, instead of fixing it at this level,
>> tools try to add another level of complexity to circumvent it. The
>> fact that python has something like 5 or 6 tools to deploy things
>> where most other languages have only one or two is quite striking.
>
> I think you are quite confused here.

I really don't think I am.

> The "metadata", as defined in PEP 314,
> are not thrown away by Distutils, but properly uploaded at PyPI.

But there is much more than what PEP 314 defines. Think about
everything which is in an egg, for example. How are files locations
are described, how are files outside site-packages described, etc...
As a concrete example, there is not enough metadata in every installer
so that you can convert from one to the other, even though this would
be very useful and definitely possible.

So it is not just metadata at the "input" stage (although there is
certainly a lot of things lost there too), but how they are
represented internally, and how they are represented in the various
installers. This matters a lot for people who extend distutils.

>
>> - Linked to the above, metadata are not enforced in pypi. This just
>> cannot work. Most other systems in existence enforce strict rules.
>
> I am not sure what you mean by enforcing strict rules. You need to be
> more precise.

For example, you can upload a distribution which does not even have a
name, or a version, easy_install tries to find tarballs by scrapping
webpages, etc...

>
> What do you want to do ? reject any package that doesn't meet some QA tests ?
> Is PyPI is a community repository, or an entreprise repository ?

That's common sense to reject packages which do not define a basic set
of metadata, and this has nothing to do with enterprise vs community.
Debian, for example, enforces metadata in its package, and it does not
qualify as enterprise software to me if the term is meant as the
opposite of community.

Those metada are needed, because other tools do use them (e.g.
easy_install, and I guess pip as well).

> I would have understood this frustration a year ago, but now, that's
> just a free rant/flame :)

I have tried to explain several times about describing how files are
installed where, and everytime I was shut down (not by you). In
particular, I think I have explained already why I believe PEP 376
does not solve the issue of interoperation with other installers
because it loses source/target directory information.

David
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-21T16:58:02Z

On Tue, 22 Dec 2009 07:48:55 +0900, David Cournapeau <cournape@...>
wrote:

> (cpan) ... is not much a
> feature problem as much as a fundamental implementation and "state of
> mind".

Yes. Their state of mind is make it simple and make it work across
the board for everything for everyone.

> Reliable packaging requires explicit handling, where the whole
> python stack for packaging relies a lot on implicit behavior.

Yep.. That's it in a nutshell.

> (distutils) .. is a design error ... , instead of fixing
> it at this level, tools try to add another level of complexity
> to circumvent it.

This now begs the beer-drinking style (cpan/perl developers understand)
question of what to actually do about this in 2010.

One thing I'm pretty certain of is that people would prefer to
accomplish something and after that drink beer together. That
was really the secret of the cpan success. They loved the
sense of accomplishment. They loved bragging too - but it
was always a welcoming type of bragging that one could easily
fall in to and feel a part of.

After reading your email David, you leave the reader with no
other conclusion other than a new explicit build tool is
needed for 2010.

Otherwise, we're all going to be sitting helplessly reading
the same old 'how can I tell distutils I want to ...?" for
the whole of the next decade.

At the same time, there won't be much of an effort to
address the design issues you talk about. Simply because
of legacy software syndrome management issues.

What can be done, if there is good delegation of the
tasks, is for a new pypi toolset (let's not fragment
things and confuse new users with lots of different
names) to be started.

At the highest level, the toolset needs to:

- build python packages/apps from explicit metadata

- upload and register them

- download and install (and the other functions)

I'd toss in a few extra things not in cpan. Like
incorporating buildout support. I don't use it myself
but so many others do.

The bigger issue is about deployment. It's about
getting python apps/libraries out to all the machines
out there with as little effort as possible.

The Package concept is based on a static snapshot
idea.

But how much python software is now in mercurial
or bzr repositories ? lots of times software is
slow moving. Yes, we do want that latest bug
fix. Say it didn't work.. jump back a revision
or two.

Is a 'package' really a good way to deploy? I'm
not against them, just suggesting that a
repository link in pypi metadata is an equally
good but perphaps more modern way.

What is python doing or going to do to help us
manage all these deployment issues. At some point
in time the stdlib has got to address lots of
'stuff' landing up on the end machine.

Batteries included has just got to include
built in bzr or mercurial or cvs or svn
support.

Just where I work I've written enough mini apps
for different machines deployment is getting
to be a challenge. Most have scm. But setup from
scratch is still un-optimal imo.

How does our fundamental distutils design cope
with the evolving landscape? including supercomputer
and server farm type requirements. Also just plain
commercial and scientific (workstation) type
implementations.

My list of future packaging/configuration/
deployment projects to interoperate with are
(but not limited to):

- buildout

- celery (http://pypi.python.org/pypi/celery/0.3.0)

- pip/distribute/setuptools

- bzr/svn/hg etc

Add to that the web frameworks..

Anyway, you've totally convinced me it's time for
a new pypi package/app build tool..

I would like to also suggest that it be built
upon some small python web framework like cherrypy
and fire up a browser for the user to work with.

I don't want another text based command line
program sorry. Maybe it is just a django app.

Anybody adventurous enough to build a package or
an app isn't going to mind installing a small
web app to run on port 8080 or whatever to do
their package build - imo.

Cheers.. drink and be merry

David

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

Re: Python people want CPAN and how the latter came about

2009-12-21T16:04:50Z

On Dec 21, 2009, at 5:48 PM, David Cournapeau wrote:

> On Mon, Dec 21, 2009 at 7:13 PM, Lennart Regebro <regebro@...> wrote:
>> What nobody still fails to explain in this discussion is what CPAN
>> "is" and Why Python doesn't already have it.
>
> That's not the right question to ask. The problem is not much a
> feature problem as much as a fundamental implementation and "state of
> mind". Reliable packaging requires explicit handling, where the whole
> python stack for packaging relies a lot on implicit behavior.

It is definitely the right question to ask, and it is very much a feature problem.

The missing feature is "install what I mean". easy_install is missing by default in most cases, and then broken by default when you install it. If a fresh python install, you cannot just type "easy_install foo", or even 'cd foo; python setup.py install' and reliably get a copy of 'foo' installed for whatever user asked for it. Instead you get piles of cryptic error messages until you learn how to use it and what command-line options to pass.

Now, CPAN is not perfect, and you can most definitely get cryptic error messages out of it. But, *most of the time*, it just works, and *most of the time*, developers can install dependencies and users can install software without really thinking about it too hard.

Everything you're saying about mindset and standardization may be good, and in fact entirely necessary to achieving this goal. But it is very important that, as a community, we:

A) keep our eyes on the prize, and try to improve the default, out-of-the-box Python package installation experience wherever possible, and
B) be clear about what the prize _is_. It's really important to nail down what it is that we all agree needs to improve. I say this because if someone wants to ask a question like "what is this thing that everyone seems to say we should work on", I think it's important to answer it.

In one sense of "not a feature problem", I think you're right. The problem here is not a particular *advanced* feature, some more sophisticated option, although many features might help fix it: the problem is that the user experience of existing functionality is bad.

We're not hearing a lot of lucid articulation of what exactly the "CPAN problem" is, and I believe the reason is that when you actually look at the problems and describe them, they're easy to work around. setup.py install doesn't work for users who aren't root? Well, maybe that's not really a feature problem, it's a documentation problem. For most of us it's pretty easy to set your PATH and PYTHONPATH and type --prefix and --single-version-externally-managed, and then everything works fine. Or use something like virtualenv. Or you can just use Python 2.6 and set only your PATH, as long as you know that Windows keeps things in one directory layout (%APPDATA%/Python) and POSIX another (~/.local/lib/site-packages).

_______________________________________________
Distutils-SIG maillist - Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig