Distribute sprint #1 wrapup

[Tarek Ziadé]

As promised, here's a quick wrapup of the work started during the
first sprint, on Distribute 0.7

Thanks to all the people that have participated (we were 6 or 7 IIRC)

If you have sprinted and see a mistake or something missing is the
wrapup, please complete it;

= The fate of 0.6 code =

First of all, we've discussed what is happening to the 0.6 code and
we've decided that it was better to keep it
under the pillow as a reference code, and code everything from
scratch. The only code that
is going to be recycled is some parts of pkg_resources.

= 0.7 philosophy =

Distribute 0.7 will be a good Distutils citizen and wants to be its incubator.
It will implement Distutils commands, without changing Distutils
behavior as it is doing now.

Another important point is that the Distribute development will be
driven by the PEP work that is
going on and drop most other concepts of Setuptools:

- PEP 376 : a *single* installation standard
- PEP 345 : Requires, Obsoletes, etc, fields
- PEP 390 : platform-dependant markers for the metadata
- PEP 382 : namespace packages
- PEP 386 : a new version scheme

Distribute will implement those PEPs by gathering all the prototypes
that have been created in the past
months and when possible, will try to push them in Distutils.

The biggest drops we are making are:
- the egg formats are removed in favor of a unique PEP 376 standard.
- easy_install is removed, and Distribute will cooperate with the pip project

What's unclear right now is what will be pushed in Distutils by the
time Python 2.7 and 3.3 are out.

= what was started during the sprint =

We started to write Distutils commands:

- develop command:
    * Created a skeleton command for develop
    * Waiting for work to complete on the install and build_egg_info
commands before work can continue here.

- build_egg_info command (in setuptools, it was called "egg_info",
we've renamed it):

    * Rewrote this command using setuptools code as a model
    * Removed all SVN support code
    * Removed obsolete egg-info writers
    * Waiting on the manifest generation feature to finish this one
    * Will implement PEP 376 egg-info writers when possible

- generate_manifest command. Not started yet, but will be used to
generate the MANIFEST file, and provide a plugin
  system for VCS support. (not an implicit system like in Setuptools)

- install command
  - building a PEP 376 compatible install command (by rewriting
pkg_resources, using the PEP 376 prototype we have written some time
ago)

- distutils.extensions : lightweight entry points, based on the PEP
376 standard taken back from the "extensions" project

 -  Currently working on testing strategy

= next sprint =

The next sprint will be next week-end, and will be announced here;
Anyone can join !


Thanks
Tarek

--
Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Eric Smith-22]

Tarek Ziadé wrote:

<exciting stuff deleted>

> What's unclear right now is what will be pushed in Distutils by the
> time Python 2.7 and 3.3 are out.

Why 3.3 and not 3.2?

Eric.
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Tarek Ziadé]

On Sat, Oct 24, 2009 at 12:50 AM, Eric Smith <eric@...> wrote:
> Tarek Ziadé wrote:
>
> <exciting stuff deleted>
>
>> What's unclear right now is what will be pushed in Distutils by the
>> time Python 2.7 and 3.3 are out.
>
> Why 3.3 and not 3.2?

Oops: that was 3.2 of course ! All releases that we should have next summer,

Tarek
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[kiorky]

Note about pep376
::
.egg-info becomes a directory

As explained earlier, the EggFormats standard from setuptools proposes two
formats to install the metadata information of a distribution:

    * A self-contained directory that can be zipped or left unzipped and
contains the distribution files and an .egg-info directory containing the metadata.
    * A distinct .egg-info directory located in the site-packages directory,
with the metadata inside.

This PEP proposes to keep just one format and make it the standard way to
install the metadata of a distribution : a distinct .egg-info directory located
in the site-packages directory, containing the metadata.


There is something i don't understand there:
Does it imply that having no access on site-packages prevent you from installing
third party libraries as a limited user? For example, on shared hosting where
you have limited privileges. Before, i could upload/build some libs and start
from there using, for exemple, buildout to scan that directory.
I make assumption that also the entry points are written along the metadata dir.
That will say that even if i don't have access to the site-packages but i have
played with sys.path to include my distribution, i won't have its entry points
(as the egg-info is not there) ?


Tarek Ziadé a écrit :

> On Sat, Oct 24, 2009 at 12:50 AM, Eric Smith <eric@...> wrote:
>> Tarek Ziadé wrote:
>>
>> <exciting stuff deleted>
>>
>>> What's unclear right now is what will be pushed in Distutils by the
>>> time Python 2.7 and 3.3 are out.
>> Why 3.3 and not 3.2?
>
> Oops: that was 3.2 of course ! All releases that we should have next summer,
>
> Tarek
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG@...
> http://mail.python.org/mailman/listinfo/distutils-sig

--
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF



_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Tarek Ziadé]

On Sat, Oct 24, 2009 at 7:57 AM, kiorky <kiorky@...> wrote:
>
> This PEP proposes to keep just one format and make it the standard way to
> install the metadata of a distribution : a distinct .egg-info directory located
> in the site-packages directory, containing the metadata.
>
>
> There is something i don't understand there:
> Does it imply that having no access on site-packages prevent you from installing
> third party libraries as a limited user?

As it s today, site-packages is just the default place, that is added and scan
when Python starts. And you need root privileges to install packages there.

There's another place where you can use when you are not root:
per-user site packages.

But those two place are just the places for "python-the-package-manager".

> For example, on shared hosting where
> you have limited privileges. Before, i could upload/build some libs and start
> from there using, for exemple, buildout to scan that directory.
> I make assumption that also the entry points are written along the metadata dir.
> That will say that even if i don't have access to the site-packages but i have
> played with sys.path to include my distribution, i won't have its entry points
> (as the egg-info is not there) ?

PEP 376 comes with a set of tools that will allow you to
install/uninstall distributions
in an arbitrary site-packages folder, and play with them. So it
basically makes almost no
differences for tools like zc.buildout-the-package-manager that is
tweaking sys.path in
the generated scripts.

I guess the simplest way will be to make the "eggs" directory a
regular site-package
like folder.  It will even simplify the scripts because you will not
have to add one entry per eggs
there as it is today.

What could be awesome is to see a branch of zc.buildout built against
distribute 0.7
when it starts to be usable, to experiment this.

zc.buildout is a package manager, so it makes it one of the target use
case for PEP 376
and other changes we will provide in distribute.

Tarek
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[kiorky]

Tarek Ziadé a écrit :
> As it s today, site-packages is just the default place, that is added and scan
> when Python starts. And you need root privileges to install packages there.
>
> There's another place where you can use when you are not root:
> per-user site packages.
>
> But those two place are just the places for "python-the-package-manager".
>
Correct me if i am wrong but what i have understood is that the egg-info is not
writed along with the "python code" but in another well known place.
From then, how can i do the following as i don't want to be defaulted.
        1. register directoy(ies) where to find distributions.
        2. Register where to install the "python code" (new egg format).
        3. Register where to install "egg-infos"
        4. Be sure that both egg-infos from default places (system, home) AND my
thirdparty place(?) are gathered together if possible. Or maybe just one place
is possible ? In this case, we may need to provide means to assemble the whole
installation to reproduce how things behave today.

Maybe, we can make some additional writing on the PEP to make things clear on
that point, i think that just things need to be clear.

On the same subject, I have another reguard,
It's for eggs actually released as only eggs on pypi, see [1].
How can i install them now ? Can we make wrappers to install them?

I have at least two hosting services that give me access to /var/foo, but the
user has no right either on python installation or its $HOME.

[1] - http://pypi.python.org/pypi/Plomino/1.4

> Tarek

--
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF



_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[kiorky]

Whow sorry, Ignore my precedent mail, i missed the end of that email., i do not
know why. Blame my eyes or thunderbird :/

> PEP 376 comes with a set of tools that will allow you to
> install/uninstall distributions
> in an arbitrary site-packages folder, and play with them. So it
> basically makes almost no
> differences for tools like zc.buildout-the-package-manager that is
> tweaking sys.path in
> the generated scripts.
Great (really) :) Entry points are working as well on those non standard places
also, aren't they?

> I guess the simplest way will be to make the "eggs" directory a
> regular site-package
> like folder.  It will even simplify the scripts because you will not
> have to add one entry per eggs
> there as it is today.
I don't think it is a really good idea because you will not have isolation at
project level. If you have A, B, C in this alternative site-packages, it would
mean that A, B, and C are importable. But when i install something with "eggs=A
C", i want that my pythonpath contains A and C but no B.I think the buildout way
to do that actually is not that bad :)

> What could be awesome is to see a branch of zc.buildout built against
> distribute 0.7
Indeed. I ll rework the minitage recipes also to use pip and prepare work for
distribute on a branch as soon a possible, i hope next we.

> when it starts to be usable, to experiment this.
>
> zc.buildout is a package manager, so it makes it one of the target use
> case for PEP 376
> and other changes we will provide in distribute.
>
How about trying to make a mirror of pypi packages rebuilt in the distribute
format. That will prevent the overhead for users to contact the maintainer in
case or breakage, to the maintainer to have to repackage its already packaged
things, and also for the user to have problems installing 'foo' even it is a
very simple package that we could just provide a rebuilt version to him.
I think about that mainly for distributions packaged only as eggs (in the actual
format). We could maybe extract this list of packages with no sdist (not that
much i think and the only one i know is plomino) and repackage them. That's very
low priority, but it can be useful and one more other step which will make the
transition as transparent as possible.

So mainly because i did not find time yet to test, and i will, my only concern
WHICH MAY BE UNFOUNDED is on namespaces conflicts between setuptools and
distribute. For me, the first one loaded module wins. And, it's low priority any
way i think as a patch may by easy to do.

Just one thing to borrow from the precedent mail would be to make one sentence
on the PEP to write in rock that we can make installation elsewhere that in
standard place for both python code & metadata.

--
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF



_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig