distribute and buildout: best practices?

[Reinout van Rees]

What's the current best way to use both distribute and buildout?  Apart from
the bootstrap + apparently needed change in buildout, I've got the following
questions:

- Do my libraries have to list a dependency on distribute or on setuptools?
  Everything zopish has a 'setuptools >= 0.6c9' in it.  I tried putting
  distribute in there, but I seemed to get less problems with leaving it at
  setuptools.

- Should just globally installing distribute be enough?  Everything ought to
  keep working.  It does lead to a bunch of runtime warnings in buildout,
  however.  (But Tarek has targeted that problem for the upcoming 0.6.4 :-)

- Do I need to change buildout's bootstrap already? Doing that tries to change
  my global setuptools, for which you need to sudo.  And it also forces
  everyone else using that buildout to switch (which in itself is OK).

- From my experiments (not exhaustive! no real debugging!) it seems that the
  only safe way is to first install distribute globally before touching a
  buildout that brings in distribute.  Is this to be expected?


Reinout

--
Reinout van Rees - reinout@... - http://reinout.vanrees.org
Software developer at http://www.thehealthagency.com
"Military engineers build missiles. Civil engineers build targets"

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Tarek Ziadé]

On Wed, Oct 7, 2009 at 1:23 PM, Reinout van Rees <reinout@...> wrote:
> What's the current best way to use both distribute and buildout?  Apart from
> the bootstrap + apparently needed change in buildout, I've got the following
> questions:
>
> - Do my libraries have to list a dependency on distribute or on setuptools?
>  Everything zopish has a 'setuptools >= 0.6c9' in it.  I tried putting
>  distribute in there, but I seemed to get less problems with leaving it at
>  setuptools.
>

No. Once we've resolved the problem you've mentioned in the hardcoded setuptools
dependency, you will have an empty Setuptools 0.6c9 egg in eggs/
so libs that ask for this dependency will  think it's met.

> - Should just globally installing distribute be enough?  Everything ought to
>  keep working.  It does lead to a bunch of runtime warnings in buildout,
>  however.  (But Tarek has targeted that problem for the upcoming 0.6.4 :-)

We still require a specific zc.buildout bootstrap.py, otherwise you
will have a local
setuptools egg that will override any global installation.

>
> - Do I need to change buildout's bootstrap already? Doing that tries to change
>  my global setuptools, for which you need to sudo.  And it also forces
>  everyone else using that buildout to switch (which in itself is OK).

It shouldn't touch anything outside your buildout, please try with the latest
one at buildout-distribute at bitbucket.

>
> - From my experiments (not exhaustive! no real debugging!) it seems that the
>  only safe way is to first install distribute globally before touching a
>  buildout that brings in distribute.  Is this to be expected?

The Distribute installation will try to patch any setuptools found in
the path so it doesn't
interfer.

In a buildout environment, if it changes anything globally, it's a bug of our
patching code and not intended, unless your buildout environment,
somehow, uses a global
setuptools. But I don't think this can happen.


Tarek
--
Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Reinout van Rees]

On 2009-10-07, Tarek Ziadé <ziade.tarek@...> wrote:

> On Wed, Oct 7, 2009 at 1:23 PM, Reinout van Rees <reinout@...> wrote:
>>
>> - Do my libraries have to list a dependency on distribute or on setuptools?
>>  Everything zopish has a 'setuptools >= 0.6c9' in it.  I tried putting
>>  distribute in there, but I seemed to get less problems with leaving it at
>>  setuptools.
>>
>
> No. Once we've resolved the problem you've mentioned in the hardcoded setuptools
> dependency, you will have an empty Setuptools 0.6c9 egg in eggs/
> so libs that ask for this dependency will  think it's met.

But to actually use distribute you need to depend on it somewhere, obviously.
Otherwise I only get a setuptools egg in my path.

> It shouldn't touch anything outside your buildout, please try with the latest
> one at buildout-distribute at bitbucket.

The latest bootstrap behaves itself wonderfully.



So: once we have new zc.buildout + bootstrap it ought to be safe to change all
buildouts without inconveniencing anyone.

I'm still not 100% sure whether it is safe to put "distribute" in the
install_requires list of a package right now, however.


Reinout

--
Reinout van Rees - reinout@... - http://reinout.vanrees.org
Software developer at http://www.thehealthagency.com
"Military engineers build missiles. Civil engineers build targets"

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Tarek Ziadé]

On Wed, Oct 7, 2009 at 2:32 PM, Reinout van Rees <reinout@...> wrote:
>
> I'm still not 100% sure whether it is safe to put "distribute" in the
> install_requires list of a package right now, however.

It depends on what you mean by safe. (beside any bootstraping bug we might find
in the near future, even if this code seems stabilized now)

Adding distribute in install_requires will have
the effect of calling the Distribute setup.py and renaming an existing
setuptools installation
detected in the environment  so it doesn't get on the way, and to add
a setuptools*.egg-info in
the environment so the Setuptools dependency is still met (to avoid
Distribute to be overriden
in turn)

By "environment" I mean here the sys.path than gets scanned by pkg_resources
when looking for Distributions.

So when your package is installed, its impact will depend on the environment its
setup.py was called in.

The impact is : now the environment is using Distribute.

This has to be done manually in Distribute's setup.py because the
micro-dependency managment
system provided by easy_install and pkg_resources doesn't know how to
deal with mutual
exclusive dependencies or interchangeable dependency.

But this is not a new problem: if you add dependencies in your
distribution, you are impacting
the whole execution environment, for every other distribution.

Now you can say it outloud in your distribution documentation, that
your distribution requires
Distribute because it has more bugfixes.

Tarek

--
Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Reinout van Rees]

On 2009-10-07, Tarek Ziadé <ziade.tarek@...> wrote:

> On Wed, Oct 7, 2009 at 2:32 PM, Reinout van Rees <reinout@...> wrote:
>>
>> I'm still not 100% sure whether it is safe to put "distribute" in the
>> install_requires list of a package right now, however.
>
> It depends on what you mean by safe. (beside any bootstraping bug we might
> find in the near future, even if this code seems stabilized now)
>
> Adding distribute in install_requires will have the effect of calling the
> Distribute setup.py and renaming an existing setuptools installation
> detected in the environment so it doesn't get on the way, and to add a
> setuptools*.egg-info in the environment so the Setuptools dependency is
> still met (to avoid Distribute to be overriden in turn)
>
> By "environment" I mean here the sys.path than gets scanned by pkg_resources
> when looking for Distributions.
>
> So when your package is installed, its impact will depend on the environment
> its setup.py was called in.
>
> The impact is : now the environment is using Distribute.

Ok, I've experimented some more and came to some sort of conclusion in the bug
report: http://tinyurl.com/yaa23yl


The warning problem occurs if the environment (which you mention above) uses
distribute.  This environment could be a buildout egg cache.  If a buildout,
which uses this cache, only knows about setuptools (as there's no distribute
depencency in that buildout), you'll get the extra "hey, I already loaded
this" UserWarnings.

But, and that's the mostly-good news, these warnings only occur if you have a
global setuptools and haven't installed distribute globally yet.  In that
case, somehow there's a fallback to the global setuptools which omits those
warnings.

IF warnings THEN install it also globally, basically.


And, from what I see, a global install works just fine with a buildout that
doesn't use the new bootstrap.  Provided the setuptools in the egg cache has
been patch already by some other buildout :-)


Reinout

--
Reinout van Rees - reinout@... - http://reinout.vanrees.org
Software developer at http://www.thehealthagency.com
"Military engineers build missiles. Civil engineers build targets"

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Chris Withers]

Reinout van Rees wrote:
> - Do my libraries have to list a dependency on distribute or on setuptools?
>   Everything zopish has a 'setuptools >= 0.6c9' in it.

They shouldn't, unless you only require setuptools after your package is
installed and don't use it in your setup.py, which seems unlikely.

If you need it in your setup.py, what's the point of specifying it? You
would have had to use it by the time you specify the requirement!

> - Do I need to change buildout's bootstrap already? Doing that tries to change
>   my global setuptools,

It shouldn't. The normal buildout bootstrap.py doesn't...

Chris

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Chris Withers]

Reinout van Rees wrote:
> I'm still not 100% sure whether it is safe to put "distribute" in the
> install_requires list of a package right now, however.

As with setuptools, why do you think you need to?

Chris


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Hanno Schlichting-4]

On Fri, Oct 9, 2009 at 1:07 PM, Chris Withers <chris@...> wrote:

> Reinout van Rees wrote:
>>
>> - Do my libraries have to list a dependency on distribute or on
>> setuptools?
>>  Everything zopish has a 'setuptools >= 0.6c9' in it.
>
> They shouldn't, unless you only require setuptools after your package is
> installed and don't use it in your setup.py, which seems unlikely.
>
> If you need it in your setup.py, what's the point of specifying it? You
> would have had to use it by the time you specify the requirement!

I assume most packages Reinout uses (like all zope.* packages) use
namespace packages. So they actually do depend during runtime on the
pkg_resources module, which happens to be available from either the
distribute  or setuptools distribution. So one of them should be
specified in install_requires.

Hanno
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Jim Fulton]

On Fri, Oct 9, 2009 at 7:57 AM, Hanno Schlichting <hanno@...> wrote:

> On Fri, Oct 9, 2009 at 1:07 PM, Chris Withers <chris@...> wrote:
>> Reinout van Rees wrote:
>>>
>>> - Do my libraries have to list a dependency on distribute or on
>>> setuptools?
>>>  Everything zopish has a 'setuptools >= 0.6c9' in it.
>>
>> They shouldn't, unless you only require setuptools after your package is
>> installed and don't use it in your setup.py, which seems unlikely.
>>
>> If you need it in your setup.py, what's the point of specifying it? You
>> would have had to use it by the time you specify the requirement!
>
> I assume most packages Reinout uses (like all zope.* packages) use
> namespace packages. So they actually do depend during runtime on the
> pkg_resources module, which happens to be available from either the
> distribute  or setuptools distribution. So one of them should be
> specified in install_requires.

This functionality should be factored out and merged with pkgutil in
the standard library.

Jim

--
Jim Fulton
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Hanno Schlichting-4]

On Fri, Oct 9, 2009 at 3:14 PM, Jim Fulton <jim@...> wrote:

> On Fri, Oct 9, 2009 at 7:57 AM, Hanno Schlichting <hanno@...> wrote:
>> On Fri, Oct 9, 2009 at 1:07 PM, Chris Withers <chris@...> wrote:
>> I assume most packages Reinout uses (like all zope.* packages) use
>> namespace packages. So they actually do depend during runtime on the
>> pkg_resources module, which happens to be available from either the
>> distribute  or setuptools distribution. So one of them should be
>> specified in install_requires.
>
> This functionality should be factored out and merged with pkgutil in
> the standard library.

Sure. That's what http://www.python.org/dev/peps/pep-0382 is about.

But both Python 2.7 and 3.2 are a bit into the future, so it doesn't
help with the current question.

Hanno
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Chris Withers]

Hanno Schlichting wrote:
> I assume most packages Reinout uses (like all zope.* packages) use
> namespace packages. So they actually do depend during runtime on the
> pkg_resources module, which happens to be available from either the
> distribute  or setuptools distribution. So one of them should be
> specified in install_requires.

You're missing my point.

If you're using buildout, you automatically have setuptools available.

If you're using easy_install, you automatically have setuptools available.

So, the only case we can even consider is when you're installing a
setuptools-based package with:

python setup.py install

In this case, which I suspect is extremely rare anyway, you'll need to
have setuptools installed already.

So, in *any* of these cases, specifying setuptools as a requirement
seems like a total waste of time...

Now, what case have I missed? ;-)

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Toshio Kuratomi-2]

On Fri, Oct 09, 2009 at 03:28:57PM +0100, Chris Withers wrote:
>
> In this case, which I suspect is extremely rare anyway, you'll need to  
> have setuptools installed already.
>
> So, in *any* of these cases, specifying setuptools as a requirement  
> seems like a total waste of time...
>
> Now, what case have I missed? ;-)
>
It's nice for people creating system packages when you specify all of the
packages that your runtime depends on in setup.py.  That allows system
packagers to read setup.py and be able to create the complete list of
runtime dependencies for their packaging metadata.  Several times I've been
asked for help debugging why a python package fails to work for a few people
only to discover that the package used entry-points or another setuptools
runtime feature but only required it for buildtime.  Note, however, that
overspeciying the *versions* you need has the opposite effect :-)

-Toshio


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Chris Withers]

Toshio Kuratomi wrote:

> On Fri, Oct 09, 2009 at 03:28:57PM +0100, Chris Withers wrote:
>> In this case, which I suspect is extremely rare anyway, you'll need to  
>> have setuptools installed already.
>>
>> So, in *any* of these cases, specifying setuptools as a requirement  
>> seems like a total waste of time...
>>
>> Now, what case have I missed? ;-)
>>
> It's nice for people creating system packages when you specify all of the
> packages that your runtime depends on in setup.py.  

...except that it causes problems that are a bit more serious than "nice
to have" because of the ridiculous situation we're in with setuptools
and distribute...

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Toshio Kuratomi-2]

On Fri, Oct 09, 2009 at 04:04:06PM +0100, Chris Withers wrote:

> Toshio Kuratomi wrote:
>> On Fri, Oct 09, 2009 at 03:28:57PM +0100, Chris Withers wrote:
>>> In this case, which I suspect is extremely rare anyway, you'll need
>>> to  have setuptools installed already.
>>>
>>> So, in *any* of these cases, specifying setuptools as a requirement  
>>> seems like a total waste of time...
>>>
>>> Now, what case have I missed? ;-)
>>>
>> It's nice for people creating system packages when you specify all of the
>> packages that your runtime depends on in setup.py.  
>
> ...except that it causes problems that are a bit more serious than "nice
> to have" because of the ridiculous situation we're in with setuptools  
> and distribute...
>

What's the issue precisely?  Once distribute is on the system, setuptools is

provided by distribute so there's no problem there, correct?

Is it that the installers don't know that there's more than one package
providing the setuptools API?  That sounds like pypi and easy_install aren't
powerful enough to recognize that an API can be provided by multiple
modules.  If you actually want to have a full-blown package manager you'll
need to fix that but at the same time I'd warn that having a full-blown
package manager means having to deal with a lot of corner cases like this.

-Toshio


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Chris Withers]

Toshio Kuratomi wrote:

> On Fri, Oct 09, 2009 at 04:04:06PM +0100, Chris Withers wrote:
>> Toshio Kuratomi wrote:
>>> On Fri, Oct 09, 2009 at 03:28:57PM +0100, Chris Withers wrote:
>>>> In this case, which I suspect is extremely rare anyway, you'll need
>>>> to  have setuptools installed already.
>>>>
>>>> So, in *any* of these cases, specifying setuptools as a requirement  
>>>> seems like a total waste of time...
>>>>
>>>> Now, what case have I missed? ;-)
>>>>
>>> It's nice for people creating system packages when you specify all of the
>>> packages that your runtime depends on in setup.py.  
>> ...except that it causes problems that are a bit more serious than "nice
>> to have" because of the ridiculous situation we're in with setuptools  
>> and distribute...
>>
> What's the issue precisely?  Once distribute is on the system, setuptools is
> provided by distribute so there's no problem there, correct?

The issue is that both the setuptools and distribute distributions
provide a setuptools package. This apparently causes problems, rather
unsurprisingly ;-)

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Toshio Kuratomi-2]

On Fri, Oct 09, 2009 at 05:13:16PM +0100, Chris Withers wrote:

> Toshio Kuratomi wrote:
>> On Fri, Oct 09, 2009 at 04:04:06PM +0100, Chris Withers wrote:
>>> Toshio Kuratomi wrote:
>>>> On Fri, Oct 09, 2009 at 03:28:57PM +0100, Chris Withers wrote:
>>>>> In this case, which I suspect is extremely rare anyway, you'll
>>>>> need to  have setuptools installed already.
>>>>>
>>>>> So, in *any* of these cases, specifying setuptools as a
>>>>> requirement  seems like a total waste of time...
>>>>>
>>>>> Now, what case have I missed? ;-)
>>>>>
>>>> It's nice for people creating system packages when you specify all of the
>>>> packages that your runtime depends on in setup.py.  
>>> ...except that it causes problems that are a bit more serious than
>>> "nice to have" because of the ridiculous situation we're in with
>>> setuptools  and distribute...
>>>
>> What's the issue precisely?  Once distribute is on the system, setuptools is
>> provided by distribute so there's no problem there, correct?
>
> The issue is that both the setuptools and distribute distributions  
> provide a setuptools package. This apparently causes problems, rather  
> unsurprisingly ;-)
>

True... but because of that people are able to specify setuptools in

setup.py and it will work with either distribute or setuptools.  Is what
you're getting at that if people didn't specify setuptools in setup.py,
distribute-0.6 could install without using the setuptools name?  I don't
think that works since you still need to take over the setuptools module
directory so import works inside the code and the setuptools egg-info so
things like plugin modules belonging to setuptools work.

-Toshio


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Reinout van Rees]

On 2009-10-09, Chris Withers <chris@...> wrote:
> Reinout van Rees wrote:
>> I'm still not 100% sure whether it is safe to put "distribute" in the
>> install_requires list of a package right now, however.
>
> As with setuptools, why do you think you need to?

Namespace packages.  You get warnings "hey, no setuptools dependency" once you
use namespaces.

Note that if I depend on distribute, I also get warnings about not having a
setuptools depencency sometimes... :-)


Reinout

--
Reinout van Rees - reinout@... - http://reinout.vanrees.org
Software developer at http://www.thehealthagency.com
"Military engineers build missiles. Civil engineers build targets"

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Reinout van Rees wrote:
> On 2009-10-09, Chris Withers <chris@...> wrote:
>> Reinout van Rees wrote:
>>> I'm still not 100% sure whether it is safe to put "distribute" in the
>>> install_requires list of a package right now, however.
>> As with setuptools, why do you think you need to?
>
> Namespace packages.  You get warnings "hey, no setuptools dependency" once you
> use namespaces.

That is a buildout bug:  whining about the lack of depenedency on the
only package which makes dependencies meaningful is silly.  No package
should *ever* need to declare that it depends on setuptools.

> Note that if I depend on distribute, I also get warnings about not having a
> setuptools depencency sometimes... :-)

Likewise.


Tres.
- --
===================================================================
Tres Seaver          [phone]          tseaver@...
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkrUuSgACgkQ+gerLs4ltQ7lewCZAS/MML7cmnzEnK1M6gbSSQHS
lrQAn2KDOhsfT8Wtj83DW9G4WTOVXoS2
=VxBP
-----END PGP SIGNATURE-----

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hanno Schlichting wrote:

> On Fri, Oct 9, 2009 at 1:07 PM, Chris Withers <chris@...> wrote:
>> Reinout van Rees wrote:
>>> - Do my libraries have to list a dependency on distribute or on
>>> setuptools?
>>>  Everything zopish has a 'setuptools >= 0.6c9' in it.
>> They shouldn't, unless you only require setuptools after your package is
>> installed and don't use it in your setup.py, which seems unlikely.
>>
>> If you need it in your setup.py, what's the point of specifying it? You
>> would have had to use it by the time you specify the requirement!
>
> I assume most packages Reinout uses (like all zope.* packages) use
> namespace packages. So they actually do depend during runtime on the
> pkg_resources module, which happens to be available from either the
> distribute  or setuptools distribution. So one of them should be
> specified in install_requires.

Why?  Nobody will check / enforce / understand what 'install_requires'
even means except setuptools / distribute.

- --
===================================================================
Tres Seaver          [phone]          tseaver@...
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkrUuWwACgkQ+gerLs4ltQ5G8wCgjfmDazDum+ClDnxDtw+nxImD
W4AAoKElE+Uelm3yL30Kdjj/6YFpL+4C
=KIbC
-----END PGP SIGNATURE-----

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Hanno Schlichting-4]

On Tue, Oct 13, 2009 at 7:31 PM, Tres Seaver <tseaver@...> wrote:
> Why?  Nobody will check / enforce / understand what 'install_requires'
> even means except setuptools / distribute.

To quote Toshio Kuratomi:
> It's nice for people creating system packages when you specify all of the
> packages that your runtime depends on in setup.py.

So apparently there are others who read and value this information.

Hanno
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig