First attempt: close but no data files!

[Kaelin Colclasure]

Greetings,

I am attempting my first Python contribution and have run into a speed  
bump getting a working setup.py using setuptools. Specifically, I  
cannot coax it to install my package data files into the site-packages  
directory. Here is a link to my project where my setup.py is available  
for browsing:

<http://bitbucket.org/kaelin/cuttlefish/>

Soooo close!

TIA,

-- Kaelin

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[ssteiner]

On Oct 22, 2009, at 8:47 PM, Kaelin Colclasure wrote:

> Greetings,
>
> I am attempting my first Python contribution and have run into a  
> speed bump getting a working setup.py using setuptools.  
> Specifically, I cannot coax it to install my package data files into  
> the site-packages directory. Here is a link to my project where my  
> setup.py is available for browsing:
>
> <http://bitbucket.org/kaelin/cuttlefish/>
>
> Soooo close!

How about some sort of description of what you're trying to do...

        • Reproduce steps: Clearly mention the steps to reproduce the bug.
        • Expected result: How application should behave on above mentioned  
steps.
        • Actual result: What is the actual result on running above steps  
i.e. the bug behavior.

Thanks,

S

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[P.J. Eby]

At 05:47 PM 10/22/2009 -0700, Kaelin Colclasure wrote:
>Greetings,
>
>I am attempting my first Python contribution and have run into a speed
>bump getting a working setup.py using setuptools. Specifically, I
>cannot coax it to install my package data files into the site-packages
>directory. Here is a link to my project where my setup.py is available
>for browsing:
>
><http://bitbucket.org/kaelin/cuttlefish/>

There's no setup.py there.


>Soooo close!
>
>TIA,
>
>-- Kaelin
>
>_______________________________________________
>Distutils-SIG maillist  -  Distutils-SIG@...
>http://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Kaelin Colclasure]

On Oct 22, 2009, at 6:05 PM, ssteinerX@... wrote:

>
> On Oct 22, 2009, at 8:47 PM, Kaelin Colclasure wrote:
>
>> Greetings,
>>
>> I am attempting my first Python contribution and have run into a  
>> speed bump getting a working setup.py using setuptools.  
>> Specifically, I cannot coax it to install my package data files  
>> into the site-packages directory. Here is a link to my project  
>> where my setup.py is available for browsing:
>>
>> <http://bitbucket.org/kaelin/cuttlefish/>
>>
>> Soooo close!
>
> How about some sort of description of what you're trying to do...
>
> • Reproduce steps: Clearly mention the steps to reproduce the bug.
> • Expected result: How application should behave on above mentioned  
> steps.
> • Actual result: What is the actual result on running above steps  
> i.e. the bug behavior.
>
> Thanks,

My project has a file (cuttlefish-config.plist) that needs to be  
installed next to the module (cuttlefish.py) in the site-packages  
directory. There is also a directory (static) containing some html  
files and png images that should go in the same place.

I first tried using

    include_package_data=True, # Requires setuptools_hg

after installing setuptools_hg to hook setuptools to understand  
Mercurial. This got all my data files added to the ...egg-info/
SOURCES.txt and I thought it was good. But then after building an egg  
and installing it with easy_setup, all the data files were missing.

I then tried adding a MANIFEST.in file:
---8<---
(Bottle-python2.6)syn:cuttlefish kaelin$ cat MANIFEST.in
include *.plist
graft static
--->8---

Same result: The files are listed in the "manifest" SOURCES.txt, but  
not installed.

Here is SOURCES.txt:
---8<---
(Bottle-python2.6)syn:cuttlefish kaelin$ cat Cuttlefish.egg-info/
SOURCES.txt
.hgignore
MANIFEST.in
README.txt
cuttlefish-config.plist
cuttlefish.py
setup.py
Cuttlefish.egg-info/PKG-INFO
Cuttlefish.egg-info/SOURCES.txt
Cuttlefish.egg-info/dependency_links.txt
Cuttlefish.egg-info/entry_points.txt
Cuttlefish.egg-info/not-zip-safe
Cuttlefish.egg-info/requires.txt
Cuttlefish.egg-info/top_level.txt
static/LICENSE.html
static/bottle-sig.png
static/cuttlefish.png
static/mako-sig.png
--->8---

Here is my setup.py:
---8<---
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Setup script for cuttlefish.
"""
from setuptools import setup, find_packages
import cuttlefish

setup(
    name="Cuttlefish",
    version=cuttlefish.__version__,
    py_modules=[
        'cuttlefish',
    ],
    include_package_data=True, # Requires setuptools_hg
    entry_points={
        'console_scripts': [
            'cuttlefish = cuttlefish:see_bottle_run',
        ],
    },
    zip_safe=False, # Needs a user-editable cuttlefish-config.plist
    install_requires=[
        'Bottle >= 0.6.4',
        'Mako >= 0.2.5',
    ],
    # PyPI metadata
    author='Kaelin Colclasure',
    author_email='kaelin@...',
    description="Browser-based search tool for quickly `grep`ing  
source code.",
    long_description="Cuttlefish is an example project using the  
Bottle flyweight Web framework and the Mako template engine in Python.  
It's intended to run on your desktop, and provide a lightweight search  
engine (using grep) for your source code.",
    license='MIT',
    platforms=['darwin', 'unix'],
    url='http://bitbucket.org/kaelin/cuttlefish/',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Environment :: Web Environment',
        #'Framework :: Bottle',
        'Intended Audience :: Developers',
        'License :: OSI Approved :: MIT License',
        'Operating System :: MacOS :: MacOS X',
        'Operating System :: Unix',
        'Programming Language :: Python :: 2.6',
        'Topic :: Software Development :: Documentation',
        'Topic :: Text Processing :: Indexing',
    ],
)
--->8---

The whole project is available from <http://bitbucket.org/kaelin/cuttlefish/ 
 >.

TIA,

-- Kaelin

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Kaelin Colclasure]

On Oct 22, 2009, at 6:14 PM, P.J. Eby wrote:

> At 05:47 PM 10/22/2009 -0700, Kaelin Colclasure wrote:
>> Greetings,
>>
>> I am attempting my first Python contribution and have run into a  
>> speed
>> bump getting a working setup.py using setuptools. Specifically, I
>> cannot coax it to install my package data files into the site-
>> packages
>> directory. Here is a link to my project where my setup.py is  
>> available
>> for browsing:
>>
>> <http://bitbucket.org/kaelin/cuttlefish/>
>
> There's no setup.py there.

Oops, forgot to `hg push`... It's there now. :-)

>
>
>> Soooo close!
>>
>> TIA,
>>
>> -- Kaelin
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG@...
>> http://mail.python.org/mailman/listinfo/distutils-sig
>

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[P.J. Eby]

At 06:26 PM 10/22/2009 -0700, Kaelin Colclasure wrote:

>On Oct 22, 2009, at 6:14 PM, P.J. Eby wrote:
>
>>At 05:47 PM 10/22/2009 -0700, Kaelin Colclasure wrote:
>>>Greetings,
>>>
>>>I am attempting my first Python contribution and have run into a
>>>speed
>>>bump getting a working setup.py using setuptools. Specifically, I
>>>cannot coax it to install my package data files into the site- packages
>>>directory. Here is a link to my project where my setup.py is
>>>available
>>>for browsing:
>>>
>>><http://bitbucket.org/kaelin/cuttlefish/>
>>
>>There's no setup.py there.
>
>
>Oops, forgot to `hg push`... It's there now. :-)

Your "package data" isn't being included because you don't have any
packages, only a module.

Move your cuttlefish.py to cuttlefish/__init__.py, and put the .plist
file in the same directory with it, and switch from py_modules to
packages in your setup() arguments.  That will make it get included
with your installs and eggs.

Finally, you may want to drop use of __file__ and use the
pkg_resources API to access your data files instead, so that your
library will work when it's installed as a zipfile:

   http://peak.telecommunity.com/DevCenter/PkgResources#basic-resource-access

For example, rather than using __file__ to find the .plist file, you might use:

     contents = resource_string(__name__, 'cuttlefish-config.plist')

to load the contents of the file as a string.

(This step is optional; if you use __file__ in your code, setuptools
will mark your project's eggs as requiring unzipping, and they will
be installed unzipped.  So, you don't *have to* do it, the __file__
use will still work.)


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Kaelin Colclasure]

On Oct 22, 2009, at 7:37 PM, P.J. Eby wrote:

> At 06:26 PM 10/22/2009 -0700, Kaelin Colclasure wrote:
>
>> On Oct 22, 2009, at 6:14 PM, P.J. Eby wrote:
>>
>>> At 05:47 PM 10/22/2009 -0700, Kaelin Colclasure wrote:
>>>> Greetings,
>>>>
>>>> I am attempting my first Python contribution and have run into a
>>>> speed
>>>> bump getting a working setup.py using setuptools. Specifically, I
>>>> cannot coax it to install my package data files into the site-  
>>>> packages
>>>> directory. Here is a link to my project where my setup.py is
>>>> available
>>>> for browsing:
>>>>
>>>> <http://bitbucket.org/kaelin/cuttlefish/>
>>>
>>> There's no setup.py there.
>>
>>
>> Oops, forgot to `hg push`... It's there now. :-)
>
> Your "package data" isn't being included because you don't have any  
> packages, only a module.
>
> Move your cuttlefish.py to cuttlefish/__init__.py, and put  
> the .plist file in the same directory with it, and switch from  
> py_modules to packages in your setup() arguments.  That will make it  
> get included with your installs and eggs.
>
> Finally, you may want to drop use of __file__ and use the  
> pkg_resources API to access your data files instead, so that your  
> library will work when it's installed as a zipfile:
>
> http://peak.telecommunity.com/DevCenter/PkgResources#basic-resource-access
>
> For example, rather than using __file__ to find the .plist file, you  
> might use:
>
>   contents = resource_string(__name__, 'cuttlefish-config.plist')
>
> to load the contents of the file as a string.
>
> (This step is optional; if you use __file__ in your code, setuptools  
> will mark your project's eggs as requiring unzipping, and they will  
> be installed unzipped.  So, you don't *have to* do it, the __file__  
> use will still work.)
>

Ah, so… Thanks for the guidance!

Restructuring as a package did indeed get things working as expected.  
It's somewhat unfortunate that this is a requirement, as it made for a  
lot of noise in my Mercurial repository and now most of my code is in  
a module with the unhelpful name __init__.py… But this too shall  
pass. :-)

Thanks again!

-- Kaelin


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[P.J. Eby]

At 09:58 PM 10/22/2009 -0700, Kaelin Colclasure wrote:
>Restructuring as a package did indeed get things working as expected.
>It's somewhat unfortunate that this is a requirement, as it made for a
>lot of noise in my Mercurial repository and now most of my code is in
>a module with the unhelpful name __init__.py…

Given how short your data file is, the fact that it's entirely text,
and the fact that your module script always needs it to be loaded, I
wonder why you don't just make it a string constant in the .py file
to start with, or better yet, simply directly create the data
structure it represents.

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Kaelin Colclasure]

On Oct 23, 2009, at 8:16 AM, P.J. Eby wrote:

> At 09:58 PM 10/22/2009 -0700, Kaelin Colclasure wrote:
>> Restructuring as a package did indeed get things working as expected.
>> It's somewhat unfortunate that this is a requirement, as it made  
>> for a
>> lot of noise in my Mercurial repository and now most of my code is in
>> a module with the unhelpful name __init__.py…
>
> Given how short your data file is, the fact that it's entirely text,  
> and the fact that your module script always needs it to be loaded, I  
> wonder why you don't just make it a string constant in the .py file  
> to start with, or better yet, simply directly create the data  
> structure it represents.

I think you're referring to cuttlefish-config.plist, which is intended  
to be edited after installation to refer to the particulars of the  
installed site's filesystem, etc. [And yes, I realize having such a  
config file live inside the package is sub-optimal. I'm just looking  
for a low-impedence solution that works with the PyPI infrastructure.  
That said, if there are "best practices" for such things established  
and supported by setuptools I'm all ears…]

There is also a 'static/' directory with a bunch of webapp resources  
in it. And it is now installing with the package perfectly too. Thanks  
again for the helpful guidance!

-- Kaelin

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[P.J. Eby]

At 08:53 AM 10/23/2009 -0700, Kaelin Colclasure wrote:

>On Oct 23, 2009, at 8:16 AM, P.J. Eby wrote:
>
>>At 09:58 PM 10/22/2009 -0700, Kaelin Colclasure wrote:
>>>Restructuring as a package did indeed get things working as expected.
>>>It's somewhat unfortunate that this is a requirement, as it made
>>>for a
>>>lot of noise in my Mercurial repository and now most of my code is in
>>>a module with the unhelpful name __init__.py…
>>
>>Given how short your data file is, the fact that it's entirely text,
>>and the fact that your module script always needs it to be loaded, I
>>wonder why you don't just make it a string constant in the .py file
>>to start with, or better yet, simply directly create the data
>>structure it represents.
>
>
>I think you're referring to cuttlefish-config.plist, which is intended
>to be edited after installation to refer to the particulars of the
>installed site's filesystem, etc.

Don't do that.  Package data is for static, read-only data used by a
library or application  at runtime.


>  [And yes, I realize having such a
>config file live inside the package is sub-optimal. I'm just looking
>for a low-impedence solution that works with the PyPI infrastructure.
>That said, if there are "best practices" for such things established
>and supported by setuptools I'm all ears…]

Put it with documentation, or use the standard distutils data_files
option, rather than package data.  User editable files should *not*
be installed adjacent to the code; it's rightly frowned on by Linux
distributions and system administrators everywhere.

If the issue is that it's a "global" configuration file and you don't
know where else to put it, base the location on the user's home
directory (on *nix ) or an APP_DATA subdirectory (on Windows).

If it's not a single per-user location, then use an environment
variable or command-line option (for command-line tools) or as an
argument (for Python APIs).

Some tools also ship with a lot of package data as *templates* for
user-configurable stuff, and include a short script to copy the
template to a user-specified location, possibly filling in a few
things specified on the command line, or via interactive
prompts.  This is a good way to provide a nice UI for your
installation/setup, especially in the case where someone might need
multiple copies of the configured data (e.g. a webapp you can install
multiple instances of, like Trac).


>There is also a 'static/' directory with a bunch of webapp resources
>in it.

If those are user-editable, it's probably a good idea to include a
script that copies and/or customizes them and places them in a
user-defined location, rather than having the code read them directly
from the package (and thus requiring the user to edit them
directly).  Of course, if those webapp resources are *not*
user-editable, then leaving them in your package data is fine.


>  And it is now installing with the package perfectly too. Thanks
>again for the helpful guidance!

You can thank me (and make a lot of sysadmins happier, or at least a
little less grumpy) by not locating any user-editable files inside
your package directory.  ;-)

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig