Module build fails because distutils adds "-isysroot"

[Patrick Näf]

Hi folks

I'm currently writing a very simple Python module in C that provides an
interface to libaprutil's MD5 routines. I compile my module using
distutils, as per instructions in the "Building C and C++ Extensions with
distutils" chapter of the official Python docs.

Since Mac OS X has libaprutil pre-installed, it seems natural that I try
to use the system version of the library. My Python module's .c file
therefore contains the following #include directive:

  #include <apr-1/apr_md5.h>

Everything works fine (module compiles and I can use it) as long as I use
the system version of Python (2.5.1) to compile the module. The problem is
that I *really* need to build the module with Python 2.6 or newer, and
this is where my trouble starts. I have Python 3.1.1 installed in
/Library/Frameworks/Python, and with this version of Python my module's
build fails miserably, like this:

tharbad:~ patrick$
/Library/Frameworks/Python.framework/Versions/3.1/bin/python3.1 ./setup.py
build_ext --inplace
running build_ext
building 'aprmd5' extension
gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk
-fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3
-I/Library/Frameworks/Python.framework/Versions/3.1/include/python3.1 -c
aprmd5.c -o build/temp.macosx-10.3-fat-3.1/aprmd5.o
aprmd5.c:43:60: aprmd5.c:43:60: error: apr-1/apr_md5.h: No such file or
directory
[...]

The problem here is that the gcc option "-isysroot" moves the root for
system includes to the SDK directory (/Developer/SDKs/MacOSX10.4u.sdk),
away from the normal root (/). Unfortunately, the SDK does not contain the
header <apr-1/apr_md5.h>, which causes the build to fail. Where does
"-isysroot" come from? Not from my module, it's added automatically by
distutils.

I believe I have correctly diagnosed the problem, and I can certainly work
around it *somehow* in my module, but since I am rather new to Python I
have trouble deciding on a solution that is appropriate. I am therefore
writing to this list, hoping someone reading this can give me some advice.

1) Is my problem a known situation for which there is a generally
accepted, best-practice solution? If so, a pointer in the right direction
would be most welcome. If not, what would seem to be a good solution? Make
my own build of libaprutil using the MacOSX10.4u SDK? Rely on a
third-party build of libaprutil (e.g. fink)? Force the compiler to use the
system's libaprutil?

2) Is this a bug in the MacOSX10.4u SDK, i.e. should the SDK contain the
headers for libaprutil? This seems possible to me, because the SDK
certainly contains headers for other libraries (e.g. openssl), so why not
for libaprutil?

3) Is this a bug in the Python distribution, i.e. should the Python 3.1.1
distutils *not* add "-isysroot"? I think this is not the case, distutils
should add the option because Python probably was built with the SDK.

4) Is my work obsolete because someone already has written a Python module
that interfaces libaprutil and has solved all of my problems? If this is
so, again, pointers would be very welcome.


Thanks for your time
Patrick


System setup:
- Mac OS X 10.5.8 (includes libaprutil 1.2.7)
- gcc --version: powerpc-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc.
build 5493)
- Yes, I *do* have the MacOSX10.4u SDK installed :-)
- Python 3.1.1, downloaded from
http://www.python.org/ftp/python/3.1.1/python-3.1.1.dmg (I have also tried
Python 2.6.2 and 3.0.1, with the same result)


_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIG@...
http://mail.python.org/mailman/listinfo/pythonmac-sig

[Dave Peterson-4]

Hi Patrick,

Patrick Näf wrote:

> ... my module's
> build fails miserably, like this:
>
> tharbad:~ patrick$
> /Library/Frameworks/Python.framework/Versions/3.1/bin/python3.1 ./setup.py
> build_ext --inplace
> running build_ext
> building 'aprmd5' extension
> gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk
> -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3
> -I/Library/Frameworks/Python.framework/Versions/3.1/include/python3.1 -c
> aprmd5.c -o build/temp.macosx-10.3-fat-3.1/aprmd5.o
> aprmd5.c:43:60: aprmd5.c:43:60: error: apr-1/apr_md5.h: No such file or
> directory
> [...]
>
> The problem here is that the gcc option "-isysroot" moves the root for
> system includes to the SDK directory (/Developer/SDKs/MacOSX10.4u.sdk),
> away from the normal root (/). Unfortunately, the SDK does not contain the
> header <apr-1/apr_md5.h>, which causes the build to fail. Where does
> "-isysroot" come from? Not from my module, it's added automatically by
> distutils.
>  

The options distutils uses come from the file at <python env
root>/lib/python2.5/config/Makefile.  I don't think there is any
distutils API provided to override/change these, just ways to append to
them when specifying your extension configuration.  These options are
used because they are the same ones used when building that Python
distribution.

So my first suggestion is that you try redefining the -isysroot option
in the flags you can specify in your extension's config.  There is a
good chance the compiler will use the last specification on the line
rather than doing something else when it gets two values.

If that doesn't work, the only other thing I can think of is to
temporarily edit the config/Makefile to remove / change the inclusion of
the -isysroot option.

HTHs,

-- Dave

_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIG@...
http://mail.python.org/mailman/listinfo/pythonmac-sig

[Patrick Näf]

Hi Dave

On Wed, September 30, 2009 18:17, Dave Peterson wrote:
> The options distutils uses come from the file at <python env
> root>/lib/python2.5/config/Makefile.  I don't think there is any

Good to know.


> So my first suggestion is that you try redefining the -isysroot option
> in the flags you can specify in your extension's config.  There is a
> good chance the compiler will use the last specification on the line
> rather than doing something else when it gets two values.

I didn't try this because 1) the Python docs say that the config file
(setup.cfg) only allows to change the same set of options that can be
specified on the command line; and 2) the "build_ext" command of setup.py
only knows a very specific and limited set of compiler options (e.g.
preprocessor defines, include paths, additional link libraries, etc.), and
-isysroot is not among these.

I figured out another way to specify -isysroot, though: Within setup.py,
the Extension object can be instantiated with the "extra_compile_args" and
"extra_link_args" arguments. For instance:

--- snip ---
from distutils.core import Extension

aprmd5 = Extension('aprmd5', [... more arguments ...],
                   extra_compile_args = ['-isysroot', '/'],
                   extra_link_args = ['-isysroot', '/'])
--- snip ---


As you suggested, compiler and linker use the last specification of
-isysroot, so I win :-) Compiler and linker now look for headers and libs
in places such as "//usr/include" and "//usr/lib" (notice the double
slash), but the module builds and I can use it!


> HTHs,

Yes it does! It's amazing, I'm still a bit dazed and waiting for the
snag... :-)


Thanks very much
Patrick


_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIG@...
http://mail.python.org/mailman/listinfo/pythonmac-sig

[Ronald Oussoren]

On 30 Sep, 2009, at 11:30, Patrick Näf wrote:

> Hi folks
>
> I'm currently writing a very simple Python module in C that provides  
> an
> interface to libaprutil's MD5 routines. I compile my module using
> distutils, as per instructions in the "Building C and C++ Extensions  
> with
> distutils" chapter of the official Python docs.
>
> Since Mac OS X has libaprutil pre-installed, it seems natural that I  
> try
> to use the system version of the library. My Python module's .c file
> therefore contains the following #include directive:
>
>  #include <apr-1/apr_md5.h>
>
> Everything works fine (module compiles and I can use it) as long as  
> I use
> the system version of Python (2.5.1) to compile the module. The  
> problem is
> that I *really* need to build the module with Python 2.6 or newer, and
> this is where my trouble starts. I have Python 3.1.1 installed in
> /Library/Frameworks/Python, and with this version of Python my  
> module's
> build fails miserably, like this:
>
> tharbad:~ patrick$
> /Library/Frameworks/Python.framework/Versions/3.1/bin/python3.1 ./
> setup.py
> build_ext --inplace
> running build_ext
> building 'aprmd5' extension
> gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk
> -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3
> -I/Library/Frameworks/Python.framework/Versions/3.1/include/
> python3.1 -c
> aprmd5.c -o build/temp.macosx-10.3-fat-3.1/aprmd5.o
> aprmd5.c:43:60: aprmd5.c:43:60: error: apr-1/apr_md5.h: No such file  
> or
> directory
> [...]
>
> The problem here is that the gcc option "-isysroot" moves the root for
> system includes to the SDK directory (/Developer/SDKs/
> MacOSX10.4u.sdk),
> away from the normal root (/). Unfortunately, the SDK does not  
> contain the
> header <apr-1/apr_md5.h>, which causes the build to fail. Where does
> "-isysroot" come from? Not from my module, it's added automatically by
> distutils.
>
> I believe I have correctly diagnosed the problem, and I can  
> certainly work
> around it *somehow* in my module, but since I am rather new to  
> Python I
> have trouble deciding on a solution that is appropriate. I am  
> therefore
> writing to this list, hoping someone reading this can give me some  
> advice.
>
> 1) Is my problem a known situation for which there is a generally
> accepted, best-practice solution? If so, a pointer in the right  
> direction
> would be most welcome. If not, what would seem to be a good  
> solution? Make
> my own build of libaprutil using the MacOSX10.4u SDK? Rely on a
> third-party build of libaprutil (e.g. fink)? Force the compiler to  
> use the
> system's libaprutil?

The OSX installers are build to explicitly use the 10.4u SDK. That's  
mostly done to ensure that people on OSX 10.4 can build extensions out  
of the box.

Python 2.6.3 (just released) contains some code that automaticly  
disables usage of the SDK when it is not present.

You can disable usage of the 10.4u SDK by adding the following  
arguments to the definition of your Extension in setup.py:

         extra_compile_args=['-isysroot', '/'],  extra_link_args=['-
isysroot', '/']

Ronald
_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIG@...
http://mail.python.org/mailman/listinfo/pythonmac-sig

[Ronald Oussoren]

On 30 Sep, 2009, at 18:17, Dave Peterson wrote:

> Hi Patrick,
>
> Patrick Näf wrote:
>> ... my module's
>> build fails miserably, like this:
>>
>> tharbad:~ patrick$
>> /Library/Frameworks/Python.framework/Versions/3.1/bin/python3.1 ./
>> setup.py
>> build_ext --inplace
>> running build_ext
>> building 'aprmd5' extension
>> gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk
>> -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3
>> -I/Library/Frameworks/Python.framework/Versions/3.1/include/
>> python3.1 -c
>> aprmd5.c -o build/temp.macosx-10.3-fat-3.1/aprmd5.o
>> aprmd5.c:43:60: aprmd5.c:43:60: error: apr-1/apr_md5.h: No such  
>> file or
>> directory
>> [...]
>>
>> The problem here is that the gcc option "-isysroot" moves the root  
>> for
>> system includes to the SDK directory (/Developer/SDKs/
>> MacOSX10.4u.sdk),
>> away from the normal root (/). Unfortunately, the SDK does not  
>> contain the
>> header <apr-1/apr_md5.h>, which causes the build to fail. Where does
>> "-isysroot" come from? Not from my module, it's added automatically  
>> by
>> distutils.
>>
>
> The options distutils uses come from the file at <python env root>/
> lib/python2.5/config/Makefile.  I don't think there is any distutils  
> API provided to override/change these, just ways to append to them  
> when specifying your extension configuration.  These options are  
> used because they are the same ones used when building that Python  
> distribution.
>
> So my first suggestion is that you try redefining the -isysroot  
> option in the flags you can specify in your extension's config.  
> There is a good chance the compiler will use the last specification  
> on the line rather than doing something else when it gets two values.
>
> If that doesn't work, the only other thing I can think of is to  
> temporarily edit the config/Makefile to remove / change the  
> inclusion of the -isysroot option.

Editing the Makefile is not necessary, distutils will automaticly do  
the right thing when you add '-isysroot' to the compile arguments in  
setup.py.

It should in general never be necessary to edit the Makefile inside  
your Python installation. Please file an issue in the tracker at python.org
  if you run into a usecase where you do have to do edit the Makefile.

BTW. I'd like to drop the usage of the 10.4u SDK for Python 2.7 and  
3.2 and use the default system headers instead (except on 10.4, where  
we have to use the SDK to enable universal builds). That annoyingly  
requires some research, it turns out that building a binary on 10.5  
with "-isysroot /" does not result in something that's fully usable on  
10.4 :-(

Ronald

>
> HTHs,
>
> -- Dave
>
> _______________________________________________
> Pythonmac-SIG maillist  -  Pythonmac-SIG@...
> http://mail.python.org/mailman/listinfo/pythonmac-sig

_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIG@...
http://mail.python.org/mailman/listinfo/pythonmac-sig