Windows CRT issues with Python 2.6

[Farshid Lashkari-3]

Hello,

I've encountered an issue with distributing a Python 2.6 and PIL based application on Windows computers that do not have the VC 9.0 CRT installed. The problem is that the _imaging.pyd module fails to load. I believe the reason is that the file has an embedded manifest specifying the CRT version to load, which causes it to ignore the CRT files in the Python exe folder. The pyd files that ship with Python (_socket.pyd, etc..) do not have this problem because they are built without an embedded manifest, which causes them to inherit the CRT that is already loaded through the Python process. You can find information about this at the following bug report (http://bugs.python.org/issue4120).

Would it be possible to modify the PIL build process to not embed a manifest for Windows pyd files?

Cheers,

Farshid

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

[Fredrik Lundh]

Make sense, but I'm not sure I can figure out what they actually ended
up doing about it from that thread.  Anyone who can summarize and/or
provide a patch?

</F>

On Thu, Sep 10, 2009 at 8:00 PM, Farshid Lashkari <flashk@...> wrote:

> Hello,
> I've encountered an issue with distributing a Python 2.6 and PIL based
> application on Windows computers that do not have the VC 9.0 CRT installed.
> The problem is that the _imaging.pyd module fails to load. I believe the
> reason is that the file has an embedded manifest specifying the CRT version
> to load, which causes it to ignore the CRT files in the Python exe folder.
> The pyd files that ship with Python (_socket.pyd, etc..) do not have this
> problem because they are built without an embedded manifest, which causes
> them to inherit the CRT that is already loaded through the Python process.
> You can find information about this at the following bug report
> (http://bugs.python.org/issue4120).
> Would it be possible to modify the PIL build process to not embed a manifest
> for Windows pyd files?
> Cheers,
> Farshid
> _______________________________________________
> Image-SIG maillist  -  Image-SIG@...
> http://mail.python.org/mailman/listinfo/image-sig
>
>

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

[Koen van de Sande]

The fix was applied to the main Python distribution (2.6.2 has it for
sure, 2.6.0/2.6.1 does not). Any extension built with 2.6.2 or higher
should be fine, as it will not get an embedded manifest.
So just rebuild with Python Windows version 2.6.2. I see in the PIL
source tree that it uses a setup.py build_ext call, so no changes to PIL
should be needed.

Koen
_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

[Fredrik Lundh]

Ah, sneaky.  I'm pretty sure the current binary distribution was built
when 2.6 was new, and I didn't really expect them to tweak
build-related issues in an incremental release :)

I'll push out a new build asap; will ping the list when it's available.

Thanks!

</F>

On Mon, Sep 28, 2009 at 3:46 PM, Koen van de Sande
<koenvandesande@...> wrote:

> The fix was applied to the main Python distribution (2.6.2 has it for sure,
> 2.6.0/2.6.1 does not). Any extension built with 2.6.2 or higher should be
> fine, as it will not get an embedded manifest.
> So just rebuild with Python Windows version 2.6.2. I see in the PIL source
> tree that it uses a setup.py build_ext call, so no changes to PIL should be
> needed.
>
> Koen
> _______________________________________________
> Image-SIG maillist  -  Image-SIG@...
> http://mail.python.org/mailman/listinfo/image-sig
>

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

[Fredrik Lundh]

I've replaced the 1.1.6 binary at effbot.org/downloads with one built
under 2.6.2.  The installer is 130k larger thanks to a 3x larger
wininst executable (!?), but the binary extensions are slightly
smaller than before, so I'll assume it's the manifests that have
disappeared.

</F>

On Mon, Sep 28, 2009 at 4:07 PM, Fredrik Lundh <fredrik@...> wrote:

> Ah, sneaky.  I'm pretty sure the current binary distribution was built
> when 2.6 was new, and I didn't really expect them to tweak
> build-related issues in an incremental release :)
>
> I'll push out a new build asap; will ping the list when it's available.
>
> Thanks!
>
> </F>
>
> On Mon, Sep 28, 2009 at 3:46 PM, Koen van de Sande
> <koenvandesande@...> wrote:
>> The fix was applied to the main Python distribution (2.6.2 has it for sure,
>> 2.6.0/2.6.1 does not). Any extension built with 2.6.2 or higher should be
>> fine, as it will not get an embedded manifest.
>> So just rebuild with Python Windows version 2.6.2. I see in the PIL source
>> tree that it uses a setup.py build_ext call, so no changes to PIL should be
>> needed.
>>
>> Koen
>> _______________________________________________
>> Image-SIG maillist  -  Image-SIG@...
>> http://mail.python.org/mailman/listinfo/image-sig
>>
>

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

[Farshid Lashkari-3]

I just tried the new installer (http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.6.exe), but the pyd files still contain embedded manifests. The installer says it was built Oct 1, 2009 with Python 2.6.2, so that should be the latest version.

I took a closer look at the changes made from that bug report. The change was only made to the PCbuild/pyd.vsprops file, which I don't believe will affect 3rd party extensions. I see that Koen submitted a patch for msvc9compiler.py that will strip the runtime from the manifest, but that doesn't appear to have been committed. So I'm not really sure how 3rd party extension developers can build pyd files without embedded manifests.

On Thu, Oct 1, 2009 at 4:10 AM, Fredrik Lundh <fredrik@...> wrote:

I've replaced the 1.1.6 binary at effbot.org/downloads with one built
under 2.6.2.  The installer is 130k larger thanks to a 3x larger
wininst executable (!?), but the binary extensions are slightly
smaller than before, so I'll assume it's the manifests that have
disappeared.

</F>

On Mon, Sep 28, 2009 at 4:07 PM, Fredrik Lundh <fredrik@...> wrote:
> Ah, sneaky.  I'm pretty sure the current binary distribution was built
> when 2.6 was new, and I didn't really expect them to tweak
> build-related issues in an incremental release :)
>
> I'll push out a new build asap; will ping the list when it's available.
>
> Thanks!
>
> </F>
>
> On Mon, Sep 28, 2009 at 3:46 PM, Koen van de Sande
> <koenvandesande@...> wrote:
>> The fix was applied to the main Python distribution (2.6.2 has it for sure,
>> 2.6.0/2.6.1 does not). Any extension built with 2.6.2 or higher should be
>> fine, as it will not get an embedded manifest.
>> So just rebuild with Python Windows version 2.6.2. I see in the PIL source
>> tree that it uses a setup.py build_ext call, so no changes to PIL should be
>> needed.
>>
>> Koen
>> _______________________________________________
>> Image-SIG maillist  -  Image-SIG@...
>> http://mail.python.org/mailman/listinfo/image-sig
>>
>
_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

[Fredrik Lundh]

Ok, I can apply Koen's patch locally and rebuild, if that makes sense.

What's the quickest way to look for embedded manifests in the PYD:s?
dumpbin doesn't seem to say anything about that, so I assume I have to
use some other tool.

</F>

On Fri, Oct 2, 2009 at 12:10 AM, Farshid Lashkari <flashk@...> wrote:

> I just tried the new installer
> (http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.6.exe), but the pyd
> files still contain embedded manifests. The installer says it was built Oct
> 1, 2009 with Python 2.6.2, so that should be the latest version.
> I took a closer look at the changes made from that bug report. The change
> was only made to the PCbuild/pyd.vsprops file, which I don't believe will
> affect 3rd party extensions. I see that Koen submitted a patch
> for msvc9compiler.py that will strip the runtime from the manifest, but that
> doesn't appear to have been committed. So I'm not really sure how 3rd party
> extension developers can build pyd files without embedded manifests.
> On Thu, Oct 1, 2009 at 4:10 AM, Fredrik Lundh <fredrik@...>
> wrote:
>>
>> I've replaced the 1.1.6 binary at effbot.org/downloads with one built
>> under 2.6.2.  The installer is 130k larger thanks to a 3x larger
>> wininst executable (!?), but the binary extensions are slightly
>> smaller than before, so I'll assume it's the manifests that have
>> disappeared.
>>
>> </F>
>>
>> On Mon, Sep 28, 2009 at 4:07 PM, Fredrik Lundh <fredrik@...>
>> wrote:
>> > Ah, sneaky.  I'm pretty sure the current binary distribution was built
>> > when 2.6 was new, and I didn't really expect them to tweak
>> > build-related issues in an incremental release :)
>> >
>> > I'll push out a new build asap; will ping the list when it's available.
>> >
>> > Thanks!
>> >
>> > </F>
>> >
>> > On Mon, Sep 28, 2009 at 3:46 PM, Koen van de Sande
>> > <koenvandesande@...> wrote:
>> >> The fix was applied to the main Python distribution (2.6.2 has it for
>> >> sure,
>> >> 2.6.0/2.6.1 does not). Any extension built with 2.6.2 or higher should
>> >> be
>> >> fine, as it will not get an embedded manifest.
>> >> So just rebuild with Python Windows version 2.6.2. I see in the PIL
>> >> source
>> >> tree that it uses a setup.py build_ext call, so no changes to PIL
>> >> should be
>> >> needed.
>> >>
>> >> Koen
>> >> _______________________________________________
>> >> Image-SIG maillist  -  Image-SIG@...
>> >> http://mail.python.org/mailman/listinfo/image-sig
>> >>
>> >
>> _______________________________________________
>> Image-SIG maillist  -  Image-SIG@...
>> http://mail.python.org/mailman/listinfo/image-sig
>
>

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

[Koen van de Sande]

Open it in a hex editor, and look for the word manifest. This is what it
looks like with dependency:

<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
    <security>
      <requestedPrivileges>
        <requestedExecutionLevel level="asInvoker"
uiAccess="false"></requestedExecutionLevel>
      </requestedPrivileges>
    </security>
  </trustInfo>
  <dependency>
    <dependentAssembly>
      <assemblyIdentity type="win32" name="Microsoft.VC90.CRT"
version="9.0.21022.8" processorArchitecture="x86"
publicKeyToken="1fc8b3b9a1e18e3b"></assemblyIdentity>
    </dependentAssembly>
  </dependency>
</assembly>PAPADDINGXXPADDINGPADDINGX

and after the patch it looks like this:

<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
    <security>
      <requestedPrivileges>
        <requestedExecutionLevel level="asInvoker"
uiAccess="false"></requestedExecutionLevel>
      </requestedPrivileges>
    </security>
  </trustInfo>
  <dependency>
    <dependentAssembly>
     
    </dependentAssembly>
  </dependency>
</assembly>PAPAD

I've pinged the issue, as the patch should be applied to the main Python
distribution.

Koen

Fredrik Lundh wrote:

> Ok, I can apply Koen's patch locally and rebuild, if that makes sense.
>
> What's the quickest way to look for embedded manifests in the PYD:s?
> dumpbin doesn't seem to say anything about that, so I assume I have to
> use some other tool.
>
> </F>
>
> On Fri, Oct 2, 2009 at 12:10 AM, Farshid Lashkari <flashk@...> wrote:
>  
>> I just tried the new installer
>> (http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.6.exe), but the pyd
>> files still contain embedded manifests. The installer says it was built Oct
>> 1, 2009 with Python 2.6.2, so that should be the latest version.
>> I took a closer look at the changes made from that bug report. The change
>> was only made to the PCbuild/pyd.vsprops file, which I don't believe will
>> affect 3rd party extensions. I see that Koen submitted a patch
>> for msvc9compiler.py that will strip the runtime from the manifest, but that
>> doesn't appear to have been committed. So I'm not really sure how 3rd party
>> extension developers can build pyd files without embedded manifests.
>> On Thu, Oct 1, 2009 at 4:10 AM, Fredrik Lundh <fredrik@...>
>> wrote:
>>    
>>> I've replaced the 1.1.6 binary at effbot.org/downloads with one built
>>> under 2.6.2.  The installer is 130k larger thanks to a 3x larger
>>> wininst executable (!?), but the binary extensions are slightly
>>> smaller than before, so I'll assume it's the manifests that have
>>> disappeared.
>>>
>>> </F>
>>>
>>> On Mon, Sep 28, 2009 at 4:07 PM, Fredrik Lundh <fredrik@...>
>>> wrote:
>>>      
>>>> Ah, sneaky.  I'm pretty sure the current binary distribution was built
>>>> when 2.6 was new, and I didn't really expect them to tweak
>>>> build-related issues in an incremental release :)
>>>>
>>>> I'll push out a new build asap; will ping the list when it's available.
>>>>
>>>> Thanks!
>>>>
>>>> </F>
>>>>
>>>> On Mon, Sep 28, 2009 at 3:46 PM, Koen van de Sande
>>>> <koenvandesande@...> wrote:
>>>>        
>>>>> The fix was applied to the main Python distribution (2.6.2 has it for
>>>>> sure,
>>>>> 2.6.0/2.6.1 does not). Any extension built with 2.6.2 or higher should
>>>>> be
>>>>> fine, as it will not get an embedded manifest.
>>>>> So just rebuild with Python Windows version 2.6.2. I see in the PIL
>>>>> source
>>>>> tree that it uses a setup.py build_ext call, so no changes to PIL
>>>>> should be
>>>>> needed.
>>>>>
>>>>> Koen
>>>>> _______________________________________________
>>>>> Image-SIG maillist  -  Image-SIG@...
>>>>> http://mail.python.org/mailman/listinfo/image-sig
>>>>>
>>>>>          
>>> _______________________________________________
>>> Image-SIG maillist  -  Image-SIG@...
>>> http://mail.python.org/mailman/listinfo/image-sig
>>>      
>>    

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

[Christoph Gohlke]

Hi,

you can use Dependency Walker <http://www.dependencywalker.com/>. If the
patch is working, pyd files will depend on MSVCR90.DLL, otherwise
%WINDIR%\winsxs\...\MSVCR90.DLL

There are two issues with Koen's patch (both easy to fix). It does not
work for 64-bit Python and it also processes exe files build with
setuptools.

Christoph

On 11:59, Fredrik Lundh wrote:

> Ok, I can apply Koen's patch locally and rebuild, if that makes sense.
>
> What's the quickest way to look for embedded manifests in the PYD:s?
> dumpbin doesn't seem to say anything about that, so I assume I have to
> use some other tool.
>
> </F>
>
> On Fri, Oct 2, 2009 at 12:10 AM, Farshid Lashkari<flashk@...>  wrote:
>> I just tried the new installer
>> (http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.6.exe), but the pyd
>> files still contain embedded manifests. The installer says it was built Oct
>> 1, 2009 with Python 2.6.2, so that should be the latest version.
>> I took a closer look at the changes made from that bug report. The change
>> was only made to the PCbuild/pyd.vsprops file, which I don't believe will
>> affect 3rd party extensions. I see that Koen submitted a patch
>> for msvc9compiler.py that will strip the runtime from the manifest, but that
>> doesn't appear to have been committed. So I'm not really sure how 3rd party
>> extension developers can build pyd files without embedded manifests.
>> On Thu, Oct 1, 2009 at 4:10 AM, Fredrik Lundh<fredrik@...>
>> wrote:
>>>
>>> I've replaced the 1.1.6 binary at effbot.org/downloads with one built
>>> under 2.6.2.  The installer is 130k larger thanks to a 3x larger
>>> wininst executable (!?), but the binary extensions are slightly
>>> smaller than before, so I'll assume it's the manifests that have
>>> disappeared.
>>>
>>> </F>
>>>
>>> On Mon, Sep 28, 2009 at 4:07 PM, Fredrik Lundh<fredrik@...>
>>> wrote:
>>>> Ah, sneaky.  I'm pretty sure the current binary distribution was built
>>>> when 2.6 was new, and I didn't really expect them to tweak
>>>> build-related issues in an incremental release :)
>>>>
>>>> I'll push out a new build asap; will ping the list when it's available.
>>>>
>>>> Thanks!
>>>>
>>>> </F>
>>>>
>>>> On Mon, Sep 28, 2009 at 3:46 PM, Koen van de Sande
>>>> <koenvandesande@...>  wrote:
>>>>> The fix was applied to the main Python distribution (2.6.2 has it for
>>>>> sure,
>>>>> 2.6.0/2.6.1 does not). Any extension built with 2.6.2 or higher should
>>>>> be
>>>>> fine, as it will not get an embedded manifest.
>>>>> So just rebuild with Python Windows version 2.6.2. I see in the PIL
>>>>> source
>>>>> tree that it uses a setup.py build_ext call, so no changes to PIL
>>>>> should be
>>>>> needed.
>>>>>
>>>>> Koen
>>>>> _______________________________________________
>>>>> Image-SIG maillist  -  Image-SIG@...
>>>>> http://mail.python.org/mailman/listinfo/image-sig
>>>>>
>>>>
>>> _______________________________________________
>>> Image-SIG maillist  -  Image-SIG@...
>>> http://mail.python.org/mailman/listinfo/image-sig
>>
>>
>

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

[Farshid Lashkari-3]

I use Resource Hacker (http://www.angusj.com/resourcehacker/), it allows you to view and edit all embedded resources of EXE/DLL files. It's a very handy application. Dependency Walker is really nice too.

Regarding the issue, wouldn't it just be better to compile the pyd without any embedded manifest file, instead of trying to parse and remove certain sections? I'm not familiar with how distutils works, but for my own Visual Studio projects I pass the /MANIFEST:NO option to the linker and it will not generate an embedded manifest file.

On Fri, Oct 2, 2009 at 8:36 AM, Christoph Gohlke <cgohlke@...> wrote:

Hi,

you can use Dependency Walker <http://www.dependencywalker.com/>. If the patch is working, pyd files will depend on MSVCR90.DLL, otherwise %WINDIR%\winsxs\...\MSVCR90.DLL

There are two issues with Koen's patch (both easy to fix). It does not work for 64-bit Python and it also processes exe files build with setuptools.

Christoph

On 11:59, Fredrik Lundh wrote:

Ok, I can apply Koen's patch locally and rebuild, if that makes sense.

What's the quickest way to look for embedded manifests in the PYD:s?
dumpbin doesn't seem to say anything about that, so I assume I have to
use some other tool.

</F>

On Fri, Oct 2, 2009 at 12:10 AM, Farshid Lashkari<flashk@...>  wrote:

I just tried the new installer
(http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.6.exe), but the pyd
files still contain embedded manifests. The installer says it was built Oct
1, 2009 with Python 2.6.2, so that should be the latest version.
I took a closer look at the changes made from that bug report. The change
was only made to the PCbuild/pyd.vsprops file, which I don't believe will
affect 3rd party extensions. I see that Koen submitted a patch
for msvc9compiler.py that will strip the runtime from the manifest, but that
doesn't appear to have been committed. So I'm not really sure how 3rd party
extension developers can build pyd files without embedded manifests.
On Thu, Oct 1, 2009 at 4:10 AM, Fredrik Lundh<fredrik@...>
wrote:

I've replaced the 1.1.6 binary at effbot.org/downloads with one built
under 2.6.2.  The installer is 130k larger thanks to a 3x larger
wininst executable (!?), but the binary extensions are slightly
smaller than before, so I'll assume it's the manifests that have
disappeared.

</F>

On Mon, Sep 28, 2009 at 4:07 PM, Fredrik Lundh<fredrik@...>
wrote:

Ah, sneaky.  I'm pretty sure the current binary distribution was built
when 2.6 was new, and I didn't really expect them to tweak
build-related issues in an incremental release :)

I'll push out a new build asap; will ping the list when it's available.

Thanks!

</F>

On Mon, Sep 28, 2009 at 3:46 PM, Koen van de Sande
<koenvandesande@...>  wrote:

The fix was applied to the main Python distribution (2.6.2 has it for
sure,
2.6.0/2.6.1 does not). Any extension built with 2.6.2 or higher should
be
fine, as it will not get an embedded manifest.
So just rebuild with Python Windows version 2.6.2. I see in the PIL
source
tree that it uses a setup.py build_ext call, so no changes to PIL
should be
needed.

Koen
_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig

[Koen van de Sande]

To check for the manifest, I just open it up in a hex editor and look
for the word "manifest". That will get you to the XML part of the file,
and you can see what's happening.
Please note that if you turn manifests off, you assume that your
extension has no other dependencies. And for things like TCL/TK, Windows
Common Controls, there can be additional dependencies in the XML file.
So, only remove the dependency on the runtimes, otherwise you break
things with additional dependencies.

Koen
_______________________________________________
Image-SIG maillist  -  Image-SIG@...
http://mail.python.org/mailman/listinfo/image-sig