Call For Help! I need Windows/Python developers!

[R. Tyler Ballance-5]

If you're running Cheetah on Windows, then you likely already know that
it is currently (unfortunately) treated as a second class citizen
insofar that unless you grab a pre-compiled version of the C-NameMapper,
you're relegated to using the slower Python version of the NameMapper.

In addition to this, I've been experimenting in the "performance" branch
(http://github.com/rtyler/cheetah/tree/performance) with rewriting other
portions of Cheetah in C (this weekend I toyed with porting
src/Utils/VerifyType.py to C, cutting ~6-10% off dynamic compilation).

I'd like to get Cheetah's C code to a state that it can execute properly
on top of Windows so Windows developers can gain all the benefits of
the C-based code that Linux and Mac OS X users can.


I'm not sure what will be needed to compile C on Windows (it's been ages
since I've developed anything lower than C# on Windows) but I'm assuming
a version of Visual Studio (the free edition should work) will be
needed. If you look at line 41 of SetupConfig.py, you can disable the:
        if os.name == 'posix':
conditional in order to have setuptools *try* to compile the C
extensions.

Past this I'm not sure what else is needed, if you join #cheetah on
Freenode, I'd love to collaborate with somebody to get this set up
properly.


If we can get the NameMapper compiling properly, then new Cheetah
releases can be pushed out as installers for Windows to make sure that
Cheetah users on Windows will get the latest and greatest without
needing to run the compiler themselves.


Any takers? :)

Cheers

-R. Tyler Ballance
Slide, Inc.


------------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[James Abbatiello]

I've just compiled NameMapper under Windows.  There is only a minor
change needed to _namemapper.c.  See attached patch.  I compiled this
on Windows XP using Visual C++ 2008 Express (that's the free version).
 The Python version is 2.6.2 installed from the python.org installer.

I've also tried to get the tests running under Windows.  There are a
few problems here:
1) Use of commands.getstatusoutput().  This isn't supported on
Windows.  I've replaced this will subprocess.Popen which seems to be
the preferred new method.
2) Some tests assume that the shell will expand globs ("*.tmpl") and
the Windows shell doesn't do this.  I'm skipping these tests on
Windows.
3) One place assumed that temp files went in /tmp.  I fixed that to be
more general.
4) After running `setup.py install` there's no command-line "cheetah"
command installed.  The tests that try to shell out and run this
command don't work.  I've made a batch file and put it on my PATH to
work around this for now but I don't know what the proper long-term
solution is.  Use setuptools to create a wrapper .exe?
A patch addressing 1 through 3 is attached.

--
James Abbatiello



------------------------------------------------------------------------------

_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[James Abbatiello]

On Fri, Jun 26, 2009 at 12:16 AM, James Abbatiello<abbeyj@...> wrote:
> 2) Some tests assume that the shell will expand globs ("*.tmpl") and
> the Windows shell doesn't do this.  I'm skipping these tests on
> Windows.

R. Tyler Ballance suggested that it might be better to manually expand
the globs if running on Windows.  This will allow the tests to pass on
all platforms.  Here's a new patch that tries to do this.

--
James Abbatiello


------------------------------------------------------------------------------

_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[R. Tyler Ballance-5]

Hey James, reply inline..

On Fri, 26 Jun 2009, James Abbatiello wrote:

> On Fri, Jun 26, 2009 at 12:16 AM, James Abbatiello<abbeyj@...> wrote:
> > 2) Some tests assume that the shell will expand globs ("*.tmpl") and
> > the Windows shell doesn't do this.  I'm skipping these tests on
> > Windows.
>
> R. Tyler Ballance suggested that it might be better to manually expand
> the globs if running on Windows.  This will allow the tests to pass on
> all platforms.  Here's a new patch that tries to do this.

I went ahead and applied all three patches and added my own (44dcbb8)
which removes the @skipIf class decorator from the CheetahWrapper.py
test.

All tests (except Unicode.JBQ_UTF8_Test8.testDynamicCompile which was
already failing) are succeeding on Linux now, mind checking out the
'next' branch and verifying similar results on Windows?

Cheers

-R. Tyler Ballance
Slide, Inc.


------------------------------------------------------------------------------

_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[R. Tyler Ballance-5]

Hey James, reply inline..

On Fri, 26 Jun 2009, James Abbatiello wrote:

> I've just compiled NameMapper under Windows.  There is only a minor
> change needed to _namemapper.c.  See attached patch.  I compiled this
> on Windows XP using Visual C++ 2008 Express (that's the free version).
>  The Python version is 2.6.2 installed from the python.org installer.
>
> I've also tried to get the tests running under Windows.  There are a
> few problems here:
> 1) Use of commands.getstatusoutput().  This isn't supported on
> Windows.  I've replaced this will subprocess.Popen which seems to be
> the preferred new method.

After Hudson found the 2.3 compat issue I switched this to use
popen2.Popen4 as we discussed on IRC, seems to be working fine now (see
build 88: http://hudson.cheetahtemplate.org/job/Cheetah%20(next)/88/)

Please let me know if this works properly on Windows. (it's committed
and pushed, attaching the patch for good measure)
 

> 2) Some tests assume that the shell will expand globs ("*.tmpl") and
> the Windows shell doesn't do this.  I'm skipping these tests on
> Windows.

Your other patch fixes this nicely :)

> 3) One place assumed that temp files went in /tmp.  I fixed that to be
> more general.

Looks good to me

> 4) After running `setup.py install` there's no command-line "cheetah"
> command installed.  The tests that try to shell out and run this
> command don't work.  I've made a batch file and put it on my PATH to
> work around this for now but I don't know what the proper long-term
> solution is.  Use setuptools to create a wrapper .exe?
> A patch addressing 1 through 3 is attached.

setuptools currently knows enought to create a windows installer, what
might be worth creating is a .bat file which will jumpstart cheetah
(instead of a .exe)?

I'm not too Windows savvy at the moment, but I'm certainly down for the
cause ;)


Cheers

-R. Tyler Ballance
Slide, Inc.

From 962ed4aeea25f70e85dadf9e425e8d364475348c Mon Sep 17 00:00:00 2001
From: R. Tyler Ballance <tyler@...>
Date: Fri, 26 Jun 2009 00:09:00 -0700
Subject: [PATCH] Replace subprocess with popen2, which is Python 2.3 compatible

---
 src/Tests/CheetahWrapper.py |   35 ++++++++++++++++++-----------------
 1 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/src/Tests/CheetahWrapper.py b/src/Tests/CheetahWrapper.py
index 3e94964..dd58834 100644
--- a/src/Tests/CheetahWrapper.py
+++ b/src/Tests/CheetahWrapper.py
@@ -11,7 +11,8 @@ Besides unittest usage, recognizes the following command-line options:
      --output
         Show the output of each subcommand.  (Normally suppressed.)
 '''
-import subprocess, os, shutil, sys, tempfile
+import os, shutil, sys, tempfile
+import popen2
 import unittest_local_copy as unittest
 
 import re                                     # Used by listTests.
@@ -149,17 +150,17 @@ Found %(result)r"""
                   test.
            out: None.
         """
-        subproc = subprocess.Popen(cmd,
-                                   stdout=subprocess.PIPE,
-                                   stderr=subprocess.STDOUT,
-                                   shell=True)
-        output = subproc.communicate()[0]
-        if subproc.returncode >= 0:
-            status = subproc.returncode
+        proc = popen2.Popen4(cmd)
+        status = proc.wait()
+        output = proc.fromchild.read()
+        proc.fromchild.close()
+
+        if status >= 0:
+            status = status
             signal = 0
         else:
             status = 0
-            signal = -subproc.returncode
+            signal = -status
         if OUTPUT:
             if output.endswith("\n"):
                 output = output[:-1]
@@ -186,17 +187,17 @@ Found %(result)r"""
            in : cmd, string, the command to run.
            out: None.
         """
-        subproc = subprocess.Popen(cmd,
-                                   stdout=subprocess.PIPE,
-                                   stderr=subprocess.STDOUT,
-                                   shell=True)
-        output = subproc.communicate()[0]
-        if subproc.returncode >= 0:
-            status = subproc.returncode
+        proc = popen2.Popen4(cmd)
+        status = proc.wait()
+        output = proc.fromchild.read()
+        proc.fromchild.close()
+
+        if status >= 0:
+            status = status
             signal = 0
         else:
             status = 0
-            signal = -subproc.returncode
+            signal = -status
         msg = "subcommand killed by signal %s: %s" % (signal, cmd)
         self.failUnlessEqual(signal, 0, msg) # Signal must be 0.
         msg = "subcommand exit status %s: %s" % (status, cmd)
--
1.6.3



------------------------------------------------------------------------------

_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[James Abbatiello]

On Fri, Jun 26, 2009 at 3:17 AM, R. Tyler Ballance<tyler@...> wrote:
> After Hudson found the 2.3 compat issue I switched this to use
> popen2.Popen4 as we discussed on IRC, seems to be working fine now (see
> build 88: http://hudson.cheetahtemplate.org/job/Cheetah%20(next)/88/)
>
> Please let me know if this works properly on Windows. (it's committed
> and pushed, attaching the patch for good measure)

Unfortunately it does not.  I missed the note in the manual but it
says popen2.Popen4 is not available on Windows.  I've changed it to
use os.popen4 which is available.  Patch attached.

>> 4) After running `setup.py install` there's no command-line "cheetah"
>> command installed.  The tests that try to shell out and run this
>> command don't work.  I've made a batch file and put it on my PATH to
>> work around this for now but I don't know what the proper long-term
>> solution is.  Use setuptools to create a wrapper .exe?
>> A patch addressing 1 through 3 is attached.
>
> setuptools currently knows enought to create a windows installer, what
> might be worth creating is a .bat file which will jumpstart cheetah
> (instead of a .exe)?
>
> I'm not too Windows savvy at the moment, but I'm certainly down for the
> cause ;)

A batch file is certainly possible.  The one I'm using right now uses
NT-specific syntax to be able to find python.exe and "cheetah" (the
small Python script).  I think the reason that setuptools makes .exe
files is probably for compatibility with Win9x.  All the .exe file
does is find the proper python.exe and launches it with the name of a
stub python script that does the proper thing.  I'll try finding some
docs on this to see how it is configured.

--
James Abbatiello


------------------------------------------------------------------------------

_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss