[PATCH] Use cStringIO instead of array in DummyTransaction

[tyler-53]

From: Arun Kumar <arunkakorp@...>

Signed-off-by: R. Tyler Ballance <tyler@...>
---
 cheetah/DummyTransaction.py |   27 +++++++++++++++++++--------
 1 files changed, 19 insertions(+), 8 deletions(-)
 mode change 100644 => 100755 cheetah/DummyTransaction.py

diff --git a/cheetah/DummyTransaction.py b/cheetah/DummyTransaction.py
old mode 100644
new mode 100755
index 26d2ea7..8b30abf
--- a/cheetah/DummyTransaction.py
+++ b/cheetah/DummyTransaction.py
@@ -10,6 +10,11 @@ specific DummyTransaction or DummyResponse behavior
 
 import types
 
+try:
+    from cStringIO import StringIO
+except ImportError:
+    import StringIO
+
 class DummyResponseFailure(Exception):
     pass
 
@@ -20,24 +25,30 @@ class DummyResponse(object):
         servlet
     '''
     def __init__(self):
-        self._outputChunks = []
-
+        self._outputChunks = StringIO()
+      
     def flush(self):
         pass
         
     def write(self, value):
-        self._outputChunks.append(value)
+        self._outputChunks.write(value)
 
     def writeln(self, txt):
         write(txt)
         write('\n')
 
     def getvalue(self, outputChunks=None):
-        chunks = outputChunks or self._outputChunks
-        try:
-            return ''.join(chunks)
+        #chunks = outputChunks or self._outputChunks
+        try:
+            if outputChunks is not None:
+                return ''.join(outputChunks)
+            else:
+                return self._outputChunks.getvalue()
         except UnicodeDecodeError, ex:
-            nonunicode = [c for c in chunks if not isinstance(c, unicode)]
+            #not sure about the best way to check for non-unicode in StringIO
+            nonunicode = ''
+            if outputChunks:
+                nonunicode = [c for c in outputChunks if not isinstance(c, unicode)]
             raise DummyResponseFailure('''Looks like you're trying to mix encoded strings with Unicode strings
             (most likely utf-8 encoded ones)
 
@@ -82,7 +93,7 @@ class TransformerResponse(DummyResponse):
             if isinstance(_filter, types.TypeType):
                 _filter = _filter()
             return _filter.filter(output)
-        return output
+         return output
 
 
 class TransformerTransaction(object):
--
1.6.0.2


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[tyler-53]

I'm uncertain if there's any lossiness in encoding a unicode()
object 'utf-8' prior to writing to the stream, but this does
appear to inadvertantly fix another issue we had with mixing UTF-8
encoded strings and unicode() objects when filling a template.
---
 cheetah/DummyTransaction.py |    7 +++++--
 cheetah/Tests/Unicode.py    |    5 ++---
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/cheetah/DummyTransaction.py b/cheetah/DummyTransaction.py
index 8b30abf..e04a604 100755
--- a/cheetah/DummyTransaction.py
+++ b/cheetah/DummyTransaction.py
@@ -31,8 +31,11 @@ class DummyResponse(object):
         pass
         
     def write(self, value):
+        if isinstance(value, unicode):
+            value = value.encode('utf-8')
         self._outputChunks.write(value)
 
+
     def writeln(self, txt):
         write(txt)
         write('\n')
@@ -43,7 +46,7 @@ class DummyResponse(object):
             if outputChunks is not None:
                 return ''.join(outputChunks)
             else:
-                return self._outputChunks.getvalue()
+                return self._outputChunks.getvalue().decode('utf-8')
         except UnicodeDecodeError, ex:
             #not sure about the best way to check for non-unicode in StringIO
             nonunicode = ''
@@ -93,7 +96,7 @@ class TransformerResponse(DummyResponse):
             if isinstance(_filter, types.TypeType):
                 _filter = _filter()
             return _filter.filter(output)
-         return output
+        return output
 
 
 class TransformerTransaction(object):
diff --git a/cheetah/Tests/Unicode.py b/cheetah/Tests/Unicode.py
index e4499d9..d627503 100644
--- a/cheetah/Tests/Unicode.py
+++ b/cheetah/Tests/Unicode.py
@@ -168,15 +168,14 @@ class Unicode_in_SearchList_Test(CommandLineTest):
             'adjective' : u'\u0e22\u0e34\u0e19\u0e14\u0e35\u0e15\u0e49\u0e2d\u0e19\u0e23\u0e31\u0e1a'}])
         assert template.respond()
 
-    def test_ErrorReporting(self):
+    def test_Thai_utf8(self):
         utf8 = '\xe0\xb8\xa2\xe0\xb8\xb4\xe0\xb8\x99\xe0\xb8\x94\xe0\xb8\xb5\xe0\xb8\x95\xe0\xb9\x89\xe0\xb8\xad\xe0\xb8\x99\xe0\xb8\xa3\xe0\xb8\xb1\xe0\xb8\x9a'
 
         source = '''This is $adjective'''
         template = self.createAndCompile(source)
         assert template and issubclass(template, Template)
         template = template(searchList=[{'adjective' : utf8}])
-        self.failUnlessRaises(DummyTransaction.DummyResponseFailure, template.respond)
-
+        assert template.respond()
 
 
 if __name__ == '__main__':
--
1.6.0.2


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[tyler-53]

On Wed, 14 Oct 2009, R. Tyler Ballance wrote:

> I'm uncertain if there's any lossiness in encoding a unicode()
> object 'utf-8' prior to writing to the stream, but this does
> appear to inadvertantly fix another issue we had with mixing UTF-8
> encoded strings and unicode() objects when filling a template.

Worth noting that this patch series has been merged and pushed to
git://github.com/rtyler/cheetah.git/next

All tests are passing as well, I would really appreciate any testing you
all could spare with this.

Tar:
        http://github.com/rtyler/cheetah/tarball/cStringIO_alpha
Zip:
        http://github.com/rtyler/cheetah/zipball/cStringIO_alpha


Cheers

> ---
>  cheetah/DummyTransaction.py |    7 +++++--
>  cheetah/Tests/Unicode.py    |    5 ++---
>  2 files changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/cheetah/DummyTransaction.py b/cheetah/DummyTransaction.py
> index 8b30abf..e04a604 100755
> --- a/cheetah/DummyTransaction.py
> +++ b/cheetah/DummyTransaction.py
> @@ -31,8 +31,11 @@ class DummyResponse(object):
>          pass
>          
>      def write(self, value):
> +        if isinstance(value, unicode):
> +            value = value.encode('utf-8')
>          self._outputChunks.write(value)
>  
> +
>      def writeln(self, txt):
>          write(txt)
>          write('\n')
> @@ -43,7 +46,7 @@ class DummyResponse(object):
>              if outputChunks is not None:
>                  return ''.join(outputChunks)
>              else:
> -                return self._outputChunks.getvalue()
> +                return self._outputChunks.getvalue().decode('utf-8')
>          except UnicodeDecodeError, ex:
>              #not sure about the best way to check for non-unicode in StringIO
>              nonunicode = ''
> @@ -93,7 +96,7 @@ class TransformerResponse(DummyResponse):
>              if isinstance(_filter, types.TypeType):
>                  _filter = _filter()
>              return _filter.filter(output)
> -         return output
> +        return output
>  
>  
>  class TransformerTransaction(object):
> diff --git a/cheetah/Tests/Unicode.py b/cheetah/Tests/Unicode.py
> index e4499d9..d627503 100644
> --- a/cheetah/Tests/Unicode.py
> +++ b/cheetah/Tests/Unicode.py
> @@ -168,15 +168,14 @@ class Unicode_in_SearchList_Test(CommandLineTest):
>              'adjective' : u'\u0e22\u0e34\u0e19\u0e14\u0e35\u0e15\u0e49\u0e2d\u0e19\u0e23\u0e31\u0e1a'}])
>          assert template.respond()
>  
> -    def test_ErrorReporting(self):
> +    def test_Thai_utf8(self):
>          utf8 = '\xe0\xb8\xa2\xe0\xb8\xb4\xe0\xb8\x99\xe0\xb8\x94\xe0\xb8\xb5\xe0\xb8\x95\xe0\xb9\x89\xe0\xb8\xad\xe0\xb8\x99\xe0\xb8\xa3\xe0\xb8\xb1\xe0\xb8\x9a'
>  
>          source = '''This is $adjective'''
>          template = self.createAndCompile(source)
>          assert template and issubclass(template, Template)
>          template = template(searchList=[{'adjective' : utf8}])
> -        self.failUnlessRaises(DummyTransaction.DummyResponseFailure, template.respond)
> -
> +        assert template.respond()
>  
>  
>  if __name__ == '__main__':
> --
> 1.6.0.2
>

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[Aahz]

On Wed, Oct 14, 2009, R. Tyler Ballance wrote:
>
> +try:
> +    from cStringIO import StringIO
> +except ImportError:
> +    import StringIO
> +

It's not clear what the goal is.  IIRC, using ''.join() is roughly
comparable in performance with cStringIO (which one is faster varies from
version to version), and if for whatever reason cStringIO is not
available, StringIO is definitely going to perform worse.
--
Aahz (aahz@...)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@...

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[tyler-53]

On Wed, 14 Oct 2009, Aahz wrote:

> On Wed, Oct 14, 2009, R. Tyler Ballance wrote:
> >
> > +try:
> > +    from cStringIO import StringIO
> > +except ImportError:
> > +    import StringIO
> > +
>
> It's not clear what the goal is.  IIRC, using ''.join() is roughly
> comparable in performance with cStringIO (which one is faster varies from
> version to version), and if for whatever reason cStringIO is not
> available, StringIO is definitely going to perform worse.

Gahh you're right, what was I thinking. I didn't actually add a
performance test prior to cherry-picking Arun's change. I had *assumed*
(first mistake) that cStringIO would be performing the majority of its
"work" on stream.getvalue() which isn't the case.

Looking at the performance tests that I just ran, as well as the
cStringIO.c code, stream.write() will actually copy (memcpy(2)) the
buffer into the stream's buffer. Whereas list.append() will add a
pointer to a list, periodically over-extending it to accomodate
additional append() calls.

What I find curious is that StringIO *effectively* does a bunch of
list.append() calls, but it is in fact slower than cStringIO (at least
on 2.6) as the code is performing some really silly ''.join() calls
and restructuring/rebuilding the array in the stream.write() call (given
certain conditions).

I'll discuss with Arun where he saw the performance improvements with
cStringIO when he wakes up, but for now I'm just going to partially back
these changes out.


Thanks for pointing the fail out Rahz :)


Cheers,
-R. Tyler Ballance


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[Aahz]

On Wed, Oct 14, 2009, tyler@... wrote:
>
> Thanks for pointing the fail out Rahz :)

You're welcome!

PS: it's not an "array", it's a "list"  ;-)

Side note: Python 2.5 adds an optimization for += on strings, but AFAIK
that only applies to CPython and of course is not available for earlier
versions.  I recommend still avoiding += for the indefinite future.
--
Aahz (aahz@...)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@...

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[Tavis Rudd]

Fyi, if I recall correctly, Cheetah originally used StringIO,
cStringIO if it was available, but I found ''.join([]) was actually
faster.  There's probably a note about it in the CHANGES file.

On Wed, 14 Oct 2009, tyler@... wrote:

>
> On Wed, 14 Oct 2009, Aahz wrote:
>
>> On Wed, Oct 14, 2009, R. Tyler Ballance wrote:
>>>
>>> +try:
>>> +    from cStringIO import StringIO
>>> +except ImportError:
>>> +    import StringIO
>>> +
>>
>> It's not clear what the goal is.  IIRC, using ''.join() is roughly
>> comparable in performance with cStringIO (which one is faster varies from
>> version to version), and if for whatever reason cStringIO is not
>> available, StringIO is definitely going to perform worse.
>
> Gahh you're right, what was I thinking. I didn't actually add a
> performance test prior to cherry-picking Arun's change. I had *assumed*
> (first mistake) that cStringIO would be performing the majority of its
> "work" on stream.getvalue() which isn't the case.
>
> Looking at the performance tests that I just ran, as well as the
> cStringIO.c code, stream.write() will actually copy (memcpy(2)) the
> buffer into the stream's buffer. Whereas list.append() will add a
> pointer to a list, periodically over-extending it to accomodate
> additional append() calls.
>
> What I find curious is that StringIO *effectively* does a bunch of
> list.append() calls, but it is in fact slower than cStringIO (at least
> on 2.6) as the code is performing some really silly ''.join() calls
> and restructuring/rebuilding the array in the stream.write() call (given
> certain conditions).
>
> I'll discuss with Arun where he saw the performance improvements with
> cStringIO when he wakes up, but for now I'm just going to partially back
> these changes out.
>
>
> Thanks for pointing the fail out Rahz :)
>
>
> Cheers,
> -R. Tyler Ballance
>

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[tyler-53]

On Thu, 15 Oct 2009, Tavis Rudd wrote:

> Fyi, if I recall correctly, Cheetah originally used StringIO,
> cStringIO if it was available, but I found ''.join([]) was actually
> faster.  There's probably a note about it in the CHANGES file.


Indeed::

        1.0rc2 (Nov 19, 2005)

          !!!THIS RELEASE REQUIRES RECOMPILATION OF ALL COMPILED CHEETAH TEMPLATES!!!

          See my email to the cheetahtemplate-discuss list on Sat. Nov. 12th for more
          details on these changes:

          - faster list-based buffering in DummyTrans, rather than StringIO (my
                benchmarks showed it to be significantly faster.  collections.deque wasn't
                any faster than a simple list.) [TR]


That was so long ago! I wasn't even aware of Python at that point I
don't think ;)


Cheers

>
> On Wed, 14 Oct 2009, tyler@... wrote:
>
> >
> >On Wed, 14 Oct 2009, Aahz wrote:
> >
> >>On Wed, Oct 14, 2009, R. Tyler Ballance wrote:
> >>>
> >>>+try:
> >>>+    from cStringIO import StringIO
> >>>+except ImportError:
> >>>+    import StringIO
> >>>+
> >>
> >>It's not clear what the goal is.  IIRC, using ''.join() is roughly
> >>comparable in performance with cStringIO (which one is faster varies from
> >>version to version), and if for whatever reason cStringIO is not
> >>available, StringIO is definitely going to perform worse.
> >
> >Gahh you're right, what was I thinking. I didn't actually add a
> >performance test prior to cherry-picking Arun's change. I had *assumed*
> >(first mistake) that cStringIO would be performing the majority of its
> >"work" on stream.getvalue() which isn't the case.
> >
> >Looking at the performance tests that I just ran, as well as the
> >cStringIO.c code, stream.write() will actually copy (memcpy(2)) the
> >buffer into the stream's buffer. Whereas list.append() will add a
> >pointer to a list, periodically over-extending it to accomodate
> >additional append() calls.
> >
> >What I find curious is that StringIO *effectively* does a bunch of
> >list.append() calls, but it is in fact slower than cStringIO (at least
> >on 2.6) as the code is performing some really silly ''.join() calls
> >and restructuring/rebuilding the array in the stream.write() call (given
> >certain conditions).
> >
> >I'll discuss with Arun where he saw the performance improvements with
> >cStringIO when he wakes up, but for now I'm just going to partially back
> >these changes out.
> >
> >
> >Thanks for pointing the fail out Rahz :)
> >
> >
> >Cheers,
> >-R. Tyler Ballance
> >

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss