Nabble - Catalyst Web Framework

Organize tests in sub-directories

2009-11-24T12:31:18Z

Hello,

I would like to structure the t directory as lib/MyApp is structured. Is it
possible to be able to configure the app somehow so `make test` checks all
the test files even if they are in sub-directories?

`prove -r -l t` can do this, but it works well only under Linux or if I also
add the -j parameter and a big enough number of parallel jobs, so I'm not
sure it will always work.

Thanks.

Octavian

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Unicode trouble with Catalyst::Engine::FastCGI

2009-11-24T08:41:03Z

* Tomas Doran <bobtfish@...> [2009-11-23 23:20]:
> This probably means (I'm guessing) that the xs part of FCGI
> doesn't correctly handle buffers which are characters rather
> than bytes.

That was my initial guess.

> We can get this fixed, but not without test cases, and I'm
> struggling to reproduce the issue here.
>
> Help?

Glad that Bernhard stepped up because I’ve never used FCGI. :-)

* Bernhard Graf <catalyst4@...> [2009-11-24 13:40]:
> As a quick fix, we could utf8::downgrade($buffer) in
> Catalyst::Engine::FastCGI right before syswrite. Doesn't hurt,
> as far as I understand.

It wouldn’t even necessarily be a quick fix in this case, IMO. It
would be preferrable to fix it in FCGI, but if the maintainer is
gone and a new version is not likely to happen, then downgrading
in the FastCGI engine would be the correct and proper fix. Any
fix in the XS code would effectively imply a transient downgrade
anyway. If the body string represents an encoded octet sequence,
then downgrading will always succeed, as well.

(If it’s not downgradeable, then the engine should probably throw
an exception, as mentioned in the other thread.)

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Re: Unicode trouble with Catalyst::Engine::FastCGI

2009-11-24T04:32:35Z

Tomas Doran schrieb:

> The FastCGI engine doesn't do anything with encoding/uft8, and nor does
> FCGI.pm
>
> This probably means (I'm guessing) that the xs part of FCGI doesn't
> correctly handle buffers which are characters rather than bytes.

I wrote a test program to keep all possible side effects away.
It is here http://scsys.co.uk:8001/36569 and it shows (using option -u),
that FCGI indeed encodes all data as soon as the utf-8 flag is on.

This is very interesting, because the latest FCGI release is nearly
seven years old. Did the utf-8 flag exist at that time at all?
Is there something in Perl's guts that does such encoding automatically?

> We can get this fixed, but not without test cases, and I'm struggling to
> reproduce the issue here.
>
> Help?

I know hardly anything about Perl XS code. Maybe someone with a clue
could have a look at FCGI.xs^H^HXL?

As a quick fix, we could utf8::downgrade($buffer) in
Catalyst::Engine::FastCGI right before syswrite. Doesn't hurt, as far as
I understand.

Bernhard

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: [ANNOUNCE] CALL FOR TESTING Catalyst-Runtime 5.80014_01

2009-11-24T02:04:14Z

At first glance all ok. I can't find any encoding issues with my
application (all templates and data in koi8-r). But I do not try to
print any chars, all data - bytes.

On Tue, Nov 24, 2009 at 02:54, Tomas Doran <bobtfish@...> wrote:

> Hot on the heels of Catalyst-Runtime 5.80014, I'd like to announce a
> development release, Catalyst 5.80014_01.
>
> This release has a potentially behavior changing fix for the CGI and FastCGI
> engines to do with outputting UTF-8. This fixes a serious bug with
> outputting utf-8 pages using Catalyst::Plugin::Unicode::Encoding at least..
>
> I _NEED_ people using FastCGI and dealing with high bit characters in their
> applications to test this release for me and let me know if they see any
> issues. If you do see any issues in your app which previously worked with
> any other 5.80 version, please reply to this mail or log a bug in RT.
>
> You can try it out by typing:
> cpan -i BOBTFISH/Catalyst-5.80014_01.tar.gz
>
> The changelog is included below as usual. Thanks in advance to anyone
> helping test this release.
>
> Cheers
> t0m
>
> --
> 5.80014_01 2009-11-22 20:01:23
>
> Bug fixes:
> - Filehandle now forced to binmode in CGI and FastCGI engines. This
> appears
> to correct some UTF-8 issues, but may break people's code which relies
> on the old behaviour.
>
> Refactoring / cleanups:
> - Plugins which inherit from Catalyst::Controller or Catalyst::Component
> are deprecated and now issue warnings.
>
>
> _______________________________________________
> List: Catalyst@...
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@.../
> Dev site: http://dev.catalyst.perl.org/
>

--
-- vovkasm

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst-dev] [ANNOUNCE] CALL FOR TESTING Catalyst-Runtime 5.80014_01

2009-11-24T01:36:33Z

Am Montag, den 23.11.2009, 23:54 +0000 schrieb Tomas Doran:
> You can try it out by typing:
> cpan -i BOBTFISH/Catalyst-5.80014_01.tar.gz

Setting TEST_HTTP=1 and TEST_THREADS=1, I got two test errors:

t/optional_http-server-restart.t ...................................... Can't locate Catalyst/Engine/HTTP/Restarter/Watcher.pm in @INC (@INC contains: /home/fw/tmp/Catalyst-Runtime-5.80014_01/inc /home/fw/tmp/Catalyst-Runtime-5.80014_01/blib/lib /home/fw/tmp/Catalyst-Runtime-5.80014_01/blib/arch /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at t/optional_http-server-restart.t line 16.
BEGIN failed--compilation aborted at t/optional_http-server-restart.t line 16.
t/optional_http-server-restart.t ...................................... Dubious, test returned 2 (wstat 512, 0x200)
No subtests run

and

t/optional_threads.t .................................................. 1/3
# Failed test 'Executed actions'
# at t/optional_threads.t line 54.
# got: 'TestApp::Controller::Action::Default->begin, TestApp::Controller::Action::Default->default, TestApp::View::Dump::Request->process, TestApp::Controller::Root->end'
# expected: 'TestApp::Controller::Action::Default->begin, TestApp::Controller::Action::Default->default, TestApp::View::Dump::Request->process, TestApp->end'
# Looks like you failed 1 test of 3.
t/optional_threads.t .................................................. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/3 subtests

The last one already has a ticket at http://rt.cpan.org/Public/Bug/Display.html?id=50039

Bye, Frank

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[ANNOUNCE] CALL FOR TESTING Catalyst-Runtime 5.80014_01

2009-11-23T15:54:14Z

Hot on the heels of Catalyst-Runtime 5.80014, I'd like to announce a
development release, Catalyst 5.80014_01.

This release has a potentially behavior changing fix for the CGI and
FastCGI engines to do with outputting UTF-8. This fixes a serious bug
with outputting utf-8 pages using Catalyst::Plugin::Unicode::Encoding
at least..

I _NEED_ people using FastCGI and dealing with high bit characters in
their applications to test this release for me and let me know if they
see any issues. If you do see any issues in your app which previously
worked with any other 5.80 version, please reply to this mail or log a
bug in RT.

You can try it out by typing:
cpan -i BOBTFISH/Catalyst-5.80014_01.tar.gz

The changelog is included below as usual. Thanks in advance to anyone
helping test this release.

Cheers
t0m

--
5.80014_01 2009-11-22 20:01:23

Bug fixes:
- Filehandle now forced to binmode in CGI and FastCGI engines.
This appears
to correct some UTF-8 issues, but may break people's code which
relies
on the old behaviour.

Refactoring / cleanups:
- Plugins which inherit from Catalyst::Controller or
Catalyst::Component
are deprecated and now issue warnings.

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: How to reduce the memory footprint?

2009-11-23T15:39:00Z

On Mon, Nov 23, 2009 at 10:45 PM, J. Shirley <jshirley@...> wrote:

I wouldn't trust the output of top/ps to determine how much memory is used.

-J

I do.
Not for individual processes, though. But for the totals top is perfectly fine for me.

I don't have many other processes running on the box. If I turn off catalyst, I use less than 250MB or memory. With Catalyst, I'm over 800MB with some occasional swapping.

When you say you go from less than 250 to over 800MB, it seems a little too much. What does "small app" mean in this case?

My ubuntu box does not have swapping (it's OpenVZ), when it goes over my allocated 256MB ram, out-of-memory hell breaks loose. So what I'm watching for is actually my total free memory.

Before starting up my Catalyst app:

top - 23:16:01 up 14 days, 4:57, 1 user, load average: 0.02, 0.02, 0.00
Tasks: 23 total,   2 running, 21 sleeping,   0 stopped,   0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:    262144k total,    25720k used,   236424k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

After startup (out-of-the-box HTTP::Prefork, small app), 30MB are gone:

top - 23:14:36 up 14 days, 4:56, 1 user, load average: 0.07, 0.03, 0.01
Tasks: 29 total,   2 running, 27 sleeping,   0 stopped,   0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:    262144k total,    55448k used,   206696k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND
3637 catw      18   0 28448 24m 1524 S 0.0 9.5   0:00.00 perl
3673 catw      20   0 28448 23m 748 S 0.0 9.2   0:00.00 perl
3675 catw      20   0 28448 23m 628 S 0.0 9.2   0:00.00 perl
3676 catw      20   0 28448 23m 628 S 0.0 9.2   0:00.00 perl
3677 catw      24   0 28448 23m 628 S 0.0 9.2   0:00.00 perl
3678 catw      25   0 28448 23m 628 S 0.0 9.2   0:00.00 perl

After a few thousand hits, total mem usage seems to level out at 62MB:

top - 23:48:13 up 14 days, 5:29, 1 user, load average: 0.02, 0.02, 0.00
Tasks: 29 total,   2 running, 27 sleeping,   0 stopped,   0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:    262144k total,    87264k used,   174880k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND
3637 catw      15   0 28560 24m 1524 S 0.0 9.5   0:00.01 perl
3673 catw      15   0 29320 25m 1616 S 0.0 9.9   0:00.24 perl
3675 catw      15   0 29308 25m 1616 S 0.0 9.9   0:00.28 perl
3676 catw      15   0 29308 25m 1616 S 0.0 9.9   0:00.25 perl
3677 catw      15   0 29308 25m 1616 S 0.0 9.9   0:00.20 perl
3678 catw      15   0 29244 25m 1616 S 0.0 9.9   0:00.22 perl

Maybe the Catalyst wiki should have some profiling and performance oriented info. That way people can have an idea on how to measure and what to expect from their apps. And what to expect after using some of the most popular modules (ie DBIC).

-rodrigo

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: How to reduce the memory footprint?

2009-11-23T14:39:39Z

On Mon, 23 Nov 2009, Julien Sobrier wrote:

> I run 5 instances of FastCGI. Each instance was taking about 90MB of memory.
> I tried to reduce the memory fooprint by reducing the number of libraries I
> used. The memory usage is now 120MB per instance! The memory increase is
> probably due to other changes.

If you can, take a look at smem (http://www.selenic.com/smem/) to see how
much memory these procs are _really_ using.

This tool looks at how much memory is shared between procs and gives you a
much better sense of how much memory it costs _per process_.

But you need to be on a relatively recent Linux kernel to use it.

-dave

/*============================================================
http://VegGuide.org http://blog.urth.org
Your guide to all that's veg House Absolute(ly Pointless)
============================================================*/

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Re: Unicode trouble with Catalyst::Engine::FastCGI

2009-11-23T14:23:17Z

On 23 Nov 2009, at 21:15, Aristotle Pagaltzis wrote:

> * Bernhard Graf <catalyst4@...> [2009-11-23 20:00]:
>> Aristotle Pagaltzis schrieb:
>>
>>>> While this fixes the problem, it is still unclear, why the
>>>> utf8 flag is set for the whole buffer.
>>>
>>> It shouldn’t matter.
>>
>> But it does.
>
> yes, because ::FastCGI is broken. :-) Is what I’m saying.

The FastCGI engine doesn't do anything with encoding/uft8, and nor
does FCGI.pm

This probably means (I'm guessing) that the xs part of FCGI doesn't
correctly handle buffers which are characters rather than bytes.

We can get this fixed, but not without test cases, and I'm struggling
to reproduce the issue here.

Help?

Cheers
t0m

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Re: Avoiding UTF8 in Catalyst

2009-11-23T14:20:05Z

On 23 Nov 2009, at 18:24, Marc SCHAEFER wrote:

> On Mon, Nov 23, 2009 at 05:43:25PM +0100, Aristotle Pagaltzis wrote:

>> I had an IRC convo with Tomas Doran last night and explained the
>> problem to him. He knocked out some tests for the broken
>
> Thank you for your time! It's nice to see the responsiveness of the
> project.

Now fixed in trunk:

http://dev.catalystframework.org/svnweb/Catalyst/revision?rev=11978

Please test it out for me to ensure this does fix your issue as
expected?

Cheers
t0m

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: How to reduce the memory footprint?

2009-11-23T13:45:41Z

On Mon, Nov 23, 2009 at 1:25 PM, Julien Sobrier <julien@...> wrote:

Right, nprocs=5

I thought 100M RES/300M VIRT meanst 100M per process, and 300M shard among perl processes.

I don't have many other processes running on the box. If I turn off catalyst, I use less than 250MB or memory. With Catalyst, I'm over 800MB with some occasional swapping.

How are you profiling your app (and determining how much memory is "free")? You can profile the memory usage of your app using the development server and get a very good idea of where the memory usage is at.

I wouldn't trust the output of top/ps to determine how much memory is used.

-J

Re: How to reduce the memory footprint?

2009-11-23T13:44:15Z

Julien Sobrier wrote on 11/23/2009 03:25 PM:
> Right, nprocs=5
>
> I thought 100M RES/300M VIRT meanst 100M per process, and 300M shard
> among perl processes.
>
> I don't have many other processes running on the box. If I turn off
> catalyst, I use less than 250MB or memory. With Catalyst, I'm over 800MB
> with some occasional swapping.

that amount of memory use doesn't seem excessive to me. I've had 250+MB
per process before for larger apps.

If you have only that much memory (~1G) available, do you really need to
be running 5 procs? Why not 2 or 3? If you run into bottlenecks with 3
preforked processes on your current hw, you'll need to be looking at
upgrading your hardware anyway.

--
Peter Karman . http://peknet.com/ . peter@...

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: How to reduce the memory footprint?

2009-11-23T13:25:05Z

Right, nprocs=5

I thought 100M RES/300M VIRT meanst 100M per process, and 300M shard among perl processes.

I don't have many other processes running on the box. If I turn off catalyst, I use less than 250MB or memory. With Catalyst, I'm over 800MB with some occasional swapping.

On Mon, Nov 23, 2009 at 1:00 PM, Tomas Doran <bobtfish@...> wrote:

On 23 Nov 2009, at 18:18, Julien Sobrier wrote:

Hello,
I have quite a small Catalyst application that runs with FastCGI + FCGI::ProcManager::MaxRequests

I run 5 instances of FastCGI. Each instance was taking about 90MB of memory. I tried to reduce the memory fooprint by reducing the number of libraries I used. The memory usage is now 120MB per instance! The memory increase is probably due to other changes.

When you say '5 instances', you mean you have nprocs = 5, right?

This _isn't_ using 100Mb per instance - all of the application is loaded into memory before fork() is called, so it will mostly be shared (by copy on write) memory.

Cheers
t0m

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Unicode trouble with Catalyst::Engine::FastCGI

2009-11-23T13:15:05Z

Hi Bernhard,

* Bernhard Graf <catalyst4@...> [2009-11-23 20:00]:
> Aristotle Pagaltzis schrieb:
>
>>> While this fixes the problem, it is still unclear, why the
>>> utf8 flag is set for the whole buffer.
>>
>> It shouldn’t matter.
>
> But it does.

yes, because ::FastCGI is broken. :-) Is what I’m saying.

>>> So Ladies and Gentleman, may I present you the culprit? It
>>> is YAML::XS! Everything read by YAML::XS
>>>
>>> perl -MYAML::XS -E '
>>> my $config = YAML::XS::LoadFile("myapp.yml");
>>> say((utf8::is_utf8($config->{name}) ? "is" : "is not"), " utf8");
>>> '
>>> is utf8
>>
>> No, that’s not the culprit.
>>
>> The culprit is Catalyst::Engine::FastCGI, which does not pay
>> attention to the UTF8 flag.
>
> Obviously YAML::XS doesn't do that either.

It does. It sets the flag to correctly reflect the state of the
internal byte buffer of the scalar so that its string value will
mean the right thing.

> But aside from that I agree with you, that something ist broken
> in F(ast)CGI. It seems more that F(ast)CGI pays attention to
> the utf8 flag where it shouldn't, because it seems to
> automatically utf8::encode the buffer due to the set utf8 flag.

Ah, that may be, yeah, it would produce the result you are
seeing, and would clearly be broken. C::E::FastCGI should just
output the string as it is and let perl worry about its meaning,
rather than (if that’s what it does) checking the UTF8 flag
itself and doing something broken in response.

> > No. YAML::XS is completely correct.
>
> No, it is not. Because it claims:
>
> This module exports the functions Dump and Load. These
> functions are intended to work exactly like YAML.pm's
> corresponding functions.
>
> And that's not the case. And therefore you can't use it as
> a drop-in-replacement for other YAML modules.

Hmm. I guess it depends on whether think of “drop-in” as
something that replicates behaviour down to internal
implementation details, or just functions identically on the
semantic level. When I see a module advertise itself like that
I assume only the latter.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Avoiding UTF8 in Catalyst

2009-11-23T13:08:32Z

* Bill Moseley <moseley@...> [2009-11-23 20:10]:
> I'd argue that when it's time to set the length it should die
> if utf8 flag is still set.

I’m of two minds about this… it may well be that a string is
correctly encoded but has gotten upgraded, and such a string will
produce the right output anyhow. I don’t know if it’s not too
stringent to demand that the UTF8 flag be off.

However, the string should be *downgradeable* by that time. If
there are wide characters in it at that time, then throwing an
exception is absolutely the right thing to do. But if there
aren’t, then you can’t decide based on the UTF8 flag whether the
string is correct or not.

As I wrote, you can read a binary file, upgrade the string, and
output it right back, and you’ll get an identical copy of the
file out of that, because a string means one and the same thing
regardless of whether it’s upgraded.

> When calculating the length the content should have already
> been encoded.

Yes.

> Again, at some point decoding and encoding should be core not
> just a plugin. It's an important part of the request cycle.

I agree.

Although it’s difficult to make it fully automatic because
browsers suck so bad about telling you what encoding the data
that they’re sending is in.

I am working on a plugin for that, but due to its dependencies
and API I don’t know if it’d be reasonable to make it core.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: How to reduce the memory footprint?

2009-11-23T13:00:31Z

On 23 Nov 2009, at 18:18, Julien Sobrier wrote:

> Hello,
> I have quite a small Catalyst application that runs with FastCGI +
> FCGI::ProcManager::MaxRequests
>
> I run 5 instances of FastCGI. Each instance was taking about 90MB of
> memory. I tried to reduce the memory fooprint by reducing the number
> of libraries I used. The memory usage is now 120MB per instance! The
> memory increase is probably due to other changes.

When you say '5 instances', you mean you have nprocs = 5, right?

This _isn't_ using 100Mb per instance - all of the application is
loaded into memory before fork() is called, so it will mostly be
shared (by copy on write) memory.

Cheers
t0m

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Re: Avoiding UTF8 in Catalyst

2009-11-23T11:08:04Z

On Mon, Nov 23, 2009 at 10:14 AM, Aristotle Pagaltzis <pagaltzis@...> wrote:

Encode the string to the destination encoding (not just character
set), so that the string represents an encoded octet stream, and
then look at the plain old character length of that string. That
will always give you the right answer, regardless of whether that
string is packed bytes or variable-width integers.

Correct. And I'd argue that when it's time to set the length it should die if utf8 flag is still set.

When calculating the length the content should have already been encoded.

Again, at some point decoding and encoding should be core not just a plugin. It's an important part of the request cycle.

--
Bill Moseley
moseley@...

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Re: Unicode trouble with Catalyst::Engine::FastCGI

2009-11-23T10:53:52Z

Aristotle Pagaltzis schrieb:

>> While this fixes the problem, it is still unclear, why the utf8
>> flag is set for the whole buffer.
>
> It shouldn’t matter.

But it does.

>> So Ladies and Gentleman, may I present you the culprit? It is
>> YAML::XS! Everything read by YAML::XS
>>
>> perl -MYAML::XS -E '
>> my $config = YAML::XS::LoadFile("myapp.yml");
>> say((utf8::is_utf8($config->{name}) ? "is" : "is not"), " utf8");
>> '
>> is utf8
>
> No, that’s not the culprit.
>
> The culprit is Catalyst::Engine::FastCGI, which does not pay
> attention to the UTF8 flag.

Obviously YAML::XS doesn't do that either.

But aside from that I agree with you, that something ist broken in
F(ast)CGI. It seems more that F(ast)CGI pays attention to the utf8 flag
where it shouldn't, because it seems to automatically utf8::encode the
buffer due to the set utf8 flag.

> No. YAML::XS is completely correct.

No, it is not. Because it claims:

This module exports the functions Dump and Load. These functions are
intended to work exactly like YAML.pm's corresponding functions.

And that's not the case. And therefore you can't use it as a
drop-in-replacement for other YAML modules.

Bernhard Graf

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Unicode trouble with Catalyst::Engine::FastCGI

2009-11-23T10:25:00Z

* Bernhard Graf <catalyst4@...> [2009-11-23 19:10]:
> Meanwhile I realized, that the final output buffer (header
> + body) actually /has/ the UTF-8 flag set. So it seems, that
> Jonathan's idea (above link) also matches my case. Everything
> seems fine again, when I insert this line
>
> utf8::downgrade($buffer) if utf8::is_utf8($buffer);
>
> into Catalyst::Engine::FastCGI::write() before
> *STDOUT->syswrite($buffer).

The conditional is superfluous. If you downgrade a string that
doesn’t need downgrading, it’s a no-op.

> While this fixes the problem, it is still unclear, why the utf8
> flag is set for the whole buffer.

It shouldn’t matter.

> Since the body has already be turned into a clean octet-stream
> by C:P:Unicode, the header must have been flagged as utf8 - and
> this was actually the case. Digging further I found out, that
> only the cookie string had the utf8 flag. The cookie is built
> with CGI::Simple::Cookie from a hash, and in this hash only one
> value had the utf8 flag: $cookie->{domain}. And this comes from
> myapp.yml, which is a YAML file, of course.
>
> So Ladies and Gentleman, may I present you the culprit? It is
> YAML::XS! Everything read by YAML::XS
>
> perl -MYAML::XS -E '
> my $config = YAML::XS::LoadFile("myapp.yml");
> say((utf8::is_utf8($config->{name}) ? "is" : "is not"), " utf8");
> '
> is utf8

No, that’s not the culprit.

The culprit is Catalyst::Engine::FastCGI, which does not pay
attention to the UTF8 flag.

> On the old installation YAML::XS was not installed. Actually
> I installed that module right after I re-installed everything
> from the snapshot.
>
> perl -MYAML::Syck -E '
> my $config = YAML::Syck::LoadFile("myapp.yml");
> say((utf8::is_utf8($config->{name}) ? "is" : "is not"), " utf8");
> '
> is not utf8
>
> So my first step is to kick YAML::XS. KICK, KICK, KICK!!!
> Restart Cat-App - now it tells me:
>
> Use of YAML::Syck or YAML to parse config files is DEPRECATED. Please
> install YAML::XS for proper YAML support at
> /opt/perl/5.10.1/lib/site_perl/Config/Any.pm line 198
>
> But my output is fixed! WoohooOO!
>
> So first of all YAML::XS must be fixed (I'll file a bug report now).

No. YAML::XS is completely correct.

> Also Catalyst::Engine(::FastCGI) should assure, that this
> implicit ut8::upgrade does not happen when the body has already
> been encoded into an octet-stream.

Nope.

> But I'm not sure if utf8::downgrade() is the right approach to
> fix this, because other Cat engines are not affected by this
> utf8-flag bug, and those don't explicitly utf8::downgrade the
> buffer either.

I’m not sure why ::FastCGI is affected specifically. Whether
a string is upgraded or not should make no difference, though. In
both cases the output should be the same. If it’s not, then
something is broken in ::FastCGI.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Re: Avoiding UTF8 in Catalyst

2009-11-23T10:24:17Z

On Mon, Nov 23, 2009 at 05:43:25PM +0100, Aristotle Pagaltzis wrote:
> If you use the `html` filter instead of `html_entity`, it will
> escape only the five characters that have to be.

Thank you. It works like a charm.

> I had an IRC convo with Tomas Doran last night and explained the
> problem to him. He knocked out some tests for the broken

Thank you for your time! It's nice to see the responsiveness of the
project.

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

How to reduce the memory footprint?

2009-11-23T10:18:50Z

Hello,
I have quite a small Catalyst application that runs with FastCGI + FCGI::ProcManager::MaxRequests

I run 5 instances of FastCGI. Each instance was taking about 90MB of memory. I tried to reduce the memory fooprint by reducing the number of libraries I used. The memory usage is now 120MB per instance! The memory increase is probably due to other changes.

I am wondering how to profile the memory usage, and how to reduce it:
* how can I figure out what is talking how much memory?
* is there a list of common things to do do to reduce the memory usage?
* is 100+MB per instance for small application usual? I hope not...
* any tip to reduce or profile my memory usage?

Thank you
Julien

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Avoiding UTF8 in Catalyst

2009-11-23T10:14:16Z

* Carl Johnstone <catalyst@...> [2009-11-23 18:50]:

> Aristotle Pagaltzis wrote:
> > Please plese don’t make statements like “not in this case”
> > without knowing what the thing you are talking about does,
> > i.e. in this case bytes::length, does. There are enough
> > misconceptions about Unicode in Perl already.
>
> As far as the usage of bytes::length. Yes I agree with you that
> the code is wrong as it's taking the byte length of perl's
> internal representation - which happens to be utf-8 and whilst
> correct in that case, isn't for any other character set and
> shouldn't be relied upon.

No: the internal representation can be either of two formats, and
which of the two you get is not reliable, because it’s purely an
implementation detail. It’s never correct. It just accidentally
works much of the time, getting the right answer by using the
wrong method.

> You *do* have to take a byte length of the string in the
> destination character set though

Yes.

> so I'm interested in what the correct solution would be.

Encode the string to the destination encoding (not just character
set), so that the string represents an encoded octet stream, and
then look at the plain old character length of that string. That
will always give you the right answer, regardless of whether that
string is packed bytes or variable-width integers.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Unicode trouble with Catalyst::Engine::FastCGI

2009-11-23T10:06:58Z

Good news everyone!
I found the gist of the matter and some places that should be fixed.

> Googling around I found
> http://www.mail-archive.com/catalyst@.../msg00051.html ,
> but I couldn't verify Jonathan's assumption in my case - even before the
> buffer is written into the FastCGI pipe in
> Catalyst::Engine::FastCGI::write the whole buffer (header and body)
> looked fine right before calling STDOUT->syswrite.

Meanwhile I realized, that the final output buffer (header + body)
actually /has/ the UTF-8 flag set. So it seems, that Jonathan's idea
(above link) also matches my case.
Everything seems fine again, when I insert this line

utf8::downgrade($buffer) if utf8::is_utf8($buffer);

into Catalyst::Engine::FastCGI::write() before
*STDOUT->syswrite($buffer).

While this fixes the problem, it is still unclear, why the utf8 flag is
set for the whole buffer.

Since the body has already be turned into a clean octet-stream by
C:P:Unicode, the header must have been flagged as utf8 - and this was
actually the case. Digging further I found out, that only the cookie
string had the utf8 flag. The cookie is built with CGI::Simple::Cookie
from a hash, and in this hash only one value had the utf8 flag:
$cookie->{domain}. And this comes from myapp.yml, which is a YAML file,
of course.

So Ladies and Gentleman, may I present you the culprit?
It is YAML::XS! Everything read by YAML::XS

perl -MYAML::XS -E '
my $config = YAML::XS::LoadFile("myapp.yml");
say((utf8::is_utf8($config->{name}) ? "is" : "is not"), " utf8");
'
is utf8

On the old installation YAML::XS was not installed. Actually I installed
that module right after I re-installed everything from the snapshot.

perl -MYAML::Syck -E '
my $config = YAML::Syck::LoadFile("myapp.yml");
say((utf8::is_utf8($config->{name}) ? "is" : "is not"), " utf8");
'
is not utf8

So my first step is to kick YAML::XS. KICK, KICK, KICK!!!
Restart Cat-App - now it tells me:

Use of YAML::Syck or YAML to parse config files is DEPRECATED. Please
install YAML::XS for proper YAML support at
/opt/perl/5.10.1/lib/site_perl/Config/Any.pm line 198

But my output is fixed! WoohooOO!

So first of all YAML::XS must be fixed (I'll file a bug report now).

Also Catalyst::Engine(::FastCGI) should assure, that this implicit
ut8::upgrade does not happen when the body has already been encoded into
an octet-stream. But I'm not sure if utf8::downgrade() is the right
approach to fix this, because other Cat engines are not affected by this
utf8-flag bug, and those don't explicitly utf8::downgrade the buffer either.

Bernhard Graf

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Re: Avoiding UTF8 in Catalyst

2009-11-23T09:54:36Z

On Mon, Nov 23, 2009 at 8:34 AM, Aristotle Pagaltzis <pagaltzis@...> wrote:

[huge snip]

Aristotle++

This was a fantastic explanation with examples. Even though I *think* I understand the unicode issues in perl, I still can find myself getting confused. These examples just help that.

Thanks for this, it was really great.

-J

--
J. Shirley :: jshirley@... :: Killing two stones with one bird.
http://our.coldhardcode.com/jshirley/

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Avoiding UTF8 in Catalyst

2009-11-23T09:42:13Z

The fact is that counting bytes from the Perl Unicode upgraded string is
wrong when using ISO-8859-1.

Maybe Catalyst dropped any support for non UTF-8 charset. By doing that
it also dropped any support for any charset having a bytesize different
than the Perl Unicode upgraded string internal format, apparently.

But I am no expert on this.

I would recommend using utf-8 throughout, even if you think you'll never need it.

The reason is you can accidentally send what appear to be correct bytes even though they are not and if you are using a English browser you will never realise that pure chance is saving you.

You sail along sticking it in a database, in files, in templates and it works... until one day it doesn't.

Then you are in big trouble with data that might-be-latin-1 or might-be-utf-8 or might-be-double-encoded.

I watched a colleague spend 3 whole months fixing an internal framework like that. Proving his fixes worked was very difficult and the historic data in Oracle, well, there was no reliable way to unbork it.

Many of us have been through a lifetime of pain with latin-1 encoding. Unless you are dead set on it, it's much easier to use utf-8 throughout and add a few simple utf-8 unit tests at the input/output boundaries of your system components.

It's not so hard, add use open ':utf8'; to your code at the top or use binmode $fh, ':utf8'; on open file handles.

Use the default utf-8 encoding in Template Toolkit.

When you want to print a variable do use Encode; print encode_utf8($foo);.

Regards, Peter

http://perl.dragonstaff.co.uk

Re: Re: Avoiding UTF8 in Catalyst

2009-11-23T09:38:29Z

Aristotle Pagaltzis wrote:
> But there’s no room for “likelies” here: that’s programming by
> coincidence.

The "likely" was correct.

When using UTF-8 whether the length of the string is different in bytes and
characters depends entirely on what the contents of the string are. Given a
particular string I could tell you exactly whether they should match, but in
the general case all I can say is that it's *likely* to be different.

In any case that's an argument about English :-)

> Either you want it or you don’t, and in this case
> you do. But bytes::length doesn’t do that.
>
> Please plese don’t make statements like “not in this case”
> without knowing what the thing you are talking about does, i.e.
> in this case bytes::length, does. There are enough misconceptions
> about Unicode in Perl already.

As far as the usage of bytes::length. Yes I agree with you that the code is
wrong as it's taking the byte length of perl's internal representation -
which happens to be utf-8 and whilst correct in that case, isn't for any
other character set and shouldn't be relied upon.

You *do* have to take a byte length of the string in the destination
character set though, so I'm interested in what the correct solution would
be.

Carl

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Avoiding UTF8 in Catalyst

2009-11-23T08:43:25Z

* Marc SCHAEFER <schaefer@...> [2009-11-23 17:20]:

> On Mon, Nov 23, 2009 at 07:42:06AM -0800, Bill Moseley wrote:
> >Still not following. You are talking about Catalyst::View::TT?
>
> It appears that the latin1 -> htmlentities conversion is done
> by View:TT's htmlentity, e.g.:
>
> [% FOREACH h IN cols %]<td>[% b.$h | html_entity %]</td>[% END %]
>
> This is perfectly OK, even if not strictly required. I thought
> it was something else doing that, but it isn't.

If you use the `html` filter instead of `html_entity`, it will
escape only the five characters that have to be.

> There is still apparently something wrong: there is absolutely
> no reason why a Perl Unicode string should be used, but I was
> unable to determine why it was created (upgraded) in the first
> place.

There is no reason why such a string should NOT be used either.
The meaning of the string doesn’t change. It’s an implementation
detail in perl whether the string has been upgraded or not.

The bug is that bytes::length is being used to get its length.

> The fact is that counting bytes from the Perl Unicode upgraded
> string is wrong when using ISO-8859-1.

Using bytes::length is ALWAYS wrong. No really, it’s ALWAYS
wrong. (See the long rant in the other mail I just sent for an
explanation.)

> Maybe Catalyst dropped any support for non UTF-8 charset. By
> doing that it also dropped any support for any charset having
> a bytesize different than the Perl Unicode upgraded string
> internal format, apparently.

It’s just plain a bug in Catalyst that it’s using bytes::length.

I had an IRC convo with Tomas Doran last night and explained the
problem to him. He knocked out some tests for the broken
behaviour. It should be all fixed in the next release, and then
you can upgrade and throw away that `before finalize_headers`
workaround.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Avoiding UTF8 in Catalyst

2009-11-23T08:34:20Z

* Carl Johnstone <catalyst@...> [2009-11-23 15:35]:
> Aristotle Pagaltzis wrote:
> > # everything should be bytes at this point, but just in case
> > $response->content_length( bytes::length( $response->body ) );
> >
> > I was shocked to discover this! Any code that uses
> > bytes::length is automatically broken.
>
> Not in this case

Yes in this case.

> the HTTP spec says that the Content-Length header should
> contain the number of octets in the body. If you're sending
> UTF-8 then this is likely different to the number of characters
> in the string.

You’re right about HTTP.

But there’s no room for “likelies” here: that’s programming by
coincidence. Either you want it or you don’t, and in this case
you do. But bytes::length doesn’t do that.

Please plese don’t make statements like “not in this case”
without knowing what the thing you are talking about does, i.e.
in this case bytes::length, does. There are enough misconceptions
about Unicode in Perl already.

Try this:

use 5.010; require utf8; require bytes; require Data::Dump;
$a = $b = chr(0xff);
utf8::upgrade($a);
utf8::downgrade($b);
say Data::Dump::pp $a, $b;
say $a eq $b ? 'ok' : 'not ok';
say length($a) == length($b) ? 'ok' : 'not ok';
say bytes::length($a) == bytes::length($b) ? 'ok' : 'not ok';

It will print the following:

("\xFF", "\xFF")
ok
ok
not ok

In other words, there are two entirely identical strings here,
their internal buffers just happen to be in different formats:
one is a packed byte array, the other is a variable-width integer
arrays. And then bytes::length goes and *IGNORES* which is which,
and just blithely looks at the size of the buffer without caring
about the (ill-named) UTF8 flag – even though both strings, when
printed, will produce the *exact same output*. Because they are
IDENTICAL.

In Perl, there are ONLY strings. Semantically, there are no “byte
strings and character strings”. Just strings. All strings are the
same: character sequences, where a a character is an arbitrarily
large integer value. That’s *all*.

Now there are, on the level of the perl implementation, two
string formats: packed byte sequence strings (which are fast but
can only store codepoints < 0x100) and variable-width integer
sequence strings (which are slower but can store all codepoints).

However, from the Perl level, there is NO difference between
those two kinds of string. If you have binary data in a string,
then it’s simply a string that happens to consist of characters
all < 0x100. Note how I didn’t talk about whether it’s a byte
array string or a variable-width integer string? That’s because
that doesn’t matter. Observe:

my $jpeg = do {
open my $fh, '<', 'some-image.jpeg' or die $!;
local $/;
<$fh>;
};

utf8::upgrade( $jpeg ); ### <------ note here

open my $fh, '>', 'output.jpeg' or die $!;
print $fh $jpeg;

If you run this code, end result will be two EXACTLY IDENTICAL
files. Because the contents in $jpeg mean the SAME THING after
upgrading as they did before. You cannot tell from just looking
at a string, whether it contains binary data or text.

However, if you ask for its bytes::length( $jpeg ), you’ll get
the wrong number! Because bytes.pm is broken! As designed!

Note that up- or downgrading a string like this will happen at
pretty random points in your code, and it won’t be obvious where
or why. It’s not actually random of course, but the point where
it happens might be hidden in some module several layers down
your call stack. It might happen only some of the time. Which is
perfectly fine, because the distinction between these two kinds
of strings is an implementation detail in perl! Just like when
you print numbers in Perl, and perl stringifies the scalar,
caches the result of that conversion in the IV slot of the
scalar, and never bothers to let you know.

Because you don’t need to know.

So it might happen that you properly Encode::encode’d your
string, but it’s passed to some routine somewhere in the guts of
some module you are using, which still causes it to get upgraded
in the course some operation. And that’s just fine. It’s not
a bug, just like it’s not a bug that perl silently stringifies
numbers and silently numifies strings. The resulting output will
always be correct in the end because every operation knows to pay
attention to all the IOK, POK, etc flags in scalars that keep
track of these conversions.

But bytes::length doesn’t! It breaks the fixed-/variable width
abstraction by blithely ignoring the UTF8 flag. (Which should
have been named UOK, to go with the IOK, POK, etc flags that
scalars already have.) It’s as if, when you asked for the length
of the number 65, and the scalar had never been stringified
before, Perl didn’t bother to stringify it, and just looked at
the length of the IV slot (integer value), and because you are
running a 32-bit perl, the answer you got was 4. Whereas if you
had stringified the scalar, then instead the answer would be
2 because "65" is two characters long. And maybe your code is
written such that it sometimes happens to stringify the scalar
(eg. by printing it in a diagnostic message) and sometimes not.
Then you get to play a lottery! Fun!

Conclusion of this much longer rant than I planned to write:

If you’re using bytes.pm or any of its functions, your code is
BROKEN. Unconditionally.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Avoiding UTF8 in Catalyst

2009-11-23T08:18:02Z

On Mon, Nov 23, 2009 at 07:42:06AM -0800, Bill Moseley wrote:
> Still not following. You are talking about Catalyst::View::TT?

It appears that the latin1 -> htmlentities conversion is done by
View:TT's htmlentity, e.g.:

[% FOREACH h IN cols %]<td>[% b.$h | html_entity %]</td>[% END %]

This is perfectly OK, even if not strictly required. I thought it was
something else doing that, but it isn't.

> BTW -- when looking at C::V::TT I see where you got that DEFAULT_ENCODING
> from -- it's documented in C::V::TT.

The simple fact that html_entity above changes é (iso-8859-1) in é
means that something must have understood I am using iso-8859-1, which
is good. But you seem to be right:

> As far as I know there's no such setting in Template Toolkit. There's
> "ENCODING" to specify the encoding of your templates.

I am using:

package MyApp::View::TT;

use strict;
use base 'Catalyst::View::TT';

__PACKAGE__->config(TEMPLATE_EXTENSION => '.tt',
FILTERS => { 'latex' => \&latex },
DEFAULT_ENCODING => 'iso-8859-1',
WRAPPER => 'wrapper.tt');

You are however right that removing the DEFAULT_ENCODING above
doesn't change anything. Replacing it by ENCODING => 'utf-8'
creates a charset conversion bug (which is expected). Replacing with
ENCODING => 'iso-8859-1' doesn't change anything. So I can safely
assume that as usually expected, iso-8859-1 is the default. I now
removed this specification altogether.

> If your templates are 8859-1 with 8 bit characters my suggestion would be to
> convert them to utf-8 and set ENCODING to utf8 for the templates, and move
> toward utf8 everywhere. Make sure you use the plugin to decode and
> encode.

Again, utf8 is out of the question here: be it in the source file, the
database, or the output. UTF-8 is unacceptable in our environment.

My problem (Catalyst sending iso-8859-1 data to the browser, but having
a wrong Content-Length: as if counting the bytes from the UTF-8
equivalent (or Perl Unicode upgraded string as mentionned in a separate
mail by Aristotle Pagaltzis)) was solved by adding the following to MyApp.pm:

before 'finalize_headers'
=> sub {
my $c = shift;

if ($c->response) {
my $s = $c->response->body;
utf8::downgrade($s);
$c->response->body($s);
}
};

There is still apparently something wrong: there is absolutely no reason
why a Perl Unicode string should be used, but I was unable to determine
why it was created (upgraded) in the first place.

The fact is that counting bytes from the Perl Unicode upgraded string is
wrong when using ISO-8859-1.

Maybe Catalyst dropped any support for non UTF-8 charset. By doing that
it also dropped any support for any charset having a bytesize different
than the Perl Unicode upgraded string internal format, apparently.

But I am no expert on this.

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Avoiding UTF8 in Catalyst

2009-11-23T07:42:06Z

On Mon, Nov 23, 2009 at 2:01 AM, Marc SCHAEFER <schaefer@...> wrote:
> On Sat, Nov 21, 2009 at 05:16:24PM -0800, Bill Moseley wrote:
>> > Apparently all diacritic characters are expanded into HTML entities.
>>
>> Where does that happen?
>
> It looks like it's TT::View's htmlentity which does this, not just for
> <> and friends. It's not a big issue for me. Maybe it could even be
> parametered.
>

Still not following. You are talking about Catalyst::View::TT?

BTW -- when looking at C::V::TT I see where you got that DEFAULT_ENCODING from -- it's documented in C::V::TT.

As far as I know there's no such setting in Template Toolkit. There's "ENCODING" to specify the encoding of your templates.

If your templates are 8859-1 with 8 bit characters my suggestion would be to convert them to utf-8 and set ENCODING to utf8 for the templates, and move toward utf8 everywhere. Make sure you use the plugin to decode and encode.

--
Bill Moseley
moseley@...

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: Re: Avoiding UTF8 in Catalyst

2009-11-23T06:29:19Z

Aristotle Pagaltzis wrote:
> # everything should be bytes at this point, but just in case
> $response->content_length( bytes::length( $response->body ) );
>
> I was shocked to discover this! Any code that uses bytes::length
> is automatically broken.

Not in this case, the HTTP spec says that the Content-Length header should
contain the number of octets in the body. If you're sending UTF-8 then this
is likely different to the number of characters in the string.

Carl

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: consuming Webservices with Catalyst?

2009-11-23T05:39:39Z

2009/11/23 Jens Schwarz <blacky6767@...>:
> Hi,
>
> I want to change my Catalyst app so that its data is not written/read to/from my MySQL-DB but to/from Webservices.
>
> I have made a simple test (outside of Catalyst) with SOAP::WSDL that works for "reading" a Webservice. This looks like this:
<snip>
> But actually I have no clue on how to consume a Webservice "the Catalyst way". The amount of CPAN modules covering SOAP, WSDL, ... is overwhelming and I am kind of lost.
>
> Are there any remommendations - especially with detailed examples? It was hard for me to even get this simple example above working as I am SOAP/WSDL newbie.

I think you have the right idea. Build a standalone web-service
client. In your example you're using a SOAP service, so I would use
XML::Compile::SOAP and XML::Compile::WSDL11, but you might have good
luck with the SOAP::* modules, I am less familiar with those. Either
way, build a standalone client which you can test on its own. Finally
build a thin Catalyst-aware shim, using Catalyst::Model::Adaptor or
Catalyst::Component::InstancePerContext, which wraps/consumes your
standalone model and captures whatever Catalyst config etc you need to
use your model within Catalyst. This approach may require slightly
more code files than the monolithic approach but will be much simpler
to test since you remove the Catalyst dependency from verifying the
majority of your model's behaviour.

/joel

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

consuming Webservices with Catalyst?

2009-11-23T05:27:52Z

Hi,

I want to change my Catalyst app so that its data is not written/read to/from my MySQL-DB but to/from Webservices.

I have made a simple test (outside of Catalyst) with SOAP::WSDL that works for "reading" a Webservice. This looks like this:

use MyInterfaces::FooService::FooPortType;

my $user = 'someuser';
my $pass = 'somepass';

*SOAP::Transport::HTTP::Client::get_basic_credentials = sub {
return ($user, $pass);
};

my $s = MyInterfaces::FooService::FooPortType->new();
my $r = $s->Foo( {
'Parameter1' => 'bar',
'Parameter2' => 'baz',
} );
print $r;

But actually I have no clue on how to consume a Webservice "the Catalyst way". The amount of CPAN modules covering SOAP, WSDL, ... is overwhelming and I am kind of lost.

Are there any remommendations - especially with detailed examples? It was hard for me to even get this simple example above working as I am SOAP/WSDL newbie.

Thanks
Jens
--
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/chbrowser

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Unicode trouble with Catalyst::Engine::FastCGI

2009-11-23T03:57:27Z

After I recently re-installed my Development-Perl (that one, that I use
apart from the production Perl installation), all pages that went
through Catalyst::Engine::FastCGI got double-utf8-encoded.

When using the standalone HTTP server, everything is fine.

Both installations used Perl 5.10.1. I did a snapshot of the old modules
at first and then re-installed everything on the new installation.

Googling around I found
http://www.mail-archive.com/catalyst@.../msg00051.html ,
but I couldn't verify Jonathan's assumption in my case - even before the
buffer is written into the FastCGI pipe in
Catalyst::Engine::FastCGI::write the whole buffer (header and body)
looked fine right before calling STDOUT->syswrite.

Is there anyone here, who has experienced similar effects and maybe
found a solution (besides kicking out the FastCGI engine)?

Meanwhile I'm proxying from the main webserver to C:E:HTTP::Prefork,
which works fine, but I would prefer keeping FastCGI as an alternative.

Bernhard Graf

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

Re: call of the dbd disconnect method via catalyst model

2009-11-23T02:12:42Z

christian4catalyst@... wrote:

> Can me someone say, where i can, who i can close the dbd-connection in line
> 09, so that line 10 reopen the connection.
>
> 01 sub foo
> 02 {
> 03 my ( $self, $c ) = @_;
> 04 my $model = $c->model('DB::TABLE') ;
> 05 my $row ;
> 06 my $res ;
> 07
> 08 $row = { ... } ;
> 09 ## $model-> method to call for dbd->disconnect()
> 10 $res = [$model->search($row)] ;
> 11
> 12 ...
> 13 }

Erm, this is a DBIx::Class question isn't it? (At a guess - you haven't
told us in any way what code you're using to provide your DB model, so
I'm just wildly speculating here).

You want to do something like $c->model('DB')->schema to get the
DBIx::Class::Schema, then call ->disconnect on that, or possibly
->schema->dbh->disconnect.

Cheers
t0m

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/