UTF-8 output
|
View:
New views
19 Messages
—
Rating Filter:
Alert me
|
 |
UTF-8 output
by Michael Lackhoff
::
Rate this Message:
I would like to switch the encoding of my app from latin1 to UTF-8 and
got it going somehow but am not sure if I am doing it right. These are my changes in my base class # seems to be necessary for form input use CGI '-utf8'; # is there no better way? sub tt_post_process { my $self = shift; my $htmlref = shift; utf8::encode($$htmlref); } sub cgiapp_init { ... $self->header_add(-charset => 'utf-8'); } I am using CAP::Dispatch and the TT-plugin. Without the header_add I get a HTTP header with an ISO-8859-1 charset definition so the browser thinks it is latin1 and even a <meta http-equiv="content-type" content="text/html; charset=UTF-8" /> in the template won't help. And is there no better way for the template output than to post_process the whole template? Is there no way to get the output of tt_process as UTF-8 so that there is no post_processing necessary? Any comments? Thanks -Michael ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Michael Peters
::
Rate this Message:
On 11/02/2009 12:43 AM, Michael Lackhoff wrote:
> Without the header_add I get > a HTTP header with an ISO-8859-1 charset definition so the browser > thinks it is latin1 and even a > <meta http-equiv="content-type" content="text/html; charset=UTF-8" /> > in the template won't help. That's very strange. Does it happen the same across all browsers? The <meta> tag should be equivalent to the HTTP header. > And is there no better way for the template output than to post_process > the whole template? Is there no way to get the output of tt_process as > UTF-8 so that there is no post_processing necessary? I have a patched HTML::Template that reads in the templates as UTF8, my db connections are all UTF8 and I decode the CGI params as UTF8. As long as all your inputs are UTF8 decoded then you don't need to explicitly encode the output. -- Michael Peters Plus Three, LP ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Emmanuel Seyman-2
::
Rate this Message:
* Michael Lackhoff [02/11/2009 09:25] :
> > I am using CAP::Dispatch and the TT-plugin. Without the header_add I get > a HTTP header with an ISO-8859-1 charset definition so the browser I believe you're seeing this because CGI::App use the CGI module, which defaults to ISO-8859-1 as a charset. >From CGI.pm : # set charset to the safe ISO-8859-1 $self->charset('ISO-8859-1'); http://cpansearch.perl.org/src/LDS/CGI.pm-3.48/lib/CGI.pm I always add the following line : $webapp->header_add(-charset => 'UTF-8'); to my CGI::App .cgi files to make this go away. Emmanuel ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Michael Lackhoff
::
Rate this Message:
On 02.11.2009 15:27 Michael Peters wrote:
> On 11/02/2009 12:43 AM, Michael Lackhoff wrote: > >> Without the header_add I get >> a HTTP header with an ISO-8859-1 charset definition so the browser >> thinks it is latin1 and even a >> <meta http-equiv="content-type" content="text/html; charset=UTF-8" /> >> in the template won't help. > > That's very strange. Does it happen the same across all browsers? The > <meta> tag should be equivalent to the HTTP header. It is with firefox, the only relevant browser for my app. If they are equivalent it might be undefined who wins: the HTTP header or the meta-tag. But I will do more tests with a very basic template. >> And is there no better way for the template output than to post_process >> the whole template? Is there no way to get the output of tt_process as >> UTF-8 so that there is no post_processing necessary? > > I have a patched HTML::Template that reads in the templates as UTF8, my > db connections are all UTF8 and I decode the CGI params as UTF8. As long > as all your inputs are UTF8 decoded then you don't need to explicitly > encode the output. Sounds very interesting. Let me try to follow: I am in the process of changing the db connection as well but have still some problems with CGI params. How do you decode them? Not one at a time, I guess? In my app they seem to be encoded twice after a roundtrip (have to do more debugging). Say, I managed to have all my inputs UTF8 decoded, how do you persuade Perl to output UTF8 without explicit encoding? Is it just your patched HTML::Template or is there a trick that would also work with Template::Toolkit? Many thanks for your response! -Michael ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Michael Lackhoff
::
Rate this Message:
On 02.11.2009 16:21 Emmanuel Seyman wrote:
>>From CGI.pm : > > # set charset to the safe ISO-8859-1 > $self->charset('ISO-8859-1'); > > http://cpansearch.perl.org/src/LDS/CGI.pm-3.48/lib/CGI.pm Thanks for the hint! But isn't it better then to tackle the problem right at the root: $webapp->query->charset('UTF-8'); # works! > I always add the following line : > > $webapp->header_add(-charset => 'UTF-8'); because this would set the header twice, first from CGI.pm to ISO-8859-1 then from the webapp to UTF-8 (no big deal but still) or are there other reasons why header_add is better? But since this is more or less academic because both approaches work, I would be much more interested in good ways to deal with the decoding of CGI-params and the encoding of TT-templates ;-) -Michael ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Cees Hek
::
Rate this Message:
On Tue, Nov 3, 2009 at 3:23 AM, Michael Lackhoff
<lackhoff@...> wrote: > Sounds very interesting. Let me try to follow: > I am in the process of changing the db connection as well but have still > some problems with CGI params. How do you decode them? Not one at a > time, I guess? In my app they seem to be encoded twice after a roundtrip > (have to do more debugging). Have a look at this perlmonks post by Rhesa: http://www.perlmonks.org/?node_id=651574 Cheers, Cees ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Michael Lackhoff
::
Rate this Message:
On 03.11.2009 01:22 Cees Hek wrote:
> Have a look at this perlmonks post by Rhesa: > > http://www.perlmonks.org/?node_id=651574 Thanks Cees (and of course Rhesa), this is exactly what I need! Since you are the author of CAP::TT, may I also ask how and when you do the encoding of the template output? Is utf8::encode in postprocess the way to go? -Michael ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Emmanuel Seyman-2
::
Rate this Message:
* Michael Lackhoff [02/11/2009 20:38] :
> > Thanks for the hint! But isn't it better then to tackle the problem > right at the root: > $webapp->query->charset('UTF-8'); # works! Probably. I've always felt that the most appropriate solution would be for CGI to default to UTF-8. > because this would set the header twice, first from CGI.pm to ISO-8859-1 > then from the webapp to UTF-8 (no big deal but still) or are there other > reasons why header_add is better? None. When I first came across the problem, the header_add() solution was the first one Google suggested. Since then, I've never looked back. Emmanuel ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Cees Hek
::
Rate this Message:
On Wed, Nov 4, 2009 at 12:15 AM, Michael Lackhoff
<lackhoff@...> wrote: > On 03.11.2009 01:22 Cees Hek wrote: > >> Have a look at this perlmonks post by Rhesa: >> >> http://www.perlmonks.org/?node_id=651574 > > Thanks Cees (and of course Rhesa), this is exactly what I need! > > Since you are the author of CAP::TT, may I also ask how and when you do > the encoding of the template output? Is utf8::encode in postprocess the > way to go? Template Toolkit works fine with utf8 templates, so I don't think you need to encode the template output yourself. if your template files contain a BOM at the start then you don't need to do anything, or you can set the ENCODING option to 'utf8' manually. See this FAQ entry for the details. http://search.cpan.org/dist/Template-Toolkit/lib/Template/FAQ.pod#Why_do_I_get_rubbish_for_my_utf-8_templates? However, I guess you should ensure that any data that you send to TT is utf8 encoded. Cheers, Cees ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Michael Lackhoff
::
Rate this Message:
On 03.11.2009 23:27 Cees Hek wrote:
> However, I guess you should ensure that any data that you send to TT > is utf8 encoded. That's exactly the problem. I pass lots of RDBO objects into the templates and those should contain decoded (UTF-8 flag on) data, not UTF-8 octets, so I guess encoding the whole thing at the end might be the only practical option left in this situation. Thanks everyone for all your input, now I am seeing things much clearer. -Michael ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Cees Hek
::
Rate this Message:
On Wed, Nov 4, 2009 at 6:15 PM, Michael Lackhoff
<lackhoff@...> wrote: > On 03.11.2009 23:27 Cees Hek wrote: > >> However, I guess you should ensure that any data that you send to TT >> is utf8 encoded. > > That's exactly the problem. I pass lots of RDBO objects into the > templates and those should contain decoded (UTF-8 flag on) data, not > UTF-8 octets, so I guess encoding the whole thing at the end might be > the only practical option left in this situation. Can't you get DBI to do the decoding for you? I think it will depend on the DBD module you use, but it is quite easy with PostgeSQL: http://search.cpan.org/dist/DBD-Pg/Pg.pm#pg_enable_utf8_(boolean) I use Rose::DB as well and have the following in my base class: __PACKAGE__->register_db( domain => 'development', type => 'main', driver => 'Pg', database => $NC::Config::database, username => $NC::Config::database_username, password => $NC::Config::database_password, host => $NC::Config::database_host, port => $NC::Config::database_port, server_time_zone => 'Australia/Sydney', european_dates => 1, # Apache::DBI and Rose::DB choke when server side prepares are turned on connect_options => { pg_server_prepare => 0, pg_enable_utf8 => 1, }, post_connect_sql => "SET CLIENT_ENCODING TO 'UTF8';", ); Cheers, Cees ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Michael Lackhoff
::
Rate this Message:
On 04.11.2009 11:24 Cees Hek wrote:
> Can't you get DBI to do the decoding for you? I think it will depend > on the DBD module you use, but it is quite easy with PostgeSQL: I am using SQLite and it is also quite easy there but then I have -- as you say -- _decoded_ values. This is what I want within my application but as soon as I produce some output (let CGI::Application print the template with embedded DB data) Perl does its evil magic and encodes everything to latin1 -- at least this is what I get. Do you do some magic with STDOUT or how do you get the UTF-8 to the browser? Whatever I try, Perl always outputs latin1 from decoded strings and the only remedy I found was to not let it output decoded strings but encoded UTF-8 octets. Sorry for the long thread but this is a horror I encounter over and over again and I am hoping for a better understanding to get it right once and for all. -Michael ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Cees Hek
::
Rate this Message:
On Wed, Nov 4, 2009 at 9:49 PM, Michael Lackhoff
<lackhoff@...> wrote: > On 04.11.2009 11:24 Cees Hek wrote: > >> Can't you get DBI to do the decoding for you? I think it will depend >> on the DBD module you use, but it is quite easy with PostgeSQL: > > I am using SQLite and it is also quite easy there but then I have -- as > you say -- _decoded_ values. This is what I want within my application > but as soon as I produce some output (let CGI::Application print the > template with embedded DB data) Perl does its evil magic and encodes > everything to latin1 -- at least this is what I get. > > Do you do some magic with STDOUT or how do you get the UTF-8 to the > browser? Whatever I try, Perl always outputs latin1 from decoded strings > and the only remedy I found was to not let it output decoded strings but > encoded UTF-8 octets. You mean something like this: __PACKAGE__->add_callback('postrun', sub { my $self = shift; # Make sure the output is utf8 encoded if it needs it if ( $_[0] && ${$_[0]} && utf8::is_utf8(${$_[0]}) ){ utf8::encode( ${$_[0]} ); } return; }); I have that in my CGI::App base class. > Sorry for the long thread but this is a horror I encounter over and over > again and I am hoping for a better understanding to get it right once > and for all. Not a problem. Utf8 support is tricky to get right, and I am still not sure if I have tackled everything myself. I'm sure others will get something out of the discussion as well. Cheers, Cees ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Michael Lackhoff
::
Rate this Message:
On 04.11.2009 12:19 Cees Hek wrote:
> You mean something like this: > > __PACKAGE__->add_callback('postrun', sub { > my $self = shift; > > # Make sure the output is utf8 encoded if it needs it > if ( $_[0] && ${$_[0]} && utf8::is_utf8(${$_[0]}) ){ > utf8::encode( ${$_[0]} ); > } > > return; > }); > > I have that in my CGI::App base class. As time goes by my base class gets more and more pieces of yours ;-) Thanks -Michael ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Is C::A::PSGI available?
by Gurunandan R. Bhat-2
::
Rate this Message:
Hi, Is CGI::Application::PSGI available? The link from the Plack website gives a 404. Thank you ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: Is C::A::PSGI available?
by Bugzilla from mark@summersault.com
::
Rate this Message:
On Sat, 07 Nov 2009 10:48:02 +0530
"Gurunandan R. Bhat" <guru@...> wrote: > > Hi, > > Is CGI::Application::PSGI available? The link from the Plack website > gives a 404. Are you referring to this link? http://search.cpan.org/dist/CGI-Application-PSGI It works now. Miyagawa wrote the module, but asked me to upload and maintain it. I endorse the Plack/PSGI project ( http://www.plackperl.org/ ) Mark -- http://mark.stosberg.com/ ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: Is C::A::PSGI available?
by Bugzilla from mark@summersault.com
::
Rate this Message:
On Sat, 07 Nov 2009 10:48:02 +0530
"Gurunandan R. Bhat" <guru@...> wrote: > > Hi, > > Is CGI::Application::PSGI available? The link from the Plack website > gives a 404. Are you referring to this link? http://search.cpan.org/dist/CGI-Application-PSGI It works now. Miyagawa wrote the module, but asked me to upload and maintain it. I endorse the Plack/PSGI project ( http://www.plackperl.org/ ) Mark -- http://mark.stosberg.com/ ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: Is C::A::PSGI available?
by Gurunandan R. Bhat-2
::
Rate this Message:
Indeed, now it does.
Thanks. On Tue, 2009-11-10 at 20:39 -0500, Mark Stosberg wrote: > On Sat, 07 Nov 2009 10:48:02 +0530 > "Gurunandan R. Bhat" <guru@...> wrote: > > > > > Hi, > > > > Is CGI::Application::PSGI available? The link from the Plack website > > gives a 404. > > Are you referring to this link? > > http://search.cpan.org/dist/CGI-Application-PSGI > > It works now. Miyagawa wrote the module, but asked me to upload and maintain it. > I endorse the Plack/PSGI project ( http://www.plackperl.org/ ) > > Mark > ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
|
|
Re: UTF-8 output
by Mike Tonks-2
::
Rate this Message:
I have the following in my cgiapp_init:
binmode STDOUT, ":utf8"; # Explicitly output utf8 - this is critical probably totally wrong, but it works for me... 2009/11/4 Michael Lackhoff <lackhoff@...>: > On 04.11.2009 12:19 Cees Hek wrote: > >> You mean something like this: >> >> __PACKAGE__->add_callback('postrun', sub { >> my $self = shift; >> >> # Make sure the output is utf8 encoded if it needs it >> if ( $_[0] && ${$_[0]} && utf8::is_utf8(${$_[0]}) ){ >> utf8::encode( ${$_[0]} ); >> } >> >> return; >> }); >> >> I have that in my CGI::App base class. > > As time goes by my base class gets more and more pieces of yours ;-) > > Thanks > -Michael > > ##### CGI::Application community mailing list ################ > ## ## > ## To unsubscribe, or change your message delivery options, ## > ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## > ## ## > ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## > ## Wiki: http://cgiapp.erlbaum.net/ ## > ## ## > ################################################################ > > ##### CGI::Application community mailing list ################ ## ## ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp ## ## ## ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ## ## ################################################################ |
| Free embeddable forum powered by Nabble | Forum Help |
