I18N with quotation marks

[Ton Voon-2]

Hi!

I wanted to find out how other people are handling this problem.

I am localising our app, which consists of strings in html and in  
dynamic javascript snippets. However, if the translated value contains  
quotations (such as: s'il vous plait), then it could break the HTML:

<select value='[% c.loc("Please select one") %]'>

or the javascript:

alert('[% c.loc("Please select one") %]');

We also sometimes use double quotes for attributes instead of single  
quotes.

What is the best practise? Always run c.loc() through a filter to  
convert to HTML entities? (Although in FF3.0 alert('Impossible  
d'exécuter snmpget pour tester la connexion'); does not give the  
single quote).

I was considering creating methods of c.hloc() (for a html  
environment) and c.jloc() (for a javascript environment), but then the  
xgettext.pl helper does not look for these method names.

Ton


_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Bill Moseley]

On Wed, Jul 1, 2009 at 10:22 AM, Ton Voon <ton.voon@...> wrote:

Hi!

I wanted to find out how other people are handling this problem.

I am localising our app, which consists of strings in html and in dynamic javascript snippets. However, if the translated value contains quotations (such as: s'il vous plait), then it could break the HTML:

<select value='[% c.loc("Please select one") %]'>

I would think the correct approach would be a filter:

     <select value='[% c.loc("Please select one") | html %]'>

but the default html filter only escapes double quotes.

--
Bill Moseley
moseley@...

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Christian Lackas-2]

* Bill Moseley <moseley@...> [090701 19:58]:

Hi Ton,

> On Wed, Jul 1, 2009 at 10:22 AM, Ton Voon <ton.voon@...> wrote:
> > I am localising our app, which consists of strings in html and in dynamic
> > javascript snippets. However, if the translated value contains quotations
> > (such as: s'il vous plait), then it could break the HTML:
> > <select value='[% c.loc("Please select one") %]'>
> I would think the correct approach would be a filter:
>      <select value='[% c.loc("Please select one") | html %]'>

you can try the 'xml' filter?

    http://template-toolkit.org/docs/manual/Filters.html#section_xml
    Same as the html filter, but adds '  which is the fifth XML
    built-in entity.

or html_entity (have not used that yet). Above manual page also contains
all other built-in filters. And you can also add your own:

    http://template-toolkit.org/docs/modules/Template/Filters.html

Christian


_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Larry Leszczynski]

Hi Ton -

> However, if the translated value contains quotations (such as: s'il
> vous plait), then it could break the HTML:
>
> <select value='[% c.loc("Please select one") %]'>
>
> or the javascript:
>
> alert('[% c.loc("Please select one") %]');

We create some custom scalar ops in a subclass of Catalyst::View::TT
(code below) that let you do:

  <select value="[% c.loc("Please select one").escape_dq %]">

  alert('[% c.loc("Please select one").escape_sq %]');

HTH,
Larry

--------------------------------------------

package Platform::View::TT;

# This is "MyApp/View/TT.pm":

use strict;
use warnings;

use base 'My::Catalyst::View::TT';

1;

--------------------------------------------

package My::Catalyst::View::TT;

use strict;
use warnings;

use base 'Catalyst::View::TT';

$Template::Stash::SCALAR_OPS->{escape_q} = sub {
    my $s = shift;
    $s =~ s/\\/\\\\/g;
    $s =~ s/"/\\"/g;
    $s =~ s/'/\\'/g;
    return $s;
};

$Template::Stash::SCALAR_OPS->{escape_dq} = sub {
    my $s = shift;
    $s =~ s/\\/\\\\/g;
    $s =~ s/"/\\"/g;
    return $s;
};

$Template::Stash::SCALAR_OPS->{escape_sq} = sub {
    my $s = shift;
    $s =~ s/\\/\\\\/g;
    $s =~ s/'/\\'/g;
    return $s;
};

1;

--------------------------------------------

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Larry Leszczynski]

> We create some custom scalar ops in a subclass of Catalyst::View::TT
> (code below) that let you do:
>
>   <select value="[% c.loc("Please select one").escape_dq %]">

Actually, escape_dq won't work here:

   <select value="[% c.loc("Please select one").escape_dq %]">

but it will work if for some reason you have a double-quote delimited
JavaScript string.
Will probably need to convert the double quotes to %22 instead.
Sorry...

>   alert('[% c.loc("Please select one").escape_sq %]');

This one does work though...

Larry

> --------------------------------------------
>
> package Platform::View::TT;
>
> # This is "MyApp/View/TT.pm":
>
> use strict;
> use warnings;
>
> use base 'My::Catalyst::View::TT';
>
> 1;
>
> --------------------------------------------
>
> package My::Catalyst::View::TT;
>
> use strict;
> use warnings;
>
> use base 'Catalyst::View::TT';
>
> $Template::Stash::SCALAR_OPS->{escape_q} = sub {
>     my $s = shift;
>     $s =~ s/\\/\\\\/g;
>     $s =~ s/"/\\"/g;
>     $s =~ s/'/\\'/g;
>     return $s;
> };
>
> $Template::Stash::SCALAR_OPS->{escape_dq} = sub {
>     my $s = shift;
>     $s =~ s/\\/\\\\/g;
>     $s =~ s/"/\\"/g;
>     return $s;
> };
>
> $Template::Stash::SCALAR_OPS->{escape_sq} = sub {
>     my $s = shift;
>     $s =~ s/\\/\\\\/g;
>     $s =~ s/'/\\'/g;
>     return $s;
> };
>
> 1;
>
> --------------------------------------------
>
> _______________________________________________
> List: Catalyst@...
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@.../
> Dev site: http://dev.catalyst.perl.org/

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Ton Voon-2]

On 1 Jul 2009, at 18:22, Ton Voon wrote:

> I am localising our app, which consists of strings in html and in  
> dynamic javascript snippets. However, if the translated value  
> contains quotations (such as: s'il vous plait), then it could break  
> the HTML:
>
> <select value='[% c.loc("Please select one") %]'>
>
> or the javascript:
>
> alert('[% c.loc("Please select one") %]');
>
> We also sometimes use double quotes for attributes instead of single  
> quotes.
>
> What is the best practise? Always run c.loc() through a filter to  
> convert to HTML entities? (Although in FF3.0 alert('Impossible  
> d'exécuter snmpget pour tester la connexion'); does not give the  
> single quote).
>
> I was considering creating methods of c.hloc() (for a html  
> environment) and c.jloc() (for a javascript environment), but then  
> the xgettext.pl helper does not look for these method names.

Hi!

Thanks for all the responses.

I think I now realise it depends on the context of the output so,  
given that the translated string is "as-is" (without any markup or  
html elements), then some filtering is required based on where the  
translated value belongs.

This is my current thinking:

For HTML text, you should pass through the html filter, eg:

<p>[% c.loc("Some text that might have < or > in it") | html %]</p>

For HTML elements, you should use double quotes for quoting attributes  
and then pass the string through the html filter, eg,

<select value="[% c.loc("May have some single or double quotes in") |  
html %]">

For javascript in <script> blocks, you should use single quotes for  
the string value and pass through an escape_js filter, eg:

<script>
var string = '[% c.loc("May have single quotes or \ in it") |  
escape_js %]';
</script>

For javascript in HTML elements, you should use double quotes for  
quoting the attributes and single quotes for the javascript strings  
and pass through the escape_js filter and the html filter, eg:

<select onclick=" alert('[% c.loc("May have all sorts of things in  
it") | escape_js | html %]') ">

The escape_js filter is defined as (From Larry Leszcznski's example):

$Template::Stash::SCALAR_OPS->{escape_js} = sub {
    my $s = shift;
    $s =~ s/\\/\\\\/g;
    $s =~ s/'/\\'/g;
    return $s;
};

Does everyone agree this makes sense? If so, any objections if I add  
this to http://dev.catalystframework.org/wiki/best_practices?

Ton



_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Jose Luis Martinez-2]

Ton Voon escribió:
> For javascript in <script> blocks, you should use single quotes for the
> string value and pass through an escape_js filter, eg:
>
> <script>
> var string = '[% c.loc("May have single quotes or \ in it") | escape_js
> %]';
> </script>

Instead of forcing yourself to use single quoted strings in javascript,
you can escape single quotes AND double quotes :)

<script>
alert('I\'m a string with \\ and lots of \"things\"');
alert("I\'m a string with \\ and lots of \"things\"");
</script>

return the same output.

And to make it more solid...

You would expect that:

<script>
alert('I\'m a </script> string');
</script>

would show you a nice alert. You're wrong :) At least FF3 and IE fail. I
suppose that it's very normal (because the browser's parser understands
nothing about the string context of the javascript, and thinks the
<script> tag ends just in the middle of your script.

The solution is as easy as to "hide" the script tag from the parser.
<script>
alert('I\'m a <\/script> string');
</script>

Note: I don't know if it's better to escape all "/", or all "</" or just
"</script>" instances in the string. Any thoughts?

 > $Template::Stash::SCALAR_OPS->{escape_js} = sub {
 >    my $s = shift;
 >    $s =~ s/\\/\\\\/g;
 >    $s =~ s/'/\\'/g;
 >    return $s;
 > };

Maybe it's more efficient to do this in one pass?
$s =~ s/(\\|'|"|\/)/\\$1/g;

Just my 2 cents,

Jose Luis Martinez
jlmartinez@...


_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Ton Voon-2]

On 2 Jul 2009, at 09:35, Ton Voon wrote:

Does everyone agree this makes sense? If so, any objections if I add this to http://dev.catalystframework.org/wiki/best_practices?

I've added a section at http://dev.catalystframework.org/wiki/best_practices now.

This includes Jose Luis Martinez's suggestions for the escape_js_string routine, implemented as a TT filter.

Final question: How do you internationalise a bit of text that does want some markup within it? For instance, I want something that outputs:

Click <a href="/about">here</a> for the about page.

If I do something like:

link = '<a href="/about">' _ c.loc("here") _ '</a>';

c.loc("Click %1 for the about page", link)

Then I cannot filter through html for the 2nd loc output.

Is there a nicer way?

Ton

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Gunnar Strand]

Ton Voon skrev:

>
> On 2 Jul 2009, at 09:35, Ton Voon wrote:
>
>> Does everyone agree this makes sense? If so, any objections if I add
>> this to http://dev.catalystframework.org/wiki/best_practices?
>
> I've added a section at
> http://dev.catalystframework.org/wiki/best_practices now.
>
> This includes Jose Luis Martinez's suggestions for the
> escape_js_string routine, implemented as a TT filter.
>
> Final question: How do you internationalise a bit of text that does
> want some markup within it? For instance, I want something that outputs:
>
> Click <a href="/about">here</a> for the about page.
>
> If I do something like:
>
> link = '<a href="/about">' _ c.loc("here") _ '</a>';
> c.loc("Click %1 for the about page", link)
>
> Then I cannot filter through html for the 2nd loc output.
>
> Is there a nicer way?

Wouldn't you need to send every part of the text through c.loc
individually? I guess something like this:

c.loc("Click ") _ link _ c.loc(" for the about page")

Not so nice maybe, but I guess it should work?

KR,
Gunnar
>
> Ton







     

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Cosimo Streppone]

Gunnar Strand <gunnarstrand@...> wrote:

> Ton Voon skrev:
>>
>> Final question: How do you internationalise a bit of text that does
>> want some markup within it? For instance, I want something that outputs:
>>
>> Click <a href="/about">here</a> for the about page.
>
> Wouldn't you need to send every part of the text through c.loc
> individually? I guess something like this:
>
> c.loc("Click ") _ link _ c.loc(" for the about page")

If you collect all your i18n messages into .po files,
that are worked on by translators, they have little or no
context information, so they are going to have a really
hard time figuring out the sense of words.

We internationalized a dynamic non-catalyst website in
19 languages now, and we found the following, in gettext notation,
to be the best for front-end and back-end developers
and translators.

   _('Click <a href="[_1]">here</a> for the about page')

So,

1) Keep the markup. It's ugly, but to us it's slightly better than
    the alternatives.

2) Variables as variables. This also conserves strings
    if/when you're changing URLs or variable content.
    Example:

    "You can upload up to [_1] Mb of pictures."

    As the string embeds the variable, your string won't need
    to be translated again if you change your upload limit.

3) Don't break sentences. In general, the longer, the better.
    In general.

--
Cosimo

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Gunnar Strand]

Cosimo Streppone skrev:

> Gunnar Strand <gunnarstrand@...> wrote:
>
>> Ton Voon skrev:
>>>
>>> Final question: How do you internationalise a bit of text that does
>>> want some markup within it? For instance, I want something that
>>> outputs:
>>>
>>> Click <a href="/about">here</a> for the about page.
>>
>> Wouldn't you need to send every part of the text through c.loc
>> individually? I guess something like this:
>>
>> c.loc("Click ") _ link _ c.loc(" for the about page")
>
> If you collect all your i18n messages into .po files,
> that are worked on by translators, they have little or no
> context information, so they are going to have a really
> hard time figuring out the sense of words.
>
> We internationalized a dynamic non-catalyst website in
> 19 languages now, and we found the following, in gettext notation,
> to be the best for front-end and back-end developers
> and translators.
>
>  _('Click <a href="[_1]">here</a> for the about page')
>
> So,
>
> 1) Keep the markup. It's ugly, but to us it's slightly better than
>   the alternatives.
>
> 2) Variables as variables. This also conserves strings
>   if/when you're changing URLs or variable content.
>   Example:
>
>   "You can upload up to [_1] Mb of pictures."
>
>   As the string embeds the variable, your string won't need
>   to be translated again if you change your upload limit.
>
> 3) Don't break sentences. In general, the longer, the better.
>   In general.

I agree completely, and I may have misunderstood the question, but I
interpreted it that the I18Ned text would be put through an HTML filter.
In that case the above example of having markup inside the text sent to
the localizer will not work.

I personally prefer to use library routines for creating html markup
instead of hard coding data. I am using Maketext for my project (I'm not
very experienced yet), but it seems that an alternative would be to
extend the language file with HTML support in it to produce
corresponding output:

_('Click [link,_1,here] for the about page')

This would be prettier (IMHO) and allows the rendering mechanism for
links to be put in one place.

KR,
Gunnar






     

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/