L20n app workflow?

[Robert Kaiser]

Hi,

My interest in L20n seems to increase every time I'm thinking about this
technology - I guess that's a good sign ;-)

What I wondered about though is how exactly the workflow of a
L20n-enabled app looks like (I think that's needed to get to a stage
where it can be implemented).

Current Mozilla L10n has two possibilities, depending on where strings
are used (using descriptive language here):

1) "the XUL workflow":
    - specify L10n file(s) for a given file
      (all strings get loaded into a single "namespace" [of entities],
      beware of string ID clashes)
    - reference the string ID only where the string is used

2) "the stringbundle workflow"
    - load one stringbundle per L10n file
      (one "namespace" [stringbundle] per L10n file)
    - reference stringbundle + string ID where the string is used

gettext usually does it the following way:

3) "the gettext workflow"
    - load a "textdomain" (with a base path), which gettext resolves to
      to an L10n file per language
      (all strings are in the "namespace" of their respective textdomain)
    - define a default textdomain/"namespace"
    - reference either only the original string (if in the default
      textdomain) or textdomain + the original string where the string is
      used

What is the plan for a workflow used by L20n?

Additionally, how is L20n supposed to find its files for a specific
language as well as the files for fallbacks?
The current Mozilla infrastructure does language selection on the chrome
registry level, but that doesn't provide for falling back to a different
language for non-defined strings (object IDs). Gettext uses a strictly
defined subdirectory architecture to be used so it can locate PO files
for the language to be used and any possible fallbacks.
Which approach do we take there for L20n?

If we want to test the concept with simple implementations written in
any languages we like, we need those concepts to build on in addition to
the file format and other logical concepts we already have in place.
(In addition, we'll one time need to define guidelines or rules for the
function names to be used for retrieving stuff from the L20n library,
but I guess that can wait a bit longer.)

I'd really like to do some crude testing implementation for PHP just to
see if L20n is really as good as I imagine it currently. From what I
know currently, the L20n concepts also make dev work much easier for me
in PHP (not needing to implement plural selectors in the code itself in
many places, leave that up to the localization layer, for example - and
of course getting rid of long original strings mixed into the code).
Would be cool to be able to test some of that ;-)

Robert Kaiser
_______________________________________________
dev-i18n mailing list
dev-i18n@...
https://lists.mozilla.org/listinfo/dev-i18n

[Axel Hecht]

Robert Kaiser wrote:

> Hi,
>
> My interest in L20n seems to increase every time I'm thinking about this
> technology - I guess that's a good sign ;-)
>
> What I wondered about though is how exactly the workflow of a
> L20n-enabled app looks like (I think that's needed to get to a stage
> where it can be implemented).
>
> Current Mozilla L10n has two possibilities, depending on where strings
> are used (using descriptive language here):
>
> 1) "the XUL workflow":
>    - specify L10n file(s) for a given file
>      (all strings get loaded into a single "namespace" [of entities],
>      beware of string ID clashes)
>    - reference the string ID only where the string is used
>
> 2) "the stringbundle workflow"
>    - load one stringbundle per L10n file
>      (one "namespace" [stringbundle] per L10n file)
>    - reference stringbundle + string ID where the string is used
>
> gettext usually does it the following way:
>
> 3) "the gettext workflow"
>    - load a "textdomain" (with a base path), which gettext resolves to
>      to an L10n file per language
>      (all strings are in the "namespace" of their respective textdomain)
>    - define a default textdomain/"namespace"
>    - reference either only the original string (if in the default
>      textdomain) or textdomain + the original string where the string is
>      used
>
> What is the plan for a workflow used by L20n?

The workflow is basically 1), mostly because we need to support that
anyway. The stringbundle workflow is similar, just that you can provide
more than one file.

I'm calling the sets of lol files that are packed up "contexts", in
which entities are resolved. Those contexts include both programmatic
parameters and fallback. How those ocntexts are set up depends on the
language mapping.

> Additionally, how is L20n supposed to find its files for a specific
> language as well as the files for fallbacks?
> The current Mozilla infrastructure does language selection on the chrome
> registry level, but that doesn't provide for falling back to a different
> language for non-defined strings (object IDs). Gettext uses a strictly
> defined subdirectory architecture to be used so it can locate PO files
> for the language to be used and any possible fallbacks.
> Which approach do we take there for L20n?

That's not clearly specified yet. In the case of, say, multiple
spanishs, it's not really clear who's falling back to what, and it might
depend on the "module" in whatever sense.

This is, again, part of the language binding. It might make sense to
support some kind of specification of fallback in the lol file itself,
though I'm not sure that's the right modularity.

> If we want to test the concept with simple implementations written in
> any languages we like, we need those concepts to build on in addition to
> the file format and other logical concepts we already have in place.
> (In addition, we'll one time need to define guidelines or rules for the
> function names to be used for retrieving stuff from the L20n library,
> but I guess that can wait a bit longer.)
>
> I'd really like to do some crude testing implementation for PHP just to
> see if L20n is really as good as I imagine it currently. From what I
> know currently, the L20n concepts also make dev work much easier for me
> in PHP (not needing to implement plural selectors in the code itself in
> many places, leave that up to the localization layer, for example - and
> of course getting rid of long original strings mixed into the code).
> Would be cool to be able to test some of that ;-)

We should get interested people together, we should poke at least Wil
Clouser and Adrian Kalla.

It seems that php doesn't support /y of js re (new in 1.9) nor \G,
implementing the parser performantly might not be able to use regular
expressions :-/.

You can see, I'm pretty interested in getting a php-based impl, as that
should really get the most exposure per impl.

Axel
_______________________________________________
dev-i18n mailing list
dev-i18n@...
https://lists.mozilla.org/listinfo/dev-i18n

[Robert Kaiser]

Axel Hecht schrieb:
> The workflow is basically 1), mostly because we need to support that
> anyway. The stringbundle workflow is similar, just that you can provide
> more than one file.

Actually, for me, the gettext solution sounds like some mid-way between
our two variants - though it doesn't merge namespaces like XUL currently
does. I'm not sure if that merging is good to do, desired or even needed
though.

> I'm calling the sets of lol files that are packed up "contexts", in
> which entities are resolved. Those contexts include both programmatic
> parameters and fallback. How those ocntexts are set up depends on the
> language mapping.

Hmm, I wonder if there should be some specified, cross-language-portable
way of accessing lol files/contexts, though, so that you can use them in
basically the same style from C/C++, PHP, JS, Java or whatever.

(Though I'd prefer an object-based solution where OO is supported well
enough, i.e. C++, PHP5, JS, etc.)

>> Additionally, how is L20n supposed to find its files for a specific
>> language as well as the files for fallbacks?
>
> That's not clearly specified yet. In the case of, say, multiple
> spanishs, it's not really clear who's falling back to what, and it might
> depend on the "module" in whatever sense.
>
> This is, again, part of the language binding. It might make sense to
> support some kind of specification of fallback in the lol file itself,
> though I'm not sure that's the right modularity.

I think we should try to have the possibility to use the same set of lol
files from different languages, so some generic specification of finding
files and doing fallback sounds like a good idea.

>> I'd really like to do some crude testing implementation for PHP just
>> to see if L20n is really as good as I imagine it currently. From what
>> I know currently, the L20n concepts also make dev work much easier for
>> me in PHP (not needing to implement plural selectors in the code
>> itself in many places, leave that up to the localization layer, for
>> example - and of course getting rid of long original strings mixed
>> into the code).
>> Would be cool to be able to test some of that ;-)
>
> We should get interested people together, we should poke at least Wil
> Clouser and Adrian Kalla.

That would be cool. I'm sure many web-based tools would profit from a
good PHP impl (even better and faster would probably be some native
support inside PHP but such stuff usually takes longer).

> It seems that php doesn't support /y of js re (new in 1.9) nor \G,
> implementing the parser performantly might not be able to use regular
> expressions :-/.

I see \G mentioned in the preg_* (PCRE-based regex) module
documentation, and from what I read, preg-* is preferred over ereg_*
(POSIX-extended regex) due to being binary-safe, faster and more powerful.

> You can see, I'm pretty interested in getting a php-based impl, as that
> should really get the most exposure per impl.

I think so as well. And it would/will probably be a good proof of the
concepts.

Robert Kaiser
_______________________________________________
dev-i18n mailing list
dev-i18n@...
https://lists.mozilla.org/listinfo/dev-i18n

[Axel Hecht]

Robert Kaiser wrote:
> Axel Hecht schrieb:
>> The workflow is basically 1), mostly because we need to support that
>> anyway. The stringbundle workflow is similar, just that you can
>> provide more than one file.
>
> Actually, for me, the gettext solution sounds like some mid-way between
> our two variants - though it doesn't merge namespaces like XUL currently
> does. I'm not sure if that merging is good to do, desired or even needed
> though.

Merging is needed to reference between application and toolkit strings
back and forth.

>> I'm calling the sets of lol files that are packed up "contexts", in
>> which entities are resolved. Those contexts include both programmatic
>> parameters and fallback. How those ocntexts are set up depends on the
>> language mapping.
>
> Hmm, I wonder if there should be some specified, cross-language-portable
> way of accessing lol files/contexts, though, so that you can use them in
> basically the same style from C/C++, PHP, JS, Java or whatever.
>
> (Though I'd prefer an object-based solution where OO is supported well
> enough, i.e. C++, PHP5, JS, etc.)

 From my own state of mind, language mappings will be most elegant in
loosely typed OO languages as JavaScript. Strongly typed languages will
have face some challenges for programmatic parameters. Non-OO languages
will likely suck, as any attempt to do objects in those language will.

That said, the algorithm to construct the contexts will be very similar
in all language bindings. Things like language selection probably wont,
I think.

>>> Additionally, how is L20n supposed to find its files for a specific
>>> language as well as the files for fallbacks?
>>
>> That's not clearly specified yet. In the case of, say, multiple
>> spanishs, it's not really clear who's falling back to what, and it
>> might depend on the "module" in whatever sense.
>>
>> This is, again, part of the language binding. It might make sense to
>> support some kind of specification of fallback in the lol file itself,
>> though I'm not sure that's the right modularity.
>
> I think we should try to have the possibility to use the same set of lol
> files from different languages, so some generic specification of finding
> files and doing fallback sounds like a good idea.

I think so, too. I just expect that packaging and such would differ
enough for this to be programming-language- and deployment-specific.

>>> I'd really like to do some crude testing implementation for PHP just
>>> to see if L20n is really as good as I imagine it currently. From what
>>> I know currently, the L20n concepts also make dev work much easier
>>> for me in PHP (not needing to implement plural selectors in the code
>>> itself in many places, leave that up to the localization layer, for
>>> example - and of course getting rid of long original strings mixed
>>> into the code).
>>> Would be cool to be able to test some of that ;-)
>>
>> We should get interested people together, we should poke at least Wil
>> Clouser and Adrian Kalla.
>
> That would be cool. I'm sure many web-based tools would profit from a
> good PHP impl (even better and faster would probably be some native
> support inside PHP but such stuff usually takes longer).
>
>> It seems that php doesn't support /y of js re (new in 1.9) nor \G,
>> implementing the parser performantly might not be able to use regular
>> expressions :-/.
>
> I see \G mentioned in the preg_* (PCRE-based regex) module
> documentation, and from what I read, preg-* is preferred over ereg_*
> (POSIX-extended regex) due to being binary-safe, faster and more powerful.

Ah, I saw it mentioned as not supported, but it is just working
differently, as php actually offers an offset argument.

>> You can see, I'm pretty interested in getting a php-based impl, as
>> that should really get the most exposure per impl.
>
> I think so as well. And it would/will probably be a good proof of the
> concepts.
>
> Robert Kaiser

Axel
_______________________________________________
dev-i18n mailing list
dev-i18n@...
https://lists.mozilla.org/listinfo/dev-i18n

[Robert Kaiser]

Axel Hecht schrieb:
> Merging is needed to reference between application and toolkit strings
> back and forth.

I would have suspected that a namespaced solution would work just as
well there, esp. when a default namespace (context, "file" or whatevery
you call it) is set.

>  From my own state of mind, language mappings will be most elegant in
> loosely typed OO languages as JavaScript. Strongly typed languages will
> have face some challenges for programmatic parameters. Non-OO languages
> will likely suck, as any attempt to do objects in those language will.

I think we agree on that. JS and PHP5 will probably look quite nice ;-)

> That said, the algorithm to construct the contexts will be very similar
> in all language bindings. Things like language selection probably wont,
> I think.

Hmm, I'd guess it should be somehow consistent though.

Would be nice to see some basic code in multiple programming languages
to find out how well this all works out. For the first tests, it would
probably be enough if they understood "simple" L20n objects, dealing
with the more complicated structures can then be implemented on top of
that (esp. as it's bascially all just in the library layer).

>> I see \G mentioned in the preg_* (PCRE-based regex) module
>> documentation, and from what I read, preg-* is preferred over ereg_*
>> (POSIX-extended regex) due to being binary-safe, faster and more
>> powerful.
>
> Ah, I saw it mentioned as not supported, but it is just working
> differently, as php actually offers an offset argument.

Yes, true. I very much like preg_match_all() actually for cases where
you need to find recurring patterns (as I expect we'll need). This often
is a good option in places where you else would go with \G-constructs, I
guess.


Robert Kaiser
_______________________________________________
dev-i18n mailing list
dev-i18n@...
https://lists.mozilla.org/listinfo/dev-i18n

[Axel Hecht]

Robert Kaiser wrote:

> Axel Hecht schrieb:
>> Merging is needed to reference between application and toolkit strings
>> back and forth.
>
> I would have suspected that a namespaced solution would work just as
> well there, esp. when a default namespace (context, "file" or whatevery
> you call it) is set.
>
>>  From my own state of mind, language mappings will be most elegant in
>> loosely typed OO languages as JavaScript. Strongly typed languages
>> will have face some challenges for programmatic parameters. Non-OO
>> languages will likely suck, as any attempt to do objects in those
>> language will.
>
> I think we agree on that. JS and PHP5 will probably look quite nice ;-)
>
>> That said, the algorithm to construct the contexts will be very
>> similar in all language bindings. Things like language selection
>> probably wont, I think.
>
> Hmm, I'd guess it should be somehow consistent though.
>
> Would be nice to see some basic code in multiple programming languages
> to find out how well this all works out. For the first tests, it would
> probably be enough if they understood "simple" L20n objects, dealing
> with the more complicated structures can then be implemented on top of
> that (esp. as it's bascially all just in the library layer).

Yeah.

>>> I see \G mentioned in the preg_* (PCRE-based regex) module
>>> documentation, and from what I read, preg-* is preferred over ereg_*
>>> (POSIX-extended regex) due to being binary-safe, faster and more
>>> powerful.
>>
>> Ah, I saw it mentioned as not supported, but it is just working
>> differently, as php actually offers an offset argument.
>
> Yes, true. I very much like preg_match_all() actually for cases where
> you need to find recurring patterns (as I expect we'll need). This often
> is a good option in places where you else would go with \G-constructs, I
> guess.

In this case, you can find false positives, for example

<key: "This looks like an entry, <key: \"foo\">, doesn't it?">

I find context sensitive parsers to be much more efficient, sadly, there
are hardly any out there.

Axel
_______________________________________________
dev-i18n mailing list
dev-i18n@...
https://lists.mozilla.org/listinfo/dev-i18n