convert to metaCharset or renderCharset?

[Franz Holzinger-2]

Hello,

when does an extension need to convert the charset and which charset
should it convert to, metaCharset or renderCharset?
Should it always convert to renderCharset, oder only if there is the
same for metaCharset?

The conversion of $v is done by this:
$v = $csConvObj->conv($v, 'utf-8', $charset);

see also the question from Masi:

http://lists.netfielders.de/pipermail/typo3-team-core/2006-June/004901.html

- Franz
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Popy]

Theorically, your extension does not need to convert charsets.
It have to generate content into the "renderCharset" (which is generally the
database charset too), then the TSFE class will convert it into metaCharset
if needed.

Of course, if you fetch external data, you may have to convert the content
you get into the renderCharset

Popy

2009/10/8 Franz Holzinger <franz@...>

> Hello,
>
> when does an extension need to convert the charset and which charset
> should it convert to, metaCharset or renderCharset?
> Should it always convert to renderCharset, oder only if there is the
> same for metaCharset?
>
> The conversion of $v is done by this:
> $v = $csConvObj->conv($v, 'utf-8', $charset);
>
> see also the question from Masi:
>
> http://lists.netfielders.de/pipermail/typo3-team-core/2006-June/004901.html
>
> - Franz
> _______________________________________________
> TYPO3-dev mailing list
> TYPO3-dev@...
> http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev
>

_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Martin Kutschker-7]

Franz Holzinger schrieb:
> Hello,
>
> when does an extension need to convert the charset

When you read data outside of the scope of TYPO3. eg when you import files. GET and POST data are
handled by TYPO3 and so is the page rendering.

> and which charset should it convert to, metaCharset or renderCharset?

In the normal flow of page generation use renderCharset. This is the internal charset TYPO3 works on
(also used in the DB). The metaCharset should perhaps be called outputCharset or maybe even
clientCharset. TYPO3 will convert from the renderCharset to the metaCharset right before sending the
page to the client (browser).

It must be a very tricky edge when when use are forced to output data directly in the metaCharset.
But I cannot image any right now.

Masi
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Franz Holzinger-2]

Hello Popy,

> Theorically, your extension does not need to convert charsets.
> It have to generate content into the "renderCharset" (which is generally the
> database charset too), then the TSFE class will convert it into metaCharset
> if needed.
>
> Of course, if you fetch external data, you may have to convert the content
> you get into the renderCharset

I must do this conversion for the data I return to the xAjax call and
data read in from the extensions tables by SQL calls.

I think I have found the answer here:
http://typo3.org/documentation/document-library/references/doc_core_tsref/4.1.0/view/7/3/


If renderCharset and metaCharset are different the output content is
automatically converted to metaCharset before output.

So metaCharset will be used if it is different to renderCharset.


- Franz
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Martin Kutschker-7]

Franz Holzinger schrieb:
>
> I must do this conversion for the data I return to the xAjax call and
> data read in from the extensions tables by SQL calls.

Ah, AJAX. In this case you must determine in which charset you want to transmit your data. Many
libraries expect JSON/XML to be in utf8 independently from the charset used in the corresponding
HTML page.

In this case must convert to and from renderCharset to utf8.

Masi
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Franz Holzinger-2]

Franz Holzinger a écrit :

>
> If renderCharset and metaCharset are different the output content is
> automatically converted to metaCharset before output.
>
> So metaCharset will be used if it is different to renderCharset.

No this does not work!
The result is wrong:

"Diese Universalgröße ist für  Erwachsene und Kinder gleichermaßen
geeignet, denn häufig"


$v = $csConvObj->conv($v, $TSFE->renderCharset, $TSFE->metaCharset);


$TSFE->renderCharset = iso-8859-1
$TSFE->metaCharset = utf-8


The output is only correct, if I do not do any charset conversion at all.

"Diese Universalgröße ist für  Erwachsene und Kinder gleichermaßen
geeignet, denn häufig"

But what is the rule for this?

- Franz
_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Franz Holzinger-2]

Hello Masi,

>> I must do this conversion for the data I return to the xAjax call and
>> data read in from the extensions tables by SQL calls.
>
> Ah, AJAX. In this case you must determine in which charset you want to transmit your data. Many
> libraries expect JSON/XML to be in utf8 independently from the charset used in the corresponding
> HTML page.
>
This can be configured. I have used:

$charset = $TSFE->renderCharset;

Which is iso-8859-1 in my case.

        // Encoding of the response to FE charset
$this->taxajax->setCharEncoding($charset);


Should this always use utf-8 instead of renderCharset?

> In this case must convert to and from renderCharset to utf8.

This still did not work.

"Diese Universalgröße ist für  Erwachsene und Kinder gleichermaßen
geeignet, denn häufig "

So maybe I should have converted it into iso-8859-1 from the renderCharset.
But conversion from renderCharset to renderCharset does nothing and I
could remove the char conversion completely. But other charsets than
utf-8 and iso-8859-1 might not be supported any more by this method.



- Franz




_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Franz Holzinger-2]

Franz Holzinger a écrit :

> Hello Masi,
>
>>> I must do this conversion for the data I return to the xAjax call and
>>> data read in from the extensions tables by SQL calls.
>>
>> Ah, AJAX. In this case you must determine in which charset you want to
>> transmit your data. Many
>> libraries expect JSON/XML to be in utf8 independently from the charset
>> used in the corresponding
>> HTML page.
>>
> This can be configured. I have used:
>
> $charset = $TSFE->renderCharset;
>
> Which is iso-8859-1 in my case.
>
>     // Encoding of the response to FE charset
> $this->taxajax->setCharEncoding($charset);
>
>
> Should this always use utf-8 instead of renderCharset?
>
>> In this case must convert to and from renderCharset to utf8.
>
> This still did not work.
>
> "Diese Universalgröße ist für  Erwachsene und Kinder gleichermaßen
> geeignet, denn häufig "
>
> So maybe I should have converted it into iso-8859-1 from the renderCharset.
> But conversion from renderCharset to renderCharset does nothing and I
> could remove the char conversion completely. But other charsets than
> utf-8 and iso-8859-1 might not be supported any more by this method.
>

If I use utf-8 for Ajax and I later use the
$v = $csConvObj->conv($v, $TSFE->renderCharset, 'utf-8');
  conversion, then I get:

"Diese Universalgre ist fr  Erwachsene und Kinder gleichermaen geeignet,
denn hufig"


  - Franz

_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev

[Franz Holzinger-2]

Hello Masi,

Martin Kutschker a écrit :
> Franz Holzinger schrieb:
>> I must do this conversion for the data I return to the xAjax call and
>> data read in from the extensions tables by SQL calls.
>
> Ah, AJAX. In this case you must determine in which charset you want to transmit your data. Many
> libraries expect JSON/XML to be in utf8 independently from the charset used in the corresponding
> HTML page.
>
> In this case must convert to and from renderCharset to utf8.

Your solution works great! I have had an error in my code using utf8 for
Ajax and iso-8859-1 for the response which resulted in wrong characters.

Now I have moved all ajax character transfers to utf-8.

The conversion call is now:

$v = $csConvObj->conv($v, $TSFE->renderCharset,
$this->ajax->taxajax->getCharEncoding());


- Franz

_______________________________________________
TYPO3-dev mailing list
TYPO3-dev@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-dev