Catalyst::Controller::REST JSON Serialization/Encoding Problem

[Alejandro Imass-3]

Hi,

The current version (0.77) seems to be encoding JSON with UTF-8 data
wrong. In fact. I have noticed this problem since earlier versions,
and it's still present in 0.77. I think that with JSON v2 many things
changed and using encode_json is actually re-encoding the data, if the
data is already in UTF-8. I have found that by changing to_json
instead of encode_json it all works perfectly. The JSON pod is very
confusing when it comes to  encoding and has a complete section on it
"A FEW NOTES ON UNICODE AND PERL" so I am pretty sure after reading it
several times that there is s a problem when using encode_json and we
should be using to_json.

The current maintainer should verify this, but I would suggest to
change encode_json for to_json like so:

In file Catalyst::Action::Serialize::JSON
-----------------------------------------------

#use JSON qw(encode_json);
use JSON qw(to_json);

In sub serialize()
---------------------

sub serialize {
    my $self = shift;
    #encode_json( shift );
    to_json( shift );
}

Also, I don't think that JSON::XS is explicitly necessary because from
the JSON::XS v2 pod

# Note that JSON version 2.0 and above will automatically use JSON::XS
        # if available, at virtually no speed overhead either, so you should
        # be able to just:

        use JSON;


Well, that's it for now, but I can verify that by using to_json it
actually solves my encoding problems.

Best,
Alejandro Imass

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Tomas Doran]

Alejandro Imass wrote:
> Well, that's it for now, but I can verify that by using to_json it
> actually solves my encoding problems.

Well, I have this working fine for me in my environment, but I'm
prepared to believe it may not work for you, as we do fairly wacky
things with uft8 (for hysterical raisins)..

Could you supply a short test to demonstrate feeding some utf8 data in
and it being double encoded when serialized?

Cheers
t0m


_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Alejandro Imass-3]

Sure, I can work out an example but I'm up to my neck in work so it
won't be immediate...

On Mon, Sep 28, 2009 at 11:58 AM, Tomas Doran <bobtfish@...> wrote:

> Alejandro Imass wrote:
>>
>> Well, that's it for now, but I can verify that by using to_json it
>> actually solves my encoding problems.
>
> Well, I have this working fine for me in my environment, but I'm prepared to
> believe it may not work for you, as we do fairly wacky things with uft8 (for
> hysterical raisins)..
>
> Could you supply a short test to demonstrate feeding some utf8 data in and
> it being double encoded when serialized?
>
> Cheers
> t0m
>
>

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Alejandro Imass-3]

I have sent you an example directly to your mail. If anyone else wants
to test, let me know and I will forward to you as well.

Thanks,
Alejandro Imass

On Mon, Sep 28, 2009 at 11:58 AM, Tomas Doran <bobtfish@...> wrote:

> Alejandro Imass wrote:
>>
>> Well, that's it for now, but I can verify that by using to_json it
>> actually solves my encoding problems.
>
> Well, I have this working fine for me in my environment, but I'm prepared to
> believe it may not work for you, as we do fairly wacky things with uft8 (for
> hysterical raisins)..
>
> Could you supply a short test to demonstrate feeding some utf8 data in and
> it being double encoded when serialized?
>
> Cheers
> t0m
>
>

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Alejandro Imass-3]

Ok. I have verified it's the encoding function's problem, or at least
the encoding problem is in the server, not JQuery or JS.

I dumped the $output just before returning and the output is all
scrambled. My shell and complete locale in Debian is UTF-8 so the
problem seems effectively to be the encode_json function.

On Mon, Sep 28, 2009 at 11:58 AM, Tomas Doran <bobtfish@...> wrote:

> Alejandro Imass wrote:
>>
>> Well, that's it for now, but I can verify that by using to_json it
>> actually solves my encoding problems.
>
> Well, I have this working fine for me in my environment, but I'm prepared to
> believe it may not work for you, as we do fairly wacky things with uft8 (for
> hysterical raisins)..
>
> Could you supply a short test to demonstrate feeding some utf8 data in and
> it being double encoded when serialized?
>
> Cheers
> t0m
>
>

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/