Doing a recursive JSON.readFull

[Mario Camou]

Hi all,

I need to read some JSON data from a server. I need the data to be parsed into Maps of Lists of Maps (or Maps of Maps, or Lists of Lists of Maps, you get the idea). I was going through the scaladocs for the Scala library and found scala.util.parsing.json.JSON.parseFull, whose docs state that it "Parse the given JSON string and return either a List[Any] if the JSON string specifies an Array, or a Map[String,Any] if the JSON string specifies an object". I thought that was just what I needed, but it turns out it works only for the first level, and it leaves any lower structures as lists of pairs instead of Maps.

Has anyone here worked with a JSON parser that does what I need? I can always write a method to convert from one representation to the other, but I thought I'd ask around first.

Now that I think about it, I'll probably have to go with a Java library or build my own, since I actually need the Lists and Maps to be the Java variants (since I'm going to have to pass them to JRuby).

Any ideas?

Thanks,
-Mario.

--
I want to change the world but they won't give me the source code.

[Lincoln-26]

I have used Twitter's implementation.  It's worked well for me and
does what I think you're looking for.

You can find it here: http://github.com/stevej/scala-json/tree/master.

Lincoln

On Sun, Jul 5, 2009 at 5:20 PM, Mario Camou<mcamou@...> wrote:

> Hi all,
> I need to read some JSON data from a server. I need the data to be parsed
> into Maps of Lists of Maps (or Maps of Maps, or Lists of Lists of Maps, you
> get the idea). I was going through the scaladocs for the Scala library and
> found scala.util.parsing.json.JSON.parseFull, whose docs state that it
> "Parse the given JSON string and return either a List[Any] if the JSON
> string specifies an Array, or a Map[String,Any] if the JSON string specifies
> an object". I thought that was just what I needed, but it turns out it works
> only for the first level, and it leaves any lower structures as lists of
> pairs instead of Maps.
> Has anyone here worked with a JSON parser that does what I need? I can
> always write a method to convert from one representation to the other, but I
> thought I'd ask around first.
> Now that I think about it, I'll probably have to go with a Java library or
> build my own, since I actually need the Lists and Maps to be the Java
> variants (since I'm going to have to pass them to JRuby).
> Any ideas?
> Thanks,
> -Mario.
>
> --
> I want to change the world but they won't give me the source code.
>

[Naftoli Gugenheim]

It's a Seq of pairs? So why not just transform it with recursive pattern matching to Map(seq: _*) or while you're at it, to java collections, also shouldn't be too hard--just foreach and say case (key, value)=>jmap.put(key,value) etc. 2.8 should have bidirectional conversions anyway.

-------------------------------------
Mario Camou<mcamou@...> wrote:

Hi all,
I need to read some JSON data from a server. I need the data to be parsed
into Maps of Lists of Maps (or Maps of Maps, or Lists of Lists of Maps, you
get the idea). I was going through the scaladocs for the Scala library and
found scala.util.parsing.json.JSON.parseFull, whose docs state that it
"Parse the given JSON string and return either a List[Any] if the JSON
string specifies an Array, or a Map[String,Any] if the JSON string specifies
an object". I thought that was just what I needed, but it turns out it works
only for the first level, and it leaves any lower structures as lists of
pairs instead of Maps.

Has anyone here worked with a JSON parser that does what I need? I can
always write a method to convert from one representation to the other, but I
thought I'd ask around first.

Now that I think about it, I'll probably have to go with a Java library or
build my own, since I actually need the Lists and Maps to be the Java
variants (since I'm going to have to pass them to JRuby).

Any ideas?

Thanks,
-Mario.

--
I want to change the world but they won't give me the source code.

[Mario Camou]

¡That was fast! Thanks for the quick answer and the pointers.

It's a List of Pairs, the first element of the Pair is a String, the second can be a List of Pairs (ad infinitum), a List of Lists of Pairs, or an atomic object. The idea would be to convert any "List of Pairs" structure to a corresponding Map.

I'll probably end up doing it just as an exercise (I'm a Scala beginner). Thanks again!
-Mario.

--
I want to change the world but they won't give me the source code.

On Sun, Jul 5, 2009 at 23:28, Naftoli Gugenhem <naftoligug@...> wrote:

It's a Seq of pairs? So why not just transform it with recursive pattern matching to Map(seq: _*) or while you're at it, to java collections, also shouldn't be too hard--just foreach and say case (key, value)=>jmap.put(key,value) etc. 2.8 should have bidirectional conversions anyway.

-------------------------------------

Mario Camou<mcamou@...> wrote:

Hi all,
I need to read some JSON data from a server. I need the data to be parsed
into Maps of Lists of Maps (or Maps of Maps, or Lists of Lists of Maps, you
get the idea). I was going through the scaladocs for the Scala library and
found scala.util.parsing.json.JSON.parseFull, whose docs state that it
"Parse the given JSON string and return either a List[Any] if the JSON
string specifies an Array, or a Map[String,Any] if the JSON string specifies
an object". I thought that was just what I needed, but it turns out it works
only for the first level, and it leaves any lower structures as lists of
pairs instead of Maps.

Has anyone here worked with a JSON parser that does what I need? I can
always write a method to convert from one representation to the other, but I
thought I'd ask around first.

Now that I think about it, I'll probably have to go with a Java library or
build my own, since I actually need the Lists and Maps to be the Java
variants (since I'm going to have to pass them to JRuby).

Any ideas?

Thanks,
-Mario.

--
I want to change the world but they won't give me the source code.

[Mario Camou]

Thanks for the quick answer! I'll have a look at it. I thought Twitter's implementation was similar to the one included in the Scala libs.

I'll probably end up using it since I assume it's quite well tested, although I'll roll my own as an exercise.

Thanks again!
-Mario.

--
I want to change the world but they won't give me the source code.

On Sun, Jul 5, 2009 at 23:28, Lincoln <linxbetter@...> wrote:

I have used Twitter's implementation.  It's worked well for me and
does what I think you're looking for.

You can find it here: http://github.com/stevej/scala-json/tree/master.

Lincoln

On Sun, Jul 5, 2009 at 5:20 PM, Mario Camou<mcamou@...> wrote:
> Hi all,
> I need to read some JSON data from a server. I need the data to be parsed
> into Maps of Lists of Maps (or Maps of Maps, or Lists of Lists of Maps, you
> get the idea). I was going through the scaladocs for the Scala library and
> found scala.util.parsing.json.JSON.parseFull, whose docs state that it
> "Parse the given JSON string and return either a List[Any] if the JSON
> string specifies an Array, or a Map[String,Any] if the JSON string specifies
> an object". I thought that was just what I needed, but it turns out it works
> only for the first level, and it leaves any lower structures as lists of
> pairs instead of Maps.
> Has anyone here worked with a JSON parser that does what I need? I can
> always write a method to convert from one representation to the other, but I
> thought I'd ask around first.
> Now that I think about it, I'll probably have to go with a Java library or
> build my own, since I actually need the Lists and Maps to be the Java
> variants (since I'm going to have to pass them to JRuby).
> Any ideas?
> Thanks,
> -Mario.
>
> --
> I want to change the world but they won't give me the source code.
>

[Naftoli Gugenheim]

Yeah, so it sounds like recursive pattern matching is what you want.
case (str, (l: List[_]) :: rest) => ...
should get a list of lists without issues of type erasure.

-------------------------------------
Mario Camou<mcamou@...> wrote:

¡That was fast! Thanks for the quick answer and the pointers.
It's a List of Pairs, the first element of the Pair is a String, the second
can be a List of Pairs (ad infinitum), a List of Lists of Pairs, or an
atomic
object. The idea would be to convert any "List of Pairs" structure to
a corresponding Map.

I'll probably end up doing it just as an exercise (I'm a Scala
beginner). Thanks again!
-Mario.

--
I want to change the world but they won't give me the source code.


On Sun, Jul 5, 2009 at 23:28, Naftoli Gugenhem <naftoligug@...> wrote:

> It's a Seq of pairs? So why not just transform it with recursive pattern
> matching to Map(seq: _*) or while you're at it, to java collections, also
> shouldn't be too hard--just foreach and say case (key,
> value)=>jmap.put(key,value) etc. 2.8 should have bidirectional conversions
> anyway.
>
> -------------------------------------
> Mario Camou<mcamou@...> wrote:
>
> Hi all,
> I need to read some JSON data from a server. I need the data to be parsed
> into Maps of Lists of Maps (or Maps of Maps, or Lists of Lists of Maps, you
> get the idea). I was going through the scaladocs for the Scala library and
> found scala.util.parsing.json.JSON.parseFull, whose docs state that it
> "Parse the given JSON string and return either a List[Any] if the JSON
> string specifies an Array, or a Map[String,Any] if the JSON string
> specifies
> an object". I thought that was just what I needed, but it turns out it
> works
> only for the first level, and it leaves any lower structures as lists of
> pairs instead of Maps.
>
> Has anyone here worked with a JSON parser that does what I need? I can
> always write a method to convert from one representation to the other, but
> I
> thought I'd ask around first.
>
> Now that I think about it, I'll probably have to go with a Java library or
> build my own, since I actually need the Lists and Maps to be the Java
> variants (since I'm going to have to pass them to JRuby).
>
> Any ideas?
>
> Thanks,
> -Mario.
>
> --
> I want to change the world but they won't give me the source code.
>

[braver]

In our experience, scala-json turned out way too slow, as it allocates a lot.  If you have lots of data to process, Xstream works well (http://xstream.codehaus.org/json-tutorial.html), and Jackson Fast JSON parser (http://jackson.codehaus.org/) is faster still.

Also, it appears that while scala-json adheres to the Stairway book, scala.util.parsing.json does not, and overall seems a premature choice for the JSON parser in the standard library.

Cheers,

Alexy

[Naftoli Gugenheim]

What's scala-json? I think the OP was talking about lift's parser.

-------------------------------------
Alexy Khrabrov<deliverable@...> wrote:

In our experience, scala-json turned out way too slow, as it allocates a
lot.  If you have lots of data to process, Xstream works well (
http://xstream.codehaus.org/json-tutorial.html), and Jackson Fast JSON
parser (http://jackson.codehaus.org/) is faster still.
Also, it appears that while scala-json adheres to the Stairway book,
scala.util.parsing.json does not, and overall seems a premature choice for
the JSON parser in the standard library.

Cheers,
Alexy

[braver]

On Mon, Jul 6, 2009 at 1:40 AM, Naftoli Gugenhem <naftoligug@...> wrote:

What's scala-json? I think the OP was talking about lift's parser.

The OP asked about scala.util.parsing.json from the standard library, and Lincoln's reply mentioned scala-json off github, which is an almost literal  implementation of the Stairway book one.

Cheers,

Alexy

[Naftoli Gugenheim]

Whoops, you're right. Don't ask where I got that impression from.
I wonder how lift's compares to the others you mentioned.

-------------------------------------
Alexy Khrabrov<deliverable@...> wrote:

On Mon, Jul 6, 2009 at 1:40 AM, Naftoli Gugenhem <naftoligug@...>wrote:

> What's scala-json? I think the OP was talking about lift's parser.


The OP asked about scala.util.parsing.json from the standard library, and
Lincoln's reply mentioned scala-json off github, which is an almost literal
 implementation of the Stairway book one.

Cheers,
Alexy

[Lincoln-26]

Thanks for those references Alexy.  I'm going to take a look at them.

On Mon, Jul 6, 2009 at 1:36 AM, Alexy Khrabrov<deliverable@...> wrote:
> In our experience, scala-json turned out way too slow, as it allocates a
> lot.  If you have lots of data to process, Xstream works well
> (http://xstream.codehaus.org/json-tutorial.html), and Jackson Fast JSON
> parser (http://jackson.codehaus.org/) is faster still.
> Also, it appears that while scala-json adheres to the Stairway book,
> scala.util.parsing.json does not, and overall seems a premature choice for
> the JSON parser in the standard library.
> Cheers,
> Alexy

[Mario Camou]

Hi again,

I ended up using Jackson as per Alexy's reference. Very nice and simple, just what I need. Thanks!

Now another question. I need an HTTP client library that can do HTTP Basic authentication and all four basic HTTP ops (GET, PUT, POST, DELETE -- I'll leave out TRACE and HEAD for the moment). All I've found so far is the Apache HTTPClient which, frankly, seems to be overkill. Does anyone around here know of a better Scala-based alternative?

Thanks again,

-Mario.

--
I want to change the world but they won't give me the source code.

On Mon, Jul 6, 2009 at 1:36 AM, Alexy Khrabrov<deliverable@...> wrote:
> In our experience, scala-json turned out way too slow, as it allocates a
> lot.  If you have lots of data to process, Xstream works well
> (http://xstream.codehaus.org/json-tutorial.html), and Jackson Fast JSON
> parser (http://jackson.codehaus.org/) is faster still.
> Also, it appears that while scala-json adheres to the Stairway book,
> scala.util.parsing.json does not, and overall seems a premature choice for
> the JSON parser in the standard library.
> Cheers,
> Alexy

[Lincoln-26]

Hi Alexy,

I'm looking at Jackson now.  Do you have any best practices for
working with it.  I'm using ObjectMapper but I'm struggling with how
to maintain immutable objects and use Jackson at the same time.  Of
course if I just make Beans and use @BeanProperty on everything then
all is well but I was hoping to use these objects elsewhere and I've
been trying to do things the scala way and keep them immutable.  Any
hints?

Thanks,
Lincoln

On Mon, Jul 6, 2009 at 1:36 AM, Alexy Khrabrov<deliverable@...> wrote:
> In our experience, scala-json turned out way too slow, as it allocates a
> lot.  If you have lots of data to process, Xstream works well
> (http://xstream.codehaus.org/json-tutorial.html), and Jackson Fast JSON
> parser (http://jackson.codehaus.org/) is faster still.
> Also, it appears that while scala-json adheres to the Stairway book,
> scala.util.parsing.json does not, and overall seems a premature choice for
> the JSON parser in the standard library.
> Cheers,
> Alexy

[Grey]

Here is a very old effort I did is an example of using Jackson with Scala.  Please don't even consider it a best practice or anything of the sort, best just an example.

http://github.com/RayRacine/Crank/blob/be3a124c09b797cbac14491c0ae4ff1efbe466bb/crank-client/src/main/scala/com/rracine/crank/client/document/json/DocValJsonSerializer.scala

Ray

On Wed, Aug 5, 2009 at 1:47 PM, Lincoln <linxbetter@...> wrote:

Hi Alexy,

I'm looking at Jackson now.  Do you have any best practices for
working with it.  I'm using ObjectMapper but I'm struggling with how
to maintain immutable objects and use Jackson at the same time.  Of
course if I just make Beans and use @BeanProperty on everything then
all is well but I was hoping to use these objects elsewhere and I've
been trying to do things the scala way and keep them immutable.  Any
hints?

Thanks,
Lincoln

On Mon, Jul 6, 2009 at 1:36 AM, Alexy Khrabrov<deliverable@...> wrote:
> In our experience, scala-json turned out way too slow, as it allocates a
> lot.  If you have lots of data to process, Xstream works well
> (http://xstream.codehaus.org/json-tutorial.html), and Jackson Fast JSON
> parser (http://jackson.codehaus.org/) is faster still.
> Also, it appears that while scala-json adheres to the Stairway book,
> scala.util.parsing.json does not, and overall seems a premature choice for
> the JSON parser in the standard library.
> Cheers,
> Alexy

[Grey]

Usage example:
http://github.com/RayRacine/Crank/blob/be3a124c09b797cbac14491c0ae4ff1efbe466bb/crank-client/src/test/scala/com/rracine/crank/client/document/json/DocValJsonSerializerTest.scala

On Wed, Aug 5, 2009 at 1:47 PM, Lincoln <linxbetter@...> wrote:

Hi Alexy,

I'm looking at Jackson now.  Do you have any best practices for
working with it.  I'm using ObjectMapper but I'm struggling with how
to maintain immutable objects and use Jackson at the same time.  Of
course if I just make Beans and use @BeanProperty on everything then
all is well but I was hoping to use these objects elsewhere and I've
been trying to do things the scala way and keep them immutable.  Any
hints?

Thanks,
Lincoln

On Mon, Jul 6, 2009 at 1:36 AM, Alexy Khrabrov<deliverable@...> wrote:
> In our experience, scala-json turned out way too slow, as it allocates a
> lot.  If you have lots of data to process, Xstream works well
> (http://xstream.codehaus.org/json-tutorial.html), and Jackson Fast JSON
> parser (http://jackson.codehaus.org/) is faster still.
> Also, it appears that while scala-json adheres to the Stairway book,
> scala.util.parsing.json does not, and overall seems a premature choice for
> the JSON parser in the standard library.
> Cheers,
> Alexy

[francisco treacy-2]

Lincoln,

I have successfully used Paranamer [1] to initialize constructors of
immutable objects via reflection (from JSON fields names and values) -
IIUC this might help.

See line 63 of http://github.com/frank06/surf/blob/e1f176a5eeaf7acccaa2a869489cc3c65d44089f/src/main/scala/surf/Json.scala

[1]: http://paranamer.codehaus.org/

Francisco


2009/8/5 Lincoln <linxbetter@...>:

> Hi Alexy,
>
> I'm looking at Jackson now.  Do you have any best practices for
> working with it.  I'm using ObjectMapper but I'm struggling with how
> to maintain immutable objects and use Jackson at the same time.  Of
> course if I just make Beans and use @BeanProperty on everything then
> all is well but I was hoping to use these objects elsewhere and I've
> been trying to do things the scala way and keep them immutable.  Any
> hints?
>
> Thanks,
> Lincoln
>
> On Mon, Jul 6, 2009 at 1:36 AM, Alexy Khrabrov<deliverable@...> wrote:
>> In our experience, scala-json turned out way too slow, as it allocates a
>> lot.  If you have lots of data to process, Xstream works well
>> (http://xstream.codehaus.org/json-tutorial.html), and Jackson Fast JSON
>> parser (http://jackson.codehaus.org/) is faster still.
>> Also, it appears that while scala-json adheres to the Stairway book,
>> scala.util.parsing.json does not, and overall seems a premature choice for
>> the JSON parser in the standard library.
>> Cheers,
>> Alexy
>

[Lincoln-26]

Thanks Ray.

After digging around I decided to do the following:

trait JSON {
  def asJMap:JMap[String,Object]
  def toJson = JSON compose asJMap
}

object JSON {
  private val mapper = new ObjectMapper
  def parse(json:String):JMap[String,Object] = mapper.readValue(json,
classOf[JMap[String,Object]])
  def compose(o:Object) = {
    val sw = new StringWriter
    mapper.writeValue(sw, o)
    sw.toString
}

Now, in each of my domain classes I just extend JSON and define the
asJMap method.

I also, by convention, define a constructor in each domain class'
companion object that takes a json string and uses JSON.parse to
create a new instance.

This seems to work pretty well.  Any thoughts on whether this is a
good approach?  My main concern is that there might be clever
scala-idiomatic ways to reduce some repetition and/or verbosity but
I'm enough of a newbie that I'm not able to think of any right now.
The main reason for the verbosity has to do with converting scala
objects to the java objects that Jackson requires.

FYI I also got Jackson working directly against my domain classes,
without requiring JMap conversion, but I ultimately decided against
the approach because I wanted more flexibility in my domain objects -
namely, I wanted to keep them immutable and I sometimes wanted the
ability to make the outputted json appear slightly different than the
class structure would allow.

Thanks,
Lincoln
       

On Wed, Aug 5, 2009 at 2:37 PM, Ray Racine<ray.racine@...> wrote:

> Usage example:
> http://github.com/RayRacine/Crank/blob/be3a124c09b797cbac14491c0ae4ff1efbe466bb/crank-client/src/test/scala/com/rracine/crank/client/document/json/DocValJsonSerializerTest.scala
>
> On Wed, Aug 5, 2009 at 1:47 PM, Lincoln <linxbetter@...> wrote:
>>
>> Hi Alexy,
>>
>> I'm looking at Jackson now.  Do you have any best practices for
>> working with it.  I'm using ObjectMapper but I'm struggling with how
>> to maintain immutable objects and use Jackson at the same time.  Of
>> course if I just make Beans and use @BeanProperty on everything then
>> all is well but I was hoping to use these objects elsewhere and I've
>> been trying to do things the scala way and keep them immutable.  Any
>> hints?
>>
>> Thanks,
>> Lincoln
>>
>> On Mon, Jul 6, 2009 at 1:36 AM, Alexy Khrabrov<deliverable@...>
>> wrote:
>> > In our experience, scala-json turned out way too slow, as it allocates a
>> > lot.  If you have lots of data to process, Xstream works well
>> > (http://xstream.codehaus.org/json-tutorial.html), and Jackson Fast JSON
>> > parser (http://jackson.codehaus.org/) is faster still.
>> > Also, it appears that while scala-json adheres to the Stairway book,
>> > scala.util.parsing.json does not, and overall seems a premature choice
>> > for
>> > the JSON parser in the standard library.
>> > Cheers,
>> > Alexy
>
>

[Lincoln-26]

Very cool stuff.  Maybe I can use this to consolidate all the
companion object constructors I'm writing.

Thanks Francisco.

Lincoln

On Thu, Aug 6, 2009 at 6:35 AM, francisco
treacy<francisco.treacy@...> wrote:

> Lincoln,
>
> I have successfully used Paranamer [1] to initialize constructors of
> immutable objects via reflection (from JSON fields names and values) -
> IIUC this might help.
>
> See line 63 of http://github.com/frank06/surf/blob/e1f176a5eeaf7acccaa2a869489cc3c65d44089f/src/main/scala/surf/Json.scala
>
> [1]: http://paranamer.codehaus.org/
>
> Francisco
>
>
> 2009/8/5 Lincoln <linxbetter@...>:
>> Hi Alexy,
>>
>> I'm looking at Jackson now.  Do you have any best practices for
>> working with it.  I'm using ObjectMapper but I'm struggling with how
>> to maintain immutable objects and use Jackson at the same time.  Of
>> course if I just make Beans and use @BeanProperty on everything then
>> all is well but I was hoping to use these objects elsewhere and I've
>> been trying to do things the scala way and keep them immutable.  Any
>> hints?
>>
>> Thanks,
>> Lincoln
>>
>> On Mon, Jul 6, 2009 at 1:36 AM, Alexy Khrabrov<deliverable@...> wrote:
>>> In our experience, scala-json turned out way too slow, as it allocates a
>>> lot.  If you have lots of data to process, Xstream works well
>>> (http://xstream.codehaus.org/json-tutorial.html), and Jackson Fast JSON
>>> parser (http://jackson.codehaus.org/) is faster still.
>>> Also, it appears that while scala-json adheres to the Stairway book,
>>> scala.util.parsing.json does not, and overall seems a premature choice for
>>> the JSON parser in the standard library.
>>> Cheers,
>>> Alexy
>>
>

[braver]

I use streaming API for Jackson, extracting some of the fields I need and then creating objects myself.  It's very fast and allows for complete control.  The object magic described earlier is fascinating though.  Jackson 1.2.0 has just come out, so the usage will only improve.

Cheers,

Alexy