[scala] XML hell

I'm hoping this is just me being dumb, but I'm desperate for help.

Basically I want to perform some modification to some existing XML. So  
far everything I've come across seems fairly painful (or at least  
excessively long).


eg

http://stackoverflow.com/questions/970675/scala-modifying-nested-elements-in-xml

http://stackoverflow.com/questions/1582244/how-to-access-parent-element-in-scala-xml



I was very much hoping for something similar to (the brilliant!)  
elementTree API on Python...  and extract out lists/arrays of Nodes,  
manipulate and regenerate the XML from that.
But it looks like everything boils down to matchers.

What would be the suggested way to add an extra <foo> node to the below?

<bar>

<foos>

<foo name = 1> foo1 </foo>
<foo name = 2> foo2 </foo>
<foo name = 3> foo3 </foo>

</foos>

</bar>

?

Any comments highly appreciated

Ken

[Marcus Downing]

I do this all the time in my Java code (using XOM), and its absense from Scala is an obstacle - and as you say, the methods presented all look pretty painful. Of course, you could just use something like XOM instead of scala.xml, but I can see why that's an ugly choice.

I agree with the decision to make scala.xml's nodes immutable, but I've argued the need before for a mutable version of them. The process would be to transform one immutable structure into another via a short-lived mutable copy.

  val start = XML.load(...)
  val mutable = start.toMutable
  mutable \ "foos" += <foo name="4">foo4</foo>
  val end = mutable.toImmutable

Fundamentally this isn't too different from the approaches presented elsewhere: filter the original xml to produce a new copy based on a set of criteria, replacing only those elements that have changed. The difference is in how the criteria are built.

Sadly my functional fu is too weak to write this myself, but I believe it could be achieved. I've added a request on UserVoice, hopefully somebody can pick it up. Go vote for it!

http://scala.uservoice.com/pages/30665-libraries/suggestions/366249-scala-xml-mutable-xml

[e-mail] wrote:

I'm hoping this is just me being dumb, but I'm desperate for help.

Basically I want to perform some modification to some existing XML. So  
far everything I've come across seems fairly painful (or at least  
excessively long).


eg

http://stackoverflow.com/questions/970675/scala-modifying-nested-elements-in-xml

http://stackoverflow.com/questions/1582244/how-to-access-parent-element-in-scala-xml



I was very much hoping for something similar to (the brilliant!)  
elementTree API on Python...  and extract out lists/arrays of Nodes,  
manipulate and regenerate the XML from that.
But it looks like everything boils down to matchers.

What would be the suggested way to add an extra <foo> node to the below?

<bar>

<foos>

<foo name = 1> foo1 </foo>
<foo name = 2> foo2 </foo>
<foo name = 3> foo3 </foo>

</foos>

</bar>

?

Any comments highly appreciated

Ken

[Thomas Chust]

2009/10/27 Ken Faulkner <ken.faulkner@...>:
> [...]
> Basically I want to perform some modification to some existing XML. So far
> everything I've come across seems fairly painful (or at least excessively
> long).
> [...]

Hello,

it would indeed be nice to have a mutable XML representation in Scala,
which could facilitate this kind of manipulation. Using the tools in
the Scala standard library, the simplest approach that comes to my
mind is to do pattern matching but delegate some of the necessary work
to the scala.xml.transform framework:

  import scala.xml._
  import scala.xml.transform._

  val xexpr =
    <bar>
      <foos>
        <foo name="1"> foo1 </foo>
        <foo name="2"> foo2 </foo>
        <foo name="3"> foo3 </foo>
      </foos>
    </bar>

  val addFoo = new RuleTransformer(new RewriteRule {
    override def transform(node: Node) = node match {
      case <foos>{foos @ _*}</foos> =>
        <foos>{foos ++ <foo name="4"> another foo </foo>}</foos>
      case node =>
        super.transform(node)
    }
  })

  print(addFoo(xexpr))

Ciao,
Thomas


--
When C++ is your hammer, every problem looks like your thumb.

[huynhjl]

I was trying to do some of the sameyesterday (filtering some tags) and so your issue was relevant to me. For your case I came up with the following solution:

import scala.xml._

val x = <bar>
<foos>
<foo name = "1"> foo1 </foo>
<foo name = "2"> foo2 </foo>
<foo name = "3"> foo3 </foo>
</foos>
</bar>

def addFoos(doc:Node, foos:Seq[Node]) : Node = {
  doc match {
    case <foos>{ff @ _*}</foos> => <foos> {ff ++ foos} </foos>
    case Elem(prefix, label, attribs, scope, children @ _*) => Elem(prefix, label, attribs, scope, children map (addFoos(_, foos)):_*)
    case other => other
  }
}

println(addFoos(x, <foo name = "4"> foo4 </foo><foo name="5">foo5</foo>))
println(addFoos(x, <foo name = "6"> foo6 </foo>))

It took me a little bit of time to figure out the recursion, but I think the end result is in fact pretty simple.

[e-mail] wrote:

I'm hoping this is just me being dumb, but I'm desperate for help.

Basically I want to perform some modification to some existing XML. So  
far everything I've come across seems fairly painful (or at least  
excessively long).


eg

http://stackoverflow.com/questions/970675/scala-modifying-nested-elements-in-xml

http://stackoverflow.com/questions/1582244/how-to-access-parent-element-in-scala-xml

[...]

Ken

[Bill Burdick-2]

Given my transform definition above, this seems simpler to me.  No recursion required:

transform(xml) {case <foos>{f @ _*}</foos> => <foos>{f ++ <foo name = "10">whatever</foo>}</foos>}

Bill

On Tue, Oct 27, 2009 at 3:45 PM, huynhjl <jlh276-gh@...> wrote:

I was trying to do some of the sameyesterday (filtering some tags) and so
your issue was relevant to me. For your case I came up with the following
solution:

import scala.xml._

val x = <bar>

<foos>
<foo name = "1"> foo1 </foo>
<foo name = "2"> foo2 </foo>
<foo name = "3"> foo3 </foo>
</foos>
</bar>

def addFoos(doc:Node, foos:Seq[Node]) : Node = {
 doc match {
   case <foos>{ff @ _*}</foos> => <foos> {ff ++ foos} </foos>
   case Elem(prefix, label, attribs, scope, children @ _*) => Elem(prefix,
label, attribs, scope, children map (addFoos(_, foos)):_*)
   case other => other
 }
}

println(addFoos(x, <foo name = "4"> foo4 </foo><foo name="5">foo5</foo>))
println(addFoos(x, <foo name = "6"> foo6 </foo>))

It took me a little bit of time to figure out the recursion, but I think the
end result is in fact pretty simple.

ken.faulkner@... wrote:
>

> I'm hoping this is just me being dumb, but I'm desperate for help.
>
> Basically I want to perform some modification to some existing XML. So
> far everything I've come across seems fairly painful (or at least
> excessively long).
>
>
> eg
>
> http://stackoverflow.com/questions/970675/scala-modifying-nested-elements-in-xml
>
> http://stackoverflow.com/questions/1582244/how-to-access-parent-element-in-scala-xml
>

> [...]
>
> Ken
>

--
View this message in context: http://www.nabble.com/-scala--XML-hell-tp26071020p26077986.html

Sent from the Scala mailing list archive at Nabble.com.

Thanks for the suggestions/code everyone.

This will definitely put me on the right track. Although I still have to admit longing for a more simple API:

 val start = XML.load(...)
 val mutable = start.toMutable
 mutable \ "foos" += <foo name="4">foo4</foo>
 val end = mutable.toImmutable

(as Marcus mentioned earlier).

Still, I'll use what I've got now...

cheers,

Ken

On 28/10/2009, at 12:48 AM, Bill Burdick wrote:

Given my transform definition above, this seems simpler to me.  No recursion required:

transform(xml) {case <foos>{f @ _*}</foos> => <foos>{f ++ <foo name = "10">whatever</foo>}</foos>}

Bill

On Tue, Oct 27, 2009 at 3:45 PM, huynhjl <jlh276-gh@...> wrote:

I was trying to do some of the sameyesterday (filtering some tags) and so
your issue was relevant to me. For your case I came up with the following
solution:

import scala.xml._

val x = <bar>

<foos>
<foo name = "1"> foo1 </foo>
<foo name = "2"> foo2 </foo>
<foo name = "3"> foo3 </foo>
</foos>
</bar>

def addFoos(doc:Node, foos:Seq[Node]) : Node = {
 doc match {
   case <foos>{ff @ _*}</foos> => <foos> {ff ++ foos} </foos>
   case Elem(prefix, label, attribs, scope, children @ _*) => Elem(prefix,
label, attribs, scope, children map (addFoos(_, foos)):_*)
   case other => other
 }
}

println(addFoos(x, <foo name = "4"> foo4 </foo><foo name="5">foo5</foo>))
println(addFoos(x, <foo name = "6"> foo6 </foo>))

It took me a little bit of time to figure out the recursion, but I think the
end result is in fact pretty simple.

ken.faulkner@... wrote:
>

> I'm hoping this is just me being dumb, but I'm desperate for help.
>
> Basically I want to perform some modification to some existing XML. So
> far everything I've come across seems fairly painful (or at least
> excessively long).
>
>
> eg
>
> http://stackoverflow.com/questions/970675/scala-modifying-nested-elements-in-xml
>
> http://stackoverflow.com/questions/1582244/how-to-access-parent-element-in-scala-xml
>

> [...]
>
> Ken
>

--
View this message in context: http://www.nabble.com/-scala--XML-hell-tp26071020p26077986.html

Sent from the Scala mailing list archive at Nabble.com.

[Bill Burdick-2]

Sorry -- I guess I didn't copy the list on this reply!

Bill

---------- Forwarded message ----------
From: Bill Burdick <bill.burdick@...>
Date: Tue, Oct 27, 2009 at 3:12 PM
Subject: Re: [scala] XML hell
To: Ken Faulkner <ken.faulkner@...>

OK then, here's a technique that preserves all of that whitespace, etc.

val xml = <bar>
<baz>

<foos>
<foo name = "1">foo1</foo><foo name = "2">foo2</foo><foo name = "3">

foo3</foo></foos></baz>
</bar>

xml match {
case <bar>{bar @ _*}</bar> => <bar>{bar map {
   case <baz>{baz @ _*}</baz> => <baz>{baz map {
       case <foos>{f @ _*}</foos> => <foos>{f ++ <foo name = "4"/>}</foos>
       case x => x
    }}</baz>
   case x => x
}}</bar>
case x => println("duh")
}


And here's a transforming visitor I wrote just for you!  It's easier to use.

def transform(n: Node)(block: PartialFunction[Node, Node]): Node = n match {

  case e: Elem => 

      val newChildren = n.child map {x => 

val filtered = transform(x)(block)

if (block.isDefinedAt(filtered)) block(filtered) else filtered}

      if ((n.child zip newChildren) forall (x => x._1 eq x._2)) e

      else Elem(n.prefix, n.label, n.attributes, n.scope, newChildren: _*)

  case x => if (block.isDefinedAt(x)) block(x) else x

}

This input:

transform(xml) {case <foos>{f @ _*}</foos> => <foos>{f ++ <foo name = "10"/>}</foos>}

produces this output:

<bar>

             <baz>

              

              <foos>

              <foo name="1">foo1</foo><foo name="2">foo2</foo><foo name="3">

              foo3</foo><foo name="10"></foo></foos></baz>

              </bar>

Bill

On Tue, Oct 27, 2009 at 12:45 PM, Ken Faulkner <ken.faulkner@...> wrote:
> Cheers,
> was similar to what I'd seen in other posts, but there are a few catches.
> 1) cant control white space. 
> 2) the real xml is more complex...   
> eg
> <bar>
> <cha>
>  ......
> </cha>
> <foos>
> <foo ....  > </foo>
> .
> .
> </foos>
> </bar>
> ie, there are other subsections within bar then need to be maintained... not
> sure if that messes up the matcher solution.
> Ken
>
> On Tue, Oct 27, 2009 at 9:18 PM, Bill Burdick <bill.burdick@...>
> wrote:
>>
>> You can use Scala's pattern matching, but it's a lot easier if you
>> eliminate the space.  This works in the version of Scala I'm using
>> right now (2.8.0.r19281-b20091026023411):
>>
>> val xml = <bar><foos><foo name = "1">foo1</foo><foo name =
>> "2">foo2</foo><foo name = "3">foo3</foo></foos></bar>
>>
>> xml match {case <bar><foos>{q @ _*}</foos></bar> => <bar><foos>{q ++
>> <foo name="4"/>}</foos></bar>}
>>
>> It returns a new XML tree with a new foo element (reusing the foo
>> elements from the original tree).
>>
>> If you're transforming documents where you can't control whitespace, I
>> think you need something closer to XSLT.  I'm not sure whether Scala
>> comes with something like that.
>>
>>
>> Bill
>>
>>
>> On Tue, Oct 27, 2009 at 4:04 AM, Ken Faulkner <ken.faulkner@...>
>> wrote:
>> > I'm hoping this is just me being dumb, but I'm desperate for help.
>> >
>> > Basically I want to perform some modification to some existing XML. So
>> > far
>> > everything I've come across seems fairly painful (or at least
>> > excessively
>> > long).
>> >
>> >
>> > eg
>> >
>> >
>> > http://stackoverflow.com/questions/970675/scala-modifying-nested-elements-in-xml
>> >
>> >
>> > http://stackoverflow.com/questions/1582244/how-to-access-parent-element-in-scala-xml
>> >
>> >
>> >
>> > I was very much hoping for something similar to (the brilliant!)
>> > elementTree
>> > API on Python...  and extract out lists/arrays of Nodes, manipulate and
>> > regenerate the XML from that.
>> > But it looks like everything boils down to matchers.
>> >
>> > What would be the suggested way to add an extra <foo> node to the below?
>> >
>> > <bar>
>> >
>> > <foos>
>> >
>> > <foo name = 1> foo1 </foo>
>> > <foo name = 2> foo2 </foo>
>> > <foo name = 3> foo3 </foo>
>> >
>> > </foos>
>> >
>> > </bar>
>> >
>> > ?
>> >
>> > Any comments highly appreciated
>> >
>> > Ken
>> >
>> >
>
>

[andreas s.]

Bill Burdick-2 wrote:

Sorry -- I guess I didn't copy the list on this reply!

Bill

---------- Forwarded message ----------
From: Bill Burdick <[e-mail]>
Date: Tue, Oct 27, 2009 at 3:12 PM
Subject: Re: [scala] XML hell
To: Ken Faulkner <[e-mail]>


OK then, here's a technique that preserves all of that whitespace, etc.

val xml = <bar>
<baz>


<foos>
<foo name = "1">foo1</foo><foo name = "2">foo2</foo><foo name = "3">
foo3</foo></foos></baz>
</bar>

xml match {
case <bar>{bar @ _*}</bar> => <bar>{bar map {
   case <baz>{baz @ _*}</baz> => <baz>{baz map {
       case <foos>{f @ _*}</foos> => <foos>{f ++ <foo name = "4"/>}</foos>
       case x => x
    }}</baz>
   case x => x
}}</bar>
case x => println("duh")
}


And here's a transforming visitor I wrote just for you!  It's easier to use.

def transform(n: Node)(block: PartialFunction[Node, Node]): Node = n match {
  case e: Elem =>
      val newChildren = n.child map {x =>
val filtered = transform(x)(block)
if (block.isDefinedAt(filtered)) block(filtered) else filtered}

      if ((n.child zip newChildren) forall (x => x._1 eq x._2)) e
      else Elem(n.prefix, n.label, n.attributes, n.scope, newChildren: _*)
  case x => if (block.isDefinedAt(x)) block(x) else x
}


This input:

transform(xml) {case <foos>{f @ _*}</foos> => <foos>{f ++ <foo name =
"10"/>}</foos>}


produces this output:

<bar>
             <baz>

              <foos>
              <foo name="1">foo1</foo><foo name="2">foo2</foo><foo name="3">
              foo3</foo><foo name="10"></foo></foos></baz>
              </bar>


Bill


On Tue, Oct 27, 2009 at 12:45 PM, Ken Faulkner <[e-mail]>
wrote:
> Cheers,
> was similar to what I'd seen in other posts, but there are a few catches.
> 1) cant control white space.
> 2) the real xml is more complex...
> eg
> <bar>
> <cha>
>  ......
> </cha>
> <foos>
> <foo ....  > </foo>
> .
> .
> </foos>
> </bar>
> ie, there are other subsections within bar then need to be maintained...
not
> sure if that messes up the matcher solution.
> Ken
>
> On Tue, Oct 27, 2009 at 9:18 PM, Bill Burdick <[e-mail]>
> wrote:
>>
>> You can use Scala's pattern matching, but it's a lot easier if you
>> eliminate the space.  This works in the version of Scala I'm using
>> right now (2.8.0.r19281-b20091026023411):
>>
>> val xml = <bar><foos><foo name = "1">foo1</foo><foo name =
>> "2">foo2</foo><foo name = "3">foo3</foo></foos></bar>
>>
>> xml match {case <bar><foos>{q @ _*}</foos></bar> => <bar><foos>{q ++
>> <foo name="4"/>}</foos></bar>}
>>
>> It returns a new XML tree with a new foo element (reusing the foo
>> elements from the original tree).
>>
>> If you're transforming documents where you can't control whitespace, I
>> think you need something closer to XSLT.  I'm not sure whether Scala
>> comes with something like that.
>>
>>
>> Bill
>>
>>
>> On Tue, Oct 27, 2009 at 4:04 AM, Ken Faulkner <[e-mail]>
>> wrote:
>> > I'm hoping this is just me being dumb, but I'm desperate for help.
>> >
>> > Basically I want to perform some modification to some existing XML. So
>> > far
>> > everything I've come across seems fairly painful (or at least
>> > excessively
>> > long).
>> >
>> >
>> > eg
>> >
>> >
>> >
http://stackoverflow.com/questions/970675/scala-modifying-nested-elements-in-xml
>> >
>> >
>> >
http://stackoverflow.com/questions/1582244/how-to-access-parent-element-in-scala-xml
>> >
>> >
>> >
>> > I was very much hoping for something similar to (the brilliant!)
>> > elementTree
>> > API on Python...  and extract out lists/arrays of Nodes, manipulate and
>> > regenerate the XML from that.
>> > But it looks like everything boils down to matchers.
>> >
>> > What would be the suggested way to add an extra <foo> node to the
below?
>> >
>> > <bar>
>> >
>> > <foos>
>> >
>> > <foo name = 1> foo1 </foo>
>> > <foo name = 2> foo2 </foo>
>> > <foo name = 3> foo3 </foo>
>> >
>> > </foos>
>> >
>> > </bar>
>> >
>> > ?
>> >
>> > Any comments highly appreciated
>> >
>> > Ken
>> >
>> >
>
>

Thanks for sharing this, it was interesting for me!

regards Andreas

[Dave Griffith]

While I will absolutely be keeping a pointer to this code and using it for my nefarious purposes, it strikes me as a piece of functionality that absolutely screams to be included in scala.xml.

--Dave Griffith

[Bill Burdick-2]

Aw shucks, thanks!

I feel I should point out that the transform method doesn't necessarily handle recursive XML trees well, where an element can have a descendant with the same label, since it doesn't pass a path along for context.  It could be changed to pass along an ancestor list, though.  Despite that, it's probably useful as-is for a good many cases and certainly helps with quick-and-dirty coding.

Bill

On Wed, Oct 28, 2009 at 10:43 PM, Dave Griffith <dave.l.griffith@...> wrote:

While I will absolutely be keeping a pointer to this code and using it for
my nefarious purposes, it strikes me as a piece of functionality that
absolutely screams to be included in scala.xml.

--Dave Griffith
--
View this message in context: http://www.nabble.com/-scala--XML-hell-tp26071020p26101727.html

Sent from the Scala mailing list archive at Nabble.com.

[Ross Judson]

I like the approach used by XAct.

http://www.brics.dk/Xact/

All XML trees are mutable, but there's an explicit "gap" node type. You can create gaps in XML trees via xpath, and fill them back in. I think it solves the hierarchy problem pretty nicely.

RJ

[Bill Burdick-2]

Rich Hickey raises some very good points about why immutability can be
very useful in concurrent programs in this talk:
http://www.infoq.com/presentations/Value-Identity-State-Rich-Hickey We
need immutability for these reasons in our project, so I like to find
immutable solutions for these things.  Mutable ones are fine too, but
it's important to have immutable solutions as well.

Bill


On Thu, Oct 29, 2009 at 2:55 AM, Ross Judson <rossjudson@...> wrote:

> I like the approach used by XAct.
>
> http://www.brics.dk/Xact/
>
> All XML trees are mutable, but there's an explicit "gap" node type. You can
> create gaps in XML trees via xpath, and fill them back in. I think it solves
> the hierarchy problem pretty nicely.
>
> RJ
>
>

[Ross Judson]

Oops. What I meant to say was, in Xact, all XML trees are IMMUTABLE. Sorry for the semantic inversion!

Here's a little more detail. The primary programming interface for Xact is a global XML class, through static methods.

Parsing XML is done with a parseXXX family of methods, including parseDocument(InputStream):

doc = XML.parseDocument(new FileInputStream("some.xml"))

Once parsed into immutable form, gap methods can be called to create a new immutable XML object with named "places" inside, based on an xpath selector:

gapped = doc.gapify("//first-name", "fname")

The opposite of the gap operation is a "plug" operation, which inserts values into named gaps. The insert operation can either replicate the same value across all of the gaps, or it can accept a list of items to be inserted, one for each gap named.

plugged = gapped.plug("fname", "Ross")

Many other operations on XML objects are possible. See http://www.brics.dk/Xact/devel/doc-public/dk/brics/xact/XML.html. Appends, inserts, deletions, and so forth are all available.

As operations are performed, Xact builds up a graph of operations against the XML. The operations are not executed until the XML must be realized into a string form:

String xmlResult = plugged.toDocument()

Xact also extends the XML syntax to include a template format, so gaps can be serialized out.

Xact is actually a language extension for Java that embeds XML processing directly into the language. The runtime system (immutable, functional, operation-based XML transformation) is separately usable.

I am not familiar with how XAct is implemented. Extending Scala's XML node classes with a "gap" class would be one way to provide a gap/plug feature.

RJ

On Wed, Oct 28, 2009 at 8:55 PM, Ross Judson <rossjudson@...> wrote:

I like the approach used by XAct.

http://www.brics.dk/Xact/

All XML trees are mutable, but there's an explicit "gap" node type. You can create gaps in XML trees via xpath, and fill them back in. I think it solves the hierarchy problem pretty nicely.

RJ

[Viktor Klang]

On Thu, Oct 29, 2009 at 6:28 PM, Ross Judson <rossjudson@...> wrote:

Oops. What I meant to say was, in Xact, all XML trees are IMMUTABLE. Sorry for the semantic inversion!

Here's a little more detail. The primary programming interface for Xact is a global XML class, through static methods.

Parsing XML is done with a parseXXX family of methods, including parseDocument(InputStream):

parseXXX, wtf?

 

doc = XML.parseDocument(new FileInputStream("some.xml"))

Once parsed into immutable form, gap methods can be called to create a new immutable XML object with named "places" inside, based on an xpath selector:

gapped = doc.gapify("//first-name", "fname")

The opposite of the gap operation is a "plug" operation, which inserts values into named gaps. The insert operation can either replicate the same value across all of the gaps, or it can accept a list of items to be inserted, one for each gap named.

plugged = gapped.plug("fname", "Ross")

Many other operations on XML objects are possible. See http://www.brics.dk/Xact/devel/doc-public/dk/brics/xact/XML.html. Appends, inserts, deletions, and so forth are all available.

As operations are performed, Xact builds up a graph of operations against the XML. The operations are not executed until the XML must be realized into a string form:

String xmlResult = plugged.toDocument()

Xact also extends the XML syntax to include a template format, so gaps can be serialized out.

Xact is actually a language extension for Java that embeds XML processing directly into the language. The runtime system (immutable, functional, operation-based XML transformation) is separately usable.

I am not familiar with how XAct is implemented. Extending Scala's XML node classes with a "gap" class would be one way to provide a gap/plug feature.

RJ

On Wed, Oct 28, 2009 at 8:55 PM, Ross Judson <rossjudson@...> wrote:

I like the approach used by XAct.

http://www.brics.dk/Xact/

All XML trees are mutable, but there's an explicit "gap" node type. You can create gaps in XML trees via xpath, and fill them back in. I think it solves the hierarchy problem pretty nicely.

RJ

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Blog: klangism.blogspot.com
Twttr: viktorklang
Code: github.com/viktorklang