Constant/reproducable order of elements in synapse.xml

[Hubert, Eric-2]

Hi devs,

Currently the implementation of the Synapse configuration model uses unordered collections to store the different configuration items (like proxy services etc.).

Thus serialization creates fixed order of main elements, but changing order of items within main elements. Diffing of XML-files created based on built-in serialization to track changes on this level is almost impossible.

Would somebody object to using sorted collection implementations instead?

Background: We are currently using a custom mechanism to create a synapse XML Configuration file. This mechanism actually guarantees a consistent order of the whole configuration file. Changes to the model can also be verified on the persisted model (synapse.xml). This turned out to be very handy.

For a couple of reasons I was now investigating whether it would make sense to rather just write a custom builder to create a synapse configuration from our specific configuration store and than use Synapse built-in support for serialization to XML:

XMLConfigurationSerializer.serializeConfiguration(synapseConfiguration,
   new FileOutputStream(synapseConfigFileName));

This way I should be able to get rid of a bunch of code while increasing flexibility. To make this work at least the issue reported yesterday as well as the ordering problem would need to be resolved.

Later on if there would be a way to configure/set the configuration builder to use (e.g. via a factory mechanism), one could have the possibility to run directly based on the external (versioned) config store without using the intermediate step of synapse.xml.

Any thoughts on this?


Regards,
  Eric

[indika kumara]

Hi Eric

My + 1 for using sorted collection implementations, if it enables to
serialize synapse XML in a consistent order.

I also prefer to encapsulate all extensible points throughout the code
base using a single consistent way ... may be factory mechanism.

Thanks
Indika

On Thu, Nov 5, 2009 at 1:44 PM, Hubert, Eric <Eric.Hubert@...> wrote:

> Hi devs,
>
> Currently the implementation of the Synapse configuration model uses unordered collections to store the different configuration items (like proxy services etc.).
>
> Thus serialization creates fixed order of main elements, but changing order of items within main elements. Diffing of XML-files created based on built-in serialization to track changes on this level is almost impossible.
>
> Would somebody object to using sorted collection implementations instead?
>
> Background: We are currently using a custom mechanism to create a synapse XML Configuration file. This mechanism actually guarantees a consistent order of the whole configuration file. Changes to the model can also be verified on the persisted model (synapse.xml). This turned out to be very handy.
>
> For a couple of reasons I was now investigating whether it would make sense to rather just write a custom builder to create a synapse configuration from our specific configuration store and than use Synapse built-in support for serialization to XML:
>
> XMLConfigurationSerializer.serializeConfiguration(synapseConfiguration,
>   new FileOutputStream(synapseConfigFileName));
>
> This way I should be able to get rid of a bunch of code while increasing flexibility. To make this work at least the issue reported yesterday as well as the ordering problem would need to be resolved.
>
> Later on if there would be a way to configure/set the configuration builder to use (e.g. via a factory mechanism), one could have the possibility to run directly based on the external (versioned) config store without using the intermediate step of synapse.xml.
>
> Any thoughts on this?
>
>
> Regards,
>  Eric
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Supun Kamburugamuva]

Hi Eric,

Synapse language is treated as a programming language at least for the out side world. For example sequences can be attributed to functions. In any programming language the order in which these symbols occur does not matter as long as the symbols can be found by the caller. So I have doubts in implementing an order for the synapse elements. By order I meant something like all the sequences are at the top then the proxy services etc.

When a user types in the synapse.xml, we should be able to preserve the order in which they have entered the elements in the synapse.xml. If you meant this one I'm +1 for implementing it.

Thanks,
Supun..

On Fri, Nov 6, 2009 at 2:32 PM, indika kumara <indika.kuma@...> wrote:

Hi Eric

My + 1 for using sorted collection implementations, if it enables to
serialize synapse XML in a consistent order.

I also prefer to encapsulate all extensible points throughout the code
base using a single consistent way ... may be factory mechanism.

Thanks
Indika

On Thu, Nov 5, 2009 at 1:44 PM, Hubert, Eric <Eric.Hubert@...> wrote:
> Hi devs,
>
> Currently the implementation of the Synapse configuration model uses unordered collections to store the different configuration items (like proxy services etc.).
>
> Thus serialization creates fixed order of main elements, but changing order of items within main elements. Diffing of XML-files created based on built-in serialization to track changes on this level is almost impossible.
>
> Would somebody object to using sorted collection implementations instead?
>
> Background: We are currently using a custom mechanism to create a synapse XML Configuration file. This mechanism actually guarantees a consistent order of the whole configuration file. Changes to the model can also be verified on the persisted model (synapse.xml). This turned out to be very handy.
>
> For a couple of reasons I was now investigating whether it would make sense to rather just write a custom builder to create a synapse configuration from our specific configuration store and than use Synapse built-in support for serialization to XML:
>
> XMLConfigurationSerializer.serializeConfiguration(synapseConfiguration,
>   new FileOutputStream(synapseConfigFileName));
>
> This way I should be able to get rid of a bunch of code while increasing flexibility. To make this work at least the issue reported yesterday as well as the ordering problem would need to be resolved.
>
> Later on if there would be a way to configure/set the configuration builder to use (e.g. via a factory mechanism), one could have the possibility to run directly based on the external (versioned) config store without using the intermediate step of synapse.xml.
>
> Any thoughts on this?
>
>
> Regards,
>  Eric
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com

[Hubert, Eric-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Supun,

 

Please see my comments inline!

Synapse language is treated as a programming language at least for the out side world. For example sequences can be attributed to functions. In any programming language the order in which these symbols occur does not matter as long as the symbols can be found by the caller.

Agreed.

So I have doubts in implementing an order for the synapse elements. By order I meant something like all the sequences are at the top then the proxy services etc.

I share these doubts, but you describe exactly the current implementation. Please have a look at: SynapseXMLConfigurationSerializer.serializeConfiguration(SynapseConfiguration synCfg). It uses a fixed order of top level elements.

Specific element serializers are then using an order depending on the way the configuration has been stored in memory. Mostly the XML information will be transferred to strongly typed data structures. While deserializing those values, currently a fixed order will be used. This might be different, from the one originally read in. Some list like structures are stored to unordered Map-Implementations for faster key-value access. While deserializing those values you end up with a random order. I think that this part could be fixed rather easily by using LinkedHashMap preserving the key order in which entries have been added to the map.

When a user types in the synapse.xml, we should be able to preserve the order in which they have entered the elements in the synapse.xml. If you meant this one I'm +1 for implementing it.
I also agree this would be the most desirable option, although the current implementation seems to be very far a way from that. Each factory/serializer pair would need to be designed for this purpose (e.g. keep a list or list of linked maps or any suitable structure of read elements/attributes, store it for deserialization purposes only and use this to retrieve the values in the exact order from structured object structures). I guess only something like this could warrant a working round-trip serialization/deserialization where the user may modify any part of the configuration, insert new stuff at any allowed place and this exact order will be preserved during later deserializations.

So my concrete suggestion for a start was just to replace all class member implementations of SynapseConfiguration which are currently of type HashMap with LinkedHashMap. From my point of view this is at least an improvement. No pain, big gain? ;-)

Regards,
    Eric

[Hiranya Jayathilaka-3]

Hi Eric,

On Thu, Nov 5, 2009 at 1:14 PM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi devs,

Currently the implementation of the Synapse configuration model uses unordered collections to store the different configuration items (like proxy services etc.).

Thus serialization creates fixed order of main elements, but changing order of items within main elements. Diffing of XML-files created based on built-in serialization to track changes on this level is almost impossible.

Would somebody object to using sorted collection implementations instead?

This sounds like a very useful feature. Most users would like to use a tool like diff to compare different versions of the Synapse configuration. Implementing this feature would hence improve the configuration management aspects of Synapse in a great deal.

So here's my +1 for the proposed changes. Please raise a JIRA issue for this as well.

Thanks,

Hiranya

 

Background: We are currently using a custom mechanism to create a synapse XML Configuration file. This mechanism actually guarantees a consistent order of the whole configuration file. Changes to the model can also be verified on the persisted model (synapse.xml). This turned out to be very handy.

For a couple of reasons I was now investigating whether it would make sense to rather just write a custom builder to create a synapse configuration from our specific configuration store and than use Synapse built-in support for serialization to XML:

XMLConfigurationSerializer.serializeConfiguration(synapseConfiguration,
  new FileOutputStream(synapseConfigFileName));

This way I should be able to get rid of a bunch of code while increasing flexibility. To make this work at least the issue reported yesterday as well as the ordering problem would need to be resolved.

Later on if there would be a way to configure/set the configuration builder to use (e.g. via a factory mechanism), one could have the possibility to run directly based on the external (versioned) config store without using the intermediate step of synapse.xml.

Any thoughts on this?


Regards,
 Eric

--
Hiranya Jayathilaka
Software Engineer;
WSO2 Inc.;  http://wso2.org
E-mail: hiranya@...;  Mobile: [phone]
Blog: http://techfeast-hiranya.blogspot.com

[Supun Kamburugamuva]

Hi Eric,

I thought about this a bit further. Synapse runs on an object model. Not on a XML configuration. As far as I understood this was the original goal. Also it makes sense. The synapse configuration can be created using XML or may be spring or may be pure programming. At the synapse object model layer I think it is logical to use a hash map. If a particular serialization and building wants to have an order it is the responsibility of the build and serialize method writer to impose it.

Isn't there a way to impose this at the serialize and builder level?

Thanks,
Supun..

On Sun, Nov 8, 2009 at 2:22 AM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Supun,

 

Please see my comments inline!

Synapse language is treated as a programming language at least for the out side world. For example sequences can be attributed to functions. In any programming language the order in which these symbols occur does not matter as long as the symbols can be found by the caller.

Agreed.

So I have doubts in implementing an order for the synapse elements. By order I meant something like all the sequences are at the top then the proxy services etc.

I share these doubts, but you describe exactly the current implementation. Please have a look at: SynapseXMLConfigurationSerializer.serializeConfiguration(SynapseConfiguration synCfg). It uses a fixed order of top level elements.

Specific element serializers are then using an order depending on the way the configuration has been stored in memory. Mostly the XML information will be transferred to strongly typed data structures. While deserializing those values, currently a fixed order will be used. This might be different, from the one originally read in. Some list like structures are stored to unordered Map-Implementations for faster key-value access. While deserializing those values you end up with a random order. I think that this part could be fixed rather easily by using LinkedHashMap preserving the key order in which entries have been added to the map.

When a user types in the synapse.xml, we should be able to preserve the order in which they have entered the elements in the synapse.xml. If you meant this one I'm +1 for implementing it.

I also agree this would be the most desirable option, although the current implementation seems to be very far a way from that. Each factory/serializer pair would need to be designed for this purpose (e.g. keep a list or list of linked maps or any suitable structure of read elements/attributes, store it for deserialization purposes only and use this to retrieve the values in the exact order from structured object structures). I guess only something like this could warrant a working round-trip serialization/deserialization where the user may modify any part of the configuration, insert new stuff at any allowed place and this exact order will be preserved during later deserializations.

So my concrete suggestion for a start was just to replace all class member implementations of SynapseConfiguration which are currently of type HashMap with LinkedHashMap. From my point of view this is at least an improvement. No pain, big gain? ;-)

Regards,
    Eric

--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com

[Hubert, Eric-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Supun,

 

If the object model does not preserve the order in which it has been created a serializer would need to store the order in some place independent from the configuration model itself which is also accessible by the deserializer at any later stage. Another implication is, that it is not enough to feed a deserializer just with the configuration model alone. It would need the additional meta data to be able to preserve the original order.

Also any serializer/factory-deserializer-pair would need to implement this.

 

Maybe others have already discussed about this at design time? Any input from the devs early involved?

 

Regards,

   Eric

---

From: Supun Kamburugamuva [mailto:supun06@...]
Sent: Monday, November 09, 2009 5:46 AM
To: dev@...
Subject: Re: Constant/reproducable order of elements in synapse.xml

 

Hi Eric,

I thought about this a bit further. Synapse runs on an object model. Not on a XML configuration. As far as I understood this was the original goal. Also it makes sense. The synapse configuration can be created using XML or may be spring or may be pure programming. At the synapse object model layer I think it is logical to use a hash map. If a particular serialization and building wants to have an order it is the responsibility of the build and serialize method writer to impose it.

Isn't there a way to impose this at the serialize and builder level?

Thanks,
Supun..

On Sun, Nov 8, 2009 at 2:22 AM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Supun,

 

Please see my comments inline!

Synapse language is treated as a programming language at least for the out side world. For example sequences can be attributed to functions. In any programming language the order in which these symbols occur does not matter as long as the symbols can be found by the caller.

Agreed.

So I have doubts in implementing an order for the synapse elements. By order I meant something like all the sequences are at the top then the proxy services etc.

I share these doubts, but you describe exactly the current implementation. Please have a look at: SynapseXMLConfigurationSerializer.serializeConfiguration(SynapseConfiguration synCfg). It uses a fixed order of top level elements.

Specific element serializers are then using an order depending on the way the configuration has been stored in memory. Mostly the XML information will be transferred to strongly typed data structures. While deserializing those values, currently a fixed order will be used. This might be different, from the one originally read in. Some list like structures are stored to unordered Map-Implementations for faster key-value access. While deserializing those values you end up with a random order. I think that this part could be fixed rather easily by using LinkedHashMap preserving the key order in which entries have been added to the map.

When a user types in the synapse.xml, we should be able to preserve the order in which they have entered the elements in the synapse.xml. If you meant this one I'm +1 for implementing it.

I also agree this would be the most desirable option, although the current implementation seems to be very far a way from that. Each factory/serializer pair would need to be designed for this purpose (e.g. keep a list or list of linked maps or any suitable structure of read elements/attributes, store it for deserialization purposes only and use this to retrieve the values in the exact order from structured object structures). I guess only something like this could warrant a working round-trip serialization/deserialization where the user may modify any part of the configuration, insert new stuff at any allowed place and this exact order will be preserved during later deserializations.

So my concrete suggestion for a start was just to replace all class member implementations of SynapseConfiguration which are currently of type HashMap with LinkedHashMap. From my point of view this is at least an improvement. No pain, big gain? ;-)

Regards,
    Eric

--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com

[Ruwan Linton]

Hi Eric,

I was on vacation and just got back. :-) Going through my email backlog.

So, First of all I do not understand the issue that you are facing with the order. I agree that synapse serializes according to a hard order while builders do not expect that order. There is no specific reason to have that hard order, it was just the way that the serialization has been implemented. Since XML doesn't care about the order in which the elements are present, the serialization just goes on the different elements.

Also if we are to preserve the order in which they were present in the synapse.xml that is used to build the configuration, making the object structure to be stored in LinkedHashMaps is not going to work since we keep separate HashMaps for the separate elements, for example proxy services are kept in its own HashMap while endpoints, sequences and local entries are on a different one.

If we are to do this change we need to rethink about the serilization process and the object store structure.

Thanks,
Ruwan
 

On Mon, Nov 9, 2009 at 11:04 AM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Supun,

 

If the object model does not preserve the order in which it has been created a serializer would need to store the order in some place independent from the configuration model itself which is also accessible by the deserializer at any later stage. Another implication is, that it is not enough to feed a deserializer just with the configuration model alone. It would need the additional meta data to be able to preserve the original order.

Also any serializer/factory-deserializer-pair would need to implement this.

 

Maybe others have already discussed about this at design time? Any input from the devs early involved?

 

Regards,

   Eric

---

From: Supun Kamburugamuva [mailto:supun06@...]
Sent: Monday, November 09, 2009 5:46 AM
To: dev@...
Subject: Re: Constant/reproducable order of elements in synapse.xml

 

Hi Eric,

I thought about this a bit further. Synapse runs on an object model. Not on a XML configuration. As far as I understood this was the original goal. Also it makes sense. The synapse configuration can be created using XML or may be spring or may be pure programming. At the synapse object model layer I think it is logical to use a hash map. If a particular serialization and building wants to have an order it is the responsibility of the build and serialize method writer to impose it.

Isn't there a way to impose this at the serialize and builder level?

Thanks,
Supun..

On Sun, Nov 8, 2009 at 2:22 AM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Supun,

 

Please see my comments inline!

Synapse language is treated as a programming language at least for the out side world. For example sequences can be attributed to functions. In any programming language the order in which these symbols occur does not matter as long as the symbols can be found by the caller.

Agreed.

So I have doubts in implementing an order for the synapse elements. By order I meant something like all the sequences are at the top then the proxy services etc.

I share these doubts, but you describe exactly the current implementation. Please have a look at: SynapseXMLConfigurationSerializer.serializeConfiguration(SynapseConfiguration synCfg). It uses a fixed order of top level elements.

Specific element serializers are then using an order depending on the way the configuration has been stored in memory. Mostly the XML information will be transferred to strongly typed data structures. While deserializing those values, currently a fixed order will be used. This might be different, from the one originally read in. Some list like structures are stored to unordered Map-Implementations for faster key-value access. While deserializing those values you end up with a random order. I think that this part could be fixed rather easily by using LinkedHashMap preserving the key order in which entries have been added to the map.

When a user types in the synapse.xml, we should be able to preserve the order in which they have entered the elements in the synapse.xml. If you meant this one I'm +1 for implementing it.

I also agree this would be the most desirable option, although the current implementation seems to be very far a way from that. Each factory/serializer pair would need to be designed for this purpose (e.g. keep a list or list of linked maps or any suitable structure of read elements/attributes, store it for deserialization purposes only and use this to retrieve the values in the exact order from structured object structures). I guess only something like this could warrant a working round-trip serialization/deserialization where the user may modify any part of the configuration, insert new stuff at any allowed place and this exact order will be preserved during later deserializations.

So my concrete suggestion for a start was just to replace all class member implementations of SynapseConfiguration which are currently of type HashMap with LinkedHashMap. From my point of view this is at least an improvement. No pain, big gain? ;-)

Regards,
    Eric

--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

[Supun Kamburugamuva]

Hi Eric,

Just an idea. Since Synapse XML is another XML, you should be able to use C14N and compare the two Synapse.xml files? Is it possibble?

Thanks,
Supun..

On Tue, Nov 10, 2009 at 3:50 AM, Ruwan Linton <ruwan.linton@...> wrote:

Hi Eric,

I was on vacation and just got back. :-) Going through my email backlog.

So, First of all I do not understand the issue that you are facing with the order. I agree that synapse serializes according to a hard order while builders do not expect that order. There is no specific reason to have that hard order, it was just the way that the serialization has been implemented. Since XML doesn't care about the order in which the elements are present, the serialization just goes on the different elements.

Also if we are to preserve the order in which they were present in the synapse.xml that is used to build the configuration, making the object structure to be stored in LinkedHashMaps is not going to work since we keep separate HashMaps for the separate elements, for example proxy services are kept in its own HashMap while endpoints, sequences and local entries are on a different one.

If we are to do this change we need to rethink about the serilization process and the object store structure.

Thanks,
Ruwan

 

On Mon, Nov 9, 2009 at 11:04 AM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Supun,

 

If the object model does not preserve the order in which it has been created a serializer would need to store the order in some place independent from the configuration model itself which is also accessible by the deserializer at any later stage. Another implication is, that it is not enough to feed a deserializer just with the configuration model alone. It would need the additional meta data to be able to preserve the original order.

Also any serializer/factory-deserializer-pair would need to implement this.

 

Maybe others have already discussed about this at design time? Any input from the devs early involved?

 

Regards,

   Eric

---

From: Supun Kamburugamuva [mailto:supun06@...]
Sent: Monday, November 09, 2009 5:46 AM
To: dev@...
Subject: Re: Constant/reproducable order of elements in synapse.xml

 

Hi Eric,

I thought about this a bit further. Synapse runs on an object model. Not on a XML configuration. As far as I understood this was the original goal. Also it makes sense. The synapse configuration can be created using XML or may be spring or may be pure programming. At the synapse object model layer I think it is logical to use a hash map. If a particular serialization and building wants to have an order it is the responsibility of the build and serialize method writer to impose it.

Isn't there a way to impose this at the serialize and builder level?

Thanks,
Supun..

On Sun, Nov 8, 2009 at 2:22 AM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Supun,

 

Please see my comments inline!

Synapse language is treated as a programming language at least for the out side world. For example sequences can be attributed to functions. In any programming language the order in which these symbols occur does not matter as long as the symbols can be found by the caller.

Agreed.

So I have doubts in implementing an order for the synapse elements. By order I meant something like all the sequences are at the top then the proxy services etc.

I share these doubts, but you describe exactly the current implementation. Please have a look at: SynapseXMLConfigurationSerializer.serializeConfiguration(SynapseConfiguration synCfg). It uses a fixed order of top level elements.

Specific element serializers are then using an order depending on the way the configuration has been stored in memory. Mostly the XML information will be transferred to strongly typed data structures. While deserializing those values, currently a fixed order will be used. This might be different, from the one originally read in. Some list like structures are stored to unordered Map-Implementations for faster key-value access. While deserializing those values you end up with a random order. I think that this part could be fixed rather easily by using LinkedHashMap preserving the key order in which entries have been added to the map.

When a user types in the synapse.xml, we should be able to preserve the order in which they have entered the elements in the synapse.xml. If you meant this one I'm +1 for implementing it.

I also agree this would be the most desirable option, although the current implementation seems to be very far a way from that. Each factory/serializer pair would need to be designed for this purpose (e.g. keep a list or list of linked maps or any suitable structure of read elements/attributes, store it for deserialization purposes only and use this to retrieve the values in the exact order from structured object structures). I guess only something like this could warrant a working round-trip serialization/deserialization where the user may modify any part of the configuration, insert new stuff at any allowed place and this exact order will be preserved during later deserializations.

So my concrete suggestion for a start was just to replace all class member implementations of SynapseConfiguration which are currently of type HashMap with LinkedHashMap. From my point of view this is at least an improvement. No pain, big gain? ;-)

Regards,
    Eric

--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb

WSO2 Inc.; http://wso2.org

email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com

[Hubert, Eric-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Ruwan,

 

thanks for your response. As described earlier it is perfectly clear to me that the usage of LinkedHashMaps would not solve the „problem“ entirely as the order of top level elements might still change. On the other hand the more annoying problem is the order of the elements below the top level elements, e.g. the list of hundreds of proxies, sequences and so on in random order.

It was just a thought that changing the implementation to LinkedHashMaps would not hurt and at least address this particular issue. I tested it out and it is really an improvement.

 

Rethinking about the serialization process and the object store structure might also be a good thought. Enhancing on this part could also ease the process of writing custom mediators. Writing factories/serializer pairs gives a lot of control but also adds a lot of boilerplate-code and forces users to get in touch with rather low level XML-API (Axiom).

 

Regards,

   Eric

 

---

From: Ruwan Linton [mailto:ruwan.linton@...]
Sent: Monday, November 09, 2009 11:20 PM
To: dev@...
Subject: Re: Constant/reproducable order of elements in synapse.xml

 

Hi Eric,

I was on vacation and just got back. :-) Going through my email backlog.

So, First of all I do not understand the issue that you are facing with the order. I agree that synapse serializes according to a hard order while builders do not expect that order. There is no specific reason to have that hard order, it was just the way that the serialization has been implemented. Since XML doesn't care about the order in which the elements are present, the serialization just goes on the different elements.

Also if we are to preserve the order in which they were present in the synapse.xml that is used to build the configuration, making the object structure to be stored in LinkedHashMaps is not going to work since we keep separate HashMaps for the separate elements, for example proxy services are kept in its own HashMap while endpoints, sequences and local entries are on a different one.

If we are to do this change we need to rethink about the serilization process and the object store structure.

Thanks,
Ruwan
 

On Mon, Nov 9, 2009 at 11:04 AM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Supun,

 

If the object model does not preserve the order in which it has been created a serializer would need to store the order in some place independent from the configuration model itself which is also accessible by the deserializer at any later stage. Another implication is, that it is not enough to feed a deserializer just with the configuration model alone. It would need the additional meta data to be able to preserve the original order.

Also any serializer/factory-deserializer-pair would need to implement this.

 

Maybe others have already discussed about this at design time? Any input from the devs early involved?

 

Regards,

   Eric

---

From: Supun Kamburugamuva [mailto:supun06@...]
Sent: Monday, November 09, 2009 5:46 AM
To: dev@...
Subject: Re: Constant/reproducable order of elements in synapse.xml

 

Hi Eric,

I thought about this a bit further. Synapse runs on an object model. Not on a XML configuration. As far as I understood this was the original goal. Also it makes sense. The synapse configuration can be created using XML or may be spring or may be pure programming. At the synapse object model layer I think it is logical to use a hash map. If a particular serialization and building wants to have an order it is the responsibility of the build and serialize method writer to impose it.

Isn't there a way to impose this at the serialize and builder level?

Thanks,
Supun..

On Sun, Nov 8, 2009 at 2:22 AM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Supun,

 

Please see my comments inline!

Synapse language is treated as a programming language at least for the out side world. For example sequences can be attributed to functions. In any programming language the order in which these symbols occur does not matter as long as the symbols can be found by the caller.

Agreed.

So I have doubts in implementing an order for the synapse elements. By order I meant something like all the sequences are at the top then the proxy services etc.

I share these doubts, but you describe exactly the current implementation. Please have a look at: SynapseXMLConfigurationSerializer.serializeConfiguration(SynapseConfiguration synCfg). It uses a fixed order of top level elements.

Specific element serializers are then using an order depending on the way the configuration has been stored in memory. Mostly the XML information will be transferred to strongly typed data structures. While deserializing those values, currently a fixed order will be used. This might be different, from the one originally read in. Some list like structures are stored to unordered Map-Implementations for faster key-value access. While deserializing those values you end up with a random order. I think that this part could be fixed rather easily by using LinkedHashMap preserving the key order in which entries have been added to the map.

When a user types in the synapse.xml, we should be able to preserve the order in which they have entered the elements in the synapse.xml. If you meant this one I'm +1 for implementing it.

I also agree this would be the most desirable option, although the current implementation seems to be very far a way from that. Each factory/serializer pair would need to be designed for this purpose (e.g. keep a list or list of linked maps or any suitable structure of read elements/attributes, store it for deserialization purposes only and use this to retrieve the values in the exact order from structured object structures). I guess only something like this could warrant a working round-trip serialization/deserialization where the user may modify any part of the configuration, insert new stuff at any allowed place and this exact order will be preserved during later deserializations.

So my concrete suggestion for a start was just to replace all class member implementations of SynapseConfiguration which are currently of type HashMap with LinkedHashMap. From my point of view this is at least an improvement. No pain, big gain? ;-)

Regards,
    Eric

--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

[Ruwan Linton]

On Tue, Nov 10, 2009 at 2:56 PM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Ruwan,

 

thanks for your response. As described earlier it is perfectly clear to me that the usage of LinkedHashMaps would not solve the „problem“ entirely as the order of top level elements might still change. On the other hand the more annoying problem is the order of the elements below the top level elements, e.g. the list of hundreds of proxies, sequences and so on in random order.

It was just a thought that changing the implementation to LinkedHashMaps would not hurt and at least address this particular issue. I tested it out and it is really an improvement.

I agree...

 

Rethinking about the serialization process and the object store structure might also be a good thought.

Yeah, I will investigate into this and propose a solution to fix the entire problem.

Enhancing on this part could also ease the process of writing custom mediators. Writing factories/serializer pairs gives a lot of control but also adds a lot of boilerplate-code and forces users to get in touch with rather low level XML-API (Axiom).

Well, if you do not want to write your own factory and serializer (enabling you to have a custom configuration for the mediator) you can use the class mediator wrapper and just write the mediator and use the properties to set the field values of the class mediator.

Thanks,
Ruwan

 

Regards,

   Eric

 

---

From: Ruwan Linton [mailto:ruwan.linton@...]
Sent: Monday, November 09, 2009 11:20 PM

To: dev@...
Subject: Re: Constant/reproducable order of elements in synapse.xml

 

Hi Eric,

I was on vacation and just got back. :-) Going through my email backlog.

So, First of all I do not understand the issue that you are facing with the order. I agree that synapse serializes according to a hard order while builders do not expect that order. There is no specific reason to have that hard order, it was just the way that the serialization has been implemented. Since XML doesn't care about the order in which the elements are present, the serialization just goes on the different elements.

Also if we are to preserve the order in which they were present in the synapse.xml that is used to build the configuration, making the object structure to be stored in LinkedHashMaps is not going to work since we keep separate HashMaps for the separate elements, for example proxy services are kept in its own HashMap while endpoints, sequences and local entries are on a different one.

If we are to do this change we need to rethink about the serilization process and the object store structure.

Thanks,
Ruwan
 

On Mon, Nov 9, 2009 at 11:04 AM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Supun,

 

If the object model does not preserve the order in which it has been created a serializer would need to store the order in some place independent from the configuration model itself which is also accessible by the deserializer at any later stage. Another implication is, that it is not enough to feed a deserializer just with the configuration model alone. It would need the additional meta data to be able to preserve the original order.

Also any serializer/factory-deserializer-pair would need to implement this.

 

Maybe others have already discussed about this at design time? Any input from the devs early involved?

 

Regards,

   Eric

---

From: Supun Kamburugamuva [mailto:supun06@...]
Sent: Monday, November 09, 2009 5:46 AM
To: dev@...
Subject: Re: Constant/reproducable order of elements in synapse.xml

 

Hi Eric,

I thought about this a bit further. Synapse runs on an object model. Not on a XML configuration. As far as I understood this was the original goal. Also it makes sense. The synapse configuration can be created using XML or may be spring or may be pure programming. At the synapse object model layer I think it is logical to use a hash map. If a particular serialization and building wants to have an order it is the responsibility of the build and serialize method writer to impose it.

Isn't there a way to impose this at the serialize and builder level?

Thanks,
Supun..

On Sun, Nov 8, 2009 at 2:22 AM, Hubert, Eric <Eric.Hubert@...> wrote:

Hi Supun,

 

Please see my comments inline!

Synapse language is treated as a programming language at least for the out side world. For example sequences can be attributed to functions. In any programming language the order in which these symbols occur does not matter as long as the symbols can be found by the caller.

Agreed.

So I have doubts in implementing an order for the synapse elements. By order I meant something like all the sequences are at the top then the proxy services etc.

I share these doubts, but you describe exactly the current implementation. Please have a look at: SynapseXMLConfigurationSerializer.serializeConfiguration(SynapseConfiguration synCfg). It uses a fixed order of top level elements.

Specific element serializers are then using an order depending on the way the configuration has been stored in memory. Mostly the XML information will be transferred to strongly typed data structures. While deserializing those values, currently a fixed order will be used. This might be different, from the one originally read in. Some list like structures are stored to unordered Map-Implementations for faster key-value access. While deserializing those values you end up with a random order. I think that this part could be fixed rather easily by using LinkedHashMap preserving the key order in which entries have been added to the map.

When a user types in the synapse.xml, we should be able to preserve the order in which they have entered the elements in the synapse.xml. If you meant this one I'm +1 for implementing it.

I also agree this would be the most desirable option, although the current implementation seems to be very far a way from that. Each factory/serializer pair would need to be designed for this purpose (e.g. keep a list or list of linked maps or any suitable structure of read elements/attributes, store it for deserialization purposes only and use this to retrieve the values in the exact order from structured object structures). I guess only something like this could warrant a working round-trip serialization/deserialization where the user may modify any part of the configuration, insert new stuff at any allowed place and this exact order will be preserved during later deserializations.

So my concrete suggestion for a start was just to replace all class member implementations of SynapseConfiguration which are currently of type HashMap with LinkedHashMap. From my point of view this is at least an improvement. No pain, big gain? ;-)

Regards,
    Eric

--
Software Engineer, WSO2 Inc
http://wso2.org
supunk.blogspot.com

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@...; cell: [phone]
blog: http://ruwansblog.blogspot.com

[Hubert, Eric-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Rethinking about the serialization process and the object store structure might also be a good thought.

Yeah, I will investigate into this and propose a solution to fix the entire problem.

 

Great!

Enhancing on this part could also ease the process of writing custom mediators. Writing factories/serializer pairs gives a lot of control but also adds a lot of boilerplate-code and forces users to get in touch with rather low level XML-API (Axiom).

Well, if you do not want to write your own factory and serializer (enabling you to have a custom configuration for the mediator) you can use the class mediator wrapper and just write the mediator and use the properties to set the field values of the class mediator.

True, but I was really thinking of using a custom configuration stored in XML (part of synapse.xml) but having just an abstraction for retrieving and storing. I must admit, that I have not yet thought about concrete implementations. Direction of thought was XSD + XML/Java-Object Mapping, something like JAXB 2.x…

 

Regards,

   Eric