a proposal for slice stitching

[Aaron Falk]

This is a note motivated by topics raised at the GEC6 Control Framework WG meeting.  Comments, criticism, feedback, and corrections are welcome. 

The issue of slice stitching has come up periodically and in the interest of making some progress on it, I wanted to propose a mechanism for stitching together aggregates using VLANs.  What is slice stitching?  Slice stitching refers to the process of interconnecting slivers between is different GENI aggregates.  In the near term, GENI needs to be able to create Ethernet VLANs that connect aggregates  (although over the longer term more diverse interconnections will be desired).

Jeff Chase, in his slides at the GEC6 CFWG meeting [1], catalogs  a very good list of questions on stitching:

•  How to join slivers/slices across different aggregates end-to-end?
•  Do we require common labels at junction points?
•  How to connect slivers?
•  Do aggregates negotiate with each other? (peer-to-peer) or a clearinghouse or service such as a slice manager coordinate? (top-down)
•  What about isolation for performance or security?

The mechanism I propose below address these questions for the narrow application of establishing end-to-end Ethernet VLANs.  Rather than try to solve the general problem, my goal is to establish a straightforward way to do stitching so that GENI aggregates and tools under development will understand what functions are required.

Let me illustrate by an example.

               +--------------------------------+
    +---------->|  Stitching Manager Service (S) |<----------+
    |           +--------------------------------+           |
    |                            |                           |
    V                            V                           V
 +--------------+        +--------------+        +--------------+
 ||AM|          |        |      |AM|    |        |          |AM||
 |+--+       +--|        |--+   +--+ +--|        |--+       +--+|
 |           |  |........|  |        |  |........|  |           |
 |           |SW|........|SW|        |SW|........|SW|           |
 |           |  |........|  |        |  |........|  |           |
 |           +--| usable |--+        +--| usable |--+           |
 |ProtoGENI (PG)| VLANs  |Internet2 (I2)| VLANs  |OpenFlow (OF) |
 +--------------+        +--------------+        +--------------+

1. A ProtoGENI cluster (PG), Internet2 (I2) and an campus OpenFlow network containing several hosts (OF) are configured to function as GENI aggregates. 
   
   
2. The ProtoGENI cluster administrator provisions connectivity to an Internet2 PoP through a regional network.  I2, PG, and the regional network administrators engineer the network, provisioning a set of VLANs for GENI use PG site and the I2 PoP.  The regional network can be thought of as a 'wire'.  The engineering of the network is worked out between the participants and is a 'local' matter.  The result is a set of VLANs known to the PG and I2 admins and pre-allocated for GENI use.
   
   
3. A similar process occurs between Internet2 and the campus OpenFlow network. 
   
   
4. A researcher now wishes to create a slice containing resources from PG and OF using I2 to provide network connectivity between them.  The researcher has acquired slice credentials allocated by a slice authority recognized by all three aggregates. 
   
   
5. The researcher (via  the GENI Aggregate Manger API) requests a sliver containing hosts connected by a topology on the ProtoGENI cluster.  The AM allocates the topology and hosts but does not yet connect them to the outside world. 
   
   
6. The above step is applied to the campus OpenFlow network.
   
   
7. The researcher now requests an I2 sliver providing Ethernet connectivity between the ProtoGENI cluster to the OpenFlow network.  The I2 AM allocates the topology but does not yet connect it to the outside world. At this point, three disconnected slivers have been established.
   
   
8. The researcher now provides his slice credentials to a stitching manager service, S, with two requests: stitch his PG and I2 slivers and stitch his I2 and OF slivers.  S, using a pre-established rule, determines the sort order for stitching is PG, OF, I2, meaning that for the PG-I2 VLAN, PG is contacted first and for the I2-OF VLAN, OF is contacted first.
   
   
9. S contacts the ProtoGENI AM, forwarding the slice credentials and the request to connect the sliver to Internet2.  The PG AM, using local policy determined by the ProtoGENI administrator, assigns a VLAN connecting the ProtoGENI cluster to Internet2 to this slice.  The PG-I2 VLAN identifying information is provided to S.  Even though the mapping has been determined, the PG switch is configured to drop traffic on the allocated VLANs until there is confirmation that the all stitching required by the slice is complete.  This is to avoid the possibility of traffic injected into a partially configured network.
   
   
10. S now contacts the I2 AM providing the slice credentials and the PG-I2 VLAN identifying information. The I2 AM prepares the mapping between the I2 internal network and the PG-I2 VLAN.  However, as within PG, the I2 switch is configured to drop traffic on the allocated VLANs until there is confirmation that the all stitching required by the slice is complete.
    
    
11. The previous two steps are repeated with OF and I2, starting with OF (as stated in step 8).  At this point S, knows the identifying information for all the stitching VLANs assigned to this slice.  This information is stored for operations and forensic use.  S also has confirmation that the stitching has been completed. 
    
    
12. S sends an indication to PG, OF, and I2 that the end-to-end network is configured.  Now the rules to drop traffic on the assigned VLANs are removed and each switch is configured to translate VLAN traffic between the assigned stitching VLAN and the internal network.  Each network sends a confirmation back to S.
    
    
13. S tells the researcher the end-to-end network is in place.
    

Some of the assumptions here:

• We assume both switches must be able to do VLAN translation.  There are other techniques for stitching together VLANs but requiring translation at every switch will add minimal constraints on how to stitching might be established.  In other words, VLANs available for inter-aggregate connectivity won't be constrained by VLAN IDs in use within either aggregate.  VLAN translation won't be uniformly available throughout GENI for several years and the things will be more complex in the intervening period.
  
• VLANs are assumed to be pre-established before the stitching process is started.  Inter-aggregate VLANs will have some isolation and performance characteristics assigned to them when created, i.e., there may be some performance guarantees that can be made for some VLANs but this model isn't intended to support on-demand per-slice QoS negotiation on the stitching VLANs. 
  
• The policy by which an internal VLAN is mapped to an stitching VLAN is entirely local and under the control of the aggregate.
  
• We assume all networks can be represented as either an aggregate or a static VLAN (a 'wire').  I believe the implication here is that the backbones behave like aggregates. 
  
• We assume that S picks the first aggregate in an unambiguous and repeatable way (e.g., in some sort order) to avoid race conditions where both A and B give out the same VLAN to different users.
• We assume that the aggregate manager can configure the switch to connect the assigned VLAN to the network resources allocated to the slice.
• Once a VLAN has been assigned to a slice for stitching, it has to be reported and recorded by the GENI clearinghouse for operations and forensic use.  Therefore, the VLAN identifying information needs to be in a standard form.
  

Questions I have:

• The AM is required since the binding is between resources allocated to a slice and the VLAN.  Will an extension to the aggregate API be required to support the protocol above?  Perhaps not: this sort of looks like a 'revise an existing slice' operation.
• It seems like the 'usable VLANs' could be multiplexed over a 802.11QinQ,  GRE, OpenVPN, or (G)MPLS tunnels. E.g., if QinQ or even an IP tunnel is used, the tunnel should be established but VLAN IDs should still be passed to permit demultiplexing at the edges.  What are the implications? 
  

Let me conclude by saying I'm sending this to the Control Framework and Experimenter Workflow and Services working groups because I believe there are implications for both groups embedded in this proposal.  This is an important experimenter service that is not part of the control plane (and thus in scope for the services-wg.  Additionally, aggregate managers and RSpecs would need to support this protocol, making it relevant to the control-wg.  I think if we can get a rough agreement that something like this would work, I'd like to hear what would be needed to prototype it.

regards,

--aaron

[1] http://groups.geni.net/geni/attachment/wiki/GEC6CFWGAgenda/gec6-cf-chase.ppt



_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Robert P Ricci]

Thanks for getting this discussion kicked off!

The model we have been pursuing is a little different than the one you
propose below. A rough sketch (in less detail than your proposal),
picking up where we diverge:

5. The user constructs an RSpec (likely with help from some user tools)
   that describes the whole slice they want to build, including the PG
   and OF nodes and link(s) between them
6. If the user *knows* the path they want to use on I2, they fill that in
   - if they don't know or don't care, they pass their request RSpec to a
   slice embedding service (SES), which fills out these details for them.
   Note that the SES isn't actually setting anything up on the users'
   behalf, it is just *annotating* the request RSpec to fill in details
   for the user
7. The user submits the RSpec to the appropriate AMs (in this case PG,
   OF, and I2) - it's trivial to extract the right set of AMs from the
   RSpec if the SES added any (for example, if it added nodes in the I2
   aggregate)
8. The AMs communicate in pairwise fashion to agree on things like
   VLAN#s for the interconnection point. They can do this because the
   request RSpec that each AM sees includes 'external references' to the
   other AMs in the topology.
9. The AMs, using the negotiated VLAN#s, etc., individually create their
   slivers and inform the user when they're ready.

We've implemented up to step 7. While 'pairwise fashion' in step 8
sounds like it would scale poorly, consider the fact that the number of
places where there are physical interconnects between various aggregates
are likely to be limited in number and fairly static, bounding the
number of pairs that have to agree. And, in fact, in the medium term,
what's likely to happen is that there are a couple of backbone AMs, with
aggregates hanging off of them. This results in a very simple, easy to
manage structure: backbones act as 'masters' for selecting VLAN#s for
their attached aggregates, the only place where real negotiation has to
occur is at connection points between backbones.

Note that this pretty much follows what Max suggested.

It is probably possible to implement a service that drives this process
for the user, to hide some of the details from them, and that service is
probably pretty straightforward.

Thus spake Aaron Falk on Mon, Feb 01, 2010 at 09:19:42AM -0500:

>    5. The researcher (via  the GENI Aggregate Manger API) requests a
>       sliver containing hosts connected by a topology on the ProtoGENI
>       cluster.  The AM allocates the topology and hosts but does not yet
>       connect them to the outside world.
>
>    6. The above step is applied to the campus OpenFlow network.
>
>    7. The researcher now requests an I2 sliver providing Ethernet
>       connectivity between the ProtoGENI cluster to the OpenFlow
>       network.  The I2 AM allocates the topology but does not yet
>       connect it to the outside world. At this point, three disconnected
>       slivers have been established.
>
>    8. The researcher now provides his slice credentials to a stitching
>       manager service, S, with two requests: stitch his PG and I2
>       slivers and stitch his I2 and OF slivers.  S, using a
>       pre-established rule, determines the sort order for stitching is
>       PG, OF, I2, meaning that for the PG-I2 VLAN, PG is contacted first
>       and for the I2-OF VLAN, OF is contacted first.
>
>    9. S contacts the ProtoGENI AM, forwarding the slice credentials and
>       the request to connect the sliver to Internet2.  The PG AM, using
>       local policy determined by the ProtoGENI administrator, assigns a
>       VLAN connecting the ProtoGENI cluster to Internet2 to this slice.
>       The PG-I2 VLAN identifying information is provided to S.  Even
>       though the mapping has been determined, the PG switch is
>       configured to drop traffic on the allocated VLANs until there is
>       confirmation that the all stitching required by the slice is
>       complete.  This is to avoid the possibility of traffic injected
>       into a partially configured network.
>
>   10. S now contacts the I2 AM providing the slice credentials and the
>       PG-I2 VLAN identifying information. The I2 AM prepares the mapping
>       between the I2 internal network and the PG-I2 VLAN.  However, as
>       within PG, the I2 switch is configured to drop traffic on the
>       allocated VLANs until there is confirmation that the all stitching
>       required by the slice is complete.
>
>   11. The previous two steps are repeated with OF and I2, starting with
>       OF (as stated in step 8).  At this point S, knows the identifying
>       information for all the stitching VLANs assigned to this slice.
>       This information is stored for operations and forensic use.  S
>       also has confirmation that the stitching has been completed.
>
>   12. S sends an indication to PG, OF, and I2 that the end-to-end
>       network is configured.  Now the rules to drop traffic on the
>       assigned VLANs are removed and each switch is configured to
>       translate VLAN traffic between the assigned stitching VLAN and the
>       internal network.  Each network sends a confirmation back to S.
>
>   13. S tells the researcher the end-to-end network is in place.

_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Aaron Falk]

Hi Max-

I don't think that most aggregates will 'directly touch'.  The common
case will be that an aggregate will be in some university lab or other
PoP and will have to traverse campus and regional networks to get to an
'adjacent' aggregate.  I think that this connectivity will be fairly
static since campus IT and regional network providers are unlikely to be
prepared (in general) to support dynamic assignment of VLANs.

The task at hand, then, is to make sure the aggregates on either end of
this static set of VLANs assign them to the same slice.  There are two
obvious approaches: peer to peer negotiation or external coordination.
I propose external coordination (while you suggest the other below)
mostly because I believe the resulting protocol will be simpler.
Additionally, the external coordinator will have a end-to-end view of
when the stitched network is complete.  Allowing slivers to send traffic
into a partially stitched network seems likely to disrupt other experiments.

--aaron

On 2/4/10 10:43 AM, Max Ott wrote:

> Hi Aaron,
>
> I'm not sure I fully understand all the details and nuances most likely driven by current constraints, but I would really like to better understand the need of a separate stitching service.
>
> If, as you suggested the interconnecting network is an aggregate in its own right, then we should have 'touch points' between distinct aggregates, or in this context, links connecting the two. The endpoints of the links can be treated as normal resources in the respective aggregates with potentially configurable attributes (direction, bandwidth, ...) and properties (e.g. identifying the peer end point and its respective resource identifier in the other aggregate). Each aggregate only deals with the respective endpoints in the context of an experimenter's slice and the link/connectivity is essentially an emerging property. Now behind the scene the two touching aggregates need to do some additional coordination as the properties of the endpoints are in most cases constraint on each other (same VLAN, bandwidth, direction, ...) and to enforce the necessary access control to make sure that traffic stays within a slice or crosses between the right slices.
>
> Is there anything I'm missing?
>
> Cheers,
>
> -max
>
>
>
> On 02/02/2010, at 1:19 AM, Aaron Falk wrote:
>
>  
>> This is a note motivated by topics raised at the GEC6 Control Framework WG meeting.  Comments, criticism, feedback, and corrections are welcome.  
>>
>> The issue of slice stitching has come up periodically and in the interest of making some progress on it, I wanted to propose a mechanism for stitching together aggregates using VLANs.  What is slice stitching?  Slice stitching refers to the process of interconnecting slivers between is different GENI aggregates.  In the near term, GENI needs to be able to create Ethernet VLANs that connect aggregates  (although over the longer term more diverse interconnections will be desired).
>>
>> Jeff Chase, in his slides at the GEC6 CFWG meeting [1], catalogs  a very good list of questions on stitching:
>> •  How to join slivers/slices across different aggregates end-to-end?
>> •  Do we require common labels at junction points?
>> •  How to connect slivers?
>> •  Do aggregates negotiate with each other? (peer-to-peer) or a clearinghouse or service such as a slice manager coordinate? (top-down)
>> •  What about isolation for performance or security?
>> The mechanism I propose below address these questions for the narrow application of establishing end-to-end Ethernet VLANs.  Rather than try to solve the general problem, my goal is to establish a straightforward way to do stitching so that GENI aggregates and tools under development will understand what functions are required.
>>
>> Let me illustrate by an example.
>>
>>                +--------------------------------+
>>     +---------->|  Stitching Manager Service (S) |<----------+
>>     |           +--------------------------------+           |
>>     |                            |                           |
>>     V                            V                           V
>>  +--------------+        +--------------+        +--------------+
>>  ||AM|          |        |      |AM|    |        |          |AM||
>>  |+--+       +--|        |--+   +--+ +--|        |--+       +--+|
>>  |           |  |........|  |        |  |........|  |           |
>>  |           |SW|........|SW|        |SW|........|SW|           |
>>  |           |  |........|  |        |  |........|  |           |
>>  |           +--| usable |--+        +--| usable |--+           |
>>  |ProtoGENI (PG)| VLANs  |Internet2 (I2)| VLANs  |OpenFlow (OF) |
>>  +--------------+        +--------------+        +--------------+
>>
>> • A ProtoGENI cluster (PG), Internet2 (I2) and an campus OpenFlow network containing several hosts (OF) are configured to function as GENI aggregates.  
>>
>> • The ProtoGENI cluster administrator provisions connectivity to an Internet2 PoP through a regional network.  I2, PG, and the regional network administrators engineer the network, provisioning a set of VLANs for GENI use PG site and the I2 PoP.  The regional network can be thought of as a 'wire'.  The engineering of the network is worked out between the participants and is a 'local' matter.  The result is a set of VLANs known to the PG and I2 admins and pre-allocated for GENI use.
>>
>> • A similar process occurs between Internet2 and the campus OpenFlow network.  
>>
>> • A researcher now wishes to create a slice containing resources from PG and OF using I2 to provide network connectivity between them.  The researcher has acquired slice credentials allocated by a slice authority recognized by all three aggregates.  
>>
>> • The researcher (via  the GENI Aggregate Manger API) requests a sliver containing hosts connected by a topology on the ProtoGENI cluster.  The AM allocates the topology and hosts but does not yet connect them to the outside world.  
>>
>> • The above step is applied to the campus OpenFlow network.
>>
>> • The researcher now requests an I2 sliver providing Ethernet connectivity between the ProtoGENI cluster to the OpenFlow network.  The I2 AM allocates the topology but does not yet connect it to the outside world. At this point, three disconnected slivers have been established.
>>
>> • The researcher now provides his slice credentials to a stitching manager service, S, with two requests: stitch his PG and I2 slivers and stitch his I2 and OF slivers.  S, using a pre-established rule, determines the sort order for stitching is PG, OF, I2, meaning that for the PG-I2 VLAN, PG is contacted first and for the I2-OF VLAN, OF is contacted first.
>>
>> • S contacts the ProtoGENI AM, forwarding the slice credentials and the request to connect the sliver to Internet2.  The PG AM, using local policy determined by the ProtoGENI administrator, assigns a VLAN connecting the ProtoGENI cluster to Internet2 to this slice.  The PG-I2 VLAN identifying information is provided to S.  Even though the mapping has been determined, the PG switch is configured to drop traffic on the allocated VLANs until there is confirmation that the all stitching required by the slice is complete.  This is to avoid the possibility of traffic injected into a partially configured network.
>>
>> • S now contacts the I2 AM providing the slice credentials and the PG-I2 VLAN identifying information. The I2 AM prepares the mapping between the I2 internal network and the PG-I2 VLAN.  However, as within PG, the I2 switch is configured to drop traffic on the allocated VLANs until there is confirmation that the all stitching required by the slice is complete.
>>
>> • The previous two steps are repeated with OF and I2, starting with OF (as stated in step 8).  At this point S, knows the identifying information for all the stitching VLANs assigned to this slice.  This information is stored for operations and forensic use.  S also has confirmation that the stitching has been completed.  
>>
>> • S sends an indication to PG, OF, and I2 that the end-to-end network is configured.  Now the rules to drop traffic on the assigned VLANs are removed and each switch is configured to translate VLAN traffic between the assigned stitching VLAN and the internal network.  Each network sends a confirmation back to S.
>>
>> • S tells the researcher the end-to-end network is in place.
>>
>> Some of the assumptions here:
>> • We assume both switches must be able to do VLAN translation.  There are other techniques for stitching together VLANs but requiring translation at every switch will add minimal constraints on how to stitching might be established.  In other words, VLANs available for inter-aggregate connectivity won't be constrained by VLAN IDs in use within either aggregate.  VLAN translation won't be uniformly available throughout GENI for several years and the things will be more complex in the intervening period.
>> • VLANs are assumed to be pre-established before the stitching process is started.  Inter-aggregate VLANs will have some isolation and performance characteristics assigned to them when created, i.e., there may be some performance guarantees that can be made for some VLANs but this model isn't intended to support on-demand per-slice QoS negotiation on the stitching VLANs.  
>> • The policy by which an internal VLAN is mapped to an stitching VLAN is entirely local and under the control of the aggregate.
>> • We assume all networks can be represented as either an aggregate or a static VLAN (a 'wire').  I believe the implication here is that the backbones behave like aggregates.  
>> • We assume that S picks the first aggregate in an unambiguous and repeatable way (e.g., in some sort order) to avoid race conditions where both A and B give out the same VLAN to different users.
>> • We assume that the aggregate manager can configure the switch to connect the assigned VLAN to the network resources allocated to the slice.
>> • Once a VLAN has been assigned to a slice for stitching, it has to be reported and recorded by the GENI clearinghouse for operations and forensic use.  Therefore, the VLAN identifying information needs to be in a standard form.
>>
>> Questions I have:
>> • The AM is required since the binding is between resources allocated to a slice and the VLAN.  Will an extension to the aggregate API be required to support the protocol above?  Perhaps not: this sort of looks like a 'revise an existing slice' operation.
>> • It seems like the 'usable VLANs' could be multiplexed over a 802.11QinQ,  GRE, OpenVPN, or (G)MPLS tunnels. E.g., if QinQ or even an IP tunnel is used, the tunnel should be established but VLAN IDs should still be passed to permit demultiplexing at the edges.  What are the implications?  
>>
>> Let me conclude by saying I'm sending this to the Control Framework and Experimenter Workflow and Services working groups because I believe there are implications for both groups embedded in this proposal.  This is an important experimenter service that is not part of the control plane (and thus in scope for the services-wg.  Additionally, aggregate managers and RSpecs would need to support this protocol, making it relevant to the control-wg.  I think if we can get a rough agreement that something like this would work, I'd like to hear what would be needed to prototype it.
>>
>> regards,
>>
>> --aaron
>>
>> [1] http://groups.geni.net/geni/attachment/wiki/GEC6CFWGAgenda/gec6-cf-chase.ppt
>>
>>
>> _______________________________________________
>> control-wg mailing list
>> control-wg@...
>> http://lists.geni.net/mailman/listinfo/control-wg
>>    
>
>  

_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Aaron Falk]

Hi Rob-

I'm glad you've been thinking about this too!

On 2/4/10 8:53 PM, Robert P Ricci wrote:

> Thanks for getting this discussion kicked off!
>
> The model we have been pursuing is a little different than the one you
> propose below. A rough sketch (in less detail than your proposal),
> picking up where we diverge:
>
> 5. The user constructs an RSpec (likely with help from some user tools)
>    that describes the whole slice they want to build, including the PG
>    and OF nodes and link(s) between them
> 6. If the user *knows* the path they want to use on I2, they fill that in
>    - if they don't know or don't care, they pass their request RSpec to a
>    slice embedding service (SES), which fills out these details for them.
>    Note that the SES isn't actually setting anything up on the users'
>    behalf, it is just *annotating* the request RSpec to fill in details
>    for the user
>  
> 7. The user submits the RSpec to the appropriate AMs (in this case PG,
>    OF, and I2) - it's trivial to extract the right set of AMs from the
>    RSpec if the SES added any (for example, if it added nodes in the I2
>    aggregate)
>  

It hadn't occurred to me that all the aggregates would see the full
RSpec.  This sounds like probably a Good Thing.

> 8. The AMs communicate in pairwise fashion to agree on things like
>    VLAN#s for the interconnection point. They can do this because the
>    request RSpec that each AM sees includes 'external references' to the
>    other AMs in the topology.
>  

As I said to Max, I'm a little concerned about the complexity in the
'pairwise fashion' negotiation.  See below.

> 9. The AMs, using the negotiated VLAN#s, etc., individually create their
>    slivers and inform the user when they're ready.
>
> We've implemented up to step 7. While 'pairwise fashion' in step 8
> sounds like it would scale poorly, consider the fact that the number of
> places where there are physical interconnects between various aggregates
> are likely to be limited in number and fairly static, bounding the
> number of pairs that have to agree. And, in fact, in the medium term,
> what's likely to happen is that there are a couple of backbone AMs, with
> aggregates hanging off of them. This results in a very simple, easy to
> manage structure: backbones act as 'masters' for selecting VLAN#s for
> their attached aggregates, the only place where real negotiation has to
> occur is at connection points between backbones.
>  

How do you handle the situation when it's not obvious who the 'master'
is?  E.g., when you have multiple backbones in a slice (we already have
2)?  What about when the two ends try to assign the same VLAN to
different slices?  Maybe I'm being unreasonably conservative.  Can you
describe the 'pairwise' protocol?  Is there code?

Can you comment on my point to Max about keeping one sliver from sending
data out of the aggregate until the end to end slice is stitched.  Do
you think this is a concern?

Finally, I suggest that the VLAN information gets logged in the
clearinghouse.  This seems like important operational information that
will be needed to help diagnose faulty slices and to assign
accountability for packets leaving an aggregate.

--aaron

_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Robert P Ricci]

For the record, I also agree with Ilia's comments.

Thus spake Aaron Falk on Fri, Feb 05, 2010 at 10:43:04AM -0500:

> > 8. The AMs communicate in pairwise fashion to agree on things like
> >    VLAN#s for the interconnection point. They can do this because the
> >    request RSpec that each AM sees includes 'external references' to the
> >    other AMs in the topology.
> >  
>
> As I said to Max, I'm a little concerned about the complexity in the
> 'pairwise fashion' negotiation.  See below.
>
>
> > 9. The AMs, using the negotiated VLAN#s, etc., individually create their
> >    slivers and inform the user when they're ready.
> >
> > We've implemented up to step 7. While 'pairwise fashion' in step 8
> > sounds like it would scale poorly, consider the fact that the number of
> > places where there are physical interconnects between various aggregates
> > are likely to be limited in number and fairly static, bounding the
> > number of pairs that have to agree. And, in fact, in the medium term,
> > what's likely to happen is that there are a couple of backbone AMs, with
> > aggregates hanging off of them. This results in a very simple, easy to
> > manage structure: backbones act as 'masters' for selecting VLAN#s for
> > their attached aggregates, the only place where real negotiation has to
> > occur is at connection points between backbones.
> >  
>
> How do you handle the situation when it's not obvious who the 'master'
> is?  E.g., when you have multiple backbones in a slice (we already have
> 2)?  What about when the two ends try to assign the same VLAN to
> different slices?  Maybe I'm being unreasonably conservative.  Can you
> describe the 'pairwise' protocol?  Is there code?

For VLAN numbers, the 'negotiation' is basically no more complicated
than set intersection - each side indicates the set of VLANs it has
available for use, you intersect the sets and pick N VLAN #s from the
result. I don't think it matters which side 'drives' this process, since
it's trivial to verify that it was done correctly. Even for more
complicated setups where there are are more than two parties that have
to agree (such as several aggregates that might share some wireless
spectrum), this is still very straightforward.

Now, as you say, longer term, we would ideally like to be in a world
where both sides don't have to *agree* on the VLAN numbers (or more
generally, labels) to use, which of course basically gets rid of the
negotiation process completely - each side simply informs the other of
the label it is using, and one or both translates between the label it
receives 'on the wire' (or over the air) and the one it uses within the
aggregate. For the wired world, technologies such as VLAN translation,
q-in-q, and MPLS will all help with this.
 
> Can you comment on my point to Max about keeping one sliver from sending
> data out of the aggregate until the end to end slice is stitched.  Do
> you think this is a concern?

I think that AMs should have the policy that if it receives a packet at
a 'touch point' with an unknown label (eg. VLAN tag #), it should
discard that packet.  This is a good idea for a number of reasons,
especially security. I would be extremely surprised if an AM *didn't*
implement this policy; I would be unwilling to implement any other
policy. I think this addresses your concern: while a link is in a 'half
set up' state, one side will not be configured to accept the new label,
and traffic it receives will just be dropped, not disruptive.
 
> Finally, I suggest that the VLAN information gets logged in the
> clearinghouse.  This seems like important operational information that
> will be needed to help diagnose faulty slices and to assign
> accountability for packets leaving an aggregate.

In our design, all logging is local to the aggregate. What we're
planning (but have not yet implemented), is that each AM *report* to the
clearninghouse to inform it about slices and slivers that exist on the
aggregate - this makes it straightforward to go around collecting the
relevant logs when you want them.

_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Jeff Chase]

Thanks Aaron.  Just a few comments.

ORCA uses what we might call the "coordinator" approach similar to what
you describe.  But rather than defining a separate stitching manager
service to function as the coordinator, the Slice Manager (SM)
representing the slice acts as the coordinator.  The SM is a piece of
software representing "the researcher" in your description.

This arrangement does not preclude various peering AMs from coordinating
directly, as in the "peer to peer" approaches ProtoGENI and OMF are
pursuing.  But we wanted to avoid requiring neighboring AMs to
pre-coordinate bindings to talk to each other.  And we wanted to allow
for scenarios involving "logical" stitches where there is no physical
adjacency, like nodes attaching to a storage server in some other domain.

The authorization questions for stitching should also be on the table.  
They are a bit subtle.  In your proposal, does the stitch manager
service run with any special privilege?  Do the AMs trust it?  Does the
CH trust it?  Presumably the "researcher" ( or its SM or experiment
control tools) must trust it.  What is the basis of this trust?  

In general, each stitch involves agreement on a common label, e.g., a
VLAN tag from one AM that is to be spliced into a VLAN from another AM
at some adjacency point.  One side of the stitch produces the label, and
the other side consumes it.  How does the consuming AM know that the
coordinator reports the label correctly?  I think this the nub of the
authorization question for stitching.

In ORCA, the producing AM signs the label, and the coordinator presents
the signed label to the consuming AM.   The public keys of the AMs are
endorsed by some mutually trusted third party, such as a CH (or a
certifying authority for a federation).  In ORCA, tickets issued by a
broker/CH can serve as the endorsement, since they contain the public
key of the AM named in the ticket, and are signed by the broker.  (I
gloss over the multi-broker case.)   This allows the SM to coordinate
stitching without any special privilege.

The ORCA protocol does not require any AM to know anything about what
the slice looks like in other domains, except for the labels imported at
the stitching points.   Only the SM/coordinator knows the entire
structure of the slice.   This is important because we want to avoid
assuming that slices are static entities, or that there is some central
controlling authority for the entire "testbed".

Jeff




Aaron Falk wrote:

> This is a note motivated by topics raised at the GEC6 Control
> Framework WG meeting.  Comments, criticism, feedback, and corrections
> are welcome.
>
> The issue of slice stitching has come up periodically and in the
> interest of making some progress on it, I wanted to propose a
> mechanism for stitching together aggregates using VLANs.  What is
> slice stitching?  Slice stitching refers to the process of
> interconnecting slivers between is different GENI aggregates.  In the
> near term, GENI needs to be able to create Ethernet VLANs that connect
> aggregates  (although over the longer term more diverse
> interconnections will be desired).
>
> Jeff Chase, in his slides at the GEC6 CFWG meeting [1], catalogs  a
> very good list of questions on stitching:
>
>     *  How to join slivers/slices across different aggregates end-to-end?
>     *  Do we require common labels at junction points?
>     *  How to connect slivers?
>     *  Do aggregates negotiate with each other? (peer-to-peer) or a
>       clearinghouse or service such as a slice manager coordinate?
>       (top-down)
>     *  What about isolation for performance or security?
>
> The mechanism I propose below address these questions for the narrow
> application of establishing end-to-end Ethernet VLANs.  Rather than
> try to solve the general problem, my goal is to establish a
> straightforward way to do stitching so that GENI aggregates and tools
> under development will understand what functions are required.
>
> Let me illustrate by an example.
>
>                +--------------------------------+
>     +---------->|  Stitching Manager Service (S) |<----------+
>     |           +--------------------------------+           |
>     |                            |                           |
>     V                            V                           V
>  +--------------+        +--------------+        +--------------+
>  ||AM|          |        |      |AM|    |        |          |AM||
>  |+--+       +--|        |--+   +--+ +--|        |--+       +--+|
>  |           |  |........|  |        |  |........|  |           |
>  |           |SW|........|SW|        |SW|........|SW|           |
>  |           |  |........|  |        |  |........|  |           |
>  |           +--| usable |--+        +--| usable |--+           |
>  |ProtoGENI (PG)| VLANs  |Internet2 (I2)| VLANs  |OpenFlow (OF) |
>  +--------------+        +--------------+        +--------------+
>
>    1. A ProtoGENI cluster (PG), Internet2 (I2) and an campus OpenFlow
>       network containing several hosts (OF) are configured to function
>       as GENI aggregates.
>
>    2. The ProtoGENI cluster administrator provisions connectivity to
>       an Internet2 PoP through a regional network.  I2, PG, and the
>       regional network administrators engineer the network,
>       provisioning a set of VLANs for GENI use PG site and the I2
>       PoP.  The regional network can be thought of as a 'wire'.  The
>       engineering of the network is worked out between the
>       participants and is a 'local' matter.  The result is a set of
>       VLANs known to the PG and I2 admins and pre-allocated for GENI use.
>
>    3. A similar process occurs between Internet2 and the campus
>       OpenFlow network.
>
>    4. A researcher now wishes to create a slice containing resources
>       from PG and OF using I2 to provide network connectivity between
>       them.  The researcher has acquired slice credentials allocated
>       by a slice authority recognized by all three aggregates.
>
>    5. The researcher (via  the GENI Aggregate Manger API) requests a
>       sliver containing hosts connected by a topology on the ProtoGENI
>       cluster.  The AM allocates the topology and hosts but does not
>       yet connect them to the outside world.
>
>    6. The above step is applied to the campus OpenFlow network.
>
>    7. The researcher now requests an I2 sliver providing Ethernet
>       connectivity between the ProtoGENI cluster to the OpenFlow
>       network.  The I2 AM allocates the topology but does not yet
>       connect it to the outside world. At this point, three
>       disconnected slivers have been established.
>
>    8. The researcher now provides his slice credentials to a stitching
>       manager service, S, with two requests: stitch his PG and I2
>       slivers and stitch his I2 and OF slivers.  S, using a
>       pre-established rule, determines the sort order for stitching is
>       PG, OF, I2, meaning that for the PG-I2 VLAN, PG is contacted
>       first and for the I2-OF VLAN, OF is contacted first.
>
>    9. S contacts the ProtoGENI AM, forwarding the slice credentials
>       and the request to connect the sliver to Internet2.  The PG AM,
>       using local policy determined by the ProtoGENI administrator,
>       assigns a VLAN connecting the ProtoGENI cluster to Internet2 to
>       this slice.  The PG-I2 VLAN identifying information is provided
>       to S.  Even though the mapping has been determined, the PG
>       switch is configured to drop traffic on the allocated VLANs
>       until there is confirmation that the all stitching required by
>       the slice is complete.  This is to avoid the possibility of
>       traffic injected into a partially configured network.
>
>   10. S now contacts the I2 AM providing the slice credentials and the
>       PG-I2 VLAN identifying information. The I2 AM prepares the
>       mapping between the I2 internal network and the PG-I2 VLAN.
>       However, as within PG, the I2 switch is configured to drop
>       traffic on the allocated VLANs until there is confirmation that
>       the all stitching required by the slice is complete.
>
>   11. The previous two steps are repeated with OF and I2, starting
>       with OF (as stated in step 8).  At this point S, knows the
>       identifying information for all the stitching VLANs assigned to
>       this slice.  This information is stored for operations and
>       forensic use.  S also has confirmation that the stitching has
>       been completed.
>
>   12. S sends an indication to PG, OF, and I2 that the end-to-end
>       network is configured.  Now the rules to drop traffic on the
>       assigned VLANs are removed and each switch is configured to
>       translate VLAN traffic between the assigned stitching VLAN and
>       the internal network.  Each network sends a confirmation back to S.
>
>   13. S tells the researcher the end-to-end network is in place.
>
>
> Some of the assumptions here:
>
>     * We assume both switches must be able to do VLAN translation.
>       There are other techniques for stitching together VLANs but
>       requiring translation at every switch will add minimal
>       constraints on how to stitching might be established.  In other
>       words, VLANs available for inter-aggregate connectivity won't be
>       constrained by VLAN IDs in use within either aggregate.  VLAN
>       translation won't be uniformly available throughout GENI for
>       several years and the things will be more complex in the
>       intervening period.
>     * VLANs are assumed to be pre-established before the stitching
>       process is started.  Inter-aggregate VLANs will have some
>       isolation and performance characteristics assigned to them when
>       created, i.e., there may be some performance guarantees that can
>       be made for some VLANs but this model isn't intended to support
>       on-demand per-slice QoS negotiation on the stitching VLANs.
>     * The policy by which an internal VLAN is mapped to an stitching
>       VLAN is entirely local and under the control of the aggregate.
>     * We assume all networks can be represented as either an aggregate
>       or a static VLAN (a 'wire').  I believe the implication here is
>       that the backbones behave like aggregates.
>     * We assume that S picks the first aggregate in an unambiguous and
>       repeatable way (e.g., in some sort order) to avoid race
>       conditions where both A and B give out the same VLAN to
>       different users.
>     * We assume that the aggregate manager can configure the switch to
>       connect the assigned VLAN to the network resources allocated to
>       the slice.
>     * Once a VLAN has been assigned to a slice for stitching, it has
>       to be reported and recorded by the GENI clearinghouse for
>       operations and forensic use.  Therefore, the VLAN identifying
>       information needs to be in a standard form.
>
>
> Questions I have:
>
>     * The AM is required since the binding is between resources
>       allocated to a slice and the VLAN.  Will an extension to the
>       aggregate API be required to support the protocol above?
>       Perhaps not: this sort of looks like a 'revise an existing
>       slice' operation.
>     * It seems like the 'usable VLANs' could be multiplexed over a
>       802.11QinQ,  GRE, OpenVPN, or (G)MPLS tunnels. E.g., if QinQ or
>       even an IP tunnel is used, the tunnel should be established but
>       VLAN IDs should still be passed to permit demultiplexing at the
>       edges.  What are the implications?
>
>
> Let me conclude by saying I'm sending this to the Control Framework
> and Experimenter Workflow and Services working groups because I
> believe there are implications for both groups embedded in this
> proposal.  This is an important experimenter service that is not part
> of the control plane (and thus in scope for the services-wg.  
> Additionally, aggregate managers and RSpecs would need to support this
> protocol, making it relevant to the control-wg.  I think if we can get
> a rough agreement that something like this would work, I'd like to
> hear what would be needed to prototype it.
>
> regards,
>
> --aaron
>
> [1]
> http://groups.geni.net/geni/attachment/wiki/GEC6CFWGAgenda/gec6-cf-chase.ppt
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> services-wg mailing list
> services-wg@...
> http://lists.geni.net/mailman/listinfo/services-wg

_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Robert P Ricci]

Thus spake Jeff Chase on Sat, Feb 06, 2010 at 04:57:36PM -0500:

> ORCA uses what we might call the "coordinator" approach similar to what
> you describe.  But rather than defining a separate stitching manager
> service to function as the coordinator, the Slice Manager (SM)
> representing the slice acts as the coordinator.  The SM is a piece of
> software representing "the researcher" in your description.
>
> This arrangement does not preclude various peering AMs from coordinating
> directly, as in the "peer to peer" approaches ProtoGENI and OMF are
> pursuing.  But we wanted to avoid requiring neighboring AMs to
> pre-coordinate bindings to talk to each other.

I probably didn't make this clear enough, but one of our main goals in
the design I outlined was to make the negotiation between neighboring
AMs invisible to the user, so that different implementations are
possible. From the user perspective, they just need to create a sliver
on every aggregate that needs to be involved in the slice, and those
aggregates will take care of stitching where it's requested.  The
peer-to-peer approach with pre-established relationships happens to be
the one we're pursuing right now. I'd be happy to have other methods
supported as well (though of course every AM that is going to
participate in a particular stitching needs to support a common method.)
For example, AMs could agree on a trusted third party that they're going
to use for negotiating labels (and this third party need not be the
researcher or the SM.)

> And we wanted to allow
> for scenarios involving "logical" stitches where there is no physical
> adjacency, like nodes attaching to a storage server in some other domain.

I'll answer these questions for the ProtoGENI design, more comments on
ORCA's below.

> The authorization questions for stitching should also be on the table.  
> They are a bit subtle.

Agreed.

> In your proposal, does the stitch manager service run with any special
> privilege? Do the AMs trust it?  Does the CH trust it?

No to all three. The main case where trust must be established is
between AMs that are physically neighboring. I think this is reasonable,
as plugging in a wire or sharing spectrum requires inherently requires
some degree of trust.

> Presumably the "researcher" ( or its SM or experiment control tools)
> must trust it.  What is the basis of this trust?  

In our design, the process of talking to all of the AMs involved falls
'by default' to the researcher. Therefore, they don't *have* to place
their trust in some stitching or slice management service. In practice,
we think it's *likely* that most users will want to use such a service.

The thing the researcher does have to trust is that the aggregates that
they ask to set up a stitched slice do so correctly (the user can
verify this to some extent, however.) When the AMs set up stitched links,
they are acting on their own authority, rather than the user's, though
they do have a signed ticked to prove that someone really did request
the stitching, and who.
 
> In general, each stitch involves agreement on a common label, e.g., a
> VLAN tag from one AM that is to be spliced into a VLAN from another AM
> at some adjacency point.  One side of the stitch produces the label, and
> the other side consumes it.  How does the consuming AM know that the
> coordinator reports the label correctly?  I think this the nub of the
> authorization question for stitching.

We sidestep this question by having aggregates communicate directly -
they don't have to trust a third party coordinator (unless they want
to.) Of course, the two (or more) sides must authenticate each other.
This works well when the aggregates are physically adjacent and have
pre-established trust, as noted above. For the case of 'logical'
connections (we support some now in the form of IP tunnels), in our
design, the public keys of all AMs are available from the CH. (So
everyone must trust the CH to give out valid public keys, which is a
pretty core assumption in our design.)
 
> In ORCA, the producing AM signs the label, and the coordinator presents
> the signed label to the consuming AM.   The public keys of the AMs are
> endorsed by some mutually trusted third party, such as a CH (or a
> certifying authority for a federation).  In ORCA, tickets issued by a
> broker/CH can serve as the endorsement, since they contain the public
> key of the AM named in the ticket, and are signed by the broker.  (I
> gloss over the multi-broker case.)   This allows the SM to coordinate
> stitching without any special privilege.

So, if I understand this right, this requires the coordinator to be the
intermediary for label information? One of our earlier designs had this
property too, but we decided to move away from it for two reasons:

1) Ordering: in some cases, link setup must be done in a specific order
    - eg. for many types of tunnels, one end must be set up to listen
    before the other can connect. If labels pass through a third party,
    this seems to require the third party to have some knowledge about
    the ordering required for every stitchable link type. The more
    aggregates a single link passes through, the worse this problem
    gets. I'd rather 'hide' all that knowledge in the AMs which have to
    know it anyway.
2) Negotiation: if there is actual negotiation required for labels rather
    than just a 'label exchange', having a third party act as
    intermediary makes things a lot more complicated. The exception to
    this statement would be if the two (or more) aggregates are
    unwilling to trust each other, but *are* willing to trust the third
    party. This seems like an unlikely scenario for GENI, and even if
    it happens, our design does support it.

How does ORCA deal with these issues?

I like the fact that the SM has no special privileges in your design -
would it be fair to say that it's more a service than a 'core' part of
the control framework?
 
> The ORCA protocol does not require any AM to know anything about what
> the slice looks like in other domains, except for the labels imported at
> the stitching points.   Only the SM/coordinator knows the entire
> structure of the slice.   This is important because we want to avoid
> assuming that slices are static entities, or that there is some central
> controlling authority for the entire "testbed".

In our current implementation, we do pass the researcher's entire
request RSpec around to every AM, and each AM simply ignores the parts
that aren't relevant to it. We could, I think, give each AM only the
relevant bits (components it controls + stitched links) - the main reason
we're not doing it now just is simplicity on the client side (it doesn't
have to worry about carving up the RSpec by AM.)

Your point about not wanting a single controlling entity is a good one,
and that's why I'm glad to see that the OMF and ORCA (as well as
ProtoGENI) designs don't have a trusted stitcher - in the limit, such a
stitcher has to have a global view of the physical substrate and be
globally trusted, which doesn't scale well.

_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Aaron Falk]

Hello all-

I've tried to summarize what's been said on this topic.  I'm sure some
of it is wrong and I left out several points that seemed IMO somewhat
off the prime topic.  Please annotate or comment as you see fit.

--aaron

1. What kind of helper services are used?

   a. An optional slice embedding service that annotates an RSpec
      description of a full slice with additional connectivity
      information to permit AMs to identify that they need to
      establish a 'stitching' VLAN. (Rob)

   b. A mandatory stitching coordinator as part of (or on behalf of) a
      clearinghouse thus representing GENI operations. It controls the
      exchange of (and logs) stitching VLAN parameters as they are
      passed between aggregates. (Aaron) (nb: I didn't say they were
      part of the clearinghouse in my original note because I thought
      I didn't have to but Jeff's thoughts about authorization
      convince me that it is probably a needed assumption.  See
      4. below.)

   c. An optional stitching coordinator as part of a slice manager
      representing the researcher that assists in the exchange of
      stitching VLAN parameters as they are passed between
      aggregates. (Jeff)

2. How does an Aggregate know to establish a stitching connection to
   it's peer?

   a. Both aggregates see an RSpec describing a connection between
      them.  Requires both AMs to understand the stitching portion of
      an RSpec. (Rob)

   b. A stitching service tells an AM to bind a sliver to a stitching
      VLAN (Aaron).

3. How does the peer Aggregate know which stitching VLAN to bind to a
   sliver?

   a. AMs negotiate with each other. Requires a coordination
      protocol. (Rob)

   b. One AM assigns a VLAN and a stitching service passes the
      information to it's peer.  Requires use of the external
      service. (Aaron)

4. What authority do external services have?

   a. The service in Rob's model (1a) is modifying request RSpecs on
      behalf of the researcher.  (Are the modifications signed?  If
      so, then by whom?)

   b. The service in Aaron's model (1b) is an intermediary between two
      AMs.  AMs have long-term trust relationships with
      clearinghouses.  In this case, the clearinghouse is trusted to
      reliably pass stitching parameters.

   c. The service in Jeff's model (1c) acts on behalf of the
      researcher and thus has no special privileges.  AMs sign
      stitching parameters so peer AMs can trust they have not been
      modified en-route.

4. Visibility goals

   a. The researcher should not have to participate in or be aware of
      stitching. (Rob)

   b. Only the researcher should have to know the full set of
      resources in a slice.  (Jeff)

   c. An operations group should know about all stitched links.
      (Aaron)

5. What are the needs to control the order in which stitching
   connections are established? ?

   a. Aggregates need to control the establishment phases of, e.g.,
      tunnels. (Rob)

   b. To prevent use of a network which is not yet fully provisioned,
      researcher should be prevented from using any stitched links
      until the end-to-end topology is established.  (Aaron)



_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Jeff Chase]

Max Ott wrote:

> On 25/02/2010, at 8:16 AM, Jeff Chase wrote:
>
>  
>> Trust is anchored
>> in identity providers outside of ORCA, and each actor controls its own
>> policies regarding what powers to give to identities endorsed by those
>> providers.
>>    
>
> What is trust really used here? Is it trusting that some 'message' comes from whoever they claim they are? Trust, that the entity I delegated some authority, too, did the right thing?
>  

By "trust is anchored in identity providers external to ORCA", I was
just referring to identity providers as external authorities to
authenticate users and other identities, and assert the attributes of
those identities for use by an authorization policy.  That statement was
a little bit glib: the local operator for each ORCA server is empowered
to define trust relationships to other actors and IdPs, e.g., what
clearinghouse to advertise your resources to, what identity providers to
accept user sessions from, etc.

Authorization policies can then be formulated using those attributes.  
For example "this principal is the PI on project XXX" can be an identity
attribute asserted by an identity provider, and an authorization policy
can use that information.   This is basically the nub of Shibboleth.
> In our world, for an entity to act on a request it needs to carry the right authorisation. An authorisation is normally a chain of authorisations which ultimately needs to lead back to the receiver (or the one acting on the request). A request comes from user A which is authorised by PI B which is authorised by organisation C which I have an agreement with. Something, like a Policy Decision Point, then needs to traverse that and ensure that they are not only authentic, but also conform to the associated policies.
>  
Right.  I think we are all in agreement about that.  For GENI, the broad
outlines of authorization policy based on chained delegations goes back
to ideas hammered out by Mike Reiter and Steve Schwab and others back in
the last round of the process, pre-BBN (2005-2006).   It is also present
in SHARP (an SOSP 2003 paper) with respect to resource control, and
SHARP is a predecessor of ORCA.  SFA suggests one instantiation of
chained delegation mechanisms as a basis for authorization policy.   I
think there is still some groping around for consensus on a general set
of mechanisms (and running code) for specifying and processing these
delegations in a way that is fully disentangled from policy. There have
been some discussions about that off on the side, involving Rob Ricci
and Steve Schwab and some Shibboleth folks and Aaron, and that probably
should be summarized on this list.

I would certainly like to hear more about the chaining/delegation
mechanisms in OMF.
> I fully agree with Jeff that in a truly federated world there is no place in the architecture for a central authority. Certain deployments may delegate all decisions to a single entity, but that's a deployment decision and not one dictated by architecture.
>  
Thanks.  Just for the record, I feel sure that Rob and the SFA designers
would agree with that too, and I didn't intend to suggest otherwise.

Jeff



_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg