Re: two issues of terminology

[Vicraj Thomas]

Jeff,

I understand the distinction you are making.  There are tools and frameworks
an experimenter may use to program slivers of components.  These include
programming language compilers/interpreters, FPGA programming tools,
frameworks such as SILO and x-kernel, and even debuggers.  These are your
"programming tools".

Then there are tools that an experimenter uses to deploy, control, monitor
and instrument experiments and even tools to collaborate with other
experimenters.  These are your "experimenter's tools" or "experiment control
tools".

I agree it is useful to distinguish the two types of tools.  I think of
"programming tools" as being very specific to the technology used to
implement the component being slivered and to the technology being
investigated by the experiment.  These tools may or may not run within a
slice.

"Experimenter tools" on the other hand are mostly independent of the
technology of the components (or slivers) used by the experiment.  They
operate at the level of a slice or sliver and don't particularly care about
how the sliver was programmed.  I say "mostly independent" since some tools
such as provisioning tools will likely need to know if they are installing
software in a machine with an OS or loading FPGA configuration files.
However the level of abstraction seen by all experimenters is roughly the
same and is independent of the technology used to implement the components
or the technology being studied.  These tools may run inside a slice or
outside or both.

I propose:
    - We distinguish between programming tools and experimenter tools
    - The distinction is not based on whether the tools run inside a slice
or outside.  Rather the distinction is based on whether the tool is used to
program slivers or to manage experiments.  (I realize this isn't a precise
distinction---I'll try to come up with something better.)
    - Experimenter tools include provisioning tools, experiment control
tools, instrumentation tools, collaboration tools, etc..
    - The focus of GENI be on experimenter tools.  There are, in addition,
operations and management tools of course but that's the OMIS WG's problem.
:-)

The Experiment Lifecycle document focuses on "Experimenter Tools".  The
document says a resource's Rspec may have pointers to tools and techniques
for programming the resource (i.e. Programming Tools).  The Resource
Discovery experimenter tool that is used to find resources suitable for an
experiment may read this Rspec and point the experimenter to relevant
programming tools.

Comments?

< Vic



On Feb 18 11:30 AM, "Jeff Chase" <chase@...> wrote:

> Vic, thanks, I'm nodding my head at the rough-cut taxonomy in your
> e-mail.  But I think it illustrates exactly the distinction I was trying
> to make, and the need for that distinction.  In particular, you said:
>
> Vicraj Thomas wrote:
>>     - Provisioning tools.  These are the first category of tool you mention
>> in your email and is used to program components: Load software into
>> components and configure them.  Raven is an example of such a tool for the
>> PlanetLab control framework.
>>  
> But no!  Raven loads code into a sliver, but it is not a tool(kit) to
> create that code.   So it is distinct from the first category of tool
> that I mentioned in my e-mail.  SILO, Click, x-kernel, etc., are
> toolkits for creating programs that will run inside slivers/slices.  For
> example, they might include libraries of off-the-shelf operators and so
> on that are useful in programming certain kinds of components (e.g.,
> routers or edge stack implementations), and various ways of combining
> those operators and coordinating them to create a program.  Then Raven
> (or whatever) loads the program into the sliver.
>
> If we consider that distinction, then all of the categories in your
> taxonomy are tools that run outside of a slice.   Raven runs outside of
> a slice.  And my view is that "experiment control tools" or
> "experimenter tools", e.g., as we use the term in the CF docs, should
> include all the things in your list but exclude the distinct category of
> programming tools, the first category in my e-mail.  It is a useful
> distinction.
>
> [There are some interesting examples that cross the boundary of course
> (e.g., for feedback-driven slice adaptation), which comes back to the
> second point about "slice controller".  In general, any of these tools
> that run "outside" a slice might have some minimal hooks "inside" the
> slice.   Analogy: "VMware tools" if you have ever worked with VMware.]
>
> Also, I note that GUSH has elements of your first two categories:
> resource discovery, and provisioning.  At least it acts as the client
> side for those services.   My point is that the distinction between
> "experimenter tools" and "experiment control tools" (as a subcategory)
> seems less useful to me.
>
> Jeff
>>     - Experiment control tools: These are the second category of tools you
>> mention in your email.  GUSH is an example of such a tool.
>>
>> I include other tools under "Experimenter Tools" such as:
>>     - Debugging tools
>>     - Collaboration tools for use by researchers based in different
>> locations
>>     - Search tools to find resources
>>
>> < Vic
>>
>>
>>
>> On Feb 17 12:41 PM, "Jeff Chase" <chase@...> wrote:
>>
>>  
>>> Folks,
>>>
>>> Two quick questions/proposals on terminology.  These might be comments
>>> for the CF requirements doc.  But they have also come up in drafting
>>> spiral 2 proposals.
>>>
>>> Both deal with what has been called "experimenter tools" or "experiment
>>> control tools" in various GENI docs.
>>>
>>> (1) There is a useful distinction between code that runs inside a slice
>>> vs. code that runs outside a slice.   Code that runs inside a slice
>>> falls under the category of "component programming" and the
>>> programmability requirement (S 5.5.5 in the CF requirements doc).   The
>>> question is: do we have an accepted name for toolkits or other
>>> off-the-shelf software artifacts whose purpose is to support
>>> easy/flexible programming of various components?   An example from the
>>> literature might be Click or Ilia's SILO framework.    The specific
>>> question that drove this concerns whether it is right to refer to SILO
>>> as "experimenter tools".  I would argue that we should not consider
>>> these as "experimenter tools" so as not to blur this useful distinction.
>>>
>>> (2) I think we have reached some kind of consensus (within services-wg
>>> and I think in my discussions with GPO folks) for a first-class
>>> entity/actor that controls and monitors a slice.  I think Gush is
>>> perhaps the best-known example.  The defining element that makes such an
>>> entity "first-class" is that it is persistent so that other actors in
>>> the control framework, or slivers in its slice, can send unsolicited
>>> messages/notifications to it.   And it may respond by taking autonomous
>>> actions to control the slice on behalf of an experimenter.  Previously
>>> we haven't had a name for this entity.  I propose that it be called
>>> "slice controller".   (Note: in Orca-world we presume/require/support
>>> such a beast: we have called it "service manager" since the SHARP days
>>> in 2003, and then started talking about "guest controller" plugin to the
>>> SM that implements a control policy.)  The specific question that drove
>>> this is whether we will be understood by GPO if we say "slice
>>> controller".  I think David Irwin had told me he though Harry's
>>> preferred term was "experiment controller", but I'm not seeing any
>>> "controller" term now in the CF req doc.
>>>
>>> Jeff
>>>
>>>
>>>
>>>
>>>
>>>    
>>
>>  
>

_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Robert P Ricci]

Thus spake Vicraj Thomas on Thu, Feb 19, 2009 at 11:19:43AM -0600:
> I propose:
>     - We distinguish between programming tools and experimenter tools
>     - The distinction is not based on whether the tools run inside a slice
> or outside.  Rather the distinction is based on whether the tool is used to
> program slivers or to manage experiments.  (I realize this isn't a precise
> distinction---I'll try to come up with something better.)
>     - Experimenter tools include provisioning tools, experiment control
> tools, instrumentation tools, collaboration tools, etc..

I think this definition is basically going in the right direction.  Let
me take a couple tries at alternate versions.

Still vague:

An experimenter tool *sets up* the environment in which a system under
test runs, a programming tool is (may be?) *part* of that environment.

More specific, but maybe not quite right:

A programming tool is one that is in the "codepath" (broadly defined) of
the system under test - that is, it's directly involved in the
experiment because my application calls it as a library, makes syscalls
into it, sends packets through it, etc. An experimenter tool is outside
this codepath in that the interaction is one way - the experimenter tool
might "call into" the SUT to start it, modify its configuration, etc,
but the SUT doesn't "call into" the experimenter tool. (And in the case
of collaboration tools and similar things, the tool may never really
interact with the SUT at all.)

--
/-----------------------------------------------------------
| Robert P Ricci <ricci@...> | <ricci@...>
| Research Associate, University of Utah Flux Group
| www.flux.utah.edu | www.emulab.net
\-----------------------------------------------------------

_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Jeff Chase]

It seems we all agree that there is a useful distinction between
"programming tools" and "experimenter tools" for a slice.  And we're
still trying to pin down the exact nature of the distinction.

IMHO, we shouldn't work too hard to make the definition precise, because
there is no bright line.  There may be tools that cross whatever line we
draw.   This definition seems pretty good:

Robert P Ricci wrote:

> ...
> An experimenter tool *sets up* the environment in which a system under
> test runs, a programming tool is (may be?) *part* of that environment.
>
> More specific, but maybe not quite right:
>
> A programming tool is one that is in the "codepath" (broadly defined) of
> the system under test - that is, it's directly involved in the
> experiment because my application calls it as a library, makes syscalls
> into it, sends packets through it, etc. An experimenter tool is outside
> this codepath in that the interaction is one way - the experimenter tool
> might "call into" the SUT to start it, modify its configuration, etc,
> but the SUT doesn't "call into" the experimenter tool. (And in the case
> of collaboration tools and similar things, the tool may never really
> interact with the SUT at all.)
>
>  

But that almost suggests that the System Under Test cannot notify the
experimenter tools (e.g., slice controller) about events within the
SUT.  We can dodge that by saying that the SUT exposes/publishes events
and the external controller subscribes to those events, so that the SUT
does not have to invoke the controller or know about it.   But that just
encapsulates the line in a publish/subscribe system somewhere.   (Not
that there's anything wrong with that.)

To me it seems most straightforward to think in terms of where code
runs, something like so:

- A "programming tool" consists of or produces
(creates/generates/combines/links) code to run within some sliver.  
Different component/sliver technologies might come with their own
programming tools or libraries of off-the-shelf programs or modules
specific to those kinds of slivers.   That's one reason why the
distinction seems useful: so we can talk about programming tools for
specific kinds of components or slivers.  (Of course, the programming
tool itself could run anywhere.)

- An "experiment tool" is a tool that manages a slice on the
experimenter's behalf, but does not run within a sliver or produce code
that runs within a sliver.   That sort of makes sense...certainly a
slice can't pull itself up by its own bootstraps, i.e., the code that
initially sets up a slice can't run within any of the slice's slivers.  
Experiment tools should strive to be general across many
component/sliver technologies, although they will have to "talk about"
various component/sliver technologies, i.e., they might have to produce
and consume various metadata or representations.

I emphasize again that it's not a bright line.  I could imagine an
experiment tool that does some management or control that is specific to
a particular kind of sliver and/or comes with an agent that runs within
specific sliver types.  And we all want long-running slices to be
self-monitoring and self-managing, so there could be families of tools
for programming some kind of reflection into slices---what are they?  
One reason to draw the line is to make it more interesting to cross it.

Jeff


_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[John Wroclawski]

I agree there's a distinction in here somewhere, but I think it's
more functional than where code runs. DETER has an experiment control
tool that closely matches Jeff's last paragraph - implemented as a
distributed agent system (agents generally run in slivers, could also
run "under" them) with a front end (runs outside silvers) and some
other outside-sliver mechanism. Much of the tool is common to any
technology; some of the agents would be technology-specific. Etc.

So the brightest line around seems to me to be Vic's "program slivers
or to manage experiments" - all else is -awfully- flexible..

--john

At 4:42 PM -0500 2/23/09, Jeff Chase wrote:

>It seems we all agree that there is a useful distinction between
>"programming tools" and "experimenter tools" for a slice.  And we're
>still trying to pin down the exact nature of the distinction.
>
>IMHO, we shouldn't work too hard to make the definition precise, because
>there is no bright line.  There may be tools that cross whatever line we
>draw.   This definition seems pretty good:
>
>Robert P Ricci wrote:
>>  ...
>>  An experimenter tool *sets up* the environment in which a system under
>>  test runs, a programming tool is (may be?) *part* of that environment.
>>
>>  More specific, but maybe not quite right:
>>
>>  A programming tool is one that is in the "codepath" (broadly defined) of
>>  the system under test - that is, it's directly involved in the
>>  experiment because my application calls it as a library, makes syscalls
>>  into it, sends packets through it, etc. An experimenter tool is outside
>>  this codepath in that the interaction is one way - the experimenter tool
>>  might "call into" the SUT to start it, modify its configuration, etc,
>>  but the SUT doesn't "call into" the experimenter tool. (And in the case
>>  of collaboration tools and similar things, the tool may never really
>>  interact with the SUT at all.)
>>
>>
>But that almost suggests that the System Under Test cannot notify the
>experimenter tools (e.g., slice controller) about events within the
>SUT.  We can dodge that by saying that the SUT exposes/publishes events
>and the external controller subscribes to those events, so that the SUT
>does not have to invoke the controller or know about it.   But that just
>encapsulates the line in a publish/subscribe system somewhere.   (Not
>that there's anything wrong with that.)
>
>To me it seems most straightforward to think in terms of where code
>runs, something like so:
>
>- A "programming tool" consists of or produces
>(creates/generates/combines/links) code to run within some sliver.
>Different component/sliver technologies might come with their own
>programming tools or libraries of off-the-shelf programs or modules
>specific to those kinds of slivers.   That's one reason why the
>distinction seems useful: so we can talk about programming tools for
>specific kinds of components or slivers.  (Of course, the programming
>tool itself could run anywhere.)
>
>- An "experiment tool" is a tool that manages a slice on the
>experimenter's behalf, but does not run within a sliver or produce code
>that runs within a sliver.   That sort of makes sense...certainly a
>slice can't pull itself up by its own bootstraps, i.e., the code that
>initially sets up a slice can't run within any of the slice's slivers.
>Experiment tools should strive to be general across many
>component/sliver technologies, although they will have to "talk about"
>various component/sliver technologies, i.e., they might have to produce
>and consume various metadata or representations.
>
>I emphasize again that it's not a bright line.  I could imagine an
>experiment tool that does some management or control that is specific to
>a particular kind of sliver and/or comes with an agent that runs within
>specific sliver types.  And we all want long-running slices to be
>self-monitoring and self-managing, so there could be families of tools
>for programming some kind of reflection into slices---what are they?
>One reason to draw the line is to make it more interesting to cross it.
>
>Jeff
>
>
>_______________________________________________
>services-wg mailing list
>services-wg@...
>http://lists.geni.net/mailman/listinfo/services-wg

_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg

[Robert P Ricci]

Yeah, I agree with John on this one, where it runs isn't the right place
to make the distinction.

As an example, Emulab offers a lot of tools that are clearly
experimenter tools - they install software, help launch processes,
distribute events, etc. We have a portal that lets you create slices on
PlanetLab using Emulab, and one of the things that comes along is all
these experimenter tools. They do exactly what you describe below, and
bootstrap themselves inside of the slice. They get installed by
'pushing' them in from outside the slice, and much of the control
infrastructure resides outside, in Emulab, but the tools themselves
pretty clearly run inside the slice.

Thus spake Jeff Chase on Mon, Feb 23, 2009 at 04:42:42PM -0500:
> But that almost suggests that the System Under Test cannot notify the
> experimenter tools (e.g., slice controller) about events within the
> SUT.

Yeah, this is definitely a weakness with this definition, but maybe it
points out the place (roughly) where something crosses from a pure
experimenter tool and starts blurring into a programming tool.

> - An "experiment tool" is a tool that manages a slice on the
> experimenter's behalf, but does not run within a sliver or produce code
> that runs within a sliver.   That sort of makes sense...certainly a
> slice can't pull itself up by its own bootstraps, i.e., the code that
> initially sets up a slice can't run within any of the slice's slivers.  
> Experiment tools should strive to be general across many
> component/sliver technologies, although they will have to "talk about"
> various component/sliver technologies, i.e., they might have to produce
> and consume various metadata or representations.

--
/-----------------------------------------------------------
| Robert P Ricci <ricci@...> | <ricci@...>
| Research Associate, University of Utah Flux Group
| www.flux.utah.edu | www.emulab.net
\-----------------------------------------------------------

_______________________________________________
services-wg mailing list
services-wg@...
http://lists.geni.net/mailman/listinfo/services-wg