plotting multiple runs as graph

[Patrick Ohly-2]

Hello,

I have imported data from multiple runs and want to plot them in
"style=graphs". The problem is that data from all runs is merged into
one source so that gnuplot draws a connecting line between unrelated
data.

My current query looks like this:

  <parameter id="p.time" show="auto">
  <parameter id="p.host" show="filter">
  <parameter id="p.peer" show="filter">

  <source id="src.offset">
    <result>time_offset</result>
   
    <input>p.time</input>
    <input>p.host</input>
    <input>p.peer</input>
  </source>

  <output target="gnuplot" format="screen" type="graphs">
    <input sweep_resolve="extern">src.offset</input>
    <option>xdata time</option>
    <option>timefmt "%H:%M:%S"</option>
    <option>format x "%H:%M:%S"</option>
    <filename unify="filter">figures/offsets</filename>
  </output>

Is there a way to separate the different runs in the graph? In gnuplot
itself this would be done by inserting a blank line in the data file,
but how can perfbase instructed (or patched, if necessary) to insert
that?

I experimented with
  <operator id="op.ridx" type="runindex">
    <input>src.offset</input>
  </operator>
and plotting that but ended up with a plot of the run index instead of
the offset...

--
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Joachim Worringen-4]

Patrick Ohly schrieb:

> Hello,
>
> I have imported data from multiple runs and want to plot them in
> "style=graphs". The problem is that data from all runs is merged into
> one source so that gnuplot draws a connecting line between unrelated
> data.
>
> My current query looks like this:
>
>   <parameter id="p.time" show="auto">
>   <parameter id="p.host" show="filter">
>   <parameter id="p.peer" show="filter">
>
>   <source id="src.offset">
>     <result>time_offset</result>
>    
>     <input>p.time</input>
>     <input>p.host</input>
>     <input>p.peer</input>
>   </source>
>
>   <output target="gnuplot" format="screen" type="graphs">
>     <input sweep_resolve="extern">src.offset</input>
>     <option>xdata time</option>
>     <option>timefmt "%H:%M:%S"</option>
>     <option>format x "%H:%M:%S"</option>
>     <filename unify="filter">figures/offsets</filename>
>   </output>
>
> Is there a way to separate the different runs in the graph? In gnuplot
> itself this would be done by inserting a blank line in the data file,
> but how can perfbase instructed (or patched, if necessary) to insert
> that?

This type of query (better: what you want it to do) does not align with
the design of perfbase conc. the <source> element: it does provide a
stream ("data vector") which contains all parameter and result values
(and run indices) of the data sets that match the specified filtering on
the parameters. This means, it is a single stream of data, no gaps or
something between different runs. Only the embedded run indices show
which run each dataset comes from.

You want to have sort of a "sweep" across the run indices of all runs
that match your filters, like it is possible to do for any paramter.
This is not possible with the current version of perfbase. What is
needed to achieve this is some operator which creates N output data
vectors from one input data vector by filtering the run indices. This is
not trivial to achieve as the query object graph would have to change on
query runtime, while it is currently created once before the actual
query is performed.

As a workaround, you could to the following: create a parameter
(occurrence="once") to which you assign a unique id when importing the
data (i.e. the number of seconds date %s if you can ensure a big enough
gap between each import operation). You can then sweep across this
parameter and get one data vector per run.

> I experimented with
>   <operator id="op.ridx" type="runindex">
>     <input>src.offset</input>
>   </operator>
> and plotting that but ended up with a plot of the run index instead of
> the offset...
>

Yes, the 'runindex' operator provides the runindex for each datapoint.
This is mostly used in combination with the 'param' operator, which
provides the value of a parameter for a runindex. See test/runindex for
an application of this. But this won't help with your problem.

  Joachim

--
Joachim Worringen, Software Architect, Dolphin Interconnect Solutions
phone ++49/(0)228/324 08 17 - http://www.dolphinics.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Patrick Ohly-2]

On Wed, 2007-03-14 at 20:50 +0100, Joachim Worringen wrote:

> Patrick Ohly schrieb:
> > Is there a way to separate the different runs in the graph? In gnuplot
> > itself this would be done by inserting a blank line in the data file,
> > but how can perfbase instructed (or patched, if necessary) to insert
> > that?
>
> This type of query (better: what you want it to do) does not align with
> the design of perfbase conc. the <source> element: it does provide a
> stream ("data vector") which contains all parameter and result values
> (and run indices) of the data sets that match the specified filtering on
> the parameters. This means, it is a single stream of data, no gaps or
> something between different runs. Only the embedded run indices show
> which run each dataset comes from.

I was hoping that the run indices would remain attached to the datasets
as long as possible while moving through the pipeline, so that when they
reach the <output> stage the code which prepares the data for gnuplot
can insert the empty lines that gnuplot needs.

> You want to have sort of a "sweep" across the run indices of all runs
> that match your filters, like it is possible to do for any paramter.

Only "sort of", because with true sweeps across the runs I would have
distinct plots with different coloring and separate entries in the graph
legend. With "sweep_resolve=extern" they would even end up in entirely
different graphs.

> As a workaround, you could to the following: create a parameter
> (occurrence="once") to which you assign a unique id when importing the
> data (i.e. the number of seconds date %s if you can ensure a big enough
> gap between each import operation). You can then sweep across this
> parameter and get one data vector per run.

Wouldn't this duplicating the already existing pb_run_index? My input
uses separators to increment the index; this would be hard to reproduce
using a occurrence="once" parameter. Incrementing on each match might
work, but still is a duplication of already existing functionality.

I wonder whether it wouldn't be simpler to expose "pb_run_index" as a
special parameter name. I had tried that, but it was rejected by the
input sanity check (due to not being a parameter declared as part of the
experiment).

Hmm, yes, simply overriding that check and providing default values for
"pb_run_index" works. The attached patch demonstrates what I mean in a
rather crude way (not sure about the "integer" type). With it I can
specify a sweep across pb_run_index, like this:

  <parameter id="p.run" show="filter">
    <value>pb_run_index</value>
    <sweep>
      <filter><equal>4</equal></filter>
      <filter><equal>5</equal></filter>
      <filter><equal>6</equal></filter>
    </sweep>
  </parameter>

    ...
    <input>p.run</input>
  </source>

This is not quite what I need because as I said above, I end up with
three plots in one graph, but it is close - if I disable the legend it
is actually quite usable. Note quite so nice is that the sweep has to be
hard-coded in the input file.

Still, passing the run index through to output stage and there using it
to separate elements looks like a better approach to me.

--
Best Regards

Patrick Ohly
Senior Software Engineer

Intel GmbH
Software & Solutions Group                
Hermuelheimer Strasse 8a                  Phone: [phone]
50321 Bruehl                              Fax: [phone]
Germany

Intel GmbH, Dornacher Strasse 1, 85622 Feldkirchen/Muenchen Germany
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
Registergericht: Muenchen HRB 47456
Ust.- IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt (BLZ 502 109 00) 600119052

[pb_run_index.patch]

---

*** /home/pohly/src/perfbase/bin/pb_query_classes.py 2007-03-15 08:37:45.680351000 +0100
--- /Projects/software/perfbase/bin/pb_query_classes.py 2007-03-15 13:57:02.091697000 +0100
***************
*** 492,509 ****
          alias = mk_label(get_attribute(val_node, self.p_name, 'alias', self.p_name, None),
                           all_nodes)
 
!         sqlexe(crs, "SELECT * FROM exp_values WHERE name = %s AND NOT is_result", None, (self.p_name, ))
!         if crs.rowcount == 0:
!             raise SpecificationError, "<value> '%s' is not a parameter in the specified experiment" % self.p_name
!         nim = build_name_idx_map(crs)
!         db_row = crs.fetchone()
!         self.val_type = get_value_type(db_row, nim)
!         self.is_constant = False
!         if db_row[nim['only_once']] > 0:
!             self.is_constant = True
!         p_unit = db_row[nim['data_unit']]
!         if not p_unit:
              p_unit = ""
          crs.close()
 
          self.showstyle = get_attribute(node, self.p_name, 'style', 'full',
--- 492,515 ----
          alias = mk_label(get_attribute(val_node, self.p_name, 'alias', self.p_name, None),
                           all_nodes)
 
!         if self.p_name != "pb_run_index":
!             sqlexe(crs, "SELECT * FROM exp_values WHERE name = %s AND NOT is_result", None, (self.p_name, ))
!             if crs.rowcount == 0:
!                 raise SpecificationError, "<value> '%s' is not a parameter in the specified experiment" % self.p_name
!             nim = build_name_idx_map(crs)
!             db_row = crs.fetchone()
!             self.val_type = get_value_type(db_row, nim)
!             self.is_constant = False
!             if db_row[nim['only_once']] > 0:
!                 self.is_constant = True
!             p_unit = db_row[nim['data_unit']]
!             if not p_unit:
!                 p_unit = ""
!         else:
!             self.val_type = "number"
!             self.is_constant = False
              p_unit = ""
+             db_row = ( "", "", "integer", "integer", "", "", "run index" )
          crs.close()
 
          self.showstyle = get_attribute(node, self.p_name, 'style', 'full',
***************
*** 577,584 ****
--- 583,593 ----
              self.sql_filter = self._parse_filter(sweep_filter, self.p_name, alias, p_unit)
              self.name += pb_sweep_suffix + str(idx)
 
+
          # SQL name, output name, pb and sql data unit, physical unit, synopsis and filter
          self.param_info = [self.p_name, alias, db_row[2], db_row[3], db_row[4], db_row[6], self.filter_str]
+
+         print self.param_info
         
          return
             

---

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Evgeny Kagan]

How can I  unsubscribe from these mails?

Thank You

-----Original Message-----
From: Patrick Ohly [mailto:patrick.ohly@...]
Sent: Thursday, March 15, 2007 3:10 PM
To: users@...
Subject: Re: [perfbase-users] plotting multiple runs as graph

On Wed, 2007-03-14 at 20:50 +0100, Joachim Worringen wrote:

> Patrick Ohly schrieb:
> > Is there a way to separate the different runs in the graph? In
> > gnuplot itself this would be done by inserting a blank line in the
> > data file, but how can perfbase instructed (or patched, if
> > necessary) to insert that?
>
> This type of query (better: what you want it to do) does not align
> with the design of perfbase conc. the <source> element: it does
> provide a stream ("data vector") which contains all parameter and
> result values (and run indices) of the data sets that match the
> specified filtering on the parameters. This means, it is a single
> stream of data, no gaps or something between different runs. Only the
> embedded run indices show which run each dataset comes from.

I was hoping that the run indices would remain attached to the datasets
as long as possible while moving through the pipeline, so that when they
reach the <output> stage the code which prepares the data for gnuplot
can insert the empty lines that gnuplot needs.

> You want to have sort of a "sweep" across the run indices of all runs
> that match your filters, like it is possible to do for any paramter.

Only "sort of", because with true sweeps across the runs I would have
distinct plots with different coloring and separate entries in the graph
legend. With "sweep_resolve=extern" they would even end up in entirely
different graphs.

> As a workaround, you could to the following: create a parameter
> (occurrence="once") to which you assign a unique id when importing the

> data (i.e. the number of seconds date %s if you can ensure a big
> enough gap between each import operation). You can then sweep across
> this parameter and get one data vector per run.

Wouldn't this duplicating the already existing pb_run_index? My input
uses separators to increment the index; this would be hard to reproduce
using a occurrence="once" parameter. Incrementing on each match might
work, but still is a duplication of already existing functionality.

I wonder whether it wouldn't be simpler to expose "pb_run_index" as a
special parameter name. I had tried that, but it was rejected by the
input sanity check (due to not being a parameter declared as part of the
experiment).

Hmm, yes, simply overriding that check and providing default values for
"pb_run_index" works. The attached patch demonstrates what I mean in a
rather crude way (not sure about the "integer" type). With it I can
specify a sweep across pb_run_index, like this:

  <parameter id="p.run" show="filter">
    <value>pb_run_index</value>
    <sweep>
      <filter><equal>4</equal></filter>
      <filter><equal>5</equal></filter>
      <filter><equal>6</equal></filter>
    </sweep>
  </parameter>

    ...
    <input>p.run</input>
  </source>

This is not quite what I need because as I said above, I end up with
three plots in one graph, but it is close - if I disable the legend it
is actually quite usable. Note quite so nice is that the sweep has to be
hard-coded in the input file.

Still, passing the run index through to output stage and there using it
to separate elements looks like a better approach to me.

--
Best Regards

Patrick Ohly
Senior Software Engineer

Intel GmbH
Software & Solutions Group                
Hermuelheimer Strasse 8a                  Phone: [phone]
50321 Bruehl                              Fax: [phone]
Germany

Intel GmbH, Dornacher Strasse 1, 85622 Feldkirchen/Muenchen Germany Sitz
der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
Registergericht: Muenchen HRB 47456
Ust.- IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ
502 109 00) 600119052

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Joachim Worringen-4]

Evgeny Kagan wrote:
> How can I  unsubscribe from these mails?

Take a look at the bottom line of each mail.

--
Joachim Worringen, Software Architect, Dolphin Interconnect Solutions
phone ++49/(0)228/324 08 17 - http://www.dolphinics.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Joachim Worringen-4]

Patrick Ohly wrote:
> I was hoping that the run indices would remain attached to the datasets
> as long as possible while moving through the pipeline, so that when they
> reach the <output> stage the code which prepares the data for gnuplot
> can insert the empty lines that gnuplot needs.

Actually, they *are* preserved as long as possible. They are not removed
until an operator creates a new value from multiple other values (say, a
"sum" reduction) in which case there is no single run_index for this new
value any more. But it was not intended to differentiate data by the run
index, only by the experiment parametes.

>> You want to have sort of a "sweep" across the run indices of all runs
>> that match your filters, like it is possible to do for any paramter.
>
> Only "sort of", because with true sweeps across the runs I would have
> distinct plots with different coloring and separate entries in the graph
> legend. With "sweep_resolve=extern" they would even end up in entirely
> different graphs.

Don't you want each run be plotted separately?

>> As a workaround, you could to the following: create a parameter
>> (occurrence="once") to which you assign a unique id when importing the
>> data (i.e. the number of seconds date %s if you can ensure a big enough
>> gap between each import operation). You can then sweep across this
>> parameter and get one data vector per run.
>
> Wouldn't this duplicating the already existing pb_run_index? My input
> uses separators to increment the index; this would be hard to reproduce
> using a occurrence="once" parameter. Incrementing on each match might
> work, but still is a duplication of already existing functionality.

Admitted - it's an ugly workaround which does not always work...

> I wonder whether it wouldn't be simpler to expose "pb_run_index" as a
> special parameter name. I had tried that, but it was rejected by the
> input sanity check (due to not being a parameter declared as part of the
> experiment).

In fact, I had already thought of exposing the "meta data" (timestamps,
author, ...) of a run in this way, making them available as a parameter
for filtering. Please go ahead! ;-)

> Hmm, yes, simply overriding that check and providing default values for
> "pb_run_index" works. The attached patch demonstrates what I mean in a
> rather crude way (not sure about the "integer" type). With it I can
> specify a sweep across pb_run_index, like this:
>
>   <parameter id="p.run" show="filter">
>     <value>pb_run_index</value>
>     <sweep>
>       <filter><equal>4</equal></filter>
>       <filter><equal>5</equal></filter>
>       <filter><equal>6</equal></filter>
>     </sweep>
>   </parameter>
>
>     ...
>     <input>p.run</input>
>   </source>
>
> This is not quite what I need because as I said above, I end up with
> three plots in one graph, but it is close - if I disable the legend it
> is actually quite usable. Note quite so nice is that the sweep has to be
> hard-coded in the input file.

Yes, this is only a partial solution for this reason. What you would
like, I think, is to specify some parameter filters and have the data of
each run that matches these filtering be processed as an individual data
vector. Right? But then again, how should the data be presented by the
output? I.e. for this case - 3 different plots in one graph, 3 different
graphs, ...? I think one single plot in one chart can be done with the
current version?

> Still, passing the run index through to output stage and there using it
> to separate elements looks like a better approach to me.

Yes, this sounds doable. The interface would be something like an
attribute to <output>: runs="merge" or runs="split". But please let us
clearify the desired output (see above).

  Joachim

--
Joachim Worringen, Software Architect, Dolphin Interconnect Solutions
phone ++49/(0)228/324 08 17 - http://www.dolphinics.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Patrick Ohly-2]

On Thu, 2007-03-15 at 14:59 -0700, Joachim Worringen wrote:

> Patrick Ohly wrote:
> > I was hoping that the run indices would remain attached to the datasets
> > as long as possible while moving through the pipeline, so that when they
> > reach the <output> stage the code which prepares the data for gnuplot
> > can insert the empty lines that gnuplot needs.
>
> Actually, they *are* preserved as long as possible. They are not removed
> until an operator creates a new value from multiple other values (say, a
> "sum" reduction) in which case there is no single run_index for this new
> value any more.

Hmm, I did't see it in debug output. Perhaps I just missed it.

> > Only "sort of", because with true sweeps across the runs I would have
> > distinct plots with different coloring and separate entries in the graph
> > legend. With "sweep_resolve=extern" they would even end up in entirely
> > different graphs.
>
> Don't you want each run be plotted separately?

No, gnuplot should treat it as one plot (= one entry in the legend, one
color and/or tick type associated with it). The improvement compared to
the current status-quo should be that in line-mode disconnected samples
should not be connected; currently they are.

> Yes, this is only a partial solution for this reason. What you would
> like, I think, is to specify some parameter filters and have the data of
> each run that matches these filtering be processed as an individual data
> vector. Right?

No, if I understand the meaning of "data vector" right this is not what
I mean. Does one "data vector" map to one gnuplot "plot" command?

>  But then again, how should the data be presented by the
> output? I.e. for this case - 3 different plots in one graph, 3 different
> graphs, ...? I think one single plot in one chart can be done with the
> current version?

One single plot, one graph, as in the current version, but without lines
connecting unrelated runs.

--
Best Regards

Patrick Ohly
Senior Software Engineer

Intel GmbH
Software & Solutions Group                
Hermuelheimer Strasse 8a                  Phone: [phone]
50321 Bruehl                              Fax: [phone]
Germany

Intel GmbH, Dornacher Strasse 1, 85622 Feldkirchen/Muenchen Germany
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
Registergericht: Muenchen HRB 47456
Ust.- IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt (BLZ 502 109 00) 600119052

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Joachim Worringen-4]

Patrick Ohly wrote:
> On Thu, 2007-03-15 at 14:59 -0700, Joachim Worringen wrote:
>> Actually, they *are* preserved as long as possible. They are not removed
>> until an operator creates a new value from multiple other values (say, a
>> "sum" reduction) in which case there is no single run_index for this new
>> value any more.
>
> Hmm, I did't see it in debug output. Perhaps I just missed it.

I have to look into it again, but this feature is i.e. used in
test/runindex (runindex operator).

>>> Only "sort of", because with true sweeps across the runs I would have
>>> distinct plots with different coloring and separate entries in the graph
>>> legend. With "sweep_resolve=extern" they would even end up in entirely
>>> different graphs.
>> Don't you want each run be plotted separately?
>
> No, gnuplot should treat it as one plot (= one entry in the legend, one
> color and/or tick type associated with it). The improvement compared to
> the current status-quo should be that in line-mode disconnected samples
> should not be connected; currently they are.

Hmm, this is a special requirement. Remember that gnuplot is not the
only output target - XML, raw text already exist, and I am adding
support for ploticus and LaTeX as time permits. But it should be possible.

>> Yes, this is only a partial solution for this reason. What you would
>> like, I think, is to specify some parameter filters and have the data of
>> each run that matches these filtering be processed as an individual data
>> vector. Right?
>
> No, if I understand the meaning of "data vector" right this is not what
> I mean. Does one "data vector" map to one gnuplot "plot" command?

In a way, yes (you can create multiple plots with one gnuplot plot
command line). Each data vector is plotted as a separate plot or graph.
What we want is to separate the elements of a single data vector by
their run indices.

As proposed, the <output> attribute 'run_indices' could have the values
'merge' (current, and default), 'single' (what you want: a single graph
- not a very intuitive name), 'graph' and 'plot' (create a separate
graph or plot for each run).

  Joachim

--
Joachim Worringen, Software Architect, Dolphin Interconnect Solutions
phone ++49/(0)228/324 08 17 - http://www.dolphinics.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Patrick Ohly-2]

On Thu, 2007-03-15 at 15:39 -0700, Joachim Worringen wrote:

> Patrick Ohly wrote:
> > On Thu, 2007-03-15 at 14:59 -0700, Joachim Worringen wrote:
> >> Actually, they *are* preserved as long as possible. They are not removed
> >> until an operator creates a new value from multiple other values (say, a
> >> "sum" reduction) in which case there is no single run_index for this new
> >> value any more.
> >
> > Hmm, I did't see it in debug output. Perhaps I just missed it.
>
> I have to look into it again, but this feature is i.e. used in
> test/runindex (runindex operator).

What I was looking at was the data passed into the gnuplot class. At
that level the runindex would be useful, but is no longer available -
the direct caller, gnuplot_output.store_data, still has access to the
full SQL table and I see a value which looks like the runindex in
db_rows[*][2].

How can I find out which column contains the runindex in that part of
the code, if any has it at all? I'd like to check whether the
segmentation of the data by runindex can be made available as meta
information to the output class.

> > No, gnuplot should treat it as one plot (= one entry in the legend, one
> > color and/or tick type associated with it). The improvement compared to
> > the current status-quo should be that in line-mode disconnected samples
> > should not be connected; currently they are.
>
> Hmm, this is a special requirement.

Indeed. I'm not sure if any of the other output targets would be able to
use this information, but at least for gnuplot line plots it would be
nice if this was done automatically.

> As proposed, the <output> attribute 'run_indices' could have the values
> 'merge' (current, and default), 'single' (what you want: a single graph
> - not a very intuitive name), 'graph' and 'plot' (create a separate
> graph or plot for each run).

I think it will be simpler to not create an additional feature which
controls whether plots are drawn in the same graph as others or in a
separate graph: instead of 'graph' and 'plot' it would be more intuitive
to expose the "run_index" as a parameter of the experiment and then use
sweeps or multiple <inputs> to get the same effect.

For the segmentation of line graphs we could add an attribute
"segmentation=["pb_run_index"|<parameter name>]" which would have the
semantic that each time this parameter changes, a new line segment is
started.

--
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Joachim Worringen-4]

[ We should move this discussion to the dev mailing list - are you
subscribed? ]

Patrick Ohly schrieb:
> What I was looking at was the data passed into the gnuplot class. At
> that level the runindex would be useful, but is no longer available -
> the direct caller, gnuplot_output.store_data, still has access to the
> full SQL table and I see a value which looks like the runindex in
> db_rows[*][2].

Yes, this is correct. The column name is '_pb_run_idx_'. In
pb_output.py:1083, you see that the first 4 columns are skipped. These
are the columns defined in pb_common.py:184-187.

> How can I find out which column contains the runindex in that part of
> the code, if any has it at all? I'd like to check whether the
> segmentation of the data by runindex can be made available as meta
> information to the output class.

_pb_run_idx_ is either > 0 if valid, or -1 if invalid (applies for data
that is is a calculuated value like sum, average, difference, ... which
does not stem from one specfic run). It's definitely possible to pass
this data down to the gnuplot class within the 'data' array (additional
dimension in this array), and either ignore or use it in the gnuplot
class. No big deal.

> Indeed. I'm not sure if any of the other output targets would be able to
> use this information, but at least for gnuplot line plots it would be
> nice if this was done automatically.

Any plot target should be able to use it, and also table targets. But
this is fully specific to the individual output target, so we can just
go ahead for gnuplot.

>> As proposed, the <output> attribute 'run_indices' could have the values
>> 'merge' (current, and default), 'single' (what you want: a single graph
>> - not a very intuitive name), 'graph' and 'plot' (create a separate
>> graph or plot for each run).
>
> I think it will be simpler to not create an additional feature which
> controls whether plots are drawn in the same graph as others or in a
> separate graph: instead of 'graph' and 'plot' it would be more intuitive
> to expose the "run_index" as a parameter of the experiment and then use
> sweeps or multiple <inputs> to get the same effect.

I agree from a software design point of view, but the usability of this
will be limited: typically, the individual run indices that you want to
see are not known. Instead, you know the content of some parameters that
you want used for filtering, and you want to handle all resulting runs
in one way or another. Thus, it will be more useful to be able to
differentiate the available runs at the "end" of the query, not at the
beginning.

BTW, there is already a way to limit a query to specific runs (<run>
element). But this does not handle these runs separately.

> For the segmentation of line graphs we could add an attribute
> "segmentation=["pb_run_index"|<parameter name>]" which would have the
> semantic that each time this parameter changes, a new line segment is
> started.

I think this is sufficient for what you want to achieve, and should be
the first step to implement.

  Joachim

--
Joachim Worringen, Software Architect, Dolphin Interconnect Solutions
phone ++49/(0)228/324 08 17 - http://www.dolphinics.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...