Adding some new transformations to gnuplot (arctan and power laws)

[Dean Foster]

Hello people;

I'm new to the gnuplot mailing list, so I don't know the ropes around
here.  I've been working on adding some other transformations to
gnuplot.  I needed them for some research I was doing, and thought
they might be of general interest.  The idea is to have a variety of
ways to transform the axis.  Currently gnuplot supports linear and
log.  I've written some code that will do arctan and a variety of
power transformations.  So I'd be happy to submit these back to
gnuplot, but haven't ever done something like that.  Since I haven't
seen any patches being sent on this news group so far, I'm not sure
what the style is either.

As an example of the type of plot you can make using these
transformations, I made a graph for the wikipedia:

http://en.wikipedia.org/wiki/Law_of_the_iterated_logarithm

It could have been made using the existing version of gnuplot by
precomputing the axis and then just adding artifical tics as needed.
But the advantage of my code is that you can try different
transformations in real time by clicking on the graph and / or using
command line changes along with replot.  I've found this to be quite
fun actually.  Kind of sad what statisticians think as "fun."  :-)

So please let me know how I should proceed.

thanks,

dean


p.s.  I tried writing up documentation, but I couldn't seem to make
any changes that ever made it back into my local version of gnuplot.
It appears I deeply don't understand texi/doc.  So for a starter,
which file should I be editing?





--
Statistics, Wharton, U Penn
215 898 8233
http://gosset.wharton.upenn.edu/~foster/

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Hans-Bernhard Bröker-2]

Dean Foster wrote:
> I've been working on adding some other transformations to
> gnuplot.

I'm afraid you could have saved yourself most of the effort.  There's
already a patch out there doing that.  See

http://sourceforge.net/tracker/?func=detail&aid=1757226&group_id=2055&atid=302055

 > Since I haven't seen any patches being
> sent on this news group so far, I'm not sure what the style is
> either.

Had you investigated that earlier, you might have stumbled over the
patches tracker the above link points into...

> p.s.  I tried writing up documentation, but I couldn't seem to make
> any changes that ever made it back into my local version of gnuplot.
> It appears I deeply don't understand texi/doc.  So for a starter,
> which file should I be editing?

docs/gnuplot.doc.  And you'll have to either 'make install' or set the
GNUHELP environment variable for it to locate the freshly built version.
  While working in the source tree in a Unix-like environment, it helps
to have a little script in gnuplot/src that goes like this:

#! /bin/sh
#GNUTERM=x11
GNUHELP=../docs/gnuplot.gih
GNUPLOT_DRIVER_DIR=.
if [ "$1" = "-d" ] ; then
   GDB=gdb
   if [ -n "$2" ] ; then
     GDB="$GDB --args"
   fi
   GNUTERM=x11
   shift
fi
#export GNUTERM GNUHELP GNUPLOT_DRIVER_DIR
export GNUHELP GNUPLOT_DRIVER_DIR
if [ -n "$1" ] ; then
   PATH=.:$PATH GNUPLOT_LIB=../demo:../../demo $GDB ./gnuplot "$@"
else
   PATH=.:$PATH GNUPLOT_LIB=../demo:../../demo $GDB ./gnuplot
fi
=== end of file ===

This runs the fresh build of gnuplot without installing it first,
referring to all new files.  The script is an extended version of how
"make check" target in the demo subdirectory works.


------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Ethan Merritt]

On Saturday 23 May 2009, Hans-Bernhard Bröker wrote:
> Dean Foster wrote:
> > I've been working on adding some other transformations to
> > gnuplot.
>
> I'm afraid you could have saved yourself most of the effort.  There's
> already a patch out there doing that.  See
>
> http://sourceforge.net/tracker/?func=detail&aid=1757226&group_id=2055&atid=302055

True.  But that patch repeats what I consider to be a design error
in how the transform is implemented.  Like the current log scale code,
it transforms the data as it is read in, and stores only the transformed
data.  That limits what you can do interactively with a plot that is
already on the screen.  I would much rather see a patch that rearranges
the order of operations so that the original, untransformed, data is
stored internally, and the transform is applied at the time of
mapping it to plot coordinates.  Some advantages that I see are:

- There is no loss of precision as you toggle the transform on/off
  interactively.  The exact original data is always available.
  NB: This was less an issue back when the log/unlog code was
  first written, because replotting necessary required re-reading
  from the original data file.  But now we can refresh/replot the
  graph directly from the internal copy, it is an extra limitation
  if you have to go back to the data file.

- An extreme case of the above occurs when the transform is toggled
  in an external viewer module rather than in the gnuplot core. This
  is partially implemented for the canvas terminal driver, but it
  would be a whole lot easier if the _untransformed_ coordinates were
  sent to the external viewer, along with the transform and its inverse.

- The code would be a lot cleaner if the transform code was entirely
  localized to the coordinate mapping functions map_position() and
  map_3dposition().  Right now we have extra code scattered all over
  the place, and extra fields in various data structures, just to
  handle the special case of data that is stored as log(data).
  As currently written, all this special case code would need to be
  extended for each new transform.  Conversely, if the transform were
  localized to a single set of map/unmap routines, virtually all of the
  special case code could go away and data structures could go away.

- The axis tic placement code for log-scale tics is complex, and is
  the source of some of our longest-standing bug reports.  I particularly
  like the idea of being able to avoid a similar mess in implementing
  new transforms.  I suspect that simplifying the tic placement code would
  all by itself be sufficient reason for storing untransformed data
  internally.

- A generalized transform/untransform step in the coordinate mapping would
  make it trivial to handle arbitrary transforms provided by the user.

So I would very much welcome any patches the work towards implementing
a generalized framework for applying an arbitrary transform at the
stage of coordinate mapping. Once that is in place, the log/unlog
transform becomes just one special case and most (all?) of the special
case code can go away.

>  > Since I haven't seen any patches being
> > sent on this news group so far, I'm not sure what the style is
> > either.

Please submit patches via the SourceForge gnuplot project site.

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[James R. Van Zandt]

Dean Foster <dean.foster@...> wrote:

> I've been working on adding some other transformations to
> gnuplot.  I needed them for some research I was doing, and thought
> they might be of general interest.  The idea is to have a variety of
> ways to transform the axis.  Currently gnuplot supports linear and
> log.

I've been looking for that functionality for some time, and I wrote
the previous version.

The patch Hans-Bernhard Broeker and Ethan Merritt pointed out
integrated a single added transform (the "probability" transform, that
makes normally distributed values appear on a straight line) into
gnuplot.  I followed the philosophy of changing the existing structure
as little as possible (AKA "do no harm").  However, Ethan thought that
was perpetuating a design error, and would rather fix the design
first.  I agree the design he describes would be better.  However,
nobody has made progress on it.

Although my patch only implements the probability transform, I spent
a fair amount of time on automatic placement of tic marks and labels
for a general transform, based on the following principles:

 - Place no tics or minitics outside the range specified by the user.
 - Don't allow tic labels to overlap.
 - Choose the "simplest" tics to label (i.e. those with the fewest
   significant digits).
 - Subject to the above, place tics with approximately even density.
 - Ensure the value corresponding to each minitic is obvious.
 - Subject to the above, place enough minitics that the user can
   easily interpolate to find the value for any point on the graph.

Several simple approaches failed, but eventually I got it working
fairly well.  You can see a couple of examples at
http://jrv.oddones.org/ .  For more examples, download
http://jrv.oddones.org/transform-1.0.tar.gz, unpack, and build with
"make plots".

The code generates a piecewise rational fit to the transform
function, so it can be used where there is no closed form for the
inverse transform.

One issue I had not solved has to do with limits.  For a log
transform, data points that are zero or negative should be dropped.
For a probability transform, data points outside (0,1) should be
dropped.  How should this be handled for a user-supplied transform?
Do we ask the user to supply the appropriate limits?  (Do we trust
him?)  Do we try to evaluate the transform, and check for NANs?

I'd look forward to working on this.

            - Jim Van Zandt

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Philipp K. Janert]

I'd like to add one thing to Ethan's list: The current
log-scale does not play well with the "smooth"
algorithms, such as "smooth frequency". I have
not analyzed the root cause of this behavior in
detail, but I can certainly see how it comes about
given the entire complex of issues that Ethan has
scoped in his email.

So, there is yet another reason why it would be
nice to be able to apply transformations for display
purposes as late in the game as possible, so that
earlier operations on the data (such as "smooth freq"
or "smooth kdens") succeed.

Best,

                Ph.



On Saturday 23 May 2009 09:40:49 am Ethan Merritt wrote:

> On Saturday 23 May 2009, Hans-Bernhard Bröker wrote:
> > Dean Foster wrote:
> > > I've been working on adding some other transformations to
> > > gnuplot.
> >
> > I'm afraid you could have saved yourself most of the effort.  There's
> > already a patch out there doing that.  See
> >
> > http://sourceforge.net/tracker/?func=detail&aid=1757226&group_id=2055&ati
> >d=302055
>
> True.  But that patch repeats what I consider to be a design error
> in how the transform is implemented.  Like the current log scale code,
> it transforms the data as it is read in, and stores only the transformed
> data.  That limits what you can do interactively with a plot that is
> already on the screen.  I would much rather see a patch that rearranges
> the order of operations so that the original, untransformed, data is
> stored internally, and the transform is applied at the time of
> mapping it to plot coordinates.  Some advantages that I see are:
>
> - There is no loss of precision as you toggle the transform on/off
>   interactively.  The exact original data is always available.
>   NB: This was less an issue back when the log/unlog code was
>   first written, because replotting necessary required re-reading
>   from the original data file.  But now we can refresh/replot the
>   graph directly from the internal copy, it is an extra limitation
>   if you have to go back to the data file.
>
> - An extreme case of the above occurs when the transform is toggled
>   in an external viewer module rather than in the gnuplot core. This
>   is partially implemented for the canvas terminal driver, but it
>   would be a whole lot easier if the _untransformed_ coordinates were
>   sent to the external viewer, along with the transform and its inverse.
>
> - The code would be a lot cleaner if the transform code was entirely
>   localized to the coordinate mapping functions map_position() and
>   map_3dposition().  Right now we have extra code scattered all over
>   the place, and extra fields in various data structures, just to
>   handle the special case of data that is stored as log(data).
>   As currently written, all this special case code would need to be
>   extended for each new transform.  Conversely, if the transform were
>   localized to a single set of map/unmap routines, virtually all of the
>   special case code could go away and data structures could go away.
>
> - The axis tic placement code for log-scale tics is complex, and is
>   the source of some of our longest-standing bug reports.  I particularly
>   like the idea of being able to avoid a similar mess in implementing
>   new transforms.  I suspect that simplifying the tic placement code would
>   all by itself be sufficient reason for storing untransformed data
>   internally.
>
> - A generalized transform/untransform step in the coordinate mapping would
>   make it trivial to handle arbitrary transforms provided by the user.
>
> So I would very much welcome any patches the work towards implementing
> a generalized framework for applying an arbitrary transform at the
> stage of coordinate mapping. Once that is in place, the log/unlog
> transform becomes just one special case and most (all?) of the special
> case code can go away.
>
> >  > Since I haven't seen any patches being
> > >
> > > sent on this news group so far, I'm not sure what the style is
> > > either.
>
> Please submit patches via the SourceForge gnuplot project site.
>
> ---------------------------------------------------------------------------
>--- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is
> a gathering of tech-side developers & brand creativity professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
> _______________________________________________
> gnuplot-beta mailing list
> gnuplot-beta@...
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Plotter-2]

Philipp K. Janert wrote:

> I'd like to add one thing to Ethan's list: The current
> log-scale does not play well with the "smooth"
> algorithms, such as "smooth frequency". I have
> not analyzed the root cause of this behavior in
> detail, but I can certainly see how it comes about
> given the entire complex of issues that Ethan has
> scoped in his email.
>
> So, there is yet another reason why it would be
> nice to be able to apply transformations for display
> purposes as late in the game as possible, so that
> earlier operations on the data (such as "smooth freq"
> or "smooth kdens") succeed.
>
> Best,
>
> Ph.

I agree will general comments about this should be done as late as
possible as display processing and original data kept.

However I just wondered about when smoothing should be applied.

If you have a logrithmic effect you presumably want smoothing of the log
not the data. But smooth unique may want to be done first?? This may not
be a case of one correct answer. I suspect it would need a switch to
specify when this processing is done.

smooth unique before , smooth cspline after ??

/Peter.


------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[James R. Van Zandt]

 Dean Foster <dean.foster@...> wrote:

> You solved the hard problem--namely the tics.
>
> I think there are really 4 different problems to solve here.
>
> (1)  When / where in the code to do the transformation.  (Ideally, as
> late as possible.)
>
> (2) Intelegent tics.
>
> (3) a variety of transformations of the axis itself.
>
> (4) dynamic exploration of transformations. (currently the L command
> is supported in gnuplot.)
>
> I think viewing doing (1) as a blocker for the other 3 doesn't make
> sense.  The other three are ready to be included in the existing code
> base and are all local changes.  I don't think they in any way make
> doing (1) harder.  Since the other 3 all call the same transformation
> functions, they shouldn't actually have to be changed at all if (1)
> were implemented.

As I remember, I had to touch the code in a lot of places to add one
transformation.  Refactoring the code to delay the transformation
would help with that.

A year ago, Ethan wrote:
>> I figured to leave the existing log-scale code in place, as you
>> suggested above, while adding a new more general mecanism that worked
>> on the original stored coordinates.  Once the general mechanism was
>> in place and working, the old log-scale code could be removed without
>> ever having to modify it to work  with "last-minute" scaling.

Here's my reply.  Do you think this makes sense?

> I think I see how that would work:
>
> Currently:
> Upon "set log x", all stored x values are transformed and on "unset
> log x" they are inverse transformed.
>
> Phase 1:
> Add a scaling "newlog".
>
> Add to AXIS a pointer to a function that transforms the data.  The
> function takes two parameters: the datum to be transformed, and a
> double.  For scaling "linear" or "log" the function is a no-op.  For
> "newlog", the extra parameter is the base of the logs.  Otherwise the
> extra parameter is ignored.
>
> Always do "last-minute" scaling: When plotting a value, call the
> function via the supplied pointer to transform it.
> On "set log x" or "unset prob x" etc., update the function pointer.
> On "(un)set log x", continue to (un)transform the stored data.
>
> Phase 2 (after initial checkout):
> Rename scaling "log" to "oldlog", and rename "newlog" to "log".
>
> Phase 3 (after a major release):
> Delete the "oldlog" commands and the code to (un)transform the stored
> data.
>
> The above only implements standard scalings (linear, log, probability,
> and maybe weibull).  To accommodate a user-defined scaling function,
> the scaling step needs to be able to point to an action table.  We
> could use that mechanism to implement the standard scalings too -
> building the action tables from static strings like "_linear(x,b)=x"
> or "_log(x,b)=log(x)/log(b)".  However, I'm worried that would be too
> slow.  The faster method would be to give the scaling function three
> parameters (datum, extra, and action table), and let the standard
> scaling functions ignore the third parameter.

           - Jim

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Ethan Merritt]

On Sunday 31 May 2009, James R. Van Zandt wrote:

>
> A year ago, Ethan wrote:
> >> I figured to leave the existing log-scale code in place, as you
> >> suggested above, while adding a new more general mechanism that worked
> >> on the original stored coordinates.  Once the general mechanism was
> >> in place and working, the old log-scale code could be removed without
> >> ever having to modify it to work  with "last-minute" scaling.
>
> Here's my reply.  Do you think this makes sense?
>
> > I think I see how that would work:
> >
> > Currently:
> > Upon "set log x", all stored x values are transformed and on "unset
> > log x" they are inverse transformed.
> >
> > Phase 1:
> > Add a scaling "newlog".
> >
> > Add to AXIS a pointer to a function that transforms the data.  The
> > function takes two parameters: the datum to be transformed, and a
> > double.  For scaling "linear" or "log" the function is a no-op.  For
> > "newlog", the extra parameter is the base of the logs.  Otherwise the
> > extra parameter is ignored.

I have been thinking a bit about this.  I think that the core requirement
is simply to store two function pointers for each axis: the transform and
its inverse.  The forward transform is used [only?] by the routines that
map plot coordinates to terminal coordinates.  It is conceptually easiest
to show for the routine map_position_double():

%%%%%%%%%%%%%%%%%% current code %%%%%%%%%%%%%%%%%%%%%
/*{{{  map_position_double */
static void
map_position_double(
    struct position *pos,
    double *x, double *y,
    const char *what)
{
    switch (pos->scalex) {
    case first_axes:
        {
            double xx = axis_log_value_checked(FIRST_X_AXIS, pos->x, what);
            *x = AXIS_MAP(FIRST_X_AXIS, xx);
            break;
        }
    ...
%%%%%%%%%%%%%%% proposed code %%%%%%%%%%%%%%%%%%%%%%%%%
/*{{{  map_position_double */
static void
map_position_double(
    struct position *pos,
    double *x, double *y,
    const char *what)
{
    switch (pos->scalex) {
    case first_axes:
        {
            double xx = pos->x;
            if (FIRST_X_AXIS.transform)
                xx = (FIRST_X_AXIS.transform)xx;
            *x = AXIS_MAP(FIRST_X_AXIS, xx);
            break;
        }
    ...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The transform is obviously not applied in the case statements handling
screen, character, or graph coordinates.  It only applies to the axial
coordinates.

Note: this edit by itself doesn't actually do much, because the data points
are usually plotted via a direct calls to the macros map_x() and map_y().
I made a very quick hack to test specifically a log transform on axis y2,
and found that for simple plots it was sufficient to modify the macros
AXIS_MAP(axis, variable) and AXIS_SETSCALE(axis, out_low, out_high)
in axis.h.   I'm sure there will turn out to be other places that need
tweaking as well, but this is looking quite do-able.

The inverse transform is obviously needed for mouse feedback. But it
would also be available for guiding the choice of sampling intervals
and tic placement if needed.

I am not following you about needing a second parameter for log scale.
The appearance of the plot is independent of the base of the log.
Choosing a different base would introduce a constant scale multiplier,
which is re-scaled out of existence by conversion to terminal coordinates.
 
> > Always do "last-minute" scaling: When plotting a value, call the
> > function via the supplied pointer to transform it.
> > On "set log x" or "unset prob x" etc., update the function pointer.
> > On "(un)set log x", continue to (un)transform the stored data.
> >
> > Phase 2 (after initial checkout):
> > Rename scaling "log" to "oldlog", and rename "newlog" to "log".

I suggest to introduce a new command
   {un}set transform {x|y|x2|y2|...} func(dummy)

Phase 1 is to start using this command.

Phase 2 is to trap the older command "set log y" and treat it as
"set transform y log(y)".

> > Phase 3 (after a major release):
> > Delete the "oldlog" commands and the code to (un)transform the stored
> > data.
> >
> > The above only implements standard scalings (linear, log, probability,
> > and maybe weibull).  To accommodate a user-defined scaling function,
> > the scaling step needs to be able to point to an action table.  We
> > could use that mechanism to implement the standard scalings too -
> > building the action tables from static strings like "_linear(x,b)=x"
> > or "_log(x,b)=log(x)/log(b)".  However, I'm worried that would be too
> > slow.  The faster method would be to give the scaling function three
> > parameters (datum, extra, and action table), and let the standard
> > scaling functions ignore the third parameter.

While I understand your concern about speed in evaluating the function,
I point out that we have extensive benchmarks on how much it costs.
Every time you put parentheses in a 'using' statement you  incur the cost
of action table evaluation.
     plot 'foo' using ($1):($2)
is slower than
     plot 'foo' using 1:2
And yes, there have been a small number of complaints about the speed hit,
but it is only noticeable for very large datasets.  A while back I showed
that unexpectedly, at least under linux the largest part of this hit comes
from from enabling/disabling FPE trapping around each point.
But that's a digression.

I suggest that if we implement this, and it works, then we can worry later
about further optimizing the speed.  For some common cases, e.g. log(), we
could store a pointer to the C library function rather than to an action
table.


Summary:

- I could make simple plots work using a last-minute log transform by modifying
  only 2 macros in axis.h.   This is looking quite do-able, although I would
  prefer to turn these macros back into real subroutines for readability if
  nothing else.

- Unresolved issues:
  + how to deal with singularities in the transform?  We won't hit them until
    we are in the middle of plotting.
  + how to pass the transform to an external helper (gnuplot_x11) or wrap it
    in the driver output (canvas terminal)?
  + tic placement (although I can report that my quick hack "just worked" for
    placing logscale tics on simple plots with no change to the tic code at all

                        - Ethan

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Ethan Merritt]

On Tuesday 02 June 2009 06:26:47 Dean Foster wrote:
> If all calls to points go through the function AXIS_MAP and
> AXIS_UNMAP, then it does seem that we are pretty close to being able
> to do a late change of axis.  

They are currently macros, not functions.
I do think that in order to pursue this we will have to change them
to be functions.

Yes, my earlier email with sample code equivalent to
   xx = (FIRST_X_AXIS.transform) ? (FIRST_X_AXIS.transform)(pos->x) : pos->x;
   *x = AXIS_MAP(FIRST_X_AXIS, xx);
would be further simplified by placing the test for a transform inside
AXIS_MAP() itself.

> I didn't know how to check that these
> functions were always used.  (I'm mostly a C++ programmer--so my only
> tricks of making things private to find who peeks at data doesn't work
> here.  But I'm sure someone else knows if these functions are always
> used as go-betweens.)  I can try using them for my transformation and
> see if it seems to work.

I grepped for use of ->term_scale as a convenient indicator, and found only
two other place in the core code that would be affected. One is the tic
placement code, which we've already agreed is a special case but maybe not
impossible to clean up. The other is
  graphics.c:  map_position_r()

As already mentioned, the mousing code in various terminals would need to
revised also, but that will have to be done on a case by case basis.

A few places test for the _sign_ of  (axis)->term_scale,  but so long as
the transform is monotonic that should be OK.  If it isn't monotonic, then
we have other problems starting with there being no 1-to-1 inverse transform.
I'm going to have to think about that, however.  Is it possible that allowing
non-monotonic axis transforms would be another way of generating parametric
plots?

--
Ethan A Merritt

------------------------------------------------------------------------------
OpenSolaris 2009.06 is a cutting edge operating system for enterprises
looking to deploy the next generation of Solaris that includes the latest
innovations from Sun and the OpenSource community. Download a copy and
enjoy capabilities such as Networking, Storage and Virtualization.
Go to: http://p.sf.net/sfu/opensolaris-get
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[James R. Van Zandt]

Dean Foster <dean.foster@...> writes:

> If all calls to points go through the function AXIS_MAP and
> AXIS_UNMAP, then it does seem that we are pretty close to being able
> to do a late change of axis.  I didn't know how to check that these
> functions were always used.  (I'm mostly a C++ programmer--so my only
> tricks of making things private to find who peeks at data doesn't work
> here.  But I'm sure someone else knows if these functions are always
> used as go-betweens.)  I can try using them for my transformation and
> see if it seems to work.
>
> I've thought a bit about the same difficulties that you mentioned.
> Let me share some of my thoughts:
>

Ethan Merritt wrote:
> > - Unresolved issues:
> >  + how to deal with singularities in the transform?  We won't hit them until
> >    we are in the middle of plotting.
...
> So my claim is, let the transformation deal with it and all the rest
> of the code has to do is check if the point is in range AFTER it has
> been transformed.

This works if the user has specified plot limits.  However, the
autoscaling code should also check for undefined == TRUE or a nonzero
imaginary.

Ethan Merritt wrote:
> >  + how to pass the transform to an external helper (gnuplot_x11) or wrap it
> >    in the driver output (canvas terminal)?

Surely that interface, in both directions, can be in terminal coordinates?

> >  + tic placement (although I can report that my quick hack "just worked" for
> >    placing logscale tics on simple plots with no change to the tic code at all
>
> tic placement is an entirely factored from transformations.  I think
> Jim Van Zandt's code makes lovely tics.  So this is now a solved
> problem.

Thanks.  A couple of caveats, though:  

The current code for the probability transformation relies on having
an analytic inverse.  There is code in there to do a numerical inverse
(see FITVAL in transform.c), but it's currently used only for tic
placement.  

The numerical inverse works only for monotonic transforms.  I don't
even want to think about non-monotonic transforms.

Currently, tic placement is the same for horizontal and vertical axes.
It ought to account for tic label size, and allow more tic marks on
vertical axes.

> I have been thinking a bit about this.  I think that the core requirement
> is simply to store two function pointers for each axis: the transform and
> its inverse.  The forward transform is used [only?] by the routines that
> map plot coordinates to terminal coordinates.  It is conceptually easiest
> to show for the routine map_position_double():

We might go through three phases here.  (1) implement only fixed
transforms for which we have analytic inverses, so the pointer could
be to a C function.  (2) allow user-defined functions, but demand
that the user also supply the inverse function.  Here we would need a
pointer to a parse table.  (3) Allow the user to
omit the inverse function, falling back on the rational interpolation.

We could go through three similar phases for range checking: (1) only
fixed transforms, for which we can hard code the tests (as in
axis_log_value_checked() in axis.c, except we'd probably switch to
pointers to separate test functions), (2) Require the user to provide
a range checking function.  Here we would need a pointer to a parse
table.  (3) If the range checking function isn't supplied, fall back
on evaluating the transform and checking for "undefined".  However,
this would not work for functions that require limits to become
monotonic.  For example, I'd like to implement f(x)=1/x, so I could
label spectra in both energy and wavelength units.  However, the range
requirement could be either "x>0" or "x<0", and if there are both
positive and negative values in the data, the program would be hard
pressed to make the right choice.


> I suggest to introduce a new command
>    {un}set transform {x|y|x2|y2|...} func(dummy)

Well, currently we have "{un}set logscale {x|y...}" so maybe we should
use "scale" instead of (or in addition to) "transform".  E.g. for
predefined transformations:

     {un}set {log|linear|probability} scale {x|y...}

and for user supplied transformations

     {un}set user scale {x|y...} func(dummy)  
     {unscale func2(dummy)}
     {rangecheck func3(dummy)}

or maybe

     {un}set user scale func({x|y...})  
     {unscale func2({x|y...})}
     {rangecheck func3({x|y...})}

Of course "unset" means to use linear scaling.


              - Jim Van Zandt

------------------------------------------------------------------------------
OpenSolaris 2009.06 is a cutting edge operating system for enterprises
looking to deploy the next generation of Solaris that includes the latest
innovations from Sun and the OpenSource community. Download a copy and
enjoy capabilities such as Networking, Storage and Virtualization.
Go to: http://p.sf.net/sfu/opensolaris-get
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Ethan Merritt]

On Tuesday 02 June 2009, James R. Van Zandt wrote:

[snipped most of it]

> Ethan Merritt wrote:
> > >  + how to pass the transform to an external helper (gnuplot_x11) or wrap it
> > >    in the driver output (canvas terminal)?
>
> Surely that interface, in both directions, can be in terminal coordinates?

The issue is clearest for the canvas terminal, where "both directions" does not
apply.  I want to toggle between linear/logscale or linear/transform in the
javascript code on the browser.  There is no back-connection to the original
gnuplot run.  

I am arguing that it may be preferable to send terminal coordinates corresponding
to linear scale, plus some clue how they should be transformed by the viewer
upon request.  

Right now the output from gnuplot is in terminal coordinates that represent either
linear or log scale depending upon what was set at the time the plot was created.
The routine gnuplot_mouse.js knows how to back-transform the stored terminal
coordinates to give back correct mousing if log-scale was in effect.
But it doesn't know how to toggle from linear to log display of the coordinates,
whatever scale they were originally on.  It wouldn't be very hard to add that,
just tedious. I was putting it off in the hope that we might come up with a general
plan for toggling transformed data, so that I could write the routines only once
instead of having to redo them later.


------------------------------------------------------------------------------
OpenSolaris 2009.06 is a cutting edge operating system for enterprises
looking to deploy the next generation of Solaris that includes the latest
innovations from Sun and the OpenSource community. Download a copy and
enjoy capabilities such as Networking, Storage and Virtualization.
Go to: http://p.sf.net/sfu/opensolaris-get
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Plotter-2]

Ethan Merritt wrote:

> On Tuesday 02 June 2009, James R. Van Zandt wrote:
>
> [snipped most of it]
>
>> Ethan Merritt wrote:
>>>>  + how to pass the transform to an external helper (gnuplot_x11) or wrap it
>>>>    in the driver output (canvas terminal)?
>> Surely that interface, in both directions, can be in terminal coordinates?
>
> The issue is clearest for the canvas terminal, where "both directions" does not
> apply.  I want to toggle between linear/logscale or linear/transform in the
> javascript code on the browser.  There is no back-connection to the original
> gnuplot run.  
>
> I am arguing that it may be preferable to send terminal coordinates corresponding
> to linear scale, plus some clue how they should be transformed by the viewer
> upon request.  
>
> Right now the output from gnuplot is in terminal coordinates that represent either
> linear or log scale depending upon what was set at the time the plot was created.
> The routine gnuplot_mouse.js knows how to back-transform the stored terminal
> coordinates to give back correct mousing if log-scale was in effect.
> But it doesn't know how to toggle from linear to log display of the coordinates,
> whatever scale they were originally on.  It wouldn't be very hard to add that,
> just tedious. I was putting it off in the hope that we might come up with a general
> plan for toggling transformed data, so that I could write the routines only once
> instead of having to redo them later.
>
>
> ------

Hi,

I'm wondering about the rationality of this. "Toggling" log/transform
scaling is really a euphormism for a total replot it seems. You
fundementally change the x or y scale transformation and have to do a
full replot of the data, axes and all.

This is feasible in a live terminal such as wxt since gnuplot is
available to do the replotting. In the case of write-once interactive
terminals I see two possibilities:

1/ export two renditions of the graph and a trivial js routine to toggle
them. This could be similar to the line visibility toggle already
working for svg but on a larger object. I suspect the coding would be
trivial and fairly portable.

I doubt that the amount of information that would be common to both
plots would merit swapping only part of the content. Gnuplot should
provide complete plots and just let the browser toggle visibility.

This approach should be valid for any arbitary future tranformations.

or

2/ export enough information and js code for the interactive terminal to
redraw new axes, tic labels , grid, plot lines, axes labels and
possibily alter the legend. It seems that this close to the entire plot
job except parsing the input data. It is reproducing most of what
gnuplot would do. Gnuplot is no longer a plotter but a preprocessor.
This seems rather off target in terms of what gnuplot should be doing.


/Peter.






------------------------------------------------------------------------------
OpenSolaris 2009.06 is a cutting edge operating system for enterprises
looking to deploy the next generation of Solaris that includes the latest
innovations from Sun and the OpenSource community. Download a copy and
enjoy capabilities such as Networking, Storage and Virtualization.
Go to: http://p.sf.net/sfu/opensolaris-get
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[James R. Van Zandt]

I wrote
...
> > monotonic. For example, I'd like to implement f(x)=1/x, so I could
> > label spectra in both energy and wavelength units. However, the range
> > requirement could be either "x>0" or "x<0", and if there are both
> > positive and negative values in the data, the program would be hard
> > pressed to make the right choice.
>
> Actually, I view this as a "monotone" transformation.  It just happens
> to wrap at zero rather than wrap at +infinity as most monotone
> functions do.  :-)  

Actually it does not satisfy either of the monotonic requirements
   whenever x ≤ y, then f(x) ≥ f(y)
or else
   whenever x ≤ y, then f(x) ≤ f(y)

> So if you are using the transformed space and
> you have both positive and negative values, the natural graph is to
> put +infinity in the middle of the graph.   No code else where even
> has to know about this weirdness--it just sees a set of points to
> plot.

True.  And sometimes it would make sense - e.g. for the position of a
virtual image.

> I tried implementing this in my version, but there are lots of range
> checking that is done in untransformed coordinates.  This would have
> to be switched to checks done using transformed coordinates.  So my
> implementation just ducked this issue.  But I think the right answer
> is to have a plot with +infinity in the middle.  So the axis would
> look something like:
>
>  --------+-------+-----+-----+------+-------+---------+--
>          5       10    20  infty   -20     -10       -5

Yes, this could work.  With some manual customization for a particular
transform.  But it would be a challenge to get autoranging to find
this solution for a general user-specified transformation.

BTW I proposed
> We might go through three phases here.  (1) implement only fixed
> transforms for which we have analytic inverses, so the pointer could
> be to a C function.  (2) allow user-defined functions, but demand
> that the user also supply the inverse function.  Here we would need a
> pointer to a parse table.  (3) Allow the user to
> omit the inverse function, falling back on the rational interpolation.
...
> and for user supplied transformations
>
>      {un}set user scale {x|y...} func(dummy)  
>      {unscale func2(dummy)}
>      {rangecheck func3(dummy)}

I would like to generalize this to make the forward transform
optional:  

      {un}set user scale {x|y...} {func(dummy)}
      {unscale func2(dummy)}
      {rangecheck func3(dummy)}

where at least one of the forward and reverse transform must be specified.

There are cases where the inverse transform has an analytic form, but
the forward transform does not.  Arguably this is true for the
probability transform, which is defined as

           set user scale x invnorm(x)

The more "natural" way to specify this might be

           set user scale x unscale norm(x)

Likewise if you want the transform for a probability distribution
where you have a function to calculate the CDF, but not its inverse.

      - Jim Van Zandt

------------------------------------------------------------------------------
OpenSolaris 2009.06 is a cutting edge operating system for enterprises
looking to deploy the next generation of Solaris that includes the latest
innovations from Sun and the OpenSource community. Download a copy and
enjoy capabilities such as Networking, Storage and Virtualization.
Go to: http://p.sf.net/sfu/opensolaris-get
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Ethan Merritt]

On Wednesday 03 June 2009 14:15:52 plotter@... wrote:
> Ethan Merritt wrote:

> > The issue is clearest for the canvas terminal, where "both directions" does not
> > apply.  I want to toggle between linear/logscale or linear/transform in the
> > javascript code on the browser.  There is no back-connection to the original
> > gnuplot run.  
> >
> > I am arguing that it may be preferable to send terminal coordinates corresponding
> > to linear scale, plus some clue how they should be transformed by the viewer
> > upon request.  
>
> I'm wondering about the rationality of this. "Toggling" log/transform
> scaling is really a euphormism for a total replot it seems. You
> fundementally change the x or y scale transformation and have to do a
> full replot of the data, axes and all.

My goal is to come as close as feasible to replicating the capabilities of
the existing interactive terminals (win x11 wxt qt) in the browser-based
terminals (canvas svg).    Except that I've kind of given up on svg :-(
 
> This is feasible in a live terminal such as wxt since gnuplot is
> available to do the replotting. In the case of write-once interactive
> terminals I see two possibilities:
>
> 1/ export two renditions of the graph and a trivial js routine to toggle
> them. This could be similar to the line visibility toggle already
> working for svg but on a larger object. I suspect the coding would be
> trivial and fairly portable.

It's a little worse than that.  The axes, including the color palette,
can all be individually toggled.  So at least in principle you might have
to export 2^N  alternative plots.  I agree that it's unlikely in practice
that you would want more than 4 = 2^2 (y or z, colorbar)

Also you are forgetting that in order to track mouse coordinates, the
browser-side code already has to know how to back-transform the
displayed coordinates.  Since we have to provide that capability
anyhow to allow mousing, why not also use it to toggle the axis scaling?

> 2/ export enough information and js code for the interactive terminal to
> redraw new axes, tic labels , grid, plot lines, axes labels and
> possibily alter the legend. It seems that this close to the entire plot
> job except parsing the input data. It is reproducing most of what
> gnuplot would do. Gnuplot is no longer a plotter but a preprocessor.
> This seems rather off target in terms of what gnuplot should be doing.

That overstates the fraction of the total task that would be handed off
to the viewer. To handle zoom/unzoom, the code in gnuplot_mouse.js already
does most of what you list.  It's just not that much code.

--
Ethan A Merritt

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Plotter-2]

Ethan Merritt wrote:

> On Wednesday 03 June 2009 14:15:52 plotter@... wrote:
>> Ethan Merritt wrote:
>
>>> The issue is clearest for the canvas terminal, where "both directions" does not
>>> apply.  I want to toggle between linear/logscale or linear/transform in the
>>> javascript code on the browser.  There is no back-connection to the original
>>> gnuplot run.  
>>>
>>> I am arguing that it may be preferable to send terminal coordinates corresponding
>>> to linear scale, plus some clue how they should be transformed by the viewer
>>> upon request.  
>> I'm wondering about the rationality of this. "Toggling" log/transform
>> scaling is really a euphormism for a total replot it seems. You
>> fundementally change the x or y scale transformation and have to do a
>> full replot of the data, axes and all.
>
> My goal is to come as close as feasible to replicating the capabilities of
> the existing interactive terminals (win x11 wxt qt) in the browser-based
> terminals (canvas svg).    Except that I've kind of given up on svg :-(
>  

That seems like a great idea and much of that can be done relatively
easily. Transforms is tricky.

>> This is feasible in a live terminal such as wxt since gnuplot is
>> available to do the replotting. In the case of write-once interactive
>> terminals I see two possibilities:
>>
>> 1/ export two renditions of the graph and a trivial js routine to toggle
>> them. This could be similar to the line visibility toggle already
>> working for svg but on a larger object. I suspect the coding would be
>> trivial and fairly portable.
>
> It's a little worse than that.  The axes, including the color palette,
> can all be individually toggled.  So at least in principle you might have
> to export 2^N  alternative plots.  I agree that it's unlikely in practice
> that you would want more than 4 = 2^2 (y or z, colorbar)
>
> Also you are forgetting that in order to track mouse coordinates, the
> browser-side code already has to know how to back-transform the
> displayed coordinates.  Since we have to provide that capability
> anyhow to allow mousing, why not also use it to toggle the axis scaling?

Mouse coords are intrinsically low res and one way. As you already noted
yourself a one off inverse transformation is easy to do off the plot.
Making this repeatable when toggling on and off requires the original
data otherwise accumulated errors and over-sensitive parts of the
mapping will rapidly cause unacceptable error.

The most obvious log transform could quickly produce significant
distortions unless the original data was reused.

>
>> 2/ export enough information and js code for the interactive terminal to
>> redraw new axes, tic labels , grid, plot lines, axes labels and
>> possibily alter the legend. It seems that this close to the entire plot
>> job except parsing the input data. It is reproducing most of what
>> gnuplot would do. Gnuplot is no longer a plotter but a preprocessor.
>> This seems rather off target in terms of what gnuplot should be doing.
>
> That overstates the fraction of the total task that would be handed off
> to the viewer. To handle zoom/unzoom, the code in gnuplot_mouse.js already
> does most of what you list.  It's just not that much code.
>

I agree stuff like toggling lines (already done for svg) zoom (browser
std function) are very feasible. My comments were aimed at the subject
of axis tranformations. That's a different kettle of fish which, like I
said, seems to involve recoding most of what gnuplot does in javescript.

This raises another issue: it is no longer within the authors control to
asses the mathematical accuracy of the platform doing the work and to
verify the graphical output before publishing. He would have to publish
without knowing what some arbitrary platform will do to his data whilst
mashing it back and forth.


I see a lot of the interactive behaviour could be ported relatively
easily and I'd love to see this but as far as axis transformation goes I
still think you're stuck with the basic issues I outlined.


regards.

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

[Ethan Merritt]

On Wednesday 10 June 2009, plotter@... wrote:

> Ethan Merritt wrote:
> > On Wednesday 03 June 2009 14:15:52 plotter@... wrote:
> >> Ethan Merritt wrote:
> >
> >>>
> >>> I am arguing that it may be preferable to send terminal coordinates corresponding
> >>> to linear scale, plus some clue how they should be transformed by the viewer
> >>> upon request.  
> >>>
> >> I'm wondering about the rationality of this. "Toggling" log/transform
> >> scaling is really a euphormism for a total replot it seems. You
> >> fundementally change the x or y scale transformation and have to do a
> >> full replot of the data, axes and all.
>
> [snip]
>
> Mouse coords are intrinsically low res and one way. As you already noted
> yourself a one off inverse transformation is easy to do off the plot.
> Making this repeatable when toggling on and off requires the original
> data otherwise accumulated errors and over-sensitive parts of the
> mapping will rapidly cause unacceptable error.

Yes, well, that was exactly what I was proposing: send the untransformed
coordinates, and do all the transforms on the fly from the original
full-resolution coordinates.
 
> The most obvious log transform could quickly produce significant
> distortions unless the original data was reused.

Exactly.  So I think we are in complete agreement.

As it may not be obvious, let me point out that the canvas terminal,
like the cairo terminals, works with much higher precision coordinates
than the actual pixel display. The nominal resolution of the scripted
plot commands is currently 10x that of the display, but that could be
further increased if desired. The only downside is that the output script
would increase in size by, say, one decimal character per coordinate.

        Ethan

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
gnuplot-beta mailing list
gnuplot-beta@...
https://lists.sourceforge.net/lists/listinfo/gnuplot-beta