"Looking back" within a singe vector

[babelproofreader]

I have a single vertical vector containing several thousand entries and what I would like to do is loop through the vector and calculate various "moving" calculations, e.g. at v(100,1) I would like to calculate a value for v(80,1) to v(100,1) inclusive, the calculations typically being averages, standard deviations, maximums and minimums and Fisher transformations. Once done, move forward and repeat for v(101,1) etc. A further complication arises in that the length of the look back period varies according to data contained in a second vector. Can anyone suggest what functions I should be looking at to complete such a task?

[Francesco Potortì]

>I have a single vertical vector containing several thousand entries and what
>I would like to do is loop through the vector and calculate various "moving"
>calculations, e.g. at v(100,1) I would like to calculate a value for v(80,1)
>to v(100,1) inclusive, the calculations typically being averages

average is a linear operation, the simplest way is using the 'filter'
function, or else you can use 'cumsum'.

>standard
>deviations, maximums and minimums and Fisher transformations.

These are not linear, so I do not see how you can use 'filter' here.

For standard deviation you can first square the vector, then do the same
as for average.

I do not have any ideas for the others.  In general, however, provided
you have enough memory, you can build a matrix like this (untested):
 M = repmat(V,21,1);
 for ii=2:21 M(ii,:) = shift(M(ii,:),1-ii); endfor
 M = M(:,1:end-20);
and then apply your transformations to M's columns.

>A further complication arises in
>that the length of the look back period varies according to data contained
>in a second vector.

All the above suggestions assume the the lookback period is constant.
However, if 20 is the maximum lookback period, you can build a matrix M
as in the last example, then invalidate an initial number of samples in
each column depending on the lookback period before applying your
transformations.  One way of doing this is to set them the invalid
values to NA and then apply your transformations to the columns using
the 'nanmean' and similar functions form the statistics package or the
'mean' and similar functions from the nan package.

All the above is very terse and untested, so ask for more info if this
isnot clear.

--
Francesco Potortì (ricercatore)        Voice: [phone] (op.2111)
ISTI - Area della ricerca CNR          Fax:   [phone]
via G. Moruzzi 1, I-56124 Pisa         Email: Potorti@...
(entrance 20, 1st floor, room C71)     Web:   http://fly.isti.cnr.it/
_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[Søren Hauberg]

lør, 14 03 2009 kl. 10:19 -0700, skrev babelproofreader:
> the calculations typically being averages, standard
> deviations, maximums and minimums and Fisher transformations.

Francesco answered the part about averages and standard deviations.
Minimums and maximums can be calculated with the 'ordfilt2' function in
the 'image' package. It is designed for images (i.e. matrix data) but
should also work with vectors.

Søren

_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[Jaroslav Hajek-2]

On Sat, Mar 14, 2009 at 6:19 PM, babelproofreader
<babelproofreader@...> wrote:

>
> I have a single vertical vector containing several thousand entries and what
> I would like to do is loop through the vector and calculate various "moving"
> calculations, e.g. at v(100,1) I would like to calculate a value for v(80,1)
> to v(100,1) inclusive, the calculations typically being averages, standard
> deviations, maximums and minimums and Fisher transformations. Once done,
> move forward and repeat for v(101,1) etc. A further complication arises in
> that the length of the look back period varies according to data contained
> in a second vector. Can anyone suggest what functions I should be looking at
> to complete such a task?

Unless you can and want to reuse parts of the computations as the
slice progresses, you can do:
v = ... your data...
ilb = ... look back indices ....

slice = cell (1, length (v));
for i = 1:n
  slice{i} = v(ilb(i):i);
endfor

and now use cellfun to apply whatever reductions you want to the
slices, e.g. sum, mean, max...
It would be nice if the loop could be vectorized, but I don't see a
way unless we write a new compiled function. It can be done with
cellfun and an anonymous function, but I don't think it will be
faster.
Anyway, with just several thousand data it runs quite fast on my machine.
The nice advantage of this approach is that with Octave 3.2, you'll
find out that the slices are not actually copied unless you attempt to
modify them; the "slice" cell array will just hold references to the
vector. This will make it memory efficient, especially if the slices
can grow large.

hth

--
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[Jaroslav Hajek-2]

On Sun, Mar 15, 2009 at 9:19 AM, Jaroslav Hajek <highegg@...> wrote:

> On Sat, Mar 14, 2009 at 6:19 PM, babelproofreader
> <babelproofreader@...> wrote:
>>
>> I have a single vertical vector containing several thousand entries and what
>> I would like to do is loop through the vector and calculate various "moving"
>> calculations, e.g. at v(100,1) I would like to calculate a value for v(80,1)
>> to v(100,1) inclusive, the calculations typically being averages, standard
>> deviations, maximums and minimums and Fisher transformations. Once done,
>> move forward and repeat for v(101,1) etc. A further complication arises in
>> that the length of the look back period varies according to data contained
>> in a second vector. Can anyone suggest what functions I should be looking at
>> to complete such a task?
>
>
> Unless you can and want to reuse parts of the computations as the
> slice progresses, you can do:
> v = ... your data...
> ilb = ... look back indices ....
>
> slice = cell (1, length (v));
> for i = 1:n
>  slice{i} = v(ilb(i):i);
> endfor
>
> and now use cellfun to apply whatever reductions you want to the
> slices, e.g. sum, mean, max...
> It would be nice if the loop could be vectorized, but I don't see a
> way unless we write a new compiled function. It can be done with
> cellfun and an anonymous function, but I don't think it will be
> faster.

Update: realizing that it's a nice generalization of mat2cell (with a
vector) and I may have uses for it myself, I created a "cellslices"
function that can replace a loop of the form
for i = 1:n
  s{i} = x(lb(i):ub(i));
endfor

So you can write your example like this:
v = ... your data...
ilb = ... look back indices ....

# create slices
slices = cellslices (v, ilb, 1:length (v));

# get sums (this would be faster using cumsum, though possibly less accurate)
sums = cellfun (@sum, slices);
# get maxs (this can't be done with cummax, though)
maxs = cellfun (@max, slices);
# get mins, or whatever else you wish...

enjoy!

--
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave