Rollout with 'Stop when result is accurate'

[Yoshito Takeuchi]

Hi all!

I rollout with 'Stop when result is accurate'.
But, Standard error enough under set value, non stop GnuBG rollout.
I use latest version GnuBG ( Today I compile gnubg from source code).

My setting.

Settings-rollouts

set 'Trials:' 12960
Check 'Stop when STDs are small enough'
set 'Minimum Trials:' 648
set 'Ratio|Standard Deviation/Value:' 0.0200

Thank you.

Yoshito Takeuchi


_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
http://lists.gnu.org/mailman/listinfo/bug-gnubg

[Philippe Michel-3]

On Sat, 31 Oct 2009, Yoshito Takeuchi wrote:

> I rollout with 'Stop when result is accurate'.
> But, Standard error enough under set value, non stop GnuBG rollout.
> I use latest version GnuBG ( Today I compile gnubg from source code).
>
> My setting.
>
> Settings-rollouts
>
> set 'Trials:' 12960
> Check 'Stop when STDs are small enough'
> set 'Minimum Trials:' 648
> set 'Ratio|Standard Deviation/Value:' 0.0200

'Stop when STDs are small enough' may not do what you seem to be
expecting. As stated, it stops when Standard Deviation/Value (not equity
sd) is below the limit you set *for every output* : singles wins, gammons,
backgammons for both players, cubeless and cubeful equity.

In practice, this criterion is useless : backgammon numbers are often very
noisy, if the equity is close to 0, sd/equity is high, etc...

Since you compile from source, you may want to apply the appended patch
that changes the criterion to 'stop when equity sd is small enough'.

With it, I get results like :

Rollout details:
Centered 1-cube:
   0.71334 0.02413 0.00064 - 0.28666 0.02212 0.00057 CL  +0.42875 CF
+0.72226
  [0.00030 0.00028 0.00006 - 0.00030 0.00025 0.00004 CL   0.00070 CF
0.00175]
Player gnubg owns 2-cube:
   0.71590 0.02384 0.00063 - 0.28410 0.02205 0.00057 CL  +0.86731 CF
+0.71301
  [0.00031 0.00029 0.00006 - 0.00031 0.00026 0.00004 CL   0.00141 CF
0.00200]
Full cubeful rollout with var.redn.
12051 games, Mersenne Twister dice gen. with seed 868626794 and
quasi-random dice
Stop when std.errs. are small enough: ratio 0.002 (min. 1296 games)

which may be what you want. With stock gnubg, it wouldn't have stopped
there since for instance sd(bg)/bg for the player on roll is about 0.1, 50
times too high!

Index: rollout.c
===================================================================
RCS file: /sources/gnubg/gnubg/rollout.c,v
retrieving revision 1.221
diff -u -r1.221 rollout.c
--- rollout.c 1 Oct 2009 21:05:54 -0000 1.221
+++ rollout.c 31 Oct 2009 19:20:15 -0000
@@ -1216,7 +1216,7 @@
  if (fNoMore[alt] || altGameCount[alt] < (rcRollout.nMinimumGames ))
  continue;
  prc = &ro_apes[alt]->rc;
- for (output = 0; output < NUM_ROLLOUT_OUTPUTS; output++) {
+ for (output = OUTPUT_EQUITY; output < NUM_ROLLOUT_OUTPUTS; output++) {
  if (output < OUTPUT_EQUITY) {
  v = fabs(aarMu[alt][output]);
  s = fabs(aarSigma[alt][output]);
@@ -1249,8 +1249,7 @@
    &aciLocal[(ro_fCubeRollout ? 0 : alt)]));
  }
  }
-
- if ((v >= .0001) && (v * rcRollout.rStdLimit < s)) {
+ if ( rcRollout.rStdLimit < s) {
  err_too_big = 1;
  break;
  }

PS: Jon, Christian, how about applying this change (or something to that
effect), plus the corresponding update the UI ? I insist : the current
behaviour of this criterion is just useless!



_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
http://lists.gnu.org/mailman/listinfo/bug-gnubg

[Yoshito Takeuchi]

Hi Philippe Michel

Great thanks for your reply.

Sorry, I didn't understand the function.
And I tried your patch and rollout.
It is just the function for me !!!

Thanks again

Yoshito Takeuchi

> 'Stop when STDs are small enough' may not do what you seem to be expecting.
> As stated, it stops when Standard Deviation/Value (not equity sd) is below
> the limit you set *for every output* : singles wins, gammons, backgammons
> for both players, cubeless and cubeful equity.
>
> In practice, this criterion is useless : backgammon numbers are often very
> noisy, if the equity is close to 0, sd/equity is high, etc...
>
> Since you compile from source, you may want to apply the appended patch that
> changes the criterion to 'stop when equity sd is small enough'.
>
> With it, I get results like :
>
> Rollout details:
> Centered 1-cube:
>  0.71334 0.02413 0.00064 - 0.28666 0.02212 0.00057 CL  +0.42875 CF +0.72226
>  [0.00030 0.00028 0.00006 - 0.00030 0.00025 0.00004 CL   0.00070 CF 0.00175]
> Player gnubg owns 2-cube:
>  0.71590 0.02384 0.00063 - 0.28410 0.02205 0.00057 CL  +0.86731 CF +0.71301
>  [0.00031 0.00029 0.00006 - 0.00031 0.00026 0.00004 CL   0.00141 CF 0.00200]
> Full cubeful rollout with var.redn.
> 12051 games, Mersenne Twister dice gen. with seed 868626794 and quasi-random
> dice
> Stop when std.errs. are small enough: ratio 0.002 (min. 1296 games)
>
> which may be what you want. With stock gnubg, it wouldn't have stopped there
> since for instance sd(bg)/bg for the player on roll is about 0.1, 50 times
> too high!
>
> Index: rollout.c
> ===================================================================
> RCS file: /sources/gnubg/gnubg/rollout.c,v
> retrieving revision 1.221
> diff -u -r1.221 rollout.c
> --- rollout.c   1 Oct 2009 21:05:54 -0000       1.221
> +++ rollout.c   31 Oct 2009 19:20:15 -0000
> @@ -1216,7 +1216,7 @@
>                                if (fNoMore[alt] || altGameCount[alt] <
> (rcRollout.nMinimumGames ))
>                                        continue;
>                                prc = &ro_apes[alt]->rc;
> -                               for (output = 0; output <
> NUM_ROLLOUT_OUTPUTS; output++) {
> +                               for (output = OUTPUT_EQUITY; output <
> NUM_ROLLOUT_OUTPUTS; output++) {
>                                        if (output < OUTPUT_EQUITY) {
>                                                v = fabs(aarMu[alt][output]);
>                                                s =
> fabs(aarSigma[alt][output]);
> @@ -1249,8 +1249,7 @@
>
> &aciLocal[(ro_fCubeRollout ? 0 : alt)]));
>                                                }
>                                        }
> -
> -                                       if ((v >= .0001) && (v *
> rcRollout.rStdLimit < s)) {
> +                                       if ( rcRollout.rStdLimit < s) {
>                                                err_too_big = 1;
>                                                break;
>                                        }
>
> PS: Jon, Christian, how about applying this change (or something to that
> effect), plus the corresponding update the UI ? I insist : the current
> behaviour of this criterion is just useless!
>
>

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
http://lists.gnu.org/mailman/listinfo/bug-gnubg

[Massimiliano Maini-3]

I tend to agree. The two criteria should be:

1. std dev smaller than a given (absolute) value

2. jsd smaller than a given value

I would also allow to combine them instead of allowing only to choose
between the 2.

For example:

max trials = 1296
min trials = 324
stop when jsd <= 2 AND sd <= 0.010

It may make sense to allow to select an OR between the two criteria.

MaX.

2009/10/31 Philippe Michel <philippe.michel7@...>:

> On Sat, 31 Oct 2009, Yoshito Takeuchi wrote:
>
> PS: Jon, Christian, how about applying this change (or something to that
> effect), plus the corresponding update the UI ? I insist : the current
> behaviour of this criterion is just useless!
>
>
>
> _______________________________________________
> Bug-gnubg mailing list
> Bug-gnubg@...
> http://lists.gnu.org/mailman/listinfo/bug-gnubg
>

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
http://lists.gnu.org/mailman/listinfo/bug-gnubg