"Joseph-ID" in benchmark db

[Mark Higgins-4]

Can anyone point me to a function to convert the keys in the benchmark dbs to a board layout pls?

They don't look like 14-char position IDs, and checking this list archive I see them occasionally referred to as Joseph-ID.




_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Michael Petch]

On 09/02/2012 12:01 AM, Mark Higgins wrote:
> Can anyone point me to a function to convert the keys in the benchmark dbs to a board layout pls?
>
> They don't look like 14-char position IDs, and checking this list archive I see them occasionally referred to as Joseph-ID.
>

Thanks to Philippe, the code I think you are looking for is now in
gnubg.c. Philippe committed a change last month that allows the "set
board" command to take a gnubg-nn ID (Which I think is what you are
referring to) and allow it to be used to setup a position in GNUBG. If
you want this feature I can build a windows release tomorrow. The
excerpt from the source code that does the work is here:

    if( strlen(pch) == 20) { /* gnubg-nn position string */
        static oldpositionkey key;

        for(i = 0; i < 10; ++i) {
           key.auch[i] = ((pch[2*i+0] - 'A') << 4) +  (pch[2*i+1] - 'A');
        }

        oldPositionFromKey( an, &key);

        return 0;
    }


The above is a snippet from GNUBG.C in the function
extern int ParsePosition( TanBoard an, char **ppch, char *pchDesc )
which was added by Philippe.

--
Michael Petch
CApp::Sysware Consulting Ltd.
OpenPGP FingerPrint=D81C 6A0D 987E 7DA5 3219 6715 466A 2ACE 5CAE 3304

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Philippe Michel-3]

On Thu, 9 Feb 2012, Mark Higgins wrote:

>
> Can anyone point me to a function to convert the keys in the benchmark dbs to a board layout pls?
>
> They don't look like 14-char position IDs, and checking this list archive I see them occasionally referred to as Joseph-ID.

The underlying binary key is the same but gnubg's position ID encodes it
in base64 while Joseph uses a simpler encoding that translates only 4 bits
per character.

See for instance
http://cvs.savannah.gnu.org/viewvc/gnubg/gnubg/gnubg.c?r1=1.905&r2=1.906

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Joseph Heled]

On 9 February 2012 20:33, Michael Petch <mpetch@...> wrote:

> On 09/02/2012 12:01 AM, Mark Higgins wrote:
>> Can anyone point me to a function to convert the keys in the benchmark dbs to a board layout pls?
>>
>> They don't look like 14-char position IDs, and checking this list archive I see them occasionally referred to as Joseph-ID.
>>
>
> Thanks to Philippe, the code I think you are looking for is now in
> gnubg.c. Philippe committed a change last month that allows the "set
> board" command to take a gnubg-nn ID (Which I think is what you are
> referring to) and allow it to be used to setup a position in GNUBG. If
> you want this feature I can build a windows release tomorrow. The
> excerpt from the source code that does the work is here:

When accepting an id from user, you do want to test that pch[XXX] -
'A' is indeed 4 bits, i.e.  'A' <= pch[XXX] <= 'A' + 15

-Joseph

>
>    if( strlen(pch) == 20) { /* gnubg-nn position string */
>        static oldpositionkey key;
>
>        for(i = 0; i < 10; ++i) {
>           key.auch[i] = ((pch[2*i+0] - 'A') << 4) +  (pch[2*i+1] - 'A');
>        }
>
>        oldPositionFromKey( an, &key);
>
>        return 0;
>    }
>
>
> The above is a snippet from GNUBG.C in the function
> extern int ParsePosition( TanBoard an, char **ppch, char *pchDesc )
> which was added by Philippe.
>
> --
> Michael Petch
> CApp::Sysware Consulting Ltd.
> OpenPGP FingerPrint=D81C 6A0D 987E 7DA5 3219 6715 466A 2ACE 5CAE 3304
>
> _______________________________________________
> Bug-gnubg mailing list
> Bug-gnubg@...
> https://lists.gnu.org/mailman/listinfo/bug-gnubg

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Mark Higgins-4]

Thx all!

One more benchmark database question, on format: I'm assuming:

* The rollout probabilities are in eg contact-train-data, not contact.bm (not sure exactly what that is).
* The format is <ID> <prob of any win> <prob of any gammon win> <prob of any gammon loss> <prob of bg win> <prob of bg loss>.
* Probabilities assume that the player is holding the dice, about to roll.

Is that correct?



On Feb 9, 2012, at 5:32 PM, Joseph Heled wrote:

> On 9 February 2012 20:33, Michael Petch <mpetch@...> wrote:
>> On 09/02/2012 12:01 AM, Mark Higgins wrote:
>>> Can anyone point me to a function to convert the keys in the benchmark dbs to a board layout pls?
>>>
>>> They don't look like 14-char position IDs, and checking this list archive I see them occasionally referred to as Joseph-ID.
>>>
>>
>> Thanks to Philippe, the code I think you are looking for is now in
>> gnubg.c. Philippe committed a change last month that allows the "set
>> board" command to take a gnubg-nn ID (Which I think is what you are
>> referring to) and allow it to be used to setup a position in GNUBG. If
>> you want this feature I can build a windows release tomorrow. The
>> excerpt from the source code that does the work is here:
>
> When accepting an id from user, you do want to test that pch[XXX] -
> 'A' is indeed 4 bits, i.e.  'A' <= pch[XXX] <= 'A' + 15
>
> -Joseph
>
>>
>>    if( strlen(pch) == 20) { /* gnubg-nn position string */
>>        static oldpositionkey key;
>>
>>        for(i = 0; i < 10; ++i) {
>>           key.auch[i] = ((pch[2*i+0] - 'A') << 4) +  (pch[2*i+1] - 'A');
>>        }
>>
>>        oldPositionFromKey( an, &key);
>>
>>        return 0;
>>    }
>>
>>
>> The above is a snippet from GNUBG.C in the function
>> extern int ParsePosition( TanBoard an, char **ppch, char *pchDesc )
>> which was added by Philippe.
>>
>> --
>> Michael Petch
>> CApp::Sysware Consulting Ltd.
>> OpenPGP FingerPrint=D81C 6A0D 987E 7DA5 3219 6715 466A 2ACE 5CAE 3304
>>
>> _______________________________________________
>> Bug-gnubg mailing list
>> Bug-gnubg@...
>> https://lists.gnu.org/mailman/listinfo/bug-gnubg
>
> _______________________________________________
> Bug-gnubg mailing list
> Bug-gnubg@...
> https://lists.gnu.org/mailman/listinfo/bug-gnubg

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Øystein Schønning-Johansen-2]

* The rollout probabilities are in eg contact-train-data, not contact.bm (not sure exactly what that is).

The positions for training and benchmarking are different. This is according to all recommandations in machine learning. contact-train-data contains the positions used for training the contact class positions.

contact.bm is the benchmark database, used to benchmark the neural nets.

 

* The format is <ID> <prob of any win> <prob of any gammon win> <prob of any gammon loss> <prob of bg win> <prob of bg loss>.

Correct!

 

* Probabilities assume that the player is holding the dice, about to roll.

Also correct!

-Øystein

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Mark Higgins-4]

Thx - what is the contact.bm format then? It didn't seem as straightforward as the training one. 

On Feb 10, 2012, at 5:42 PM, Øystein Schønning-Johansen <oysteijo@...> wrote:

* The rollout probabilities are in eg contact-train-data, not contact.bm (not sure exactly what that is).

The positions for training and benchmarking are different. This is according to all recommandations in machine learning. contact-train-data contains the positions used for training the contact class positions.

contact.bm is the benchmark database, used to benchmark the neural nets.

 

* The format is <ID> <prob of any win> <prob of any gammon win> <prob of any gammon loss> <prob of bg win> <prob of bg loss>.

Correct!

 

* Probabilities assume that the player is holding the dice, about to roll.

Also correct!

-Øystein

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Øystein Schønning-Johansen-2]

2012/2/10 Mark Higgins <miggins@...>

Thx - what is the contact.bm format then? It didn't seem as straightforward as the training one. 

A bit more complex...

lines starting with 's', is a settings string. These ase settings for nn-gnubg, that's Josephs tools.

lines starting with 'm' is a move benchmark.

Following the m, is a position and to integers representing a roll. Then comes the resulting board after the best move and the corresponding equity after this move. Following that again comes the resulting position after the candidate moves and a equity difference to the best move. 

You can see the few lines ahead of a m-line, the rollout with the probablility breakdowns. Lines starting with #R.

Lines starting with 'o', are cube evaluation bechmarks, I'll have to check what each following number is.

Lines starting with 'r' , I really don't remember.....

-Øystein

 

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Joseph Heled]

'r' is the seed used for the rollout, I think

On 10 February 2012 23:02, Øystein Schønning-Johansen
<oysteijo@...> wrote:

> 2012/2/10 Mark Higgins <miggins@...>
>>
>> Thx - what is the contact.bm format then? It didn't seem as
>> straightforward as the training one.
>>
>
> A bit more complex...
>
> lines starting with 's', is a settings string. These ase settings for
> nn-gnubg, that's Josephs tools.
>
> lines starting with 'm' is a move benchmark.
> Following the m, is a position and to integers representing a roll. Then
> comes the resulting board after the best move and the corresponding equity
> after this move. Following that again comes the resulting position after the
> candidate moves and a equity difference to the best move.
>
> You can see the few lines ahead of a m-line, the rollout with the
> probablility breakdowns. Lines starting with #R.
>
> Lines starting with 'o', are cube evaluation bechmarks, I'll have to check
> what each following number is.
>
> Lines starting with 'r' , I really don't remember.....
>
> -Øystein
>
>
>
> _______________________________________________
> Bug-gnubg mailing list
> Bug-gnubg@...
> https://lists.gnu.org/mailman/listinfo/bug-gnubg
>

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Øystein Schønning-Johansen-2]

'r' is the seed used for the rollout, I think

Sound likely, since there is an 'r'-line for every rollout result. But there is no code lines in perr.py to conferm it. I guess we can trust your memory on that.

'o' is the cube rollout, and the numbers are the rollout values of the outcome probability,

-Øystein

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Mark Higgins-4]

Does anyone have the average error stats for 0-ply gnubg on the contact benchmarks?

I see race & crashed results at Joseph's page here:

http://homepages.ihug.co.nz/~peps/ngb/index-top.html

but can't find the contact result anywhere. (Though I'd guess it's pretty close to the crashed error, ie around 0.01ppg/move.)

On Feb 10, 2012, at 6:26 AM, Øystein Schønning-Johansen wrote:

'r' is the seed used for the rollout, I think

Sound likely, since there is an 'r'-line for every rollout result. But there is no code lines in perr.py to conferm it. I guess we can trust your memory on that.

'o' is the cube rollout, and the numbers are the rollout values of the outcome probability,

-Øystein

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Øystein Schønning-Johansen-2]

I've looped through all 'm'-positionsThe following way:

For each postion I find if the best move with my evaluator, and find if my move is among the candidates in the list of moves. If it does not make the best move, I add the error to the total. If my evaluators move is not among the candidaes at all, I assign the same error as the worst move among the candidates. 

For all positions in contact.bm, GNU backgammon will have an error of about 1125, (IIRC)

Please report how your players make it.

-Øystein

2012/2/12 Mark Higgins <miggins@...>

Does anyone have the average error stats for 0-ply gnubg on the contact benchmarks?

I see race & crashed results at Joseph's page here:

http://homepages.ihug.co.nz/~peps/ngb/index-top.html

but can't find the contact result anywhere. (Though I'd guess it's pretty close to the crashed error, ie around 0.01ppg/move.)

On Feb 10, 2012, at 6:26 AM, Øystein Schønning-Johansen wrote:

'r' is the seed used for the rollout, I think

Sound likely, since there is an 'r'-line for every rollout result. But there is no code lines in perr.py to conferm it. I guess we can trust your memory on that.

'o' is the cube rollout, and the numbers are the rollout values of the outcome probability,

-Øystein

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Øystein Schønning-Johansen-2]

2012/2/12 Øystein Schønning-Johansen <oysteijo@...>

For each postion I find if the best move with my evaluator, and find if my move is among the candidates in the list of moves. If it does not make the best move, I add the error to the total. If my evaluators move is not among the candidaes at all, I assign the same error as the worst move among the candidates. 

If you evaluator makes a move that is not among the candidates, check that your move generator creates all legal move. I once found a terrible bug in my move generator in this way. I've trained the neural net for weeks, and found the move generator created some illegal moves in some positions.

-Øystein

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Mark Higgins-4]

My best player (TD trained, race & contact networks, a couple extra inputs beyond the standard Tesauro ones) has an average error of 0.0164ppg/move in the contact set, so not surprisingly worse than GNUbg (I assume 1125 means 0.01125ppg/move?).

I also was curious which benchmark set was most relevant for predicting match score, since of course a real game is a mixture of the positions. I took a bunch of my players, of varying skills, and calculated the average error rate for the three benchmark sets; and also played each against PubEval for 40k cubeless money games. Then I regressed the score in those games against the benchmark ERs to see which was most important (using R^2 as a proxy for importance).

Turns out the contact benchmark is most relevant, followed by crashed. Race is not that important.

Details here:

http://compgammon.blogspot.com/2012/02/gnubg-benchmark-results.html

On Feb 12, 2012, at 8:51 AM, Øystein Schønning-Johansen wrote:

I've looped through all 'm'-positionsThe following way:

For each postion I find if the best move with my evaluator, and find if my move is among the candidates in the list of moves. If it does not make the best move, I add the error to the total. If my evaluators move is not among the candidaes at all, I assign the same error as the worst move among the candidates. 

For all positions in contact.bm, GNU backgammon will have an error of about 1125, (IIRC)

Please report how your players make it.

-Øystein

2012/2/12 Mark Higgins <miggins@...>

Does anyone have the average error stats for 0-ply gnubg on the contact benchmarks?

I see race & crashed results at Joseph's page here:

http://homepages.ihug.co.nz/~peps/ngb/index-top.html

but can't find the contact result anywhere. (Though I'd guess it's pretty close to the crashed error, ie around 0.01ppg/move.)

On Feb 10, 2012, at 6:26 AM, Øystein Schønning-Johansen wrote:

'r' is the seed used for the rollout, I think

Sound likely, since there is an 'r'-line for every rollout result. But there is no code lines in perr.py to conferm it. I guess we can trust your memory on that.

'o' is the cube rollout, and the numbers are the rollout values of the outcome probability,

-Øystein

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Øystein Schønning-Johansen-2]

2012/2/12 Mark Higgins <miggins@...>

My best player (TD trained, race & contact networks, a couple extra inputs beyond the standard Tesauro ones) has an average error of 0.0164ppg/move in the contact set, so not surprisingly worse than GNUbg (I assume 1125 means 0.01125ppg/move?).

No, I mean total error over the file.

> grep -e '^m ' contact.bm | wc -l

107485

1125 is the total error over all these 107485 positions. That makes an average error of 0.010466577 Equity points pr positon in the contact.bm file.

-Øystein

 

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Ian Shaw]

Some parts of this message have been removed. Learn more about Nabble's security policy.

1125 is a great result. Gnubg contact benchmarks at about 1122, so your net is pretty much as good as gnubg. 1 point of contact-benchmark error is about 156.5 micropoints/game advantage.

 

We’ve managed to get as low as 1073 with the standard 250 inputs, but it took a 512 hidden nodes to do it, which would kill performance if we used it in the production gnubg.

 

Are you prepared to discuss your extra inputs?

 

n  Ian

 

 

From: bug-gnubg-bounces+ian.shaw=riverauto.co.uk@... [mailto:bug-gnubg-bounces+ian.shaw=riverauto.co.uk@...] On Behalf Of Øystein Schønning-Johansen
Sent: 12 February 2012 16:39
To: Mark Higgins
Cc: bug-gnubg@...
Subject: Re: [Bug-gnubg] "Joseph-ID" in benchmark db

 

 

2012/2/12 Mark Higgins <miggins@...>

My best player (TD trained, race & contact networks, a couple extra inputs beyond the standard Tesauro ones) has an average error of 0.0164ppg/move in the contact set, so not surprisingly worse than GNUbg (I assume 1125 means 0.01125ppg/move?).

 

 

No, I mean total error over the file.

 

> grep -e '^m ' contact.bm | wc -l

107485

 

1125 is the total error over all these 107485 positions. That makes an average error of 0.010466577 Equity points pr positon in the contact.bm file.

 

-Øystein

 

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Mark Higgins-4]

Some parts of this message have been removed. Learn more about Nabble's security policy.

I think Øystein was talking about gnubg 0-ply when he quoted 1125.

My best player scores much worse than that - around 1495. 

On Feb 14, 2012, at 3:52 PM, Ian Shaw <Ian.Shaw@...> wrote:

1125 is a great result. Gnubg contact benchmarks at about 1122, so your net is pretty much as good as gnubg. 1 point of contact-benchmark error is about 156.5 micropoints/game advantage.

 

We’ve managed to get as low as 1073 with the standard 250 inputs, but it took a 512 hidden nodes to do it, which would kill performance if we used it in the production gnubg.

 

Are you prepared to discuss your extra inputs?

 

n  Ian

 

 

From: bug-gnubg-bounces+ian.shaw=riverauto.co.uk@... [mailto:bug-gnubg-bounces+ian.shaw=riverauto.co.uk@...] On Behalf Of Øystein Schønning-Johansen
Sent: 12 February 2012 16:39
To: Mark Higgins
Cc: bug-gnubg@...
Subject: Re: [Bug-gnubg] "Joseph-ID" in benchmark db

 

 

2012/2/12 Mark Higgins <miggins@...>

My best player (TD trained, race & contact networks, a couple extra inputs beyond the standard Tesauro ones) has an average error of 0.0164ppg/move in the contact set, so not surprisingly worse than GNUbg (I assume 1125 means 0.01125ppg/move?).

 

 

No, I mean total error over the file.

 

> grep -e '^m ' contact.bm | wc -l

107485

 

1125 is the total error over all these 107485 positions. That makes an average error of 0.010466577 Equity points pr positon in the contact.bm file.

 

-Øystein

 

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Ian Shaw]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Yes, that’s 0-ply.

 

I think I gave results for higher plies in another thread.

 

n  Ian

 

From: Mark Higgins [mailto:miggins@...]
Sent: 14 February 2012 16:48
To: Ian Shaw
Cc: Øystein Schønning-Johansen; bug-gnubg@...
Subject: Re: [Bug-gnubg] "Joseph-ID" in benchmark db

 

I think Øystein was talking about gnubg 0-ply when he quoted 1125.

 

My best player scores much worse than that - around 1495. 

 

On Feb 14, 2012, at 3:52 PM, Ian Shaw <Ian.Shaw@...> wrote:

1125 is a great result. Gnubg contact benchmarks at about 1122, so your net is pretty much as good as gnubg. 1 point of contact-benchmark error is about 156.5 micropoints/game advantage.

 

We’ve managed to get as low as 1073 with the standard 250 inputs, but it took a 512 hidden nodes to do it, which would kill performance if we used it in the production gnubg.

 

Are you prepared to discuss your extra inputs?

 

n  Ian

 

 

From: bug-gnubg-bounces+ian.shaw=riverauto.co.uk@... [mailto:bug-gnubg-bounces+ian.shaw=riverauto.co.uk@...] On Behalf Of Øystein Schønning-Johansen
Sent: 12 February 2012 16:39
To: Mark Higgins
Cc: bug-gnubg@...
Subject: Re: [Bug-gnubg] "Joseph-ID" in benchmark db

 

 

2012/2/12 Mark Higgins <miggins@...>

My best player (TD trained, race & contact networks, a couple extra inputs beyond the standard Tesauro ones) has an average error of 0.0164ppg/move in the contact set, so not surprisingly worse than GNUbg (I assume 1125 means 0.01125ppg/move?).

 

 

No, I mean total error over the file.

 

> grep -e '^m ' contact.bm | wc -l

107485

 

1125 is the total error over all these 107485 positions. That makes an average error of 0.010466577 Equity points pr positon in the contact.bm file.

 

-Øystein

 

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Philippe Michel-3]

On Tue, 14 Feb 2012, Ian Shaw wrote:

> 1 point of contact-benchmark error is about 156.5 micropoints/game

This is for checkers play errors only apparently. Do you have a similar
ratio for cube errors and for the crashed net benchmark ?

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg

[Ian Shaw]

Hi Philippe,

I don't think we do. Øystein performed tests with some nets he was experimenting with. He only looked at cubeless play, and  I think he only tested the Contact nets.

-- Ian

-----Original Message-----
From: Philippe Michel [mailto:philippe.michel7@...]
Sent: 15 February 2012 20:07
To: Ian Shaw
Cc: bug-gnubg@...
Subject: Benchmark error to ppg loss

On Tue, 14 Feb 2012, Ian Shaw wrote:

> 1 point of contact-benchmark error is about 156.5 micropoints/game

This is for checkers play errors only apparently. Do you have a similar ratio for cube errors and for the crashed net benchmark ?

_______________________________________________
Bug-gnubg mailing list
Bug-gnubg@...
https://lists.gnu.org/mailman/listinfo/bug-gnubg