Regression test: Quadrics

View: New views
20 Messages — Rating Filter:   Alert me  
< Prev | 1 - 2 | Next >

Regression test: Quadrics

by Trevor Lovett :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Just ran a regression test against the whole trunk and got some failures back for quadrics.  Should I only be concerned with the top 3?  For the rest, I expect that the images would be visibly different since we are switching from using standard library sin(), cos() to series-approximations of those functions.  Attached: screen shot of the quadrics test results, and the whole tree summary (a random sampling of failures revealed only "images are visibly different" messages).  -Trevor

Summary pasted here:
Job directories:

    * RIBs - 229 jobs, 127 failures
          o Camera - 8 jobs, 5 failures
          o DepthOfField - 3 jobs, 1 failures
          o Flower - 1 jobs, no failures
          o Misc - 19 jobs, 12 failures
          o MotionBlur - 12 jobs, 11 failures
          o ObjectInstance - 3 jobs, 3 failures
          o Occlusion - 2 jobs, 2 failures
          o Primitives - 66 jobs, 35 failures
                + Blobby - 2 jobs, 1 failures
                + Curves - 15 jobs, no failures
                + NURBS - 4 jobs, 2 failures
                + Patches - 1 jobs, 1 failures
                + Points - 5 jobs, 3 failures
                + Polygons - 13 jobs, 10 failures
                + Quadrics - 8 jobs, 6 failures
                + Subdivision - 16 jobs, 11 failures 
          o Procedurals - 3 jobs, no failures
          o RIBProcessing - 4 jobs, no failures
          o Scenes - 2 jobs, no failures
          o Shading - 52 jobs, 27 failures
                + SLFunctions - 14 jobs, no failures 
          o Sidedness - 6 jobs, 5 failures
          o SolidModeling - 3 jobs, 1 failures
          o SurfaceParameters - 11 jobs, 3 failures
          o Texturing - 32 jobs, 21 failures
                + Environment - 8 jobs, 5 failures
                + PlainTex - 14 jobs, 10 failures
                + Shadows - 10 jobs, 6 failures 
          o Transformations - 2 jobs, 1 failures 



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

regression_trevor_080709.png (276K) Download Attachment

Re: Regression test: Quadrics

by Paul Gregory-4 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Trevor,

In that screengrab, the first three fail often for other reasons, don't worry too much about them, it's to do with Python and the RunProcedural, unless your machine is properly setup, they will fail.

The others look fine from a cursory glance.


Paul

2009/8/7 Trevor Lovett <trevlovett@...>
Just ran a regression test against the whole trunk and got some failures back for quadrics.  Should I only be concerned with the top 3?  For the rest, I expect that the images would be visibly different since we are switching from using standard library sin(), cos() to series-approximations of those functions.  Attached: screen shot of the quadrics test results, and the whole tree summary (a random sampling of failures revealed only "images are visibly different" messages).  -Trevor

Summary pasted here:
Job directories:

    * RIBs - 229 jobs, 127 failures
          o Camera - 8 jobs, 5 failures
          o DepthOfField - 3 jobs, 1 failures
          o Flower - 1 jobs, no failures
          o Misc - 19 jobs, 12 failures
          o MotionBlur - 12 jobs, 11 failures
          o ObjectInstance - 3 jobs, 3 failures
          o Occlusion - 2 jobs, 2 failures
          o Primitives - 66 jobs, 35 failures
                + Blobby - 2 jobs, 1 failures
                + Curves - 15 jobs, no failures
                + NURBS - 4 jobs, 2 failures
                + Patches - 1 jobs, 1 failures
                + Points - 5 jobs, 3 failures
                + Polygons - 13 jobs, 10 failures
                + Quadrics - 8 jobs, 6 failures
                + Subdivision - 16 jobs, 11 failures 
          o Procedurals - 3 jobs, no failures
          o RIBProcessing - 4 jobs, no failures
          o Scenes - 2 jobs, no failures
          o Shading - 52 jobs, 27 failures
                + SLFunctions - 14 jobs, no failures 
          o Sidedness - 6 jobs, 5 failures
          o SolidModeling - 3 jobs, 1 failures
          o SurfaceParameters - 11 jobs, 3 failures
          o Texturing - 32 jobs, 21 failures
                + Environment - 8 jobs, 5 failures
                + PlainTex - 14 jobs, 10 failures
                + Shadows - 10 jobs, 6 failures 
          o Transformations - 2 jobs, 1 failures 


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Chris Foster-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, Aug 8, 2009 at 7:47 AM, Trevor Lovett<trevlovett@...> wrote:
> Just ran a regression test against the whole trunk and got some failures
> back for quadrics.  Should I only be concerned with the top 3?  For the
> rest, I expect that the images would be visibly different since we are
> switching from using standard library sin(), cos() to series-approximations
> of those functions.

No, I don't think they should be different.  The "approximations" for sin and
cos are actually *exact* on our grid, apart from any finite precision rounding
errors!  Neat huh?

If we did find rounding error was a problem, we should switch to using double
precision inside the sinCosGrid() function.  In fact, that's probably a good
idea anyway.

> Attached: screen shot of the quadrics test results, and
> the whole tree summary (a random sampling of failures revealed only "images
> are visibly different" messages).

Oh dear, something is really wrong here.  I think there's likely to be two
problems:

1) Do you have pdiff in your path?  That's the perceptual difference program
which we normally use to compare images.  For consistency, you should use the
version included in the aqsis source.  You've should enable the
AQSIS_USE_PDIFF flag inside the cmake build config to get pdiff built.
Without pdiff, IIRC a simple pixel-per-pixel similarity test is used for
comparison which results in very many non-failures being marked as failures as
you can see here.

2) Did you install cgkit2?  You'll need it, because the three main quadrics
tests which currently display "no output" have the input RIB generated by
cgkit on the fly.  (IMHO it's a bad thing to have dynamic content in test
cases, but that's how it is.)

If everything is working smoothly, you should see only a few known failures -
in total no more than about 5.  Please let me know which ones.

Thanks,
~Chris.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Trevor Lovett :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message



On Sat, Aug 8, 2009 at 12:35 AM, Chris Foster <chris42f@...> wrote:
On Sat, Aug 8, 2009 at 7:47 AM, Trevor Lovett<trevlovett@...> wrote:
> Just ran a regression test against the whole trunk and got some failures
> back for quadrics.  Should I only be concerned with the top 3?  For the
> rest, I expect that the images would be visibly different since we are
> switching from using standard library sin(), cos() to series-approximations
> of those functions.

No, I don't think they should be different.  The "approximations" for sin and
cos are actually *exact* on our grid, apart from any finite precision rounding
errors!  Neat huh?

That's right, forgot it was exact.  Neat indeed--I'll have to go through the derivation when I have a spare moment.  




If we did find rounding error was a problem, we should switch to using double
precision inside the sinCosGrid() function.  In fact, that's probably a good
idea anyway.

I think it's a rounding issue--I'll try making them doubles.  Could also be an "off by 1" issue with an array length or index somewhere.  



> Attached: screen shot of the quadrics test results, and
> the whole tree summary (a random sampling of failures revealed only "images
> are visibly different" messages).

Oh dear, something is really wrong here.  I think there's likely to be two
problems:

1) Do you have pdiff in your path?  That's the perceptual difference program
which we normally use to compare images.  For consistency, you should use the
version included in the aqsis source.  You've should enable the
AQSIS_USE_PDIFF flag inside the cmake build config to get pdiff built.
Without pdiff, IIRC a simple pixel-per-pixel similarity test is used for
comparison which results in very many non-failures being marked as failures as
you can see here.

I didn't know about that flag when I did the tests--I just built pdiff from what I got from their svn repo after I saw error messages about not being able to find pdiff.
    


2) Did you install cgkit2?  You'll need it, because the three main quadrics
tests which currently display "no output" have the input RIB generated by
cgkit on the fly.  (IMHO it's a bad thing to have dynamic content in test
cases, but that's how it is.)

No, I will try this.
 


If everything is working smoothly, you should see only a few known failures -
in total no more than about 5.  Please let me know which ones.

Thanks,
~Chris.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Chris Foster-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, Aug 8, 2009 at 8:25 AM, Paul Gregory<pgregory@...> wrote:
> Trevor,
>
> In that screengrab, the first three fail often for other reasons, don't
> worry too much about them, it's to do with Python and the RunProcedural,
> unless your machine is properly setup, they will fail.

I think it's important that these are made to work, because he's
specifically testing the quadrics refactoring.  The reason they fail
is because of a missing install of cgkit.

In fact I haven't been able to make these work for a long time,
because for some strange reason I can't get cgkit working properly on
my machine.  Given that, I'd very much like to replace them with plain
old static RIB (the scripts needed to regenerate them should be kept
around of course).  What do you think Paul?

~Chris.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Paul Gregory-4 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

We could, the main purpose of those is to test quadrics, not procedurals. As long as we have something in the RTS that tests procedurals, that's fine.

Paul

2009/8/8 Chris Foster <chris42f@...>
On Sat, Aug 8, 2009 at 8:25 AM, Paul Gregory<pgregory@...> wrote:
> Trevor,
>
> In that screengrab, the first three fail often for other reasons, don't
> worry too much about them, it's to do with Python and the RunProcedural,
> unless your machine is properly setup, they will fail.

I think it's important that these are made to work, because he's
specifically testing the quadrics refactoring.  The reason they fail
is because of a missing install of cgkit.

In fact I haven't been able to make these work for a long time,
because for some strange reason I can't get cgkit working properly on
my machine.  Given that, I'd very much like to replace them with plain
old static RIB (the scripts needed to regenerate them should be kept
around of course).  What do you think Paul?

~Chris.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Chris Foster-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, Aug 8, 2009 at 9:12 AM, Paul Gregory<pgregory@...> wrote:
> We could, the main purpose of those is to test quadrics, not procedurals. As
> long as we have something in the RTS that tests procedurals, that's fine.

AFAIK this has nothing to do with procedurals.  What happens is that
rendertester.py runs the python script itself, and feeds the resulting
RIB into aqsis.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Trevor Lovett :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hm built and installed py-cgkit (from macports), but those first 3 in the quadrics regression aren't producing output.  I'll wait until I get some raw .rib's to test with for those.

I'm still getting the same 127 failures and finding this a bit puzzling.  I see that the script is using pdiff.  I think what must be happening is that my MF code is being used in every case that makes use of AdjustedShadingRate because of a misunderstanding on my part about how the attribute system works.

Getting a bit late here--I think I can sort that one out in the AM.    

On Sat, Aug 8, 2009 at 1:14 AM, Chris Foster <chris42f@...> wrote:
On Sat, Aug 8, 2009 at 9:12 AM, Paul Gregory<pgregory@...> wrote:
> We could, the main purpose of those is to test quadrics, not procedurals. As
> long as we have something in the RTS that tests procedurals, that's fine.

AFAIK this has nothing to do with procedurals.  What happens is that
rendertester.py runs the python script itself, and feeds the resulting
RIB into aqsis.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Chris Foster-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, Aug 8, 2009 at 9:01 AM, Trevor Lovett<trevlovett@...> wrote:
>> If we did find rounding error was a problem, we should switch to using
>> double
>> precision inside the sinCosGrid() function.  In fact, that's probably a
>> good
>> idea anyway.
>
> I think it's a rounding issue--I'll try making them doubles.  Could also be
> an "off by 1" issue with an array length or index somewhere.

Maybe you should try making just the variables which are used in the
calculation into doubles, and just save the results into float arrays?  That's
probably accurate enough.  The main problem with 32-bit floats in a
circumstance like this is that the rounding errors tend to accumulate over
many iterations.

>> > Attached: screen shot of the quadrics test results, and
>> > the whole tree summary (a random sampling of failures revealed only
>> > "images
>> > are visibly different" messages).
>>
>> Oh dear, something is really wrong here.  I think there's likely to be two
>> problems:
>>
>> 1) Do you have pdiff in your path?  That's the perceptual difference
>> program
>> which we normally use to compare images.  For consistency, you should use
>> the
>> version included in the aqsis source.  You've should enable the
>> AQSIS_USE_PDIFF flag inside the cmake build config to get pdiff built.
>> Without pdiff, IIRC a simple pixel-per-pixel similarity test is used for
>> comparison which results in very many non-failures being marked as
>> failures as
>> you can see here.
>
> I didn't know about that flag when I did the tests--I just built pdiff from
> what I got from their svn repo after I saw error messages about not being
> able to find pdiff.

Right.  Make sure you're using the pdiff included with aqsis, since the latest
version from upstream is rather too strict for us, and we haven't figured out
how to get around that yet.

~Chris.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Matthias Baas-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Chris Foster wrote:
> On Sat, Aug 8, 2009 at 9:12 AM, Paul Gregory<pgregory@...> wrote:
>> We could, the main purpose of those is to test quadrics, not procedurals. As
>> long as we have something in the RTS that tests procedurals, that's fine.
>
> AFAIK this has nothing to do with procedurals.  What happens is that
> rendertester.py runs the python script itself, and feeds the resulting
> RIB into aqsis.

Just to confirm, those Python scripts in the Quadrics directory are no
procedurals. I wrote those tests as Python scripts because then I could
easily create this "parameter matrix", i.e. test a range of values on
several parameters and also check all combinations.
But of course, there's no reason why this has to be regenerated each
time the tests are run. You can just as well dump the RIB and replace
the script by the RIB (and probably move the script elsewhere in case
you have to regenerate the RIB for some reason).

The procedural tests are in the Procedurals directory. They also use
cgkit to issue the Ri commands. This could be replaced by pure prints or
stdout.write as well.

There is another Python script in the Occlusion directory which was not
written by me. That one uses the ri module and the cgtypes module. It
looks like this is not run during the tests but apparently it was used
to generate occlmap.rib, so I suppose that one is fine already.
(However, I noticed another thing, the test seems to be split up into
two separate jobs which always need to be run together and in the
correct order. This is not how things should be set up. The rendertester
tool assumes that jobs are always independent from each other. So all
passes that are required for running a test should be part of the same jobs)

- Matthias -


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Trevor Lovett :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hm, I just nuked my build directory did:
mkdir build
ccmake ../aqsis_gsoc     (the source)

Now I'm going through the options and I don't see AQSIS_USE_PDIFF.

Has anyone who was building under OS X had this difficulty?

I pulled the aqsis source maybe two weeks ago so it should be relatively up to date.

These are the options I get:

 AQSIS_BISON_EXECUTABLE           /usr/bin/bison                               
 AQSIS_ENABLE_THREADING           OFF                                          
 AQSIS_FLEX_EXECUTABLE            /usr/bin/flex                                
 AQSIS_FLTK_INCLUDE_DIR           /opt/local/include                           
 AQSIS_FLTK_LIBRARIES_DIR                                                      
 AQSIS_NSIS_EXECUTABLE            /opt/local/bin/makensis                      
 AQSIS_OPENEXR_INCLUDE_DIR        /opt/local/include/OpenEXR                   
 AQSIS_OPENEXR_LIBRARIES_DIR                                                   
 AQSIS_OSACOMPILE_EXECUTABLE      /usr/bin/osacompile                          
 AQSIS_PACKAGE_NAME               Aqsis-1.5.0-Darwin.dmg                       
 AQSIS_RESOURCE_EXECUTABLE        /usr/bin/Rez                                 
 AQSIS_TIFF_INCLUDE_DIR           /opt/local/include                           
 AQSIS_TIFF_LIBRARIES_DIR                                                      
 AQSIS_USE_FLTK                   ON                                           
 AQSIS_USE_OPENEXR                ON                                           
 AQSIS_USE_TIMERS                 ON                                           
 AQSIS_XCODEBUILD_EXECUTABLE      /usr/bin/xcodebuild   
 AQSIS_XSLTPROC_EXECUTABLE        /usr/bin/xsltproc                            
 AQSIS_ZLIB_INCLUDE_DIR           /usr/include                                 
 AQSIS_ZLIB_LIBRARIES_DIR                                                      
 ARCHIVEPATH                      %AQSISHOME%/Contents/Resources               
 BINDIR                           bin                                          
 BOOST_ROOT                                                                    
 BUNDLEDIR                        /Users/trevor/Project/soc/build/bundle       
 CMAKE_BUILD_TYPE                 Release                                      
 CMAKE_INSTALL_PREFIX             /usr/local                                   
 CMAKE_OSX_ARCHITECTURES          i386                                         
 CMAKE_OSX_SYSROOT                /Developer/SDKs/MacOSX10.5.sdk               
 CMAKE_PROJECT_NAME               Aqsis                                        
 DISPLAYPATH                      %AQSISHOME%/Contents/Resources/lib           
 EXAMPLESDIR                      share/aqsis/examples                         
 FLTK_FLUID_EXECUTABLE            /opt/local/bin/fluid                         
 INCLUDEDIR                       include                                      
 LIBDIR                           lib           
 PLUGINDIR                        lib                                          
 PROCEDURALPATH                   %AQSISHOME%/Contents/Resources/lib           
 RESOURCEPATH                     %AQSISHOME%/Contents/Resources               
 SCRIPTSDIR                       share/aqsis/scripts                          
 SHADERDIR                        share/aqsis/shaders                          
 SHADERPATH                       %AQSISHOME%/Contents/Resources/share/aqsis/sh
 SYSCONFDIR                       etc/aqsis                                    
 TEXTUREPATH                      %AQSISHOME%/Contents/Resources               
 aqsis_enable_testing             OFF                                          
                                                      


On Sat, Aug 8, 2009 at 1:47 AM, Chris Foster <chris42f@...> wrote:
On Sat, Aug 8, 2009 at 9:01 AM, Trevor Lovett<trevlovett@...> wrote:
>> If we did find rounding error was a problem, we should switch to using
>> double
>> precision inside the sinCosGrid() function.  In fact, that's probably a
>> good
>> idea anyway.
>
> I think it's a rounding issue--I'll try making them doubles.  Could also be
> an "off by 1" issue with an array length or index somewhere.

Maybe you should try making just the variables which are used in the
calculation into doubles, and just save the results into float arrays?  That's
probably accurate enough.  The main problem with 32-bit floats in a
circumstance like this is that the rounding errors tend to accumulate over
many iterations.

>> > Attached: screen shot of the quadrics test results, and
>> > the whole tree summary (a random sampling of failures revealed only
>> > "images
>> > are visibly different" messages).
>>
>> Oh dear, something is really wrong here.  I think there's likely to be two
>> problems:
>>
>> 1) Do you have pdiff in your path?  That's the perceptual difference
>> program
>> which we normally use to compare images.  For consistency, you should use
>> the
>> version included in the aqsis source.  You've should enable the
>> AQSIS_USE_PDIFF flag inside the cmake build config to get pdiff built.
>> Without pdiff, IIRC a simple pixel-per-pixel similarity test is used for
>> comparison which results in very many non-failures being marked as
>> failures as
>> you can see here.
>
> I didn't know about that flag when I did the tests--I just built pdiff from
> what I got from their svn repo after I saw error messages about not being
> able to find pdiff.

Right.  Make sure you're using the pdiff included with aqsis, since the latest
version from upstream is rather too strict for us, and we haven't figured out
how to get around that yet.

~Chris.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Trevor Lovett :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Ah nevermind.  I needed to be in "advanced mode".

On Sat, Aug 8, 2009 at 11:32 AM, Trevor Lovett <trevlovett@...> wrote:
Hm, I just nuked my build directory did:
mkdir build
ccmake ../aqsis_gsoc     (the source)

Now I'm going through the options and I don't see AQSIS_USE_PDIFF.

Has anyone who was building under OS X had this difficulty?

I pulled the aqsis source maybe two weeks ago so it should be relatively up to date.

These are the options I get:

 AQSIS_BISON_EXECUTABLE           /usr/bin/bison                               
 AQSIS_ENABLE_THREADING           OFF                                          
 AQSIS_FLEX_EXECUTABLE            /usr/bin/flex                                
 AQSIS_FLTK_INCLUDE_DIR           /opt/local/include                           
 AQSIS_FLTK_LIBRARIES_DIR                                                      
 AQSIS_NSIS_EXECUTABLE            /opt/local/bin/makensis                      
 AQSIS_OPENEXR_INCLUDE_DIR        /opt/local/include/OpenEXR                   
 AQSIS_OPENEXR_LIBRARIES_DIR                                                   
 AQSIS_OSACOMPILE_EXECUTABLE      /usr/bin/osacompile                          
 AQSIS_PACKAGE_NAME               Aqsis-1.5.0-Darwin.dmg                       
 AQSIS_RESOURCE_EXECUTABLE        /usr/bin/Rez                                 
 AQSIS_TIFF_INCLUDE_DIR           /opt/local/include                           
 AQSIS_TIFF_LIBRARIES_DIR                                                      
 AQSIS_USE_FLTK                   ON                                           
 AQSIS_USE_OPENEXR                ON                                           
 AQSIS_USE_TIMERS                 ON                                           
 AQSIS_XCODEBUILD_EXECUTABLE      /usr/bin/xcodebuild   
 AQSIS_XSLTPROC_EXECUTABLE        /usr/bin/xsltproc                            
 AQSIS_ZLIB_INCLUDE_DIR           /usr/include                                 
 AQSIS_ZLIB_LIBRARIES_DIR                                                      
 ARCHIVEPATH                      %AQSISHOME%/Contents/Resources               
 BINDIR                           bin                                          
 BOOST_ROOT                                                                    
 BUNDLEDIR                        /Users/trevor/Project/soc/build/bundle       
 CMAKE_BUILD_TYPE                 Release                                      
 CMAKE_INSTALL_PREFIX             /usr/local                                   
 CMAKE_OSX_ARCHITECTURES          i386                                         
 CMAKE_OSX_SYSROOT                /Developer/SDKs/MacOSX10.5.sdk               
 CMAKE_PROJECT_NAME               Aqsis                                        
 DISPLAYPATH                      %AQSISHOME%/Contents/Resources/lib           
 EXAMPLESDIR                      share/aqsis/examples                         
 FLTK_FLUID_EXECUTABLE            /opt/local/bin/fluid                         
 INCLUDEDIR                       include                                      
 LIBDIR                           lib           
 PLUGINDIR                        lib                                          
 PROCEDURALPATH                   %AQSISHOME%/Contents/Resources/lib           
 RESOURCEPATH                     %AQSISHOME%/Contents/Resources               
 SCRIPTSDIR                       share/aqsis/scripts                          
 SHADERDIR                        share/aqsis/shaders                          
 SHADERPATH                       %AQSISHOME%/Contents/Resources/share/aqsis/sh
 SYSCONFDIR                       etc/aqsis                                    
 TEXTUREPATH                      %AQSISHOME%/Contents/Resources               
 aqsis_enable_testing             OFF                                          
                                                      


On Sat, Aug 8, 2009 at 1:47 AM, Chris Foster <chris42f@...> wrote:
On Sat, Aug 8, 2009 at 9:01 AM, Trevor Lovett<trevlovett@...> wrote:
>> If we did find rounding error was a problem, we should switch to using
>> double
>> precision inside the sinCosGrid() function.  In fact, that's probably a
>> good
>> idea anyway.
>
> I think it's a rounding issue--I'll try making them doubles.  Could also be
> an "off by 1" issue with an array length or index somewhere.

Maybe you should try making just the variables which are used in the
calculation into doubles, and just save the results into float arrays?  That's
probably accurate enough.  The main problem with 32-bit floats in a
circumstance like this is that the rounding errors tend to accumulate over
many iterations.

>> > Attached: screen shot of the quadrics test results, and
>> > the whole tree summary (a random sampling of failures revealed only
>> > "images
>> > are visibly different" messages).
>>
>> Oh dear, something is really wrong here.  I think there's likely to be two
>> problems:
>>
>> 1) Do you have pdiff in your path?  That's the perceptual difference
>> program
>> which we normally use to compare images.  For consistency, you should use
>> the
>> version included in the aqsis source.  You've should enable the
>> AQSIS_USE_PDIFF flag inside the cmake build config to get pdiff built.
>> Without pdiff, IIRC a simple pixel-per-pixel similarity test is used for
>> comparison which results in very many non-failures being marked as
>> failures as
>> you can see here.
>
> I didn't know about that flag when I did the tests--I just built pdiff from
> what I got from their svn repo after I saw error messages about not being
> able to find pdiff.

Right.  Make sure you're using the pdiff included with aqsis, since the latest
version from upstream is rather too strict for us, and we haven't figured out
how to get around that yet.

~Chris.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Matthias Baas-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Trevor Lovett wrote:
> Hm built and installed py-cgkit (from macports), but those first 3 in
> the quadrics regression aren't producing output.  I'll wait until I get
> some raw .rib's to test with for those.

The MacPorts version of cgkit is still cgkit1 which is fairly old. The
tests assume cgkit2. The functionality in cgkit1 should actually be fine
for the regression tests but the package layout has changed. In cgkit2,
every module is inside a package "cgkit" which was not the case in
cgkit1, so all the imports have changed (i.e. from ri import * -> from
cgkit.ri import *)

- Matthias -

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Trevor Lovett :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Ok--I'll try building cgkit2 from source then.

On Sat, Aug 8, 2009 at 11:36 AM, Matthias Baas <matthias.baas@...> wrote:
Trevor Lovett wrote:
> Hm built and installed py-cgkit (from macports), but those first 3 in
> the quadrics regression aren't producing output.  I'll wait until I get
> some raw .rib's to test with for those.

The MacPorts version of cgkit is still cgkit1 which is fairly old. The
tests assume cgkit2. The functionality in cgkit1 should actually be fine
for the regression tests but the package layout has changed. In cgkit2,
every module is inside a package "cgkit" which was not the case in
cgkit1, so all the imports have changed (i.e. from ri import * -> from
cgkit.ri import *)

- Matthias -

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Trevor Lovett :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Tried installing cgkit2 via the .dmg but it complains that it requires python 2.5 even though I'm running 2.5, it's active and all other python's on my system have been uninstalled.

Also tried building cgkit-2.0.0alpha8 from src but when building the supportlib via scons I get a bunch of errors.  

Perhaps you could send me an .egg Mattias (IIRC you were running OS X) ?  Otherwise I will wait until I can get my hands on some .rib's.

On Sat, Aug 8, 2009 at 11:40 AM, Trevor Lovett <trevlovett@...> wrote:
Ok--I'll try building cgkit2 from source then.


On Sat, Aug 8, 2009 at 11:36 AM, Matthias Baas <matthias.baas@...> wrote:
Trevor Lovett wrote:
> Hm built and installed py-cgkit (from macports), but those first 3 in
> the quadrics regression aren't producing output.  I'll wait until I get
> some raw .rib's to test with for those.

The MacPorts version of cgkit is still cgkit1 which is fairly old. The
tests assume cgkit2. The functionality in cgkit1 should actually be fine
for the regression tests but the package layout has changed. In cgkit2,
every module is inside a package "cgkit" which was not the case in
cgkit1, so all the imports have changed (i.e. from ri import * -> from
cgkit.ri import *)

- Matthias -

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Trevor Lovett :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Nevermind about the .egg--that won't help. 

On Sat, Aug 8, 2009 at 2:28 PM, Trevor Lovett <trevlovett@...> wrote:
Tried installing cgkit2 via the .dmg but it complains that it requires python 2.5 even though I'm running 2.5, it's active and all other python's on my system have been uninstalled.

Also tried building cgkit-2.0.0alpha8 from src but when building the supportlib via scons I get a bunch of errors.  

Perhaps you could send me an .egg Mattias (IIRC you were running OS X) ?  Otherwise I will wait until I can get my hands on some .rib's.


On Sat, Aug 8, 2009 at 11:40 AM, Trevor Lovett <trevlovett@...> wrote:
Ok--I'll try building cgkit2 from source then.


On Sat, Aug 8, 2009 at 11:36 AM, Matthias Baas <matthias.baas@...> wrote:
Trevor Lovett wrote:
> Hm built and installed py-cgkit (from macports), but those first 3 in
> the quadrics regression aren't producing output.  I'll wait until I get
> some raw .rib's to test with for those.

The MacPorts version of cgkit is still cgkit1 which is fairly old. The
tests assume cgkit2. The functionality in cgkit1 should actually be fine
for the regression tests but the package layout has changed. In cgkit2,
every module is inside a package "cgkit" which was not the case in
cgkit1, so all the imports have changed (i.e. from ri import * -> from
cgkit.ri import *)

- Matthias -

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development




------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Chris Foster-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi Matthias,

On 8/8/09, Matthias Baas <matthias.baas@...> wrote:
> Just to confirm, those Python scripts in the Quadrics directory are no
> procedurals. I wrote those tests as Python scripts because then I could
> easily create this "parameter matrix", i.e. test a range of values on
> several parameters and also check all combinations.
> But of course, there's no reason why this has to be regenerated each
> time the tests are run. You can just as well dump the RIB and replace
> the script by the RIB (and probably move the script elsewhere in case
> you have to regenerate the RIB for some reason).

Would you be able to do this Matthias?  ie, dump to some appropriately
named RIBs, and replace the calls to the python scripts in the config
files with the RIBs?  Alternatively I'm happy to do it myself if you
(or someone) can generate and send me the RIB files in question.

Of course, I'd generate them myself, but my cgkit install was
collateral damage from me upgrading some unknown package a year or so
ago.  After that I was never been able to get cgkit to run again
without segfaulting in the most odd of places :-(  I'm sure it's
something odd I did to my system, but I ended up giving up trying to
figure out what.

It would be nice if we keep the test cases as simple as possible (ie,
plain RIB), since it's nice for the test suite to run with a minimum
of fuss.  Having the original python sources available too is good of
course.

> There is another Python script in the Occlusion directory which was not
> written by me. That one uses the ri module and the cgtypes module. It
> looks like this is not run during the tests but apparently it was used
> to generate occlmap.rib, so I suppose that one is fine already.
> (However, I noticed another thing, the test seems to be split up into
> two separate jobs which always need to be run together and in the
> correct order. This is not how things should be set up. The rendertester
> tool assumes that jobs are always independent from each other. So all
> passes that are required for running a test should be part of the same jobs)

Fair enough, then this needs to be changed.

Thanks,
~Chris.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Matthias Baas-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi Chris,

>> But of course, there's no reason why this has to be regenerated each
>> time the tests are run. You can just as well dump the RIB and replace
>> the script by the RIB (and probably move the script elsewhere in case
>> you have to regenerate the RIB for some reason).
>
> Would you be able to do this Matthias?  ie, dump to some appropriately
> named RIBs, and replace the calls to the python scripts in the config
> files with the RIBs?

Is the RTS still in svn or in git? I just tried to commit the update to
the svn repository but got an error message. It looks like I don't have
permission to commit stuff anymore.
Anyway, I have attached the files to this mail instead. You can extract
the archive in the Quadrics directory and then commit the stuff.

- Matthias -



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

quadrics.tar.gz (4K) Download Attachment

Re: Regression test: Quadrics

by Matthias Baas-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Trevor Lovett wrote:
> Tried installing cgkit2 via the .dmg but it complains that it requires
> python 2.5 even though I'm running 2.5, it's active and all other
> python's on my system have been uninstalled.

As you mentioned MacPorts before, I'm assuming you are running the
MacPorts Python version, right? As far as I know this one doesn't work
with the dmg as its built differently. I think you would have to use the
version from python.org (which I think is a frameworks build... but
frankly, I'm not quite sure myself what the exact differences are).

> Also tried building cgkit-2.0.0alpha8 from src but when building the
> supportlib via scons I get a bunch of errors.  

Hm, if you want you can send me the errors and I'll see if it's
something obvious....
But I understand that you actually want to work on other stuff.
Anyway, for the Aqsis RTS you could just install the "light" version of
cgkit which doesn't require any compilation at all (see the first
paragraph on this page to find out how to do it:
http://cgkit.sourceforge.net/doc2/node10.html)

Or you can just grab the archive from my previous mail and use the RIBs
instead of the Python scripts.

- Matthias -


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

Re: Regression test: Quadrics

by Trevor Lovett :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Thanks for the advice.  Cgkit looks handy so I'll want to get that working eventually but doing the configuration-gymnastics will have to wait until after google's deadline.  I ran the .rib's you sent and the output from my build matches the reference images so I'm satisfied for now.

Trevor

On Sat, Aug 8, 2009 at 8:55 PM, Matthias Baas <matthias.baas@...> wrote:
Trevor Lovett wrote:
> Tried installing cgkit2 via the .dmg but it complains that it requires
> python 2.5 even though I'm running 2.5, it's active and all other
> python's on my system have been uninstalled.

As you mentioned MacPorts before, I'm assuming you are running the
MacPorts Python version, right? As far as I know this one doesn't work
with the dmg as its built differently. I think you would have to use the
version from python.org (which I think is a frameworks build... but
frankly, I'm not quite sure myself what the exact differences are).

> Also tried building cgkit-2.0.0alpha8 from src but when building the
> supportlib via scons I get a bunch of errors.

Hm, if you want you can send me the errors and I'll see if it's
something obvious....
But I understand that you actually want to work on other stuff.
Anyway, for the Aqsis RTS you could just install the "light" version of
cgkit which doesn't require any compilation at all (see the first
paragraph on this page to find out how to do it:
http://cgkit.sourceforge.net/doc2/node10.html)

Or you can just grab the archive from my previous mail and use the RIBs
instead of the Python scripts.

- Matthias -


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development
< Prev | 1 - 2 | Next >