Attempting to run qmtests

[Mark O'Donohue]

Hi

It has been a while since I last ran qmtest for firebird, so I was
following the  instructions at:

http://www.firebirdsql.org/index.php?op=devel&sub=qa&id=qmtest_howto


All is dandy,

$python qadbm.py
   > repository /home/fbqa
   > platform Windows
   > target 1.5.2
   > create /home/fbtests
   > copy /home/fbtests
   > exit


Then you get to "Configuration" :

    It lists a group of key value pairs. They look nice, but what does
    one do with them?   They really look like they want to go in a
    config file or be environment variables. There is a
    /home/fbtest/QMtest/configuration file but that is full of XML so
    I dont think they below there - or do they?

The final step is to run qmtest, and list the cmd options.

    Does someone mind filling me in on what to do, in terms of completing
    the above howto example. I assume it then goes something like:

    $cd /home/fbtests
    $qmtest ls
    $qmtest run

    $qmtest report

And I guess it would be handy to run a subset of tests or an
invidiual test, presumaby that is by:

    $qmtest ls functional.arno
    $qmtest run functional.arno


But at the moment I am stuck on configuration :-).

Cheers - Mark




------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Firebird-test mailing list
Firebird-test@...
https://lists.sourceforge.net/lists/listinfo/firebird-test

[Pavel Cisar]

Mark O'Donohue napsal(a):
>
> Then you get to "Configuration" :
>
>     It lists a group of key value pairs. They look nice, but what does
>     one do with them?   They really look like they want to go in a
>     config file or be environment variables. There is a
>     /home/fbtest/QMtest/configuration file but that is full of XML so
>     I dont think they below there - or do they?

You need to create a text file with these variables with values valid
for your system. Then you will specify this file when running qmtest
using the -C option.

> The final step is to run qmtest, and list the cmd options.
>
>     Does someone mind filling me in on what to do, in terms of completing
>     the above howto example. I assume it then goes something like:
>
>     $cd /home/fbtests
>     $qmtest ls
>     $qmtest run
>
>     $qmtest report

The report command is not so important. The run command should produce
the results.qmr file that you can rename (to have platform, fb arch. a
run number, for example wcs1.qmr fro Windows Classic run 1) and send to
me for further processing.

If you want to look at results, I would advise to use the web gui
(qmtest qui command).

> And I guess it would be handy to run a subset of tests or an
> invidiual test, presumaby that is by:
>
>     $qmtest ls functional.arno
>     $qmtest run functional.arno

Yes.

best regards
Pavel Cisar
IBPhoenix


------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Firebird-test mailing list
Firebird-test@...
https://lists.sourceforge.net/lists/listinfo/firebird-test

[Mark O'Donohue]

Hi Pavel

Thanks for the tip, I have got further, but now I am getting groups of
failed tests.

I am running against the standard ubuntu fb2.0.3 package install,

   $qmtest run -C context.txt functional

gives:
    --- STATISTICS --------------------------------

      358        tests total
       88 ( 25%) tests FAIL
      270 ( 75%) tests PASS


88 failures seems a little high.  Looking at the first failure, which is


      test case: functional.arno.derived_tables.derived_tables_06

That test is:

    SELECT
       dt.*
     FROM
       Table_10 t10
       JOIN (SELECT * FROM Table_10 t2 WHERE t2.ID = t10.ID)
               dt ON (1 = 1);

When I ran that test using isql it gives the "Column Unknown: T10.ID" as
expected.


The QMTest result complained it found an extra "=":

      -Token unknown - line 5, column 2
      -=

So QMTest has an extra "=" inserted somewhere.

So any idea of what is going wrong (I have included more details below) ?

Cheers - Mark



Setup is :

./qadbm.py
QADBM>platform Linux
QADBM>target 2.0.3
QADBM>create fbtest2.0.3
QADBM>copy fbtest2.0.3
QADBM>exit

I create a context.txt file for ubuntu/2.0.3 as :

temp_directory=/var/tmp/
server_location=localhost:
database_location=/var/firebird/test/
 
suite_database_location=/home/odonohue/work/firebird/testing/qa/qa/trunk/testsuite/fdb/
backup_location=/home/odonohue/work/firebird/testing/qa/qa/trunk/testsuite/fdk/
files_location=/home/odonohue/work/firebird/testing/qa/qa/trunk/testsuite/files/
isc4_path=/var/lib/firebird/2.0/system/security2.fdb
user_name=SYSDBA
user_password=<a secret noone will ever guess>
isql_path=/usr/bin/isql-fb
gsec_path=/usr/bin/gsec
gstat_path=/usr/bin/gstat
gbak_path=/usr/bin/gbak
nbackup_path=/usr/bin/nbackup
gfix_path=/usr/bin/gfix
gpre_path=/usr/bin/gpre


Then I run :

   cd fbtest2.0.3
   qmtest run -C context.txt functional

It produced the results file:

   $ ls -lh results.qmr
   -rw-r--r-- 1 odonohue odonohue 165K 2008-12-13 12:03 results.qmr

I was hoping for a basic report and tried most of the options for:

    $qmtest report -f results.qmr

    but it just kept replying with :
        usage: qmtest report [ OPTIONS ] [ result [-e expected] ]+
    and although -e is optional, I wasnt sure where the was.

So I did the gui thing.

    $qmtest gui

And fired up the browser.   I then chose File/Load Results, selected the
results.qmr file, and that seemed to work.

navagating to the first failue:
        functional.arno.derived_tables.derived_tables_06

The difference is:

Actual result:
      Statement failed, SQLCODE = -104
      Dynamic SQL Error
      -SQL error code = -104
      -Token unknown - line 5, column 2
      -=

Expected result:
      Statement failed, SQLCODE = -206
      Dynamic SQL Error
      -SQL error code = -206
      -Column unknown
      -T10.ID
      -At line 5, column 53

>
> The report command is not so important. The run command should produce
> the results.qmr file that you can rename (to have platform, fb arch. a
> run number, for example wcs1.qmr fro Windows Classic run 1) and send to
> me for further processing.
>

What further processing do you do?




------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Firebird-test mailing list
Firebird-test@...
https://lists.sourceforge.net/lists/listinfo/firebird-test

[Helen Borrie]

Mark,

At 01:36 PM 13/12/2008, you wrote:

>Hi Pavel
>
>Thanks for the tip, I have got further, but now I am getting groups of
>failed tests.
>
>I am running against the standard ubuntu fb2.0.3 package install,

Wouldn't you do better installing and testing with the official project distro, so that you actually know what you're working with?  Ubuntu's stuff comes from Debian and Debian was calling Firebird 1.5 "Firebird 2" for a long time, possibly still is...

>   $qmtest run -C context.txt functional
>
>gives:
>    --- STATISTICS --------------------------------
>
>      358        tests total
>       88 ( 25%) tests FAIL
>      270 ( 75%) tests PASS
>
>
>88 failures seems a little high.  

Firebird 1.5 (posing as Firebird 2) wouldn't pass any tests set up to test new language features in (the real) Firebird 2.0.

Just a thought...on the opposite side of the coin, at least you might be able to demonstrate how much up the creek the Debian/Firebird situation has been at defined points in its evolution. ;-)

Helen


------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Firebird-test mailing list
Firebird-test@...
https://lists.sourceforge.net/lists/listinfo/firebird-test

[Mark O'Donohue]

Hi Helen

Helen Borrie wrote:
> Mark,
>
>>
>> I am running against the standard ubuntu fb2.0.3 package install,
>
> Wouldn't you do better installing and testing with the official
> project distro, so that you actually know what you're working with?
>

Careful, Helen they will hear you :-).

The they in debian in terms of who maintains the packages there is
generally us; at least thats my take from what I read in
pkg-firebird-general; and packaging within constrains imposed on you
ain't easy.

Besides what is wrong with a package name of firebird2_1.5.3 :-).


However, my general aim is to test "my" build, but to do that I wanted
to run a base test set for reference.

The Ubuntu 2.0.3 firebird looks legit it reports:

$ isql-fb -z
ISQL Version: LI-V2.0.3.12981 Firebird 2.0
Use CONNECT or CREATE DATABASE to specify a database
SQL>


And the actual test ran fine, when I did it by hand.


So I actually suspect some trickery in qmtest. It would be nice to be
able to get in and trace it a little better - bring back DECTest thats
what I say :-).


Cheers - Mark




  .

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Firebird-test mailing list
Firebird-test@...
https://lists.sourceforge.net/lists/listinfo/firebird-test

[Philippe Makowski]

Mark,
> And the actual test ran fine, when I did it by hand.
>
and what result if you run only one test, for example
$qmtest run -C context.txt functional.arno.derived_tables.derived_tables_06

?

may be you have a qmtest problem
which version are you using ?





------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Firebird-test mailing list
Firebird-test@...
https://lists.sourceforge.net/lists/listinfo/firebird-test

[Mark O'Donohue]

I found the problem, and Helen will give me an i-told-you-so for it I am
sure!

It seemed that any test with lines > 64 characters in length fails.

even a:

/usr/bin/isql-fb < test.sql

will ignore the first 64 characters for lines that are longer than 64
characters - this is a problem with the libeditline provided in Unbuntu
(and Debian).

Mind you it took a bit of digging though and looking at the internals of
qmtest and fbqa.py before I figured that out.

The bug doesnt occur in our older localized version of libeditline.
There isnt any reported bug for Debian or Ubuntu but someone fixed
something similar on minix-editline.  So I created a bug report for Unbuntu.

    https://bugs.launchpad.net/ubuntu/+source/editline/+bug/307652

Then having gone that far, it wasn't too hard then to then give them a
patch to fix is as well.


Philippe Makowski wrote:
> Mark,
>> And the actual test ran fine, when I did it by hand.
>>
> and what result if you run only one test, for example
> $qmtest run -C context.txt functional.arno.derived_tables.derived_tables_06
>

yep, I was doing that, and comparing out put to one that worked, then I
added some code within FirebirdTest to debug what was actually sent
through to isql, since it does several substitutions to the input.
Finally when dumping and comparing stdin/stderr/stdout for a working and
non-working cases, I tried the line length.

Anyway, a side effect is I've added some debugging code that will with a
switch print the SQL code, used for setup and execution when you run the
tests.

Now where was I...

Cheers - Mark


------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Firebird-test mailing list
Firebird-test@...
https://lists.sourceforge.net/lists/listinfo/firebird-test

[Pavel Cisar]

Hi Mark,

Mark O'Donohue napsal(a):

> Hi Pavel
>
> Thanks for the tip, I have got further, but now I am getting groups of
> failed tests.
>
> I am running against the standard ubuntu fb2.0.3 package install,
>
>    $qmtest run -C context.txt functional
>
> gives:
>     --- STATISTICS --------------------------------
>
>       358        tests total
>        88 ( 25%) tests FAIL
>       270 ( 75%) tests PASS
>
>
> 88 failures seems a little high.  Looking at the first failure, which is
>
>
>       test case: functional.arno.derived_tables.derived_tables_06
>
> That test is:
>
>     SELECT
>        dt.*
>      FROM
>        Table_10 t10
>        JOIN (SELECT * FROM Table_10 t2 WHERE t2.ID = t10.ID)
>                dt ON (1 = 1);
>
> When I ran that test using isql it gives the "Column Unknown: T10.ID" as
> expected.
>
>
> The QMTest result complained it found an extra "=":
>
>       -Token unknown - line 5, column 2
>       -=
>
> So QMTest has an extra "=" inserted somewhere.
>
> So any idea of what is going wrong (I have included more details below) ?

Now I realized that you may use wrong (outdated) version of qmtest, if
you downloaded qmtest from original site. It doesn't contain recent
patches and might have trouble with Python 2.5.

Please download this version and try again
http://www.firebirdsql.org/download/prerelease/python/qmtest-2.4.1070.tar.gz

There is also a Windows installer if you want to run it under Windows too.

best regards
Pavel Cisar
IBPhoenix

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Firebird-test mailing list
Firebird-test@...
https://lists.sourceforge.net/lists/listinfo/firebird-test

[Mark O'Donohue]

Hi Pavel

On Sat, Dec 13, 2008 at 9:39 PM, Pavel Cisar <pcisar@...> wrote:
>
> Now I realized that you may use wrong (outdated) version of qmtest, if
> you downloaded qmtest from original site. It doesn't contain recent
> patches and might have trouble with Python 2.5.
>
> Please download this version and try again
> http://www.firebirdsql.org/download/prerelease/python/qmtest-2.4.1070.tar.gz
>


Thanks, I will use that, I am on qmtest 2.4, but probably that is a
later build.   But for this problem I am 100% sure it was editline,
starting with :

http://packages.qa.debian.org/e/editline.html

and linking their own test program with Ubuntu provided libraries, and
then building the ones in the package, I got the same error, entirely
without firebird or qmtest.

Now, to download a version from our website, and will start Today over
again  ....

Cheers - Mark

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Firebird-test mailing list
Firebird-test@...
https://lists.sourceforge.net/lists/listinfo/firebird-test