Nabble - monetdb-bugs

[ monetdb-Bugs-1607210 ] XQ: server-side compilation crash (member benchmark)

2009-11-21T06:35:03Z

Bugs item #1607210, was opened at 2006-12-02 01:27
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1607210&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/runtime
Group: Pathfinder CVS Head
Status: Open
Resolution: Fixed
Priority: 8
Private: No
Submitted By: Peter Boncz (boncz)
Assigned to: Peter Boncz (boncz)
Summary: XQ: server-side compilation crash (member benchmark)

Initial Comment:
Philippe Michiels reported at dagstuhl that the simple CHILD test of the MemBeR benchmark crashes when run in MapiClient -lx; crashing during the *compilation* process

It works with pf standalone, translating to MIL explicitly

http://ilps.science.uva.nl/Resources/MemBeR/CHILD.html

The query used in this micro-benchmark navigates downward from the root, performing n successive child navigation steps, where n is to be varied.
For every integer n, the corresponding query, named CHILD(n), is:

/t1/t2/.../tn

THE BUG ONLY OCCURS WITH n>15
AND WITH SERVER SIDE COMPILATION

may well be some buffer overflow..

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-21 15:35

Message:
===================================================================
2009/11/21 - stmane: pathfinder/runtime/pathfinder.mx,1.462.4.12

use more conservative stack limit (PFmaxstack)
for too deep recusion check (PFrecursion_fence())
to prevent segfault (due to too deep recursion and
consequent stack overflow) in tests
tests/BugTracker/Tests/collection_management.SF-1726599.*
tests/BugTracker/Tests/crash_on_concatenated_query.SF-1730547.*
cf.,
https://sourceforge.net/tracker/?func=detail&aid=1607210&group_id=56967&atid=482468
https://sourceforge.net/tracker/?func=detail&aid=1730547&group_id=56967&atid=482468
===================================================================

Report can be closed once nightly testing confirms the fix for all
platforms.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-16 22:18

Message:
re-opened as the respective test fails on all platform with timeout or
segfault at least since 2009.03.25
cf.,
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/server-side_compilation_crash.SF-1607210.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/server-side_compilation_crash.SF-1607210.out.00.html

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2008-02-10 14:12

Message:
Logged In: YES
user_id=572415
Originator: NO

Appears to work fine on most platforms, again;
exceptions:

Segfault on RedHat (on Itanium):
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.64.d-RedHat4WS/tests_BugTracker/server-side_compilation_crash.SF-1607210.err.00.html

Timeout on Windows:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.32.d-Windows5.2/tests_BugTracker/server-side_compilation_crash.SF-1607210.err.00.html

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2008-01-22 14:58

Message:
Logged In: YES
user_id=572415
Originator: NO

re-opened as it fails, again, since 2008.01.12, i.e., after checkins on
2008.01.11, now with

ERROR = !fatal error: aborted too deep recursion

http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/GNU.32.32.d-Fedora6/tests_BugTracker/server-side_compilation_crash.SF-1607210.err.00.html

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2007-06-05 18:37

Message:
Logged In: YES
user_id=591107
Originator: YES

fixed

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-06-04 21:19

Message:
Logged In: YES
user_id=572415
Originator: NO

See also
[ 1730547 ] XQ: Mserver crashes on concatenated query
http://sourceforge.net/tracker/index.php?func=detail&aid=1730547&group_id=56967&atid=482468

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-05-29 10:24

Message:
Logged In: YES
user_id=572415
Originator: NO

See also
http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/GNU.64.64.d-Fedora6/tests_BugTracker/server-side_compilation_crash.SF-1607210.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/GNU.64.64.d-Fedora6/tests_BugTracker/server-side_compilation_crash.SF-1607210.out.00.html

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-05-28 12:33

Message:
Logged In: YES
user_id=572415
Originator: NO

Added test in
pathfinder/tests/BugTracker/Tests/server-side_compilation_crash.SF-1607210*

As opposed to
[ 1207048 ] XQuery: crashtest
http://sourceforge.net/tracker/index.php?func=detail&aid=1207048&group_id=56967&atid=482468
http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/GNU.64.64.d-Fedora6/tests_BugDay_2005-11-09_0.9.3/xquery_crashtest.SF-1207048.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/GNU.64.64.d-Fedora6/tests_BugDay_2005-11-09_0.9.3/xquery_crashtest.SF-1207048.err.00.html

it still crashes with a segfault, i.e.,
the new fence does not catch the stack overflow in this case, yet.

Not analysed, yet.

Re-opend to keep us reminded to have another look at this one.

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2007-05-25 23:55

Message:
Logged In: YES
user_id=591107
Originator: YES

fixed using a fence

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2006-12-11 16:10

Message:
Logged In: YES
user_id=572415
Originator: NO

I am pretty sure, this is a problem with recursive burg-generated code
running out of stack space, which in unfortunately out of out stack space
control, and hence IMHO very hard to prevent/fix...
With "pf" standalone, it happens as well, just with a longer path
expression (29 child steps in my case):

$ n=28 ; i=0 ; q='doc("MyDoc.xml")' ; while [ $i -lt $n ] ; do i=$[i+1] ;
q="$q`printf "/t%02d" $i`" ; done ; echo $q | tee /tmp/q.xq ; pf /tmp/q.xq
>/dev/null
doc("MyDoc.xml")/t01/t02/t03/t04/t05/t06/t07/t08/t09/t10/t11/t12/t13/t14/t15/t16/t17/t18/t19/t20/t21/t22/t23/t24/t25/t26/t27/t28

$ n=29 ; i=0 ; q='doc("MyDoc.xml")' ; while [ $i -lt $n ] ; do i=$[i+1] ;
q="$q`printf "/t%02d" $i`" ; done ; echo $q | tee /tmp/q.xq ; pf /tmp/q.xq
>/dev/null
doc("MyDoc.xml")/t01/t02/t03/t04/t05/t06/t07/t08/t09/t10/t11/t12/t13/t14/t15/t16/t17/t18/t19/t20/t21/t22/t23/t24/t25/t26/t27/t28/t29
Segmentation fault

$ gdb --args pf /tmp/q.xq
GNU gdb Red Hat Linux (6.3.0.0-1.84rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
libthread_db library "/lib64/libthread_db.so.1".

(gdb) r
Starting program:
/net/corona.ins.cwi.nl/export/scratch0/manegold/Monet/Testing/Stable/.GNU.64.32.d.--disable-optimize_--enable-debug_--enable-assert.PREFIX./PATHFINDER/bin/pf
/tmp/q.xq
[Thread debugging using libthread_db enabled]
[New Thread 46912496328800 (LWP 30516)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 46912496328800 (LWP 30516)]
0x0000000000483666 in reduce (p=0x2aaaaaae0b90, goalnt=50) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/core/fs.brg:874
874 rule = PFfs_rule (STATE_LABEL (p), goalnt);
(gdb) li
869 PFpnode_t *kids[MAX_KIDS]; /* leaf nodes of this rule */
870 bool topdown; /* is this a top-down rule? */
871 PFcnode_t *c; /* temporary helper variable */
872
873 /* determine rule that matches for this non-terminal */
874 rule = PFfs_rule (STATE_LABEL (p), goalnt);
875
876 /* PFinfo (OOPS_NOTICE, "match for rule %i", rule); */
877 assert (rule);
878
(gdb) up
#1 0x00000000004837d9 in reduce (p=0x2aaaaaae0b90, goalnt=49) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/core/fs.brg:929
929 reduce (kids[i], nts[i]);
(gdb) li
924 /*
925 * Recursively invoke compilation. This means bottom-up
compilation.
926 */
927 if (!topdown)
928 for (unsigned short i = 0; nts[i]; i++)
929 reduce (kids[i], nts[i]);
930
931 /*
932 PFinfo (OOPS_NOTICE, "processing rule %i", rule);
933 */
(gdb) bt
#0 0x0000000000483666 in reduce (p=0x2aaaaaae0b90, goalnt=50) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/core/fs.brg:874
#1 0x00000000004837d9 in reduce (p=0x2aaaaaae0b90, goalnt=49) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/core/fs.brg:929
[...]
#680 0x00000000004837d9 in reduce (p=0x2aaaaaae5f00, goalnt=1) at #680
0x00000000004837d9 in reduce (p=0x2aaaaaae5f00, goalnt=1) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/core/fs.brg:929
#681 0x0000000000491eaf in PFfs (r=0x2aaaaaae5f00) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/core/fs.brg:4209
#682 0x00000000004065da in PFcompile (url=0x7ffffffb672e "/tmp/q.xq",
pfout=0x38ecf318c0, status=0x731f40) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/compile.c:381
#683 0x0000000000405f11 in main (argc=2, argv=0x7ffffffb4e78) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/main.c:905
(gdb)
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/core/fs.brg:929
#681 0x0000000000491eaf in PFfs (r=0x2aaaaaae5f00) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/core/fs.brg:4209
#682 0x00000000004065da in PFcompile (url=0x7ffffffb672e "/tmp/q.xq",
pfout=0x38ecf318c0, status=0x731f40) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/compile.c:381
#683 0x0000000000405f11 in main (argc=2, argv=0x7ffffffb4e78) at
/ufs/manegold/_/scratch0/Monet/Testing/Stable/pathfinder/compiler/main.c:905
(gdb)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2006-12-02 08:38

Message:
Logged In: YES
user_id=572415
Originator: NO

How does it crash?
Is there any error message or exit status?
Does it work for stand-alone pf also for n >> 15?

Might well be a stack-overflow in some burg generated code...

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1607210&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-1730547 ] XQ: Mserver crashes on concatenated query

2009-11-21T06:34:54Z

Bugs item #1730547, was opened at 2007-06-04 10:42
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1730547&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/runtime
Group: Pathfinder CVS Head
Status: Open
Resolution: Fixed
Priority: 8
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Stefan Manegold (stmane)
Summary: XQ: Mserver crashes on concatenated query

Initial Comment:
The attached query looks like:

"aap",
"aap",
"aap",
"aap",
... (1000 times)
"aap",
"aap"

When given to Mserver, it says: Segmentation Fault (0.12 as well as 0.17 (build last week)).

This bug should not have high priority.

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-21 15:34

Message:
===================================================================
2009/11/21 - stmane: pathfinder/runtime/pathfinder.mx,1.462.4.12

use more conservative stack limit (PFmaxstack)
for too deep recusion check (PFrecursion_fence())
to prevent segfault (due to too deep recursion and
consequent stack overflow) in tests
tests/BugTracker/Tests/collection_management.SF-1726599.*
tests/BugTracker/Tests/crash_on_concatenated_query.SF-1730547.*
cf.,
https://sourceforge.net/tracker/?func=detail&aid=1607210&group_id=56967&atid=482468
https://sourceforge.net/tracker/?func=detail&aid=1730547&group_id=56967&atid=482468
===================================================================
2009/11/21 - stmane:

pathfinder/tests/BugTracker/Tests/crash_on_concatenated_query.SF-1730547.timeout,1.1.2.1

extended timeout for slow(er) systems
===================================================================

Report can be closed once nightly testing confirms the fix for all
platforms.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-17 10:15

Message:
Instead of a "ERROR = !fatal error: aborted too deep recursion"
the test now (again) triggers a timeout or segfault on all platform:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/crash_on_concatenated_query.SF-1730547.err.00.html

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2008-11-19 14:52

Message:
It was the "expected" output, since this test used to trigger a "ERROR =
!fatal error: aborted too deep recursion" (and hence produce no output) on
all platforms.

Apparently, this is no longer the case but now some platforms are able to
run the test without error, i.e., producing some (correct? -> to be
checked! ...) output.

Maybe, we need to split this test into two, a small one that works on all
platforms and a huge one that fails (i.e., triggers a reasonable error
message) on all platforms ...

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2008-11-19 14:42

Message:
Hmm I'm not sure if the 'correct' output is indeed the correct one:
http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/Int.64.32.d.1-Fedora8/tests_BugTracker/crash_on_concatenated_query.SF-1730547.out.00.html

@ Stefan could you please verify.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-06-05 20:44

Message:
Logged In: YES
user_id=572415
Originator: NO

Test added in
pathfinder/tests/BugTracker/Tests/Attic/crash_on_concatenated_query.SF-1730547.*

Thanks to Peter's latest fix, it now correctly triggers
"ERROR = !fatal error: aborted too deep recursion"
instead of crashing the server.

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2007-06-05 18:31

Message:
Logged In: YES
user_id=591107
Originator: NO

fixed

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2007-06-05 18:31

Message:
Logged In: YES
user_id=591107
Originator: NO

fixed

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-06-04 21:19

Message:
Logged In: YES
user_id=572415
Originator: NO

Debugging releaved that the segfault is indeed caused by burg running out
of stack space due to too deep recursion.

See also
[ 1607210 ] XQ: server-side compilation crash (member benchmark)
http://sourceforge.net/tracker/index.php?func=detail&aid=1607210&group_id=56967&atid=482468

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-06-04 18:21

Message:
Logged In: YES
user_id=572415
Originator: NO

for got to mention:
pf "eventually" produces some MIL plan; no error.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-06-04 18:20

Message:
Logged In: YES
user_id=572415
Originator: NO

Running this through plain pf (0.18), i.e.,

{ for i in `seq 1 999` ; do echo '"aap",' ; done ; echo '"aap"' ; } | pf

makes pf grow to 2.7 GB virt (1.8 GB res) on my 64-bit FC6 machine;
hence, this might well be a memory and/or burg problem,
especially on 32-bit machines.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1730547&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-1726599 ] XQuery: collection management broken

2009-11-21T02:42:22Z

Bugs item #1726599, was opened at 2007-05-27 22:32
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1726599&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/runtime
Group: Pathfinder CVS Head
>Status: Closed
Resolution: Fixed
Priority: 8
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Peter Boncz (boncz)
Summary: XQuery: collection management broken

Initial Comment:
Today's MonetDB 4.17

Given a small document: <aap id="1"/>

shredding it once, deleting it, and shredding it again corrupts the database.

[wouter@localhost test]$
/opt/MonetDB-4.17/bin/MapiClient -lxq -p50170
xquery>pf:add-doc("/home/wouter/test.xml","doc1.xml")
more>xquery>pf:del-doc("doc1.xml")
more>xquery>pf:add-doc("/home/wouter/test.xml","doc1.xml")
more>xquery>pf:collection("doc1.xml")
more><aap id="1"/><aap id="1"/>
xquery>

At first sight the document still looks intact:

xquery>doc("doc1.xml")
more><?xml version="1.0" encoding="utf-8"?>
<aap id="1"/>

But then it looks like the document's document node has gone, because:

xquery>doc("doc1.xml")/aap
more>
xquery>

gives an empty result.

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-21 11:42

Message:
===================================================================
2009/11/20 - boncz:

pathfinder/tests/BugTracker/Tests/collection_management.SF-1726599.stable.err,1.3.24.1

pathfinder/tests/BugTracker/Tests/collection_management.SF-1726599.stable.out,1.3.24.1
approving test output
===================================================================

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-16 22:20

Message:
re-opened as the test fails on all platforms at least since 2009.04.09
cf.,
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/collection_management.SF-1726599.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.32.32.d.1-Debian4.0/tests_BugTracker/collection_management.SF-1726599.err.00.html

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2007-06-08 02:23

Message:
Logged In: YES
user_id=591107
Originator: NO

fixed

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-05-29 10:26

Message:
Logged In: YES
user_id=572415
Originator: NO

See also
http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/GNU.64.64.d-Fedora6/tests_BugTracker/collection_management.SF-1726599.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/GNU.64.64.d-Fedora6/tests_BugTracker/collection_management.SF-1726599.out.00.html

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-05-28 12:55

Message:
Logged In: YES
user_id=572415
Originator: NO

Test added in
pathfinder/tests/BugTracker/Tests/collection_management.SF-1726599*

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2007-05-28 01:16

Message:
Logged In: YES
user_id=591107
Originator: NO

yes indeed, it seems that:

- pf:del-doc() does not cleanup empty-collections, while MIL delete_doc()
does this
as a result, the second add-doc adds the document to the previous
collection

- there seems to be a bug in index filtering for deleted documents in a
multi-doc collection
as the functinality is new, so it is not that surprising

will fix soon

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1726599&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2808353 ] SQL: not (...) goes far out of hand

2009-11-20T06:18:28Z

Bugs item #2808353, was opened at 2009-06-18 12:48
Message generated for change (Settings changed) made by nielsnes
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2808353&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SQL/Core
Group: SQL CVS Head
>Status: Closed
>Resolution: Rejected
Priority: 3
Private: Yes
Submitted By: Stefan de Konink (skinkie)
Assigned to: Niels Nes (nielsnes)
Summary: SQL: not (...) goes far out of hand

Initial Comment:
To create the table; /export refers to loki.

create table kvk (kvk bigint, bedrijfsnaam varchar(255), adres varchar(64),
postcode varchar(6), plaats varchar(32), type varchar(16));
create table anbikvk (naam varchar(256), vestigingsplaats varchar(32), besch
ikking date, intrekking date, kvk bigint);
create table anbi (naam varchar(256), vestigingsplaats varchar(32), beschikk
ing timestamp, intrekking timestamp);
COPY 27598 RECORDS INTO anbi FROM '/export/scratch1/konink/anbi.csv' USING DELIMITERS ',', '\n', '""';
COPY 1621594 RECORDS INTO kvk FROM '/export/scratch1/konink/allesn' USING DELIMITERS ',', '\n', '""';
insert into anbikvk select naam, vestigingsplaats, beschikking, intrekking,
kvk from kvk,anbi where lower(naam) = lower(bedrijfsnaam) and lower(plaats) = lo
wer(vestigingsplaats);
select count(*) from kvk,anbi where not (lower(naam) = lower(bedrijfsnaam) and lower(plaats) = lowers(vestigingsplaats));

Known errors are:
- Segmentation fault (64/32)
- BUN error
- !ERROR: GDKsave: error on: name=54/5412, ext=head, mode=1
!OS: No space left on device
!ERROR: GDKload: failed name=54/5412, ext=head

The database is by far smaller than main memory.

----------------------------------------------------------------------

>Comment By: Niels Nes (nielsnes)
Date: 2009-11-20 14:18

Message:
closing as the generated plan is correct. The crossproduct is in this case
needed because of the not and and lower. As long as we do not have a case
insensitive join, rewriting lower(x) = lower(y) isn't possible/usefull.
No test possible as the crash (caused by the large crossproduct result)
would re quire a bit table.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-08-14 11:58

Message:
tagged subject

----------------------------------------------------------------------

Comment By: Stefan de Konink (skinkie)
Date: 2009-08-11 00:30

Message:
Points:
1) Can lower = lower (or upper = upper) be recognized in the parser as
case insensitive comparison? Instead of transforming two entire columns to
another value.

Plan:
#project (
#| group by (
#| | select (
#| | | crossproduct (
#| | | | table(sys.kvk) [ kvk.bedrijfsnaam, kvk.plaats, kvk.%TID% NOT NULL
],
#| | | | table(sys.anbi) [ anbi.naam, anbi.vestigingsplaats, anbi.%TID%
NOT NULL ]
#| | | )
#| | ) [ not(and(=(lower(anbi.naam), lower(kvk.bedrijfsnaam)),
=(lower(kvk.plaats), lower(anbi.vestigingsplaats)))) = true ]
#| ) [ ] [ count NOT NULL as L1 ]
#) [ L1 NOT NULL ]

Trace (doesn't run on a full database) as will be attached.

----------------------------------------------------------------------

Comment By: Martin Kersten (mlkersten)
Date: 2009-08-06 14:03

Message:
To be further investigated when the minimal database + testscript is
provided.
Alternatively, a TRACE of this query against the original might provide
the information needed.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2808353&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2686008 ] SQL: multi-attribute GROUP BY may ignore sortedness

2009-11-20T05:05:37Z

Bugs item #2686008, was opened at 2009-03-13 09:43
Message generated for change (Settings changed) made by nielsnes
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2686008&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SQL/Core
Group: SQL "stable"
>Status: Closed
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Roberto Cornacchia (cornuz)
Assigned to: Niels Nes (nielsnes)
Summary: SQL: multi-attribute GROUP BY may ignore sortedness

Initial Comment:
-- A table with unsorted values in a and b
create table unsorted (a int,b int);
insert into unsorted values (2, 3);
insert into unsorted values (1, 2);
insert into unsorted values (4, 1);
insert into unsorted values (3, 2);
insert into unsorted values (2, 3);
insert into unsorted values (3, 3);
insert into unsorted values (3, 1);
insert into unsorted values (4, 3);

-- Store it in a new table with tuples sorted on a,b
create table sorted (a int, b int);
insert into sorted
select * from unsorted
order by a,b;

-- these tho are semantically equivalent (the group by attributes are swapped)
trace select a,b from sorted group by a,b;
trace select a,b from sorted group by b,a;

The first version is much faster, as it exploits the sortedness on a,b when grouping on a,b.
The second version identifies the same groups. Therefore, the sortedness on a,b could be exploited when grouping on b,a, but it isn't.

Even on such a small example, the trace shows already a performance difference, which becomes dramatic on larger data.

In general, when a table is sorted on a1,a2,...,aN ,
grouping on any permutation of a1,a2,...,aK with K<=N should exploit sortedness.

----------------------------------------------------------------------

>Comment By: Niels Nes (nielsnes)
Date: 2009-11-20 13:05

Message:
added a test to
src/test/BugTracker-2009/Tests/use_order_column_first.SF-2686008.sql

fixed by an optimizer which checks for sorted columns in the group by
(only works when we can easily derive the base table)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-03-13 09:51

Message:
Good point --- while performance can be considered bugs, a solution might
only become available as new feature in the CVS HEAD and hence in the next
feature release --- Martin's recently added "derivepath" MAL optimizer
provides a framework that can be exploited to do the proposed (and other,
e.g., exploit also uniqueness) optimization for multi-attribute grouping,
very similar to what the "joinpath" optimizer does for (binary- / mapping-
/ pivot-) join paths.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2686008&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-19T12:15:56Z

Bugs item #2898944, was opened at 2009-11-17 09:27
Message generated for change (Comment added) made by tsheyar
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "candidate"
Status: Open
Resolution: Fixed
Priority: 6
Private: No
Submitted By: Axel Belinfante (axelbel)
Assigned to: Jan Rittinger (tsheyar)
Summary: pf takes 'forever' (> 1 hour) to compile particular query

Initial Comment:
I don't know whether the following really qualifies as bug, or just is a case of 'user error by giving stupid input.
In any case, I would like to bring this to your attention.

When I compile a query for attached module definition with pf (may or aug or pre-nov version), compilation takes 'forever'.
When I replace the clause ecv:to-human($g0, $human_type) by simple $g0
and remove definitions of $human_type and to-human it compiles almost instantly.
It is the huge difference in compilation time that surprises me.

The query I compile is:

import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation.xq"; ecv:view-from-save("eprints-export-conv-fmt.xml")

where tmp/long-compilation.xq is the attached module.

What the query attempts to do: obtain results from the database, order these on two criteria, and when listing the results, insert human-readable headers.
The human-readable headers are obtained by the to-human mapping from field value (used as ordering criterium) to human-readable string.
At some point I just used if-then-else-if-then-else etc. to do the mapping, but that seemed slower to compile than the approach I took here.

Suggestions for rewriting the query to a fast-compiling one would be considered as a fix too, I guess.

----------------------------------------------------------------------

>Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-19 21:15

Message:
Alex,

if you want to toy around with the algebraic optimizations (option -o of
pf) and its effect on compile and runtime, then feel free to try out the
following optimizations:

-oOIKCGP
-oOIKCG_VG_IS_GYSVR_DC_GP

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-19 18:22

Message:
time pf -oOIKCG_VG_IS_GYECSVR_DC_GP /tmp/long-compilation3.req >
/tmp/long-compilation3.mil
15.868u 0.472s 0:16.33 100.0% 0+0k 0+0io 0pf+0w

time mclient -lmil -p 51012 /tmp/long-compilation3.mil > /dev/null
0.016u 0.032s 0:03.80 1.0% 0+0k 0+0io 0pf+0w

compilation time is indeed much, much better (though still a bit long)
run time of resulting .mil is kind of ok (even faster would be even
better, of course :-)

anything else I could try?
otherwise, I guess I will just use the options you suggested when
compiling those generated queries,
and that should do for now.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-19 18:04

Message:
Axel,

you hit different problems here. Your original bug caused the planner to
take generate millions of plans.

Your latest example however generates a plan that features 5420 operators
out of which 660 are joins. Here the join optimization takes a little
bit...

What happens if you don't optimize much? -- Try e.g., the following (if
you leave out the 'E' letter compilation time will probably halve):

echo 'import module namespace ecv =
"http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation3.xq";
ecv:view-from-save("eprints-export-conv-fmt.xml")' | pf
-oOIKCG_VG_IS_GYECSVR_DC_GP > foo; Mserver foo

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-19 14:13

Message:
I have tried the latest fix.
there is a good thing, and there is a bad thing.

the good thing is that for these queries the fix works.
that is really, really great.

the bad thing is that it is easy to add complexity (add more of the same
thing),
and again have a relatively slow compilation - although, due to the fix
now compilation does finish.

fact is, those queries are not hand-written, but generated from more
high-level user input.
it is easy for the user to make a slight change to the intended
presentation of the query results
(add more refined structuring/grouping of the results), which can result
in query for which
again compilation time is long.

to give an example:
in long-compilation2.xq I have 3 levels of structure. it compiles in about
3 secs.
file long-compilation3.xq (attached) has 6 levels of structure. it
compiles in approx. 37 minutes.
running the compiled query takes iat most a couple of seconds

I tried rewriting the queries.
I could only obtain reduced compilation time (only 2 minutes) by adding
additional
(pre=-processed) data to the database.
even then, running the compiled query takes approximately a minute...

(I also tried compiling with -M -- this is fast, but I was unable to run
it
MAPI = dbguest@localhost:51012
QUERY = proc_vid.insert("fnC8CAED69_print_4_eprint_xs_element1",
1059825001LL);
...
ERROR = !ERROR: interpret: unknown variable 'proc_vid'.
!ERROR: interpret_params: insert(param 1): evaluation error.
)

I'm not sure you want to close this issue while such 'slight' changes to a
query
have such huge effects on compilation time.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-19 10:00

Message:
Test works fine:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.err.00.html

Axel,

could you please check and report whether Jan's fix also works for you,
and if so, close this bug report?

Thanks!

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-18 19:35

Message:
If this fix works out, it will be in the upcoming release. (We had some
show stoppers with yesterday's release candidate.)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 18:34

Message:
===================================================================
2009/11/18 - stmane: pathfinder/tests/BugTracker/Tests/All,1.146.2.11

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.bat,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod1.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod2.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q1.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q2.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.sh,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.err,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.out,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.timeout,1.1.2.1

added test (compilation via pf, only) for
ID: 2898944 "pf takes 'forever' (> 1 hour) to compile particular query"
https://sourceforge.net/tracker/?func=detail&aid=2898944&group_id=56967&atid=482468

Seems to work fine, now;
compiling both queries with both `pf -A` & `pf -M`
takes less than 2 secs on my machine with a debug build.
Set timeout to 6 secs.
===================================================================

report can be closed, once nightly testing and Axel confirm the fix.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 16:51

Message:
Plan generation with permutations gets you into trouble.

Planning the distinct operator requires a full ordering of the table as we
use the order extend in MIL to remove duplicates. To calculate the plans
for all possible orders we generate all permutations. (In the first
long-compilation example distinct was applied to a table with 7 columns and
for the second example distinct operated on 8 columns.)

Now we first eliminate all functionally dependent and constant columns as
they do not add any order information. If we still have too many
permutations we ignore all orderings (plans) after a magic boundary.

With this changes in place the compile times are (almost) acceptable:

pf -T query-long-compilation.xq
...
overall compilation time: 974ms

pf -T query-long-compilation2.xq
...
overall compilation time: 02s 781ms

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:53

Message:
Just a few general notes:

1)
A user program can at most trigger a machine crash, but never cause it.
If a user program triggers a machine crash (e.g., by using resources
(too?) many resources, it's in the end a problem (bug) in the OS (or the
hardware) that cannot properly handle (or prevent) such (over-)load
properly.

2)
If you kill (via keyboard interrupt ^C or by sending, say, SIGTERM or
SIGKILL) a process that uses lots of memory (and possibly virtual memory,
i.e., swap), it might indeed take some time until that process is gone and
your prompt is back, since the systems needs to clean up that memory (and
swap) and possibly swap your shell back into main memory ...

3)
Yes, a process that is using much memory (i.e., close to or more than the
machine has physical memory) might (usually will) affect the interactive
responsiveness of the machine.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 12:36

Message:
the original long-compilation.xq now finishes in about 10 secs, which is
still longer than I would like,
but much shorter than it used to.

regarding time measurements, I may have caused some confusion, in the
sense that:
- the original long-compilation.xq definitely took 'too long', but likely
I always interruped it
long before 1 hour passed
- the query that prompted me change the bug summmary line to include '> 1
hour' was a
different one, I'm sure that was long-compilation2.xq

long-compilation2.xq might have taken down a machine before the fix
(machine did crash while that pf was running, but we do not know whether
that was the cause)
after applying the fix and rerunning the pf on long-compilation2.xq I do
notice that
at some point the machine stops being responsive,
only to become responsive again after the segfault -
maybe the segfault saved us from a second machine crash :-)

I wonder whether the memory consumption is what so badly affects
responsivity of the machine.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:03

Message:
Axel,

what about your original long-compilation.xq?
Does that now work fine for you (i.e., with Jan's fix)?

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 11:30

Message:
I don't want to spoil the fun, but after the fix pf still runs pretty long
on the file I just attached
(long-compilation2.xq)

top gives me:
PID USER NI VIRT RES SHR P S %CPU %MEM TIME COMMAND
26940 dbguest 0 7164m 3.7g 172 4 R 27 96.5 14:07
/local/dbguest/Install/MonetDB-XQuery_Nov2009/bin/pf -T /tmp/talt11

I tried to kill it with ^C, without too much success
(and I already started to fear that the machine was hung up on it),
until a little later I saw (I started it as: time /path/to/pf
xqueryfile)

Killed
828.295u 39.446s 16:38.94 86.8% 0+0k 0+0io 5pf+0w
Segmentation fault

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 11:03

Message:
I think, adding a test (also) for this one should be done; it should be not
only possible, but also easy:
the query that (used to) compile for for than one hour is given, just make
a test that calls pf to compile the query, sending the output to, say,
/dev/null/ .
If that finishes without error and within the default 1 minute timeout (or
less, if desired/required; add a TSTNME.timeout file with a <1 scale
factor), the test succeeded and everything is fine; otherwise, the test
fails ...

... just an idea to help assessing and monitoring this issue and hence the
stability and usability of MonetDB as such ...

... does not apply to this bug report, only; see also my email of last
weekend ...

... if there is no time right no to add the test, keep the bug report open
to remind us ...

... in fact, I do exactly that ...

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 10:20

Message:
The reason for the 'forever' compile time is that the physical plan space
blows up in a heuristic that tries to find good plan orders. I now added a
check that avoids generating to many plans.

I checked in the fix in the Nov2009 stable branch. I'm however not sure
whether it makes it into the release -- probably not. So please get it from
either CVS or nightly-builds.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-17 09:32

Message:
not sure about category or group - chose seemingly most appropriate.

even though I did not change default priority , having some answer for
this
(whether fix in code, or guideline for fast compiling query) would be very
helpful.
In my application the pf compilation run is triggered from a web-gui -
fast compilation
is very important there, and predicatable compilation times maybe even
more.
(i.e. it would be nice when all compilations would take the same order of
time).

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-19T09:22:46Z

Bugs item #2898944, was opened at 2009-11-17 09:27
Message generated for change (Comment added) made by axelbel
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "candidate"
Status: Open
Resolution: Fixed
Priority: 6
Private: No
Submitted By: Axel Belinfante (axelbel)
Assigned to: Jan Rittinger (tsheyar)
Summary: pf takes 'forever' (> 1 hour) to compile particular query

Initial Comment:
I don't know whether the following really qualifies as bug, or just is a case of 'user error by giving stupid input.
In any case, I would like to bring this to your attention.

When I compile a query for attached module definition with pf (may or aug or pre-nov version), compilation takes 'forever'.
When I replace the clause ecv:to-human($g0, $human_type) by simple $g0
and remove definitions of $human_type and to-human it compiles almost instantly.
It is the huge difference in compilation time that surprises me.

The query I compile is:

import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation.xq"; ecv:view-from-save("eprints-export-conv-fmt.xml")

where tmp/long-compilation.xq is the attached module.

What the query attempts to do: obtain results from the database, order these on two criteria, and when listing the results, insert human-readable headers.
The human-readable headers are obtained by the to-human mapping from field value (used as ordering criterium) to human-readable string.
At some point I just used if-then-else-if-then-else etc. to do the mapping, but that seemed slower to compile than the approach I took here.

Suggestions for rewriting the query to a fast-compiling one would be considered as a fix too, I guess.

----------------------------------------------------------------------

>Comment By: Axel Belinfante (axelbel)
Date: 2009-11-19 18:22

Message:
time pf -oOIKCG_VG_IS_GYECSVR_DC_GP /tmp/long-compilation3.req >
/tmp/long-compilation3.mil
15.868u 0.472s 0:16.33 100.0% 0+0k 0+0io 0pf+0w

time mclient -lmil -p 51012 /tmp/long-compilation3.mil > /dev/null
0.016u 0.032s 0:03.80 1.0% 0+0k 0+0io 0pf+0w

compilation time is indeed much, much better (though still a bit long)
run time of resulting .mil is kind of ok (even faster would be even
better, of course :-)

anything else I could try?
otherwise, I guess I will just use the options you suggested when
compiling those generated queries,
and that should do for now.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-19 18:04

Message:
Axel,

you hit different problems here. Your original bug caused the planner to
take generate millions of plans.

Your latest example however generates a plan that features 5420 operators
out of which 660 are joins. Here the join optimization takes a little
bit...

What happens if you don't optimize much? -- Try e.g., the following (if
you leave out the 'E' letter compilation time will probably halve):

echo 'import module namespace ecv =
"http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation3.xq";
ecv:view-from-save("eprints-export-conv-fmt.xml")' | pf
-oOIKCG_VG_IS_GYECSVR_DC_GP > foo; Mserver foo

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-19 14:13

Message:
I have tried the latest fix.
there is a good thing, and there is a bad thing.

the good thing is that for these queries the fix works.
that is really, really great.

the bad thing is that it is easy to add complexity (add more of the same
thing),
and again have a relatively slow compilation - although, due to the fix
now compilation does finish.

fact is, those queries are not hand-written, but generated from more
high-level user input.
it is easy for the user to make a slight change to the intended
presentation of the query results
(add more refined structuring/grouping of the results), which can result
in query for which
again compilation time is long.

to give an example:
in long-compilation2.xq I have 3 levels of structure. it compiles in about
3 secs.
file long-compilation3.xq (attached) has 6 levels of structure. it
compiles in approx. 37 minutes.
running the compiled query takes iat most a couple of seconds

I tried rewriting the queries.
I could only obtain reduced compilation time (only 2 minutes) by adding
additional
(pre=-processed) data to the database.
even then, running the compiled query takes approximately a minute...

(I also tried compiling with -M -- this is fast, but I was unable to run
it
MAPI = dbguest@localhost:51012
QUERY = proc_vid.insert("fnC8CAED69_print_4_eprint_xs_element1",
1059825001LL);
...
ERROR = !ERROR: interpret: unknown variable 'proc_vid'.
!ERROR: interpret_params: insert(param 1): evaluation error.
)

I'm not sure you want to close this issue while such 'slight' changes to a
query
have such huge effects on compilation time.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-19 10:00

Message:
Test works fine:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.err.00.html

Axel,

could you please check and report whether Jan's fix also works for you,
and if so, close this bug report?

Thanks!

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-18 19:35

Message:
If this fix works out, it will be in the upcoming release. (We had some
show stoppers with yesterday's release candidate.)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 18:34

Message:
===================================================================
2009/11/18 - stmane: pathfinder/tests/BugTracker/Tests/All,1.146.2.11

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.bat,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod1.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod2.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q1.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q2.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.sh,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.err,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.out,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.timeout,1.1.2.1

added test (compilation via pf, only) for
ID: 2898944 "pf takes 'forever' (> 1 hour) to compile particular query"
https://sourceforge.net/tracker/?func=detail&aid=2898944&group_id=56967&atid=482468

Seems to work fine, now;
compiling both queries with both `pf -A` & `pf -M`
takes less than 2 secs on my machine with a debug build.
Set timeout to 6 secs.
===================================================================

report can be closed, once nightly testing and Axel confirm the fix.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 16:51

Message:
Plan generation with permutations gets you into trouble.

Planning the distinct operator requires a full ordering of the table as we
use the order extend in MIL to remove duplicates. To calculate the plans
for all possible orders we generate all permutations. (In the first
long-compilation example distinct was applied to a table with 7 columns and
for the second example distinct operated on 8 columns.)

Now we first eliminate all functionally dependent and constant columns as
they do not add any order information. If we still have too many
permutations we ignore all orderings (plans) after a magic boundary.

With this changes in place the compile times are (almost) acceptable:

pf -T query-long-compilation.xq
...
overall compilation time: 974ms

pf -T query-long-compilation2.xq
...
overall compilation time: 02s 781ms

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:53

Message:
Just a few general notes:

1)
A user program can at most trigger a machine crash, but never cause it.
If a user program triggers a machine crash (e.g., by using resources
(too?) many resources, it's in the end a problem (bug) in the OS (or the
hardware) that cannot properly handle (or prevent) such (over-)load
properly.

2)
If you kill (via keyboard interrupt ^C or by sending, say, SIGTERM or
SIGKILL) a process that uses lots of memory (and possibly virtual memory,
i.e., swap), it might indeed take some time until that process is gone and
your prompt is back, since the systems needs to clean up that memory (and
swap) and possibly swap your shell back into main memory ...

3)
Yes, a process that is using much memory (i.e., close to or more than the
machine has physical memory) might (usually will) affect the interactive
responsiveness of the machine.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 12:36

Message:
the original long-compilation.xq now finishes in about 10 secs, which is
still longer than I would like,
but much shorter than it used to.

regarding time measurements, I may have caused some confusion, in the
sense that:
- the original long-compilation.xq definitely took 'too long', but likely
I always interruped it
long before 1 hour passed
- the query that prompted me change the bug summmary line to include '> 1
hour' was a
different one, I'm sure that was long-compilation2.xq

long-compilation2.xq might have taken down a machine before the fix
(machine did crash while that pf was running, but we do not know whether
that was the cause)
after applying the fix and rerunning the pf on long-compilation2.xq I do
notice that
at some point the machine stops being responsive,
only to become responsive again after the segfault -
maybe the segfault saved us from a second machine crash :-)

I wonder whether the memory consumption is what so badly affects
responsivity of the machine.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:03

Message:
Axel,

what about your original long-compilation.xq?
Does that now work fine for you (i.e., with Jan's fix)?

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 11:30

Message:
I don't want to spoil the fun, but after the fix pf still runs pretty long
on the file I just attached
(long-compilation2.xq)

top gives me:
PID USER NI VIRT RES SHR P S %CPU %MEM TIME COMMAND
26940 dbguest 0 7164m 3.7g 172 4 R 27 96.5 14:07
/local/dbguest/Install/MonetDB-XQuery_Nov2009/bin/pf -T /tmp/talt11

I tried to kill it with ^C, without too much success
(and I already started to fear that the machine was hung up on it),
until a little later I saw (I started it as: time /path/to/pf
xqueryfile)

Killed
828.295u 39.446s 16:38.94 86.8% 0+0k 0+0io 5pf+0w
Segmentation fault

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 11:03

Message:
I think, adding a test (also) for this one should be done; it should be not
only possible, but also easy:
the query that (used to) compile for for than one hour is given, just make
a test that calls pf to compile the query, sending the output to, say,
/dev/null/ .
If that finishes without error and within the default 1 minute timeout (or
less, if desired/required; add a TSTNME.timeout file with a <1 scale
factor), the test succeeded and everything is fine; otherwise, the test
fails ...

... just an idea to help assessing and monitoring this issue and hence the
stability and usability of MonetDB as such ...

... does not apply to this bug report, only; see also my email of last
weekend ...

... if there is no time right no to add the test, keep the bug report open
to remind us ...

... in fact, I do exactly that ...

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 10:20

Message:
The reason for the 'forever' compile time is that the physical plan space
blows up in a heuristic that tries to find good plan orders. I now added a
check that avoids generating to many plans.

I checked in the fix in the Nov2009 stable branch. I'm however not sure
whether it makes it into the release -- probably not. So please get it from
either CVS or nightly-builds.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-17 09:32

Message:
not sure about category or group - chose seemingly most appropriate.

even though I did not change default priority , having some answer for
this
(whether fix in code, or guideline for fast compiling query) would be very
helpful.
In my application the pf compilation run is triggered from a web-gui -
fast compilation
is very important there, and predicatable compilation times maybe even
more.
(i.e. it would be nice when all compilations would take the same order of
time).

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-19T09:04:38Z

Bugs item #2898944, was opened at 2009-11-17 09:27
Message generated for change (Comment added) made by tsheyar
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "candidate"
Status: Open
Resolution: Fixed
Priority: 6
Private: No
Submitted By: Axel Belinfante (axelbel)
Assigned to: Jan Rittinger (tsheyar)
Summary: pf takes 'forever' (> 1 hour) to compile particular query

Initial Comment:
I don't know whether the following really qualifies as bug, or just is a case of 'user error by giving stupid input.
In any case, I would like to bring this to your attention.

When I compile a query for attached module definition with pf (may or aug or pre-nov version), compilation takes 'forever'.
When I replace the clause ecv:to-human($g0, $human_type) by simple $g0
and remove definitions of $human_type and to-human it compiles almost instantly.
It is the huge difference in compilation time that surprises me.

The query I compile is:

import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation.xq"; ecv:view-from-save("eprints-export-conv-fmt.xml")

where tmp/long-compilation.xq is the attached module.

What the query attempts to do: obtain results from the database, order these on two criteria, and when listing the results, insert human-readable headers.
The human-readable headers are obtained by the to-human mapping from field value (used as ordering criterium) to human-readable string.
At some point I just used if-then-else-if-then-else etc. to do the mapping, but that seemed slower to compile than the approach I took here.

Suggestions for rewriting the query to a fast-compiling one would be considered as a fix too, I guess.

----------------------------------------------------------------------

>Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-19 18:04

Message:
Axel,

you hit different problems here. Your original bug caused the planner to
take generate millions of plans.

Your latest example however generates a plan that features 5420 operators
out of which 660 are joins. Here the join optimization takes a little
bit...

What happens if you don't optimize much? -- Try e.g., the following (if
you leave out the 'E' letter compilation time will probably halve):

echo 'import module namespace ecv =
"http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation3.xq";
ecv:view-from-save("eprints-export-conv-fmt.xml")' | pf
-oOIKCG_VG_IS_GYECSVR_DC_GP > foo; Mserver foo

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-19 14:13

Message:
I have tried the latest fix.
there is a good thing, and there is a bad thing.

the good thing is that for these queries the fix works.
that is really, really great.

the bad thing is that it is easy to add complexity (add more of the same
thing),
and again have a relatively slow compilation - although, due to the fix
now compilation does finish.

fact is, those queries are not hand-written, but generated from more
high-level user input.
it is easy for the user to make a slight change to the intended
presentation of the query results
(add more refined structuring/grouping of the results), which can result
in query for which
again compilation time is long.

to give an example:
in long-compilation2.xq I have 3 levels of structure. it compiles in about
3 secs.
file long-compilation3.xq (attached) has 6 levels of structure. it
compiles in approx. 37 minutes.
running the compiled query takes iat most a couple of seconds

I tried rewriting the queries.
I could only obtain reduced compilation time (only 2 minutes) by adding
additional
(pre=-processed) data to the database.
even then, running the compiled query takes approximately a minute...

(I also tried compiling with -M -- this is fast, but I was unable to run
it
MAPI = dbguest@localhost:51012
QUERY = proc_vid.insert("fnC8CAED69_print_4_eprint_xs_element1",
1059825001LL);
...
ERROR = !ERROR: interpret: unknown variable 'proc_vid'.
!ERROR: interpret_params: insert(param 1): evaluation error.
)

I'm not sure you want to close this issue while such 'slight' changes to a
query
have such huge effects on compilation time.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-19 10:00

Message:
Test works fine:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.err.00.html

Axel,

could you please check and report whether Jan's fix also works for you,
and if so, close this bug report?

Thanks!

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-18 19:35

Message:
If this fix works out, it will be in the upcoming release. (We had some
show stoppers with yesterday's release candidate.)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 18:34

Message:
===================================================================
2009/11/18 - stmane: pathfinder/tests/BugTracker/Tests/All,1.146.2.11

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.bat,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod1.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod2.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q1.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q2.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.sh,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.err,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.out,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.timeout,1.1.2.1

added test (compilation via pf, only) for
ID: 2898944 "pf takes 'forever' (> 1 hour) to compile particular query"
https://sourceforge.net/tracker/?func=detail&aid=2898944&group_id=56967&atid=482468

Seems to work fine, now;
compiling both queries with both `pf -A` & `pf -M`
takes less than 2 secs on my machine with a debug build.
Set timeout to 6 secs.
===================================================================

report can be closed, once nightly testing and Axel confirm the fix.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 16:51

Message:
Plan generation with permutations gets you into trouble.

Planning the distinct operator requires a full ordering of the table as we
use the order extend in MIL to remove duplicates. To calculate the plans
for all possible orders we generate all permutations. (In the first
long-compilation example distinct was applied to a table with 7 columns and
for the second example distinct operated on 8 columns.)

Now we first eliminate all functionally dependent and constant columns as
they do not add any order information. If we still have too many
permutations we ignore all orderings (plans) after a magic boundary.

With this changes in place the compile times are (almost) acceptable:

pf -T query-long-compilation.xq
...
overall compilation time: 974ms

pf -T query-long-compilation2.xq
...
overall compilation time: 02s 781ms

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:53

Message:
Just a few general notes:

1)
A user program can at most trigger a machine crash, but never cause it.
If a user program triggers a machine crash (e.g., by using resources
(too?) many resources, it's in the end a problem (bug) in the OS (or the
hardware) that cannot properly handle (or prevent) such (over-)load
properly.

2)
If you kill (via keyboard interrupt ^C or by sending, say, SIGTERM or
SIGKILL) a process that uses lots of memory (and possibly virtual memory,
i.e., swap), it might indeed take some time until that process is gone and
your prompt is back, since the systems needs to clean up that memory (and
swap) and possibly swap your shell back into main memory ...

3)
Yes, a process that is using much memory (i.e., close to or more than the
machine has physical memory) might (usually will) affect the interactive
responsiveness of the machine.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 12:36

Message:
the original long-compilation.xq now finishes in about 10 secs, which is
still longer than I would like,
but much shorter than it used to.

regarding time measurements, I may have caused some confusion, in the
sense that:
- the original long-compilation.xq definitely took 'too long', but likely
I always interruped it
long before 1 hour passed
- the query that prompted me change the bug summmary line to include '> 1
hour' was a
different one, I'm sure that was long-compilation2.xq

long-compilation2.xq might have taken down a machine before the fix
(machine did crash while that pf was running, but we do not know whether
that was the cause)
after applying the fix and rerunning the pf on long-compilation2.xq I do
notice that
at some point the machine stops being responsive,
only to become responsive again after the segfault -
maybe the segfault saved us from a second machine crash :-)

I wonder whether the memory consumption is what so badly affects
responsivity of the machine.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:03

Message:
Axel,

what about your original long-compilation.xq?
Does that now work fine for you (i.e., with Jan's fix)?

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 11:30

Message:
I don't want to spoil the fun, but after the fix pf still runs pretty long
on the file I just attached
(long-compilation2.xq)

top gives me:
PID USER NI VIRT RES SHR P S %CPU %MEM TIME COMMAND
26940 dbguest 0 7164m 3.7g 172 4 R 27 96.5 14:07
/local/dbguest/Install/MonetDB-XQuery_Nov2009/bin/pf -T /tmp/talt11

I tried to kill it with ^C, without too much success
(and I already started to fear that the machine was hung up on it),
until a little later I saw (I started it as: time /path/to/pf
xqueryfile)

Killed
828.295u 39.446s 16:38.94 86.8% 0+0k 0+0io 5pf+0w
Segmentation fault

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 11:03

Message:
I think, adding a test (also) for this one should be done; it should be not
only possible, but also easy:
the query that (used to) compile for for than one hour is given, just make
a test that calls pf to compile the query, sending the output to, say,
/dev/null/ .
If that finishes without error and within the default 1 minute timeout (or
less, if desired/required; add a TSTNME.timeout file with a <1 scale
factor), the test succeeded and everything is fine; otherwise, the test
fails ...

... just an idea to help assessing and monitoring this issue and hence the
stability and usability of MonetDB as such ...

... does not apply to this bug report, only; see also my email of last
weekend ...

... if there is no time right no to add the test, keep the bug report open
to remind us ...

... in fact, I do exactly that ...

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 10:20

Message:
The reason for the 'forever' compile time is that the physical plan space
blows up in a heuristic that tries to find good plan orders. I now added a
check that avoids generating to many plans.

I checked in the fix in the Nov2009 stable branch. I'm however not sure
whether it makes it into the release -- probably not. So please get it from
either CVS or nightly-builds.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-17 09:32

Message:
not sure about category or group - chose seemingly most appropriate.

even though I did not change default priority , having some answer for
this
(whether fix in code, or guideline for fast compiling query) would be very
helpful.
In my application the pf compilation run is triggered from a web-gui -
fast compilation
is very important there, and predicatable compilation times maybe even
more.
(i.e. it would be nice when all compilations would take the same order of
time).

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2900365 ] SQL: concurrent tracelog()

2009-11-19T07:42:43Z

Bugs item #2900365, was opened at 2009-11-19 11:46
Message generated for change (Comment added) made by vzzzbx
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900365&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SQL/Core
Group: SQL "stable"
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Martin Kersten (mlkersten)
Summary: SQL: concurrent tracelog()

Initial Comment:
When running 2 mclient instances simultaneously, the 1st mclient can run tracelog(), the second cannot (the error message is "SELECT: '(null)' does not return a table").

$ mclient -lsql -p51234 -daap
Welcome to mclient, the MonetDB/SQL interactive terminal
Database: MonetDB v5.16.0, 'aap'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>SELECT * FROM tracelog();
0 tuples
sql>

On another prompt:

$ mclient -lsql -p51234 -daap
Welcome to mclient, the MonetDB/SQL interactive terminal
Database: MonetDB v5.16.0, 'aap'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>SELECT * FROM tracelog();
SELECT: '(null)' does not return a table
sql>

----------------------------------------------------------------------

>Comment By: Wouter Alink (vzzzbx)
Date: 2009-11-19 16:42

Message:
when running trace using JDBC, multiple connections to the same database do
seem to be able to query tracelog(). (if no mclient is running)
however, they are not isolated from each other; work done in one
transaction, can be found in a call to tracelog() in another transaction.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900365&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-19T05:13:17Z

Bugs item #2898944, was opened at 2009-11-17 09:27
Message generated for change (Comment added) made by axelbel
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "candidate"
Status: Open
Resolution: Fixed
Priority: 6
Private: No
Submitted By: Axel Belinfante (axelbel)
Assigned to: Jan Rittinger (tsheyar)
Summary: pf takes 'forever' (> 1 hour) to compile particular query

Initial Comment:
I don't know whether the following really qualifies as bug, or just is a case of 'user error by giving stupid input.
In any case, I would like to bring this to your attention.

When I compile a query for attached module definition with pf (may or aug or pre-nov version), compilation takes 'forever'.
When I replace the clause ecv:to-human($g0, $human_type) by simple $g0
and remove definitions of $human_type and to-human it compiles almost instantly.
It is the huge difference in compilation time that surprises me.

The query I compile is:

import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation.xq"; ecv:view-from-save("eprints-export-conv-fmt.xml")

where tmp/long-compilation.xq is the attached module.

What the query attempts to do: obtain results from the database, order these on two criteria, and when listing the results, insert human-readable headers.
The human-readable headers are obtained by the to-human mapping from field value (used as ordering criterium) to human-readable string.
At some point I just used if-then-else-if-then-else etc. to do the mapping, but that seemed slower to compile than the approach I took here.

Suggestions for rewriting the query to a fast-compiling one would be considered as a fix too, I guess.

----------------------------------------------------------------------

>Comment By: Axel Belinfante (axelbel)
Date: 2009-11-19 14:13

Message:
I have tried the latest fix.
there is a good thing, and there is a bad thing.

the good thing is that for these queries the fix works.
that is really, really great.

the bad thing is that it is easy to add complexity (add more of the same
thing),
and again have a relatively slow compilation - although, due to the fix
now compilation does finish.

fact is, those queries are not hand-written, but generated from more
high-level user input.
it is easy for the user to make a slight change to the intended
presentation of the query results
(add more refined structuring/grouping of the results), which can result
in query for which
again compilation time is long.

to give an example:
in long-compilation2.xq I have 3 levels of structure. it compiles in about
3 secs.
file long-compilation3.xq (attached) has 6 levels of structure. it
compiles in approx. 37 minutes.
running the compiled query takes iat most a couple of seconds

I tried rewriting the queries.
I could only obtain reduced compilation time (only 2 minutes) by adding
additional
(pre=-processed) data to the database.
even then, running the compiled query takes approximately a minute...

(I also tried compiling with -M -- this is fast, but I was unable to run
it
MAPI = dbguest@localhost:51012
QUERY = proc_vid.insert("fnC8CAED69_print_4_eprint_xs_element1",
1059825001LL);
...
ERROR = !ERROR: interpret: unknown variable 'proc_vid'.
!ERROR: interpret_params: insert(param 1): evaluation error.
)

I'm not sure you want to close this issue while such 'slight' changes to a
query
have such huge effects on compilation time.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-19 10:00

Message:
Test works fine:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.err.00.html

Axel,

could you please check and report whether Jan's fix also works for you,
and if so, close this bug report?

Thanks!

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-18 19:35

Message:
If this fix works out, it will be in the upcoming release. (We had some
show stoppers with yesterday's release candidate.)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 18:34

Message:
===================================================================
2009/11/18 - stmane: pathfinder/tests/BugTracker/Tests/All,1.146.2.11

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.bat,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod1.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod2.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q1.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q2.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.sh,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.err,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.out,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.timeout,1.1.2.1

added test (compilation via pf, only) for
ID: 2898944 "pf takes 'forever' (> 1 hour) to compile particular query"
https://sourceforge.net/tracker/?func=detail&aid=2898944&group_id=56967&atid=482468

Seems to work fine, now;
compiling both queries with both `pf -A` & `pf -M`
takes less than 2 secs on my machine with a debug build.
Set timeout to 6 secs.
===================================================================

report can be closed, once nightly testing and Axel confirm the fix.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 16:51

Message:
Plan generation with permutations gets you into trouble.

Planning the distinct operator requires a full ordering of the table as we
use the order extend in MIL to remove duplicates. To calculate the plans
for all possible orders we generate all permutations. (In the first
long-compilation example distinct was applied to a table with 7 columns and
for the second example distinct operated on 8 columns.)

Now we first eliminate all functionally dependent and constant columns as
they do not add any order information. If we still have too many
permutations we ignore all orderings (plans) after a magic boundary.

With this changes in place the compile times are (almost) acceptable:

pf -T query-long-compilation.xq
...
overall compilation time: 974ms

pf -T query-long-compilation2.xq
...
overall compilation time: 02s 781ms

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:53

Message:
Just a few general notes:

1)
A user program can at most trigger a machine crash, but never cause it.
If a user program triggers a machine crash (e.g., by using resources
(too?) many resources, it's in the end a problem (bug) in the OS (or the
hardware) that cannot properly handle (or prevent) such (over-)load
properly.

2)
If you kill (via keyboard interrupt ^C or by sending, say, SIGTERM or
SIGKILL) a process that uses lots of memory (and possibly virtual memory,
i.e., swap), it might indeed take some time until that process is gone and
your prompt is back, since the systems needs to clean up that memory (and
swap) and possibly swap your shell back into main memory ...

3)
Yes, a process that is using much memory (i.e., close to or more than the
machine has physical memory) might (usually will) affect the interactive
responsiveness of the machine.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 12:36

Message:
the original long-compilation.xq now finishes in about 10 secs, which is
still longer than I would like,
but much shorter than it used to.

regarding time measurements, I may have caused some confusion, in the
sense that:
- the original long-compilation.xq definitely took 'too long', but likely
I always interruped it
long before 1 hour passed
- the query that prompted me change the bug summmary line to include '> 1
hour' was a
different one, I'm sure that was long-compilation2.xq

long-compilation2.xq might have taken down a machine before the fix
(machine did crash while that pf was running, but we do not know whether
that was the cause)
after applying the fix and rerunning the pf on long-compilation2.xq I do
notice that
at some point the machine stops being responsive,
only to become responsive again after the segfault -
maybe the segfault saved us from a second machine crash :-)

I wonder whether the memory consumption is what so badly affects
responsivity of the machine.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:03

Message:
Axel,

what about your original long-compilation.xq?
Does that now work fine for you (i.e., with Jan's fix)?

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 11:30

Message:
I don't want to spoil the fun, but after the fix pf still runs pretty long
on the file I just attached
(long-compilation2.xq)

top gives me:
PID USER NI VIRT RES SHR P S %CPU %MEM TIME COMMAND
26940 dbguest 0 7164m 3.7g 172 4 R 27 96.5 14:07
/local/dbguest/Install/MonetDB-XQuery_Nov2009/bin/pf -T /tmp/talt11

I tried to kill it with ^C, without too much success
(and I already started to fear that the machine was hung up on it),
until a little later I saw (I started it as: time /path/to/pf
xqueryfile)

Killed
828.295u 39.446s 16:38.94 86.8% 0+0k 0+0io 5pf+0w
Segmentation fault

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 11:03

Message:
I think, adding a test (also) for this one should be done; it should be not
only possible, but also easy:
the query that (used to) compile for for than one hour is given, just make
a test that calls pf to compile the query, sending the output to, say,
/dev/null/ .
If that finishes without error and within the default 1 minute timeout (or
less, if desired/required; add a TSTNME.timeout file with a <1 scale
factor), the test succeeded and everything is fine; otherwise, the test
fails ...

... just an idea to help assessing and monitoring this issue and hence the
stability and usability of MonetDB as such ...

... does not apply to this bug report, only; see also my email of last
weekend ...

... if there is no time right no to add the test, keep the bug report open
to remind us ...

... in fact, I do exactly that ...

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 10:20

Message:
The reason for the 'forever' compile time is that the physical plan space
blows up in a heuristic that tries to find good plan orders. I now added a
check that avoids generating to many plans.

I checked in the fix in the Nov2009 stable branch. I'm however not sure
whether it makes it into the release -- probably not. So please get it from
either CVS or nightly-builds.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-17 09:32

Message:
not sure about category or group - chose seemingly most appropriate.

even though I did not change default priority , having some answer for
this
(whether fix in code, or guideline for fast compiling query) would be very
helpful.
In my application the pf compilation run is triggered from a web-gui -
fast compilation
is very important there, and predicatable compilation times maybe even
more.
(i.e. it would be nice when all compilations would take the same order of
time).

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2900358 ] SQL: tracelog() is undefined

2009-11-19T03:36:50Z

Bugs item #2900358, was opened at 2009-11-19 11:29
Message generated for change (Comment added) made by mlkersten
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900358&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SQL/Core
Group: SQL "stable"
>Status: Closed
>Resolution: Wont Fix
Priority: 5
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Martin Kersten (mlkersten)
Summary: SQL: tracelog() is undefined

Initial Comment:
The function tracelog() is not defined by default. If the sql query 'create function ... sql.dump_trace;'
(provided in http://www.monetdb.nl/projects/monetdb/SQL/Documentation/TRACE-Statement.html) is first executed then it works fine.
(I added a test SQL:src/test/Tests/trace)

$ monetdb create aap
created database in maintenance mode: aap
$ monetdb start aap
starting database 'aap'... done
$ mclient -lsql -p51234 -daap -umonetdb -Pmonetdb
Welcome to mclient, the MonetDB/SQL interactive terminal
Database: MonetDB v5.16.0, 'aap'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>SELECT * FROM tracelog();
SELECT: no such operator 'tracelog'

$ mserver5 --version
MonetDB server v5.16.0 (64-bit), based on kernel v1.34.0 (64-bit oids)
Copyright (c) 1993-July 2008 CWI
Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved
Visit http://monetdb.cwi.nl/ for further information
Configured for prefix: /ufs/alink/opt/MonetDB-Nov2009
Libraries:
libpcre: 7.8 2008-09-05 (compiled with 7.8)
openssl: OpenSSL 0.9.8g 19 Oct 2007 (compiled with OpenSSL 0.9.8g 19 Oct 2007)
libxml2: 2.7.6 (compiled with 2.7.6)
Compiled by: alink@... (x86_64-unknown-linux-gnu)
Compilation: gcc -Wall -Wextra -g -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef Wp,-D_FORTIFY_SOURCE=2
Linking : ld -IPA -m elf_x86_64

----------------------------------------------------------------------

>Comment By: Martin Kersten (mlkersten)
Date: 2009-11-19 12:36

Message:
Correct. There is no hardcoded definition for this function in the SQL
catalog.
It is part of a collection of sql scripts in .../src/sql/*.sql that have
to be loaded once for a database.
That's why the definition was also repeated in the documentation

----------------------------------------------------------------------

Comment By: Wouter Alink (vzzzbx)
Date: 2009-11-19 11:31

Message:
i think this is martin's area ;) re-assiging to martin.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900358&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2900365 ] SQL: concurrent tracelog()

2009-11-19T02:47:22Z

Bugs item #2900365, was opened at 2009-11-19 11:46
Message generated for change (Settings changed) made by vzzzbx
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900365&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SQL/Core
Group: SQL "stable"
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Wouter Alink (vzzzbx)
>Assigned to: Martin Kersten (mlkersten)
Summary: SQL: concurrent tracelog()

Initial Comment:
When running 2 mclient instances simultaneously, the 1st mclient can run tracelog(), the second cannot (the error message is "SELECT: '(null)' does not return a table").

$ mclient -lsql -p51234 -daap
Welcome to mclient, the MonetDB/SQL interactive terminal
Database: MonetDB v5.16.0, 'aap'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>SELECT * FROM tracelog();
0 tuples
sql>

On another prompt:

$ mclient -lsql -p51234 -daap
Welcome to mclient, the MonetDB/SQL interactive terminal
Database: MonetDB v5.16.0, 'aap'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>SELECT * FROM tracelog();
SELECT: '(null)' does not return a table
sql>

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900365&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2900365 ] SQL: concurrent tracelog()

2009-11-19T02:46:50Z

Bugs item #2900365, was opened at 2009-11-19 11:46
Message generated for change (Tracker Item Submitted) made by vzzzbx
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900365&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SQL/Core
Group: SQL "stable"
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Niels Nes (nielsnes)
Summary: SQL: concurrent tracelog()

Initial Comment:
When running 2 mclient instances simultaneously, the 1st mclient can run tracelog(), the second cannot (the error message is "SELECT: '(null)' does not return a table").

$ mclient -lsql -p51234 -daap
Welcome to mclient, the MonetDB/SQL interactive terminal
Database: MonetDB v5.16.0, 'aap'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>SELECT * FROM tracelog();
0 tuples
sql>

On another prompt:

$ mclient -lsql -p51234 -daap
Welcome to mclient, the MonetDB/SQL interactive terminal
Database: MonetDB v5.16.0, 'aap'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>SELECT * FROM tracelog();
SELECT: '(null)' does not return a table
sql>

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900365&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2900358 ] SQL: tracelog() is undefined

2009-11-19T02:31:41Z

Bugs item #2900358, was opened at 2009-11-19 11:29
Message generated for change (Comment added) made by vzzzbx
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900358&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SQL/Core
Group: SQL "stable"
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Wouter Alink (vzzzbx)
>Assigned to: Martin Kersten (mlkersten)
Summary: SQL: tracelog() is undefined

Initial Comment:
The function tracelog() is not defined by default. If the sql query 'create function ... sql.dump_trace;'
(provided in http://www.monetdb.nl/projects/monetdb/SQL/Documentation/TRACE-Statement.html) is first executed then it works fine.
(I added a test SQL:src/test/Tests/trace)

$ monetdb create aap
created database in maintenance mode: aap
$ monetdb start aap
starting database 'aap'... done
$ mclient -lsql -p51234 -daap -umonetdb -Pmonetdb
Welcome to mclient, the MonetDB/SQL interactive terminal
Database: MonetDB v5.16.0, 'aap'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>SELECT * FROM tracelog();
SELECT: no such operator 'tracelog'

$ mserver5 --version
MonetDB server v5.16.0 (64-bit), based on kernel v1.34.0 (64-bit oids)
Copyright (c) 1993-July 2008 CWI
Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved
Visit http://monetdb.cwi.nl/ for further information
Configured for prefix: /ufs/alink/opt/MonetDB-Nov2009
Libraries:
libpcre: 7.8 2008-09-05 (compiled with 7.8)
openssl: OpenSSL 0.9.8g 19 Oct 2007 (compiled with OpenSSL 0.9.8g 19 Oct 2007)
libxml2: 2.7.6 (compiled with 2.7.6)
Compiled by: alink@... (x86_64-unknown-linux-gnu)
Compilation: gcc -Wall -Wextra -g -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef Wp,-D_FORTIFY_SOURCE=2
Linking : ld -IPA -m elf_x86_64

----------------------------------------------------------------------

>Comment By: Wouter Alink (vzzzbx)
Date: 2009-11-19 11:31

Message:
i think this is martin's area ;) re-assiging to martin.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900358&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2900358 ] SQL: tracelog() is undefined

2009-11-19T02:29:51Z

Bugs item #2900358, was opened at 2009-11-19 11:29
Message generated for change (Tracker Item Submitted) made by vzzzbx
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900358&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SQL/Core
Group: SQL "stable"
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Niels Nes (nielsnes)
Summary: SQL: tracelog() is undefined

Initial Comment:
The function tracelog() is not defined by default. If the sql query 'create function ... sql.dump_trace;'
(provided in http://www.monetdb.nl/projects/monetdb/SQL/Documentation/TRACE-Statement.html) is first executed then it works fine.
(I added a test SQL:src/test/Tests/trace)

$ monetdb create aap
created database in maintenance mode: aap
$ monetdb start aap
starting database 'aap'... done
$ mclient -lsql -p51234 -daap -umonetdb -Pmonetdb
Welcome to mclient, the MonetDB/SQL interactive terminal
Database: MonetDB v5.16.0, 'aap'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>SELECT * FROM tracelog();
SELECT: no such operator 'tracelog'

$ mserver5 --version
MonetDB server v5.16.0 (64-bit), based on kernel v1.34.0 (64-bit oids)
Copyright (c) 1993-July 2008 CWI
Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved
Visit http://monetdb.cwi.nl/ for further information
Configured for prefix: /ufs/alink/opt/MonetDB-Nov2009
Libraries:
libpcre: 7.8 2008-09-05 (compiled with 7.8)
openssl: OpenSSL 0.9.8g 19 Oct 2007 (compiled with OpenSSL 0.9.8g 19 Oct 2007)
libxml2: 2.7.6 (compiled with 2.7.6)
Compiled by: alink@... (x86_64-unknown-linux-gnu)
Compilation: gcc -Wall -Wextra -g -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef Wp,-D_FORTIFY_SOURCE=2
Linking : ld -IPA -m elf_x86_64

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900358&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2900355 ] M5: crash time

2009-11-19T02:24:32Z

Bugs item #2900355, was opened at 2009-11-19 11:24
Message generated for change (Tracker Item Submitted) made by vzzzbx
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900355&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Documentation
Group: MonetDB5 "stable"
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Sjoerd Mullender (sjoerd)
Summary: M5: crash time

Initial Comment:
Very minor issue: The output of the monetdb utility might be a little bit confusing. The 'last crash' time, seems to be the start time of the previous mserver session that crashed. Because the server crashed a few seconds before I started it again, while it seems to show that there are 13 minutes in between the two sessions. Perhaps the description should be changed to "start time of last session that crashed " or something similar.

$ monetdb status -l aap
aap:
location: /export/scratch0/roberto/wouter/dbfarm-MonetDB-Nov2009/aap
database name: aap
state: running
locked: no
scenarios: mal sql msql
connections: mapi:monetdb://127.0.0.1:51236/
start count: 3
stop count: 0
crash count: 2
current uptime: 10s
average uptime: 0s
maximum uptime: 0s
minimum uptime: 0s
last crash: 2009-11-19 10:47:18
last start: 2009-11-19 11:00:13
average of crashes in the last start attempt: 0
average of crashes in the last 10 start attempts: 0.20
average of crashes in the last 30 start attempts: 0.07

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2900355&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-19T01:00:29Z

Bugs item #2898944, was opened at 2009-11-17 09:27
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "candidate"
Status: Open
Resolution: Fixed
Priority: 6
Private: No
Submitted By: Axel Belinfante (axelbel)
Assigned to: Jan Rittinger (tsheyar)
Summary: pf takes 'forever' (> 1 hour) to compile particular query

Initial Comment:
I don't know whether the following really qualifies as bug, or just is a case of 'user error by giving stupid input.
In any case, I would like to bring this to your attention.

When I compile a query for attached module definition with pf (may or aug or pre-nov version), compilation takes 'forever'.
When I replace the clause ecv:to-human($g0, $human_type) by simple $g0
and remove definitions of $human_type and to-human it compiles almost instantly.
It is the huge difference in compilation time that surprises me.

The query I compile is:

import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation.xq"; ecv:view-from-save("eprints-export-conv-fmt.xml")

where tmp/long-compilation.xq is the attached module.

What the query attempts to do: obtain results from the database, order these on two criteria, and when listing the results, insert human-readable headers.
The human-readable headers are obtained by the to-human mapping from field value (used as ordering criterium) to human-readable string.
At some point I just used if-then-else-if-then-else etc. to do the mapping, but that seemed slower to compile than the approach I took here.

Suggestions for rewriting the query to a fast-compiling one would be considered as a fix too, I guess.

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-19 10:00

Message:
Test works fine:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.err.00.html

Axel,

could you please check and report whether Jan's fix also works for you,
and if so, close this bug report?

Thanks!

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-18 19:35

Message:
If this fix works out, it will be in the upcoming release. (We had some
show stoppers with yesterday's release candidate.)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 18:34

Message:
===================================================================
2009/11/18 - stmane: pathfinder/tests/BugTracker/Tests/All,1.146.2.11

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.bat,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod1.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod2.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q1.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q2.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.sh,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.err,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.out,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.timeout,1.1.2.1

added test (compilation via pf, only) for
ID: 2898944 "pf takes 'forever' (> 1 hour) to compile particular query"
https://sourceforge.net/tracker/?func=detail&aid=2898944&group_id=56967&atid=482468

Seems to work fine, now;
compiling both queries with both `pf -A` & `pf -M`
takes less than 2 secs on my machine with a debug build.
Set timeout to 6 secs.
===================================================================

report can be closed, once nightly testing and Axel confirm the fix.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 16:51

Message:
Plan generation with permutations gets you into trouble.

Planning the distinct operator requires a full ordering of the table as we
use the order extend in MIL to remove duplicates. To calculate the plans
for all possible orders we generate all permutations. (In the first
long-compilation example distinct was applied to a table with 7 columns and
for the second example distinct operated on 8 columns.)

Now we first eliminate all functionally dependent and constant columns as
they do not add any order information. If we still have too many
permutations we ignore all orderings (plans) after a magic boundary.

With this changes in place the compile times are (almost) acceptable:

pf -T query-long-compilation.xq
...
overall compilation time: 974ms

pf -T query-long-compilation2.xq
...
overall compilation time: 02s 781ms

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:53

Message:
Just a few general notes:

1)
A user program can at most trigger a machine crash, but never cause it.
If a user program triggers a machine crash (e.g., by using resources
(too?) many resources, it's in the end a problem (bug) in the OS (or the
hardware) that cannot properly handle (or prevent) such (over-)load
properly.

2)
If you kill (via keyboard interrupt ^C or by sending, say, SIGTERM or
SIGKILL) a process that uses lots of memory (and possibly virtual memory,
i.e., swap), it might indeed take some time until that process is gone and
your prompt is back, since the systems needs to clean up that memory (and
swap) and possibly swap your shell back into main memory ...

3)
Yes, a process that is using much memory (i.e., close to or more than the
machine has physical memory) might (usually will) affect the interactive
responsiveness of the machine.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 12:36

Message:
the original long-compilation.xq now finishes in about 10 secs, which is
still longer than I would like,
but much shorter than it used to.

regarding time measurements, I may have caused some confusion, in the
sense that:
- the original long-compilation.xq definitely took 'too long', but likely
I always interruped it
long before 1 hour passed
- the query that prompted me change the bug summmary line to include '> 1
hour' was a
different one, I'm sure that was long-compilation2.xq

long-compilation2.xq might have taken down a machine before the fix
(machine did crash while that pf was running, but we do not know whether
that was the cause)
after applying the fix and rerunning the pf on long-compilation2.xq I do
notice that
at some point the machine stops being responsive,
only to become responsive again after the segfault -
maybe the segfault saved us from a second machine crash :-)

I wonder whether the memory consumption is what so badly affects
responsivity of the machine.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:03

Message:
Axel,

what about your original long-compilation.xq?
Does that now work fine for you (i.e., with Jan's fix)?

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 11:30

Message:
I don't want to spoil the fun, but after the fix pf still runs pretty long
on the file I just attached
(long-compilation2.xq)

top gives me:
PID USER NI VIRT RES SHR P S %CPU %MEM TIME COMMAND
26940 dbguest 0 7164m 3.7g 172 4 R 27 96.5 14:07
/local/dbguest/Install/MonetDB-XQuery_Nov2009/bin/pf -T /tmp/talt11

I tried to kill it with ^C, without too much success
(and I already started to fear that the machine was hung up on it),
until a little later I saw (I started it as: time /path/to/pf
xqueryfile)

Killed
828.295u 39.446s 16:38.94 86.8% 0+0k 0+0io 5pf+0w
Segmentation fault

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 11:03

Message:
I think, adding a test (also) for this one should be done; it should be not
only possible, but also easy:
the query that (used to) compile for for than one hour is given, just make
a test that calls pf to compile the query, sending the output to, say,
/dev/null/ .
If that finishes without error and within the default 1 minute timeout (or
less, if desired/required; add a TSTNME.timeout file with a <1 scale
factor), the test succeeded and everything is fine; otherwise, the test
fails ...

... just an idea to help assessing and monitoring this issue and hence the
stability and usability of MonetDB as such ...

... does not apply to this bug report, only; see also my email of last
weekend ...

... if there is no time right no to add the test, keep the bug report open
to remind us ...

... in fact, I do exactly that ...

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 10:20

Message:
The reason for the 'forever' compile time is that the physical plan space
blows up in a heuristic that tries to find good plan orders. I now added a
check that avoids generating to many plans.

I checked in the fix in the Nov2009 stable branch. I'm however not sure
whether it makes it into the release -- probably not. So please get it from
either CVS or nightly-builds.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-17 09:32

Message:
not sure about category or group - chose seemingly most appropriate.

even though I did not change default priority , having some answer for
this
(whether fix in code, or guideline for fast compiling query) would be very
helpful.
In my application the pf compilation run is triggered from a web-gui -
fast compilation
is very important there, and predicatable compilation times maybe even
more.
(i.e. it would be nice when all compilations would take the same order of
time).

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-18T10:35:06Z

Bugs item #2898944, was opened at 2009-11-17 09:27
Message generated for change (Comment added) made by sjoerd
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "candidate"
Status: Open
Resolution: Fixed
Priority: 6
Private: No
Submitted By: Axel Belinfante (axelbel)
Assigned to: Jan Rittinger (tsheyar)
Summary: pf takes 'forever' (> 1 hour) to compile particular query

Initial Comment:
I don't know whether the following really qualifies as bug, or just is a case of 'user error by giving stupid input.
In any case, I would like to bring this to your attention.

When I compile a query for attached module definition with pf (may or aug or pre-nov version), compilation takes 'forever'.
When I replace the clause ecv:to-human($g0, $human_type) by simple $g0
and remove definitions of $human_type and to-human it compiles almost instantly.
It is the huge difference in compilation time that surprises me.

The query I compile is:

import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation.xq"; ecv:view-from-save("eprints-export-conv-fmt.xml")

where tmp/long-compilation.xq is the attached module.

What the query attempts to do: obtain results from the database, order these on two criteria, and when listing the results, insert human-readable headers.
The human-readable headers are obtained by the to-human mapping from field value (used as ordering criterium) to human-readable string.
At some point I just used if-then-else-if-then-else etc. to do the mapping, but that seemed slower to compile than the approach I took here.

Suggestions for rewriting the query to a fast-compiling one would be considered as a fix too, I guess.

----------------------------------------------------------------------

>Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-18 19:35

Message:
If this fix works out, it will be in the upcoming release. (We had some
show stoppers with yesterday's release candidate.)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 18:34

Message:
===================================================================
2009/11/18 - stmane: pathfinder/tests/BugTracker/Tests/All,1.146.2.11

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.bat,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod1.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod2.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q1.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q2.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.sh,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.err,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.out,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.timeout,1.1.2.1

added test (compilation via pf, only) for
ID: 2898944 "pf takes 'forever' (> 1 hour) to compile particular query"
https://sourceforge.net/tracker/?func=detail&aid=2898944&group_id=56967&atid=482468

Seems to work fine, now;
compiling both queries with both `pf -A` & `pf -M`
takes less than 2 secs on my machine with a debug build.
Set timeout to 6 secs.
===================================================================

report can be closed, once nightly testing and Axel confirm the fix.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 16:51

Message:
Plan generation with permutations gets you into trouble.

Planning the distinct operator requires a full ordering of the table as we
use the order extend in MIL to remove duplicates. To calculate the plans
for all possible orders we generate all permutations. (In the first
long-compilation example distinct was applied to a table with 7 columns and
for the second example distinct operated on 8 columns.)

Now we first eliminate all functionally dependent and constant columns as
they do not add any order information. If we still have too many
permutations we ignore all orderings (plans) after a magic boundary.

With this changes in place the compile times are (almost) acceptable:

pf -T query-long-compilation.xq
...
overall compilation time: 974ms

pf -T query-long-compilation2.xq
...
overall compilation time: 02s 781ms

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:53

Message:
Just a few general notes:

1)
A user program can at most trigger a machine crash, but never cause it.
If a user program triggers a machine crash (e.g., by using resources
(too?) many resources, it's in the end a problem (bug) in the OS (or the
hardware) that cannot properly handle (or prevent) such (over-)load
properly.

2)
If you kill (via keyboard interrupt ^C or by sending, say, SIGTERM or
SIGKILL) a process that uses lots of memory (and possibly virtual memory,
i.e., swap), it might indeed take some time until that process is gone and
your prompt is back, since the systems needs to clean up that memory (and
swap) and possibly swap your shell back into main memory ...

3)
Yes, a process that is using much memory (i.e., close to or more than the
machine has physical memory) might (usually will) affect the interactive
responsiveness of the machine.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 12:36

Message:
the original long-compilation.xq now finishes in about 10 secs, which is
still longer than I would like,
but much shorter than it used to.

regarding time measurements, I may have caused some confusion, in the
sense that:
- the original long-compilation.xq definitely took 'too long', but likely
I always interruped it
long before 1 hour passed
- the query that prompted me change the bug summmary line to include '> 1
hour' was a
different one, I'm sure that was long-compilation2.xq

long-compilation2.xq might have taken down a machine before the fix
(machine did crash while that pf was running, but we do not know whether
that was the cause)
after applying the fix and rerunning the pf on long-compilation2.xq I do
notice that
at some point the machine stops being responsive,
only to become responsive again after the segfault -
maybe the segfault saved us from a second machine crash :-)

I wonder whether the memory consumption is what so badly affects
responsivity of the machine.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:03

Message:
Axel,

what about your original long-compilation.xq?
Does that now work fine for you (i.e., with Jan's fix)?

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 11:30

Message:
I don't want to spoil the fun, but after the fix pf still runs pretty long
on the file I just attached
(long-compilation2.xq)

top gives me:
PID USER NI VIRT RES SHR P S %CPU %MEM TIME COMMAND
26940 dbguest 0 7164m 3.7g 172 4 R 27 96.5 14:07
/local/dbguest/Install/MonetDB-XQuery_Nov2009/bin/pf -T /tmp/talt11

I tried to kill it with ^C, without too much success
(and I already started to fear that the machine was hung up on it),
until a little later I saw (I started it as: time /path/to/pf
xqueryfile)

Killed
828.295u 39.446s 16:38.94 86.8% 0+0k 0+0io 5pf+0w
Segmentation fault

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 11:03

Message:
I think, adding a test (also) for this one should be done; it should be not
only possible, but also easy:
the query that (used to) compile for for than one hour is given, just make
a test that calls pf to compile the query, sending the output to, say,
/dev/null/ .
If that finishes without error and within the default 1 minute timeout (or
less, if desired/required; add a TSTNME.timeout file with a <1 scale
factor), the test succeeded and everything is fine; otherwise, the test
fails ...

... just an idea to help assessing and monitoring this issue and hence the
stability and usability of MonetDB as such ...

... does not apply to this bug report, only; see also my email of last
weekend ...

... if there is no time right no to add the test, keep the bug report open
to remind us ...

... in fact, I do exactly that ...

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 10:20

Message:
The reason for the 'forever' compile time is that the physical plan space
blows up in a heuristic that tries to find good plan orders. I now added a
check that avoids generating to many plans.

I checked in the fix in the Nov2009 stable branch. I'm however not sure
whether it makes it into the release -- probably not. So please get it from
either CVS or nightly-builds.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-17 09:32

Message:
not sure about category or group - chose seemingly most appropriate.

even though I did not change default priority , having some answer for
this
(whether fix in code, or guideline for fast compiling query) would be very
helpful.
In my application the pf compilation run is triggered from a web-gui -
fast compilation
is very important there, and predicatable compilation times maybe even
more.
(i.e. it would be nice when all compilations would take the same order of
time).

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-18T09:34:17Z

Bugs item #2898944, was opened at 2009-11-17 09:27
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "candidate"
Status: Open
Resolution: Fixed
Priority: 6
Private: No
Submitted By: Axel Belinfante (axelbel)
Assigned to: Jan Rittinger (tsheyar)
Summary: pf takes 'forever' (> 1 hour) to compile particular query

Initial Comment:
I don't know whether the following really qualifies as bug, or just is a case of 'user error by giving stupid input.
In any case, I would like to bring this to your attention.

When I compile a query for attached module definition with pf (may or aug or pre-nov version), compilation takes 'forever'.
When I replace the clause ecv:to-human($g0, $human_type) by simple $g0
and remove definitions of $human_type and to-human it compiles almost instantly.
It is the huge difference in compilation time that surprises me.

The query I compile is:

import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation.xq"; ecv:view-from-save("eprints-export-conv-fmt.xml")

where tmp/long-compilation.xq is the attached module.

What the query attempts to do: obtain results from the database, order these on two criteria, and when listing the results, insert human-readable headers.
The human-readable headers are obtained by the to-human mapping from field value (used as ordering criterium) to human-readable string.
At some point I just used if-then-else-if-then-else etc. to do the mapping, but that seemed slower to compile than the approach I took here.

Suggestions for rewriting the query to a fast-compiling one would be considered as a fix too, I guess.

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 18:34

Message:
===================================================================
2009/11/18 - stmane: pathfinder/tests/BugTracker/Tests/All,1.146.2.11

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.bat,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod1.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod2.xq,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q1.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q2.xq.in,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.sh,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.err,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.out,1.1.2.1

pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.timeout,1.1.2.1

added test (compilation via pf, only) for
ID: 2898944 "pf takes 'forever' (> 1 hour) to compile particular query"
https://sourceforge.net/tracker/?func=detail&aid=2898944&group_id=56967&atid=482468

Seems to work fine, now;
compiling both queries with both `pf -A` & `pf -M`
takes less than 2 secs on my machine with a debug build.
Set timeout to 6 secs.
===================================================================

report can be closed, once nightly testing and Axel confirm the fix.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 16:51

Message:
Plan generation with permutations gets you into trouble.

Planning the distinct operator requires a full ordering of the table as we
use the order extend in MIL to remove duplicates. To calculate the plans
for all possible orders we generate all permutations. (In the first
long-compilation example distinct was applied to a table with 7 columns and
for the second example distinct operated on 8 columns.)

Now we first eliminate all functionally dependent and constant columns as
they do not add any order information. If we still have too many
permutations we ignore all orderings (plans) after a magic boundary.

With this changes in place the compile times are (almost) acceptable:

pf -T query-long-compilation.xq
...
overall compilation time: 974ms

pf -T query-long-compilation2.xq
...
overall compilation time: 02s 781ms

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:53

Message:
Just a few general notes:

1)
A user program can at most trigger a machine crash, but never cause it.
If a user program triggers a machine crash (e.g., by using resources
(too?) many resources, it's in the end a problem (bug) in the OS (or the
hardware) that cannot properly handle (or prevent) such (over-)load
properly.

2)
If you kill (via keyboard interrupt ^C or by sending, say, SIGTERM or
SIGKILL) a process that uses lots of memory (and possibly virtual memory,
i.e., swap), it might indeed take some time until that process is gone and
your prompt is back, since the systems needs to clean up that memory (and
swap) and possibly swap your shell back into main memory ...

3)
Yes, a process that is using much memory (i.e., close to or more than the
machine has physical memory) might (usually will) affect the interactive
responsiveness of the machine.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 12:36

Message:
the original long-compilation.xq now finishes in about 10 secs, which is
still longer than I would like,
but much shorter than it used to.

regarding time measurements, I may have caused some confusion, in the
sense that:
- the original long-compilation.xq definitely took 'too long', but likely
I always interruped it
long before 1 hour passed
- the query that prompted me change the bug summmary line to include '> 1
hour' was a
different one, I'm sure that was long-compilation2.xq

long-compilation2.xq might have taken down a machine before the fix
(machine did crash while that pf was running, but we do not know whether
that was the cause)
after applying the fix and rerunning the pf on long-compilation2.xq I do
notice that
at some point the machine stops being responsive,
only to become responsive again after the segfault -
maybe the segfault saved us from a second machine crash :-)

I wonder whether the memory consumption is what so badly affects
responsivity of the machine.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:03

Message:
Axel,

what about your original long-compilation.xq?
Does that now work fine for you (i.e., with Jan's fix)?

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 11:30

Message:
I don't want to spoil the fun, but after the fix pf still runs pretty long
on the file I just attached
(long-compilation2.xq)

top gives me:
PID USER NI VIRT RES SHR P S %CPU %MEM TIME COMMAND
26940 dbguest 0 7164m 3.7g 172 4 R 27 96.5 14:07
/local/dbguest/Install/MonetDB-XQuery_Nov2009/bin/pf -T /tmp/talt11

I tried to kill it with ^C, without too much success
(and I already started to fear that the machine was hung up on it),
until a little later I saw (I started it as: time /path/to/pf
xqueryfile)

Killed
828.295u 39.446s 16:38.94 86.8% 0+0k 0+0io 5pf+0w
Segmentation fault

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 11:03

Message:
I think, adding a test (also) for this one should be done; it should be not
only possible, but also easy:
the query that (used to) compile for for than one hour is given, just make
a test that calls pf to compile the query, sending the output to, say,
/dev/null/ .
If that finishes without error and within the default 1 minute timeout (or
less, if desired/required; add a TSTNME.timeout file with a <1 scale
factor), the test succeeded and everything is fine; otherwise, the test
fails ...

... just an idea to help assessing and monitoring this issue and hence the
stability and usability of MonetDB as such ...

... does not apply to this bug report, only; see also my email of last
weekend ...

... if there is no time right no to add the test, keep the bug report open
to remind us ...

... in fact, I do exactly that ...

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 10:20

Message:
The reason for the 'forever' compile time is that the physical plan space
blows up in a heuristic that tries to find good plan orders. I now added a
check that avoids generating to many plans.

I checked in the fix in the Nov2009 stable branch. I'm however not sure
whether it makes it into the release -- probably not. So please get it from
either CVS or nightly-builds.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-17 09:32

Message:
not sure about category or group - chose seemingly most appropriate.

even though I did not change default priority , having some answer for
this
(whether fix in code, or guideline for fast compiling query) would be very
helpful.
In my application the pf compilation run is triggered from a web-gui -
fast compilation
is very important there, and predicatable compilation times maybe even
more.
(i.e. it would be nice when all compilations would take the same order of
time).

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-18T07:51:28Z

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-18T03:53:34Z

Bugs item #2898944, was opened at 2009-11-17 09:27
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "candidate"
Status: Open
Resolution: Fixed
Priority: 6
Private: No
Submitted By: Axel Belinfante (axelbel)
Assigned to: Jan Rittinger (tsheyar)
Summary: pf takes 'forever' (> 1 hour) to compile particular query

Initial Comment:
I don't know whether the following really qualifies as bug, or just is a case of 'user error by giving stupid input.
In any case, I would like to bring this to your attention.

When I compile a query for attached module definition with pf (may or aug or pre-nov version), compilation takes 'forever'.
When I replace the clause ecv:to-human($g0, $human_type) by simple $g0
and remove definitions of $human_type and to-human it compiles almost instantly.
It is the huge difference in compilation time that surprises me.

The query I compile is:

import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation.xq"; ecv:view-from-save("eprints-export-conv-fmt.xml")

where tmp/long-compilation.xq is the attached module.

What the query attempts to do: obtain results from the database, order these on two criteria, and when listing the results, insert human-readable headers.
The human-readable headers are obtained by the to-human mapping from field value (used as ordering criterium) to human-readable string.
At some point I just used if-then-else-if-then-else etc. to do the mapping, but that seemed slower to compile than the approach I took here.

Suggestions for rewriting the query to a fast-compiling one would be considered as a fix too, I guess.

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:53

Message:
Just a few general notes:

1)
A user program can at most trigger a machine crash, but never cause it.
If a user program triggers a machine crash (e.g., by using resources
(too?) many resources, it's in the end a problem (bug) in the OS (or the
hardware) that cannot properly handle (or prevent) such (over-)load
properly.

2)
If you kill (via keyboard interrupt ^C or by sending, say, SIGTERM or
SIGKILL) a process that uses lots of memory (and possibly virtual memory,
i.e., swap), it might indeed take some time until that process is gone and
your prompt is back, since the systems needs to clean up that memory (and
swap) and possibly swap your shell back into main memory ...

3)
Yes, a process that is using much memory (i.e., close to or more than the
machine has physical memory) might (usually will) affect the interactive
responsiveness of the machine.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 12:36

Message:
the original long-compilation.xq now finishes in about 10 secs, which is
still longer than I would like,
but much shorter than it used to.

regarding time measurements, I may have caused some confusion, in the
sense that:
- the original long-compilation.xq definitely took 'too long', but likely
I always interruped it
long before 1 hour passed
- the query that prompted me change the bug summmary line to include '> 1
hour' was a
different one, I'm sure that was long-compilation2.xq

long-compilation2.xq might have taken down a machine before the fix
(machine did crash while that pf was running, but we do not know whether
that was the cause)
after applying the fix and rerunning the pf on long-compilation2.xq I do
notice that
at some point the machine stops being responsive,
only to become responsive again after the segfault -
maybe the segfault saved us from a second machine crash :-)

I wonder whether the memory consumption is what so badly affects
responsivity of the machine.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 12:03

Message:
Axel,

what about your original long-compilation.xq?
Does that now work fine for you (i.e., with Jan's fix)?

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-18 11:30

Message:
I don't want to spoil the fun, but after the fix pf still runs pretty long
on the file I just attached
(long-compilation2.xq)

top gives me:
PID USER NI VIRT RES SHR P S %CPU %MEM TIME COMMAND
26940 dbguest 0 7164m 3.7g 172 4 R 27 96.5 14:07
/local/dbguest/Install/MonetDB-XQuery_Nov2009/bin/pf -T /tmp/talt11

I tried to kill it with ^C, without too much success
(and I already started to fear that the machine was hung up on it),
until a little later I saw (I started it as: time /path/to/pf
xqueryfile)

Killed
828.295u 39.446s 16:38.94 86.8% 0+0k 0+0io 5pf+0w
Segmentation fault

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 11:03

Message:
I think, adding a test (also) for this one should be done; it should be not
only possible, but also easy:
the query that (used to) compile for for than one hour is given, just make
a test that calls pf to compile the query, sending the output to, say,
/dev/null/ .
If that finishes without error and within the default 1 minute timeout (or
less, if desired/required; add a TSTNME.timeout file with a <1 scale
factor), the test succeeded and everything is fine; otherwise, the test
fails ...

... just an idea to help assessing and monitoring this issue and hence the
stability and usability of MonetDB as such ...

... does not apply to this bug report, only; see also my email of last
weekend ...

... if there is no time right no to add the test, keep the bug report open
to remind us ...

... in fact, I do exactly that ...

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-18 10:20

Message:
The reason for the 'forever' compile time is that the physical plan space
blows up in a heuristic that tries to find good plan orders. I now added a
check that avoids generating to many plans.

I checked in the fix in the Nov2009 stable branch. I'm however not sure
whether it makes it into the release -- probably not. So please get it from
either CVS or nightly-builds.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-17 09:32

Message:
not sure about category or group - chose seemingly most appropriate.

even though I did not change default priority , having some answer for
this
(whether fix in code, or guideline for fast compiling query) would be very
helpful.
In my application the pf compilation run is triggered from a web-gui -
fast compilation
is very important there, and predicatable compilation times maybe even
more.
(i.e. it would be nice when all compilations would take the same order of
time).

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-18T03:36:00Z

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-18T03:03:34Z

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-18T02:30:36Z

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-18T02:03:05Z

[ monetdb-Bugs-2009556 ] XQ: "Zombie" document in collection

2009-11-18T01:52:38Z

Bugs item #2009556, was opened at 2008-07-03 11:41
Message generated for change (Settings changed) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2009556&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/runtime
Group: Pathfinder "stable"
>Status: Closed
Resolution: Fixed
Priority: 8
Private: No
Submitted By: Luc Touraille (touraillel)
Assigned to: Lefteris Sidirourgos (lsidir)
Summary: XQ: "Zombie" document in collection

Initial Comment:
[Jun 2008 release]

A deleted document from a collection where there is more than 1 document is still visible using the collection("mycoll") function.

On Feb 2008 release, the pf:collection function worked correctly, but not the fn:collection function, that produced the same bug.

Stefan reproduced this bug, giving the following trace :

========
$ cat /tmp/MyDoc1.xml
--------
<a/>
========
$ cat /tmp/MyDoc2.xml
--------

========
========
$ mclient -lx -s'pf:add-doc("/tmp/MyDoc1.xml","MyDoc1","MyCol")'
--------
========
$ mclient -lx -s'pf:collections()'
--------
<collection updatable="false" size="99 KiB" numDocs="1">MyCol</collection>
========
$ mclient -lx -s'pf:collection("MyCol")'
--------
<a/>
========
$ mclient -lx -g -s'fn:collection("MyCol")'
--------
<?xml version="1.0" encoding="utf-8"?>
<a/>

========
$ mclient -lx -s'pf:documents()'
--------
<document updatable="false" url="/tmp/MyDoc1.xml" collection="MyCol">MyDoc1</document>
========
$ mclient -lx -s'pf:documents("MyCol")'
--------
<document updatable="false" url="/tmp/MyDoc1.xml" collection="MyCol">MyDoc1</document>
========
========
$ mclient -lx -s'pf:del-doc("MyDoc1")'
--------
========
$ mclient -lx -s'pf:collections()'
--------

========
$ mclient -lx -s'pf:collection("MyCol")'
--------

========
$ mclient -lx -g -s'fn:collection("MyCol")'
--------

========
$ mclient -lx -s'pf:documents()'
--------

========
$ mclient -lx -s'pf:documents("MyCol")'
--------

========
========
$ mclient -lx -s'pf:add-doc("/tmp/MyDoc1.xml","MyDoc1","MyCol")'
--------
========
$ mclient -lx -s'pf:add-doc("/tmp/MyDoc2.xml","MyDoc2","MyCol")'
--------
========
$ mclient -lx -s'pf:collections()'
--------
<collection updatable="false" size="99 KiB" numDocs="2">MyCol</collection>
========
$ mclient -lx -s'pf:collection("MyCol")'
--------
<a/>
========
$ mclient -lx -g -s'fn:collection("MyCol")'
--------
<?xml version="1.0" encoding="utf-8"?>
<a/>
,
<?xml version="1.0" encoding="utf-8"?>


========
$ mclient -lx -s'pf:documents()'
--------
<document updatable="false" url="/tmp/MyDoc1.xml" collection="MyCol">MyDoc1</document>,
<document updatable="false" url="/tmp/MyDoc2.xml" collection="MyCol">MyDoc2</document>
========
$ mclient -lx -s'pf:documents("MyCol")'
--------
<document updatable="false" url="/tmp/MyDoc2.xml" collection="MyCol">MyDoc2</document>,
<document updatable="false" url="/tmp/MyDoc1.xml" collection="MyCol">MyDoc1</document>
========
========
$ mclient -lx -s'pf:del-doc("MyDoc2")'
--------
========
$ mclient -lx -s'pf:collections()'
--------
<collection updatable="false" size="99 KiB" numDocs="1">MyCol</collection>
========
$ mclient -lx -s'pf:collection("MyCol")'
--------
<a/>
========
$ mclient -lx -g -s'fn:collection("MyCol")'
--------
<?xml version="1.0" encoding="utf-8"?>
<a/>
,
<?xml version="1.0" encoding="utf-8"?>


========
$ mclient -lx -s'pf:documents()'
--------
<document updatable="false" url="/tmp/MyDoc1.xml" collection="MyCol">MyDoc1</document>
========
$ mclient -lx -s'pf:documents("MyCol")'
--------
<document updatable="false" url="/tmp/MyDoc1.xml" collection="MyCol">MyDoc1</document>
========
========
$ mclient -lx -s'pf:del-doc("MyDoc1")'
--------
========
$ mclient -lx -s'pf:collections()'
--------

========
$ mclient -lx -s'pf:collection("MyCol")'
--------

========
$ mclient -lx -g -s'fn:collection("MyCol")'
--------

========
$ mclient -lx -s'pf:documents()'
--------

========
$ mclient -lx -s'pf:documents("MyCol")'
--------

========

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 10:52

Message:
confirmed.
closing.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-17 09:56

Message:
Thanks!
I made the test's output invariant (independent of local paths) by loading
documents from the MonetDB web site instead of from local files,
and moved it to be first in this directory, to not require
delete_all_docs() to empty the DB --- the latter might take several minutes
to remove all docs accumulated by the numerous tests in this directory.

Seems to work fine, now.
Can be closed, once nightly testing confirms this.

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2009-11-17 01:05

Message:
the output differs as follows:
- the collection function given an error on a nonexistent collection,
which is ok
- the attenpt to serialize a collection gives an error - which is ok

approved the output, this test seems to work correctly on my platform

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-16 22:29

Message:
the test has been failing at least since 2009-01-01, now resulting in a
timeout
cf.,
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/Zombie_document.SF-2009556.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/Zombie_document.SF-2009556.out.00.html

----------------------------------------------------------------------

Comment By: Fabian (mr-meltdown)
Date: 2009-04-07 11:34

Message:
debug spam, sorry for the noise.

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2009-04-07 10:48

Message:
I think we should give an run-time error in the serializer if the
collection root gets serialized..

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-02-16 01:08

Message:
cf.
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora8/tests_BugTracker/Zombie_document.SF-2009556.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsg103/GNU.64.64.d.1-Fedora8/tests_BugTracker/Zombie_document.SF-2009556.out.00.html

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-02-15 21:58

Message:
Lefteris,

what is the status of this one?

Stefan

----------------------------------------------------------------------

Comment By: Lefteris Sidirourgos (lsidir)
Date: 2008-12-16 09:12

Message:
During the pathfinder skype meeting on Monday 15/12 the solution that was
proposed was to apply the docfilter proc after the pf:collection+step. This
will fix the fn:collection results. However if a query serializes the
results of pf:collection, the delete documents will still appear. This is
acceptable since pf:collection was not designed for such use, and users
should use the built in functions of XQuery (i.e., fn:collection) rather
than pf:collection.

In addition docfilter should break to 2 new functions to separate the
"swizzling" from the filtering.

----------------------------------------------------------------------

Comment By: Lefteris Sidirourgos (lsidir)
Date: 2008-12-15 16:45

Message:
Hi, the bug is still around indeed. The bug characteristics are as follows:
* If the mil proc ws_collection_root() is invoked, delete documents still
appear
* if the mil proc ws_collection() is invoked, delete documents do not
appear.

By investigating a little more, after the second document is deleted the
database looks like that:

mil>print(doc_name);
#---------------------------------#
# h alias # name
# oid str # type
#---------------------------------#
[ 1000000000@0, "MyDoc1" ]
[ 1000000001@0, nil ]

Which is expected. On the other hand,
mil>print(doc_collection);
#---------------------------------#
# h t # name
# oid oid # type
#---------------------------------#
[ 1@0, 14@0 ]
[ 1000000000@0, 1000000000@0 ]
[ 0@0, 1000000002@0 ]
[ 1000000001@0, 1000000000@0 ]

thus the delete document appears in the tail, which it looks that this is
known and correct since the implementation of ws_collection() knows that
and has the following two lines of code:
var docs := doct.leftjoin(doc_name).select(str_nil,str_nil); # selected
existing docs
doc := doch.leftjoin(docs.mirror().leftjoin(doct)); # reduced doc to
existing docs

thus from the doc_name BAT decides which document still exists, but in the
implementation of ws_collection_root this is not the case, since no
filtering is happening. Notice that the PRE_Table, after deleting the
document and after restarting the logger looks like:

PRE table
[ 0@0 ]
#-------------------------------------------------------------------------#
# h nid size level prop kind tag cont
# name
# void oid int chr oid str str oid
# type
#-------------------------------------------------------------------------#
[ 0@0, nil, nil, 'þ', nil, "COLLECTION", nil, nil
]
[ 1@0 ]
#-------------------------------------------------------------------------#
# h nid size level prop kind tag cont
# name
# void void int chr oid str str oid
# type
#-------------------------------------------------------------------------#
[ 0@0, 0@0, 4, 'þ', nil, "COLLECTION", nil, 1@0
]
[ 1@0, 1@0, 1, 'ÿ', nil, "DOCUMENT", nil, 1@0
]
[ 2@0, 2@0, 0, '\000', 0@0, "ELEMENT", "||a", 1@0
]
[ 3@0, 3@0, 1, 'ÿ', nil, "DOCUMENT", nil, 1@0
]
[ 4@0, 4@0, 0, '\000', 1@0, "ELEMENT", "||b", 1@0
]

thus the elements etc. still exist

I am not sure what is the best way to fix this problem, so I am open to
suggestions, I am also not sure what are the steps taken after a document
is deleted, so I will need Peter's help on this one

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2008-12-12 00:02

Message:
Lefteris, can you check if the output is correct (the extra <aap>??) and
this bug really fixed? If so, close it. If not, re-open it. Thanks!

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2008-11-18 11:48

Message:
The URLs are indeed to be ignored --- not trivial with Mtest.py right now,
though, I'll check and think about it.

The changes sizes in KB look very suspicious to me --- I would strongly
object to ignoring or approving them unless we know for sure why they
differ, or better, which size they represent --- I honestly don't know
either, yet ...

Moreover, the current output shows an extra

<aap>
Hello World
</aap>

compared to the stable/approved output --- also that needs to be checked
before considering this bug fixed ...

----------------------------------------------------------------------

Comment By: Lefteris Sidirourgos (lsidir)
Date: 2008-11-18 11:31

Message:
Hi,

the test Zombie_document.SF-2009556 seems to be failing not because the
bug is still there, but because of some dynamic(?) information in the
documents, such as size in kb and URL. I am not aware if and how can we
make testweb to ignore these differences. Assigning to Stefan:) I
considering this bug fixed though. We just have to manage the tests a
little, the same holds for the rest, of the related to this bug, reports.

lefteris

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2008-07-03 22:34

Message:
Logged In: YES
user_id=572415
Originator: NO

Might be related to
[ 1976341 ] XQ: leftovers after deleting document
http://sourceforge.net/tracker/index.php?func=detail&aid=1976341&group_id=56967&atid=482468

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2008-07-03 22:33

Message:
Logged In: YES
user_id=572415
Originator: NO

The problem seems to be "even worse":

* add 2 documents to a single collection "MyCol"
* delete one of them, again
=> + pf:colections(), pf:documents(), & pf:documents("MyCol") correctly
show only one remaining document
- pf:collection("MyCol") & fn:collection("MyCol") wrongly show both
documents

* stop Mserver
* restart Mserver
=> - all above functions wrongly show *both* documents, again!

This happens both with the (now) default Algebra back-end and with the
(old) "milprint_summer" back-end.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2008-07-03 14:48

Message:
Logged In: YES
user_id=572415
Originator: NO

Added test in
pathfinder/tests/BugTracker/Tests/Zombie_document.SF-2009556.*

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2009556&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-1911209 ] XQ: MonetDB 'hangs' after requesting non-existing doc

2009-11-18T01:45:25Z

Bugs item #1911209, was opened at 2008-03-10 17:37
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1911209&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/runtime
Group: (zombie: MonetDB4 4.22)
>Status: Closed
Resolution: Fixed
Priority: 8
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Nobody/Anonymous (nobody)
Summary: XQ: MonetDB 'hangs' after requesting non-existing doc

Initial Comment:
Using mps.

Querying 'pf:documents()' will never return ('hang') after having inserted a node and then querying for a non-existing doc a few times.

Reproduce by inserting a document "aap.xml" in the database (with content '<aap/>' and shred it updatable):

pf:add-doc("/tmp/aap.xml","aap.xml","aap.xml",10)

The following script will execute fine (can be run many times):

N=0
while [[ $N -lt 1000 ]]
do
echo "Step $N"
echo 'pf:documents()' | mclient -lxq > /dev/null
echo 'doc("aap.xml")' | mclient -lxq > /dev/null
N=$(($N + 1))
done

This script will execute fine (it returns an error for each document, can also be run many times):

N=0
while [[ $N -lt 1000 ]]
do
echo "Step $N"
echo 'pf:documents()' | mclient -lxq > /dev/null
echo 'doc("does_not_exist.xml")' | mclient -lxq > /dev/null
N=$(($N + 1))
done

perform the update:

do insert <beer/> into doc("aap.xml")/aap

now execute the above script again:

N=0
while [[ $N -lt 1000 ]]
do
echo "Step $N"
echo 'pf:documents()' | mclient -lxq > /dev/null
echo 'doc("does_not_exist.xml")' | mclient -lxq > /dev/null
N=$(($N + 1))
done

on my system this script 'hangs' on the 'pf:documents()' query after a few tries (between 10 and 250).

System: Fedora Core 8, 64bit, MonetDB 4.22 64bit-oids, Feb2008 supertarball.

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 10:45

Message:
confirmed.
closing.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-17 10:57

Message:
"fixed" by moving this test to be the second in this directory such that it
runs on an empty DB and hence the output is identical as when run in
isolation.

To be closed once nightly testing confirms the "fix".

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-16 22:24

Message:
re-opened as the test fails since it has been added on 2009.08.01
cf.,
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/hang_non_existing_doc.SF-1911209.out.00.html

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-07-31 10:03

Message:
Finally added test in pathfinder/tests/BugTracker:
hang_non_existing_doc.SF-1911209.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-02-16 13:36

Message:
re-opened to remind us that we should consider adding a test

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2008-03-30 03:00

Message:
Logged In: YES
user_id=591107
Originator: NO

a recent checking in gdk_bbp.x (deadlock resolved) seems to solve this
bug.

If you update MonetDB to the HEAD, I would expect the problem to disappear

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2008-03-30 01:58

Message:
Logged In: YES
user_id=591107
Originator: NO

a recent checking in gdk_bbp.x (deadlock resolved) seems to solve this
bug.

If you update MonetDB to the HEAD, I would expect the problem to disappear

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2008-03-29 21:22

Message:
Logged In: YES
user_id=591107
Originator: NO

a recent checking in gdk_bbp.x (deadlock resolved) seems to solve this
bug.

If you update MonetDB to the HEAD, I would expect the problem to disappear

----------------------------------------------------------------------

Comment By: Wouter Alink (vzzzbx)
Date: 2008-03-12 13:15

Message:
Logged In: YES
user_id=621590
Originator: YES

forgot to say: it doesn't happen with 4.20

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1911209&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2893875 ] PF: 82 tests fail on Auf2009_NFI/Nov2009 but work on Aug2009

2009-11-18T01:43:47Z

Bugs item #2893875, was opened at 2009-11-07 17:19
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2893875&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/tests
Group: Pathfinder "stable"
>Status: Closed
>Resolution: Fixed
Priority: 9
Private: No
Submitted By: Stefan Manegold (stmane)
Assigned to: Stefan Manegold (stmane)
Summary: PF: 82 tests fail on Auf2009_NFI/Nov2009 but work on Aug2009

Initial Comment:
The following 82 tests work(ed) fine with the Aug2009 branch,
but appear to fail with the Aug2009 branch,
and hence most probaly also with the Nov2009 branch
(which is currently even more instable wrt. testing for yet undiscovered reasons):

benchmarks/MBench/qa05
benchmarks/MBench/
benchmarks/XBench/DC/MD/q19
benchmarks/XBench/DC/MD/
modules/pftijah/procs
modules/pftijah/colltest1
modules/pftijah/colltest2
modules/pftijah/
runtime/xrpc/admin/add_del_doc_noxrpc
runtime/xrpc/admin/backup_restore_noxrpc
runtime/xrpc/admin/
tests/BugTracker/treat_as.SF-1586454
tests/BugTracker/insert_multiple.SF-1590580
tests/BugTracker/inserting_multiple_elements.SF-1590583
tests/BugTracker/insert_into_transient.SF-1626208
tests/BugTracker/clear_attrs_on_delete.SF-1612739
tests/BugTracker/clear_attrs_on_delete.SF-1612739-b
tests/BugTracker/clear_attrs_on_delete.SF-1612739-c
tests/BugTracker/corrupt_after_update.SF-1706640
tests/BugTracker/accessing_renamed_inserted_deleted_node.SF-1718622-1718635-1718709
tests/BugTracker/insert_large_doc.SF-1726954
tests/BugTracker/replace-corrupts.SF-1758902
tests/BugTracker/swizzle-bug.SF-1760811
tests/BugTracker/swizzle-bug2.SF-1763495
tests/BugTracker/insert_attribute_gives_ERROR_in_merged_union.SF-1763575
tests/BugTracker/immune_for_updates.SF-1766259
tests/BugTracker/repeated-insert.SF-1814911
tests/BugTracker/insert-new-page.SF-1854215
tests/BugTracker/immune_for_updates.SF-1981852
tests/BugTracker/hang_non_existing_doc.SF-1911209
tests/BugTracker/function_parameter_without_type.SF-1898518
tests/BugTracker/copybug.SF-2642003
tests/BugTracker/update-with-timing.SF-2852928
tests/BugTracker/compilation_error.SF-2860574_1
tests/BugTracker/compilation_error.SF-2860574_2
tests/BugsViaSourgeforce/invisible_error_messages.SF-1409122
tests/BugsViaSourgeforce/
tests/StandOff/update/basic_insert
tests/StandOff/update/basic_test
tests/StandOff/update/
tests/Update/setattr-1
tests/Update/insert-1
tests/Update/insert-2
tests/Update/symm
tests/Update/replacevaluetest
tests/Update/replacevaluetest2
tests/Update/update
tests/Update/illegal-insert
tests/Update/illegal-delete
tests/Update/insert_test_order
tests/Update/insert_test_order_seq
tests/Update/replacenode-corruption.SF-2153245
tests/W3C_use_cases/XQUF/AddressBook/Q1x
tests/W3C_use_cases/XQUF/AddressBook/check_docs
tests/W3C_use_cases/XQUF/AddressBook/
tests/W3C_use_cases/XQUF/Parts/Q1
tests/W3C_use_cases/XQUF/Parts/Q2
tests/W3C_use_cases/XQUF/Parts/Q3a1
tests/W3C_use_cases/XQUF/Parts/Q3a2
tests/W3C_use_cases/XQUF/Parts/Q4a
tests/W3C_use_cases/XQUF/Parts/Q4ax
tests/W3C_use_cases/XQUF/Parts/Q4b
tests/W3C_use_cases/XQUF/Parts/Q4bx
tests/W3C_use_cases/XQUF/Parts/Q4c
tests/W3C_use_cases/XQUF/Parts/Q4cx
tests/W3C_use_cases/XQUF/Parts/Q6
tests/W3C_use_cases/XQUF/Parts/Q6x
tests/W3C_use_cases/XQUF/Parts/
tests/W3C_use_cases/XQUF/R/Q1
tests/W3C_use_cases/XQUF/R/Q2
tests/W3C_use_cases/XQUF/R/Q3
tests/W3C_use_cases/XQUF/R/Q4
tests/W3C_use_cases/XQUF/R/Q4x
tests/W3C_use_cases/XQUF/R/Q5b
tests/W3C_use_cases/XQUF/R/Q6a
tests/W3C_use_cases/XQUF/R/Q6b
tests/W3C_use_cases/XQUF/R/Q7
tests/W3C_use_cases/XQUF/R/Q7x
tests/W3C_use_cases/XQUF/R/Q8
tests/W3C_use_cases/XQUF/R/Q9
tests/XQuery/is-before4
tests/XQuery/union

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-18 10:43

Message:
confirmed.
closing.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-17 10:08

Message:
forgot to finish:

"I already" ...
... approved ALG-specific output of the insert-order test on Lefteris'
advice and closed the bug report:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1964365&group_id=56967

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-17 10:05

Message:
Thanks, Peter!

Your checkins were "too late" for last night's (this morning's) testing,
but local testing shows pathfinder/benchmarks/MBench/Tests/qa05.* and
pathfinder/tests/Update/Tests/update.* working fine, again.

I already

Can be closed once confirmed by nightly testing.
(For now, system-specific problems are also documented in the TestWeb, and
should be handled later; they're not considered crucial for the upcoming
release.)

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2009-11-17 00:32

Message:
both problems are fixed now

I did spot in tests/Update/insert_test_order_seq
another instance of differences accross all platforms.

The test is a little vauge, inserting a sequence of values; I am not sure
what the XQUF semantics says about this. Is the order
implementation-dependent? This this order might also be ok and the test
could be approved.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-16 18:42

Message:
In MBench/qa05 Peter's heuristic rewrite leads to an (unnecessary) cross
product. (For all eNest nodes it applies pf:attribute.) I guess that might
be the problem.

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-16 15:42

Message:
This is a show stopper.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-16 15:28

Message:
Peter,

the two "crucial" problems remaining (which both started on the Nov2009
branch after propagations of NFI changes from the Aug2009_NFI branch to the
Nov2009 branch) are

timeout in
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/benchmarks_MBench/qa05.err.00.html
(since 2009.10.09)

missing updates in
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_Update/update.out.00.html
(since 2009.10.08)
They might actually be missing, because the query parts of the update
queries involving an attribute predicate yield an empty result.
The missing updates are

for $elem in $testdoc/document/element[@attribute = 50]
return do rename exactly-one($elem) into "Element"

for $elem in $testdoc/document/element[@attribute = 10]
return (do insert <a/> as first into exactly-one($elem) treat as
element(),
do insert as first into exactly-one($elem) treat as element())

for $elem in $testdoc/document/element[@attribute = 20]
return (do insert <a/> as last into exactly-one($elem) treat as
element(),
do insert as last into exactly-one($elem) treat as element())

for $elem in $testdoc/document/element[@attribute = 30]
return (do insert <a/> before exactly-one($elem),
do insert before exactly-one($elem))

for $elem in $testdoc/document/element[@attribute = 40]
return (do insert <a/> after exactly-one($elem),
do insert after exactly-one($elem))

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-15 13:57

Message:
below the details what I did so far:

> - tests/BugTracker/empty_file.SF-2017862: SunOS specific version only!!
approved.
> - tests/BugTracker/XML_document_cache_broken.SF-1414720
approved.
> - tests/W3C_use_cases/XQ/NS/* (or remove tests!)
disabled.
> - tests/XQuery/{div.is-before4,union}
div:
approved MIL error messages iso. XQuery error message

is-before4, union:
nothing, as I'm not sure, whether the new output is correct.
Since the NFI change were propagated to the Nov2009 branch on Oct 7,
these tests show again the output as it was before
===================================================================
2009/05/20 - tsheyar:
pathfinder/tests/XQuery/Tests/is-before4.stable.out,1.5.30.1
pathfinder/tests/XQuery/Tests/union.stable.out,1.6.12.1
-- Make the error message for missing documents more informative:
Include the document name in the error message.

-- Adjust new error messages in the testweb.

-- Adjust stable output (for tests with a modified inter-document order).

-- Fix/Extend a cross product rewrite (for unreferenced inputs).

-- Simplify the 'missing document' error message in the algebra
for constant document names (which is the default).
===================================================================
cf.,
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_XQuery/is-before4.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_XQuery/union.out.00.html

Peter and/or Jan R., could you please comment?

> -
tests/BugDay_2005-12-19_0.9.3/attribute_after_non-attribute_content.SF-1351516
disabled.
> - tests/BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372: windows
only
minor --- either approve or change access rights on Windows machine:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.32.d.1-Windows6.0/tests_BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.64.d.1-Windows6.0/tests_BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372.out.00.html
> - tests/clients/php_monetdb
fixed.
> - runtime/procs/approve
approved.
> - modules/pftijah/{load,sigs,procs}
approved.
> -
tests/BugsViaSourgeforce/{invisible_error_messages.SF-1409122,ID.1766173}
approved.

> (2) remove or disable the following tests, which seem incorrect or
outdated:
> - tests/BugTracker/xs_untypedAtomic.SF-1509806.mps
disabled.
> - tests/BugTracker/Zombie_document.SF-2009556
Nothing yet; we get a timeout:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/Zombie_document.SF-2009556.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/Zombie_document.SF-2009556.err.00.html
> - tests/BugTracker/function_parameter_without_type.SF-1898518
approved.
> - tests/BugTracker/fn-root_fn-id_on_attribute_nodes.SF-1
disabled.
> - tests/BugTracker/975028,non-existing_collection.SF-1991726
approved.
> - tests/BugTracker/port_busy.SF-1809586
Nothing, yet; output might actually be OK / as expected:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/port_busy.SF-1809586.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/port_busy.SF-1809586.err.00.html
> - tests/BugTracker/collection_management
Nothing, yet; need to understand what's (supposed to be) going on:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/collection_management.SF-1726599.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/collection_management.SF-1726599.err.00.html
> - tests/BugDay_2005-12-19_0.9.3/multiple_servers_2.SF-1385152
fixed.
> - runtime/smack
Nothing, yet; smack00 test works fine for MIL, MAL, SQL, but never worked
(timout: hangs "forever") for XQuery
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime/smack00.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime/smack00.err.00.html
> - benchmarks/mbench/{qa02,qa06} are now only run for Darwin9.8.0
G.32.32.d.1
> -- should also set NOT_ALGEBRA for this platform
these are (and have been) skipped with Algebra on all platforms.

> (3) ask Jennie to look at the following tests in runtime/xrpc/admin/:
> - add_del_doc_norpc: approve?
> - backup_restore_xrpc: empty, check this pls??
Since 12.11., these now fail with

ERROR = !module import: error loading module from
http://$HOST:47966/admin/admin.xq

http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime_xrpc_admin/add_del_doc_noxrpc.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime_xrpc_admin/backup_restore_noxrpc.err.00.html

> (4) ask Jan Flokstra to look at the following tests in
modules/pftijah/:
> - coltest1 hang
> - test_lms-or: order
> - test_lms_rmoverlap, test_select_start*: empty
fixed by Sjoerd's backport of Hennigs fixes from the head to the Nov2009
branch

> (5) adapt or disable these test scripts to work on windows:
> - tests/BugDay_2005-12-19_0.9.3/file_locked_after_shredding.SF-1238352
> (windows6 only) -
> - tests/BugDay_2005-12-19_0.9.3/shredding_on_the_fly.SF-1377006: idem
> - tests/BugTracker/predicate_selects_too_few_nodes.SF-1636588 - windows
fails
> to parse, because of Mtest substitution failure (for $v becomes for 4)
fixed by Sjoerd.

> (6) ignore output of tests (possibly for certain platforms only), as
these are
> system limits related (too deep recursion):
> - tests/BugTracker/crash_on_concatenated_query.SF-1730547 - for all
systems
> - tests/BugTracker/server-side_compilation_crash.SF-1607210 - for all
systems
> - tests/BugsViaSourgeforce/ID.1015172{a,b,c}: timeout on windows5
(windows6
> works)
> - tests/XQuery/step_attr_nametest: Darwin9.8.0 G.32.32.d too deep
recursion,
> - tests/XQuery/typeswitch3 - too deep recursion on various systems..
> - tests/BugDay_2005-11-09_0.9.3/xquery_crashtest.SF-1207048: pf
recursion
> check not triggered on Fedora10 G.32.32.d.1
to be done ...

didn't get further than here, so far ...

Stefan

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2009-11-11 11:45

Message:
Thanks. I looked into this and fixed a problem with XQuery updates that
indeed was present. This will reduce the amount of problems, removing
(almost) all of those tests that are XQuery updates from the list.

The update bug was caused by a later modification that succeeded my own
check of the testweb that I performed before checking in the code. At that
time, already, I found it very hard to work with the test web. Note, this
is not a complaint to anyone in particular. However a matter of fact.
Simply too many tests fail, such that one is forced to spend an hour just
to compare the state of the testweb before and after a check-in, to detect
which of the tests that fail were actually already failing before the
check-in. Let alone analyze what is happening.

my suggestons are as follows:

(1) approve output:
- tests/BugTracker/empty_file.SF-2017862: SunOS specific version only!!
- tests/BugTracker/XML_document_cache_broken.SF-1414720
- tests/W3C_use_cases/XQ/NS/* (or remove tests!)
- tests/XQuery/{div.is-before4,union}
-
tests/BugDay_2005-12-19_0.9.3/attribute_after_non-attribute_content.SF-1351516
- tests/BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372: windows
only
- tests/clients/php_monetdb
- runtime/procs/approve
- modules/pftijah/{load,sigs,procs}
-
tests/BugsViaSourgeforce/{invisible_error_messages.SF-1409122,ID.1766173}

(2) remove or disable the following tests, which seem incorrect or
outdated:
- tests/BugTracker/xs_untypedAtomic.SF-1509806.mps
- tests/BugTracker/Zombie_document.SF-2009556
- tests/BugTracker/function_parameter_without_type.SF-1898518
- tests/BugTracker/fn-root_fn-id_on_attribute_nodes.SF-1
- tests/BugTracker/975028,non-existing_collection.SF-1991726
- tests/BugTracker/port_busy.SF-1809586
- tests/BugTracker/collection_management
- tests/BugDay_2005-12-19_0.9.3/multiple_servers_2.SF-1385152
- runtime/smack
- benchmarks/mbench/{qa02,qa06} are now only run for Darwin9.8.0
G.32.32.d.1 -- should also set NOT_ALGEBRA for this platform

(3) ask Jennie to look at the following tests in runtime/xrpc/admin/:
- add_del_doc_norpc: approve?
- backup_restore_xrpc: empty, check this pls??

(4) ask Jan Flokstra to look at the following tests in modules/pftijah/:
- coltest1 hang
- test_lms-or: order
- test_lms_rmoverlap, test_select_start*: empty

(5) adapt or disable these test scripts to work on windows:
- tests/BugDay_2005-12-19_0.9.3/file_locked_after_shredding.SF-1238352
(windows6 only) -
- tests/BugDay_2005-12-19_0.9.3/shredding_on_the_fly.SF-1377006: idem
- tests/BugTracker/predicate_selects_too_few_nodes.SF-1636588 - windows
fails to parse, because of Mtest substitution failure (for $v becomes for
4)

(6) ignore output of tests (possibly for certain platforms only), as these
are system limits related (too deep recursion):
- tests/BugTracker/crash_on_concatenated_query.SF-1730547 - for all
systems
- tests/BugTracker/server-side_compilation_crash.SF-1607210 - for all
systems
- tests/BugsViaSourgeforce/ID.1015172{a,b,c}: timeout on windows5
(windows6 works)
- tests/XQuery/step_attr_nametest: Darwin9.8.0 G.32.32.d too deep
recursion,
- tests/XQuery/typeswitch3 - too deep recursion on various systems..
- tests/BugDay_2005-11-09_0.9.3/xquery_crashtest.SF-1207048: pf recursion
check not triggered on Fedora10 G.32.32.d.1

what will be left, then
=======================

The only problems that will be left after following the above are in fact
problems that occur only on a single system.
Rather than indicating a platform deficiency, however, it is still very
likely that these differences indicate the
precense of a bug, that is only triggered under certain circumstances that
happen to hold on that platform.

so these are interesting to look at; all of them were already present in
the last Stable

Fedora10.G.64.64.d.0 - merge_union2: non dense head
- tests/XQuery/merge_union2
Fedora10 G.64.64.d.0 - crash
- tests/XQuery/step_attribute8,

Fedora10 I.64.64.d.1: crash
- tests/XQuery/orderby6

Fedora10 G.64.64.s.1 - crash
- tests/XQuery/step_descendant-or-self3

Fedora10.G.32.32.d.1 -- segmentation fault
- xbench/tc/md/q10
- tests/W3C_use_cases/XQ/XMP/Q07

windows - !ERROR: couldn't read name (1000000152_qn_uri) 1
- tests/BugTracker/32_docs.SF-1730617:

sunos5 g32.32.d.1 - segmentation fault
- mbench/{qs28,31}
- tests/XQuery/{step_duplicates,transient_upward_steps}
- benchmarks/XPathMark/{Q27,28}
- benchmarks/XPath/q13
- benchmarks/XMark/q04
- tests/XQuery/step_attr_nametest (also on Darwin9.8.0 G.32.32.d.1)
sunos5 g32.32.d.1 - 4 times inserted nil due to errors at tuples 0, 1, 2,
3
- tests/BugTracker/child-steps-and-replace.SF-2716723:

Debian4.0 G.32.32.d.1 - !fatal error: columns referenced in trace message
operator not found
- tests/BugTracker/fn-trace.SF-2805513 (also Gentoo2.0.1 G.64.64.d.1)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-08 18:13

Message:
The same test that fail with Aug2009_NFI, while working with Aug2009,
indeed also fail with Nov2009.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-08 16:50

Message:
For completeness:

I checked this on my 64-bit Fedora 10 Linux desktop, using a "testing"
build of MonetDB, i.e., configured with --disbale-debug --enable-optimize
--enable-assert .

I ran Mtest.py (more precisely make check, respectively RunMtest in the
build directory) seperately for the Aug2009 and the Aug2009_NFI brach
(builds) of pathfinder, collected the console output in files, and run diff
over those (as well as over the times.sql files that Mtest.py creates in
its <TSTTRGBASE> directory.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2893875&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-18T01:20:05Z

[ monetdb-Bugs-2898944 ] pf takes 'forever' (> 1 hour) to compile particular query

2009-11-17T12:09:55Z

Bugs item #2898944, was opened at 2009-11-17 09:27
Message generated for change (Settings changed) made by axelbel
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "candidate"
Status: Open
Resolution: None
>Priority: 6
Private: No
Submitted By: Axel Belinfante (axelbel)
Assigned to: Nobody/Anonymous (nobody)
>Summary: pf takes 'forever' (> 1 hour) to compile particular query

Initial Comment:
I don't know whether the following really qualifies as bug, or just is a case of 'user error by giving stupid input.
In any case, I would like to bring this to your attention.

When I compile a query for attached module definition with pf (may or aug or pre-nov version), compilation takes 'forever'.
When I replace the clause ecv:to-human($g0, $human_type) by simple $g0
and remove definitions of $human_type and to-human it compiles almost instantly.
It is the huge difference in compilation time that surprises me.

The query I compile is:

import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv" at "/tmp/long-compilation.xq"; ecv:view-from-save("eprints-export-conv-fmt.xml")

where tmp/long-compilation.xq is the attached module.

What the query attempts to do: obtain results from the database, order these on two criteria, and when listing the results, insert human-readable headers.
The human-readable headers are obtained by the to-human mapping from field value (used as ordering criterium) to human-readable string.
At some point I just used if-then-else-if-then-else etc. to do the mapping, but that seemed slower to compile than the approach I took here.

Suggestions for rewriting the query to a fast-compiling one would be considered as a fix too, I guess.

----------------------------------------------------------------------

Comment By: Axel Belinfante (axelbel)
Date: 2009-11-17 09:32

Message:
not sure about category or group - chose seemingly most appropriate.

even though I did not change default priority , having some answer for
this
(whether fix in code, or guideline for fast compiling query) would be very
helpful.
In my application the pf compilation run is triggered from a web-gui -
fast compilation
is very important there, and predicatable compilation times maybe even
more.
(i.e. it would be nice when all compilations would take the same order of
time).

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-1911209 ] XQ: MonetDB 'hangs' after requesting non-existing doc

2009-11-17T01:57:58Z

Bugs item #1911209, was opened at 2008-03-10 17:37
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1911209&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/runtime
Group: (zombie: MonetDB4 4.22)
Status: Open
Resolution: Fixed
Priority: 8
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Nobody/Anonymous (nobody)
Summary: XQ: MonetDB 'hangs' after requesting non-existing doc

Initial Comment:
Using mps.

Querying 'pf:documents()' will never return ('hang') after having inserted a node and then querying for a non-existing doc a few times.

Reproduce by inserting a document "aap.xml" in the database (with content '<aap/>' and shred it updatable):

pf:add-doc("/tmp/aap.xml","aap.xml","aap.xml",10)

The following script will execute fine (can be run many times):

N=0
while [[ $N -lt 1000 ]]
do
echo "Step $N"
echo 'pf:documents()' | mclient -lxq > /dev/null
echo 'doc("aap.xml")' | mclient -lxq > /dev/null
N=$(($N + 1))
done

This script will execute fine (it returns an error for each document, can also be run many times):

N=0
while [[ $N -lt 1000 ]]
do
echo "Step $N"
echo 'pf:documents()' | mclient -lxq > /dev/null
echo 'doc("does_not_exist.xml")' | mclient -lxq > /dev/null
N=$(($N + 1))
done

perform the update:

do insert <beer/> into doc("aap.xml")/aap

now execute the above script again:

N=0
while [[ $N -lt 1000 ]]
do
echo "Step $N"
echo 'pf:documents()' | mclient -lxq > /dev/null
echo 'doc("does_not_exist.xml")' | mclient -lxq > /dev/null
N=$(($N + 1))
done

on my system this script 'hangs' on the 'pf:documents()' query after a few tries (between 10 and 250).

System: Fedora Core 8, 64bit, MonetDB 4.22 64bit-oids, Feb2008 supertarball.

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-17 10:57

Message:
"fixed" by moving this test to be the second in this directory such that it
runs on an empty DB and hence the output is identical as when run in
isolation.

To be closed once nightly testing confirms the "fix".

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-16 22:24

Message:
re-opened as the test fails since it has been added on 2009.08.01
cf.,
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/hang_non_existing_doc.SF-1911209.out.00.html

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-07-31 10:03

Message:
Finally added test in pathfinder/tests/BugTracker:
hang_non_existing_doc.SF-1911209.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-02-16 13:36

Message:
re-opened to remind us that we should consider adding a test

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2008-03-30 03:00

Message:
Logged In: YES
user_id=591107
Originator: NO

a recent checking in gdk_bbp.x (deadlock resolved) seems to solve this
bug.

If you update MonetDB to the HEAD, I would expect the problem to disappear

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2008-03-30 01:58

Message:
Logged In: YES
user_id=591107
Originator: NO

a recent checking in gdk_bbp.x (deadlock resolved) seems to solve this
bug.

If you update MonetDB to the HEAD, I would expect the problem to disappear

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2008-03-29 21:22

Message:
Logged In: YES
user_id=591107
Originator: NO

a recent checking in gdk_bbp.x (deadlock resolved) seems to solve this
bug.

If you update MonetDB to the HEAD, I would expect the problem to disappear

----------------------------------------------------------------------

Comment By: Wouter Alink (vzzzbx)
Date: 2008-03-12 13:15

Message:
Logged In: YES
user_id=621590
Originator: YES

forgot to say: it doesn't happen with 4.20

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1911209&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-1730547 ] XQ: Mserver crashes on concatenated query

2009-11-17T01:15:22Z

Bugs item #1730547, was opened at 2007-06-04 10:42
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1730547&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/runtime
Group: Pathfinder CVS Head
Status: Open
Resolution: Fixed
>Priority: 8
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Stefan Manegold (stmane)
Summary: XQ: Mserver crashes on concatenated query

Initial Comment:
The attached query looks like:

"aap",
"aap",
"aap",
"aap",
... (1000 times)
"aap",
"aap"

When given to Mserver, it says: Segmentation Fault (0.12 as well as 0.17 (build last week)).

This bug should not have high priority.

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-17 10:15

Message:
Instead of a "ERROR = !fatal error: aborted too deep recursion"
the test now (again) triggers a timeout or segfault on all platform:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/crash_on_concatenated_query.SF-1730547.err.00.html

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2008-11-19 14:52

Message:
It was the "expected" output, since this test used to trigger a "ERROR =
!fatal error: aborted too deep recursion" (and hence produce no output) on
all platforms.

Apparently, this is no longer the case but now some platforms are able to
run the test without error, i.e., producing some (correct? -> to be
checked! ...) output.

Maybe, we need to split this test into two, a small one that works on all
platforms and a huge one that fails (i.e., triggers a reasonable error
message) on all platforms ...

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2008-11-19 14:42

Message:
Hmm I'm not sure if the 'correct' output is indeed the correct one:
http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/Int.64.32.d.1-Fedora8/tests_BugTracker/crash_on_concatenated_query.SF-1730547.out.00.html

@ Stefan could you please verify.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-06-05 20:44

Message:
Logged In: YES
user_id=572415
Originator: NO

Test added in
pathfinder/tests/BugTracker/Tests/Attic/crash_on_concatenated_query.SF-1730547.*

Thanks to Peter's latest fix, it now correctly triggers
"ERROR = !fatal error: aborted too deep recursion"
instead of crashing the server.

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2007-06-05 18:31

Message:
Logged In: YES
user_id=591107
Originator: NO

fixed

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2007-06-05 18:31

Message:
Logged In: YES
user_id=591107
Originator: NO

fixed

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-06-04 21:19

Message:
Logged In: YES
user_id=572415
Originator: NO

Debugging releaved that the segfault is indeed caused by burg running out
of stack space due to too deep recursion.

See also
[ 1607210 ] XQ: server-side compilation crash (member benchmark)
http://sourceforge.net/tracker/index.php?func=detail&aid=1607210&group_id=56967&atid=482468

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-06-04 18:21

Message:
Logged In: YES
user_id=572415
Originator: NO

for got to mention:
pf "eventually" produces some MIL plan; no error.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2007-06-04 18:20

Message:
Logged In: YES
user_id=572415
Originator: NO

Running this through plain pf (0.18), i.e.,

{ for i in `seq 1 999` ; do echo '"aap",' ; done ; echo '"aap"' ; } | pf

makes pf grow to 2.7 GB virt (1.8 GB res) on my 64-bit FC6 machine;
hence, this might well be a memory and/or burg problem,
especially on 32-bit machines.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1730547&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-1981735 ] PF/alg: some error messages differ from PF/mps

2009-11-17T01:11:18Z

Bugs item #1981735, was opened at 2008-06-02 08:43
Message generated for change (Comment added) made by sjoerd
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1981735&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/tests
Group: Pathfinder "stable"
Status: Closed
Resolution: Fixed
Priority: 2
Private: No
Submitted By: Stefan Manegold (stmane)
Assigned to: Peter Boncz (boncz)
Summary: PF/alg: some error messages differ from PF/mps

Initial Comment:
With the Algebra back-end, some error messages seem to differ from the milprint_summer back-end.

In case this is intended, we should (must) approved the Algebra-specific output
(as I already did with
tests/BugDay_2005-12-19_0.9.3/Tests/tag_name_validation.SF-1221984.stable.err{,.Algebra}
tests/BugDay_2005-12-19_0.9.3/Tests/zero-or-one_merged-union.SF-1221336.stable.err{,.Algebra}
).
Otherwise, we should (must) fix the error messages.

Examples:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_i18n/bad-001.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_XQuery/matches02.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_XQuery/matches01.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_XQuery/attr_err.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_XQuery/div.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_XQuery/cast2.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_W3C_use_cases_XQUF_R/Q4.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_BugsViaSourgeforce/invisible_error_messages.SF-1409122.err.00.html

----------------------------------------------------------------------

>Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-17 10:11

Message:
I now have a patch ready for the problem I identified in my last comment.
It's too late for the release build I'm doing now, so if this patch proofs
to be good, it'll come in the next release.
The error message I now get contains the complete invalid character, not
just the first byte of its UTF-8 encoding.

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-17 09:31

Message:
Actually, I think this does show a (small) problem in the compiler: when
confronted with a multi-byte character that it doesn't expect, it only
prints the first byte. The compiler should treat the document as a
sequence of Unicode code point, not as a sequence of bytes.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-16 23:38

Message:
Thanks.
I approved the Linux-specific error message.

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2009-11-16 23:23

Message:
Stefan, I'd say this seems to have to do with different locales on
different test machines.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-14 22:37

Message:
The issues is not gone, yet.
The question is still, why the same query triggers different error
messages on different systems, and/or whether this matters or whether we
just accept the differences as given.

We seems to get either
"
ERROR = !parse error: syntax error, unexpected invalid_character on line
5, column 19 (next token is `�')
!parse error: XQuery parsing failed
"
or simply
"
ERROR = ! !parse error: XQuery parsing failed
"

cf.
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_i18n/bad-001.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsg103/GNU.64.64.d.1-Fedora10/tests_i18n/bad-001.err.00.html

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2009-11-12 10:37

Message:
after my latest pass over the web, such issues should be gone

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-02-14 19:20

Message:
for this one, the error message appears to be identical (now) with ALG &
MPS, but differs from what it used to be, giving less information than
before:

ALG:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.32.32.d.1-Debian4.0/tests_i18n/bad-001.err.00.html
MPS:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsg103/GNU.32.32.d.1-Debian4.0/tests_i18n/bad-001.err.00.html

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-02-14 18:41

Message:
For the records:

Only one seems to remain:

http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora8/tests_XQuery/div.err.00.html

----------------------------------------------------------------------

Comment By: Lefteris Sidirourgos (lsidir)
Date: 2008-06-12 14:34

Message:
Logged In: YES
user_id=1856546
Originator: NO

Lower the priority since the remaining two different error messages are
not important

----------------------------------------------------------------------

Comment By: Lefteris Sidirourgos (lsidir)
Date: 2008-06-09 15:11

Message:
Logged In: YES
user_id=1856546
Originator: NO

The following tests are now producing the correct error msg:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_XQuery/matches02.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_XQuery/matches01.err.00.html

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2008-06-03 13:43

Message:
Logged In: YES
user_id=572415
Originator: YES

Still differing:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_i18n/bad-001.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_XQuery/matches02.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_XQuery/matches01.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTestsG103/GNU.64.64.d-Fedora8/tests_XQuery/div.err.00.html

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2008-06-03 11:33

Message:
Logged In: YES
user_id=572415
Originator: YES

With the updated error message of invisible_error_messages.SF-
1409122.stable.err, the MPS test now fails

http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.32.d-Fedora8/tests_BugsViaSourgeforce/invisible_error_messages.SF-1409122.err.00.html

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2008-06-02 15:57

Message:
Logged In: YES
user_id=993208
Originator: NO

Fixed attr_err.err, cast2.err, Q4.out, and invisible_error_messages.SF-
1409122.err.

Still remaining:
bad-001.err (could it be that mclient has problems with the characters?)
matches (CATCH is missing in the code generation)
div (check for denominator in milgen is missing)

Re-assigned to Lefteris as he can fix the MIL specific problems (and he
started fn:matches).

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1981735&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2893875 ] PF: 82 tests fail on Auf2009_NFI/Nov2009 but work on Aug2009

2009-11-17T01:08:36Z

Bugs item #2893875, was opened at 2009-11-07 17:19
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2893875&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/tests
Group: Pathfinder "stable"
Status: Open
Resolution: None
Priority: 9
Private: No
Submitted By: Stefan Manegold (stmane)
Assigned to: Stefan Manegold (stmane)
Summary: PF: 82 tests fail on Auf2009_NFI/Nov2009 but work on Aug2009

Initial Comment:
The following 82 tests work(ed) fine with the Aug2009 branch,
but appear to fail with the Aug2009 branch,
and hence most probaly also with the Nov2009 branch
(which is currently even more instable wrt. testing for yet undiscovered reasons):

benchmarks/MBench/qa05
benchmarks/MBench/
benchmarks/XBench/DC/MD/q19
benchmarks/XBench/DC/MD/
modules/pftijah/procs
modules/pftijah/colltest1
modules/pftijah/colltest2
modules/pftijah/
runtime/xrpc/admin/add_del_doc_noxrpc
runtime/xrpc/admin/backup_restore_noxrpc
runtime/xrpc/admin/
tests/BugTracker/treat_as.SF-1586454
tests/BugTracker/insert_multiple.SF-1590580
tests/BugTracker/inserting_multiple_elements.SF-1590583
tests/BugTracker/insert_into_transient.SF-1626208
tests/BugTracker/clear_attrs_on_delete.SF-1612739
tests/BugTracker/clear_attrs_on_delete.SF-1612739-b
tests/BugTracker/clear_attrs_on_delete.SF-1612739-c
tests/BugTracker/corrupt_after_update.SF-1706640
tests/BugTracker/accessing_renamed_inserted_deleted_node.SF-1718622-1718635-1718709
tests/BugTracker/insert_large_doc.SF-1726954
tests/BugTracker/replace-corrupts.SF-1758902
tests/BugTracker/swizzle-bug.SF-1760811
tests/BugTracker/swizzle-bug2.SF-1763495
tests/BugTracker/insert_attribute_gives_ERROR_in_merged_union.SF-1763575
tests/BugTracker/immune_for_updates.SF-1766259
tests/BugTracker/repeated-insert.SF-1814911
tests/BugTracker/insert-new-page.SF-1854215
tests/BugTracker/immune_for_updates.SF-1981852
tests/BugTracker/hang_non_existing_doc.SF-1911209
tests/BugTracker/function_parameter_without_type.SF-1898518
tests/BugTracker/copybug.SF-2642003
tests/BugTracker/update-with-timing.SF-2852928
tests/BugTracker/compilation_error.SF-2860574_1
tests/BugTracker/compilation_error.SF-2860574_2
tests/BugsViaSourgeforce/invisible_error_messages.SF-1409122
tests/BugsViaSourgeforce/
tests/StandOff/update/basic_insert
tests/StandOff/update/basic_test
tests/StandOff/update/
tests/Update/setattr-1
tests/Update/insert-1
tests/Update/insert-2
tests/Update/symm
tests/Update/replacevaluetest
tests/Update/replacevaluetest2
tests/Update/update
tests/Update/illegal-insert
tests/Update/illegal-delete
tests/Update/insert_test_order
tests/Update/insert_test_order_seq
tests/Update/replacenode-corruption.SF-2153245
tests/W3C_use_cases/XQUF/AddressBook/Q1x
tests/W3C_use_cases/XQUF/AddressBook/check_docs
tests/W3C_use_cases/XQUF/AddressBook/
tests/W3C_use_cases/XQUF/Parts/Q1
tests/W3C_use_cases/XQUF/Parts/Q2
tests/W3C_use_cases/XQUF/Parts/Q3a1
tests/W3C_use_cases/XQUF/Parts/Q3a2
tests/W3C_use_cases/XQUF/Parts/Q4a
tests/W3C_use_cases/XQUF/Parts/Q4ax
tests/W3C_use_cases/XQUF/Parts/Q4b
tests/W3C_use_cases/XQUF/Parts/Q4bx
tests/W3C_use_cases/XQUF/Parts/Q4c
tests/W3C_use_cases/XQUF/Parts/Q4cx
tests/W3C_use_cases/XQUF/Parts/Q6
tests/W3C_use_cases/XQUF/Parts/Q6x
tests/W3C_use_cases/XQUF/Parts/
tests/W3C_use_cases/XQUF/R/Q1
tests/W3C_use_cases/XQUF/R/Q2
tests/W3C_use_cases/XQUF/R/Q3
tests/W3C_use_cases/XQUF/R/Q4
tests/W3C_use_cases/XQUF/R/Q4x
tests/W3C_use_cases/XQUF/R/Q5b
tests/W3C_use_cases/XQUF/R/Q6a
tests/W3C_use_cases/XQUF/R/Q6b
tests/W3C_use_cases/XQUF/R/Q7
tests/W3C_use_cases/XQUF/R/Q7x
tests/W3C_use_cases/XQUF/R/Q8
tests/W3C_use_cases/XQUF/R/Q9
tests/XQuery/is-before4
tests/XQuery/union

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-17 10:08

Message:
forgot to finish:

"I already" ...
... approved ALG-specific output of the insert-order test on Lefteris'
advice and closed the bug report:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1964365&group_id=56967

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-17 10:05

Message:
Thanks, Peter!

Your checkins were "too late" for last night's (this morning's) testing,
but local testing shows pathfinder/benchmarks/MBench/Tests/qa05.* and
pathfinder/tests/Update/Tests/update.* working fine, again.

I already

Can be closed once confirmed by nightly testing.
(For now, system-specific problems are also documented in the TestWeb, and
should be handled later; they're not considered crucial for the upcoming
release.)

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2009-11-17 00:32

Message:
both problems are fixed now

I did spot in tests/Update/insert_test_order_seq
another instance of differences accross all platforms.

The test is a little vauge, inserting a sequence of values; I am not sure
what the XQUF semantics says about this. Is the order
implementation-dependent? This this order might also be ok and the test
could be approved.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-16 18:42

Message:
In MBench/qa05 Peter's heuristic rewrite leads to an (unnecessary) cross
product. (For all eNest nodes it applies pf:attribute.) I guess that might
be the problem.

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-16 15:42

Message:
This is a show stopper.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-16 15:28

Message:
Peter,

the two "crucial" problems remaining (which both started on the Nov2009
branch after propagations of NFI changes from the Aug2009_NFI branch to the
Nov2009 branch) are

timeout in
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/benchmarks_MBench/qa05.err.00.html
(since 2009.10.09)

missing updates in
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_Update/update.out.00.html
(since 2009.10.08)
They might actually be missing, because the query parts of the update
queries involving an attribute predicate yield an empty result.
The missing updates are

for $elem in $testdoc/document/element[@attribute = 50]
return do rename exactly-one($elem) into "Element"

for $elem in $testdoc/document/element[@attribute = 10]
return (do insert <a/> as first into exactly-one($elem) treat as
element(),
do insert as first into exactly-one($elem) treat as element())

for $elem in $testdoc/document/element[@attribute = 20]
return (do insert <a/> as last into exactly-one($elem) treat as
element(),
do insert as last into exactly-one($elem) treat as element())

for $elem in $testdoc/document/element[@attribute = 30]
return (do insert <a/> before exactly-one($elem),
do insert before exactly-one($elem))

for $elem in $testdoc/document/element[@attribute = 40]
return (do insert <a/> after exactly-one($elem),
do insert after exactly-one($elem))

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-15 13:57

Message:
below the details what I did so far:

> - tests/BugTracker/empty_file.SF-2017862: SunOS specific version only!!
approved.
> - tests/BugTracker/XML_document_cache_broken.SF-1414720
approved.
> - tests/W3C_use_cases/XQ/NS/* (or remove tests!)
disabled.
> - tests/XQuery/{div.is-before4,union}
div:
approved MIL error messages iso. XQuery error message

is-before4, union:
nothing, as I'm not sure, whether the new output is correct.
Since the NFI change were propagated to the Nov2009 branch on Oct 7,
these tests show again the output as it was before
===================================================================
2009/05/20 - tsheyar:
pathfinder/tests/XQuery/Tests/is-before4.stable.out,1.5.30.1
pathfinder/tests/XQuery/Tests/union.stable.out,1.6.12.1
-- Make the error message for missing documents more informative:
Include the document name in the error message.

-- Adjust new error messages in the testweb.

-- Adjust stable output (for tests with a modified inter-document order).

-- Fix/Extend a cross product rewrite (for unreferenced inputs).

-- Simplify the 'missing document' error message in the algebra
for constant document names (which is the default).
===================================================================
cf.,
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_XQuery/is-before4.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_XQuery/union.out.00.html

Peter and/or Jan R., could you please comment?

> -
tests/BugDay_2005-12-19_0.9.3/attribute_after_non-attribute_content.SF-1351516
disabled.
> - tests/BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372: windows
only
minor --- either approve or change access rights on Windows machine:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.32.d.1-Windows6.0/tests_BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.64.d.1-Windows6.0/tests_BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372.out.00.html
> - tests/clients/php_monetdb
fixed.
> - runtime/procs/approve
approved.
> - modules/pftijah/{load,sigs,procs}
approved.
> -
tests/BugsViaSourgeforce/{invisible_error_messages.SF-1409122,ID.1766173}
approved.

> (2) remove or disable the following tests, which seem incorrect or
outdated:
> - tests/BugTracker/xs_untypedAtomic.SF-1509806.mps
disabled.
> - tests/BugTracker/Zombie_document.SF-2009556
Nothing yet; we get a timeout:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/Zombie_document.SF-2009556.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/Zombie_document.SF-2009556.err.00.html
> - tests/BugTracker/function_parameter_without_type.SF-1898518
approved.
> - tests/BugTracker/fn-root_fn-id_on_attribute_nodes.SF-1
disabled.
> - tests/BugTracker/975028,non-existing_collection.SF-1991726
approved.
> - tests/BugTracker/port_busy.SF-1809586
Nothing, yet; output might actually be OK / as expected:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/port_busy.SF-1809586.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/port_busy.SF-1809586.err.00.html
> - tests/BugTracker/collection_management
Nothing, yet; need to understand what's (supposed to be) going on:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/collection_management.SF-1726599.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/collection_management.SF-1726599.err.00.html
> - tests/BugDay_2005-12-19_0.9.3/multiple_servers_2.SF-1385152
fixed.
> - runtime/smack
Nothing, yet; smack00 test works fine for MIL, MAL, SQL, but never worked
(timout: hangs "forever") for XQuery
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime/smack00.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime/smack00.err.00.html
> - benchmarks/mbench/{qa02,qa06} are now only run for Darwin9.8.0
G.32.32.d.1
> -- should also set NOT_ALGEBRA for this platform
these are (and have been) skipped with Algebra on all platforms.

> (3) ask Jennie to look at the following tests in runtime/xrpc/admin/:
> - add_del_doc_norpc: approve?
> - backup_restore_xrpc: empty, check this pls??
Since 12.11., these now fail with

ERROR = !module import: error loading module from
http://$HOST:47966/admin/admin.xq

http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime_xrpc_admin/add_del_doc_noxrpc.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime_xrpc_admin/backup_restore_noxrpc.err.00.html

> (4) ask Jan Flokstra to look at the following tests in
modules/pftijah/:
> - coltest1 hang
> - test_lms-or: order
> - test_lms_rmoverlap, test_select_start*: empty
fixed by Sjoerd's backport of Hennigs fixes from the head to the Nov2009
branch

> (5) adapt or disable these test scripts to work on windows:
> - tests/BugDay_2005-12-19_0.9.3/file_locked_after_shredding.SF-1238352
> (windows6 only) -
> - tests/BugDay_2005-12-19_0.9.3/shredding_on_the_fly.SF-1377006: idem
> - tests/BugTracker/predicate_selects_too_few_nodes.SF-1636588 - windows
fails
> to parse, because of Mtest substitution failure (for $v becomes for 4)
fixed by Sjoerd.

> (6) ignore output of tests (possibly for certain platforms only), as
these are
> system limits related (too deep recursion):
> - tests/BugTracker/crash_on_concatenated_query.SF-1730547 - for all
systems
> - tests/BugTracker/server-side_compilation_crash.SF-1607210 - for all
systems
> - tests/BugsViaSourgeforce/ID.1015172{a,b,c}: timeout on windows5
(windows6
> works)
> - tests/XQuery/step_attr_nametest: Darwin9.8.0 G.32.32.d too deep
recursion,
> - tests/XQuery/typeswitch3 - too deep recursion on various systems..
> - tests/BugDay_2005-11-09_0.9.3/xquery_crashtest.SF-1207048: pf
recursion
> check not triggered on Fedora10 G.32.32.d.1
to be done ...

didn't get further than here, so far ...

Stefan

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2009-11-11 11:45

Message:
Thanks. I looked into this and fixed a problem with XQuery updates that
indeed was present. This will reduce the amount of problems, removing
(almost) all of those tests that are XQuery updates from the list.

The update bug was caused by a later modification that succeeded my own
check of the testweb that I performed before checking in the code. At that
time, already, I found it very hard to work with the test web. Note, this
is not a complaint to anyone in particular. However a matter of fact.
Simply too many tests fail, such that one is forced to spend an hour just
to compare the state of the testweb before and after a check-in, to detect
which of the tests that fail were actually already failing before the
check-in. Let alone analyze what is happening.

my suggestons are as follows:

(1) approve output:
- tests/BugTracker/empty_file.SF-2017862: SunOS specific version only!!
- tests/BugTracker/XML_document_cache_broken.SF-1414720
- tests/W3C_use_cases/XQ/NS/* (or remove tests!)
- tests/XQuery/{div.is-before4,union}
-
tests/BugDay_2005-12-19_0.9.3/attribute_after_non-attribute_content.SF-1351516
- tests/BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372: windows
only
- tests/clients/php_monetdb
- runtime/procs/approve
- modules/pftijah/{load,sigs,procs}
-
tests/BugsViaSourgeforce/{invisible_error_messages.SF-1409122,ID.1766173}

(2) remove or disable the following tests, which seem incorrect or
outdated:
- tests/BugTracker/xs_untypedAtomic.SF-1509806.mps
- tests/BugTracker/Zombie_document.SF-2009556
- tests/BugTracker/function_parameter_without_type.SF-1898518
- tests/BugTracker/fn-root_fn-id_on_attribute_nodes.SF-1
- tests/BugTracker/975028,non-existing_collection.SF-1991726
- tests/BugTracker/port_busy.SF-1809586
- tests/BugTracker/collection_management
- tests/BugDay_2005-12-19_0.9.3/multiple_servers_2.SF-1385152
- runtime/smack
- benchmarks/mbench/{qa02,qa06} are now only run for Darwin9.8.0
G.32.32.d.1 -- should also set NOT_ALGEBRA for this platform

(3) ask Jennie to look at the following tests in runtime/xrpc/admin/:
- add_del_doc_norpc: approve?
- backup_restore_xrpc: empty, check this pls??

(4) ask Jan Flokstra to look at the following tests in modules/pftijah/:
- coltest1 hang
- test_lms-or: order
- test_lms_rmoverlap, test_select_start*: empty

(5) adapt or disable these test scripts to work on windows:
- tests/BugDay_2005-12-19_0.9.3/file_locked_after_shredding.SF-1238352
(windows6 only) -
- tests/BugDay_2005-12-19_0.9.3/shredding_on_the_fly.SF-1377006: idem
- tests/BugTracker/predicate_selects_too_few_nodes.SF-1636588 - windows
fails to parse, because of Mtest substitution failure (for $v becomes for
4)

(6) ignore output of tests (possibly for certain platforms only), as these
are system limits related (too deep recursion):
- tests/BugTracker/crash_on_concatenated_query.SF-1730547 - for all
systems
- tests/BugTracker/server-side_compilation_crash.SF-1607210 - for all
systems
- tests/BugsViaSourgeforce/ID.1015172{a,b,c}: timeout on windows5
(windows6 works)
- tests/XQuery/step_attr_nametest: Darwin9.8.0 G.32.32.d too deep
recursion,
- tests/XQuery/typeswitch3 - too deep recursion on various systems..
- tests/BugDay_2005-11-09_0.9.3/xquery_crashtest.SF-1207048: pf recursion
check not triggered on Fedora10 G.32.32.d.1

what will be left, then
=======================

The only problems that will be left after following the above are in fact
problems that occur only on a single system.
Rather than indicating a platform deficiency, however, it is still very
likely that these differences indicate the
precense of a bug, that is only triggered under certain circumstances that
happen to hold on that platform.

so these are interesting to look at; all of them were already present in
the last Stable

Fedora10.G.64.64.d.0 - merge_union2: non dense head
- tests/XQuery/merge_union2
Fedora10 G.64.64.d.0 - crash
- tests/XQuery/step_attribute8,

Fedora10 I.64.64.d.1: crash
- tests/XQuery/orderby6

Fedora10 G.64.64.s.1 - crash
- tests/XQuery/step_descendant-or-self3

Fedora10.G.32.32.d.1 -- segmentation fault
- xbench/tc/md/q10
- tests/W3C_use_cases/XQ/XMP/Q07

windows - !ERROR: couldn't read name (1000000152_qn_uri) 1
- tests/BugTracker/32_docs.SF-1730617:

sunos5 g32.32.d.1 - segmentation fault
- mbench/{qs28,31}
- tests/XQuery/{step_duplicates,transient_upward_steps}
- benchmarks/XPathMark/{Q27,28}
- benchmarks/XPath/q13
- benchmarks/XMark/q04
- tests/XQuery/step_attr_nametest (also on Darwin9.8.0 G.32.32.d.1)
sunos5 g32.32.d.1 - 4 times inserted nil due to errors at tuples 0, 1, 2,
3
- tests/BugTracker/child-steps-and-replace.SF-2716723:

Debian4.0 G.32.32.d.1 - !fatal error: columns referenced in trace message
operator not found
- tests/BugTracker/fn-trace.SF-2805513 (also Gentoo2.0.1 G.64.64.d.1)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-08 18:13

Message:
The same test that fail with Aug2009_NFI, while working with Aug2009,
indeed also fail with Nov2009.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-08 16:50

Message:
For completeness:

I checked this on my 64-bit Fedora 10 Linux desktop, using a "testing"
build of MonetDB, i.e., configured with --disbale-debug --enable-optimize
--enable-assert .

I ran Mtest.py (more precisely make check, respectively RunMtest in the
build directory) seperately for the Aug2009 and the Aug2009_NFI brach
(builds) of pathfinder, collected the console output in files, and run diff
over those (as well as over the times.sql files that Mtest.py creates in
its <TSTTRGBASE> directory.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2893875&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

[ monetdb-Bugs-2893875 ] PF: 82 tests fail on Auf2009_NFI/Nov2009 but work on Aug2009

2009-11-17T01:05:24Z

Bugs item #2893875, was opened at 2009-11-07 17:19
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2893875&group_id=56967

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/tests
Group: Pathfinder "stable"
Status: Open
Resolution: None
Priority: 9
Private: No
Submitted By: Stefan Manegold (stmane)
Assigned to: Stefan Manegold (stmane)
Summary: PF: 82 tests fail on Auf2009_NFI/Nov2009 but work on Aug2009

Initial Comment:
The following 82 tests work(ed) fine with the Aug2009 branch,
but appear to fail with the Aug2009 branch,
and hence most probaly also with the Nov2009 branch
(which is currently even more instable wrt. testing for yet undiscovered reasons):

benchmarks/MBench/qa05
benchmarks/MBench/
benchmarks/XBench/DC/MD/q19
benchmarks/XBench/DC/MD/
modules/pftijah/procs
modules/pftijah/colltest1
modules/pftijah/colltest2
modules/pftijah/
runtime/xrpc/admin/add_del_doc_noxrpc
runtime/xrpc/admin/backup_restore_noxrpc
runtime/xrpc/admin/
tests/BugTracker/treat_as.SF-1586454
tests/BugTracker/insert_multiple.SF-1590580
tests/BugTracker/inserting_multiple_elements.SF-1590583
tests/BugTracker/insert_into_transient.SF-1626208
tests/BugTracker/clear_attrs_on_delete.SF-1612739
tests/BugTracker/clear_attrs_on_delete.SF-1612739-b
tests/BugTracker/clear_attrs_on_delete.SF-1612739-c
tests/BugTracker/corrupt_after_update.SF-1706640
tests/BugTracker/accessing_renamed_inserted_deleted_node.SF-1718622-1718635-1718709
tests/BugTracker/insert_large_doc.SF-1726954
tests/BugTracker/replace-corrupts.SF-1758902
tests/BugTracker/swizzle-bug.SF-1760811
tests/BugTracker/swizzle-bug2.SF-1763495
tests/BugTracker/insert_attribute_gives_ERROR_in_merged_union.SF-1763575
tests/BugTracker/immune_for_updates.SF-1766259
tests/BugTracker/repeated-insert.SF-1814911
tests/BugTracker/insert-new-page.SF-1854215
tests/BugTracker/immune_for_updates.SF-1981852
tests/BugTracker/hang_non_existing_doc.SF-1911209
tests/BugTracker/function_parameter_without_type.SF-1898518
tests/BugTracker/copybug.SF-2642003
tests/BugTracker/update-with-timing.SF-2852928
tests/BugTracker/compilation_error.SF-2860574_1
tests/BugTracker/compilation_error.SF-2860574_2
tests/BugsViaSourgeforce/invisible_error_messages.SF-1409122
tests/BugsViaSourgeforce/
tests/StandOff/update/basic_insert
tests/StandOff/update/basic_test
tests/StandOff/update/
tests/Update/setattr-1
tests/Update/insert-1
tests/Update/insert-2
tests/Update/symm
tests/Update/replacevaluetest
tests/Update/replacevaluetest2
tests/Update/update
tests/Update/illegal-insert
tests/Update/illegal-delete
tests/Update/insert_test_order
tests/Update/insert_test_order_seq
tests/Update/replacenode-corruption.SF-2153245
tests/W3C_use_cases/XQUF/AddressBook/Q1x
tests/W3C_use_cases/XQUF/AddressBook/check_docs
tests/W3C_use_cases/XQUF/AddressBook/
tests/W3C_use_cases/XQUF/Parts/Q1
tests/W3C_use_cases/XQUF/Parts/Q2
tests/W3C_use_cases/XQUF/Parts/Q3a1
tests/W3C_use_cases/XQUF/Parts/Q3a2
tests/W3C_use_cases/XQUF/Parts/Q4a
tests/W3C_use_cases/XQUF/Parts/Q4ax
tests/W3C_use_cases/XQUF/Parts/Q4b
tests/W3C_use_cases/XQUF/Parts/Q4bx
tests/W3C_use_cases/XQUF/Parts/Q4c
tests/W3C_use_cases/XQUF/Parts/Q4cx
tests/W3C_use_cases/XQUF/Parts/Q6
tests/W3C_use_cases/XQUF/Parts/Q6x
tests/W3C_use_cases/XQUF/Parts/
tests/W3C_use_cases/XQUF/R/Q1
tests/W3C_use_cases/XQUF/R/Q2
tests/W3C_use_cases/XQUF/R/Q3
tests/W3C_use_cases/XQUF/R/Q4
tests/W3C_use_cases/XQUF/R/Q4x
tests/W3C_use_cases/XQUF/R/Q5b
tests/W3C_use_cases/XQUF/R/Q6a
tests/W3C_use_cases/XQUF/R/Q6b
tests/W3C_use_cases/XQUF/R/Q7
tests/W3C_use_cases/XQUF/R/Q7x
tests/W3C_use_cases/XQUF/R/Q8
tests/W3C_use_cases/XQUF/R/Q9
tests/XQuery/is-before4
tests/XQuery/union

----------------------------------------------------------------------

>Comment By: Stefan Manegold (stmane)
Date: 2009-11-17 10:05

Message:
Thanks, Peter!

Your checkins were "too late" for last night's (this morning's) testing,
but local testing shows pathfinder/benchmarks/MBench/Tests/qa05.* and
pathfinder/tests/Update/Tests/update.* working fine, again.

I already

Can be closed once confirmed by nightly testing.
(For now, system-specific problems are also documented in the TestWeb, and
should be handled later; they're not considered crucial for the upcoming
release.)

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2009-11-17 00:32

Message:
both problems are fixed now

I did spot in tests/Update/insert_test_order_seq
another instance of differences accross all platforms.

The test is a little vauge, inserting a sequence of values; I am not sure
what the XQUF semantics says about this. Is the order
implementation-dependent? This this order might also be ok and the test
could be approved.

----------------------------------------------------------------------

Comment By: Jan Rittinger (tsheyar)
Date: 2009-11-16 18:42

Message:
In MBench/qa05 Peter's heuristic rewrite leads to an (unnecessary) cross
product. (For all eNest nodes it applies pf:attribute.) I guess that might
be the problem.

----------------------------------------------------------------------

Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-11-16 15:42

Message:
This is a show stopper.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-16 15:28

Message:
Peter,

the two "crucial" problems remaining (which both started on the Nov2009
branch after propagations of NFI changes from the Aug2009_NFI branch to the
Nov2009 branch) are

timeout in
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/benchmarks_MBench/qa05.err.00.html
(since 2009.10.09)

missing updates in
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_Update/update.out.00.html
(since 2009.10.08)
They might actually be missing, because the query parts of the update
queries involving an attribute predicate yield an empty result.
The missing updates are

for $elem in $testdoc/document/element[@attribute = 50]
return do rename exactly-one($elem) into "Element"

for $elem in $testdoc/document/element[@attribute = 10]
return (do insert <a/> as first into exactly-one($elem) treat as
element(),
do insert as first into exactly-one($elem) treat as element())

for $elem in $testdoc/document/element[@attribute = 20]
return (do insert <a/> as last into exactly-one($elem) treat as
element(),
do insert as last into exactly-one($elem) treat as element())

for $elem in $testdoc/document/element[@attribute = 30]
return (do insert <a/> before exactly-one($elem),
do insert before exactly-one($elem))

for $elem in $testdoc/document/element[@attribute = 40]
return (do insert <a/> after exactly-one($elem),
do insert after exactly-one($elem))

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-15 13:57

Message:
below the details what I did so far:

> - tests/BugTracker/empty_file.SF-2017862: SunOS specific version only!!
approved.
> - tests/BugTracker/XML_document_cache_broken.SF-1414720
approved.
> - tests/W3C_use_cases/XQ/NS/* (or remove tests!)
disabled.
> - tests/XQuery/{div.is-before4,union}
div:
approved MIL error messages iso. XQuery error message

is-before4, union:
nothing, as I'm not sure, whether the new output is correct.
Since the NFI change were propagated to the Nov2009 branch on Oct 7,
these tests show again the output as it was before
===================================================================
2009/05/20 - tsheyar:
pathfinder/tests/XQuery/Tests/is-before4.stable.out,1.5.30.1
pathfinder/tests/XQuery/Tests/union.stable.out,1.6.12.1
-- Make the error message for missing documents more informative:
Include the document name in the error message.

-- Adjust new error messages in the testweb.

-- Adjust stable output (for tests with a modified inter-document order).

-- Fix/Extend a cross product rewrite (for unreferenced inputs).

-- Simplify the 'missing document' error message in the algebra
for constant document names (which is the default).
===================================================================
cf.,
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_XQuery/is-before4.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_XQuery/union.out.00.html

Peter and/or Jan R., could you please comment?

> -
tests/BugDay_2005-12-19_0.9.3/attribute_after_non-attribute_content.SF-1351516
disabled.
> - tests/BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372: windows
only
minor --- either approve or change access rights on Windows machine:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.32.d.1-Windows6.0/tests_BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.64.d.1-Windows6.0/tests_BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372.out.00.html
> - tests/clients/php_monetdb
fixed.
> - runtime/procs/approve
approved.
> - modules/pftijah/{load,sigs,procs}
approved.
> -
tests/BugsViaSourgeforce/{invisible_error_messages.SF-1409122,ID.1766173}
approved.

> (2) remove or disable the following tests, which seem incorrect or
outdated:
> - tests/BugTracker/xs_untypedAtomic.SF-1509806.mps
disabled.
> - tests/BugTracker/Zombie_document.SF-2009556
Nothing yet; we get a timeout:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/Zombie_document.SF-2009556.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/Zombie_document.SF-2009556.err.00.html
> - tests/BugTracker/function_parameter_without_type.SF-1898518
approved.
> - tests/BugTracker/fn-root_fn-id_on_attribute_nodes.SF-1
disabled.
> - tests/BugTracker/975028,non-existing_collection.SF-1991726
approved.
> - tests/BugTracker/port_busy.SF-1809586
Nothing, yet; output might actually be OK / as expected:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/port_busy.SF-1809586.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/port_busy.SF-1809586.err.00.html
> - tests/BugTracker/collection_management
Nothing, yet; need to understand what's (supposed to be) going on:
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/collection_management.SF-1726599.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/collection_management.SF-1726599.err.00.html
> - tests/BugDay_2005-12-19_0.9.3/multiple_servers_2.SF-1385152
fixed.
> - runtime/smack
Nothing, yet; smack00 test works fine for MIL, MAL, SQL, but never worked
(timout: hangs "forever") for XQuery
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime/smack00.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime/smack00.err.00.html
> - benchmarks/mbench/{qa02,qa06} are now only run for Darwin9.8.0
G.32.32.d.1
> -- should also set NOT_ALGEBRA for this platform
these are (and have been) skipped with Algebra on all platforms.

> (3) ask Jennie to look at the following tests in runtime/xrpc/admin/:
> - add_del_doc_norpc: approve?
> - backup_restore_xrpc: empty, check this pls??
Since 12.11., these now fail with

ERROR = !module import: error loading module from
http://$HOST:47966/admin/admin.xq

http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime_xrpc_admin/add_del_doc_noxrpc.err.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/runtime_xrpc_admin/backup_restore_noxrpc.err.00.html

> (4) ask Jan Flokstra to look at the following tests in
modules/pftijah/:
> - coltest1 hang
> - test_lms-or: order
> - test_lms_rmoverlap, test_select_start*: empty
fixed by Sjoerd's backport of Hennigs fixes from the head to the Nov2009
branch

> (5) adapt or disable these test scripts to work on windows:
> - tests/BugDay_2005-12-19_0.9.3/file_locked_after_shredding.SF-1238352
> (windows6 only) -
> - tests/BugDay_2005-12-19_0.9.3/shredding_on_the_fly.SF-1377006: idem
> - tests/BugTracker/predicate_selects_too_few_nodes.SF-1636588 - windows
fails
> to parse, because of Mtest substitution failure (for $v becomes for 4)
fixed by Sjoerd.

> (6) ignore output of tests (possibly for certain platforms only), as
these are
> system limits related (too deep recursion):
> - tests/BugTracker/crash_on_concatenated_query.SF-1730547 - for all
systems
> - tests/BugTracker/server-side_compilation_crash.SF-1607210 - for all
systems
> - tests/BugsViaSourgeforce/ID.1015172{a,b,c}: timeout on windows5
(windows6
> works)
> - tests/XQuery/step_attr_nametest: Darwin9.8.0 G.32.32.d too deep
recursion,
> - tests/XQuery/typeswitch3 - too deep recursion on various systems..
> - tests/BugDay_2005-11-09_0.9.3/xquery_crashtest.SF-1207048: pf
recursion
> check not triggered on Fedora10 G.32.32.d.1
to be done ...

didn't get further than here, so far ...

Stefan

----------------------------------------------------------------------

Comment By: Peter Boncz (boncz)
Date: 2009-11-11 11:45

Message:
Thanks. I looked into this and fixed a problem with XQuery updates that
indeed was present. This will reduce the amount of problems, removing
(almost) all of those tests that are XQuery updates from the list.

The update bug was caused by a later modification that succeeded my own
check of the testweb that I performed before checking in the code. At that
time, already, I found it very hard to work with the test web. Note, this
is not a complaint to anyone in particular. However a matter of fact.
Simply too many tests fail, such that one is forced to spend an hour just
to compare the state of the testweb before and after a check-in, to detect
which of the tests that fail were actually already failing before the
check-in. Let alone analyze what is happening.

my suggestons are as follows:

(1) approve output:
- tests/BugTracker/empty_file.SF-2017862: SunOS specific version only!!
- tests/BugTracker/XML_document_cache_broken.SF-1414720
- tests/W3C_use_cases/XQ/NS/* (or remove tests!)
- tests/XQuery/{div.is-before4,union}
-
tests/BugDay_2005-12-19_0.9.3/attribute_after_non-attribute_content.SF-1351516
- tests/BugDay_2005-12-19_0.9.3/shred_doc_with_space.SF-1211372: windows
only
- tests/clients/php_monetdb
- runtime/procs/approve
- modules/pftijah/{load,sigs,procs}
-
tests/BugsViaSourgeforce/{invisible_error_messages.SF-1409122,ID.1766173}

(2) remove or disable the following tests, which seem incorrect or
outdated:
- tests/BugTracker/xs_untypedAtomic.SF-1509806.mps
- tests/BugTracker/Zombie_document.SF-2009556
- tests/BugTracker/function_parameter_without_type.SF-1898518
- tests/BugTracker/fn-root_fn-id_on_attribute_nodes.SF-1
- tests/BugTracker/975028,non-existing_collection.SF-1991726
- tests/BugTracker/port_busy.SF-1809586
- tests/BugTracker/collection_management
- tests/BugDay_2005-12-19_0.9.3/multiple_servers_2.SF-1385152
- runtime/smack
- benchmarks/mbench/{qa02,qa06} are now only run for Darwin9.8.0
G.32.32.d.1 -- should also set NOT_ALGEBRA for this platform

(3) ask Jennie to look at the following tests in runtime/xrpc/admin/:
- add_del_doc_norpc: approve?
- backup_restore_xrpc: empty, check this pls??

(4) ask Jan Flokstra to look at the following tests in modules/pftijah/:
- coltest1 hang
- test_lms-or: order
- test_lms_rmoverlap, test_select_start*: empty

(5) adapt or disable these test scripts to work on windows:
- tests/BugDay_2005-12-19_0.9.3/file_locked_after_shredding.SF-1238352
(windows6 only) -
- tests/BugDay_2005-12-19_0.9.3/shredding_on_the_fly.SF-1377006: idem
- tests/BugTracker/predicate_selects_too_few_nodes.SF-1636588 - windows
fails to parse, because of Mtest substitution failure (for $v becomes for
4)

(6) ignore output of tests (possibly for certain platforms only), as these
are system limits related (too deep recursion):
- tests/BugTracker/crash_on_concatenated_query.SF-1730547 - for all
systems
- tests/BugTracker/server-side_compilation_crash.SF-1607210 - for all
systems
- tests/BugsViaSourgeforce/ID.1015172{a,b,c}: timeout on windows5
(windows6 works)
- tests/XQuery/step_attr_nametest: Darwin9.8.0 G.32.32.d too deep
recursion,
- tests/XQuery/typeswitch3 - too deep recursion on various systems..
- tests/BugDay_2005-11-09_0.9.3/xquery_crashtest.SF-1207048: pf recursion
check not triggered on Fedora10 G.32.32.d.1

what will be left, then
=======================

The only problems that will be left after following the above are in fact
problems that occur only on a single system.
Rather than indicating a platform deficiency, however, it is still very
likely that these differences indicate the
precense of a bug, that is only triggered under certain circumstances that
happen to hold on that platform.

so these are interesting to look at; all of them were already present in
the last Stable

Fedora10.G.64.64.d.0 - merge_union2: non dense head
- tests/XQuery/merge_union2
Fedora10 G.64.64.d.0 - crash
- tests/XQuery/step_attribute8,

Fedora10 I.64.64.d.1: crash
- tests/XQuery/orderby6

Fedora10 G.64.64.s.1 - crash
- tests/XQuery/step_descendant-or-self3

Fedora10.G.32.32.d.1 -- segmentation fault
- xbench/tc/md/q10
- tests/W3C_use_cases/XQ/XMP/Q07

windows - !ERROR: couldn't read name (1000000152_qn_uri) 1
- tests/BugTracker/32_docs.SF-1730617:

sunos5 g32.32.d.1 - segmentation fault
- mbench/{qs28,31}
- tests/XQuery/{step_duplicates,transient_upward_steps}
- benchmarks/XPathMark/{Q27,28}
- benchmarks/XPath/q13
- benchmarks/XMark/q04
- tests/XQuery/step_attr_nametest (also on Darwin9.8.0 G.32.32.d.1)
sunos5 g32.32.d.1 - 4 times inserted nil due to errors at tuples 0, 1, 2,
3
- tests/BugTracker/child-steps-and-replace.SF-2716723:

Debian4.0 G.32.32.d.1 - !fatal error: columns referenced in trace message
operator not found
- tests/BugTracker/fn-trace.SF-2805513 (also Gentoo2.0.1 G.64.64.d.1)

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-08 18:13

Message:
The same test that fail with Aug2009_NFI, while working with Aug2009,
indeed also fail with Nov2009.

----------------------------------------------------------------------

Comment By: Stefan Manegold (stmane)
Date: 2009-11-08 16:50

Message:
For completeness:

I checked this on my 64-bit Fedora 10 Linux desktop, using a "testing"
build of MonetDB, i.e., configured with --disbale-debug --enable-optimize
--enable-assert .

I ran Mtest.py (more precisely make check, respectively RunMtest in the
build directory) seperately for the Aug2009 and the Aug2009_NFI brach
(builds) of pathfinder, collected the console output in files, and run diff
over those (as well as over the times.sql files that Mtest.py creates in
its <TSTTRGBASE> directory.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2893875&group_id=56967

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
Monetdb-bugs@...
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs