Nabble - Derby

[jira] Commented: (DERBY-4272) SQL Authorization Support for dblook

2009-12-15T19:21:18Z

[ https://issues.apache.org/jira/browse/DERBY-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791157#action_12791157 ]

Dag H. Wanvik commented on DERBY-4272:
--------------------------------------

Thank, Hiranya!
Hopefully we can by joined efforts make those missing parts of the test code ready in time for 10.6. I'll have a look asap.

> SQL Authorization Support for dblook
> ------------------------------------
>
> Key: DERBY-4272
> URL: https://issues.apache.org/jira/browse/DERBY-4272
> Project: Derby
> Issue Type: Improvement
> Components: Tools
> Environment: Any
> Reporter: Hiranya Jayathilaka
> Assignee: Hiranya Jayathilaka
> Attachments: DERBY-4272-changes-u1.txt, DERBY-4272-changes-u2.txt, DERBY-4272-changes-u3.txt, DERBY-4272-changes-u4.txt, DERBY-4272-changes-u5.txt, DERBY-4272-changes-u6.txt, DERBY-4272-u1.patch, DERBY-4272-u2.patch, DERBY-4272-u3.patch, DERBY-4272-u4.patch, DERBY-4272-u5.patch, DERBY-4272-u6.patch, dhw-sample-1.sql, new.sql, old.sql
>
>
> Currently dblook suffers from two major shortcomings.
> 1. dblook doesn't take the object dependencies into consideration when generating DDL scripts
> 2. dblook doesn't have any support for SQL authorization
> I intend to fix these two issues and improve dblook so that the DDL scripts generated by dblook can be executed without errors under all conditions.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Commented: (DERBY-4471) Left outer join reassociation rewrite gives wrong result

2009-12-15T14:45:18Z

[ https://issues.apache.org/jira/browse/DERBY-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791002#action_12791002 ]

Dag H. Wanvik commented on DERBY-4471:
--------------------------------------

Seen all the way back to 10.0; not a regression.

> Left outer join reassociation rewrite gives wrong result
> --------------------------------------------------------
>
> Key: DERBY-4471
> URL: https://issues.apache.org/jira/browse/DERBY-4471
> Project: Derby
> Issue Type: Bug
> Components: SQL
> Affects Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.2.1, 10.1.3.1, 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0
> Reporter: Dag H. Wanvik
> Attachments: derby-4471-junit-repro.diff
>
>
> The following script and output shows the problem:
> > create table r(c1 char(1));
> > create table s(c1 char(1), c2 char(1));
> > create table t(c1 char(1));
> > insert into r values 'a';
> > insert into s values ('b', default);
> > insert into t values ('c');
> > select * from s left outer join t on s.c2=t.c1 or s.c2 is null;
> C1 |C2 |C1
> --------------
> b |NULL|c
> > select * from r left outer join s on r.c1=s.c1;
> C1 |C1 |C2
> --------------
> a |NULL|NULL
> > select * from (r left outer join s on r.c1=s.c1) left outer join t on s.c2=t.c1 or s.c2 is null;
> C1 |C1 |C2 |C1
> -------------------
> a |NULL|NULL|c
> > select * from r left outer join (s left outer join t on s.c2=t.c1 or s.c2 is null) on r.c1=s.c1;
> C1 |C1 |C2 |C1
> -------------------
> a |NULL|NULL|c
> The last result is wrong. The correct answer should be:
> C1 |C1 |C2 |C1
> -------------------
> a |NULL|NULL|NULL
> since in the last form, the left table r has the value 'a', which does
> not match any row in result of the compound inner given the join
> predicate ("r.c1=s.c1"), so all nulls should be appended to the 'a'
> from the outer table r.
> This happens because internally the last form is rewritten to the
> second but the last form (left-deep), but this rewrite is not
> justified here unless the join predicate on s rejects null, which the
> present one explicitly does not ("or s.c2 is null"). Cf. for example
> [1], page 52, which describes this transform and its prerequisite
> condition as indentity #7.
> [1] Galindo-Legaria, C. & Rosenthal, A.: "Outerjoin simplification and
> reordering for query optimization", ACM Transactions on Database
> Systems, Vol 22, No 1, March 1997.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Updated: (DERBY-4471) Left outer join reassociation rewrite gives wrong result

2009-12-15T14:43:18Z

[ https://issues.apache.org/jira/browse/DERBY-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dag H. Wanvik updated DERBY-4471:
---------------------------------

Affects Version/s: 10.0.2.0
10.0.2.1
10.1.1.0
10.1.2.1
10.1.3.1
10.2.1.6
10.2.2.0
10.3.1.4
10.3.2.1
10.3.3.0
10.4.1.3
10.4.2.0
10.5.1.1
10.5.2.0
10.5.3.0

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Updated: (DERBY-4471) Left outer join reassociation rewrite gives wrong result

2009-12-15T14:37:18Z

[ https://issues.apache.org/jira/browse/DERBY-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dag H. Wanvik updated DERBY-4471:
---------------------------------

Attachment: derby-4471-junit-repro.diff

Thanks, Knut!
Uploading a patch to JoinTest.java which shows the problem in JUnit form.

> Left outer join reassociation rewrite gives wrong result
> --------------------------------------------------------
>
> Key: DERBY-4471
> URL: https://issues.apache.org/jira/browse/DERBY-4471
> Project: Derby
> Issue Type: Bug
> Components: SQL
> Reporter: Dag H. Wanvik
> Attachments: derby-4471-junit-repro.diff
>
>
> The following script and output shows the problem:
> > create table r(c1 char(1));
> > create table s(c1 char(1), c2 char(1));
> > create table t(c1 char(1));
> > insert into r values 'a';
> > insert into s values ('b', default);
> > insert into t values ('c');
> > select * from s left outer join t on s.c2=t.c1 or s.c2 is null;
> C1 |C2 |C1
> --------------
> b |NULL|c
> > select * from r left outer join s on r.c1=s.c1;
> C1 |C1 |C2
> --------------
> a |NULL|NULL
> > select * from (r left outer join s on r.c1=s.c1) left outer join t on s.c2=t.c1 or s.c2 is null;
> C1 |C1 |C2 |C1
> -------------------
> a |NULL|NULL|c
> > select * from r left outer join (s left outer join t on s.c2=t.c1 or s.c2 is null) on r.c1=s.c1;
> C1 |C1 |C2 |C1
> -------------------
> a |NULL|NULL|c
> The last result is wrong. The correct answer should be:
> C1 |C1 |C2 |C1
> -------------------
> a |NULL|NULL|NULL
> since in the last form, the left table r has the value 'a', which does
> not match any row in result of the compound inner given the join
> predicate ("r.c1=s.c1"), so all nulls should be appended to the 'a'
> from the outer table r.
> This happens because internally the last form is rewritten to the
> second but the last form (left-deep), but this rewrite is not
> justified here unless the join predicate on s rejects null, which the
> present one explicitly does not ("or s.c2 is null"). Cf. for example
> [1], page 52, which describes this transform and its prerequisite
> condition as indentity #7.
> [1] Galindo-Legaria, C. & Rosenthal, A.: "Outerjoin simplification and
> reordering for query optimization", ACM Transactions on Database
> Systems, Vol 22, No 1, March 1997.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Resolved: (DERBY-4405) Transformation to inner join not performed for certain three-way joins

2009-12-15T13:51:18Z

[ https://issues.apache.org/jira/browse/DERBY-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dag H. Wanvik resolved DERBY-4405.
----------------------------------

Resolution: Fixed
Fix Version/s: 10.6.0.0
Issue & fix info: (was: [Patch Available])

> Transformation to inner join not performed for certain three-way joins
> ----------------------------------------------------------------------
>
> Key: DERBY-4405
> URL: https://issues.apache.org/jira/browse/DERBY-4405
> Project: Derby
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 10.6.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Dag H. Wanvik
> Priority: Minor
> Fix For: 10.6.0.0
>
> Attachments: derby-4405-2.diff, derby-4405-2.stat, derby-4405.diff, derby-4405.stat
>
>
> In the CROSS JOIN section in the reference manual (http://db.apache.org/derby/docs/dev/ref/rrefsqljcrossjoin.html) there are three examples that are supposed to be equivalent. However, the performance differs significantly between the different queries.
> The queries use the tours db and look like this:
> (1)
> SELECT * FROM CITIES LEFT OUTER JOIN
> (FLIGHTS CROSS JOIN COUNTRIES)
> ON CITIES.AIRPORT = FLIGHTS.ORIG_AIRPORT
> WHERE COUNTRIES.COUNTRY_ISO_CODE = 'US'
> (2)
> SELECT * FROM CITIES LEFT OUTER JOIN
> FLIGHTS INNER JOIN COUNTRIES ON 1=1
> ON CITIES.AIRPORT = FLIGHTS.ORIG_AIRPORT
> WHERE COUNTRIES.COUNTRY_ISO_CODE = 'US'
> (3)
> SELECT * FROM CITIES LEFT OUTER JOIN
> (SELECT * FROM FLIGHTS, COUNTRIES) S
> ON CITIES.AIRPORT = S.ORIG_AIRPORT
> WHERE S.COUNTRY_ISO_CODE = 'US'
> When executed in ij, (1) and (2) need 6 seconds to complete, whereas (3) completes in 50 ms.
> The query plans for (1) and (2) use nested loop joins and table scans. (3) uses a combination of hash join and nested loop join, and index scans as well as table scans.
> It looks like (3) has been rewritten from a left outer join to an inner join internally. This is fine because all rows that have the right-side columns filled with NULLs will be filtered out by the predicate S.COUNTRY_ISO_CODE='US', so the extra rows generated by the outer join will not be returned.
> This optimization should also be possible for (1) and (2). We should improve the logic so that those joins are transformed too. The transformation happens in HalfOuterJoinNode.transformOuterJoins().

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Commented: (DERBY-4405) Transformation to inner join not performed for certain three-way joins

2009-12-15T13:49:18Z

[ https://issues.apache.org/jira/browse/DERBY-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790975#action_12790975 ]

Dag H. Wanvik commented on DERBY-4405:
--------------------------------------

Regressions passed, committed as svn 891015.

> Transformation to inner join not performed for certain three-way joins
> ----------------------------------------------------------------------
>
> Key: DERBY-4405
> URL: https://issues.apache.org/jira/browse/DERBY-4405
> Project: Derby
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 10.6.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Dag H. Wanvik
> Priority: Minor
> Attachments: derby-4405-2.diff, derby-4405-2.stat, derby-4405.diff, derby-4405.stat
>
>
> In the CROSS JOIN section in the reference manual (http://db.apache.org/derby/docs/dev/ref/rrefsqljcrossjoin.html) there are three examples that are supposed to be equivalent. However, the performance differs significantly between the different queries.
> The queries use the tours db and look like this:
> (1)
> SELECT * FROM CITIES LEFT OUTER JOIN
> (FLIGHTS CROSS JOIN COUNTRIES)
> ON CITIES.AIRPORT = FLIGHTS.ORIG_AIRPORT
> WHERE COUNTRIES.COUNTRY_ISO_CODE = 'US'
> (2)
> SELECT * FROM CITIES LEFT OUTER JOIN
> FLIGHTS INNER JOIN COUNTRIES ON 1=1
> ON CITIES.AIRPORT = FLIGHTS.ORIG_AIRPORT
> WHERE COUNTRIES.COUNTRY_ISO_CODE = 'US'
> (3)
> SELECT * FROM CITIES LEFT OUTER JOIN
> (SELECT * FROM FLIGHTS, COUNTRIES) S
> ON CITIES.AIRPORT = S.ORIG_AIRPORT
> WHERE S.COUNTRY_ISO_CODE = 'US'
> When executed in ij, (1) and (2) need 6 seconds to complete, whereas (3) completes in 50 ms.
> The query plans for (1) and (2) use nested loop joins and table scans. (3) uses a combination of hash join and nested loop join, and index scans as well as table scans.
> It looks like (3) has been rewritten from a left outer join to an inner join internally. This is fine because all rows that have the right-side columns filled with NULLs will be filtered out by the predicate S.COUNTRY_ISO_CODE='US', so the extra rows generated by the outer join will not be returned.
> This optimization should also be possible for (1) and (2). We should improve the logic so that those joins are transformed too. The transformation happens in HalfOuterJoinNode.transformOuterJoins().

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

Re: [jira] Commented: (DERBY-4424) Convert outerjoin.sql into JUnit

2009-12-15T12:57:06Z

"Bryan Pendleton (JIRA)" <jira@...> writes:

> 1) testBug2897(org.apache.derbyTesting.functionTests.tests.lang.OuterJoinTest)java.lang.NullPointerException

Btw, the number here, 2897, refers to pre-JIRA days. Should we make up
some systematic way of referring to such old error number so as not
to confuse them with JIRA numbering? I imagine we don't want to remove
the old number information since I have seen the IBM folks sometimes make
use of the old numbers.. Maybe it's enough that we always refer to
JIRA errors as DERBY-nnnn in comments and some adapted string for
identifiers, e.g. [dD]erbyNNNN.

Dag

From forum: Apache Derby Developers

[jira] Updated: (DERBY-4038) On Z/OS store/access.sql fails with encryptionAES and encryptionDES

2009-12-15T11:21:18Z

[ https://issues.apache.org/jira/browse/DERBY-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Matrigali updated DERBY-4038:
----------------------------------

comments from review of the patch:

> - removed the call to procedure WAIT_FOR_POST_COMMIT as it didn't appear to
> be used anywhere.
sounds good, it is not called so remove all reference to it.
>
> - Was I misinterpreting the use of WAIT_FOR_POST_COMMIT? What would be the pur
pose of it?
It is not called, leftover cruft.

comments on TODO's:
+ // TODO: check on what the following comment from the original test
+ // means. I don't think this test is currently checking whether the
+ // optimizer is picking the index, might be worthwhile adding it.

At this point not sure history of this, I would just remove comment.

+ // as subset of columns, with qualifier not in list
+ //TODO: this comment is not matching the expected results. OK?
+ // should be 5,3,1 and 50,30,10
+ doQuery(st, "select e, c, a from foo where foo.b = 20",
comment looks bad, just fix the comment.

+ // TODO: how does this check that rows are reclaimed or not? How?
looks like all work was moved to the subroutine, so probably should move
comments to the subroutine from the caller. At least the part of the comments
that make sense.

The code just tries to exercise the post commit path, but agree that it does
not specifically verify the reclaim. This is hard to do from a user level
test, in a way that is reproducible across different hardware, jvms. The
values are just hand picked to exercise different paths through the code.
One could try to use space table to verify sizes of the tables, but this
has caused non-reproducible results in cases where timing of background threads
are different.

// TODO: change this call into a call to SYSCS_DIAG.SPACE_TABLE
I agree, it should use the current supported interface.

+ // TODO: original test had 1 pages visited.
+ // But now that it's converted, we get 0 pages, which also
+ // seems reasonable to me. Or not?
This seems like it might be a problem. Can you print out the full query
plan on failure of the assert. It is hard to know what is going on from
just this one piece of info.

The main point of this test is that is should use the index whether there
are zero or 1 rows. We choose to force this because if we use index we get
row level locking, but if we use table scan we are going to get table level
locking - so better concurrency in the edge case.

+ // TODO: in which way does this check qualifiers?
qualifiers are just things like a = 1 and b = 20. When test was written the
code in language and store to evaluate these were changing. The changes
especially had some new logic to handle "and" and "or". So the test just
has some of these cases and tests them by making sure the right results
come back.

> On Z/OS store/access.sql fails with encryptionAES and encryptionDES
> -------------------------------------------------------------------
>
> Key: DERBY-4038
> URL: https://issues.apache.org/jira/browse/DERBY-4038
> Project: Derby
> Issue Type: Bug
> Components: Test
> Affects Versions: 10.3.3.0
> Environment: java version "1.6.0"
> Java(TM) SE Runtime Environment (build pmz6460sr3-20081108_01(SR3))
> IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 z/OS s390x-64 jvmmz6460-20081107_25433 (JIT enabled, AOT enabled)
> J9VM - 20081105_025433_BHdSMr
> JIT - r9_20081031_1330
> GC - 20081027_AB)
> JCL - 20081106_01
> $ pwd
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: access.diff, access.out, DERBY-4038.diff_1
>
>
> On Z/OS access.sql fails for encryptionAES and encryptionDES. The diff is big but I can't quite make out what the problem is. Maybe just a change of query plans.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Updated: (DERBY-4463) JMX test in nightly test suite failed with: JMXTest:clientjava.lang.InterruptedException:

2009-12-15T09:44:18Z

[ https://issues.apache.org/jira/browse/DERBY-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Matrigali updated DERBY-4463:
----------------------------------

doesn't look like latest patch fixed in 889875 fixed the ibm results. Here are results against
a build against 890158:

http://people.apache.org/~myrnavl/derby_test_results/main/windows/testlog/ibm15/890158-suites.All_diff.txt
http://people.apache.org/~myrnavl/derby_test_results/main/windows/testlog/ibm16/890158-suites.All_diff.txt

> JMX test in nightly test suite failed with: JMXTest:clientjava.lang.InterruptedException:
> --------------------------------------------------------------------------------------------
>
> Key: DERBY-4463
> URL: https://issues.apache.org/jira/browse/DERBY-4463
> Project: Derby
> Issue Type: Bug
> Components: Test
> Affects Versions: 10.6.0.0
> Environment: failed on windows with ibm15 and ibm16 jvms
> Reporter: Mike Matrigali
> Assignee: Dag H. Wanvik
> Fix For: 10.6.0.0
>
> Attachments: derby-4463-2.diff, derby-4463.diff
>
>
> 7) JMXTest:clientjava.lang.InterruptedException:
> at org.apache.derbyTesting.junit.SpawnedProcess.complete(SpawnedProcess.java:137)
> at org.apache.derbyTesting.junit.NetworkServerTestSetup.tearDown(NetworkServerTestSetup.java:392)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:20)
> at junit.extensions.TestSetup.run(TestSetup.java:23)
> at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
> at junit.extensions.TestSetup.run(TestSetup.java:23)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
> at junit.extensions.TestSetup.run(TestSetup.java:23)
> Started failing with build 887040. Here are links for the errors the first time it was encountered:
> http://people.apache.org/~myrnavl/derby_test_results/main/windows/testlog/ibm15/887040-suites.All_diff.txt
> http://people.apache.org/~myrnavl/derby_test_results/main/windows/testlog/ibm16/887040-suites.All_diff.txt
> And it still occurred in the next main build:
> http://people.apache.org/~myrnavl/derby_test_results/main/windows/testlog/ibm15/887483-suites.All_diff.txt
> http://people.apache.org/~myrnavl/derby_test_results/main/windows/testlog/ibm16/887483-suites.All_diff.txt

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

Re: SQLException Table does not exist although just created

2009-12-15T08:58:27Z

Rick,

Again, thanks for your very generous and detailed help!

I am attempting to use the same data file that I used with mySQL. It has no column names at all, just values. It is in CSV format with strings having embedded spaces and commas surrounded by double quotes.

I guess I had better start reading the Derby documentation seriously -- it appears to be quite different from mySQL in many ways.

-Bill

From forum: Apache Derby Users

[jira] Updated: (DERBY-4477) Selecting / projecting a column whose value is represented by a stream more than once fails

2009-12-15T08:58:18Z

[ https://issues.apache.org/jira/browse/DERBY-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kristian Waagan updated DERBY-4477:
-----------------------------------

Attachment: derby-4477-0a-prototype.diff

Attached a prototype patch 0a.

The logic is isolated in ProjectRestrictResultSet, the rest of the patch is code from the patch attached to DERBY-3650.
Currently, the prototype tried to implement something along the lines of phase 2.
I'm running the regressions tests, and tomorrow I will post a performance test and some results.

I'd like some feedback on how we want Derby to behave:
- what should the clone stream threshold be?
- is it okay to always materialize [[LONG] VAR]CHAR [FOR BIT DATA]?
- the DataValueDescriptor.getLengthIfKnow was something I added just before posting the patch, to optimize where possible. Keep it or ditch it? Useful in other scenarios?
(as a side note, getCharLengthIfKnown was added to InternalClob)
- the second check could be done in the constructor if there was a way to reliably find the types of the relevant columns (I was only able to find the descriptors for the top-level result set, but then I don't know the available structures very well)

Finally, with the exception of the code in the constructor, the added code should only be activated if the user selects a "store streamable" [1] column more than once. I don't know if that is very common, and I guess the most important issue is that Derby is able to handle it without crashing.

[1] The store streamable are the various CHAR and CHAR FOR BIT DATA types, BLOB, and CLOB.
(Hmm, what about XML?)

> Selecting / projecting a column whose value is represented by a stream more than once fails
> -------------------------------------------------------------------------------------------
>
> Key: DERBY-4477
> URL: https://issues.apache.org/jira/browse/DERBY-4477
> Project: Derby
> Issue Type: Bug
> Components: Store
> Affects Versions: 10.3.3.0, 10.4.2.0, 10.5.3.0
> Reporter: Kristian Waagan
> Assignee: Kristian Waagan
> Attachments: derby-4477-0a-prototype.diff
>
>
> Selecting / projecting a column whose value is represented as a stream more than once crashes Derby, i.e.:
> ResultSet rs = stmt.executeQuery("SELECT clobValue AS clobOne, clobValue AS clobTwo FROM mytable");
> rs.getString(1);
> rs.getString(2);
> After having looked at the class of bugs having to do with reuse of stream data types, I now have a possible fix. It fixes DERBY-3645, DERBY-3646 and DERBY-2349 (there may be more Jiras).
> The core of the fix is cloning certain DVDs being selected/projected in multiple columns. There are two types of cloning:
> A) materializing clone
> B) stream clone
> (A) can be implemented already, (B) requires code to clone a stream without materializing it. Note that the streams I'm talking about are streams originating from the store.
> Testing revealed the following:
> - the cost of the checks performed to figure out if cloning is required seems acceptable (negligible?)
> - in some cases (A) has better performance than (B) because the raw data only has to be decoded once
> - stream clones are preferred when the data value is above a certain size for several reasons:
> * avoids potential out-of-memory errors (and in case of a server environment, it lowers the memory pressure)
> * avoids decoding the whole value if the JDBC streaming APIs are used to access only parts of the value
> * avoids decoding overall in cases where the value isn't accessed by the client / user
> (this statement conflicts with the performance observation above)
> We don't always know the size of a value, and since the fix code deals with all kinds of data types, it is slightly more costly to try to obtain the size.
> What do people think about the following goal statement?
> Goals:
> ----- Phase 1
> 1) No crashes or wrong results due to stream reuse when executing duplicate column selections (minus goal 4)
> 2) Minimal performance degradation for non-duplicate column selections
> 3) Only a minor performance degradation for duplicate [[LONG] VAR]CHAR [FOR BIT DATA] column selections
> ----- Phase 2
> 4) No out-of-memory exceptions during execution of duplicate column selections of BLOB/CLOB
> 5) Optimize BLOB/CLOB cloning
> I think phase 1 can proceed by reviewing and discussing the prototype patch. Phase 2 requires more discussion and work (see DERBY-3650).
> A note about the bug behavior facts:
> Since this issue is the underlying cause for several other reported issues, I have decided to be liberal when setting the bug behavior facts. Depending on where the duplicate column selection is used, it can cause both crashes, wrong results and data corruption.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

Regression Test Report - Daily 890403 - Sun DBTG

2009-12-15T08:55:52Z

[Auto-generated mail]

*Daily* 890403/2009-12-14 18:02:12 MET

Failed Tests OK Skip Duration Suite
-------------------------------------------------------
*Jvm: 1.6*
lin
0 11774 11774 0 1040.46% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 57.22% derbyall
0 2 2 0 657.42% compatibility
0 2 2 0 .% demoSuite
sles
0 11774 11774 0 680.96% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 49.37% derbyall
0 2 2 0 225.37% compatibility
0 2 2 0 .% demoSuite
sol
0 11774 11774 0 817.35% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 43.10% derbyall
0 2 2 0 307.48% compatibility
0 2 2 0 .% demoSuite
solN+1
0 11774 11774 0 142.44% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 67.31% derbyall
0 2 2 0 300.42% compatibility
0 2 2 0 .% demoSuite
sparc
0 11774 11774 0 530.46% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 47.44% derbyall
0 2 2 0 269.52% compatibility
0 2 2 0 .% demoSuite
vista
F:41,E:8 11768 11719 0 130.69% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 59.02% derbyall
NA NA NA NA compatibility
0 2 2 0 .% demoSuite
vista-64
0 11768 11768 0 188.98% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 57.48% derbyall
NA NA NA NA compatibility
0 2 2 0 .% demoSuite
w2003
0 11768 11768 0 231.30% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 69.89% derbyall
NA NA NA NA compatibility
0 2 2 0 .% demoSuite
Details in http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.6/testing/Limited/testSummary-890403.html
Attempted failure analysis in
http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.6/FailReports/890403_bySig.html

*Jvm: 1.5*
lin
0 222 222 0 52.99% derbyall
0 9984 9984 0 1730.60% suitesAll
sles
0 222 222 0 48.42% derbyall
0 9984 9984 0 1092.74% suitesAll
sol
0 222 222 0 45.48% derbyall
0 9984 9984 0 1528.31% suitesAll
solN+1
0 222 222 0 60.37% derbyall
0 9984 9984 0 1422.45% suitesAll
sparc
0 222 222 0 48.16% derbyall
0 9984 9984 0 1220.08% suitesAll
vista
0 222 222 0 45.39% derbyall
F:0,E:1 9978 9977 0 916.37% suitesAll
vista-64
0 222 222 0 58.19% derbyall
0 9978 9978 0 212.57% suitesAll
w2003
0 222 222 0 55.03% derbyall
0 9978 9978 0 569.97% suitesAll
Details in http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.5/testing/Limited/testSummary-890403.html
Attempted failure analysis in
http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.5/FailReports/890403_bySig.html

*Jvm: 1.4*
lin
0 219 219 2 52.82% derbyall
0 9831 9831 0 1586.06% suitesAll
sles
0 219 219 2 43.22% derbyall
0 9831 9831 0 913.14% suitesAll
sol
0 219 219 2 42.43% derbyall
0 9831 9831 0 1230.49% suitesAll
solN+1
0 219 219 2 59.00% derbyall
0 9831 9831 0 1328.31% suitesAll
sparc
0 219 219 2 46.97% derbyall
0 9831 9831 0 1232.11% suitesAll
vista
0 219 219 2 44.53% derbyall
0 9825 9825 0 899.64% suitesAll
vista-64
0 219 219 2 56.76% derbyall
0 9825 9825 0 216.07% suitesAll
w2003
0 219 219 2 56.80% derbyall
0 9831 9831 0 569.30% suitesAll
Details in http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.4/testing/Limited/testSummary-890403.html
Attempted failure analysis in
http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.4/FailReports/890403_bySig.html

.......................................................
*Jvm: 1.7*
lin
0 11774 11774 0 .% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 .% derbyall
0 2 2 0 .% compatibility
0 2 2 0 .% demoSuite
sol
0 11774 11774 0 .% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 .% derbyall
0 2 2 0 .% compatibility
0 2 2 0 .% demoSuite
vista-64
0 11768 11768 0 .% suitesAll
0 13 13 0 .% jdbcapiAutoLoad
0 12 12 0 .% JDBCDriversAll
0 13 13 0 .% JDBCDriversClient
0 12 12 0 .% JDBCDriversEmbedded
0 221 221 0 .% derbyall
NA NA NA NA compatibility
0 2 2 0 .% demoSuite
Details in http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.7/testing/Limited/testSummary-890403.html
Attempted failure analysis in
http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.7/FailReports/890403_bySig.html

*Jvm: phoneME*
ubuntu
0 4466 4466 0 .% suitesAll
Details in http://dbtg.foundry.sun.com/derby/test/Daily/javaME/testing/Limited/testSummary-890403.html
Attempted failure analysis in
http://dbtg.foundry.sun.com/derby/test/Daily/javaME/FailReports/890403_bySig.html

-------------------------------------------------------

Changes in http://dbtg.foundry.sun.com/derby/test/Daily/UpdateInfo/890403.txt

( All results in http://dbtg.foundry.sun.com/derby/test/ )

From forum: Apache Derby Developers

Re: SQLException Table does not exist although just created

2009-12-15T08:54:14Z

Hi Bill,

It's hard to say without seeing the SQL you used to create the tables
and the SQL of the query which is failing. You may have another
capitalization problem here: column names obey the same casing rules as
table names.

Hope this helps,
-Rick

Wildman wrote:

> After having fixed the table name capitalization problem, when I attempt to
> execute the CALL SYSCS_UTIL.SYSCS_IMPORT_TABLE call, I get this error:
>
> java.sql.SQLException: The exception 'java.sql.SQLException: Column
> 'COLUMN2' is either not in any table in the FROM list or appears within a
> join specification and is outside the scope of the join specification or
> appears in a HAVING clause and is not in the GROUP BY list. If this is a
> CREATE or ALTER TABLE statement then 'COLUMN2' is not a column in the
> target table.' was thrown while evaluating an expression.
> at
> org.apache.derby.client.am.SQLExceptionFactory40.getSQLException(Unknown
> Source)
>
> This is quite a helpful error message, yes? Can anyone help in interpreting
> it?
>
> Thanks!
> -Bill
>
>

From forum: Apache Derby Users

Re: SQLException Table does not exist although just created

2009-12-15T08:19:42Z

After having fixed the table name capitalization problem, when I attempt to execute the CALL SYSCS_UTIL.SYSCS_IMPORT_TABLE call, I get this error:

java.sql.SQLException: The exception 'java.sql.SQLException: Column 'COLUMN2' is either not in any table in the FROM list or appears within a join specification and is outside the scope of the join specification or appears in a HAVING clause and is not in the GROUP BY list. If this is a CREATE or ALTER TABLE statement then 'COLUMN2' is not a column in the target table.' was thrown while evaluating an expression.
at org.apache.derby.client.am.SQLExceptionFactory40.getSQLException(Unknown Source)

This is quite a helpful error message, yes? Can anyone help in interpreting it?

Thanks!
-Bill

From forum: Apache Derby Users

[jira] Updated: (DERBY-4272) SQL Authorization Support for dblook

2009-12-15T08:12:18Z

[ https://issues.apache.org/jira/browse/DERBY-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hiranya Jayathilaka updated DERBY-4272:
---------------------------------------

Attachment: DERBY-4272-changes-u6.txt
DERBY-4272-u6.patch

Uploading the patch for all my workings so far. Includes:

* Complete source code including some of the recent fixes I have done to the code for the bugs discovered during testing
* Partially completed test cases and related resources

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Commented: (DERBY-4455) Prepared statement failure with CLOB: Stream has already been read and end-of-file reached and cannot be re-used.

2009-12-15T08:04:18Z

[ https://issues.apache.org/jira/browse/DERBY-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790786#action_12790786 ]

Knut Anders Hatlen commented on DERBY-4455:
-------------------------------------------

Sounds reasonable, although I don't quite understand how the length argument is used. Could this somehow make the length parameter passed to one of the top-level JDBC streaming methods getting lost?

> Prepared statement failure with CLOB: Stream has already been read and end-of-file reached and cannot be re-used.
> -----------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-4455
> URL: https://issues.apache.org/jira/browse/DERBY-4455
> Project: Derby
> Issue Type: Bug
> Components: Network Server
> Affects Versions: 10.5.3.0
> Environment: Mac OS X 10.6.2, Java 6, Bitronix JTA
> Reporter: Brett Wooldridge
> Assignee: Kristian Waagan
> Attachments: derby-4455-1a.diff, derby-4455-1b.diff, DerbyFailure.zip
>
>
> Possibly related to #4332?
> We have encountered an error when using Prepared Statements and CLOBs. I have read:
> http://db.apache.org/derby/papers/JDBCImplementation.html#setAsciiStream%2CsetBinaryStream%2CsetCharacterStream
> But it does not seem applicable, as we are not re-using a stream.
> The environment is this:
> 1. Java 6
> 2. Derby 10.5.3.0
> 3. Bitronix JTA 1.3.3
> We're actually using Hibernate, but I eliminated it from the equation (and the problem persists).
> A summary of the failure flow is this:
> 1. Start a transaction
> 2. Obtain a connection from a pool of connections (for this test, the pool size is pinned at 1)
> 3. Prepare a statement that inserts a CLOB.
> 4. Set the parameters
> 5. Add the prepared statement to a batch (but we only batch 1 -- this is to emulate what hibernate is doing as closely as possible).
> 6. Execute the batch.
> Everything up to this point works.
> 7. Repeat steps 1-6. But this time, the connection will be reused from the pool, and the statement will be gotten from a prepared statement cache (maintained by bitronix). I.e. the prepared statement is re-used.
> 8. Observe the following failure:
> org.apache.derby.client.am.BatchUpdateException: Non-atomic batch failure. The batch was submitted, but at least one exception occurred on an individual member of the batch. Use getNextException() to retrieve the exceptions for specific batched elements.
> at org.apache.derby.client.am.Agent.endBatchedReadChain(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatchRequestX(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatchX(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatch(Unknown Source)
> at bitronix.tm.resource.jdbc.JdbcPreparedStatementHandle.executeBatch(JdbcPreparedStatementHandle.java:248)
> at org.dancernetworks.TestFailure.doInsert(TestFailure.java:134)
> at org.dancernetworks.TestFailure.doPrepared(TestFailure.java:110)
> at org.dancernetworks.TestFailure.main(TestFailure.java:55)
> Nov 30, 2009 10:29:31 PM bitronix.tm.BitronixTransactionManager shutdown
> INFO: shutting down Bitronix Transaction Manager
> An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> java.sql.SQLException: An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedResultSet.noStateChangeException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.transferParameters(Unknown Source)
> at org.apache.derby.jdbc.XAStatementControl.getRealPreparedStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.getPreparedStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.getStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredStatement.close(Unknown Source)
> at org.apache.derby.impl.drda.DRDAStatement.close(Unknown Source)
> at org.apache.derby.impl.drda.Database.close(Unknown Source)
> at org.apache.derby.impl.drda.Session.close(Unknown Source)
> at org.apache.derby.impl.drda.DRDAConnThread.closeSession(Unknown Source)
> at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
> Caused by: java.sql.SQLException: An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
> ... 15 more
> Caused by: java.sql.SQLException: Java exception: 'Stream has already been read and end-of-file reached and cannot be re-used.: java.io.EOFException'.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.javaException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> ... 12 more
> Caused by: java.io.EOFException: Stream has already been read and end-of-file reached and cannot be re-used.
> at org.apache.derby.iapi.types.ReaderToUTF8Stream.read(Unknown Source)
> at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:320)
> at org.apache.derby.iapi.types.SQLChar.readExternal(Unknown Source)
> at org.apache.derby.iapi.types.SQLChar.getString(Unknown Source)
> at org.apache.derby.iapi.types.SQLChar.setFrom(Unknown Source)
> at org.apache.derby.iapi.types.DataType.setValue(Unknown Source)
> at org.apache.derby.impl.sql.GenericParameterValueSet.transferDataValues(Unknown Source)
> at org.apache.derby.impl.sql.execute.BaseActivation.setParameters(Unknown Source)
> at org.apache.derby.impl.sql.GenericActivationHolder.setParameters(Unknown Source)
> ... 10 more
> Attached is an archived Eclipse project of a self-contained reproduction. It includes everything needed to run, including the Bitronix 1.3.3 jar.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

Re: SQLException Table does not exist although just created

2009-12-15T07:19:18Z

Hi Bill,

Some responses inline..

Wildman wrote:
> Rick,
>
> Again, this is very helpful! One might well ask why single quotes are
> allowed in the SYSCS_UTIL call (and in fact, Apache's examples show single
> quotes in that application) but not in the SQL create table.
This is because the ANSI/ISO syntax makes a distinction between string
literals and SQL identifiers. String literals are enclosed in single
quotes. SQL identifiers are double-quoted if you need them to be
case-sensitive. It helps to keep in mind that SQL is a very old language
which goes back to the days when case-insensitive languages were still
in fashion. Mixing case-insensitive SQL with case-sensitive Java
requires some patience.
> Actually, I
> think the REAL question is, "Why does the SQL standard not include I/O to
> load tables from files" -- this forces every database to implement this
> functionality in its own, idiosyncratic way.
>
I suspect this is because by the time the language was standardized,
each major vendor had already solved this problem in a proprietary way.
There was no good way to harmonize these different approaches.

Hope this helps,
-Rick
> -Bill
>

From forum: Apache Derby Users

[jira] Updated: (DERBY-3092) Use java.util.concurrent in TransactionTable to improve scalability

2009-12-15T07:03:18Z

[ https://issues.apache.org/jira/browse/DERBY-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Knut Anders Hatlen updated DERBY-3092:
--------------------------------------

Summary: Use java.util.concurrent in TransactionTable to improve scalability (was: Use java.util.concurrent in TransactionTable and XactFactory to improve scalability)

I've filed a separate issue for the XactFactory part of this issue. See DERBY-4478.

> Use java.util.concurrent in TransactionTable to improve scalability
> -------------------------------------------------------------------
>
> Key: DERBY-3092
> URL: https://issues.apache.org/jira/browse/DERBY-3092
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.3.1.4
> Reporter: Dyre Tjeldvoll
> Assignee: Knut Anders Hatlen
> Attachments: xact-concept.diff, xact-concept.png
>
>
> Running scalability tests with the client and buffer manager from DERBY-2911 shows that access to the TransactionTable.trans (a Hashtable) and XactFactory.tranId (a shared long) are the next major sources of contention.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Commented: (DERBY-4424) Convert outerjoin.sql into JUnit

2009-12-15T06:55:18Z

[ https://issues.apache.org/jira/browse/DERBY-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790765#action_12790765 ]

Bryan Pendleton commented on DERBY-4424:
----------------------------------------

Hi Eranda, thanks for attaching the updated patch with the license granted.

The patch applies and builds cleanly in my environment, but when I run
the test, I get an error.

Can you see if you can figure out what might be making this test case fail?

thanks,

bryan

...........E...............
Time: 30.233
There was 1 error:
1) testBug2897(org.apache.derbyTesting.functionTests.tests.lang.OuterJoinTest)java.lang.NullPointerException
at org.apache.derbyTesting.junit.RuntimeStatisticsParser.<init>(RuntimeStatisticsParser.java:49)
at org.apache.derbyTesting.functionTests.tests.lang.OuterJoinTest.testBug2897(OuterJoinTest.java:2293)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:109)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
at junit.extensions.TestSetup.run(TestSetup.java:23)
at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)

FAILURES!!!
Tests run: 26, Failures: 0, Errors: 1

> Convert outerjoin.sql into JUnit
> --------------------------------
>
> Key: DERBY-4424
> URL: https://issues.apache.org/jira/browse/DERBY-4424
> Project: Derby
> Issue Type: Improvement
> Reporter: Eranda Sooriyabandara
> Assignee: Eranda Sooriyabandara
> Priority: Minor
> Fix For: 10.6.0.0
>
> Attachments: OuterJoinTest.diff, OuterJoinTest.diff, OuterJoinTest.diff
>
>
> Conversion of the outerjoin test into JUnit

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Created: (DERBY-4478) Use AtomicLong for XactFactory.tranId

2009-12-15T06:49:18Z

Use AtomicLong for XactFactory.tranId
-------------------------------------

Key: DERBY-4478
URL: https://issues.apache.org/jira/browse/DERBY-4478
Project: Derby
Issue Type: Improvement
Components: Store
Affects Versions: 10.6.0.0
Reporter: Knut Anders Hatlen
Priority: Minor

Dyre Tjeldvoll posted some results on DERBY-3092 that indicated that some types of load might cause contention on XactFactory.tranId (a shared long), and suggested that it was replaced with a java.util.concurrent.atomic.AtomicLong on the platforms that support java.util.concurrent.*.

I'm splitting this issue out from DERBY-3092 so that the two possible improvements reported there can be addressed independently.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Assigned: (DERBY-4455) Prepared statement failure with CLOB: Stream has already been read and end-of-file reached and cannot be re-used.

2009-12-15T06:43:18Z

[ https://issues.apache.org/jira/browse/DERBY-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kristian Waagan reassigned DERBY-4455:
--------------------------------------

Assignee: Kristian Waagan

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Commented: (DERBY-4455) Prepared statement failure with CLOB: Stream has already been read and end-of-file reached and cannot be re-used.

2009-12-15T06:37:18Z

[ https://issues.apache.org/jira/browse/DERBY-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790759#action_12790759 ]

Kristian Waagan commented on DERBY-4455:
----------------------------------------

Knut Anders> ... It may be more robust to create a new setValue() method with no length argument and with well-defined behaviour.

I agree adding another value would be more robust.
Looking at the existing code, it seems -1 is already used to indicate that the length is unknown. I found a total of 7 uses of the existing method, so it isn't heavily used.
Would it make sense to improve the documentation of the existing method (in DataValueDescriptor) and verifying the current use, instead of adding yet another method?

After agreeing on this issue, I plan to commit the patch to trunk and backport it to the 10.5 branch. I might also merge it with 10.4 if it merges cleanly.
I think it is an incremental improvement, and it would be nice to follow up on the XA behavior later.

> Prepared statement failure with CLOB: Stream has already been read and end-of-file reached and cannot be re-used.
> -----------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-4455
> URL: https://issues.apache.org/jira/browse/DERBY-4455
> Project: Derby
> Issue Type: Bug
> Components: Network Server
> Affects Versions: 10.5.3.0
> Environment: Mac OS X 10.6.2, Java 6, Bitronix JTA
> Reporter: Brett Wooldridge
> Attachments: derby-4455-1a.diff, derby-4455-1b.diff, DerbyFailure.zip
>
>
> Possibly related to #4332?
> We have encountered an error when using Prepared Statements and CLOBs. I have read:
> http://db.apache.org/derby/papers/JDBCImplementation.html#setAsciiStream%2CsetBinaryStream%2CsetCharacterStream
> But it does not seem applicable, as we are not re-using a stream.
> The environment is this:
> 1. Java 6
> 2. Derby 10.5.3.0
> 3. Bitronix JTA 1.3.3
> We're actually using Hibernate, but I eliminated it from the equation (and the problem persists).
> A summary of the failure flow is this:
> 1. Start a transaction
> 2. Obtain a connection from a pool of connections (for this test, the pool size is pinned at 1)
> 3. Prepare a statement that inserts a CLOB.
> 4. Set the parameters
> 5. Add the prepared statement to a batch (but we only batch 1 -- this is to emulate what hibernate is doing as closely as possible).
> 6. Execute the batch.
> Everything up to this point works.
> 7. Repeat steps 1-6. But this time, the connection will be reused from the pool, and the statement will be gotten from a prepared statement cache (maintained by bitronix). I.e. the prepared statement is re-used.
> 8. Observe the following failure:
> org.apache.derby.client.am.BatchUpdateException: Non-atomic batch failure. The batch was submitted, but at least one exception occurred on an individual member of the batch. Use getNextException() to retrieve the exceptions for specific batched elements.
> at org.apache.derby.client.am.Agent.endBatchedReadChain(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatchRequestX(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatchX(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatch(Unknown Source)
> at bitronix.tm.resource.jdbc.JdbcPreparedStatementHandle.executeBatch(JdbcPreparedStatementHandle.java:248)
> at org.dancernetworks.TestFailure.doInsert(TestFailure.java:134)
> at org.dancernetworks.TestFailure.doPrepared(TestFailure.java:110)
> at org.dancernetworks.TestFailure.main(TestFailure.java:55)
> Nov 30, 2009 10:29:31 PM bitronix.tm.BitronixTransactionManager shutdown
> INFO: shutting down Bitronix Transaction Manager
> An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> java.sql.SQLException: An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedResultSet.noStateChangeException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.transferParameters(Unknown Source)
> at org.apache.derby.jdbc.XAStatementControl.getRealPreparedStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.getPreparedStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.getStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredStatement.close(Unknown Source)
> at org.apache.derby.impl.drda.DRDAStatement.close(Unknown Source)
> at org.apache.derby.impl.drda.Database.close(Unknown Source)
> at org.apache.derby.impl.drda.Session.close(Unknown Source)
> at org.apache.derby.impl.drda.DRDAConnThread.closeSession(Unknown Source)
> at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
> Caused by: java.sql.SQLException: An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
> ... 15 more
> Caused by: java.sql.SQLException: Java exception: 'Stream has already been read and end-of-file reached and cannot be re-used.: java.io.EOFException'.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.javaException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> ... 12 more
> Caused by: java.io.EOFException: Stream has already been read and end-of-file reached and cannot be re-used.
> at org.apache.derby.iapi.types.ReaderToUTF8Stream.read(Unknown Source)
> at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:320)
> at org.apache.derby.iapi.types.SQLChar.readExternal(Unknown Source)
> at org.apache.derby.iapi.types.SQLChar.getString(Unknown Source)
> at org.apache.derby.iapi.types.SQLChar.setFrom(Unknown Source)
> at org.apache.derby.iapi.types.DataType.setValue(Unknown Source)
> at org.apache.derby.impl.sql.GenericParameterValueSet.transferDataValues(Unknown Source)
> at org.apache.derby.impl.sql.execute.BaseActivation.setParameters(Unknown Source)
> at org.apache.derby.impl.sql.GenericActivationHolder.setParameters(Unknown Source)
> ... 10 more
> Attached is an archived Eclipse project of a self-contained reproduction. It includes everything needed to run, including the Bitronix 1.3.3 jar.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

Re: SQLException Table does not exist although just created

2009-12-15T06:21:14Z

Rick,

Again, this is very helpful! One might well ask why single quotes are allowed in the SYSCS_UTIL call (and in fact, Apache's examples show single quotes in that application) but not in the SQL create table. Actually, I think the REAL question is, "Why does the SQL standard not include I/O to load tables from files" -- this forces every database to implement this functionality in its own, idiosyncratic way.

-Bill

From forum: Apache Derby Users

Re: SQLException Table does not exist although just created

2009-12-15T06:08:14Z

Hi Bill,

You need to use double-quotes rather than single-quotes. See the
following section in the Derby Reference Guide:
http://db.apache.org/derby/docs/10.5/ref/ref-single.html#crefsqlj34834

Hope this helps,
-Rick

Wildman wrote:

> Thanks, Rick -- that is indeed quite helpful.
>
> I am a bit unsure of how to specify the table name so that it retains
> lower-case. If I quote the name when creating the table; e.g. "create table
> 'part_type' ( .... )" then I get a SQL error complaining about the quotes.
> Is there another way to specify a table name including casing?
>
> Thanks!
> -Bill
>
>

From forum: Apache Derby Users

[jira] Assigned: (DERBY-3092) Use java.util.concurrent in TransactionTable and XactFactory to improve scalability

2009-12-15T06:05:19Z

[ https://issues.apache.org/jira/browse/DERBY-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Knut Anders Hatlen reassigned DERBY-3092:
-----------------------------------------

Assignee: Knut Anders Hatlen

> Use java.util.concurrent in TransactionTable and XactFactory to improve scalability
> -----------------------------------------------------------------------------------
>
> Key: DERBY-3092
> URL: https://issues.apache.org/jira/browse/DERBY-3092
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.3.1.4
> Reporter: Dyre Tjeldvoll
> Assignee: Knut Anders Hatlen
> Attachments: xact-concept.diff, xact-concept.png
>
>
> Running scalability tests with the client and buffer manager from DERBY-2911 shows that access to the TransactionTable.trans (a Hashtable) and XactFactory.tranId (a shared long) are the next major sources of contention.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

Re: SQLException Table does not exist although just created

2009-12-15T05:58:01Z

Wildman <wleue1@...> writes:

> Thanks, Rick -- that is indeed quite helpful.
>
> I am a bit unsure of how to specify the table name so that it retains
> lower-case. If I quote the name when creating the table; e.g. "create table
> 'part_type' ( .... )" then I get a SQL error complaining about the quotes.
> Is there another way to specify a table name including casing?

Hi Bill,

You need to use double-quotes for SQL identifiers (single-quotes are
only for string literals):

create table "part_type" ( .... )

--
Knut Anders

From forum: Apache Derby Users

Re: SQLException Table does not exist although just created

2009-12-15T05:53:04Z

Thanks, Rick -- that is indeed quite helpful.

I am a bit unsure of how to specify the table name so that it retains lower-case. If I quote the name when creating the table; e.g. "create table 'part_type' ( .... )" then I get a SQL error complaining about the quotes. Is there another way to specify a table name including casing?

Thanks!
-Bill

From forum: Apache Derby Users

[jira] Commented: (DERBY-4455) Prepared statement failure with CLOB: Stream has already been read and end-of-file reached and cannot be re-used.

2009-12-15T05:29:18Z

[ https://issues.apache.org/jira/browse/DERBY-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790739#action_12790739 ]

Knut Anders Hatlen commented on DERBY-4455:
-------------------------------------------

One comment to the patch:

+ // NOTE: The length is ignored (use a non-sense value, it
+ // hopefully fails if required in the future).
+ pvstarget.getParameterForSet(i).setValue(is, -3);

The length is only ignored by SQLChar.setValue() (and, by inheritance, SQLClob). SQLBinary.setValue() uses the length argument, as far as I can see. It may be more robust to create a new setValue() method with no length argument and with well-defined behaviour.

> Prepared statement failure with CLOB: Stream has already been read and end-of-file reached and cannot be re-used.
> -----------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-4455
> URL: https://issues.apache.org/jira/browse/DERBY-4455
> Project: Derby
> Issue Type: Bug
> Components: Network Server
> Affects Versions: 10.5.3.0
> Environment: Mac OS X 10.6.2, Java 6, Bitronix JTA
> Reporter: Brett Wooldridge
> Attachments: derby-4455-1a.diff, derby-4455-1b.diff, DerbyFailure.zip
>
>
> Possibly related to #4332?
> We have encountered an error when using Prepared Statements and CLOBs. I have read:
> http://db.apache.org/derby/papers/JDBCImplementation.html#setAsciiStream%2CsetBinaryStream%2CsetCharacterStream
> But it does not seem applicable, as we are not re-using a stream.
> The environment is this:
> 1. Java 6
> 2. Derby 10.5.3.0
> 3. Bitronix JTA 1.3.3
> We're actually using Hibernate, but I eliminated it from the equation (and the problem persists).
> A summary of the failure flow is this:
> 1. Start a transaction
> 2. Obtain a connection from a pool of connections (for this test, the pool size is pinned at 1)
> 3. Prepare a statement that inserts a CLOB.
> 4. Set the parameters
> 5. Add the prepared statement to a batch (but we only batch 1 -- this is to emulate what hibernate is doing as closely as possible).
> 6. Execute the batch.
> Everything up to this point works.
> 7. Repeat steps 1-6. But this time, the connection will be reused from the pool, and the statement will be gotten from a prepared statement cache (maintained by bitronix). I.e. the prepared statement is re-used.
> 8. Observe the following failure:
> org.apache.derby.client.am.BatchUpdateException: Non-atomic batch failure. The batch was submitted, but at least one exception occurred on an individual member of the batch. Use getNextException() to retrieve the exceptions for specific batched elements.
> at org.apache.derby.client.am.Agent.endBatchedReadChain(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatchRequestX(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatchX(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatch(Unknown Source)
> at bitronix.tm.resource.jdbc.JdbcPreparedStatementHandle.executeBatch(JdbcPreparedStatementHandle.java:248)
> at org.dancernetworks.TestFailure.doInsert(TestFailure.java:134)
> at org.dancernetworks.TestFailure.doPrepared(TestFailure.java:110)
> at org.dancernetworks.TestFailure.main(TestFailure.java:55)
> Nov 30, 2009 10:29:31 PM bitronix.tm.BitronixTransactionManager shutdown
> INFO: shutting down Bitronix Transaction Manager
> An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> java.sql.SQLException: An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedResultSet.noStateChangeException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.transferParameters(Unknown Source)
> at org.apache.derby.jdbc.XAStatementControl.getRealPreparedStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.getPreparedStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.getStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredStatement.close(Unknown Source)
> at org.apache.derby.impl.drda.DRDAStatement.close(Unknown Source)
> at org.apache.derby.impl.drda.Database.close(Unknown Source)
> at org.apache.derby.impl.drda.Session.close(Unknown Source)
> at org.apache.derby.impl.drda.DRDAConnThread.closeSession(Unknown Source)
> at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
> Caused by: java.sql.SQLException: An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
> ... 15 more
> Caused by: java.sql.SQLException: Java exception: 'Stream has already been read and end-of-file reached and cannot be re-used.: java.io.EOFException'.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.javaException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> ... 12 more
> Caused by: java.io.EOFException: Stream has already been read and end-of-file reached and cannot be re-used.
> at org.apache.derby.iapi.types.ReaderToUTF8Stream.read(Unknown Source)
> at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:320)
> at org.apache.derby.iapi.types.SQLChar.readExternal(Unknown Source)
> at org.apache.derby.iapi.types.SQLChar.getString(Unknown Source)
> at org.apache.derby.iapi.types.SQLChar.setFrom(Unknown Source)
> at org.apache.derby.iapi.types.DataType.setValue(Unknown Source)
> at org.apache.derby.impl.sql.GenericParameterValueSet.transferDataValues(Unknown Source)
> at org.apache.derby.impl.sql.execute.BaseActivation.setParameters(Unknown Source)
> at org.apache.derby.impl.sql.GenericActivationHolder.setParameters(Unknown Source)
> ... 10 more
> Attached is an archived Eclipse project of a self-contained reproduction. It includes everything needed to run, including the Bitronix 1.3.3 jar.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Commented: (DERBY-4477) Selecting / projecting a column whose value is represented by a stream more than once fails

2009-12-15T05:21:18Z

[ https://issues.apache.org/jira/browse/DERBY-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790733#action_12790733 ]

Kristian Waagan commented on DERBY-4477:
----------------------------------------

Removed unused imports with revision 890789.

> Selecting / projecting a column whose value is represented by a stream more than once fails
> -------------------------------------------------------------------------------------------
>
> Key: DERBY-4477
> URL: https://issues.apache.org/jira/browse/DERBY-4477
> Project: Derby
> Issue Type: Bug
> Components: Store
> Affects Versions: 10.3.3.0, 10.4.2.0, 10.5.3.0
> Reporter: Kristian Waagan
> Assignee: Kristian Waagan
>
> Selecting / projecting a column whose value is represented as a stream more than once crashes Derby, i.e.:
> ResultSet rs = stmt.executeQuery("SELECT clobValue AS clobOne, clobValue AS clobTwo FROM mytable");
> rs.getString(1);
> rs.getString(2);
> After having looked at the class of bugs having to do with reuse of stream data types, I now have a possible fix. It fixes DERBY-3645, DERBY-3646 and DERBY-2349 (there may be more Jiras).
> The core of the fix is cloning certain DVDs being selected/projected in multiple columns. There are two types of cloning:
> A) materializing clone
> B) stream clone
> (A) can be implemented already, (B) requires code to clone a stream without materializing it. Note that the streams I'm talking about are streams originating from the store.
> Testing revealed the following:
> - the cost of the checks performed to figure out if cloning is required seems acceptable (negligible?)
> - in some cases (A) has better performance than (B) because the raw data only has to be decoded once
> - stream clones are preferred when the data value is above a certain size for several reasons:
> * avoids potential out-of-memory errors (and in case of a server environment, it lowers the memory pressure)
> * avoids decoding the whole value if the JDBC streaming APIs are used to access only parts of the value
> * avoids decoding overall in cases where the value isn't accessed by the client / user
> (this statement conflicts with the performance observation above)
> We don't always know the size of a value, and since the fix code deals with all kinds of data types, it is slightly more costly to try to obtain the size.
> What do people think about the following goal statement?
> Goals:
> ----- Phase 1
> 1) No crashes or wrong results due to stream reuse when executing duplicate column selections (minus goal 4)
> 2) Minimal performance degradation for non-duplicate column selections
> 3) Only a minor performance degradation for duplicate [[LONG] VAR]CHAR [FOR BIT DATA] column selections
> ----- Phase 2
> 4) No out-of-memory exceptions during execution of duplicate column selections of BLOB/CLOB
> 5) Optimize BLOB/CLOB cloning
> I think phase 1 can proceed by reviewing and discussing the prototype patch. Phase 2 requires more discussion and work (see DERBY-3650).
> A note about the bug behavior facts:
> Since this issue is the underlying cause for several other reported issues, I have decided to be liberal when setting the bug behavior facts. Depending on where the duplicate column selection is used, it can cause both crashes, wrong results and data corruption.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Created: (DERBY-4477) Selecting / projecting a column whose value is represented by a stream more than once fails

2009-12-15T05:15:18Z

Selecting / projecting a column whose value is represented by a stream more than once fails
-------------------------------------------------------------------------------------------

Key: DERBY-4477
URL: https://issues.apache.org/jira/browse/DERBY-4477
Project: Derby
Issue Type: Bug
Components: Store
Affects Versions: 10.5.3.0, 10.4.2.0, 10.3.3.0
Reporter: Kristian Waagan
Assignee: Kristian Waagan

Selecting / projecting a column whose value is represented as a stream more than once crashes Derby, i.e.:
ResultSet rs = stmt.executeQuery("SELECT clobValue AS clobOne, clobValue AS clobTwo FROM mytable");
rs.getString(1);
rs.getString(2);

After having looked at the class of bugs having to do with reuse of stream data types, I now have a possible fix. It fixes DERBY-3645, DERBY-3646 and DERBY-2349 (there may be more Jiras).
The core of the fix is cloning certain DVDs being selected/projected in multiple columns. There are two types of cloning:
A) materializing clone
B) stream clone

(A) can be implemented already, (B) requires code to clone a stream without materializing it. Note that the streams I'm talking about are streams originating from the store.

Testing revealed the following:
- the cost of the checks performed to figure out if cloning is required seems acceptable (negligible?)
- in some cases (A) has better performance than (B) because the raw data only has to be decoded once
- stream clones are preferred when the data value is above a certain size for several reasons:
* avoids potential out-of-memory errors (and in case of a server environment, it lowers the memory pressure)
* avoids decoding the whole value if the JDBC streaming APIs are used to access only parts of the value
* avoids decoding overall in cases where the value isn't accessed by the client / user
(this statement conflicts with the performance observation above)

We don't always know the size of a value, and since the fix code deals with all kinds of data types, it is slightly more costly to try to obtain the size.

What do people think about the following goal statement?
Goals:
----- Phase 1
1) No crashes or wrong results due to stream reuse when executing duplicate column selections (minus goal 4)
2) Minimal performance degradation for non-duplicate column selections
3) Only a minor performance degradation for duplicate [[LONG] VAR]CHAR [FOR BIT DATA] column selections
----- Phase 2
4) No out-of-memory exceptions during execution of duplicate column selections of BLOB/CLOB
5) Optimize BLOB/CLOB cloning

I think phase 1 can proceed by reviewing and discussing the prototype patch. Phase 2 requires more discussion and work (see DERBY-3650).

A note about the bug behavior facts:
Since this issue is the underlying cause for several other reported issues, I have decided to be liberal when setting the bug behavior facts. Depending on where the duplicate column selection is used, it can cause both crashes, wrong results and data corruption.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Commented: (DERBY-4455) Prepared statement failure with CLOB: Stream has already been read and end-of-file reached and cannot be re-used.

2009-12-15T04:39:18Z

[ https://issues.apache.org/jira/browse/DERBY-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790725#action_12790725 ]

Brett Wooldridge commented on DERBY-4455:
-----------------------------------------

I consider this bug fixed by the provided patch. Therefore you are free to close the issue.

However, if there is indeed *unnecessary* server-side "re-preparation" of the statements (see org.apache.derby.impl.jdbc.EmbedPreparedStatement.transferParameters in the stack trace) in the case of XA connections, I would like a separate issue to be opened. Other than that efficiency concern, the actual defect appears to be fixed by the patch and I have not encountered this exception since applying the patch.

I would like to see this fix integrated into the next release because I prefer not to run non-released versions of jars.

> Prepared statement failure with CLOB: Stream has already been read and end-of-file reached and cannot be re-used.
> -----------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-4455
> URL: https://issues.apache.org/jira/browse/DERBY-4455
> Project: Derby
> Issue Type: Bug
> Components: Network Server
> Affects Versions: 10.5.3.0
> Environment: Mac OS X 10.6.2, Java 6, Bitronix JTA
> Reporter: Brett Wooldridge
> Attachments: derby-4455-1a.diff, derby-4455-1b.diff, DerbyFailure.zip
>
>
> Possibly related to #4332?
> We have encountered an error when using Prepared Statements and CLOBs. I have read:
> http://db.apache.org/derby/papers/JDBCImplementation.html#setAsciiStream%2CsetBinaryStream%2CsetCharacterStream
> But it does not seem applicable, as we are not re-using a stream.
> The environment is this:
> 1. Java 6
> 2. Derby 10.5.3.0
> 3. Bitronix JTA 1.3.3
> We're actually using Hibernate, but I eliminated it from the equation (and the problem persists).
> A summary of the failure flow is this:
> 1. Start a transaction
> 2. Obtain a connection from a pool of connections (for this test, the pool size is pinned at 1)
> 3. Prepare a statement that inserts a CLOB.
> 4. Set the parameters
> 5. Add the prepared statement to a batch (but we only batch 1 -- this is to emulate what hibernate is doing as closely as possible).
> 6. Execute the batch.
> Everything up to this point works.
> 7. Repeat steps 1-6. But this time, the connection will be reused from the pool, and the statement will be gotten from a prepared statement cache (maintained by bitronix). I.e. the prepared statement is re-used.
> 8. Observe the following failure:
> org.apache.derby.client.am.BatchUpdateException: Non-atomic batch failure. The batch was submitted, but at least one exception occurred on an individual member of the batch. Use getNextException() to retrieve the exceptions for specific batched elements.
> at org.apache.derby.client.am.Agent.endBatchedReadChain(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatchRequestX(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatchX(Unknown Source)
> at org.apache.derby.client.am.PreparedStatement.executeBatch(Unknown Source)
> at bitronix.tm.resource.jdbc.JdbcPreparedStatementHandle.executeBatch(JdbcPreparedStatementHandle.java:248)
> at org.dancernetworks.TestFailure.doInsert(TestFailure.java:134)
> at org.dancernetworks.TestFailure.doPrepared(TestFailure.java:110)
> at org.dancernetworks.TestFailure.main(TestFailure.java:55)
> Nov 30, 2009 10:29:31 PM bitronix.tm.BitronixTransactionManager shutdown
> INFO: shutting down Bitronix Transaction Manager
> An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> java.sql.SQLException: An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedResultSet.noStateChangeException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.transferParameters(Unknown Source)
> at org.apache.derby.jdbc.XAStatementControl.getRealPreparedStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.getPreparedStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.getStatement(Unknown Source)
> at org.apache.derby.iapi.jdbc.BrokeredStatement.close(Unknown Source)
> at org.apache.derby.impl.drda.DRDAStatement.close(Unknown Source)
> at org.apache.derby.impl.drda.Database.close(Unknown Source)
> at org.apache.derby.impl.drda.Session.close(Unknown Source)
> at org.apache.derby.impl.drda.DRDAConnThread.closeSession(Unknown Source)
> at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
> Caused by: java.sql.SQLException: An IOException was thrown when reading a 'java.sql.String' from an InputStream.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
> ... 15 more
> Caused by: java.sql.SQLException: Java exception: 'Stream has already been read and end-of-file reached and cannot be re-used.: java.io.EOFException'.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.javaException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> ... 12 more
> Caused by: java.io.EOFException: Stream has already been read and end-of-file reached and cannot be re-used.
> at org.apache.derby.iapi.types.ReaderToUTF8Stream.read(Unknown Source)
> at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:320)
> at org.apache.derby.iapi.types.SQLChar.readExternal(Unknown Source)
> at org.apache.derby.iapi.types.SQLChar.getString(Unknown Source)
> at org.apache.derby.iapi.types.SQLChar.setFrom(Unknown Source)
> at org.apache.derby.iapi.types.DataType.setValue(Unknown Source)
> at org.apache.derby.impl.sql.GenericParameterValueSet.transferDataValues(Unknown Source)
> at org.apache.derby.impl.sql.execute.BaseActivation.setParameters(Unknown Source)
> at org.apache.derby.impl.sql.GenericActivationHolder.setParameters(Unknown Source)
> ... 10 more
> Attached is an archived Eclipse project of a self-contained reproduction. It includes everything needed to run, including the Bitronix 1.3.3 jar.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Resolved: (DERBY-4472) [patch] non short circuit op, could cause eof problems

2009-12-15T00:50:18Z

[ https://issues.apache.org/jira/browse/DERBY-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kristian Waagan resolved DERBY-4472.
------------------------------------

Resolution: Fixed
Fix Version/s: 10.6.0.0
Issue & fix info: (was: [Patch Available])

Thanks for the update.

Committed the patch to trunk with revision 890692.

> [patch] non short circuit op, could cause eof problems
> ------------------------------------------------------
>
> Key: DERBY-4472
> URL: https://issues.apache.org/jira/browse/DERBY-4472
> Project: Derby
> Issue Type: Bug
> Components: JDBC
> Affects Versions: 10.5.3.0
> Reporter: Dave Brosius
> Priority: Trivial
> Fix For: 10.6.0.0
>
> Attachments: sc.diff
>
>
> (from patch)
> int offset = 0;
> - while( offset < dataDirectory.length() & Character.isSpaceChar( dataDirectory.charAt( offset)))
> + while( offset < dataDirectory.length() && Character.isSpaceChar( dataDirectory.charAt( offset)))
> offset ++;
> int leftParen = -1;
> int rightParen = -1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

From forum: Apache Derby Developers

[jira] Subscription: Derby: JIRA issues with patch available

2009-12-15T00:36:18Z

Issue Subscription
Filter: Derby: JIRA issues with patch available (15 issues)
Subscriber: derby-dev

Key Summary
DERBY-4405 Transformation to inner join not performed for certain three-way joins
https://issues.apache.org/jira/browse/DERBY-4405
DERBY-651 Re-enable the storing of java objects in the database
https://issues.apache.org/jira/browse/DERBY-651
DERBY-4472 [patch] non short circuit op, could cause eof problems
https://issues.apache.org/jira/browse/DERBY-4472
DERBY-4475 [patch] roleName isn't trimmed as expected
https://issues.apache.org/jira/browse/DERBY-4475
DERBY-4463 JMX test in nightly test suite failed with: JMXTest:clientjava.lang.InterruptedException:
https://issues.apache.org/jira/browse/DERBY-4463
DERBY-4191 Lack of SELECT privilege does not prevent SELECT COUNT(*)
https://issues.apache.org/jira/browse/DERBY-4191
DERBY-4241 Improve transition from read-only to writable Clob representation
https://issues.apache.org/jira/browse/DERBY-4241
DERBY-4455 Prepared statement failure with CLOB: Stream has already been read and end-of-file reached and cannot be re-used.
https://issues.apache.org/jira/browse/DERBY-4455
DERBY-4357 TableFunctions provide no information to limit underlying query
https://issues.apache.org/jira/browse/DERBY-4357
DERBY-4183 Our regression tests use various jar files for which we don't have build scripts.
https://issues.apache.org/jira/browse/DERBY-4183
DERBY-1191 Some SQLExceptions, for example those generated from BrokeredStatements, do not print to derby.log even when derby.stream.error.logSeverityLevel=0
https://issues.apache.org/jira/browse/DERBY-1191
DERBY-4272 SQL Authorization Support for dblook
https://issues.apache.org/jira/browse/DERBY-4272
DERBY-873 OSGi bundle activator not JSR169 compliant
https://issues.apache.org/jira/browse/DERBY-873
DERBY-4137 OOM issue using XA with timeouts
https://issues.apache.org/jira/browse/DERBY-4137
DERBY-3788 Provide a zero-admin way of updating the statisitcs of an index
https://issues.apache.org/jira/browse/DERBY-3788

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=10396&filterId=12310751

From forum: Apache Derby Developers

Re: Questions concerning the implementation

2009-12-14T19:42:30Z

mkoendling <m_koendl@...> writes:

> hi derby team,
>
> I already posted some of my questions concerning the implementation of jdbc
> in the user list:
> http://old.nabble.com/Questions-about-the-implementation-to26689131.html
>
> I guess one of my problems is to figure out where the jdbc-driver starts and
> where it ends. Is it correct to say that the jdbc-driver of derby (embedded
> or client) is quite intertwined with the whole engine? Or perhaps, is the

The embedded driver does, of course, make calls on the Derby engine,
but the driver code is fairly well separated out, cf. the directory

java/engine/org/apache/derby/impl/jdbc

For example, EmbedResultSet.java:

public abstract class EmbedResultSet extends ConnectionChild
implements EngineResultSet, Comparable {

implements java.sql.Resultset (through implementation of the the
interface EngineResultSet).

> jdbc-driver the whole engine?

No, that is not a fair statement.

>
> My second question concerns ResultSets. How is the data actually fetched in
> Client-Server mode. Is the whole ResultSet loaded into one Object/Table/...
> and sent over the network? Is there some kind of cache for the ResultSet and
> subsequent rows are loaded dynamically?... .

The client does prefetch rows, but will ask for more rows when the
cache is empty. The server will then use JDBC API (and hence the
embedded driver) to get more rows from the engine and return them to
the client, see Cursor.java/NetCursor.java. The client driver uses the
DRDA protocol [1], which can make the code a bit hard to understand at
first. I suggest running a debugger to get a feel for the dynamic
structure of the code.

[1] http://en.wikipedia.org/wiki/DRDA

Hope this helps a bit.

Dag

From forum: Apache Derby Developers

Re: SQLException Table does not exist although just created

2009-12-14T16:36:10Z

Hi Bill,

The schema and table name arguments are case sensitive. In addition,
since you did not use quoted identifiers when you created the table, the
table name is PART_TYPE, not part_type. Try your experiment with an
uppercased table name.

Hope this helps,
-Rick

Wildman wrote:

> I am trying to move a Java database application from mySQL to the embedded
> version of the Java Database (nee Derby). I successfully create an instance
> of the DriverManager and make a connection. Next, I create several tables.
> Next I try to use the SYSCS_UTIL.SYSCS_IMPORT utility to load data from a
> CSV file into the tables I have just created. I get the following error:
>
> java.sql.SQLException: Table 'BILL.part_type' does not exist.
>
> ...even though I have just created the table 'part_type' a few lines above.
>
> Is this possibly related to the 'BILL' schema? When I create the tables, I
> don't specify or create any schema, although I do log in as 'Bill' when I
> open the database connection.
>
> A version of my source, edited for brevity, follows.
>
> Thanks!
> -Bill
> - - - - - - - - - - - - - -
> Properties props = new Properties();
> props.put("user", "Bill");
> props.put("password", "******"); // password masked in this copy
> String full_url = URL + DEFAULT_DB + ";create=true";
> try {
> con = DriverManager.getConnection(full_url, props);
> } catch (Exception e2) {
> JOptionPane.showMessageDialog(this.gui,
> "Sorry, got an exception creating the new database: " +
> e2,
> "Fatal", JOptionPane.INFORMATION_MESSAGE);
> e2.printStackTrace(System.out);
> return -1;
> }
>
> // drop any existing tables - ignore errors
> try {
> sendSQL("drop table part_type");
> } // etc..
> try {
> sql = "create table part_type ("
> + "part_type_id INTEGER NOT NULL PRIMARY KEY GENERATED
> ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1),"
> + "partnum varchar(20),"
> + "name varchar(255),"
> + "family varchar(40),"
> + "genus varchar(20),"
> + "image varchar(1)"
> + ")";
> sendSQL(sql); // this sends theSQL string to the connection
>
> String infile = getStartDir() + File.separator + "Partsref.csv";
> sql = "CALL SYSCS_UTIL.SYSCS_IMPORT_TABLE (null, 'part_type', '"
> + infile + "', null, null, null, 1)";
> System.err.println(sql);
> sendSQL(sql); // <-- exception here
> - - - - - - - - - - - - - - - - - -
>
>

From forum: Apache Derby Users