[jira] Created: (DERBY-3541) quit; on ij with authentication enabled does not shutdown the derby modules that are loaded

quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
-------------------------------------------------------------------------------------------

                 Key: DERBY-3541
                 URL: https://issues.apache.org/jira/browse/DERBY-3541
             Project: Derby
          Issue Type: Bug
          Components: Security
    Affects Versions: 10.4.0.0
            Reporter: V.Narayanan




--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.Narayanan updated DERBY-3541:
-------------------------------

    Attachment: derby.properties
                WithoutAuthentication_StackTrace.txt
                PrintStackTrace.diff

I modified the derby codebase to print a stack trace (new Throwable().printStackTrace();) in
the TopService#shutdown() method (pls find patch attached.)

I then tried getting a connection *with* authentication enabled (pls find derby.properties
attached) with the following connection url

ij>  connect 'jdbc:derby:replicationdb2;create=true;user=oystein;password=pass';
ij> quit;
vn@vn-laptop:~/work/workspaces/PrintStackTrace/trials$

There was no stack trace printed.

I then tried a connection *without* authentication enabled

There was a stack trace printed (pls find attached stack trace)

ij> connect 'jdbc:derby:replicationdb1;create=true';
ij> quit;
java.lang.Throwable
        at org.apache.derby.impl.services.monitor.TopService.shutdown(TopService.java:327)
        at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:237)
        at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:203)
        at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:231)
        at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:119)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:207)
        at org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416)
        at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:249)
        at org.apache.derby.impl.tools.ij.Main.go(Main.java:215)
        at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:181)
        at org.apache.derby.impl.tools.ij.Main.main(Main.java:73)
        at org.apache.derby.tools.ij.main(ij.java:59)

        ............... and so on (attached is the full stack trace)

Not shutting down the modules when authentication is enabled I believe is not correct.

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Security
>    Affects Versions: 10.4.0.0
>            Reporter: V.Narayanan
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579040#action_12579040 ]

V.Narayanan commented on DERBY-3541:
------------------------------------

Upon calling quit in ij, ij tries to shutdown the connection. The code relevant to this
is found here

org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416)

here we do the following

if (d!=null) { // do we have a driver running? shutdown on exit.
        try {
                DriverManager.getConnection("jdbc:derby:;shutdown=true");
        } catch (SQLException e) {
                // ignore the errors, they are expected.
        }
}

now with authentication enabled trying to take a connection to shutdown like done in
the above code will result in an exception

ij> connect 'jdbc:derby:;shutdown=true';
ERROR 08004: Connection authentication failure occurred.  Reason: Invalid authentication..

This is the reason why upon a quit; in ij with authentication enabled and in the
embedded mode the modules are never shutdown.

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Security
>    Affects Versions: 10.4.0.0
>            Reporter: V.Narayanan
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579042#action_12579042 ]

V.Narayanan commented on DERBY-3541:
------------------------------------

We get a better picture of the above error if we replace the catch clause in

org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416)

if (d!=null) { // do we have a driver running? shutdown on exit.
try {
DriverManager.getConnection("jdbc:derby:;shutdown=true");
} catch (SQLException e) {
// ignore the errors, they are expected.
}
}

with a e.printStackTrace()

I get the following everytime I try to take a connection and quit ij with
authentication enabled

ij version 10.5
ij> connect 'jdbc:derby:mydb;user=oystein;password=pass';
ij> quit;
java.sql.SQLNonTransientConnectionException: Connection authentication failure occurred.  Reason: Invalid authentication..
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:76)
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:87)
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:93)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:172)
        at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:220)
        at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:119)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:207)
        at org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416)
        at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:249)
        at org.apache.derby.impl.tools.ij.Main.go(Main.java:215)
        at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:181)
        at org.apache.derby.impl.tools.ij.Main.main(Main.java:73)
        at org.apache.derby.tools.ij.main(ij.java:59)
Caused by: java.sql.SQLException: Connection authentication failure occurred.  Reason: Invalid authentication..
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:119)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:70)
        ... 13 more
vn@vn-laptop:~/work/workspaces/PrintStackTrace$

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Security
>    Affects Versions: 10.4.0.0
>            Reporter: V.Narayanan
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579181#action_12579181 ]

V.Narayanan commented on DERBY-3541:
------------------------------------

I get an authentication exception similar to the one above in a client/server
environment too (with the same modification to the derby codebase as above).

I however don't think I can fault derby here.

I believe when we call quit; on ij (after booting a database in the embedded mode)
we are trying to shutdown ij without  worrying about the database that we might
have booted earlier. I do not think we can restrict the user from exiting (shutting
down) the application (ij) he has started just because he has taken an authenticated
derby connection inside the application(ij).

This seems like a perfectly legal operation to me.

ij relying on taking a successful system shutdown derby connection (without proper
user name and password) when authentication is enabled causes the modules in the
derby system to not be shutdown in a clean fashion.

Basically ij tries its best to do a clean shutdown of the system upon exit.

From the analysis above I am tempted to close this JIRA issue as invalid (unless someone
has an objection).

I however think this behavior of ij when we call a quit (of not doing a clean shutdown when authentication
is enabled) can be documented somewhere (if not already documented). If the community feels the same
way I will raise a JIRA requesting this documentation to be included.

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Security
>    Affects Versions: 10.4.0.0
>            Reporter: V.Narayanan
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579332#action_12579332 ]

Jørgen Løland commented on DERBY-3541:
--------------------------------------

I guess this (quiting derby without shutting down the databases) has other implications as well. E.g., when a booted database is not properly shut down, the checkpoint that is normally made on shutdown will not be there, increasing the time to recover the next time that database is booted.

Combining what you say here with the discussion on 3447, my suggestion is twofold:

* I agree that derby should not hang if the user shuts down an application without shutting down the databases. Your suggestion of using daemon threads for replication sounds good to me.
* On the other hand, we control the ij application, and we may want to improve it. Documenting this problematic behavior like suggested may be enough, but it would also be possible to abort the 'quit;' call with an exception saying:

ij > quit;
Error: could not quit ij because the following databases have not been shut down: "test1", "test2". Shut down these databases first and then quit ij.

I don't feel strongly about either option (only document / fail 'quit;'), but I think that in any case, the severity can be set lower now that we know the cause of the problem.

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Security
>    Affects Versions: 10.4.0.0
>            Reporter: V.Narayanan
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579338#action_12579338 ]

V.Narayanan commented on DERBY-3541:
------------------------------------

>* On the other hand, we control the ij application, and we may want to improve it.
>Documenting this problematic behavior like suggested may be enough, but it would
>also be possible to abort the 'quit;' call with an exception saying:

Depends on how much of derby specific code you want to put in ij, Jorgen.

I must admit I was surprised at seeing the derby system shutdown being attempted
when we do a quit itself.

My impression about ij was ij is a general jdbc front end tool. It can be used with any database
that supports a jdbc driver. This shutdown attempt with use to any other database (different
jdbc url) will surely fail. Well maybe that is why we ignore the SQLExceptions upon doing
this.

Adding more and more derby specific code to ij, makes ij more and more a
derby specific tool and lesser and lesser a general jdbc interface supporting front-end.

For now I will lower the severity of the issue. I think we can take it from there.

I will also leave the issue open.

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Security
>    Affects Versions: 10.4.0.0
>            Reporter: V.Narayanan
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.Narayanan updated DERBY-3541:
-------------------------------

    Priority: Minor  (was: Major)

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Security
>    Affects Versions: 10.4.0.0
>            Reporter: V.Narayanan
>            Priority: Minor
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579350#action_12579350 ]

Øystein Grøvlen commented on DERBY-3541:
----------------------------------------

I think it is unfortunate that ij behaves differently with and without authentication.

I also feel that is very convenient that ij shut down the database when I exit.  Lately, when I have done some work in ij on a database with authentication/authorization turned on, it has been very annoying that if I quit I ij after doing a large update operation, it has taken a long time to boot the database the next time.  (Now I understand why)

I see the following possible solutions:

1. Change ij so the database is never shut down on quit, not just when authentication is turned on.  Document that users need to shutdown databases before they exit ij, similarly to the message to JDBC programmers.
2. Somehow fix the case with authentication so that the database is shut down.  (I am not sure how)
3. Let quit report error if ij is not able to shut down a database due to authentication.  Introduce a new command that can be use to force quit ij without shutting down databases.

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Security
>    Affects Versions: 10.4.0.0
>            Reporter: V.Narayanan
>            Priority: Minor
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dag H. Wanvik updated DERBY-3541:
---------------------------------

    Component/s: Tools

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Tools
>    Affects Versions: 10.4.1.3
>            Reporter: V.Narayanan
>            Priority: Minor
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tiago R. Espinha updated DERBY-3541:
------------------------------------


Triaged for 10.5.2. No changes made.

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Tools
>    Affects Versions: 10.4.1.3
>            Reporter: V.Narayanan
>            Priority: Minor
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729056#action_12729056 ]

Lily Wei commented on DERBY-3541:
---------------------------------

I tried this bug on trunk version(10.6). I can not reproduce this bug.
$ ij -p derby.properties
ij version 10.6
ij> connect 'jdbc:derby:test;create=true;user=johndoe;password=pass';
ij> quit;
I am welcome to any suggestion to reproduce this bug.

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Tools
>    Affects Versions: 10.4.1.3
>            Reporter: V.Narayanan
>            Priority: Minor
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729072#action_12729072 ]

V.Narayanan commented on DERBY-3541:
------------------------------------

Thank you for trying out the reproducible.
when you said "I can not reproduce this bug."
did you mean that the modules are currently being
closed? Are you sure that TopService#shutdown()
is now being called with authentication enabled?

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Tools
>    Affects Versions: 10.4.1.3
>            Reporter: V.Narayanan
>            Priority: Minor
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729414#action_12729414 ]

Lily Wei commented on DERBY-3541:
---------------------------------

Thank you, too. This is a good bug to look at. I mean that when authentication is enable, quit does not do shutdown. However, I don't see how quit can shutdown when authentication is enable. In such case, shutdown process will hit authentication failure error first before successfully shutting down the database. I think that is the point 2 on Oystin's point if I may quote him.

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Tools
>    Affects Versions: 10.4.1.3
>            Reporter: V.Narayanan
>            Priority: Minor
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729545#action_12729545 ]

V.Narayanan commented on DERBY-3541:
------------------------------------

Oystein says:

"2. Somehow fix the case with authentication so that the database is shut down. (I am not sure how)"

Although he does mean that it is difficult to shutdown the database when authentication is enabled
(the same way shutdown is done when authentication is not enabled) he does not mean this is not
a bug.

Like in my comments before, ij is a generic jdbc front end tool, I am not happy with a derby specific
call here at all. Ideally speaking you should be capable of using ij with any DB that supports a jdbc
driver.

If everyone agrees we can probably document that we need to shutdown DB's before quitting and
prevent ij from going down without closing the DB's. That would be one probable solution. But again
I am not sure I will be happy with such a crude fix.

Here is my 2 paisa :)

(IJ is a simple and elegant tool. There is a innocent simplicity to the tool that I would never want it
to lose. If a user wants a sophisticated front end there are other options. We should not clutter ij
unnecessarily.)

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Tools
>    Affects Versions: 10.4.1.3
>            Reporter: V.Narayanan
>            Priority: Minor
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729625#action_12729625 ]

Kathey Marsden commented on DERBY-3541:
---------------------------------------

V. Narayanan said:
>If everyone agrees we can probably document that we need to shutdown DB's before quitting and prevent ij from going down without closing the DB's.

I think that is just too dangerous. There are a lot of applications that use ij scripts during their install that might hang if we do this.  We do document that exit (quit) does a shutdown. http://db.apache.org/derby/docs/dev/tools/rtoolsijcomref33358.html

I think the best we can do is document that  with authentication, shutdown should be performed explicitly because the ij request will not succeed.  I think even adding a warning that shutdown was not successful might cause automated scripts to fail.

> quit; on ij with authentication enabled does not shutdown the derby modules that are loaded
> -------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3541
>                 URL: https://issues.apache.org/jira/browse/DERBY-3541
>             Project: Derby
>          Issue Type: Bug
>          Components: Tools
>    Affects Versions: 10.4.1.3
>            Reporter: V.Narayanan
>            Priority: Minor
>         Attachments: derby.properties, PrintStackTrace.diff, WithoutAuthentication_StackTrace.txt
>
>

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.