Derby in-memory back end - where to go next?

[Kristian Waagan-4]

Hello,

In Derby 10.5 an in-memory back end, or storage engine, was included. It
stores all the data in main memory, with the exception of derby.log. If
this is news to you, and you want a quick intro to it, see [1] and [2].

I'm trying to gather some feedback on whether the current implementation
is found acceptable, or if there are additional features people would
like to see. I expect some wishes to emerge, and I plan to record these
on the wiki page [1]. The page can then be used to guide further work in
this area.

To start the discussion, I'll list some potential features and tasks.
Feel free to comment on any one of them either by replying to this
thread, or by adding your comments to [1]. It can be a +1 or -1 on the
feature itself, a suggestion for a new feature, or details on what a
feature should look like.


* Documentation
Must at least document the JDBC subsubprotocol, and also explain how to
delete in-memory databases.
If new features are added, these must be documented as well.

* Deletion of in-memory databases
Currently the only ways to delete an in-memory database are to restart
the JVM or use a static method that isn't part of Derby's public API. A
proper mechanism for deletion should be added.

* Automatic deletion on database shutdown (or when last connection
disconnects)

* "Anonymous in-memory databases"
A database which only the connection creating it can access, and when
the connection goes away the database goes away.

* Automatic persistence
The database could be persisted to disk automatically based on certain
criteria. The most obvious ones are perhaps on a fixed interval and on
JVM shutdown.

* Monitoring
The most basic information is how many in-memory databases exist in the
current JVM, and how big they are. How should this information be
presented? Should it be available to anyone having a connection to the
current JVM?

* No derby.log
Include a class in Derby that will discard everything written to derby.log.


Thank you for your feedback,
--
Kristian

[1] http://wiki.apache.org/db-derby/InMemoryBackEndPrimer
[2] http://blogs.sun.com/kah/

[Bernt M. Johnsen]

>>>>>>>>>>>> Kristian Waagan wrote (2009-09-07 11:26:13):

> Hello,
>
> In Derby 10.5 an in-memory back end, or storage engine, was included. It  
> stores all the data in main memory, with the exception of derby.log. If  
> this is news to you, and you want a quick intro to it, see [1] and [2].
>
> I'm trying to gather some feedback on whether the current implementation  
> is found acceptable, or if there are additional features people would  
> like to see. I expect some wishes to emerge, and I plan to record these  
> on the wiki page [1]. The page can then be used to guide further work in  
> this area.
>
> To start the discussion, I'll list some potential features and tasks.  
> Feel free to comment on any one of them either by replying to this  
> thread, or by adding your comments to [1]. It can be a +1 or -1 on the  
> feature itself, a suggestion for a new feature, or details on what a  
> feature should look like.

And here's my opinion (+1: yes, +0/-0 don't care but with a
positive/negative bias, -1: no).

>
>
> * Documentation
> Must at least document the JDBC subsubprotocol, and also explain how to  
> delete in-memory databases.
> If new features are added, these must be documented as well.

+1

> * Deletion of in-memory databases
> Currently the only ways to delete an in-memory database are to restart  
> the JVM or use a static method that isn't part of Derby's public API. A  
> proper mechanism for deletion should be added.

+1

> * Automatic deletion on database shutdown (or when last connection  
> disconnects)

-1

I certainly don't want this feature. An in-memory database should (at
least by default) live as long as the VM or until explicitely deleted.

> * "Anonymous in-memory databases"
> A database which only the connection creating it can access, and when  
> the connection goes away the database goes away.

+0

> * Automatic persistence
> The database could be persisted to disk automatically based on certain  
> criteria. The most obvious ones are perhaps on a fixed interval and on  
> JVM shutdown.

+0

> * Monitoring
> The most basic information is how many in-memory databases exist in the  
> current JVM, and how big they are. How should this information be  
> presented? Should it be available to anyone having a connection to the  
> current JVM?

+1: Use JMX

> * No derby.log
> Include a class in Derby that will discard everything written to
> derby.log.

-0

--
Bernt Marius Johnsen, Staff Engineer
Database Technology Group, Sun Microsystems, Trondheim, Norway

[Rick Hillegas-2]

Thanks for starting this discussion, Kristian. My thoughts inline...

Kristian Waagan wrote:

> Hello,
>
> In Derby 10.5 an in-memory back end, or storage engine, was included.
> It stores all the data in main memory, with the exception of
> derby.log. If this is news to you, and you want a quick intro to it,
> see [1] and [2].
>
> I'm trying to gather some feedback on whether the current
> implementation is found acceptable, or if there are additional
> features people would like to see. I expect some wishes to emerge, and
> I plan to record these on the wiki page [1]. The page can then be used
> to guide further work in this area.
>
> To start the discussion, I'll list some potential features and tasks.
> Feel free to comment on any one of them either by replying to this
> thread, or by adding your comments to [1]. It can be a +1 or -1 on the
> feature itself, a suggestion for a new feature, or details on what a
> feature should look like.
>
>
> * Documentation
> Must at least document the JDBC subsubprotocol, and also explain how
> to delete in-memory databases.
> If new features are added, these must be documented as well.

+1
>
> * Deletion of in-memory databases
> Currently the only ways to delete an in-memory database are to restart
> the JVM or use a static method that isn't part of Derby's public API.
> A proper mechanism for deletion should be added.
+1
>
> * Automatic deletion on database shutdown (or when last connection
> disconnects)
I can see that this would be useful for some people. I wouldn't make
this the default behavior, however. There are some operations like
changing persistent database-wide properties that don't take effect
until you bounce the database--and those operations would be disabled if
auto-delete were the default behavior.
>
> * "Anonymous in-memory databases"
> A database which only the connection creating it can access, and when
> the connection goes away the database goes away.
This sounds like a refinement of the previous option and I can see that
this would be a useful security feature.
>
> * Automatic persistence
> The database could be persisted to disk automatically based on certain
> criteria. The most obvious ones are perhaps on a fixed interval and on
> JVM shutdown.
+1
>
> * Monitoring
> The most basic information is how many in-memory databases exist in
> the current JVM, and how big they are. How should this information be
> presented? Should it be available to anyone having a connection to the
> current JVM?
+1

As Bernt says, JMX is attractive.
>
> * No derby.log
> Include a class in Derby that will discard everything written to
> derby.log.
The current mechanism for redirecting the error log to a user defined
stream seems easy and flexible enough to me. But the user documentation
should point out that applications need to redirect the error log if
they really want to run without leaving any fingerprints.

Thanks!
-Rick
>
>
> Thank you for your feedback,

[Rick Hillegas-2]

Hi Kristian,

Here's another piece of feedback: Last night I gave an overview of Derby
to the San Francisco Java User's Group. A developer asked whether the
growth of the in-memory database could be bounded. He had a use case
which we didn't explore in depth but which involved periodically
truncating the database. I asked him to bring his requirements to the
Derby user list so that we could feed them into your spec effort. Here
are my takeaways:

* It would be great to be able to bound the growth of the in-memory db

* It would be great if the memory occupied by deleted records could be
released

Thanks,
-Rick

Kristian Waagan wrote:

> Hello,
>
> In Derby 10.5 an in-memory back end, or storage engine, was included.
> It stores all the data in main memory, with the exception of
> derby.log. If this is news to you, and you want a quick intro to it,
> see [1] and [2].
>
> I'm trying to gather some feedback on whether the current
> implementation is found acceptable, or if there are additional
> features people would like to see. I expect some wishes to emerge, and
> I plan to record these on the wiki page [1]. The page can then be used
> to guide further work in this area.
>
> To start the discussion, I'll list some potential features and tasks.
> Feel free to comment on any one of them either by replying to this
> thread, or by adding your comments to [1]. It can be a +1 or -1 on the
> feature itself, a suggestion for a new feature, or details on what a
> feature should look like.
>
>
> * Documentation
> Must at least document the JDBC subsubprotocol, and also explain how
> to delete in-memory databases.
> If new features are added, these must be documented as well.
>
> * Deletion of in-memory databases
> Currently the only ways to delete an in-memory database are to restart
> the JVM or use a static method that isn't part of Derby's public API.
> A proper mechanism for deletion should be added.
>
> * Automatic deletion on database shutdown (or when last connection
> disconnects)
>
> * "Anonymous in-memory databases"
> A database which only the connection creating it can access, and when
> the connection goes away the database goes away.
>
> * Automatic persistence
> The database could be persisted to disk automatically based on certain
> criteria. The most obvious ones are perhaps on a fixed interval and on
> JVM shutdown.
>
> * Monitoring
> The most basic information is how many in-memory databases exist in
> the current JVM, and how big they are. How should this information be
> presented? Should it be available to anyone having a connection to the
> current JVM?
>
> * No derby.log
> Include a class in Derby that will discard everything written to
> derby.log.
>
>
> Thank you for your feedback,

[Lily Wei]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Rick:

     Thank you so much for sharing the information with the group.

>* It would be great to be able to bound the growth of the in-memory db

Is there a trend for need of in-memory db on JAVA world? Is it mainly for applications, i.e. ERP, CRM, SRM? What kind of solution JAVA can provide for smart device like iPhone, RIMM or Plam? i.e. Will JAVA play well with WindowMobile or Arnoid?

 

 

Thank you for shed the lights for us in advance,

Lily

---

From: Rick Hillegas <Richard.Hillegas@...>
To: Derby Discussion <derby-user@...>
Sent: Wednesday, September 9, 2009 11:13:05 AM
Subject: Re: Derby in-memory back end - where to go next?

Hi Kristian,

Here's another piece of feedback: Last night I gave an overview of Derby to the San Francisco Java User's Group. A developer asked whether the growth of the in-memory database could be bounded. He had a use case which we didn't explore in depth but which involved periodically truncating the database. I asked him to bring his requirements to the Derby user list so that we could feed them into your spec effort. Here are my takeaways:

* It would be great to be able to bound the growth of the in-memory db

* It would be great if the memory occupied by deleted records could be released

Thanks,
-Rick

Kristian Waagan wrote:

> Hello,
>
> In Derby 10.5 an in-memory back end, or storage engine, was included. It stores all the data in main memory, with the exception of derby.log. If this is news to you, and you want a quick intro to it, see [1] and [2].
>
> I'm trying to gather some feedback on whether the current implementation is found acceptable, or if there are additional features people would like to see. I expect some wishes to emerge, and I plan to record these on the wiki page [1]. The page can then be used to guide further work in this area.
>
> To start the discussion, I'll list some potential features and tasks. Feel free to comment on any one of them either by replying to this thread, or by adding your comments to [1]. It can be a +1 or -1 on the feature itself, a suggestion for a new feature, or details on what a feature should look like.
>
>
> * Documentation
> Must at least document the JDBC subsubprotocol, and also explain how to delete in-memory databases.
> If new features are added, these must be documented as well.
>
> * Deletion of in-memory databases
> Currently the only ways to delete an in-memory database are to restart the JVM or use a static method that isn't part of Derby's public API. A proper mechanism for deletion should be added.
>
> * Automatic deletion on database shutdown (or when last connection disconnects)
>
> * "Anonymous in-memory databases"
> A database which only the connection creating it can access, and when the connection goes away the database goes away.
>
> * Automatic persistence
> The database could be persisted to disk automatically based on certain criteria. The most obvious ones are perhaps on a fixed interval and on JVM shutdown.
>
> * Monitoring
> The most basic information is how many in-memory databases exist in the current JVM, and how big they are. How should this information be presented? Should it be available to anyone having a connection to the current JVM?
>
> * No derby.log
> Include a class in Derby that will discard everything written to derby.log.
>
>
> Thank you for your feedback,

[Rick Hillegas-2]

Hi Lily,

Some comments inline...

Lily Wei wrote:
>
> Hi Rick:
>
>      Thank you so much for sharing the information with the group.
>
> >* It would be great to be able to bound the growth of the in-memory db
>
> Is there a trend for need of in-memory db on JAVA world?
>
I find that this consistently generates a lot of discussion whenever I
talk about 10.5 features with users.
>
> Is it mainly for applications, i.e. ERP, CRM, SRM?
>
The top use-cases which keep coming up are:

o Middle-tier caching -- here people use Derby in the middle tier in
order to scale out access to a big back end like Oracle or DB2. Running
in memory makes this perform even better.

o Monitoring transient data streams - here you slice and dice the data
while the monitoring application is up but you don't necessarily need to
keep the data after the monitoring session ends.

o Test rigs -- here you can use Derby on your laptop to run regression
tests against an application which will run in production on a big back
end like Oracle or DB2; the rig is lightweight and cleans up after itself.
>
> What kind of solution JAVA can provide for smart device like iPhone,
> RIMM or Plam? i.e. Will JAVA play well with WindowMobile or Arnoid?
>
Our small device story is our ability to run on the stripped-down CDC
VM. Being able to run completely in memory gives this story extra appeal
too.

Thanks,
-Rick

>
>  
>
>  
>
> Thank you for shed the lights for us in advance,
>
> Lily
>
>
> *From:* Rick Hillegas <Richard.Hillegas@...>
> *To:* Derby Discussion <derby-user@...>
> *Sent:* Wednesday, September 9, 2009 11:13:05 AM
> *Subject:* Re: Derby in-memory back end - where to go next?
>
> Hi Kristian,
>
> Here's another piece of feedback: Last night I gave an overview of
> Derby to the San Francisco Java User's Group. A developer asked
> whether the growth of the in-memory database could be bounded. He had
> a use case which we didn't explore in depth but which involved
> periodically truncating the database. I asked him to bring his
> requirements to the Derby user list so that we could feed them into
> your spec effort. Here are my takeaways:
>
> * It would be great to be able to bound the growth of the in-memory db
>
> * It would be great if the memory occupied by deleted records could be
> released
>
> Thanks,
> -Rick
>
> Kristian Waagan wrote:
> > Hello,
> >
> > In Derby 10.5 an in-memory back end, or storage engine, was
> included. It stores all the data in main memory, with the exception of
> derby.log. If this is news to you, and you want a quick intro to it,
> see [1] and [2].
> >
> > I'm trying to gather some feedback on whether the current
> implementation is found acceptable, or if there are additional
> features people would like to see. I expect some wishes to emerge, and
> I plan to record these on the wiki page [1]. The page can then be used
> to guide further work in this area.
> >
> > To start the discussion, I'll list some potential features and
> tasks. Feel free to comment on any one of them either by replying to
> this thread, or by adding your comments to [1]. It can be a +1 or -1
> on the feature itself, a suggestion for a new feature, or details on
> what a feature should look like.
> >
> >
> > * Documentation
> > Must at least document the JDBC subsubprotocol, and also explain how
> to delete in-memory databases.
> > If new features are added, these must be documented as well.
> >
> > * Deletion of in-memory databases
> > Currently the only ways to delete an in-memory database are to
> restart the JVM or use a static method that isn't part of Derby's
> public API. A proper mechanism for deletion should be added.
> >
> > * Automatic deletion on database shutdown (or when last connection
> disconnects)
> >
> > * "Anonymous in-memory databases"
> > A database which only the connection creating it can access, and
> when the connection goes away the database goes away.
> >
> > * Automatic persistence
> > The database could be persisted to disk automatically based on
> certain criteria. The most obvious ones are perhaps on a fixed
> interval and on JVM shutdown.
> >
> > * Monitoring
> > The most basic information is how many in-memory databases exist in
> the current JVM, and how big they are. How should this information be
> presented? Should it be available to anyone having a connection to the
> current JVM?
> >
> > * No derby.log
> > Include a class in Derby that will discard everything written to
> derby.log.
> >
> >
> > Thank you for your feedback,
>
>

[Bryan Pendleton]

>> >* It would be great to be able to bound the growth of the in-memory db

What would happen when you reached that bound? Would it be similar to
running out of disk space for a disk-resident DB (i.e., your transaction
is aborted, and you get an exception)?

If so, then what do you do? Would the application then increase the bound,
and expect to resume adding data to their in-memory DB?

Or would the application then drop tables, delete rows from tables, etc.,
and then expect to resume adding data to their in-memory DB?

thanks,

bryan

[Rick Hillegas-2]

Bryan Pendleton wrote:

>>> >* It would be great to be able to bound the growth of the in-memory db
>
> What would happen when you reached that bound? Would it be similar to
> running out of disk space for a disk-resident DB (i.e., your transaction
> is aborted, and you get an exception)?
>
> If so, then what do you do? Would the application then increase the
> bound,
> and expect to resume adding data to their in-memory DB?
>
> Or would the application then drop tables, delete rows from tables, etc.,
> and then expect to resume adding data to their in-memory DB?

Hi Bryan,

Thanks for calling out these possibilities. I can imagine different
applications which answer that question differently. I hope that the guy
who requested this feature can come into the community and discuss his
use-case with us. I can imagine some users would want us to add the
ability to boost the db size on the fly and some users would want us to
add a bulk-truncate command.

Thanks,
-Rick
>
> thanks,
>
> bryan
>
>

[Lily Wei]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi Rick:

     I have some follow up questions.
Middle-tier caching, monitoring transient data streams and test rigs totally make sense.

Do you see any benchmark in turn of how derby helps these applications?

In aspect such as performance, totally memory consumption or reduce hardware cost?

 

     Do you see other embedded databases that also provide solution on the stripped-down CDC VM?

Do you have any data point for Derby?

 

Thank you so much for shed some lights for people like me,

Lily

---

From: Rick Hillegas <Richard.Hillegas@...>
To: Derby Discussion <derby-user@...>
Sent: Wednesday, September 9, 2009 2:01:01 PM
Subject: Re: Derby in-memory back end - where to go next?

Hi Lily,

Some comments inline...

Lily Wei wrote:
>
> Hi Rick:
>
>      Thank you so much for sharing the information with the group.
>
> >* It would be great to be able to bound the growth of the in-memory db
>
> Is there a trend for need of in-memory db on JAVA world?
>
I find that this consistently generates a lot of discussion whenever I talk about 10.5 features with users.
>
> Is it mainly for applications, i.e. ERP, CRM, SRM?
>
The top use-cases which keep coming up are:

o Middle-tier caching -- here people use Derby in the middle tier in order to scale out access to a big back end like Oracle or DB2. Running in memory makes this perform even better.

o Monitoring transient data streams - here you slice and dice the data while the monitoring application is up but you don't necessarily need to keep the data after the monitoring session ends.

o Test rigs -- here you can use Derby on your laptop to run regression tests against an application which will run in production on a big back end like Oracle or DB2; the rig is lightweight and cleans up after itself.
>
> What kind of solution JAVA can provide for smart device like iPhone, RIMM or Plam? i.e. Will JAVA play well with WindowMobile or Arnoid?
>
Our small device story is our ability to run on the stripped-down CDC VM. Being able to run completely in memory gives this story extra appeal too.

Thanks,
-Rick

>
> 
> 
> Thank you for shed the lights for us in advance,
>
> Lily
>
>
> *From:* Rick Hillegas <Richard.Hillegas@...>
> *To:* Derby Discussion <derby-user@...>
> *Sent:* Wednesday, September 9, 2009 11:13:05 AM
> *Subject:* Re: Derby in-memory back end - where to go next?
>
> Hi Kristian,
>
> Here's another piece of feedback: Last night I gave an overview of Derby to the San Francisco Java User's Group. A developer asked whether the growth of the in-memory database could be bounded. He had a use case which we didn't explore in depth but which involved periodically truncating the database. I asked him to bring his requirements to the Derby user list so that we could feed them into your spec effort. Here are my takeaways:
>
> * It would be great to be able to bound the growth of the in-memory db
>
> * It would be great if the memory occupied by deleted records could be released
>
> Thanks,
> -Rick
>
> Kristian Waagan wrote:
> > Hello,
> >
> > In Derby 10.5 an in-memory back end, or storage engine, was included. It stores all the data in main memory, with the exception of derby.log. If this is news to you, and you want a quick intro to it, see [1] and [2].
> >
> > I'm trying to gather some feedback on whether the current implementation is found acceptable, or if there are additional features people would like to see. I expect some wishes to emerge, and I plan to record these on the wiki page [1]. The page can then be used to guide further work in this area.
> >
> > To start the discussion, I'll list some potential features and tasks. Feel free to comment on any one of them either by replying to this thread, or by adding your comments to [1]. It can be a +1 or -1 on the feature itself, a suggestion for a new feature, or details on what a feature should look like.
> >
> >
> > * Documentation
> > Must at least document the JDBC subsubprotocol, and also explain how to delete in-memory databases.
> > If new features are added, these must be documented as well.
> >
> > * Deletion of in-memory databases
> > Currently the only ways to delete an in-memory database are to restart the JVM or use a static method that isn't part of Derby's public API. A proper mechanism for deletion should be added.
> >
> > * Automatic deletion on database shutdown (or when last connection disconnects)
> >
> > * "Anonymous in-memory databases"
> > A database which only the connection creating it can access, and when the connection goes away the database goes away.
> >
> > * Automatic persistence
> > The database could be persisted to disk automatically based on certain criteria. The most obvious ones are perhaps on a fixed interval and on JVM shutdown.
> >
> > * Monitoring
> > The most basic information is how many in-memory databases exist in the current JVM, and how big they are. How should this information be presented? Should it be available to anyone having a connection to the current JVM?
> >
> > * No derby.log
> > Include a class in Derby that will discard everything written to derby.log.
> >
> >
> > Thank you for your feedback,
>
>

[GregB]

Kristian,

Quick question on the in-memory DB.  How do you specify the database should be in-memory for an instance of a EmbeddedConnectionPoolDataSource?

Example url - jdbc:derby:memory:testdb;create=true

Connection pool -
EmbeddedConnectionPoolDataSource ds = new EmbeddedConnectionPoolDataSource();
ds.setDatabaseName("testdb");
ds.setCreateDatabase("create");
// Now what to specify in-memory?


Thanks,

Greg

Kristian Waagan-4 wrote:

Hello,

In Derby 10.5 an in-memory back end, or storage engine, was included. It
stores all the data in main memory, with the exception of derby.log. If
this is news to you, and you want a quick intro to it, see [1] and [2].

I'm trying to gather some feedback on whether the current implementation
is found acceptable, or if there are additional features people would
like to see. I expect some wishes to emerge, and I plan to record these
on the wiki page [1]. The page can then be used to guide further work in
this area.

To start the discussion, I'll list some potential features and tasks.
Feel free to comment on any one of them either by replying to this
thread, or by adding your comments to [1]. It can be a +1 or -1 on the
feature itself, a suggestion for a new feature, or details on what a
feature should look like.


* Documentation
Must at least document the JDBC subsubprotocol, and also explain how to
delete in-memory databases.
If new features are added, these must be documented as well.

* Deletion of in-memory databases
Currently the only ways to delete an in-memory database are to restart
the JVM or use a static method that isn't part of Derby's public API. A
proper mechanism for deletion should be added.

* Automatic deletion on database shutdown (or when last connection
disconnects)

* "Anonymous in-memory databases"
A database which only the connection creating it can access, and when
the connection goes away the database goes away.

* Automatic persistence
The database could be persisted to disk automatically based on certain
criteria. The most obvious ones are perhaps on a fixed interval and on
JVM shutdown.

* Monitoring
The most basic information is how many in-memory databases exist in the
current JVM, and how big they are. How should this information be
presented? Should it be available to anyone having a connection to the
current JVM?

* No derby.log
Include a class in Derby that will discard everything written to derby.log.


Thank you for your feedback,
--
Kristian

[1] http://wiki.apache.org/db-derby/InMemoryBackEndPrimer
[2] http://blogs.sun.com/kah/

[Knut Anders Hatlen]

GregB <gregburrow@...> writes:

> Kristian,
>
> Quick question on the in-memory DB.  How do you specify the database should
> be in-memory for an instance of a EmbeddedConnectionPoolDataSource?
>
> Example url - jdbc:derby:memory:testdb;create=true
>
> Connection pool -
> EmbeddedConnectionPoolDataSource ds = new
> EmbeddedConnectionPoolDataSource();
> ds.setDatabaseName("testdb");
> ds.setCreateDatabase("create");
> // Now what to specify in-memory?

I haven't tested it, but I think ds.setDatabaseName("memory:testdb")
should do the trick.

--
Knut Anders

[Rick Hillegas-2]

Hi Lily,

Some responses inline...

Lily Wei wrote:
>
> Hi Rick:
>
>      I have some follow up questions.
> Middle-tier caching, monitoring transient data streams and test rigs
> totally make sense.
>
> Do you see any benchmark in turn of how derby helps these applications?
>
I don't think we've published any figures on the performance boost you
get from running in memory. My anecdotal recollection is that you see a
significant boost once you've gotten past database creation. Kristian
has done the most extensive testing and may have some figures that he
can share. Unfortunately, he suffered an accident earlier this week and
is up on the blocks for a while.
>
> In aspect such as performance, totally memory consumption or reduce
> hardware cost?
>
>  
>
>      Do you see other embedded databases that also provide solution on
> the stripped-down CDC VM?
>
I don't think that H2 or HSQLDB run on CDC.

Regards,
-Rick

>
> Do you have any data point for Derby?
>
>  
>
> Thank you so much for shed some lights for people like me,
>
> Lily
>
>
>
> *From:* Rick Hillegas <Richard.Hillegas@...>
> *To:* Derby Discussion <derby-user@...>
> *Sent:* Wednesday, September 9, 2009 2:01:01 PM
> *Subject:* Re: Derby in-memory back end - where to go next?
>
> Hi Lily,
>
> Some comments inline...
>
> Lily Wei wrote:
> >
> > Hi Rick:
> >
> >      Thank you so much for sharing the information with the group.
> >
> > >* It would be great to be able to bound the growth of the in-memory db
> >
> > Is there a trend for need of in-memory db on JAVA world?
> >
> I find that this consistently generates a lot of discussion whenever I
> talk about 10.5 features with users.
> >
> > Is it mainly for applications, i.e. ERP, CRM, SRM?
> >
> The top use-cases which keep coming up are:
>
> o Middle-tier caching -- here people use Derby in the middle tier in
> order to scale out access to a big back end like Oracle or DB2.
> Running in memory makes this perform even better.
>
> o Monitoring transient data streams - here you slice and dice the data
> while the monitoring application is up but you don't necessarily need
> to keep the data after the monitoring session ends.
>
> o Test rigs -- here you can use Derby on your laptop to run regression
> tests against an application which will run in production on a big
> back end like Oracle or DB2; the rig is lightweight and cleans up
> after itself.
> >
> > What kind of solution JAVA can provide for smart device like iPhone,
> RIMM or Plam? i.e. Will JAVA play well with WindowMobile or Arnoid?
> >
> Our small device story is our ability to run on the stripped-down CDC
> VM. Being able to run completely in memory gives this story extra
> appeal too.
>
> Thanks,
> -Rick
> >
> >
> >
> > Thank you for shed the lights for us in advance,
> >
> > Lily
> >
> >
> > *From:* Rick Hillegas <Richard.Hillegas@...
> <mailto:Richard.Hillegas@...>>
> > *To:* Derby Discussion <derby-user@...
> <mailto:derby-user@...>>
> > *Sent:* Wednesday, September 9, 2009 11:13:05 AM
> > *Subject:* Re: Derby in-memory back end - where to go next?
> >
> > Hi Kristian,
> >
> > Here's another piece of feedback: Last night I gave an overview of
> Derby to the San Francisco Java User's Group. A developer asked
> whether the growth of the in-memory database could be bounded. He had
> a use case which we didn't explore in depth but which involved
> periodically truncating the database. I asked him to bring his
> requirements to the Derby user list so that we could feed them into
> your spec effort. Here are my takeaways:
> >
> > * It would be great to be able to bound the growth of the in-memory db
> >
> > * It would be great if the memory occupied by deleted records could
> be released
> >
> > Thanks,
> > -Rick
> >
> > Kristian Waagan wrote:
> > > Hello,
> > >
> > > In Derby 10.5 an in-memory back end, or storage engine, was
> included. It stores all the data in main memory, with the exception of
> derby.log. If this is news to you, and you want a quick intro to it,
> see [1] and [2].
> > >
> > > I'm trying to gather some feedback on whether the current
> implementation is found acceptable, or if there are additional
> features people would like to see. I expect some wishes to emerge, and
> I plan to record these on the wiki page [1]. The page can then be used
> to guide further work in this area.
> > >
> > > To start the discussion, I'll list some potential features and
> tasks. Feel free to comment on any one of them either by replying to
> this thread, or by adding your comments to [1]. It can be a +1 or -1
> on the feature itself, a suggestion for a new feature, or details on
> what a feature should look like.
> > >
> > >
> > > * Documentation
> > > Must at least document the JDBC subsubprotocol, and also explain
> how to delete in-memory databases.
> > > If new features are added, these must be documented as well.
> > >
> > > * Deletion of in-memory databases
> > > Currently the only ways to delete an in-memory database are to
> restart the JVM or use a static method that isn't part of Derby's
> public API. A proper mechanism for deletion should be added.
> > >
> > > * Automatic deletion on database shutdown (or when last connection
> disconnects)
> > >
> > > * "Anonymous in-memory databases"
> > > A database which only the connection creating it can access, and
> when the connection goes away the database goes away.
> > >
> > > * Automatic persistence
> > > The database could be persisted to disk automatically based on
> certain criteria. The most obvious ones are perhaps on a fixed
> interval and on JVM shutdown.
> > >
> > > * Monitoring
> > > The most basic information is how many in-memory databases exist
> in the current JVM, and how big they are. How should this information
> be presented? Should it be available to anyone having a connection to
> the current JVM?
> > >
> > > * No derby.log
> > > Include a class in Derby that will discard everything written to
> derby.log.
> > >
> > >
> > > Thank you for your feedback,
> >
> >
>
>

[Kristian Waagan-4]

Rick Hillegas wrote:

> Hi Lily,
>
> Some responses inline...
>
> Lily Wei wrote:
>>
>> Hi Rick:
>>
>>      I have some follow up questions.
>> Middle-tier caching, monitoring transient data streams and test rigs
>> totally make sense.
>>
>> Do you see any benchmark in turn of how derby helps these applications?
>>
> I don't think we've published any figures on the performance boost you
> get from running in memory. My anecdotal recollection is that you see
> a significant boost once you've gotten past database creation.
> Kristian has done the most extensive testing and may have some figures
> that he can share. Unfortunately, he suffered an accident earlier this
> week and is up on the blocks for a while.

Hello Rick and Lily,

The performance benefit you'll see with the in-memory back end is highly
dependent on the load and the underlying disk subsystem.
For write intensive loads the boost can be in orders of magnitude.
For read intensive loads the boost can be close to zero.

If you have a read-only database, it may be better in some cases to keep
the database on disk, maximize the page cache size and then prime the
cache (pulling all pages into the cache).
The downside of using the in-memory back end in such a scenario, is that
some of the data will be stored twice: once in the "virtual in-memory
file system" and once in the page cache. For the same reason, you should
tweak the page cache size accordingly to your amount of data and heap
size. Minimizing the page cache  (i.e. allowing only 40 pages) to avoid
the "data duplication" problem is not a good idea for optimal performance...
For some more information about the effects of page cache size and page
size, see [1]. It is really a comparison between two implementations of
an in-memory back end, but closer to the end of the document there are
some relevant experiments.

Unfortunately I'm unable to find the numbers I had comparing the disk
based back end with the in-memory back end.
If anyone wants some hard numbers, they can try running the various
performance clients found in the source code repository (under
trunk/testing/.../perf/clients). The simplest ones are the single record
operation clients and the bank_tx load.

In my opinion, the primary use cases for the current in-memory back end
are testing and development. In the next release it may be better suited
for storing purely transient data in a production environment as well
(with a proper delete mechanism and maybe a size limit feature).


--
Kristian

[1]
https://issues.apache.org/jira/secure/attachment/12400859/derby-646-performance_comparison_1a.txt

>>
>> In aspect such as performance, totally memory consumption or reduce
>> hardware cost?
>>
>>  
>>
>>      Do you see other embedded databases that also provide solution
>> on the stripped-down CDC VM?
>>
> I don't think that H2 or HSQLDB run on CDC.
>
> Regards,
> -Rick
>>
>> Do you have any data point for Derby?
>>
>>  
>>
>> Thank you so much for shed some lights for people like me,
>>
>> Lily
>>
>>
>>
>> *From:* Rick Hillegas <Richard.Hillegas@...>
>> *To:* Derby Discussion <derby-user@...>
>> *Sent:* Wednesday, September 9, 2009 2:01:01 PM
>> *Subject:* Re: Derby in-memory back end - where to go next?
>>
>> Hi Lily,
>>
>> Some comments inline...
>>
>> Lily Wei wrote:
>> >
>> > Hi Rick:
>> >
>> >      Thank you so much for sharing the information with the group.
>> >
>> > >* It would be great to be able to bound the growth of the
>> in-memory db
>> >
>> > Is there a trend for need of in-memory db on JAVA world?
>> >
>> I find that this consistently generates a lot of discussion whenever
>> I talk about 10.5 features with users.
>> >
>> > Is it mainly for applications, i.e. ERP, CRM, SRM?
>> >
>> The top use-cases which keep coming up are:
>>
>> o Middle-tier caching -- here people use Derby in the middle tier in
>> order to scale out access to a big back end like Oracle or DB2.
>> Running in memory makes this perform even better.
>>
>> o Monitoring transient data streams - here you slice and dice the
>> data while the monitoring application is up but you don't necessarily
>> need to keep the data after the monitoring session ends.
>>
>> o Test rigs -- here you can use Derby on your laptop to run
>> regression tests against an application which will run in production
>> on a big back end like Oracle or DB2; the rig is lightweight and
>> cleans up after itself.
>> >
>> > What kind of solution JAVA can provide for smart device like
>> iPhone, RIMM or Plam? i.e. Will JAVA play well with WindowMobile or
>> Arnoid?
>> >
>> Our small device story is our ability to run on the stripped-down CDC
>> VM. Being able to run completely in memory gives this story extra
>> appeal too.
>>
>> Thanks,
>> -Rick
>> >
>> > > > Thank you for shed the lights for us in advance,
>> >
>> > Lily
>> >
>> >
>> > *From:* Rick Hillegas <Richard.Hillegas@...
>> <mailto:Richard.Hillegas@...>>
>> > *To:* Derby Discussion <derby-user@...
>> <mailto:derby-user@...>>
>> > *Sent:* Wednesday, September 9, 2009 11:13:05 AM
>> > *Subject:* Re: Derby in-memory back end - where to go next?
>> >
>> > Hi Kristian,
>> >
>> > Here's another piece of feedback: Last night I gave an overview of
>> Derby to the San Francisco Java User's Group. A developer asked
>> whether the growth of the in-memory database could be bounded. He had
>> a use case which we didn't explore in depth but which involved
>> periodically truncating the database. I asked him to bring his
>> requirements to the Derby user list so that we could feed them into
>> your spec effort. Here are my takeaways:
>> >
>> > * It would be great to be able to bound the growth of the in-memory db
>> >
>> > * It would be great if the memory occupied by deleted records could
>> be released
>> >
>> > Thanks,
>> > -Rick
>> >
>> > Kristian Waagan wrote:
>> > > Hello,
>> > >
>> > > In Derby 10.5 an in-memory back end, or storage engine, was
>> included. It stores all the data in main memory, with the exception
>> of derby.log. If this is news to you, and you want a quick intro to
>> it, see [1] and [2].
>> > >
>> > > I'm trying to gather some feedback on whether the current
>> implementation is found acceptable, or if there are additional
>> features people would like to see. I expect some wishes to emerge,
>> and I plan to record these on the wiki page [1]. The page can then be
>> used to guide further work in this area.
>> > >
>> > > To start the discussion, I'll list some potential features and
>> tasks. Feel free to comment on any one of them either by replying to
>> this thread, or by adding your comments to [1]. It can be a +1 or -1
>> on the feature itself, a suggestion for a new feature, or details on
>> what a feature should look like.
>> > >
>> > >
>> > > * Documentation
>> > > Must at least document the JDBC subsubprotocol, and also explain
>> how to delete in-memory databases.
>> > > If new features are added, these must be documented as well.
>> > >
>> > > * Deletion of in-memory databases
>> > > Currently the only ways to delete an in-memory database are to
>> restart the JVM or use a static method that isn't part of Derby's
>> public API. A proper mechanism for deletion should be added.
>> > >
>> > > * Automatic deletion on database shutdown (or when last
>> connection disconnects)
>> > >
>> > > * "Anonymous in-memory databases"
>> > > A database which only the connection creating it can access, and
>> when the connection goes away the database goes away.
>> > >
>> > > * Automatic persistence
>> > > The database could be persisted to disk automatically based on
>> certain criteria. The most obvious ones are perhaps on a fixed
>> interval and on JVM shutdown.
>> > >
>> > > * Monitoring
>> > > The most basic information is how many in-memory databases exist
>> in the current JVM, and how big they are. How should this information
>> be presented? Should it be available to anyone having a connection to
>> the current JVM?
>> > >
>> > > * No derby.log
>> > > Include a class in Derby that will discard everything written to
>> derby.log.
>> > >
>> > >
>> > > Thank you for your feedback,
>> >
>> >
>>
>>
>

[Lily Wei]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Thank you so much for the information.

With more smart phone on the market, I think people will use the in-memory feature more and appricate the performance improvment.

 

+1

 

Thanks again,

Lily

---

From: Kristian Waagan <Kristian.Waagan@...>
To: Derby Discussion <derby-user@...>
Sent: Friday, September 25, 2009 2:45:53 AM
Subject: Re: Derby in-memory back end - where to go next?

Rick Hillegas wrote:

> Hi Lily,
>
> Some responses inline...
>
> Lily Wei wrote:
>>
>> Hi Rick:
>>
>>      I have some follow up questions.
>> Middle-tier caching, monitoring transient data streams and test rigs totally make sense.
>>
>> Do you see any benchmark in turn of how derby helps these applications?
>>
> I don't think we've published any figures on the performance boost you get from running in memory. My anecdotal recollection is that you see a significant boost once you've gotten past database creation. Kristian has done the most extensive testing and may have some figures that he can share. Unfortunately, he suffered an accident earlier this week and is up on the blocks for a while.

Hello Rick and Lily,

The performance benefit you'll see with the in-memory back end is highly dependent on the load and the underlying disk subsystem..
For write intensive loads the boost can be in orders of magnitude.
For read intensive loads the boost can be close to zero.

If you have a read-only database, it may be better in some cases to keep the database on disk, maximize the page cache size and then prime the cache (pulling all pages into the cache).
The downside of using the in-memory back end in such a scenario, is that some of the data will be stored twice: once in the "virtual in-memory file system" and once in the page cache. For the same reason, you should tweak the page cache size accordingly to your amount of data and heap size. Minimizing the page cache  (i.e. allowing only 40 pages) to avoid the "data duplication" problem is not a good idea for optimal performance...
For some more information about the effects of page cache size and page size, see [1]. It is really a comparison between two implementations of an in-memory back end, but closer to the end of the document there are some relevant experiments.

Unfortunately I'm unable to find the numbers I had comparing the disk based back end with the in-memory back end.
If anyone wants some hard numbers, they can try running the various performance clients found in the source code repository (under trunk/testing/.../perf/clients). The simplest ones are the single record operation clients and the bank_tx load.

In my opinion, the primary use cases for the current in-memory back end are testing and development. In the next release it may be better suited for storing purely transient data in a production environment as well (with a proper delete mechanism and maybe a size limit feature).


-- Kristian

[1] https://issues.apache.org/jira/secure/attachment/12400859/derby-646-performance_comparison_1a.txt

>>
>> In aspect such as performance, totally memory consumption or reduce hardware cost?
>>
>> 
>>      Do you see other embedded databases that also provide solution on the stripped-down CDC VM?
>>
> I don't think that H2 or HSQLDB run on CDC.
>
> Regards,
> -Rick
>>
>> Do you have any data point for Derby?
>>
>> 
>> Thank you so much for shed some lights for people like me,
>>
>> Lily
>>
>>
>>
>> *From:* Rick Hillegas <Richard.Hillegas@...>
>> *To:* Derby Discussion <derby-user@...>
>> *Sent:* Wednesday, September 9, 2009 2:01:01 PM
>> *Subject:* Re: Derby in-memory back end - where to go next?
>>
>> Hi Lily,
>>
>> Some comments inline...
>>
>> Lily Wei wrote:
>> >
>> > Hi Rick:
>> >
>> >      Thank you so much for sharing the information with the group.
>> >
>> > >* It would be great to be able to bound the growth of the in-memory db
>> >
>> > Is there a trend for need of in-memory db on JAVA world?
>> >
>> I find that this consistently generates a lot of discussion whenever I talk about 10.5 features with users.
>> >
>> > Is it mainly for applications, i.e. ERP, CRM, SRM?
>> >
>> The top use-cases which keep coming up are:
>>
>> o Middle-tier caching -- here people use Derby in the middle tier in order to scale out access to a big back end like Oracle or DB2. Running in memory makes this perform even better.
>>
>> o Monitoring transient data streams - here you slice and dice the data while the monitoring application is up but you don't necessarily need to keep the data after the monitoring session ends.
>>
>> o Test rigs -- here you can use Derby on your laptop to run regression tests against an application which will run in production on a big back end like Oracle or DB2; the rig is lightweight and cleans up after itself.
>> >
>> > What kind of solution JAVA can provide for smart device like iPhone, RIMM or Plam? i.e. Will JAVA play well with WindowMobile or Arnoid?
>> >
>> Our small device story is our ability to run on the stripped-down CDC VM. Being able to run completely in memory gives this story extra appeal too.
>>
>> Thanks,
>> -Rick
>> >
>> > > > Thank you for shed the lights for us in advance,
>> >
>> > Lily
>> >
>> >
>> > *From:* Rick Hillegas <Richard.Hillegas@... <mailto:Richard.Hillegas@...>>
>> > *To:* Derby Discussion <derby-user@... <mailto:derby-user@...>>
>> > *Sent:* Wednesday, September 9, 2009 11:13:05 AM
>> > *Subject:* Re: Derby in-memory back end - where to go next?
>> >
>> > Hi Kristian,
>> >
>> > Here's another piece of feedback: Last night I gave an overview of Derby to the San Francisco Java User's Group. A developer asked whether the growth of the in-memory database could be bounded. He had a use case which we didn't explore in depth but which involved periodically truncating the database. I asked him to bring his requirements to the Derby user list so that we could feed them into your spec effort. Here are my takeaways:
>> >
>> > * It would be great to be able to bound the growth of the in-memory db
>> >
>> > * It would be great if the memory occupied by deleted records could be released
>> >
>> > Thanks,
>> > -Rick
>> >
>> > Kristian Waagan wrote:
>> > > Hello,
>> > >
>> > > In Derby 10.5 an in-memory back end, or storage engine, was included. It stores all the data in main memory, with the exception of derby.log. If this is news to you, and you want a quick intro to it, see [1] and [2].
>> > >
>> > > I'm trying to gather some feedback on whether the current implementation is found acceptable, or if there are additional features people would like to see. I expect some wishes to emerge, and I plan to record these on the wiki page [1]. The page can then be used to guide further work in this area.
>> > >
>> > > To start the discussion, I'll list some potential features and tasks. Feel free to comment on any one of them either by replying to this thread, or by adding your comments to [1]. It can be a +1 or -1 on the feature itself, a suggestion for a new feature, or details on what a feature should look like.
>> > >
>> > >
>> > > * Documentation
>> > > Must at least document the JDBC subsubprotocol, and also explain how to delete in-memory databases.
>> > > If new features are added, these must be documented as well.
>> > >
>> > > * Deletion of in-memory databases
>> > > Currently the only ways to delete an in-memory database are to restart the JVM or use a static method that isn't part of Derby's public API. A proper mechanism for deletion should be added.
>> > >
>> > > * Automatic deletion on database shutdown (or when last connection disconnects)
>> > >
>> > > * "Anonymous in-memory databases"
>> > > A database which only the connection creating it can access, and when the connection goes away the database goes away.
>> > >
>> > > * Automatic persistence
>> > > The database could be persisted to disk automatically based on certain criteria. The most obvious ones are perhaps on a fixed interval and on JVM shutdown.
>> > >
>> > > * Monitoring
>> > > The most basic information is how many in-memory databases exist in the current JVM, and how big they are. How should this information be presented? Should it be available to anyone having a connection to the current JVM?
>> > >
>> > > * No derby.log
>> > > Include a class in Derby that will discard everything written to derby.log.
>> > >
>> > >
>> > > Thank you for your feedback,
>> >
>> >
>>
>>
>

[Kristian Waagan-4]

Rick Hillegas wrote:

> Hi Kristian,
>
> Here's another piece of feedback: Last night I gave an overview of
> Derby to the San Francisco Java User's Group. A developer asked
> whether the growth of the in-memory database could be bounded. He had
> a use case which we didn't explore in depth but which involved
> periodically truncating the database. I asked him to bring his
> requirements to the Derby user list so that we could feed them into
> your spec effort. Here are my takeaways:
>
> * It would be great to be able to bound the growth of the in-memory db
>
> * It would be great if the memory occupied by deleted records could be
> released

Hi Rick,

I'm not quite sure how easy it is to implement the second feature. I see
a possibility if the in-memory back end recognizes the Derby binary
format, but removing the deleted records at the storage layer will have
performance implications. We also have to deal with the checksums
somehow... Maybe a quick prototype could tell us if this is feasible at all.

In addition to the actual "early deletion" of records, which would
probably happen after Derby has written out a complete page, a kind of
sparse byte array implementation would be needed. An easier options is
using
These two features could be implemented individually (the spare byte
array implementation first). Is perhaps java.util.zip.Deflater an option?

Regarding the processing, we may be able to utilize the field storing
the number of deleted records in the page header to decide if we need to
scan the page for deleted records, and use the slot table to calculate
the amount of free space on the page. I don't know what is the best
approach regarding performance; use the existing "meta-structures" to
decide whether to scan, or just scan the byte array unconditionally.

And finally, is this added complexity worth the gain?
Is there an easier way?


Regards,
--
Kristian

>
> Thanks,
> -Rick
>
> Kristian Waagan wrote:
>> Hello,
>>
>> In Derby 10.5 an in-memory back end, or storage engine, was included.
>> It stores all the data in main memory, with the exception of
>> derby.log. If this is news to you, and you want a quick intro to it,
>> see [1] and [2].
>>
>> I'm trying to gather some feedback on whether the current
>> implementation is found acceptable, or if there are additional
>> features people would like to see. I expect some wishes to emerge,
>> and I plan to record these on the wiki page [1]. The page can then be
>> used to guide further work in this area.
>>
>> To start the discussion, I'll list some potential features and tasks.
>> Feel free to comment on any one of them either by replying to this
>> thread, or by adding your comments to [1]. It can be a +1 or -1 on
>> the feature itself, a suggestion for a new feature, or details on
>> what a feature should look like.
>>
>>
>> * Documentation
>> Must at least document the JDBC subsubprotocol, and also explain how
>> to delete in-memory databases.
>> If new features are added, these must be documented as well.
>>
>> * Deletion of in-memory databases
>> Currently the only ways to delete an in-memory database are to
>> restart the JVM or use a static method that isn't part of Derby's
>> public API. A proper mechanism for deletion should be added.
>>
>> * Automatic deletion on database shutdown (or when last connection
>> disconnects)
>>
>> * "Anonymous in-memory databases"
>> A database which only the connection creating it can access, and when
>> the connection goes away the database goes away.
>>
>> * Automatic persistence
>> The database could be persisted to disk automatically based on
>> certain criteria. The most obvious ones are perhaps on a fixed
>> interval and on JVM shutdown.
>>
>> * Monitoring
>> The most basic information is how many in-memory databases exist in
>> the current JVM, and how big they are. How should this information be
>> presented? Should it be available to anyone having a connection to
>> the current JVM?
>>
>> * No derby.log
>> Include a class in Derby that will discard everything written to
>> derby.log.
>>
>>
>> Thank you for your feedback,
>

[Kristian Waagan-4]

Kristian Waagan wrote:

> Hello,
>
> In Derby 10.5 an in-memory back end, or storage engine, was included.
> It stores all the data in main memory, with the exception of
> derby.log. If this is news to you, and you want a quick intro to it,
> see [1] and [2].
>
> I'm trying to gather some feedback on whether the current
> implementation is found acceptable, or if there are additional
> features people would like to see. I expect some wishes to emerge, and
> I plan to record these on the wiki page [1]. The page can then be used
> to guide further work in this area.

Hello all,

Thanks to everyone that chimed in on this thread earlier.
I have now updated the wiki page with some more information. It is very
crude, I just wanted to get something out before I leave for two weeks
of vacation.
Feel free to read through the section "Future features" and give your
feedback on this thread.
Does the suggested features look reasonable?
Has something really important (for you) been left out?

When I get back, I think I'll start working on the new delete mechanism
and maybe the monitoring. Besides from that, documentation must be added.
If time permits, or somebody else wants to put down some effort, maybe
one or more of the remaining features can be added.
In addition to the wiki page, I suppose a separate Jira will be created
for each feature. More detailed specifications will be attached to the
individual Jiras.

The wiki page is at: http://wiki.apache.org/db-derby/InMemoryBackEndPrimer


Thanks,
--
Kristian

[ snip ]

[Rick Hillegas-2]

Kristian Waagan wrote:
> Kristian Waagan wrote:
> ...
> The wiki page is at:
> http://wiki.apache.org/db-derby/InMemoryBackEndPrimer
>
Thanks for pulling this spec together, Kristian. Some comments:

Proper Delete Mechanism

I agree that option (1) looks fine: the mechanism for deleting a
database then looks like the mechanisms for creating and halting
databases. I think it's ok to restrict this privilege to the DBO. For
most use-cases that will probably be fine. There might be an oddball
case of a System Administrator needing to delete an in-memory database
without bouncing the VM. This sounds like something we could add when we
add system privileges.


Automatic database persistence on JVM shutdown

I think we should say something about how the user experiences this feature:

1) It makes sense to me that snapshot-on-exit is an attribute of a
database which is set when the database is initially booted. Can you
change the setting and if so, who can do this and how?

2) It would be nice to say a few words about how you reboot one of these
snapshots. Is this done via the restoreFrom connection attribute? Do you
need to specify snapshot-on-exit when you reboot one of these in-memory
databases or is it assumed that if you reboot one of these in-memory
databases then you want the original exit behavior to continue in force?


Automatic database persistence on database shutdown

Similar questions about this feature. In addition, it becomes possible
for a database to have two states: the in-memory version and the
snapshot created when the database was shutdown.

3) If you reboot the database, do you get the version that's still in
memory? In the meantime, someone could have booted the snapshot as a
persistent database, fiddled with it, and re-written the snapshot.

4) Can you reboot the database with a restoreFrom setting? Does this
override the version in memory or does this raise an error?

I think there's definitely an opportunity to share code between
snapshot-on-exit and snapshot-on-shutdown. But they look like separate
features to me. My gut feeling is that snapshot-on-shutdown is more
important than snapshot-on-exit and has fewer problems. I think that
most people who need to persist an in-memory database could write their
application in such a way that they could gracefully park the database
before bringing down the vm. The more I think about it, the more that
shutdown-on-exit seems to refer to an undefined state of the database
and cannot be guaranteed to work properly.

Thanks,
-Rick