RFC: Major changes in ODS 12

[Dmitry Yemanov]

All,

Below is the initial proposal (to be corrected/extended by others) that
describes the physical ODS changes we'd like to see in ODS 12.

   Header page

     * rework implementation ID to simplify the porting
       (split into endianness and swap_double flags?)

     * deprecate page buffers and sweep interval
       (in favor of the per-database values defined in .conf)

   Pointer page / data page

     * add "swept" flag
      (allows to skip pages that have no dead or backversions,
       thus improving the GC / sweep performance)

   Page inventory page

     * use its own field instead of common "pag::reserved"
       (related to the misuse by PAG_allocate() happened in ODS 11.1)

   New page type (bitmap?) to link SCNs to page numbers.
     This may noticeable improve the incremental NBackup performance,
     avoiding reading pages not changed since the last backup.

   Switch from SLONG to ULONG for transaction numbers and page numbers.

   Extend blob length field from ULONG to UINT64
   (and maybe allow blob levels higher than 2?).

   Consider better alternative (instead of SQZ) for record compression.
   Tests are required. If not found, extend SQZ for sequences longer
   than 127 bytes.

   Consider better packing of index key data (record numbers)
   and compound index keys (described by Jim in fb-architect).
   Tests are required.

   Extend the ACL format to cover new permission types and objects.

Additions are welcome, as well as questions/comments re. the
aforementioned items.


Dmitry

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Dimitry Sibiryakov-3]

>    Consider better alternative (instead of SQZ) for record compression.
>    Tests are required. If not found, extend SQZ for sequences longer
>    than 127 bytes.

   What's the purpose of this change? Smaller size, faster compression
or what?

> Additions are welcome, as well as questions/comments re. the
> aforementioned items.

   Add "Min transaction number" field to header to allow to drop old
TIPs and/or reuse transactions numbers to remove limit of transactions
in database.
   Change offset fields from USHORT to ULONG to allow page sizes >64k.
Change meaning of hdr_page_size from number of bytes to number of
kilobytes for the same purpose.
   I'm not sure, but, perhaps, reference counter for BLOBs could be
useful in the future.

   SY, SD.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Dimitry Sibiryakov-3]

> Additions are welcome

   Forgot to add: change record size from USHORT to ULONG to remove 64k
limit for record length.

   SY, SD.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Rustam Gadjimuradov]

Dmitry Yemanov>
>     * deprecate page buffers and sweep interval
>       (in favor of the per-database values defined in .conf)

One conf-file for all databases (i.e. current firebird.conf)
or per-database.conf file (i.e. my_db.conf) ? May be
this was already discussed early (i don't remeber) ?

WBR, GR.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Mark Rotteveel]

>      * deprecate page buffers and sweep interval
>        (in favor of the per-database values defined in .conf)

What is the rationale behind this proposal? I have the feeling that moving this configuration to a separate conf file will make (remote) management of these configuration options harder, not simpler; and will place a burden on the systems administrator instead of the database owner.

If these options are moved to a separate file (or database?) they should also be remotely configurable by the database owner, not just by manually editting a conf file (manually editing conf files IMHO should be a last resort, and only for very low-level configuration).

I would even like to see that Firebird eventually deprecates support for loose, unmanaged database-files (except for embedded) and move to a central repository of databases with extended management and configuration options. So database are (by default) only accessible through their alias (not the full filename). This would make management and naming of various databases more like Oracle, MS SQL Server etc.

Mark



--
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Roman Rokytskyy-2]

> Additions are welcome, as well as questions/comments re. the
> aforementioned items.

Regarding the schema support. If the ODS change is needed here, please
consider this feature as well.

Roman

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Claudio Valderrama C.]

> -----Original Message-----
> From: Dmitry Yemanov [mailto:firebird2@...]
> Sent: Lunes, 26 de Octubre de 2009 5:52
>
> Below is the initial proposal (to be corrected/extended by
> others) that
> describes the physical ODS changes we'd like to see in ODS 12.
>
>    Header page
>
>      * rework implementation ID to simplify the porting
>        (split into endianness and swap_double flags?)

Centralize that thing in a single place if possible, intead of several files
that have to be changed now.


>      * deprecate page buffers and sweep interval
>        (in favor of the per-database values defined in .conf)

Sorry for being dumb, but is this an enhancement? Why moving them out of the
db?


>    New page type (bitmap?) to link SCNs to page numbers.
>      This may noticeable improve the incremental NBackup performance,
>      avoiding reading pages not changed since the last backup.

Not sure if this is intended to work only while gbak is not used on the same
db (perhaps the question is too crazy?).

 
>    Switch from SLONG to ULONG for transaction numbers and
> page numbers.

Good, although it only moves the bar higher (later in time) when the db will
need a backup/restore cycle.


>    Extend blob length field from ULONG to UINT64
>    (and maybe allow blob levels higher than 2?).

Not sure we want these, as the db grows too fast with big chunks, blobs are
always subject to txn support (unlike other engines), each modification is
indeed a new blob, etc. Maybe allowing a second class of blob placed
directly in the file system would be more interesting.

 
>    Consider better alternative (instead of SQZ) for record
> compression.
>    Tests are required. If not found, extend SQZ for sequences longer
>    than 127 bytes.

Perhaps being able to cope with UNICODE is all that's needed?

C.



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Calin Pirtea(RDS)]

I agree with Mark.

Instead of moving valuable settings out of the DB I would lean towards
moving more of them in.

Maybe a new RDB$ table would be an improvement?

Cheers,
Calin.

----- Original Message -----
From: "Mark Rotteveel" <Avalanche1979@...>
To: "For discussion among Firebird Developers"
<firebird-devel@...>
Sent: Monday, October 26, 2009 7:21 PM
Subject: Re: [Firebird-devel] RFC: Major changes in ODS 12


>      * deprecate page buffers and sweep interval
>        (in favor of the per-database values defined in .conf)

What is the rationale behind this proposal? I have the feeling that moving
this configuration to a separate conf file will make (remote) management of
these configuration options harder, not simpler; and will place a burden on
the systems administrator instead of the database owner.

If these options are moved to a separate file (or database?) they should
also be remotely configurable by the database owner, not just by manually
editting a conf file (manually editing conf files IMHO should be a last
resort, and only for very low-level configuration).

I would even like to see that Firebird eventually deprecates support for
loose, unmanaged database-files (except for embedded) and move to a central
repository of databases with extended management and configuration options.
So database are (by default) only accessible through their alias (not the
full filename). This would make management and naming of various databases
more like Oracle, MS SQL Server etc.

Mark



--
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel 


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Alexander Peshkoff]

On Monday 26 October 2009 12:22:57 Dimitry Sibiryakov wrote:
> >    Consider better alternative (instead of SQZ) for record compression.
> >    Tests are required. If not found, extend SQZ for sequences longer
> >    than 127 bytes.
>
>    What's the purpose of this change? Smaller size, faster compression
> or what?

Ideally both.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Alexander Peshkoff]

On Monday 26 October 2009 11:52:16 Dmitry Yemanov wrote:
>      * rework implementation ID to simplify the porting
>        (split into endianness and swap_double flags?)

I've planned to use CPU/OS/compiler schema instead of current CLASSes +
endianness/swap_double flags.

>    New page type (bitmap?) to link SCNs to page numbers.
>      This may noticeable improve the incremental NBackup performance,
>      avoiding reading pages not changed since the last backup.

How will it affect performance in all other moments?

>    Switch from SLONG to ULONG for transaction numbers and page numbers.

May be it's better to use FB_UINT64 for transaction numbers?

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Dmitry Yemanov]

Dimitry Sibiryakov wrote:

>>    Consider better alternative (instead of SQZ) for record compression.
>>    Tests are required. If not found, extend SQZ for sequences longer
>>    than 127 bytes.
>
> What's the purpose of this change? Smaller size, faster compression?

I was primarily thinking about the latter. But if proven impossible, I'd
agree to favor smaller size, especially for longish strings.

> Add "Min transaction number" field to header to allow to drop old
> TIPs and/or reuse transactions numbers to remove limit of transactions
> in database.

I will let Vlad to comment here ;-)

> Change offset fields from USHORT to ULONG to allow page sizes >64k.
> Change meaning of hdr_page_size from number of bytes to number of
> kilobytes for the same purpose.

Do you believe in faster performance using bigger pages?


Dmitry

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Dmitry Yemanov]

Roman Rokytskyy wrote:
>
> Regarding the schema support. If the ODS change is needed here, please
> consider this feature as well.

AFAIU, it's needed in the system catalog only. But almost all tables
will be affected. And amount of internal changes is also likely to be
big enough. And API change is required as well. And it's preferred to
have this feature bound to longer metadata identifiers (i.e. implemented
together).

I seriously doubt this fits the v3 time frame.


Dmitry

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Vlad Khorsun-2]

>> Add "Min transaction number" field to header to allow to drop old
>> TIPs

    It is exists and called OST

>> and/or reuse transactions numbers to remove limit of transactions
>> in database.

    I not sure we have urgent need to do something in this regards.
Also it just will not work.

> I will let Vlad to comment here ;-)

    See above ;)
 
>> Change offset fields from USHORT to ULONG to allow page sizes >64k.
>> Change meaning of hdr_page_size from number of bytes to number of
>> kilobytes for the same purpose.
>
> Do you believe in faster performance using bigger pages?

    I also think we dont need huge pages. Read speed could be reached by
another approach and write speed must not be destroyed.

Regards,
Vlad

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Dmitry Yemanov]

Alexander Peshkoff wrote:
>
> I've planned to use CPU/OS/compiler schema instead of current CLASSes +
> endianness/swap_double flags.

Well, I believe you're a better expert in this area :-)

>>    New page type (bitmap?) to link SCNs to page numbers.
>>      This may noticeable improve the incremental NBackup performance,
>>      avoiding reading pages not changed since the last backup.
>
> How will it affect performance in all other moments?

AFAIK, Vlad has some premature results to share.

>> Switch from SLONG to ULONG for transaction numbers and page numbers.
>
> May be it's better to use FB_UINT64 for transaction numbers?

And thus make the every record version four bytes larger? I doubt this
is a good idea.


Dmitry

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Vlad Khorsun-2]

>>    New page type (bitmap?) to link SCNs to page numbers.
>>      This may noticeable improve the incremental NBackup performance,
>>      avoiding reading pages not changed since the last backup.
>
> How will it affect performance in all other moments?

    Testing will show. What is maximum allowed performance drop, except of 0% ? :)  
Note, performance drop occurs only after SCN change. I have draft implementation of
this feature and see less than 10% of performance drop in TPCC test. And it can be
improved.

>>    Switch from SLONG to ULONG for transaction numbers and page numbers.
>
> May be it's better to use FB_UINT64 for transaction numbers?

    We already have a very fat record header. How many real cases you know when
database reached current limit ?

Regards,
Vlad

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Dmitry Yemanov]

Dimitry Sibiryakov wrote:
>
> Forgot to add: change record size from USHORT to ULONG to remove 64k
> limit for record length.

I'm not sure it would be as easy as it sounds. I have no problem however
to make this ODS change now to be utilized by some further FB version.


Dmitry

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Dmitry Yemanov]

Rustam Gadjimuradov wrote:
>
> One conf-file for all databases (i.e. current firebird.conf)
> or per-database.conf file (i.e. my_db.conf) ?

You may have a common firebird.conf with defaults as well as a special
section inside firebird.conf that define custom settings for my_db.fdb,
or even a separate my_db.conf that's included by firebird.conf.


Dmitry

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Dmitry Yemanov]

Mark Rotteveel wrote:
>
> What is the rationale behind this proposal?

Avoiding both semantical and management conflicts between in-database
settings and the ones inside the per-database configuration file.

Believe or not, a lot of our users (especially newcomers) have no idea
that a few settings may be specified inside the database and it often
becomes a support issue. With two independent ways to specify
per-database options it's going to become even worse.

Also, the configuration file provides much more settings and IMHO should
be preferred. Someone might suggest to go the opposite route and migrate
all these options into the database file, but I'm not convinced this is
a good idea.

> I have the feeling that moving
> this configuration to a separate conf file will make (remote) management of
> these configuration options harder, not simpler; and will place a burden on
> the systems administrator instead of the database owner.

In turn, I have a feeling that the page cache setting has more to do
with system administrators rather than with database owners ;-)

> If these options are moved to a separate file (or database?) they should
> also be remotely configurable by the database owner

What if the .conf files would be editable by the Services API?

Let's hear other opinions as well, this is a really questionable topic.
I just want to see some consistency in the database settings and thus
open to all ideas.


Dmitry

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Dmitry Yemanov]

Claudio Valderrama C. wrote:
>
>>      * deprecate page buffers and sweep interval
>>        (in favor of the per-database values defined in .conf)
>
> Sorry for being dumb, but is this an enhancement? Why moving them out of the
> db?

Answered in another reply.

>>    New page type (bitmap?) to link SCNs to page numbers.
>>      This may noticeable improve the incremental NBackup performance,
>>      avoiding reading pages not changed since the last backup.
>
> Not sure if this is intended to work only while gbak is not used on the same
> db (perhaps the question is too crazy?).

Not sure I got your point. GBAK is just a regular client application, it
doesn't directly interact with NBackup.

>> Switch from SLONG to ULONG for transaction numbers and
>> page numbers.
>
> Good, although it only moves the bar higher (later in time) when the db will
> need a backup/restore cycle.

Correct, but it's the simplest thing we can do and for some customers 2x
longer uptime is a noticeable improvement.

>>    Extend blob length field from ULONG to UINT64
>>    (and maybe allow blob levels higher than 2?).
>
> Not sure we want these

I'm not sure either. I just don't feel myself comfortable with a 4GB
limit for something designed to be really big :-) This can surely be
deferred till the next ODS upgrade.

>> Consider better alternative (instead of SQZ) for record
>> compression.
>> Tests are required. If not found, extend SQZ for sequences longer
>> than 127 bytes.
>
> Perhaps being able to cope with UNICODE is all that's needed?

I haven't thought about Unicode. But wasting 500 bytes for an empty
varchar makes me crazy :-)


Dmitry

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Dimitry Sibiryakov-3]

> Dimitry Sibiryakov wrote:
>
>>>    Consider better alternative (instead of SQZ) for record compression.
>>>    Tests are required. If not found, extend SQZ for sequences longer
>>>    than 127 bytes.
>> What's the purpose of this change? Smaller size, faster compression?
>
> I was primarily thinking about the latter. But if proven impossible, I'd
> agree to favor smaller size, especially for longish strings.

   No, using short instead of byte won't speed up compression (as much
as further optimization of placement), IMHO. What to smaller size... In
the worst case (64k record, absolutely not compressible) we loose about
500 bytes. Is it too much?

>> Change offset fields from USHORT to ULONG to allow page sizes >64k.
>> Change meaning of hdr_page_size from number of bytes to number of
>> kilobytes for the same purpose.
>
> Do you believe in faster performance using bigger pages?

   Up to 65k - yes. Such page can hold whole record of any available
size or whole biggest BLOB segment.

Wlad Horsun wrote:
>>> >> Add "Min transaction number" field to header to allow to drop old
>>> >> TIPs
>
>     It is exists and called OST

   May be you are right. Tell me then: TI pages which holds states of
transactions with numbers less than OST are marked as free? When?

>>> >> and/or reuse transactions numbers to remove limit of transactions
>>> >> in database.
>
>     I not sure we have urgent need to do something in this regards.
> Also it just will not work.

   Do you mean that it won't work within current code or you doubt in
conception of circular buffers in common?

   SY, SD.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel