Nabble - PostgreSQL - hackers

Re: Range types

2009-12-16T13:37:43Z

Martijn van Oosterhout <kleptog@...> writes:
> However, it does seem reasonable to allow people to restrict, either by
> typmod or a check constraint the kinds of values that can be stored in
> a particular column. Then an application can decide which way they want
> their intervals to work and have the database enforce it.

Sure --- the range datatype should absolutely provide inquiry functions
that let you determine all the properties of a range, so something like
"CHECK (is_open_on_right(col))" would work for that. I'm of the opinion
that we must not usurp typmod for range behavior --- the right thing is
to pass that through to the contained type, just as we do with arrays.

(Note that a range over timestamp(0) would eliminate at least some of
the platform dependencies we've been arguing about. I'm still quite
dubious that "next timestamp" is anything except evidence that you've
misformulated your problem, though.)

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T13:32:59Z

On Wed, 2009-12-16 at 15:46 -0500, Tom Lane wrote:
> > Huh? We're miscommunicating somewhere.
>
> Yeah, apparently. By open-ended I meant -infinity left bound, or null
> left bound if you prefer. Not sure if there's a better term.

But my proposal allowed both of those things with various flag settings
(because it has a flag byte in the latest proposal). I said so
explicitly.

Regards,
Jeff Davis

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Update on true serializable techniques in MVCC

2009-12-16T13:28:27Z

Robert, Please forgive a couple editorial inserts to your statement
-- I hope it clarifies. If I've distorted your meaning, feel free
to straighten me out. :-)

Robert Haas <robertmhaas@...> wrote:

> This thread veered off into a discussion of the traditional
> [predicate locking] technique, rather than the [serializable] one
> in the paper. I think that's the part Alvaro was responding to.

If you're right about Alvaro's concern -- my rough understanding is
that HOT creates a linked lists of tuples which are mutations of one
another without altering any value which is part of an index. If
that's anywhere near a correct understanding, I can't see how there
would be a problem with using the described locking techniques with
any tuple on the list.

As an aside, the thesis mentions smart use of multiple locking
granularities only under the "future work" section. I can't see an
implementation being considered production quality without that, as
without it there would be no way to constrain the space required to
track the locks. But there is no shortage of literature on how to
do that, so I view such discussions as more appropriate to low-level
implementation discussions should we ever get serious about using
the techniques which are the main thrust of Dr. Cahill's thesis.

If anyone is interested in reviewing recent literature on these
techniques, Dr. Cahill seemed to like (Hellerstein et al., 2007), to
the point where I may well track it down when I'm done pondering the
work which I referenced at the start of this thread.

-Kevin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T13:17:16Z

On Wed, Dec 16, 2009 at 03:57:44PM -0500, Tom Lane wrote:

> Jeff Davis <pgsql@...> writes:
> > I still have not seen an answer to the problem of changing the
> > representation of a continuous range. If you have the continuous range
> > [5, 10], you're pretty much stuck with that representation, even if the
> > application is expecting things in the form [ ).
>
> That is not our problem. It's the application's problem if it can't
> handle the concept. You might as well be complaining that type numeric
> is broken because it can represent values that will fail to fit into
> float8 when some application tries to force them into that form.

However, it does seem reasonable to allow people to restrict, either by
typmod or a check constraint the kinds of values that can be stored in
a particular column. Then an application can decide which way they want
their intervals to work and have the database enforce it.

(Intermediate values may become a different kind, just as long as the
result being stored it the right kind.)

Have a nice day,
--
Martijn van Oosterhout <kleptog@...> http://svana.org/kleptog/
> Please line up in a tree and maintain the heap invariant while
> boarding. Thank you for flying nlogn airlines.

signature.asc (196 bytes) Download Attachment

Re: Range types

2009-12-16T12:57:44Z

Jeff Davis <pgsql@...> writes:
> On Wed, 2009-12-16 at 13:59 -0500, Tom Lane wrote:
>> The argument for having
>> granularity wired into the datatype seems to boil down to just space
>> savings. I don't find that compelling enough to justify code
>> contortions and user-visible restrictions on functionality.

> The argument (at least from me) is that discrete ranges have better
> semantics. The counterargument was that the granularity of a timestamp
> is an implementation detail. So I countered by making it explicit.

Making it explicit doesn't fix the fact that you can't rely on the
arithmetic to be exact.

> I still have not seen an answer to the problem of changing the
> representation of a continuous range. If you have the continuous range
> [5, 10], you're pretty much stuck with that representation, even if the
> application is expecting things in the form [ ).

That is not our problem. It's the application's problem if it can't
handle the concept. You might as well be complaining that type numeric
is broken because it can represent values that will fail to fit into
float8 when some application tries to force them into that form.

> And to further make the case for allowing user-defined discrete ranges,
> what about ip4r?

What about it? I don't have a problem with the concept that next() is
well defined for some datatypes. I just have a problem with the concept
that it's well-defined for timestamps. It's not, and I don't believe
that forcing it to have some definition is useful in the real world
(which, as a rule, isn't going to fit the simplifying assumptions you
have to make to make it even sort-of work).

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T12:46:58Z

Jeff Davis <pgsql@...> writes:
> On Wed, 2009-12-16 at 12:50 -0500, Tom Lane wrote:
>> I'm still not exactly clear on what the use-case is for discrete
>> timestamp ranges, and I wonder how many people are going to be happy
>> with a representation that can't handle a range that's open-ended
>> on the left.

> Huh? We're miscommunicating somewhere.

Yeah, apparently. By open-ended I meant -infinity left bound, or null
left bound if you prefer. Not sure if there's a better term.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T12:42:27Z

Martijn van Oosterhout <kleptog@...> writes:
> But a period type will take just one or two more bytes if you don't
> require alignment. Alignment on a varlena type seems silly anyway,
> since you'll be aligning the header byte rather than the content.

You might still end up paying the alignment overhead after the field,
of course. But avoiding embedding the alignment in the type itself
seems worth doing.

One idea that might be interesting to consider in this regard is
force-packing varlena range values. Consider a text range('a', 'b').
The datums are likely to come in with 4-byte headers requiring
alignment. If we have the smarts to force them to 1-byte header form
inside the varlena range value, not only do we save bytes right there,
but we don't have to expend cycles to copy them somewhere to re-align
them before we can pass them to the datatype-specific functions.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T12:41:31Z

Tom Lane wrote:
> Alvaro Herrera <alvherre@...> writes:
> > In short, I think that while it is possible to define ranges of strings,
> > it is not as useful as one would like.
>
> Note it is not the *range* that is the problem, it is the assumption
> that there's a unique "next" string. There's no unique next in the
> reals or rationals either, but we have no problem considering intervals
> over those sets.

Yeah, agreed. It's easy (I think) to define more useful ranges of
strings if you don't insist in having "next".

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T12:35:38Z

On Wed, 2009-12-16 at 13:59 -0500, Tom Lane wrote:
> The argument for having
> granularity wired into the datatype seems to boil down to just space
> savings. I don't find that compelling enough to justify code
> contortions and user-visible restrictions on functionality.

The argument (at least from me) is that discrete ranges have better
semantics. The counterargument was that the granularity of a timestamp
is an implementation detail. So I countered by making it explicit.

Space savings is not crucial, but it would be frustrating to needlessly
waste space.

I still have not seen an answer to the problem of changing the
representation of a continuous range. If you have the continuous range
[5, 10], you're pretty much stuck with that representation, even if the
application is expecting things in the form [ ).

>From an application's standpoint, you probably want to get the
information about a range as separate columns (as opposed to parsing a
string). So an application expecting data in [) format might use a query
like:

select ..., first(myperiod), next(myperiod) from mytable;

That gives the application complete information about the range. You can
even make a view over a table like that to make it even more transparent
to the application. It's not entirely unreasonable that many such
applications exist; there are many presentations and tutorials that have
been telling people to use a "start" and "end" column, and assume that
the start is inclusive and the end is exclusive.

If there is some other application that expects data in (] format, you
just use the query:

select ..., prior(myperiod), last(myperiod) from mytable;

With discrete ranges, that all just works (barring overflow or special
values).

With continuous ranges, first() or next() might fail on some values that
were produced by some other application. Really, for continuous ranges,
you'd need to have a query like:

select ..., start(myperiod), start_inclusive(myperiod),
end(myperiod), end_inclusive(myperiod) from mytable;

in order to have all of the information. And that means that the
application needs a full implementation of a range type to understand
the inclusivity and produce a correct result.

And to further make the case for allowing user-defined discrete ranges,
what about ip4r? That's a discrete range, and it's user-defined. And
that probably means other useful discrete ranges will be invented, as
well.

If we want to say that there is no discrete TIMESTAMP range by default,
and that the superuser has to define it, that's one thing. But if we say
that the only acceptable base types for discrete ranges will be
hard-wired, that's way too restrictive. If nothing else, it makes some
system types "special" which we have not done very many other places.

Regards,
Jeff Davis

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Does "verbose" Need to be Reserved?

2009-12-16T12:31:11Z

"David E. Wheeler" <david@...> writes:
> I asked on IRC, and Andrew RhodiumToad Gierth pointed out that
> it became a reserved word at some point.

"Some point" would have been around the time VACUUM VERBOSE got
invented, ie January 1997 according to the CVS logs. We can't unreserve
it until you're ready to break that syntax (is VERBOSE a table name
or a flag for a database-wide vacuum?). Don't hold your breath.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Update on true serializable techniques in MVCC

2009-12-16T12:27:13Z

On Wed, Dec 16, 2009 at 1:30 PM, Kevin Grittner
<Kevin.Grittner@...> wrote:

> Alvaro Herrera <alvherre@...> wrote:
>
>> So you'd have to disable HOT updates when true serializability was
>> active?
>
> I wouldn't think so; but someone familiar with HOT logic could
> probably determine whether the unmodified algorithm could be used by
> reviewing the "simplifying assumptions" near the bottom of page 42,
> and the page about "Generalizing to other database engines" (section
> 4.8). I think those portions might stand on their own without
> reading the rest of the document.

This thread veered off into a discussion of the traditional technique,
rather than the one in the paper. I think that's the part Alvaro was
responding to.

...Robert

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T12:26:23Z

Alvaro Herrera <alvherre@...> writes:
> In short, I think that while it is possible to define ranges of strings,
> it is not as useful as one would like.

Note it is not the *range* that is the problem, it is the assumption
that there's a unique "next" string. There's no unique next in the
reals or rationals either, but we have no problem considering intervals
over those sets.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Update on true serializable techniques in MVCC

2009-12-16T12:26:22Z

On Wed, Dec 16, 2009 at 1:29 PM, Boszormenyi Zoltan <zb@...> wrote:

> Robert Haas írta:
>> On Wed, Dec 16, 2009 at 1:25 PM, Robert Haas <robertmhaas@...> wrote:
>>
>>> On Wed, Dec 16, 2009 at 1:14 PM, Alvaro Herrera
>>> <alvherre@...> wrote:
>>>
>>>> Robert Haas escribió:
>>>>
>>>>> On Wed, Dec 16, 2009 at 10:29 AM, Andres Freund <andres@...> wrote:
>>>>>
>>>>>> On Wednesday 16 December 2009 16:24:42 Robert Haas wrote:
>>>>>>
>>>>>>>> Inserts and deletes follow the same protocol, obtaining an exclusive
>>>>>>>> lock on the row after the one being inserted or deleted. The result
>>>>>>>> of this locking protocol is that a range scan prevents concurrent
>>>>>>>> inserts or delete within the range of the scan, and vice versa.
>>>>>>>>
>>>>>>>> That sounds like it should actually work.
>>>>>>>>
>>>>>>> Only if you can guarantee that the database will access the rows using
>>>>>>> some particular index. If it gets to the data some other way it might
>>>>>>> accidentally circumvent the lock. That's kind of a killer in terms of
>>>>>>> making this work for PostgreSQL.
>>>>>>>
>>>>>> Isnt the whole topic only relevant for writing access? There you have to
>>>>>> access the index anyway.
>>>>>>
>>>>> Yeah, I guess you have to insert the new tuple. I guess while you
>>>>> were at it you might check whether the next tuple is locked...
>>>>>
>>>> So you'd have to disable HOT updates when true serializability was
>>>> active?
>>>>
>>> I thought about that, but I don't think so. HOT only applies to
>>> updates, and predicate locking only applies to inserts. Unless I have
>>> my head in the sand?
>>>
>>
>> Err, no, wait. Predicate locking can apply to updates, but since HOT
>> updates never update an indexed column, I think we might still be OK?
>>
>
> A predicate can include columns from an index plus others.
> Am I missing something?

Hmm, interesting point. In that case you couldn't use the index to
enforce predicate locking under MVCC without disabling HOT. But there
will be other cases where that wouldn't help anyway - a predicate
could also include unindexed columns exclusively. For those, the
traditional approach (not the one discussed in this paper) probably
requires locking against any heap insert, or checking each new heap
insert against the constraint, or... something.

...Robert

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T12:24:36Z

On Wed, Dec 16, 2009 at 10:57:19AM -0800, Scott Bailey wrote:
> Ok, silly question here. But how do you determine the length of a
> continuous range? By definition length of [a, b) and (a, b] = b-a. But
> what about (a,b) and [a,b]? Are we saying that because they are
> continuous, the difference between values included in the range and
> those excluded are so infinitesimally small so as not to matter? Thus
> length (a,b) == length [a,b] == length [a,b)? And if that is the case,
> does the inclusiveness of the range really even matter?

Short answer: Yes

Longer answer: You need to decide on your definition of "length" and
what you usually use is the "measure". And yes, the difference between
the two is so called "measure 0" and thus has no effect on the length.

Note the measure has to be done considering the intervals as intervals
on a real line. The integers by themselves have no measure (they are
countable). So for the "length" of a set of integers you might consider
the count of the set.

http://planetmath.org/encyclopedia/ProofThatTheOuterLebesgueMeasureOfAnIntervalIsItsLength.html
http://en.wikipedia.org/wiki/Outer_measure

As for "continuous", as you use it above is not a way I recognise.
There are contiguous sets, but they are something else.

Have a nice day,
--
Martijn van Oosterhout <kleptog@...> http://svana.org/kleptog/
> Please line up in a tree and maintain the heap invariant while
> boarding. Thank you for flying nlogn airlines.

signature.asc (196 bytes) Download Attachment

Re: Does "verbose" Need to be Reserved?

2009-12-16T12:20:32Z

>>>>> "David" == "David E Wheeler" <david@...> writes:

David> Hey All,
David> I was just getting a new version of pgTAP ready for release, and while testing it on HEAD, I got this error:

David> + psql:pgtap.sql:5789: ERROR: syntax error at end of input
David> + LINE 28: IF verbose THEN RETURN NEXT diag(tests[i] ||...
David> + ^

David> I asked on IRC, and Andrew “RhodiumToad” Gierth pointed out
David> that it became a reserved word at some point. I'm fine to
David> rename my variable, but Andew and I were wondering if it's
David> really necessary for "verbose" to be reserved, since it's not
David> in the spec.

Looking at it more closely, this is likely to be fallout from the
plpgsql lexer/parser changes; it probably worked before only because
plpgsql was doing its own thing rather than using the main lexer.

VERBOSE has been reserved all along in order to distinguish
'vacuum verbose;' from 'vacuum tablename;' and so on; but it's still an
interesting pitfall for plpgsql users.

--
Andrew (irc:RhodiumToad)

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T12:07:39Z

On Tue, Dec 15, 2009 at 04:29:26PM -0800, Jeff Davis wrote:

> On Tue, 2009-12-15 at 18:06 -0600, decibel wrote:
> > Now that varlena's don't have an enormous fixed overhead, perhaps it's
> > worth looking at using them. Obviously some operations would be
> > slower, but for your stated examples of auditing and history, I
> > suspect that you're not going to notice the overhead that much.
>
> For most varvarlena types, you only get stuck with the full alignment
> burden if you get unlucky. In this case, we're moving from 16 bytes to
> 17, which really means 24 bytes with alignment. Try creating two tables:
>
> create table foo(i int8, t1 timestamp, t2 timestamp);
> create table bar(i int8, c "char", t1 timestamp, t2 timestamp);

But a period type will take just one or two more bytes if you don't
require alignment. Alignment on a varlena type seems silly anyway,
since you'll be aligning the header byte rather than the content.

In the implementation you may need to copy the content before
processing to satisfy the alignment of the contained type, but that's
just a SMOP.

Have a nice day,
--
Martijn van Oosterhout <kleptog@...> http://svana.org/kleptog/
> Please line up in a tree and maintain the heap invariant while
> boarding. Thank you for flying nlogn airlines.

signature.asc (196 bytes) Download Attachment

Does "verbose" Need to be Reserved?

2009-12-16T12:05:10Z

Hey All,

I was just getting a new version of pgTAP ready for release, and while testing it on HEAD, I got this error:

+ psql:pgtap.sql:5789: ERROR: syntax error at end of input
+ LINE 28: IF verbose THEN RETURN NEXT diag(tests[i] ||...
+ ^

I asked on IRC, and Andrew “RhodiumToad” Gierth pointed out that it became a reserved word at some point. I'm fine to rename my variable, but Andew and I were wondering if it's really necessary for "verbose" to be reserved, since it's not in the spec.

Thanks,

David

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T11:45:54Z

tomas@... wrote:

> (and as Andrew Dunstan pointed out off-list: I was wrong with my bold
> assertion that one can squeeze infinitely many (arbitrary length)
> strings between two given. This is not always the case).

Of course you can do that if you assume lexicographical order, or any
other arbitrary order. The interesting point is whether there exists
some ordering on which this does not happen. And in fact there is:
order strings by length first, and then lexicographically. If you do
this then you have next() and prev() for any given string. If you use
ASCII only, you have a.next = b, and so on.

There is the argument that some languages do not sort lexicographically
but this is also besides the point -- you only need to find *some* way
to sort the characters in the alphabet. If you dictate that in your
ordering "á" comes before "à" and both after "a", and all of them before
b, then you know that a.next = á and á.next = à and à.next = b. (Note
that I have also dictated that there is no other character that sorts
after a and before b, which is perfectly possible because the alphabet
is fixed for any given language. You could use the complete list of
characters coded in a given set of Unicode planes, or even extant all
planes, to obtain the same result).

Defining strings with this ordering means you can have some useful
ranges like [a-z], but then you cannot meaningfully use ranges for
things like [aardvark - zulu] --- note that in this particular example,
the order is reversed, because zulu comes before aardvark which is
probably not what you want. zzz.next = aaaa

In short, I think that while it is possible to define ranges of strings,
it is not as useful as one would like.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: PATCH: Spurious "22" in hstore.sgml

2009-12-16T11:39:14Z

On Wed, Dec 16, 2009 at 20:34, David E. Wheeler <david@...> wrote:
> *** a/doc/src/sgml/hstore.sgml
> --- b/doc/src/sgml/hstore.sgml

Heh, interesting. That clearly shouldn't be there. Applied.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Patch: Remove gcc dependency in definition of inline functions

2009-12-16T11:36:47Z

On ons, 2009-12-16 at 10:49 -0500, Tom Lane wrote:

> Marko Kreen <markokr@...> writes:
> > So the plain-C89 compilers would be downgraded to "second-class"
> > targets, not worth getting max performance out of them.
>
> Hm? Failing to inline is already a performance hit, which is why
> Kurt got interested in this in the first place.
>
> I think you're way overthinking this. Where we started was just
> a proposal to try to expand the set of inline-ing compilers beyond
> "gcc only". I don't see why we need to do anything but that. The
> code is fine as-is except for the control #ifdefs.

I think the ifdefs should just be HAVE_INLINE && !MSVC, right?

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

PATCH: Spurious "22" in hstore.sgml

2009-12-16T11:34:33Z

*** a/doc/src/sgml/hstore.sgml
--- b/doc/src/sgml/hstore.sgml
***************
*** 278,284 ****
<entry>get <type>hstore</>'s keys as a set</entry>
<entry><literal>skeys('a=>1,b=>2')</literal></entry>
<entry>
! 22<programlisting>
a
b
</programlisting></entry>
--- 278,284 ----
<entry>get <type>hstore</>'s keys as a set</entry>
<entry><literal>skeys('a=>1,b=>2')</literal></entry>
<entry>
! <programlisting>
a
b
</programlisting></entry>

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T11:29:55Z

On Wed, 2009-12-16 at 12:50 -0500, Tom Lane wrote:
> I'm still not exactly clear on what the use-case is for discrete
> timestamp ranges, and I wonder how many people are going to be happy
> with a representation that can't handle a range that's open-ended
> on the left.

Huh? We're miscommunicating somewhere. Discrete ranges are values, and
those values can be displayed a number of different ways. That's one of
the biggest advantages.

The very same discrete range can be displayed as open-open, closed-open,
open-closed, or closed-closed. There are edge cases, like how infinity
is never closed, and overflow conditions. But generally speaking, you
have more options for presenting a discrete range than a continuous
range.

The range [5, 7) is equal to the set {5, 6} and equal to the ranges:
(4,7), (4,6], [5,7), and [5,6]. One application can insert it as [5,7)
and another can read it as (4,6].

That's the use case: the application's preferences don't have to match.
It's OK to mix various representation preferences, because you can
convert between them. The on disk format happens to hint at one
particular canonical form, but doesn't enforce that on anyone.

> Huh? You're not going to be able to have a special case data
> representation for one or two data types at the same time as you have a
> function-based datatype-independent concept of a parameterized range
> type. Well, maybe you could have special code paths for just date and
> timestamp but it'd be horrid.

They aren't supposed to be exactly the same API, I said that from the
start. There are API differences between continuous and discrete ranges,
and we shouldn't ignore them.

One important differences is that (barring overflow conditions and
special values) prior, first, last, and next are defined for all
discrete range values, but not for all continuous range values.

For instance, the discrete range [5,7) has prior=4, first=5, last=6,
next=7.

Whereas the continuous range [5,7) has prior=undef, first=5, last=undef,
next=7.

We could define one API, that treats discrete and continuous ranges
differently. But you'll never be able to transform a continuous range to
a different representation, while you can do so with a discrete range.

> More importantly, the notion of a representation granule is still 100%
> wishful thinking for any inexact-representation datatype, which is going
> to be a severe crimp in getting this accepted for timestamp, let alone
> defining it in a way that would allow users to try to apply it to
> floats. Float timestamps might not be the default case anymore but they
> are still supported.

If the only objection is that a superuser can confuse the system by
poorly defining a range type on a non-default build, I think that
objection can be overcome.

> I think you should let go of the feeling that you have to shave bytes
> off the storage format. You're creating a whole lot of work for
> yourself and a whole lot of user-visible corner cases in return for
> what ultimately isn't much.

This isn't just to shave bytes. It's also because I like the semantics
of discrete ranges for many cases.

Regards,
Jeff Davis

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

pinging Dano

2009-12-16T11:28:26Z

Hi,

who is the main editor named "Dano" of the wiki page about Parallel
Query Execution
(http://wiki.postgresql.org/wiki/Parallel_Query_Execution), please speak up.

Is there any code or patch available ATM? What "discussion with Tom and
Simon" is that page referring to?

Regards

Markus Wanner

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

PATCH: Add hstore_to_json()

2009-12-16T11:28:09Z

I just realized that this was easy to do, and despite my complete lack of C skillz was able to throw this together in a couple of hours. It might be handy to some, though the possible downsides are:

* No json_to_hstore().
* Leads to requests for hstore_to_yaml(), hstore_to_xml(), etc.
* Andrew Gierth said “no” when I suggested it.

But it's kind of handy, too. Thoughts?

Best,

David

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

hstore_to_json.patch (7K) Download Attachment

Re: Range types

2009-12-16T11:25:12Z

On Wed, 2009-12-16 at 12:42 -0500, Robert Haas wrote:

> On Wed, Dec 16, 2009 at 12:31 PM, Jeff Davis <pgsql@...> wrote:
> > There's one problem, and that's for timestamptz ranges with intervals
> > that include days and months. Timezone adjustments are just not
> > well-defined for that kind of granule (nor would it be particularly
> > useful even if it magically worked), so this would have to be blocked
> > somehow. I think that's a special case, and we could provide the user
> > with a nice error message telling the user to use a date or timestamp
> > range instead.
>
> This seems like a fairly special-purpose type. You'd be targeting it
> at people who are very concerned with storing large numbers of these
> (so they really care about space consumption) but for some reason
> don't need to mix days and months (actually, the current interval
> representation stores days, months, and seconds separately). I
> certainly think this might be useful to some people but it doesn't
> really sounds like a general range type facility, since it seems to
> involve some hacks that are fairly datatype-specific.

My statement should have read "days or months". In other words, you
can't have a timestamptz range with a granularity of '3 days'. But if
that's your granularity, as Scott says, you should be using a date
range, not a timestamptz range.

Timestamptz ranges are only really useful when you have a granularity
measured in seconds (or some fraction or multiple thereof). Otherwise,
the timezone adjustment doesn't make any sense.

So this isn't a case of limited functionality, just that we need to
inform the user that a timestamptz range with granularity '1 day' or '1
month' makes no sense.

Regards,
Jeff Davis

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: XLogInsert

2009-12-16T11:14:21Z

On Wednesday 16 December 2009 20:07:07 Gurjeet Singh wrote:

> 2009/12/15 Greg Smith <greg@...>
>
> > Jaime Casanova wrote:
> >> So in this extreme case avg tps is just 6 transactions better
> >
> > Great job trying to find the spot where the code worked better. I'm not
> > so sure I trust pgbench results where the TPS was so low though. Which
> > leads us right back to exactly how Jeff measured his original results.
> >
> > As I said already, I think we need more insight into Jeff's performance
> > report, a way to replicate that test, to look a bit at the latency as
> > reported by the updated LWLock patch that Pierre submitted. Tweaking
> > your test to give more useful results is a nice second opinion on top of
> > that. But we're out of time for now, so this patch is getting returned
> > with feedback. I encourage Jeff to resubmit the same patch or a better
> > one with a little more data on performance measurements to our final 8.5
> > CommitFest in hopes we can confirm this an improvement worth committing.
>
> Last week I worked on a FUSE based filesystem, which I call BlackholeFS.
> Its similar to /dev/null, but for directories. Basically it simply returns
> success for all the writes, but doesn't do any writes on the files under
> it.

I doubt that it will be faster than a tmpfs - the additional context switches
et al probably will hurt already.
If you constrain the checkpoint_segments to something sensible it shouldnt use
too much memory.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: XLogInsert

2009-12-16T11:07:07Z

2009/12/15 Greg Smith <greg@...>

Jaime Casanova wrote:

So in this extreme case avg tps is just 6 transactions better

Great job trying to find the spot where the code worked better. I'm not so sure I trust pgbench results where the TPS was so low though. Which leads us right back to exactly how Jeff measured his original results.

As I said already, I think we need more insight into Jeff's performance report, a way to replicate that test, to look a bit at the latency as reported by the updated LWLock patch that Pierre submitted. Tweaking your test to give more useful results is a nice second opinion on top of that. But we're out of time for now, so this patch is getting returned with feedback. I encourage Jeff to resubmit the same patch or a better one with a little more data on performance measurements to our final 8.5 CommitFest in hopes we can confirm this an improvement worth committing.

Last week I worked on a FUSE based filesystem, which I call BlackholeFS. Its similar to /dev/null, but for directories. Basically it simply returns success for all the writes, but doesn't do any writes on the files under it.

Would moving the pg_xlog/ (and possibly table data too) to such a filesystem exercise this patch better?

Best regards,

--
Lets call it Postgres

EnterpriseDB http://www.enterprisedb.com

gurjeet[.singh]@EnterpriseDB.com

singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com
Twitter: singh_gurjeet
Skype: singh_gurjeet

Mail sent from my BlackLaptop device

Re: Range types

2009-12-16T10:59:02Z

Scott Bailey <artacus@...> writes:
> As I pointed out off-list, I think the granularity for timestamp range
> should be limited to hours and smaller. Anything larger is asking for
> trouble. And quite honestly if they wanted day granularity, they should
> use date range.

I'm still not real clear on what the expected use-case is for this.
You're evidently envisioning applications where the allowed form of
an interval is constrained, but in the cases I can think of, the
constraints are a lot more convoluted than what you're proposing.
For example, if you're trying to do classroom scheduling, it might be
useful to constrain the periods to start and end on hour boundaries
--- but the next thing you'll want is to have it know that the "next"
slot after 5pm Friday is 8am Monday. Except on holidays. And then
there's the fact that my alma mater starts most hour-long classes on
the half hour.

I think that wiring such constraints into the low-level datatype is
doomed to failure. What you'd be better off with is a function that
finds the "next" period given a current period and some suitable
representation of the off-limits intervals. The argument for having
granularity wired into the datatype seems to boil down to just space
savings. I don't find that compelling enough to justify code
contortions and user-visible restrictions on functionality.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T10:57:19Z

Ok, silly question here. But how do you determine the length of a
continuous range? By definition length of [a, b) and (a, b] = b-a. But
what about (a,b) and [a,b]? Are we saying that because they are
continuous, the difference between values included in the range and
those excluded are so infinitesimally small so as not to matter? Thus
length (a,b) == length [a,b] == length [a,b)? And if that is the case,
does the inclusiveness of the range really even matter?

And can anyone point me to a reference for working with continuous
ranges? Google just insists that I spelled contiguous wrong.

Scott

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Update on true serializable techniques in MVCC

2009-12-16T10:30:19Z

Alvaro Herrera <alvherre@...> wrote:

> So you'd have to disable HOT updates when true serializability was
> active?

I wouldn't think so; but someone familiar with HOT logic could
probably determine whether the unmodified algorithm could be used by
reviewing the "simplifying assumptions" near the bottom of page 42,
and the page about "Generalizing to other database engines" (section
4.8). I think those portions might stand on their own without
reading the rest of the document.

-Kevin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Update on true serializable techniques in MVCC

2009-12-16T10:29:43Z

Robert Haas írta:

> On Wed, Dec 16, 2009 at 1:25 PM, Robert Haas <robertmhaas@...> wrote:
>
>> On Wed, Dec 16, 2009 at 1:14 PM, Alvaro Herrera
>> <alvherre@...> wrote:
>>
>>> Robert Haas escribió:
>>>
>>>> On Wed, Dec 16, 2009 at 10:29 AM, Andres Freund <andres@...> wrote:
>>>>
>>>>> On Wednesday 16 December 2009 16:24:42 Robert Haas wrote:
>>>>>
>>>>>>> Inserts and deletes follow the same protocol, obtaining an exclusive
>>>>>>> lock on the row after the one being inserted or deleted. The result
>>>>>>> of this locking protocol is that a range scan prevents concurrent
>>>>>>> inserts or delete within the range of the scan, and vice versa.
>>>>>>>
>>>>>>> That sounds like it should actually work.
>>>>>>>
>>>>>> Only if you can guarantee that the database will access the rows using
>>>>>> some particular index. If it gets to the data some other way it might
>>>>>> accidentally circumvent the lock. That's kind of a killer in terms of
>>>>>> making this work for PostgreSQL.
>>>>>>
>>>>> Isnt the whole topic only relevant for writing access? There you have to
>>>>> access the index anyway.
>>>>>
>>>> Yeah, I guess you have to insert the new tuple. I guess while you
>>>> were at it you might check whether the next tuple is locked...
>>>>
>>> So you'd have to disable HOT updates when true serializability was
>>> active?
>>>
>> I thought about that, but I don't think so. HOT only applies to
>> updates, and predicate locking only applies to inserts. Unless I have
>> my head in the sand?
>>
>
> Err, no, wait. Predicate locking can apply to updates, but since HOT
> updates never update an indexed column, I think we might still be OK?
>

A predicate can include columns from an index plus others.
Am I missing something?

> ...Robert
>
>

--
Bible has answers for everything. Proof:
"But let your communication be, Yea, yea; Nay, nay: for whatsoever is more
than these cometh of evil." (Matthew 5:37) - basics of digital technology.
"May your kingdom come" - superficial description of plate tectonics

----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
http://www.postgresql.at/

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Update on true serializable techniques in MVCC

2009-12-16T10:27:08Z

On Wed, Dec 16, 2009 at 1:25 PM, Robert Haas <robertmhaas@...> wrote:

> On Wed, Dec 16, 2009 at 1:14 PM, Alvaro Herrera
> <alvherre@...> wrote:
>> Robert Haas escribió:
>>> On Wed, Dec 16, 2009 at 10:29 AM, Andres Freund <andres@...> wrote:
>>> > On Wednesday 16 December 2009 16:24:42 Robert Haas wrote:
>>> >> > Inserts and deletes follow the same protocol, obtaining an exclusive
>>> >> > lock on the row after the one being inserted or deleted. The result
>>> >> > of this locking protocol is that a range scan prevents concurrent
>>> >> > inserts or delete within the range of the scan, and vice versa.
>>> >> >
>>> >> > That sounds like it should actually work.
>>> >>
>>> >> Only if you can guarantee that the database will access the rows using
>>> >> some particular index. If it gets to the data some other way it might
>>> >> accidentally circumvent the lock. That's kind of a killer in terms of
>>> >> making this work for PostgreSQL.
>>> > Isnt the whole topic only relevant for writing access? There you have to
>>> > access the index anyway.
>>>
>>> Yeah, I guess you have to insert the new tuple. I guess while you
>>> were at it you might check whether the next tuple is locked...
>>
>> So you'd have to disable HOT updates when true serializability was
>> active?
>
> I thought about that, but I don't think so. HOT only applies to
> updates, and predicate locking only applies to inserts. Unless I have
> my head in the sand?

Err, no, wait. Predicate locking can apply to updates, but since HOT
updates never update an indexed column, I think we might still be OK?

...Robert

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Update on true serializable techniques in MVCC

2009-12-16T10:25:48Z

On Wed, Dec 16, 2009 at 1:14 PM, Alvaro Herrera
<alvherre@...> wrote:

> Robert Haas escribió:
>> On Wed, Dec 16, 2009 at 10:29 AM, Andres Freund <andres@...> wrote:
>> > On Wednesday 16 December 2009 16:24:42 Robert Haas wrote:
>> >> > Inserts and deletes follow the same protocol, obtaining an exclusive
>> >> > lock on the row after the one being inserted or deleted. The result
>> >> > of this locking protocol is that a range scan prevents concurrent
>> >> > inserts or delete within the range of the scan, and vice versa.
>> >> >
>> >> > That sounds like it should actually work.
>> >>
>> >> Only if you can guarantee that the database will access the rows using
>> >> some particular index. If it gets to the data some other way it might
>> >> accidentally circumvent the lock. That's kind of a killer in terms of
>> >> making this work for PostgreSQL.
>> > Isnt the whole topic only relevant for writing access? There you have to
>> > access the index anyway.
>>
>> Yeah, I guess you have to insert the new tuple. I guess while you
>> were at it you might check whether the next tuple is locked...
>
> So you'd have to disable HOT updates when true serializability was
> active?

I thought about that, but I don't think so. HOT only applies to
updates, and predicate locking only applies to inserts. Unless I have
my head in the sand?

...Robert

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Range types

2009-12-16T10:21:43Z

Tom Lane wrote:
> Jeff Davis <pgsql@...> writes:
>> [ hacky special-case representation for discrete timestamp ranges ]
>
> I'm still not exactly clear on what the use-case is for discrete
> timestamp ranges, and I wonder how many people are going to be happy
> with a representation that can't handle a range that's open-ended
> on the left.

They wouldn't. But the timestamp data would be the anchor, not
necessarily the start point. As long as we ranges unbounded on both ends
we'd be ok.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Update on true serializable techniques in MVCC

2009-12-16T10:14:22Z

Robert Haas escribió:

> On Wed, Dec 16, 2009 at 10:29 AM, Andres Freund <andres@...> wrote:
> > On Wednesday 16 December 2009 16:24:42 Robert Haas wrote:
> >> > Inserts and deletes follow the same protocol, obtaining an exclusive
> >> > lock on the row after the one being inserted or deleted. The result
> >> > of this locking protocol is that a range scan prevents concurrent
> >> > inserts or delete within the range of the scan, and vice versa.
> >> >
> >> > That sounds like it should actually work.
> >>
> >> Only if you can guarantee that the database will access the rows using
> >> some particular index. If it gets to the data some other way it might
> >> accidentally circumvent the lock. That's kind of a killer in terms of
> >> making this work for PostgreSQL.
> > Isnt the whole topic only relevant for writing access? There you have to
> > access the index anyway.
>
> Yeah, I guess you have to insert the new tuple. I guess while you
> were at it you might check whether the next tuple is locked...

So you'd have to disable HOT updates when true serializability was
active?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers