Upgrading our minimum required flex version for 8.5

[Tom Lane-2]

"Kevin Grittner" <Kevin.Grittner@...> writes:
> I seem to be able to "sneak up on it from behind" by doing a regular
> make and then "make distclean" and copying the results.  Perhaps
> someone knows off-hand what I'm missing that prevents "make dist" from
> working.  The attempt ends with:

> jade  -D .  -d stylesheet.dsl -i output-text -t sgml -V nochunks
> tempfile_HISTORY.sgml >HISTORY.html
> /bin/sh: jade: not found

Looks like you're missing openjade.  However, that's only used to
produce the HISTORY file which is hardly critical, so copying your
source tree after building is probably a perfectly good answer.

                        regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Kevin Grittner]

Tom Lane <tgl@...> wrote:
> "Kevin Grittner" <Kevin.Grittner@...> writes:
 
>> jade  -D .  -d stylesheet.dsl -i output-text -t sgml -V nochunks
>> tempfile_HISTORY.sgml >HISTORY.html
>> /bin/sh: jade: not found
>
> Looks like you're missing openjade.  However, that's only used to
> produce the HISTORY file which is hardly critical, so copying your
> source tree after building is probably a perfectly good answer.
 
Thanks.  I was able to install jade, but it throws pages of errors.
Rather than try to get that working, I'll just count on the other
approach.  I'd want to make it and check it on my machine first,
anyway, so the distclean is no big deal at that point.
 
-Kevin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Peter Eisentraut-2]

On Thursday 02 July 2009 19:46:04 Tom Lane wrote:

> "Kevin Grittner" <Kevin.Grittner@...> writes:
> > Then I take it back -- the new flex versions would have little or no
> > impact on me.  Worst case, I might need to download a snapshot to
> > apply my patch for testing on the "big" machines.  If I understood
> > what make options I could use on my machine to create derived files
> > for other machines, even that would go away.
>
> Peter or Marc could clue you in a bit better, but I think it's as simple
> as saying "make dist" at the top level of a modified source tree.  This
> gets you a source tarball the same way the release tarballs are made.

make distprep is what prepares those files.  make dist depends on make
distprep and then builds a tarball out of everything.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Jaime Casanova-2]

On Thu, Jul 2, 2009 at 12:13 PM, Tom Lane<tgl@...> wrote:
> However I'm a bit worried about the situation for Windows --- does
> anyone know whether a newer flex is readily available for Windows?
>

MSYS Suplementary Tools (for mingw) includes flex-2.5.33
http://sourceforge.net/projects/mingw/files/

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. [phone]

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Kevin Grittner]

Peter Eisentraut <peter_e@...> wrote:
 
> make distprep is what prepares those files.
 
Perfect!  Flex files built.  No errors.  Thanks much!
 
-Kevin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Zdenek Kotala]

Tom Lane píše v čt 02. 07. 2009 v 11:32 -0400:

> Andrew Dunstan <andrew@...> writes:
> > Tom Lane wrote:
> >> Right, this will only affect people doing development or otherwise
> >> building from a CVS pull.
>
> > Of course, that includes the whole buildfarm. We might need to ask some
> > people to upgrade there.
>
> Yes.  What I was thinking of doing was committing a configure change to
> reject flex < 2.5.31, and waiting to see how much of the buildfarm goes
> red.

Solaris 10 has 2.5.4 version. OpenSolaris already has 2.5.33.

                Zdenek


--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Zdenek Kotala]

Tom Lane píše v čt 02. 07. 2009 v 13:13 -0400:

> I wrote:
> > Yes.  What I was thinking of doing was committing a configure change to
> > reject flex < 2.5.31, and waiting to see how much of the buildfarm goes
> > red.
>
> Actually, most of the buildfarm members show which flex version they are
> running in the configure output.  A quick look shows that of the 45
> members that have reported on HEAD in the past 2 days, 22 are running
> 2.5.4, which is a lot higher than I was expecting.  Most of these are
> the Solaris boxen, which I imagine can be updated fairly painlessly
> since there are some of them that are already running something newer.

I hope, need to check/compile, but I don't expect any problems.

        Zdenek


--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Zdenek Kotala]

Tom Lane píše v čt 02. 07. 2009 v 13:13 -0400:

>
> Actually, most of the buildfarm members show which flex version they are
> running in the configure output.  A quick look shows that of the 45
> members that have reported on HEAD in the past 2 days, 22 are running
> 2.5.4, which is a lot higher than I was expecting.  Most of these are
> the Solaris boxen, which I imagine can be updated fairly painlessly
> since there are some of them that are already running something newer.

My three S10 are updated. All of them use version 2.5.35 now. S8, S9 is
not yet under my control and they are not migrated yet to new location.

                Zdenek


--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Tom Lane-2]

Andrew Dunstan <andrew@...> writes:
> Tom Lane wrote:
>> Andrew Dunstan <andrew@...> writes:
>>> I think it would need to be benchmarked. My faint recollection is that
>>> the re-entrant lexers are slower.
>>
>> The flex documentation states in so many words:
>>    The option `--reentrant' does not affect the performance of the scanner.
>> Do you feel a need to verify their claim?

> No, I'll take their word for it. I must have been thinking of something
> else.

As I got further into this, it turned out that Andrew's instinct was
right: it does need to be benchmarked.  Although the inner loops of the
lexer seem to be the same with or without --reentrant, once you buy into
the whole nine yards of --reentrant, --bison-bridge, and a "pure" bison
parser, you find out that the lexer's API changes: there are more
parameters to yylex() than there used to be.  It's also necessary to
pass around a yyscanner pointer to all the subroutines in scan.l.  (But
on the other hand, this eliminates accesses to global variables, which
are often not that cheap.)  So the "no performance impact" claim isn't
telling the whole truth.

As best I can tell after some casual testing on a couple of machines,
the actual bottom line is that "raw_parser" (ie, the bison and flex
processing) is going to be a couple of percent slower with a reentrant
grammar and lexer, for typical queries involving a lot of short tokens.
Now this disappears into the noise as soon as you include parse analysis
(let alone planning and execution), but it is possible to measure the
slowdown in a test harness that calls raw_parser only.

A possible compromise that I think would avoid most or all of the
slowdown is to make the lexer reentrant but not the grammar (so that
yylval and yylloc remain as global variables instead of being parameters
to yylex).  I haven't actually benchmarked that, though.  It strikes
me as a fairly silly thing to do.  If we're going to go for reentrancy
I think we should fix both components.

I'm willing to live with the small slowdown.  Comments?

                        regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Pavel Stehule]

2009/7/12 Tom Lane <tgl@...>:

> Andrew Dunstan <andrew@...> writes:
>> Tom Lane wrote:
>>> Andrew Dunstan <andrew@...> writes:
>>>> I think it would need to be benchmarked. My faint recollection is that
>>>> the re-entrant lexers are slower.
>>>
>>> The flex documentation states in so many words:
>>>    The option `--reentrant' does not affect the performance of the scanner.
>>> Do you feel a need to verify their claim?
>
>> No, I'll take their word for it. I must have been thinking of something
>> else.
>
> As I got further into this, it turned out that Andrew's instinct was
> right: it does need to be benchmarked.  Although the inner loops of the
> lexer seem to be the same with or without --reentrant, once you buy into
> the whole nine yards of --reentrant, --bison-bridge, and a "pure" bison
> parser, you find out that the lexer's API changes: there are more
> parameters to yylex() than there used to be.  It's also necessary to
> pass around a yyscanner pointer to all the subroutines in scan.l.  (But
> on the other hand, this eliminates accesses to global variables, which
> are often not that cheap.)  So the "no performance impact" claim isn't
> telling the whole truth.
>
> As best I can tell after some casual testing on a couple of machines,
> the actual bottom line is that "raw_parser" (ie, the bison and flex
> processing) is going to be a couple of percent slower with a reentrant
> grammar and lexer, for typical queries involving a lot of short tokens.
> Now this disappears into the noise as soon as you include parse analysis
> (let alone planning and execution), but it is possible to measure the
> slowdown in a test harness that calls raw_parser only.
>
> A possible compromise that I think would avoid most or all of the
> slowdown is to make the lexer reentrant but not the grammar (so that
> yylval and yylloc remain as global variables instead of being parameters
> to yylex).  I haven't actually benchmarked that, though.  It strikes
> me as a fairly silly thing to do.  If we're going to go for reentrancy
> I think we should fix both components.

when we don't use reentrant grammar, then we cannot use main sql parser in SQL?

Pavel

>
> I'm willing to live with the small slowdown.  Comments?
>
>                        regards, tom lane
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@...)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Andrew Dunstan]

Tom Lane wrote:

> As best I can tell after some casual testing on a couple of machines,
> the actual bottom line is that "raw_parser" (ie, the bison and flex
> processing) is going to be a couple of percent slower with a reentrant
> grammar and lexer, for typical queries involving a lot of short tokens.
> Now this disappears into the noise as soon as you include parse analysis
> (let alone planning and execution), but it is possible to measure the
> slowdown in a test harness that calls raw_parser only.
>
> A possible compromise that I think would avoid most or all of the
> slowdown is to make the lexer reentrant but not the grammar (so that
> yylval and yylloc remain as global variables instead of being parameters
> to yylex).  I haven't actually benchmarked that, though.  It strikes
> me as a fairly silly thing to do.  If we're going to go for reentrancy
> I think we should fix both components.
>
> I'm willing to live with the small slowdown.  Comments?
>
>
>  

If we're going to have a reentrant lexer, I think we should go the whole
nine yards. I agree that a couple of percent slowdown on just the lexing
and parsing will be lost in the noise. So +1 from me.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Tom Lane-2]

Pavel Stehule <pavel.stehule@...> writes:
> 2009/7/12 Tom Lane <tgl@...>:
>> If we're going to go for reentrancy
>> I think we should fix both components.

> when we don't use reentrant grammar, then we cannot use main sql parser in SQL?

It wouldn't be a problem for the immediate application I have in mind,
which is to re-use the core lexer in plpgsql.  But it does seem like
it might be a problem down the road as plpgsql gets smarter.

                        regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Pavel Stehule]

2009/7/12 Tom Lane <tgl@...>:

> Pavel Stehule <pavel.stehule@...> writes:
>> 2009/7/12 Tom Lane <tgl@...>:
>>> If we're going to go for reentrancy
>>> I think we should fix both components.
>
>> when we don't use reentrant grammar, then we cannot use main sql parser in SQL?
>
> It wouldn't be a problem for the immediate application I have in mind,
> which is to re-use the core lexer in plpgsql.  But it does seem like
> it might be a problem down the road as plpgsql gets smarter.
>

it's bad. I thing so integration main parser into plpgsql should be
the most important feature of plpgsql from trapping exception time. I
have to ask - we need it necessary reetrant grammer? We need
integration only in complilation time - for CREATE FUNCTION statement.
Can be nonreetrant grammer problem (but we have to store some info
from validation time somewhere - maybe in probin column) ?

>                        regards, tom lane
>

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Tom Lane-2]

Andrew Dunstan <andrew@...> writes:
> If we're going to have a reentrant lexer, I think we should go the whole
> nine yards. I agree that a couple of percent slowdown on just the lexing
> and parsing will be lost in the noise. So +1 from me.

I found a couple of places where a few cycles could be shaved.  The
current version of the patch (attached) seems indistinguishable in
speed from 8.4 on my HPPA box, though still a percent or two slower on
my x86_64 box.

This is ready to go except for changing the minimum flex version test
in configure (and associated documentation).  I see that a good-sized
fraction of the buildfarm is still on flex 2.5.4 and will therefore go
red when this goes in.  Should I hold off a bit longer, or is committing
the best way to get their attention?

                        regards, tom lane




--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Andrew Dunstan]

Tom Lane wrote:
> I see that a good-sized
> fraction of the buildfarm is still on flex 2.5.4 and will therefore go
> red when this goes in.  Should I hold off a bit longer, or is committing
> the best way to get their attention?
>
>  

Probably the latter. I will update my various members. I see that 2.5.4
is the default on my FBSD install, which is the latest, so for this and
maybe some other current platforms we'll be imposing a bit of extra
burden to build, but I guess that's the price of progress.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Josh Berkus]

> This is ready to go except for changing the minimum flex version test
> in configure (and associated documentation).  I see that a good-sized
> fraction of the buildfarm is still on flex 2.5.4 and will therefore go
> red when this goes in.  Should I hold off a bit longer, or is committing
> the best way to get their attention?

Oh, I didn't think about the Flex version.  That's going to be a far
more widespread problem.  OSX 10.5, for example, still ships with
2.5.33.  I suspect that a *lot* of OSes won't have anything up-to-date.

--
Josh Berkus
PostgreSQL Experts Inc.
www.pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Andrew Dunstan]

Josh Berkus wrote:

>
>> This is ready to go except for changing the minimum flex version test
>> in configure (and associated documentation).  I see that a good-sized
>> fraction of the buildfarm is still on flex 2.5.4 and will therefore go
>> red when this goes in.  Should I hold off a bit longer, or is committing
>> the best way to get their attention?
>
> Oh, I didn't think about the Flex version.  That's going to be a far
> more widespread problem.  OSX 10.5, for example, still ships with
> 2.5.33.  I suspect that a *lot* of OSes won't have anything up-to-date.
>

That's the version Tom is proposing to make the minimum.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Tom Lane-2]

Andrew Dunstan <andrew@...> writes:
> Josh Berkus wrote:
>> Oh, I didn't think about the Flex version.  That's going to be a far
>> more widespread problem.  OSX 10.5, for example, still ships with
>> 2.5.33.  I suspect that a *lot* of OSes won't have anything up-to-date.

> That's the version Tom is proposing to make the minimum.

Yeah, 2.5.33 is fine (in fact it's what I installed on my HPUX box to
replace 2.5.4).  AFAICS OSX 10.5 is fine, but 10.4 will need a newer
flex version.  This is not anything very new to Mac users.  10.4 also
shipped with bison 1.28, which is too old to build Postgres and has been
too old for many years now.  So anyone building from CVS on that
platform has already installed at least one nondefault tool.

(For comparison's sake: flex 2.5.4 was released in 1996.  2.5.31, which
I'm proposing to make the new minimum, was released in 2003.  Bison
1.875, our current minimum for that tool, was released in 2002, and
we made it the minimum required version in 2003.)

                        regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Kevin Grittner]

Tom Lane <tgl@...> wrote:
 
> Now this disappears into the noise as soon as you include parse
> analysis (let alone planning and execution)
 
No need to go for silly options to avoid a performance issue at that
level.  Time wasted dealing with subsequent silliness would almost
certainly have a higher "lost opportunity" cost regarding things that
matter more.
 
-Kevin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Andrew Dunstan]

Andrew Dunstan wrote:

>
>
> Tom Lane wrote:
>> I see that a good-sized
>> fraction of the buildfarm is still on flex 2.5.4 and will therefore go
>> red when this goes in.  Should I hold off a bit longer, or is committing
>> the best way to get their attention?
>>
>>  
>
> Probably the latter. I will update my various members. I see that
> 2.5.4 is the default on my FBSD install, which is the latest, so for
> this and maybe some other current platforms we'll be imposing a bit of
> extra burden to build, but I guess that's the price of progress.
>
>

OK, the fly in this ointment turns out to be MSVC. The latest flex from
GnuWin32 is 2.5.4a, and building 2.5.35 for Windows is turning out to be
quite a pain. Luckily, MinGW has a pre-built modified 2.5.33 available,
and I have installed this (also needed msys-regex), and then I jury
rigged my MSVC build to use it, so we still have one MSVC working OK in
the buildfarm. But we can hardly ask people to install MinGW/MSys so
they can build with MSVC, that's horribly ugly. I'll work on getting
version 2.5.35 build for Windows in a way that works standalone, and
push it somewhere (maybe the dev wiki).

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers