Core team statement on replication in PostgreSQL

[Merlin Moncure-2]

On Sun, Jun 1, 2008 at 11:58 AM, Robert Hodges
<robert.hodges@...> wrote:
> Hi Merlin,
>
> My point here is that with reasonably small extensions to the core you can
> build products that are a lot better than SLONY.   Triggers do not cover
> DDL, among other issues, and it's debatable whether they are the best way to
> implement quorum policies like Google's semi-synchronous replication.  As I
> mentioned separately this topic deserves another thread which I promise to
> start.

These issues are much discussed and well understood.  At this point,
the outstanding points of discussion are technical...how to make this
thing work.

merlin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Tom Lane-2]

"Merlin Moncure" <mmoncure@...> writes:
> On Sun, Jun 1, 2008 at 11:58 AM, Robert Hodges
> <robert.hodges@...> wrote:
>> My point here is that with reasonably small extensions to the core you can
>> build products that are a lot better than SLONY.

> These issues are much discussed and well understood.

Well, what we know is that previous attempts to define replication hooks
to be added to the core have died for lack of interest.  Maybe Robert
can start a new discussion that will actually get somewhere; if so, more
power to him.  (Is the replica-hooks-discuss list still working?)  But
that is entirely orthogonal to what is proposed in this thread, which
is to upgrade the existing PITR support into a reasonably useful
replication feature.

                        regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Dawid Kuroczko]

On Thu, May 29, 2008 at 4:12 PM, Tom Lane <tgl@...> wrote:
> The Postgres core team met at PGCon to discuss a few issues, the largest
> of which is the need for simple, built-in replication for PostgreSQL.
[...]
> We believe that the most appropriate base technology for this is
1> probably real-time WAL log shipping, as was demoed by NTT OSS at PGCon.

> We hope that such a feature can be completed for 8.4.  Ideally this
> would be coupled with the ability to execute read-only queries on the
> slave servers, but we see technical difficulties that might prevent that
> from being completed before 8.5 or even further out.  (The big problem
> is that long-running slave-side queries might still need tuples that are
> vacuumable on the master, and so replication of vacuuming actions would
> cause the slave's queries to deliver wrong answers.)
>
> Again, this will not replace Slony, pgPool, Continuent, Londiste, or
> other systems for many users, as it will be not be highly scalable nor
> support long-distance replication nor replicating less than an entire
> installation.  But it is time to include a simple, reliable basic
> replication feature in the core system.

Hello!

I thought I would share a few thoughts of my own about the issue.
I have a hands-on experience with Oracle and MySQL apart from
PostgreSQL so I hope it will be a bit interesting.

The former has a feature called "physical standby", which looks
quite like our WAL-shipping based replication.  Simply archived
logs are replayed on the standby database.  A primary database
and standby database are connected, and can stream the logs
directly.  They either copy the log when its finished (as we do now)
or can do it in coninuous manner (as I hope we will be able to).

It is possible to have a synchronous replication (where "COMMIT"
on primary database succeeds when the data is safely stored on
the standby database).  I think such a feature would be a great
advantage for PostgreSQL (where you cannot afford to loose
any transactions).

Their standby database is not accessible.  It can be opened read-only,
but during that time replication stops.  So PostgreSQL having
read-only and still replicating standby database would be great.

The other method is "logical standby" which works by dissecting
WAL-logs and recreating DDLs/DMLs from it.  Never seen anyone
use it. ;-)

Then we have a mysql replication -- done by replaying actual DDLs/DMLs
on the slaves.  This approach has issues, most notably when slaves are
highly loaded and lag behind the master -- so you end up with infrastructure
to monitor lags and turn off slaves which lag too much.  Also it is painful
to setup -- you have to stop, copy, configure and run.

* Back to PostgreSQL world

As for PostgreSQL solutions we have a slony-I, which is great as long as
you don't have too many people managing the database and/or your
schema doesn't change too frequently.  Perhaps it would be maintainable
more easily if there would be to get DDLs (as DDL triggers or similar).
Its main advantages for me is ability to prepare complex setups and
easily add new slaves).  The pgpool solution is quite nice but then
again adding a new slave is not so easy.  And being a filtering
layer between client and server it feels a bit "fragile" (I know it is not,
but then again it is harder to convince someone that "yes it will work
100% right all the time").

* How I would like PostgreSQL WAL-replication to evolve:

First of all it would be great if a slave/standby would contact the master
and maintain the state with it (tell it its xmin, request a log to stream,
go online-streaming).  Especially I hope that it should be possible
to make a switchover (where the two databases exchange roles),
and in this the direct connection between the two should help.

In detail, I think it should go like this:
* A slave database starts up, checks that it works as a "replica"
(hopefully it would not be a postgresql.conf constant, but rather
some file maintained by the database).
* It would connect to the master database, tell where in the WAL
it is now, and request a log N.
* If log N is not available, request a log from external supplied
script (so that it could be fetched from log archive repository
somewhere, recovered from a backup tape, etc).
* Continue asking, until we get to the logs which are available
at master database.
* Continue replaying until we get within max_allowed_replication_lag
time, and open our slave for read-only queries.
* If we start lagging too much perhaps close the read-only access
to the database (perhaps configurable?).

I think that replication should be easy to set up.  I think our
archive_command is quite easy, but many a person come
with a lot of misconceptions how it works (and it takes time
to explain them how it actually work, especially what is
archive_command for, and that pg_start_backup() doesn't
actually _do_ backup, but just tells PostgreSQL that
backup is being done).

Easy to setup and easy to switchover (change the direction
of replication) and failover.

Also, it would be interesting to be able to have a shared-storage
solution where a single database would modify data and multiple
read-only slaves would access the data.  The WAL-shipping would
be used then to "push" modified pages into read-only slaves' memory.
And each checkpoint would allow read-only slaves to release these
pages from memory.  Kind of replication without actually doing any
writes.

* high available  libpq?

Other thing worth to consider is perhaps adding a HA-capability
to libpq.  Currently I use a setup where I have machine with
database and pg_bouncer (for connection pooling at database
level) and on client machines I have a pgpool on localhost
(for transparent failover).  I think some level of replication
awareness could be built-in into libpq.  It would allow to make
simpler, but still HA, setups.

Perhaps ability to specify a standby in connect string, something like:

  "host=foo1.bar;db=pgdb;failover=yes;stbyhost=foo2.bar"

...a failover="yes" would allow libpq to try connecting to stbyhost
if host is not available AND succeed if stbyhost announces itself
as a new master.

This would make switchover easier (clients would be told, that
we are closing, and that they should connect to a new master).

Also some way of supplying multiple hosts, where client should
pick a host at random, would be quite useful, say:

  "host=foo1.bar,foo2.bar,foo3.bar;db=pgdb;loadbalance=random"

Why in libpq and not in [language/framework/application]?  Well
it would help with not reinventing the wheel.  And if we want
a painless switchover, client should be aware that there is
a standby ready.  Also, being optional it wouldn't change anything
for those who don't want it.

   Regards,
      Dawid

PS: As for pgpool/pg_bouncer -- I wish there would be a way for
them to tell the database that they are proxying for someone
else (so that I could check in pg_stat_activity who is on the far
end of the database connection, not "localhost" ;)).
--
Solving [site load issues] with [more database replication] is a lot
like solving your own personal problems with heroin - at first it
sorta works, but after a while things just get out of hand.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[James Mansion-2]

David Fetter wrote:
> This part is a deal-killer.  It's a giant up-hill slog to sell warm
> standby to those in charge of making resources available because the
> warm standby machine consumes SA time, bandwidth, power, rack space,
> etc., but provides no tangible benefit, and this feature would have
> exactly the same problem.
>
> IMHO, without the ability to do read-only queries on slaves, it's not
> worth doing this feature at all.
>  
That's not something that squares with my experience *at all*, which
admitedly is entirely in
investment banks. Business continuity is king, and in some places the
warm standby rep
from the database vendor is trusted more than block-level rep from the
SAN vendor
(though that may be changing to some extent in favour of the SAN).

James


--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[James Mansion-2]

Aidan Van Dyk wrote:
> The whole single-threaded WAL replay problem is going to rear it's ugly
> head here too, and mean that a slave *won't* be able to keep up with a
> busy master if it's actually trying to apply all the changes in
> real-time.
Is there a reason to commit at the same points that the master
committed?  Wouldn't relaxing
that mean that at least you would get 'big' commits and some economy of
scale?  It might
not be too bad.  All I can say is that Sybase warm standby is useful,
even though the rep
for an update that changes a hundred rows is a hundred updates keyed on
primary key,
which is pretty sucky in terms of T-SQL performance.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Hannu Krosing-3]

On Fri, 2008-05-30 at 15:16 -0400, Robert Treat wrote:

> On Friday 30 May 2008 01:10:20 Tom Lane wrote:
> > Greg Smith <gsmith@...> writes:
> > > I fully accept that it may be the case that it doesn't make technical
> > > sense to tackle them in any order besides sync->read-only slaves because
> > > of dependencies in the implementation between the two.
> >
> > Well, it's certainly not been my intention to suggest that no one should
> > start work on read-only-slaves before we finish the other part.  The
> > point is that I expect the log shipping issues will be done first
> > because they're easier, and it would be pointless to not release that
> > feature if we had it.
> >
> > But since you mention it: one of the plausible answers for fixing the
> > vacuum problem for read-only slaves is to have the slaves push an xmin
> > back upstream to the master to prevent premature vacuuming.  The current
> > design of pg_standby is utterly incapable of handling that requirement.
> > So there might be an implementation dependency there, depending on how
> > we want to solve that problem.
> >
>
> Sure, but whose to say that after synchronous wal shipping is "finished" it
> wont need a serious re-write due to new needs from the hot standby feature. I
> think going either way carries some risk.

The simplest form of synchronous wal shipping would not even need
postgresql running on slave, just a small daemon which reports when wal
blocks are a) received and b) synced to disk.

This setup would just guarantee "no data loss on single machine
failure". form there on you could add various features, including
support for both switchover and failover, async replication to multiple
slaves, etc.

the only thing that needs anything additional from slave wal-receiving
daemon is when you want the kind of wal-sync which would guarantee that
read-only query on slave issued after commit returns from master sees
latest data. for this kinds of guarantees you need at least feedback
about wal-replay, but possibly also shared transaction numbers and
shared snapshots, to be sure that OLTP type queries see the latest and
OLAP queries are not denied seeing VACUUMED on master.

--------------
Hannu



--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Hannu Krosing-3]

On Thu, 2008-05-29 at 13:37 -0400, Tom Lane wrote:

> David Fetter <david@...> writes:
> > On Thu, May 29, 2008 at 08:46:22AM -0700, Joshua D. Drake wrote:
> >> The only question I have is... what does this give us that PITR
> >> doesn't give us?
>
> > It looks like a wrapper for PITR to me, so the gain would be ease of
> > use.
>
> A couple of points about that:
>
> * Yeah, ease of use is a huge concern here.  We're getting beat up
> because people have to go find a separate package (and figure out
> which one they want), install it, learn how to use it, etc.  It doesn't
> help that the most mature package is Slony which is, um, not very
> novice-friendly or low-admin-complexity.  I personally got religion
> on this about two months ago when Red Hat switched their bugzilla
> from Postgres to MySQL because the admins didn't want to deal with Slony
> any more.  People want simple.
>
> * The proposed approach is trying to get to "real" replication
> incrementally.  Getting rid of the loss window involved in file-by-file
> log shipping is step one,

Actually we can already do better than file-by-file by using
pg_xlogfile_name_offset() which was added sometime in 2006. SkyTools for
example does this to get no more than a few seconds failure window.

Doing this synchronously would be of course better.

probably we should use the same "modes/protocols" as DRBD when
determining when a "sync" wal write is "done"

quote from 
http://www.slackworks.com/~dkrovich/DRBD/usingdrbdsetup.html#AEN76


Table 1. DRBD Protocols

Protocol
            Description
                 A
A write operation is complete as
soon as the data is written to disk
and sent to the network.
                 B
A write operation is complete as
soon as a reception acknowledgement
arrives.
                 C
A write operation is complete as
soon as a write acknowledgement
arrives.

There are also additional paramaters you can pass to the disk and net
options. See the drbdsetup man page for additional information

/end quote

> and I suspect that step two is going to be
> fixing performance issues in WAL replay to ensure that slaves can keep
> up.  After that we'd start thinking about how to let slaves run
> read-only queries.  But even without read-only queries, this will be
> a useful improvement for HA/backup scenarios.
>
> regards, tom lane
>


--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Hannu Krosing-3]

On Thu, 2008-05-29 at 12:05 -0700, Robert Hodges wrote:

> Hi everyone,
>
> First of all, I’m absolutely delighted that the PG community is
> thinking seriously about replication.  
>
> Second, having a solid, easy-to-use database availability solution
> that works more or less out of the box would be an enormous benefit to
> customers.  Availability is the single biggest problem for customers
> in my experience and as other people have commented the alternatives
> are not nice.  It’s an excellent idea to build off an existing feature
> —PITR is already pretty useful and the proposed features are solid
> next steps.  The fact that it does not solve all problems is not a
> drawback but means it’s likely to get done in a reasonable timeframe.
>
> Third, you can’t stop with just this feature.  (This is the BUT part
> of the post.)  The use cases not covered by this feature area actually
> pretty large.  Here are a few that concern me:
>
> 1.) Partial replication.
> 2.) WAN replication.

1.) & 2.) are better done asunc, the domain of Slony-I/Londiste

> 3.) Bi-directional replication.  (Yes, this is evil but there are
> problems where it is indispensable.)

Sure, it is also a lot harder and always has several dimensions
(performanse/availability7locking) which play against each other

> 4.) Upgrade support.  Aside from database upgrade (how would this ever
> really work between versions?), it would not support zero-downtime app
> upgrades, which depend on bi-directional replication tricks.

Or you could use zero-downtime  app upgrades, which don't depend on
this :P

> 5.) Heterogeneous replication.
> 6.) Finally, performance scaling using scale-out over large numbers of
> replicas.  I think it’s possible to get tunnel vision on this—it’s not
> a big requirement in the PG community because people don’t use PG in
> the first place when they want to do this.  They use MySQL, which has
> very good replication for performance scaling, though it’s rather weak
> for availability.  

Again, doing scale-out over large number of replicas should either be
async or for sync use some broadcast channel to all slaves (and still be
a performance problem on master, as it has to wait for slowest slave).

> As a consequence, I don’t see how you can get around doing some sort
> of row-based replication like all the other databases.

Is'nt WAL-base replication "some sort of row-based replication" ?

>  Now that people are starting to get religion on this issue I would
> strongly advocate a parallel effort to put in a change-set extraction
> API that would allow construction of comprehensive master/slave
> replication.

Triggers. see pgQ's logtrigga()/logutrigga(). See slides for Marko
Kreen's presentation at pgCon08.

>  (Another approach would be to make it possible for third party apps
> to read the logs and regenerate SQL.)

which logs ? WAL or SQL command logs ?

> There are existing models for how to do change set extraction; we have
> done it several times at my company already.  There are also research
> projects like GORDA that have looked fairly comprehensively at this
> problem.

pgQ with its triggers does a pretty good job of change-set extraction.

------------------
Hannu



--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Robert Hodges]

Hi Hannu,

Hi Hannu,


On 6/1/08 2:14 PM, "Hannu Krosing" <hannu@...> wrote:

>
>> As a consequence, I don¹t see how you can get around doing some sort
>> of row-based replication like all the other databases.
>
> Is'nt WAL-base replication "some sort of row-based replication" ?
>
Yes, in theory.  However, there's a big difference between replicating
physical WAL records and doing logical replication with SQL statements.
Logical replication requires extra information to reconstruct primary keys.
(Somebody tell me if this is already in the WAL; I'm learning the code as
fast as possible but assuming for now it's not.)

>
>>  Now that people are starting to get religion on this issue I would
>> strongly advocate a parallel effort to put in a change-set extraction
>> API that would allow construction of comprehensive master/slave
>> replication.
>
> Triggers. see pgQ's logtrigga()/logutrigga(). See slides for Marko
> Kreen's presentation at pgCon08.
>
>

Thanks very much for the pointer.  The slides look interesting.

Robert


--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Chris Browne]

adsmail@... ("Andreas 'ads' Scherbaum") writes:

> On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote:
>
>> Well, yes, but you do know about archive_timeout, right? No need to wait
>> 2 hours.
>
> Then you ship 16 MB binary stuff every 30 second or every minute but
> you only have some kbyte real data in the logfile. This must be taken
> into account, especially if you ship the logfile over the internet
> (means: no high-speed connection, maybe even pay-per-traffic) to the
> slave.

If you have that kind of scenario, then you have painted yourself into
a corner, and there isn't anything that can be done to extract you
from it.

Consider: If you have so much update traffic that it is too much to
replicate via WAL-copying, why should we expect that other mechanisms
*wouldn't* also overflow the connection?

If you haven't got enough network bandwidth to use this feature, then
nobody is requiring that you use it.  It seems like a perfectly
reasonable prerequisite to say "this requires that you have enough
bandwidth."
--
(reverse (concatenate 'string "ofni.secnanifxunil" "@" "enworbbc"))
http://www3.sympatico.ca/cbbrowne/
"There's nothing worse than having only one drunk head."
-- Zaphod Beeblebrox

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Hannu Krosing-3]

On Thu, 2008-05-29 at 23:37 +0200, Mathias Brossard wrote:

> I pointed out that the NTT solution is synchronous because Tom said in
> the first part of his email that:
>
>  > In practice, simple asynchronous single-master-multiple-slave
>  > replication covers a respectable fraction of use cases, so we have
>  > concluded that we should allow such a feature to be included in the
>  > core project.
>
> ... and yet "the most appropriate base technology for this" is
> synchronous and maybe I should have also pointed out in my previous mail
> is that it doesn't support multiple slaves.

I don't think that you need too many slaves in sync mode.

Probably 1-st slave sync and others async from there on will be good
enough.

> Also, as other have pointed out there are different interpretations of
> "synchronous" depending on wether the WAL data has reached the other end
> of the network connection, a safe disk checkpoint or the slave DB itself.

Probably all DRBD-s levels ( A) data sent to network, B) data received,
C) data written to disk) should be supported + C1) data replayed in
slave DB. C1 meaning that it can be done in parallel with C)

Then each DBA can set it up depending on what he trusts - network,
slave's power supply or slaves' disk.

Also, the case of slave failure should be addressed. I don't think that
the best solution is halting all ops on master if slave/network fails.

Maybe we should allow also a setup with 2-3 slaves, where operations can
continue when at least 1 slave is "syncing" ?

--------------
Hannu


--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Andreas 'ads' Scherbaum]

On Mon, 02 Jun 2008 11:52:05 -0400 Chris Browne wrote:

> adsmail@... ("Andreas 'ads' Scherbaum") writes:
> > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote:
> >
> >> Well, yes, but you do know about archive_timeout, right? No need to wait
> >> 2 hours.
> >
> > Then you ship 16 MB binary stuff every 30 second or every minute but
> > you only have some kbyte real data in the logfile. This must be taken
> > into account, especially if you ship the logfile over the internet
> > (means: no high-speed connection, maybe even pay-per-traffic) to the
> > slave.
>
> If you have that kind of scenario, then you have painted yourself into
> a corner, and there isn't anything that can be done to extract you
> from it.

You are misunderstanding something. It's perfectly possible that you
have a low-traffic database with changes every now and then. But you
have to copy a full 16 MB logfile every 30 seconds or every minute just
to have the slave up-to-date.


> Consider: If you have so much update traffic that it is too much to
> replicate via WAL-copying, why should we expect that other mechanisms
> *wouldn't* also overflow the connection?

For some MB real data you copy several GB logfiles per day - that's a
lot overhead, isn't it?


> If you haven't got enough network bandwidth to use this feature, then
> nobody is requiring that you use it.  It seems like a perfectly
> reasonable prerequisite to say "this requires that you have enough
> bandwidth."

If you have a high-traffic database, then of course you need an other
connection as if you only have a low-traffic or a mostly read-only
database. But that's not the point. Copying an almost unused 16 MB WAL
logfile is just overhead - especially because the logfile is not
compressable very much because of all the leftovers from earlier use.


Kind regards

--
                                Andreas 'ads' Scherbaum
German PostgreSQL User Group

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Hannu Krosing-3]

On Mon, 2008-06-02 at 22:40 +0200, Andreas 'ads' Scherbaum wrote:

> On Mon, 02 Jun 2008 11:52:05 -0400 Chris Browne wrote:
>
> > adsmail@... ("Andreas 'ads' Scherbaum") writes:
> > > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote:
> > >
> > >> Well, yes, but you do know about archive_timeout, right? No need to wait
> > >> 2 hours.
> > >
> > > Then you ship 16 MB binary stuff every 30 second or every minute but
> > > you only have some kbyte real data in the logfile. This must be taken
> > > into account, especially if you ship the logfile over the internet
> > > (means: no high-speed connection, maybe even pay-per-traffic) to the
> > > slave.
> >
> > If you have that kind of scenario, then you have painted yourself into
> > a corner, and there isn't anything that can be done to extract you
> > from it.
>
> You are misunderstanding something. It's perfectly possible that you
> have a low-traffic database with changes every now and then. But you
> have to copy a full 16 MB logfile every 30 seconds or every minute just
> to have the slave up-to-date.

To repeat my other post in this thread:

Actually we can already do better than file-by-file by using
pg_xlogfile_name_offset() which was added sometime in 2006. walmgr.py
from SkyTools package for example does this to get no more than a few
seconds failure window and it copies just the changed part of WAL to
slave.

pg_xlogfile_name_offset() was added just for this purpose - to enable
WAL shipping scripts to query, where inside the logfile current write
pointer is.

It is not synchronous, but it can be made very close, within subsecond
if you poll it frequently enough.

-------------------
Hannu



--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Andrew Sullivan-5]

On Sun, Jun 01, 2008 at 01:43:22PM -0400, Tom Lane wrote:
> power to him.  (Is the replica-hooks-discuss list still working?)  But

Yes.  And silent as ever. :-)

A

--
Andrew Sullivan
ajs@...
[phone] x104
http://www.commandprompt.com/

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Stephen Denne]

Hannu Krosing wrote:
> The simplest form of synchronous wal shipping would not even need
> postgresql running on slave, just a small daemon which
> reports when wal
> blocks are a) received and b) synced to disk.

While that does sound simple, I'd presume that most people would want the guarantee of the same version of postgresql installed wherever the logs are ending up, with the log receiver speaking the same protocol version as the log sender. I imagine that would be most easily achieved through using something like the continuously restoring startup mode of current postgresql.

However variations on this kind of daemon can be used to perform testing, configuring it to work well, go slow, pause, not respond, disconnect, or fail in particular ways, emulating disk full, etc.

Regards,
Stephen Denne.
--
At the Datamail Group we value teamwork, respect, achievement, client focus, and courage.
This email with any attachments is confidential and may be subject to legal privilege.  
If it is not intended for you please advise by replying immediately, destroy it and do not
copy, disclose or use it in any way.

The Datamail Group, through our GoGreen programme, is committed to environmental sustainability.  
Help us in our efforts by not printing this email.
__________________________________________________________________
  This email has been scanned by the DMZGlobal Business Quality
              Electronic Messaging Suite.
Please see http://www.dmzglobal.com/dmzmessaging.htm for details.
__________________________________________________________________



--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Markus Wanner-2]

Hello Andrew,

Andrew Sullivan wrote:
> Yes.  And silent as ever. :-)

Are the slides of your PgCon talk available for download somewhere?

BTW: up until recently, there was yet another mailing list:
pgreplication-general@.... It was less focused on hooks
and got at least some traffic. :-) Are those mails still archived somewhere?

Regards

Markus




--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Heikki Linnakangas-2]

Stephen Denne wrote:
> Hannu Krosing wrote:
>> The simplest form of synchronous wal shipping would not even need
>> postgresql running on slave, just a small daemon which
>> reports when wal
>> blocks are a) received and b) synced to disk.
>
> While that does sound simple, I'd presume that most people would want the guarantee of the same version of postgresql installed wherever the logs are ending up, with the log receiver speaking the same protocol version as the log sender. I imagine that would be most easily achieved through using something like the continuously restoring startup mode of current postgresql.

Hmm, WAL version compatibility is an interesting question. Most minor
releases hasn't changed the WAL format, and it would be nice to allow
running different minor versions in the master and slave in those cases.
But it's certainly not unheard of to change the WAL format. Perhaps we
should introduce a WAL version number, similar to catalog version?

--
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Teodor Sigaev-2]

>
> Hmm, WAL version compatibility is an interesting question. Most minor
> releases hasn't changed the WAL format, and it would be nice to allow
As I remember, high minor version should read all WALs from lowers, but it isn't
true for opposite case and between different major versions.

> running different minor versions in the master and slave in those cases.
> But it's certainly not unheard of to change the WAL format. Perhaps we
> should introduce a WAL version number, similar to catalog version?

Agree. Right now it only touches warm-stand-by servers, but introducing simple
log-shipping and based on it replication will cause a lot of unobvious
errors/bugs. Is it possible to use catalog version number as WAL version?

--
Teodor Sigaev                                   E-mail: teodor@...
                                                    WWW: http://www.sigaev.ru/

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Csaba Nagy]

On Wed, 2008-06-04 at 11:13 +0300, Heikki Linnakangas wrote:
> Hmm, WAL version compatibility is an interesting question. Most minor
> releases hasn't changed the WAL format, and it would be nice to allow
> running different minor versions in the master and slave in those cases.
> But it's certainly not unheard of to change the WAL format. Perhaps we
> should introduce a WAL version number, similar to catalog version?

Would that also cover possible differences in page size, 32bit OS vs.
64bit OS, different timestamp flavour, etc. issues ? AFAIR, all these
things can have an influence on how the data is written and possibly
make the WAL incompatible with other postgres instances, even if the
exact same version...

Cheers,
Csaba.



--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[Heikki Linnakangas-2]

Csaba Nagy wrote:

> On Wed, 2008-06-04 at 11:13 +0300, Heikki Linnakangas wrote:
>> Hmm, WAL version compatibility is an interesting question. Most minor
>> releases hasn't changed the WAL format, and it would be nice to allow
>> running different minor versions in the master and slave in those cases.
>> But it's certainly not unheard of to change the WAL format. Perhaps we
>> should introduce a WAL version number, similar to catalog version?
>
> Would that also cover possible differences in page size, 32bit OS vs.
> 64bit OS, different timestamp flavour, etc. issues ? AFAIR, all these
> things can have an influence on how the data is written and possibly
> make the WAL incompatible with other postgres instances, even if the
> exact same version...

These are already covered by the information in pg_control.

--
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers