Core team statement on replication in PostgreSQL

View: New views
20 Messages — Rating Filter:   Alert me  
< Prev | 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 | Next >

Re: Core team statement on replication in PostgreSQL

by Robert Treat :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Friday 30 May 2008 01:10:20 Tom Lane wrote:

> Greg Smith <gsmith@...> writes:
> > I fully accept that it may be the case that it doesn't make technical
> > sense to tackle them in any order besides sync->read-only slaves because
> > of dependencies in the implementation between the two.
>
> Well, it's certainly not been my intention to suggest that no one should
> start work on read-only-slaves before we finish the other part.  The
> point is that I expect the log shipping issues will be done first
> because they're easier, and it would be pointless to not release that
> feature if we had it.
>
> But since you mention it: one of the plausible answers for fixing the
> vacuum problem for read-only slaves is to have the slaves push an xmin
> back upstream to the master to prevent premature vacuuming.  The current
> design of pg_standby is utterly incapable of handling that requirement.
> So there might be an implementation dependency there, depending on how
> we want to solve that problem.
>

Sure, but whose to say that after synchronous wal shipping is "finished" it
wont need a serious re-write due to new needs from the hot standby feature. I
think going either way carries some risk.

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Robert Treat :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Thursday 29 May 2008 20:31:31 Greg Smith wrote:

> On Thu, 29 May 2008, Tom Lane wrote:
> > There's no point in having read-only slave queries if you don't have a
> > trustworthy method of getting the data to them.
>
> This is a key statement that highlights the difference in how you're
> thinking about this compared to some other people here.  As far as some
> are concerned, the already working log shipping *is* a trustworthy method
> of getting data to the read-only slaves.  There are plenty of applications
> (web oriented ones in particular) where if you could direct read-only
> queries against a slave, the resulting combination would be a giant
> improvement over the status quo even if that slave was as much as
> archive_timeout behind the master.  That quantity of lag is perfectly fine
> for a lot of the same apps that have read scalability issues.
>
> If you're someone who falls into that camp, the idea of putting the sync
> replication job before the read-only slave one seems really backwards.
>

Just looking at it from an overall market perspective, synchronous log
shipping pretty much only addresses failover needs, where as read-only slaves
address both failover and scaling issues. (Note I say address, not solve).

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Robert Treat :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Thursday 29 May 2008 22:59:21 Merlin Moncure wrote:

> On Thu, May 29, 2008 at 9:26 PM, Josh Berkus <josh@...> wrote:
> >> I fully accept that it may be the case that it doesn't make technical
> >> sense to tackle them in any order besides sync->read-only slaves because
> >> of dependencies in the implementation between the two.  If that's the
> >> case, it would be nice to explicitly spell out what that was to deflect
> >> criticism of the planned prioritization.
> >
> > There's a very simple reason to prioritize the synchronous log shipping
> > first; NTT may open source their solution and we'll get it a lot sooner
> > than the other components.
>
> That's a good argument.  I just read the NTT document and the stuff
> looks fantastic.  You've convinced me...

It would be a better argument if the NTT guys hadn't said that they estimated
6 months time before the code would be released, which puts us beyond 8.4.
Now it is possible that the time frame could be sooner, but unless someone
already has the patch, this reminds me a little too much of the arguments for
including windows support in a single release because we already had a work
port/patch set to go from.

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Simon Riggs :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message


On Fri, 2008-05-30 at 11:12 -0700, Robert Hodges wrote:
> This is clearly an important use case but it also seems clear that
> the WAL approach is not a general-purpose approach to replication.

I think we cannot make such a statement yet, if ever.

I would note that log-based replication is now the mainstay of
commercial database replication techniques for loosely-coupled groups of
servers. It would seem strange to assume that it should not be good for
us too, simply because we know it to be difficult.

IMHO the project has a pretty good track record of delivering
functionality that looked hard at first glance.

--
 Simon Riggs           www.2ndQuadrant.com
 PostgreSQL Training, Services and Support


--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by singh.gurjeet :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Thu, May 29, 2008 at 7:42 PM, Tom Lane <tgl@...> wrote:
The big problem
is that long-running slave-side queries might still need tuples that are
vacuumable on the master, and so replication of vacuuming actions would
cause the slave's queries to deliver wrong answers.

Another issue with read-only slaves just popped up in my head.

How do we block the readers on the slave while it is replaying an ALTER TABLE or similar command that requires Exclusive lock and potentially alters the table's structure. Or does the WAL replay already takes an x-lock on such a table?

Best regards,
--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device

Re: Core team statement on replication in PostgreSQL

by Greg Smith-12 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote:

> Then you ship 16 MB binary stuff every 30 second or every minute but
> you only have some kbyte real data in the logfile.

Not if you use pg_clearxlogtail (
http://www.2ndquadrant.com/replication.htm ), which got lost in the giant
March commitfest queue but should probably wander into contrib as part of
8.4.

--
* Greg Smith gsmith@... http://www.gregsmith.com Baltimore, MD

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Andrew Dunstan :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message



Andreas 'ads' Scherbaum wrote:

> On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote:
>
>  
>> Well, yes, but you do know about archive_timeout, right? No need to wait
>> 2 hours.
>>    
>
> Then you ship 16 MB binary stuff every 30 second or every minute but
> you only have some kbyte real data in the logfile. This must be taken
> into account, especially if you ship the logfile over the internet
> (means: no high-speed connection, maybe even pay-per-traffic) to the
> slave.
>
>
>
>  

Sure there's a price to pay. But that doesn't mean the facility doesn't
exist. And I rather suspect that most of Josh's customers aren't too
concerned about traffic charges or affected by such bandwidth
restrictions. Certainly, none of my clients are, and they aren't in the
giant class. Shipping a 16Mb file, particularly if compressed, every
minute or so, is not such a huge problem for a great many commercial
users, and even many domestic users.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by singh.gurjeet :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, May 31, 2008 at 1:52 AM, Greg Smith <gsmith@...> wrote:
On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote:

Then you ship 16 MB binary stuff every 30 second or every minute but
you only have some kbyte real data in the logfile.

Not if you use pg_clearxlogtail ( http://www.2ndquadrant.com/replication.htm ), which got lost in the giant March commitfest queue but should probably wander into contrib as part of 8.4.

This means we need to modify pg_standby to not check for filesize when reading XLogs.

Best regards,

--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device

Re: Core team statement on replication in PostgreSQL

by Greg Smith-12 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, 31 May 2008, Gurjeet Singh wrote:

>> Not if you use pg_clearxlogtail
>
> This means we need to modify pg_standby to not check for filesize when
> reading XLogs.

No, the idea is that you run the segments through pg_clearxlogtail | gzip,
which then compresses lightly used segments massively because all the
unused bytes are 0.  File comes out the same size at the other side, but
you didn't ship a full 16MB if there was only a few KB used.

--
* Greg Smith gsmith@... http://www.gregsmith.com Baltimore, MD

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Andreas 'ads' Scherbaum :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, 30 May 2008 16:22:41 -0400 (EDT) Greg Smith wrote:

> On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote:
>
> > Then you ship 16 MB binary stuff every 30 second or every minute but
> > you only have some kbyte real data in the logfile.
>
> Not if you use pg_clearxlogtail (
> http://www.2ndquadrant.com/replication.htm ), which got lost in the giant
> March commitfest queue but should probably wander into contrib as part of
> 8.4.

Yes, this topic was discussed several times in the past but to
solve this it needs a patch/solution which is integrated into PG
itself, not contrib.


Kind regards

--
                                Andreas 'ads' Scherbaum
German PostgreSQL User Group

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Andreas 'ads' Scherbaum :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote:

> Andreas 'ads' Scherbaum wrote:
> > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote:
> >
> >> Well, yes, but you do know about archive_timeout, right? No need to wait
> >> 2 hours.
> >
> > Then you ship 16 MB binary stuff every 30 second or every minute but
> > you only have some kbyte real data in the logfile. This must be taken
> > into account, especially if you ship the logfile over the internet
> > (means: no high-speed connection, maybe even pay-per-traffic) to the
> > slave.
>
> Sure there's a price to pay. But that doesn't mean the facility doesn't
> exist. And I rather suspect that most of Josh's customers aren't too
> concerned about traffic charges or affected by such bandwidth
> restrictions. Certainly, none of my clients are, and they aren't in the
> giant class. Shipping a 16Mb file, particularly if compressed, every
> minute or so, is not such a huge problem for a great many commercial
> users, and even many domestic users.

The real problem is not the 16 MB, the problem is: you can't compress
this file. If the logfile is rotated it still contains all the
old binary data which is not a good starter for compression.

So you may have some kB changes in the wal logfile every minute but you
still copy 16 MB data. Sure, it's not so much - but if you rotate a
logfile every minute this still transfers 16*60*24 = ~23 GB a day.


Kind regards

--
                                Andreas 'ads' Scherbaum
German PostgreSQL User Group

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Josh Berkus :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Andrew,

> Sure there's a price to pay. But that doesn't mean the facility doesn't
> exist. And I rather suspect that most of Josh's customers aren't too
> concerned about traffic charges or affected by such bandwidth
> restrictions. Certainly, none of my clients are, and they aren't in the
> giant class. Shipping a 16Mb file, particularly if compressed, every
> minute or so, is not such a huge problem for a great many commercial
> users, and even many domestic users.

The issue is that when you're talking about telecommunications companies
(and similar) once a minute isn't adequate.  Those folks want at least
every second, or even better synchronous.

Anyway, this is a pretty pointless discussion given that we want both
capabilities, and stuff will get implemented in the order it makes
technical sense.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Joshua D. Drake :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message



On Sat, 2008-05-31 at 02:48 +0530, Gurjeet Singh wrote:

>        
>         Not if you use pg_clearxlogtail
>         ( http://www.2ndquadrant.com/replication.htm ), which got lost
>         in the giant March commitfest queue but should probably wander
>         into contrib as part of 8.4.
>
> This means we need to modify pg_standby to not check for filesize when
> reading XLogs.
>
> Best regards,
>

It does.

Joshua D. Drake

>


--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Tatsuo Ishii-4 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

> Andreas 'ads' Scherbaum wrote:
> > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote:
> >
> >  
> >> Well, yes, but you do know about archive_timeout, right? No need to wait
> >> 2 hours.
> >>    
> >
> > Then you ship 16 MB binary stuff every 30 second or every minute but
> > you only have some kbyte real data in the logfile. This must be taken
> > into account, especially if you ship the logfile over the internet
> > (means: no high-speed connection, maybe even pay-per-traffic) to the
> > slave.
> >
> >
> >
> >  
>
> Sure there's a price to pay. But that doesn't mean the facility doesn't
> exist. And I rather suspect that most of Josh's customers aren't too
> concerned about traffic charges or affected by such bandwidth
> restrictions. Certainly, none of my clients are, and they aren't in the
> giant class. Shipping a 16Mb file, particularly if compressed, every
> minute or so, is not such a huge problem for a great many commercial
> users, and even many domestic users.

Sumitomo Electric Co., Ltd., a 20 billion dollars selling company in
Japan (parent company of Sumitomo Electric Information Systems Co.,
Ltd., which is one of the Recursive SQL development support company)
uses 100 PostgreSQL servers. They are doing backups by using log
shipping to another data center and have problems with the amount of
the transferring log data. They said this is one of the big problems
they have with PostgreSQL and hope it will be solved in the near
future.
--
Tatsuo Ishii
SRA OSS, Inc. Japan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Andrew Dunstan :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message



Tatsuo Ishii wrote:

>> Andreas 'ads' Scherbaum wrote:
>>    
>>> On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote:
>>>
>>>  
>>>      
>>>> Well, yes, but you do know about archive_timeout, right? No need to wait
>>>> 2 hours.
>>>>    
>>>>        
>>> Then you ship 16 MB binary stuff every 30 second or every minute but
>>> you only have some kbyte real data in the logfile. This must be taken
>>> into account, especially if you ship the logfile over the internet
>>> (means: no high-speed connection, maybe even pay-per-traffic) to the
>>> slave.
>>>
>>>
>>>
>>>  
>>>      
>> Sure there's a price to pay. But that doesn't mean the facility doesn't
>> exist. And I rather suspect that most of Josh's customers aren't too
>> concerned about traffic charges or affected by such bandwidth
>> restrictions. Certainly, none of my clients are, and they aren't in the
>> giant class. Shipping a 16Mb file, particularly if compressed, every
>> minute or so, is not such a huge problem for a great many commercial
>> users, and even many domestic users.
>>    
>
> Sumitomo Electric Co., Ltd., a 20 billion dollars selling company in
> Japan (parent company of Sumitomo Electric Information Systems Co.,
> Ltd., which is one of the Recursive SQL development support company)
> uses 100 PostgreSQL servers. They are doing backups by using log
> shipping to another data center and have problems with the amount of
> the transferring log data. They said this is one of the big problems
> they have with PostgreSQL and hope it will be solved in the near
> future.
>
>  

Excellent data point. Now, what I'd like to know is whether they are
getting into trouble simply because of the volume of log data generated
or because they have a short archive_timeout set. If it's the former
(which seems more likely) then none of the ideas I have seen so far in
this discussion seemed likely to help, and that would indeed be a major
issue we should look at. Another question is this: are they being
overwhelmed by the amount of network traffic generated, or by difficulty
in postgres producers and/or consumers to keep up? If it's network
traffic, then perhaps compression would help us.

Maybe we need to set some goals for the level of log volumes we expect
to be able to create/send/comsume.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by singh.gurjeet :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, May 31, 2008 at 3:41 AM, Greg Smith <gsmith@...> wrote:
On Sat, 31 May 2008, Gurjeet Singh wrote:

Not if you use pg_clearxlogtail

This means we need to modify pg_standby to not check for filesize when
reading XLogs.

No, the idea is that you run the segments through pg_clearxlogtail | gzip, which then compresses lightly used segments massively because all the unused bytes are 0.  File comes out the same size at the other side, but you didn't ship a full 16MB if there was only a few KB used.

Got it. I remember reading about pg_clearxlogtail in these mailing lists; but somehow forgot how it actually worked!

--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device

Re: Core team statement on replication in PostgreSQL

by Mike Rylander :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum
<adsmail@...> wrote:

> On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote:
>> Andreas 'ads' Scherbaum wrote:
>> > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote:
>> >
>> >> Well, yes, but you do know about archive_timeout, right? No need to wait
>> >> 2 hours.
>> >
>> > Then you ship 16 MB binary stuff every 30 second or every minute but
>> > you only have some kbyte real data in the logfile. This must be taken
>> > into account, especially if you ship the logfile over the internet
>> > (means: no high-speed connection, maybe even pay-per-traffic) to the
>> > slave.
>>
>> Sure there's a price to pay. But that doesn't mean the facility doesn't
>> exist. And I rather suspect that most of Josh's customers aren't too
>> concerned about traffic charges or affected by such bandwidth
>> restrictions. Certainly, none of my clients are, and they aren't in the
>> giant class. Shipping a 16Mb file, particularly if compressed, every
>> minute or so, is not such a huge problem for a great many commercial
>> users, and even many domestic users.
>
> The real problem is not the 16 MB, the problem is: you can't compress
> this file. If the logfile is rotated it still contains all the
> old binary data which is not a good starter for compression.

Using bzip2 in my archive_command script, my WAL files are normally
compressed to between 2MB and 5MB, depending on the write load
(larger, and more of them, in the middle of the day).  bzip2
compression is more expensive and rotated WAL files are not
particularly compressable to be sure, but due to (and given) the
nature of the data bzip2 works pretty well, and much better than gzip.

>
> So you may have some kB changes in the wal logfile every minute but you
> still copy 16 MB data. Sure, it's not so much - but if you rotate a
> logfile every minute this still transfers 16*60*24 = ~23 GB a day.
>

I archived 1965 logs yesterday on one instance of my app totalling
8.5GB ... not to bad, really.

--
Mike Rylander
 | VP, Research and Design
 | Equinox Software, Inc. / The Evergreen Experts
 | phone: 1-877-OPEN-ILS (673-6457)
 | email: miker@...
 | web: http://www.esilibrary.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Merlin Moncure-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, May 31, 2008 at 2:18 AM, Mike Rylander <mrylander@...> wrote:

> On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum
> <adsmail@...> wrote:
>> On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote:
>>> Andreas 'ads' Scherbaum wrote:
>>> > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote:
>>> >
>>> >> Well, yes, but you do know about archive_timeout, right? No need to wait
>>> >> 2 hours.
>>> >
>>> > Then you ship 16 MB binary stuff every 30 second or every minute but
>>> > you only have some kbyte real data in the logfile. This must be taken
>>> > into account, especially if you ship the logfile over the internet
>>> > (means: no high-speed connection, maybe even pay-per-traffic) to the
>>> > slave.
>>>
>>> Sure there's a price to pay. But that doesn't mean the facility doesn't
>>> exist. And I rather suspect that most of Josh's customers aren't too
>>> concerned about traffic charges or affected by such bandwidth
>>> restrictions. Certainly, none of my clients are, and they aren't in the
>>> giant class. Shipping a 16Mb file, particularly if compressed, every
>>> minute or so, is not such a huge problem for a great many commercial
>>> users, and even many domestic users.
>>
>> The real problem is not the 16 MB, the problem is: you can't compress
>> this file. If the logfile is rotated it still contains all the
>> old binary data which is not a good starter for compression.
>
> Using bzip2 in my archive_command script, my WAL files are normally
> compressed to between 2MB and 5MB, depending on the write load
> (larger, and more of them, in the middle of the day).  bzip2
> compression is more expensive and rotated WAL files are not
> particularly compressable to be sure, but due to (and given) the
> nature of the data bzip2 works pretty well, and much better than gzip.

Compression especially is going to negate one of the big advantages of
wal shipping, namely that it is cheap investment in terms of load to
the main.  A gigabit link can ship a lot of log files, you can always
bond and 10gige is coming.  IMO the key trick is to make sure you
don't send the log file more than once from the same source...i.e
cascading relay.

merlin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Joshua D. Drake :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Merlin Moncure wrote:
> On Sat, May 31, 2008 at 2:18 AM, Mike Rylander <mrylander@...> wrote:
>> On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum

> Compression especially is going to negate one of the big advantages of
> wal shipping, namely that it is cheap investment in terms of load to
> the main.  A gigabit link can ship a lot of log files, you can always

Who has a gigabit link between Dallas and Atlanta? That is the actual
problem here. Switch to Switch compression is a waste of time (if you
aren't running GiGE, what are you doing???).

Sincerely,

Joshua D. Drake

--
Sent via pgsql-hackers mailing list (pgsql-hackers@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Core team statement on replication in PostgreSQL

by Robert Hodges :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Re: [HACKERS] Core team statement on replication in PostgreSQL Hi Merlin,

My point here is that with reasonably small extensions to the core you can build products that are a lot better than SLONY.   Triggers do not cover DDL, among other issues, and it’s debatable whether they are the best way to implement quorum policies like Google’s semi-synchronous replication.  As I mentioned separately this topic deserves another thread which I promise to start.  

It is of course possible to meet some of these needs with an appropriate client interface to WAL shipping.  There’s no a-priori reason why built-in PostgreSQL slaves need to be the only client.  I would put a vote in for covering this possibility in the initial replication design.  We are using a very similar approach in our own master/slave replication product.  

Thanks, Robert

P.S., No offense intended to Jan Wieck et al.  There are some pretty cool things in SLONY.  

On 5/29/08 8:16 PM, "Merlin Moncure" <mmoncure@...> wrote:

On Thu, May 29, 2008 at 3:05 PM, Robert Hodges
<robert.hodges@...> wrote:
> Third, you can't stop with just this feature.  (This is the BUT part of the
> post.)  The use cases not covered by this feature area actually pretty
> large.  Here are a few that concern me:
>
> 1.) Partial replication.
> 2.) WAN replication.
> 3.) Bi-directional replication.  (Yes, this is evil but there are problems
> where it is indispensable.)
> 4.) Upgrade support.  Aside from database upgrade (how would this ever
> really work between versions?), it would not support zero-downtime app
> upgrades, which depend on bi-directional replication tricks.
> 5.) Heterogeneous replication.
> 6.) Finally, performance scaling using scale-out over large numbers of
> replicas.  I think it's possible to get tunnel vision on this—it's not a big
> requirement in the PG community because people don't use PG in the first
> place when they want to do this.  They use MySQL, which has very good
> replication for performance scaling, though it's rather weak for
> availability.

These type of things are what Slony is for.  Slony is trigger based.
This makes it more complex than log shipping style replication, but
provides lots of functionality.

wal shipping based replication is maybe the fastest possible
solution...you are already paying the overhead so it comes virtually
for free from the point of view of the master.

mysql replication is imo nearly worthless from backup standpoint.

merlin



--
Robert Hodges, CTO, Continuent, Inc.
Email:  robert.hodges@...
Mobile:  +1-510-501-3728  Skype:  hodgesrm
< Prev | 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 | Next >