|
View:
New views
20 Messages
—
Rating Filter:
Alert me
|
| < Prev | 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 | Next > |
|
|
Re: Core team statement on replication in PostgreSQLOn Friday 30 May 2008 01:10:20 Tom Lane wrote:
> Greg Smith <gsmith@...> writes: > > I fully accept that it may be the case that it doesn't make technical > > sense to tackle them in any order besides sync->read-only slaves because > > of dependencies in the implementation between the two. > > Well, it's certainly not been my intention to suggest that no one should > start work on read-only-slaves before we finish the other part. The > point is that I expect the log shipping issues will be done first > because they're easier, and it would be pointless to not release that > feature if we had it. > > But since you mention it: one of the plausible answers for fixing the > vacuum problem for read-only slaves is to have the slaves push an xmin > back upstream to the master to prevent premature vacuuming. The current > design of pg_standby is utterly incapable of handling that requirement. > So there might be an implementation dependency there, depending on how > we want to solve that problem. > Sure, but whose to say that after synchronous wal shipping is "finished" it wont need a serious re-write due to new needs from the hot standby feature. I think going either way carries some risk. -- Robert Treat Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Thursday 29 May 2008 20:31:31 Greg Smith wrote:
> On Thu, 29 May 2008, Tom Lane wrote: > > There's no point in having read-only slave queries if you don't have a > > trustworthy method of getting the data to them. > > This is a key statement that highlights the difference in how you're > thinking about this compared to some other people here. As far as some > are concerned, the already working log shipping *is* a trustworthy method > of getting data to the read-only slaves. There are plenty of applications > (web oriented ones in particular) where if you could direct read-only > queries against a slave, the resulting combination would be a giant > improvement over the status quo even if that slave was as much as > archive_timeout behind the master. That quantity of lag is perfectly fine > for a lot of the same apps that have read scalability issues. > > If you're someone who falls into that camp, the idea of putting the sync > replication job before the read-only slave one seems really backwards. > Just looking at it from an overall market perspective, synchronous log shipping pretty much only addresses failover needs, where as read-only slaves address both failover and scaling issues. (Note I say address, not solve). -- Robert Treat Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Thursday 29 May 2008 22:59:21 Merlin Moncure wrote:
> On Thu, May 29, 2008 at 9:26 PM, Josh Berkus <josh@...> wrote: > >> I fully accept that it may be the case that it doesn't make technical > >> sense to tackle them in any order besides sync->read-only slaves because > >> of dependencies in the implementation between the two. If that's the > >> case, it would be nice to explicitly spell out what that was to deflect > >> criticism of the planned prioritization. > > > > There's a very simple reason to prioritize the synchronous log shipping > > first; NTT may open source their solution and we'll get it a lot sooner > > than the other components. > > That's a good argument. I just read the NTT document and the stuff > looks fantastic. You've convinced me... It would be a better argument if the NTT guys hadn't said that they estimated 6 months time before the code would be released, which puts us beyond 8.4. Now it is possible that the time frame could be sooner, but unless someone already has the patch, this reminds me a little too much of the arguments for including windows support in a single release because we already had a work port/patch set to go from. -- Robert Treat Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Fri, 2008-05-30 at 11:12 -0700, Robert Hodges wrote: > This is clearly an important use case but it also seems clear that > the WAL approach is not a general-purpose approach to replication. I think we cannot make such a statement yet, if ever. I would note that log-based replication is now the mainstay of commercial database replication techniques for loosely-coupled groups of servers. It would seem strange to assume that it should not be good for us too, simply because we know it to be difficult. IMHO the project has a pretty good track record of delivering functionality that looked hard at first glance. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Thu, May 29, 2008 at 7:42 PM, Tom Lane <tgl@...> wrote:
The big problem Another issue with read-only slaves just popped up in my head. How do we block the readers on the slave while it is replaying an ALTER TABLE or similar command that requires Exclusive lock and potentially alters the table's structure. Or does the WAL replay already takes an x-lock on such a table? Best regards, -- gurjeet[.singh]@EnterpriseDB.com singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com EnterpriseDB http://www.enterprisedb.com Mail sent from my BlackLaptop device |
|
|
Re: Core team statement on replication in PostgreSQLOn Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote:
> Then you ship 16 MB binary stuff every 30 second or every minute but > you only have some kbyte real data in the logfile. Not if you use pg_clearxlogtail ( http://www.2ndquadrant.com/replication.htm ), which got lost in the giant March commitfest queue but should probably wander into contrib as part of 8.4. -- * Greg Smith gsmith@... http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLAndreas 'ads' Scherbaum wrote: > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: > > >> Well, yes, but you do know about archive_timeout, right? No need to wait >> 2 hours. >> > > Then you ship 16 MB binary stuff every 30 second or every minute but > you only have some kbyte real data in the logfile. This must be taken > into account, especially if you ship the logfile over the internet > (means: no high-speed connection, maybe even pay-per-traffic) to the > slave. > > > > Sure there's a price to pay. But that doesn't mean the facility doesn't exist. And I rather suspect that most of Josh's customers aren't too concerned about traffic charges or affected by such bandwidth restrictions. Certainly, none of my clients are, and they aren't in the giant class. Shipping a 16Mb file, particularly if compressed, every minute or so, is not such a huge problem for a great many commercial users, and even many domestic users. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Sat, May 31, 2008 at 1:52 AM, Greg Smith <gsmith@...> wrote:
This means we need to modify pg_standby to not check for filesize when reading XLogs. Best regards, -- gurjeet[.singh]@EnterpriseDB.com singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com EnterpriseDB http://www.enterprisedb.com Mail sent from my BlackLaptop device |
|
|
Re: Core team statement on replication in PostgreSQLOn Sat, 31 May 2008, Gurjeet Singh wrote:
>> Not if you use pg_clearxlogtail > > This means we need to modify pg_standby to not check for filesize when > reading XLogs. No, the idea is that you run the segments through pg_clearxlogtail | gzip, which then compresses lightly used segments massively because all the unused bytes are 0. File comes out the same size at the other side, but you didn't ship a full 16MB if there was only a few KB used. -- * Greg Smith gsmith@... http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Fri, 30 May 2008 16:22:41 -0400 (EDT) Greg Smith wrote:
> On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote: > > > Then you ship 16 MB binary stuff every 30 second or every minute but > > you only have some kbyte real data in the logfile. > > Not if you use pg_clearxlogtail ( > http://www.2ndquadrant.com/replication.htm ), which got lost in the giant > March commitfest queue but should probably wander into contrib as part of > 8.4. Yes, this topic was discussed several times in the past but to solve this it needs a patch/solution which is integrated into PG itself, not contrib. Kind regards -- Andreas 'ads' Scherbaum German PostgreSQL User Group -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote:
> Andreas 'ads' Scherbaum wrote: > > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: > > > >> Well, yes, but you do know about archive_timeout, right? No need to wait > >> 2 hours. > > > > Then you ship 16 MB binary stuff every 30 second or every minute but > > you only have some kbyte real data in the logfile. This must be taken > > into account, especially if you ship the logfile over the internet > > (means: no high-speed connection, maybe even pay-per-traffic) to the > > slave. > > Sure there's a price to pay. But that doesn't mean the facility doesn't > exist. And I rather suspect that most of Josh's customers aren't too > concerned about traffic charges or affected by such bandwidth > restrictions. Certainly, none of my clients are, and they aren't in the > giant class. Shipping a 16Mb file, particularly if compressed, every > minute or so, is not such a huge problem for a great many commercial > users, and even many domestic users. The real problem is not the 16 MB, the problem is: you can't compress this file. If the logfile is rotated it still contains all the old binary data which is not a good starter for compression. So you may have some kB changes in the wal logfile every minute but you still copy 16 MB data. Sure, it's not so much - but if you rotate a logfile every minute this still transfers 16*60*24 = ~23 GB a day. Kind regards -- Andreas 'ads' Scherbaum German PostgreSQL User Group -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLAndrew,
> Sure there's a price to pay. But that doesn't mean the facility doesn't > exist. And I rather suspect that most of Josh's customers aren't too > concerned about traffic charges or affected by such bandwidth > restrictions. Certainly, none of my clients are, and they aren't in the > giant class. Shipping a 16Mb file, particularly if compressed, every > minute or so, is not such a huge problem for a great many commercial > users, and even many domestic users. The issue is that when you're talking about telecommunications companies (and similar) once a minute isn't adequate. Those folks want at least every second, or even better synchronous. Anyway, this is a pretty pointless discussion given that we want both capabilities, and stuff will get implemented in the order it makes technical sense. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Sat, 2008-05-31 at 02:48 +0530, Gurjeet Singh wrote: > > Not if you use pg_clearxlogtail > ( http://www.2ndquadrant.com/replication.htm ), which got lost > in the giant March commitfest queue but should probably wander > into contrib as part of 8.4. > > This means we need to modify pg_standby to not check for filesize when > reading XLogs. > > Best regards, > It does. Joshua D. Drake > -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQL> Andreas 'ads' Scherbaum wrote:
> > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: > > > > > >> Well, yes, but you do know about archive_timeout, right? No need to wait > >> 2 hours. > >> > > > > Then you ship 16 MB binary stuff every 30 second or every minute but > > you only have some kbyte real data in the logfile. This must be taken > > into account, especially if you ship the logfile over the internet > > (means: no high-speed connection, maybe even pay-per-traffic) to the > > slave. > > > > > > > > > > Sure there's a price to pay. But that doesn't mean the facility doesn't > exist. And I rather suspect that most of Josh's customers aren't too > concerned about traffic charges or affected by such bandwidth > restrictions. Certainly, none of my clients are, and they aren't in the > giant class. Shipping a 16Mb file, particularly if compressed, every > minute or so, is not such a huge problem for a great many commercial > users, and even many domestic users. Sumitomo Electric Co., Ltd., a 20 billion dollars selling company in Japan (parent company of Sumitomo Electric Information Systems Co., Ltd., which is one of the Recursive SQL development support company) uses 100 PostgreSQL servers. They are doing backups by using log shipping to another data center and have problems with the amount of the transferring log data. They said this is one of the big problems they have with PostgreSQL and hope it will be solved in the near future. -- Tatsuo Ishii SRA OSS, Inc. Japan -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLTatsuo Ishii wrote: >> Andreas 'ads' Scherbaum wrote: >> >>> On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: >>> >>> >>> >>>> Well, yes, but you do know about archive_timeout, right? No need to wait >>>> 2 hours. >>>> >>>> >>> Then you ship 16 MB binary stuff every 30 second or every minute but >>> you only have some kbyte real data in the logfile. This must be taken >>> into account, especially if you ship the logfile over the internet >>> (means: no high-speed connection, maybe even pay-per-traffic) to the >>> slave. >>> >>> >>> >>> >>> >> Sure there's a price to pay. But that doesn't mean the facility doesn't >> exist. And I rather suspect that most of Josh's customers aren't too >> concerned about traffic charges or affected by such bandwidth >> restrictions. Certainly, none of my clients are, and they aren't in the >> giant class. Shipping a 16Mb file, particularly if compressed, every >> minute or so, is not such a huge problem for a great many commercial >> users, and even many domestic users. >> > > Sumitomo Electric Co., Ltd., a 20 billion dollars selling company in > Japan (parent company of Sumitomo Electric Information Systems Co., > Ltd., which is one of the Recursive SQL development support company) > uses 100 PostgreSQL servers. They are doing backups by using log > shipping to another data center and have problems with the amount of > the transferring log data. They said this is one of the big problems > they have with PostgreSQL and hope it will be solved in the near > future. > > Excellent data point. Now, what I'd like to know is whether they are getting into trouble simply because of the volume of log data generated or because they have a short archive_timeout set. If it's the former (which seems more likely) then none of the ideas I have seen so far in this discussion seemed likely to help, and that would indeed be a major issue we should look at. Another question is this: are they being overwhelmed by the amount of network traffic generated, or by difficulty in postgres producers and/or consumers to keep up? If it's network traffic, then perhaps compression would help us. Maybe we need to set some goals for the level of log volumes we expect to be able to create/send/comsume. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Sat, May 31, 2008 at 3:41 AM, Greg Smith <gsmith@...> wrote:
Got it. I remember reading about pg_clearxlogtail in these mailing lists; but somehow forgot how it actually worked! -- gurjeet[.singh]@EnterpriseDB.com singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com EnterpriseDB http://www.enterprisedb.com Mail sent from my BlackLaptop device |
|
|
Re: Core team statement on replication in PostgreSQLOn Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum
<adsmail@...> wrote: > On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote: >> Andreas 'ads' Scherbaum wrote: >> > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: >> > >> >> Well, yes, but you do know about archive_timeout, right? No need to wait >> >> 2 hours. >> > >> > Then you ship 16 MB binary stuff every 30 second or every minute but >> > you only have some kbyte real data in the logfile. This must be taken >> > into account, especially if you ship the logfile over the internet >> > (means: no high-speed connection, maybe even pay-per-traffic) to the >> > slave. >> >> Sure there's a price to pay. But that doesn't mean the facility doesn't >> exist. And I rather suspect that most of Josh's customers aren't too >> concerned about traffic charges or affected by such bandwidth >> restrictions. Certainly, none of my clients are, and they aren't in the >> giant class. Shipping a 16Mb file, particularly if compressed, every >> minute or so, is not such a huge problem for a great many commercial >> users, and even many domestic users. > > The real problem is not the 16 MB, the problem is: you can't compress > this file. If the logfile is rotated it still contains all the > old binary data which is not a good starter for compression. Using bzip2 in my archive_command script, my WAL files are normally compressed to between 2MB and 5MB, depending on the write load (larger, and more of them, in the middle of the day). bzip2 compression is more expensive and rotated WAL files are not particularly compressable to be sure, but due to (and given) the nature of the data bzip2 works pretty well, and much better than gzip. > > So you may have some kB changes in the wal logfile every minute but you > still copy 16 MB data. Sure, it's not so much - but if you rotate a > logfile every minute this still transfers 16*60*24 = ~23 GB a day. > I archived 1965 logs yesterday on one instance of my app totalling 8.5GB ... not to bad, really. -- Mike Rylander | VP, Research and Design | Equinox Software, Inc. / The Evergreen Experts | phone: 1-877-OPEN-ILS (673-6457) | email: miker@... | web: http://www.esilibrary.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Sat, May 31, 2008 at 2:18 AM, Mike Rylander <mrylander@...> wrote:
> On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum > <adsmail@...> wrote: >> On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote: >>> Andreas 'ads' Scherbaum wrote: >>> > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: >>> > >>> >> Well, yes, but you do know about archive_timeout, right? No need to wait >>> >> 2 hours. >>> > >>> > Then you ship 16 MB binary stuff every 30 second or every minute but >>> > you only have some kbyte real data in the logfile. This must be taken >>> > into account, especially if you ship the logfile over the internet >>> > (means: no high-speed connection, maybe even pay-per-traffic) to the >>> > slave. >>> >>> Sure there's a price to pay. But that doesn't mean the facility doesn't >>> exist. And I rather suspect that most of Josh's customers aren't too >>> concerned about traffic charges or affected by such bandwidth >>> restrictions. Certainly, none of my clients are, and they aren't in the >>> giant class. Shipping a 16Mb file, particularly if compressed, every >>> minute or so, is not such a huge problem for a great many commercial >>> users, and even many domestic users. >> >> The real problem is not the 16 MB, the problem is: you can't compress >> this file. If the logfile is rotated it still contains all the >> old binary data which is not a good starter for compression. > > Using bzip2 in my archive_command script, my WAL files are normally > compressed to between 2MB and 5MB, depending on the write load > (larger, and more of them, in the middle of the day). bzip2 > compression is more expensive and rotated WAL files are not > particularly compressable to be sure, but due to (and given) the > nature of the data bzip2 works pretty well, and much better than gzip. Compression especially is going to negate one of the big advantages of wal shipping, namely that it is cheap investment in terms of load to the main. A gigabit link can ship a lot of log files, you can always bond and 10gige is coming. IMO the key trick is to make sure you don't send the log file more than once from the same source...i.e cascading relay. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLMerlin Moncure wrote:
> On Sat, May 31, 2008 at 2:18 AM, Mike Rylander <mrylander@...> wrote: >> On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum > Compression especially is going to negate one of the big advantages of > wal shipping, namely that it is cheap investment in terms of load to > the main. A gigabit link can ship a lot of log files, you can always Who has a gigabit link between Dallas and Atlanta? That is the actual problem here. Switch to Switch compression is a waste of time (if you aren't running GiGE, what are you doing???). Sincerely, Joshua D. Drake -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLMy point here is that with reasonably small extensions to the core you can build products that are a lot better than SLONY. Triggers do not cover DDL, among other issues, and it’s debatable whether they are the best way to implement quorum policies like Google’s semi-synchronous replication. As I mentioned separately this topic deserves another thread which I promise to start. It is of course possible to meet some of these needs with an appropriate client interface to WAL shipping. There’s no a-priori reason why built-in PostgreSQL slaves need to be the only client. I would put a vote in for covering this possibility in the initial replication design. We are using a very similar approach in our own master/slave replication product. Thanks, Robert P.S., No offense intended to Jan Wieck et al. There are some pretty cool things in SLONY. On 5/29/08 8:16 PM, "Merlin Moncure" <mmoncure@...> wrote: On Thu, May 29, 2008 at 3:05 PM, Robert Hodges -- Robert Hodges, CTO, Continuent, Inc. Email: robert.hodges@... Mobile: +1-510-501-3728 Skype: hodgesrm |
| < Prev | 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 | Next > |
| Free embeddable forum powered by Nabble | Forum Help |