|
View:
New views
20 Messages
—
Rating Filter:
Alert me
|
| < Prev | 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 | Next > |
|
|
Re: Core team statement on replication in PostgreSQLOn Thu, May 29, 2008 at 01:55:42PM -0700, Josh Berkus wrote:
> David, > >>> I think the consensus in the core team was that having synchronous >>> log shipping in 8.4 would already be a worthwhile feature by itself. >> >> If that was in fact the consensus of the core team, > > It is. > >> and what I've been seeing from several core members in this thread >> makes that idea unclear, it's out of step with the stated goal of >> the feature. Having some kind of half-way, >> doesn't-actually-quite-work-out-of-the-box "replication" will make >> things worse and not better. > > So, you've got a better implementation up your sleeve? Nope. > I really don't get where you're coming from on this. Frankly, your > e-mails seem gauged to be merely disruptive without any intent of > constructive input. I'm sorry they've come across that way. That was not my intent. > If you're opposed to working on replication in the core, then just > say so. If you think that there's an easier way to develop M-S > replication in the core than using WAL, then please present your > solution. I think having master-slave replication in the core using WAL is a *great* thing to do, doable, a good path to go on, etc., and I think it's worth holding up 8.4 until we have at least one actual out-of-the-box version of same. People have hinted that we might be able to get both a synchronous one and an asynchronous one based on WAL, which would be even better. :) Cheers, David. -- David Fetter <david@...> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@... Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLDavid,
> I think having master-slave replication in the core using WAL is a > *great* thing to do, doable, a good path to go on, etc., and I think > it's worth holding up 8.4 until we have at least one actual > out-of-the-box version of same. Ah, ok. Well, I can tell you that the core team is also united on the value of time-based as opposed to feature-based release cycles. *if* we can develop this in time for 8.4, everybody would be overjoyed, but I also think we should be realistic. > People have hinted that we might be able to get both a synchronous one > and an asynchronous one based on WAL, which would be even better. :) That's the idea, yes. No reason to dump asynch WAL copying when it's already working. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Thu, May 29, 2008 at 01:39:29PM -0700, David Fetter wrote:
> > I think the consensus in the core team was that having synchronous > > log shipping in 8.4 would already be a worthwhile feature by itself. > > If that was in fact the consensus of the core team, and what I've been > seeing from several core members in this thread makes that idea > unclear, it's out of step with the stated goal of the feature. Having > some kind of half-way, doesn't-actually-quite-work-out-of-the-box > "replication" will make things worse and not better. I don't agree with that. It seems to me that the stated goal is fourfold: 1. Get something that is simple (out of the box). 2. Get something that is generally useful, even if not universally useful. 3. Get something that is implementable in stages, with each stage representing an increase in functionality. 4. Aim for 8.4, but be realistic about chances. It seems to me that sync. replication to a write-only standby actually meets those goals. I can think of more than one case where I have spent lots of time, money, or both attempting to achieve this goal with various add-on hardware and software systems that mostly worked except when they didn't. These additions were complicated to set up, painful to use, and documented by sadists who couldn't write. I agree that there are plenty of people unwilling to use a system that requires additional hardware that mostly can't be used for other things. But those people already have alternatives (maybe even more, and simple ones, soon). The synchronous-needing crowd currently have nothing. The proposed feature would be a huge improvement. A -- Andrew Sullivan ajs@... +1 503 667 4564 x104 http://www.commandprompt.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLPeter Eisentraut wrote:
> Mathias Brossard wrote: >> From what I gather from those slides it seems to me that the NTT solution >> is synchronous not asynchronous. In my opinion it's even better, but I do >> understand that others might prefer asynchronous. I'm going to speculate, >> but I would think it should be possible (without a substancial rewrite) to >> support both modes (or even some intermediate modes, like DRBD on Linux). > > Well, we already have asynchronous. The whole point is adding synchronous. I don't think the current log shipping solution matches, as Tom puts it, the « need for simple, built-in replication » (from a user perspective, I wouldn't call it "simple"). I pointed out that the NTT solution is synchronous because Tom said in the first part of his email that: > In practice, simple asynchronous single-master-multiple-slave > replication covers a respectable fraction of use cases, so we have > concluded that we should allow such a feature to be included in the > core project. ... and yet "the most appropriate base technology for this" is synchronous and maybe I should have also pointed out in my previous mail is that it doesn't support multiple slaves. Also, as other have pointed out there are different interpretations of "synchronous" depending on wether the WAL data has reached the other end of the network connection, a safe disk checkpoint or the slave DB itself. Sincerely, -- Mathias Brossard -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Thursday 29 May 2008 12:13:20 Bruce Momjian wrote:
> David Fetter wrote: > > On Thu, May 29, 2008 at 11:58:31AM -0400, Bruce Momjian wrote: > > > Josh Berkus wrote: > > > > Publishing the XIDs back to the master is one possibility. We > > > > also looked at using "spillover segments" for vacuumed rows, but > > > > that seemed even less viable. > > > > > > > > I'm also thinking, for *async replication*, that we could simply > > > > halt replication on the slave whenever a transaction passes minxid > > > > on the master. However, the main focus will be on synchrounous > > > > hot standby. > > > > > > Another idea I discussed with Tom is having the slave _delay_ > > > applying WAL files until all slave snapshots are ready. > > > > Either one of these would be great, but something that involves > > machines that stay useless most of the time is just not going to work. > > Right, the ultimate target is to have the slave be read-only, but we > need to get the streaming of WAL logs done first. > I would have thought the read only piece would have been more important than the synchronous piece. In my experience readable slaves is the big selling point in both Oracle and MySQL's implementations, and people are not nearly as concerned if there is a small asynchronous window. It would also be easier to implement on some level; we have already solved the asynchronus wal shipping problem, so we would just need to solve the read-only bits. For synchronus hot standby, you have to solve both the synchronus shipping and the read-only bits. Seems like more work with less upside that read-only slaves vs. pitr warm standby we have now. Interesting that core views this differently. -- Robert Treat Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLRobert Treat <xzilla@...> writes:
> I would have thought the read only piece would have been more important than > the synchronous piece. In my experience readable slaves is the big selling > point in both Oracle and MySQL's implementations, and people are not nearly > as concerned if there is a small asynchronous window. Actually, it's the "easy to use" piece that's near the front of my own priority list. In any case, the expectation that we can get the log shipping fixed sooner than we can get read-only slaves has nothing to do with the relative desirability of the pieces; it's a technical judgment based on what we can see of the problems' difficulty. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Thu, 2008-05-29 at 17:42 -0400, Robert Treat wrote: > > I would have thought the read only piece would have been more important than > the synchronous piece. In my experience readable slaves is the big selling > point in both Oracle and MySQL's implementations, and people are not nearly > as concerned if there is a small asynchronous window. The read only piece is the more important piece from a market perspective. Joshua D. Drake -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLtgl@... (Tom Lane) writes:
> As I said originally, we have no expectation that the proposed features > will displace the existing replication projects for "high end" > replication problems ... and I'd characterize all of Robert's concerns > as "high end" problems. We are happy to let those be solved outside > the core project. I expect that one of the effects of having an "improved PITR" as a 'core' replication solution will be that Slony-I will become less popular amongst "naive users," and that is *not* a bad thing. I am quite sure that there are a lot of cases today where users would be better served by what falls out of this. It will mean that those that are left will, despite being fewer, be more cognizant that they are getting what they require. We should see somewhat fewer "expectation gaps," and that should allow more people to be more satisfied with what they are getting. -- let name="cbbrowne" and tld="cbbrowne.com" in name ^ "@" ^ tld;; http://linuxdatabases.info/info/linux.html Lisp stoppped itself FEP Command: -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQL-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1 Hi, I'd first want to applaud core decision: having bare PostgreSQL propose a reliable and simple to set-up synchronous replication solution is an excellent perspective! ... Le 29 mai 08 à 23:42, Robert Treat a écrit : > I would have thought the read only piece would have been more > important than > the synchronous piece. In my experience readable slaves is the big > selling > point in both Oracle and MySQL's implementations, and people are not > nearly > as concerned if there is a small asynchronous window. ... Even more so when you're confronted to this exact problem. A fellow PG user ended up having both the WAL and the data replicated by DRBD (protocol C) and some heartbeat scripts to do the automatic failover. This wasn't easy to setup, and to some extend we're still concerned about the reliability part of it. We know about the "easy to use" part of it: we didn't get it. While at it, would it be possible for the "simple" part of the core team statement to include automatic failover? That would mean for current master when it's going to stop on error (fatal) to tell the slave to warm-up. Of course in case of more severe crash the slave would have to get started by other means, but covering the fatal error path and have the master restart as a slave would only add up to the reliability... wouldn't it? > It would also be easier to implement on some level; we have already > solved the > asynchronus wal shipping problem, so we would just need to solve the > read-only bits. For synchronus hot standby, you have to solve both the > synchronus shipping and the read-only bits. Seems like more work > with less > upside that read-only slaves vs. pitr warm standby we have now. > > Interesting that core views this differently. core seems to think read-only slave is more complex than synchronous slave, in term of slave read only long transaction and master vacuums for example. Regards, - -- dim -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkg/Kk8ACgkQlBXRlnbh1bneKACeMK+fSp8VExctndo46X76NTxV atIAn2UYw1g/4RPddypqirrZcqg5C7gm =JeA6 -----END PGP SIGNATURE----- -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLDimitri Fontaine <dim@...> writes:
> While at it, would it be possible for the "simple" part of the core > team statement to include automatic failover? No, I think it would be a useless expenditure of energy. Failover includes a lot of things that are not within our purview: switching IP addresses to point to the new server, some kind of STONITH solution to keep the original master from coming back to life, etc. Moreover there are already projects/products concerned with those issues. It might be useful to document where to find solutions to that problem, but we can't take it on as part of core Postgres. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLJoshua D. Drake wrote: > On Thu, 2008-05-29 at 17:42 -0400, Robert Treat wrote: > > >> I would have thought the read only piece would have been more important than >> the synchronous piece. In my experience readable slaves is the big selling >> point in both Oracle and MySQL's implementations, and people are not nearly >> as concerned if there is a small asynchronous window. >> > > The read only piece is the more important piece from a market > perspective. > > > You must be gauging a different market from the one I'm in. I have just come back from a meeting with a (quite technically savvy) customer who was quite excited by the news and saw the possibility of read-only slaves as a nice to have extra rather than a must-have-or-it's-not-worth anything feature. I'm really quite astounded and rather saddened by the waves of negativity I have seen today. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Thu, 2008-05-29 at 18:39 -0400, Andrew Dunstan wrote: > > Joshua D. Drake wrote: > > On Thu, 2008-05-29 at 17:42 -0400, Robert Treat wrote: > > > > You must be gauging a different market from the one I'm in. I have just > come back from a meeting with a (quite technically savvy) customer who One customer does not make a hundred. I am not saying that the shipping isn't valid, just that those that I talk to are more interested in the read only slave. Consider that we have any number of ways to solve the problem we are considering implementing now. DRBD being just one of them. Especially considering we aren't talking about failover etc.. > was quite excited by the news and saw the possibility of read-only > slaves as a nice to have extra rather than a must-have-or-it's-not-worth > anything feature. > > I'm really quite astounded and rather saddened by the waves of > negativity I have seen today. I have seen much more positive than negative. I think most are just trying to figure out exactly what we are talking about. I for example am not really against the feature at all. I am suspect of the idea of only shipping 2 components of a 3 component feature though. Sincerely, Joshua D. Drake -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQL"Joshua D. Drake" <jd@...> writes:
> One customer does not make a hundred. I am not saying that the shipping > isn't valid, just that those that I talk to are more interested in the > read only slave. Consider that we have any number of ways to solve the > problem we are considering implementing now. DRBD being just one of > them. In principle, sure, but there's a big gap between theory and practice here. What you are arguing for is the "give the users a toolkit and let them figure it out" approach that David was lambasting upthread. People want the bits to go from point A to point B; they don't want to have to research, design, test, and administer their own solution for moving the bits. Furthermore, as several people have pointed out from their own bitter experiences, getting a reliable and easy-to-use solution is not as easy as it sounds. I think we have nontrivial work in front of us to build a simple, reliable, community-tested log shipping solution; and it's not very sexy work either. But it needs to get done, and it really needs to get done first. There's no point in having read-only slave queries if you don't have a trustworthy method of getting the data to them. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Thu, 2008-05-29 at 19:02 -0400, Tom Lane wrote: > "Joshua D. Drake" <jd@...> writes: > > One customer does not make a hundred. I am not saying that the shipping > > isn't valid, just that those that I talk to are more interested in the > > read only slave. Consider that we have any number of ways to solve the > > problem we are considering implementing now. DRBD being just one of > > them. > > In principle, sure, but there's a big gap between theory and practice > here. What you are arguing for is the "give the users a toolkit and > let them figure it out" approach that David was lambasting upthread. I think maybe my actual argument isn't coming through. What I am arguing for is not shipping XY without Z. That is all. (and no, I don't think we should hold up 8.4). In short. Let's do the feature but don't ship half of it. Ship the whole thing. If that means it doesn't get into 8.4, then it doesn't get into 8.4. > People want the bits to go from point A to point B; they don't want > to have to research, design, test, and administer their own solution > for moving the bits. You don't have to convince me. I agree with you. > I think we have nontrivial > work in front of us to build a simple, reliable, community-tested > log shipping solution; and it's not very sexy work either. But it > needs to get done, and it really needs to get done first. There's > no point in having read-only slave queries if you don't have a > trustworthy method of getting the data to them. O.k. I was with you until here. Log shipping ala pg_standby works fine now sans read-only slave. No, it isn't out of the box which I can see an argument for but it is certainly trustworthy. Or do you mean the synchronous part? Sincerely, Joshua D. rake -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQL"Joshua D. Drake" <jd@...> writes:
> I think maybe my actual argument isn't coming through. What I am arguing > for is not shipping XY without Z. That is all. (and no, I don't think we > should hold up 8.4). So we should keep all the work out of the tree until every part is done? No thanks; especially not when there is a perfectly respectable use-case for parts X and Y alone (whether it suits *your* uses or not). >> There's no point in having read-only slave queries if you don't have a >> trustworthy method of getting the data to them. > O.k. I was with you until here. Log shipping ala pg_standby works fine > now sans read-only slave. No, it isn't out of the box which I can see an > argument for but it is certainly trustworthy. Or do you mean the > synchronous part? How much testing has pg_standby really gotten? Some, sure, but it's a contrib module that wasn't even there before 8.3. Even ignoring the lag issue, I wouldn't trust it a whole lot if I were a DBA responsible for valuable data. As much as some folk would like to think that contrib is mainstream, it's not really in the same league as far as testing coverage goes. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Thu, May 29, 2008 at 7:12 PM, Joshua D. Drake <jd@...> wrote:
> On Thu, 2008-05-29 at 19:02 -0400, Tom Lane wrote: >> I think we have nontrivial >> work in front of us to build a simple, reliable, community-tested >> log shipping solution; and it's not very sexy work either. But it >> needs to get done, and it really needs to get done first. There's >> no point in having read-only slave queries if you don't have a >> trustworthy method of getting the data to them. > > O.k. I was with you until here. Log shipping ala pg_standby works fine > now sans read-only slave. No, it isn't out of the box which I can see an > argument for but it is certainly trustworthy. Or do you mean the > synchronous part? I disagree...setting up pg_standby is more complex than it really has to be. There are several examples in the archives of people getting their standby solutions busted with partial wal files, etc. I helped beta test pg_standby and there are a few 'gotchas' in getting it set up properly. pg_standby is not the problem (although there are some odd things about it), it's getting files from point a to point b. It would be nice to have 'pg_archive' which mirrors pg_standby and handles the work on the client side for example. While some of us can work magic with rsync, etc. It would be nice to get things running with few .conf settings and no external dependencies. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLOn Thu, 29 May 2008, Tom Lane wrote:
> There's no point in having read-only slave queries if you don't have a > trustworthy method of getting the data to them. This is a key statement that highlights the difference in how you're thinking about this compared to some other people here. As far as some are concerned, the already working log shipping *is* a trustworthy method of getting data to the read-only slaves. There are plenty of applications (web oriented ones in particular) where if you could direct read-only queries against a slave, the resulting combination would be a giant improvement over the status quo even if that slave was as much as archive_timeout behind the master. That quantity of lag is perfectly fine for a lot of the same apps that have read scalability issues. If you're someone who falls into that camp, the idea of putting the sync replication job before the read-only slave one seems really backwards. I fully accept that it may be the case that it doesn't make technical sense to tackle them in any order besides sync->read-only slaves because of dependencies in the implementation between the two. If that's the case, it would be nice to explicitly spell out what that was to deflect criticism of the planned prioritization. -- * Greg Smith gsmith@... http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQL* Tom Lane <tgl@...> [080529 20:22]:
> "Joshua D. Drake" <jd@...> writes: > > I think maybe my actual argument isn't coming through. What I am arguing > > for is not shipping XY without Z. That is all. (and no, I don't think we > > should hold up 8.4). > > So we should keep all the work out of the tree until every part is done? > No thanks; especially not when there is a perfectly respectable use-case > for parts X and Y alone (whether it suits *your* uses or not). Thank you. I would be really disappointed if some sort of synchronous WAL shipping patch was done and considered good, but just not applied because some semi-related "read only queries while WAL applying" patch wasn't yet considered done and good... Then again, the opposite is also true. If someone sat down and did the hard work and made read-only queries runnable on a slave applying WAL and the patch was considered done and good, I would be disappointed if it wasn't applied because the synchronous WAL shipping wasn't done. Both features are worthy on their own. But the first done should not be held up because the other is not ready. But because core specifically mentioned "ease of use" as one of the goals, I really think the "built-in WAL shipping" should be *their* focuss first. Having read-only queries that no-one can use (well, sure, some of us could set it up reliably) isn't that useful to "general PostgreSQL" community that I think they were thinking of when deciding on this. But their focus doesn't force anybody interested in other features to not work on their features. It's an OSS community... And the fact that NTT had a nice presentation on it means it's probably actually doable for 8.4: http://www.pgcon.org/2008/schedule/attachments/61_Synchronous%20Log%20Shipping%20Replication.pdf a. -- Aidan Van Dyk Create like a god, aidan@... command like a king, http://www.highrise.ca/ work like a slave. |
|
|
Re: Core team statement on replication in PostgreSQLTom Lane wrote:
>>> There's no point in having read-only slave queries if you don't have a >>> trustworthy method of getting the data to them. > >> O.k. I was with you until here. Log shipping ala pg_standby works fine >> now sans read-only slave. No, it isn't out of the box which I can see an >> argument for but it is certainly trustworthy. Or do you mean the >> synchronous part? > > How much testing has pg_standby really gotten? Some, sure, but it's a > contrib module that wasn't even there before 8.3. Even ignoring the lag > issue, I wouldn't trust it a whole lot if I were a DBA responsible for > valuable data. As much as some folk would like to think that contrib > is mainstream, it's not really in the same league as far as testing > coverage goes. This is a pretty telling statement. Sincerely, Joshua D. Drake -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
|
|
Re: Core team statement on replication in PostgreSQLGreg,
> I fully accept that it may be the case that it doesn't make technical > sense to tackle them in any order besides sync->read-only slaves because > of dependencies in the implementation between the two. If that's the > case, it would be nice to explicitly spell out what that was to deflect > criticism of the planned prioritization. There's a very simple reason to prioritize the synchronous log shipping first; NTT may open source their solution and we'll get it a lot sooner than the other components. That is, we expect that synch log shipping is *easier* than read-only slaves and will get done sooner. Since there are quite a number of users who could use this, whether or not they can run queries on the slaves, why not ship that feature as soon as its done? There's also a number of issues with using the currently log shipping method for replication. In additon to the previously mentioned setup pains, there's the 16MB chunk size for shipping log segments, which is fine for data warehouses but kind of sucks for a web application with a 3GB database which may take 2 hours to go though 16MB. So we have to change the shipping method anyway, and if we're doing that, why not work on synch? Mind you, if someone wanted to get started on read-only slaves *right now* I can't imagine anyone would object. There's a number of problems to solve with recovery mode, table locking etc. that can use some work even before we deal with changes to log shipping, or XID writeback or any of the other issues. So, volunteers? -- Josh Berkus PostgreSQL @ Sun San Francisco -- Sent via pgsql-hackers mailing list (pgsql-hackers@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |
| < Prev | 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 | Next > |
| Free embeddable forum powered by Nabble | Forum Help |