Nabble - PostgreSQL - patches

Re: still alive?

2008-10-01T17:29:35Z

Bruce Momjian wrote:
>
> Marc, care to do the honors?
>

Note:

1. there are several lists to kill, not just pgsql-patches. The
database says:

pgsql-chat
pgsql-benchmarks
pgsql-hackers-win32
pgsql-hackers-pitr
pgsql-cygwin
pgsql-ports

2. The archives must, obviously, survive the kill, and still be
fetchable via rsync to the archives server.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: still alive?

2008-10-01T17:08:00Z

On Wed, 1 Oct 2008 20:05:00 -0400 (EDT)
Bruce Momjian <bruce@...> wrote:

>
> Marc, care to do the honors?

KILL IT!!!!!! :P

>
> ---------------------------------------------------------------------------
>
> Peter Eisentraut wrote:
> > Simon Riggs wrote:
> > > On Thu, 2008-09-11 at 15:39 +0300, Peter Eisentraut wrote:
> > >> Bruce Momjian wrote:
> > >>> Abhijit Menon-Sen wrote:
> > >>>> I thought -patches was supposed to die. What happened?
> > >>> I was wondering the same thing. Peter?
> > >> Hmm, let's try this:
> > >>
> > >> Anyone who thinks the patches list should remain as separate
> > >> from hackers, shout now (with rationale)!
> > >
> > > Kill it now, long enough before the next patchfest for it to
> > > stick.
> >
> > I think what we need now, for patches, ports, and the others, is
> > someone to actually kill it. All the talk has been talked,
> > everything has been decided, now someone with the right permission
> > bits just turn it off.
>

--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: still alive?

2008-10-01T17:05:00Z

Marc, care to do the honors?

---------------------------------------------------------------------------

Peter Eisentraut wrote:

> Simon Riggs wrote:
> > On Thu, 2008-09-11 at 15:39 +0300, Peter Eisentraut wrote:
> >> Bruce Momjian wrote:
> >>> Abhijit Menon-Sen wrote:
> >>>> I thought -patches was supposed to die. What happened?
> >>> I was wondering the same thing. Peter?
> >> Hmm, let's try this:
> >>
> >> Anyone who thinks the patches list should remain as separate from
> >> hackers, shout now (with rationale)!
> >
> > Kill it now, long enough before the next patchfest for it to stick.
>
> I think what we need now, for patches, ports, and the others, is someone
> to actually kill it. All the talk has been talked, everything has been
> decided, now someone with the right permission bits just turn it off.

--
Bruce Momjian <bruce@...> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: still alive?

2008-10-01T17:04:40Z

Peter Eisentraut wrote:

Re: still alive?

2008-10-01T16:53:10Z

Simon Riggs wrote:

> On Thu, 2008-09-11 at 15:39 +0300, Peter Eisentraut wrote:
>> Bruce Momjian wrote:
>>> Abhijit Menon-Sen wrote:
>>>> I thought -patches was supposed to die. What happened?
>>> I was wondering the same thing. Peter?
>> Hmm, let's try this:
>>
>> Anyone who thinks the patches list should remain as separate from
>> hackers, shout now (with rationale)!
>
> Kill it now, long enough before the next patchfest for it to stick.

I think what we need now, for patches, ports, and the others, is someone
to actually kill it. All the talk has been talked, everything has been
decided, now someone with the right permission bits just turn it off.

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: libpq not linked against libgssapi

2008-10-01T08:35:50Z

Markus Schaaf wrote:
> src/interfaces/libpq/Makefile is missing a reference to libgssapi.
> The problem occurred while compiling 8.3.3 on NetBSD --with-gssapi,
> and is still present in HEAD. A patch is attached.

Applied and backpatched to 8.3.

Thanks!

//Magnus

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

libpq not linked against libgssapi

2008-10-01T07:13:24Z

src/interfaces/libpq/Makefile is missing a reference to libgssapi.
The problem occurred while compiling 8.3.3 on NetBSD --with-gssapi,
and is still present in HEAD. A patch is attached.

Index: src/interfaces/libpq/Makefile
===================================================================
RCS file: /projects/cvsroot/pgsql/src/interfaces/libpq/Makefile,v
retrieving revision 1.167
diff -b -u -r1.167 Makefile
--- src/interfaces/libpq/Makefile 17 Sep 2008 04:31:08 -0000 1.167
+++ src/interfaces/libpq/Makefile 1 Oct 2008 13:57:49 -0000
@@ -56,7 +56,7 @@
# shared library link. (The order in which you list them here doesn't
# matter.)
ifneq ($(PORTNAME), win32)
-SHLIB_LINK += $(filter -lcrypt -ldes -lcom_err -lcrypto -lk5crypto -lkrb5 -lgssapi_krb5 -lgss -lssl -lsocket -lnsl -lresolv -lintl, $(LIBS)) $(LDAP_LIBS_FE) $(PTHREAD_LIBS)
+SHLIB_LINK += $(filter -lcrypt -ldes -lcom_err -lcrypto -lk5crypto -lkrb5 -lgssapi_krb5 -lgss -lgssapi -lssl -lsocket -lnsl -lresolv -lintl, $(LIBS)) $(LDAP_LIBS_FE) $(PTHREAD_LIBS)
else
SHLIB_LINK += $(filter -lcrypt -ldes -lcom_err -lcrypto -lk5crypto -lkrb5 -lgssapi32 -lssl -lsocket -lnsl -lresolv -lintl $(PTHREAD_LIBS), $(LIBS)) $(LDAP_LIBS_FE)
endif

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Infrastructure changes for recovery (v8)

2008-09-30T15:52:31Z

Patch now includes all previous agreed changes, plus I've found what
looks to be a workable method of removing the shutdown checkpoint
without loss of robustness.

Patch summary

Tuning
* Bgwriter performs dirty block cleaning during recovery
* Bgwriter performs restartpoints, offloading this task from Startup
process to allow it to continue with recovery actions
* Shutdown checkpoint removed at end of recovery. Bgwriter performs
immediate checkpoint instead, so we have same protection, but
connections and transactions can be started earlier than previously.
* PreAllocXLogs() not performed by startup process, so we do not delay
startup while we write zeroes to next WAL file. bgwriter does that now.
* XLogCtl structure padding for enhanced scalability

Recovery State Changes
* If archive recovery proceeds past a safe stopping point we signal the
postmaster that database is now in a consistent state, PM_RECOVERY. This
state change is also linked to startup of the bgwriter and stats
processes (and will in the future be the place where read only backends
may connect also)
* optional recovery_safe_start_location parameter now provided in
recovery.conf, to allow a consistency point to be manually defined if a
base backup was not taken using standard pg_start/stop backup functions
* New minSafeStopPoint added to controlfile to allow us to determine
consistency if archive recovery crashes/restarts. Value is updated each
time we access new WAL file.
* stats file removed earlier in recovery, so we may accumulate new stats
during recovery
* End of recovery is now marked by a clear global state change. Change
is global, atomic and fast - tested for using IsRecoveryProcessingMode()

Additional Safeguards
* Locks are placed around all ControlFile operations
* XLogInsert() and AssignTransactionId() now have specific checks to
prevent their use during recovery
* Makes StartupMultiXact() atomic. Adds comments to show that
StartCLOG() is already atomic, though StartupSUBTRANS() is not (this
will be addressed in a later patch, so not touched here)
* recovery.conf is not removed until slightly later now, to protect
against crash at the end of startup
* New WAL record XLOG_RECOVERY_END is now only place where timelineid
may change

Other Changes
* log_restartpoints removed, use log_checkpoints in postgresql.conf
* pg_controldata and pg_resetxlog changed to show safe start point
* designed to work in EXEC_BACKEND mode for Windows
* additional function signature for pg_start_backup('label', true |
false) to allow definition of immediate checkpoint/not
* doc changes for recovery.conf parameters
* fixes bug discovered while other testing: if pg_stop_backup() is run
when xlogswitch has just occurred then we do not switch log files, yet
we return current filename even though nothing of value in it. If
archive_timeout not enabled we would wait forever for pg_stop_backup()
to return.
* Substantial comments throughout

Patch is now v8.

doc/src/sgml/backup.sgml | 30 !
doc/src/sgml/func.sgml | 12
src/backend/access/transam/clog.c | 3
src/backend/access/transam/multixact.c | 14
src/backend/access/transam/subtrans.c | 3
src/backend/access/transam/xact.c | 3
src/backend/access/transam/xlog.c | 783 ++++++++++++++-!!!!!!!!!!!!!!!
src/backend/postmaster/bgwriter.c | 418 +++--!!!!!!!!!
src/backend/postmaster/postmaster.c | 62 +!
src/backend/storage/buffer/README | 9
src/bin/pg_controldata/pg_controldata.c | 3
src/bin/pg_resetxlog/pg_resetxlog.c | 2
src/include/access/xlog.h | 14
src/include/access/xlog_internal.h | 4
src/include/catalog/pg_control.h | 3
src/include/postmaster/bgwriter.h | 6
src/include/storage/pmsignal.h | 1
src/test/regress/expected/opr_sanity.out | 7
18 files changed, 579 insertions(+), 79 deletions(-), 719 modifications(!)

Please review everybody. Many thanks.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

[recovery_infrastruc.v8.patch]

Index: doc/src/sgml/backup.sgml =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/backup.sgml,v retrieving revision 2.120 diff -c -r2.120 backup.sgml *** doc/src/sgml/backup.sgml 18 Jul 2008 17:33:17 -0000 2.120 --- doc/src/sgml/backup.sgml 30 Sep 2008 17:15:15 -0000 *************** *** 1200,1205 **** --- 1200,1229 ---- </listitem> </varlistentry> + <varlistentry id="recovery-safe-start-location" + xreflabel="recovery_safe_start_location"> + <term><varname>recovery_safe_start_location</varname> + (<type>string</type>) + </term> + <listitem> + <para> + Allows user to optionally specify a safe start location for a base + backup that was not made online using <function>pg_start_backup()</> + and <function>pg_stop_backup()</>. If those functions were used, + this parameter need not be set because the server sets this for you + automatically to avoid error. You cannot use this parameter to move + the safe stopping point to an earlier transaction log location. The + format for this parameter is identical to the output of + <function>pg_current_xlog_insert_location()</>, example: + <programlisting> + recovery_safe_start_location = '0/D4445B8' + </programlisting> + The location always has a forward slash, even on Windows, since it + is not a file path. + </para> + </listitem> + </varlistentry> + <varlistentry id="log-restartpoints" xreflabel="log_restartpoints"> <term><varname>log_restartpoints</varname> *************** *** 1207,1215 **** </term> <listitem> <para> ! Specifies whether to log each restart point as it occurs. This ! can be helpful to track the progress of a long recovery. ! Default is <literal>false</>. </para> </listitem> </varlistentry> --- 1231,1239 ---- </term> <listitem> <para> ! This parameter has now been deprecated. Instead, please set ! <varname>log_checkpoints</varname> in <filename>postgresql.conf</> ! if you want similar log entries during recovery. </para> </listitem> </varlistentry> Index: doc/src/sgml/func.sgml =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/func.sgml,v retrieving revision 1.447 diff -c -r1.447 func.sgml *** doc/src/sgml/func.sgml 11 Sep 2008 17:32:33 -0000 1.447 --- doc/src/sgml/func.sgml 30 Sep 2008 17:15:15 -0000 *************** *** 12262,12267 **** --- 12262,12275 ---- </row> <row> <entry> + <literal><function>pg_start_backup</function>(<parameter>label</> <type>text</>)</literal> + </entry> + <entry><type>text</type>, <type>boolean</type></entry> + <entry>Set up for performing on-line backup, specifying if + we want an immediate checkpoint or not.</entry> + </row> + <row> + <entry> <literal><function>pg_stop_backup</function>()</literal> </entry> <entry><type>text</type></entry> *************** *** 12333,12338 **** --- 12341,12350 ---- interest). After noting the ending location, the current transaction log insertion point is automatically advanced to the next transaction log file, so that the ending transaction log file can be archived immediately to complete the backup. + <function>pg_start_backup</> issues a checkpoint while we wait. + <function>pg_start_backup</> can also be specified with two parameters, + the second parameter defining whether the checkpoint is an immediate + checkpoint or whether we write out buffers smoothly over a short period. </para> <para> Index: src/backend/access/transam/clog.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/clog.c,v retrieving revision 1.47 diff -c -r1.47 clog.c *** src/backend/access/transam/clog.c 1 Aug 2008 13:16:08 -0000 1.47 --- src/backend/access/transam/clog.c 30 Sep 2008 17:15:15 -0000 *************** *** 260,265 **** --- 260,268 ---- /* * This must be called ONCE during postmaster or standalone-backend startup, * after StartupXLOG has initialized ShmemVariableCache->nextXid. + * + * We access just a single clog page, so this action is atomic and safe + * for use if other processes are active during recovery. */ void StartupCLOG(void) Index: src/backend/access/transam/multixact.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/multixact.c,v retrieving revision 1.28 diff -c -r1.28 multixact.c *** src/backend/access/transam/multixact.c 1 Aug 2008 13:16:08 -0000 1.28 --- src/backend/access/transam/multixact.c 30 Sep 2008 17:15:15 -0000 *************** *** 1413,1420 **** * MultiXactSetNextMXact and/or MultiXactAdvanceNextMXact. Note that we * may already have replayed WAL data into the SLRU files. * ! * We don't need any locks here, really; the SLRU locks are taken ! * only because slru.c expects to be called with locks held. */ void StartupMultiXact(void) --- 1413,1423 ---- * MultiXactSetNextMXact and/or MultiXactAdvanceNextMXact. Note that we * may already have replayed WAL data into the SLRU files. * ! * We want this operation to be atomic to ensure that other processes can ! * use MultiXact while we complete recovery. We access one page only from the ! * offset and members buffers, so once locks are acquired they will not be ! * dropped and re-acquired by SLRU code. So we take both locks at start, then ! * hold them all the way to the end. */ void StartupMultiXact(void) *************** *** 1426,1431 **** --- 1429,1435 ---- /* Clean up offsets state */ LWLockAcquire(MultiXactOffsetControlLock, LW_EXCLUSIVE); + LWLockAcquire(MultiXactMemberControlLock, LW_EXCLUSIVE); /* * Initialize our idea of the latest page number. *************** *** 1452,1461 **** MultiXactOffsetCtl->shared->page_dirty[slotno] = true; } - LWLockRelease(MultiXactOffsetControlLock); - /* And the same for members */ - LWLockAcquire(MultiXactMemberControlLock, LW_EXCLUSIVE); /* * Initialize our idea of the latest page number. --- 1456,1462 ---- *************** *** 1483,1488 **** --- 1484,1490 ---- } LWLockRelease(MultiXactMemberControlLock); + LWLockRelease(MultiXactOffsetControlLock); /* * Initialize lastTruncationPoint to invalid, ensuring that the first *************** *** 1543,1549 **** * SimpleLruTruncate would get confused. It seems best not to risk * removing any data during recovery anyway, so don't truncate. */ ! if (!InRecovery) TruncateMultiXact(); TRACE_POSTGRESQL_MULTIXACT_CHECKPOINT_DONE(true); --- 1545,1551 ---- * SimpleLruTruncate would get confused. It seems best not to risk * removing any data during recovery anyway, so don't truncate. */ ! if (!IsRecoveryProcessingMode()) TruncateMultiXact(); TRACE_POSTGRESQL_MULTIXACT_CHECKPOINT_DONE(true); Index: src/backend/access/transam/subtrans.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/subtrans.c,v retrieving revision 1.23 diff -c -r1.23 subtrans.c *** src/backend/access/transam/subtrans.c 1 Aug 2008 13:16:08 -0000 1.23 --- src/backend/access/transam/subtrans.c 30 Sep 2008 17:15:15 -0000 *************** *** 226,231 **** --- 226,234 ---- * * oldestActiveXID is the oldest XID of any prepared transaction, or nextXid * if there are none. + * + * Note that this is not atomic and is not yet safe to perform while other + * processes might access subtrans. */ void StartupSUBTRANS(TransactionId oldestActiveXID) Index: src/backend/access/transam/xact.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/xact.c,v retrieving revision 1.265 diff -c -r1.265 xact.c *** src/backend/access/transam/xact.c 11 Aug 2008 11:05:10 -0000 1.265 --- src/backend/access/transam/xact.c 30 Sep 2008 17:15:15 -0000 *************** *** 393,398 **** --- 393,401 ---- bool isSubXact = (s->parent != NULL); ResourceOwner currentOwner; + if (IsRecoveryProcessingMode()) + elog(FATAL, "cannot assign TransactionIds during recovery"); + /* Assert that caller didn't screw up */ Assert(!TransactionIdIsValid(s->transactionId)); Assert(s->state == TRANS_INPROGRESS); Index: src/backend/access/transam/xlog.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/xlog.c,v retrieving revision 1.319 diff -c -r1.319 xlog.c *** src/backend/access/transam/xlog.c 23 Sep 2008 09:20:35 -0000 1.319 --- src/backend/access/transam/xlog.c 30 Sep 2008 22:32:49 -0000 *************** *** 113,119 **** /* * ThisTimeLineID will be same in all backends --- it identifies current ! * WAL timeline for the database system. */ TimeLineID ThisTimeLineID = 0; --- 113,120 ---- /* * ThisTimeLineID will be same in all backends --- it identifies current ! * WAL timeline for the database system. Zero is always a bug, so we ! * start with that to allow us to spot any errors. */ TimeLineID ThisTimeLineID = 0; *************** *** 123,128 **** --- 124,133 ---- /* Are we recovering using offline XLOG archives? */ static bool InArchiveRecovery = false; + /* Local copy of shared RecoveryProcessingMode state */ + static bool LocalRecoveryProcessingMode = true; + static bool knownProcessingMode = false; + /* Was the last xlog file restored from archive, or local? */ static bool restoredFromArchive = false; *************** *** 131,137 **** static bool recoveryTarget = false; static bool recoveryTargetExact = false; static bool recoveryTargetInclusive = true; - static bool recoveryLogRestartpoints = false; static TransactionId recoveryTargetXid; static TimestampTz recoveryTargetTime; static TimestampTz recoveryLastXTime = 0; --- 136,141 ---- *************** *** 141,146 **** --- 145,153 ---- static TimestampTz recoveryStopTime; static bool recoveryStopAfter; + /* is the database proven consistent yet? */ + bool reachedSafeStartPoint = false; + /* * During normal operation, the only timeline we care about is ThisTimeLineID. * During recovery, however, things are more complicated. To simplify life *************** *** 240,248 **** * ControlFileLock: must be held to read/update control file or create * new log file. * ! * CheckpointLock: must be held to do a checkpoint (ensures only one ! * checkpointer at a time; currently, with all checkpoints done by the ! * bgwriter, this is just pro forma). * *---------- */ --- 247,256 ---- * ControlFileLock: must be held to read/update control file or create * new log file. * ! * CheckpointLock: must be held to do a checkpoint or restartpoint, ensuring ! * we get just one of those at any time. In 8.4+ recovery, both startup and ! * bgwriter processes may take restartpoints, so this locking must be strict ! * to ensure there are no mistakes. * *---------- */ *************** *** 285,295 **** --- 293,310 ---- /* * Total shared-memory state for XLOG. + * + * This small structure is accessed by many backends, so we take care to + * pad out the parts of the structure so they can be accessed by separate + * CPUs without causing false sharing cache flushes. Padding is generous + * to allow for a wide variety of CPU architectures. */ + #define XLOGCTL_BUFFER_SPACING 128 typedef struct XLogCtlData { /* Protected by WALInsertLock: */ XLogCtlInsert Insert; + char InsertPadding[XLOGCTL_BUFFER_SPACING - sizeof(XLogCtlInsert)]; /* Protected by info_lck: */ XLogwrtRqst LogwrtRqst; *************** *** 297,305 **** --- 312,327 ---- uint32 ckptXidEpoch; /* nextXID & epoch of latest checkpoint */ TransactionId ckptXid; XLogRecPtr asyncCommitLSN; /* LSN of newest async commit */ + /* add data structure padding for above info_lck declarations */ + char InfoPadding[XLOGCTL_BUFFER_SPACING - sizeof(XLogwrtRqst) + - sizeof(XLogwrtResult) + - sizeof(uint32) + - sizeof(TransactionId) + - sizeof(XLogRecPtr)]; /* Protected by WALWriteLock: */ XLogCtlWrite Write; + char WritePadding[XLOGCTL_BUFFER_SPACING - sizeof(XLogCtlWrite)]; /* * These values do not change after startup, although the pointed-to pages *************** *** 311,316 **** --- 333,356 ---- int XLogCacheBlck; /* highest allocated xlog buffer index */ TimeLineID ThisTimeLineID; + /* + * IsRecoveryProcessingMode shows whether the postmaster is in a + * postmaster state earlier than PM_RUN, or not. This is a globally + * accessible state to allow EXEC_BACKEND case. + * + * We also retain a local state variable InRecovery. InRecovery=true + * means the code is being executed by Startup process and therefore + * always during Recovery Processing Mode. This allows us to identify + * code executed *during* Recovery Processing Mode but not necessarily + * by Startup process itself. + * + * Protected by mode_lck + */ + bool SharedRecoveryProcessingMode; + slock_t mode_lck; + + char InfoLockPadding[XLOGCTL_BUFFER_SPACING]; + slock_t info_lck; /* locks shared variables shown above */ } XLogCtlData; *************** *** 397,404 **** --- 437,446 ---- static void readRecoveryCommandFile(void); static void exitArchiveRecovery(TimeLineID endTLI, uint32 endLogId, uint32 endLogSeg); + static void exitRecovery(void); static bool recoveryStopsHere(XLogRecord *record, bool *includeThis); static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags); + static XLogRecPtr GetRedoLocationForCheckpoint(void); static bool XLogCheckBuffer(XLogRecData *rdata, bool doPageWrites, XLogRecPtr *lsn, BkpBlock *bkpb); *************** *** 480,485 **** --- 522,532 ---- bool updrqst; bool doPageWrites; bool isLogSwitch = (rmid == RM_XLOG_ID && info == XLOG_SWITCH); + bool isRecoveryEnd = (rmid == RM_XLOG_ID && info == XLOG_RECOVERY_END); + + /* cross-check on whether we should be here or not */ + if (IsRecoveryProcessingMode() && !isRecoveryEnd) + elog(FATAL, "cannot make new WAL entries during recovery"); /* info's high bits are reserved for use by me */ if (info & XLR_INFO_MASK) *************** *** 1720,1727 **** XLogRecPtr WriteRqstPtr; XLogwrtRqst WriteRqst; ! /* Disabled during REDO */ ! if (InRedo) return; /* Quick exit if already known flushed */ --- 1767,1773 ---- XLogRecPtr WriteRqstPtr; XLogwrtRqst WriteRqst; ! if (IsRecoveryProcessingMode()) return; /* Quick exit if already known flushed */ *************** *** 1809,1817 **** * the bad page is encountered again during recovery then we would be * unable to restart the database at all! (This scenario has actually * happened in the field several times with 7.1 releases. Note that we ! * cannot get here while InRedo is true, but if the bad page is brought in ! * and marked dirty during recovery then CreateCheckPoint will try to ! * flush it at the end of recovery.) * * The current approach is to ERROR under normal conditions, but only * WARNING during recovery, so that the system can be brought up even if --- 1855,1863 ---- * the bad page is encountered again during recovery then we would be * unable to restart the database at all! (This scenario has actually * happened in the field several times with 7.1 releases. Note that we ! * cannot get here while IsRecoveryProcessingMode(), but if the bad page is ! * brought in and marked dirty during recovery then if a checkpoint were ! * performed at the end of recovery it will try to flush it. * * The current approach is to ERROR under normal conditions, but only * WARNING during recovery, so that the system can be brought up even if *************** *** 1821,1827 **** * and so we will not force a restart for a bad LSN on a data page. */ if (XLByteLT(LogwrtResult.Flush, record)) ! elog(InRecovery ? WARNING : ERROR, "xlog flush request %X/%X is not satisfied --- flushed only to %X/%X", record.xlogid, record.xrecoff, LogwrtResult.Flush.xlogid, LogwrtResult.Flush.xrecoff); --- 1867,1873 ---- * and so we will not force a restart for a bad LSN on a data page. */ if (XLByteLT(LogwrtResult.Flush, record)) ! elog(ERROR, "xlog flush request %X/%X is not satisfied --- flushed only to %X/%X", record.xlogid, record.xrecoff, LogwrtResult.Flush.xlogid, LogwrtResult.Flush.xrecoff); *************** *** 2094,2100 **** unlink(tmppath); } ! elog(DEBUG2, "done creating and filling new WAL file"); /* Set flag to tell caller there was no existent file */ *use_existent = false; --- 2140,2147 ---- unlink(tmppath); } ! XLogFileName(tmppath, ThisTimeLineID, log, seg); ! elog(DEBUG2, "done creating and filling new WAL file %s", tmppath); /* Set flag to tell caller there was no existent file */ *use_existent = false; *************** *** 2400,2405 **** --- 2447,2474 ---- xlogfname); set_ps_display(activitymsg, false); + /* + * Calculate and write out a new safeStartPoint. This defines + * the latest LSN that might appear on-disk while we apply + * the WAL records in this file. If we crash during recovery + * we must reach this point again before we can prove + * database consistency. Not a restartpoint! Restart points + * define where we should start recovery from, if we crash. + */ + if (InArchiveRecovery) + { + uint32 nextLog = log; + uint32 nextSeg = seg; + + NextLogSeg(nextLog, nextSeg); + + LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); + ControlFile->minSafeStartPoint.xlogid = nextLog; + ControlFile->minSafeStartPoint.xrecoff = nextSeg * XLogSegSize; + UpdateControlFile(); + LWLockRelease(ControlFileLock); + } + return fd; } if (errno != ENOENT) /* unexpected failure? */ *************** *** 4228,4233 **** --- 4297,4303 ---- XLogCtl->XLogCacheBlck = XLOGbuffers - 1; XLogCtl->Insert.currpage = (XLogPageHeader) (XLogCtl->pages); SpinLockInit(&XLogCtl->info_lck); + SpinLockInit(&XLogCtl->mode_lck); /* * If we are not in bootstrap mode, pg_control should already exist. Read *************** *** 4532,4548 **** ereport(LOG, (errmsg("recovery_target_inclusive = %s", tok2))); } else if (strcmp(tok1, "log_restartpoints") == 0) { - /* - * does nothing if a recovery_target is not also set - */ - if (!parse_bool(tok2, &recoveryLogRestartpoints)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("parameter \"log_restartpoints\" requires a Boolean value"))); ereport(LOG, ! (errmsg("log_restartpoints = %s", tok2))); } else ereport(FATAL, --- 4602,4642 ---- ereport(LOG, (errmsg("recovery_target_inclusive = %s", tok2))); } + else if (strcmp(tok1, "recovery_safe_start_location") == 0) + { + unsigned int uxlogid; + unsigned int uxrecoff; + XLogRecPtr NewSafeStartPtr; + + if (sscanf(tok2, "%X/%X", &uxlogid, &uxrecoff) != 2) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not parse transaction log location \"%s\"", + tok2))); + + NewSafeStartPtr.xlogid = uxlogid; + NewSafeStartPtr.xrecoff = uxrecoff; + if (XLByteLE(ControlFile->minSafeStartPoint, NewSafeStartPtr)) + { + ControlFile->minSafeStartPoint.xlogid = uxlogid; + ControlFile->minSafeStartPoint.xrecoff = uxrecoff; + + ereport(LOG, + (errmsg("recovery_safe_start_location = '%s'", tok2))); + } + else if (ControlFile->state != DB_IN_ARCHIVE_RECOVERY) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("recovery_safe_start_location = '%s' is earlier than control file %X/%X", + tok2, + ControlFile->minSafeStartPoint.xlogid, + ControlFile->minSafeStartPoint.xrecoff))); + } else if (strcmp(tok1, "log_restartpoints") == 0) { ereport(LOG, ! (errcode(ERRCODE_INVALID_PARAMETER_VALUE), ! errmsg("parameter \"log_restartpoints\" has been deprecated"))); } else ereport(FATAL, *************** *** 4678,4692 **** unlink(recoveryPath); /* ignore any error */ /* ! * Rename the config file out of the way, so that we don't accidentally ! * re-enter archive recovery mode in a subsequent crash. */ - unlink(RECOVERY_COMMAND_DONE); - if (rename(RECOVERY_COMMAND_FILE, RECOVERY_COMMAND_DONE) != 0) - ereport(FATAL, - (errcode_for_file_access(), - errmsg("could not rename file \"%s\" to \"%s\": %m", - RECOVERY_COMMAND_FILE, RECOVERY_COMMAND_DONE))); ereport(LOG, (errmsg("archive recovery complete"))); --- 4772,4784 ---- unlink(recoveryPath); /* ignore any error */ /* ! * As of 8.4 we no longer rename the recovery.conf file out of the ! * way until after we have performed a full checkpoint. This ensures ! * that any crash between now and the end of the checkpoint does not ! * attempt to restart from a WAL file that is no longer available to us. ! * As soon as we remove recovery.conf we lose our recovery_command and ! * cannot reaccess WAL files from the archive. */ ereport(LOG, (errmsg("archive recovery complete"))); *************** *** 4813,4818 **** --- 4905,4911 ---- CheckPoint checkPoint; bool wasShutdown; bool reachedStopPoint = false; + bool performedRecovery = false; bool haveBackupLabel = false; XLogRecPtr RecPtr, LastRec, *************** *** 4825,4830 **** --- 4918,4925 ---- uint32 freespace; TransactionId oldestActiveXID; + XLogCtl->SharedRecoveryProcessingMode = true; + /* * Read control file and check XLOG status looks valid. * *************** *** 5038,5046 **** --- 5133,5147 ---- if (minRecoveryLoc.xlogid != 0 || minRecoveryLoc.xrecoff != 0) ControlFile->minRecoveryPoint = minRecoveryLoc; ControlFile->time = (pg_time_t) time(NULL); + /* No need to hold ControlFileLock yet, we aren't up far enough */ UpdateControlFile(); /* + * Reset pgstat data, because it may be invalid after recovery. + */ + pgstat_reset_all(); + + /* * If there was a backup label file, it's done its job and the info * has now been propagated into pg_control. We must get rid of the * label file so that if we crash during recovery, we'll pick up at *************** *** 5150,5155 **** --- 5251,5282 ---- LastRec = ReadRecPtr; + /* + * Have we reached our safe starting point? If so, we can + * signal Postmaster to enter consistent recovery mode. + * + * There are two point in the log we must pass. The first is + * the minRecoveryPoint, which is the LSN at the time the + * base backup was taken that we are about to rollfoward from. + * If recovery has ever crashed or was stopped there is + * another point also: minSafeStartPoint, which we know the + * latest LSN that recovery could have reached prior to crash. + */ + if (!reachedSafeStartPoint && + XLByteLE(ControlFile->minSafeStartPoint, EndRecPtr) && + XLByteLE(ControlFile->minRecoveryPoint, EndRecPtr)) + { + reachedSafeStartPoint = true; + if (InArchiveRecovery) + { + ereport(LOG, + (errmsg("consistent recovery state reached at %X/%X", + EndRecPtr.xlogid, EndRecPtr.xrecoff))); + if (IsUnderPostmaster) + SendPostmasterSignal(PMSIGNAL_RECOVERY_START); + } + } + record = ReadRecord(NULL, LOG); } while (record != NULL && recoveryContinue); *************** *** 5171,5176 **** --- 5298,5304 ---- /* there are no WAL records following the checkpoint */ ereport(LOG, (errmsg("redo is not required"))); + reachedSafeStartPoint = true; } } *************** *** 5184,5192 **** /* * Complain if we did not roll forward far enough to render the backup ! * dump consistent. */ ! if (XLByteLT(EndOfLog, ControlFile->minRecoveryPoint)) { if (reachedStopPoint) /* stopped because of stop request */ ereport(FATAL, --- 5312,5320 ---- /* * Complain if we did not roll forward far enough to render the backup ! * dump consistent and start safely. */ ! if (InRecovery && !reachedSafeStartPoint) { if (reachedStopPoint) /* stopped because of stop request */ ereport(FATAL, *************** *** 5308,5346 **** XLogCheckInvalidPages(); /* ! * Reset pgstat data, because it may be invalid after recovery. */ ! pgstat_reset_all(); ! /* ! * Perform a checkpoint to update all our recovery activity to disk. ! * ! * Note that we write a shutdown checkpoint rather than an on-line ! * one. This is not particularly critical, but since we may be ! * assigning a new TLI, using a shutdown checkpoint allows us to have ! * the rule that TLI only changes in shutdown checkpoints, which ! * allows some extra error checking in xlog_redo. ! */ ! CreateCheckPoint(CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_IMMEDIATE); } - /* - * Preallocate additional log files, if wanted. - */ - PreallocXlogFiles(EndOfLog); - - /* - * Okay, we're officially UP. - */ - InRecovery = false; - - ControlFile->state = DB_IN_PRODUCTION; - ControlFile->time = (pg_time_t) time(NULL); - UpdateControlFile(); - - /* start the archive_timeout timer running */ - XLogCtl->Write.lastSegSwitchTime = ControlFile->time; - /* initialize shared-memory copy of latest checkpoint XID/epoch */ XLogCtl->ckptXidEpoch = ControlFile->checkPointCopy.nextXidEpoch; XLogCtl->ckptXid = ControlFile->checkPointCopy.nextXid; --- 5436,5449 ---- XLogCheckInvalidPages(); /* ! * Finally exit recovery and mark that in WAL. Pre-8.4 we wrote ! * a shutdown checkpoint here, but we ask bgwriter to do that now. */ ! exitRecovery(); ! performedRecovery = true; } /* initialize shared-memory copy of latest checkpoint XID/epoch */ XLogCtl->ckptXidEpoch = ControlFile->checkPointCopy.nextXidEpoch; XLogCtl->ckptXid = ControlFile->checkPointCopy.nextXid; *************** *** 5374,5379 **** --- 5477,5565 ---- readRecordBuf = NULL; readRecordBufSize = 0; } + + /* + * Prior to 8.4 we wrote a Shutdown Checkpoint at the end of recovery. + * This could add minutes to the startup time, so we want bgwriter + * to perform it. This then frees the Startup process to complete so we can + * allow transactions and WAL inserts. We still write a checkpoint, but + * it will be an online checkpoint. Online checkpoints have a redo + * location that can be prior to the actual checkpoint record. So we want + * to derive that redo location *before* we let anybody else write WAL, + * otherwise we might miss some WAL records if we crash. + */ + if (performedRecovery) + { + XLogRecPtr redo; + + /* + * We must grab the pointer before anybody writes WAL + */ + redo = GetRedoLocationForCheckpoint(); + + /* + * Tell the bgwriter + */ + SetRedoLocationForArchiveCheckpoint(redo); + + /* + * Okay, we can come up now. Allow others to write WAL. + */ + XLogCtl->SharedRecoveryProcessingMode = false; + + /* + * Now request checkpoint + */ + RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_IMMEDIATE); + } + else + { + /* + * No recovery, so lets just get on with it. + */ + LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); + ControlFile->state = DB_IN_PRODUCTION; + ControlFile->time = (pg_time_t) time(NULL); + UpdateControlFile(); + LWLockRelease(ControlFileLock); + + /* + * Okay, we're officially UP. + */ + XLogCtl->SharedRecoveryProcessingMode = false; + } + + /* start the archive_timeout timer running */ + XLogCtl->Write.lastSegSwitchTime = (pg_time_t) time(NULL); + + } + + /* + * IsRecoveryProcessingMode() + * + * Fast test for whether we're still in recovery or not. We test the shared + * state each time only until we leave recovery mode. After that we never + * look again, relying upon the settings of our local state variables. This + * is designed to avoid the need for a separate initialisation step. + */ + bool + IsRecoveryProcessingMode(void) + { + if (knownProcessingMode && !LocalRecoveryProcessingMode) + return false; + + { + /* use volatile pointer to prevent code rearrangement */ + volatile XLogCtlData *xlogctl = XLogCtl; + + SpinLockAcquire(&xlogctl->mode_lck); + LocalRecoveryProcessingMode = XLogCtl->SharedRecoveryProcessingMode; + SpinLockRelease(&xlogctl->mode_lck); + } + + knownProcessingMode = true; + + return LocalRecoveryProcessingMode; } /* *************** *** 5631,5650 **** static void LogCheckpointStart(int flags) { ! elog(LOG, "checkpoint starting:%s%s%s%s%s%s", ! (flags & CHECKPOINT_IS_SHUTDOWN) ? " shutdown" : "", ! (flags & CHECKPOINT_IMMEDIATE) ? " immediate" : "", ! (flags & CHECKPOINT_FORCE) ? " force" : "", ! (flags & CHECKPOINT_WAIT) ? " wait" : "", ! (flags & CHECKPOINT_CAUSE_XLOG) ? " xlog" : "", ! (flags & CHECKPOINT_CAUSE_TIME) ? " time" : ""); } /* * Log end of a checkpoint. */ static void ! LogCheckpointEnd(void) { long write_secs, sync_secs, --- 5817,5840 ---- static void LogCheckpointStart(int flags) { ! if (flags & CHECKPOINT_RESTARTPOINT) ! elog(LOG, "restartpoint starting:%s", ! (flags & CHECKPOINT_IMMEDIATE) ? " immediate" : ""); ! else ! elog(LOG, "checkpoint starting:%s%s%s%s%s%s", ! (flags & CHECKPOINT_IS_SHUTDOWN) ? " shutdown" : "", ! (flags & CHECKPOINT_IMMEDIATE) ? " immediate" : "", ! (flags & CHECKPOINT_FORCE) ? " force" : "", ! (flags & CHECKPOINT_WAIT) ? " wait" : "", ! (flags & CHECKPOINT_CAUSE_XLOG) ? " xlog" : "", ! (flags & CHECKPOINT_CAUSE_TIME) ? " time" : ""); } /* * Log end of a checkpoint. */ static void ! LogCheckpointEnd(int flags) { long write_secs, sync_secs, *************** *** 5667,5683 **** CheckpointStats.ckpt_sync_end_t, &sync_secs, &sync_usecs); ! elog(LOG, "checkpoint complete: wrote %d buffers (%.1f%%); " ! "%d transaction log file(s) added, %d removed, %d recycled; " ! "write=%ld.%03d s, sync=%ld.%03d s, total=%ld.%03d s", ! CheckpointStats.ckpt_bufs_written, ! (double) CheckpointStats.ckpt_bufs_written * 100 / NBuffers, ! CheckpointStats.ckpt_segs_added, ! CheckpointStats.ckpt_segs_removed, ! CheckpointStats.ckpt_segs_recycled, ! write_secs, write_usecs / 1000, ! sync_secs, sync_usecs / 1000, ! total_secs, total_usecs / 1000); } /* --- 5857,5882 ---- CheckpointStats.ckpt_sync_end_t, &sync_secs, &sync_usecs); ! if (flags & CHECKPOINT_RESTARTPOINT) ! elog(LOG, "restartpoint complete: wrote %d buffers (%.1f%%); " ! "write=%ld.%03d s, sync=%ld.%03d s, total=%ld.%03d s", ! CheckpointStats.ckpt_bufs_written, ! (double) CheckpointStats.ckpt_bufs_written * 100 / NBuffers, ! write_secs, write_usecs / 1000, ! sync_secs, sync_usecs / 1000, ! total_secs, total_usecs / 1000); ! else ! elog(LOG, "checkpoint complete: wrote %d buffers (%.1f%%); " ! "%d transaction log file(s) added, %d removed, %d recycled; " ! "write=%ld.%03d s, sync=%ld.%03d s, total=%ld.%03d s", ! CheckpointStats.ckpt_bufs_written, ! (double) CheckpointStats.ckpt_bufs_written * 100 / NBuffers, ! CheckpointStats.ckpt_segs_added, ! CheckpointStats.ckpt_segs_removed, ! CheckpointStats.ckpt_segs_recycled, ! write_secs, write_usecs / 1000, ! sync_secs, sync_usecs / 1000, ! total_secs, total_usecs / 1000); } /* *************** *** 5702,5718 **** XLogRecPtr recptr; XLogCtlInsert *Insert = &XLogCtl->Insert; XLogRecData rdata; - uint32 freespace; uint32 _logId; uint32 _logSeg; TransactionId *inCommitXids; int nInCommit; /* * Acquire CheckpointLock to ensure only one checkpoint happens at a time. ! * (This is just pro forma, since in the present system structure there is ! * only one process that is allowed to issue checkpoints at any given ! * time.) */ LWLockAcquire(CheckpointLock, LW_EXCLUSIVE); --- 5901,5916 ---- XLogRecPtr recptr; XLogCtlInsert *Insert = &XLogCtl->Insert; XLogRecData rdata; uint32 _logId; uint32 _logSeg; TransactionId *inCommitXids; int nInCommit; + bool leavingArchiveRecovery = false; /* * Acquire CheckpointLock to ensure only one checkpoint happens at a time. ! * That shouldn't be happening, but checkpoints are an important aspect ! * of our resilience, so we take no chances. */ LWLockAcquire(CheckpointLock, LW_EXCLUSIVE); *************** *** 5727,5741 **** --- 5925,5948 ---- CheckpointStats.ckpt_start_t = GetCurrentTimestamp(); /* + * Find out if this is the first checkpoint after archive recovery. + */ + LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); + leavingArchiveRecovery = (ControlFile->state == DB_IN_ARCHIVE_RECOVERY); + LWLockRelease(ControlFileLock); + + /* * Use a critical section to force system panic if we have trouble. */ START_CRIT_SECTION(); if (shutdown) { + LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); ControlFile->state = DB_SHUTDOWNING; ControlFile->time = (pg_time_t) time(NULL); UpdateControlFile(); + LWLockRelease(ControlFileLock); } /* *************** *** 5750,5840 **** checkPoint.ThisTimeLineID = ThisTimeLineID; checkPoint.time = (pg_time_t) time(NULL); ! /* ! * We must hold WALInsertLock while examining insert state to determine ! * the checkpoint REDO pointer. ! */ ! LWLockAcquire(WALInsertLock, LW_EXCLUSIVE); ! ! /* ! * If this isn't a shutdown or forced checkpoint, and we have not inserted ! * any XLOG records since the start of the last checkpoint, skip the ! * checkpoint. The idea here is to avoid inserting duplicate checkpoints ! * when the system is idle. That wastes log space, and more importantly it ! * exposes us to possible loss of both current and previous checkpoint ! * records if the machine crashes just as we're writing the update. ! * (Perhaps it'd make even more sense to checkpoint only when the previous ! * checkpoint record is in a different xlog page?) ! * ! * We have to make two tests to determine that nothing has happened since ! * the start of the last checkpoint: current insertion point must match ! * the end of the last checkpoint record, and its redo pointer must point ! * to itself. ! */ ! if ((flags & (CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_FORCE)) == 0) { ! XLogRecPtr curInsert; ! INSERT_RECPTR(curInsert, Insert, Insert->curridx); ! if (curInsert.xlogid == ControlFile->checkPoint.xlogid && ! curInsert.xrecoff == ControlFile->checkPoint.xrecoff + ! MAXALIGN(SizeOfXLogRecord + sizeof(CheckPoint)) && ! ControlFile->checkPoint.xlogid == ! ControlFile->checkPointCopy.redo.xlogid && ! ControlFile->checkPoint.xrecoff == ! ControlFile->checkPointCopy.redo.xrecoff) { ! LWLockRelease(WALInsertLock); ! LWLockRelease(CheckpointLock); ! END_CRIT_SECTION(); ! return; ! } ! } ! /* ! * Compute new REDO record ptr = location of next XLOG record. ! * ! * NB: this is NOT necessarily where the checkpoint record itself will be, ! * since other backends may insert more XLOG records while we're off doing ! * the buffer flush work. Those XLOG records are logically after the ! * checkpoint, even though physically before it. Got that? ! */ ! freespace = INSERT_FREESPACE(Insert); ! if (freespace < SizeOfXLogRecord) ! { ! (void) AdvanceXLInsertBuffer(false); ! /* OK to ignore update return flag, since we will do flush anyway */ ! freespace = INSERT_FREESPACE(Insert); ! } ! INSERT_RECPTR(checkPoint.redo, Insert, Insert->curridx); ! /* ! * Here we update the shared RedoRecPtr for future XLogInsert calls; this ! * must be done while holding the insert lock AND the info_lck. ! * ! * Note: if we fail to complete the checkpoint, RedoRecPtr will be left ! * pointing past where it really needs to point. This is okay; the only ! * consequence is that XLogInsert might back up whole buffers that it ! * didn't really need to. We can't postpone advancing RedoRecPtr because ! * XLogInserts that happen while we are dumping buffers must assume that ! * their buffer changes are not included in the checkpoint. ! */ ! { ! /* use volatile pointer to prevent code rearrangement */ ! volatile XLogCtlData *xlogctl = XLogCtl; ! SpinLockAcquire(&xlogctl->info_lck); ! RedoRecPtr = xlogctl->Insert.RedoRecPtr = checkPoint.redo; ! SpinLockRelease(&xlogctl->info_lck); } /* - * Now we can release WAL insert lock, allowing other xacts to proceed - * while we are flushing disk buffers. - */ - LWLockRelease(WALInsertLock); - - /* * If enabled, log checkpoint start. We postpone this until now so as not * to log anything if we decided to skip the checkpoint. */ --- 5957,6025 ---- checkPoint.ThisTimeLineID = ThisTimeLineID; checkPoint.time = (pg_time_t) time(NULL); ! if (leavingArchiveRecovery) ! checkPoint.redo = GetRedoLocationForArchiveCheckpoint(); ! else { ! /* ! * We must hold WALInsertLock while examining insert state to determine ! * the checkpoint REDO pointer. ! */ ! LWLockAcquire(WALInsertLock, LW_EXCLUSIVE); ! /* ! * If this isn't a shutdown or forced checkpoint, and we have not inserted ! * any XLOG records since the start of the last checkpoint, skip the ! * checkpoint. The idea here is to avoid inserting duplicate checkpoints ! * when the system is idle. That wastes log space, and more importantly it ! * exposes us to possible loss of both current and previous checkpoint ! * records if the machine crashes just as we're writing the update. ! * (Perhaps it'd make even more sense to checkpoint only when the previous ! * checkpoint record is in a different xlog page?) ! * ! * We have to make two tests to determine that nothing has happened since ! * the start of the last checkpoint: current insertion point must match ! * the end of the last checkpoint record, and its redo pointer must point ! * to itself. ! */ ! if ((flags & (CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_FORCE)) == 0) { ! XLogRecPtr curInsert; ! INSERT_RECPTR(curInsert, Insert, Insert->curridx); ! if (curInsert.xlogid == ControlFile->checkPoint.xlogid && ! curInsert.xrecoff == ControlFile->checkPoint.xrecoff + ! MAXALIGN(SizeOfXLogRecord + sizeof(CheckPoint)) && ! ControlFile->checkPoint.xlogid == ! ControlFile->checkPointCopy.redo.xlogid && ! ControlFile->checkPoint.xrecoff == ! ControlFile->checkPointCopy.redo.xrecoff) ! { ! LWLockRelease(WALInsertLock); ! LWLockRelease(CheckpointLock); ! END_CRIT_SECTION(); ! return; ! } ! } ! /* ! * Compute new REDO record ptr = location of next XLOG record. ! * ! * NB: this is NOT necessarily where the checkpoint record itself will be, ! * since other backends may insert more XLOG records while we're off doing ! * the buffer flush work. Those XLOG records are logically after the ! * checkpoint, even though physically before it. Got that? ! */ ! checkPoint.redo = GetRedoLocationForCheckpoint(); ! /* ! * Now we can release WAL insert lock, allowing other xacts to proceed ! * while we are flushing disk buffers. ! */ ! LWLockRelease(WALInsertLock); } /* * If enabled, log checkpoint start. We postpone this until now so as not * to log anything if we decided to skip the checkpoint. */ *************** *** 5941,5958 **** XLByteToSeg(ControlFile->checkPointCopy.redo, _logId, _logSeg); /* ! * Update the control file. */ LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); if (shutdown) ControlFile->state = DB_SHUTDOWNED; ControlFile->prevCheckPoint = ControlFile->checkPoint; ControlFile->checkPoint = ProcLastRecPtr; ControlFile->checkPointCopy = checkPoint; ControlFile->time = (pg_time_t) time(NULL); UpdateControlFile(); LWLockRelease(ControlFileLock); /* Update shared-memory copy of checkpoint XID/epoch */ { /* use volatile pointer to prevent code rearrangement */ --- 6126,6168 ---- XLByteToSeg(ControlFile->checkPointCopy.redo, _logId, _logSeg); /* ! * Update the control file. In 8.4, this routine becomes the primary ! * point for recording changes of state in the control file at the ! * end of recovery. Postmaster state already shows us being in ! * normal running mode, but it is only after this point that we ! * are completely free of reperforming a recovery if we crash. Note ! * that this is executed by bgwriter after the death of Startup process. */ LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); + if (shutdown) ControlFile->state = DB_SHUTDOWNED; + else + ControlFile->state = DB_IN_PRODUCTION; + ControlFile->prevCheckPoint = ControlFile->checkPoint; ControlFile->checkPoint = ProcLastRecPtr; ControlFile->checkPointCopy = checkPoint; ControlFile->time = (pg_time_t) time(NULL); UpdateControlFile(); + LWLockRelease(ControlFileLock); + if (leavingArchiveRecovery) + { + /* + * Rename the config file out of the way, so that we don't accidentally + * re-enter archive recovery mode in a subsequent crash. Prior to + * 8.4 this step was performed at end of exitArchiveRecovery(). + */ + unlink(RECOVERY_COMMAND_DONE); + if (rename(RECOVERY_COMMAND_FILE, RECOVERY_COMMAND_DONE) != 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not rename file \"%s\" to \"%s\": %m", + RECOVERY_COMMAND_FILE, RECOVERY_COMMAND_DONE))); + } + /* Update shared-memory copy of checkpoint XID/epoch */ { /* use volatile pointer to prevent code rearrangement */ *************** *** 5999,6014 **** * in subtrans.c). During recovery, though, we mustn't do this because * StartupSUBTRANS hasn't been called yet. */ ! if (!InRecovery) ! TruncateSUBTRANS(GetOldestXmin(true, false)); /* All real work is done, but log before releasing lock. */ if (log_checkpoints) ! LogCheckpointEnd(); LWLockRelease(CheckpointLock); } /* * Flush all data in shared memory to disk, and fsync * --- 6209,6268 ---- * in subtrans.c). During recovery, though, we mustn't do this because * StartupSUBTRANS hasn't been called yet. */ ! TruncateSUBTRANS(GetOldestXmin(true, false)); /* All real work is done, but log before releasing lock. */ if (log_checkpoints) ! LogCheckpointEnd(flags); LWLockRelease(CheckpointLock); } + /* + * GetRedoLocationForCheckpoint() + * + * When !IsRecoveryProcessingMode() this must be called while holding + * WALInsertLock(). + */ + static XLogRecPtr + GetRedoLocationForCheckpoint() + { + XLogCtlInsert *Insert = &XLogCtl->Insert; + uint32 freespace; + XLogRecPtr redo; + + freespace = INSERT_FREESPACE(Insert); + if (freespace < SizeOfXLogRecord) + { + (void) AdvanceXLInsertBuffer(false); + /* OK to ignore update return flag, since we will do flush anyway */ + freespace = INSERT_FREESPACE(Insert); + } + INSERT_RECPTR(redo, Insert, Insert->curridx); + + /* + * Here we update the shared RedoRecPtr for future XLogInsert calls; this + * must be done while holding the insert lock AND the info_lck. + * + * Note: if we fail to complete the checkpoint, RedoRecPtr will be left + * pointing past where it really needs to point. This is okay; the only + * consequence is that XLogInsert might back up whole buffers that it + * didn't really need to. We can't postpone advancing RedoRecPtr because + * XLogInserts that happen while we are dumping buffers must assume that + * their buffer changes are not included in the checkpoint. + */ + { + /* use volatile pointer to prevent code rearrangement */ + volatile XLogCtlData *xlogctl = XLogCtl; + + SpinLockAcquire(&xlogctl->info_lck); + RedoRecPtr = xlogctl->Insert.RedoRecPtr = redo; + SpinLockRelease(&xlogctl->info_lck); + } + + return redo; + } + /* * Flush all data in shared memory to disk, and fsync * *************** *** 6073,6101 **** } } /* ! * OK, force data out to disk */ ! CheckPointGuts(checkPoint->redo, CHECKPOINT_IMMEDIATE); /* ! * Update pg_control so that any subsequent crash will restart from this ! * checkpoint. Note: ReadRecPtr gives the XLOG address of the checkpoint ! * record itself. */ ControlFile->prevCheckPoint = ControlFile->checkPoint; ! ControlFile->checkPoint = ReadRecPtr; ! ControlFile->checkPointCopy = *checkPoint; ControlFile->time = (pg_time_t) time(NULL); UpdateControlFile(); ! ereport((recoveryLogRestartpoints ? LOG : DEBUG2), (errmsg("recovery restart point at %X/%X", ! checkPoint->redo.xlogid, checkPoint->redo.xrecoff))); if (recoveryLastXTime) ! ereport((recoveryLogRestartpoints ? LOG : DEBUG2), ! (errmsg("last completed transaction was at log time %s", ! timestamptz_to_str(recoveryLastXTime)))); } /* --- 6327,6395 ---- } } + RequestRestartPoint(ReadRecPtr, checkPoint, reachedSafeStartPoint); + } + + /* + * As of 8.4, RestartPoints are always created by the bgwriter + * once we have reachedSafeStartPoint. We use bgwriter's shared memory + * area wherever we call it from, to keep better code structure. + */ + void + CreateRestartPoint(const XLogRecPtr ReadPtr, const CheckPoint *restartPoint, int flags) + { + if (log_checkpoints) + { + /* + * Prepare to accumulate statistics. + */ + + MemSet(&CheckpointStats, 0, sizeof(CheckpointStats)); + CheckpointStats.ckpt_start_t = GetCurrentTimestamp(); + + LogCheckpointStart(CHECKPOINT_RESTARTPOINT | flags); + } + /* ! * Acquire CheckpointLock to ensure only one restartpoint happens at a time. ! * We rely on this lock to ensure that the startup process doesn't exit ! * Recovery while we are half way through a restartpoint. */ ! LWLockAcquire(CheckpointLock, LW_EXCLUSIVE); ! ! CheckPointGuts(restartPoint->redo, CHECKPOINT_RESTARTPOINT | flags); /* ! * Update pg_control, using current time */ + LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); ControlFile->prevCheckPoint = ControlFile->checkPoint; ! ControlFile->checkPoint = ReadPtr; ! ControlFile->checkPointCopy = *restartPoint; ControlFile->time = (pg_time_t) time(NULL); UpdateControlFile(); + LWLockRelease(ControlFileLock); ! /* ! * Currently, there is no need to truncate pg_subtrans during recovery. ! * If we did do that, we will need to have called StartupSUBTRANS() ! * already and then TruncateSUBTRANS() would go here. ! */ ! ! /* All real work is done, but log before releasing lock. */ ! if (log_checkpoints) ! LogCheckpointEnd(CHECKPOINT_RESTARTPOINT); ! ! ereport((log_checkpoints ? LOG : DEBUG2), (errmsg("recovery restart point at %X/%X", ! restartPoint->redo.xlogid, restartPoint->redo.xrecoff))); ! if (recoveryLastXTime) ! ereport((log_checkpoints ? LOG : DEBUG2), ! (errmsg("last completed transaction was at log time %s", ! timestamptz_to_str(recoveryLastXTime)))); ! ! LWLockRelease(CheckpointLock); } /* *************** *** 6160,6166 **** } /* ! * XLOG resource manager's routines */ void xlog_redo(XLogRecPtr lsn, XLogRecord *record) --- 6454,6516 ---- } /* ! * exitRecovery() ! * ! * Exit recovery state and write a XLOG_RECOVERY_END record. This is the ! * only record type that can record a change of timelineID. We assume ! * caller has already set ThisTimeLineID, if appropriate. ! */ ! static void ! exitRecovery(void) ! { ! XLogRecData rdata; ! ! rdata.buffer = InvalidBuffer; ! rdata.data = (char *) (&ThisTimeLineID); ! rdata.len = sizeof(TimeLineID); ! rdata.next = NULL; ! ! /* ! * If a restartpoint is in progress, we will not be able to successfully ! * acquire CheckpointLock. If bgwriter is still in progress then send ! * a second signal to nudge bgwriter to go faster so we can avoid delay. ! * Then wait for lock, so we know the restartpoint has completed. We do ! * this because we don't want to interrupt the restartpoint half way ! * through, which might leave us in a mess and we want to be robust. We're ! * going to checkpoint soon anyway, so not it's not wasted effort. ! */ ! if (LWLockConditionalAcquire(CheckpointLock, LW_EXCLUSIVE)) ! LWLockRelease(CheckpointLock); ! else ! { ! RequestRestartPointCompletion(); ! ereport(LOG, ! (errmsg("startup process waiting for restartpoint to complete"))); ! LWLockAcquire(CheckpointLock, LW_EXCLUSIVE); ! LWLockRelease(CheckpointLock); ! } ! ! /* ! * This is the only type of WAL message that can be inserted during ! * recovery. This ensures that we don't allow others to get access ! * until after we have changed state. ! */ ! (void) XLogInsert(RM_XLOG_ID, XLOG_RECOVERY_END, &rdata); ! ! /* ! * We don't XLogFlush() here otherwise we'll end up zeroing the WAL ! * file ourselves. So just let bgwriter's forthcoming checkpoint do ! * that for us. ! */ ! ! InRecovery = false; ! } ! ! /* ! * XLOG resource manager's routines. ! * ! * Definitions of message info are in include/catalog/pg_control.h, ! * though not all messages relate to control file processing. */ void xlog_redo(XLogRecPtr lsn, XLogRecord *record) *************** *** 6195,6215 **** ControlFile->checkPointCopy.nextXid = checkPoint.nextXid; /* ! * TLI may change in a shutdown checkpoint, but it shouldn't decrease */ ! if (checkPoint.ThisTimeLineID != ThisTimeLineID) { ! if (checkPoint.ThisTimeLineID < ThisTimeLineID || !list_member_int(expectedTLIs, ! (int) checkPoint.ThisTimeLineID)) ereport(PANIC, ! (errmsg("unexpected timeline ID %u (after %u) in checkpoint record", ! checkPoint.ThisTimeLineID, ThisTimeLineID))); /* Following WAL records should be run with new TLI */ ! ThisTimeLineID = checkPoint.ThisTimeLineID; } - - RecoveryRestartPoint(&checkPoint); } else if (info == XLOG_CHECKPOINT_ONLINE) { --- 6545,6582 ---- ControlFile->checkPointCopy.nextXid = checkPoint.nextXid; /* ! * TLI no longer changes at shutdown checkpoint, since as of 8.4, ! * shutdown checkpoints only occur at shutdown. Much less confusing. */ ! ! RecoveryRestartPoint(&checkPoint); ! } ! else if (info == XLOG_RECOVERY_END) ! { ! TimeLineID tli; ! ! memcpy(&tli, XLogRecGetData(record), sizeof(TimeLineID)); ! ! /* ! * TLI may change when recovery ends, but it shouldn't decrease. ! * ! * This is the only WAL record that can tell us to change timelineID ! * while we process WAL records. ! * ! * We can *choose* to stop recovery at any point, generating a ! * new timelineID which is recorded using this record type. ! */ ! if (tli != ThisTimeLineID) { ! if (tli < ThisTimeLineID || !list_member_int(expectedTLIs, ! (int) tli)) ereport(PANIC, ! (errmsg("unexpected timeline ID %u (after %u) at recovery end record", ! tli, ThisTimeLineID))); /* Following WAL records should be run with new TLI */ ! ThisTimeLineID = tli; } } else if (info == XLOG_CHECKPOINT_ONLINE) { *************** *** 6232,6238 **** ControlFile->checkPointCopy.nextXidEpoch = checkPoint.nextXidEpoch; ControlFile->checkPointCopy.nextXid = checkPoint.nextXid; ! /* TLI should not change in an on-line checkpoint */ if (checkPoint.ThisTimeLineID != ThisTimeLineID) ereport(PANIC, (errmsg("unexpected timeline ID %u (should be %u) in checkpoint record", --- 6599,6605 ---- ControlFile->checkPointCopy.nextXidEpoch = checkPoint.nextXidEpoch; ControlFile->checkPointCopy.nextXid = checkPoint.nextXid; ! /* TLI must not change at a checkpoint */ if (checkPoint.ThisTimeLineID != ThisTimeLineID) ereport(PANIC, (errmsg("unexpected timeline ID %u (should be %u) in checkpoint record", *************** *** 6290,6296 **** } #ifdef WAL_DEBUG - static void xlog_outrec(StringInfo buf, XLogRecord *record) { --- 6657,6662 ---- *************** *** 6310,6316 **** } #endif /* WAL_DEBUG */ - /* * Return the (possible) sync flag used for opening a file, depending on the * value of the GUC wal_sync_method. --- 6676,6681 ---- *************** *** 6449,6454 **** --- 6814,6820 ---- uint32 _logSeg; struct stat stat_buf; FILE *fp; + bool immediate_checkpoint = false; if (!superuser()) ereport(ERROR, *************** *** 6502,6516 **** /* Ensure we release forcePageWrites if fail below */ PG_ENSURE_ERROR_CLEANUP(pg_start_backup_callback, (Datum) 0); { /* * Force a CHECKPOINT. Aside from being necessary to prevent torn * page problems, this guarantees that two successive backup runs will * have different checkpoint positions and hence different history * file names, even if nothing happened in between. - * - * We don't use CHECKPOINT_IMMEDIATE, hence this can take awhile. */ ! RequestCheckpoint(CHECKPOINT_FORCE | CHECKPOINT_WAIT); /* * Now we need to fetch the checkpoint record location, and also its --- 6868,6905 ---- /* Ensure we release forcePageWrites if fail below */ PG_ENSURE_ERROR_CLEANUP(pg_start_backup_callback, (Datum) 0); { + bool flags = CHECKPOINT_FORCE | CHECKPOINT_WAIT; + + /* + * We support both variants of the pg_start_backup() SQL function + * with a single C function. If we requested two parameter variant, + * then get the value for the second parameter. + */ + if (PG_NARGS() == 2) + { + immediate_checkpoint = PG_GETARG_BOOL(1); + + /* By default, this can take some time */ + if (immediate_checkpoint) + { + flags |= CHECKPOINT_IMMEDIATE; + ereport(NOTICE, + (errmsg("pg_start_backup() signalling for immediate checkpoint"))); + } + else + ereport(NOTICE, + (errmsg("pg_start_backup() signalling for smooth checkpoint" + ", may last up to %u s", + (int) (CheckPointTimeout * CheckPointCompletionTarget)))); + } + /* * Force a CHECKPOINT. Aside from being necessary to prevent torn * page problems, this guarantees that two successive backup runs will * have different checkpoint positions and hence different history * file names, even if nothing happened in between. */ ! RequestCheckpoint(flags); /* * Now we need to fetch the checkpoint record location, and also its *************** *** 6639,6651 **** LWLockRelease(WALInsertLock); /* ! * Force a switch to a new xlog segment file, so that the backup is valid * as soon as archiver moves out the current segment file. We'll report * the end address of the XLOG SWITCH record as the backup stopping point. */ stoppoint = RequestXLogSwitch(); XLByteToSeg(stoppoint, _logId, _logSeg); XLogFileName(stopxlogfilename, ThisTimeLineID, _logId, _logSeg); /* Use the log timezone here, not the session timezone */ --- 7028,7049 ---- LWLockRelease(WALInsertLock); /* ! * Request switch to a new xlog segment file, so that the backup is valid * as soon as archiver moves out the current segment file. We'll report * the end address of the XLOG SWITCH record as the backup stopping point. */ stoppoint = RequestXLogSwitch(); XLByteToSeg(stoppoint, _logId, _logSeg); + + /* + * If we didn't actually switch xlog files then there is nothing in + * this file for us to wait for, so set stopxlogfilename to be the + * previous file instead. We still report the same ending location. + */ + if ((stoppoint.xrecoff % XLogSegSize) == 0) + PrevLogSeg(_logId, _logSeg); + XLogFileName(stopxlogfilename, ThisTimeLineID, _logId, _logSeg); /* Use the log timezone here, not the session timezone */ *************** *** 6741,6747 **** BackupHistoryFileName(histfilepath, ThisTimeLineID, _logId, _logSeg, startpoint.xrecoff % XLogSegSize); ! seconds_before_warning = 60; waits = 0; while (XLogArchiveIsBusy(stopxlogfilename) || --- 7139,7145 ---- BackupHistoryFileName(histfilepath, ThisTimeLineID, _logId, _logSeg, startpoint.xrecoff % XLogSegSize); ! seconds_before_warning = 10; waits = 0; while (XLogArchiveIsBusy(stopxlogfilename) || Index: src/backend/postmaster/bgwriter.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/postmaster/bgwriter.c,v retrieving revision 1.51 diff -c -r1.51 bgwriter.c *** src/backend/postmaster/bgwriter.c 11 Aug 2008 11:05:11 -0000 1.51 --- src/backend/postmaster/bgwriter.c 30 Sep 2008 18:33:55 -0000 *************** *** 49,54 **** --- 49,55 ---- #include <unistd.h> #include "access/xlog_internal.h" + #include "catalog/pg_control.h" #include "libpq/pqsignal.h" #include "miscadmin.h" #include "pgstat.h" *************** *** 130,135 **** --- 131,143 ---- int ckpt_flags; /* checkpoint flags, as defined in xlog.h */ + /* + * When the Startup process wants bgwriter to perform a restartpoint, it + * sets these fields so that we can update the control file afterwards. + */ + XLogRecPtr ReadPtr; /* Requested log pointer */ + CheckPoint restartPoint; /* restartPoint data for ControlFile */ + uint32 num_backend_writes; /* counts non-bgwriter buffer writes */ int num_requests; /* current # of requests */ *************** *** 166,172 **** /* these values are valid when ckpt_active is true: */ static pg_time_t ckpt_start_time; ! static XLogRecPtr ckpt_start_recptr; static double ckpt_cached_elapsed; static pg_time_t last_checkpoint_time; --- 174,180 ---- /* these values are valid when ckpt_active is true: */ static pg_time_t ckpt_start_time; ! static XLogRecPtr ckpt_start_recptr; /* not used if IsRecoveryProcessingMode */ static double ckpt_cached_elapsed; static pg_time_t last_checkpoint_time; *************** *** 198,203 **** --- 206,212 ---- { sigjmp_buf local_sigjmp_buf; MemoryContext bgwriter_context; + bool BgWriterRecoveryMode; BgWriterShmem->bgwriter_pid = MyProcPid; am_bg_writer = true; *************** *** 356,371 **** */ PG_SETMASK(&UnBlockSig); /* * Loop forever */ for (;;) { - bool do_checkpoint = false; - int flags = 0; - pg_time_t now; - int elapsed_secs; - /* * Emergency bailout if postmaster has died. This is to avoid the * necessity for manual cleanup of all postmaster children. --- 365,381 ---- */ PG_SETMASK(&UnBlockSig); + BgWriterRecoveryMode = IsRecoveryProcessingMode(); + + if (BgWriterRecoveryMode) + elog(DEBUG1, "bgwriter starting during recovery, pid = %u", + BgWriterShmem->bgwriter_pid); + /* * Loop forever */ for (;;) { /* * Emergency bailout if postmaster has died. This is to avoid the * necessity for manual cleanup of all postmaster children. *************** *** 383,501 **** got_SIGHUP = false; ProcessConfigFile(PGC_SIGHUP); } - if (checkpoint_requested) - { - checkpoint_requested = false; - do_checkpoint = true; - BgWriterStats.m_requested_checkpoints++; - } - if (shutdown_requested) - { - /* - * From here on, elog(ERROR) should end with exit(1), not send - * control back to the sigsetjmp block above - */ - ExitOnAnyError = true; - /* Close down the database */ - ShutdownXLOG(0, 0); - DumpFreeSpaceMap(0, 0); - /* Normal exit from the bgwriter is here */ - proc_exit(0); /* done */ - } ! /* ! * Force a checkpoint if too much time has elapsed since the last one. ! * Note that we count a timed checkpoint in stats only when this ! * occurs without an external request, but we set the CAUSE_TIME flag ! * bit even if there is also an external request. ! */ ! now = (pg_time_t) time(NULL); ! elapsed_secs = now - last_checkpoint_time; ! if (elapsed_secs >= CheckPointTimeout) { ! if (!do_checkpoint) ! BgWriterStats.m_timed_checkpoints++; ! do_checkpoint = true; ! flags |= CHECKPOINT_CAUSE_TIME; } ! ! /* ! * Do a checkpoint if requested, otherwise do one cycle of ! * dirty-buffer writing. ! */ ! if (do_checkpoint) { ! /* use volatile pointer to prevent code rearrangement */ ! volatile BgWriterShmemStruct *bgs = BgWriterShmem; /* ! * Atomically fetch the request flags to figure out what kind of a ! * checkpoint we should perform, and increase the started-counter ! * to acknowledge that we've started a new checkpoint. */ ! SpinLockAcquire(&bgs->ckpt_lck); ! flags |= bgs->ckpt_flags; ! bgs->ckpt_flags = 0; ! bgs->ckpt_started++; ! SpinLockRelease(&bgs->ckpt_lck); /* ! * We will warn if (a) too soon since last checkpoint (whatever ! * caused it) and (b) somebody set the CHECKPOINT_CAUSE_XLOG flag ! * since the last checkpoint start. Note in particular that this ! * implementation will not generate warnings caused by ! * CheckPointTimeout < CheckPointWarning. */ ! if ((flags & CHECKPOINT_CAUSE_XLOG) && ! elapsed_secs < CheckPointWarning) ! ereport(LOG, ! (errmsg("checkpoints are occurring too frequently (%d seconds apart)", ! elapsed_secs), ! errhint("Consider increasing the configuration parameter \"checkpoint_segments\"."))); ! /* ! * Initialize bgwriter-private variables used during checkpoint. ! */ ! ckpt_active = true; ! ckpt_start_recptr = GetInsertRecPtr(); ! ckpt_start_time = now; ! ckpt_cached_elapsed = 0; ! /* ! * Do the checkpoint. ! */ ! CreateCheckPoint(flags); ! ! /* ! * After any checkpoint, close all smgr files. This is so we ! * won't hang onto smgr references to deleted files indefinitely. ! */ ! smgrcloseall(); ! ! /* ! * Indicate checkpoint completion to any waiting backends. ! */ ! SpinLockAcquire(&bgs->ckpt_lck); ! bgs->ckpt_done = bgs->ckpt_started; ! SpinLockRelease(&bgs->ckpt_lck); ! ! ckpt_active = false; ! ! /* ! * Note we record the checkpoint start time not end time as ! * last_checkpoint_time. This is so that time-driven checkpoints ! * happen at a predictable spacing. ! */ ! last_checkpoint_time = now; } - else - BgBufferSync(); - - /* Check for archive_timeout and switch xlog files if necessary. */ - CheckArchiveTimeout(); - - /* Nap for the configured time. */ - BgWriterNap(); } } --- 393,599 ---- got_SIGHUP = false; ProcessConfigFile(PGC_SIGHUP); } ! if (BgWriterRecoveryMode) { ! if (shutdown_requested) ! { ! /* ! * From here on, elog(ERROR) should end with exit(1), not send ! * control back to the sigsetjmp block above ! */ ! ExitOnAnyError = true; ! /* Normal exit from the bgwriter is here */ ! proc_exit(0); /* done */ ! } ! ! if (!IsRecoveryProcessingMode()) ! { ! elog(DEBUG2, "bgwriter changing from recovery to normal mode"); ! ! InitXLOGAccess(); ! BgWriterRecoveryMode = false; ! ! /* ! * Start time-driven events from now ! */ ! last_checkpoint_time = last_xlog_switch_time = (pg_time_t) time(NULL); ! ! /* ! * Notice that we do *not* act on a checkpoint_requested ! * state at this point. We have changed mode, so we wish to ! * perform a checkpoint not a restartpoint. ! */ ! continue; ! } ! ! if (checkpoint_requested) ! { ! XLogRecPtr ReadPtr; ! CheckPoint restartPoint; ! ! checkpoint_requested = false; ! ! /* ! * Initialize bgwriter-private variables used during checkpoint. ! */ ! ckpt_active = true; ! ckpt_start_time = (pg_time_t) time(NULL); ! ckpt_cached_elapsed = 0; ! ! /* ! * Get the requested values from shared memory that the ! * Startup process has put there for us. ! */ ! SpinLockAcquire(&BgWriterShmem->ckpt_lck); ! ReadPtr = BgWriterShmem->ReadPtr; ! memcpy(&restartPoint, &BgWriterShmem->restartPoint, sizeof(CheckPoint)); ! SpinLockRelease(&BgWriterShmem->ckpt_lck); ! ! /* Use smoothed writes, until interrupted if ever */ ! CreateRestartPoint(ReadPtr, &restartPoint, 0); ! ! /* ! * After any checkpoint, close all smgr files. This is so we ! * won't hang onto smgr references to deleted files indefinitely. ! */ ! smgrcloseall(); ! ! ckpt_active = false; ! checkpoint_requested = false; ! } ! else ! { ! /* Clean buffers dirtied by recovery */ ! BgBufferSync(); ! ! /* Nap for the configured time. */ ! BgWriterNap(); ! } } ! else /* Normal processing */ { ! bool do_checkpoint = false; ! int flags = 0; ! pg_time_t now; ! int elapsed_secs; ! ! Assert(!IsRecoveryProcessingMode()); ! ! if (checkpoint_requested) ! { ! checkpoint_requested = false; ! do_checkpoint = true; ! BgWriterStats.m_requested_checkpoints++; ! } ! if (shutdown_requested) ! { ! /* ! * From here on, elog(ERROR) should end with exit(1), not send ! * control back to the sigsetjmp block above ! */ ! ExitOnAnyError = true; ! /* Close down the database */ ! ShutdownXLOG(0, 0); ! DumpFreeSpaceMap(0, 0); ! /* Normal exit from the bgwriter is here */ ! proc_exit(0); /* done */ ! } /* ! * Force a checkpoint if too much time has elapsed since the last one. ! * Note that we count a timed checkpoint in stats only when this ! * occurs without an external request, but we set the CAUSE_TIME flag ! * bit even if there is also an external request. */ ! now = (pg_time_t) time(NULL); ! elapsed_secs = now - last_checkpoint_time; ! if (elapsed_secs >= CheckPointTimeout) ! { ! if (!do_checkpoint) ! BgWriterStats.m_timed_checkpoints++; ! do_checkpoint = true; ! flags |= CHECKPOINT_CAUSE_TIME; ! } /* ! * Do a checkpoint if requested, otherwise do one cycle of ! * dirty-buffer writing. */ ! if (do_checkpoint) ! { ! /* use volatile pointer to prevent code rearrangement */ ! volatile BgWriterShmemStruct *bgs = BgWriterShmem; ! ! /* ! * Atomically fetch the request flags to figure out what kind of a ! * checkpoint we should perform, and increase the started-counter ! * to acknowledge that we've started a new checkpoint. ! */ ! SpinLockAcquire(&bgs->ckpt_lck); ! flags |= bgs->ckpt_flags; ! bgs->ckpt_flags = 0; ! bgs->ckpt_started++; ! SpinLockRelease(&bgs->ckpt_lck); ! ! /* ! * We will warn if (a) too soon since last checkpoint (whatever ! * caused it) and (b) somebody set the CHECKPOINT_CAUSE_XLOG flag ! * since the last checkpoint start. Note in particular that this ! * implementation will not generate warnings caused by ! * CheckPointTimeout < CheckPointWarning. ! */ ! if ((flags & CHECKPOINT_CAUSE_XLOG) && ! elapsed_secs < CheckPointWarning) ! ereport(LOG, ! (errmsg("checkpoints are occurring too frequently (%d seconds apart)", ! elapsed_secs), ! errhint("Consider increasing the configuration parameter \"checkpoint_segments\"."))); ! ! /* ! * Initialize bgwriter-private variables used during checkpoint. ! */ ! ckpt_active = true; ! ckpt_start_recptr = GetInsertRecPtr(); ! ckpt_start_time = now; ! ckpt_cached_elapsed = 0; ! ! /* ! * Do the checkpoint. ! */ ! CreateCheckPoint(flags); ! ! /* ! * After any checkpoint, close all smgr files. This is so we ! * won't hang onto smgr references to deleted files indefinitely. ! */ ! smgrcloseall(); ! ! /* ! * Indicate checkpoint completion to any waiting backends. ! */ ! SpinLockAcquire(&bgs->ckpt_lck); ! bgs->ckpt_done = bgs->ckpt_started; ! SpinLockRelease(&bgs->ckpt_lck); ! ! ckpt_active = false; ! ! /* ! * Note we record the checkpoint start time not end time as ! * last_checkpoint_time. This is so that time-driven checkpoints ! * happen at a predictable spacing. ! */ ! last_checkpoint_time = now; ! } ! else ! BgBufferSync(); ! /* Check for archive_timeout and switch xlog files if necessary. */ ! CheckArchiveTimeout(); ! /* Nap for the configured time. */ ! BgWriterNap(); } } } *************** *** 588,594 **** (ckpt_active ? ImmediateCheckpointRequested() : checkpoint_requested)) break; pg_usleep(1000000L); ! AbsorbFsyncRequests(); udelay -= 1000000L; } --- 686,693 ---- (ckpt_active ? ImmediateCheckpointRequested() : checkpoint_requested)) break; pg_usleep(1000000L); ! if (!IsRecoveryProcessingMode()) ! AbsorbFsyncRequests(); udelay -= 1000000L; } *************** *** 642,647 **** --- 741,759 ---- if (!am_bg_writer) return; + /* Perform minimal duties during recovery and skip wait if requested */ + if (IsRecoveryProcessingMode()) + { + BgBufferSync(); + + if (!shutdown_requested && + !checkpoint_requested && + IsCheckpointOnSchedule(progress)) + BgWriterNap(); + + return; + } + /* * Perform the usual bgwriter duties and take a nap, unless we're behind * schedule, in which case we just try to catch up as quickly as possible. *************** *** 716,731 **** * However, it's good enough for our purposes, we're only calculating an * estimate anyway. */ ! recptr = GetInsertRecPtr(); ! elapsed_xlogs = ! (((double) (int32) (recptr.xlogid - ckpt_start_recptr.xlogid)) * XLogSegsPerFile + ! ((double) recptr.xrecoff - (double) ckpt_start_recptr.xrecoff) / XLogSegSize) / ! CheckPointSegments; ! ! if (progress < elapsed_xlogs) { ! ckpt_cached_elapsed = elapsed_xlogs; ! return false; } /* --- 828,846 ---- * However, it's good enough for our purposes, we're only calculating an * estimate anyway. */ ! if (!IsRecoveryProcessingMode()) { ! recptr = GetInsertRecPtr(); ! elapsed_xlogs = ! (((double) (int32) (recptr.xlogid - ckpt_start_recptr.xlogid)) * XLogSegsPerFile + ! ((double) recptr.xrecoff - (double) ckpt_start_recptr.xrecoff) / XLogSegSize) / ! CheckPointSegments; ! ! if (progress < elapsed_xlogs) ! { ! ckpt_cached_elapsed = elapsed_xlogs; ! return false; ! } } /* *************** *** 967,972 **** --- 1082,1158 ---- } /* + * Always runs in Startup process (see xlog.c) + */ + void + RequestRestartPoint(const XLogRecPtr ReadPtr, const CheckPoint *restartPoint, bool sendToBGWriter) + { + /* + * Should we just do it ourselves? + */ + if (!IsPostmasterEnvironment || !sendToBGWriter) + { + CreateRestartPoint(ReadPtr, restartPoint, CHECKPOINT_IMMEDIATE); + return; + } + + /* + * Push requested values into shared memory, then signal to request restartpoint. + */ + if (BgWriterShmem->bgwriter_pid == 0) + elog(LOG, "could not request restartpoint because bgwriter not running"); + + #ifdef NOT_USED + elog(LOG, "tli = %u nextXidEpoch = %u nextXid = %u nextOid = %u", + restartPoint->ThisTimeLineID, + restartPoint->nextXidEpoch, + restartPoint->nextXid, + restartPoint->nextOid); + #endif + + SpinLockAcquire(&BgWriterShmem->ckpt_lck); + BgWriterShmem->ReadPtr = ReadPtr; + memcpy(&BgWriterShmem->restartPoint, restartPoint, sizeof(CheckPoint)); + SpinLockRelease(&BgWriterShmem->ckpt_lck); + + if (kill(BgWriterShmem->bgwriter_pid, SIGINT) != 0) + elog(LOG, "could not signal for restartpoint: %m"); + } + + /* + * Sends another checkpoint request signal to bgwriter, which causes it + * to avoid smoothed writes and continue processing as if it had been + * called with CHECKPOINT_IMMEDIATE. This is used at the end of recovery. + */ + void + RequestRestartPointCompletion(void) + { + if (BgWriterShmem->bgwriter_pid != 0 && + kill(BgWriterShmem->bgwriter_pid, SIGINT) != 0) + elog(LOG, "could not signal for restartpoint immediate: %m"); + } + + XLogRecPtr + GetRedoLocationForArchiveCheckpoint(void) + { + XLogRecPtr redo; + + SpinLockAcquire(&BgWriterShmem->ckpt_lck); + redo = BgWriterShmem->ReadPtr; + SpinLockRelease(&BgWriterShmem->ckpt_lck); + + return redo; + } + + void + SetRedoLocationForArchiveCheckpoint(XLogRecPtr redo) + { + SpinLockAcquire(&BgWriterShmem->ckpt_lck); + BgWriterShmem->ReadPtr = redo; + SpinLockRelease(&BgWriterShmem->ckpt_lck); + } + + /* * ForwardFsyncRequest * Forward a file-fsync request from a backend to the bgwriter * Index: src/backend/postmaster/postmaster.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/postmaster/postmaster.c,v retrieving revision 1.565 diff -c -r1.565 postmaster.c *** src/backend/postmaster/postmaster.c 23 Sep 2008 20:35:38 -0000 1.565 --- src/backend/postmaster/postmaster.c 30 Sep 2008 17:15:15 -0000 *************** *** 254,259 **** --- 254,264 ---- { PM_INIT, /* postmaster starting */ PM_STARTUP, /* waiting for startup subprocess */ + PM_RECOVERY, /* consistent recovery mode; state only + * entered for archive and streaming recovery, + * and only after the point where the + * all data is in consistent state. + */ PM_RUN, /* normal "database is alive" state */ PM_WAIT_BACKUP, /* waiting for online backup mode to end */ PM_WAIT_BACKENDS, /* waiting for live backends to exit */ *************** *** 1302,1308 **** * state that prevents it, start one. It doesn't matter if this * fails, we'll just try again later. */ ! if (BgWriterPID == 0 && pmState == PM_RUN) BgWriterPID = StartBackgroundWriter(); /* --- 1307,1313 ---- * state that prevents it, start one. It doesn't matter if this * fails, we'll just try again later. */ ! if (BgWriterPID == 0 && (pmState == PM_RUN || pmState == PM_RECOVERY)) BgWriterPID = StartBackgroundWriter(); /* *************** *** 2116,2122 **** if (pid == StartupPID) { StartupPID = 0; ! Assert(pmState == PM_STARTUP); /* FATAL exit of startup is treated as catastrophic */ if (!EXIT_STATUS_0(exitstatus)) --- 2121,2127 ---- if (pid == StartupPID) { StartupPID = 0; ! Assert(pmState == PM_STARTUP || pmState == PM_RECOVERY); /* FATAL exit of startup is treated as catastrophic */ if (!EXIT_STATUS_0(exitstatus)) *************** *** 2157,2167 **** load_role(); /* ! * Crank up the background writer. It doesn't matter if this ! * fails, we'll just try again later. */ ! Assert(BgWriterPID == 0); ! BgWriterPID = StartBackgroundWriter(); /* * Likewise, start other special children as needed. In a restart --- 2162,2172 ---- load_role(); /* ! * Check whether we need to start background writer, if not ! * already running. */ ! if (BgWriterPID == 0) ! BgWriterPID = StartBackgroundWriter(); /* * Likewise, start other special children as needed. In a restart *************** *** 3845,3850 **** --- 3850,3900 ---- PG_SETMASK(&BlockSig); + if (CheckPostmasterSignal(PMSIGNAL_RECOVERY_START)) + { + Assert(pmState == PM_STARTUP); + + /* + * Go to shutdown mode if a shutdown request was pending. + */ + if (Shutdown > NoShutdown) + { + pmState = PM_WAIT_BACKENDS; + /* PostmasterStateMachine logic does the rest */ + } + else + { + /* + * Startup process has entered recovery + */ + pmState = PM_RECOVERY; + + /* + * Load the flat authorization file into postmaster's cache. The + * startup process won't have recomputed this from the database yet, + * so we it may change following recovery. + */ + load_role(); + + /* + * Crank up the background writer. It doesn't matter if this + * fails, we'll just try again later. + */ + Assert(BgWriterPID == 0); + BgWriterPID = StartBackgroundWriter(); + + /* + * Likewise, start other special children as needed. + */ + Assert(PgStatPID == 0); + PgStatPID = pgstat_start(); + + /* XXX at this point we could accept read-only connections */ + ereport(DEBUG1, + (errmsg("database system is in consistent recovery mode"))); + } + } + if (CheckPostmasterSignal(PMSIGNAL_PASSWORD_CHANGE)) { /* Index: src/backend/storage/buffer/README =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/storage/buffer/README,v retrieving revision 1.14 diff -c -r1.14 README *** src/backend/storage/buffer/README 21 Mar 2008 13:23:28 -0000 1.14 --- src/backend/storage/buffer/README 30 Sep 2008 17:15:15 -0000 *************** *** 264,266 **** --- 264,275 ---- This ensures that the page image transferred to disk is reasonably consistent. We might miss a hint-bit update or two but that isn't a problem, for the same reasons mentioned under buffer access rules. + + As of 8.4, background writer starts during recovery mode when there is + some form of potentially extended recovery to perform. It performs an + identical service to normal processing, except that checkpoints it + writes are technically restartpoints. Flushing outstanding WAL for dirty + buffers is also skipped, though there shouldn't ever be new WAL entries + at that time in any case. We could choose to start background writer + immediately but we hold off until we can prove the database is in a + consistent state so that postmaster has a single, clean state change. Index: src/bin/pg_controldata/pg_controldata.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_controldata/pg_controldata.c,v retrieving revision 1.41 diff -c -r1.41 pg_controldata.c *** src/bin/pg_controldata/pg_controldata.c 24 Sep 2008 08:59:42 -0000 1.41 --- src/bin/pg_controldata/pg_controldata.c 30 Sep 2008 17:15:15 -0000 *************** *** 197,202 **** --- 197,205 ---- printf(_("Minimum recovery ending location: %X/%X\n"), ControlFile.minRecoveryPoint.xlogid, ControlFile.minRecoveryPoint.xrecoff); + printf(_("Minimum safe starting location: %X/%X\n"), + ControlFile.minSafeStartPoint.xlogid, + ControlFile.minSafeStartPoint.xrecoff); printf(_("Maximum data alignment: %u\n"), ControlFile.maxAlign); /* we don't print floatFormat since can't say much useful about it */ Index: src/bin/pg_resetxlog/pg_resetxlog.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_resetxlog/pg_resetxlog.c,v retrieving revision 1.68 diff -c -r1.68 pg_resetxlog.c *** src/bin/pg_resetxlog/pg_resetxlog.c 24 Sep 2008 09:00:44 -0000 1.68 --- src/bin/pg_resetxlog/pg_resetxlog.c 30 Sep 2008 17:15:15 -0000 *************** *** 595,600 **** --- 595,602 ---- ControlFile.prevCheckPoint.xrecoff = 0; ControlFile.minRecoveryPoint.xlogid = 0; ControlFile.minRecoveryPoint.xrecoff = 0; + ControlFile.minSafeStartPoint.xlogid = 0; + ControlFile.minSafeStartPoint.xrecoff = 0; /* Now we can force the recorded xlog seg size to the right thing. */ ControlFile.xlog_seg_size = XLogSegSize; Index: src/include/access/xlog.h =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/include/access/xlog.h,v retrieving revision 1.88 diff -c -r1.88 xlog.h *** src/include/access/xlog.h 12 May 2008 08:35:05 -0000 1.88 --- src/include/access/xlog.h 30 Sep 2008 17:15:15 -0000 *************** *** 133,139 **** } XLogRecData; extern TimeLineID ThisTimeLineID; /* current TLI */ ! extern bool InRecovery; extern XLogRecPtr XactLastRecEnd; /* these variables are GUC parameters related to XLOG */ --- 133,148 ---- } XLogRecData; extern TimeLineID ThisTimeLineID; /* current TLI */ ! ! /* ! * Prior to 8.4, all activity during recovery were carried out by Startup ! * process. This local variable continues to be used in many parts of the ! * code to indicate actions taken by RecoveryManagers. Other processes who ! * potentially perform work during recovery should check ! * IsRecoveryProcessingMode(), see XLogCtl notes in xlog.c ! */ ! extern bool InRecovery; ! extern XLogRecPtr XactLastRecEnd; /* these variables are GUC parameters related to XLOG */ *************** *** 166,171 **** --- 175,181 ---- /* These indicate the cause of a checkpoint request */ #define CHECKPOINT_CAUSE_XLOG 0x0010 /* XLOG consumption */ #define CHECKPOINT_CAUSE_TIME 0x0020 /* Elapsed time */ + #define CHECKPOINT_RESTARTPOINT 0x0040 /* Restartpoint during recovery */ /* Checkpoint statistics */ typedef struct CheckpointStatsData *************** *** 197,202 **** --- 207,214 ---- extern void xlog_redo(XLogRecPtr lsn, XLogRecord *record); extern void xlog_desc(StringInfo buf, uint8 xl_info, char *rec); + extern bool IsRecoveryProcessingMode(void); + extern void UpdateControlFile(void); extern Size XLOGShmemSize(void); extern void XLOGShmemInit(void); Index: src/include/access/xlog_internal.h =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/include/access/xlog_internal.h,v retrieving revision 1.24 diff -c -r1.24 xlog_internal.h *** src/include/access/xlog_internal.h 11 Aug 2008 11:05:11 -0000 1.24 --- src/include/access/xlog_internal.h 30 Sep 2008 17:15:15 -0000 *************** *** 17,22 **** --- 17,23 ---- #define XLOG_INTERNAL_H #include "access/xlog.h" + #include "catalog/pg_control.h" #include "fmgr.h" #include "pgtime.h" #include "storage/block.h" *************** *** 245,250 **** --- 246,254 ---- extern pg_time_t GetLastSegSwitchTime(void); extern XLogRecPtr RequestXLogSwitch(void); + extern void CreateRestartPoint(const XLogRecPtr ReadPtr, + const CheckPoint *restartPoint, int flags); + /* * These aren't in xlog.h because I'd rather not include fmgr.h there. */ Index: src/include/catalog/pg_control.h =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/include/catalog/pg_control.h,v retrieving revision 1.42 diff -c -r1.42 pg_control.h *** src/include/catalog/pg_control.h 23 Sep 2008 09:20:39 -0000 1.42 --- src/include/catalog/pg_control.h 30 Sep 2008 17:15:15 -0000 *************** *** 46,52 **** #define XLOG_NOOP 0x20 #define XLOG_NEXTOID 0x30 #define XLOG_SWITCH 0x40 ! /* System status indicator */ typedef enum DBState --- 46,52 ---- #define XLOG_NOOP 0x20 #define XLOG_NEXTOID 0x30 #define XLOG_SWITCH 0x40 ! #define XLOG_RECOVERY_END 0x50 /* System status indicator */ typedef enum DBState *************** *** 102,107 **** --- 102,108 ---- CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr minRecoveryPoint; /* must replay xlog to here */ + XLogRecPtr minSafeStartPoint; /* safe point after recovery crashes */ /* * This data is used to check for hardware-architecture compatibility of Index: src/include/postmaster/bgwriter.h =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/include/postmaster/bgwriter.h,v retrieving revision 1.12 diff -c -r1.12 bgwriter.h *** src/include/postmaster/bgwriter.h 11 Aug 2008 11:05:11 -0000 1.12 --- src/include/postmaster/bgwriter.h 30 Sep 2008 17:15:15 -0000 *************** *** 12,17 **** --- 12,18 ---- #ifndef _BGWRITER_H #define _BGWRITER_H + #include "catalog/pg_control.h" #include "storage/block.h" #include "storage/relfilenode.h" *************** *** 25,30 **** --- 26,36 ---- extern void BackgroundWriterMain(void); extern void RequestCheckpoint(int flags); + extern void RequestRestartPoint(const XLogRecPtr ReadPtr, const CheckPoint *restartPoint, bool sendToBGWriter); + extern void RequestRestartPointCompletion(void); + extern XLogRecPtr GetRedoLocationForArchiveCheckpoint(void); + extern void SetRedoLocationForArchiveCheckpoint(XLogRecPtr redo); + extern void CheckpointWriteDelay(int flags, double progress); extern bool ForwardFsyncRequest(RelFileNode rnode, ForkNumber forknum, Index: src/include/storage/pmsignal.h =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/include/storage/pmsignal.h,v retrieving revision 1.20 diff -c -r1.20 pmsignal.h *** src/include/storage/pmsignal.h 19 Jun 2008 21:32:56 -0000 1.20 --- src/include/storage/pmsignal.h 30 Sep 2008 17:15:15 -0000 *************** *** 22,27 **** --- 22,28 ---- */ typedef enum { + PMSIGNAL_RECOVERY_START, /* move to PM_RECOVERY state */ PMSIGNAL_PASSWORD_CHANGE, /* pg_auth file has changed */ PMSIGNAL_WAKEN_ARCHIVER, /* send a NOTIFY signal to xlog archiver */ PMSIGNAL_ROTATE_LOGFILE, /* send SIGUSR1 to syslogger to rotate logfile */ Index: src/test/regress/expected/opr_sanity.out =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/test/regress/expected/opr_sanity.out,v retrieving revision 1.84 diff -c -r1.84 opr_sanity.out *** src/test/regress/expected/opr_sanity.out 16 Aug 2008 00:01:38 -0000 1.84 --- src/test/regress/expected/opr_sanity.out 30 Sep 2008 17:15:15 -0000 *************** *** 109,117 **** p1.proretset != p2.proretset OR p1.provolatile != p2.provolatile OR p1.pronargs != p2.pronargs); ! oid | proname | oid | proname ! -----+---------+-----+--------- ! (0 rows) -- Look for uses of different type OIDs in the argument/result type fields -- for different aliases of the same built-in function. --- 109,118 ---- p1.proretset != p2.proretset OR p1.provolatile != p2.provolatile OR p1.pronargs != p2.pronargs); ! oid | proname | oid | proname ! ------+-----------------+------+----------------- ! 2172 | pg_start_backup | 2176 | pg_start_backup ! (1 row) -- Look for uses of different type OIDs in the argument/result type fields -- for different aliases of the same built-in function.

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: still alive?

2008-09-30T00:02:35Z

On Thu, 2008-09-11 at 15:39 +0300, Peter Eisentraut wrote:

> Bruce Momjian wrote:
> > Abhijit Menon-Sen wrote:
> >> I thought -patches was supposed to die. What happened?
> >
> > I was wondering the same thing. Peter?
>
> Hmm, let's try this:
>
> Anyone who thinks the patches list should remain as separate from
> hackers, shout now (with rationale)!

Kill it now, long enough before the next patchfest for it to stick.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: Proposed patch to change TOAST compression strategy for 8.3.4

2008-09-30T00:00:48Z

On Mon, 2008-09-29 at 12:32 +0200, Jérôme Jouanin wrote:

> Upgrade 8.3.4 is available. Before compiling, I have to apply the
> optimized toast patch : bin7hetTGkMRL.bin.
>
> There are differences in 1 of the 3 files patched : tuptoaster.c
>
> The patch runs successfully but before installing on production
> servers, I have to ask : what about the compatibility of this patch in
> 8.3.4 ?

We use patches to build the next releases of PostgreSQL. We don't make
any attempt to maintain past patches as current so that people can apply
them to previous branches.

If you want someone to do that you will probably need to hire a
development and support company, or do it yourself.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-29T08:53:55Z

On Mon, 2008-09-29 at 11:24 -0400, Tom Lane wrote:

> Simon Riggs <simon@...> writes:
> > On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote:
> >> ... If we crash and restart, we'll have to get to the end
> >> of this file before we start letting backends in; which might be further
> >> than we actually got before the crash, but not too much further because
> >> we already know the whole WAL file is available.
>
> > Don't want to make it per file though. Big systems can whizz through WAL
> > files very quickly, so we either make it a big number e.g. 255 files per
> > xlogid, or we make it settable (and recorded in pg_control).
>
> I think you are missing the point I made above. If you set the
> okay-to-resume point N files ahead, and then the master stops generating
> files so quickly, you've got a problem --- it might be a long time until
> the slave starts letting backends in after a crash/restart.
>
> Fetching a new WAL segment from the archive is expensive enough that an
> additional write/fsync per cycle doesn't seem that big a problem to me.
> There's almost certainly a few fsync-equivalents going on in the
> filesystem to create and delete the retrieved segment files.

Didn't miss yer point, just didn't agree. :-)

I'll put it at one (1) and then wait for any negative perf reports. No
need to worry about things like that until later.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-29T08:24:08Z

Simon Riggs <simon@...> writes:
> On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote:
>> ... If we crash and restart, we'll have to get to the end
>> of this file before we start letting backends in; which might be further
>> than we actually got before the crash, but not too much further because
>> we already know the whole WAL file is available.

> Don't want to make it per file though. Big systems can whizz through WAL
> files very quickly, so we either make it a big number e.g. 255 files per
> xlogid, or we make it settable (and recorded in pg_control).

I think you are missing the point I made above. If you set the
okay-to-resume point N files ahead, and then the master stops generating
files so quickly, you've got a problem --- it might be a long time until
the slave starts letting backends in after a crash/restart.

Fetching a new WAL segment from the archive is expensive enough that an
additional write/fsync per cycle doesn't seem that big a problem to me.
There's almost certainly a few fsync-equivalents going on in the
filesystem to create and delete the retrieved segment files.

regards, tom lane

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-29T08:05:49Z

On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote:

> Simon Riggs <simon@...> writes:
> > I think we can get away with writing the LSN value to disk, as you
> > suggested, but only every so often. No need to do it after every WAL
> > record, just consistently every so often, so it gives us a point at
> > which we know we are safe.
>
> Huh? How does that make you safe? What you need to know is the max
> LSN that could possibly be on disk.
>
> Hmm, actually we could get away with tying this to fetching WAL files
> from the archive. When switching to a new WAL file, write out the
> *ending* WAL address of that file to pg_control. Then process the WAL
> records in it. Whether or not any of the affected pages get to disk,
> we know that there is no LSN on disk exceeding what we already put in
> pg_control. If we crash and restart, we'll have to get to the end
> of this file before we start letting backends in; which might be further
> than we actually got before the crash, but not too much further because
> we already know the whole WAL file is available.
>
> Or is that the same thing you were saying? The detail about using
> the end address seems fairly critical, and you didn't mention it...

Same! Just said safe point was "LSN + 1", and since end = next start.

Looks we've got a solution, no matter how it's described. (I actually
have a more detailed proof of safety using snapshots/MVCC considerations
so I wasn't overly worried but what we've discussed is much easier to
understand and agree. Proof of safety is all we need, and this simpler
proof is more secure.)

Don't want to make it per file though. Big systems can whizz through WAL
files very quickly, so we either make it a big number e.g. 255 files per
xlogid, or we make it settable (and recorded in pg_control).

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-29T07:13:59Z

Simon Riggs <simon@...> writes:
> I think we can get away with writing the LSN value to disk, as you
> suggested, but only every so often. No need to do it after every WAL
> record, just consistently every so often, so it gives us a point at
> which we know we are safe.

Huh? How does that make you safe? What you need to know is the max
LSN that could possibly be on disk.

Hmm, actually we could get away with tying this to fetching WAL files
from the archive. When switching to a new WAL file, write out the
*ending* WAL address of that file to pg_control. Then process the WAL
records in it. Whether or not any of the affected pages get to disk,
we know that there is no LSN on disk exceeding what we already put in
pg_control. If we crash and restart, we'll have to get to the end
of this file before we start letting backends in; which might be further
than we actually got before the crash, but not too much further because
we already know the whole WAL file is available.

Or is that the same thing you were saying? The detail about using
the end address seems fairly critical, and you didn't mention it...

regards, tom lane

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-29T06:52:28Z

On Mon, 2008-09-29 at 08:46 -0400, Tom Lane wrote:
> Simon Riggs <simon@...> writes:
> > ... That kinda works, but the problem is that restartpoints are time based,
> > not log based. We need them to be deterministic for us to rely upon them
> > in the above way.
>
> Right, but the performance disadvantages of making them strictly
> log-distance-based are pretty daunting. We don't really want slaves
> doing that while they're in catchup mode.

I don't think we need to perform restartpoints actually, now I think
about it. It's only the LSN that is important.

I think we can get away with writing the LSN value to disk, as you
suggested, but only every so often. No need to do it after every WAL
record, just consistently every so often, so it gives us a point at
which we know we are safe. We will need to have Startup process block
momentarily while the value is written.

Propose Startup process writes/flushes LSN to pg_control every time we
change xlogid. That's independent of WAL file size and fairly clear.

When we reach that LSN + 1 we will know that no LSNs higher than that
value can have reached disk.

OK?

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-29T05:46:46Z

Simon Riggs <simon@...> writes:
> ... That kinda works, but the problem is that restartpoints are time based,
> not log based. We need them to be deterministic for us to rely upon them
> in the above way.

Right, but the performance disadvantages of making them strictly
log-distance-based are pretty daunting. We don't really want slaves
doing that while they're in catchup mode.

regards, tom lane

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-29T05:32:17Z

On Sun, 2008-09-28 at 21:16 -0400, Tom Lane wrote:

> Simon Riggs <simon@...> writes:
> >> It does nothing AFAICS for the
> >> problem that when restarting archive recovery from a restartpoint,
> >> it's not clear when it is safe to start letting in backends. You need
> >> to get past the highest LSN that has made it out to disk, and there is
> >> no good way to know what that is.
>
> > AFAICS when we set minRecoveryLoc we *never* unset it. It's recorded in
> > the controlfile, so whenever we restart we can see that it has been set
> > previously and now we are beyond it.
>
> Right ...
>
> > So if we crash during recovery and
> > then restart *after* we reached minRecoveryLoc then we resume in safe
> > mode almost immediately.
>
> Wrong.

OK, see where you're coming from now. Solution is needed, I agree.

> What minRecoveryLoc is is an upper bound for the LSNs that might be
> on-disk in the filesystem backup that an archive recovery starts from.
> (Defined as such, it never changes during a restartpoint crash/restart.)
> Once you pass that, the on-disk state as modified by any dirty buffers
> inside the recovery process represents a consistent database state.
> However, the on-disk state alone is not guaranteed consistent. As you
> flush some (not all) of your shared buffers you enter other
> not-certainly-consistent on-disk states. If we crash in such a state,
> we know how to use the last restartpoint plus WAL replay to recover to
> another state in which disk + dirty buffers are consistent. However,
> we reach such a state only when we have read WAL to beyond the highest
> LSN that has reached disk --- and in recovery mode there is no clean
> way to determine what that was.
>
> Perhaps a solution is to make XLogFLush not be a no-op in recovery mode,
> but have it scribble a highest-LSN somewhere on stable storage (maybe
> scribble on pg_control itself, or maybe better someplace else). I'm
> not totally sure about that. But I am sure that doing nothing will
> be unreliable.

No need to write highest LSN to disk constantly...

If we restart from a restartpoint then initially the current apply LSN
will be potentially/probably earlier than the latest on-disk LSN, as you
say. But once we have completed the next restartpoint *after* the value
pg_control says then we will be guaranteed that the two LSNs are the
same, since otherwise we would have restarted at a later point.

That kinda works, but the problem is that restartpoints are time based,
not log based. We need them to be deterministic for us to rely upon them
in the above way. If we crash and then replay we can only be certain we
are safe when we have found a restartpoint that the previous recovery
will definitely have reached.

So we must have log-based restartpoints, using either a constant LSN
offset, or a parameter like checkpoint_segments. But if it is changeable
then it needs to be written into the control file, so we don't make a
mistake about it.

So we need to:
* add an extra test to delay safe point if required
* write restart_segments value to control file
* force a restartpoint on first valid checkpoint WAL record after we
have passed restart_segments worth of log

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Proposed patch to change TOAST compression strategy for 8.3.4

2008-09-29T03:32:48Z

Hello,

Upgrade 8.3.4 is available. Before compiling, I have to apply the optimized toast patch : bin7hetTGkMRL.bin.

There are differences in 1 of the 3 files patched : tuptoaster.c

The patch runs successfully but before installing on production servers, I have to ask : what about the compatibility of this patch in 8.3.4 ?

Thanks,

Jérôme Jouanin

Re: [PgFoundry] Unsigned Data Types [1 of 2]

2008-09-28T22:12:38Z

Hello all,

Just wanted to let everyone know I have committed this patch to the
PgFoundry uint project.
I have also updated the commit-fest wiki with this status.

Thanks to everyone (especially Jaime) for the feedback and reviews.

- Ryan

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-28T18:16:01Z

Simon Riggs <simon@...> writes:
>> It does nothing AFAICS for the
>> problem that when restarting archive recovery from a restartpoint,
>> it's not clear when it is safe to start letting in backends. You need
>> to get past the highest LSN that has made it out to disk, and there is
>> no good way to know what that is.

> AFAICS when we set minRecoveryLoc we *never* unset it. It's recorded in
> the controlfile, so whenever we restart we can see that it has been set
> previously and now we are beyond it.

Right ...

> So if we crash during recovery and
> then restart *after* we reached minRecoveryLoc then we resume in safe
> mode almost immediately.

Wrong.

What minRecoveryLoc is is an upper bound for the LSNs that might be
on-disk in the filesystem backup that an archive recovery starts from.
(Defined as such, it never changes during a restartpoint crash/restart.)
Once you pass that, the on-disk state as modified by any dirty buffers
inside the recovery process represents a consistent database state.
However, the on-disk state alone is not guaranteed consistent. As you
flush some (not all) of your shared buffers you enter other
not-certainly-consistent on-disk states. If we crash in such a state,
we know how to use the last restartpoint plus WAL replay to recover to
another state in which disk + dirty buffers are consistent. However,
we reach such a state only when we have read WAL to beyond the highest
LSN that has reached disk --- and in recovery mode there is no clean
way to determine what that was.

Perhaps a solution is to make XLogFLush not be a no-op in recovery mode,
but have it scribble a highest-LSN somewhere on stable storage (maybe
scribble on pg_control itself, or maybe better someplace else). I'm
not totally sure about that. But I am sure that doing nothing will
be unreliable.

regards, tom lane

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-28T17:53:19Z

On Sun, 2008-09-28 at 14:02 -0400, Tom Lane wrote:

> It does nothing AFAICS for the
> problem that when restarting archive recovery from a restartpoint,
> it's not clear when it is safe to start letting in backends. You need
> to get past the highest LSN that has made it out to disk, and there is
> no good way to know what that is.
>
> Unless we can get past this problem the whole thing seems a bit dead
> in
> the water :-(

I agree the importance of your a problem but don't fully understand the
circumstances under which you see a problem arising.

AFAICS when we set minRecoveryLoc we *never* unset it. It's recorded in
the controlfile, so whenever we restart we can see that it has been set
previously and now we are beyond it. So if we crash during recovery and
then restart *after* we reached minRecoveryLoc then we resume in safe
mode almost immediately. If we crash during recovery before we reached
minRecoveryLoc then we continue until we find it.

There is a loophole, as described on separate post, but that can be
plugged by offering explicit setting of the minRecoveryLoc from
recovery.conf. Most people use pg_start_backup() so do not experience
the need for that.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] get_relation_stats_hook()

2008-09-28T12:57:49Z

Simon Riggs <simon@...> writes:
> New version of Postgres patch, v5. Implements suggested changes.
> Ready for review and apply.

Applied with some revisions. The method for passing back freefunc
didn't work, so I made it pass the whole VariableStatsData struct
instead; this might allow some additional flexibility by changing other
fields besides the intended statsTuple and freefunc. Also, I was still
unhappy about adding a hook in the midst of code that clearly needs
improvement, without making it possible for the hook to override the
adjacent broken code paths; so I refactored the API a bit for that too.

The plugin function would now be something like this:

static bool
plugin_get_relation_stats(PlannerInfo *root,
RangeTblEntry *rte,
AttrNumber attnum,
VariableStatData *vardata)
{
HeapTuple statstup = NULL;

/* For now, we only cover the simple-relation case */
if (rte->rtekind != RTE_RELATION || rte->inh)
return false;

if (!get_tom_stats_tupletable(rte->relid, attnum))
return false;

/*
* Get stats if present. We asked for only one row, so no need for loops.
*/
if (SPI_processed > 0)
statstup = SPI_copytuple(SPI_tuptable->vals[0]);

SPI_freetuptable(SPI_tuptable);
SPI_finish();

if (!statstup)
return false; /* should this happen? */

vardata->statsTuple = statstup;
/* define function to use when time to free the tuple */
vardata->freefunc = heap_freetuple;

return true;
}

and if you want to insert stats for expression indexes then there's a
separate get_index_stats_hook for that.

regards, tom lane

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-28T11:02:25Z

Simon Riggs <simon@...> writes:

> On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote:
>> After reading this for awhile, I realized that there is a rather
>> fundamental problem with it: it switches into "consistent recovery"
>> mode as soon as it's read WAL beyond ControlFile->minRecoveryPoint.
>> In a crash recovery situation that typically is before the last
>> checkpoint (if indeed it's not still zero), and what that means is
>> that this patch will activate the bgwriter and start letting in
>> backends instantaneously after a crash, long before we can have any
>> certainty that the DB state really is consistent.
>>
>> In a normal crash recovery situation this would be easily fixed by
>> simply not letting it go to "consistent recovery" state at all, but
>> what about recovery from a restartpoint? We don't want a slave that's
>> crashed once to never let backends in again. But I don't see how to
>> determine that we're far enough past the restartpoint to be consistent
>> again. In crash recovery we assume (without proof ;-)) that we're
>> consistent once we reach the end of valid-looking WAL, but that rule
>> doesn't help for a slave that's following a continuing WAL sequence.
>>
>> Perhaps something could be done based on noting when we have to pull in
>> a WAL segment from the recovery_command, but it sounds like a pretty
>> fragile assumption.

> Seems like we just say we only signal the postmaster if
> InArchiveRecovery. Archive recovery from a restartpoint is still archive
> recovery, so this shouldn't be a problem in the way you mention. The
> presence of recovery.conf overrides all other cases.

What that implements is my comment that we don't have to let anyone in
at all during a plain crash recovery. It does nothing AFAICS for the
problem that when restarting archive recovery from a restartpoint,
it's not clear when it is safe to start letting in backends. You need
to get past the highest LSN that has made it out to disk, and there is
no good way to know what that is.

Unless we can get past this problem the whole thing seems a bit dead in
the water :-(

>> * I'm a bit uncomfortable with the fact that the
>> IsRecoveryProcessingMode flag is read and written with no lock.

> It's not a dynamic state, so I can fix that inside
> IsRecoveryProcessingMode() with a local state to make check faster.

Erm, this code doesn't look like it can allow IsRecoveryProcessingMode
to become locally true in the first place? I guess you could fix it
by initializing IsRecoveryProcessingMode to true, but that seems likely
to break other places. Maybe better is to have an additional local
state variable showing whether the flag has ever been fetched from
shared memory.

The other issues don't seem worth arguing about ...

regards, tom lane

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] get_relation_stats_hook()

2008-09-26T07:51:53Z

On Wed, 2008-08-06 at 23:38 +0100, Simon Riggs wrote:

> On Wed, 2008-08-06 at 16:37 +0100, Simon Riggs wrote:
>
> > I'll submit the fully working plugin once we've stabilised the API. It's
> > designed as a contrib module, so it can go in pgfoundry or contrib.
>
> OK, here's fully working plugin, plus API patch.
>
> I expect to open a pgfoundry project for the plugin, but will wait for
> the main patch review.
>
> I've tried the APIs three different ways and this seems cleanest and
> most widely applicable approach. It's possible to add calls in more
> places, but I haven't done this for reasons discussed previously.

New version of Postgres patch, v5. Implements suggested changes.
Ready for review and apply.

New version of stats plugin, v3. Works with v5.
Corrected problems:
* now loads using preload_shared_libraries as well as LOAD
* example test script fix

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

[stat_hooks.v5.patch]

Index: src/backend/utils/adt/selfuncs.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/utils/adt/selfuncs.c,v retrieving revision 1.253 diff -c -r1.253 selfuncs.c *** src/backend/utils/adt/selfuncs.c 25 Aug 2008 22:42:34 -0000 1.253 --- src/backend/utils/adt/selfuncs.c 25 Sep 2008 15:57:58 -0000 *************** *** 118,123 **** --- 118,126 ---- #include "utils/selfuncs.h" #include "utils/syscache.h" + /* Hook for plugins to get control when we ask for stats */ + get_relation_stats_hook_type get_relation_stats_hook = NULL; + release_relation_stats_hook_type release_relation_stats_hook = NULL; static double var_eq_const(VariableStatData *vardata, Oid operator, Datum constval, bool constisnull, *************** *** 3996,4005 **** } else if (rte->rtekind == RTE_RELATION) { ! vardata->statsTuple = SearchSysCache(STATRELATT, ObjectIdGetDatum(rte->relid), Int16GetDatum(var->varattno), 0, 0); } else { --- 3999,4017 ---- } else if (rte->rtekind == RTE_RELATION) { ! if (get_relation_stats_hook) ! vardata->statsTuple = (*get_relation_stats_hook) ! (rte->relid, ! var->varattno, ! vardata->freefunc); ! if (!vardata->statsTuple) ! { ! vardata->statsTuple = SearchSysCache(STATRELATT, ObjectIdGetDatum(rte->relid), Int16GetDatum(var->varattno), 0, 0); + vardata->freefunc = ReleaseSysCache; + } } else { *************** *** 4116,4125 **** index->indpred == NIL) vardata->isunique = true; /* Has it got stats? */ ! vardata->statsTuple = SearchSysCache(STATRELATT, ObjectIdGetDatum(index->indexoid), ! Int16GetDatum(pos + 1), ! 0, 0); if (vardata->statsTuple) break; } --- 4128,4147 ---- index->indpred == NIL) vardata->isunique = true; /* Has it got stats? */ ! if (get_relation_stats_hook) ! vardata->statsTuple = (*get_relation_stats_hook) ! (index->indexoid, ! pos + 1, ! vardata->freefunc); ! if (!vardata->statsTuple) ! { ! vardata->statsTuple = SearchSysCache(STATRELATT, ObjectIdGetDatum(index->indexoid), ! Int16GetDatum(pos + 1), ! 0, 0); ! vardata->freefunc = ReleaseSysCache; ! } ! if (vardata->statsTuple) break; } *************** *** 5551,5557 **** double *indexCorrelation = (double *) PG_GETARG_POINTER(7); Oid relid; AttrNumber colnum; ! HeapTuple tuple; double numIndexTuples; List *indexBoundQuals; int indexcol; --- 5573,5580 ---- double *indexCorrelation = (double *) PG_GETARG_POINTER(7); Oid relid; AttrNumber colnum; ! HeapTuple tuple = NULL; ! void (*freefunc) (HeapTuple tuple) = NULL; double numIndexTuples; List *indexBoundQuals; int indexcol; *************** *** 5756,5765 **** colnum = 1; } ! tuple = SearchSysCache(STATRELATT, ! ObjectIdGetDatum(relid), ! Int16GetDatum(colnum), ! 0, 0); if (HeapTupleIsValid(tuple)) { --- 5779,5795 ---- colnum = 1; } ! if (get_relation_stats_hook) ! tuple = (*get_relation_stats_hook) (relid, colnum, freefunc); ! ! if (!tuple) ! { ! tuple = SearchSysCache(STATRELATT, ! ObjectIdGetDatum(relid), ! Int16GetDatum(colnum), ! 0, 0); ! freefunc = ReleaseSysCache; ! } if (HeapTupleIsValid(tuple)) { *************** *** 5800,5806 **** free_attstatsslot(InvalidOid, NULL, 0, numbers, nnumbers); } ! ReleaseSysCache(tuple); } PG_RETURN_VOID(); --- 5830,5837 ---- free_attstatsslot(InvalidOid, NULL, 0, numbers, nnumbers); } ! ! ReleaseStatsTuple(tuple, freefunc); } PG_RETURN_VOID(); Index: src/backend/utils/cache/lsyscache.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/utils/cache/lsyscache.c,v retrieving revision 1.159 diff -c -r1.159 lsyscache.c *** src/backend/utils/cache/lsyscache.c 2 Aug 2008 21:32:00 -0000 1.159 --- src/backend/utils/cache/lsyscache.c 25 Sep 2008 14:16:01 -0000 *************** *** 27,32 **** --- 27,33 ---- #include "catalog/pg_proc.h" #include "catalog/pg_statistic.h" #include "catalog/pg_type.h" + #include "optimizer/plancat.h" #include "miscadmin.h" #include "nodes/makefuncs.h" #include "utils/array.h" *************** *** 35,40 **** --- 36,43 ---- #include "utils/lsyscache.h" #include "utils/syscache.h" + /* Hook for plugins to get control in get_attavgwidth() */ + get_attavgwidth_hook_type get_attavgwidth_hook = NULL; /* ---------- AMOP CACHES ---------- */ *************** *** 2492,2507 **** * * Given the table and attribute number of a column, get the average * width of entries in the column. Return zero if no data available. */ int32 get_attavgwidth(Oid relid, AttrNumber attnum) { HeapTuple tp; tp = SearchSysCache(STATRELATT, ! ObjectIdGetDatum(relid), ! Int16GetDatum(attnum), ! 0, 0); if (HeapTupleIsValid(tp)) { int32 stawidth = ((Form_pg_statistic) GETSTRUCT(tp))->stawidth; --- 2495,2524 ---- * * Given the table and attribute number of a column, get the average * width of entries in the column. Return zero if no data available. + * + * Calling a hook at this point looks somewhat strange, but is required + * because the optimizer handles inheritance relations by calling for + * the avg width later in the planner than get_relation_info_hook(). + * So the APIs and call points of hooks must match the optimizer. */ int32 get_attavgwidth(Oid relid, AttrNumber attnum) { HeapTuple tp; + int32 stawidth; + + if (get_attavgwidth_hook) + { + stawidth = (*get_attavgwidth_hook) (relid, attnum); + if (stawidth > 0) + return stawidth; + } tp = SearchSysCache(STATRELATT, ! ObjectIdGetDatum(relid), ! Int16GetDatum(attnum), ! 0, 0); ! if (HeapTupleIsValid(tp)) { int32 stawidth = ((Form_pg_statistic) GETSTRUCT(tp))->stawidth; Index: src/include/optimizer/plancat.h =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/include/optimizer/plancat.h,v retrieving revision 1.51 diff -c -r1.51 plancat.h *** src/include/optimizer/plancat.h 16 Aug 2008 00:01:38 -0000 1.51 --- src/include/optimizer/plancat.h 25 Sep 2008 15:57:06 -0000 *************** *** 14,19 **** --- 14,20 ---- #ifndef PLANCAT_H #define PLANCAT_H + #include "access/htup.h" #include "nodes/relation.h" #include "utils/relcache.h" *************** *** 24,29 **** --- 25,43 ---- RelOptInfo *rel); extern PGDLLIMPORT get_relation_info_hook_type get_relation_info_hook; + /* Hooks for plugins to get control in lsyscache.c and selfuncs.c */ + typedef HeapTuple (*get_relation_stats_hook_type) (Oid relid, AttrNumber attnum, + void (*freefunc) (HeapTuple tuple)); + extern PGDLLIMPORT get_relation_stats_hook_type get_relation_stats_hook; + + /* must match ReleaseSysCache call signature */ + typedef void (*release_relation_stats_hook_type) (HeapTuple tuple); + extern PGDLLIMPORT release_relation_stats_hook_type release_relation_stats_hook; + + typedef int32 (*get_attavgwidth_hook_type) (Oid relid, AttrNumber attnum); + extern PGDLLIMPORT get_attavgwidth_hook_type get_attavgwidth_hook; + + extern void get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, RelOptInfo *rel); Index: src/include/utils/selfuncs.h =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/include/utils/selfuncs.h,v retrieving revision 1.46 diff -c -r1.46 selfuncs.h *** src/include/utils/selfuncs.h 16 Aug 2008 00:01:38 -0000 1.46 --- src/include/utils/selfuncs.h 25 Sep 2008 15:55:05 -0000 *************** *** 74,85 **** Oid atttype; /* type to pass to get_attstatsslot */ int32 atttypmod; /* typmod to pass to get_attstatsslot */ bool isunique; /* true if matched to a unique index */ } VariableStatData; #define ReleaseVariableStats(vardata) \ do { \ if (HeapTupleIsValid((vardata).statsTuple)) \ ! ReleaseSysCache((vardata).statsTuple); \ } while(0) --- 74,102 ---- Oid atttype; /* type to pass to get_attstatsslot */ int32 atttypmod; /* typmod to pass to get_attstatsslot */ bool isunique; /* true if matched to a unique index */ + void (*freefunc) (HeapTuple tuple); /* funct ptr to free statsTuple */ } VariableStatData; + #define ReleaseStatsTuple(tuple, freefunc) \ + do { \ + if (HeapTupleIsValid(tuple)) \ + { \ + if (freefunc) \ + freefunc(tuple); \ + else \ + elog(ERROR, "unable to release variable stats correctly"); \ + } \ + } while(0) + #define ReleaseVariableStats(vardata) \ do { \ if (HeapTupleIsValid((vardata).statsTuple)) \ ! { \ ! if ((vardata).freefunc) \ ! (vardata).freefunc((vardata).statsTuple); \ ! else \ ! elog(ERROR, "unable to release variable stats correctly"); \ ! } \ } while(0)

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

TOM.v3.tar (41K) Download Attachment

Re: [HACKERS] Infrastructure changes for recovery

2008-09-26T04:41:55Z

On Fri, 2008-09-26 at 11:20 +0100, Simon Riggs wrote:

> > After reading this for awhile, I realized that there is a rather
> > fundamental problem with it: it switches into "consistent recovery"
> > mode as soon as it's read WAL beyond ControlFile->minRecoveryPoint.
> > In a crash recovery situation that typically is before the last
> > checkpoint (if indeed it's not still zero), and what that means is
> > that this patch will activate the bgwriter and start letting in
> > backends instantaneously after a crash, long before we can have any
> > certainty that the DB state really is consistent.
> >
> > In a normal crash recovery situation this would be easily fixed by
> > simply not letting it go to "consistent recovery" state at all, but
> > what about recovery from a restartpoint? We don't want a slave that's
> > crashed once to never let backends in again. But I don't see how to
> > determine that we're far enough past the restartpoint to be consistent
> > again. In crash recovery we assume (without proof ;-)) that we're
> > consistent once we reach the end of valid-looking WAL, but that rule
> > doesn't help for a slave that's following a continuing WAL sequence.
> >
> > Perhaps something could be done based on noting when we have to pull in
> > a WAL segment from the recovery_command, but it sounds like a pretty
> > fragile assumption.
>
> Seems like we just say we only signal the postmaster if
> InArchiveRecovery. Archive recovery from a restartpoint is still archive
> recovery, so this shouldn't be a problem in the way you mention. The
> presence of recovery.conf overrides all other cases.

Anticipating your possible reponses, I would add this also:

There has long been an annoying hole in the PITR scheme which is the
support of recovery using a crashed database. That is there to support
split mirror snapshots, but it creates a loophole where we don't know
the min recovery location, circumventing the care we (you!) took to put
stop/start backup in place.

I think we need to add a parameter to recovery.conf that people can use
to specify a minRecoveryPoint iff there in no backup label file. They
can work out what this should be by following this procedure, which we
should document:
* split mirror, so you have offline copy of crashed database
* copy database away to backup
* go to running database and run pg_current_xlog_insert_location()
* use the value to specify recovery_min_location

If they don't specify this, then bgwriter will not start and you cannot
run in Hot Standby mode. Their choice, so we need not worry then about
the loophole any more.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-26T03:19:54Z

On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote:
> Simon Riggs <simon@...> writes:
> > Version 7

> Anyway, that's sufficiently bad that I'm bouncing the patch for
> reconsideration.

No problem, I understand this needs discussion.

There's less detail here than first appears. There are some basic points
to consider from which all else follows.

> After reading this for awhile, I realized that there is a rather
> fundamental problem with it: it switches into "consistent recovery"
> mode as soon as it's read WAL beyond ControlFile->minRecoveryPoint.
> In a crash recovery situation that typically is before the last
> checkpoint (if indeed it's not still zero), and what that means is
> that this patch will activate the bgwriter and start letting in
> backends instantaneously after a crash, long before we can have any
> certainty that the DB state really is consistent.
>
> In a normal crash recovery situation this would be easily fixed by
> simply not letting it go to "consistent recovery" state at all, but
> what about recovery from a restartpoint? We don't want a slave that's
> crashed once to never let backends in again. But I don't see how to
> determine that we're far enough past the restartpoint to be consistent
> again. In crash recovery we assume (without proof ;-)) that we're
> consistent once we reach the end of valid-looking WAL, but that rule
> doesn't help for a slave that's following a continuing WAL sequence.
>
> Perhaps something could be done based on noting when we have to pull in
> a WAL segment from the recovery_command, but it sounds like a pretty
> fragile assumption.

Seems like we just say we only signal the postmaster if
InArchiveRecovery. Archive recovery from a restartpoint is still archive
recovery, so this shouldn't be a problem in the way you mention. The
presence of recovery.conf overrides all other cases.

> Some other issues I noted before giving up:

All of these issues raised can be addressed, but I think the main
decision we need to make is not so much about running other processes
but about when it can start and when they have to change mode.

When they can start seems solvable, as above.

When/how they must change state from recovery to normal mode seems more
difficult. State change must be atomic across all processes, but also
done at a micro level so that XLogFlush tests for the state change. The
overhead of continually checking seems high, so I am tempted to say lets
just kick 'em all off and then let them back on again. That's easily
accomplished for bgwriter without anybody noticing much. For Hot Standby
that would mean that a failover would kick off all query backends. I can
see why that would be both desirable and undesirable.

Anyway, from here I propose:
* we keep the shutdown checkpoint
* we kick off bgwriter (and any children) then let 'em back on again so
they can initialise in a different mode.

To do that, I just need to dust off a previous version of the patch. So
we can sort this out quickly if we have a clear way to proceed.

------------------------------------------------------------------
other comments relate to this current patch, so further discussion of
the points below may not be required, if we agree how to proceed as
above.

> * I'm a bit uncomfortable with the fact that the
> IsRecoveryProcessingMode flag is read and written with no lock.
> There's no atomicity or concurrent-write problem, sure, but on
> a multiprocessor with weak memory ordering guarantees (eg PPC)
> readers could get a fairly stale value of the flag. The false
> to true transition happens before anyone except the startup process is
> running, so that's no problem; the net effect is then that backends
> might think that recovery mode was still active for awhile after it
> wasn't. This seems a bit scary, eg in the patch as it stands that'll
> disable XLogFlush calls that should have happened. You could fix that
> by grabbing/releasing some spinlock (any old one) around the accesses,
> but are any of the call sites performance-critical? The one in
> XLogInsert looks like it is, if nothing else.

Agreed.

It's not a dynamic state, so I can fix that inside
IsRecoveryProcessingMode() with a local state to make check faster.

bool
IsRecoveryProcessingMode(void)
{
if (!IsRecoveryProcessingMode)
return false;

{
/* use volatile pointer to prevent code rearrangement */
volatile XLogCtlData *xlogctl = XLogCtl;

SpinLockAcquire(&xlogctl->mode_lck);
RecoveryProcessingMode = XLogCtl->IsRecoveryProcessingMode;
SpinLockRelease(&xlogctl->mode_lck);
}

return IsRecoveryProcessingMode;
}

This only applies if we decide not to kick everybody off, change state
and then let them back on again.

> * I kinda think you broke XLogFlush anyway. It's certainly clear
> that the WARNING case at the bottom is unreachable with the patch,
> and I think that means that you've messed up an important error
> robustness behavior. Is it still possible to get out of recovery mode
> if there are any bad LSNs in the shared buffer pool?

Perhaps. But the WARNING could only occur during shutdown checkpoints.
This specifically patch avoids those, so the case would never arise with
this patch and needs no avoidance. Yes, you can still leave recovery
mode if there are bad LSNs with this patch, but you won't know what they
are because of the lack of the shutdown checkpoint. Probably an argument
in favour of allowing shutdown checkpoints.

> * The use of InRecovery in CreateCheckPoint seems pretty bogus, since
> that function can be called from the bgwriter in which the flag will
> never be true. Either this needs to be IsRecoveryProcessingMode(),
> or it's useless because we'll never create ordinary checkpoints until
> after subtrans.c is up anyway.

Exactly. bgwriter never needs this to be set because it writes
restorepoints before this, using different code path.

> * The patch moves the clearing of InRecovery from after to before
> StartupCLOG, RecoverPreparedTransactions, etc. Is that really a
> good idea? It makes it hard for those modules to know if they are
> coming up after a normal or recovery startup. I think they may not
> care at the moment, but I'd leave that alone without a darn good
> reason to change it.

I didn't move this as you say. It already was before StartupClog.

I moved it into exitRecovery() only, so it was unset in the same way
exitArchiveRecovery sets InArchiveRecovery to false. So refactoring, no
change of sequencing.

> * The comment about CheckpointLock being only pro forma is now wrong,
> if you are going to use locking it to implement a delay in exitRecovery.
> But I don't understand why the delay there. If it's needed, seems like
> the path where a restartpoint *isn't* in progress is wrong --- don't you
> actually need to start one and wait for it?

All of this ducking and diving is because of the bgwriter needing to
perform a state change. That's the ball to keep our eye on.

After much thrashing, I decided that interrupting a restartpoint is too
dangerous a thing to want to do. If we're in the middle of one, we
finish it, if not there's no need to interrupt it.

> Also I note that if the
> LWLockConditionalAcquire succeeds, you neglect to release the lock,
> which surely can't be right.

Doh. Thanks.

> * The tail end of StartupXLOG() looks pretty unsafe to me. Surely
> we mustn't clear IsRecoveryProcessingMode until after we have
> established the safe-recovery checkpoint. The comments there seem to
> be only vaguely related to the current state of the patch, too.

The whole point was to remove the ShutdownCheckpoint, but it sounds like
you're not keen on that any more.

I'm neutral on this point: I can see why people would want it removed -
it will speed up failover. I can see why people would want it kept -
there is a slight window where if we crash we will need to go back to
archive recovery.

I had a workable solution that kept it, so will revert to it.

> * Logging of recoveryLastXTime seems pretty bogus now. It's supposed to
> be associated with a restartpoint completion report, but now it's just
> out in the ether somewhere and doesn't represent a guarantee that we're
> synchronized that far.

That last one was there before so we knew where the log ended. It was
not supposed to be associated with a restartpoint, just with end of log
purely for information (by user request, for when a log file is
corrupted and we need to know "when" we are up to).

> * backup.sgml needs to be updated to say that log_restartpoints is
> obsolete.

Yes

> * There are a bunch of disturbing assumptions in the SLRU-related
> modules about their StartUp calls being executed without any concurrent
> access. This isn't really a problem that needs to be dealt with in this
> patch, I think, but that will all have to be cleaned up before we dare
> allow any backends to run concurrently with recovery.

Well spotted, thanks.

> btree's
> suppression of relcache invals for metapage updates will be a problem
> too.

Again thanks. This patch is stand-alone from later work, thats why.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Infrastructure changes for recovery

2008-09-25T15:28:39Z

Simon Riggs <simon@...> writes:
> Version 7

After reading this for awhile, I realized that there is a rather
fundamental problem with it: it switches into "consistent recovery"
mode as soon as it's read WAL beyond ControlFile->minRecoveryPoint.
In a crash recovery situation that typically is before the last
checkpoint (if indeed it's not still zero), and what that means is
that this patch will activate the bgwriter and start letting in
backends instantaneously after a crash, long before we can have any
certainty that the DB state really is consistent.

In a normal crash recovery situation this would be easily fixed by
simply not letting it go to "consistent recovery" state at all, but
what about recovery from a restartpoint? We don't want a slave that's
crashed once to never let backends in again. But I don't see how to
determine that we're far enough past the restartpoint to be consistent
again. In crash recovery we assume (without proof ;-)) that we're
consistent once we reach the end of valid-looking WAL, but that rule
doesn't help for a slave that's following a continuing WAL sequence.

Perhaps something could be done based on noting when we have to pull in
a WAL segment from the recovery_command, but it sounds like a pretty
fragile assumption.

Anyway, that's sufficiently bad that I'm bouncing the patch for
reconsideration. Some other issues I noted before giving up:

* I'm a bit uncomfortable with the fact that the
IsRecoveryProcessingMode flag is read and written with no lock.
There's no atomicity or concurrent-write problem, sure, but on
a multiprocessor with weak memory ordering guarantees (eg PPC)
readers could get a fairly stale value of the flag. The false
to true transition happens before anyone except the startup process is
running, so that's no problem; the net effect is then that backends
might think that recovery mode was still active for awhile after it
wasn't. This seems a bit scary, eg in the patch as it stands that'll
disable XLogFlush calls that should have happened. You could fix that
by grabbing/releasing some spinlock (any old one) around the accesses,
but are any of the call sites performance-critical? The one in
XLogInsert looks like it is, if nothing else.

* I kinda think you broke XLogFlush anyway. It's certainly clear
that the WARNING case at the bottom is unreachable with the patch,
and I think that means that you've messed up an important error
robustness behavior. Is it still possible to get out of recovery mode
if there are any bad LSNs in the shared buffer pool?

* The use of InRecovery in CreateCheckPoint seems pretty bogus, since
that function can be called from the bgwriter in which the flag will
never be true. Either this needs to be IsRecoveryProcessingMode(),
or it's useless because we'll never create ordinary checkpoints until
after subtrans.c is up anyway.

* The patch moves the clearing of InRecovery from after to before
StartupCLOG, RecoverPreparedTransactions, etc. Is that really a
good idea? It makes it hard for those modules to know if they are
coming up after a normal or recovery startup. I think they may not
care at the moment, but I'd leave that alone without a darn good
reason to change it.

* The comment about CheckpointLock being only pro forma is now wrong,
if you are going to use locking it to implement a delay in exitRecovery.
But I don't understand why the delay there. If it's needed, seems like
the path where a restartpoint *isn't* in progress is wrong --- don't you
actually need to start one and wait for it? Also I note that if the
LWLockConditionalAcquire succeeds, you neglect to release the lock,
which surely can't be right.

* The tail end of StartupXLOG() looks pretty unsafe to me. Surely
we mustn't clear IsRecoveryProcessingMode until after we have
established the safe-recovery checkpoint. The comments there seem to
be only vaguely related to the current state of the patch, too.

* Logging of recoveryLastXTime seems pretty bogus now. It's supposed to
be associated with a restartpoint completion report, but now it's just
out in the ether somewhere and doesn't represent a guarantee that we're
synchronized that far.

* backup.sgml needs to be updated to say that log_restartpoints is
obsolete.

* There are a bunch of disturbing assumptions in the SLRU-related
modules about their StartUp calls being executed without any concurrent
access. This isn't really a problem that needs to be dealt with in this
patch, I think, but that will all have to be cleaned up before we dare
allow any backends to run concurrently with recovery. btree's
suppression of relcache invals for metapage updates will be a problem
too.

regards, tom lane

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Subtransaction commits and Hot Standby

2008-09-24T14:44:19Z

On Wed, 2008-09-24 at 13:48 +0100, Simon Riggs wrote:
> On Tue, 2008-09-23 at 22:47 +0100, Simon Riggs wrote:
>
> > I've tested this some more and am much happier with it now.
>
> The concept is fine, but I've found a coding bug in further testing.
> Please wait now for new version before review.

OK, spent long time testing various batching scenarios for this using a
custom test harness to simulate various spreads of xids in transaction
trees. All looks fine now.

The main work is done in new clog.c functions:
TransactionIdSetTreeStatus() which sets whole tree atomically by calling
TransactionIdSetPageStatus(), which in turn calls
TransactionIdSetStatusBit() for each xid status change.

TransactionIdSetPageStatus() performs locking and handles write_ok
problem, as did code it replaces. TransactionIdSetPageStatus() is called
theoretical minimum number of times for any transaction tree.

Patch slightly fumbles diff-ing new and replacement code, so there are
two chunks that appear to show I'm removing locking. I'm not!!

Everything else is just API changes.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

[atomic_subxids.v4.patch]

Index: src/backend/access/transam/README =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/README,v retrieving revision 1.11 diff -c -r1.11 README *** src/backend/access/transam/README 21 Mar 2008 13:23:28 -0000 1.11 --- src/backend/access/transam/README 24 Sep 2008 17:33:23 -0000 *************** *** 342,351 **** an XID. A transaction can be in progress, committed, aborted, or "sub-committed". This last state means that it's a subtransaction that's no longer running, but its parent has not updated its state yet (either it is ! still running, or the backend crashed without updating its status). A ! sub-committed transaction's status will be updated again to the final value as ! soon as the parent commits or aborts, or when the parent is detected to be ! aborted. Savepoints are implemented using subtransactions. A subtransaction is a transaction inside a transaction; its commit or abort status is not only --- 342,360 ---- an XID. A transaction can be in progress, committed, aborted, or "sub-committed". This last state means that it's a subtransaction that's no longer running, but its parent has not updated its state yet (either it is ! still running, or the backend crashed without updating its status). Prior ! to 8.4 we updated the status to sub-committed in clog as soon as ! sub-commit had happened. It was later realised that this is not actually ! required for any purpose and the action can be deferred until transaction ! commit. The main role of marking transactions as sub-committed is to ! provide an atomic commit protocol when transaction status is spread across ! multiple clog pages. As a result whenever transaction status spreads ! across multiple pages we must use a two-phase commit protocol. The first ! phase is to mark the subtransactions as sub-committed, then we mark the ! top level transaction and all its subtransactions committed (in that order). ! So in 8.4 sub-committed state still exists, but as a transitory state as ! part of final commit. Subtransaction abort is always marked in clog as ! soon as it occurs, to allow locks to be released. Savepoints are implemented using subtransactions. A subtransaction is a transaction inside a transaction; its commit or abort status is not only Index: src/backend/access/transam/clog.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/clog.c,v retrieving revision 1.47 diff -c -r1.47 clog.c *** src/backend/access/transam/clog.c 1 Aug 2008 13:16:08 -0000 1.47 --- src/backend/access/transam/clog.c 24 Sep 2008 21:24:02 -0000 *************** *** 80,89 **** static bool CLOGPagePrecedes(int page1, int page2); static void WriteZeroPageXlogRec(int pageno); static void WriteTruncateXlogRec(int pageno); ! ! ! /* ! * Record the final state of a transaction in the commit log. * * lsn must be the WAL location of the commit record when recording an async * commit. For a synchronous commit it can be InvalidXLogRecPtr, since the --- 80,105 ---- static bool CLOGPagePrecedes(int page1, int page2); static void WriteZeroPageXlogRec(int pageno); static void WriteTruncateXlogRec(int pageno); ! static void TransactionIdSetPageStatus(TransactionId xid, int nsubxids, ! TransactionId *subxids, XidStatus status, XLogRecPtr lsn, int pageno, bool subcommit); ! static void TransactionIdSetStatusBit(TransactionId xid, XidStatus status, ! XLogRecPtr lsn, int slotno); ! ! /* ! * TransactionIdSetTreeStatus ! * ! * Record the final state of transaction entries in the commit log for ! * a transaction and its subtransaction tree. Take care to ensure this is ! * both atomic and efficient. Prior to 8.4, this capability was provided ! * by the non-atomic TransactionIdSetStatus, which is replaced by this ! * new atomic version. ! * ! * xid is a single xid to set status for. This will typically be ! * the top level transactionid for a top level commit or abort. It can ! * also be a subtransaction when we record transaction aborts. ! * ! * subxids is an array of xids of length nsubxids, representing subtransactions ! * in the tree of xid. In various cases nsubxids may be zero. * * lsn must be the WAL location of the commit record when recording an async * commit. For a synchronous commit it can be InvalidXLogRecPtr, since the *************** *** 91,107 **** * should be InvalidXLogRecPtr for abort cases, too. * * NB: this is a low-level routine and is NOT the preferred entry point ! * for most uses; TransactionLogUpdate() in transam.c is the intended caller. */ void ! TransactionIdSetStatus(TransactionId xid, XidStatus status, XLogRecPtr lsn) { - int pageno = TransactionIdToPage(xid); - int byteno = TransactionIdToByte(xid); - int bshift = TransactionIdToBIndex(xid) * CLOG_BITS_PER_XACT; int slotno; ! char *byteptr; ! char byteval; Assert(status == TRANSACTION_STATUS_COMMITTED || status == TRANSACTION_STATUS_ABORTED || --- 107,238 ---- * should be InvalidXLogRecPtr for abort cases, too. * * NB: this is a low-level routine and is NOT the preferred entry point ! * for most uses; functions in in transam.c are the intended callers. ! * ! * Note that no lock requests are made at this level, only lower functions. ! * ! * XXX Think about issuing FADVISE_WILLNEED on pages that we will need, ! * but aren't yet in cache, as well as hinting pages not to fall out of ! * cache yet. */ void ! TransactionIdSetTreeStatus(TransactionId xid, int nsubxids, ! TransactionId *subxids, XidStatus status, XLogRecPtr lsn) ! { ! int pageno = TransactionIdToPage(xid); /* get page of parent */ ! int i; ! ! Assert(status == TRANSACTION_STATUS_COMMITTED || ! status == TRANSACTION_STATUS_ABORTED); ! ! /* ! * See how many subxids, if any, are on the same page as the parent, if any. ! */ ! for (i = 0; i < nsubxids; i++) ! { ! if (TransactionIdToPage(subxids[i]) != pageno) ! break; ! } ! ! /* ! * Do all items fit on a single page? ! */ ! if (i == nsubxids) ! { ! /* ! * Set the parent and any subtransactions on same page as it ! */ ! TransactionIdSetPageStatus(xid, nsubxids, subxids, status, lsn, ! pageno, true); ! } ! else ! { ! int num_on_first_page = i; ! int num_on_page = 0; ! int offset; ! ! if (status == TRANSACTION_STATUS_COMMITTED) ! { ! /* ! * If this is a commit then we care about doing this atomically. ! * By here, we know we're updating more than one page of clog, ! * so we must mark entries that are *not* on the first page so ! * that they show as subcommitted before we then return to ! * update the status to fully committed. ! * We don't mark the first page because we will be doing that ! * when we mark the main commit, so we wish to avoid touching ! * that page twice. ! */ ! num_on_page = 0; ! i = offset = num_on_first_page; ! pageno = TransactionIdToPage(subxids[num_on_first_page]); ! while (i < nsubxids) ! { ! while (TransactionIdToPage(subxids[i]) == pageno && i < nsubxids) ! { ! num_on_page++; ! i++; ! } ! ! TransactionIdSetPageStatus(InvalidTransactionId, ! num_on_page, subxids + offset, ! TRANSACTION_STATUS_SUB_COMMITTED, lsn, pageno, true); ! offset = i; ! num_on_page = 0; ! pageno = TransactionIdToPage(subxids[offset]); ! } ! } ! ! /* ! * Now set the parent and subtransactions on same page as it, if any ! */ ! pageno = TransactionIdToPage(xid); ! TransactionIdSetPageStatus(xid, num_on_first_page, subxids, status, lsn, ! pageno, true); ! ! /* ! * By now, all subtransactions have been subcommitted, so all calls ! * to TransactionIdSetPageStatus() will use subcommit=false after ! * this point for this transaction tree. ! */ ! ! /* ! * Now work through the rest of the subxids one clog page at a time, ! * starting from the second page onwards, like we did above. ! */ ! num_on_page = 0; ! i = offset = num_on_first_page; ! pageno = TransactionIdToPage(subxids[num_on_first_page]); ! while (i < nsubxids) ! { ! while (TransactionIdToPage(subxids[i]) == pageno && i < nsubxids) ! { ! num_on_page++; ! i++; ! } ! ! TransactionIdSetPageStatus(InvalidTransactionId, ! num_on_page, subxids + offset, ! status, lsn, pageno, false); ! offset = i; ! num_on_page = 0; ! pageno = TransactionIdToPage(subxids[offset]); ! } ! } ! } ! ! /* ! * Record the final state of transaction entries in the commit log for ! * all entries on *one* page only. Atomic only on this page. ! * ! * Otherwise API is same as TransactionIdSetTreeStatus() ! */ ! static void ! TransactionIdSetPageStatus(TransactionId xid, int nsubxids, ! TransactionId *subxids, XidStatus status, XLogRecPtr lsn, int pageno, bool subcommit) { int slotno; ! int i; Assert(status == TRANSACTION_STATUS_COMMITTED || status == TRANSACTION_STATUS_ABORTED || *************** *** 116,124 **** * mustn't let it reach disk until we've done the appropriate WAL flush. * But when lsn is invalid, it's OK to scribble on a page while it is * write-busy, since we don't care if the update reaches disk sooner than ! * we think. Hence, pass write_ok = XLogRecPtrIsInvalid(lsn). */ slotno = SimpleLruReadPage(ClogCtl, pageno, XLogRecPtrIsInvalid(lsn), xid); byteptr = ClogCtl->shared->page_buffer[slotno] + byteno; /* Current state should be 0, subcommitted or target state */ --- 247,303 ---- * mustn't let it reach disk until we've done the appropriate WAL flush. * But when lsn is invalid, it's OK to scribble on a page while it is * write-busy, since we don't care if the update reaches disk sooner than ! * we think. */ slotno = SimpleLruReadPage(ClogCtl, pageno, XLogRecPtrIsInvalid(lsn), xid); + + /* + * If we synch commit more than one xid on this page while write busy + * we might find that some of the bits go to disk and others don't. + * That would break atomicity, so if we haven't already subcommitted + * the xids for this commit, we do that first and then come back + * to start marking commits. If using async commit then we already + * waited for the write I/O to complete by this point, so nothing to do. + */ + if (subcommit && status == TRANSACTION_STATUS_COMMITTED && + XLogRecPtrIsInvalid(lsn)) + { + for (i = 0; i < nsubxids; i++) + { + Assert(ClogCtl->shared->page_number[slotno] == TransactionIdToPage(subxids[i])); + TransactionIdSetStatusBit(subxids[i], + TRANSACTION_STATUS_SUB_COMMITTED, lsn, slotno); + } + } + + + /* Set the main transaction id, if any */ + if (TransactionIdIsValid(xid)) + TransactionIdSetStatusBit(xid, status, lsn, slotno); + + /* Set the subtransactions on this page only */ + for (i = 0; i < nsubxids; i++) + { + Assert(ClogCtl->shared->page_number[slotno] == TransactionIdToPage(subxids[i])); + TransactionIdSetStatusBit(subxids[i], status, lsn, slotno); + } + + ClogCtl->shared->page_dirty[slotno] = true; + + LWLockRelease(CLogControlLock); + } + + /* + * Must be called with CLogControlLock held + */ + static void + TransactionIdSetStatusBit(TransactionId xid, XidStatus status, XLogRecPtr lsn, int slotno) + { + int byteno = TransactionIdToByte(xid); + int bshift = TransactionIdToBIndex(xid) * CLOG_BITS_PER_XACT; + char *byteptr; + char byteval; + byteptr = ClogCtl->shared->page_buffer[slotno] + byteno; /* Current state should be 0, subcommitted or target state */ *************** *** 132,139 **** byteval |= (status << bshift); *byteptr = byteval; - ClogCtl->shared->page_dirty[slotno] = true; - /* * Update the group LSN if the transaction completion LSN is higher. * --- 311,316 ---- *************** *** 149,156 **** if (XLByteLT(ClogCtl->shared->group_lsn[lsnindex], lsn)) ClogCtl->shared->group_lsn[lsnindex] = lsn; } - - LWLockRelease(CLogControlLock); } /* --- 326,331 ---- Index: src/backend/access/transam/transam.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/transam.c,v retrieving revision 1.76 diff -c -r1.76 transam.c *** src/backend/access/transam/transam.c 26 Mar 2008 18:48:59 -0000 1.76 --- src/backend/access/transam/transam.c 24 Sep 2008 17:33:23 -0000 *************** *** 40,54 **** /* Local functions */ static XidStatus TransactionLogFetch(TransactionId transactionId); - static void TransactionLogUpdate(TransactionId transactionId, - XidStatus status, XLogRecPtr lsn); /* ---------------------------------------------------------------- * Postgres log access method interface * * TransactionLogFetch ! * TransactionLogUpdate * ---------------------------------------------------------------- */ --- 40,58 ---- /* Local functions */ static XidStatus TransactionLogFetch(TransactionId transactionId); /* ---------------------------------------------------------------- * Postgres log access method interface * * TransactionLogFetch ! * ! * Prior to 8.4, we also had TransactionLogUpdate and ! * TransactionLogMultiUpdate. These have now been merged ! * into a single command TransactionIdSetTreeStatus(), ! * though that is now part of clog.c because of the need ! * for closer integration with clog code to achieve ! * atomic clog updates for subtransactions. * ---------------------------------------------------------------- */ *************** *** 100,140 **** return xidstatus; } - /* - * TransactionLogUpdate - * - * Store the new status of a transaction. The commit record LSN must be - * passed when recording an async commit; else it should be InvalidXLogRecPtr. - */ - static inline void - TransactionLogUpdate(TransactionId transactionId, - XidStatus status, XLogRecPtr lsn) - { - /* - * update the commit log - */ - TransactionIdSetStatus(transactionId, status, lsn); - } - - /* - * TransactionLogMultiUpdate - * - * Update multiple transaction identifiers to a given status. - * Don't depend on this being atomic; it's not. - */ - static inline void - TransactionLogMultiUpdate(int nxids, TransactionId *xids, - XidStatus status, XLogRecPtr lsn) - { - int i; - - Assert(nxids != 0); - - for (i = 0; i < nxids; i++) - TransactionIdSetStatus(xids[i], status, lsn); - } - - /* ---------------------------------------------------------------- * Interface functions * --- 104,109 ---- *************** *** 143,154 **** * ======== * these functions test the transaction status of * a specified transaction id. ! * ! * TransactionIdCommit ! * TransactionIdAbort * ======== * these functions set the transaction status ! * of the specified xid. * * See also TransactionIdIsInProgress, which once was in this module * but now lives in procarray.c. --- 112,125 ---- * ======== * these functions test the transaction status of * a specified transaction id. ! * ! * TransactionIdCommitTree ! * TransactionIdAsyncCommitTree ! * TransactionIdAbortTree * ======== * these functions set the transaction status ! * of the specified transaction tree. As of 8.4, these ! * are now atomic so we set the whole tree in a single call. * * See also TransactionIdIsInProgress, which once was in this module * but now lives in procarray.c. *************** *** 287,374 **** return false; } - - /* - * TransactionIdCommit - * Commits the transaction associated with the identifier. - * - * Note: - * Assumes transaction identifier is valid. - */ - void - TransactionIdCommit(TransactionId transactionId) - { - TransactionLogUpdate(transactionId, TRANSACTION_STATUS_COMMITTED, - InvalidXLogRecPtr); - } - - /* - * TransactionIdAsyncCommit - * Same as above, but for async commits. The commit record LSN is needed. - */ - void - TransactionIdAsyncCommit(TransactionId transactionId, XLogRecPtr lsn) - { - TransactionLogUpdate(transactionId, TRANSACTION_STATUS_COMMITTED, lsn); - } - - /* - * TransactionIdAbort - * Aborts the transaction associated with the identifier. - * - * Note: - * Assumes transaction identifier is valid. - * No async version of this is needed. - */ - void - TransactionIdAbort(TransactionId transactionId) - { - TransactionLogUpdate(transactionId, TRANSACTION_STATUS_ABORTED, - InvalidXLogRecPtr); - } - - /* - * TransactionIdSubCommit - * Marks the subtransaction associated with the identifier as - * sub-committed. - * - * Note: - * No async version of this is needed. - */ - void - TransactionIdSubCommit(TransactionId transactionId) - { - TransactionLogUpdate(transactionId, TRANSACTION_STATUS_SUB_COMMITTED, - InvalidXLogRecPtr); - } - /* * TransactionIdCommitTree ! * Marks all the given transaction ids as committed. ! * ! * The caller has to be sure that this is used only to mark subcommitted ! * subtransactions as committed, and only *after* marking the toplevel ! * parent as committed. Otherwise there is a race condition against ! * TransactionIdDidCommit. */ void ! TransactionIdCommitTree(int nxids, TransactionId *xids) { ! if (nxids > 0) ! TransactionLogMultiUpdate(nxids, xids, TRANSACTION_STATUS_COMMITTED, ! InvalidXLogRecPtr); } /* * TransactionIdAsyncCommitTree ! * Same as above, but for async commits. The commit record LSN is needed. */ void ! TransactionIdAsyncCommitTree(int nxids, TransactionId *xids, XLogRecPtr lsn) { ! if (nxids > 0) ! TransactionLogMultiUpdate(nxids, xids, TRANSACTION_STATUS_COMMITTED, ! lsn); } /* --- 258,284 ---- return false; } /* * TransactionIdCommitTree ! * Marks all the given transaction ids as committed, atomically. */ void ! TransactionIdCommitTree(TransactionId xid, int nxids, TransactionId *xids) { ! return TransactionIdSetTreeStatus(xid, nxids, xids, ! TRANSACTION_STATUS_COMMITTED, InvalidXLogRecPtr); } /* * TransactionIdAsyncCommitTree ! * Same as above, but for async commits, atomically. The commit record ! * LSN is needed. */ void ! TransactionIdAsyncCommitTree(TransactionId xid, int nxids, TransactionId *xids, XLogRecPtr lsn) { ! return TransactionIdSetTreeStatus(xid, nxids, xids, ! TRANSACTION_STATUS_COMMITTED, lsn); } /* *************** *** 379,392 **** * will consider all the xacts as not-yet-committed anyway. */ void ! TransactionIdAbortTree(int nxids, TransactionId *xids) { ! if (nxids > 0) ! TransactionLogMultiUpdate(nxids, xids, TRANSACTION_STATUS_ABORTED, ! InvalidXLogRecPtr); } - /* * TransactionIdPrecedes --- is id1 logically < id2? */ --- 289,300 ---- * will consider all the xacts as not-yet-committed anyway. */ void ! TransactionIdAbortTree(TransactionId xid, int nxids, TransactionId *xids) { ! TransactionIdSetTreeStatus(xid, nxids, xids, ! TRANSACTION_STATUS_ABORTED, InvalidXLogRecPtr); } /* * TransactionIdPrecedes --- is id1 logically < id2? */ Index: src/backend/access/transam/twophase.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/twophase.c,v retrieving revision 1.45 diff -c -r1.45 twophase.c *** src/backend/access/transam/twophase.c 11 Aug 2008 11:05:10 -0000 1.45 --- src/backend/access/transam/twophase.c 24 Sep 2008 17:33:23 -0000 *************** *** 1745,1753 **** XLogFlush(recptr); /* Mark the transaction committed in pg_clog */ ! TransactionIdCommit(xid); ! /* to avoid race conditions, the parent must commit first */ ! TransactionIdCommitTree(nchildren, children); /* Checkpoint can proceed now */ MyProc->inCommit = false; --- 1745,1751 ---- XLogFlush(recptr); /* Mark the transaction committed in pg_clog */ ! TransactionIdCommitTree(xid, nchildren, children); /* Checkpoint can proceed now */ MyProc->inCommit = false; *************** *** 1822,1829 **** * Mark the transaction aborted in clog. This is not absolutely necessary * but we may as well do it while we are here. */ ! TransactionIdAbort(xid); ! TransactionIdAbortTree(nchildren, children); END_CRIT_SECTION(); } --- 1820,1826 ---- * Mark the transaction aborted in clog. This is not absolutely necessary * but we may as well do it while we are here. */ ! TransactionIdAbortTree(xid, nchildren, children); END_CRIT_SECTION(); } Index: src/backend/access/transam/xact.c =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/xact.c,v retrieving revision 1.265 diff -c -r1.265 xact.c *** src/backend/access/transam/xact.c 11 Aug 2008 11:05:10 -0000 1.265 --- src/backend/access/transam/xact.c 24 Sep 2008 17:33:23 -0000 *************** *** 254,260 **** static TransactionId RecordTransactionAbort(bool isSubXact); static void StartTransaction(void); - static void RecordSubTransactionCommit(void); static void StartSubTransaction(void); static void CommitSubTransaction(void); static void AbortSubTransaction(void); --- 254,259 ---- *************** *** 952,962 **** * Now we may update the CLOG, if we wrote a COMMIT record above */ if (markXidCommitted) ! { ! TransactionIdCommit(xid); ! /* to avoid race conditions, the parent must commit first */ ! TransactionIdCommitTree(nchildren, children); ! } } else { --- 951,957 ---- * Now we may update the CLOG, if we wrote a COMMIT record above */ if (markXidCommitted) ! TransactionIdCommitTree(xid, nchildren, children); } else { *************** *** 974,984 **** * flushed before the CLOG may be updated. */ if (markXidCommitted) ! { ! TransactionIdAsyncCommit(xid, XactLastRecEnd); ! /* to avoid race conditions, the parent must commit first */ ! TransactionIdAsyncCommitTree(nchildren, children, XactLastRecEnd); ! } } /* --- 969,975 ---- * flushed before the CLOG may be updated. */ if (markXidCommitted) ! TransactionIdAsyncCommitTree(xid, nchildren, children, XactLastRecEnd); } /* *************** *** 1156,1191 **** s->maxChildXids = 0; } - /* - * RecordSubTransactionCommit - */ - static void - RecordSubTransactionCommit(void) - { - TransactionId xid = GetCurrentTransactionIdIfAny(); - - /* - * We do not log the subcommit in XLOG; it doesn't matter until the - * top-level transaction commits. - * - * We must mark the subtransaction subcommitted in the CLOG if it had a - * valid XID assigned. If it did not, nobody else will ever know about - * the existence of this subxact. We don't have to deal with deletions - * scheduled for on-commit here, since they'll be reassigned to our parent - * (who might still abort). - */ - if (TransactionIdIsValid(xid)) - { - /* XXX does this really need to be a critical section? */ - START_CRIT_SECTION(); - - /* Record subtransaction subcommit */ - TransactionIdSubCommit(xid); - - END_CRIT_SECTION(); - } - } - /* ---------------------------------------------------------------- * AbortTransaction stuff * ---------------------------------------------------------------- --- 1147,1152 ---- *************** *** 1288,1301 **** * waiting for already-aborted subtransactions. It is OK to do it without * having flushed the ABORT record to disk, because in event of a crash * we'd be assumed to have aborted anyway. - * - * The ordering here isn't critical but it seems best to mark the parent - * first. This assures an atomic transition of all the subtransactions to - * aborted state from the point of view of concurrent - * TransactionIdDidAbort calls. */ ! TransactionIdAbort(xid); ! TransactionIdAbortTree(nchildren, children); END_CRIT_SECTION(); --- 1249,1256 ---- * waiting for already-aborted subtransactions. It is OK to do it without * having flushed the ABORT record to disk, because in event of a crash * we'd be assumed to have aborted anyway. */ ! TransactionIdAbortTree(xid, nchildren, children); END_CRIT_SECTION(); *************** *** 3791,3798 **** /* Must CCI to ensure commands of subtransaction are seen as done */ CommandCounterIncrement(); ! /* Mark subtransaction as subcommitted */ ! RecordSubTransactionCommit(); /* Post-commit cleanup */ if (TransactionIdIsValid(s->transactionId)) --- 3746,3757 ---- /* Must CCI to ensure commands of subtransaction are seen as done */ CommandCounterIncrement(); ! /* ! * Prior to 8.4 we marked subcommit in clog at this point. ! * We now only perform that step, if required, as part of the ! * atomic update of the whole transaction tree at top level ! * commit or abort. ! */ /* Post-commit cleanup */ if (TransactionIdIsValid(s->transactionId)) *************** *** 4259,4269 **** TransactionId max_xid; int i; - TransactionIdCommit(xid); - /* Mark committed subtransactions as committed */ sub_xids = (TransactionId *) &(xlrec->xnodes[xlrec->nrels]); ! TransactionIdCommitTree(xlrec->nsubxacts, sub_xids); /* Make sure nextXid is beyond any XID mentioned in the record */ max_xid = xid; --- 4218,4226 ---- TransactionId max_xid; int i; /* Mark committed subtransactions as committed */ sub_xids = (TransactionId *) &(xlrec->xnodes[xlrec->nrels]); ! TransactionIdCommitTree(xid, xlrec->nsubxacts, sub_xids); /* Make sure nextXid is beyond any XID mentioned in the record */ max_xid = xid; *************** *** 4299,4309 **** TransactionId max_xid; int i; - TransactionIdAbort(xid); - /* Mark subtransactions as aborted */ sub_xids = (TransactionId *) &(xlrec->xnodes[xlrec->nrels]); ! TransactionIdAbortTree(xlrec->nsubxacts, sub_xids); /* Make sure nextXid is beyond any XID mentioned in the record */ max_xid = xid; --- 4256,4264 ---- TransactionId max_xid; int i; /* Mark subtransactions as aborted */ sub_xids = (TransactionId *) &(xlrec->xnodes[xlrec->nrels]); ! TransactionIdAbortTree(xid, xlrec->nsubxacts, sub_xids); /* Make sure nextXid is beyond any XID mentioned in the record */ max_xid = xid; Index: src/include/access/clog.h =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/include/access/clog.h,v retrieving revision 1.21 diff -c -r1.21 clog.h *** src/include/access/clog.h 1 Jan 2008 19:45:56 -0000 1.21 --- src/include/access/clog.h 24 Sep 2008 17:33:23 -0000 *************** *** 32,38 **** #define NUM_CLOG_BUFFERS 8 ! extern void TransactionIdSetStatus(TransactionId xid, XidStatus status, XLogRecPtr lsn); extern XidStatus TransactionIdGetStatus(TransactionId xid, XLogRecPtr *lsn); extern Size CLOGShmemSize(void); --- 32,39 ---- #define NUM_CLOG_BUFFERS 8 ! extern void TransactionIdSetTreeStatus(TransactionId xid, int nsubxids, ! TransactionId *subxids, XidStatus status, XLogRecPtr lsn); extern XidStatus TransactionIdGetStatus(TransactionId xid, XLogRecPtr *lsn); extern Size CLOGShmemSize(void); Index: src/include/access/transam.h =================================================================== RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/include/access/transam.h,v retrieving revision 1.65 diff -c -r1.65 transam.h *** src/include/access/transam.h 11 Mar 2008 20:20:35 -0000 1.65 --- src/include/access/transam.h 24 Sep 2008 17:33:23 -0000 *************** *** 139,151 **** extern bool TransactionIdDidCommit(TransactionId transactionId); extern bool TransactionIdDidAbort(TransactionId transactionId); extern bool TransactionIdIsKnownCompleted(TransactionId transactionId); - extern void TransactionIdCommit(TransactionId transactionId); - extern void TransactionIdAsyncCommit(TransactionId transactionId, XLogRecPtr lsn); extern void TransactionIdAbort(TransactionId transactionId); ! extern void TransactionIdSubCommit(TransactionId transactionId); ! extern void TransactionIdCommitTree(int nxids, TransactionId *xids); ! extern void TransactionIdAsyncCommitTree(int nxids, TransactionId *xids, XLogRecPtr lsn); ! extern void TransactionIdAbortTree(int nxids, TransactionId *xids); extern bool TransactionIdPrecedes(TransactionId id1, TransactionId id2); extern bool TransactionIdPrecedesOrEquals(TransactionId id1, TransactionId id2); extern bool TransactionIdFollows(TransactionId id1, TransactionId id2); --- 139,148 ---- extern bool TransactionIdDidCommit(TransactionId transactionId); extern bool TransactionIdDidAbort(TransactionId transactionId); extern bool TransactionIdIsKnownCompleted(TransactionId transactionId); extern void TransactionIdAbort(TransactionId transactionId); ! extern void TransactionIdCommitTree(TransactionId xid, int nxids, TransactionId *xids); ! extern void TransactionIdAsyncCommitTree(TransactionId xid, int nxids, TransactionId *xids, XLogRecPtr lsn); ! extern void TransactionIdAbortTree(TransactionId xid, int nxids, TransactionId *xids); extern bool TransactionIdPrecedes(TransactionId id1, TransactionId id2); extern bool TransactionIdPrecedesOrEquals(TransactionId id1, TransactionId id2); extern bool TransactionIdFollows(TransactionId id1, TransactionId id2);

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: hash index improving v3

2008-09-24T09:22:54Z

On Wed, 2008-09-24 at 12:04 -0400, Bruce Momjian wrote:

> Can we consider this hash thread closed/completed?

Please

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: hash index improving v3

2008-09-24T09:04:22Z

Can we consider this hash thread closed/completed?

---------------------------------------------------------------------------

Tom Lane wrote:

> Simon Riggs <simon@...> writes:
> > Thinks: Why not just sort all of the time and skip the debate entirely?
>
> The sort is demonstrably a loser for smaller indexes. Admittedly,
> if the index is small then the sort can't cost all that much, but if
> the (correct) threshold is some large fraction of shared_buffers then
> it could still take awhile on installations with lots-o-buffers.
>
> The other side of that coin is that it's not clear this is really worth
> arguing about, much less exposing a separate parameter for.
>
> regards, tom lane
>
> --
> Sent via pgsql-patches mailing list (pgsql-patches@...)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-patches

Re: Solve a problem of LC_TIME of windows.

2008-09-24T07:55:18Z

Hi.

----- Original Message -----
From: "Alvaro Herrera" <alvherre@...>
> What about this line?
>
> #define STRLEN_MAX 255
>
> Are we really sure the strftime format cannot be longer than that?

Ahh, although the place to replace is here, is it said that MAX_L10 N_DATA is suitable?
--
cache_locale_time(void)
{
..
char buf[MAX_L10N_DATA];
..
strftime(buf, MAX_L10N_DATA, "%a", timeinfo);

Regards,
Hiroshi Saito

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: Solve a problem of LC_TIME of windows.

2008-09-24T07:55:10Z

Hi.

----- Original Message -----
From: "Magnus Hagander" <magnus@...>

> In principle, I think this patch looks good.
>
> Do you (or somebody else) have an example where this breaks in an
> encoding where I can actually understand the characters, though ;-) That
> would make testing a whole lot easier...
>
> Also, the patch needs error checking. strftime() can fail, and the
> multibyte conversion functions can certainly fail. That will need to be
> added.

Ok, thanks!

strftime return to 0.
http://msdn.microsoft.com/en-us/library/fe06s4ak(VS.71).aspx
MultiByteToWideChar and WideCharToMultiByte return to GetLastError.
http://msdn.microsoft.com/en-us/library/ms776413(VS.85).aspx

I will proposal the next patch.:-)

BTW, this is SQL for a check.
http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/LC_TIME_PATCH/DATECHECK.sql
Probably, all are included.
http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/LC_TIME_PATCH/LC_TIME_CHECK_LOCALE.sql

Regards,
Hiroshi Saito

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: Solve a problem of LC_TIME of windows.

2008-09-24T07:15:19Z

Magnus Hagander wrote:

> Also, the patch needs error checking. strftime() can fail, and the
> multibyte conversion functions can certainly fail. That will need to be
> added.

What about this line?

#define STRLEN_MAX 255

Are we really sure the strftime format cannot be longer than that?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [HACKERS] Subtransaction commits and Hot Standby

2008-09-24T05:48:45Z

On Tue, 2008-09-23 at 22:47 +0100, Simon Riggs wrote:

> I've tested this some more and am much happier with it now.

The concept is fine, but I've found a coding bug in further testing.
Please wait now for new version before review.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: Solve a problem of LC_TIME of windows.

2008-09-24T02:11:41Z

Hiroshi Saito wrote:

> Hi.
>
> I have problem of LC_TIME of windows.(CVS-HEAD)
>
> As for Version 8.3.3. It is edited by wrong gettext and is. (But, it is
> expressed.)
> http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/LC_TIME_PATCH/pg8.3.3-to_char_gettext_format.png
>
>
> As for Version 8.4. It came to be used by Tom-san in strftime of
> Native-windowsAPI.
> It is good improvement.! But, strftime of Native returns a result by
> CODEPAGE of
> environment of operation by Windows with LC_TIME. In Japanese
> environment, return
> value is SJIS(CP932). Then, SJIS(CP932) can't be chosen by
> SERVER_ENCODING.:-(
> http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/pg84beta-03-to_char.png
>
> Then, I'm proposal patch. It is solved splendidly.
> http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/LC_TIME_PATCH/DATECHECK_EUCJP.txt
>
> http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/LC_TIME_PATCH/DATECHECK_UTF8.txt

In principle, I think this patch looks good.

Do you (or somebody else) have an example where this breaks in an
encoding where I can actually understand the characters, though ;-) That
would make testing a whole lot easier...

Also, the patch needs error checking. strftime() can fail, and the
multibyte conversion functions can certainly fail. That will need to be
added.

//Magnus

--
Sent via pgsql-patches mailing list (pgsql-patches@...)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches