Savannah is down v2

[Sylvain Beucler-2]

Hi,

Savannah was unreachable again, with the same errors:

[50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=54200245, block=108396607
[50919.317501] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=60835372, block=121667635
[50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=60835372, block=121667635
[50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=45304252, block=90603599
[50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=54200420, block=108396613
[50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=54200245, block=108396607
[50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=60835372, block=121667635
[50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=60835372, block=121667635
[50919.318106] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=45304252, block=90603599
[50919.318448] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=54200420, block=108396613
[50926.450881] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=48185395, block=96370691
[50926.450881] EXT3-fs error (device md3): ext3_get_inode_loc: unable to read in
ode block - inode=48185395, block=96370691


It doesn't reboot either:

   6144 MB Detected                                                            
                                                                               
ProLiant System BIOS - P29 (03/25/2003)                                        
Copyright 1982, 2003 Hewlett-Packard Development Group, L.P.                    
                                                                               
                                                                               
Processor 1 initialized at 2.80 GHz/400 MHz(512 Kbyte L2)                      
Processor 2 initialized at 2.80 GHz/400 MHz(512 Kbyte L2)                      
                                                                               
 207-Memory Configuration Warning - DIMM In DIMM Socket 1A.                    
     does not have Primary Width of 4 and only supports standard ECC.          
 207-Memory Configuration Warning - DIMM In DIMM Socket 2A.                    
     does not have Primary Width of 4 and only supports standard ECC.          
 207-Memory Configuration Warning - DIMM In DIMM Socket 3B.                    
     does not have Primary Width of 4 and only supports standard ECC.          
 207-Memory Configuration Warning - DIMM In DIMM Socket 4B.                    
     does not have Primary Width of 4 and only supports standard ECC.          
 207-Memory Configuration Warning - DIMM In DIMM Socket 5C.                    
     does not have Primary Width of 4 and only supports standard ECC.          
 207-Memory Configuration Warning - DIMM In DIMM Socket 6C.                    
     does not have Primary Width of 4 and only supports standard ECC.          
Advanced Memory Protection Mode: Advanced ECC Support                          
Redundant ROM Detected - This system contains a valid backup system ROM.      
                                                                               
                                                                               
602-Diskette Boot Record Error


Maybe it tries to boot on the USB drive, but the memory errors don't
look good.

--
Sylvain

[Ward Vandewege via RT]

> [beuc - Sat May 30 03:30:01 2009]:
>
> Hi,
>
> Savannah was unreachable again, with the same errors:
>
> [50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=54200245, block=108396607
> [50919.317501] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=60835372, block=121667635
> [50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=60835372, block=121667635
> [50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=45304252, block=90603599
> [50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=54200420, block=108396613
> [50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=54200245, block=108396607
> [50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=60835372, block=121667635
> [50919.316366] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=60835372, block=121667635
> [50919.318106] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=45304252, block=90603599
> [50919.318448] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=54200420, block=108396613
> [50926.450881] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=48185395, block=96370691
> [50926.450881] EXT3-fs error (device md3): ext3_get_inode_loc: unable
> to read in
> ode block - inode=48185395, block=96370691
>
>
> It doesn't reboot either:
>
>    6144 MB Detected
>
> ProLiant System BIOS - P29 (03/25/2003)
> Copyright 1982, 2003 Hewlett-Packard Development Group, L.P.
>
>
> Processor 1 initialized at 2.80 GHz/400 MHz(512 Kbyte L2)
> Processor 2 initialized at 2.80 GHz/400 MHz(512 Kbyte L2)
>
>  207-Memory Configuration Warning - DIMM In DIMM Socket 1A.
> >  does not have Primary Width of 4 and only supports standard ECC.
>  207-Memory Configuration Warning - DIMM In DIMM Socket 2A.
> >  does not have Primary Width of 4 and only supports standard ECC.
>  207-Memory Configuration Warning - DIMM In DIMM Socket 3B.
> >  does not have Primary Width of 4 and only supports standard ECC.
>  207-Memory Configuration Warning - DIMM In DIMM Socket 4B.
> >  does not have Primary Width of 4 and only supports standard ECC.
>  207-Memory Configuration Warning - DIMM In DIMM Socket 5C.
> >  does not have Primary Width of 4 and only supports standard ECC.
>  207-Memory Configuration Warning - DIMM In DIMM Socket 6C.
> >  does not have Primary Width of 4 and only supports standard ECC.

I saw that yesterday and looked around; from what I can tell this memory
is not exactly the right type for the machine. But since we have not
changed it, my guess is that it's been like this from day one, which
means it can't be *that* horrible. It's complaining about all six memory
slots, each of which have a 1GB stick in them I believe.

The manual says:

Error Code                      

207-Memory Configuration
Warning - DIMM In DIMM Socket
X does not have Primary Width of
4 and only supports standard
ECC.

Audible Possible Problem      
Installed DIMMs have
a primary width of x8.  

Possible Action
Install DIMMs that have a
primary width of x4.

Looks like getting the right kind of ram would be  

> Advanced Memory Protection Mode: Advanced ECC Support
> Redundant ROM Detected - This system contains a valid backup system
> ROM.
>
>
> 602-Diskette Boot Record Error
>
>
> Maybe it tries to boot on the USB drive,

Yes, this is a ridiculous bios bug. We need to disable diskette/usb
booting from the bios config screen, but you can't get there because
that only pops up after it gets beyond this point....

Thanks,
Ward.

--
Ward Vandewege <ward@...>
Free Software Foundation - Senior System Administrator

[Alex Fernandez]

Hi Sylvain,

On Sat, May 30, 2009 at 9:26 AM, Sylvain Beucler <beuc@...> wrote:
> Savannah was unreachable again, with the same errors:

Any news? I'm sure lots of people are anxious to do new releases.

Thanks,

Alex.

[Sylvain Beucler-2]

On Sat, May 30, 2009 at 11:42:20PM +0200, Alex Fernandez wrote:
> Hi Sylvain,
>
> On Sat, May 30, 2009 at 9:26 AM, Sylvain Beucler <beuc@...> wrote:
> > Savannah was unreachable again, with the same errors:
>
> Any news? I'm sure lots of people are anxious to do new releases.

Hi,

http://identi.ca/group/fsfstatus

Join #savannah for more, I can elaborate right now.

--
Sylvain

[Ward Vandewege via RT]

Update:

1. danny or peabo will bring disk with last backup to colo tomorrow,
sometime before noon EDT
2. it will be provided to savannah over nfs at lan speed
3. they will then also move the usb disk to another machine with usb2
(after checking in on #savannah to make sure no rsyncs/etc are running
anymore), and then you'll also get that over nfs at lan speed

I've ordered some adapters for SCA to 68 pin scsi which should hopefully
get to the office by Tuesday afternoon. We have a scsi controller that
we can use, so then we can take another stab at the old disks in the
disk array.

If you want, we can also give you some more disk space for savannah over
nfs, in the mean time. Just let us know.

--
Ward Vandewege <ward@...>
Free Software Foundation - Senior System Administrator

[Sylvain Beucler-2]

On Sun, May 31, 2009 at 11:02:19PM -0400, Ward Vandewege via RT wrote:

>
> Update:
>
> 1. danny or peabo will bring disk with last backup to colo tomorrow,
> sometime before noon EDT
> 2. it will be provided to savannah over nfs at lan speed
> 3. they will then also move the usb disk to another machine with usb2
> (after checking in on #savannah to make sure no rsyncs/etc are running
> anymore), and then you'll also get that over nfs at lan speed
>
> I've ordered some adapters for SCA to 68 pin scsi which should hopefully
> get to the office by Tuesday afternoon. We have a scsi controller that
> we can use, so then we can take another stab at the old disks in the
> disk array.
>
> If you want, we can also give you some more disk space for savannah over
> nfs, in the mean time. Just let us know.

We'll need space to host all of 'vcs-noshell', yes.
Our du's said it weighs ~41GB, we've got 39 free here.
CVS itself is 24GB afaicr.

Do we have more disks?

--
Sylvain

[Noah Slater-4]

On Sun, May 31, 2009 at 11:02:19PM -0400, Ward Vandewege via RT wrote:
> Update:

Is there any reason the headers for these recent emails omit the usual:

  savannah-hackers-public@...

My list filtering rules are getting very confused.

Thanks,

--
Noah Slater, http://tumbolia.org/nslater

[Sylvain Beucler-2]

On Mon, Jun 01, 2009 at 01:16:10PM +0100, Noah Slater wrote:
> On Sun, May 31, 2009 at 11:02:19PM -0400, Ward Vandewege via RT wrote:
> > Update:
>
> Is there any reason the headers for these recent emails omit the usual:
>
>   savannah-hackers-public@...
>
> My list filtering rules are getting very confused.

Yes, because I'm mail sysadmin@... which is an RT queue and
doesn't support Cc:s correctly. Hence I bounce the mail when I get it.

--
Sylvain

[Sylvain Beucler-2]

On Mon, Sep 28, 2009 at 12:16:05PM -0400, Daniel Clark via RT wrote:

> > [beuc - Mon Jun 01 04:09:41 2009]:
> >
> > On Sun, May 31, 2009 at 11:02:19PM -0400, Ward Vandewege via RT wrote:
> > >
> > > Update:
> > >
> > > 1. danny or peabo will bring disk with last backup to colo tomorrow,
> > > sometime before noon EDT
> > > 2. it will be provided to savannah over nfs at lan speed
> > > 3. they will then also move the usb disk to another machine with usb2
> > > (after checking in on #savannah to make sure no rsyncs/etc are running
> > > anymore), and then you'll also get that over nfs at lan speed
> > >
> > > I've ordered some adapters for SCA to 68 pin scsi which should hopefully
> > > get to the office by Tuesday afternoon. We have a scsi controller that
> > > we can use, so then we can take another stab at the old disks in the
> > > disk array.
> > >
> > > If you want, we can also give you some more disk space for savannah over
> > > nfs, in the mean time. Just let us know.
> >
> > We'll need space to host all of 'vcs-noshell', yes.
> > Our du's said it weighs ~41GB, we've got 39 free here.
> > CVS itself is 24GB afaicr.
> >
> > Do we have more disks?
>
> I believe all issues in this ticket were resolved via IRC or by the new
> machine with a crapload of disk you guys now have - please tell me if
> you think there is anthing more to do on this (and if so, specifically
> what it is). Thanks.

I think we can close this ticket.  The only remaining issue is a way
to modify the DNS "live" during a migration.

In anticipation the TTL could be set to 60s already.

--
Sylvain

[Sylvain Beucler-2]

On Wed, Sep 30, 2009 at 05:31:11PM -0400, Daniel Clark via RT wrote:

> > [beuc - Wed Sep 30 14:23:57 2009]:
> >
> > On Mon, Sep 28, 2009 at 12:16:05PM -0400, Daniel Clark via RT wrote:
> > > > [beuc - Mon Jun 01 04:09:41 2009]:
> > > >
> > > > On Sun, May 31, 2009 at 11:02:19PM -0400, Ward Vandewege via RT
> > wrote:
> > > > >
> > > > > Update:
> > > > >
> > > > > 1. danny or peabo will bring disk with last backup to colo
> > tomorrow,
> > > > > sometime before noon EDT
> > > > > 2. it will be provided to savannah over nfs at lan speed
> > > > > 3. they will then also move the usb disk to another machine with
> > usb2
> > > > > (after checking in on #savannah to make sure no rsyncs/etc are
> > running
> > > > > anymore), and then you'll also get that over nfs at lan speed
> > > > >
> > > > > I've ordered some adapters for SCA to 68 pin scsi which should
> > hopefully
> > > > > get to the office by Tuesday afternoon. We have a scsi
> > controller that
> > > > > we can use, so then we can take another stab at the old disks in
> > the
> > > > > disk array.
> > > > >
> > > > > If you want, we can also give you some more disk space for
> > savannah over
> > > > > nfs, in the mean time. Just let us know.
> > > >
> > > > We'll need space to host all of 'vcs-noshell', yes.
> > > > Our du's said it weighs ~41GB, we've got 39 free here.
> > > > CVS itself is 24GB afaicr.
> > > >
> > > > Do we have more disks?
> > >
> > > I believe all issues in this ticket were resolved via IRC or by the
> > new
> > > machine with a crapload of disk you guys now have - please tell me
> > if
> > > you think there is anthing more to do on this (and if so,
> > specifically
> > > what it is). Thanks.
> >
> > I think we can close this ticket.  The only remaining issue is a way
> > to modify the DNS "live" during a migration.
> >
> > In anticipation the TTL could be set to 60s already.
>
> It was already 300 seconds for gnu.org; we just set it to 300 seconds
> (down from a day) for nongnu.org; I think that should be sufficient, but
> if you really want less time just ping me on IRC 10min before and I can
> put it down to 60s temporarily.
>
> Also we should be sure to schedule the changeover to a time where FSF
> Sys Admins, you, and any other volunteers who want to help are available.

Ward suggested rsync'ing the zone from somewhere, so we could do it
without schedule, and possibly switch back and forth quickly if
there's a problem.

This is a non-trivial move, so flexibility is important.

--
Sylvain

[Sylvain Beucler-2]

On Thu, Oct 01, 2009 at 05:27:34PM -0400, Daniel Clark via RT wrote:

> As per IRC, just the ability to update these forward DNS records would
> be sufficient?
>
> savannah.gnu.org
> *.savannah.gnu.org
> savannah.nongnu.org
> *.savannah.nongnu.org
> sv.gnu.org
> *.sv.gnu.org
> sv.nongnu.org
> *.sv.nongnu.org

This sounds right.

There are also cvs.gnu.org, arch.gnu.org, etc., but those are CNAMEs,
so there's no need to update them :)

--
Sylvain

[Sylvain Beucler-2]

On Thu, Oct 01, 2009 at 11:26:31PM +0200, Sylvain Beucler wrote:

> On Thu, Oct 01, 2009 at 05:27:34PM -0400, Daniel Clark via RT wrote:
> > As per IRC, just the ability to update these forward DNS records would
> > be sufficient?
> >
> > savannah.gnu.org
> > *.savannah.gnu.org
> > savannah.nongnu.org
> > *.savannah.nongnu.org
> > sv.gnu.org
> > *.sv.gnu.org
> > sv.nongnu.org
> > *.sv.nongnu.org
>
> This sounds right.
>
> There are also cvs.gnu.org, arch.gnu.org, etc., but those are CNAMEs,
> so there's no need to update them :)

Btw, there's also audio-video.gnu.org but I suggest we don't worry
about it.

--
Sylvain

[Ward Vandewege via RT]

These subdomains are now all split off in separate zone files

  sv.nongnu.org
  savannah.nongnu.org
  sv.gnu.org
  savannah.gnu.org

Ns1/ns2 are configured to accept 140.186.70.68 as master for those
domains. Beuc's going to configure that box (savannah-backup.gnu.org, a
domU on another of our machines) with an authoritative nameserver.

I put initial zone files in /root/ on savannah, they should be used to
start out.

We're also still waiting on the ns4 folks to accept these new sub zones.
Once that is done I can remove the savannah/sv data from the gnu.org
zone. For nongnu.org, we only have ns1/ns2 listed as nameservers, so
those domains will be active once the new nameserver on .68 is up.

Thanks,
Ward.

--
Ward Vandewege <ward@...>
Free Software Foundation - Senior System Administrator