Bug#554349: apt: Method rred has died unexpectedly!

[Faheem Mitha]

Package: apt
Version: 0.7.20.2+lenny1
Severity: normal


I am currently getting the following. My guess is a malformed pdiff
file. A couple of other people of #debian (oftc) have been seeing
similar errors. Possibly related to
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=545694 but this is
supposedly fixed on 0.7.24 and one of my co-bug-reporters can
reproduce this on sid.

Note the same pdiff, 2009-11-04-0016.39.pdiff, occurs before each
error.

This is the first joint bug report I've filed, and I've been using
Debian and filing bug reports since 2001. :-)

                                                   Regards, Faheem.

My error

Get:8 http://debian.csail.mit.edu unstable/main
2009-11-04-0016.39.pdiff [9307kB]
Ign http://people.debian.org stable/enslyon Packages/DiffIndex
Ign http://people.debian.org stable/main Sources/DiffIndex
Ign http://people.debian.org stable/enslyon Sources/DiffIndex
Hit http://jonas.genannt.name lenny/restricted Packages
Hit http://security.debian.org squeeze/updates/non-free Sources
Hit http://www.backports.org lenny-backports/main Packages
Ign http://people.debian.org stable/main Packages
Hit http://www.backports.org lenny-backports/contrib Packages
Ign http://people.debian.org stable/enslyon Packages
Ign http://people.debian.org stable/main Sources
Hit http://www.backports.org lenny-backports/non-free Packages
Ign http://people.debian.org stable/enslyon Sources
Hit http://people.debian.org stable/main Packages
Hit http://people.debian.org stable/enslyon Packages
Hit http://people.debian.org stable/main Sources
Hit http://people.debian.org stable/enslyon Sources
Get:9 http://debian.csail.mit.edu squeeze/main
2009-11-04-0016.39.pdiff [10.2MB]
Get:10 http://debian.csail.mit.edu squeeze/non-free
2009-11-04-0016.39.pdiff [139kB]
Get:11 http://debian.csail.mit.edu squeeze/contrib
2009-11-04-0016.39.pdiff [66.8kB]
Get:12 http://debian.csail.mit.edu squeeze/main
2009-11-04-0016.39.pdiff [4844kB]
E: Method rred has died unexpectedly!

from cockroach (stefan at ott dot net) on #debian on oftc (running sid - 0.7.24)

$ sudo apt-get update
Hit ftp://ftp.ch.debian.org sid Release.gpg
Get:1 ftp://ftp.ch.debian.org sid/main Translation-en_US
Ign ftp://ftp.ch.debian.org sid/main Translation-en_US
Get:2 ftp://ftp.ch.debian.org sid/contrib Translation-en_US
Ign ftp://ftp.ch.debian.org sid/contrib Translation-en_US
Get:3 ftp://ftp.ch.debian.org sid/non-free Translation-en_US
Ign ftp://ftp.ch.debian.org sid/non-free Translation-en_US
Hit ftp://ftp.ch.debian.org sid Release
Hit ftp://ftp.ch.debian.org sid/main Packages/DiffIndex
Hit ftp://ftp.ch.debian.org sid/contrib Packages/DiffIndex
Hit ftp://ftp.ch.debian.org sid/non-free Packages/DiffIndex
Get:4 ftp://ftp.ch.debian.org sid/main 2009-11-02-2358.21.pdiff [37.2kB]
Get:5 ftp://ftp.ch.debian.org sid/main 2009-11-02-2358.21.pdiff [37.2kB]
Get:6 ftp://ftp.ch.debian.org sid/contrib 2009-11-04-0016.39.pdiff
[70.1kB]
Get:7 ftp://ftp.ch.debian.org sid/contrib 2009-11-04-0016.39.pdiff
[70.1kB]
Get:8 ftp://ftp.ch.debian.org sid/main 2009-11-04-0016.39.pdiff
[9,274kB]
Get:9 ftp://ftp.ch.debian.org sid/main 2009-11-04-0016.39.pdiff
[9,274kB]
Get:10 ftp://ftp.ch.debian.org sid/main 2009-11-04-0016.39.pdiff
[9,274kB]
E: Method rred has died unexpectedly!
E: Sub-process rred received a segmentation fault.

from Hideo on #debian on oftc

Hit http://www.debian-multimedia.org unstable/main Packages/DiffIndex
Hit http://ftp.us.debian.org stable/contrib Sources
Hit http://ftp.us.debian.org stable/non-free Sources
Hit http://www.debian-multimedia.org unstable/main Sources/DiffIndex
Get:1 http://ftp.us.debian.org testing/main 2009-11-04-0016.39.pdiff
[4,844kB]
Get:2 http://ftp.us.debian.org testing/main 2009-11-04-0016.39.pdiff
[4,844kB]
Get:3 http://ftp.us.debian.org unstable/main 2009-11-04-0016.39.pdiff
[9,307kB]
Get:4 http://ftp.us.debian.org unstable/non-free
2009-11-04-0016.39.pdiff [86.4kB]
Get:5 http://ftp.us.debian.org experimental/main
2009-11-04-0016.39.pdiff [307kB]
Get:6 http://ftp.us.debian.org testing/main 2009-11-04-0016.39.pdiff
[4,844kB]
Get:7 http://ftp.us.debian.org unstable/main 2009-11-04-0016.39.pdiff
[9,307kB]
E: Method rred has died unexpectedly!
E: Sub-process rred received a segmentation fault.

-- Package-specific info:

-- apt-config dump --

APT "";
APT::Architecture "i386";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Acquire "";
APT::Acquire::Translation "environment";
APT::Authentication "";
APT::Authentication::TrustCDROM "true";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^linux-image.*";
APT::NeverAutoRemove:: "^linux-restricted-modules.*";
APT::Cache-Limit "56777216";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::userstatus "status.user";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
DPkg "";
DPkg::Pre-Invoke "";
DPkg::Pre-Invoke:: "if [ -x /usr/sbin/etckeeper ]; then etckeeper pre-install; fi";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "if [ -x /usr/sbin/etckeeper ]; then etckeeper post-install; fi";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Tools "";
DPkg::Tools::Options "";
DPkg::Tools::Options::/usr/bin/apt-listchanges "";
DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
RPM "";
RPM::Pre-Invoke "";
RPM::Pre-Invoke:: "if [ -x /usr/sbin/etckeeper ]; then etckeeper pre-install; fi";
RPM::Post-Invoke "";
RPM::Post-Invoke:: "if [ -x /usr/sbin/etckeeper ]; then etckeeper post-install; fi";

-- /etc/apt/preferences --

Package: *
Pin: release a=testing
Pin-Priority: 50

Package: *
Pin: release a=squeeze
Pin-Priority: 50

Package: *
Pin: release a=unstable
Pin-Priority: 50

Package: *
Pin: release a=experimental
Pin-Priority: 1

Package: *
Pin: release a=snapshot.debian.net
Pin-Priority: 1

-- /etc/apt/sources.list --

#deb http://debian.csail.mit.edu/debian/ sarge main non-free contrib
#deb-src http://debian.csail.mit.edu/debian/ sarge main non-free contrib

#deb http://debian.csail.mit.edu/debian/ etch main non-free contrib
#deb-src http://debian.csail.mit.edu/debian/ etch main non-free contrib

#deb http://security.debian.org/ etch/updates main contrib non-free
#deb-src http://security.debian.org/ etch/updates main contrib non-free

deb http://debian.csail.mit.edu/debian/ lenny main non-free contrib
deb-src http://debian.csail.mit.edu/debian/ lenny main non-free contrib

deb http://security.debian.org/ lenny/updates main contrib non-free
deb-src http://security.debian.org/ lenny/updates main contrib non-free

deb http://debian.csail.mit.edu/debian/ squeeze main non-free contrib
deb-src http://debian.csail.mit.edu/debian/ squeeze main non-free contrib

deb http://security.debian.org/ squeeze/updates main contrib non-free
deb-src http://security.debian.org/ squeeze/updates main contrib non-free

deb http://debian.csail.mit.edu/debian/ unstable main non-free contrib
deb-src http://debian.csail.mit.edu/debian/ unstable main non-free contrib

deb http://debian.csail.mit.edu/debian/ experimental main non-free contrib
deb-src http://debian.csail.mit.edu/debian/ experimental main non-free contrib

#deb http://snapshot.debian.net/archive pool linux-2.6
#deb http://snapshot.debian.net/archive pool gcc-4.0
#deb http://snapshot.debian.net/archive pool xen-3.0

deb http://www.debian-multimedia.org lenny main

deb http://www.backports.org/debian/ lenny-backports main contrib non-free

#deb http://www.prato.linux.it/~mnencia/debian nvidia/
#deb-src http://www.prato.linux.it/~mnencia/debian nvidia/

#deb http://cran.r-project.org/bin/linux/debian etch/

#deb http://galen.dulci.org/debian etch main
#deb-src http://galen.dulci.org/debian etch main
       
deb http://people.debian.org/~vdanjean/debian stable main enslyon
deb-src http://people.debian.org/~vdanjean/debian stable main enslyon

deb http://download.skype.com/linux/repos/debian/ stable non-free

deb http://jonas.genannt.name/debian lenny restricted

#deb http://people.debian.org/~preining/TeX/ tl2007/
#deb http://people.debian.org/~preining/TeX/ context/
#deb http://people.debian.org/~preining/TeX/ lmodern/

-- System Information:
Debian Release: 5.0.3
  APT prefers stable
  APT policy: (500, 'stable'), (50, 'unstable'), (50, 'testing'), (1, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.26-2-vserver-686-bigmem (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages apt depends on:
ii  debian-archive-keyring       2009.01.31  GnuPG archive keys of the Debian a
ii  libc6                        2.7-18      GNU C Library: Shared libraries
ii  libgcc1                      1:4.3.2-1.1 GCC support library
ii  libstdc++6                   4.3.2-1.1   The GNU Standard C++ Library v3

apt recommends no packages.

Versions of packages apt suggests:
pn  apt-doc               <none>             (no description available)
ii  aptitude              0.4.11.11-1~lenny1 terminal-based package manager
ii  bzip2                 1.0.5-1            high-quality block-sorting file co
ii  dpkg-dev              1.14.25            Debian package development tools
ii  lzma                  4.43-14            Compression method of 7z format in
ii  python-apt            0.7.7.1+nmu1       Python interface to libapt-pkg
ii  synaptic              0.62.1+nmu1        Graphical package manager

-- no debconf information



--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[David Kalnischkies-2]

tags 554349 + confirmed
thanks

Hello Faheem Mitha & all other co-bug-reporters :)

First of all: Thanks for the report!

I have currently no time to dig into it a bit deeper,
(hopefully i can do it tonight or tomorrow)
but my first guess is an overflow of an internal variable:
apt-get update -o Debug::pkgAcquire::RRed=1 -qq
results in 65,442 lines of output which is near the value
of a signed integer (65,535).

A patch of this size (~9 MB) is a bit uncommon as it
is bigger than the complete download of the new file.
It would be good if APT could detect it... we will see. :)

As a workaround you can disable pdiffs temporary with
apt-get update -o Acquire::Pdiffs=false



Best regards / Mit freundlichen Grüßen,

David "DonKult" Kalnischkies



--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Michael Vogt-5]

On Tue, Nov 03, 2009 at 11:21:48PM -0500, Faheem Mitha wrote:

> Package: apt
> Version: 0.7.20.2+lenny1
> Severity: normal
>
>
> I am currently getting the following. My guess is a malformed pdiff
> file. A couple of other people of #debian (oftc) have been seeing
> similar errors. Possibly related to
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=545694 but this is
> supposedly fixed on 0.7.24 and one of my co-bug-reporters can
> reproduce this on sid.
>
> Note the same pdiff, 2009-11-04-0016.39.pdiff, occurs before each
> error.
>
> This is the first joint bug report I've filed, and I've been using
> Debian and filing bug reports since 2001. :-)

Thanks for your bugreport.

I can reproduce the failure here and I'm looking into the problem
now. As a workaround you can run:

# apt-get update -o Acquire::PDiffs=false

thanks,
 Michael



--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Michael Vogt-5]

It turns out that the unusual size of the diff causes a the stack to
overflow, a different workaround is "ulimit -s 12288" (or something
like that). Clearly its still a bug in rred.cc.



--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Julian Gilbey]

On Wed, Nov 04, 2009 at 11:54:22AM +0100, Michael Vogt wrote:
> Thanks for your bugreport.
>
> I can reproduce the failure here and I'm looking into the problem
> now. As a workaround you can run:
>
> # apt-get update -o Acquire::PDiffs=false

Looks like the diff file is corrupt; applying it manually to the
Packages file using ed leaves an invalid Packages file.

   Julian



--
To UNSUBSCRIBE, email to debian-bugs-rc-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Michael Vogt-9]

On Wed, Nov 04, 2009 at 11:12:25AM +0100, David Kalnischkies wrote:

> tags 554349 + confirmed
> thanks
>
> Hello Faheem Mitha & all other co-bug-reporters :)
>
> First of all: Thanks for the report!
>
> I have currently no time to dig into it a bit deeper,
> (hopefully i can do it tonight or tomorrow)
> but my first guess is an overflow of an internal variable:
> apt-get update -o Debug::pkgAcquire::RRed=1 -qq
> results in 65,442 lines of output which is near the value
> of a signed integer (65,535).
>
> A patch of this size (~9 MB) is a bit uncommon as it
> is bigger than the complete download of the new file.
> It would be good if APT could detect it... we will see. :)

[..]

Attached is a patch that should add detection for patches that are
bigger than the actual file to patch. Unfortuantely I can no longer
reproduce the failure. Testing/feedback is welcome.

Cheers,
 Michael

---

=== modified file 'apt-pkg/acquire-item.cc'
--- apt-pkg/acquire-item.cc 2009-06-30 19:37:30 +0000
+++ apt-pkg/acquire-item.cc 2009-11-04 17:07:24 +0000
@@ -247,7 +247,7 @@
     std::clog << "SHA1-Current: " << ServerSha1 << std::endl;
 
  // check the historie and see what patches we need
- string history = Tags.FindS("SHA1-History");    
+ string history = Tags.FindS("SHA1-History");
  std::stringstream hist(history);
  while(hist >> d.sha1 >> size >> d.file)
  {
@@ -258,6 +258,16 @@
     // from that point on, we probably need all diffs
     if(found)
     {
+       string sha1, size, file;
+       std::stringstream patches(Tags.FindS("SHA1-Patches"));
+       while(patches >> sha1 >> size >> file)  {
+  if (sha1 == d.sha1 && atoi(size.c_str()) > fd.Size()) {
+     std::clog << "patch size bigger than package file size?!?" << std::endl;
+     return false;
+  }
+       }
+    
+
        if(Debug)
   std::clog << "Need to get diff: " << d.file << std::endl;
        available_patches.push_back(d);

=== modified file 'apt-pkg/acquire-item.h'
--- apt-pkg/acquire-item.h 2009-06-30 19:37:30 +0000
+++ apt-pkg/acquire-item.h 2009-11-04 16:55:25 +0000
@@ -267,10 +267,10 @@
    /** The filename of the diff. */
    string file;
 
-   /** The sha1 hash of the diff. */
+   /** The sha1 hash of the patched file. */
    string sha1;
 
-   /** The size of the diff. */
+   /** The size of the patched file. */
    unsigned long size;
 };
  /*}}}*/

=== modified file 'debian/changelog'
--- debian/changelog 2009-10-29 10:53:01 +0000
+++ debian/changelog 2009-11-04 14:56:53 +0000
@@ -9,6 +9,8 @@
   * apt-pkg/packagemanager.cc:
     - add output about pre-depends configuring when debug::pkgPackageManager
       is used
+  * apt-pkg/acquire-item.cc:
+    - ignore patches if the patch is larger than the file
 
   [ Brian Murray ]
   * apt-pkg/depcache.cc, apt-pkg/indexcopy.cc:

---

[Faheem Mitha]

Hi Michael, David,

Thanks for the prompt response and the patch. See below.

On Wed, 4 Nov 2009, Michael Vogt wrote:

> Attached is a patch that should add detection for patches that are
> bigger than the actual file to patch. Unfortuantely I can no longer
> reproduce the failure. Testing/feedback is welcome.

I can't reproduce this any longer, but would it not be possible to create
an 'artificial' apt repos to reproduce this behavior (before and after
the patch) since it seems you know what causes it?

                                                           Regards, Faheem.



--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...