Compatibility of security mirror

[Lee Winter]

The security mirror at security.debian.org appears to have a structure
that is compatible with the main debian mirrors.  If that appearance
is an accurate reflection of reality then the updates/main/* tree
should be compatible with the main/* tree.  I use the term compatible
to mean that a local mirror could be composed of the main archive plus
the security updates and thus require a single entry in the client
machine's sources.list file.

Is that appearance accurate?  Can security update be stored in the
same directory tree and accessed by clients transparently?  If so, is
it possible to merge the updates/* folders into the main repository or
do they need to remain underneath updates/ ?

Thanks for any info or suggestions,

Lee Winter
NP Engineering
Nashua, New Hampshire


--
To UNSUBSCRIBE, email to debian-security-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Goswin von Brederlow-2]

Lee Winter <lee.j.i.winter@...> writes:

> The security mirror at security.debian.org appears to have a structure
> that is compatible with the main debian mirrors.  If that appearance
> is an accurate reflection of reality then the updates/main/* tree
> should be compatible with the main/* tree.  I use the term compatible
> to mean that a local mirror could be composed of the main archive plus
> the security updates and thus require a single entry in the client
> machine's sources.list file.
>
> Is that appearance accurate?  Can security update be stored in the
> same directory tree and accessed by clients transparently?  If so, is
> it possible to merge the updates/* folders into the main repository or
> do they need to remain underneath updates/ ?
>
> Thanks for any info or suggestions,
>
> Lee Winter
> NP Engineering
> Nashua, New Hampshire

Yes and no.

You can easily use reprepro to fetch packages from both ftp.debian.org
and security.debian.org and combine them into a single apt
repository. That works reasonably well with the exception of screwups
where different builds of the same source are uploaded to security and
stable-proposed-updates. This has happened a few times in the past and
can need a one time manual intervention in reprepro.

This has one minor drawback though: The combined apt repository will
be unsigned (you do not want to do that) or signed by a local
key. This has 2 effects:

  1) You need to "apt-key add" your local key on every client (or
     build a keyring deb and install that).
  2) The Debian-Installer will not work from the local mirror.
     You need to do the initial install from an official mirror or use
     a netinst or full CD1/DVD1 image. Netboot and business card
     images will not work.


You can not create 1:1 mirrors of ftp.debian.org and
security.debian.org in the same place because the Release and
Release.gpg files overlap. If you need a true mirror then you need to
keep them seperate and have 2 lines in sources.list.

MfG
        Goswin

PS: reprepro can also trivially handle locally compiled debs, e.g. in
a local/ section.


--
To UNSUBSCRIBE, email to debian-security-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Lee Winter]

On Wed, Sep 16, 2009 at 3:45 AM, Goswin von Brederlow <goswin-v-b@...> wrote:

> Lee Winter <lee.j.i.winter@...> writes:
>
>> The security mirror at security.debian.org appears to have a structure
>> that is compatible with the main debian mirrors.  If that appearance
>> is an accurate reflection of reality then the updates/main/* tree
>> should be compatible with the main/* tree.  I use the term compatible
>> to mean that a local mirror could be composed of the main archive plus
>> the security updates and thus require a single entry in the client
>> machine's sources.list file.
>>
>> Is that appearance accurate?  Can security update be stored in the
>> same directory tree and accessed by clients transparently?  If so, is
>> it possible to merge the updates/* folders into the main repository or
>> do they need to remain underneath updates/ ?
>
> Yes and no.
>
> You can easily use reprepro to fetch packages from both ftp.debian.org
> and security.debian.org and combine them into a single apt
> repository. That works reasonably well with the exception of screwups
> where different builds of the same source are uploaded to security and
> stable-proposed-updates. This has happened a few times in the past and
> can need a one time manual intervention in reprepro.

I'm using the updated debmirror, but the same issues would arise.

>
> This has one minor drawback though: The combined apt repository will
> be unsigned (you do not want to do that) or signed by a local
> key.

Why is that?  If there are no file collissions such as described
above, then it should be possible to have two logical repositories in
one file structure.  The issue is not how to integrate the
repositories.  The issue is how to simplify the sources.list files on
the client systems.  Right now the client's need three entries for
lenny, security, and volatile as if they came from separate mirrors.
I would like to keep the repositories independent but have one
"debian" section in sources.list.  Is that not possible?

> This has 2 effects:
>
>  1) You need to "apt-key add" your local key on every client (or
>     build a keyring deb and install that).
>  2) The Debian-Installer will not work from the local mirror.
>     You need to do the initial install from an official mirror or use
>     a netinst or full CD1/DVD1 image. Netboot and business card
>     images will not work.

This is a key issue for me because installs happen here often.  The
updated debmirror is supposed to accomplish that and it appears to be
getting the installer files into the mirror, but I have not yet gotten
it to work during an actual install (work in progress).

>
> You can not create 1:1 mirrors of ftp.debian.org and
> security.debian.org in the same place because the Release and
> Release.gpg files overlap.

Where?  AFAICT all of the paths are different.  E.g,, security inserts
"updates" so that "dists/stable/main" becomes
"dists/stable/updates/main".  Same for pool.  And volatile does the
same thing.

> If you need a true mirror then you need to
> keep them seperate and have 2 lines in sources.list.

OK, I can do that.  I just want a simple-minded file structure.

Thanks for the help.

-- Lee


--
To UNSUBSCRIBE, email to debian-security-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Russ Allbery-2]

Lee Winter <lee.j.i.winter@...> writes:
> Goswin von Brederlow <goswin-v-b@...> wrote:

>> This has one minor drawback though: The combined apt repository will be
>> unsigned (you do not want to do that) or signed by a local key.

> Why is that?

Because the package lists from the two separate repositories are
independently signed, and since you don't have access to the signing key,
there's no way to combine them into a single package list and still have a
valid signature without changing keys.

> Right now the client's need three entries for lenny, security, and
> volatile as if they came from separate mirrors.  I would like to keep
> the repositories independent but have one "debian" section in
> sources.list.  Is that not possible?

There's a one-to-one correspondance between an entry in sources.list and
the metadata that apt expects to find in the repository, which in turn is
signed.  You would have to combine the metadata in order to combine the
sources.list lines, which would then require resigning the metadata.

--
Russ Allbery (rra@...)               <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-security-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Lee Winter]

On Wed, Sep 16, 2009 at 3:54 PM, Russ Allbery <rra@...> wrote:

> Lee Winter <lee.j.i.winter@...> writes:
>> Goswin von Brederlow <goswin-v-b@...> wrote:
>
>>> This has one minor drawback though: The combined apt repository will be
>>> unsigned (you do not want to do that) or signed by a local key.
>
>> Why is that?
>
> Because the package lists from the two separate repositories are
> independently signed, and since you don't have access to the signing key,
> there's no way to combine them into a single package list and still have a
> valid signature without changing keys.
>
>> Right now the client's need three entries for lenny, security, and
>> volatile as if they came from separate mirrors.  I would like to keep
>> the repositories independent but have one "debian" section in
>> sources.list.  Is that not possible?
>
> There's a one-to-one correspondance between an entry in sources.list and
> the metadata that apt expects to find in the repository, which in turn is
> signed.  You would have to combine the metadata in order to combine the
> sources.list lines, which would then require resigning the metadata.

OK, this is where it starts to get interesting.  I didn't see much
more than passing references to this in the apt doc.  Did I miss it or
are there other docs that describe the repository structure?  Should I
be looking at the doc about creating packages or for creating
releases?

BTW, thanks for the clear/concise response.

Lee Winter
NP Engineering
Nashua, New Hampsire


--
To UNSUBSCRIBE, email to debian-security-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Russ Allbery-2]

Lee Winter <lee.j.i.winter@...> writes:
> On Wed, Sep 16, 2009 at 3:54 PM, Russ Allbery <rra@...> wrote:

>> There's a one-to-one correspondance between an entry in sources.list
>> and the metadata that apt expects to find in the repository, which in
>> turn is signed.  You would have to combine the metadata in order to
>> combine the sources.list lines, which would then require resigning the
>> metadata.

> OK, this is where it starts to get interesting.  I didn't see much more
> than passing references to this in the apt doc.  Did I miss it or are
> there other docs that describe the repository structure?  Should I be
> looking at the doc about creating packages or for creating releases?

I'm afraid I have no idea where it might be documented.  The above
statement is from experience dealing with apt and various local
repositories rather than taken from documentation.  :/

Back when I made a foray into writing tools to generate a local repository
and then patching debarchiver, I found it rather difficult to find
coherent documentation of all of the features of the Debian archive layout
and mostly resorted to looking at the Debian archives and reading source
code.

--
Russ Allbery (rra@...)               <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-security-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Goswin von Brederlow-2]

Russ Allbery <rra@...> writes:

> Lee Winter <lee.j.i.winter@...> writes:
>> On Wed, Sep 16, 2009 at 3:54 PM, Russ Allbery <rra@...> wrote:
>
>>> There's a one-to-one correspondance between an entry in sources.list
>>> and the metadata that apt expects to find in the repository, which in
>>> turn is signed.  You would have to combine the metadata in order to
>>> combine the sources.list lines, which would then require resigning the
>>> metadata.
>
>> OK, this is where it starts to get interesting.  I didn't see much more
>> than passing references to this in the apt doc.  Did I miss it or are
>> there other docs that describe the repository structure?  Should I be
>> looking at the doc about creating packages or for creating releases?
>
> I'm afraid I have no idea where it might be documented.  The above
> statement is from experience dealing with apt and various local
> repositories rather than taken from documentation.  :/
>
> Back when I made a foray into writing tools to generate a local repository
> and then patching debarchiver, I found it rather difficult to find
> coherent documentation of all of the features of the Debian archive layout
> and mostly resorted to looking at the Debian archives and reading source
> code.

Actualy I might have been wrong about file conflicts between debian
and security. I thought the Release and Release.gpg files would
collide. But as it turns out the Release(.gpg) files for security are
also under updates.

So it is possible to combine the two into

/dists/lenny/Release
/dists/lenny/Release.gpg
/dists/lenny/{main,contrib,non-free}
/dists/lenny/updates/Release
/dists/lenny/updates/Release.gpg
/dists/lenny/updates/{main,contrib,non-free}
/pool/{main,contrib,non-free}
/pool/updates/{main,contrib,non-free}

You can probably convince debmirror to do this with the right ignore
options.

BUT that doesn't really help. You still need:

deb http://mirror/debian lenny main contrib non-free
deb http://mirror/debian lenny/updates main contrib non-free

And if you need 2 lines in sources.list anyway then you might as well
use

deb http://mirror/debian lenny main contrib non-free
deb http://mirror/debian-security lenny/updates main contrib non-free

and keep the debmirror calls simpler.



Just for information an apt repository is structured like this:

deb http://HOST/PATH SUITE DIST [DIST [...]]

/PATH/dists/SUITE/Release                   <- checksums of Packages files
/PATH/dists/SUITE/Release.gpg               <- signature of above
/PATH/dists/SUITE/DIST/binary-ARCH/Packages <- lists debs
/PATH/dists/SUITE/DIST/binary-ARCH/Release  <- pining infos
/PATH/FILENAME                              <- FILENAME from Packages file

or (iirc)

deb http://HOST/BASE PATH/

/BASE/PATH/Release                   <- checksums of Packages files
/BASE/PATH/Release.gpg               <- signature of above
/BASE/PATH/Packages                  <- lists debs
/BASE/FILENAME                       <- FILENAME from Packages file

The later form is not often used and not by Debian. Anything else,
from the point of apt, of the archive structure is determined by the
contents of the Packages (and Sources) file.

MfG
        Goswin


--
To UNSUBSCRIBE, email to debian-security-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...