Publishing metalinks on the download page

[Bram Neijt]

Dear developers,

My name is Bram and I'm a member of the metalinks developers. Metalinks
are an XML format describing mirror links which helps downloading files
more easily using a download client.

I've recently launched an online service which makes publishing
metalinks as easy as adding an url to your downloads page, and I'm
looking for somebody willing to give feedback or better yet, try it out.

More information on metalinks and the new online service can be found at
the metalinks tools project trac:
http://sourceforge.net/apps/trac/metalinks/
and http://www.dynmirror.net/help/developers/

Of course I'm happy to answer any and all question in this thread also.
Feel free to just mail 'no' if you are not interested in making changes
to your project's download homepage.


Greetings,
  Bram


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Stefan Bodewig]

Hi Bram,

On 2009-09-01, Bram Neijt <bneijt@...> wrote:

> Of course I'm happy to answer any and all question in this thread also.
> Feel free to just mail 'no' if you are not interested in making changes
> to your project's download homepage.

Our download pages are created by CGI scripts that already selects the
user's closest mirror (based on IP number and some GeoIP database) and
recommend a few others, so I don't think we'd ebenefit from the
metalinks link at all.  In particular since the list of ASF mirrors is
maintained centrally.

I'm not sure whether I understood the description of "getting the
checksums from a single source" in the links you posted correctly.  We
wouldn't want users to trust any signatures or checksums downloaded from
anywhere but the central ASF site and thus our download page explicitly
sends users to the main site only for those sort of files.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Bram Neijt]

Hi Stefan,

You are correct that you won't benefit from publishing metalinks, but
your users might. One of the failures a redirect/GeoIP based approach
can't solve for the user is possible firewall problems. The user may be
behind a firewall which restricts to HTTP connetions, not knowing that
for sure he/she would have to try the GeoIP suggestion. If that fails,
manually retry. That would be solved by having a smarter and better
informed download program, which metalinks can help create. Another
benefit is that the download client can also verify the download
afterwards, without the user having to run any extra commands after the
download finished.

About the digest information coming from a reliable central source, you
are already doing that with your download page by pointing only to the
central MD4/SHA1/signature files. I've rewritten the page to keep other
people from getting confused about that :)

Thank you very much for your comments on the links. If you ever end up
trying metalinks, feel free to drop the metalink community[1] a line or
mail me directly.

All the best!
  Bram

[1] http://sourceforge.net/apps/trac/metalinks/wiki/CommunityInformation

On Thu, 2009-09-03 at 10:46 +0200, Stefan Bodewig wrote:

> Hi Bram,
>
> On 2009-09-01, Bram Neijt <bneijt@...> wrote:
>
> > Of course I'm happy to answer any and all question in this thread also.
> > Feel free to just mail 'no' if you are not interested in making changes
> > to your project's download homepage.
>
> Our download pages are created by CGI scripts that already selects the
> user's closest mirror (based on IP number and some GeoIP database) and
> recommend a few others, so I don't think we'd ebenefit from the
> metalinks link at all.  In particular since the list of ASF mirrors is
> maintained centrally.
>
> I'm not sure whether I understood the description of "getting the
> checksums from a single source" in the links you posted correctly.  We
> wouldn't want users to trust any signatures or checksums downloaded from
> anywhere but the central ASF site and thus our download page explicitly
> sends users to the main site only for those sort of files.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Stefan Bodewig]

On 2009-09-03, Bram Neijt <bneijt@...> wrote:

> You are correct that you won't benefit from publishing metalinks, but
> your users might.

Which would benefit Ant ...  But I'm not convinced 8-)

> One of the failures a redirect/GeoIP based approach can't solve for
> the user is possible firewall problems. The user may be behind a
> firewall which restricts to HTTP connetions, not knowing that for sure
> he/she would have to try the GeoIP suggestion.

If you look at the actual download page it will list HTTP mirrors first
(since ftp mirrors are so 20th century anyway, I guess).  Most users
will have stopped before reaching the ftp mirrors.

> Another benefit is that the download client can also verify the
> download afterwards,

This depends on my level of paranoia.  Do I want to trust md5 or sha1
hashes at all?  Does the client speak OpenPGP for a stronger checksum
algorithm (unless the signer is in my web of trust, the signature isn't
more than that)?

> without the user having to run any extra commands after the download
> finished.

Do I tust the download client? ;-;

> About the digest information coming from a reliable central source, you
> are already doing that with your download page by pointing only to the
> central MD4/SHA1/signature files. I've rewritten the page to keep other
> people from getting confused about that :)

Thanks.

I think in the ASF's case dynmirror doesn't really help.  The list of
mirrors is dynamic and mirrors come and go (e.g. they may get removed if
they don't sync fast enough) and we like to keep it that way.  The
specific ASF projects (like Ant) aren't even aware of the process.

If I understand dynmirror correctly it would accept any download URL as
a mirror if it can provide matching filenames and MD5 checksums, is that
correct?  This would allow mirrors to add themselves that are not
"approved" (they may want to show ads we don't like for example).

If my understanding is correct it would also allow me to create a trojan
distribution of some software if I manage to create MD5 checksums that
match the original distribution - given that creating hash collisions in
MD5 isn't that difficult for a well-funded bad-guy, this is something
I'd be concerned about.  Given its adoption Apache httpd looks like a
very attractive target for inserting a backdoor, so the well-funded
bad-guy isn't that far-fetched IMHO.

Let's say I hope my understanding is wrong.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Bram Neijt]

Hi Stefan,

On Thu, 2009-09-03 at 16:54 +0200, Stefan Bodewig wrote:
> > Another benefit is that the download client can also verify the
> > download afterwards,
>
> This depends on my level of paranoia.  Do I want to trust md5 or sha1
> hashes at all?  Does the client speak OpenPGP for a stronger checksum
> algorithm (unless the signer is in my web of trust, the signature isn't
> more than that)?
That is true, the security of the download depends on the strength of
the published verification method (e.g. SHA-512 would be better then
SHA-1), the security of each of the mirrors you have approved, and the
method/response time to break-ins on those mirrors. And then there is
the security of the transmission. I don't think we would disagree on any
of those :).

>
> > without the user having to run any extra commands after the download
> > finished.
>
> Do I tust the download client? ;-;
I trust my download client, just as much as I trust my md5sums
binary ;). If the users' software can't be trusted, then nothing about
the download can be, simply because any kind of trojan could even mod it
after the download and verification. It's all very complicated and
depends on how paranoid you are I guess, I've actually never looked at
most of the core-utils source, so all my `ls` results may just be lies.
Anyway, I trust my download client (aria2c).

>
> > About the digest information coming from a reliable central source, you
> > are already doing that with your download page by pointing only to the
> > central MD4/SHA1/signature files. I've rewritten the page to keep other
> > people from getting confused about that :)
>
> Thanks.
>
> I think in the ASF's case dynmirror doesn't really help.  The list of
> mirrors is dynamic and mirrors come and go (e.g. they may get removed if
> they don't sync fast enough) and we like to keep it that way.  The
> specific ASF projects (like Ant) aren't even aware of the process.

Which I can understand is big no-no for large projects, but hopefully a
step up for small projects that want to do their own mirroring in
collaboration with their users. I'll see what smaller projects have to
say about it. But we can safely say that dynmirrors is not for you.

> If I understand dynmirror correctly it would accept any download URL as
> a mirror if it can provide matching filenames and MD5 checksums, is that
> correct?  This would allow mirrors to add themselves that are not
> "approved" (they may want to show ads we don't like for example).
It is not possible for them to show ads, as the links are only mentioned
in the metalink and the download client won't open them in a browser. If
they host an add there, the download client will simply check the
server, see that the content-length doesn't match up and drop it. If
everything goes well, the user won't even notice dynmirror.net is
involved. However, as we already said, dynmirror is not for you :)


> If my understanding is correct it would also allow me to create a trojan
> distribution of some software if I manage to create MD5 checksums that
> match the original distribution - given that creating hash collisions in
> MD5 isn't that difficult for a well-funded bad-guy, this is something
> I'd be concerned about.  Given its adoption Apache httpd looks like a
> very attractive target for inserting a backdoor, so the well-funded
> bad-guy isn't that far-fetched IMHO.
>
> Let's say I hope my understanding is wrong.
You are correct, a well funded bad-guy would be able to do so creating a
hash collision on MD5 or any other kind of verification method you can
muster. A really well-funded bad-guy would be better off becoming a
dictator, and taking control of most of the countries DNS servers. But
yes, you are completely right if you assume that the download security
is completely dictated by the security of the verification method (which
is what dynmirror.net does). Again, don't use dynmirror.net if you feel
uncomfortable with this.

That said, you could host your own metalink with only one or two
mirrors, anybody using aria2c for example, would no-longer require to
hand-check the digest after download. Consider this example metalink:

<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/">
  <files>
    <file name="apache-ivy-2.1.0-rc2-bin.zip">
      <verification>
        <hash type="sha-1">f3e10f28d15c59276d6808742018af2f2e1f13e6</hash>
      </verification>
      <resources>
        <url
type="http">http://mirror.hostfuss.com/apache/ant/ivy/2.1.0-rc2/apache-ivy-2.1.0-rc2-bin.zip</url>
      </resources>
    </file>
    <file name="apache-ivy-2.1.0-rc2-bin.zip.asc">
      <resources>
        <url
type="http">http://www.apache.org/dist/ant/ivy/2.1.0-rc2/apache-ivy-2.1.0-rc2-bin.zip.asc</url>
      </resources>
    </file>
  </files>
</metalink>

This would allow people to download both the file and the detatched
signature, while their download client will check the SHA-1 (I suggest
aria2c -M http://link_to_the_above_metalink).

I think that would actually help increase security, as people are more
likely to check the detached signature if it's automatically downloaded
with next to the file.

Greets,
  Bram




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Stefan Bodewig]

On 2009-09-03, Bram Neijt <bneijt@...> wrote:

> On Thu, 2009-09-03 at 16:54 +0200, Stefan Bodewig wrote:

>> Do I tust the download client? ;-;
> I trust my download client, just as much as I trust my md5sums
> binary ;).

Agreed.

>> If my understanding is correct it would also allow me to create a trojan
>> distribution of some software if I manage to create MD5 checksums that
>> match the original distribution

> You are correct, a well funded bad-guy would be able to do so creating a
> hash collision on MD5 or any other kind of verification method you can
> muster. A really well-funded bad-guy would be better off becoming a
> dictator, and taking control of most of the countries DNS servers.

Maybe.  But the amount of funds required is very different.  If MD5 was
the only checksum I'm pretty sure my notebook would be able to create a
zip or tar with matching checksums in a few hours.

> That said, you could host your own metalink with only one or two
> mirrors, anybody using aria2c for example, would no-longer require to
> hand-check the digest after download.

The way we create the download page could probably be used to create a
metalink XML file as well (i.e. an XML file that contained exactly the
same mirrors that are shown on the download page).

I don't have a strong opinion on whether we want to do that.  Others?

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Stefan Bodewig]

On 2009-09-04, Stefan Bodewig <bodewig@...> wrote:

> On 2009-09-03, Bram Neijt <bneijt@...> wrote:

>> You are correct, a well funded bad-guy would be able to do so creating a
>> hash collision on MD5 or any other kind of verification method you can
>> muster. A really well-funded bad-guy would be better off becoming a
>> dictator, and taking control of most of the countries DNS servers.

> Maybe.  But the amount of funds required is very different.  If MD5 was
> the only checksum I'm pretty sure my notebook would be able to create a
> zip or tar with matching checksums in a few hours.

Here I am clearly wrong, sorry.

My notebook should be able to find MD5 collisions within a few minutes
<http://eprint.iacr.org/2006/105.pdf> but going from there to creating
an archive with malicious content and matching a given MD5 checksum
would be a whole lot more difficult and take way longer - still not out
of reach of a really well-funded bad-guy, though.

Note that I can append some bytes of random junk to ZIPs and TARs
without changing their contents, so this gives me some freedom to create
colliding archives, but it will still take much longer than "a few
hours".

The report I had in the back of my mind when I wrote the paragraph above
<http://www.win.tue.nl/hashclash/SoftIntCodeSign/> requires the attacker
to be able to modify the "good" archive before the checksum is created -
which in general is not the case in the way dynmirror works.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...