IPv6 troubleshooting help needed

[Florian Weimer]

Now that security.debian.org has got AAAA records, reports regarding
connectivity issues pop up in odd places.  The level of complaints is
still rather low (quite unexpected to me), and it's not affecting user
experience in a significant way (I think), but we should still try to
improve things where we can.

In short, we're looking for Debian Developers who help us to debug
IPv6 connectivity issues.  Here's the basic skill set (which may seem
a bit daunting, but if everything were easy, we wouldn't have anything
to troubleshoot, right?):

  - basic knowledge of BGP and real-world Internet WAN routing
    and traffic exchange policies

  - ability to interpret looking glass output (show ip bgp, show
    route) and traceroutes

  - some IPv6 experience preferred

  - basic understanding of DNS (A/AAAA records,
    proxy/forwarding/recursive/authoritative server distinction)

  - Internet2 routing knowledge is a plus (but certainly not a must)

  - non-English language skills would help as well

  - DD status required

Tasks include:

  - subscribing to the various mailing lists (debian-user, NANOG and
    other *NOGs, outages, dns-operations, language-specific lists) and
    monitor them for occurrences of security.debian.org

  - following up on reports, requesting more information if necessary

  - obtaining the reverse view from Debian's own hosts

  - try to guess where the problem is located (the hard part!)

  - contacting network operators in case of persistent issues

  - interacting with DSA if action on Debian's part is required

  - ask native speakers within Debian for help in case there's are
    language barrier towards the reporter

The most frequent issues I've seen are related to partial transit from
Internet2, and broken DNS proxies taking AAAA records and serving
their first four bytes as IPv4 addresses.  I don't think we've fully
debugged either set of issues, unfortunately.  The upside is that
large parts of the IPv6 community are friendly, accessible and eager
to help out when there are problems.

If you're interested, drop me a message.

Access to proper troubleshooting tools will require LDAP privileges
which are subject to DSA and security@ approval (hence the DD status
requirement).


--
To UNSUBSCRIBE, email to debian-project-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Andre Felipe Machado-3]

Hello,
I am being unable to update systems (security and volatile) both at home and at
work (Brasil).
At work is very critical issue, as it is a Debian Partner, in the course of
deploying Debian systems across Brasil federal key government agencies.
A security mistake now will cost years of convincing effort...
Where do I keep track of such issue?
Regards.
Andre Felipe Machado

[Stephen Gran]

This one time, at band camp, Andre Felipe Machado said:
> Hello,
> I am being unable to update systems (security and volatile) both at home and at
> work (Brasil).
> At work is very critical issue, as it is a Debian Partner, in the course of
> deploying Debian systems across Brasil federal key government agencies.
> A security mistake now will cost years of convincing effort...

Can you send the output of `mtr security.debian.org`?  The same for
volatile would be helpful as well.

Cheers,
--
 -----------------------------------------------------------------
|   ,''`.                                            Stephen Gran |
|  : :' :                                        sgran@... |
|  `. `'                        Debian user, admin, and developer |
|    `-                                     http://www.debian.org |
 -----------------------------------------------------------------

[Andre Felipe Machado-3]

Hello, Stephen
At the end of email, you could find the command output.
It may be useful to add some information.
Yesterday I tried update at another location, with other adsl modem (an old USR
9001), but at same adsl isp (www.gvt.com.br). It was successful.
My home adsl connection to the same isp is made using a new Dlink DSL-2640T adsl
modem router wifi, that may affect the ipv6 connection.
At my personal server located in a data center in USA, all updates went smoothly.
At work (sorry, mtr report only next tuesday), we have layers of routers,
switches, balancers, proxies, firewalls, ids, ips, etc. So, there may be filters
acting.

This security update is what really worries me, as it is a debian partner.
Is it possible to reverse the servers configuration until all factors are known,
using another less critical site/repo?

Is it better to open a bug report about the pseudo-package "security.debian.org"
regarding this behaviour?

Regards.
Andre Felipe



andremachado@debian:~/temp$ mtr --report --report-cycles 2 security.debian.org
HOST: debian                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. mygateway1.ar7                0.0%     2    0.6   0.6   0.6   0.6   0.0
  2. 189.27.208.1.dynamic.adsl.gv  0.0%     2    5.4   5.8   5.4   6.1   0.5
  3. gvt                           0.0%     2    5.8   5.9   5.8   6.0   0.1
  4. gvt                           0.0%     2    6.4   6.5   6.4   6.5   0.0
  5. gvt                           0.0%     2    6.4   6.4   6.4   6.5   0.1
  6. gvt                           0.0%     2   20.5  20.6  20.5  20.8   0.2
  7. gvt                           0.0%     2   20.7  20.7  20.7  20.7   0.0
  8. Ge4                           0.0%     2   20.5  20.8  20.5  21.0   0.3
  9. So6                           0.0%     2  138.0 135.1 132.3 138.0   4.1
 10. So6                           0.0%     2  137.8 150.7 137.8 163.6  18.3
 11. Xe4-0-0-0-grtpartv1.red.tele  0.0%     2  171.4 204.8 171.4 238.2  47.3
 12. So3                           0.0%     2  242.6 242.7 242.6 242.7   0.1
 13. ???                          100.0     2    0.0   0.0   0.0   0.0   0.0
andremachado@debian:~/temp$ mtr --report --report-cycles 2 volatile.debian.org
HOST: debian                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. mygateway1.ar7                0.0%     2    0.6   0.6   0.6   0.6   0.1
  2. 189.27.208.1.dynamic.adsl.gv  0.0%     2    5.6   5.7   5.6   5.8   0.1
  3. gvt                           0.0%     2    6.0   6.0   6.0   6.0   0.0
  4. 189.59.252.205                0.0%     2    6.3   6.4   6.3   6.5   0.1
  5. gvt-host.gvt.net.br           0.0%     2    5.9   6.6   5.9   7.3   0.9
  6. gvt-po-7-1-4-rc01.spo.gvt.ne  0.0%     2   20.0  20.1  20.0  20.1   0.1
  7. Ge4                           0.0%     2   20.9  23.2  20.9  25.5   3.2
  8. So6                           0.0%     2  138.1 191.3 138.1 244.5  75.2
  9. Xe-1-3-0-0-grtwaseq2.red.tel  0.0%     2  158.3 169.5 158.3 180.6  15.8
 10. So5-0-0-0-grtparix3.red.tele  0.0%     2  175.0 214.8 175.0 254.6  56.3
 11. So5-1-0-0-grtlontl1.red.tele  0.0%     2  244.5 243.5 242.6 244.5   1.3
 12. ns.de.prserv.net              0.0%     2  254.0 333.4 254.0 412.8 112.3
 13. ns.de.prserv.net              0.0%     2  252.1 254.3 252.1 256.4   3.0
 14. ???                          100.0     2    0.0   0.0   0.0   0.0   0.0
andremachado@debian:~/temp$

[Andre Felipe Machado-3]

Hello, Simon
Thanks for the guidance.
The new commands outputs are below.
Should I open a bug report to pseudo-package "security.debian.org" to track this
issue instead of using this list?
Regards.
Andre Felipe




andremachado@debian:~$ mtr -6 -n --report --report-cycles 2 volatile.debian.org
HOST: debian                      Loss%   Snt   Last   Avg  Best  Wrst StDev
andremachado@debian:~$ mtr -6 -n --report --report-cycles 2 security.debian.org
HOST: debian                      Loss%   Snt   Last   Avg  Best  Wrst StDev
andremachado@debian:~$

andremachado@debian:~$ mtr  -n --report --report-cycles 2 security.debian.org
HOST: debian                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 10.1.1.1                      0.0%     2    0.6   0.8   0.6   1.0   0.3
  2. 189.27.208.1                  0.0%     2    5.4   5.8   5.4   6.1   0.5
  3. 200.175.124.193               0.0%     2    6.5   6.1   5.8   6.5   0.5
  4. 189.59.252.205                0.0%     2    6.5   6.6   6.5   6.7   0.2
  5. 189.59.252.185                0.0%     2    6.3   7.5   6.3   8.7   1.7
  6. 189.59.248.5                  0.0%     2   20.4  20.5  20.4  20.6   0.1
  7. 84.16.10.209                  0.0%     2   21.0  21.0  21.0  21.0   0.0
  8. 213.140.36.65                 0.0%     2  179.9 155.8 131.7 179.9  34.1
  9. 84.16.13.61                   0.0%     2  134.4 158.0 134.4 181.6  33.3
 10. 213.140.37.14                 0.0%     2  261.9 264.0 261.9 266.2   3.1
 11. 213.140.38.30                 0.0%     2  242.3 251.7 242.3 261.1  13.3
 12. 195.66.226.27                 0.0%     2  258.5 250.8 243.2 258.5  10.8
 13. 195.66.224.27                 0.0%     2  256.1 249.7 243.3 256.1   9.1
 14. ???                          100.0     2    0.0   0.0   0.0   0.0   0.0
andremachado@debian:~$ mtr  -n --report --report-cycles 2 volatile.debian.org
HOST: debian                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 10.1.1.1                      0.0%     2    0.6   0.6   0.6   0.6   0.0
  2. 189.27.208.1                  0.0%     2    5.8   5.8   5.8   5.8   0.0
  3. 200.175.124.193               0.0%     2    6.0   5.9   5.9   6.0   0.1
  4. 189.59.252.205                0.0%     2    6.4   6.5   6.4   6.7   0.2
  5. 189.59.252.185                0.0%     2    6.5   6.7   6.5   6.8   0.2
  6. 189.59.248.5                  0.0%     2   19.8  19.8  19.8  19.9   0.0
  7. 84.16.10.209                  0.0%     2   21.7  21.5  21.3  21.7   0.3
  8. 213.140.36.69                 0.0%     2  137.9 137.9 137.8 137.9   0.1
  9. 213.140.36.50                 0.0%     2  134.2 143.0 134.2 151.8  12.5
 10. 84.16.15.121                  0.0%     2  175.0 216.2 175.0 257.3  58.2
 11. 213.140.36.205                0.0%     2  261.1 254.9 248.8 261.1   8.7
 12. 213.140.38.54                 0.0%     2  232.4 235.2 232.4 238.0   3.9
 13. ???                          100.0     2    0.0   0.0   0.0   0.0   0.0
andremachado@debian:~$

[Simon Paillard]

On Sun, Nov 01, 2009 at 12:28:41PM -0200, Andre Felipe Machado wrote:
> The new commands outputs are below.
> Should I open a bug report to pseudo-package "security.debian.org" to track this
> issue

security.debian.org is the way to reach security team, while such issues
is in DSA's scope.

> instead of using this list?

We may track this using 'mirrors' pseudopackage, or RT in DSA queue
since such issues are not specific to mirrors but affect the whole
Debian infrastructure.

> andremachado@debian:~$ mtr -6 -n --report --report-cycles 2 volatile.debian.org
> HOST: debian                      Loss%   Snt   Last   Avg  Best  Wrst StDev
> andremachado@debian:~$ mtr -6 -n --report --report-cycles 2 security.debian.org
> HOST: debian                      Loss%   Snt   Last   Avg  Best  Wrst StDev

If the issues you meet are really v6 specific, then ensure your system
is configured for v6.
IPv4 traceroute is not very usefull to debug routing issues in v6.
Or the problem you meet applies to v4 as well ?

--
Simon Paillard


--
To UNSUBSCRIBE, email to debian-project-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Andre Felipe Machado-3]

Hello,
That is it! Thanks a lot!
I disabled the dns relay function at the Dlink DSL-2640T and now my home Debian
machine can update using security.debian.org and volatile.debian.org that
changed to ipv6 addresses some days ago.
I will submit a bug report to Dlink site.

But what still really worries me is how to exactly track, prove and demonstrate,
that the problem at company is some kind of error at dns caching.
As a country wide, servicing multiple agencies, I guess it is using some high
end network hardware and some kind of software and security infrastructure.
I *must* have 100% demonstrable, 100% sure, log and or report to submit a ticket
to their net/security teams.
What commands could prove and register the dns caching problem at their network?
Also, it have layers of ips, ids, routers, swithches, firewalls, etc, and
something in that setup could also be causing the problem.
Suggestions?
Regards.
Andre Felipe


Luca Bruno  wrote ..

> Andre Felipe Machado scrisse:
>
> > Hello, Stephen
> > At the end of email, you could find the command output.
>
> That is an ipv4 connections not going well, isn't it? Why are you
> talking about ipv6?
>
> > It may be useful to add some information.
> > Yesterday I tried update at another location, with other adsl modem
> > (an old USR 9001), but at same adsl isp (www.gvt.com.br). It was
> > successful. My home adsl connection to the same isp is made using a
> > new Dlink DSL-2640T adsl modem router wifi, that may affect the ipv6
> > connection.
>
> Out of the mind, you could be experiencing something similar to
> http://blog.bofh.it/id_202 (which, if you can't read Italian, is
> basically your modem doing DNS-caching badly) which already happened on
> some other Dlink modems.
>
> Can you please check to which IP are your packets trying to go? (eg.
> via dig, ping and traceroute).
>
> Ciao, Luca
>
> --
>  .''`.  ** Debian GNU/Linux **  | Luca Bruno (kaeso)
> : :'  :   The Universal O.S.    | lucab (AT) debian.org
> `. `'`   | GPG Key ID: 3BFB9FB3
>   `-     http://www.debian.org  | Debian GNU/Linux Developer

[Luca Bruno-7]

Andre Felipe Machado scrisse:

> Hello,
> That is it! Thanks a lot!
> I disabled the dns relay function at the Dlink DSL-2640T and now my
> home Debian machine can update using security.debian.org and
> volatile.debian.org that changed to ipv6 addresses some days ago.
> I will submit a bug report to Dlink site.

Good to have it pin-pointed. Just to take notes, which firmware version
is that using? And by the way, Florian's mail already mentioned that
under "most frequent issues".

> But what still really worries me is how to exactly track, prove and
> demonstrate, that the problem at company is some kind of error at dns
> caching. As a country wide, servicing multiple agencies, I guess it
> is using some high end network hardware and some kind of software and
> security infrastructure. I *must* have 100% demonstrable, 100% sure,
> log and or report to submit a ticket to their net/security teams.
> What commands could prove and register the dns caching problem at
> their network? Also, it have layers of ips, ids, routers, swithches,
> firewalls, etc, and something in that setup could also be causing the
> problem. Suggestions?

It may be almost anything in-between, like dns-cacher, firewall, load
balancer, etc...
https://bugzilla.redhat.com/show_bug.cgi?id=505105 already contains
quite a discrete list of things, including Cisco, Juniper and Foundry
stuff.

You can start comparing a `dig ANY volatile.debian.org` from your
company and from another good-working place, checking for
inconsistencies. If there's a dns wrong-resolution, pinging/reaching it
will reveal an IP not pertaining to a proper repository-hosting
machine.
Otherwise there could be something weird in routing, which can be
spotted via mtr or traceroute. If you have ipv6 connectivity at company
(ie. a configured v6 route) it may be something different.

But as we've pin-pointed first issue, and the second one looks like
something due to internal network configuration, I think we can move
this discussion away from -project to private, if nobody else
interested (maybe summarising back the result at the end).

Ciao, Luca

--
 .''`.  ** Debian GNU/Linux **  | Luca Bruno (kaeso)
: :'  :   The Universal O.S.    | lucab (AT) debian.org
`. `'`   | GPG Key ID: 3BFB9FB3
  `-     http://www.debian.org  | Debian GNU/Linux Developer

[Andre Felipe Machado-3]

Hello, Luca
Thanks for the hints!
I will move this discussion to debian-users list at next msg, as it may be
useful for others too.

The Dlink DSL-2640T have firmware V3.02B01T01.TF-A.20060804
It seems that there is not a newer firmware version at their brazilian site.

I will compare dig outputs tomorrow and start to trace there.
Regards.
Andre Felipe