SSH connections stall expecting SSH2_MSG_KEX_DH_GEX_REPLY

[martin f krafft]

Hi,

every once in a while, I am stuck in a crap wifi network and often
cannot even establish SSH connections. What happens is that the
socket connection is established, but the client then just waits for
a server reply during the DH key exhchange:

  debug1: SSH2_MSG_KEXINIT sent
  debug1: SSH2_MSG_KEXINIT received
  debug1: kex: server->client 3des-cbc hmac-md5 none
  debug1: kex: client->server 3des-cbc hmac-md5 none
  debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<2048<8192) sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
  debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

there it sits forever, eventually doing TCP retransmissions of the
DH GEX Init sequence.

In the tcpdump output, I see a lot of duplicate packets, but
otherwise can't figure out what's going on.

  2.996410 192.168.254.246 -> 213.203.238.82 TCP 59448 > 22 [SYN] Seq=0 Win=5440 Len=0 MSS=1360 TSV=4283727 TSER=0 WS=6
  4.443188 213.203.238.82 -> 192.168.254.246 TCP 22 > 59448 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1380 TSV=747572639 TSER=4283727 WS=7
  4.443250 192.168.254.246 -> 213.203.238.82 TCP 59448 > 22 [ACK] Seq=1 Ack=1 Win=5440 Len=0 TSV=4284088 TSER=747572639
  5.620407 213.203.238.82 -> 192.168.254.246 SSH Server Protocol: SSH-2.0-OpenSSH_4.3p2 Debian-9
  5.620536 192.168.254.246 -> 213.203.238.82 TCP 59448 > 22 [ACK] Seq=1 Ack=32 Win=5440 Len=0 TSV=4284383 TSER=747573010
  5.620750 192.168.254.246 -> 213.203.238.82 SSH Client Protocol: SSH-2.0-OpenSSH_4.7p1 Debian-8
  6.889086 213.203.238.82 -> 192.168.254.246 TCP 22 > 59448 [ACK] Seq=32 Ack=32 Win=5888 Len=0 TSV=747573317 TSER=4284383
  6.889130 192.168.254.246 -> 213.203.238.82 SSHv2 Client: Key Exchange Init
  6.975096 213.203.238.82 -> 192.168.254.246 SSHv2 Server: Key Exchange Init
  7.012395 192.168.254.246 -> 213.203.238.82 TCP 59448 > 22 [ACK] Seq=592 Ack=736 Win=6848 Len=0 TSV=4284731 TSER=747573317
  7.711829 213.203.238.82 -> 192.168.254.246 TCP 22 > 59448 [ACK] Seq=736 Ack=592 Win=6912 Len=0 TSV=747573644 TSER=4284700
  7.711873 192.168.254.246 -> 213.203.238.82 SSHv2 Client: Diffie-Hellman GEX Request
  8.741589 213.203.238.82 -> 192.168.254.246 SSHv2 Server: Diffie-Hellman Key Exchange Reply
  8.741634 192.168.254.246 -> 213.203.238.82 TCP 59448 > 22 [ACK] Seq=616 Ack=1016 Win=8256 Len=0 TSV=4285163 TSER=747573824
  8.781014 192.168.254.246 -> 213.203.238.82 SSHv2 Client: Diffie-Hellman GEX Init
 14.908203 192.168.254.246 -> 213.203.238.82 SSHv2 [TCP Retransmission] Client: Diffie-Hellman GEX Init
 16.227454 213.203.238.82 -> 192.168.254.246 TCP [TCP Previous segment lost] 22 > 59448 [ACK] Seq=1608 Ack=888 Win=8064 Len=0 TSV=747575655 TSER=4286705 SLE=616 SRE=888
 22.576404 192.168.254.246 -> 213.203.238.82 SSH Encrypted request packet len=560
 23.876222 213.203.238.82 -> 192.168.254.246 TCP [TCP ACKed lost segment] 22 > 59447 [ACK] Seq=1 Ack=586 Win=54 Len=0 TSV=747577555 TSER=4288622
 23.942641 213.203.238.82 -> 192.168.254.246 SSH Encrypted response packet len=280

I tried varying the MTU but that didn't seem to have any effect.

Does anyone have any clue what's going on here? Is
SSH2_MSG_KEX_DH_GEX_INIT so complex that it manages to screw over
crap networks?

--
 .''`.   martin f. krafft <madduck@...>
: :'  :  proud Debian developer, author, administrator, and user
`. `'`   http://people.debian.org/~madduck - http://debiansystem.info
  `-  Debian - when you have better things to do than fixing systems
 
"women, when they are not in love,
 have all the cold blood of an experienced attorney."
                                                   -- honoré de balzac

[NN_il_Confusionario]

On Wed, Apr 09, 2008 at 03:34:13PM +0200, martin f krafft wrote:
> every once in a while, I am stuck in a crap wifi network and often
> cannot even establish SSH connections.
> I tried varying the MTU but that didn't seem to have any effect.

perhaps you coud try tuning parameters in /proc/sys/net/ipv4/
(lartc shoud have a guide about the most used ones).

For me too ssh in crap networks has more problems than other protocols
(but I have never meet your specific error)

--
Chi usa software non libero avvelena anche te. Digli di smettere.
Informatica=arsenico: minime dosi in rari casi patologici, altrimenti letale.
Informatica=bomba: intelligente solo per gli stupidi che ci credono.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Bernardo Dal Seno]

On 09/04/2008, martin f krafft <madduck@...> wrote:
>  every once in a while, I am stuck in a crap wifi network and often
>  cannot even establish SSH connections. What happens is that the
>  socket connection is established, but the client then just waits for
>  a server reply during the DH key exhchange:
[snip]


>  In the tcpdump output, I see a lot of duplicate packets, but
>  otherwise can't figure out what's going on.

I can see only one duplicate packet:

>   14.908203 192.168.254.246 -> 213.203.238.82 SSHv2 [TCP Retransmission] Client: Diffie-Hellman GEX Init

Have you filtered the tcpdump (Wireshark?) output?


>   22.576404 192.168.254.246 -> 213.203.238.82 SSH Encrypted request packet len=560

And this is an encrypted packet, so the Diffie-Hellman exchange should
be completed.


>   23.876222 213.203.238.82 -> 192.168.254.246 TCP [TCP ACKed lost segment] 22 > 59447 [ACK] Seq=1 Ack=586 Win=54 Len=0 TSV=747577555 TSER=4288622

This packet is very strange.  It's an ACK for a previous connection,
with sequence number 1, i.e., the server has not sent any byte, while
acknowledges 585 bytes sent from the client.  This is strange because
even if you tenet to an Ssh server you get a response containing the
version of the server.  Do you remember if you have done something in
particular to get that?


>  Does anyone have any clue what's going on here? Is
>  SSH2_MSG_KEX_DH_GEX_INIT so complex that it manages to screw over
>  crap networks?

I don't understand what's happening, but maybe some packet has been
filtered from the dump.  Do you have a firewall? Does it reject any
packet?

Ciao,
Bernardo


--
To UNSUBSCRIBE, email to debian-user-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[martin f krafft]

also sprach Bernardo Dal Seno <dibbex@...> [2008.04.11.0035 +0200]:
> >  In the tcpdump output, I see a lot of duplicate packets, but
> >  otherwise can't figure out what's going on.
>
> I can see only one duplicate packet:
>
> >   14.908203 192.168.254.246 -> 213.203.238.82 SSHv2 [TCP Retransmission] Client: Diffie-Hellman GEX Init

There'll be more if I just keep waiting.

> Have you filtered the tcpdump (Wireshark?) output?

Nope.

> >   22.576404 192.168.254.246 -> 213.203.238.82 SSH Encrypted
> >   request packet len=560
>
> And this is an encrypted packet, so the Diffie-Hellman exchange
> should be completed.

Okay, so let's assume the DH exchange completes fine, why is the
session then not established?

> >   23.876222 213.203.238.82 -> 192.168.254.246 TCP [TCP ACKed lost segment] 22 > 59447 [ACK] Seq=1 Ack=586 Win=54 Len=0 TSV=747577555 TSER=4288622
>
> This packet is very strange.  It's an ACK for a previous connection,
> with sequence number 1, i.e., the server has not sent any byte, while
> acknowledges 585 bytes sent from the client.  This is strange because
> even if you tenet to an Ssh server you get a response containing the
> version of the server.  Do you remember if you have done something in
> particular to get that?

Nope. Well, I flew into Barcelona and tried to connect to the
KubiWireless network here.

> >  Does anyone have any clue what's going on here? Is
> >  SSH2_MSG_KEX_DH_GEX_INIT so complex that it manages to screw over
> >  crap networks?
>
> I don't understand what's happening, but maybe some packet has
> been filtered from the dump.  Do you have a firewall? Does it
> reject any packet?

Well, a packet filter runs on 213.203.238.82, but it allows SSH
traffic and RELATED,ESTABLISHED.

Thanks for your time,

--
 .''`.   martin f. krafft <madduck@...>
: :'  :  proud Debian developer, author, administrator, and user
`. `'`   http://people.debian.org/~madduck - http://debiansystem.info
  `-  Debian - when you have better things to do than fixing systems
 
"one should never do anything that
 one cannot talk about after dinner."
                                                        -- oscar wilde

[martin f krafft]

also sprach martin f krafft <madduck@...> [2008.04.16.1857 +0200]:
> > I don't understand what's happening, but maybe some packet has
> > been filtered from the dump.  Do you have a firewall? Does it
> > reject any packet?
>
> Well, a packet filter runs on 213.203.238.82, but it allows SSH
> traffic and RELATED,ESTABLISHED.

FWIW, I get pretty much exactly the same behaviour, whether I try my
machines, university ones, .debian.org ones, or any other SSH server
out there, basically.

Unfortunately, I can't really cooperate with anyone out there on it,
since, well, I cannot talk to the net. UDP kinda works. TCP
like HTTP, SMTP, IRC, or SSH do not.

--
 .''`.   martin f. krafft <madduck@...>
: :'  :  proud Debian developer, author, administrator, and user
`. `'`   http://people.debian.org/~madduck - http://debiansystem.info
  `-  Debian - when you have better things to do than fixing systems
 
obviously i was either onto something, or on something.
                                 -- larry wall on the creation of perl

[Bernardo Dal Seno]

On 16/04/2008, martin f krafft <madduck@...> wrote:
> > > I don't understand what's happening, but maybe some packet has
>  > > been filtered from the dump.  Do you have a firewall? Does it
>  > > reject any packet?
>  >
>  > Well, a packet filter runs on 213.203.238.82, but it allows SSH
>  > traffic and RELATED,ESTABLISHED.

I asked this question because not long ago I had a problem with TCP
connections hanging and I was sure that all packets should pass the
firewall because of rules like those you mentioned.  But when I added
a logging rule, I discovered that packets were actually rejected. (The
cause was a lousy router that mangled packets in way violating an RFC,
and in the end I had to disable the SACK option).

So I thought that maybe your "crap wifi network" had also a crap
router mangling packets in some way or another.  But this is unlikely,
if you don't have a firewall on your local machine, and you say you
experience problems with any other machine:

> FWIW, I get pretty much exactly the same behaviour, whether I try my
>  machines, university ones, .debian.org ones, or any other SSH server
>  out there, basically.


No other clues, sorry.  :-(

Bernardo


--
To UNSUBSCRIBE, email to debian-user-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Chris Davies-2]

martin f krafft <madduck@...> wrote:
> every once in a while, I am stuck in a crap wifi network and often
> cannot even establish SSH connections. What happens is that the
> socket connection is established, but the client then just waits for
> a server reply during the DH key exhchange:

Are you running seahorse? I've not logged a bug, yet, as I can't
reproduce it reliably, but there seems to be a problem if seahorse fires
up without network access and it's delivering your ssh keys.

To "prove" this, ssh localhost. If it hangs then seahorse is the
culprit. (I can't determine the combination of settings to force this
broken behaviour, so it /may/ be just my machine.)

Chris


--
To UNSUBSCRIBE, email to debian-user-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[martin f krafft]

also sprach Chris Davies <chris-usenet@...> [2008.04.18.1034 +0200]:
> Are you running seahorse? I've not logged a bug, yet, as I can't
> reproduce it reliably, but there seems to be a problem if seahorse
> fires up without network access and it's delivering your ssh keys.

I am not using seahorse.

--
 .''`.   martin f. krafft <madduck@...>
: :'  :  proud Debian developer, author, administrator, and user
`. `'`   http://people.debian.org/~madduck - http://debiansystem.info
  `-  Debian - when you have better things to do than fixing systems
 
"you know you're a hopeless geek when you misspell 'date' as 'data'"
                                                   -- branden robinson

[Christofer C. Bell]

On Fri, Apr 18, 2008 at 4:07 AM, martin f krafft <madduck@...> wrote:
> also sprach Chris Davies <chris-usenet@...> [2008.04.18.1034 +0200]:
>
> > Are you running seahorse? I've not logged a bug, yet, as I can't
>  > reproduce it reliably, but there seems to be a problem if seahorse
>  > fires up without network access and it's delivering your ssh keys.
>
>  I am not using seahorse.

Martin, I had an issue similar to yours recently, affecting only ssh
(not other protocols) and it turns out that I needed to set my eth0's
MTU to a lower value (I used 576) because the routers my traffic was
passing through were set lower than the (usual) 1500.  You might give
that a try.

--
Chris


--
To UNSUBSCRIBE, email to debian-user-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[martin f krafft]

also sprach Christofer C. Bell <christofer.c.bell@...> [2008.04.19.0723 +0200]:
> Martin, I had an issue similar to yours recently, affecting only ssh
> (not other protocols) and it turns out that I needed to set my eth0's
> MTU to a lower value (I used 576) because the routers my traffic was
> passing through were set lower than the (usual) 1500.  You might give
> that a try.

I've tried MTU sizes down to 512 without any effect.

--
 .''`.   martin f. krafft <madduck@...>
: :'  :  proud Debian developer, author, administrator, and user
`. `'`   http://people.debian.org/~madduck - http://debiansystem.info
  `-  Debian - when you have better things to do than fixing systems
 
"i believe that the moment is near when by a procedure
 of active paranoiac thought, it will be possible
 to systematise confusion and contribute to
 the total discrediting of the world of reality."
                                                      -- salvador dali