sshd hangs after SSH2_MSG_KEXINIT sent - Fedora Core 5 update

[Davin Flatten]

Hello-

I am having a strange problem ever since we applied the Fedora Core 5
update to the Openssh RPM's.  Ever since the update when some users
connect thru a NAT gateway to the NAT'ed server the connection hangs.
This occurs only for some combinations of firewalls.  Below is all the
information I could gather on the subject.  Has anyone had this same
problem and found a solution?

The setup is as follows:
     ssh server <---Nat firewall #1 <--Internet <--Nat firewall #2<--ssh
client
Firewall #1 is an OpenBSD gateway running m0n0wall and the Firewall #2
depends on which client is connecting.
Only on some client firewalls the problem arises that the connection
hangs after the server sends the SSH2_MSG_KEXINIT.

-- yum upgrades --
Jan 04 11:45:24 Updated: openssh-askpass.x86_64 4.3p2-4.11.fc5
Jan 04 11:45:29 Updated: openssh-server.x86_64 4.3p2-4.11.fc5
Jan 04 11:45:37 Updated: openssh-clients.x86_64 4.3p2-4.11.fc5
Jan 04 11:47:39 Updated: openssh.x86_64 4.3p2-4.11.fc5

-- sshd server logs --
Feb  5 17:07:17 jeeves sshd[21270]: debug1: rexec start in 4 out 4
newsock 4 pipe 6 sock 7
Feb  5 17:07:17 jeeves sshd[20894]: debug1: Forked child 21270.
Feb  5 17:07:17 server sshd[21270]: debug1: inetd sockets after dupping:
3, 3
Feb  5 17:07:17 server sshd[21270]: Connection from xxx.xxx.xxx.xxx port
62175
Feb  5 17:07:17 server sshd[21270]: debug1: Client protocol version 2.0;
client software version OpenSSH_4.2
Feb  5 17:07:17 server sshd[21270]: debug1: match: OpenSSH_4.2 pat OpenSSH*
Feb  5 17:07:17 server sshd[21270]: debug1: Enabling compatibility mode
for protocol 2.0
Feb  5 17:07:17 server sshd[21270]: debug1: Local version string
SSH-2.0-OpenSSH_4.3
Feb  5 17:07:17 server sshd[21271]: debug1: permanently_set_uid: 74/74
Feb  5 17:07:17 server sshd[21271]: debug1: list_hostkey_types:
ssh-rsa,ssh-dss
Feb  5 17:07:17 server sshd[21271]: debug1: SSH2_MSG_KEXINIT sent

-- sshd packet logs --
No.     Time        Source                Destination           Protocol
Info
      1 0.000000    client-ip-address        server-ip-address
           TCP      50938 > ssh [FIN, ACK] Seq=0 Ack=0 Win=65535 Len=0
TSV=326215631 TSER=277581882
      2 0.000006    server-ip-address            client-ip-address
       TCP      ssh > 50938 [ACK] Seq=0 Ack=4294966560 Win=46 Len=0
TSV=277653781 TSER=326215487 SLE=0 SRE=1
      3 0.939549    client-ip-address        server-ip-address
           TCP      57188 > ssh [SYN] Seq=0 Len=0 MSS=1460 WS=0
TSV=326215633 TSER=0
      4 0.939576    server-ip-address            client-ip-address
       TCP      ssh > 57188 [SYN, ACK] Seq=0 Ack=1 Win=741376 Len=0
MSS=1460 TSV=277654721 TSER=326215633 WS=7
      5 0.941794    client-ip-address        server-ip-address
           TCP      57188 > ssh [ACK] Seq=1 Ack=1 Win=65535 Len=0
TSV=326215633 TSER=277654721
      6 0.951588    server-ip-address            client-ip-address
       SSHv2    Server Protocol: SSH-1.99-OpenSSH_4.3
      7 0.955164    client-ip-address        server-ip-address
           TCP      57188 > ssh [ACK] Seq=1 Ack=22 Win=65535 Len=0
TSV=326215633 TSER=277654733
      8 0.956787    client-ip-address        server-ip-address
           SSHv2    Client Protocol: SSH-2.0-OpenSSH_4.2
      9 0.956802    server-ip-address            client-ip-address
       TCP      ssh > 57188 [ACK] Seq=22 Ack=21 Win=5888 Len=0
TSV=277654738 TSER=326215633
     10 0.957918    server-ip-address            client-ip-address
       SSHv2    Server: Key Exchange Init[Packet size limited during
capture]
     11 0.961538    client-ip-address        server-ip-address
           TCP      57188 > ssh [ACK] Seq=21 Ack=726 Win=65535 Len=0
TSV=326215633 TSER=277654739
     12 48.095708   server-ip-address            client-ip-address
       TCP      ssh > 50938 [FIN, ACK] Seq=0 Ack=4294966560 Win=46 Len=0
TSV=277701878 TSER=326215487 SLE=0 SRE=1
     13 48.121979   client-ip-address        server-ip-address
           TCP      50938 > ssh [FIN, ACK] Seq=0 Ack=1 Win=65535 Len=0
TSV=326215727 TSER=277701878
     14 48.122001   server-ip-address            client-ip-address
       TCP      [TCP ACKed lost segment] ssh > 50938 [RST] Seq=1 Len=0
     15 48.460033   client-ip-address        server-ip-address
           TCP      [TCP Previous segment lost] 57188 > ssh [FIN, ACK]
Seq=757 Ack=726 Win=65535 Len=0 TSV=326215728 TSER=277654739
     16 48.460043   server-ip-address            client-ip-address
       TCP      [TCP Dup ACK 10#1] ssh > 57188 [ACK] Seq=726 Ack=21
Win=5888 Len=0 TSV=277702242 TSER=326215633 SLE=757 SRE=758

-- ssh client logs --
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to server [xxx.xxx.xxx.xxx] port 22.
debug1: Connection established.
debug1: identity file /Users/xxxxx/.ssh/identity type -1
debug1: identity file /Users/xxxxx/.ssh/id_rsa type 1
debug1: identity file /Users/xxxxx/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.2
debug1: Miscellaneous failure
No credentials cache found

debug1: Miscellaneous failure
No credentials cache found

debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

-- ssh client packet logs --
No.     Time        Source                Destination           Protocol
Info
      1 0.000000    client-ip-address          server-ip-address        
TCP      51475 > ssh [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460 WS=0
TSV=326210863 TSER=0
      2 0.006043    server-ip-address         client-ip-address        
TCP      ssh > 51475 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460
TSV=275222302 TSER=326210863 WS=7
      3 0.006132    client-ip-address          server-ip-address        
TCP      51475 > ssh [ACK] Seq=1 Ack=1 Win=65535 Len=0 TSV=326210863
TSER=275222302
      4 0.016366    server-ip-address         client-ip-address        
SSHv2    Server Protocol: SSH-2.0-OpenSSH_4.3
      5 0.016483    client-ip-address          server-ip-address        
TCP      51475 > ssh [ACK] Seq=1 Ack=21 Win=65535 Len=0 TSV=326210863
TSER=275222312
      6 0.017673    client-ip-address          server-ip-address        
SSHv2    Client Protocol: SSH-2.0-OpenSSH_4.2
      7 0.021603    server-ip-address         client-ip-address        
TCP      ssh > 51475 [ACK] Seq=21 Ack=21 Win=5888 Len=0 TSV=275222317
TSER=326210863
      8 0.024625    server-ip-address         client-ip-address        
SSHv2    Server: Key Exchange Init[Short Frame]
      9 0.024721    client-ip-address          server-ip-address        
TCP      51475 > ssh [ACK] Seq=21 Ack=725 Win=65535 Len=0 TSV=326210863
TSER=275222318
     10 0.152480    client-ip-address          server-ip-address        
SSHv2    Client: Key Exchange Init[Short Frame]
     11 0.155474    server-ip-address         client-ip-address        
ICMP     Destination unreachable (Host unreachable)
     12 1.551705    client-ip-address          server-ip-address        
SSHv2    [TCP Retransmission] Client: Key Exchange Init
     13 4.552823    client-ip-address          server-ip-address        
SSHv2    [TCP Retransmission] Client: Key Exchange Init
     14 10.554255   client-ip-address          server-ip-address        
SSHv2    [TCP Retransmission] Client: Key Exchange Init
     15 22.556332   client-ip-address          server-ip-address        
SSHv2    [TCP Retransmission] Encrypted request packet len=736
     16 46.559552   client-ip-address          server-ip-address        
SSHv2    [TCP Retransmission] Encrypted request packet len=736
     17 51.549655   client-ip-address          server-ip-address        
TCP      51475 > ssh [FIN, ACK] Seq=757 Ack=725 Win=65535 Len=0
TSV=326210966 TSER=275222318
     18 51.555941   server-ip-address         client-ip-address        
TCP      [TCP Dup ACK 7#1] ssh > 51475 [ACK] Seq=725 Ack=21 Win=5888
Len=0 TSV=275273850 TSER=326210863 SLE=757 SRE=758  

[Darren Tucker]

Davin Flatten wrote:

> Hello-
>
> I am having a strange problem ever since we applied the Fedora Core 5
> update to the Openssh RPM's.  Ever since the update when some users
> connect thru a NAT gateway to the NAT'ed server the connection hangs.
> This occurs only for some combinations of firewalls.  Below is all the
> information I could gather on the subject.  Has anyone had this same
> problem and found a solution?
>
> The setup is as follows:
>      ssh server <---Nat firewall #1 <--Internet <--Nat firewall #2<--ssh
> client

I would put money on it being this:
http://www.snailbook.com/faq/mtu-mismatch.auto.html

[quote]
Short Answer
You probably have an MTU/fragmentation problem. For each network
interface on both client and server set the MTU to 576, eg ifconfig eth0
mtu 576. If the problem goes away, read on.

Long Answer
Long answer: At each routing hop, IP packets bigger than the outgoing
interface’s Maximum Transmission Unit (MTU) get fragmented. Only the
first fragment has TCP port numbers. Firewalls often behave badly in the
presence of packet fragmentation, dropping everything but the first
fragment since the subsequent ones can’t be matched against the firewall
rules. Some NAT configuration (eg many-to-one NAT or port address
translation) can’t match the fragments against their translation state
tables.

Arguably, such devices should perform packet reassembly first so as to
properly consider fragmented packets. However, this is more complicated
and so is often not done. Also, this feature would raise a possible
starvation attack against the packet filter, by sending many bogus
initial fragments and causing the device to store them for reassembly
with subsequent packets which will never come.

Logging in and using the shell will normally generate relatively small
packets, and so the initial connection proceeds normally ; however if do
you something that generates a lot of data (eg cat'ing a big file or
starting an X Windows application), you may generate a packet bigger
than the MTU.

Let's say it’s a 1500 byte IP packet and the router has 2 different
MTU's (say 1500 & 1484) and no firewall. When the router goes to forward
it, the packet is too big for the interface MTU (1484), so the router
breaks it into 2 fragments, 0 and 1. Fragment 0 contains the first 1484
bytes (including the TCP source and dest ports) and fragment 1 contains
the remaining 16 bytes. Both fragments are sent on to their destinations.

When the first fragment reaches its target, it’s held by the IP stack
until the remaining fragments arrive, at which time the IP packet is
reassembled and passed up the stack to TCP. If all fragments are not
received by the timeout, the entire IP packet is discarded and an ICMP
"timeout during reassembly" error is sent back.

Now add your firewall, which drops fragment 1. Your 1500 byte IP packet
times out during reassembly and TCP retries, by sending another 1500
byte packet. Repeat. Eventually, TCP will time out and you’ll get a
connection termination.

IP stack parameters (such as Path MTU Discovery) and external variables
(such as the MTU's of all the hops between hosts) can also affect
whether or not a given connection will have this problem.
[/quote]

> Firewall #1 is an OpenBSD gateway running m0n0wall and the Firewall #2
> depends on which client is connecting.

I thought m0n0wall was FreeBSD based, but if it uses PF and you're using
"keep state" rules then make sure you're only creating state on the TCP
SYN packets, eg:

pass in [...] proto tcp [...] flags S/SA keep state

If you omit the "flags S/SA" then depending on your rules you may end up
creating state on a reply packet which may result in PF's idea of the
TCP window scaling being wrong.  This will mostly work but can cause all
kinds of weird problems...

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
     Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

[dkensiski]

The "No credentials found" error is because your client is trying to look up Kerberos credentials, but you are probably not using Kerberos.  I was able to resolve the issue by simply adding the following to the end of /etc/ssh_config:

    GSSAPIAuthentication no
    GSSAPIKeyExchange no

No restart necessary -- the next time you initate an SSH conneciton, it should take effect.

I had one or two stubborn systems where it still happened and I resolved those by putting the same lines in /etc/sshd_config so it changed the way the server behaves.  In this case, you'll have to restart the SSH daemon for the changes to take effect.

Hope it helps!
--Dave