xen and networking problems

[Brook Milligan]

Now that I have gotten a xen kernel to boot (thanks!)[1], I am having
trouble with the network interface under xen.  In short, the kernel
boots and apparently configures the interface, but no packets can be
transmitted.  The identical configuration works fine with a MONOLITHIC
kernel compiled from current source as of 2009-08-01 (which is the
same as for the xen kernel and the entire system).

When booted with 64Mb (of a total 16Gb) of RAM for dom0, the
XEN3PAE_DOM0 kernel probes the network interfaces as follows:

     fxp0 at pci7 dev 2 function 0: i82550 Ethernet, rev 13
     ioapic0: int17 0x1a9c0<vector=0xc0,delmode=0x1,logical,actlo,level,masked,dest=0x0> 0xf000000<target=0xf>
     fxp0: interrupting at ioapic0 pin 17, event channel 6
     fxp0: Ethernet address 00:30:48:23:3a:81
     inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
     inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
     fxp1 at pci7 dev 3 function 0: i82550 Ethernet, rev 13
     fxp1: interrupting at ioapic0 pin 18, event channel 5
     xen_alloc_contig: XENMEM_decrease_reservation failed!
     fxp1: unable to allocate control data, error = 12

Note the error with the second interface.  When booted with either
256Mb or 1024Mb of RAM the same error shows up, but for both network
interfaces.

The one physical interface is configured as follows:

     # ifconfig.fxp0
     media autoselect
     inet 172.16.0.2/24

and a bridge is configured as follows for later use by domUs:

     # ifconfig.bridge0
     create
     !brconfig $int add fxp0
     !brconfig $int stp fxp0
     !brconfig $int up

'ifconfig -a' reports the following (which is essentially identical,
barring the missing fxp1 interface, to the report when running
successfully under the MONOLITHIC kernel):

     fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
             capabilities=3f00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx>
             enabled=0
             address: 00:30:48:23:3a:81
             media: Ethernet autoselect (100baseTX full-duplex)
             status: active
             inet 172.16.0.2 netmask 0xffffff00 broadcast 172.16.0.255
             inet6 fe80::230:48ff:fe23:3a81%fxp0 prefixlen 64 scopeid 0x1
     lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33184
             inet 127.0.0.1 netmask 0xff000000
             inet6 ::1 prefixlen 128
             inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
     bridge0: flags=41<UP,RUNNING> mtu 1500

However, no packets can be transmitted.  For example, ping reports
'host is down' errors.

Where shall I be looking for solutions to this?  Are there known
networking problems with xen?  Is this something special with the fxp
driver?  with my hardware?  Any suggestions on where to start are
welcome.  I very much appreciate the help.

Thanks alot.

Cheers,
Brook

[1] See the thread starting with
    http://mail-index.netbsd.org/port-xen/2009/10/29/msg005516.html

[Manuel Bouyer]

On Thu, Oct 29, 2009 at 09:48:17PM -0600, Brook Milligan wrote:

> Now that I have gotten a xen kernel to boot (thanks!)[1], I am having
> trouble with the network interface under xen.  In short, the kernel
> boots and apparently configures the interface, but no packets can be
> transmitted.  The identical configuration works fine with a MONOLITHIC
> kernel compiled from current source as of 2009-08-01 (which is the
> same as for the xen kernel and the entire system).
>
> When booted with 64Mb (of a total 16Gb) of RAM for dom0, the
> XEN3PAE_DOM0 kernel probes the network interfaces as follows:
>
>      fxp0 at pci7 dev 2 function 0: i82550 Ethernet, rev 13
>      ioapic0: int17 0x1a9c0<vector=0xc0,delmode=0x1,logical,actlo,level,masked,dest=0x0> 0xf000000<target=0xf>
>      fxp0: interrupting at ioapic0 pin 17, event channel 6
>      fxp0: Ethernet address 00:30:48:23:3a:81
>      inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
>      inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
>      fxp1 at pci7 dev 3 function 0: i82550 Ethernet, rev 13
>      fxp1: interrupting at ioapic0 pin 18, event channel 5
>      xen_alloc_contig: XENMEM_decrease_reservation failed!
>      fxp1: unable to allocate control data, error = 12
>
> Note the error with the second interface.  When booted with either
> 256Mb or 1024Mb of RAM the same error shows up, but for both network
> interfaces.
>
> The one physical interface is configured as follows:
>
>      # ifconfig.fxp0
>      media autoselect
>      inet 172.16.0.2/24
>
> and a bridge is configured as follows for later use by domUs:
>
>      # ifconfig.bridge0
>      create
>      !brconfig $int add fxp0
>      !brconfig $int stp fxp0
>      !brconfig $int up
>
> 'ifconfig -a' reports the following (which is essentially identical,
> barring the missing fxp1 interface, to the report when running
> successfully under the MONOLITHIC kernel):
>
>      fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
>     capabilities=3f00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx>
>     enabled=0
>     address: 00:30:48:23:3a:81
>     media: Ethernet autoselect (100baseTX full-duplex)
>     status: active
>     inet 172.16.0.2 netmask 0xffffff00 broadcast 172.16.0.255
>     inet6 fe80::230:48ff:fe23:3a81%fxp0 prefixlen 64 scopeid 0x1
>      lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33184
>     inet 127.0.0.1 netmask 0xff000000
>     inet6 ::1 prefixlen 128
>     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>      bridge0: flags=41<UP,RUNNING> mtu 1500
>
> However, no packets can be transmitted.  For example, ping reports
> 'host is down' errors.

Please check with netstat if you have interrupt for this device.
You could also run tcpdump and see if you receive some packets.

--
Manuel Bouyer <bouyer@...>
     NetBSD: 26 ans d'experience feront toujours la difference
--

[Brook Milligan]

Manuel Bouyer writes:
 > Please check with netstat if you have interrupt for this device.

# vmstat -i
interrupt                                     total     rate
vcpu0 ioapic0 pin 17                           1205        9
vcpu0 ioapic0                                  2644       21
vcpu0 ioapic0                                    10        0
vcpu0 ioapic0                                   485        3
vcpu0 clock                                   13057      104
vcpu0 xenbus                                     53        0
Total                                         17454      139

Note that the ioapic0 pin 17 device corresponds to the fxp0 network
interface.  This command was run while pinging the interface, and the
number of interrupts increases each time the command is run.  I
presume this means that the hardware is generating interrupts as
packets arrive.

In addition, the results of netstat are below.  The number of packets
in increased with each run of the command and the number of packets
out increased if I pinged from this machine (although outgoing pings
always resulted in 'ping: sendto: Host is down' errors).

# netstat -i
Name  Mtu   Network       Address              Ipkts Ierrs    Opkts Oerrs Colls
fxp0  1500  <Link>        00:30:48:23:3a:81     1702     0       14     0     0
fxp0  1500  176.16.0.2/24 172.16.0.2            1702     0       14     0     0
fxp0  1500  fe80::/64     fe80::230:48ff:fe     1702     0       14     0     0
lo0   33184 <Link>                                 0     0        0     0     0
lo0   33184 127/8         localhost                0     0        0     0     0
lo0   33184 localhost/128 ::1                      0     0        0     0     0
lo0   33184 fe80::/64     fe80::1                  0     0        0     0     0

 > You could also run tcpdump and see if you receive some packets.

tcpdump never captured any packets under these tests, which were all
done without creating the bridge0 interface.  (The bridge was present
when I generated output for earlier posts.)  Before I disabled
bridge0, however, tcpdump reported only STP packets.  For these tests
I disabled the bridge to make sure that it wasn't interfering with the
primary network interface.

Does all of this mean that the packets are arriving at the hardware
interface, are being partially processed by the kernel network stack,
but are being lost somewhere along the way before making it to any
applications?

What else can be done to track down what is going on here?

By the way, without a serial console is there a way to capture the
boot messages from the xen hypervisor?  Alternatively, is there a way
to pause the boot process so I can review the messages before they
scroll off the screen?

Thanks again for your help.

Cheers,
Brook

[Manuel Bouyer]

On Fri, Oct 30, 2009 at 02:43:26PM -0600, Brook Milligan wrote:

> Manuel Bouyer writes:
>  > Please check with netstat if you have interrupt for this device.
>
> # vmstat -i
> interrupt                                     total     rate
> vcpu0 ioapic0 pin 17                           1205        9
> vcpu0 ioapic0                                  2644       21
> vcpu0 ioapic0                                    10        0
> vcpu0 ioapic0                                   485        3
> vcpu0 clock                                   13057      104
> vcpu0 xenbus                                     53        0
> Total                                         17454      139
>
> Note that the ioapic0 pin 17 device corresponds to the fxp0 network
> interface.  This command was run while pinging the interface, and the
> number of interrupts increases each time the command is run.  I
> presume this means that the hardware is generating interrupts as
> packets arrive.

So interrupts are probably working

>
> In addition, the results of netstat are below.  The number of packets
> in increased with each run of the command and the number of packets
> out increased if I pinged from this machine (although outgoing pings
> always resulted in 'ping: sendto: Host is down' errors).
>
> # netstat -i
> Name  Mtu   Network       Address              Ipkts Ierrs    Opkts Oerrs Colls
> fxp0  1500  <Link>        00:30:48:23:3a:81     1702     0       14     0     0
> fxp0  1500  176.16.0.2/24 172.16.0.2            1702     0       14     0     0
> fxp0  1500  fe80::/64     fe80::230:48ff:fe     1702     0       14     0     0
> lo0   33184 <Link>                                 0     0        0     0     0
> lo0   33184 127/8         localhost                0     0        0     0     0
> lo0   33184 localhost/128 ::1                      0     0        0     0     0
> lo0   33184 fe80::/64     fe80::1                  0     0        0     0     0
>
>  > You could also run tcpdump and see if you receive some packets.
>
> tcpdump never captured any packets under these tests, which were all
> done without creating the bridge0 interface.  (The bridge was present
> when I generated output for earlier posts.)  Before I disabled
> bridge0, however, tcpdump reported only STP packets.  For these tests
> I disabled the bridge to make sure that it wasn't interfering with the
> primary network interface.

Did you run it with -n ? Otherwise, IP reverse resolution could cause it
to stall.

>
> Does all of this mean that the packets are arriving at the hardware
> interface, are being partially processed by the kernel network stack,
> but are being lost somewhere along the way before making it to any
> applications?

tcpdump being silent makes it worse than that, as the packets are
passed to bpf from the driver itself, in the interrupt routine.


>
> What else can be done to track down what is going on here?

check that it's not running out of mbufs. vmstat -m will tell you
if mbuf allocation fail (mbuf and mbuf cluster).

>
> By the way, without a serial console is there a way to capture the
> boot messages from the xen hypervisor?  Alternatively, is there a way
> to pause the boot process so I can review the messages before they
> scroll off the screen?

'xm dmesg' should give them to you

--
Manuel Bouyer <bouyer@...>
     NetBSD: 26 ans d'experience feront toujours la difference
--

[Brook Milligan]

Manuel Bouyer writes:
 > Did you run it with -n ? Otherwise, IP reverse resolution could cause it
 > to stall.

Good point.  I ran it before without -n.  This time I tried it again
with -n.  Same thing; no packets captured.  When I terminate tcpdump
with ^C, it reports 0 packets captured/received/dropped.

 > tcpdump being silent makes it worse than that, as the packets are
 > passed to bpf from the driver itself, in the interrupt routine.

So, I'm guessing this implies a problem early in the packet
processing?  Is there a way to verify that?

 > check that it's not running out of mbufs. vmstat -m will tell you
 > if mbuf allocation fail (mbuf and mbuf cluster).

See the output of vmstat -m below.  I'm not sure what to look at
exactly.  Does this indicate any problems?

 > 'xm dmesg' should give them to you

Thanks.  Just for reference, I've included that output as well.

I hope there are some clues in the output I'm providing.  I greatly
appreciate all the help and hope we can find a solution.

Cheers,
Brook

===========================================================================

# vmstat -m
Memory statistics by bucket size
    Size   In Use   Free   Requests  HighWater  Couldfree
      16      273    239       1759    1280          0
      32      492     20       2191     640          0
      64      787     45        870     320          0
     128      676     28       1216     160          0
     256      551      9        596      80          0
     512       36     12         45      40          0
    1024       94      2        306      20          0
    2048       27      3         40      10          0
    4096       25      2        545       5          0
    8192        5      0          6       5          0
   16384        2      0          2       5          0
  131072        2      0          2       5          0

Memory usage type by bucket size
    Size  Type(s)
      16  ptyfs mount, kernfs mount, prop string, VM swap, USB, USB device,
          soname, sockaddr, vmem, NETSMBDEV, sysctldata, devbuf, pcb, temp,
          routetbl, UFS mount, in_multi, ifaddr, acpi
      32  VM swap, USB, soname, sockaddr, tcpcongctl, blist, RAIDframe,
          sysctldata, devbuf, pcb, temp, routetbl, ifmedia, ifaddr, ether_multi,
          acpi
      64  USB, vmem, NDP, sysctldata, devbuf, DMA map, temp, ifaddr, acpi
     128  prop dictionary, VM swap, USB, USB device, soname, NDP, sysctlnode,
          sysctldata, devbuf, pcb, temp, routetbl, in_multi, ifaddr, acpi
     256  NTFS mount, MSDOSFS mount, VM page, USB, sysctldata, devbuf, DMA map,
          temp, routetbl, UFS mount, ifaddr, acpi
     512  VM map, vmem, sysctldata, devbuf, temp, ifaddr, crypto, acpi
    1024  USB, sysctlnode, sysctldata, devbuf, temp, acpi
    2048  USB device, vmem, RAIDframe, sysctlnode, sysctldata, temp, UFS mount
    4096  vmem, RAIDframe, sysctlnode, devbuf, temp, UFS mount
    8192  sysctlnode, devbuf
   16384  sysctlnode, devbuf
  131072  blist, devbuf

Memory statistics by type                                Type  Kern
           Type InUse  MemUse HighUse   Limit   Requests Limit Limit Size(s)
    ptyfs mount     1      1K      1K  39322K          1     0     0 16
     NTFS mount     0      0K      1K  39322K          1     0     0 256
  MSDOSFS mount     0      0K      1K  39322K          1     0     0 256
   kernfs mount     1      1K      1K  39322K          1     0     0 16
prop dictionary   123     16K     16K  39322K        131     0     0 128
    prop string   120      2K      2K  39322K        120     0     0 16
        VM swap     3      1K      1K  39322K          3     0     0 16,32,128
        VM page     1      1K      1K  39322K          1     0     0 256
         VM map     3      2K      2K  39322K          3     0     0 512
            USB    46      5K      5K  39322K         52     0     0 16,32,64,128,256,1024
     USB device    18     19K     19K  39322K         18     0     0 16,128,2048
         soname    30      4K      4K  39322K         55     0     0 16,32,128
       sockaddr    52      2K      2K  39322K         91     0     0 16,32
     tcpcongctl     2      1K      1K  39322K          2     0     0 32
           vmem    10     15K     15K  39322K         15     0     0 16,64,512,2048,4096
          blist     2     73K     73K  39322K          2     0     0 32,131072
      NETSMBDEV     1      1K      1K  39322K          1     0     0 16
      RAIDframe    10     35K     35K  39322K         10     0     0 32,2048,4096
            NDP     4      1K      1K  39322K          5     0     0 64,128
     sysctlnode   111    163K    163K  39322K        142     0     0 128,1024,2048,4096,8192,16384
     sysctldata    59      6K      6K  39322K         74     0     0 16,32,64,128,256,512,1024,2048
         devbuf   145    157K    158K  39322K        190     0     0 16,32,64,128,256,512,1024,4096,8192,16384,131072
        DMA map   816    148K    148K  39322K        816     0     0 64,256
            pcb    89     11K     12K  39322K        120     0     0 16,32,128
           temp    15      6K     14K  39322K        758     0     0 16,32,64,128,256,512,1024,2048,4096
       routetbl    32      3K      4K  39322K         69     0     0 16,32,128,256
      UFS mount     7     13K     13K  39322K          7     0     0 16,256,2048,4096
       in_multi    22      2K      2K  39322K         22     0     0 16,128
        ifmedia     5      1K      1K  39322K          5     0     0 32
         ifaddr    17      4K      4K  39322K         17     0     0 16,32,64,128,256,512
    ether_multi     4      1K      1K  39322K          4     0     0 32
         crypto     1      1K      1K  39322K          1     0     0 512
           acpi  1220     86K     86K  39322K       4840     0     0 16,32,64,128,256,512,1024

Memory totals:  In Use    Free    Requests
                  762K     35K        7578

Memory resource pool statistics
Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
amappl        48      417    0        0     5     0     5     5     0   inf    0
anonpl        16     5620    0        0    23     0    23    23     0   inf    0
aobj          72        7    0        0     1     0     1     1     0   inf    0
ataspl        56     2318    0     2318     1     0     1     1     0   inf    1
biopl        176       12    0        0     1     0     1     1     0   inf    0
buf16k      16384     139    0       17    34     1    33    34     1     1    0
buf1k       1024        6    0        6     1     0     1     1     1     1    1
buf2k       2048      110    0        5     4     0     4     4     1     1    0
buf4k       4096        7    0        0     1     0     1     1     1     1    0
buf512b      512        1    0        1     1     0     1     1     1     1    1
buf8k       8192       10    0        5     2     0     2     2     1     1    1
bufpl        176      252    0        0    11     0    11    11     0   inf    0
cwdi          24       21    0        0     1     0     1     1     0   inf    0
execargs    262144    201    0      201     4     0     4     4     0    16    4
extent        20        2    0        1     1     0     1     1     0   inf    0
fdfile        28      299    0        0     3     0     3     3     0   inf    0
ffsdino1     128     3262    0        0   106     0   106   106     0   inf    0
ffsino       168     3262    0        0   136     0   136   136     0   inf    0
file          64      164    0        0     3     0     3     3     0   inf    0
filedesc     652       21    0        0     4     0     4     4     0   inf    0
fstrans       20       26    0        0     1     0     1     1     0   inf    0
in6pcbpl     144       15    0       14     1     0     1     1     0   inf    0
inmltpl       32        2    0        0     1     0     1     1     0   inf    0
inpcbpl      112       12    0       10     1     0     1     1     0   inf    0
kcredpl      164       59    0        0     3     0     3     3     0   inf    0
kmem-100     100       31    0        0     7     0     7     7     0   inf    0
kmem-1024   1024      112    0        0   112     0   112   112     0   inf    0
kmem-12       12      490    0        0    12     0    12    12     0   inf    0
kmem-128     128       15    0        0     4     0     4     4     0   inf    0
kmem-1280   1280       26    0        0    26     0    26    26     0   inf    0
kmem-16       16      131    0        0     5     0     5     5     0   inf    0
kmem-1792   1792        2    0        0     2     0     2     2     0   inf    0
kmem-20       20       76    0        0     4     0     4     4     0   inf    0
kmem-2048   2048        4    0        0     4     0     4     4     0   inf    0
kmem-2304   2304        3    0        0     3     0     3     3     0   inf    0
kmem-24       24      148    0        0     8     0     8     8     0   inf    0
kmem-2560   2560        5    0        0     5     0     5     5     0   inf    0
kmem-28       28        7    0        0     1     0     1     1     0   inf    0
kmem-32       32     2195    0        0   138     0   138   138     0   inf    0
kmem-3328   3328        1    0        0     1     0     1     1     0   inf    0
kmem-3584   3584        5    0        0     5     0     5     5     0   inf    0
kmem-36       36     2182    0        0   156     0   156   156     0   inf    0
kmem-40       40      127    0        0    11     0    11    11     0   inf    0
kmem-44       44        6    0        0     1     0     1     1     0   inf    0
kmem-48       48       20    0        0     2     0     2     2     0   inf    0
kmem-512     512       27    0        0    27     0    27    27     0   inf    0
kmem-56       56       13    0        0     2     0     2     2     0   inf    0
kmem-64       64        7    0        0     1     0     1     1     0   inf    0
kmem-72       72      306    0        0    44     0    44    44     0   inf    0
kmem-768     768       70    0        0    70     0    70    70     0   inf    0
kmem-8         8        1    0        0     1     0     1     1     0   inf    0
kmem-84       84      112    0        0    19     0    19    19     0   inf    0
ksiginfo      48        1    0        0     1     0     1     1     0   inf    0
kvakernel   4096      610    0        0    20     0    20    20     0   inf    0
kvakmem     4096       72    0        0     3     0     3     3     0   inf    0
lockf         64        8    0        0     1     0     1     1     0   inf    0
lwppl        672       61    0        0    11     0    11    11     0   inf    0
lwpucpl      776       10    0       10     1     0     1     1     0   inf    1
mbpl         256      136    0        0    10     0    10    10     1   inf    1
mclpl       2048      128    0        0    68     0    68    68     4   512    4
mutex         12      415    0        0     2     0     2     2     0   inf    0
ncache        96     3277    0        0    79     0    79    79     0   inf    0
pcache       576      103    0       18    13     0    13    13     0   inf    0
pcglarge     756       91    0       23    14     0    14    14     0   inf    0
pcgnormal    180      174    0        7     8     0     8     8     0   inf    0
pdict16       44        9    0        0     1     0     1     1     0   inf    0
pdict32       60        3    0        0     1     0     1     1     0   inf    0
pdppl       16384      26    0        5    21     0    21    21     0   inf    0
phpool-0      32      149    0        1     2     0     2     2     0   inf    0
phpool-64     36      759    0        0     7     0     7     7     0   inf    0
piperd       188        6    0        0     1     0     1     1     0   inf    0
pipewr       188        6    0        0     1     0     1     1     0   inf    0
plimitpl     196       11    0        0     1     0     1     1     0   inf    0
pmappl       124       21    0        0     1     0     1     1     0   inf    0
pnbufpl     1024       17    0        0     5     0     5     5     0   inf    0
procpl       476       21    0        0     3     0     3     3     0   inf    0
propdict      32      132    0        8     1     0     1     1     0   inf    0
propnmbr      32       14    0        1     1     0     1     1     0   inf    0
propstng      20      186    0        4     1     0     1     1     0   inf    0
pstatspl     268       21    0        0     2     0     2     2     0   inf    0
ptimerpl     164        4    0        0     1     0     1     1     0   inf    0
ptimerspl    140        4    0        0     1     0     1     1     0   inf    0
pvpl          20     7269    0        0    36     0    36    36     0   inf    0
ractx         24      657    0        0     4     0     4     4     0   inf    0
rndsample    528       45    0       41     1     0     1     1     0   inf    0
rtentpl      144       26    0        0     1     0     1     1     0   inf    0
sigacts     2056       21    0        0    21     0    21    21     0   inf    0
socket       320       92    0        0     8     0     8     8     0   inf    0
tcpcbpl      500       13    0       10     1     0     1     1     0   inf    0
tstilepl      52       61    0        0     1     0     1     1     0   inf    0
uaoeltpl      84        2    0        0     1     0     1     1     0   inf    0
uarea       12288      61    0        0    61     0    61    61     0   inf    0
ufsdir       264        2    0        0     1     0     1     1     0   inf    0
vmembt        28     1060    0        0     8     0     8     8     0   inf    0
vmmpepl       84      847    0        0    18     0    18    18     0   inf    0
vmsppl       212       21    0        0     2     0     2     2     0   inf    0
vnodepl      184     3277    0        0   149     0   149   149     0   inf    0

In use 6680K, total allocated 8638K; utilization 77.3%

===========================================================================

# xm dmesg
 __  __            _____  _____  ____  
 \ \/ /___ _ __   |___ / |___ / |___ \
  \  // _ \ '_ \    |_ \   |_ \   __) |
  /  \  __/ | | |  ___) | ___) | / __/
 /_/\_\___|_| |_| |____(_)____(_)_____|
                                       
(XEN) Xen version 3.3.2 (root@) (gcc version 4.1.3 20080704 prerelease (NetBSD nb2 20081120)) Wed Oct 28 18:11:20 MDT 2009
(XEN) Latest ChangeSet: unavailable
(XEN) Command line: dom0_mem=64M
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 2 seconds
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009f800 (usable)
(XEN)  000000000009f800 - 00000000000a0000 (reserved)
(XEN)  00000000000d8000 - 00000000000e0000 (reserved)
(XEN)  00000000000e4000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000f7ef0000 (usable)
(XEN)  00000000f7ef0000 - 00000000f7efc000 (ACPI data)
(XEN)  00000000f7efc000 - 00000000f7f00000 (ACPI NVS)
(XEN)  00000000f7f00000 - 00000000f7f80000 (usable)
(XEN)  00000000f7f80000 - 00000000f8000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec10000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff800000 - 00000000ffc00000 (reserved)
(XEN)  00000000fff00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000400000000 (usable)
(XEN) System RAM: 16255MB (16645180kB)
(XEN) ACPI: RSDP 000F6720, 0014 (r0 PTLTD )
(XEN) ACPI: RSDT F7EF8195, 0034 (r1 PTLTD    RSDT    6040000  LTP        0)
(XEN) ACPI: FACP F7EFBE60, 0074 (r1 INTEL  K_CANYON  6040000 PTL         8)
(XEN) ACPI: DSDT F7EF81C9, 3C97 (r1  INTEL   PLUMAS  6040000 MSFT  100000E)
(XEN) ACPI: FACS F7EFCFC0, 0040
(XEN) ACPI: APIC F7EFBED4, 00B4 (r1 PTLTD   APIC    6040000  LTP        0)
(XEN) ACPI: BOOT F7EFBF88, 0028 (r1 PTLTD  $SBFTBL$  6040000  LTP        1)
(XEN) ACPI: SPCR F7EFBFB0, 0050 (r1 PTLTD  $UCRTBL$  6040000 PTL         1)
(XEN) Xen heap: 9MB (9444kB)
(XEN) Domain heap initialised
(XEN) Processor #0 15:2 APIC version 20
(XEN) Processor #6 15:2 APIC version 20
(XEN) Processor #1 15:2 APIC version 20
(XEN) Processor #7 15:2 APIC version 20
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 3, version 32, address 0xfec80000, GSI 24-47
(XEN) IOAPIC[2]: apic_id 4, version 32, address 0xfec80400, GSI 48-71
(XEN) IOAPIC[3]: apic_id 5, version 32, address 0xfec81000, GSI 72-95
(XEN) IOAPIC[4]: apic_id 8, version 32, address 0xfec81400, GSI 96-119
(XEN) Enabling APIC mode:  Flat.  Using 5 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 1996.662 MHz processor.
(XEN) CPU0: Intel(R) XEON(TM) CPU 2.00GHz stepping 04
(XEN) Booting processor 1/6 eip 8c000
(XEN) CPU1: Intel(R) XEON(TM) CPU 2.00GHz stepping 04
(XEN) Booting processor 2/1 eip 8c000
(XEN) CPU2: Intel(R) XEON(TM) CPU 2.00GHz stepping 04
(XEN) Booting processor 3/7 eip 8c000
(XEN) CPU3: Intel(R) XEON(TM) CPU 2.00GHz stepping 04
(XEN) Total of 4 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) checking TSC synchronization across 4 CPUs: passed.
(XEN) Platform timer is 3.579MHz ACPI PM Timer
(XEN) Brought up 4 CPUs
(XEN) I/O virtualisation disabled
(XEN) mtrr: your CPUs had inconsistent fixed MTRR settings
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 32-bit, PAE, lsb
(XEN)  Dom0 kernel: 32-bit, PAE, lsb, paddr 0xc0100000 -> 0xc0ab54fc
(XEN)  Dom0 symbol map 0xc0ab54fc -> 0xc0b89dc8
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   000000003e000000->000000003f000000 (12288 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: c0100000->c0b89dc8
(XEN)  Init. ramdisk: c0b8a000->c0b8a000
(XEN)  Phys-Mach map: c0b8a000->c0b9a000
(XEN)  Start info:    c0b9a000->c0b9a474
(XEN)  Page tables:   c0b9b000->c0ba8000
(XEN)  Boot stack:    c0ba8000->c0ba9000
(XEN)  TOTAL:         c0000000->c1000000
(XEN)  ENTRY ADDRESS: c0100000
(XEN) Dom0 has maximum 4 VCPUs
(XEN) Scrubbing Free RAM: ................................................................................................................................................................done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 96kB init memory.
(XEN) ioapic_guest_write: apic=0, pin=2, old_irq=0, new_irq=-1
(XEN) ioapic_guest_write: old_entry=000009f0, new_entry=00010900
(XEN) ioapic_guest_write: Attempt to remove IO-APIC pin of in-use IRQ!

[Manuel Bouyer]

On Fri, Oct 30, 2009 at 03:32:04PM -0600, Brook Milligan wrote:
> Manuel Bouyer writes:
>  > Did you run it with -n ? Otherwise, IP reverse resolution could cause it
>  > to stall.
>
> Good point.  I ran it before without -n.  This time I tried it again
> with -n.  Same thing; no packets captured.  When I terminate tcpdump
> with ^C, it reports 0 packets captured/received/dropped.

very strange ...

>
>  > tcpdump being silent makes it worse than that, as the packets are
>  > passed to bpf from the driver itself, in the interrupt routine.
>
> So, I'm guessing this implies a problem early in the packet
> processing?  Is there a way to verify that?

Next step would be to put instrumentation in dev/ic/i82557.c,
fxp_start() and fxp_rxintr() are called (and if so, what they
do with the mbuf).

>
>  > check that it's not running out of mbufs. vmstat -m will tell you
>  > if mbuf allocation fail (mbuf and mbuf cluster).
>
> See the output of vmstat -m below.  I'm not sure what to look at
> exactly.  Does this indicate any problems?

I didn't see. There's no failed allocation for mbpl and mclpl.

--
Manuel Bouyer <bouyer@...>
     NetBSD: 26 ans d'experience feront toujours la difference
--

[Brook Milligan]

Manuel Bouyer writes:
 > Next step would be to put instrumentation in dev/ic/i82557.c,
 > fxp_start() and fxp_rxintr() are called (and if so, what they
 > do with the mbuf).

Ok, I've started that.  

For reference, I'm working with i82577.c v1.129.

I added some printf() statements at the beginning and end of
fxp_start, fxp_rxintr, and fxp_txintr.  fxp_rxintr is called many
times, presumably for every packet received by the hardware.

For each outgoing packet, for example created by ping, fxp_start is
called and completes, then fxp_txintr is called and completes.

At this point, I'm not certain how to investigate the fate of mbufs.

Any help from people who know something about this driver or kernel
buffer handling is greatly appreciated.

Thanks alot.

Cheers,
Brook

[Manuel Bouyer]

On Tue, Nov 03, 2009 at 04:57:10PM -0700, Brook Milligan wrote:

> Manuel Bouyer writes:
>  > Next step would be to put instrumentation in dev/ic/i82557.c,
>  > fxp_start() and fxp_rxintr() are called (and if so, what they
>  > do with the mbuf).
>
> Ok, I've started that.  
>
> For reference, I'm working with i82577.c v1.129.
>
> I added some printf() statements at the beginning and end of
> fxp_start, fxp_rxintr, and fxp_txintr.  fxp_rxintr is called many
> times, presumably for every packet received by the hardware.
>
> For each outgoing packet, for example created by ping, fxp_start is
> called and completes, then fxp_txintr is called and completes.
>
> At this point, I'm not certain how to investigate the fate of mbufs.
>
> Any help from people who know something about this driver or kernel
> buffer handling is greatly appreciated.

Did you check with tcpdump on the remote end that the remote end is
getting the ICMP echo request sent by the fxp adapter when ping is
running ? That would confirm that transmit is working.

For receive, you could add a printf to check that in fxp_rxintr,
the code does call (*ifp->if_input)(ifp, m);
If it does not, the next step is to add more printf in fxp_rxintr to
see at which point the packet is dropped (there are many places for this,
basically each time that FXP_INIT_RFABUF() is called.

--
Manuel Bouyer, LIP6, Universite Paris VI.           Manuel.Bouyer@...
     NetBSD: 26 ans d'experience feront toujours la difference
--

[Brook Milligan]

Manuel Bouyer writes:
 > Did you check with tcpdump on the remote end that the remote end is
 > getting the ICMP echo request sent by the fxp adapter when ping is
 > running ? That would confirm that transmit is working.

The remote end is seeing (and responding to) arp requests from the xen
machine.

 > For receive, you could add a printf to check that in fxp_rxintr,
 > the code does call (*ifp->if_input)(ifp, m);
 > If it does not, the next step is to add more printf in fxp_rxintr to
 > see at which point the packet is dropped (there are many places for this,
 > basically each time that FXP_INIT_RFABUF() is called.

With more instrumentation it is clear that fxp_rxintr is exiting
immediately.  The value of rfa->rfa_status (and thus rxstat) is always
0 on entry to the function.  Consequently, the early checks cause an
immediate exit.

So, it seems that transmission of packets is ok but incoming packets
are never seen by the driver because their status information is
wrong.  Does this imply that the hardware is not providing the correct
information (it does fine without xen) or that xen is interfering in
some way?

What is the next step to debugging this?

Thanks again for the help.

Cheers,
Brook

[Brook Milligan]

Here is some additional information.  It seems that the presence or
absence of PAE support may be related to the lack of a functioning fxp
interface.  I have now tested the machine with a series of xen/kernel
combinations.  These were all built from cvs checkout of source as of
2009-08-01 for src and pkgsrc-2009Q3 for pkgsrc.

xen kernel fxp
none MONOLITHIC functional
3.1.4 XEN3_DOM0 functional
3.1.4/PAE XEN3PAE_DOM0 nonfunctional
3.3.2 XEN3PAE_DOM0 nonfunctional

As far as I can tell, xen v3.3.2 will only boot PAE kernels.

By a nonfunctional fxp interface, I mean that upon entry into the
fxp_rxintr function the value of rfa->rfa_status is always zero, which
is interpreted by the driver to mean that no packets are available.

What about the PAE support in xen, the kernel, or both could lead to
this problem?

Any help is greatly appreciated.

Thanks alot.

Cheers,
Brook

[Manuel Bouyer]

On Thu, Nov 05, 2009 at 08:02:22AM -0700, Brook Milligan wrote:

> Here is some additional information.  It seems that the presence or
> absence of PAE support may be related to the lack of a functioning fxp
> interface.  I have now tested the machine with a series of xen/kernel
> combinations.  These were all built from cvs checkout of source as of
> 2009-08-01 for src and pkgsrc-2009Q3 for pkgsrc.
>
> xen kernel fxp
> none MONOLITHIC functional
> 3.1.4 XEN3_DOM0 functional
> 3.1.4/PAE XEN3PAE_DOM0 nonfunctional
> 3.3.2 XEN3PAE_DOM0 nonfunctional
>
> As far as I can tell, xen v3.3.2 will only boot PAE kernels.
>
> By a nonfunctional fxp interface, I mean that upon entry into the
> fxp_rxintr function the value of rfa->rfa_status is always zero, which
> is interpreted by the driver to mean that no packets are available.
>
> What about the PAE support in xen, the kernel, or both could lead to
> this problem?

How much RAM does you machine have ? If this systeme supports it it would also
be usefull to test amd64.

It looks like this driver doesn't deal well with 64bit physical addresses.
Would the attached patch help ?

--
Manuel Bouyer, LIP6, Universite Paris VI.           Manuel.Bouyer@...
     NetBSD: 26 ans d'experience feront toujours la difference
--

Index: i82557.c
===================================================================
RCS file: /cvsroot/src/sys/dev/ic/i82557.c,v
retrieving revision 1.130
diff -u -p -u -r1.130 i82557.c
--- i82557.c 15 Sep 2009 19:20:30 -0000 1.130
+++ i82557.c 5 Nov 2009 15:56:42 -0000
@@ -912,7 +912,7 @@ fxp_start(struct ifnet *ifp)
  tbdp++;
  for (seg = 0; seg < nsegs; seg++) {
  tbdp[seg].tb_addr =
-    htole32(dmamap->dm_segs[seg].ds_addr);
+    htole32((uint32_t)dmamap->dm_segs[seg].ds_addr);
  tbdp[seg].tb_size =
     htole32(dmamap->dm_segs[seg].ds_len);
  }
@@ -1150,7 +1150,7 @@ fxp_intr(void *arg)
  rxmap = M_GETCTX(sc->sc_rxq.ifq_head, bus_dmamap_t);
  fxp_scb_wait(sc);
  CSR_WRITE_4(sc, FXP_CSR_SCB_GENERAL,
-    rxmap->dm_segs[0].ds_addr +
+    (uint32_t)rxmap->dm_segs[0].ds_addr +
     RFA_ALIGNMENT_FUDGE);
  fxp_scb_cmd(sc, FXP_SCB_COMMAND_RU_START);
  }
@@ -2003,7 +2003,7 @@ fxp_init(struct ifnet *ifp)
  rxmap = M_GETCTX(sc->sc_rxq.ifq_head, bus_dmamap_t);
  fxp_scb_wait(sc);
  CSR_WRITE_4(sc, FXP_CSR_SCB_GENERAL,
-    rxmap->dm_segs[0].ds_addr + RFA_ALIGNMENT_FUDGE);
+    (uint32_t)rxmap->dm_segs[0].ds_addr + RFA_ALIGNMENT_FUDGE);
  fxp_scb_cmd(sc, FXP_SCB_COMMAND_RU_START);
 #endif
 
Index: i82557var.h
===================================================================
RCS file: /cvsroot/src/sys/dev/ic/i82557var.h,v
retrieving revision 1.47
diff -u -p -u -r1.47 i82557var.h
--- i82557var.h 12 May 2009 14:25:17 -0000 1.47
+++ i82557var.h 5 Nov 2009 15:56:42 -0000
@@ -179,7 +179,7 @@ struct fxp_softc {
  * fxp_rxdesc DMA map on a per-mbuf basis.
  */
  bus_dmamap_t sc_dmamap;
-#define sc_cddma sc_dmamap->dm_segs[0].ds_addr
+#define sc_cddma (uint32_t)sc_dmamap->dm_segs[0].ds_addr
 
  /*
  * Software state for transmit descriptors.
@@ -335,7 +335,7 @@ do { \
  \
  if ((__p_m = (sc)->sc_rxq.ifq_tail) != NULL) { \
  __p_rfa = FXP_MTORFA(__p_m); \
- __v = htole32(__rxmap->dm_segs[0].ds_addr + \
+ __v = htole32((uint32_t)__rxmap->dm_segs[0].ds_addr + \
     RFA_ALIGNMENT_FUDGE); \
  FXP_RFASYNC((sc), __p_m, \
     BUS_DMASYNC_POSTREAD|BUS_DMASYNC_POSTWRITE); \

[Brook Milligan]

Manuel Bouyer writes:
 > How much RAM does you machine have ? If this systeme supports it it would also
 > be usefull to test amd64.

I have tested the machine with 4GB, 8GB and 16GB RAM.  The problems
all occur whenever there is more than 4GB of RAM.  Networking works in
all the various xen/kernel combinations with only 4GB.  

The CPU is an Intel Xeon that apparently lacks EM64T support so I
assume that amd64 kernels are out.

 > It looks like this driver doesn't deal well with 64bit physical addresses.
 > Would the attached patch help ?

No, this seems to make no difference.  See below for notes on the
patch, which would not compile as originally given.

Clearly, the combination of PAE support and >4GB memory triggers the
problems.  Does this mean that xen is not really supporting PAE or
that the hardware is not really correct or that there is some specific
interaction with the network interface or ... ?  How can any of this
be tested?  In all cases everything about the machine seems to work,
except for the network interface when PAE support is involved and
there is >4GB RAM.

Concrete guidance for troubleshooting is really needed as this is way
beyond my understanding of the internals of the kernel.  I am,
however, willing to test out suggestions and try new code on the
machine.

Thanks alot for your help.

Cheers,
Brook

===========================================================================

For future reference, the following portion of the patch will not
compile because of the context in which the macro is used.

      > Index: i82557var.h
      > ===================================================================
      > RCS file: /cvsroot/src/sys/dev/ic/i82557var.h,v
      > retrieving revision 1.47
      > diff -u -p -u -r1.47 i82557var.h
      > --- i82557var.h 12 May 2009 14:25:17 -0000 1.47
      > +++ i82557var.h 5 Nov 2009 15:56:42 -0000
      > @@ -179,7 +179,7 @@ struct fxp_softc {
      >   * fxp_rxdesc DMA map on a per-mbuf basis.
      >   */
      >   bus_dmamap_t sc_dmamap;
      > -#define sc_cddma sc_dmamap->dm_segs[0].ds_addr
      > +#define sc_cddma (uint32_t)sc_dmamap->dm_segs[0].ds_addr
      >  
      >   /*
      >   * Software state for transmit descriptors.

Instead, the cast must be done when the macro is used as follows:

     --- i82557var.h.orig 2009-11-06 12:12:48.000000000 -0700
     +++ i82557var.h 2009-11-06 13:10:22.000000000 -0700
     @@ -255,9 +255,9 @@
      #define FXP_RXMAP_GET(sc) ((sc)->sc_rxmaps[(sc)->sc_rxfree++])
      #define FXP_RXMAP_PUT(sc, map) (sc)->sc_rxmaps[--(sc)->sc_rxfree] = (map)

     -#define FXP_CDTXADDR(sc, x) ((sc)->sc_cddma + FXP_CDTXOFF((x)))
     -#define FXP_CDTBDADDR(sc, x) ((sc)->sc_cddma + FXP_CDTBDOFF((x)))
     -#define FXP_CDTXPADADDR(sc) ((sc)->sc_cddma + FXP_CDTXPADOFF)
     +#define FXP_CDTXADDR(sc, x) ((uint32_t)(sc)->sc_cddma + FXP_CDTXOFF((x)))
     +#define FXP_CDTBDADDR(sc, x) ((uint32_t)(sc)->sc_cddma + FXP_CDTBDOFF((x)))
     +#define FXP_CDTXPADADDR(sc) ((uint32_t)(sc)->sc_cddma + FXP_CDTXPADOFF)

      #define FXP_CDTX(sc, x) (&(sc)->sc_control_data->fcd_txdescs[(x)])

[Thor Lancelot Simon-2]

On Fri, Nov 06, 2009 at 02:57:29PM -0700, Brook Milligan wrote:
>
> Clearly, the combination of PAE support and >4GB memory triggers the
> problems.  Does this mean that xen is not really supporting PAE or
> that the hardware is not really correct or that there is some specific
> interaction with the network interface or ... ?  How can any of this
> be tested?  In all cases everything about the machine seems to work,
> except for the network interface when PAE support is involved and
> there is >4GB RAM.

It could be the device.  Does it claim to support 64-bit DMA (dual address
cycle)?  If not, we should never be feeding it 64-bit addresses; we should
always be bouncing (the x86 PCI code is supposed to take care of this
automatically).  But if so, maybe the device's DAC support is broken;
try whacking the pci code to bounce all transfers > 4GB and see wha
happens, would be my suggestion -- it is pretty easy to test.

Thor

[Manuel Bouyer]

On Fri, Nov 06, 2009 at 02:53:03PM -0700, Brook Milligan wrote:
> Manuel Bouyer writes:
>  > How much RAM does you machine have ? If this systeme supports it it would also
>  > be usefull to test amd64.
>
> I have tested the machine with 4GB, 8GB and 16GB RAM.  The problems
> all occur whenever there is more than 4GB of RAM.  Networking works in
> all the various xen/kernel combinations with only 4GB.  


OK, so as I understant it, fxp works with a PAE kernel too, but only
if there's 4GB or less RAM in the system ?

If so then it's probably an issue with bus_dma(9) not being used
properly.

> Clearly, the combination of PAE support and >4GB memory triggers the
> problems.  Does this mean that xen is not really supporting PAE or
> that the hardware is not really correct or that there is some specific
> interaction with the network interface or ... ?  How can any of this
> be tested?  In all cases everything about the machine seems to work,
> except for the network interface when PAE support is involved and
> there is >4GB RAM.

As it's i386-only, a non-PAE kernel will never use ram above 4GB. So
it's not PAE ws non-PAE, it's really the 4Gb limit that triggers the
issue.

The bus_dma(9) usage in fxp needs to be audited. I've tested i386PAE on
a system with 8GB RAM and various devices, without problems. Unfortunably
I don't have a add-on fxp to test.

--
Manuel Bouyer <bouyer@...>
     NetBSD: 26 ans d'experience feront toujours la difference
--

[Brook Milligan]

Manuel Bouyer writes:
 > OK, so as I understant it, fxp works with a PAE kernel too, but only
 > if there's 4GB or less RAM in the system ?

Yes, networking works with all combinations of kernels with <4GB RAM.
Packet receipt does not work (but transmission does) with xen/PAE
kernels with >4GB RAM.

 > If so then it's probably an issue with bus_dma(9) not being used
 > properly.

 > As it's i386-only, a non-PAE kernel will never use ram above 4GB. So
 > it's not PAE ws non-PAE, it's really the 4Gb limit that triggers the
 > issue.

Yes.

 > The bus_dma(9) usage in fxp needs to be audited. I've tested i386PAE on
 > a system with 8GB RAM and various devices, without problems. Unfortunably
 > I don't have a add-on fxp to test.

Can you give any guidance on how to audit the usage of bus_dma(9)?  I
am happy to try patches if someone can provide them but doesn't have
suitable hardware.

Thanks for your help.

Cheers,
Brook