howto re-create XEN network bridge?

View: New views
11 Messages — Rating Filter:   Alert me  

howto re-create XEN network bridge?

by SoftDux :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi all,

Is it possible to re-create the XEN network bridge? On one of my XEN
servers, somce domU's can't connect to the outside world, yet I can
connect to the VPS' from the internet on SSH. It seems that outgoing
traffic is not working as it should, but incoming is, and I think it
might have todo with the bridge. How would I re-create the bridge to
see if it sovles the problem?

--
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

Re: howto re-create XEN network bridge?

by Fajar A. Nugraha-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, Nov 6, 2009 at 6:58 AM, Rudi Ahlers <Rudi@...> wrote:
> Hi all,
>
> Is it possible to re-create the XEN network bridge? On one of my XEN
> servers, somce domU's can't connect to the outside world, yet I can
> connect to the VPS' from the internet on SSH. It seems that outgoing
> traffic is not working as it should, but incoming is, and I think it
> might have todo with the bridge. How would I re-create the bridge to
> see if it sovles the problem?

Take a look at the output of "brctl show", and also see the contents
of "/etc/xen/scripts/network-bridge".
IMHO you should first make sure that it's not iptables problem. See
the content of
/proc/sys/net/ipv4/netfilter/ip_conntrack_count
/proc/sys/net/ipv4/netfilter/ip_conntrack_max

if count is close to max, you could either:
- increase ip_conntrack_max, or
- disable iptables for bridged traffic (echo 0 >
/proc/sys/net/bridge/bridge-nf-call-iptables)

--
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

Re: howto re-create XEN network bridge?

by SoftDux :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, Nov 6, 2009 at 11:48 AM, Fajar A. Nugraha <fajar@...> wrote:

> On Fri, Nov 6, 2009 at 6:58 AM, Rudi Ahlers <Rudi@...> wrote:
>> Hi all,
>>
>> Is it possible to re-create the XEN network bridge? On one of my XEN
>> servers, somce domU's can't connect to the outside world, yet I can
>> connect to the VPS' from the internet on SSH. It seems that outgoing
>> traffic is not working as it should, but incoming is, and I think it
>> might have todo with the bridge. How would I re-create the bridge to
>> see if it sovles the problem?
>
> Take a look at the output of "brctl show", and also see the contents
> of "/etc/xen/scripts/network-bridge".
> IMHO you should first make sure that it's not iptables problem. See
> the content of
> /proc/sys/net/ipv4/netfilter/ip_conntrack_count
> /proc/sys/net/ipv4/netfilter/ip_conntrack_max
>
> if count is close to max, you could either:
> - increase ip_conntrack_max, or
> - disable iptables for bridged traffic (echo 0 >
> /proc/sys/net/bridge/bridge-nf-call-iptables)
>
> --
> Fajar
>
> _______________________________________________


Thank you Fajar,


I tried to look at the ip_contrack files that you suggested, but there
are no such files on the server:


root@zaxen01:[~]$ locate ip_conntrack_count
root@zaxen01:[~]$ locate ip_conntrack_max
root@zaxen01:[~]$

root@zaxen01:[~]$ uname -a
Linux zaxen01.softdux.com 2.6.18-128.1.16.el5xen #1 SMP Tue Jun 30
06:39:23 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
root@zaxen01:[~]$ cat /etc/redhat-release
CentOS release 5.3 (Final)


Also, there's no firewall running ont he dom0 that blocks anything for
the domU's - I don't quite know how todo this even.

Any other suggestions?



--
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

Re: howto re-create XEN network bridge?

by Fajar A. Nugraha-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, Nov 6, 2009 at 5:25 PM, Rudi Ahlers <Rudi@...> wrote:

>> IMHO you should first make sure that it's not iptables problem. See
>> the content of
>> /proc/sys/net/ipv4/netfilter/ip_conntrack_count
>> /proc/sys/net/ipv4/netfilter/ip_conntrack_max


> I tried to look at the ip_contrack files that you suggested, but there
> are no such files on the server:
>
>
> root@zaxen01:[~]$ locate ip_conntrack_count
> root@zaxen01:[~]$ locate ip_conntrack_max

err ... I DIDN'T tell you to use "locate".
I DID give you the exact file path.

> Also, there's no firewall running ont he dom0 that blocks anything for
> the domU's - I don't quite know how todo this even.

The default xen bridge setup adds some iptables rules. Even when you
don't explicitly set them up.

--
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

Re: howto re-create XEN network bridge?

by SoftDux :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, Nov 6, 2009 at 12:38 PM, Fajar A. Nugraha <fajar@...> wrote:

> On Fri, Nov 6, 2009 at 5:25 PM, Rudi Ahlers <Rudi@...> wrote:
>
>>> IMHO you should first make sure that it's not iptables problem. See
>>> the content of
>>> /proc/sys/net/ipv4/netfilter/ip_conntrack_count
>>> /proc/sys/net/ipv4/netfilter/ip_conntrack_max
>
>
>> I tried to look at the ip_contrack files that you suggested, but there
>> are no such files on the server:
>>
>>
>> root@zaxen01:[~]$ locate ip_conntrack_count
>> root@zaxen01:[~]$ locate ip_conntrack_max
>
> err ... I DIDN'T tell you to use "locate".
> I DID give you the exact file path.

Well, no need to be rude, but to me more specific, those files don't exist:

root@zaxen01:[~]$ more /proc/sys/net/ipv4/netfilter/ip_conntrack_count
/proc/sys/net/ipv4/netfilter/ip_conntrack_count: No such file or directory
root@zaxen01:[~]$ locate ip_conntrack_count


Nor is there a /proc/sys/net/ipv4/netfilter/ folder.....
ll /proc/sys/net/ipv4/netfilter/
ls: /proc/sys/net/ipv4/netfilter/: No such file or directory



>
>> Also, there's no firewall running ont he dom0 that blocks anything for
>> the domU's - I don't quite know how todo this even.
>
> The default xen bridge setup adds some iptables rules. Even when you
> don't explicitly set them up.

AND, as I said in my first post, the firewall is disabled:

root@zaxen01:[~]$ iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination



>
> --
> Fajar
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@...
> http://lists.xensource.com/xen-users
>



--
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

Re: howto re-create XEN network bridge?

by SoftDux :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, Nov 6, 2009 at 12:38 PM, Fajar A. Nugraha <fajar@...> wrote:

> On Fri, Nov 6, 2009 at 5:25 PM, Rudi Ahlers <Rudi@...> wrote:
>
>>> IMHO you should first make sure that it's not iptables problem. See
>>> the content of
>>> /proc/sys/net/ipv4/netfilter/ip_conntrack_count
>>> /proc/sys/net/ipv4/netfilter/ip_conntrack_max
>
>
>> I tried to look at the ip_contrack files that you suggested, but there
>> are no such files on the server:
>>
>>
>> root@zaxen01:[~]$ locate ip_conntrack_count
>> root@zaxen01:[~]$ locate ip_conntrack_max
>
> err ... I DIDN'T tell you to use "locate".
> I DID give you the exact file path.
>
>> Also, there's no firewall running ont he dom0 that blocks anything for
>> the domU's - I don't quite know how todo this even.
>
> The default xen bridge setup adds some iptables rules. Even when you
> don't explicitly set them up.
>
> --
> Fajar
>
> _______________________________________________



Here's the output of brctl show:


root@zaxen01:[~]$ brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.feffffffffff       no              vifpict0
                                                        vifandr0
                                                        vifzafe0
                                                        vifserv0
                                                        vifraft0
                                                        vifplut0
                                                        viferis0
                                                        vifbyra0
                                                        vifanim0
                                                        vifacti0
                                                        peth0
                                                        vif0.0



All the interfaces that should be there, is there.


--
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

Re: howto re-create XEN network bridge?

by Fajar A. Nugraha-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, Nov 6, 2009 at 5:50 PM, Rudi Ahlers <Rudi@...> wrote:

> Here's the output of brctl show:
>
>
> root@zaxen01:[~]$ brctl show
> bridge name     bridge id               STP enabled     interfaces
> xenbr0          8000.feffffffffff       no              vifpict0
>                                                        vifandr0
>                                                        vifzafe0
>                                                        vifserv0
>                                                        vifraft0
>                                                        vifplut0
>                                                        viferis0
>                                                        vifbyra0
>                                                        vifanim0
>                                                        vifacti0
>                                                        peth0
>                                                        vif0.0
>
You can try by sniffing packets on uplink physical interface (should
be peth0 in your case). Test with simple packets (like ping) and snoop
it with tcpdump (something like "tcpdump -n -i peth0 icmp"). That
would at least tell you whether packets can get to physical interface
or not.

If the traffic is there but somehow you still have problems, the
source of problem might be elsewhere and not in your bridge setup.
Might be switch blocking ports with many MACs, or router caching old
MAC addresses.

--
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

Re: howto re-create XEN network bridge?

by SoftDux :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, Nov 6, 2009 at 12:59 PM, Fajar A. Nugraha <fajar@...> wrote:

> On Fri, Nov 6, 2009 at 5:50 PM, Rudi Ahlers <Rudi@...> wrote:
>
>> Here's the output of brctl show:
>>
>>
>> root@zaxen01:[~]$ brctl show
>> bridge name     bridge id               STP enabled     interfaces
>> xenbr0          8000.feffffffffff       no              vifpict0
>>                                                        vifandr0
>>                                                        vifzafe0
>>                                                        vifserv0
>>                                                        vifraft0
>>                                                        vifplut0
>>                                                        viferis0
>>                                                        vifbyra0
>>                                                        vifanim0
>>                                                        vifacti0
>>                                                        peth0
>>                                                        vif0.0
>>
>
> You can try by sniffing packets on uplink physical interface (should
> be peth0 in your case). Test with simple packets (like ping) and snoop
> it with tcpdump (something like "tcpdump -n -i peth0 icmp"). That
> would at least tell you whether packets can get to physical interface
> or not.
>
> If the traffic is there but somehow you still have problems, the
> source of problem might be elsewhere and not in your bridge setup.
> Might be switch blocking ports with many MACs, or router caching old
> MAC addresses.
>
> --
> Fajar
>
> _______________________________________________

Hi Fajar,

The packets does indeed get to the physical interface, but I get no
return ICMP packets, but it's almost asif that originating packets are
incomplete:

I got this from "tcpdump -n -i peth0 icmp" on the dom0 hostnode:

18:06:56.043744 IP 196.34.136.58 > 66.197.167.226: ICMP echo request,
id 34310, seq 1, length 64
18:06:57.053851 IP 196.34.136.58 > 66.197.167.226: ICMP echo request,
id 34310, seq 2, length 64


Yet, when I ping another VPS on the same dom0, I see an icmp reply:

[root@vps ~]# ping -c2 196.34.136.51
PING 196.34.136.51 (196.34.136.51) 56(84) bytes of data.
64 bytes from 196.34.136.51: icmp_seq=1 ttl=64 time=0.903 ms
64 bytes from 196.34.136.51: icmp_seq=2 ttl=64 time=0.115 ms




These IP's is completely open on the network firewall as well.



--
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

Re: howto re-create XEN network bridge?

by SoftDux :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, Nov 6, 2009 at 6:19 PM, Rudi Ahlers <Rudi@...> wrote:

> On Fri, Nov 6, 2009 at 12:59 PM, Fajar A. Nugraha <fajar@...> wrote:
>> On Fri, Nov 6, 2009 at 5:50 PM, Rudi Ahlers <Rudi@...> wrote:
>>
>>> Here's the output of brctl show:
>>>
>>>
>>> root@zaxen01:[~]$ brctl show
>>> bridge name     bridge id               STP enabled     interfaces
>>> xenbr0          8000.feffffffffff       no              vifpict0
>>>                                                        vifandr0
>>>                                                        vifzafe0
>>>                                                        vifserv0
>>>                                                        vifraft0
>>>                                                        vifplut0
>>>                                                        viferis0
>>>                                                        vifbyra0
>>>                                                        vifanim0
>>>                                                        vifacti0
>>>                                                        peth0
>>>                                                        vif0.0
>>>
>>
>> You can try by sniffing packets on uplink physical interface (should
>> be peth0 in your case). Test with simple packets (like ping) and snoop
>> it with tcpdump (something like "tcpdump -n -i peth0 icmp"). That
>> would at least tell you whether packets can get to physical interface
>> or not.
>>
>> If the traffic is there but somehow you still have problems, the
>> source of problem might be elsewhere and not in your bridge setup.
>> Might be switch blocking ports with many MACs, or router caching old
>> MAC addresses.
>>
>> --
>> Fajar
>>
>> _______________________________________________
>
> Hi Fajar,
>
> The packets does indeed get to the physical interface, but I get no
> return ICMP packets, but it's almost asif that originating packets are
> incomplete:
>
> I got this from "tcpdump -n -i peth0 icmp" on the dom0 hostnode:
>
> 18:06:56.043744 IP 196.34.136.58 > 66.197.167.226: ICMP echo request,
> id 34310, seq 1, length 64
> 18:06:57.053851 IP 196.34.136.58 > 66.197.167.226: ICMP echo request,
> id 34310, seq 2, length 64
>
>
> Yet, when I ping another VPS on the same dom0, I see an icmp reply:
>
> [root@vps ~]# ping -c2 196.34.136.51
> PING 196.34.136.51 (196.34.136.51) 56(84) bytes of data.
> 64 bytes from 196.34.136.51: icmp_seq=1 ttl=64 time=0.903 ms
> 64 bytes from 196.34.136.51: icmp_seq=2 ttl=64 time=0.115 ms
>
>
>
>
> These IP's is completely open on the network firewall as well.
>
>
>
> --


I see now that none of the VPS's can establish any outbound
connections, yet the server itself can. I have just upgraded the OS,
as follows:

root@zaxen01:[~]$ uname -a
Linux zaxen01.softdux.com 2.6.18-164.6.1.el5xen #1 SMP Tue Nov 3
16:48:13 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
root@zaxen01:[~]$ xm dmesg
 __  __            _____  _   ____     _  __   _  _    __    _       _ ____
 \ \/ /___ _ __   |___ / / | |___ \   / |/ /_ | || |  / /_  / |  ___| | ___|
  \  // _ \ \047_ \    |_ \ | |   __) |__| | \047_ \| || |_| \047_ \ |
| / _ \ |___ \
  /  \  __/ | | |  ___) || |_ / __/|__| | (_) |__   _| (_) || ||  __/ |___) |
 /_/\_\___|_| |_| |____(_)_(_)_____|  |_|\___/   |_|(_)___(_)_(_)___|_|____/

 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.1.2-164.6.1.el5 (mockbuild@...) (gcc version
4.1.2 20080704 (Red Hat 4.1.2-46)) Tue Nov  3 16:04:14 EST 2009
 Latest ChangeSet: unavailable








--
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

Re: howto re-create XEN network bridge?

by SoftDux :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Fri, Nov 6, 2009 at 8:28 PM, Rudi Ahlers <Rudi@...> wrote:

> On Fri, Nov 6, 2009 at 6:19 PM, Rudi Ahlers <Rudi@...> wrote:
>> On Fri, Nov 6, 2009 at 12:59 PM, Fajar A. Nugraha <fajar@...> wrote:
>>> On Fri, Nov 6, 2009 at 5:50 PM, Rudi Ahlers <Rudi@...> wrote:
>>>
>>>> Here's the output of brctl show:
>>>>
>>>>
>>>> root@zaxen01:[~]$ brctl show
>>>> bridge name     bridge id               STP enabled     interfaces
>>>> xenbr0          8000.feffffffffff       no              vifpict0
>>>>                                                        vifandr0
>>>>                                                        vifzafe0
>>>>                                                        vifserv0
>>>>                                                        vifraft0
>>>>                                                        vifplut0
>>>>                                                        viferis0
>>>>                                                        vifbyra0
>>>>                                                        vifanim0
>>>>                                                        vifacti0
>>>>                                                        peth0
>>>>                                                        vif0.0
>>>>
>>>
>>> You can try by sniffing packets on uplink physical interface (should
>>> be peth0 in your case). Test with simple packets (like ping) and snoop
>>> it with tcpdump (something like "tcpdump -n -i peth0 icmp"). That
>>> would at least tell you whether packets can get to physical interface
>>> or not.
>>>
>>> If the traffic is there but somehow you still have problems, the
>>> source of problem might be elsewhere and not in your bridge setup.
>>> Might be switch blocking ports with many MACs, or router caching old
>>> MAC addresses.
>>>
>>> --
>>> Fajar
>>>
>>> _______________________________________________
>>

Are there any limits on the amount of network interfaces one can setup on XEN?

I'm really at whit's end with this problem and don't know how to fix it.


--
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users

Re: howto re-create XEN network bridge?

by Fajar A. Nugraha-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Sat, Nov 7, 2009 at 1:28 AM, Rudi Ahlers <Rudi@...> wrote:

> I see now that none of the VPS's can establish any outbound
> connections, yet the server itself can.


Are the domUs still accessible from outside world? You mentioned
earlier that you can ssh to domU, but domU can't access anywhere. If
that's the case I suggest you try using bridge + NAT. You should have
a bridge called virbr0 by default on RHEL/Centos (created by libvirt),
with NAT already setup (check with "iptables -nL -t nat"). It should
show something like this

# iptables -nL -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  192.168.122.0/24    !192.168.122.0/24

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination



After that, create a domU that uses virbr0 bridge (use DHCP in domU).
Then check whether you can access outside world from domU. If you can,
then what I said earlier is probably true : the source of problem
might be elsewhere and not in your bridge setup.
It's also possible that your ISP/uplink provider is blocking packets
from your domU IP address.

--
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@...
http://lists.xensource.com/xen-users