prerequisite: Oracle VM Server configuration: multiple networks, multipathed SAN storage

The Xen State Database

Troubleshooting tip: if the instructions below are followed, but "old" interfaces appear to persiste between reboots even though no configuration is found in /etc/sysconfig/network-scripts, the Xen State Directory may be interfering.

Take a look at:

1. The Xen State Databse directory: /var/lib/xend/state/
2. The network configuration file: /var/lib/xend/state/network.xml

Configure Networking

WARNING: Beware of Baseboard Management Controllers (BMCs)

Baseboard Management Controllers (BMC) are increasingly shipped on rackmount servers. While these can getbe youa outgreat ofmanagement trouble,tool, theythe versions that share the system's ethernet ports often don't react well with bonded interfaces. With the Broadcom NetXtreme II, you must disable the management firmware in order to enable bonding, less a broadcast storm occur on the connected switch. In the Broadcom case, a DOS utility "xdiag" is provided on the firmware update disk which can disable the management firmware (MF) code. If "xdiag -ver" shows "MF" active on a controller, you need to run "xdiag -c <controller#> -mfw 0" to disable it.

This will probably have to be accomplished via the system's console - or a kickstart installation file (not covered in this wiki page), as these actions are likely to drop network connectivity.

This configuration has the operating system configuring all aspects of the system's network connectivity. This differs from the original Xen model of using a "network-script <something>" to configure networking at virtual machine initialization time via /etc/xen/xend-config.sxp. While there's nothing inherently wrong with this method, it is more difficult to scale and troubleshoot as it splits network configuration between the operating system and the xend daemon. Hence why when many interfaces and bridges are required to support a bonding + trunking system, it was far more straightforward to consolidate network configuration with OS-supplied mechanisms.

We assume that VLANs are shared between all relevant switches via an inter-switch link of some sorts. Not sure about this? Ask your network people. These configurations generally only function with managed switches.

The VM hosts will be connected to one switch with 2 or more ethernet (probably gigabit ethernet) links, these will be:

1. aggregated (aka bonded, teamed) using the IEEE 802.2ad protocol
   
2. trunked, using the IEEE 802.1q vlan trunking protocol

see: Network Switch Configuration- bonding and trunking

With the switch configured, the VM host systems can now be configured. These steps are valid for all RHEL derivatives (Oracle EL, Oracle VM Server, CentOS, etc). All relevant files are in the /etc/sysconfig/network-scripts/ directory.

Debian Linux and derivative (Ubuntu) users will want to look into the format of their distribution's /etc/network/interfaces file.

The end result is that the system will end up with a large group of interfaces:

1. the physical interfaces (ethX: eth0, eth1, ...)
2. bonding interfaces to aggregate the physical interfaces (bondX: bond0, bond1, ...) - also a VLAN trunk!
   
3. interfaces for each individual VLAN on the bonded trunk (bondX.vlan#: bond0.1, bond0.50)
4. per-vlan bridges, these are the bridges that virtual machines will attach their native virtual interfaces (vifs) to, I have used the vlan# convention for the bridge names (vlan1, vlan50, vlan51...) where interface vlan50 particilates in vlan ID 50 placed on the trunk.
   

Load Bonding Module

By default, the bonding module only supports the creation of one bonding interface. The "options bonding max_bonds=#" is used to increase this value to whatever # is required, which will generally be between 1 and 4 depending on your site and needs.

Add the bonding module to /etc/modprobe.conf:

alias bond0 bonding
alias bond1 bonding
options bonding max_bonds=2

Configure the bonding interface

bond0 via /etc/sysconfig/network-scripts/ifcfg-bond0
# 802.3ad bonded link
# switch po1: Gi1/1, Gi1/2
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
BONDING_OPTS="mode=4 miimon=100"

Configure the physical interfaces (eth0 and eth1) that are part of bond0:

• /etc/sysconfig/network-scripts/ifcfg-eth0

# First interface in bond0
DEVICE=eth0
BOOTPROTO=static
HWADDR=<INTERFACE MAC ADDRESS>
TYPE=Ethernet
ONBOOT=yes
MASTER=bond0
SLAVE=yes

• /etc/sysconfig/network-scripts/ifcfg-eth1

# Second interface in bond0
DEVICE=eth1
BOOTPROTO=static
HWADDR=<INTERFACE MAC ADDRESS>
TYPE=Ethernet
ONBOOT=yes
MASTER=bond0
SLAVE=yes

Configure per-vlan bridges (vlanX: vlan50, vlan51, vlan100...):

• /etc/sysconfig/network-scripts/ifcfg-vlan50

DEVICE=vlan50
BOOTPROTO=none
ONBOOT=yes
TYPE=Bridge

• /etc/sysconfig/network-scripts/ifcfg-vlan51

DEVICE=vlan51
BOOTPROTO=none
ONBOOT=yes
TYPE=Bridge

• /etc/sysconfig/network-scripts/ifcfg-vlan100

DEVICE=vlan100
BOOTPROTO=none
ONBOOT=yes
TYPE=Bridge

Configure per-vlan interfaces on the bonded interface, and attach them to their respective vlan bridge interfaces:

• /etc/sysconfig/network-scripts/ifcfg-bond0.50

DEVICE=bond0.50
ONBOOT=yes
BOOTPROTO=none
VLAN=yes
TYPE=Ethernet
BRIDGE=vlan50

• /etc/sysconfig/network-scripts/ifcfg-bond0.51

DEVICE=bond0.51
ONBOOT=yes
BOOTPROTO=none
VLAN=yes
TYPE=Ethernet
BRIDGE=vlan51

• /etc/sysconfig/network-scripts/ifcfg-bond0.100

DEVICE=bond0.100
ONBOOT=yes
BOOTPROTO=none
VLAN=yes
TYPE=Ethernet
BRIDGE=vlan100

Configure VM server host connectivity

Here it's assumed that there is at least one management VLAN, say Vlan 100. Note that many sites use another dedicated physical interface for management. Whatever way it's done, it is strongly suggested to secure and segregate the host (Xen dom0) management access.

In this example, VLAN100 is used for management, hence it the only interface that's assigned an IP address.

Will add virtual interface vlan#:X, here vlan100:1 to the appropriate vlan bridge (vlan100):

• /etc/sysconfig/network-scripts/ifcfg-vlan100:1

DEVICE=vlan100:1
BOOTPROTO=static
ONBOOT=yes
IPADDR=10.1.1.101
NETMASK=255.255.255.0

Disable the Xend network-scripts

Note: when configuring virtual machines, they will now be attached to the "vlan###" bridge, and not the generic "xenbr###" interfaces (who will no longer be created once this process is complete).

As previously discussed, we are using operating system tools to build the various virtual networks. Disabling the "network-scripts" funcion of /etc/xen/xend-config.sxp is accomplished by creating a "dummy" or "do nothing" script that xend can harmlessly continue to run when initializing a virtual machine.

• build the dummy script: /etc/xen/scripts/network-dummy

#!/bin/bash
exit 0

• make it executable: chmod 755 /etc/xen/scripts/network-dummy
• edit /etc/xen/xend-config.sxp, and change:

(network-script network-dummy)

Configuration Complete

A series of "ifup" commands will bring the above configuration into production without a reboot. Nevertheless, a reboot is strongly recommended to ensure that the network configuration restores itself properly

The following interfaces should be seen when running an "ifconfig":

bond0 Link encap:Ethernet HWaddr 00:14:5E:C0:FF:EE
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:33617546 errors:0 dropped:0 overruns:0 frame:0
TX packets:12192931 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4023299390 (3.7 GiB) TX bytes:1022184796 (974.8 MiB)

bond0.50 Link encap:Ethernet HWaddr 00:14:5E:C0:FF:EE
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:22507118 errors:0 dropped:0 overruns:0 frame:0
TX packets:604 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2307351805 (2.1 GiB) TX bytes:59276 (57.8 KiB)

bond0.51 Link encap:Ethernet HWaddr 00:14:5E:C0:FF:EE
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:22507118 errors:0 dropped:0 overruns:0 frame:0
TX packets:604 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2307351805 (2.1 GiB) TX bytes:59276 (57.8 KiB)

bond0.100 Link encap:Ethernet HWaddr 00:14:5E:C0:FF:EE
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:22507118 errors:0 dropped:0 overruns:0 frame:0
TX packets:604 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2307351805 (2.1 GiB) TX bytes:59276 (57.8 KiB)

eth0 Link encap:Ethernet HWaddr 00:14:5E:C0:FF:EE
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:6783365 errors:0 dropped:0 overruns:0 frame:0
TX packets:2132508 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:731753941 (697.8 MiB) TX bytes:226889202 (216.3 MiB)
Interrupt:23 Memory:ce000000-ce011100

eth1 Link encap:Ethernet HWaddr 00:14:5E:C0:FF:EE
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:26834181 errors:0 dropped:0 overruns:0 frame:0
TX packets:10060423 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3291545449 (3.0 GiB) TX bytes:795295594 (758.4 MiB)
Interrupt:16 Memory:ca000000-ca011100

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:59021 errors:0 dropped:0 overruns:0 frame:0
TX packets:59021 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:14567768 (13.8 MiB) TX bytes:14567768 (13.8 MiB)

vlan50 Link encap:Ethernet HWaddr 00:14:5E:C0:FF:EE
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:20511279 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2122732011 (1.9 GiB) TX bytes:0 (0.0 b)

vlan51 Link encap:Ethernet HWaddr 00:14:5E:C0:FF:EE
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:20511279 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2122732011 (1.9 GiB) TX bytes:0 (0.0 b)

vlan100 Link encap:Ethernet HWaddr 00:14:5E:C0:FF:EE
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:20511279 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2122732011 (1.9 GiB) TX bytes:0 (0.0 b)

vlan100:1 Link encap:Ethernet HWaddr 00:14:5E:C0:FF:EE
inet addr:10.1.1.101 Bcast:10.200.40.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1



The following bridges should be seen when running a "brctl show":

[root@virtual4 network-scripts]# brctl show
bridge name bridge id STP enabled interfaces
vlan50 8000.00145e187fa4 no bond0.50
vlan51 8000.00145e187fa4 no bond0.51
vlan100 8000.00145e187fa4 no bond0.100


The status of the bonded interface can be seen by using the procfs and sysfs:

• /proc/net/bonding/bond0
• /sys/class/net/bonding_masters
• the /sys/class/net/bond0/ directory

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 17
Partner Key: 101
Partner Mac Address: C0:FF:EE:25:60:00

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:14:5e:C0:FF:EE
Aggregator ID: 1

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:14:5e:C0:FF:EF
Aggregator ID: 1

cat /sys/class/net/bonding_masters

bond0 bond1

cat /sys/class/net/bond0/operstate

up