Description

AHV Networking is covered in the documentation available on the Nutanix Support Portal.

• Initial setup is covered in the Field Installation Guide:
  Foundation 4.6.x Field Installation Guide
• Basic configuration for most use cases is covered in the following:
  AHV Admin Guide (AOS 5.18 version)
  AHV Best Practices Guide
• Advanced configuration for more in-depth needs is covered in the following:
  AHV Networking Best Practices Guide.

Solution

Note: Do not use any of those commands on User VM or CVM interfaces (vnet interfaces).

Topics Covered:

• Open vSwitch Configuration
• manage_ovs Utility
• Steps to add or remove the VLAN tag of CVM and AHV host
• LACP Configuration
• How to select active NIC in active-backup bond mode
• Separate 1G interfaces from a bond with the 10G interfaces
• Bridges with a single uplink
• Known issues

Open vSwitch Configuration

Nutanix AHV hosts come with Open vSwitch pre-configured. The 'ovs-vsctl show' command will print an overview of the Open vSwitch configuration:

[root@ahv ~]#  ovs-vsctl show
e0d89b99-335d-417f-9fd1-4a0cb8db711e
    Bridge "br0"
        Port "br0"
            Interface "br0"
                type: internal
        Port "vnet0"
            Interface "vnet0"
        Port "bond0"
            Interface "eth2"
            Interface "eth1"
            Interface "eth0"
    ovs_version: "1.10.0"

There is a Bridge named "br0" with several Ports.

• Port "br0" is an internal interface on bridge br0. This is the AHV host's external management interface.
• Port "vnet0" is a virtual interface port that the CVM (Controller VM) uses.
• Port "bond0" or port "br0-up" is a bonded port containing multiple physical interfaces from the AHV host.

Note: In AOS 5.5, bridge chaining was introduced and as a result, you can see more bridges in the "ovs-vsctl show" output.
[Back to Top]

manage_ovs Utility

Nutanix provides a utility called 'manage_ovs', which is installed on each CVM and should be used to manage the Open vSwitch configuration on the AHV host. See the "manage_ovs --helpshort" output for details on usage.

Sample output from AOS 5.10.2:

nutanix@cvm$ manage_ovs --helpshort

USAGE: manage_ovs [flags] <action>

Where <action> is one of the following:

  show_bridges: Shows a list of the uplink bridges.
  show_interfaces: Shows a list of host physical interfaces.
  show_uplinks: Shows the current uplink configuration for the OVS bridge.
  update_uplinks: Updates the uplink configuration for the OVS bridge.
  enable_bridge_chain: Enables bridge chaining on the host.
  disable_bridge_chain: Disables bridge chaining on the host.
  create_single_bridge: Adds one OVS bridge to the host.
  delete_single_bridge: Removes one OVS bridge from the host.

The update_uplinks action requires the --interfaces flag, which indicates the
desired set of uplinks for the OVS bridge. The script will remove any existing
uplinks from the bridge, and replace them with the specified set of uplinks on
a single bonded port.

flags:

/usr/local/nutanix/cluster/bin/manage_ovs:
  --bond_mode: Bond mode to use: active-backup, balance-slb, or balance-tcp
    (default: 'active-backup')
  --bond_name: Bond name to use
  --bridge_name: Openvswitch on which to operate
    (default: '')
  --[no]dry_run: Just print what would be done instead of doing it
    (default: 'false')
  --[no]enable_vlan_splinters: Enable VLAN splintering on uplink interfaces
    (default: 'true')
  --[no]force: Reconfigure the bridge even if the the set of uplinks has not changed
    (default: 'false')
  -?,--[no]help: show this help
  --[no]helpshort: show usage only for this module
  --[no]helpxml: like --help, but generates XML output
  --host: Host on which to operate
    (default: '192.168.5.1')
  --interfaces: Comma-delimited list of interfaces to configure as bridge uplinks, or a keyword based on the NIC speed:
    all, 100g, 40g, 10g, 1g, etc.
  --[no]json: Output in json format.
    (default: 'false')
  --lacp_fallback: If LACP negotiation fails, set the bond_mode to active-backup: true, false
    (default: 'true')
  --lacp_mode: LACP mode for the uplink bond: off, slow, or fast
    (default: 'off')
  --mtu: Maximum transmission unit
    (an integer)
  --num_arps: Number of gratuitous ARPs to send on the bridge interface after updating uplinks
    (default: '3')
    (an integer)
  --[no]prevent_network_loop: Enables network loop prevention when bridge chain is enabled.
    (default: 'false')
  --[no]require_link: Require that at least one uplink has link status
    (default: 'true')

AOS 5.19 or newer

It is possible to manage the host network configuration using Prism Element or Prism Central UI. Refer to the About virtual switch chapter for more information about Virtual switches. Refer to Creating or updating a virtual switch chapter for information on how to manage Virtual switches.
[Back to Top]

Additional Information

Steps to add or remove the VLAN tag of CVM and AHV host

Perform all AHV host and CVM network changes by connecting to the host using IPMI and opening a remote console. Making VLAN changes at the physical switch level without co-ordinating the host changes per these steps can lead to CVM access and connectivity issues and potentially lead to UVM downtime and the cluster not being accessible from Prism Central or Prism Element until the VLAN configuration is completed or any mismatch is resolved. It is recommended to perform such changes one host/CVM/uplink bond/switchport at a time whilst under maintenance mode, completing a single host/CVM and bringing it back online before moving to the next. See KB4639 - How to place CVM and host in maintenance mode
Also see: AHV Admin Guide / Layer 2 Network Management with Open vSwitch / VLAN Configuration

To change the VLAN on the AHV host, run the following command:

[root@ahv ~]# ovs-vsctl set port br0 tag=<vlan tag>

Note: Make sure that you are updating br0 and not br0-up or bond0 port. If VLAN is assigned to br0-up or bond0, then CVM and host might lose network connectivity. In case VLAN was assigned to br-up or bond0, run the following command to remove it:

[root@ahv ~]# ovs-vsctl remove port br0-up tag <vlan tag>

Use "ovs-vsctl show" command to verify changes. The correct configuration should look like this:

[root@ahv ~]# ovs-vsctl show
...
    Bridge "br0"
...
        Port "br0-up"
            Interface "eth2"
            Interface "eth3"
        Port "br0"
            tag: 1234    <<<<< Correct vlan tag location for AHV host
            Interface "br0"
                type: internal
 ...

To change the VLAN on the CVM, run the following command with the VLAN tag (marked X) you would like to configure:

nutanix@cvm$ change_cvm_vlan <vlan tag>

Note: Do not use the ovs-vsctl command from the AHV host to set the VLAN tag for the CVM (on vnet0). Instead, use the change_cvm_vlan command from CVM.

To remove the VLAN tag from the AHV host, run the following command after opening the remote console:

[root@ahv ~]# ovs-vsctl set port br0 tag=0

To remove VLAN tagging from CVM, follow this procedure:

• Log in to the AHV host via remote console
• SSH from the host to the CVM (always SSH to the 192.168.5.254 address):

  [root@ahv ~]# ssh nutanix@192.168.5.254

• Execute the following command on the CVM:

  nutanix@cvm$ change_cvm_vlan --remove

[Back to Top]

LACP Configuration

Follow KB-3263  How to Enable, Disable, and Verify LACP in AHV.

How to find the MAC address of a host NIC

• Execute the following command on AHV host:

[root@ahv ~]# ethtool -P <interface>

Sample output:

[root@ahv ~]# ethtool -P eth3
Permanent address: 00:25:90:cb:39:27

• Execute the following command on AHV host:

[root@ahv ~]# ifconfig <interface>

Sample output:

[root@ahv ~]# ifconfig eth3
eth3   Link encap:Ethernet  HWaddr 00:25:90:CB:39:27  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:46857327754 errors:0 dropped:228250 overruns:0 frame:0
          TX packets:49134503170 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:49893674683483 (45.3 TiB)  TX bytes:54855610562476 (49.8 TiB)

[Back to Top]

How to select active NIC in active-backup bond mode

By default, during boot time AHV randomly selects one of the NICs in bond and makes it active. All other NICs are assigned backup roles and they will only be activated if active NIC fails.
If you want to set a preferred NIC to be selected as primary, you can use the following command (AOS 5.15.4 and later):

ovs-vsctl set port <port> other_config:bond-primary=<nic_interface>

For example:

ovs-vsctl set port br0-up other_config:bond-primary=eth2

To check which NIC is active and/or preferred, connect to the AHV host and run the following command:

[root@ahv ~]# ovs-appctl bond/show

In the command output, the active interface will be marked as an active slave. In the example below eth3 is an active interface in br0-up bond:

[root@ahv ~]# ovs-appctl bond/show
---- br0-up ----
bond_mode: active-backup
active-backup primary: eth2
...
slave eth2: enabled
        may_enable: true

slave eth3: enabled
        active slave
        may_enable: true

To change active NIC please connect to the AHV host and run the following command:

[root@ahv ~]# ovs-appctl bond/set-active-slave <bond name> <interface name>

Example:

[root@ahv ~]# ovs-appctl bond/set-active-slave br0-up eth2
done

Now, when you check the bond/show output, you can see eth2 interface appears as active slave:

Example:

[root@ahv ~]# ovs-appctl bond/show
---- br0-up ----
bond_mode: active-backup
...
slave eth2: enabled
        active slave
        may_enable: true

slave eth3: enabled
        may_enable: true

[Back to Top]

Separate 1G interfaces from a bond with the 10G interfaces

Check KB 8835 for a detailed description of the workflow.
[Back to Top]

Bridges with a single uplink

Check KB 8015 for a detailed description of the workflow.
[Back to Top]

Known issues

Symptom: manage_ovs fails with "ovs-vsctl: not found" error:

CRITICAL manage_ovs:166 Failed to execute ovs command: list-br
...
stderr: sh: ovs-vsctl: not found
...

Resolution: Make sure that the underlying hypervisor is AHV. If another hypervisor is used, refer to its vendor's documentation on how to manage the network from the command line.

Symptom: manage_ovs fails with "Cluster ARP and DHCP IP not initialized." error when trying to create an additional bridge:

2019-10-24 01:55:57 ERROR manage_ovs:602 OVS error (192.168.5.1 create_single_bridge): Cluster ARP and DHCP ip not initialized.

Resolution: Make sure that the node, where the bridge is being added, is part of an existing cluster. manage_ovs does not support creating new bridges on nodes that are not joined to the cluster.
[Back to Top]

Related Articles

• Original article in Nutanix Portal: Nutanix KB Article: 2090
• Nutanix landing page