This blog post is written by Amit Sharma, Shiva Katta and Krishnaprasad K from Dell Hypervisor Engineering team.

What is SR-IOV (Single Root I/O Virtualization)?

SR-IOV is the specification created and maintained by PCI SIG. It allows a single PCIe device to appear to be as multiple separate PCIe devices.  SR-IOV introduces the idea of physical functions and virtual functions. Physical functions (PFs) are full-featured PCIe functions; virtual functions (VFs) are “lightweight” functions that lack configuration resources.  SR-IOV requires support from BIOS, adapter, processors and Operating system. PFs have full configuration resources as it is possible to configure or control the PCIe device via the PF, and the PF has full ability to move data in and out of the device. VFs lack configuration resources; they only have the ability to move data in and out.

Support matrix of SR-IOV from Dell w.r.t VMware ESXi

Dell PowerEdge Server support

    • Dell PowerEdge 12th Generation servers support SR-IOV.
    • Below is a list of Dell 11th Generation platforms that support the SR-IOV feature: 
      • R910, R210 II, T110 II
    • The following platforms are also supported and must be an 11G Generation II server: 
      • T410, R410, R510, R610, T610, R710, T710

Network Controllers support

    • Intel X520 10GB Ethernet adapters
    • Intel X540 10GB Ethernet adapters

NOTE 1: The above support list is not finite. The list will be extended as and when applicable.

NOTE 2: SR-IOV is supported from Intel Xeon 56xx series processor onwards.

Enabling SR-IOV

The support for SR-IOV for VMware ESXi starts from 5.1 onwards. Enabling SR-IOV for VMware ESXi 5.1 is a two-step process. The first step involves enabling the necessary BIOS tokens and checking the compatibility from the hardware perspective. The second step is enabling the NIC driver for IO Virtualization.

Step1

            The first step talks about the settings to be done from the hardware level to enable SR-IOV.

      • Make sure that you have the latest BIOS installed from Dell support.
      • Enable SR-IOV BIOS token (SR-IOV Global Enable) by traversing to Integrated Devices section in BIOS as below.                           

Step2

Step 2 talks about enabling SR-IOV from VMware ESXi 5.1. The below steps detail about enabling this feature from Intel driver.

    1. Enable virtual functions for Intel ixgbe driver as below.

 ~# esxcfg-module –s “max_vfs=8” ixgbe

Here max_vfs is the parameter passed to the driver ‘ixgbe’. This parameter is used to enable the virtual functions from the driver. If the network controller used is a dual port card, the virtual functions for both ports can be enabled by passing the parameter max_vfs=8, 8

NOTE: The maximum virtual functions supported in VMware ESXi 5.1 are 32. So make sure that the total number of virtual functions enabled combining all ports is less than or equal to 32. Refer VMware configuration maximums guide

2. Reboot host to re-load the driver with VFs enabled.

3. Connect host from vCenter server/ client. Navigate to ‘Configuration --> Advanced Settings’ tab to see the virtual functions enabled as below:-

4. Edit the VM settings and assign the Virtual functions as a PCI device similar to PCI passthrough feature.
5. Once the VFs are assigned to the VM, the guest OS driver enables it to be available for the virtual machines.

NOTE: You may have to install the required drivers from Guest OS to make the virtual functions visible. RHEL 6.3 already contains a native driver ‘ixgbevf’.

Disabling SR-IOV

This section talks about disabling SR-IOV from Intel 'ixgbe' driver.

1. First step is to understand the total number of VFs enabled /port using the below command

~# esxcfg-module –g ixgbe

This command shows the max_vfs parameter which shows the number of ports and number of virtual functions already enabled.

2. Disable SR-IOV for the specific port by executing the below command

~# esxcfg-module –s max_vfs=0, 0 ixgbe  # Assuming that VFs were enabled for two ports.

3. Reboot the system to take this in to effect.