This blog post is written by Amit Sharma, Shiva Katta and Krishnaprasad K from Dell Hypervisor Engineering team.
What is SR-IOV (Single Root I/O Virtualization)?
SR-IOV is the specification created and maintained by PCI SIG. It allows a single PCIe device to appear to be as multiple separate PCIe devices. SR-IOV introduces the idea of physical functions and virtual functions. Physical functions (PFs) are full-featured PCIe functions; virtual functions (VFs) are “lightweight” functions that lack configuration resources. SR-IOV requires support from BIOS, adapter, processors and Operating system. PFs have full configuration resources as it is possible to configure or control the PCIe device via the PF, and the PF has full ability to move data in and out of the device. VFs lack configuration resources; they only have the ability to move data in and out.
Support matrix of SR-IOV from Dell w.r.t VMware ESXi
Dell PowerEdge Server support
Network Controllers support
NOTE 1: The above support list is not finite. The list will be extended as and when applicable.
NOTE 2: SR-IOV is supported from Intel Xeon 56xx series processor onwards.
The support for SR-IOV for VMware ESXi starts from 5.1 onwards. Enabling SR-IOV for VMware ESXi 5.1 is a two-step process. The first step involves enabling the necessary BIOS tokens and checking the compatibility from the hardware perspective. The second step is enabling the NIC driver for IO Virtualization.
The first step talks about the settings to be done from the hardware level to enable SR-IOV.
Step 2 talks about enabling SR-IOV from VMware ESXi 5.1. The below steps detail about enabling this feature from Intel driver.
~# esxcfg-module –s “max_vfs=8” ixgbe
Here max_vfs is the parameter passed to the driver ‘ixgbe’. This parameter is used to enable the virtual functions from the driver. If the network controller used is a dual port card, the virtual functions for both ports can be enabled by passing the parameter max_vfs=8, 8
NOTE: The maximum virtual functions supported in VMware ESXi 5.1 are 32. So make sure that the total number of virtual functions enabled combining all ports is less than or equal to 32. Refer VMware configuration maximums guide .
2. Reboot host to re-load the driver with VFs enabled.
3. Connect host from vCenter server/ client. Navigate to ‘Configuration --> Advanced Settings’ tab to see the virtual functions enabled as below:-
4. Edit the VM settings and assign the Virtual functions as a PCI device similar to PCI passthrough feature.5. Once the VFs are assigned to the VM, the guest OS driver enables it to be available for the virtual machines.
NOTE: You may have to install the required drivers from Guest OS to make the virtual functions visible. RHEL 6.3 already contains a native driver ‘ixgbevf’.
This section talks about disabling SR-IOV from Intel 'ixgbe' driver.
1. First step is to understand the total number of VFs enabled /port using the below command
~# esxcfg-module –g ixgbe
This command shows the max_vfs parameter which shows the number of ports and number of virtual functions already enabled.
2. Disable SR-IOV for the specific port by executing the below command
~# esxcfg-module –s max_vfs=0, 0 ixgbe # Assuming that VFs were enabled for two ports.
3. Reboot the system to take this in to effect.