Brief about SR-IOV

SR-IOV is a specification that can present single PCIe device (this is called Physical Function) as multiple independent PCIe devices (each one is called Virtual Function) to OS or hypervisor. Dedicating a virtual function to a VM can be accomplished by a mechanism called PCI pass-through with which I/O operations can be done by accessing the virtual function directly from the VM.

By providing a virtual function directly to a VM, the hypervisor’s virtual switch (vSwitch) is no longer required to process network traffic. This hypervisor bypass increases network throughput, lowers latency, and reduces overall CPU utilization.

Limitations in ESXi 5.1 and 5.5 releases

In ESXi 5.1/5.5 host, we have limited number of interrupt vectors available for pass-through devices. So the ESXi host supports limited number of virtual functions. The total numbers of interrupt vectors allocated to pass-through devices are less than 128.

SR-IOV enhancements in ESXi 6.0 release

In ESXi 6.0 host, the total number of interrupt vectors can scale up to 4096 if the system is having 32 CPUs. When the host boots, devices on the host such as storage controllers, physical network adapters, and USB controllers consume a subset of the 4096 vectors. If these devices require more than 1024 vectors, the maximum number of potentially supported virtual function is reduced.

Simple formula to estimate the maximum number of virtual functions for use if all 3072 interrupt vectors is available for pass-through:

3X + 2Y < 3072

3 interrupt vectors per virtual function for Intel adapters (X) and 2 interrupt vectors per VF for Emulex adapters (Y).

The number 3072 might be smaller if other types of devices on the host use more than 1024 interrupt vectors from the total of 4096 vectors on the host.

Description ESXi 5.1/5.5 ESXi 6.0
Interrupt vectors 256 4096
Host devices consumption 128 1024
Available interrupt vector for virtual function 128 3072
Interrupts required per virtual function 3 (Intel), 2 (Emulex) 3 (Intel), 2 (Emulex)
Usable virtual function 63 (Intel), 31 (Emulex) 63 (Intel), 31 (Emulex)

Supported adapters

  • Products based on the Intel 82599ES 10 Gigabit Ethernet Controller Family (Niantic). Example, Intel X520 CNA adapter
  • Products based on the Intel Ethernet Controller X540 Family (Twinville).
  • Emulex OneConnect (Blade Engine 3). Example, Emulex OCe14102-UX-D (12th generation servers), OCe14102-U1-D (13th generation servers)

References

SR-IOV in VMware ESXi

SR-IOV feature enhancements in VMware vSphere ESXi 5.5

Comparison between NPAR and SR-IOV in VMware ESXi