fiber channel

The following post is written by Ujjwal Rajbhandari, from our Storage Product Marketing Group. 

There are number of discussions, blogs and articles comparing iSCSI, FCoE and FC. Many of them share a common belief that FCoE and FC are better suited as core datacenter SAN and iSCSI is ideal for Tier 2 storage or for SAN deployments in ROBO/SMB environment. That is because iSCSI is characterized as “low performing”, “lossy” and “unpredictable”. In this blog I will tackle the mis-information around iSCSI performance as compared to FC and FCoE. I will also compare effective efficiency of the various SAN protocols since efficiency is an aspect of performance

Both iSCSI and FCoE share the same 10Gb Ethernet at the transport layer. However, the perception is that the TCP/IP overhead makes iSCSI inefficient as compared to FCoE and FC (both having better payload to packet size ratio) thus leading to lower performance and efficiency. Figure 1 shows protocol efficiency calculation for iSCSI (both 1.5K MTU and 9K MTU), FC and FCoE (2.5K MTU). It can be seen that when jumbo frames are enabled, iSCSI has the best protocol efficiency.

Figure 1: Protocol efficiency comparisons

In regards to performance, iSCSI having low performance might have been true when 1Gbps was maximum throughput available per iSCSI port where as FC was delivering 2Gbps, 4Gbps and 8Gbps per port, but with the availability of 10GbE the commonly held belief that iSCSI performance is not being up-to-par as compared to FCoE or FC is no longer true.

The Office of CTO at Dell conducted a series of performance test to compare 10GbE iSCSI, FCoE and 4Gb FC. To ensure similar workload the application throughput was limited to 4Gb. The host adapters used for the different protocols were as follows:

1. 10GbE NIC with iSCSI offload for iSCSI traffic

2. 10GbE converged network adapter (CNA) for FCoE traffic

3. 4Gbps FC HBA for fibre channel traffic

The goal of the testing was to capture achieved throughput and CPU utilization for a given SAN protocol.

The protocol efficiency comparisons from Figure 1 might be theoretical in nature, Figure 2 shows results from an IO workload study compare throughput of 10GbE iSCSI, FCoE and 4Gb FC HBAs. To keep the results easy to visualize, the results show the throughput achieved when application generated 4Gb throughput. It can be clearly seen that iSCSI outperforms FCoE and FC regardless of read or write operation for various IO block size.

Figure 2: Throughput Performance Comparisons (MB/s)

Alongside capturing the throughput, let’s examine the host CPU utilization to better assess the performance and efficiency of specific SAN protocol. All the host adapters are comprised of hardware based offload capability to process the protocol specific traffic minimizing use of CPU resources. Figure 3 shows the effective CPU utilization for various workloads. It can be seen from the figure that all the host adapters have similar CPU utilization metrics, again reinforcing the fact that iSCSI is as efficient as FCoE and FC.

Finally, Figure 4 shows throughput efficiency, defined as MBps/%CPU, for the various storage protocols. The chart clearly shows 10GbE iSCSI having the best throughput efficiency across the workload types clearly outperforming FCoE and FC.

From the test results we can undoubtedly summarize that iSCSI as a SAN protocol is not “lower performing” or “inefficient” as compared to FC or FCoE. On the contrary, iSCSI outperforms both FC and FCoE. Customers who are planning to purchase storage for their datacenter can consider iSCSI SAN as a viable option knowing iSCSI performance is at par or even better than FCoE and FC. Also, customers considering unifying their datacenter network over Ethernet can start doing so now with iSCSI. While FCoE can also deliver storage traffic over Ethernet it is still under development and is not ready for prime-time.

Figure 3: CPU utilization (%) for iSCSI Offload, FCoE and FC

Figure 4: Overall protocol throughput efficiency (MBps/%CPU) for iSCSI Offload, FCoE and FC

Talk about incendiary... especially considering the instructor's audience: mostly Dell technical personnel. (MD3000i, EqualLogic ring a bell?) This statement was delivered, probably without much thought on how it would be received by the attendees, during a recent training session.  The course focused on products that play almost exclusively in a fiber channel SAN arena.  The teacher was "kinda like a nut living amongst squirrels."  More than once that day, a heated discussion flared on iSCSI vs fiber channel and eventually spanned to cover FCoE as well.

Naturally, I tweeted about the entertaining debate. The twitter response was equally interesting. It was as if I had mentioned Microsoft in a Linux forum. Ford vs. Chevy. You get the idea --it's a religion. I began to think about my last 10 years at another company. The kool-aid I was drinking was the color of orange wires.  The instructor's glasses, understandably, seem to be a little orange tinted as well. Its what he and his company grew up with. Over the next few days, I discussed this topic with several folks in and out of the DellTechCenter. I also read a number of related interesting blogs and forum posts. What did I learn?  Here's a quick summary of the various opinions I ran into this week:

Fiber Channel:
FC SANS are thought of for performance and reliability.  Management and implementation of FC SANS requires a specific knowledge set and can be viewed as expensive to get started.  Another common opinion was that customers with FC investments likely won't be quick to adopt or move to iSCSI. 

iSCSI:
iSCSI is generally viewed as easy and less expensive to implement from both a training and hardware perspective. It's likely the infrastructure skills required already exist within the companies evaluating the technologies.  Although, sometimes the intricacies of infrastructure configuration for best performance are somewhat glossed over.  The iSCSI SANS found their way into the smaller shops initially, but has quickly been moving up the food chain. The predominant opinion is that when 10 Gb Ethernet becomes widely accepted, so will iSCSI.

FCoE:
There were tentative opinions in the group on FCoE.  The common thread was that the cost for the needed additional hardware might hinder market acceptance.

Fortunately, I have multi-colored kool-aid these days. As a storage evangelist for DellTechCenter, I have the opportunity to evaluate and write about the various technologies. I previously did not have hands on experience with iSCSI.  Now that I've had a chance to connect a few servers to both EqualLogic and MD3000i arrays, I was truly surprised at how easy it was to get going. Factor in the ease of implementation, equivalent storage features (mpio, replication, snapshot, etc), scalability and cost of the solutions --the 'not enterprise' opinion seems a bit uninformed. I personally don't see it as an end to fiber though. In this industry these type of debates will always come up.  I'm sure there were token-ring proponents that had similar opinions on Ethernet's viability in the enterprise.