fcoe

For those of you who would enjoy a little balance to all the FCoE buzz in the industry, let me introduce you to Robert Winter and Gaurav Chawla - two of Dell's strategic engineers in this space. 

Read below for their take on the realities of FCoE.  Thanks again, guys.

       
Robert Winter    Gaurav Chawla

FCoE has been positioned by some as a future primary data center fabric and by others as simply a transitional technology. We believe that the ultimate position of FCoE will probably be somewhere between these two extremes. At this point, FCoE is not a full-formed solution and will probably require several years to achieve the same maturity level as iSCSI or FC.

The fact is that FCoE represents an attempt to completely detach an existing, purpose-built storage protocol, namely Fibre Channel (FC), and move it to a completely different physical fabric, Ethernet. To say that this migration is seamless ignores some very real practical challenges.

Let’s take a look at the technology facts.

The FCoE Storage Protocol and the OSI Model Are Not The Best of Friends

The OSI Layered Model (Open Systems Interconnect) is a well-known architectural abstraction that helps to describe the operation of protocols. In looking at FC, FCoE, and iSCSI, one consideration that mustn’t be lost is that Fibre Channel protocol layers cannot be mapped to OSI layers in a straight forward manner because the FC protocol originates from a much different historical context. FCoE, which leverages the FC protocol, has an inherent awkwardness when applied to Ethernet networks that iSCSI does not. The iSCSI protocol, which originated from a traditional Ethernet and IP environment, was designed with those technologies in mind. (Figure 1)

Figure 1: Storage Protocols mapped to the OSI model

Fibre Channel is a layered protocol and may consist of 5 layers if all are implemented. iSCSI is also a layered protocol but folds more easily into the OSI model and has a determinate and consistent implementation. The iSCSI layer can be interpreted as an application protocol that exists over a native TCP and IP infrastructure. SCSI can then be thought of as a presentation layer as it adapts data to an iSCSI format and the iSCSI layer adapts SCSI messages to the TCP/IP protocols.

FCoE Needs DCB to Work

There’s been an unfortunate blurring of FCoE and the standards that attempt to evolve Ethernet fabrics to the point where they can better support a lossless network, namely, DCB (Data Center Bridging), DCE (Cisco’s version of DCB; Data Center Ethernet) or CEE (Brocade’s version of DCB; Converged Enhanced Ethernet). Data Center Bridging improves the Ethernet fabric irrespective of what protocol (be it iSCSI, NFS, TCP or FCoE) runs over the top of it. The important qualifier here is that FCoE won’t work at all without this new type of Ethernet. iSCSI works with any Ethernet but the same benefits that come from DCB may also accrue to iSCSI when running over a DCB network. You don’t need FCoE to have DCB but you MUST have DCB to support FCoE.

FCoE Lacks Native Routing

FCoE is deployed on a layer 2 Ethernet network and does not seamlessly work across WANs or IP subnets. Both FC and FCoE require the use of other companion protocols to connect multiple FC or FCoE SANs. FCIP (Fibre Channel over IP) and FC-IFR (Fibre Channel Inter-Fabric Routing) are two such protocols. FCIP is well defined and it tunnels FC over IP to connect two FC fabrics. However, it is not widely deployed in practice. FC-IFR is another protocol, currently under development.

FCoE is More Expensive

There is a fork-lift upgrade for FCoE – new switches, new adapters, new software and, at some point, new targets. See Figure 2.

Figure 2: FCoE fork-lift upgrade

Several additional cost components come into play with FCoE. FCoE switches are more costly due to additionally needed functionality and they require access to FC services, either co-located on the switch or accessed via a remote FC switch. The only reusable component is the FC target and possibly some FC switches.

FCoE Requires New Hardware

Figure 3: FCoE New Hardware transition

There are several components that need to be deployed for an FCoE SAN. This is necessary because FCoE is a new protocol that maps FC onto a non-FC link layer and that link-layer (Ethernet) does not have the same capabilities as the FC physical layer. See figure above.

The focus of the first version of the FCoE standards specification (FC-BB-5) was to allow FCoE initiators (FCoE enabled servers) to connect to existing FC SANs via Ethernet. This requires both DCB capabilities and FC services to be available on the Ethernet network. In the initial deployments, there is a single link hop on the Ethernet network. FCoE enabled servers run a new adapter called CNA (Converged Network Adapter). The CNA runs FCoE initiator functionality in the firmware or as a combination of software and firmware. The FCoE initiator connects to the edge switch via Ethernet. This edge switch is typically a Top of Rack (ToR) switch. It supports DCB functionality to ensure there are no packet drops on the CNA-to- switch link. The edge switch also contains FCF (FCoE Forwarder) functionality. i.e. it contains both DCB Ethernet and native Fibre Channel ports. The FC ports connect to an FC-SAN. Due to the above components such a single hop deployment requires new CNAs and switches (FCoE Forwarder). The Cisco Nexus 5000 is an example of such a switch. Since 10GBase-T is not widely available these deployments use SFP+ direct attach schemes for cabling.

As a next step in its evolution FCoE may evolve to multiple hops on a DCB network. As the DCB standards mature customers will have the opportunity to further reduce cabling complexity by converging Ethernet and FC cables from a ToR switch to a core switch. In this environment FCF functionality will move to the core network. The chassis switches in the core network will have Fibre Channel blades which will connect to an existing FC SAN. This will require new core switches. The Cisco Nexus 7000 is an example of such a switch.

As a final step in the evolution of FCoE native FCoE may become viable. In this environment, both FCoE initiators and FCoE targets will connect to a DCB enabled Ethernet network. It will not require FCoE Forwarder devices, but will require FC services to run on the Ethernet network. The standards work for this is currently in progress but it will require FC services and may require new capabilities on the DCB enabled edge switches.

iSCSI, on the other hand, does not necessitate the use of new hardware components. iSCSI can run over a standard Ethernet network. The server may run iSCSI stack in software (SW-iSCSI stack is part of OS) or may run it on an adapter (full iSCSI offload to hardware). However, if the underlying network infrastructures (CNA, switches) are DCB capable then iSCSI will take advantage of all the DCB capabilities. It will leverage the DCB layer 2 flow control (802.1Qbb) and will also be able to provide a guaranteed QoS (bandwidth reservation) using 803.1Qaz.

FCoE must overlay specific services, provided by new types of equipment, over a new kind of Ethernet, DCB. To say this is convergence stretches the point a bit as FCoE is an overlay network and only converges Ethernet to the extent that Ethernet frames are used in data transfers.

FCoE Still the New Kid on the Block

Figure 3: FCoE and iSCSI maturity timeline

The timeline indicates that FCoE is still an emerging technology with wider deployment expected around 2011. FCoE will take time to mature and be inter-operable due to its dependencies on multiple standard’s efforts. Expectations are for most IEEE DCB standards to be finished by April 2010. iSCSI was approved by IETF in 2002 and has a three to five year edge over FCoE in terms of maturity and use in deployed user networks.

Summary

A technology such as FCoE that attempts to bring simplification to the data center by converging on a single fabric, namely Ethernet, is an effort that we fully support. We support the effort, however, with our eyes fully open to the difficulties of porting the FC protocol to a completely different underlying fabric. The FC protocol was designed to be a reliable, lossless fabric from the ground up. The physical layers of FC were in large part responsible for its success. The migration of FC to Ethernet requires Ethernet to have same level of lossless behavior as native FC.

FCoE will require time for maturity and interoperability. FCoE depends on multiple standards activities (T11 FCOE, IEEE DCB) and all these standards need to be ratified and implemented to ensure inter-operability. Given enough money and time we believe all these concerns can be alleviated but this will not occur easily, cheaply or simply.

Convergence to a single Ethernet fabric really means that an overlaid FC protocol network rides over the top of native Ethernet. Convergence is, therefore, somewhat mythical at this point

iSCSI was built, from the ground up, to run on Ethernet. While it is true iSCSI doesn’t need an expensive Ethernet upgrade such as DCB (DCE, CEE) it is also true that a better experience occurs over DCB with iSCSI. iSCSI can take advantage of better Ethernet but doesn’t require it as FCoE does.

iSCSI has evolved to adapt to the advantages that 10Gbps Ethernet, protocol offload and TCP protocol efficiencies have provided. iSCSI today is not your Father’s iSCSI. It is a reliable, fast, native storage protocol that continues to improve and evolve.

Where FCoE and iSCSI are in 3 years, when FCoE reaches iSCSI’s current maturity level, will be critical to which storage fabric wins the data center. FCoE may be more mature but iSCSI won’t stand still and will continue to evolve into a superior storage protocol

The following post is written by Ujjwal Rajbhandari, from our Storage Product Marketing Group. 

There are number of discussions, blogs and articles comparing iSCSI, FCoE and FC. Many of them share a common belief that FCoE and FC are better suited as core datacenter SAN and iSCSI is ideal for Tier 2 storage or for SAN deployments in ROBO/SMB environment. That is because iSCSI is characterized as “low performing”, “lossy” and “unpredictable”. In this blog I will tackle the mis-information around iSCSI performance as compared to FC and FCoE. I will also compare effective efficiency of the various SAN protocols since efficiency is an aspect of performance

Both iSCSI and FCoE share the same 10Gb Ethernet at the transport layer. However, the perception is that the TCP/IP overhead makes iSCSI inefficient as compared to FCoE and FC (both having better payload to packet size ratio) thus leading to lower performance and efficiency. Figure 1 shows protocol efficiency calculation for iSCSI (both 1.5K MTU and 9K MTU), FC and FCoE (2.5K MTU). It can be seen that when jumbo frames are enabled, iSCSI has the best protocol efficiency.

Figure 1: Protocol efficiency comparisons

In regards to performance, iSCSI having low performance might have been true when 1Gbps was maximum throughput available per iSCSI port where as FC was delivering 2Gbps, 4Gbps and 8Gbps per port, but with the availability of 10GbE the commonly held belief that iSCSI performance is not being up-to-par as compared to FCoE or FC is no longer true.

The Office of CTO at Dell conducted a series of performance test to compare 10GbE iSCSI, FCoE and 4Gb FC. To ensure similar workload the application throughput was limited to 4Gb. The host adapters used for the different protocols were as follows:

1. 10GbE NIC with iSCSI offload for iSCSI traffic

2. 10GbE converged network adapter (CNA) for FCoE traffic

3. 4Gbps FC HBA for fibre channel traffic

The goal of the testing was to capture achieved throughput and CPU utilization for a given SAN protocol.

The protocol efficiency comparisons from Figure 1 might be theoretical in nature, Figure 2 shows results from an IO workload study compare throughput of 10GbE iSCSI, FCoE and 4Gb FC HBAs. To keep the results easy to visualize, the results show the throughput achieved when application generated 4Gb throughput. It can be clearly seen that iSCSI outperforms FCoE and FC regardless of read or write operation for various IO block size.

Figure 2: Throughput Performance Comparisons (MB/s)

Alongside capturing the throughput, let’s examine the host CPU utilization to better assess the performance and efficiency of specific SAN protocol. All the host adapters are comprised of hardware based offload capability to process the protocol specific traffic minimizing use of CPU resources. Figure 3 shows the effective CPU utilization for various workloads. It can be seen from the figure that all the host adapters have similar CPU utilization metrics, again reinforcing the fact that iSCSI is as efficient as FCoE and FC.

Finally, Figure 4 shows throughput efficiency, defined as MBps/%CPU, for the various storage protocols. The chart clearly shows 10GbE iSCSI having the best throughput efficiency across the workload types clearly outperforming FCoE and FC.

From the test results we can undoubtedly summarize that iSCSI as a SAN protocol is not “lower performing” or “inefficient” as compared to FC or FCoE. On the contrary, iSCSI outperforms both FC and FCoE. Customers who are planning to purchase storage for their datacenter can consider iSCSI SAN as a viable option knowing iSCSI performance is at par or even better than FCoE and FC. Also, customers considering unifying their datacenter network over Ethernet can start doing so now with iSCSI. While FCoE can also deliver storage traffic over Ethernet it is still under development and is not ready for prime-time.

Figure 3: CPU utilization (%) for iSCSI Offload, FCoE and FC

Figure 4: Overall protocol throughput efficiency (MBps/%CPU) for iSCSI Offload, FCoE and FC

Let’s start off by stating that Unified Fabric is not 10GbE alone.  Unified Fabric can be defined as a wire-once backbone that unifies computing and I/O resources.  And this includes the complete integration of 10GbE, iSCSI, and FCoE.  The main idea is that 10GbE can provide the infrastructure to run both iSCSI and FCoE.   

Both 10GbE and iSCSI are standards.  However, FCoE is still being finalized as a standard.  The missing piece for FCoE is DCB, DCE or CEE.  Regardless of the name, they all represent the same thing- flow control in the network.  So what options does one have while waiting for standardization?  The answer is quite simple- consolidate your Ethernet to 10GbE since it can support both iSCSI and FCoE.  And this will ease the transition towards unifying your data center fabric once the IEEE standards are defined.

On the storage side, on 8/25 Dell announced 10GbE support for the Dell/EMC CX4 series storage arrays (See Greg White’s blog posting) and has been demonstrating 10GbE on them at VMWorld 2009.  On the networking front, Dell has announced a new high-port count switch, a pass-thru blade I/O module and a mezzanine card that all support 10GbE.  All these enhancements will provide more bandwidth and more options for virtualized environments. Dell is also the only vendor providing support for advanced features enabled by VMware’s vStorage APIs for both the #1 iSCSI (Dell EqualLogic) and #1 Fibre Channel (CX4) SANs providing advanced vStorage APIs for Multipathing (MEM) support.  The MEM allows customers to intelligently use all paths between the SAN and vSphere 4.0 for improved scalability (think about 10GbE with the MEM!).  These are examples of what Dell is doing with 10GbE.  In addition, VMware vSphere features such as VMotion, storage VMotion, HA, FT, and DRS will benefit from the larger pipe.  The net effect makes the data center more efficient to deploy and manage.  Overall, 10GbE will be an important enabler of data center infrastructures and Dell is actively integrating it into our solutions, including the adoption of computing pods.  Computing pods are designed to offer IT organizations significantly enhanced data center capabilities, including efficient aggregation and networking, increased space savings, unified maintenance and training, and simplified infrastructure design and deployment. Redundant, high-bandwidth connectivity internal to blade enclosures within the pod helps speed communication and reduce congestion on the core data center network layer.  We see 10GbE as a key enabling technology to a more modular, standardized, building block approach to the data center.

Talk about incendiary... especially considering the instructor's audience: mostly Dell technical personnel. (MD3000i, EqualLogic ring a bell?) This statement was delivered, probably without much thought on how it would be received by the attendees, during a recent training session.  The course focused on products that play almost exclusively in a fiber channel SAN arena.  The teacher was "kinda like a nut living amongst squirrels."  More than once that day, a heated discussion flared on iSCSI vs fiber channel and eventually spanned to cover FCoE as well.

Naturally, I tweeted about the entertaining debate. The twitter response was equally interesting. It was as if I had mentioned Microsoft in a Linux forum. Ford vs. Chevy. You get the idea --it's a religion. I began to think about my last 10 years at another company. The kool-aid I was drinking was the color of orange wires.  The instructor's glasses, understandably, seem to be a little orange tinted as well. Its what he and his company grew up with. Over the next few days, I discussed this topic with several folks in and out of the DellTechCenter. I also read a number of related interesting blogs and forum posts. What did I learn?  Here's a quick summary of the various opinions I ran into this week:

Fiber Channel:
FC SANS are thought of for performance and reliability.  Management and implementation of FC SANS requires a specific knowledge set and can be viewed as expensive to get started.  Another common opinion was that customers with FC investments likely won't be quick to adopt or move to iSCSI. 

iSCSI:
iSCSI is generally viewed as easy and less expensive to implement from both a training and hardware perspective. It's likely the infrastructure skills required already exist within the companies evaluating the technologies.  Although, sometimes the intricacies of infrastructure configuration for best performance are somewhat glossed over.  The iSCSI SANS found their way into the smaller shops initially, but has quickly been moving up the food chain. The predominant opinion is that when 10 Gb Ethernet becomes widely accepted, so will iSCSI.

FCoE:
There were tentative opinions in the group on FCoE.  The common thread was that the cost for the needed additional hardware might hinder market acceptance.

Fortunately, I have multi-colored kool-aid these days. As a storage evangelist for DellTechCenter, I have the opportunity to evaluate and write about the various technologies. I previously did not have hands on experience with iSCSI.  Now that I've had a chance to connect a few servers to both EqualLogic and MD3000i arrays, I was truly surprised at how easy it was to get going. Factor in the ease of implementation, equivalent storage features (mpio, replication, snapshot, etc), scalability and cost of the solutions --the 'not enterprise' opinion seems a bit uninformed. I personally don't see it as an end to fiber though. In this industry these type of debates will always come up.  I'm sure there were token-ring proponents that had similar opinions on Ethernet's viability in the enterprise.