For those of you who would enjoy a little balance to all the FCoE buzz in the industry, let me introduce you to Robert Winter and Gaurav Chawla - two of Dell's strategic engineers in this space.
Read below for their take on the realities of FCoE. Thanks again, guys.
Robert Winter Gaurav Chawla
FCoE has been positioned by some as a future primary data center fabric and by others as simply a transitional technology. We believe that the ultimate position of FCoE will probably be somewhere between these two extremes. At this point, FCoE is not a full-formed solution and will probably require several years to achieve the same maturity level as iSCSI or FC.
The fact is that FCoE represents an attempt to completely detach an existing, purpose-built storage protocol, namely Fibre Channel (FC), and move it to a completely different physical fabric, Ethernet. To say that this migration is seamless ignores some very real practical challenges.
Let’s take a look at the technology facts.
The FCoE Storage Protocol and the OSI Model Are Not The Best of Friends
The OSI Layered Model (Open Systems Interconnect) is a well-known architectural abstraction that helps to describe the operation of protocols. In looking at FC, FCoE, and iSCSI, one consideration that mustn’t be lost is that Fibre Channel protocol layers cannot be mapped to OSI layers in a straight forward manner because the FC protocol originates from a much different historical context. FCoE, which leverages the FC protocol, has an inherent awkwardness when applied to Ethernet networks that iSCSI does not. The iSCSI protocol, which originated from a traditional Ethernet and IP environment, was designed with those technologies in mind. (Figure 1)
Figure 1: Storage Protocols mapped to the OSI model
Fibre Channel is a layered protocol and may consist of 5 layers if all are implemented. iSCSI is also a layered protocol but folds more easily into the OSI model and has a determinate and consistent implementation. The iSCSI layer can be interpreted as an application protocol that exists over a native TCP and IP infrastructure. SCSI can then be thought of as a presentation layer as it adapts data to an iSCSI format and the iSCSI layer adapts SCSI messages to the TCP/IP protocols.
FCoE Needs DCB to Work
There’s been an unfortunate blurring of FCoE and the standards that attempt to evolve Ethernet fabrics to the point where they can better support a lossless network, namely, DCB (Data Center Bridging), DCE (Cisco’s version of DCB; Data Center Ethernet) or CEE (Brocade’s version of DCB; Converged Enhanced Ethernet). Data Center Bridging improves the Ethernet fabric irrespective of what protocol (be it iSCSI, NFS, TCP or FCoE) runs over the top of it. The important qualifier here is that FCoE won’t work at all without this new type of Ethernet. iSCSI works with any Ethernet but the same benefits that come from DCB may also accrue to iSCSI when running over a DCB network. You don’t need FCoE to have DCB but you MUST have DCB to support FCoE.
FCoE Lacks Native Routing
FCoE is deployed on a layer 2 Ethernet network and does not seamlessly work across WANs or IP subnets. Both FC and FCoE require the use of other companion protocols to connect multiple FC or FCoE SANs. FCIP (Fibre Channel over IP) and FC-IFR (Fibre Channel Inter-Fabric Routing) are two such protocols. FCIP is well defined and it tunnels FC over IP to connect two FC fabrics. However, it is not widely deployed in practice. FC-IFR is another protocol, currently under development.
FCoE is More Expensive
There is a fork-lift upgrade for FCoE – new switches, new adapters, new software and, at some point, new targets. See Figure 2.
Figure 2: FCoE fork-lift upgrade
Several additional cost components come into play with FCoE. FCoE switches are more costly due to additionally needed functionality and they require access to FC services, either co-located on the switch or accessed via a remote FC switch. The only reusable component is the FC target and possibly some FC switches.
FCoE Requires New Hardware
Figure 3: FCoE New Hardware transition
There are several components that need to be deployed for an FCoE SAN. This is necessary because FCoE is a new protocol that maps FC onto a non-FC link layer and that link-layer (Ethernet) does not have the same capabilities as the FC physical layer. See figure above.
The focus of the first version of the FCoE standards specification (FC-BB-5) was to allow FCoE initiators (FCoE enabled servers) to connect to existing FC SANs via Ethernet. This requires both DCB capabilities and FC services to be available on the Ethernet network. In the initial deployments, there is a single link hop on the Ethernet network. FCoE enabled servers run a new adapter called CNA (Converged Network Adapter). The CNA runs FCoE initiator functionality in the firmware or as a combination of software and firmware. The FCoE initiator connects to the edge switch via Ethernet. This edge switch is typically a Top of Rack (ToR) switch. It supports DCB functionality to ensure there are no packet drops on the CNA-to- switch link. The edge switch also contains FCF (FCoE Forwarder) functionality. i.e. it contains both DCB Ethernet and native Fibre Channel ports. The FC ports connect to an FC-SAN. Due to the above components such a single hop deployment requires new CNAs and switches (FCoE Forwarder). The Cisco Nexus 5000 is an example of such a switch. Since 10GBase-T is not widely available these deployments use SFP+ direct attach schemes for cabling.
As a next step in its evolution FCoE may evolve to multiple hops on a DCB network. As the DCB standards mature customers will have the opportunity to further reduce cabling complexity by converging Ethernet and FC cables from a ToR switch to a core switch. In this environment FCF functionality will move to the core network. The chassis switches in the core network will have Fibre Channel blades which will connect to an existing FC SAN. This will require new core switches. The Cisco Nexus 7000 is an example of such a switch.
As a final step in the evolution of FCoE native FCoE may become viable. In this environment, both FCoE initiators and FCoE targets will connect to a DCB enabled Ethernet network. It will not require FCoE Forwarder devices, but will require FC services to run on the Ethernet network. The standards work for this is currently in progress but it will require FC services and may require new capabilities on the DCB enabled edge switches.
iSCSI, on the other hand, does not necessitate the use of new hardware components. iSCSI can run over a standard Ethernet network. The server may run iSCSI stack in software (SW-iSCSI stack is part of OS) or may run it on an adapter (full iSCSI offload to hardware). However, if the underlying network infrastructures (CNA, switches) are DCB capable then iSCSI will take advantage of all the DCB capabilities. It will leverage the DCB layer 2 flow control (802.1Qbb) and will also be able to provide a guaranteed QoS (bandwidth reservation) using 803.1Qaz.
FCoE must overlay specific services, provided by new types of equipment, over a new kind of Ethernet, DCB. To say this is convergence stretches the point a bit as FCoE is an overlay network and only converges Ethernet to the extent that Ethernet frames are used in data transfers.
FCoE Still the New Kid on the Block
Figure 3: FCoE and iSCSI maturity timeline
The timeline indicates that FCoE is still an emerging technology with wider deployment expected around 2011. FCoE will take time to mature and be inter-operable due to its dependencies on multiple standard’s efforts. Expectations are for most IEEE DCB standards to be finished by April 2010. iSCSI was approved by IETF in 2002 and has a three to five year edge over FCoE in terms of maturity and use in deployed user networks.
Summary
A technology such as FCoE that attempts to bring simplification to the data center by converging on a single fabric, namely Ethernet, is an effort that we fully support. We support the effort, however, with our eyes fully open to the difficulties of porting the FC protocol to a completely different underlying fabric. The FC protocol was designed to be a reliable, lossless fabric from the ground up. The physical layers of FC were in large part responsible for its success. The migration of FC to Ethernet requires Ethernet to have same level of lossless behavior as native FC.
FCoE will require time for maturity and interoperability. FCoE depends on multiple standards activities (T11 FCOE, IEEE DCB) and all these standards need to be ratified and implemented to ensure inter-operability. Given enough money and time we believe all these concerns can be alleviated but this will not occur easily, cheaply or simply.
Convergence to a single Ethernet fabric really means that an overlaid FC protocol network rides over the top of native Ethernet. Convergence is, therefore, somewhat mythical at this point
iSCSI was built, from the ground up, to run on Ethernet. While it is true iSCSI doesn’t need an expensive Ethernet upgrade such as DCB (DCE, CEE) it is also true that a better experience occurs over DCB with iSCSI. iSCSI can take advantage of better Ethernet but doesn’t require it as FCoE does.
iSCSI has evolved to adapt to the advantages that 10Gbps Ethernet, protocol offload and TCP protocol efficiencies have provided. iSCSI today is not your Father’s iSCSI. It is a reliable, fast, native storage protocol that continues to improve and evolve.
Where FCoE and iSCSI are in 3 years, when FCoE reaches iSCSI’s current maturity level, will be critical to which storage fabric wins the data center. FCoE may be more mature but iSCSI won’t stand still and will continue to evolve into a superior storage protocol