iscsi

Let’s start off by stating that Unified Fabric is not 10GbE alone.  Unified Fabric can be defined as a wire-once backbone that unifies computing and I/O resources.  And this includes the complete integration of 10GbE, iSCSI, and FCoE.  The main idea is that 10GbE can provide the infrastructure to run both iSCSI and FCoE.   

Both 10GbE and iSCSI are standards.  However, FCoE is still being finalized as a standard.  The missing piece for FCoE is DCB, DCE or CEE.  Regardless of the name, they all represent the same thing- flow control in the network.  So what options does one have while waiting for standardization?  The answer is quite simple- consolidate your Ethernet to 10GbE since it can support both iSCSI and FCoE.  And this will ease the transition towards unifying your data center fabric once the IEEE standards are defined.

On the storage side, on 8/25 Dell announced 10GbE support for the Dell/EMC CX4 series storage arrays (See Greg White’s blog posting) and has been demonstrating 10GbE on them at VMWorld 2009.  On the networking front, Dell has announced a new high-port count switch, a pass-thru blade I/O module and a mezzanine card that all support 10GbE.  All these enhancements will provide more bandwidth and more options for virtualized environments. Dell is also the only vendor providing support for advanced features enabled by VMware’s vStorage APIs for both the #1 iSCSI (Dell EqualLogic) and #1 Fibre Channel (CX4) SANs providing advanced vStorage APIs for Multipathing (MEM) support.  The MEM allows customers to intelligently use all paths between the SAN and vSphere 4.0 for improved scalability (think about 10GbE with the MEM!).  These are examples of what Dell is doing with 10GbE.  In addition, VMware vSphere features such as VMotion, storage VMotion, HA, FT, and DRS will benefit from the larger pipe.  The net effect makes the data center more efficient to deploy and manage.  Overall, 10GbE will be an important enabler of data center infrastructures and Dell is actively integrating it into our solutions, including the adoption of computing pods.  Computing pods are designed to offer IT organizations significantly enhanced data center capabilities, including efficient aggregation and networking, increased space savings, unified maintenance and training, and simplified infrastructure design and deployment. Redundant, high-bandwidth connectivity internal to blade enclosures within the pod helps speed communication and reduce congestion on the core data center network layer.  We see 10GbE as a key enabling technology to a more modular, standardized, building block approach to the data center.

Last week, I posted an article on what makes EqualLogic architecture different from other iSCSI/FC offerings and why the switch interlink bandwidth is important. In Part 2, the dramatic conclusion to the series, I'm focusing on cables and switch settings.

I had an opportunity to spend a few hours in the Dell Interoperability Lab last week.  We discussed what they were working on and afterwards was introduced to a neat little Chinese restaurant nearby with some excellent General Tso's... I suppose I should save my restaurant reviews for when I replace that guy on Man Vs. Food.  As I was saying, the team was busy installing things that I'm not at liberty to discuss yet; but let’s just say some of them had 10GbE ports. The 10GbE sparked an interesting conversation on cables. Arguably, with 1Gb networking, quality of cables became important in the datacenter. With 10GbE on the crest of becoming mainstream, it’s even more important now. The rigorous testing the interoperability team puts various components through tends to show that what you may think about cable quality and reliability isn't always true. They asked I not discuss the testing details as that information might be used to tune products to beat the tests.

My favorite quote of the day, in true lab-geek-guru fashion, was: "One word on making your own cables: Don't." I chuckled a bit at this statement, but it’s true. It's likely you may get most of the cables crimped right, but the ones that don't will cause you the most pain.  When 10Mbs and 100Mbs were mainstream, handmade wasn’t really an issue.  Starting with 1Gb and faster, it really can be a major factor.  Because TCP connections have error recovery, your marginal home-made cable might just impact performance, but not actually break completely.  In many cases, you won’t get any error messages on those packet retries, so just use quality cables.   Another insight from the team was that even if you aren't getting 10GbE anytime soon, start ordering the correct cables now.  If you have the choice when ordering, go ahead and get Cat 6 or 6a. The difference in cost is negligible and Cat 6/6a is ready for 10GbE.  Better yet, it's better insulated and provides better noise rejection.  Per one of the interoperability team members, this statement from Wikipedia (with a few edits) is valid:

10GBASE-T should work up to 55 m (180 ft) with current Cat 6 cabling. In order to allow deployment at the usual 100 m (328 ft), the standard uses a new partitioned Category 6a (a.k.a "augmented Cat6") cable specification, designed to reduce crosstalk between UTP cables, known as alien crosstalk. Some manufacturers of cable have released "Cat7" cable which is specifically marketed for 10GBASE-T compatibility and claims greater shielding beyond cat6a specifications. FYI the difference…cat6a is 550MHZ and cat 7 is 600MHZ. And though they SAY you can use cat 7 on 40GbaseT, nobody has as of yet, of course.  For now…the big deal is that a 7ft cat 6 cable is $5 and a cat-7 is $15 or more.  Talk is that cat-7e will be even better, and prices will go down in a few years.

Cable testing and qualification should be at the top of your list of considerations. Some basics from the lab: Make sure the cable is tested to the right standards. Try aiming for 55-micron gold plating as the better coating is more resistant to multiple plug ins and plug outs and will provide a clean and clear transmission. Just because a cable says it meets specs unfortunately doesn't mean they are good enough for your environment. Our testers said the only way to really know how a cable is to buy a couple and test them thoroughly with a QUALITY cable tester before you place the big order. The guys went on to say that most shops probably use a run of the mill tester from a local computer supply that may not be telling everything. In general they found they had to get at least +2 DB headroom over spec to be a truly robust cable. In fact, some of the best cables they have go over +17DB in headroom.   One of the geeks noted that he had a cable with 0 headroom, but did test out as a perfectly good CAT6 cable, and yet was causing hundreds of errors (retransmits of packets) at the switch port.  The cable was replaced with one that had +2.3 DB headroom, and the errors went away!   

I asked for some signs that might indicate the presence of suspect cables: As with anything in Tech, 'it depends' and can vary switch by switch, but they did say seeing a high number of transmission retries or intermittent LUN drops usually means its time to do some cable testing. By the way, the guys in the lab use a neat little cable tester, that costs about $10K! This is why they buy quality cables and STILL test them first, to insure that they are not going to find issues the hard way… when something breaks!

Where possible, jumbo frames (size set to 9000 or 9014 bytes depending on your NIC) and flow control should be enabled. Set them on your switch and server levels, and the EqualLogic arrays will adjust accordingly. Additional network considerations include minimizing switch hops and maximizing the bandwidth on the inter-switch links if present. Reducing the number of switch hops between the server(s) and the storage reduces the chances of storage traffic competing with other data traffic on congested interswitch links.

Also, a word on firmware of PowerConnect switches, since many Dell servers and storage products are sold with them -- schedule some time to update them to current firmware versions. After EqualLogic joined Dell, some significant changes were implemented that provide for more performance and reliability. In fact, in the lab it was pointed out the reliability testing of these switches was on par or actually better than many of the costlier solutions out there. 

Last week, I dropped in on our sales force at an internal training event to make sure they are telling their customers about the wonders of DellTechCenter. Once my duty as a booth-babe was over, I attended an interesting session on proper network design for EqualLogic SANs given by Marc Keating, Principal SE at Dell (and one of the waaaay back EqualLogic guys.) I went in thinking my lab setup was fine and came out thinking I need to make some serious changes! Although it's a lab and we have yet to have performance issues, I feel its my responsibility as a pocket protector carrying member of the technical workforce to correct this egregious slight against the capability of our EqualLogic PS Series arrays. Its akin to limiting a Ferrari to 70 MPH because that's the maximum speed on the roads around here.  What if I drive through West Texas where its 80 MPH or better yet get some track time over in College Station?  

Back to reality. I'll temporarily pause my dreaming of fast cars and finish my point. According to Marc, the network setup is probably the most important and likely least understood aspect of getting the much heralded performance out of an EqualLogic group. Most other iSCSI (and FC for that matter) products subscribe to the typical frame based multi-path failover concept, or 'hard path link' between components. You have one port on your adapter going to one switch and another port on another card going to another switch. Then each one of those switches in turn has a connection going to each controller on your iSCSI device. Should any one of those components fail (adapter, switch or controller), then the multi-pathing software makes a 'hard-path' switch to maintain connectivity. This is the way the world worked for me before I ran into EqualLogic SANs. The world was flat... But then, in the network session, I began to understand the advantages of the EqualLogic architecture; and why networking setup is important to take advantage of it all.

PS Series arrays optimize resources by automating volume and network load balancing. It is a frame-less architecture.  Instead of a single head unit with two controllers, there can be multiple arrays all in a single group.  Despite the multiple arrays, its still a single management interface, single discovery address for the clients, single... OK, you get the idea. Each array has two controllers. Data is intelligently managed across the entire group of arrays and the luns/volumes may span across any or all of the arrays in the group.  I/O is load balanced across the interfaces - and the assumption is made that your network is sound and has equivalent bandwidth regardless of the path data takes to and from the iSCSI initiators. Redundancy is inherent. If the path through one particular switch or controller or adapter goes down (or is overloaded ), it'll re-route automatically. 

Keating's presentation also pointed out that this breaks the typical 'hard path' scenario I described earlier because in that scenario the two switches typically do not have interlinks that can support this type of bandwidth. That's really the key, either use stacking or trunking (depending on your switch capability and size of your group) to make sure there is enough bandwidth between the switches. For a smaller environment (i.e. 2 switch environment) trunking is probably sufficient.  It's recommended that you have a trunk capacity equal to the network capacity of the arrays, although 70-80% is considered acceptable. 10-Gig links are most efficient if available. To maintain performance scalability with more than a few arrays (and/or switches) in the group, its likely you'll want to make sure your switches support stacking. As you grow and scale your datacenter environment, you may get to the point where stackable switches (even a couple of stacks that are trunked together) may not be enough.  At that point you will have to move towards some kind of chassis based switch.  The technical report titled "Dell EqualLogic PS Series Network Performance Guidelines", found at the EqualLogic support site (under resources/tech reports after you register and log in) covers the network requirements and recommendations in detail.

In part 2 I'll discuss some interesting insights from the Dell Interopability Lab into cables and specific switch settings. Now back to my regularly scheduled obsession with fast cars.

Talk about incendiary... especially considering the instructor's audience: mostly Dell technical personnel. (MD3000i, EqualLogic ring a bell?) This statement was delivered, probably without much thought on how it would be received by the attendees, during a recent training session.  The course focused on products that play almost exclusively in a fiber channel SAN arena.  The teacher was "kinda like a nut living amongst squirrels."  More than once that day, a heated discussion flared on iSCSI vs fiber channel and eventually spanned to cover FCoE as well.

Naturally, I tweeted about the entertaining debate. The twitter response was equally interesting. It was as if I had mentioned Microsoft in a Linux forum. Ford vs. Chevy. You get the idea --it's a religion. I began to think about my last 10 years at another company. The kool-aid I was drinking was the color of orange wires.  The instructor's glasses, understandably, seem to be a little orange tinted as well. Its what he and his company grew up with. Over the next few days, I discussed this topic with several folks in and out of the DellTechCenter. I also read a number of related interesting blogs and forum posts. What did I learn?  Here's a quick summary of the various opinions I ran into this week:

Fiber Channel:
FC SANS are thought of for performance and reliability.  Management and implementation of FC SANS requires a specific knowledge set and can be viewed as expensive to get started.  Another common opinion was that customers with FC investments likely won't be quick to adopt or move to iSCSI. 

iSCSI:
iSCSI is generally viewed as easy and less expensive to implement from both a training and hardware perspective. It's likely the infrastructure skills required already exist within the companies evaluating the technologies.  Although, sometimes the intricacies of infrastructure configuration for best performance are somewhat glossed over.  The iSCSI SANS found their way into the smaller shops initially, but has quickly been moving up the food chain. The predominant opinion is that when 10 Gb Ethernet becomes widely accepted, so will iSCSI.

FCoE:
There were tentative opinions in the group on FCoE.  The common thread was that the cost for the needed additional hardware might hinder market acceptance.

Fortunately, I have multi-colored kool-aid these days. As a storage evangelist for DellTechCenter, I have the opportunity to evaluate and write about the various technologies. I previously did not have hands on experience with iSCSI.  Now that I've had a chance to connect a few servers to both EqualLogic and MD3000i arrays, I was truly surprised at how easy it was to get going. Factor in the ease of implementation, equivalent storage features (mpio, replication, snapshot, etc), scalability and cost of the solutions --the 'not enterprise' opinion seems a bit uninformed. I personally don't see it as an end to fiber though. In this industry these type of debates will always come up.  I'm sure there were token-ring proponents that had similar opinions on Ethernet's viability in the enterprise.

Most heard phrase this week after my return from family excursion:  "Wow, new guy and already taking vacation." I'd feebly attempt to explain how I had purchased the cruise tickets well in advance of joining Dell; however the finger wagging didn't seem to let up much.  Oh well, last week's sunburns and margaritas were well worth the ribbing.

Speaking of vacation -just before you catch the yard on fire with bottle rockets and black cats, join us for a special special July 2 "TechThursday" Chat, iSCSI Security and Best Practices, at 3:00 PM CDT.  We'll have several KEY members of the Dell storage team on hand to field any questions or comments. 

The picture shows behind the scenes of our last chat:  Kong, Scott & I hard at work.